Advanced Storage Area Network Design
Edward Mazurek Technical Lead Data Center Storage Networking [email protected] @TheRealEdMaz
BRKSAN-2883 Agenda
• Introduction
• Technology Overview
• Design Principles
• Storage Fabric Design Considerations
• Data Center SAN Topologies
• Intelligent SAN Services
• Q&A
3 Introduction
6 An Era of Massive Data Growth Creating New Business Imperatives for IT
10X Increase in Data Produced (From 4.4T GB to 44T GB)
32B IoT Devices (Will be Connected to Internet) By 2020 40% of Data Will Be “Touched” by Cloud
85% of Data for Which Enterprises Will Have Liability and Responsibility
IDC April 2014: The Digital Universe of Opportunities: Rich Data and Increasing Value of Internet of Things
7 Evolution of Storage Networking…. Enterprise Apps: OLTP, VDI, etc. Big Data, Scale-Out NAS Cloud Storage (Object)
Compute Nodes
REST API Fabric
Fabric
Block and/or File Arrays
Multi-Protocol (FC, FICON, FCIP, FCoE, NAS, iSCSI, HTTP) Performance (16G FC, 10GE, 40GE, 100GE) Scale (Tens of Thousands P/V Devices, Billions of Objects) 8 Operational Simplicity (Automation, Self-Service Provisioning) Enterprise Flash Drives = More IO Significantly More IO/s per Drive at Much Lower Response Time
• Drive performance hasn’t 110 changed since 2003 (15K 100 SATA drives drives) 90 (8 drives) • Supports new application 80 100% Random Read Miss 8KB performance requirements 70 One Drive per DA Processor - 8 processors
60 • Price/performance making 15K rpm drives SSD more affordable 50 (8 drives) 40 • Solid state drives dramatically
Response Time Msec Time Response 30 Enterprise Flash increase IOPS that a given Drives (8 drives) 20 array can support 10 • Increased IO directly translates 0 to increased throughput 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 IOPs 9 Technology Overview
15 Fibre Channel – Foundations Based on SCSI
• Foundational protocol, forms the basis of Host Disk (Initiator) (Target) an I/O transaction
SCSI READ Operation • Communications are based upon Point to Point SCSI I/O Channel • Storage is accessed at a block-level via SCSI Host Disk (Initiator) (Target) • High-performance interconnect providing high I/O throughput SCSI WRITE Operation • SCSI I/O Channel The foundation for all block-based storage connectivity • Mature - SCSI-1 developed in 1986 16 Fibre Channel - Communications Point-to-point oriented Transmitter Receiver • Facilitated through device login N_port N_port Receiver Transmitter N_Port-to-N_Port connection Host Disk • Logical node connection point (Initiator) (Target) Flow controlled • Buffer-to-buffer credits and end-to-end basis Transmitter Receiver N_port Acknowledged Receiver Transmitter • For certain classes of traffic, none for others Host SAN (Initiator) (Switch) Multiple connections allowed per device
17 Fibre Channel Addressing Dual Port HBA Every Fibre Channel port and node has two 10:00:00:00:c9:6e:a8:16 64 bit hard-coded addresses called World 10:00:00:00:c9:6e:a8:17 Wide Names (WWN)
50:0a:09:83:9d:53:43:54 • NWWN(node) uniquely identify devices
• PWWN(port) uniquely identify each port in a device
• Allocated to manufacturer by IEEE Host Switch Disk phx2-9513# show int fc 1/1 • Coded into each device when manufactured fc1/1 is up Hardware is Fibre Channel, SFP is short wave laser Switch Name Server maps PWWN to FCID Port WWN is 20:01:00:05:9b:29:e8:80 4 bits 12 bits 24 bits 24 bits N-port or IEEE Organizational Unique ID 0002 Locally Assigned Identifier F_port Identifier (OUI) Format Identifier Port Identifier Assigned to each vendor Vendor-Unique Assignment 18 Port Initialization – FLOGI and PLOGIGIs/PLOGIs Target Step 1: Fabric Login (FLOGI) • Determines the presence or absence of a Fabric FC Fabric 3 • Exchanges Service Parameters with the Fabric • Switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID) • Initializes the buffer-to-buffer credits E_Port Step 2: Port Login (PLOGI) F_Port • Required between nodes that want to communicate 1 • Similar to FLOGI – Transports a PLOGI frame to the N_Port designation node port 2 • In P2P topology (no fabric present), initializes buffer- HBA to-buffer credits
Initiator
19 FC_ID Address Model
• FC_ID address models help speed up FC routing
• Switches assign FC_ID addresses to N_Ports
• Some addresses are reserved for fabric services
• Private loop devices only understand 8-bit address (0x0000xx)
• FL_Port can provide proxy service for public address translation
• Maximum switch domains = 239 (based on standard) 8 Bits 8 Bits 8 Bits Switch Area Device Switch Topology Model Domain Private Loop Device 00 00 Arbitrated Loop Address Model Physical Address (AL_PA) Switch Public Loop Device Area Arbitrated Loop Address Model Domain Physical Address (AL_PA)
20 FSPF Fabric Shortest Path First
• Provides routing services within any FC fabric
• Supports multipath routing
• Bases path status on a link state protocol similar to OSPF
• Routes hop by hop, based only on the domain ID
• Runs on E ports or TE ports and provides a loop free topology
• Runs on a per VSAN basis. Connectivity in a given VSAN in a fabric is guaranteed only for the switches configured in that VSAN.
• Uses a topology database to keep track of the state of the links on all switches in the fabric and associates a cost with each link
• Fibre Channel standard ANSI T11 FC-SW2
21 FSPF phx2-5548-3# show fsp database vsan 12
FSPF Link State Database for VSAN 12 Domain 0x02(2) LSR Type = 1 Advertising domain ID = 0x02(2) LSR Age = 1400 16G Number of links = 4 Port-Channel NbrDomainId IfIndex NbrIfIndex Link Type Cost ------0x01(1) 0x00010101 0x00010003 1 125 0x01(1) 0x00010100 0x00010002 1 125 0x03(3) 0x00040000 0x00040000 1 62 16G 0x03(3) 0x00040001 0x00040001 1 125
FSPF Link State Database for VSAN 12 Domain 0x03(3) 8G LSR Type = 1 Advertising domain ID = 0x03(3) LSR Age = 1486 9148-2 Number of links = 2 5548 NbrDomainId IfIndex NbrIfIndex Link Type Cost 1/1 ------2/13 0x02(2) 0x00040000 0x00040000 1 62 2/14 1 1/2 0x02(2) 0x00040001 0x00040001 1 125 phx2-5548-3# 2/15 1/3 D2 2/16 2 1/4 D3 22 Fibre Channel FC-2 Hierarchy
• Multiple exchanges are initiated between initiators (hosts) and targets (disks)
• Each exchange consists of one or more bidirectional sequences
• Each sequence consists of one or more frames
• For the SCSI3 ULP, each exchange maps to a SCSI command
OX_ID and RX_ID Exchange
SEQ_ID Sequence Sequence Sequence
SEQ_CNT Frame Frame Frame
Frame Fields ULP Information Unit
23 What Is FCoE? It’s Fibre Channel From a Fibre Channel standpoint it’s • FC connectivity over a new type of cable called… Ethernet From an Ethernet standpoints it’s • Yet another ULP (Upper Layer Protocol) to be transported
FC-4 ULP Mapping FC-4 ULP Mapping FC-3 Generic Services FC-3 Generic Services FC-2 Framing & Flow Control FC-2 Framing & Flow Control FC-1 Encoding FCoE Logical End Point FC-0 Physical Interface Ethernet Media Access Control Ethernet Physical Layer
24 Standards for FCoE FCoE is fully defined in FC-BB-5 standard FCoE works alongside additional technologies to make I/O Consolidation a reality
FCoE IEEE 802.1 T11 DCB
FC on FC on Other other Network network PFC ETS Media DCBX media
Lossless Priority Configuration Ethernet Grouping Verification
FC-BB-5 802.1Qbb 802.1Qaz 802.1Qaz
Technically stable October, 2008 Standard Sponsor Ballot July 2010 Sponsor Ballot October 2010 Sponsor Ballot October25 2010 Completed in June 2009 Published Fall 2011 Published Fall 2011 Published Fall 2011 Status Published in May, 2010 25 • VLAN Tag enables 8 priorities FCoE Flow Control for Ethernet traffic IEEE 802.1Qbb Priority Flow Control 3.3ms • PFC enables Flow Control on a Per-Priority basis using PAUSE frames (IEEE 802.1p) Resume • Receiving device/switch sends Pause frame when receiving buffer passes threshold
• Two types of pause frames • Quanta = 65535 = 3.3ms • Quanta = 0 = Immediate resume
• Distance support is determined FCoE by how much buffer is available to absorb data in Ethernet Wire flight after Pause frame sent
26 ETS: Enhanced Transmission Selection IEEE 802.1Qaz • Allows you to create priority groups
• Can guarantee bandwidth
• Can assign bandwidth percentages to groups
• Not all priorities need to be used or in groups
80%20% 80% 20% FCoE Ethernet Wire
27 FCoE Is Really Two Different Protocols FIP (FCoE Initialization Protocol) FCoE Itself
• It is the control plane protocol • Is the data plane protocol
• It is used to discover the FC entities • It is used to carry most of the FC connected to an Ethernet cloud frames and all the SCSI traffic
• It is also used to login to and logout • Ethertype 0x8906 from the FC fabric
• Uses unique BIA on CNA for MAC
• Ethertype 0x8914 The Two Protocols Have • Two different Ethertypes • Two different frame formats • Both are defined in FC-BB-5
28 http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html FPMA - Fabric Provided MAC Address
Fibre Channel over Ethernet Addressing Scheme Domain ID . FPMA assigned for each FCID FC Fabric Domain ID 11 . FPMA composed of a FC-MAP and FCID FCID Domain 11.00.01 . FC-MAP – Mapped Address Prefix ID 10 . the upper 24 bits of the FPMA FCID . FCID is the lower 24 bits of the FPMA 10.00.01 . FCoE forwarding decisions still made based on Fibre Channel FSPF and the FCID within the FPMA FCID Addressing
FC-MAP FC-ID (0E-FC-xx) 10.00.01
FC-MAP FC-ID FPMA (0E-FC-xx) 10.00.01
29 What is an FCoE Switch? • FCF (Fibre Channel Forwarder) accepts a Fibre Channel frame encapsulated in an Ethernet packet and forwards that packet over a VLAN across an Ethernet network to a remote FCoE end FCoE device Attached Storage • FCF is a logical FC switch inside an FCoE switch • Fibre Channel login happens at the FCF Nexus • Contains an FCF-MAC address FCF • Consumes a Domain ID
• FCoE encapsulation/decapsulation happens within the FCF Nexus FCoE MDS • NPV devices are not FCF’s and do not have domains FCF
FC
30 FCoE is Operationally Identical
• Supports both FC and FCoE
• FCoE is treated exactly the same as FC
• After zoning device perform registration and then performs discovery
phx2-9513# show fcns database vsan 42 VSAN 42: ------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE ------Which 0xac0600 N 50:0a:09:83:8d:53:43:54 (NetApp) scsi-fcp:target 0xac0700 N 50:0a:09:84:9d:53:43:54 (NetApp) scsi-fcp:target are 0xac0c00 N 20:41:54:7f:ee:07:9c:00 (Cisco) npv FCoE 0xac1800 N 10:00:00:00:c9:6e:b7:f0 scsi-fcp:init fc-gs 0xef0000 N 20:01:a0:36:9f:0d:eb:25 scsi-fcp:init fc-gs hosts?
33 After Link Is Up, Accessing Storage FIP and FCoE Login Process Target Step 1: FIP Discovery Process FC or FCoE • Enables FCoE adapters to discover which VLAN to Fabric transmit & receive FCoE frames • Enables FCoE adapters and FCoE switches to discover other FCoE capable devices • Occurs over Lossless Ethernet E_Ports or Step 2: FIP Login Process VE_Port • Similar to existing Fibre Channel Login (FLOGI) process – Sent to upstream FCF VF_Port • FCF assigns the host a FCID and FPMA to be used for FCoE forwarding VN_Port • Returns the FCID and the Fabric Provided MAC Address (FPMA) to the ENode FC-MAC CNA
ENode FIP Discovery 34 SCSI is the foundation for all Operating System SCSI SCSI SCSI SCSI SCSI
FCP FCP FCP i S C S I
FCP FCP FCP
FCIP
TCP TCP TCP F C o E IP IP
L o s s l e s s E t h e r n e t E t h e r n e t E t h e r n e t Physical Wire
35 Connectivity Types
FC FCoE
N F F N VN VF VF VN
Target Switch Initiator Target Switch Initiator
E E F NP VE VE VF VNP TE TE
Switch Switch Blade Server Switch Switch Blade Server Chassis Chassis
36 Fibre Channel Port Types Summary
Fibre Channel Switch
Fabric NPV E_Port F_Port NP_Port Switch E_Port Switch
Fabric TE_Port TE_Port VNP_Port NPV Switch VF_Port Switch
Fabric End VE_Port VE_Port F_Port N_Port Switch Node
G_Port End VF_Port VN_Port Node
37 The Story of Interface Speeds
Clocking Encoding Data Rate • Comparing speeds is more Protocol Gbps Data/Sent Gbps MB/s complex than just the 8G FC 8.500 8b/10b 6.8 850 “apparent” speed • Data throughput is based on 10G FC 10.51875 64b/66b 10.2 1,275 both the interface clocking
10G FCoE 10.3125 64b/66b 10.0 1,250 (how fast the interface transmits) and how efficient 16G FC 14.025 64b/66b 13.6 1,700 the interface transmits (how much encoding overhead) 32G FC 28.050 64b/66b 27.2 3,400
40G FCoE 41.250 64b/66b 40.0 5,000
38
38 Design Principles
39 VSANs Introduced in 2002 • A Virtual SAN (VSAN) Provides a Method to Allocate Ports within a Physical Fabric and Create Virtual Fabrics
• Analogous to VLANs in Ethernet Per Port Allocation • Virtual fabrics created from larger cost-effective redundant physical fabric
• Reduces wasted ports of a SAN island approach
• Fabric events are isolated per VSAN which gives further isolation for High Availability
• FC Features can be configured on a per VSAN basis.
• ANSI T.11 committee and is now part of Fibre Channel standards as Virtual Fabrics
40 VSAN • Assign ports to VSANs • Logically separate fabrics
• Hardware enforced
• Prevents fabric disruptions • RSCN sent within fabric only
• Each fabric service (zone server, name server, login server, etc.) operates independently in each VSAN
• Each VSAN is configured and managed independently
phx2-9513# show fspf vsan 43 vsan database FSPF routing for VSAN 43 vsan 2 interface fc1/1 FSPF routing administration status is enabled vsan 2 interface fc1/2 FSPF routing operational status is UP vsan 4 interface fc1/8 It is an intra-domain router vsan 4 interface fc1/9 Autonomous region is 0 MinLsArrival = 1000 msec , MinLsInterval = 2000 msec phx2-9513# show zoneset active vsan 43 Local Domain is 0xe6(230) zoneset name UCS-Fabric-B vsan 43 Number of LSRs = 3, Total Checksum = 0x00012848 zone name UCS-B-VMware-Netapp vsan 43 41 Zoning & VSANs 1. Assign physical ports to VSANs VSAN 2 2. Configure zones within each VSAN Disk2 Disk3 • A zone consists of multiple zone members Zone A Host1 Disk1 Zone C Zone B 3. Assign zones to zoneset Host2 Disk4 • Each VSAN has its own zoneset Zoneset 1 4. Activate zoneset in VSAN VSAN 3 • Members in a zone can access each other; Zone A Host4 members in different zones cannot access Zone B Host3 Disk5 each other Disk6 Zoneset 1 • Devices can belong to more than one zone 42 Zoning examples
• Non-zoned devices are members of zone name AS01_NetApp vsan 42 the default zone member pwwn 20:03:00:25:b5:0a:00:06 member pwwn 50:0a:09:84:9d:53:43:54 • A physical fabric can have a maximum of 16,000 zones (9700-only network) device-alias name AS01 • Attributes can include pWWN, FC pwwn 20:03:00:25:b5:0a:00:06 alias, FCID, FWWN, Switch Interface device-alias name NTAP fc x/y, Symbolic node name, Device member pwwn 50:0a:09:84:9d:53:43:54 zone name AS01_NetApp vsan 42 alias member device-alias AS01 member device-alias NTAP
43 The Trouble with sizable Zoning All Zone Members are Created Equal Number of ACLs . Standard zoning model just has “members” 10,000 . Any member can talk to any 8,000 other member 6,000 . Recommendation: 1-1 zoning 4,000 . Each pair consumes an ACL entry in TCAM 2,000
. Result: n*(n-1) entries 0
Number of ACL Entries ACL of Number
0
60 10 20 30 40 50 70 80 90 100 Number of Members
44 Smart Zoning Operation Today – 1:1Operation Zoning Today – ManyOperation - Many Smart Zoning 8 x I Zones Cmds ACLs Zones Cmds ACLs Zones Cmds ACLs 4 x T Create 32 96 Create64 1 13 Create132 1 13 64 zones(s) zones(s) zones(s) Add an +4 +12 Add +8an +1 Add+24 an +1 +8 initiator initiator initiator Add a +8 +24 Add+16 a +1 Add+24 a +1 +16 target target target
• Feature added in NX-OS 5.2(6)
• Allows storage admins to create larger zones while still keeping premise of single initiator & single target
• Dramatic reduction SAN administrative time for zoning
• Utility to convert existing zone or zoneset to Smart Zoning
45 How to enable Smart Zoning New Zone Existing Zone
46 Zoning Best Practices
• zone mode enhanced • Acquires lock on all switches while zoning changes are underway • Enables full zoneset distribution
• zone confirm-commit • Causes zoning changes to be displayed during zone commit
• zoneset overwrite-control – New in NX-OS 6.2(13) • Prevents a different zoneset than the currently activated zoneset from being inadvertently activated
Note: Above setting are per-VSAN
48 IVR - Inter-VSAN Routing
• Enables devices in different VSANs to VSAN 2 communicate Disk2 Disk3 • Allows selective routing between specific Zone A Host1 Disk1 Zone C members of two or Zone B Disk4 Host2 more VSANs • Traffic flow between selective devices Zoneset 1 • Resource sharing, i.e., tape libraries and VSAN 3 disks
Host4 Zone A • IVR Zoneset Zone B Host3 Disk5 • A collection of IVR zones that must be activated Disk6 to be operational Zoneset 1 49 Forward Error Correction - FEC
• Allows for the correction of some errors in frames
• Almost zero latency penalty 9710-2# show interface fc1/8 fc1/8 is trunking • Can prevent SCSI timeouts and aborts … Port mode is TE • Applies to MDS 9700 FC and MDS 9396S Port vsan is 1 Speed is 16 Gbps • Applies to 16G fixed speed FC ISLs only Rate mode is dedicated switchport speed 16000 Transmit B2B Credit is 500 Receive B2B Credit is 500 • Configured via: B2B State Change Number is 14 Receive data field Size is 2112 switchport fec tts Beacon is turned off admin fec state is up • No reason not to use it! oper fec state is up Trunk vsans (admin allowed and active) (1-2,20,237)
50 Trunking & Port Channels Trunking Single-link ISL or PortChannel ISL can be configured to become EISL – (TE_Port) Trunk
VSAN1 VSAN1 Traffic engineering with pruning VSANs on/off the trunk
VSAN2 VSAN2 Efficient use of ISL bandwidth VSAN3 VSAN3
TE Port TE Port Up to 16 links can be combined into a PortChannel increasing the aggregate bandwidth by distributing traffic granularly among all functional links in the channel Port Channel Port Load balances across multiple links and maintains optimum Channel TE Port TE Port bandwidth utilization. Load balancing is based on the source ID, destination ID, and exchange ID
If one link fails, traffic previously carried on this link is switched to the remaining links. To the upper protocol, the link is still there, although the bandwidth is diminished. The E Port E Port routing tables are not affected by link failure 51 N-Port Virtualization Scaling Fabrics with Stability • N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a “switch” to act like a server/HBA performing multiple fabric logins through a single physical link
• Physical servers connect to the NPV switch and login to the upstream NPIV core switch
• No local switching is done on an FC switch in NPV mode
• FC edge switch in NPV mode does not take up a domain ID phx2-9513 (config)# feature npiv • Helps to alleviate domain ID exhaustion in large fabrics NPV Switch FC NPIV Blade Server Core Switch F-Port Server1 F-Port FC1/1 N_Port_ID 1 Server2 FC1/2 N_Port_ID 2 NP-Port Server3 FC1/3 N_Port_ID 3 F_Port N-Port 52 Comparison Between NPIV and NPV
NPIV (N-Port ID Virtualization) NPV (N-Port Virtualizer) •Used by HBA and FC •Used by FC (MDS 9124, 9148, switches 9148S, etc.), FCOE switches (Nexus 5K), blade switches and •Enables multiple logins on a Cisco UCS Fabric InterConnects single interface (UCS6100) •Allows SAN to control and •Aggregate multiple physical/logical monitor virtual machines logins to the core switch (VMs) •Addresses the explosion of number of FC switches •Used for VMWare, MS Virtual Server and Linux Xen •Used for server consolidation applications applications
53 NPV Uplink Selection
NPV supports automatic selection of NP uplinks. When a server interface is brought up, the NP uplink interface with the minimum load is selected from the available NP uplinks in the same VSAN as the server interface.
When a new NP uplink interface becomes operational, the existing load is not redistributed automatically to include the newly available uplink. Server interfaces that become operational after the NP uplink can select the new NP uplink.
Manual method with NPV Traffic-Maps associates one or more NP uplink interfaces with a server interface.
Note: Use of parallel NPV links will pin traffic to one NPV link. Use of SAN Portchannels with NPV actual traffic will be load balanced.
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_0_1a/CLIConfigurationGuide/npv.html#wp1534672
54 NPV Uplink Selection – UCS Example
• NPV uplink selection can be automatic or manual
• With UCS autoselection, the vHBAs will be uniformly assigned to the available uplinks depending on the number of logins on each uplink Cisco UCS FC NPIV Blade Server NPV Switch NP-Port Core Switch F-Port
FC1/1 FC1/2 F_Port FC1/3 FC1/4 F_Port FC1/5 FC1/6
58 Uplink Port Failure
• Failure of an uplink moves pinned hosts from failed port to up port(s)
• Path selection is the same as when new hosts join NPV switch and pathing decision is made 2 devices re-login Cisco UCS FC NPIV Blade Server NPV Switch NP-Port Core Switch F-Port
FC1/1 FC1/2 F_Port FC1/3 Port is Down FC1/4 F_Port FC1/5 FC1/6
59 Uplink Port Recovery
• No automatic redistribution of hosts to recovered NP port
Cisco UCS FC NPIV Blade Server NPV Switch NP-Port Core Switch F-Port
FC1/1 FC1/2 F_Port FC1/3 Port is Up FC1/4 F_Port FC1/5 FC1/6
60 New F-Port Attached Host
• New host entering fabric is automatically pinned to recovered NP_Port
• Previously pinned hosts are still not automatically redistributed Cisco UCS FC NPIV Blade Server NPV Switch NP-Port Core Switch F-Port
FC1/1 FC1/2 F_Port FC1/3 FC1/4 F_Port FC1/5 FC1/6
61 New NP_Port & New F-Port Attached Host
• NPV continues to distribute new hosts joining fabric
Cisco UCS FC NPIV Blade Server NPV Switch NP-Port Core Switch F-Port
FC1/1 FC1/2 F_Port FC1/3 FC1/4 F_Port FC1/5 FC1/6 F_Port New Port Added
62 Auto-Load-Balance npv_switch(config)# npv auto-load-balance disruptive
This is Disruptive
Disruptive load balance works independent of automatic selection of interfaces and a configured traffic map of external interfaces. This feature forces reinitialization of the server interfaces to achieve load balance when this feature is enabled and whenever a new external interface comes up. To avoid flapping the server interfaces too often, enable this feature once and then disable it whenever the needed load balance is achieved.
If disruptive load balance is not enabled, you need to manually flap the server interface to move some of the load to a new external interface.
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/6_2/configuration/guides/interf aces/nx-os/cli_interfaces/npv.html#pgfId-1072790
63 F-Port Port Channel and F-Port Trunking Enhanced Blade Switch Resiliency F-Port Port Channel F-Port Port Channel w/ NPV .Bundle multiple ports in to 1 logical link F-Port Port Core Channel Director Storage Any port, any module
Blade N . High-Availability (HA)
System Blade 2 Blade Servers are transparent if a cable, port, or line
Blade 1 cards fails Blade . Traffic Management N-Ports F-Ports Higher aggregate bandwidth Hardware-based load balancing F-Port Trunking F-Port Trunking w/ NPV Core Director .Partition F-Port to carry traffic for multiple VSANs F-Port Trunking . Extend VSAN benefits to Blade VSAN1 Blade N Servers VSAN2 Blade 2 Separate management domains Blade 1 VSAN3 Blade System Blade Separate fault isolation domains N-Port F-Port Differentiated services: QoS, Security 64 Port Channeling & Trunking - Configuration
phx2-5548-3# show run interface san-port-channel 1 interface san-port-channel 1 switchport trunk allowed vsan 1 switchport trunk allowed vsan add 12 phx2-5548-3# show run interface fc 2/13-14 interface fc2/13 channel-group 1 force Nexus MDS no shutdown 5548 9148 fc2/13 fc1/1 interface fc2/14 fc2/14 1 fc1/2 channel-group 1 force no shutdown D2 D3
67 Port Channeling & Trunking - Configuration
phx2-9148-2# show run interface port-channel 1 interface port-channel1 switchport trunk allowed vsan 1 switchport trunk allowed vsan add 12 phx2-9148-2# show run interface fc1/1-2 interface fc1/1 channel-group 1 force Nexus MDS no shutdown 5548 9148 fc2/13 fc1/1 interface fc1/2 fc2/14 1 fc1/2 channel-group 1 force no shutdown D2 D3
68 Port Channel – Nexus switch config phx2-5548-3# show run int san-port-channel 3 Nexus 5548 interface san-port-channel 3 channel mode active switchport mode F switchport trunk allowed vsan 1 switchport trunk allowed vsan add 12 D2 phx2-5548-3# show run int fc 2/9-10 fc2/9 fc2/10 interface fc2/9 3 switchport mode F channel-group 3 force fc2/2 no shutdown fc2/1 interface fc2/10 switchport mode F Fabric channel-group 3 force no shutdown Interconnect
71 Port Channel – FI Config
5548
D2 fc2/9 fc2/10 3
fc2/1 fc2/2
Fabric Interconnect
72 FLOGI – Before Port Channel phx2-5548-3# show flogi database 5548 ------INTERFACE VSAN FCID PORT NAME NODE NAME ------fc2/9 12 0x020000 20:41:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/9 12 0x020001 20:02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 fc2/9 12 0x020002 20:02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 D2 fc2/9 12 0x020003 20:02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 fc2/10 12 0x020020 20:42:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/9 fc2/10 fc2/10 12 0x020021 20:02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 fc2/10 12 0x020022 20:02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00
Total number of flogi = 7 fc2/1 fc2/2 phx2-5548-3#
Fabric Interconnect
73 FLOGI- After port channel
phx2-5548-3# show flogi database 5548 ------INTERFACE VSAN FCID PORT NAME NODE NAME ------San-po3 12 0x020040 24:0c:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 San-po3 12 0x020001 20:02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 San-po3 12 0x020002 20:02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 D2 San-po3 12 0x020003 20:02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 San-po3 12 0x020021 20:02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 2/9 2/10 San-po3 12 0x020022 20:02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00
Total number of flogi = 6 2/2 phx2-5548-3# 2/1
Fabric Interconnect
74 Port-channel design considerations
• All types of switches • Name port-channels the same on both sides • Common port allocation in both fabrics • ISL speeds should be >= edge device speeds • Maximum 16 members per port-channel allowed • Multiple port-channels to same adjacent switch should be equal cost • Member of VSAN 1 + trunk other VSANs • Check TCAM usage: • show system internal acl tcam-usage
75 port-channel design considerations
• Director class • Split port-channel members across multiple line cards • When possible use same port on each LC: • Ex. fc1/5, fc2/5, fc3/5, fc4/5, etc. • If multiple members per linecard distribute across port-groups • show port-resources module x
76 Port-channel design considerations
• Fabric switches • Ensure enough credits for distance • Can “rob” buffers from other ports in port-group that are “out-of-service” • Split port-channel member across different forwarding engines to distribute ACLTCAM • For F port-channels to NPV switches (like UCS FIs) • Each device’s zoning ACLTCAM programming will be repeated on each PC member • For E port-channels using IVR • Each host/target session that gets translated will take up ACLTCAM on each member • Use following table: • Ex. On a 9148S a six member port-channel could be allocated across the 3 fwd engines as follows: • fc1/1, fc1/2, fc1/17, fc1/18, fc1/33 and fc1/34
• Consider MDS 9396S for larger scale deployments
77 F port-channel design considerations Ports are allocated to fwd-engines according the following table:
Fwd Zoning Region Bottom Region Switch/Module Port Range(s) Fwd-Eng Number Engines Entries Entries MDS 9148 3 fc1/25-36 & fc1/45-48 1 2852 407 fc1/5-12 & fc1/37-44 2 2852 407 1-4 & 13-24 3 2852 407 MDS 9250i 4 fc1/5-12 & eth1/1-8 1 2852 407 fc1/1-4 & fc1/13-20 & 2 2852 407 fc1/37-40 fc1/21-36 3 2852 407 ips1/1-2 4 2852 407 MDS 9148S 3 1-16 1 2852 407 17-32 2 2852 407 33-48 3 2852 407
78 F port-channel design considerations
Fwd Zoning Region Bottom Region Switch/Module Port Range(s) Fwd-Eng Number Engines Entries Entries MDS 9396S 12 1-8 0 49136 19664 9-16 1 49136 19664 17-24 2 49136 19664 25-32 3 49136 19664 33-40 4 49136 19664 41-48 5 49136 19664 49-56 6 49136 19664 57-64 7 49136 19664 65-72 8 49136 19664 73-80 9 49136 19664 81-88 10 49136 19664 89-96 11 49136 19664
79 F port-channel design considerations
Fwd Zoning Region Bottom Region Switch/Module Port Range(s) Fwd-Eng Number Engines Entries Entries DS-X9248-48K9 1 1-48 0 27168 2680 DS-X9248-96K9 2 1-24 0 27168 2680 25-48 1 27168 2680 DS-X9224-96K9 2 1-12 0 27168 2680 13-24 1 27168 2680 DS-X9232-256K9 4 1-8 0 49136 19664 9-16 1 49136 19664 17-24 2 49136 19664 25-32 3 49136 19664 DS-X9248-256K9 4 1-12 0 49136 19664 13-24 1 49136 19664 25-36 2 49136 19664 37-48 3 49136 19664
80 F port-channel design considerations
Fwd Zoning Region Bottom Region Switch/Module Port Range(s) Fwd-Eng Number Engines Entries Entries DS-X9448-768K9 6 1-8 0 49136 19664 9-16 1 49136 19664 17-24 2 49136 19664 25-32 3 49136 19664 33-40 4 49136 19664 41-48 5 49136 19664
81 Internal CRC handling
• New feature to handle frames internally corrupted due to bad HW
• Frames that are received corrupted are dropped at the ingress port • These frames are not included in this feature
• In rare cases frames can get corrupted internally due to bad hardware • These are then dropped • Sometimes difficult to detect
• New feature detects the condition and isolates hardware
• 5 possible stages where frames can get corrupted
82 Internal CRC handling
• Stages of Internal CRC Detection and Isolation
The five possible stages at which internal CRC errors may occur in a switch: 1. Ingress buffer of a module 2. Ingress crossbar of a module 3. Crossbar of a fabric module 4. Egress crossbar of a module 5. Egress buffer of a module
83 Internal CRC handling
• The modules that support this functionality are: • Cisco MDS 9700 48-Port 16-Gbps Fibre Channel Switching Module • Cisco MDS 9700 48-Port 10-Gbps Fibre Channel over Ethernet Switching Module • Cisco MDS 9700 Fabric Module 1 • Cisco MDS 9700 Supervisors
• Enabled via the following configuration command: • hardware fabric crc threshold 1-100
• When detected failing module is powered down
• New in NX-OS 6.2(13)
84 Device-alias
• device-alias(DA) is a way of naming PWWNs
• DAs are distributed on a fabric basis via CFS
• device-alias database is independent of VSANs • If a device is moved from one VSAN to another no DA changes are needed
• device-alias can run in two modes: • Basic – device-alias names can be used but PWWNs are substituted in config • Enhanced – device-alias names exist in configuration natively – Allows rename without zoneset re-activations
• device-alias are used in zoning, IVR zoning and port-security
• copy running-config startup-config fabric after making changes!
85 Device-alias
• device-alias confirm-commit • Displays the changes and prompts for confirmation
MDS9710-2(config)# device-alias confirm-commit enable MDS9710-2(config)# device-alias database MDS9710-2(config-device-alias-db)# device-alias name edm pwwn 1000000011111111 MDS9710-2(config-device-alias-db)# device-alias commit The following device-alias changes are about to be committed + device-alias name edm pwwn 10:00:00:00:11:11:11:11 Do you want to continue? (y/n) [n]
86 Device-alias
• Note: To prevent problems the same device-alias is only allowed once per commit.
• Example: MDS9148s-1(config)# device-alias database MDS9148s-1(config-device-alias-db)# device-alias name test pwwn 1122334455667788 MDS9148s-1(config-device-alias-db)# device-alias rename test test1 Command rejected. Device-alias reused in current session :test Please use 'show device-alias session rejected' to display the rejected set of commands and for the device-alias best-practices recommendation.
87 Cisco Prime Data Center Network Manager Feature Support and User Interface VMpath Analysis provides VM connectivity to network and storage across Unified Compute and Unified Fabric
• Visibility past physical access (switch) layer
• Standard & Custom Reports
• On Nexus and MDS platforms
• Dynamic Topology Views
• Rule-based event filtering and forwarding
• Threshold Alerting
• Integration via vCenter API
88 SAN Design Security Challenges SAN design security is often overlooked as an area of concern • Application integrity and security is addressed, but not back-end storage network carrying actual data • SAN extension solutions now push SANs outside datacenter boundaries Not all compromises are intentional FC • Accidental breaches can still have the same consequences SAN design security is only one part of complete data center solution • Host access security—one-time passwords, auditing, VPNs • Storage security—data-at-rest encryption, LUN security Theft External DOS Privilege Escalation/ Unintended Privilege or Other Unauthorized Data Intrusion Application Connections Tampering Tampering (Internal) (Trojans, etc.) SAN
LAN 89 SAN Security Device/SAN Secure management access Management Security Via SSH, • Role-based access control SFTP, SNMPv3, and • CLI, SNMP, and web access User Roles RADIUS or TACACS+ or LDAP Secure management protocols Server for Authentication • SSH, SFTP, and SNMPv3 Secure switch control protocols • TrustSec SAN • FC-SP (DH-CHAP) Protocol Security AAA - RADIUS, TACACS+ and LDAP (FC-SP) iSCSI- • User, switch and iSCSI host authentication VSANs Attached Provide Servers Fabric Binding Secure Hardware-Based Isolation • Prevent unauthorized switches from joining the Zoning Via Port and fabric WWN
Shared Physical Storage 90 Slow Drain Slow Drain Device Detection and Congestion Avoidance
• Devices can impart slowness in a fabric
• Feature of the fabric that’ll expose that device for remediation
• BRKSAN-3446 SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric White paper (2013) http://www.cisco.com/en/US/prod/collateral/ps4159/ps6409/ps12970/white_paper_c11-729444.pdf
91 Storage Fabric Topology Considerations
92 The Importance of “Architecture”
SAN designs traditionally robust: Latency dual fabrics, data loss is not • Initiator to target tolerated • Slow drain Must manage ratios • Performance under load: does my fabric perform the same • Fan in/out • ISL oversubscription Application independence • Virtualized storage IO streams • Consistent fabric performance (NPIV attached devices, server regardless of changes to SCSI profile RDM, LPARs, etc.) • Number of frames • Frame size • Queue depth • Speed or throughput
93 SAN Major Design Factors High Performance Port density Crossbar • How many now, how many later? 2 • Topology to accommodate port requirements Large Port QoS, Count Network performance Congestion Directors Control, • What is acceptable? Unavoidable? Reduce FSPF 3 Routes 1 Traffic management 8 8 8 8 8 8 8 8 8 8 8 8 • Preferential routing or resource allocation Fault isolation • Consolidation while maintaining isolation Management • Secure, simplified management 4 Failure of One Device Has No Impact on Others 94
94 Scalability—Port Density Topology Requirements
Considerations
• Number of ports for end devices Large Port • How many ports are needed now? Count Directors • What is the expected life of the SAN?
• How many will be needed in the future? 8 8 8 8 8 8 8 8 8 8 8 8 • Hierarchical SAN design Best Practice
• Design to cater for future requirements
• Doesn’t imply “build it all now,” but means “cater for it” and avoids costly retrofits tomorrow
95 Scalability—Port Density – MDS Switch selection
• MDS 9148S – 48 ports 16G FC
• MDS 9250i – 40 ports 16G FC + 8 port 10G FCoE + 2 FCIP ports
• MDS 9396S – 96 ports 16G FC
• MDS 9706 – Up to 192 ports 16G FC and/or 10G FCoE and/or 40G FCoE
• MDS 9710 – Up to 384ports 16G FC and/or 10G FCoE and/or 40G FCoE
• MDS 9718 – Up to 768 ports 16G FC and/or 10G FCoE and/or 40G FCoE
• All MDS 97xx chassis are 32G ready!
• All 16G MDS platforms are full line rate
96 Scalability—Port Density – Nexus Switch selection
• Nexus 55xx – Up to 96 ports 10G FCoE and/or 8G FC ports
• Nexus 5672UP – Up to 48 10G FCoE and/or 16 8G FC ports
• Nexus 5672UP-16G – Up to 48 10G FCoE and/or 24 16G FC ports
• Nexus5624Q – 12 ports 40G or 48 ports 10G FCoE
• Nexus5648Q – 24 ports 40G or 96 ports 10G FCoE
• Nexus5696Q – Up to 32 ports 100G / 96 ports 40G / 384 ports 10G FCoE or 60 8G FC
• Nexus 56128P – Up to 96 10G FCoE and/or 48 8G FC ports
• All Nexus platforms are full line rate
97 Traffic Management Do different apps/servers have different performance requirements? • Should bandwidth be reserved for specific applications? QoS, Congestion • Is preferential treatment/ Control, QoS necessary? Reduce FSPF Routes
8 8 8 8 8 8 8 8 8 8 8 8 Given two alternate paths for traffic between data centers, should traffic use one path in preference to the other? • Preferential routes
98 Network Performance Oversubscription Design Considerations All SAN Designs Have Some Degree of Oversubscription
• Without oversubscription, SANs would Tape Oversubscription be too costly Disk Oversubscription Disk do not sustain wire-rate I/O Need to sustain close to with ‘realistic’ I/O mixtures maximum data rate • Oversubscription is introduced at Vendors may recommend a 6:1 to LTO-6 Native Transfer multiple points as high as 20:1 host to disk Rate ~ 160 MBps fan-out ratio • Switches are rarely the bottleneck Highly application dependent in SAN implementations
• Device capabilities (peak and sustained) 8 8 8 8 8 8 Port Channels must be considered along with network Help Reduce oversubscription Oversubscription While Maintaining HA Requirements • Must consider oversubscription during a network failure event ISL Oversubscription Two-tier design ratio less than fan-out ratio Host Oversubscription Largest variance observed at this level. DB servers close to line rate, others highly oversubscribed 16Gb line cards non-oversubscribed
99 Fault Isolation Consolidation of Storage
• Single Fabric = Increased Storage Utilization + Reduced Administration Overhead Major Drawback
• Faults Are No Longer Isolated Physical SAN Islands Are Virtualized onto Common • Technologies such as VSANs enable consolidation SAN Infrastructure and scalability while maintaining security and stability
• VSANs constrain fault impacts
Fabric • Faults in one virtual fabric (VSAN) are contained #3 and do not impact other virtual fabrics Fabric #1 Fabric #2
100 Data Center SAN Topologies
101 Denser Server Cabinets What are the implications? Uplinks change from 40 GE servers Vertical Horizontal to 4x 10G servers Cabling Cabling EoR X-Connect ToR Main X-Connect
DC Infrastructure Changes Denser: cabinets, cross-connects cable runs From 42U to ~58U Horizontal Cabling: from 10G, through 40G to 100G – longer distances Vertical Cable: match appropriate server connectivity choice Is SAN EoR economical now? 102 Structured Cabling Supporting new EoR & ToR designs
• Pricing advantage for manufactured cabling systems
• Removes guessing game of how many strands to pull per cabinet
• Growth at 6 or 12 LC ports per cassette
• Fiber-only cable plant designs possible
103 Core-Edge Highly Scalable Network Design
• Traditional SAN design for growing SANs
• High density directors in core and fabric switches, directors or blade switches on edge
• Predictable performance
• Scalable growth up to core and ISL capacity
• Evolves to support EoR & ToR
End of Row Top of Rack Blade Server • MDS 9718 as core
105 Large Edge-Core-Edge/End-of-Row Design “A” Large Edge/Core/Edge Fabric (2496 End Device Ports per Fabric) 240 Storage ports at 16Gb Shown, • Traditional Edge-Core-Edge design Is ideal for (optional 480 @ 8Gb without Repeat very large centralized services and consistent changing bandwidth ratios) host-disk performance regardless of location for “B” Fabric • Full line rate ports, no fabric oversubscription • 8Gb or 16Gb hosts and targets 240 ISLs from storage edge • Services consolidated in the core 120
to core @ 16Gb MDS9710 • Easy expansion
Ports Deployed 3456 per fabric 6,912 total
Used Ports 5,760 total @ 16Gb 240 ISLs from host 6,240 total @ 8Gb edge to core @ 16Gb 24 Storage Ports 480 total @ 16Gb, or 960 total @ 8Gb
Host Ports 3360 total 1680 hosts @ 8Gb or 16Gb ISL ports 960 total
Host ISL Oversubscription 7:1 @ 16Gb
End to End 7:1 @ 16Gb storage Oversubscription 7:1 @ 8Gb storage
106 Very Large Edge-Core/End-of-Row Design “A” Fabric Shown, Very Large Edge/Core/Edge 576(288 per switch) Repeat for “B” Fabric (6144 End Device Ports per Fabric) Storage ports at 16Gb MDS 9718 • Traditional Core-Edge design Is ideal for very large centralized services and consistent host- disk performance regardless of location • Full line rate ports, no fabric oversubscription • 16Gb hosts and targets • Services consolidated in the core • Easy expansion 768(48 per switch) 24 Ports Deployed 12,288 ISLs from host edge to core @ 16Gb Used Ports 10,368 @ 16Gb
Storage Ports 1152 @ 16Gb MDS 9710
Host Ports 8064 4032 (252 per ISL ports 768 switch) hosts @ 8Gb Host ISL Oversubscription 7:1 @ 16Gb or 16Gb
End to End Oversubscription 7:1 @ 16Gb storage
107 SAN Top of Rack – MDS 9148S SAN Top of Rack (5,376 Usable Ports) 352 Storage ports at 16Gb • Ideal for centralized services while reducing cabling MDS 9710 requirements • Consistent host/target performance regardless of location in rack • 8Gb hosts & 16Gb targets
• Easy edge expansion A B • Massive cabling infrastructure avoided as compared to EoR designs 4 ISLs from each edge to core @ 16Gb • Additional efficiencies with in rack IO convergence MDS 9148S Ports Deployed 5,376 4,224 hosts @ 16Gb Used Ports 5,344
Storage Ports 352 @ 16Gb
Host Ports 4,224
Host ISL Oversubscription 12:1 @ 16Gb 48 Racks End to End Oversubscription 12:1 @ 16G hosts 44 Dual-attached servers per rack 108 Rack Top-of-Rack Design - Blade Centers SAN Top of Rack – Blade Centers (1,920 Usable Ports per Fabric) 96 Storage ports • Ideal for centralized services at 16Gb MDS 9710 • Consistent host/target performance regardless of location in blade enclosure or rack • 8Gb hosts & 16Gb targets • Need to manage more SAN Edge switches/Blade Switches • NPV attachment reduces fabric complexity A B
• Assumes little east-west SAN traffic 8 ISLs from each edge to core @ 8Gb • Add blade server ISLs to reduce fabric oversubscription Blade Center Ports Deployed 1,920
Used Ports 192 @ 16Gb 960 hosts @ 8Gb 1056 @ 8Gb
Storage Ports 192 @ 16Gb, or 192 @ 8Gb
Host Ports 2304
Host ISL Oversubscription 4:1 @ 8G 12 Racks, 72 chassis 96 Dual-attached blade servers per rack End to End Oversubscription 6:1 @ 16Gb Storage 12:1 @ 8Gb Storage Rack 109 Medium Scale Dual Fabric Collapsed Core Design “A” Fabric Shown, Medium Scale Dual Fabric Repeat for “B” Fabric (768 Usable Ports per Fabric) 96 Storage ports at 16Gb MDS 9710 • Ideal for centralized services • Consistent host/target performance regardless of location • 8Gb or 16Gb hosts & targets (if they exist) • Relatively easy edge expansion to Core/Edge • EoR design • Supports blade centers connectivity
Ports Deployed 768
Used Ports 768@ 16Gb
Storage Ports 96 @ 16Gb 672 hosts @ 16Gb
Host Ports 672 @ 16Gb
Host ISL Oversubscription N/A
End to End Oversubscription 7:1 @ 16Gb
110 POD SAN Design POD SAN Design Ideal for centralized services 36-48 Storage • Consistent host/target performance regardless of ports at 16Gb location in blade enclosure or rack • 10/16Gb hosts & 16Gb targets • Need to manage more SAN Edge switches/Blade MDS 9396S MDS 9396S Switches
• NPV attachment reduces fabric complexity A B • Add blade server ISLs to reduce fabric 6 ISLs from each edge 8 ISLs from each edge oversubscription to core @ 16Gb to core @ 8Gb
MDS 9148S UCS FI 6248UP
6 Racks, 252 chassis 6 Racks, 288 blades 42 Dual-attached servers 48 Dual-attached blade per rack servers per rack
252 hosts @ 16Gb or 288 hosts @ 10Gb 111 FI 6332-16UP, FI 6332 UCS SAN Design
FI 6332-16UP Use Case FI 6332 Use Case
40G 40G Nexus Nexus 7K/9K 7K/9K 16G FC 40G FCoE
FI 6332-16UP FI 6332 40G 40G 40G 40G Storage Storage MDS Array MDS Array UCS UCS 9700 UCS UCS 9700 B-Series C-Series B-Series C-Series B200 C220 B200 C220 B260 C240 B260 C240 B460 C460 B460 C460 and and IOM 2304 IOM 2304
40G 40G 16G FC 40G FCoE
112 Intelligent SAN Services
113 Enhancing SAN Design with Services
Extend Fabrics • FCIP • Extended Buffer to Buffer credits • Encrypt the pipe
SAN Services extend the effective distance for remote applications • SAN IO acceleration • Write acceleration SAN Extension with FCIP • Tape acceleration
Enhance array replication requirements
Reduces WAN-induced latency
Improves application performance over distance Data Migration with IO Acceleration Data Migration DMM
Fabric is aware of all data frames from initiator to target 114 SAN Extension with FCIP Fibre Channel over IP
• Encapsulation of Fibre Channel frames into IP packets and tunneling through an existing TCP/IP network infrastructure, in order to connect geographically distant islands
• Write Acceleration to improve throughput and latency
• Hardware-based compression
• Hardware-based IPSec encryption
Array to Array Replication
FCIP Tunnel TE Port 115 FC Redirect - How IOA Works Replication Replication Starts Starts
Initiator to Flow redirected to Flow accelerated and target IOA Engine sent towards normal routing path
IOA IOA
MAN/WAN
IOA IOA
IOA= I/O Accelerator
Initiator Target Initiator Target
Virtual Initiator Virtual Target 116 Data Acceleration A fabric service to accelerate I/O between SAN devices
• Accelerate SCSI I/O Over both Fibre Channel (FC) and Fibre Channel over IP (FCIP) links For both Write Acceleration (WA) and Tape Acceleration (TA)
• I/O Acceleration Node platforms: MSM-18/4, SSN-16, MDS-9222i, MDS-9250i
• Uses FC Redirect
IOA IOA
MAN/WAN
IOA IOA
IOA= I/O Accelerator
117 IOA FCIP Tape Backup Large Health Insurance Firm MDS IOA Results
92% throughputFCIP increase
. Highly resilient– Clustering of IOA engines allows for load balancing and failover
. Improved Scalability- Scale without increasing management overhead
. Significant reutilization of existing infrastructure- All chassis and common equipment re-utilized
. Flat VSAN topology- Simple capacity and availability planning
118 SAN Extension – FC over long distance BB_Credits and Distance ~1 km per Frame 2 Gbps FC
~0.5 km per Frame 4 Gbps FC
8 Gbps FC ~0.25 km per Frame
16 Gbps FC ~0.125 km per Frame
16 Km
phx2-9513(config)# feature fcrxbbcredit extended • BB_Credits are used to ensure enough FC frames in flight phx2-9513(config)# interface 1/1 phx2-9513(config-if)# switchport fcrxbbcredit extended 1000 • A full (2112 byte) FC frame is approx 1 km long @ 2 Gbps, phx2-9513# show interface 1/1 ½ km long @ 4 Gbps ¼ km long at 8 Gbps fc1/1 is up ….. • As distance increases, the number of available BB_Credits Transmit B2B Credit is 128 need to increase as well Receive B2B Credit is 1000 • Insufficient BB_Credits will throttle performance - no data will be transmitted until R_RDY is returned 119 SAN Extension – FCoE over long distance FCoE Flow Control For long distance FCoE, receiving switch Ingress Buffer must be large enough to absorb all packets in flight from the time the Pause frame is sent to the to time the Pause Frame is received Buffer Threshold • A 10GE, 50 km link can hold ~300 frames • That means 600+ frames could be either in flight or will be transmitted by the time the receiver detects buffer congestion and sends a Pause frame to the time the Pause frame is received and the sender stops transmitting
Egress Buffer Frame Latency Buffer Frame
Frame Frame Pause threshold
Frame Frame Frame Frame Ingress Buffer Frame Frame Frame Frame Frame Frame
Pause
Latency Buffer turning is platform specific
120 Data Mobility Application Servers • Migrates data between storage arrays for • Technology refreshes • Workload balancing • Storage consolidation • DMM offers • Online migration of heterogeneous arrays Data Mobility Manager • Simultaneous migration of multiple LUNs
Application Data • Unequal size LUN migration I/O Migration • Rate adjusted migration • Verification of migrated data • Dual fabric support • CLI and wizard-based management with Cisco Fabric Manager • Not metered on no. of terabytes migrated or no. of arrays Old New • Requires no SAN reconfiguration or rewiring Array Array • Uses FC Redirect 121 SAN Extension - CWDM Course Wavelength Division Multiplexing TX RX Transmission TX RX TX Optical fiber pair RX TX RX Optical OADM Optical transmitters receivers • 8 channels WDM using 20nm spacing
• Colored CWDM SFPs used in FC switch
• Optical multiplexing done in OADM • Passive device
122 SAN Extension - DWDM Dense Wavelength Division Multiplexing TX Transmission RX TX RX Optical Splitter Protection TX RX Optical fiber pair TX RX Optical DWDM devices Optical transmitters receivers
• DWDM systems use optical devices to combine the output of several optical transmitters
• Higher density technology compared with CWDM, <1nm spacing
123 Dense vs Coarse (DWDM vs CWDM)
DWDM CWDM Application Long Haul Metro Amplifiers Typically EDFAs Almost Never # Channels Up to 80 Up to 8 Channel Spacing 0.4 nm 20nm Distance Up to 3000km Up to 80km Spectrum 1530nm to 1560nm 1270nm to 1610nm Filter Technology Intelligent Passive
Site 1 Site 2 Site 1 Site 2
MDS ONS
Array DWDM CWDM 124 Summary
Drivers in DC are forcing change Many design options
• 10G convergence & server virtualization • Optimized for performance
• It's not just about FCP anymore. FCoE, NFS, iSCSI are • Some for management being adopted • Others for cable plant optimization Proper SAN design is holistic in the approach
• Performance, Scale, Management attributes all play critical roles
• Not all security issues are external
• Fault isolation goes beyond SAN A/B separation
• Consider performance under load
• Design for SAN services
125 Additional Relevant Sessions Storage Networking – Cisco Live Berlin
• BRKSAN-3446 - SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric
• Friday 9AM
126 Call to Action
• Visit the World of Solutions for: • Multiprotocol Storage Networking booth • See the MDS 9718, Nexus 5672UP, 2348UPQ, and MDS 40G FCoE blade • Data Center Switching Whisper Suite • Strategy & Roadmap (Product portfolio includes: Cisco Nexus 2K, 5K, 6K, 7K, and MDS products). • Technical Solution Clinics
• Meet the Engineer • Available Tuesday and Thursday
128 Complete Your Online Session Evaluation
• Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt.
• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations
129 Thank you
130