We're Ready. Are You?
Total Page:16
File Type:pdf, Size:1020Kb
We’re ready. Are you? FCoE for Small and Mid-size Enterprise Hernan Vukovic Consulting Systems Engineer BRKSAN-2101 The Session Objectives: • Provide a refresh of FCoE and DCBX • Understand the basic FCoE implementation on Nexus 5K • FCoE design options for small and mid-size enterprise • FCoE deployment with Cisco Unified architecture • Step-by-step configuration examples The Session Non-Objectives: • Nexus hardware architecture deep dive • UCS storage architecture • SAN distance extension using FCoE • FCOE or iSCSI, who is better for small and mid-size customers Related Sessions Storage Contents • BRKCOM-2007 - UCS Storage Integration, Technologies, and Topologies Traditional Data Center Design Ethernet LAN and Fibre Channel SAN Ethernet FC FC Fabric ‘A’ Fabric ‘B’ L3 L2 LAN NIC HBA SAN Agenda • Introduction to FCoE Technology • FCoE SAN Design for Small and Mid-size Enterprise • Basic FCoE Configuration and Troubleshooting • Conclusion Agenda • Introduction to FCoE Technology • FCoE SAN Design for Small and Mid-size Enterprise • Basic FCoE Configuration and Troubleshooting • Conclusion Block Storage Protocols (FC/FCoE/iSCSI) What is SAN (Storage Area Network) LAN (TCP/IP) SAN (Fibre Channel, iSCSI, FCoE) A dedicated network that provides access to consolidated, block level data storage. The SCSI I/O Transaction • The SCSI protocol defines a bus based system used to carry block based storage commands • The channel provides connectivity between server and storage The following shows two sample SCSI exchanges: Host (Initiator) Disk (Target) SCSI READ OPERATION DATA STATUSDATA DATA READ SCSI I/O Channel Host (Initiator) Disk (Target) SCSI WRITE OPERATION STATUS WRITE SCSI I/O Channel DATA DATA DATA SAN Protocols Network Stack Comparison SCSI iSCSI FCoE FibreChannel SCSI SCSI SCSI SCSI iSCSI FCP FCP FC FC TCP IP FCoE Lossless Ethernet Ethernet PHYSICAL WIRE Block Storage Networking Protocols FC FCoE iSCSI • SCSI transport protocol that operates • Mapping of Fibre Channel frames over • SCSI transport protocol that operates over Fiber Channel Ethernet over TCP • FC frames with the SCSI CDB • Fibre Channel is enabled to run on a • Encapsulation of SCSI command payload are transported over Fiber lossless Ethernet network descriptor blocks and data in TCP/IP channel Protocol byte streams • Works on Ethernet switches with FCF • Works on Fiber Channel switches capability • Works on any Ethernet switch • OXID/RXID generated for every I_T • OXID/RXID generated for every I_T pair • ISID/TSID generated for every I_T pair conversation conversation pair conversation • Needs Zoning • Needs Zoning • Zoning not required • Runs on dedicated lossless Fiber • Uses Ethernet and needs lossless • Works on TCP and subject to losses Channel networks network in network • Limited by distance • Limited by distance • No Distance limitations • Well suited for latency sensitive and • Reduces the TCO of the fabrics by • Well suited for applications with less high I/O applications preserving the advantages of FC I/O requirements while reducing the networks TCO Unified Fabric and FCoE All Data accessed over a common fabric iSCSI iSCSI NAS NAS FC SAN FCoE SAN Appliance Gateway Appliance Gateway Computer System Computer System Computer System Computer System Computer System Computer System Host/ Application Application Application Application Application Application Server File System File System File System File System File System File System Volume Manager Volume Manager Volume Manager Volume Manager I/O Redirector I/O Redirector SCSI Device Driver SCSI Device Driver SCSI Device Driver SCSI Device Driver NFS/CIFS iSCSI Driver iSCSI Driver NFS/CIFS FC Driver FCoE Driver TCP/IP Stack TCP/IP Stack TCP/IP Stack TCP/IP Stack HBA CNA NIC NIC Storage NIC NIC Transport SAN Unified Fabric IP Block I/O File I/O NIC NIC NIC NIC TCP/IP Stack TCP/IP Stack TCP/IP Stack TCP/IP Stack iSCSI Layer iSCSI Layer File System File System Storage FC FCoE Bus Adapter FC HBA Device Driver FC HBA Media FC FC Basics of FCoE What is Fibre Channel over Ethernet (FCoE) FCoE Benefits • Mapping of FC Frames over • Fewer Cables Ethernet •Both block I/O & Ethernet traffic co-exist on same cable • Enables FC on a Lossless Ethernet Network • Fewer adapters needed • Overall less power Ethernet • Interoperates with existing Fibre SAN’s Channel •Management of SAN’s remains Traffic constant • No Gateway FCoE Protocol Fundamentals Protocol Organization – Data and Control Plane FC-BB-5 defines two protocols required for an FCoE enabled Fabric FIP (FCoE Initialization FCoE Protocol) • Data Plane • It is the control plane protocol • It is used to carry most of the • It is used to discover the FC entities FC frames and all the SCSI traffic connected to an Ethernet cloud • Uses Fabric Assigned MAC • It is also used to login to and logout address (dynamic) : FPMA from the FC fabric • Uses unique BIA on CNA for MAC • IEEE-assigned Ethertype for FCoE traffic is 0x8906 • IEEE-assigned Ethertype for FCoE traffic is 0x8914 FCoE Protocol Fundamentals Fibre Channel over Ethernet (FCoE) Bit 0 FCoE Frame Format Bit 31 Destination MAC Address . Fibre Channel over Ethernet provides a high capacity and lower cost transport Source MAC Address option for block based storage (IEEE 802.1Q Tag) . Two protocols defined in the standard ET = FCoE Ver Reserved Reserved . FCoE – Data Plane Protocol Reserved . FIP – Control Plane Protocol Reserved SOF . FCoE is a standard - June 3rd 2009, the Encapsulated FC Frame (with CRC) FC-BB-5 working group of T11 completed its work and unanimously EOF Reserved approved a final standard for FCoE FCS . FCoE ‘is’ Fibre Channel Byte 0 Byte 2197 FC FCS FC Payload EOF CRC FCoE Header Header Header Ethernet FCoE Protocol Fundamentals FCoE Initialization Protocol (FIP) . Neighbor Discovery and Configuration Enode FCoE Switch (VN – VF and VE to VE) Initiator FCF . Step 1: FCoE VLAN Discovery VLAN VLAN Discovery Discovery FIP sends out a multicast to ALL_FCF_MAC address looking for the FCoE VLAN FCoE FIP frames use the native VLAN Initializatio FCF FCF n Discovery Discovery Protocol . Step 2: FCF Discovery (FIP) FIP sends out a multicast to the ALL_FCF_MAC address on the FCoE VLAN to find the FCFs answering for that FCoE VLAN FLOGI/FDISC FCF’s responds back with their MAC address FLOGI/FDISC Accept . Step 3: Fabric Login FIP sends a FLOGI request to the FCF_MAC found in Step 2 FC Establishes a virtual link between host and FCF FC Command FCoE Command FCF assigns the host a Enode MAC address to be used for FCoE Responses Protocol forwarding ** FIP does not carry any Fibre Channel data frames What happens after FLOGI . FCoE Protocol carries FC model Initiator Target FC . Name server registration/query PLOGI . PLOGI from initiator to target ACCEPT PRLI . SCSI Commands and Data ACCEPT transmission Report LUNs SCSI Command SCSI Command (1) (Read) SCSI_FCP_DATA SCSI Data (1) SCSI Status (1) STATUS (=good) SCSI Inquiry Command (LUN n) SCSI SCSI Target Initiator SCSI Command (1) (Write) SCSI_FCP_DATA SCSI Data (1) SCSI Status (1) STATUS (=good) Recap of Fibre Channel Concepts Name server WWNs Fabric controller FCIDs FSPF Zone Server VSANs Initiator target Fibre Channel Fabric SCSI Fibre Channel HBA Host System Disk Array Fibre Channel Addressing • 64-bit WWNs are used as burnt-in unique FCoE Addressing Scheme: addresses assigned to fabric switches, • A fabric-provided Mac address ports, and nodes by the manufacturer (FPMA) is assigned to each Enode • These addresses are registered in the • Enode MAC composed of a FC-MAP fabric and mapped to an 24-bit FC_ID and FCID 8 bits 8 bits 8 bits • FCoE forwarding decisions will still be made based on FSPF and the Switch Domain Area Device FCID within the Enode MAC • FC Switches assign FC_ID addresses to FC-MAP FC-ID N_Ports; (0E-FC-xx) 10.00.01 • FSPF Forwarding decisions are made on FC-MAC FC-MAP FC-ID domain ID Address (0E-FC-xx) 7.8.9 Basic Fibre Channel Port Types FC E_Port E_Port F_Port NP_Port FC NPV Switch Switch F_Port N_Port Node FC Switch F_Port N_Port Node FCF VE_Port VE_Port VF_Port VNP_Port FCoE_NPV Switch Switch End VF_Port VN_Port Node End VF_Port VN_Port FCoE Switch : FCF Node Fibre Channel VSAN Cisco MDS 9000 A Virtual SAN (VSAN) Provides Family with VSAN Service a Method to Allocate Ports Within a Physical Fabric to Create Virtual Fabrics Physical SAN Islands Are • Analogous to VLANs in Ethernet Virtualized onto Common SAN • Virtual fabrics created from larger Infrastructure cost-effective redundant physical fabric • Reduces wasted ports of island approach • Fabric events are isolated per VSAN— maintains isolation for HA (i.e., RSCNs) pwwn 50:06:01:61:3c:e0:1a:f6 Fibre Channel Zoning Target . Zones are the basic form of data path security FC/FCoE Fabric . zone members can only “see” and talk to other members of the zone . devices can be members of more than one zone . Default zoning is “deny” . Zones belong to a zoneset FCF with Domain ID 10 . Zoneset must be “active” to enforce zoning . Only one active zoneset per fabric or per VSAN SAN pwwn 10:00:00:00:c9:76:fd:31 Disk2 Disk3 Initiator ZoneA Host1 Disk1 ZoneC zoneset name ZONESET_V1 vsan 1 zone name Z_FC1_b1_FC1_e1_V1 vsan 1 Disk4 Host2 fcid 0x10.00.01 [pwwn 10:00:00:00:c9:76:fd:31] [initiator] ZoneB fcid 0x11.00.01 [pwwn 50:06:01:61:3c:e0:1a:f6] [target] Fibre Channel Flow Control . B2B Credits used to ensure that FC transport is lossless Fibre Channel Switch . Number of credits negotiated between ports when link is brought up . Each side informs the other side of the number of buffer credits it has 16 F ports - In the Fabric Login(FLOGI) R_RDY E ports – In the Exchange Link Parameters(ELP) . # Credits decremented with each packet placed on the wire Packet .