Storage Area Network

Section 2 : Storage Networking Technologies and Virtualization Storage Area Network

Chapter 6

EMC Proven Professional

The #1 Certification Program in the information storage and management industry

© 2009 EMC Corporation. All rights reserved. Chapter Objectives Upon completion of this chapter, you will be able to: .Describe SAN and its benefits .Discuss components of SAN .Describe connectivity options of SAN .Describe FC protocol stack and FC addressing .List common FC topologies

© 2009 EMC Corporation. All rights reserved. Business Needs and Technology Challenges .Just-in-time information to business users .Integration of information infrastructure with business processes .Flexible and resilient storage architecture .DAS is inefficient to meet these challenges . Storage Networking emerged as a solution . FC SAN . NAS . IP SAN

© 2009 EMC Corporation. All rights reserved. 6.1 Overview of Fibre Channel . Fibre Channel is a high-speed network technology uses: . Optical fiber cables (for front end connectivity) . Serial copper cables (for back end connectivity) . Latest FC implementations support 8GFC - 8Gb/s . Servers are attached to 2 distinct networks . Back-end . Front-end

IP FC SAN network

Users and Servers and Storage and Application Applications Application Clients Data © 2009 EMC Corporation. All rights reserved. Fibre Channel contd… . FC standardization began in 1988 by ANSI. They chartered Fibre Channel Working Group (FCWG). . In 1994, high speed standard was developed and Fibre Channel Association (FCA) was founded with 70 members of companies. . Offers High speed compared to Ultra SCSI.

© 2009 EMC Corporation. All rights reserved. 6.2 SAN and Its Evolution . Dedicated high speed network of Servers servers and shared storage devices across the globe. . Provide block level data access . Resource Consolidation FC SAN . Centralized storage and management . Scalability . Theoretical limit: Appx. 15 million devices

FC SAN Evolution Server Servers Servers

FC Switch FC Switch

FC Hub FC Switch FC Switch

FC Hub FC Switch

FC Switch FC Switch

Storage Array Storage Arrays Storage Arrays SAN Islands Interconnected SANs Enterprise SANs FC Arbitrated Loop FC Switched Fabric FC Switched Fabric Fibre Channel SAN Evolution

© 2009 EMC Corporation. All rights reserved. 6.3 Components of SAN . Three Basic components: . Servers . Network Infrastructure and . Storage . Later divided into following key elements: . Node ports . Cabling . Interconnecting devices ( FC switches) . Storage Arrays and . SAN management software

© 2009 EMC Corporation. All rights reserved. 6.3.1 Node ports . Examples of nodes . Hosts, storage and tape library Node Tx Port 0 . Ports are available on: Rx Port 0 . HBA in host Link Port 1 . Front-end adapters in storage . Each port has transmit (Tx) link and receive (Rx) link Port n . HBAs perform low-level interface functions automatically to minimize impact on host performance

© 2009 EMC Corporation. All rights reserved. 6.3.2 Cabling . SAN implementation uses: . Copper cables for short distance . Optical fiber cables for long distance . Two types of optical cables . Single-mode . Can carry single beams of light . Distance up to 10 KM . Multi-mode . Can carry multiple beams of light MMF cables- OM1,OM2, Laser simultaneously optimized OM3 . Distance up to 500 meters.

© 2009 EMC Corporation. All rights reserved. Cabling (Connectors) .Node Connectors: . Standard Connectors(SC) . SC Duplex Connectors . Speed up to 1Gb/s SC Connector . Lucent Connectors(LC) . LC Duplex Connectors LC Connector .Patch panel Connectors . ST Simplex Connectors Small Form-factor Pluggable(SFP) ST Connector optical transceiver. Speed- 10Gb/s

© 2009 EMC Corporation. All rights reserved. 6.3.3 Interconnecting devices . Basis for SAN communication 1. Hubs . Logical loop or Physical star topology. . Bandwidth is shared. FC HUB FC Switch 2. Switches . More intelligent than Hubs. . Do not share bandwidth. 3. Directors . Similar to FC switches – high port count and fault tolerance. . Deployed for Data Centers. Director

© 2009 EMC Corporation. All rights reserved. 6.3.4 Storage Arrays . Provides storage consolidation and centralization . Features of an array . High Availability/Redundancy . Performance . Business Continuity . Multiple host connectivity.

. A suite of tools used in a SAN to manage the interface between host and storage arrays . Provides integrated management of SAN environment . Mapping devices, servers and switches. . Monitoring and generating alerts. . Web based GUI or CLI

© 2009 EMC Corporation. All rights reserved. 6.4.1 Point-to-Point . Direct connection between the devices. . Limited connectivity. . Cannot be scaled to accommodate large number of network devices . DAS.

Storage Array

© 2009 EMC Corporation. All rights reserved. Servers 6.4.2 Fibre Channel –Arbitrated Loop . In FC-AL, devices attached to shared loop. . Token ring topology and physical star topology. . Each device competes with others to perform I/O operations. . Devices are connected via hubs.

FC Hub Storage Array

Servers © 2009 EMC Corporation. All rights reserved. FC-AL Limitations 1. Shares bandwidth in loop. Data transmission is low. 2. Uses 8-bit addressing, supports up to 127 devices. 3. Loop re-initialization in adding or removing devices.

HBA HBAHBA HBA BypByp Receive Transmit

9 8 4 Node B 7 5 Node C NL_Port NL_Port #4 Transmit Receive NL_Port NL_Port #4 BypByp #3 #3 6 ArrayHBA Port BypByp HBAFA Receive Transmit Hub_Pt Hub_Pt

Target

© 2009 EMC Corporation. All rights reserved. 6.4.3 Fibre Channel – Switched Fabric . FC-SW provides dedicated bandwidth and scalability between devices. . Also called Fibre Connect. . A Fabric is a logical/virtual space where all nodes communicate in a network. . Switch creates logical space . Domain ID - addressing . Each port in fabric has 24-bit Fibre channel address. FC Switch . Fabric topology described as Storage Array tiers.

. Tires based on number of switches used. . Number is based on infrastructure constructed. . Number of tiers determine time for message propagation.

FC Switch Node A Node D

Port Port N_Port #2 N_PortNL_Port #1 Transmit Receive NL_Port #1 #2 Storage HBA HBAPort Receive Transmit

Node B Node C

N_PortNL_Port #4 Transmit Receive N_Port #3 #4

HBA HBA Storage Port Receive Transmit Port Port

Host NL-Port NL-Port

Tape Library FC Hub Host NL-Port

Host FC Switch N-Port F-Port FL-Port FC Switch

F-Port E-Port E-Port F-Port

N-Port N-Port

Storage Array Storage Array © 2009 EMC Corporation. All rights reserved. Types of Ports 1.N_port: node port, HBA/ storage array connected to switch/ switch fabric. 2.NL_port: supports arbitrated loop topology. 3.E_port: expansion port, connection between two switches through Inter-Switch links (ISL) . ISL transfers host-storage data and fabric management traffic. 4.F_port: fabric port connecting N_port. 5.FL_port: fabric port that participate in FC-AL. . Connects to NL_ports on an FC-AL loop. 6.G_port: generic port acts as E_port or F_port. . Determines functionality during initialization.

© 2009 EMC Corporation. All rights reserved. 6.6 FC Architecture Overview . FC uses channel technology for interconnecting devices. . Provide high performance with low protocol overheads. . Fibre Channel Protocol (FCP) is implementation of SCSI-3 over FC network. . Sustained transmission bandwidth over long distances . Support larger number of addressable devices . Provides speeds up to 8.5 Gb/s (8 GFC) . FCP specified by standard T10. . FCP-3 is latest issued standard. FCP-4 under development. . Fibre Channel – Physical and Signaling Interface (FC-PH): used to transmit SCSI commands.

© 2009 EMC Corporation. All rights reserved. 6.6.1 Fibre Channel Protocol Stack . FC standard maps Upper Layer Protocols (ULP) . SCSI, IP, High Performance Parallel Interface (HIPPI), Enterprise System Connection (ESCON), and ATM.

Application

FC-4 SCSI HIPPI ESCON ATM IP

FC-2 Framing/Flow Control

FC-1 Encode/Decode

FC-0 1 Gb/s 2 Gb/s 4 Gb/s 8 Gb/s

FC-4 Mapping Mapping upper layer protocol (e.g. SCSI-3 to interface FC transport FC-3 Common Not implemented services FC-2 Routing, flow Frame structure, ports, FC addressing, buffer control credits FC-1 Encode/decode 8b/10b encoding, bit and frame synchronization FC-0 Physical layer Media, cables, connector

© 2009 EMC Corporation. All rights reserved. . FC Address is assigned during Fabric Login . Used to communicate between nodes within SAN . Similar in functionality to an IP address on NICs . Address Format: 24 bit address, dynamically assigned. 1. N_port address: . Domain ID: 8bits, 256 possible domains. Of which 239 are available. FFFFFC – reserved for name server. . Area ID: identifies group of F_ports. . Port ID: identifies specific F_port within a group.

Maximum N_ports that can be addressed = 239 domains * 256 areas * 256 ports = 15,663,104 FC addresses.

© 2009 EMC Corporation. All rights reserved. 2. FC Address of an NL_port: . Private loop: 2 upper bytes are unused (all 0s). . Public loop: when FL_port connects a AL. . Loop ID identifies an Arbitrated loop. Remains same for all NL_ports in a loop. . AL-PA: addresses each FL_port connecting the private/ public loop. 127 allowable AL-PA addresses.

© 2009 EMC Corporation. All rights reserved. World Wide Names . Unique 64 bit identifier. . 2 types of WWNs: WWNN – World Wide Node Name. WWPN – World Wide Port Name. . Static to the port . Used to physically identify ports or nodes within SAN . Similar to NIC’s MAC address . WWN are used to identify storage devices and HBAs. . Name Server maps WWNs with dynamic FC addresses.

World Wide Name - Array 5 0 0 6 0 1 6 0 0 0 6 0 0 1 B 2 0101 0000 0000 0110 0000 0001 0110 0000 0000 0000 0110 0000 0000 0001 1011 0010 Company ID Port Model Seed 24 bits 32 bits

World Wide Name - HBA 1 0 0 0 0 0 0 0 c 9 2 0 d c 4 0 Reserved Company ID Company Specific 12 bits 24 bits 24 bits

© 2009 EMC Corporation. All rights reserved. Frame Header fields . R_CTL: Routing Control – control frame or data frame. . CS_CTL: Class Specific Control – link speeds for class 1 and 4. . TYPE: ULP to be carried in frame. . Destination ID and Source ID: to identify source and destination. . F_CTL: 3-byte field has control information related to frame content. . SEQ_ID: identifies the sequence of frame. . DF_CTL: Data Field Control – 1-byte field indicate optional headers . Sequence Count: maintains count of frames sent for a sequence. . OX_ID and RX_ID: Originating Exchange ID and Responder Exchange ID.

© 2009 EMC Corporation. All rights reserved. 6.6.4 Structure and Organization of FC Data FC data is organized as: 1. Exchange operations . Enables two N_ports to identify and manage a set of information units . Maps to sequence 2. Sequence . Contiguous set of frames sent from one port to another 3. Frames . Fundamental unit of data transfer . Each frame can contain up to 2112 bytes of payload

© 2009 EMC Corporation. All rights reserved. 6.6.5 Flow Control . Uses 2 flow- control mechanisms: . buffer – to – buffer credit – BB_Credit . end – to end credit – EE_Credit . BB_Credit: h/w based flow control. Controls maximum number of frames to be available on link. . EE_Credit: Initiator and Target exchange EE_Credit parameters at the start of communication. . Affect class 1 and 2 traffic only.

o Class F – used by switches. o Class 4,5 and 6 used for specific applications.

FC SAN

Storage Array Array port

. When device logs onto fabric, registers to name server.

. When port logs onto fabric, performs device discovery process with other registered devices in name server. . Only the members in the same zone establish the link- level services.

Zone sets Zone set (Library)

Zone Zone Zone Zone (Library)

Member Member Member Member Member Member Member WWN's

. Multiple zones defines in a fabric, only one zone can be active at one time. . Members, zones and zone sets form the hierarchy. . A port / node can be a member of multiple zones. . Zone sets also referred to as Zone configurations.

Servers Switch Domain ID = 15 Port 5

Port 7 Zone 2 Port 1 WWN 10:00:00:00:C9:20:DC:40 Storage Array FC Switch

Zone 3 Port 9 Port 12

WWN 10:00:00:00:C9:20:DC:56

WWN 50:06:04:82:E8:91:2B:9E

Zone 1 Zone 1 (WWN Zone) = 10:00:00:00:C9:20:DC:82 ; 50:06:04:82:E8:91:2B:9E Zone 2 (Port Zone) = 15,5 ; 15,7 WWN 10:00:00:00:C9:20:DC:82 Zone 3 (Mixed Zone) = 10:00:00:00:C9:20:DC:56 ; 15,12

© 2009 EMC Corporation. All rights reserved. Types of Zoning 1. WWN zoning: . Uses WWNs. . Also referred as soft zoning. . Major advantage- allows SAN reconfiguration without changing zone information 2. Port zoning: . Uses FC addresses of ports to define zones. . FC address is dynamically assigned when port logs on to fabric. . Hard zoning – any changes in fabric configuration affects zoning. 3. Mixed zoning: . Combines qualities of WWN and port zoning.

Zoning can be combined with LUN masking. Zoning at fabric level. LUN masking at storage level.

© 2009 EMC Corporation. All rights reserved. 6.8 Fibre Channel Login Types 3 login types: 1. Fabric login (FLOGI): Login between N_port and a F_port. a. Device sends FLOGI frame with WWNN and WWPN parameters at the well-known FC address FFFFFE. b. Switch accepts login and returns Accept (ACC) frame with the assigned FC address for the device. 2. Port login (PLOGI): Login between two N_ports. a. Initiator sends PLOGI frame. Target acknowledges by ACC frame. 3. Process login (PRLI): between N_ports, but related to FC-4 ULPs a. N_ports exchange SCSI- 3- related service parameters.

© 2009 EMC Corporation. All rights reserved. 6.9 FC Topologies . Standard topologies to connect devices. . Core – Edge fabric – most popular fabric topology designs . Mesh Topology . Variations of both are commonly deployed in SAN implementations.

Server Core Tier Director Single-core topology Storage Array

Edge Tier FC Switch FC Switch FC Switch

Core Tier Director Director Server Storage Array Dual-core topology © 2009 EMC Corporation. All rights reserved. Benefits and limitations of Core- Edge Fabric . Benefits: . One – hop storage access to all storage in the system. . Easy calculations of ISL loading and traffic patterns. . Easier to develop set of rules for scaling and apportioning. . Can be scaled to larger environments by linking core switches. . Limitations: . Scaling leads to more ISLs, increasing Hop count. Leads to transmission delay. Solution is to keep hop count unchanged. . Maintaining ISLs becomes difficult as number of cores increases. When this happens, change design to compound core-edge design. © 2009 EMC Corporation. All rights reserved. 6.9.2 Mesh Topology

FC Switches FC Switches

Server Partial Mesh Server Full Mesh

Storage Array Storage Array

© 2009 EMC Corporation. All rights reserved. Chapter Summary Key topics covered in this chapter: o SAN features and benefits o SAN connectivity options o Port types and inter switch links o FC protocol stack and addressing o FC fabric topologies o Fabric management by zoning