Data Center Bridging
Total Page:16
File Type:pdf, Size:1020Kb
Data Center Bridging IEEE 802 Tutorial 12th November 2007 Contributors and Supporters Hugh Barrass (Cisco) Zhi Hern-Loh (Fulcrum Micro) Jan Bialkowski (Infinera) Mike Ko (IBM) Bob Brunner (Ericsson) Menu Menuchehry (Marvell) Craig Carlson (Qlogic) Joe Pelissier (Cisco) Mukund Chavan (Emulex) Renato Recio (IBM) Rao Cherukuri (Juniper Networks) Guenter Roeck (Teak Uri Cummings (Fulcrum Micro) Technologies) Norman Finn (Cisco) Ravi Shenoy (Emulex) Anoop Ghanwani (Brocade) John Terry (Brocade) Mitchell Gusat (IBM) Pat Thaler (Broadcom) Asif Hazarika (Fujitsu Manoj Wadekar (Intel) Microelectronics) Fred Worley (HP) DCB Tutorial Nov 2007 2 Agenda Introduction: Pat Thaler Background: Manoj Wadekar Gap Analysis: Anoop Ghanwani Solution Framework: Hugh Barrass Potential Challenges and Solutions: Joe Pelissier 802.1 Architecture for DCB: Norm Finn Q&A DCB Tutorial Nov 2007 3 Data Center Bridging Related Projects 802.1Qau Congestion Notification ¾In draft development 802.1Qaz Enhanced Transmission Selection ¾PAR submitted for IEEE 802 approval at this meeting Priority-Based Flow Control ¾Congestion Management task group is developing a PAR DCB Tutorial Nov 2007 4 Background: Data Center I/O Consolidation Manoj Wadekar Data Center Topology NAS transaction NAS web state content access packet processing (load balance, DB queries firewall, proxy, etc) business responses transactions SAN commits, secured data integrity open interactions client Back End requests, DAS Intra-Tier: lock mechanics, data sharing, responses web consistency content Inter-Tier: database query/response access Mid Tier Intra-Tier: State preservation Front End Inter-Tier: Business transaction, database query or commit Intra-Tier: Load balancing, dynamic content creation, caching Inter-Tier: Business transaction Networking Characteristics: Application Smaller Larger I/O Size: # of Connections: Many Few DCB Tutorial Nov 2007 6 Characteristics of Data Center Market Segments Internet Portal Data Centers ¾ Internet based services to consumers and businesses ¾ Concentrated industry, with small number of high growth customers Enterprise Servers Data Centers ¾ Business workflow, Database, Web 2.0/SOA ¾ Large enterprise to medium Business HPC and Analytics Data Centers ¾ Large 1000s node clusters for HPC (DOD, Seismic,…) ¾ Medium 100s node clusters for Analytics (e.g. FSIs …) ¾ Complex scientific and technical DCB Tutorial Nov 2007 7 Data Centers Today Type Internet Portal Data CenterEnterprise Data Center HPCC Data Center ¾SW based enterprises, Non ¾ Non mission critical HW mission critical HW base ¾Large desktop client base architecture ¾10K to 100K servers ¾ 100 to 10K servers ¾100 to 10K servers Character Primary Needs: Primary Needs: Primary Needs: istics ¾Low capital cost ¾Robust RAS ¾Low Latency ¾Reduced power and cooling ¾Security and QOS control ¾High throughput ¾Configuration solution ¾Simplified management flexibility Presentation LogicBusiness Logic Database Topology examples 100s Æ 1000s 1000s Servers >100 DB Storage Web Servers Servers Ethernet Routers Ethernet Switches Fibre Channel Switches and Appliances and Appliances ¾Fabric preference is ¾Low Cost ¾Looks nice on the surface, but… ¾Standard high volume ¾Lots of cables (cost and power) underneath DCB Tutorial Nov 2007 8 HPC Cluster Network Market Overview - Interconnects in Top 500 Interconnect Family – Top500 Machines 100 Other 90 Cray HiPPI Future 80 IBM Other Directions 70 Quadrics InfiniPath IB 20 - 40 Gb/s IB 60 Myrinet- InfiniBand 2K 50 Myrinet Cray Ethernet Myri-10G (Enet) 40 30 IBM SP 1 & 10G Enet Switch Gigabit 20 Ethernet 10 0 1997 2000 2002 2007 Standard networks Ethernet & Infiniband (IB) replacing proprietary networks: ¾ IB leading in aggregate performance ¾ Ethernet dominates in volume Adoption of 10 Gbps Ethernet in the HPC market will likely be fueled by: ¾ 10 Gbps NICs on the motherboard, 10 GBASE-T/10GBASE-KR ¾ Storage convergence over Ethernet (iSCSI, FCoE) There will always be need for special solution for high end (E.g. IB, Myrinet) ¾ But 10 GigE will also play a role in the HPC market in the future Challenges: Need Ethernet enhancements - IB claims better technology ¾ Low Latency, traffic differentiation, “no-drop” fabric, multi-path, bi-sectional bandwidth DCB Tutorial Nov 2007 9 Data Center: Blade Servers 12,00012,000 35%35% ProjectedProjected BladeBlade UnitsUnits (k)(k) andand ServerServer ShareShare (%)(%) BLADE 30% 10,000 31%31% 30% 10,000 29% BLADE 29% Enet 25%25% BLADE 8,0008,000 Enet 23%23% LAN BLADE FC 20%20% Enet 19%19% 6,0006,000 FC Enet SW Enet 15% 15 % Storage 15% 15 % FC IB 4,0004,000 11%11% IB FC SW 1010 % % FC IPC 9%9% IB 2,0002,000 5%5% IB IB SW 00 0%0% 20062006 2007 2007 2008 2008 2009 2009 2010 2010 2011 2011 2012 2012 TotalTotal Servers Servers BladesBlades BladeBlade Attach Attach Rate Rate Challenges: IO Consolidation is Blade Servers are gaining strong requirement for Blade momentum in market Servers: Second Generation Blade Servers ¾ Multiple fabrics, mezzanine cards, in market ¾ Power/thermal envelope, ¾ Management complexity Provides server, cable ¾ Backplane complexity consolidation Ethernet default fabric Optional fabrics for SAN and HPC REF: Total Servers: Worldwide and Regional Server 2007-2012 Forecast, April, 2007, IDC Blade Servers: Worldwide and US Blade Server 2006-2010 Forecast, Oct’06, IDC DCB Tutorial Nov 2007 10 Data Center: Storage Trends Lot of people using SAN: SANSAN Growth:Growth: iSCSIiSCSI andand FCFC Networked Storage growing 60% year 5,0005,000 FC is incumbent and growing: FCFC iSCSIiSCSI Entrenched enterprise customers 4,0004,000 iSCSI is taking off…: SMB and 3,0003,000 Greenfield deployment - choice driven by targets 2,0002,000 External Terabytes External Terabytes Storage convergence over Ethernet: 1,1, 0 0 0 0 0 0 iSCSI for new SAN installations 00 2005 2006 2007 2008 2009 2010 FC tunneled over Ethernet (FCoE) for 2005 2006 2007 2008 2009 2010 expanding FC SAN installation IDC Storage and FC HBA analyst reports, 2006 Challenges: Too many ports, fabrics, cables, power/thermal Need to address FC as well as iSCSI DCB Tutorial Nov 2007 11 SAN Protocols FCoE: App Applications ¾FC Tunneled through Ethernet SCSI SCSI ¾Addresses customer investment in iSCSI FCP legacy FC storage ¾Expects FC equivalent no-drop Encapsulation Layer TCP behavior in underlying Ethernet interconnect IP FC/FCoE ¾ Needs Ethernet enhancements for Base link convergence and “no-drop” Transport Ethernet (DC enhanced) performance FC FC Headers CRC iSCSI: Ethernet Ethernet Headers FCS/PAD ¾ SCSI over TCP/IP that provides reliability FC Payload ¾ High speed protocol acceleration FC solutions benefit from reduced FCoE Frame packet drops DCB Tutorial Nov 2007 12 Fabric convergence needs: Processor Traffic Differentiation: Memory ¾LAN, SAN, IPC traffic needs differentiation in converged fabric Lossless Fabric: I/O I/O I/O ¾FC does not have transport layer – retransmissions are at SCSI! IPCStg LAN ¾iSCSI acceleration may benefit from lossless fabric too Seamless deployment: ¾Backward compatibility ¾Plug and play Ethernet needs these enhancements to be true converged fabric for Data Center DCB Tutorial Nov 2007 13 Data Center Bridging What is DCB? ¾Data Center Bridging provides Ethernet enhancements for Data Center needs (Storage, IPC, Blade Processor Servers etc.) Memory ¾Enhancements apply to bridges as well as end stations I/O Why DCB? ¾DC market demands converged fabric ¾Ethernet needs enhancements to be successful converged fabric of choice Scope of DCB: DCB ¾Should provide convergence DCB capabilities for Data Center – short range networks DCB Tutorial Nov 2007 14 Data Center Bridging Æ Value Proposition Simpler Management Single physical fabric to manage. Simpler to deploy, upgrade and maintain. Fabric Convergence Servers Multiple Fabrics Converged Fabric (DCB) Lower Cost Less adapters, cables & switches Lower power/thermals Improved RAS Reduced failure points, time, misconnections, bumping. DCB Tutorial Nov 2007 15 Comparison of convergence levels No convergence Converged Fabric DCB (dedicated) Management Cluster Cluster Memory Memory Memory Memory Network Memory Network Memory uP uP uP Chip Chip Chip SAN SAN 1 PCIe PCIe PCIe Link System LAN System LAN System Number of 2-3 1 1 Fabric Types IO interference No No Yes Technologies 3 1 to 2 1 to 2 Managed HW cost 3x adapters 3x adapters 1x adapter RAS More HW More HW Least HW Cable mess 3-4x 3-4x 1x DCB Tutorial Nov 2007 16 Data Center Bridging Summary DCB provides IO Consolidation: Lower CapEx, Lower OpEx, Unified management Needs standards-based Ethernet enhancements: Need to support multiple traffic types and provide traffic differentiation “No-drop” option for DC applications Deterministic network behavior for IPC Does not disrupt existing infrastructure Should allow “plug-and-play” for enhanced devices Maintain backward compatibility for legacy devices DCB Tutorial Nov 2007 17 Gap Analysis: Data Center Current Infrastructure Anoop Ghanwani Overview Data Center Bridging (DCB) requirements What do 802.1 and 802.3 already provide? What can be done to make bridges more suited for the data center? DCB Tutorial Nov 2007 19 Recap of DCB Requirements Single physical infrastructure for different traffic types Traffic in existing bridged LANs ¾Tolerant to loss – apps that care about loss use TCP ¾QoS achieved by using traffic classes; e.g. voice vs data traffic Data center traffic has different needs ¾Some apps expect lossless transport; e.g. FCoE This requirement cannot