Unified Networking on 10 Gigabit Ethernet and NetApp provide a simple and flexible path to cost- effective performance of next-generation storage networks.

Executive Summary Contents Unified networking over 10 Gigabit Ethernet (10GbE) in the data center Executive Summary ...... 1 offers compelling benefits, including a simplified infrastructure, lower Simplifying the Network with 10GbE . . 2 equipment and power costs, and the flexibility to meet the needs of the Evolving with the Data Center . . . . .2 evolving, virtualized data center. In recent years, growth in Ethernet-based The Promise of Ethernet Storage . . . .3 storage has surpassed that of storage-specific fabrics, driven in large part by Ethernet Enhancements for Storage . . 4 the increase in server virtualization. The ubiquity of (SAN) access will be critical as virtualization deployments continue to Data Center Bridging for Lossless Ethernet ...... 4 grow and new, on-demand data center models emerge. over Ethernet: With long histories in Ethernet networking and storage, Intel and NetApp Extending Consolidation ...... 4 are two leaders in the transition to 10GbE unified networking. This paper Contrasting FCoE Architectures . . . 4 explores the many benefits of unified networking, the approaches to Open FCoE: Momentum enabling it, and the pivotal roles Intel and NetApp are playing in helping in the * Community ...... 4 to bring important, consolidation-driving technologies to enterprise data Intel Ethernet Unified Networking . . . 5 center customers. Reliability ...... 5

Cost-Effectiveness ...... 5

Scalable Performance ...... 6

Ease of Use ...... 6

NetApp Ethernet Storage: Proven, I/O Unification Engine . . . . .6

Conclusion ...... 7

Solution provided by: Simplifying the As server virtualization continues to 16 times the network capacity, and over 20 Network with 10GbE grow, 10GbE and unified networking times the compute capacity by 2015.4 A are simplifying server connectivity. new infrastructure must emerge to power As IT departments look to reduce Consolidating the traffic of multiple GbE this growth and enable the most efficient costs and improve server efficiency, connections onto a single 10GbE adapter use of resources; this is cloud computing. they are turning increasingly to server significantly reduces cable and infrastructure The cloud is an evolution of computing virtualization and consolidation. The complexity and overall TCO. Recent that delivers services over the Internet benefits of virtualization are widely enhancements to the Ethernet standard to consumers and enterprises. Services understood: less server hardware to enable 10GbE support for both LAN and scale—as needed and only when needed— purchase, lower power and cooling SAN traffic, allowing IT to realize further without user intervention. A highly scalable benefits by converging data and storage and efficient cloud architecture is needed needs, and centralized management. infrastructures. Thanks to its ubiquity, cost to provide both the technical attributes Today’s servers are based on powerful effectiveness, flexibility, and ease of use, and the extreme resource utilization and new processors, including the Intel® Ethernet has emerged as the unified data efficiency cloud computing promises. Xeon® processor 5600 and 7500 center fabric. With its reduced hardware requirements, series, that support more virtual Evolving with the fewer points of management, and broad machines (VMs) per physical host Data Center ecosystem support, 10GbE delivers the than ever before, helping IT realize flexible, simplified network infrastructure greater consolidation ratios. The growth in server virtualization has needed to support cloud computing. These helped data center networks evolve from key characteristics make 10GbE the ideal • The latest generation of Intel® Xeon® discrete, siloed infrastructures to more fabric for cloud infrastructures: processors enables IT to consolidate servers at a 15:1 ratio, delivering power savings of up to 90 percent and a five- Over 2.5 billion users will connect to the Internet in the month return on investment. next five years2 with over 10 billion devices.3 This usage • New four-socket processors are will require eight times the amount of storage capacity, 16 delivering 20 times the performance of previous-generation processors. times the network capacity, and over 20 times the compute • Nearly 50 percent of the four-socket capacity by 2015.4 A new infrastructure must emerge to servers shipped today are being used power this growth and enable the most efficient use of for virtualization.1 Unfortunately, the success achieved by many resources; this is cloud computing. organizations in attaining these benefits has been limited by the complications flexible fabrics with the scalability and Ubiquity. Ethernet connectivity ships of networking virtualized servers. As agility necessary to address the needs of standard on nearly every server today, and VM density increases, a physical server’s new usage models and provide an excellent Ethernet infrastructures are a universal data networking needs also increase, adding both foundation for enterprise cloud computing. center component. 10GbE products have cost and complexity. A typical virtualized been available for several years, but when server uses eight to 10 GbE local area Over 2.5 billion users will connect to the 10GbE LAN on motherboard (LOM) 2 network (LAN) ports and two dedicated Internet in the next five years with over connections are integrated in the next 3 SAN ports. 10 billion devices. This usage will require generation of servers, true unified LAN and eight times the amount of storage capacity, SAN connectivity will be available by default.

2 Unified Networking for 10 Gigabit Ethernet Advanced virtualization support. The Promise of Given its flexibility and long history, it Advanced server virtualization enables Ethernet Storage is not surprising that Ethernet storage dynamic resource allocation and is required is the fastest growing segment of the New usage models and the explosive for any cloud computing infrastructure. storage systems market. The industry growth of data in their organizations 5 Technologies from companies such as Intel research firm IDC estimates that the have forced IT administrators to deal and NetApp, and hypervisor vendors are worldwide Ethernet-based storage systems with complicated technical and business delivering line-rate 10GbE throughput (NAS and iSCSI SAN) market grew at a challenges. Today, most IT departments compounded annual growth rate (CAGR) and support for platform virtualization deploy separate LAN and storage networks, of approximately 23 percent between enhancements. with storage often divided between NAS 2005 and 2009. In comparison, unit Unified networking. A 10GbE unified and SAN, which requires multiple data shipments for FC storage shipments grew fabric simplifies the network infrastructure recovery solutions, a variety of data at approximately 10 percent CAGR during by consolidating LAN and SAN traffic. management models, and, potentially, the same period. different support teams. The goal of unified Internet small computer system interface iSCSI storage shipments experienced the networking is to allow a single fabric— (iSCSI) and network (NFS) highest growth rates (70 percent) during Ethernet—to carry these disparate are examples of storage protocols that use 2005–2009, driven by broad iSCSI traffic types. Ethernet, and the recent ratification of adoption in Windows*, virtual the Fibre Channel over Ethernet (FCoE) Ethernet has served as a unified data server, and blade server environments. NAS standard extends this capability. Recent center fabric for years, supporting LAN, unit shipments grew at a 14 percent CAGR Ethernet enhancements ensure quality of NAS (NFS, common Internet file system during this time period due to continued service (QoS) for critical traffic. (CIFS)) and iSCSI SAN traffic. With NAS deployments for business-critical recent Ethernet enhancements and the database applications and large-scale virtual Intel and NetApp are two companies ratification of the FCoE specification, server deployments. helping to advance the shift to 10GbE- standard Ethernet adapters can now based unified networking in the data As shown in Figure 1, Ethernet storage unit connect servers to Fibre Channel (FC) center. The latest Intel® Ethernet 10 Gigabit shipments surpassed FC storage shipments SANs. Extending Ethernet’s inherent controller and server adapters include in 2008. Industry analysts project continued advantages, including proven reliability, gains in the Ethernet storage market share virtualization optimizations and advanced ubiquity, and wide familiarity, to FC due to increasing deployment of “Ethernet unified networking features, including SAN traffic will help accelerate the move only” data centers (which use a unified optimizations for lossless Ethernet, to 10GbE-based I/O consolidation in 10GbE infrastructure for all data and storage intelligent, hardware-based accelerations virtualized data centers, reduce costs, and traffic), the emergence of cloud computing, for FCoE and iSCSI, and support for Open improve simplification and agility. and as FCoE solutions enter the mainstream. FCoE, which is discussed later in the paper.

NetApp is a leader in supporting Ethernet Ethernet versus Fibre Channel Target Unit Shipments storage, first as a pioneer for network area storage (NAS), next as an early 300,000 proponent of iSCSI, and now as a leader 250,000 with convergence-ready 10GbE FCoE. 200,000 NetApp’s Ethernet-based storage systems, through their unified, multi-protocol 150,000 architecture, have delivered consolidation 100,000 and unification to customers for over 50,000 10 years. 0 These solutions help to simplify network 2002 2003 2004 2005 2006 2007 2008 2009 connectivity for today’s virtualized servers Calendar Year FC Ethernet (iSCSI & NAS) and lay the foundation for the next- generation data center. Figure 1. Actual unit shipments for Ethernet and Fibre Channel storage (IDC, 2010).

Unified Networking for 10 Gigabit Ethernet 3 Ethernet Enhancements infrastructure, lower equipment and power In comparison, incumbent FC vendors for Storage costs, and universal SAN connectivity have developed converged network across the data center over the trusted adapters (CNAs) that offload FCoE Data Center Bridging Ethernet fabric. functions in hardware. The CNAs leverage for Lossless Ethernet those vendors’ existing FC software, Contrasting FCoE Architectures To strengthen 10GbE as a unified data including drivers, APIs, and management center fabric, the IEEE has developed Today we see two approaches for enabling applications. and ratified standards for Ethernet servers with FCoE connectivity, with each Open FCoE: Momentum in approach differing significantly according enhancements to support storage traffic. the Linux* Community These enhancements strengthen 10GbE as a to accessibility, ease of use, and cost Initiated by Intel, the Open FCoE project unified data center fabric for running FCoE of ownership. was accepted by the Linux community and iSCSI. Known collectively as “Data The first approach, Open FCoE, consists in November 2007 with the goal of Center Bridging” (DCB), these extensions of native FCoE initiators in OSs including accelerating development of a native FCoE enable better traffic prioritization over a Linux*, Microsoft Windows, and VMware initiator in the Linux kernel. The industry single interface and an advanced means for ESX*, which enable FCoE in standard responded enthusiastically, and today there shaping traffic on the network to decrease 10GbE adapters. This approach provides are over 190 active participants in the congestion. In short, DCB provides the a robust, scalable, and high-performance community who are contributing code, QoS that delivers a lossless Ethernet fabric server connectivity option without providing review comments, and testing for storage traffic. For more information, expensive, proprietary hardware. As shown the Open FCoE stack. To date, the Open see the DCB white paper from the Ethernet in Figure 2, Open FCoE implements the FCoE Source Web site (www.open-fcoe.org) Alliance: http://ethernetalliance.org/files/ complete FC protocol in the OS kernel. has received over 20,000 hits. Open static_page_files/83AD0BBC-C299- It provides libraries for different system- industry standards and Open Source play a B906-8F5985957E3327AA/Data Center level implementations, allowing vendors significant role in the modern data center, Bridging.pdf to implement data plane functions of as they lower research and development Fibre Channel over Ethernet: the FCoE stack in hardware to deliver costs and enable access to a multi- Extending Consolidation optimum performance. FCoE is a logical extension of Ethernet that uses FC’s Network, Service, and Open FCoE Solution FCoE HBA (CNA) Solution Protocol layers to carry data packets over Ethernet’s physical and data link layers. Application I/O Application I/O Using FC’s upper layers smoothes the SCSI storage interface SCSI storage interface transition to FCoE because existing SAN- System Operating based applications do not need to change Fibre Channel protocol to benefit from the performance and cost FCoE transport protocol benefits of FCoE. System Operating Fibre Channel protocol Many Enterprises have extensive FC FCoE transport protocol installations, and FCoE provides easy FCoE data path offloads FC SAN access for any server with a Ethernet MAC capable 10GbE port. By using standard Ethernet MAC Device Driver

Ethernet fabrics, FCoE eliminates the Adapter Host Bus Ethernet PHY Ethernet PHY need for dedicated FC host bus adapters Device Driver (HBAs), reducing cabling and switch-port requirements, while coexisting with existing FC hardware and software infrastructures. Figure 2. Overview of Fibre Channel over Ethernet (FCoE) host bus adapters (converged network adapters) and Open FCoE initiator solutions The result is a simplified data center

4 Unified Networking for 10 Gigabit Ethernet vendor supply chain that is designed for Native Initiator Success: iSCSI Intel Ethernet heterogeneous interoperability, ultimately iSCSI provides an excellent example of Unified Networking resulting in greater choice and lower the success of native storage initiators As discussed earlier, Ethernet-based equipment costs. integrated into the OS. storage has been growing at a faster rate The Open FCoE approach offers a number In the early days of iSCSI, proponents than traditional FC networks, particularly of advantages in terms of accessibility and of iSCSI HBAs claimed that these because Ethernet is a trusted, widely ease of use. dedicated adapters were necessary to deployed, and well-understood technology. deliver acceptable performance. Much Ethernet is the foundation of unified Accessibility. The Open FCoE approach like CNAs, iSCSI HBAs offload storage networking, and with over 25 years of makes FC SAN access available to any protocol processing to a separate processor experience delivering quality Ethernet server with a standard, compatible 10GbE on the adapter, rather than allowing the products, Intel is uniquely positioned to adapter installed. Rather than having to host processor and OS to handle these drive the transition to 10GbE unified purchase proprietary hardware, IT can tasks. While early HBAs did offer some networking continue using its current Ethernet adapters benefits in terms of throughput, they did to gain this additional functionality. Reliability so at levels beyond what was needed in real With FCoE support integrated into the world performance. They also suffered from Intel is the volume leader in Ethernet OS, IT can select compatible Ethernet many of the same issues as today’s CNAs, adapter shipments and has led the adapters from multiple vendors, helping to including high costs, vendor lock-in, industry through speed transitions and avoid vendor lock-in. With Open FCoE, reliability, and stability. enhancements to the Ethernet standard, FC SAN access will become universal including iSCSI, DCB, and FCoE. Intel Today, all major server OSs include once 10GbE LOM technology becomes Ethernet products have the broadest native iSCSI support, delivering the same pervasive. operating system support in the industry, benefits as described above for Open and Intel’s long-term product roadmaps Ease of use. Because the Open FCoE FCoE. The maturity of native iSCSI align future products with new server approach uses standard 10GbE adapters, initiators along with advanced adapter platform capabilities and upcoming data IT can leverage existing knowledge to and platform features now provides center trends, including cloud computing. configure and manage these adapters enterprise-level performance with standard Intel has expanded the capabilities for FCoE deployments. In fact, IT can Ethernet products. In fact, Intel recently of its trusted Ethernet product line standardize on a single product or product demonstrated a standard Intel® Ethernet to incorporate FCoE for true unified family for all LAN and SAN connectivity. 10 Gigabit Server Adapter driving 1.25 networking. By contrast, CNA vendors are FCoE initiator integration into the OS also million IOPS (input/output operations per acquiring Ethernet technology and adding means common OS-based tool support second) using the native iSCSI initiator in it to their existing FC-centric products. across a product family or even adapters Windows Server* 2008 R2.6 from multiple vendors. Using CNAs Cost-Effectiveness requires learning new tools and installing IT departments standardizing on Intel® Ethernet Server Adapters for iSCSI additional, proprietary software. Use of Standardizing on Intel Ethernet 10 Gigabit connectivity are able to use a single CNAs may also require using different Server Adapters takes advantage of Ethernet initiator, TCP/IP stack, and set of products for LAN and SAN connectivity economics to deliver cost-effective broad- management tools and IT policies. This because CNAs use the Ethernet fabric but based unified networking deployment. A standardization can deliver easier server are optimized for storage functions. single Intel Ethernet 10 Gigabit Server provisioning, lower the likelihood of Adapter supports FCoE, iSCSI, NAS, and human error, and simplify management LAN traffic for true unified networking while reducing capital and operational without the need for additional, expensive expenditures. As FCoE initiators are hardware or upgrades. integrated into OSs and 10GbE LOM implementations grow, Intel expects Open FCoE-based solutions to deliver these same benefits.

Unified Networking for 10 Gigabit Ethernet 5 Scalable Performance FCoE CNAs, on the other hand, require NetApp’s robust iSCSI storage, along In terms of performance, Intel Ethernet vendor lock-in. The management with powerful data management software, 10 Gigabit Server Adapters combine high interfaces, APIs, and drivers, of each provides the best of both worlds: throughput, intelligent hardware-based FCoE CNA brand typically differ, which performance plus the familiar Internet offloads with native OS initiators, and adds complexity. Protocol. Today, more than 25,000 customers rely on NetApp iSCSI solutions. stable hardware, while leaving processor NetApp Ethernet Storage: NetApp is also an active participant in the cycles available for application processing. Proven, I/O Unification Engine These adapters offload the main data paths development and marketing of the FCoE NetApp Ethernet-based storage systems, to improve FCoE throughput. The net result standard and was the first vendor to ship through their unified, multi-protocol is comparable FCoE performance for real- native FCoE storage systems. FCoE is a architecture, have delivered efficient, high- world workloads compared to that of CNAs. logical progression of NetApp’s unified performing storage solutions to enterprises storage architecture. The Open FCoE In real-world implementations, SAN for over a dozen years. NetApp has been approach is supported by NetApp and performance is determined by a number a leader in supporting Ethernet storage, will help to accelerate broad server to of factors, including application threading, first as a NAS pioneer, and next as an early SAN connectivity and enable SANs to storage target processor speed, and disk proponent of iSCSI and now as a leader be more accessible. speeds. At typical IOP levels, processor with FCoE. Thousands of enterprises utilization is typically quite low, especially have deployed NetApp Ethernet storage As a pioneer and leader in 10GbE Storage in systems with the latest multicore that simplifies data management, protects since calendar 2006, NetApp has shipped processors and memory controllers. investments, and reduces total cost of over 18,000 10GbE target ports since ownership. NetApp Ethernet Storage first offering them in 2006 and has seen In contrast, CNA solutions based on systems deliver industry-leading efficiency the adoption bloom to an increase of over proprietary offload engines may show through an extensive suite of virtualized 100 percent in total shipments over the little if any improvement in IOPS on new tools enabling administrators to store the last nine months. As customers look to hardware platforms (although processor maximum amount of data for the lowest consolidate data center resources, they can utilization may improve). possible cost, retain data on disk for longer increase value and efficiency by sharing the bandwidth of a high-performance 10GbE- Ease of Use periods of time, and reduce data center power, cooling and space costs. based unified network infrastructure. By using the native storage initiators The increased bandwidth and added integrated into the OS, Intel Ethernet I/O unification requires storage solutions enhancements to support FCoE make 10 Gigabit Server Adapters make it easy to that support both file and block protocols 10GbE the ideal interconnect for modern connect any server to the SAN without the and take full advantage of the benefits of data centers. need for complicated proprietary solutions. Ethernet. NetApp storage systems work Standardizing on Intel Ethernet 10 Gigabit as a “unification engine” supporting NFS Server Adapters delivers a number of and CIFS, iSCSI, and FCoE in the same benefits to enterprise IT: system, as well as leverages traditional FC storage investments. • Trusted and validated OS-based initiators With NetApp NAS solutions, customers • Single interface to configure an can solve the data management challenges adapter for LAN or SAN traffic of scalability, availability, and consolidation of existing Windows*, Linux*, and UNIX* • Plug-and-Play with existing file servers. NetApp NAS solutions offer management tools flexible provisioning, reliable backup, • No proprietary software to install archiving, business continuity, and virtualization.

6 Unified Networking for 10 Gigabit Ethernet Conclusion 10GbE unified networking provides a simple, flexible, and well-understood fabric for today’s virtualized data centers and lays the groundwork for new computing models, including Cloud Computing, which will deliver more intelligent, responsive data centers and greater business agility. With their long histories of leadership in Ethernet networking and storage, respectively, Intel and NetApp are two companies that are helping IT organizations transition to a unified data center fabric on 10GbE. Both companies are early supporters of FCoE, with NetApp being the first storage vendor to support native FCoE storage and Intel the first Ethernet adapter vendor to support Open FCoE. Proven and reliable products, standards leadership, and multi-protocol architectures are a few key examples of why Intel and NetApp will be instrumental in leading the way to the next-generation data center infrastructure.

Unified Networking for 10 Gigabit Ethernet 7 1 Source: Intel estimates as of January 2010. Performance comparison using SPECjbb*2005 bops (business operations per second). Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. For detailed calculations, configurations and assumptions refer to the legal information slide in backup. 2 IDC, “Server Workloads Forecast,” 2009. 3 IDC, “The Internet Reaches Late Adolescence,” December 2009. 4 8x Network: 800 terabytes per second of IP traffic estimated on internal Intel analysis “Network Supply/Demand 2010-2020” forecast; 16x Storage: 60 exabytes of data stored from Barclays Capital “Storage Bits” September 2009, extrapolation by Intel for 2015; 20x Compute: Intel internal long-range planning forecast. Extrapolated to one billion virtual servers using one virtual machine per core 5 IDC, WW Storage Systems Tracker, December 2009. 6 Test Configuration: Iometer v. 2006.7.27, number of managers = 1, number of workers/manager = 30 , (total number of workers) = 30, number of LUNs = 30 , number of outstanding I/Os = 50 , IO size = 512B. 10 iSCSI targets with 3 LUNs per target. Target is StarWind Enterprise configured with RAM disk. SUT: Supermicro 6026T-NTR+ , Intel® Xeon® processor W 5680 (12 M Cache, 3.33 GHz, 6.40 GT/s Intel® QPI), 24 GB DDR3, Windows Server* 2008 R2 x64 Network Configuration: Cisco Nexus* 5020, Intel 82599EB 10 Gigabit Ethernet Controller connected @ 10 Gbps.

Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit www.intel.com/performance/resources/limits.htm

© 2010 Intel Corporation. All rights reserved. Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and other countries.

*Other names and brands may be claimed as the property of others.

0610/BY/MESH/PDF 323964-001US