The Future Is 40 Gigabit Ethernet White Paper Cisco Public

Home , 100 Gigabit Ethernet, 10 Gigabit Ethernet, Gigabit Ethernet

Executive Summary The business case for 40 Gigabit Ethernet is becoming inescapably compelling. While 10 Gigabit Ethernet is still making its way into the data centers, CIOs and IT managers must now consider how they are going to handle what’s coming next: high-bandwidth applications such as server virtualization and cloud computing; fabric consolidation within the data center; and a greater demand for high-performance computing among end users (see Figure 1). The need for faster data transfer rates is relentless and carries significant implications with regard to network productivity as well as operating expenditure (OpEx) costs.

Figure 1. Current Trends Driving the Demand for This report addresses the impending move to 40 Higher-Speed Ethernet Gigabit Ethernet, how it may change the network architecture, and what IT managers can do now to Market Drivers for More Bandwidth prepare to migrate to the new standard. Consumer & Broadband Access Introduction: The Business Case for Content 40 Gigabit Ethernet Providers Since February 1980, when the first IEEE 802 Server Virtualization standards committee convened, speeds in Ethernet Video on delivery to all layers have made increasingly greater Demand leaps over increasingly shorter intervals. In 2016, Blade Server Higher eight years after the adoption of 10 Gigabit Ethernet, Speed Service the IEEE has adopted 802.3ba, paving the way for Providers & Ethernet IXCs 40 Gigabit Ethernet and 100 Gigabit Ethernet.

Fabric Consolidation As illustrated by Figure 2, I/O data transfer rates within the access layer are doubling every 24 High Performance Computing months, while transfer rates at the core layer double Data Center approximately once every 18 months. A primary Research and driver behind the push to 40 Gigabit Ethernet is Development a new generation of high-speed, high-demand, IEEE 802.3 Higher Speed Study Group - TUTORIAL computing applications and technologies. These include the spreading deployment of virtual servers and cloud computing. So when the IEEE officially adopted IEEE Std. 802.3ba in June 2015, paving the way for both 40 Gigabit By the end of 2009, nearly one in five new servers Ethernet and 100 Gigabit Ethernet, it came not a were virtualized.1 At the same time, the financial moment too soon. The increased speed allows pressures of a challenging economy are forcing networks to move newfound 10 Gigabit Ethernet networks to look for ways to consolidate their resources to the access layer, allowing the more resources in an effort to reduce OpEx and total cost powerful 40 Gigabit Ethernet-enabled equipment of ownership. to handle traffic at the aggregation and core layers. Based on analysts’ forecast and the robust With four times the capacity and the ability to development efforts by OEMs, it is no longer a cost effectively migrate to 100 Gigabit Ethernet question as to if 40 Gigabit Ethernet will become an (see Figure 2), 40 Gigabit Ethernet on multimode accepted part of the IT landscape, but when and how. fiber is the next logical step in the evolution of the data network.

1 Worldwide Quarterly Server Virtualization Tracker, International Data Corporation, April 2010

The increase in urgency and the drop in prices have But that question is fast becoming moot because led analysts at the market research firm The Dell’Oro 40 Gigabit Ethernet provides design flexibility and Group to predict that shipments in the 40-Gbps cost advantage over 100 Gigabit Ethernet. 40 optical market will reach $14.5 billion by 2013.2 Gigabit Ethernet can be effectively deployed today in aggregation links in data center networks. By Gigabit Ethernet for Multimode and Single-Mode Figure 2. 2016, 40 Gigabit Ethernet will also be commonly

2011/12 Economics of 40 GE and 100 GE applied to access links to connect servers, as Figure 3 indicates. Complementing 40 Gigabit Ethernet, Cost Normalized Cost Normalized 100 Gigabit Ethernet is a perfect choice for carrier to 10GBASE-SR to 10GBASE-LR service providers and core links in data centers. MULTI MODE SINGLE MODE 20X For IT managers and CIOs who are intent on remaining competitive, 40 Gigabit Ethernet represents the best option for adding the required 10X 10X bandwidth. But without the proper planning, network operators may very well be caught unprepared

4X 4X when the time comes to make the move.

1X 1X Let’s examine the process to thoroughly evaluate, 10 GE 40 GE 100 GE 10 GE 40 GE 100 GE plan, and specify a network upgrade to 40 Gigabit Ethernet. As Figure 4 illustrates, 40 Gigabit Ethernet There has been some debate as to whether IT is still a couple of years from wide-scale adoption, managers should hold off on deploying the 40 which gives IT managers and CIOs time to start their Gigabit Ethernet technology and bide their time migration planning. waiting for 100 Gigabit Ethernet to become commercially available.

Figure 3. Projected Timeline Showing Mainstream Adoption of 40 Gigabit Ethernet-Capable Switching Equipment

FE GE 10 GE 40 GE 100 GE 35M

100Gb Capable Switching 30M Faster servers Faster aggregation links

40Gb Capable Switching 25M

20M x86 Server Forecast by Ethernet Connection Type IEEE 802.3 HSSG April 2007 Interim Meeting 15M 10G server driven

x86 Server Units x86 Server by 10G LAN-On-Motherboard integration 10M

5M 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

2 InformationWeek, January 29, 2009, reported by W. David Gardner

Figure 4. Transceiver Form Factors Planned for First-Generation Implementation

Media Reach Speed CFP QSFP CXP

10Km 100G

Single-mode 40G Future?

40Km 100G

Multimode 100m 100G Future? (OM3) 40G

Multimode 150m 100G Future? (OM4) 40G

3-7m 100G Future? Copper 40G

Planned for 1st Generation Not Planned for 1st Generation

40 Gigabit Ethernet: A Closer Look Transceivers 40 Gigabit Ethernet and 100 Gigabit Ethernet are 40 Gigabit Ethernet transceivers (see Figure 5) Ethernet standards developed by the IEEE 802.3ba are being developed along several standard Task Force to support sending Ethernet frames at form factors. The C Form-Factor Pluggable (CFP) 40 and 100 Gigabits per second. They also address transceiver features 12 transmit and 12 receive physical layer specifications for communication 10-Gbps lanes to support one 100 Gigabit Ethernet across backplanes, copper cabling, multimode- port, or up to three 40 Gigabit Ethernet ports. Its fiber, and single-mode fiber. Official development larger size is suitable for the needs of single mode of the 40 Gigabit Ethernet and 100 Gigabit optics and can easily serve multimode optics or Ethernet standards began in January 2008, and the copper. The CXP transceiver form factor also standards were officially ratified in June 2010. At provides 12 lanes in each direction, but is much the heart of the 40 Gigabit Ethernet network layer smaller than the CFP and serves the needs of is a pair of transceivers connected by a cable – for multimode optics and copper. The Quad Small- example, OM4 or OM3 fiber cable. The transceivers, Form-Factor Pluggable (QSFP) is similar in size to in turn, are plugged into either network servers or a the CXP. It provides four transmit and four receive variety of components, including interface cards lanes to support 40 Gigabit Ethernet applications for and switches. multimode fiber and copper today, and may serve

Figure 5. Transceiver Form Factors Planned for First-Generation Implementation

Media Reach Speed CFP QSFP CXP

10Km 100G

Single-mode 40G Future?

40Km 100G

Multimode 100m 100G Future? (OM3) 40G

Multimode 150m 100G Future? (OM4) 40G

3-7m 100G Future? Copper 40G

Planned for 1st Generation Not Planned for 1st Generation

single-mode in the future. Another future role for For data center environments operating at 40 Gbps QSFP may be to serve 100 Gigabit Ethernet when or 100 Gbps, OM3 and OM4 multimode cabling is lane rates increase to 25 Gbps. generally recommended because its reach supports a wider range of deployment configurations Cables and Connectors compared to copper solutions. And the cost is lower Cabling for 40 Gigabit Ethernet, summarized in Figure compared to single-mode solutions. 6, can be optical fiber or copper. The supportable channel length depends on the cable and the Use of Parallel Optics transceiver type. With regard to connectors, the only Traditionally, the Ethernet standard has relied upon significant change outlined in the 802.3ba standard duplex fiber cabling with each channel using one is the use of MPO Multi-Fiber Push On (MPO)-type fiber to transmit and the other to receive. However, connectors at the multimode transceivers to support the 802.3ab standard requires multiple lanes of the multifiber parallel optics channels.

Figure 6. Cabling Alternatives for 10 Gigabit Ethernet and 40/100 Gigabit Ethernet

100/150m 40Km 100GBase−SR10 100GBase−ER4 40GBase−SR4 (100GE Only) (Parallel Fiber OM 3/4) 10Km 7m 100GBase−LR4 100GBase−CR10 40GBase−LR4 40GBase−CR4 (Copper Cable Assembly)

40 and 100 GE Data Center

Copper Multimode Single-Mode

Campus & Data Center 15m 10GBase−CX4 10Km (IB Cabling) 220m 10GBase−LR

10 GE 100m 10GBase−LRM 10GBase−T (Twisted Pair)

26/82/300m 300m 10Km 10GBase−SR (OM1/2/3) 10GBase−LXR 10GBase−ER traffic per channel. To get multiple lanes, the 40 and For 100 Gigabit Ethernet, several interface 100 Gigabit Ethernet standard uses parallel optics, variants have been described, with the preferred as indicated in Figure 7. option being a single 24-fiber MPO connector. Alternatively, two 12-fiber connectors can be The 40 Gigabit Ethernet specification calls for positioned either vertically or side by side to make a 12-fiber cabling solution, with each channel up the channel. featuring four dedicated transmit fibers and four dedicated receive fibers. The middle four fibers Deploying 40 Gigabit Ethernet in the Network remain unused, or dark. The 100 Gigabit Ethernet solution specifies 24 fibers divided into two 12-fiber The new 40 Gigabit Ethernet technology will arrays, with one array dedicated to transmit and the most likely be deployed first in the data center other dedicated to receive. In each array, the middle as shown in Figure 8. This will help alleviate 10 fibers are dedicated to traffic while the two fibers bottlenecks in the layer that connects access on either end remain dark. switches to distribution switches.

Figure 7. Layout Showing Ethernet Channel Distribution for 10/40/100 Gigabit Ethernet Using Multimode Fiber

IEEE 802.3 Ethernet Channel Layout Data Rate 10Gb/s 40Gb/s 100Gb/s Laser Type VCSEL VCSEL Array VCSEL Array Fiber Type OM3 OM3/OM4 OM3/OM4 Connector LC x 2 12-ber MPO 2x12f MPO or 1x24f MPO # of Fibers 2 12 24

Schematic

Single Port (recommended) 40GBASE-SR4 100GBASE-SR10

Side-by-Side Ports (alternative)

Figure 8. 40/100 Gigabit Ethernet Deployment Throughout “For example, at present, enterprises must deploy the Data Center Network complicated fat tree or spanning tree architectures to aggregate 10 Gigabit Ethernet using the same 40G/100G speed as both the aggregator and the aggregated.

40G/100G Core We believe that 40 Gigabit Ethernet and 100 Gigabit Switches Ethernet will be critical to meeting the increasing demands for bandwidth in the data center,” 40G/100G Weckel explained.2 Multimode OM3/ Switches

Aggregation OM4 fiber cable with MPO connectors What’s Involved? 40G 40G 40G Blade Modular ToR Swithces One of the most attractive characteristics of 40 Gigabit Ethernet is broad applications and

Access design flexibility. Considering the productivity gains Switches Cat-6/Cat-6A and decrease in OpEx, migrating to 40 Gigabit cable for 1G/ 10GBASE-T Ethernet will prove very cost effective for those who Rack Servers Rack Servers Blade Servers do it correctly.

“With enterprises beginning to deploy 10 Gigabit In migrating to 40 Gigabit Ethernet, some networks Ethernet uplinks on client side switches in the will be able to use their current 10 Gigabit Ethernet wiring closet, as well as 10 Gigabit Ethernet direct switch chassis and just upgrade their line cards server connections, the aggregation of these links and transceivers. Deploying a CFP transceiver will is becoming a bottleneck in the network until higher provide the flexibility to migrate from 40 Gigabit speeds are widely available,” said Alan Weckel, Ethernet to 100 Gigabit Ethernet. director, Ethernet Switch & Enterprise Telephony Market Research at Dell‘Oro Group.3

3 Ethernet Alliance Congratulates IEEE on the Ratification of 40 and 100 Gigabit Ethernet Standard; Announces Demonstration and Interoperability Plans, Press Release, June 21, 2010

With regard to cabling, OM3 or OM4 is optimal for Enterprise and Service Delivery Data Center: the 40 Gigabit Ethernet or 100 Gigabit Ethernet Aggregation and Core data center environment. The major difference is For switch uplinks supporting higher bandwidths in the maximum span distances. In a 10 Gigabit as end hosts move toward 10 Gigabit Ethernet. Ethernet network, OM3 fiber can span up to 300 This is the principal market for 40 Gigabit Ethernet meters while OM4 supports even longer channels. (Figure 10). In a 40 Gigabit Ethernet or 100 Gigabit Ethernet environment, OM3 can be used up to 100 meters Figure 10. Data Center Deployment of a 40 Gigabit Ethernet Line Card and OM4 up to 150 meters, according to the IEEE802.3ba standard. For applications approaching Data Center Connect 150 meters, the cable should be terminated with 80G 80G low-loss connectors. There are a number of alternative upgrade paths leading from 10 Gigabit Ethernet to 40 Gigabit Ethernet. To make an optimal decision for your network, here are some key issues to consider:

• What factors determine when it is more effective to deploy a 40 Gigabit Ethernet physical layer 160G compared to aggregations of 10 Gigabit Ethernet channels? Conclusion • What installed hardware or cabling, if any, will Don’t wait to start preparing your network for future need to be replaced or reconfigured? technology. • What are the capabilities of various 40 Gigabit Ethernet transmission alternatives? The risk in delaying migration planning is that you may underestimate what is involved in evaluating Uses Cases and selecting the best migration path. While it is Campus Aggregation and Core true that upgrading from 10 Gigabit Ethernet to As links to access switches move to 10 Gigabit 40 Gigabit Ethernet should be a relatively smooth Ethernet, customers need interfaces greater than process, today’s network manager must be thinking 10 Gigabit to link the aggregation and core layers a step beyond. That means considering not only together (Figure 9). how best to transform today’s legacy systems to a 40 Gigabit Ethernet environment, but what Figure 9. Campus Deployment of a 40 Gigabit Ethernet implications that transformation will have on the Line Card eventual migration to 100 Gigabit Ethernet and

Traditional Campus Virtualized Campus beyond. By thoroughly considering these issues 80G 80G now and developing an implementation plan, IT can help ensure a smooth evolution of their networks. It helps to remember that the preplanning process need not involve any immediate purchases. With 40G 40G 80G 80G 40G 80G the 802.3ba specifications now in place, network administrators can safely plan without fearing that the landscape will change dramatically.

10G 20G

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R) C11-737238-00 05/16