The Case for 25G and 100G
Total Page:16
File Type:pdf, Size:1020Kb
WHITE PAPER The case for 25G and 100G WHITE PAPER More bandwidth for data centers WHITE PAPER Contents Introduction ........................................................................................................................................................................ 3 Top-of-Rack Datacenter Design .......................................................................................................................... 4 25 Gigabit Ethernet (25GE) and 50 Gigabit Ethernet (50GE) Standardization ................................................................... 5 Protocol Stack ..................................................................................................................................................................... 6 25GE and 50GE switches .................................................................................................................................................... 7 25GE and 50GE NICs ........................................................................................................................................................... 8 Available Electrical and Optical Modules ............................................................................................................................ 8 25GE and 50GE Test Requirements .................................................................................................................................... 9 Xena Networks 25GE and 50GE Test Solutions .................................................................................................................. 9 Unique Eye Diagram ........................................................................................................................................ 10 Testing Above Layer 1 ...................................................................................................................................... 11 Testing from 10 Gigabit/s to 100 Gigabit/s ........................................................................................................ 12 Conclusion ........................................................................................................................................................................ 13 www.xenanetworks.com 2 WHITE PAPER INTRODUCTION For a number of years the upgrade path for higher speed Ethernet has been 10GE -> 40GE/100GE, which has been well accepted for many applications. However a couple of new speeds have recently been added between 10 Gigabit/s and 100 Gigabit/s: 25 Gigabit/s (25GE) and 50 Gigabit/s (50GE). As there an ongoing development of standards for Ethernet speeds higher than 100 Gigabit, it may seem a bit odd that lower speeds are not only discussed but in fact also standardized. The drive for 25GE and 50GE comes from datacenters, where multiple 10GE connections are used to interface to the servers in the datacenters. In high density datacenters numerous 10GE “Top of Rack” (ToR) switches are required to connect to the datacenter servers. Each switch and each switch port has a cost of its own. To this adds the cost of requirements for space, maintenance, power and cooling of the switches. By introducing 25GE single lane connections instead of the 10GE connections, the datacenters will get 2.5 times more capacity per connection, while equipment cost and space, power, maintenance and cooling requirements will be reduced compared with datacenters using only 10GE to provide the same bandwidth, simply because the datacenter with 25GE connections will be built with less equipment. In addition the CPUs in datacenter servers can now be so powerful that they easily can fill up a 10GE connection. To provide more transmission capacity the next step has until now been to upgrade to 40GE, which is considerably more expensive than a 10GE connection as 40GE is based on sending the data on 4 10 Gigabit/s lanes in parallel. As an alternative to 40GE an upgrade to 25GE will be a cost effective solution because fewer lanes are used. The general expectation in the industry is that 25GE deployment will grow rapidly in the coming years. Analysts forecast that by 2020 25GE and 50GE will together have a market share of approx. 25% in the high speed NIC market (from 10GE to 100GE) (based on data from Crehan Research). Using 25 Gigabit lanes for transport of Ethernet is not new: Most current 100GE implementations use four 25 Gigabit lanes to transport the Ethernet signals. Therefore the 25GE standardization bodies have standardized a 25 Gigabit lane based on the 100GE technology. The current standardization of 50GE is based on using two 25 Gigabit lanes. This will be an alternative to the current 40G implementations, which use four 10 Gigabit lanes to transport the Ethernet signal. The reduction of lanes will reduce the cost of the network equipment needed in the datacenter, which leads to reduced CAPEX and OPEX. In the future the Ethernet upgrade path may change to 10GE -> 25GE -> 50GE -> 100GE. In any case it will be easy and cost effective to upgrade datacenters from multiple 25GE lanes to 50GE or 100GE networks. www.xenanetworks.com 3 WHITE PAPER Figure 1: Current and future high speed Ethernet upgrade path Top-of-Rack Datacenter Design A popular datacenter design is the ”Top-of-Rack” (ToR) design: Each server rack is equipped with one Ethernet switch (or two for redundancy). The switch could be placed in the top of the rack, which has given the design its name; the switch may however be placed elsewhere in the rack. Another design is the “End-of-Rack” design, where one Ethernet switch, placed outside the server racks, connects to all the servers in the datacenter. In both cases with the increased need for bandwidth you will either need more switch ports or more bandwidth per port. Going from 10 GE ports to 25 GE ports will provide the increased bandwidth in a more cost effectively compared with adding 10 GE ports. www.xenanetworks.com 4 WHITE PAPER Figure 2: Top-of-Rack datacenter design 25 GIGABIT ETHERNET (25GE) AND 50 GIGABIT ETHERNET (50GE) STANDARDIZATION 25 Gigabit Ethernet (25GE) is a new standard for Ethernet connectivity in a datacenter environment, developed by the IEEE 802.3by 25 Gb/s Ethernet Task Force. The work was started in the IEEE standards body with a successful Call for Interest (CFI) in July, 2014. In the CFI the 25 Gigabit Ethernet Motivation was outlined: • Provide cost optimized server capability beyond 10G • Provide a 25Gbit/s MAC rate that: o Leverages single-lane 25Gbit/s physical layer technology developed to support 100GE o Maximize efficiency of server to access switch interconnect The IEEE 802.3by standard is based on technology defined for 100 Gigabit Ethernet implemented as four 25-Gbit/s lanes. As the main application of 25GE are connections inside a datacenter and inside a server rack in the data center the IEEE 802.3by standard focus on very short range connections. The IEEE 802.3by standard objectives (approved November 2014) included: • Support a MAC data rate of 25 Gb/s • Define a single-lane 25 Gb/s PHY for operation over a printed circuit board backplane • Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 3m • Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 5m • Define a single-lane 25 Gb/s PHY for operation over MMF consistent with IEEE P802.3bm Clause 95 The IEEE 802.3by standard was approved in June 2016. In addition to the IEEE 802.3by 25 Gb/s Ethernet Task Force an industry consortium (the 25G Ethernet Consortium) was formed in July 2014 by Arista, Broadcom, Google, Mellanox and Microsoft to support the specification of single www.xenanetworks.com 5 WHITE PAPER lane 25 Gigabit/s Ethernet and dual lane 50 Gigabit/s Ethernet technology. The 25G Ethernet Consortium draft specification was completed in September 2015. The following table is based on a table in the 25G Ethernet Consortium specification showing a summary of the two solutions: Port Mode Lane Speed per Lane PCS Lane Bonding Auto- Count (GHz) negotiation A. Single Lane 25G 1 25.78125 N/A Yes B. Two lane 50G 2 25.78125 MLD4 or RS FEC (see Yes end of next section) Table 1: 25G and 50G solutions summary In 2016 IEEE initiated work on defining 50 Gigabit Ethernet by setting up the IEEE 802.3cd task force. The objective of the task force is however to define 50 Gigabit Ethernet over a single lane instead of the dual lane solution as outlined in the 25G Ethernet Consortium specification as a single lane technology is expected to be more cost effective than the dual lane technology. To achieve this it will be necessary to use a Pulse Amplitude Modulation technique PAM-4, where 4 different pulse amplitudes are used to transfer the information. The IEEE 802.3cd task force target is to complete the 50 Gigabit Ethernet single lane standard in the 2018-2019 time frame. PROTOCOL STACK The Ethernet protocol stack is based on the seven-layer OSI model of computer networking: Layer 7 Application layer Layer 6 Presentation layer Layer 5 Session layer Layer 4 Transport layer Layer 3 Network layer Layer 2 Data link layer Layer 1 Physical layer Table 2: The OSI model of computer networking The 25G Ethernet Consortium specification for 25 Gigabit Ethernet (25GE) and 50 Gigabit Ethernet (50GE) addresses Layer 1, which contains the following sublayers: www.xenanetworks.com 6 WHITE PAPER Sublayer Functions • PCS (Physical Coding auto-negotiation •