WHITE PAPER

The case for 25G and 100G

WHITE PAPER

More for data centers

WHITE PAPER

Contents

Introduction ...... 3

Top-of-Rack Datacenter Design ...... 4

25 Gigabit (25GE) and 50 (50GE) Standardization ...... 5

Protocol Stack ...... 6

25GE and 50GE switches ...... 7

25GE and 50GE NICs ...... 8

Available Electrical and Optical Modules ...... 8

25GE and 50GE Test Requirements ...... 9

Xena Networks 25GE and 50GE Test Solutions ...... 9

Unique Eye Diagram ...... 10

Testing Above Layer 1 ...... 11

Testing from 10 Gigabit/s to 100 Gigabit/s ...... 12

Conclusion ...... 13

www.xenanetworks.com 2

WHITE PAPER

INTRODUCTION For a number of years the upgrade path for higher speed Ethernet has been 10GE -> 40GE/100GE, which has been well accepted for many applications. However a couple of new speeds have recently been added between 10 Gigabit/s and 100 Gigabit/s: 25 Gigabit/s (25GE) and 50 Gigabit/s (50GE). As there an ongoing development of standards for Ethernet speeds higher than 100 Gigabit, it may seem a bit odd that lower speeds are not only discussed but in fact also standardized.

The drive for 25GE and 50GE comes from datacenters, where multiple 10GE connections are used to interface to the servers in the datacenters. In high density datacenters numerous 10GE “Top of Rack” (ToR) switches are required to connect to the datacenter servers. Each switch and each switch port has a cost of its own. To this adds the cost of requirements for space, maintenance, power and cooling of the switches. By introducing 25GE single lane connections instead of the 10GE connections, the datacenters will get 2.5 times more capacity per connection, while equipment cost and space, power, maintenance and cooling requirements will be reduced compared with datacenters using only 10GE to provide the same bandwidth, simply because the datacenter with 25GE connections will be built with less equipment. In addition the CPUs in datacenter servers can now be so powerful that they easily can fill up a 10GE connection. To provide more transmission capacity the next step has until now been to upgrade to 40GE, which is considerably more expensive than a 10GE connection as 40GE is based on sending the data on 4 10 Gigabit/s lanes in parallel. As an alternative to 40GE an upgrade to 25GE will be a cost effective solution because fewer lanes are used. The general expectation in the industry is that 25GE deployment will grow rapidly in the coming years. Analysts forecast that by 2020 25GE and 50GE will together have a market share of approx. 25% in the high speed NIC market (from 10GE to 100GE) (based on data from Crehan Research).

Using 25 Gigabit lanes for transport of Ethernet is not new: Most current 100GE implementations use four 25 Gigabit lanes to transport the Ethernet signals. Therefore the 25GE standardization bodies have standardized a 25 Gigabit lane based on the 100GE technology.

The current standardization of 50GE is based on using two 25 Gigabit lanes. This will be an alternative to the current 40G implementations, which use four 10 Gigabit lanes to transport the Ethernet signal. The reduction of lanes will reduce the cost of the network equipment needed in the datacenter, which leads to reduced CAPEX and OPEX.

In the future the Ethernet upgrade path may change to 10GE -> 25GE -> 50GE -> 100GE. In any case it will be easy and cost effective to upgrade datacenters from multiple 25GE lanes to 50GE or 100GE networks.

www.xenanetworks.com 3

WHITE PAPER

Figure 1: Current and future high speed Ethernet upgrade path

Top-of-Rack Datacenter Design A popular datacenter design is the ”Top-of-Rack” (ToR) design: Each server rack is equipped with one Ethernet switch (or two for redundancy). The switch could be placed in the top of the rack, which has given the design its name; the switch may however be placed elsewhere in the rack. Another design is the “End-of-Rack” design, where one Ethernet switch, placed outside the server racks, connects to all the servers in the datacenter. In both cases with the increased need for bandwidth you will either need more switch ports or more bandwidth per port. Going from 10 GE ports to 25 GE ports will provide the increased bandwidth in a more cost effectively compared with adding 10 GE ports.

www.xenanetworks.com 4

WHITE PAPER

Figure 2: Top-of-Rack datacenter design

25 GIGABIT ETHERNET (25GE) AND 50 GIGABIT ETHERNET (50GE) STANDARDIZATION (25GE) is a new standard for Ethernet connectivity in a datacenter environment, developed by the IEEE 802.3by 25 Gb/s Ethernet Task Force. The work was started in the IEEE standards body with a successful Call for Interest (CFI) in July, 2014. In the CFI the 25 Gigabit Ethernet Motivation was outlined:

• Provide cost optimized server capability beyond 10G • Provide a 25Gbit/s MAC rate that: o Leverages single-lane 25Gbit/s physical layer technology developed to support 100GE o Maximize efficiency of server to access switch interconnect The IEEE 802.3by standard is based on technology defined for implemented as four 25-Gbit/s lanes. As the main application of 25GE are connections inside a datacenter and inside a server rack in the data center the IEEE 802.3by standard focus on very short range connections. The IEEE 802.3by standard objectives (approved November 2014) included:

• Support a MAC data rate of 25 Gb/s • Define a single‐lane 25 Gb/s PHY for operation over a backplane • Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 3m • Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 5m • Define a single-lane 25 Gb/s PHY for operation over MMF consistent with IEEE P802.3bm Clause 95

The IEEE 802.3by standard was approved in June 2016.

In addition to the IEEE 802.3by 25 Gb/s Ethernet Task Force an industry consortium (the 25G Ethernet Consortium) was formed in July 2014 by Arista, Broadcom, , Mellanox and Microsoft to support the specification of single

www.xenanetworks.com 5

WHITE PAPER

lane 25 Gigabit/s Ethernet and dual lane 50 Gigabit/s Ethernet technology. The 25G Ethernet Consortium draft specification was completed in September 2015.

The following table is based on a table in the 25G Ethernet Consortium specification showing a summary of the two solutions:

Port Mode Lane Speed per Lane PCS Lane Bonding Auto- Count (GHz) negotiation A. Single Lane 25G 1 25.78125 N/A Yes B. Two lane 50G 2 25.78125 MLD4 or RS FEC (see Yes end of next section) Table 1: 25G and 50G solutions summary

In 2016 IEEE initiated work on defining 50 Gigabit Ethernet by setting up the IEEE 802.3cd task force. The objective of the task force is however to define 50 Gigabit Ethernet over a single lane instead of the dual lane solution as outlined in the 25G Ethernet Consortium specification as a single lane technology is expected to be more cost effective than the dual lane technology. To achieve this it will be necessary to use a Pulse Amplitude Modulation technique PAM-4, where 4 different pulse amplitudes are used to transfer the information. The IEEE 802.3cd task force target is to complete the 50 Gigabit Ethernet single lane standard in the 2018-2019 time frame.

PROTOCOL STACK The Ethernet protocol stack is based on the seven-layer OSI model of computer networking:

Layer 7 Application layer Layer 6 Presentation layer Layer 5 Session layer Layer 4 Transport layer Layer 3 Network layer Layer 2 Data link layer Layer 1 Physical layer Table 2: The OSI model of computer networking

The 25G Ethernet Consortium specification for 25 Gigabit Ethernet (25GE) and 50 Gigabit Ethernet (50GE) addresses Layer 1, which contains the following sublayers:

www.xenanetworks.com 6

WHITE PAPER

Sublayer Functions • PCS (Physical Coding auto-negotiation • determine a functional link has been established (Link Sublayer) Training) • coding (64b/66b encoding in the case of 25 Gigabit and 50 Gigabit Ethernet) FEC (Forward Error • optional for 25 and 50 Gigabit Ethernet Correction Sublayer) • the 25G Ethernet Consortium specification lists the following possible FECs: BASE-R FEC “Fire code” (IEEE 802.3 Clause 74) o o Reed Solomon FEC (IEEE 802.3 Clause 91) o No FEC In addition to the FECs above IEEE 802.3by includes Reed Solomon FEC (IEEE 802.3 Clause 108) auto- negotiation can be used to determine which FEC is used • PMA (Physical Medium framing • octet synchronization/detection Attachment Sublayer) • scrambling/descrambling PMD (Physical Medium • the for the physical medium Dependent Sublayer) Table 3: Layer 1 sublayers

The 25G Ethernet Consortium specification in many cases refers to clauses in IEEE 802.3 for other rates to define the layer 1 functions for 25GE and 50GE.

For 50GE a Multi-Lane Distribution (MLD) technique is required to distribute data into two lanes. When RS FEC is used for the 50GE solution MLD is handled by the interface between the PCS sublayer and the FEC sublayer. Otherwise MLD4, defined for 40G systems, is used as a part of the functionality in the PMA sublayer. In both cases the data from the upper layers are distributed into 4 “logical” (or virtual) 12.89063 Gigabit/s lanes, two of each sent on the physical 25.78125 Gigabit/s lanes. Alignment markers are inserted into the data stream to handle the multi-lane distribution, one unique alignment marker for each virtual lane.

25GE AND 50GE SWITCHES Several vendors offer switches with 25GE and 50GE capabilities, typically implemented as very flexible solutions supporting 100GE, 50GE, 40GE, 25GE and in most cases also 10GE from one port. Some examples are:

Product Maximum number of ports 100GE 50GE 40GE 25GE 10GE Arista 7260CX-64 64 128 64 256 258 Broadcom StrataXGS® Tomahawk BCM56960 Series 32 64 64 128 N/A Broadcom StrataXGS® Tomahawk II BCM56970 Series 64 128 128 256 N/A Cavium XPliant® CNX88091 32 64 32 128 128 Cisco Nexus 3232C 32 64 32 128 128 Mellanox SN2700 32 64 32 64 64 Table 4: Switches supporting 25GE and 50GE

www.xenanetworks.com 7

WHITE PAPER

25GE AND 50GE NICS A Network Interface Controller (NIC) (or network interface card, network adapter) connects a computer (e.g. servers in a datacenter) to a network. NIC products supporting 25GE and 50GE include:

• Broadcom NetXtreme® C-Series: Dual port 10/25G Ethernet • Mellanox ConnectX® -4 Lx EN 10/25/40/50 Gigabit Ethernet Adapter: Single and dual port versions • QLogic® FastLinQ™ 45000 Series Intelligent Ethernet Adapters: will include adapters that support 10GE, 25GE, 40GE, 50GE, 100GE

AVAILABLE ELECTRICAL AND OPTICAL MODULES For deployment of 25GE vendors like Finisar, Avago and Innolight offer SFP28 modules, which support one lane with bandwidth up to 28 Gigabit/s. SFP28 modules have the same form factor as SFP and SFP+ modules. Available modules include:

• 25 Gigabit/s copper o Direct Attach Cables Cobber twinax cables (DACs) terminated with SFP28 modules in each end, 0.5 to 5 meters in length. • 25 Gigabit/s fiber optics o 850 nm SR supporting distances up to 100 meters on OM4 multimode cable or distances up to 70 m OM3 multimode cable o 1310 nm LR supporting single mode fiber and distances from 5 to 20 km Another module that will be relevant for 25GE is QSFP28, which supports four 28 Gigabit/s lanes in the QSFP+ form factor. The QSFP28 module was originally designed to support 100 Gigabit/s connections. However for datacenters with 25GE connections, direct attach cobber twinax cables (DACs) connecting a QSFP28 100GbE port to four independent 25GE SFP28 connections are available. The DACs will have a QSFP28 module in one end breaking out to four cables, each terminated with a SFP28 module.

This means high capacity Ethernet switch (ToR) with 100 Gigabit/s ports can interface to 4 servers with 25 Gigabit/s ports through one 100 Gigabit/s port in the ToR. For optical interfaces 100 Gigabit/s QSFP28 to 4 x 25Gigabit/s SFP28 break-out Active Optical Cables (AOC) are available. In addition passive optical breakout cables are available splitting four 25 Gigabit/s optical signals from a QSFP28 with MPO connector into four individual cable pairs to be connected to SFP28 optical modules with LC connectors.

Figure 4: 100 Gigabit/s QSFP28 to 4 x 25 Gigabit/s SFP28 DAC electrical break- Figure 3: 100 GE to 4*25 GE break-out cable out cable

www.xenanetworks.com 8

WHITE PAPER

Figure 5: 100 Gigabit/s QSFP28 to 4 x 25 Gigabit/s SFP28 break-out Active Optical Cable (AOC)

Figure 6: MTP to 4 dual LC passive optical break-out cable

25GE AND 50GE TEST REQUIREMENTS Testing Ethernet systems is required to ensure the proper functioning and quality of the systems. This of course also applies to the new 25GE and 50GE solutions as they are developed and deployed. To test the layer 1 functions defined in the 25G Ethernet Consortium specification verification of functions like auto-negotiation, Link Training and FEC must be performed.

In addition an eye diagram of the signal should be checked to identify physical layer issues, which may be caused e.g. by noise and signals that are too high, too low, too long or too short.

As ToR switches typically are implemented as very flexible products that support 100GE, 50GE, 40GE, 25GE and 10GE from one port it is of course also important to test Ethernet at all rates supported by the port under is test.

XENA NETWORKS 25GE AND 50GE TEST SOLUTIONS The Xena Loki-100G-5S-1P 5-speed dual-media test module provides powerful testing capabilities to test and verify the layer 1 functions defined for 25GE and 50GE.

www.xenanetworks.com 9

WHITE PAPER

Figure 7: The powerful and versatile Xena Loki-100G-5S-1P 5-speed dual-media test module

You can set the Loki-100G-5S-1P to perform auto-negotiation so when connected to a Device Under Test (DUT) the Loki-100G-5S-1P can verify that the DUT reacts correctly to the auto-negotiation process. The Loki-100G-5S-1P will also do Link Training with the DUT, verifying the correct functioning of this feature. Signals can be configured with or without FEC, verifying that the DUT use the correct FEC. Loki-100G-5S-1P supports the FECs specified by IEEE 802.3by and by the 25G Ethernet Consortium:

• BASE-R FEC “Fire code” (IEEE 802.3 Clause 74) • Reed Solomon FEC (IEEE 802.3 Clause 91) • Reed Solomon FEC (IEEE 802.3 Clause 108)

Unique Eye Diagram The Loki-100G-5S-1P includes the “eye diagram”, a unique feature for analyzing signal quality. For this the Loki-100G- 5S-1P controls and monitors the link at the physical level and creates bit-error-rate (BER) eye diagrams, estimates the link BER from the vertical and horizontal BER bathtub curves and controls the PHY tuning in the transmit and the receive directions. The eye-data table provides an estimate of several parameters of the eye, including width, height and jitter.

Figure 8: The unique eye diagram you get with Xena 100G test modules

www.xenanetworks.com 10

WHITE PAPER

Testing Above Layer 1 Ethernet testing with the Loki-100G-5S-1P is of course not limited to the layer 1 tests described in the previous sections. Advanced layer 2 and layer 3 test scenarios can be performed using the free test applications for the Loki- 100G-5S-1P:

• ValkyrieManager test software is used to configure and generate streams of Ethernet traffic between Xena test equipment and Devices Under Test (DUTs) and analyze the results.

• Valkyrie2544 offers full support for the 4 test-types specified in RFC2544 o Throughput o Latency o Frame loss o Back-to-back frames o Jitter (Frame Delay Variation) is also supported

• Valkyrie3918 makes it easy to create, edit and execute all test-types specified in RFC 3918. RFC3918 describes tests for measuring and reporting the throughput, forwarding, latency and Group Management Protocol (IGMP) group membership characteristics of devices that support IP multicast protocols.

• Valkyrie2889 is an application for benchmarking the performance of Layer 2 LAN switches. The following RFC 2889 test types are supported: o All Throughput and Forwarding rate tests (both Fully and Partially meshed) o Congestion Control o Address Caching Capacity o Address Learning Rate o Broadcast Frame Forwarding and Latency o Forward Pressure and Maximum Forwarding Rate

• Valkyrie1564 provides full support for both the configuration and performance test types described in Y.1564 for complete validation of Ethernet service-level agreements (SLAs) in a single test

Another free application for the Loki-100G-5S-1P (and all other Xena test modules):

• ValkyrieCLI - a powerful and easy-to-use command-line-interface (CLI) scripting API that makes test automation easier for test engineers.

www.xenanetworks.com 11

WHITE PAPER

Testing from 10 Gigabit/s to 100 Gigabit/s

Figure 9: Loki-100G-5S-1P can test four 25GE ports through a 100 Gigabit/s QSFP28 transceiver and a break out cable terminated with four SFP28 or two 25GE ports connected directly to 25 Gigabit/s transceivers in the Loki-100G-5S-1P

In addition 25GE and 50GE testing Loki-100G-5S-1P also supports test of 100GE, 40GE and 10GE, so both the new and the existing Ethernet rates in the range from 10 Gigabit/s to 100 Gigabit/s can be tested. This is needed to test and verify 25GE and 50 GE capable switch ports. The unique test module lets users dynamically choose between two different physical transceiver cages and form factors. The first is a single QSFP28/QSFP+ transceiver cage, and the second is two SFP28/SFP+ transceiver cages.

The unique combination of five different Ethernet network speeds and multiple physical optical transceiver form factors makes the Loki-100G-5S-1P a versatile solution for performance and functional testing of network infrastructure and Ethernet equipment such as taps, switches, routers, NICs, packet-brokers, and backhaul platforms.

www.xenanetworks.com 12

WHITE PAPER

CONCLUSION New Ethernet data rates – 25GE and 50GE - are coming, driven by the need for cost effective expansion of datacenter capacity to fulfil the huge increase in bandwidth requirements from cloud based solutions. Therefore the deployment of 25GE and 50GE is expected to grow rapidly in the coming years.

Datacenter switches with 25GE and 50 GE capabilities are typically implemented as very flexible solutions that support 100GE, 50GE, 40GE, 25GE and 10GE from one port. To test the product all rates need to be tested.

With the Xena Loki-100G-5S-1P 5-speed dual-media test module the user gets powerful and versatile Ethernet test capabilities up to layer 3 to test and verify both new and existing Ethernet rates in the range from 10 Gigabit/s to 100 Gigabit/s: 10GE, 25GE, 40GE, 50GE and 100GE.

www.xenanetworks.com 13