The Case for 25 and 50 Gigabit Ethernet

GIGABIT ETHERNET GIGABIT THE CASE FOR But u But environment. datacenter for which Ethernet, 10 Gigabit servers in the data centers.The new CPUs can easilysupport data rates higher than the and fully utilize the high capacity of the new generations CPUs of being used in the This has created a demand for cost hungry for bandwidth. Xena need, this meet To that support the new speeds. equipment test need will centers data To evaluate and benchmark the performance of any new 25GE and 50GE equipment, Gigabit Ethernet solutionsstandardized and ready for deployment. This is why the industry has pushed to get cost effective 25 tag price T OVERVIEW five speedswith just one testmodule represents a significant cost saving. 1P he demand for cloud- Xena Networks Networks Xena

whic

h can generate traffic at 100GE/50GE /40GE/25GE/ 100GE/50GE at traffic generate can h

due to the capital expenditure and power consumption.

–

Global Price/PerformanceLeaders in Gig

based solutions

has launched a new 5

a long time has been used for connections in the pgrading to pgrading - deliver extra deliver can that solutions effective

has grown has 25 AND 50 AND 25

40 Gigabit Ethernet 40 Gigabit - speed test module – module test speed

rapidly,

leaving many many leaving

Gigabit Ethernet and 50 and Ethernet Gigabit 10GE. 10GE. abit abit

comes with an extra extra an with comes

E Being able to test all test all to able Being thernet

the the

datacenters datacenters Loki

Testing Testing

bandwidth 100G - 5S – www.xenanetworks.com -

WHITE PAPER 50GE

it easy to test the speeds. the new test to easy it rates Ethernet new for need the drives optimization Datacenter

. Xena makes makes Xena .

–

25GE and and 25GE

WHITE PAPER

Xena Networks Networks Xena Conclusion Xena Networks 25GE and 50GE Test Solutions 50GE and 25GE Networks Xena 25GE and 50GE Test Requirements Test 50GE and 25GE Modules Optical and Electrical Available NICs 50GE and 25GE switches 50GE and 25GE Protocol Stack Standardization (50GE) Ethernet 50Gigabit and (25GE) Ethernet 25 Gigabit Introduction OVERVIEW Contents

Testing from 10 Gigabit/s to 100 Gigabit/s 100 to 10Gigabit/s from Testing Testing Above Layer 1 Unique Eye Diagram Eye Unique Top - of - Rack Datacenter Design Datacenter Rack –

......

Global Price/PerformanceLeaders in Gig ......

The Case for Case The ......

......

25 and 50 Gigabit Ethernet 50Gigabit and 25

......

abit abit E thernet

Testing Testing

...... – www.xenanetworks.com

10 13 12 11 10

9 8 5 8 7 6 4 3 1

WHITE PAPER

Xena Networks Networks Xena networks. 100GE or 50GE to lanes 25GE multiple from datacenters upgrade to effective cost and easy be will it case added been recently have speeds new of couple a However applications. many for accepted well been has which 40GE/100GE, For a number of years the upgrade path for higher speed Ethernet has been 10GE - I In the future the Ethernet upgrade path may change to 10GE - the datacenter, which leads reduced to CAPEX and OPEX. Ethernet signal. The reduction of lanes will reduce theof cost the network equipment nee the transport to lanes 10Gigabit four use which implementations, 40G current the to alternative The current standardiz bodies havestandardized a 25 Gigabit lane based on the 100GE technology. use four 25Gigabit lanes to transport the Ethernet signals.Therefore the 25GE standardization trans for lanes Gigabit 25 Using to 100GE) (based from data on Crehan Research). 10GE (from market NIC speed high the in 25% approx. of share market a have together will 50GE and 202025GE by that forecast Analysts years. coming the in rapidly will grow nt deployme 25GE effectivesolution becausefewer lanes are used. The general expectation in the industry is that cost a will be 25GE to upgrade an 40GE to alternative As an parallel. in lanes 4 10Gigabit/s 10GE a than expensive more considerably is provide more transmission capacity the next step has until now been to upgrade to 40GE, which datacenter servers can now so be powerful that they easilycan fill upa 10GE connection. To equipmen less with built be will connections 25GE with datacenter compared with datacenters using only 10GE to providesame the bandwidth, simply because the equipment cost and space, power, maintenance and cooling requirements will be reduced conne per capacity more times 2.5 get will datacenters the connections, 10GE the of instead connections lane single 25GE introducing By switches. the of cooling and power port switch each and switch Each servers. datacenter the to connect to required are switches (ToR) Rack” to interface to the servers in the datacenters. In high density datacenters numerous10GE “Top of used are connections 10GE multiple where datacenters, from comes 50GE and 25GE for drive The fact also standardized. higher than 100 Gigabit, it seem may a bit odd that lower speeds are not only discussed but in and 50 Gigabit/s (50GE). As there an ongoing development of standards for Ethernet speeds N

TRODUCTION –

has a cost of its own. To this adds the cost of requirements for space, maintenance, Global Price/PerformanceLeaders in Gig

ation of 50GE is based on using two 25 Gigabit lanes. This will be an an be will This lanes. 25 Gigabit two using on based is 50GE of ation

port of Ethernet is not new: Most current 100GE implementations between 10 Gigabit/s and 100 Gigabit/s: 25 Gigabit/s ( Gigabit/s 25 100Gigabit/s: and 10Gigabit/s between

connection as 40GE is based on sending the data on

abit abit

E thernet > 25GE > 25GE t. In addition the CPUs in

Testing Testing

- > 50GE - > 50GE

–

ction, while while ction, www.xenanetworks.com > 100GE.> In any

> 25GE) 25GE) ded in

WHITE PAPER

Xena Networks Networks Xena with adding 10 GE ports. 10GE adding with ports to 25 GE ports will provide the increased bandwidth in a more cost effectively c bandwidth you will either need more switch ports or more bandwidth per port. Going from 10 GE for need increased the with cases both In datacenter. the in racks, servers connects the to all Another design is the “End- rack. the in elsewhere placed be however may switch the name; its design the given has which one Ethernet switch (or two for redundancy). The s A popular datacenter design is the ”Top- Top Figure 1: Current andfuture high speed Ethernet upgrade path

- of - – Rack Datacenter Design

Global Price/PerformanceLeaders in Gig

of - Rack” design, where one Ethernet switch, placed outside the server of -

Rack” (ToR) design: Each server rack is equipped with with equipped is server rack Each design: (ToR) Rack”

witch could be placed in the top of the rack, abit abit E thernet

Testing Testing

– www.xenanetworks.com

ompared

WHITE PAPER

Xena Networks Networks Xena was outlined: Motivation Ethernet Gigabit in the IEEE standards body with a successfulCall for Interest (CFI) in July, 2014. In the CFI 25 the develop environment, 25 Gigabit Ethernet (25GE) is a new standard for Ethernet connectivity in a datacenter STANDARDIZATION 25 GIGABIT ETHERNET 50GIGABI AND (25GE) Figure 2: Top- The IEEE 802.3by standard was approved in June 2016. connections. The IEEE 802.3by standard 802.3by IEEE The connections. inside a server rack in the data center the IEEE 802.3by standard focus on very short range as four 25 The IEEE 802.3by standard is based implemented on technology defined Ethernet for 100 Gigabit

• • • • • • •

Clause 95 Clause Define a single a Define 5m least at to up lengths with cables, axial single a Define a single a Define single a Define that: rate MAC 25Gbit/s a Provide capabilit server optimized cost Provide Support aMAC rate data of 25 Gb/s - xial cables, with lengths up to at least 3m least at to up lengths with cables, xial Gbit/s lanes. As the main application of 25GE are connections inside a datacenter and and datacenter a inside connections are 25GE of application main the As lanes. Gbit/s –

Global Price/PerformanceLeaders in Gig o o

- Rack datacenter design Maximize efficiency of server to access switch interconnect switch access to server of efficiency Maximize 100GE single Leverages

ed by the IEEE 802.3by 25 Gb/s Ethernet Task Force. The work was started started was work The Force. Task Ethernet Gb/s 25 802.3by IEEE the by ed

- - - ‐ lane 25 Gb/s PHY for operation over MMF consistent with IEEE P802 IEEE with consistent MMF over operation for PHY Gb/s 25 lane lane Gb/s 25 PHY for operation over links consistentwith copper twin lane Gb/s 25 PHY for operation over links consistentwith copper twin backplane board circuit printed a over operation for PHY Gb/s 25 lane

- lane 25Gbit/s physical layer technology developed to support

objectives (approved November 2014) included:

y beyond 10G

abit abit

E thernet

T ETHERNET (50GE)T

Testing Testing

–

www.xenanetworks.com

.3bm .3bm

WHITE PAPER

Xena Networks Networks Xena target is to complete the 50 Gigabit Ethernet single lane standard in the 2018- the in standard lane single Ethernet Gigabit 50 the complete is to target force task 802.3cd IEEE The information. the transfer to used are amplitudes pulse different Ethernet (50GE) addresses Layer 1, which contains the following sublayers: 50Gigabit and (25GE) Ethernet Gigabit 25 for specification Consortium Ethernet The 25G The Ethernet protocol stack based th on is PROTOCOL STACK achieve this it will be necessary to use a Pulse Amplitude Modulation technique PAM To lane technology. dual the than effective more cost be to is expected technology lane single instead of the dual l lane single a over Ethernet Gigabit 50 define to however is force task the of objective The force. task 802.3cd IEEE the up setting by Ethernet 50Gigabit on defining work initiated 2016IEEE In Table 1: 25G and50G solutions summary summary of the two solutions: The following table is based a table a on in the showing 25Gspecification Ethernet Consortium 2015. September in completed was specification draft Consortium Ethernet 25G The technology. Ethernet Gigabit/s Microsoft Ethernet Consortium)wasformed in July 2014 byArista, Broadcom, Google, Mellanox and In addition to the IEEE 802.3by 25 Gb/s Ethernet Task Force an industry consortium (the 25G Table 2: The OSI model of computer networking B. Two lane 50G lane Two B. 25G Lane A. Single

Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer Port Mode

to support the specification of single lane 25 Gigabit/s Ethernet and dual lane 50 lane dual and Ethernet Gigabit/s 25 lane single of specification the support to –

Global Price/PerformanceLeaders in Gig

Presentation layer Presentation

Application layer Application Transport layer

Data link layer link Data ane solution as outlined in the 25G Ethernet Consortiumspecification as a Network layer Network Physical layer Physical

Session layer Session

Count Lane Lane

1 2

Speed per Lane

25.78125 25.78125 (GHz)

e seven e

layer OSI model of computer networking: computer of model OSI layer abit abit MLD4 or RS FEC (see (see RSFEC or MLD4 end of next section) PCS Lane Bonding E thernet N/A

Testing Testing

–

www.xenanetworks.com 2019 time frame. time 2019

negotiation - Auto 4 Yes Yes , where 4 4 where ,

WHITE PAPER

Xena Networks Networks Xena distribution, one unique align Table 4: Switches supporting 25GE and50GE ex Some port. one from 10GE also cases most in and 25GE 40GE, 50GE, 100GE, supporting solutions flexible very as implemented typically capabilities, 50GE and 25GE with switches offer vendors Several SWITCH 50GE AND 25GE multi the handle to stream data the into inserted are markers Alignment lanes. Gigabit/s 25.78125 physical the on sent each of two lanes, Gigabit/s 12.89063 virtual) (or 4“logical” into functionality in the PMA sublayer. In both cases the data from sublayer and the FEC sublayer. Otherwise MLD4, defined for 40G systems, is used as of a part the When RS FEC is used for the 50GE solution MLD is handled bythe interface betweenPCS the Multi a 50GE For 50GE. and 25GE for functions 1 layer the define to rates other for 802.3 IEEE in clauses to refers cases many in specification Consortium Ethernet The 25G Table 3: Layer 1 sublayers Dependent Sublayer) PMD (Physical Medium A PMA Correction Sublayer) FEC (Forward Error Sublayer) Coding (Physical PCS Sublayer Mellanox SN2700 Mellanox 3232C Cisco Nexus XPliant Cavium Broadcom StrataXGS® II BCM56970 Tomahawk Series Broadcom StrataXGS® BCM56960 Tomahawk Series 7260CX Arista Product

ttachment Sublayer)ttachment (Physical Medium

–

Global Price/PerformanceLeaders in Gig

amples are: amples - 64 ® - LaneDistribution (MLD) technique is required to distribute data into two lanes.

CNX88091

ment marker for each virtual lane. virtual each for marker ment • Functions • • • • • • • • ES

octet synchronization/detection octet and 25Gigabit case of the in encoding (64b/66b coding the transceiverfor the physical medium scrambling/descrambling framing Ethernet) 50 Gigabit following possible FECs: possible following Consortium Ethernet 25G the determine a functional link has been established (Link auto optional for 25 and 50 Gigabit Ethernet Ethernet 50Gigabit 25and for optional Training) o o Solomon FEC (IEEE 802.3 Clause 108) Clause 802.3 (IEEE FEC Solomon In addition to the FECs above can be used to determine which FEC is used is FEC which determine to used be can o

- No FEC No 91) Clause 802.3 (IEEE FEC Solomon Reed

BASE negotiation

- R FEC “Fire code” (IEEE 802.3 Clause 74) Clause 802.3 (IEEE code” “Fire R FEC

abit abit

E thernet

the upper layers are distributed 100GE

32 32 32 64 32 64

Testing Testing Reed Reed includes 802.3by IEEE specification lists the the lists specification Maximum number of number Maximum

50GE 128 128 64 64 64 64

auto –

www.xenanetworks.com

40GE negotiation negotiation

128 32 32 32 64 64

lane lane 25GE 128 128 256 128 256 64

ports

10GE N/A N/A 128 128 258 64

WHITE PAPER

Xena Networks Networks Xena modules. SFP+ and SFP as support one lane with bandwidth upto 28 For deployment of 25GE vendors like Finisar, Avago and Innolight offer SFP28modules, whi AVAILABLE ELECTRICAL include: computer (e.g. servers in a datacenter) to a network. NIC products supporting 25GE and 50GE A Network Interface Controller (NIC) (or network network interface card, adapter) connects a 25GE AND 50GE NICS connectors connec connector into four individual cable pairs to be MPO with module optical QSFP28 from a signals optical 25Gigabit/s four splitting available addition passive optical cables breakout are In available. are (AOC) Cables Active Optical out Another Gigabit/s QSFP28 to 4 x 25Gigabit/ 4x to QSFP28 Gigabit/s 100 interfaces optical For ToR. the in port 100Gigabit/s one through ports 25Gigabit/s with means This module. SFP28 a with terminated each cables, four to out breaking end one in module QSFP28 a will have The DACs available. are connections SFP28 25GE independent four to port 100GbE QSFP28 a connecting (DACs) connections connections. 25GE with However centers for data in the QSFP+ form factor.The QSFP28 module was originally designed to support 100 Gigabit/ Figure 3:

• • • • •

ted to SFP28 optical modules optical SFP28 to ted

module that will be relevant for 25GE is QSFP28, which supports four 28 four supports which QSFP28, is 25GE for relevant will be that module dual port versions port dual that support 10GE, 25GE, 40GE, 50GE, 100GE that 50GE, 40GE, 25GE, support 10GE, Q Mellanox ConnectX® - Broadcom NetXtreme® C- 25 Gigabit/s fiber optics 25 Gigabit/s copper Logic® FastLinQ™ 45000 Series Intelligent Ethernet Adapters: will includeadapters – . 100 GE

high capacity Ethernet switch (ToR) with 100 Gigabit/s ports can interface to 4 servers 4servers to interface can ports Gigabit/s 100 with (ToR) switch Ethernet capacity high

Global Price/PerformanceLeaders in Gig o o o

1310 70 to up distances or 850 each in modules Direct Attach Cables Cobber twinax cables (DACs) terminated with SFP28

to 4*25 to

nm SR supporting distances up to 100 meters on OM4 multimode cable cable multimode OM4 on meters 100 to up distances supporting nm SR nm supporting LR single modeand fiber distances from 5 to 20

Available modules include: include: modules Available GE

break

4 Lx EN 10/25/40/50 Gigabit 10/25/40/50 Ethernet4 Lx EN Adapter:and Single

AND OPTICAL MODULES OPTICAL AND s SFP28 break s SFP28 end, 0.5 to 5 meters in length. in meters 5 to 0.5 end, Series: DualSeries: Ethernet port 10/25G -

out cable with LC LC with

m OM3 multimode cable multimode OM3 m

Gigabit/s. SFP28 modules have the same form factor factor form same the have modules SFP28 Gigabit/s.

abit abit

E thernet ,

cobber twinax cables cables twinax cobber attach direct

Testing Testing

– www.xenanetworks.com Gigabit/s lanes lanes Gigabit/s

km ch ch

WHITE PAPER

Xena Networks Networks Xena may be caused noise e.g. by signals and are that too high, too low, too long or too short. In addition an eye diagramthe of sign al should be checked to identify physical layer issues, which auto like functions of verification deployed. To test the layerfunctions 1 defined in the 25G Ethernet Consortiumspecification so 50GE and 25GE new the to applies also course of This Testing Ethernet systems is required to ensure the proper functioning and quality of the systems. R TEST 50GE AND 25GE

Figure 6: MTP to 4 dual LC passive optical break –

Global Price/PerformanceLeaders in Gig

- EQUIREMENTS negotiation, Link Training and FEC must be performed.

- out cable abit abit lutions as they are developed and E Cable (AOC) Gigabit/s SFP28 break 25 4x to QSFP28 Gigabit/s 5: 100 Figure out cable break electrical DAC SFP28 Gigabit/s 25 4x to Figure QSFP28 Gigabit/s 4: 100 thernet

Testing Testing

– www.xenanetworks.com - out Active Optical

WHITE PAPER

Xena Networks Networks Xena pro curves and controls the PHY tuning in the transmit and the receive directions.The eye rate (BER) eye diagrams, estimates BER the from link vertical the horizontal and BER bathtub Under Test (DUT) the Loki the DUT use the correct FEC. FEC. correct the use DUT the correct functioning of thisfeature. Signals can be configured with or without FEC, verifying that The Loki process. negotiation this the Loki The Unique Loki Xena The A 25GE XENA NETWORKS supported bythe port under is test. 25G 40GE, 50GE, 100GE, support that products very flexible as implemented are typically switches As ToR You can set the Loki Figure 7: The powerful andversatile Xena Loki to test and verify the layerfunctions 1 defined for 25GE and 50GE. by the 25G Ethernet Consortium:

vides an estimate of several parameters of the eye, including width, height and jitter. and height width, including eye, the of parameters several of estimate an vides Loki • • •

- Reed Clause (IEEE 802.3 FEC Solomon 108) Reed SolomonFEC (IEEE 802.3 Clause 91) BASE 100G

Eye Diagram E and 10GE and E port it from is one of course to test important also Ethernet at all rates –

- Global Price/PerformanceLeaders in Gig 100G

- - - 100G R FEC “Fire (IEEE code” R FEC 5S - - 1P 5S - - 5S- 100G

includesthe “eye diagram”, a unique feature for analyzing signal quality. For - 1P

monitors the link at the physical level and creates bit creates and level physical the at link the controls monitors and - -

100G 5S 5

- speed dual - - Loki 1P 100G -

5S- to perform auto perform to - 100G -

1P 5S ND 50GE TEST SOLUTIO TEST 50GE ND

- 802.3 Clause802.3 74) can verifycan the DUT that reacts correctly the to auto 1P - - 5S- media test module provides powerful testing capabilities capabilities testing powerful provides module test media

the the verifying DUT, the with Training Link do will also 1P

- supports FECs the specified byIEEE 802.3by 100G - negotiation so when connected to a Device Device a to connected so when negotiation

5S abit abit -

5 E thernet - speed dual

Testing Testing

- media test module

– www.xenanetworks.com

- data table data

- and and - error -

WHITE PAPER

Xena Networks Networks Xena free test applications for the Loki the for applications test free the previoussections. Advanced layer 2 and layer 3 test scenarios can be performed using the Ethernet testing the Loki with 1 Layer Above Testing Figure 8: The unique eye diagram you get with Xena 100G test modules

• • • •

Valkyrie protocols. multicast IP support that devices of characteristics latency and Internet Group Management Protocol (IG forwarding, throughput, the reporting and measuring for tests describes RFC3918 3918. Valkyrie switches. The following RFC 2889 test types are supported: are types test 2889 RFC following The switches. Valkyrie

Valkyrie results. traffic between Xena test equipment and Devices Under Test (DUTs) and analyze the –

Global Price/PerformanceLeaders in Gig o o o o o

2544 offers full support for the 4 test 4 the for support full offers 2544 Manager 3918 2889 (Frame Jitter

Back Frame loss Frame Latency Throughput

- is an application for benchmarking the performance of Layer 2 LAN makes it easy to create, edit and execute all test all execute and edit create, to easy it makes to - back frames back

generate streams of Ethernet Ethernet of streams generate and configure to used is software test

- 100G Delay Variation) is also supported is also Variation) Delay - 100G - 5S-

- 1P 5S-

iscourse of not limited to the layer 1 tests described in 1P

- abit abit types specified in RFC2544 in specified types E thernet MP) group membership

Testing Testing

types specified in RFC in specified types – www.xenanetworks.com

WHITE PAPER

Xena Networks Networks Xena directly to 25 Gigabit/s transceivers in the Loki and abreak out cable terminated with four SFP28 transceivers or two 25GE ports connected Figure 9: Loki Another free application for the Loki the for application free Another

test module lets u lets module test can be tested.This is neededto test and verify 25GE and 50 GE capable switch ports.Theunique Gigabit/s 100 to Gigabit/s from10 range the in rates Ethernet existing the new and the so both Loki testing 50GE and 25GE addition In Gigabit/sTesting to 100Gigabit/s from 10

• •

engineers. test for easier automation test makes ValkyrieCLI Valkyrie (SLAs) in a single test single a in (SLAs) service Ethernet of validation complete for Y.1564 in described types –

Global Price/PerformanceLeaders in Gig - o o o o o o 100G

1564 Forward Pressure Maximum and Forwarding Rate Broadcast Frame Forwarding Latency and Address Learning Rate Capacity Caching Address Congestion Control All Throughput and Forwarding rate t sers dynamically choose between two different physical transceiver cages and - 5S

- a powerful easy and

provides full support for both the configuration and performance test 1P

can testcan four 25GE ports through

100G - 100G

- - to

5S-

- - 5S - use command use 1P 100G - 1P

(and all other Xena test

also supports test of 100GE, 40GE and 10GE, 10GE, and 40GE 100GE, of test supports also - 5S abit abit - ests (both Fully and Partially meshed) 1P

E a 100 Gigabit/s QSFP28 transceiver transceiver QSFP28 100Gigabit/s a

- thernet line - interface (CLI) scripting API that that API scripting (CLI) interface

Testing Testing

modules) –

- www.xenanetworks.com level agreements agreements level

WHITE PAPER

Xena Networks Networks Xena routers, NICs, packet functional testing of network infrastructure and Ethernet equipment such as taps, switches, 100GE. and 50GE 40GE, 25GE, 10GE, Gigabit/s: 100 to Gigabit/s from 10 range verify both new and existing Ethernet rates in the Ethernet test layer to capabilities 3 up test to and versatile and powerful gets user the module test theWith Xena Loki tested. one To port. test the prod from 10GE and 25GE 40GE, 50GE, 100GE, support that solutions very flexible as implemented typically are capabilities 50GE and 25GE with switches Datacenter years. coming the in rapidly grow to cloud based expected is solutions. 50GE and Therefore the 25GE deploy of ment expansion of datacenter capacity to fulfil the huge increasein bandwidth requirements from – rates data Ethernet New CONCLUSION Loki the makes factors form transceiver sp network Ethernet different five of combination unique The cages. transceiver SFP28/SFP+ two is second the and cage, transceiver QSFP28/QSFP+ single a is first The factors. form

–

Global Price/PerformanceLeaders in Gig

- 100G

- brokers, and ba

- 5S- uct all rates need to be

25GE and 50GE - 50GE and 25GE 1P

5 - speed dual ckhaul platforms. ckhaul - 100G

are coming, driven by the need for cost effective effective cost for need the by driven coming, are - media media - 5S - 1P

a versatile solution for performance and and performance for solution versatile a abit abit

E thernet eeds and multiple physical optical optical physical multiple and eeds

Testing Testing

– www.xenanetworks.com

WHITE PAPER