Protocol Solutions Group

Protocol Solutions Group - The Protocol of the PHY WHITE PAPER High-speed designs require protocol knowledge. Introduction David J. Rodgers Sr. Product Marketing Mgr. The framework for Ethernet communications is not new. The first IEEE SAN/Fabric Protocol Tools 802.3 Ethernet standard achieved ratification in 1983 for 10Mbit/s over thick coax cabling. The next 20 years saw a modest advancement of Date: 30 November 2018 standards and speeds, resulting in the 10Gb/s rate ratification in 2002; Summary IEEE 802.3ae. From 2002 to today, Ethernet has rapidly expanded with ratified specifications for data rates up to 100Gb/s, and a pending 400Gb/s “The SierraNet™ device specification expected to be ratified by the end of 2017. helped us tremendously at a time, where we had zero Along with the increase in Ethernet speeds, the corresponding visibility into the 802.3BJ AN specifications have included advancements (and complexities) in physical and LT process with other link bring-up; two of these being Auto-Negotiation (AN) and Training vendors and link-partners. Sequences (TS). The IEEE 802.3bj specification for 100GbE (4x25GbE) With its help, we were able to Ethernet and the related 802.3by standard for 25GbE connections, and pinpoint the problem very fast, contributions from the 25G Ethernet Consortium added new PHY layer fix it and optimize our AN/LT requirements for AN and TS dialogs to ensure optimized communications. flow even further. We also could show to the customer Traditional signal integrity tools for hardware development are proving that the other “link-partner” insufficient in the effective determination and debug of PHY layer needed improvements, which specification nonconformance. The challenge presented is how to was perceived as ’standard’ in incorporate current tools and practices with new offerings to achieve industry. For Cavium; with our success. dedication to offer world-class The data rates of today’s technology have added to the test tool’s products, we are happy to complexity and protocol analyzers have become a “must have” instead of have a world-class and a “nice to have” in the lab. Fortunately, the Ethernet test and innovative partner like measurement community response to the hardware engineering Teledyne LeCroy on our side.” challenges is robust with innovative new solutions. Amir Motamedi, Sr. Staff Hardware/SI Engineer, Hardware vs Software Protocol Analysis Cavium The Ethernet community is very familiar with the idea of “packet inspection” or “protocol analysis” tools, for example, the ubiquitous and well-known “Wire Shark” is the de-facto standard for use in troubleshooting Ethernet frames. Ethernet frames, however, only transit upon stable connections and fabrics and Wire Shark captures are lossy at higher data rates when used in conjunction with a NIC.

Network management utilities in use today sense the reduced throughput of the NIC when used as the Wire Shark tap and will reroute the impacted

Teledyne LeCroy Protocol Solutions Group - The Protocol of the PHY High-speed Ethernet designs require protocol page | 1 of 7 knowledge.

traffic to more stable connections. Subsequently, Wire Shark does not capture the intended Ethernet traffic.

More importantly to the discussion, Wire Shark does not provide the “complete” Ethernet picture. as it has no visibility for debugging Ethernet anomalies which may be the root cause in a failed/failing link.

The challenge for understanding and debugging the physical layer transactions of today’s new, high-speed networks requires “bump in the wire” observation techniques coupled with comprehensive analysis utilities; hence the introduction of hardware protocol analysis tools.

The use of hardware-based protocol analysis tools for Ethernet applications specifically is gaining broad acceptance, largely due to the complexities of physically connecting at higher data rates and application layer protocols. The aim of these hardware-based solutions and accompanying applications is to facilitate testing for conformance and compliance with the relevant industry standards.

The IEEE802.3by specification for 25GbE, and IEEE802.3bj for 100GbE (4x25Gb) added new PHY layer requirements to ensure stable connections. In addition to the IEEE802.3by specification for 25GbE, the 25G Ethernet Consortium specifications were folded into the final release. The new PHY layer requirements included enhancements to the auto-negotiation, link training, and Forward Error Correction (FEC) dialogs to ensure optimized communications.

Adherence to these new requirements is increasingly challenging, and traditional signal integrity and software tools fall short when debugging nonconformance. Summarily, the challenge of capturing and analyzing data on high-speed fabrics diminishes with a purpose built analyzer.

Interpretation of Specifications – Auto-Negotiation Auto-negotiation (AN) for Ethernet connections is not new. Beyond simple speed connection requirements, AN as evolved for today’s Ethernet includes additional layers of connection configuration information. For instance, new to 25GbE and 100GbE applications is the optional use of both Firecode or BASE-R FEC and Reed Solomon (RS-FEC) or no FEC at all (desirable when low latency is critical, and a potentially lossy connection is permissible). The Technology Ability Field (TAF) is included during the AN sequence, allowing end points to advertise their capabilities, request FEC implementations, and introduce additional Organizationally Unique Identifier (OUI) Figure 1 – 802.3bj Link Speed Options information. An example of the TAF is shown here.

Teledyne LeCroy Protocol Solutions Group - The Protocol of the PHY High-speed Ethernet designs require protocol page | 2 of 7 knowledge.

The AN sequences of the 25GbE and 100GbE specifications are, however, increasingly important to efficient, expeditious establishment and maintenance of a link. IEEE and Consortium specifications are concerned with the result of the implementation, and not necessarily the means to accomplish those results. The physical layer protocol information may reside in either hardware, firmware, software, or a combination thereof. Regardless of the PHY implementation, it is important the link partners communicate their capabilities and agree on how they will transmit and receive packets.

As represented in Figure 2, not only may the new link operate at specified rates, Extended Technology Abilities fields are exchanged for establishing the best possible link between the negotiating ports, presenting the FEC capabilities, and introduction of additional Organizationally Unique Identifier (OUI) information. Each end of the link must be programmed correctly to ensure the proper dialog and information is transmitted. Figure 2 – Additional AN Information Once the ports in the link have exchanged the requisite AN information and reached agreement, the ports acknowledge each other, the link speed and supported characteristics are established, and the link moves to the next step, Training Sequences (TS).

The IEEE 802.3 specifications mandate the time from when AN concludes and when TS must begin. If too much time elapses between the conclusion of AN sequencing and the onset of TS exchanges, the link will time out and revert to AN again. The initialization of TS requires the linking ports to move to the negotiated speed and then commence the training. Depending on the vendors’ implementation preference (e.g. ASIC, firmware, or software) the timing may be an issue.

Teledyne LeCroy Protocol Solutions Group - The Protocol of the PHY High-speed Ethernet designs require protocol page | 3 of 7 knowledge.

Link Training and Transition Timing There are new and significant challenges in link training to achieve optimal frame transmission. The initialization of TS requires the linking ports to move to the negotiated speed and then commence the training. The most common issue observed today is simply a timing mismatch in the transition from AN to the TS processes. The IEEE 802.3 specifications mandate a window of 500ms from when AN concludes and when TS must begin. If too much time elapses between the conclusion of the AN and the onset of TS exchanges the link will time out and revert to AN again.

The representative example here is the conclusion of the AN and the onset of TS. Once AN completes, both ports issue a “Loss of Sync” and reconnect at the Figure 3 - Transition from AN to TS, determine delta time from AN completion to TS inception. negotiated speed (25Gb in this example) and begin the TS link optimization. The analyzer captures the state transitions allowing precise calculations to determine AN completion and 25GbE SERDES inception to ensure the start training is within specification compliance.

If the timing is not an issue, another suspect may be the copper connection between ports. High speed Ethernet data rates are pushing the limits of copper. However, Feed Forward Equalization (FFE) for the transmitter has enabled higher data rates than were previously possible.

The “old way” of hardware testing The Hardware Developer has traditionally relied on oscilloscopes (BERTs and PERTs) in combination with in- house applications to test and troubleshoot physical layer connections. The limitation, however, is these tools do not decipher the link layer transactions for root cause determination for the breakdown in basic link communications. Key among the issues we need to inspect:

• The Link did not come up. How does the developer reliably determine the root cause of the failure? • The Link participants acknowledge 25GbE capabilities during AN, however the link was established at 10GbE. How did this occur? • The AN sequence appears to have completed, and the transition to link training is indicated, however, the link reverted to AN again. What caused the link to fail training and start AN again? • The link partners have moved into the training phase and the developer can see the changes in the physical signal on the scope. How does the developer know if the transmitter changes correspond to the coefficient change request(s)?

Legacy hardware tools can measure the speeds and represent the eye diagram of the link under observation, but they provide no insight into the link connection dialog. Insight into the protocol is required to determine if and how the requested change was or was not applied.

Protocol analysis, or packet inspection tools (e.g. Wireshark) are well known in the Ethernet community. The advantage of the protocol analyzer is the ability to capture and examine specific frames and traffic types and content to determine conformance to the associated specification or expected mode(s) of operation.

Teledyne LeCroy Protocol Solutions Group - The Protocol of the PHY High-speed Ethernet designs require protocol page | 4 of 7 knowledge.

Traditional packet inspection software uses the output of standard, off the shelf, NIC or SPAN ports, or the diagnostic interface of the ASIC to provide the tap for data stream capture for analysis. This approach may convey operational metrics, packet counts, and bit error rates, however, it also relies on a stable and active link to provide the information. When the link is not present or is unstable, these same capture access points and analytics are blind to transactions occurring on the link, especially those during AN and TS phases.

The key challenges today are debugging the new, high-speed network physical layer transaction properties, understanding the requirements necessitating different test and measurement methodologies, and integrating hardware-based protocol analysis tools to address those demands.

How does the Hardware Analyzer help? The hardware-based Ethernet fabric analyzer provides a complete and unfettered “view” of the line-rate traffic. Whatever crosses the wire is recorded and can be observed. Ideally, the analyzer probe, or “passive tap”, needs to be discreet, transparent, and neither introduce or mask behavior of the link under observation. A truly analog probe design will tap the signal at extremely low impedance yet miss nothing of the ongoing transactions.

A passive tap protocol analyzer captures and displays the AN sequence from both ends of the link without affecting or altering the content of the exchange. The passive probe captures the real transactions, in real time. The importance of the passive tap cannot be overstated when examining the link layer transactions. by way of example, maintaining the timing tolerance of the AN exchange is essential to a successful speed negotiation. If the endpoint times-out before AN completes, the sequence will restart, potentially in an endless loop. The passive probing, also known as “analog pass through” captures the events of the transaction(s) as they occur, without the need to de-embed the probe or adjust timing to remove retransmission effects.

The hardware protocol analyzer represents the traffic with nanosecond level granularity and precision. It simultaneously records both ends of the link, which is not possible with any other software or embedded analytics tools.

Determining implementation differences between vendors and specifications The advantage of the protocol analyzer is the ability to drill down into specific frames and traffic types to examine the vendors’ interpretation and subsequent implementation of the associated specification. In reference to training sequences, the PHY must adjust the transmitter coefficients, via Feed Forward Equalization (FFE), to establish to optimal settings. A given vendor may be able to determine the settings of their own device from their embedded analytic reports. However, without the “bump in the wire”, determining the behavior of both devices simultaneously is unachievable.

Noted above, not only are NEMs faced with conformance to IEEE 802.3 standards, the 25G Ethernet Consortium contributions to the Ethernet ecosystem added complexity to interoperation by early adopters. The Consortium provisions have interdependencies with IEEE specifications and must be considered when designing and testing new equipment. Figure 4 - Example of Training Sequence frame information

Teledyne LeCroy Protocol Solutions Group - The Protocol of the PHY High-speed Ethernet designs require protocol page | 5 of 7 knowledge.

Evolution of Specifications Specifications for all technologies take on a life of their own. Initial ratification and release are the start of this life. As specifications evolve, there are iterative updates to address feature enhancements and additions the ecosystems require. Typically, the requirement of new upper layer protocols drives specification changes in the Ethernet community. Key among these, of late, is the advancement of NVMe over Fabrics storage interfaces.

In the case of Ethernet, the 25GbE specifications noted herein derive from multiple sources; the IEEE and the 25G Consortium. NEMs are very concerned with ensuring interoperability with other members of the Ethernet community as well as conformance with the related specifications they are supporting. As discussed, the ability to determine the actual link traffic, at line rate, enables rapid and concise evaluation and understanding of specification compliance or deviation.

The test and measurement vendor teams need to have a comprehensive understanding of the Ethernet specifications, and actively participate in the associated standards bodies. The basis of the test tools must be specification conformance, and that does not mean only ratified versions. Attendance at the SIG functions, like the IEEE or T11, and the ability to foresee the evolution and advancement of specifications is a must to ensure timely delivery of updated or new functional requirements.

The Wrap Up The use of a hardware based, “bump in the wire” protocol analyzer is not new to Ethernet or other communications technologies. The evolution of high-speed specifications has introduced a new set of requirements, and subsequently, customers and environments that may take advantage of the features and benefits offered by the hardware analyzer. Frankly speaking, a protocol analyzer like the SierraNet™ family of Ethernet analyzer and jammer products is the only real means to determine the behavior of the traffic on a given link or fabric.

The complexities of specification adherence within the myriad of implementation designs by NEMs equates to interoperability nightmares, as already experienced in the Ethernet ecosystem. Design validation and support demands further the fact that new approaches and tools are mandatory in the successful deployment of new Ethernet products.

From ASIC designs, to cabling and connector implementations, all participants in the Ethernet Eco-System are impacted by the increasing need for speed.

For more information regarding the Teledyne LeCroy family of SierraNet protocol analyzer tools and InFusion jammer utilities, please visit http://teledynelecroy.com/protocolanalyzer/.

For more information regarding the Cavium family of semiconductor product solutions and services, please visit http://Cavium.com/.

Teledyne LeCroy Protocol Solutions Group - The Protocol of the PHY High-speed Ethernet designs require protocol page | 6 of 7 knowledge.

Acknowledgments: 1 – “IEEE 802.3 25G Ethernet SG – Auto-Negotiation Considerations. A thought-starter on AN” (Baden, Begin, Booth, Kim, Nicholl, November 2015) http://ieee802.org/3/25GSG/public/Nov14/baden_25GE_02_1114.pdf /

2 – “25G Ethernet consortium releases 25G/50G Ethernet specification, focuses on multivendor interoperability” (Buckley, January 2017) http://www.fiercetelecom.com/telecom/25g-ethernet-consortium-releases-25g-50g-ethernet-specification-focuses-multi- vendor

3 – “IEEE Standard for Ethernet – Amendment 2: Media Access Control Parameters, Physical Layers, and Management Parameters for 25Gb/s Operation” (IEEE Std 802.3by™, 2016) http://standards.ieee.org/getieee802/download/802.3by-2016.pdf

4 – Interview of Amir Motamedi, Sr. Staff Hardware and SI Engineer, by David J. Rodgers, Sr Program Marketing Manager

5 – “25 Consortium Members Validate Multi-Vendor Interoperability” Press Release of 30 January 2017 http://25gethernet.org/25-gigabit-ethernet-consortium-members-validate-multi-vendor-interoperability-0

Teledyne LeCroy Protocol Solutions Group - The Protocol of the PHY High-speed Ethernet designs require protocol page | 7 of 7 knowledge.