40 and 100 Gigabit Ethernet
Total Page:16
File Type:pdf, Size:1020Kb
BEYOND 10 GIG ETHERNET 40 and 100 Gigabit Ethernet: Ready for Real-Time? With 10 Gigabit Ethernet just ramping up in deployment, the natural question is, “What SOLUTIONS ENGINEERING about 40GbE and 100GbE?” In terms of real-time applications, the needed tweaks are being made to 10GbE, but for now 40 and 100GbE remain in the server space—for now. by Rob Kraft AdvancedIO Systems hile 10GbE is barely off the 10GbE Physical launching pad in terms of broad Interfaces Wdeployment in just about any market space, recent excitement has turned to 40GbE and 100GbE. It’s not a stretch to Interface to Time Stamp FPGA External Time Interface acknowledge that real-time embedded en- Lag gineers, who are often also technophiles, Application- are likely to be seduced by the allure of Optimized Memory Buffering such high bandwidth technology. Connectivity Control For Line Modified Stack Engine Input So, the question is: should those of us Behavior and in the real-time space shelve the just-or- Additional Application-Level dered 10GbE technology and start design- Offload Memory ing 40GbE or 100GbE into our next-gen- eration bandwidth-hungry applications? Host Fabric Interfaces Has 10GbE already become passé? In December 2007, the IEEE P802.3ba 40 Gbit/s and 100 Gbit/s Ethernet Task Force Figure 1 Elements of an FPGA-based solution that implements features required was formed out of the High Speed Study for real-time 10GbE connectivity. Group (HSSG) that had been working since 2006. At the time of writing this article, the most recent release from the task force was enterprises migrating from private networks and improve the flow of information, they P8023.ba Draft 1.2, on February 10, 2009. to Internet-based VPNs. To adequately ser- also can reduce cabling infrastructure—a The targeted date for a ratified standard vice the increasing bandwidth demands of non-trivial problem given the number of is June 2010. Current predictions are that their customers, ISPs need to increase their servers in a data center. 100GbE will “take off” in 2013. 40GbE is backbone bandwidth at a ratio that may be A recent report cited the number of expected to ramp up sooner. For reference, 4x to 10x the customer’s needs. In addition, servers in 2007 to be 11.8 million in the Table 1 shows how 10GbE rollouts com- large search engine and social networking U.S. and 30.3 million worldwide, up more pared to predictions and gives some data companies are eyeing 100GbE as a solution than 4 times from a decade earlier. Ana- about 1GbE rollouts at selected points after to their bandwidth needs for inter-data-cen- lysts also predict that the U.S.’s 6600 data the respective standards’ ratification dates. ter network aggregation. centers (where most servers are hosted) Simultaneously, 40GbE (and good will need replacing or retrofitting in the Factors Driving 40GbE and “old” 10GbE) are being driven by the next several years. Clearly, there are lots 100GbE growing needs for data movement between of servers to interconnect and lots of Increases in Internet video traffic are servers and other computing applications “voltage” behind the market force. driven by sources such as YouTube, high- within data centers. The higher band- To support the 40GbE and 100GbE definition IPTV, video conferencing, and widths not only solve data aggregation data rates, there have been several inno- Reprinted from March 2009 SOLUTIONS Engineering vations. Some have been driven by the characteristics are not encountered in the when the receiving Ethernet interface is fact that current technology does not per- server space, so server space solutions do momentarily unable to access host system mit the transmission of 40 Gbits/s or 100 not address them. memory to store the incoming data. The Gbits/s in a single stream over any optical Real-time test and measurement, result is unacceptable permanent packet fiber or copper wire. For instance, in the scientific, defense, hardware-in-the-loop loss. This characteristic is relatively un- current baseline, 100GbE would be trans- simulation, and other multi-sensor systems common in server systems that have much ported in parallel over cables consisting rely on incoming data being accurately softer real-time requirements, thus giving of 10 fibers, or 10 wires, or using 4 wave- time stamped. The time-stamping aligns opportunities for flow control or retrans- lengths in the case of single-mode fiber. data arriving from multiple sources or sen- mission when required. Therefore, the Except for increased data rate, there are sors to permit detailed off-line analysis, as cost-optimized solutions targeted at the no changes proposed at the Ethernet MAC inputs into models in real-time processing, server space do not have to be designed to layer (compared to 10GbE). Still, the base- or to tightly control the release of data in accommodate these regular long-duration line has introduced the concept of multiple complex simulation systems. At the higher- bursts. lanes and multi-lane distribution (MLD) performance end, the CPU cannot time-tag To address the requirements above, at the Physical Coding Sublayer (PCS). data with accuracy and precision since, by 10GbE technology must implement fea- This was done to accommodate combina- the time the packets reach this point, they tures including interfaces for precision tions of differing numbers and speeds of have gone through several non-determin- time-stamping, local memory to accom- parallel electrical lanes and media lanes istic interfaces. A solution is to stamp the modate large full-rate inbound bursts and (fibers, wires, or light wavelengths), and to packets at the 10GbE interface ingress outbound data staging, and the ability to decouple those two numbers since electri- point, before they ever get to the CPU. customize stack behavior for receiving cal and optical technologies will develop Most real-time applications begin as real-time sensor data (Figure 1). at different rates. a stream of digitized analog real-world These real-time adaptations will be- Among other changes is an evolution sensor signals in a control, automation, come even more complicated to imple- of the now-familiar XAUI (10 Gigabit communications or measurement/analysis ment for 40GbE and 100GbE and need Attachment Unit Interface), used for on- system. The sampled data passes through to be solved before solutions are matured. board signaling, into XLAUI (40 Gigabit) signal processing algorithms such as fil- Among the challenges: and CAUI (100 Gigabit). The ‘XL’ and tering, FFT, decoding and many others, • Incoming buffers need to be larger— ‘C’ correspond to Roman numerals for and is subsequently passed on to other 4x to 10x the data arrives in the same 40 and 100. To accommodate the higher processing functions. Many signal pro- time period as before. rates (10.3125 Gbaud per lane, compared cessing algorithms correct for or tolerate • Likewise, external memory and con- to XAUI’s 3.125 Gbaud per lane), XLAUI scattered errors and noise in the signal trollers need to operate at a higher and CAUI use 64B/66B encoding, which stream, but choke when faced with a con- bandwidth. has a reduced overhead (3%) compared to secutive stretch of missing data. • Interfaces like CAUI require 2.5x the 8B/10B (20%) used in XAUI. 40GbE is Ironically, the typical behavior of number of high-speed pins of XAUI, handled in 4 XLAUI lanes, and 100GbE the standard Ethernet protocol stack can increasing the number of high-speed in 10 CAUI lanes. transform benign scattered errors into I/O pins required in devices and com- a swath of missing data that chokes sig- plicating PCB routing. Application-Level nal processing algorithms. This occurs • Internal processing bandwidth needs Characteristics because the protocol will discard entire to increase correspondingly to realize At the moment, the driving applica- packets or messages, which could be protocol offload and other processing tions involve aggregation and distribution 1500, 9000, or up to 64000 bytes long, required for Ethernet termination. of data. But eventually, the data has to ter- if an error is detected in a checksum or minate at processors or other devices. The if a message arrives incomplete. And yet Implications for Real-Time termination problem is challenging even at the source of the checksum error may be 40GbE and 100GbE 10GbE (sometimes even at 1GbE), where just one or two data bytes, or in the packet Because 40GbE and 100GbE technol- processors get clobbered by the effort of header. The solution is to modify the stack ogies are currently driven by the massive running the protocol stacks. It stands to behavior to avoid dropping the packets in server-type markets, the ASIC-based so- reason that the termination problem will the presence of the CRC or checksum er- lutions, out of sheer volume necessity, will be 4x to 10x worse for the case of 40GbE rors, and to allow them to pass to the sig- target those applications, just as they have and 100GbE. In the commercial space, nal processing stage. in the 10GbE case. As pointed out, those the solution at 10GbE typically involves High-throughput real-time instru- applications have some fundamentally forms of protocol offload, whereby some mentation, communication and other sen- different requirements from the higher- or all of the protocol processing elements sor processing systems can receive multi- end real-time embedded applications. The are farmed out to a coprocessing ASIC. ple back-to-back packets burst at full line latter will therefore need to use solutions However, there are a variety of application speed on a regular or even a sustained ba- based on programmable logic, which al- characteristics unique to many real-time sis.