Reprinted from and conferencing, video IPTV, definition YouTube, as high- such sources by driven 100GbE Factors Driving and 40GbE dates. respective ratification the standards’ about1GbE rollouts atselected points after data some gives predictions and to pared com rollouts 10GbE how shows 1 Table expectedsooner.uprampto Forreference, 2013.100GbEin off”“takewill 40GbE is that predictions 2010. are Current June is standard ratified a for date targeted The P8023.ba1.2,10,FebruaryDrafton2009. was force task the from release recent most 2006. At the time of writing this article, the Group Study (HSSG) Speed that had High been working the since of out formed was 40 Gbit/s and 100 Gbit/s Task Force passé? 10GbE become Has already applications? -hungry eration next-gen our into 100GbE or 40GbE ing dered 10GbE technology design and start just-or the shelve space real-time the in such bandwidth technology. high of allure the by seduced be to likely are technophiles, also often are who gineers, acknowledge en embedded real-time that to 40GbE and 100GbE. It’s not a stretch to market space, recent excitement has turned W solutions engineering Increases in video traffic are traffic videoInternet in Increases In December 2007, the IEEE P802.3ba So, questionthe is: should ofthose us by  made to10GbE,butfornow 40and100GbEremain intheserver space—fornow. about 40GbEand100GbE?”In termsofreal-time applications,theneededtweaks are being With 10Gigabit Ethernet justrampingupindeployment, thenaturalquestionis,“What Ready forReal-Time? 40 and100GigabitEthernet: Beyond 10gigEthernet elyet n ut bu any about just in deployment the offbroad of terms in pad launching barely is 10GbE hile AdvancedIO Systems Rob Kraft March 2009

- - - - - its o ol sle aa aggregation data solve band only not higher widths The centers. data within applications computing other and servers the by growing drivenneeds for data movement being between are 10GbE) “old” networkter aggregation. inter-data-cen for needs bandwidth their to companies are eyeing 100GbE as a solution networkingsocialengineandsearchlarge 4x to 10x customer’s the needs. addition, In backbonebandwidth ratioathatatmay be their increase customers, to ISPsneed their increasingthevicebandwidth demands of VPNs.Toser adequatelyInternet-based to enterprises migrating from private networks F i gure 1 gure Simultaneously, 40GbE (and good good (and 40GbE Simultaneously, for real-time 10GbE connectivity. 10GbE real-time for required features implements that solution FPGA-based an of Elements Application-Level Modified Stack External Time Behavior and Behavior Interface to Additional Offload Lag Time Stamp Connectivity Application- Interface Optimized Host Fabric Interfaces Engine 10GbE Physical Interfaces - - - data rates, there have been several inno several been have there rates, data of lots and “voltage” force. market the behind interconnect to servers of lots are Clearly,there years. several next the in retrofitting or replacing need will hosted) are servers most (where centers U.S.’s the lysts that predict also data 6600 Ana earlier. decade a from times 4 than U.S. worldwide, 30.3and million up more the in 11.8million be to 2007 in servers center. adata in servers of number the given problem non-trivial infrastructure—a cabling reduce can also improve and flow the theyinformation, of Memory Control FPGA o upr te 0b ad 100GbE and 40GbE the support To of number the cited report recent A Memory Buffering For Line Input

- - SOLUTIONS Engineering vations. Some have been driven by the characteristics are not encountered in the when the receiving Ethernet interface is fact that current technology does not per- server space, so server space solutions do momentarily unable to access host system mit the transmission of 40 Gbits/s or 100 not address them. memory to store the incoming data. The Gbits/s in a single stream over any optical Real-time test and measurement, result is unacceptable permanent packet fiber or copper wire. For instance, in the scientific, defense, hardware-in-the-loop loss. This characteristic is relatively un- current baseline, 100GbE would be trans- simulation, and other multi-sensor systems common in server systems that have much ported in parallel over cables consisting rely on incoming data being accurately softer real-time requirements, thus giving of 10 fibers, or 10 wires, or using 4 wave- time stamped. The time-stamping aligns opportunities for flow control or retrans- lengths in the case of single-mode fiber. data arriving from multiple sources or sen- mission when required. Therefore, the Except for increased data rate, there are sors to permit detailed off-line analysis, as cost-optimized solutions targeted at the no changes proposed at the Ethernet MAC inputs into models in real-time processing, server space do not have to be designed to layer (compared to 10GbE). Still, the base- or to tightly control the release of data in accommodate these regular long-duration line has introduced the concept of multiple complex simulation systems. At the higher- bursts. lanes and multi-lane distribution (MLD) performance end, the CPU cannot time-tag To address the requirements above, at the (PCS). data with accuracy and precision since, by 10GbE technology must implement fea- This was done to accommodate combina- the time the packets reach this point, they tures including interfaces for precision tions of differing numbers and speeds of have gone through several non-determin- time-stamping, local memory to accom- parallel electrical lanes and media lanes istic interfaces. A solution is to stamp the modate large full-rate inbound bursts and (fibers, wires, or light wavelengths), and to packets at the 10GbE interface ingress outbound data staging, and the ability to decouple those two numbers since electri- point, before they ever get to the CPU. customize stack behavior for receiving cal and optical technologies will develop Most real-time applications begin as real-time sensor data (Figure 1). at different rates. a stream of digitized analog real-world These real-time adaptations will be- Among other changes is an evolution sensor signals in a control, automation, come even more complicated to imple- of the now-familiar XAUI (10 Gigabit communications or measurement/analysis ment for 40GbE and 100GbE and need Attachment Unit Interface), used for on- system. The sampled data passes through to be solved before solutions are matured. board signaling, into XLAUI (40 Gigabit) signal processing algorithms such as fil- Among the challenges: and CAUI (100 Gigabit). The ‘XL’ and tering, FFT, decoding and many others, • Incoming buffers need to be larger— ‘C’ correspond to Roman numerals for and is subsequently passed on to other 4x to 10x the data arrives in the same 40 and 100. To accommodate the higher processing functions. Many signal pro- time period as before. rates (10.3125 Gbaud per lane, compared cessing algorithms correct for or tolerate • Likewise, external memory and con- to XAUI’s 3.125 Gbaud per lane), XLAUI scattered errors and noise in the signal trollers need to operate at a higher and CAUI use 64B/66B encoding, which stream, but choke when faced with a con- bandwidth. has a reduced overhead (3%) compared to secutive stretch of missing data. • Interfaces like CAUI require 2.5x the 8B/10B (20%) used in XAUI. 40GbE is Ironically, the typical behavior of number of high-speed pins of XAUI, handled in 4 XLAUI lanes, and 100GbE the standard Ethernet protocol stack can increasing the number of high-speed in 10 CAUI lanes. transform benign scattered errors into I/O pins required in devices and com- a swath of missing data that chokes sig- plicating PCB routing. Application-Level nal processing algorithms. This occurs • Internal processing bandwidth needs Characteristics because the protocol will discard entire to increase correspondingly to realize At the moment, the driving applica- packets or messages, which could be protocol offload and other processing tions involve aggregation and distribution 1500, 9000, or up to 64000 bytes long, required for Ethernet termination. of data. But eventually, the data has to ter- if an error is detected in a checksum or minate at processors or other devices. The if a message arrives incomplete. And yet Implications for Real-Time termination problem is challenging even at the source of the checksum error may be 40GbE and 100GbE 10GbE (sometimes even at 1GbE), where just one or two data bytes, or in the packet Because 40GbE and 100GbE technol- processors get clobbered by the effort of header. The solution is to modify the stack ogies are currently driven by the massive running the protocol stacks. It stands to behavior to avoid dropping the packets in server-type markets, the ASIC-based so- reason that the termination problem will the presence of the CRC or checksum er- lutions, out of sheer volume necessity, will be 4x to 10x worse for the case of 40GbE rors, and to allow them to pass to the sig- target those applications, just as they have and 100GbE. In the commercial space, nal processing stage. in the 10GbE case. As pointed out, those the solution at 10GbE typically involves High-throughput real-time instru- applications have some fundamentally forms of protocol offload, whereby some mentation, communication and other sen- different requirements from the higher- or all of the protocol processing elements sor processing systems can receive multi- end real-time embedded applications. The are farmed out to a coprocessing ASIC. ple back-to-back packets burst at full line latter will therefore need to use solutions However, there are a variety of application speed on a regular or even a sustained ba- based on programmable logic, which al- characteristics unique to many real-time sis. In such cases, there are no spare cycles lows the solutions to be customized to the I/O applications that use 10GbE. These for flow control or retransmission requests problems in the application space, while

Reprinted from March 2009 SOLUTIONS Engineering

Switch Port Shipments Specification Ratification Year Year y0+3 Year y0+4 Year y0+5 Year y0+7 Year y0+9 maintaining a standard external software (y0) interface for compatibility with the Ether- 10GbE (802.3ae) - 2002 1.8M 12M net ecosystem. Figure 2 shows one such predicted from 2006 solution for 10GbE systems: AdvancedIO 10GbE (802.3ae) – actuals 2002 150K 320K <1M 7M Systems V1120 dual-channel conduction- and 2008 predictions cooled 10GbE interface module, based on 1GbE actuals 1999 20M 80M the Xilinx Virtex-5. (1998) Without question, 40GbE and 100GbE will arrive in the real-time em- Table 1 The 10GbE switch port shipment predictions and actual shipments in bedded space, but the arrival is still some January 2006, and revised predictions for 2011. The actual switch port years away and needs to be accompanied shipments for 1GbE are also shown. Source: ’Oro, Infonetics by the same kinds of innovation that make 10GbE suitable for the space. The good news is that, at 10 Gbits/s, Ethernet finally has sufficient bandwidth for most real- time high-speed applications, and there are real-time focused solutions existing today that you can use to get on board the Ethernet bandwagon. Once aboard, you can smoothly ride the Ethernet speed curve to 40GbE and beyond as your re- quirements scale, avoiding the software and architectural upheaval that resulted from previous iterations of integrating different high-speed technologies.

AdvancedIO Vancouver, BC. (604) 331-1600. [www.advancedio.com].

Figure 2 The AdvancedIO Systems V1120 Dual-port 10GbE conduction-cooled rugged XMC module, designed following the new VITA 42.6 standard, implements the architecture shown in the block diagram of Figure 1.

Reprinted from March 2009