VLASOV LAYOUT 1/25/12 5:10 PM Page 93 BEYOND 100G OPTICAL COMMUNICATIONS Silicon CMOS-Integrated Nano-Photonics for Computer and Data Communications Beyond 100G Yurii A. Vlasov, IBM T. J. Watson Research Center ABSTRACT dictated by tremendous growth of data traffic in networks due to bandwidth-intensive applica- Five criteria that are usually considered by tions such as virtualization, video on demand, IEEE standards committees for development of the need for massive storage, and the rise of next generation standards are broad market social networking. To meet these demands, the potential, distinct identity, and compatibility, as concept of warehouse-scale datacenters (WSCs) well as technical and economic feasibility. We is being developed [2] with large numbers of consider these criteria separately and show that commodity servers interconnected by massive the new emerging large-volume markets loosely numbers of high-bandwidth optical links [3]. defined as Computercom will demand new stan- While performance, reliability, power efficiency, dards and new technologies. We discuss how the and the like are very important attributes, the balance between single-channel bit rate, and main driving force for wide acceptance of high- number of wavelength multiplexed and spatially bandwidth optical interconnects in WSC data- multiplexed optical channels can help to satisfy centers is the cost efficiency. Novel technologies the need for huge total bandwidth, while keep- like silicon complementary metal oxide semicon- ing cost low and power efficiency high. Silicon ductor (CMOS) integrated nanophotonics CMOS-integrated photonics holds promise to promise to slash transceiver cost from the cur- become a technology of choice for wide deploy- rent few dollars per gigabit per second of ment of low-power and cost-effective optical input/output (I/O) bandwidth to less than a few interconnects for these new markets, and to cents per gigabit per second. These technologies, become a single solution addressing distances if developed in time, will be prone to dominate spanning from just a meter to 10km. the market. The second area, HPCS, is also becoming BROAD MARKET POTENTIAL ready for wide acceptance of massively parallel optical interconnects [4–7]. This is mostly neces- At least two markets are becoming poised for sitated by the ever increasing mismatch between wide deployment of broadband optical commu- computational operations currently reaching 10 nications — burgeoning Internet data centers Peta-floating point operations per second (flops) (IDCs) and high-performance computing sys- and available memory bandwidth that is limited tems (HPCSs). Both are experiencing in recent by severe constraints in providing high-band- years sustainable growth in volumes, revenues width electrical I/O links on cards and between and performance that is envisioned to be limit- racks [5–7]. With this tendency likely to extend ed, among other constraints, by availability of to the next decade, massive numbers of parallel cost-effective and power-efficient optical inter- optical links, on the order of 100 million, is connects [1]. In addition, if a cost-effective opti- expected in the HPCS that will be used to con- cal solution can become readily available, the nect racks, boards, modules, and chips together high-end and mid-range servers market will also [5, 6]. In order to be massively deployed in benefit from understanding that optical intercon- HPCS, optical links must provide reliable com- nects can be a viable business solution for munication while maintaining extremely low replacement of costly and bulky electrical cables power dissipation, on the order of just a few and backplanes. These new opportunities, with pico-Joules per transmitted bit. It is worth men- potential market volumes on the order of mil- tioning that, besides accounting for just electri- lions of parts per year, are forming a new appli- cal-to-optical (EO) and optical-to-electrical The views expressed in cation area that can be loosely called, as a (OE) conversion in transceivers, this power bud- this document are those of follow-up to the more traditional telecom and get should also include all electrical high-speed the author and do not datacom markets, a Computercom. serial (HSS) circuitry required to provide high- necessarily represent the Although these markets’ needs are similar, integrity electrical signaling on a card or module views of IBM Corpora- they are dictated by different economic drivers. for the I/O link. As with cost efficiency, the tion. The expansion of datacenter businesses is mostly power efficiency numbers required for wide IEEE Communications Magazine • February 2012 0163-6804/12/$25.00 © 2012 IEEE S67 VLASOV LAYOUT 1/25/12 5:10 PM Page 94 adoption of optical interconnects in the HPCS of HSS links supports 10 Gb/s data rates and, With the new are approximately 20–50 times lower than avail- with the maturation of scaled CMOS technology, able with today’s technology. 28 Gb/s HSS links were developed. However, generation of MM further bandwidth scaling is severely limited by fibers, it is possible DISTINCT IDENTITY CMOS device performance even in the advanced to envision that 28 nm CMOS node, as well as by increasingly AGGREGATE BANDWIDTH difficult and power-hungry equalization that is traditional With the recent adoption of new IEEE 802.3ba required to restore signal integrity and retime at VCSEL/MM links standards for 40 Gb/100 Gb Ethernet [8], some the end of the on-card HSS link even for very of these growth areas are covered, and technolo- short distances of a few inches. Besides consider- can be extended gies and solutions are being currently developed. able additional power dissipation, the retiming beyond current However, it is quite possible that actual growth increases the link latency, which for some appli- 100 m; however, in network traffic will demand earlier deploy- cations is highly undesirable. Therefore, it is ment of WSC datacenters [1–3]. In addition, the likely that the line rates for HSS links will not it is difficult to growth in network traffic will put serious stress increase significantly beyond 25 Gb/s, and the expect the reach on the capability of network switches and routers required IO bandwidth has to be provided to keep up with the pace, and will stress the abil- instead by an increasing degree of paralleliza- approaching 1000m. ity of current backplane architectures to provide tion. necessary connectivity [9]. Current plans for the development of the next generation HPC sys- WAVELENGTH-DIVISION MULTIPLEXING tems that can deliver performance at the Exaflops level (1018 FLOPs) by the end of the To satisfy the bandwidth demands, the degree of decade are also under consideration by govern- WDM has to increase beyond the current four ments in the United States and Japan, followed WDM channels of the 100 GbE standard. WDM by China, Russia, and India. All of these trends technology can potentially provide the terabit will soon call for new standardization cycles for bandwidth offered by an SM fiber, especially 400 Gb and 1 Tb optical interconnects, and will promising at relatively short distances of just a demand an investment in the development of few kilometers where dispersion penalties do not novel technologies that can provide 10–50 times yet play a significant role. However, the number improvement in cost and power efficiency to of WDM channels and, correspondingly, the meet these new market demands [9]. Although it channel spacing will most likely be limited to less is expected that further development of next- than 10 due to cost and power limitations associ- generation optical transceivers based on tradi- ated with the technical solutions that can pro- tional multimode optical fibers and directly vide reliable uncooled operation and a high modulated vertical cavity surface-emitting laser degree of integration. (VCSEL)/multimode (MM) might provide some of this scaling, serious difficulties are envisioned LINK BUDGET to meet aggressive power and cost savings Maintaining the link integrity at longer distances required for future WSC and HPCS, while simul- is always more difficult, and requires more opti- taneously maintaining reliability. cal power and extended link budget than for short-reach interconnects. Since the deployment REACH of a new generation of optical links will happen The current 100GbE standard regulates either in a controlled environment of a large datacen- short-reach VCSEL/MM-based links for less ter or, even better, in controlled buildings specif- than 100 m distances or single-mode (SM) links ically designed to host HPCSs, the 6–11 dB link serving over 10 km distances. For WSC and budget typical for 100 GbE long reach applica- Exascale HPCS markets, intermediate distances tions might be excessive and not economically are of interest covering distances from a few viable. Even assuming the optimistic power slope meters up to 2 km [10]. Growth in these markets efficiency of the next generation DFB laser has already been identified as a gap needing to arrays exceeding 20–30 percent and assuming be filled by new standards and new technologies 3–5 dBm of laser power per channel, the inser- [9]. With the new generation of MM fibers, it is tion loss for transmitter and receiver would most possible to envision that traditional VCSEL/MM likely reduce the budget to a moderate 2–3 dB links can be extended beyond the current 100 m; [10]. Cost-effective packaging can be a dominat- however, it is difficult to expect the reach to ing factor for improvement of the link budget approach 1000 m. New technologies are needed since it will be mostly limited by coupling losses to fill this gap that starts to appear, including from the DFB laser to the photonic integrated directly modulated SM VCSELs, directly modu- circuit chip. lated (DML) distributed feedback (DFB) lasers, and silicon photonics [11].