Compute Communication Interfaces and Active Optical Cables

Home , Compute Express Link, ESATAp, Expansion card, Parallel SCSI, Serial Attached SCSI, System bus

MIT Microphotonics Center

Petre Popescu - Fellow Design Engineer AMD Technology Group

April 5, 2011 Agenda

1. Compute Communication Interfaces – overview, requirements, and trends 2. Compute Communication Interfaces - optical interconnect 3. Chip-to-chip Optical Interconnect – AOC challenges 4. Compute Communication Interfaces and optical interconnect - summary

Note: Any opinions or recommendations expressed in this presentation are those of the author and do not necessarily reflect the views of AMD.

2 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Compute Communication Interfaces

Overview, Requirements, and Trends

3 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Evolution of Compute Communication Interfaces (Wire-line)

. The evolution of compute communication interfaces is shown in the figure. A higher- speed generation emerges every ~3.5 years. . Ethernet interfaces (GbE, 10GbE and 100GbE) are governed by various IEEE 802.3 standards. . SATA (Gen2 and Gen3) is a working document of T10, a technical committee of International Committee for Information Technology Standards (INCITS).

. All other interfaces shown in the figure H. Wang and J. Lee, JSSCC April 2010 (USB, PCI-E) and others discussed in this section are governed by their respective special interest groups (SIG).

4 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | SATA/SAS • Serial advanced technology attachment (serial ATA or SATA) is an interface connecting host bus adapters to mass storage devices (SSD, hard disk drives (HDD), and optical drives (DVD-RW)). Industry compatibility specifications can be found at serialata.org. • SATA host adapters and devices communicate via a high- speed serial cable over two pairs of conductors. • SATA Revision 3.0 6 Gb/s standard was released in 2009. • Serial Attached SCSI (SAS) is a computer bus used to move data to and from computer and storage devices. SAS depends on a point-to-point serial protocol that replaces the parallel SCSI bus technology. SAS offers backwards- compatibility with SATA Revision 3.0 drives. • SAS bandwidth will increase: 12 Gb/s and 24 Gb/s revisions are expected in 2013 and 2017 respectively. • External SATA (eSATA) provides a variant of SATA meant for external connectivity. It has revised electrical requirements, cables and connectors: • Power over eSATA (eSATAp) or eSATA/USB combo. • eSATAp port can supply 12V to power a HDD or DVD-RW (desktop) or 5V to power up a HDD/SSD (notebook).

5 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | USB

. Universal serial bus (USB) is a specification to establish communication between devices and a host controller (usually a personal computer). . USB connectors also supply electric power, so many devices connected by USB do not need a power source of their own. . The USB 1.1 specification was released in September 1998 and is currently in use. It allowed for a 12 Mb/s data rate for higher-speed devices such as disk drives, and a lower 1.5 Mb/s rate for low- bandwidth devices such as joysticks. . The USB 2.0 specification was released in April 2000 and is currently in use. It allowed for 480 Mb/s, a forty-fold increase over 12 Mb/s for the original USB 1.1. . The USB 3.0 Specification was published in 2008. Its main goals were increased data transfer rate (up to 5 Gb/s, SuperSpeed), decreased power consumption, increased power output, and backwards compatibility with USB 2.0. The first USB 3.0-equipped devices were presented in January 2010. . The next generation USB 4.0 (?) most likely will be 10 Gb/s (?).

6 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | DisplayPort • Video Electronics Standards Association (VESA - website VESA.org) • Embedded DisplayPort (eDP), a companion standard to the regular DisplayPort interface, is intended for use in netbook, notebook, and tablet mobile PCs, as well as all-in-one desktop systems. • eDP v1.3 also provides information on how to use the new 5.4 Gb/s (per lane) DisplayPort main link data rate High Bit Rate 2 or HBR2. • eDP v1.3 includes a feature for panel self-refresh (PSR) intended to reduce system power usage, and therefore improve battery life in portable systems. PSR features are expected to be available by 2012. • As a display interface, DisplayPort uses a unique data structure (four data lanes, no clock) that allows for ongoing expansion of capabilities. As new system capabilities and applications are developed, features and capabilities can be added to DisplayPort without affecting older DisplayPort PC systems or monitors. • I don’t expect lane data rate increase from 5.4 Gb/s in the foreseeable future.

7 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | HDMI

. High-definition multimedia interface (HDMI) is a compact audio/video interface for transmitting uncompressed digital data. . HDMI specification is developed by HDMI Consortium. HDMI Licensing LLC is responsible for licensing the HDMI specification. . HDMI connects digital audio/video sources (such as set-top boxes, HD-DVD (Blu-ray) players, camcorders, personal computers (PCs), video game consoles such as the PlayStation 3, Xbox 360) to compatible digital audio devices, computer monitors, video projectors, and digital televisions. . The most-used version, HDMI 1.3a (3 lanes up to 2.97 Gb/s per lane plus clock) supports refresh rates up to 120 Hz and has the ability to automatically and accurately adjust the audio to maintain lip-sync with the video image. . HDMI 1.4 (4K resolutions, up to 3840X2160) supports color spaces designed specifically for digital still cameras. 4k2k provides the same resolution as many state-of-the-art digital theaters – up to 4 times the resolution of 1080p. . The bandwidth is sufficient for the near-term market needs. Higher resolutions, 3D, and deep color content at 240 Hz refresh rate may push the bandwidth higher.

8 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | PCI-E

. Peripheral Component Interconnect Express (PCI Express), officially abbreviated as PCIe (also written as PCI-E), is a computer expansion card standard designed to replace the older PCI and PCI-X bus standards. . Format specifications are maintained and developed by the PCI Special Interest Group (PCI-SIG), a group of more than 900 companies that also maintain the PCIe specifications. . PCIe has numerous improvements over the older bus standards, including higher maximum system bus throughput, lower I/O pin count, smaller physical footprint, better performance-scaling for bus devices, a more detailed error detection and reporting mechanism, and native hot plugging. More recent revisions of the PCIe standard support hardware I/O virtualization. . The PCIe electrical interface is also used in a variety of other standards, most notably ExpressCard, a laptop expansion card interface. . PCIe Gen 3.0 (8 Gb/s per lane full duplex) is the latest standard for expansion cards that is available on mainstream personal computers. . Next-generation PCIe Gen 4 is expected in 2014 and most likely will run at 16 Gb/s per lane full-duplex.

9 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | HyperTransport

. HyperTransport (HT), formerly known as Lightning Data Transport (LDT), is a technology for interconnection of computer processors. It is a bidirectional serial/parallel high-bandwidth, low-latency, point-to-point technology. . HyperTransport Consortium is a membership-based, non-profit organization that is in charge of promoting and developing HyperTransport Technology. . Hyper Transport Consortium website: http://www.hypertransport.org . Coherent HyperTransport links are used to communicate cache- coherency information between multiple processors in a multi-processor system configuration that share data. . HT3, currently in use, has a data rate 6.4 Gb/s per lane. Next generations will increase the data rate to 10 Gb/s or 12 Gb/s. The increase is expected in 2014. . There is a trend to make HT physical implementation and HT PHY requirements the same or as close as possible as PCIe. PCIe Gen 3 has a data rate of 8 Gb/s per lane.

10 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Compute Communication Interfaces - Summary

. We reviewed six communication interfaces: – Computer to peripherals: SATA/SAS, USB, DP, and HDMI. – Compute chip-to-chip communication: PCI-E and HT. . A common characteristic is continuous data rate increase. PCI-E has the highest data rate of 8 Gb/s per lane. Data rates of 16 Gb/s per lane and higher are expected in 2014 and beyond. . Interface specifications are controlled by standards organizations, special interest groups (SIG), and consortiums, or are proprietary. . Interfaces are standardized and optimized for a specific functionality in the compute environment (PC, laptop, server, etc.). They will continue to be used in the foreseeable future. . Through-silicon via (TSV) technology and die stacking will impact the way we architect our systems. Memory interfaces will be significantly changed and have not been included in this review.

11 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Compute Communication Interfaces

Optical Interconnect

12 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Active Optical Cables and Optical Engines

. Computer-to-peripherals specifications define copper cable-based interfaces. As the data rate is moving higher, the maximum length of the copper cable is moving lower. . Repeaters are available for applications in which the copper cable length exceeds the specified maximum length. . Active optical cables, or AOC, are available for most computer-to- Hitachi Cable peripherals interfaces (HDMI, USB, DP, and SATA). For SATA/SAS, InfiniGreen interface specifications define the use of AOC. . Proprietary AOC and Optical Engines (MicroPod and others) have been developed for HPC applications and high-end servers. . AOCs and Optical Engines have been optimized depending on application: – AOC with O/E conversion embedded and fixed cable length (like Hitachi Cable – InfiniGreen). Since O/E conversion is embedded, there is no need to clean optical connectors and overall system testing is maintained electrically. – Optical Engines with miniature detachable connector (like PRIZM, developed by US Conec for MicroPod). Custom-length optical cables can be used to lower cable cost and better use of the available space. . AOC adoption will continue to grow due to lower power and better Avago - MicroPod signaling performance compared to copper cables. Higher volumes may result in lower cost and will increase adoption.

13 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Compute Communication Interfaces - Chip-to-Chip Interconnect

SoC2

. The most common chip-to-chip interconnect specifications are PCI-E and HT/cHT. Electrical links (lanes) are full-duplex differential connections and can include one (as shown) or two connectors. The total link length (no repeaters) is up to 26”. . There are differences between PCI-E and HT specifications. The trend is to make HT PHY electrical requirements similar to PCI-E such that, from a physical implementation point of view, they will look the same. Achieving low latency is critical! . Another common characteristic is the capability to shutdown all lanes except one. The active lane can be programmed to operate at the minimum data rate in a low-power mode. . Current compute chip-to-chip interconnects have a maximum data rate of 8 Gb/s and are implemented as copper-based differential transmission lines. The expectation is that the required data rate will grow to 16 Gb/s in 2014 and beyond.

14 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Chip-to-Chip Optical Interconnect – AOC Challenges

SoC2

P1=P11+P12 P12 P11

Added Cost . We can assume that AOC ball assignment (or electrical connector) will allow placing AOC terminals very close to the SoC. . The main challenges for AOC adoption for chip-to-chip interconnect are the energy efficiency and cost.

. To make it attractive, we have to bring the overall energy efficiency (P1) to10 pJ/bit (5 pJ/bit+5 pJ/bit) or lower. The power required for multiplexing and de-multiplexing, clock

generation, and CDR functions are included in P11. . AOC is an added cost and must be compensated for by reducing the cost of the boards (fewer layers and less complexity), lower-cost connector (if used), and better energy efficiency (cost of ownership).

15 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Compute Communication Interfaces and Optical Interconnect - Summary

• AOC solutions have been developed for computer-to-peripherals interfaces, replacing copper cables when they become uneconomical (repeater added and/or higher power dissipation). The adoption is limited by AOC cost. • The data rate on computer-to-peripherals interfaces is continuously growing and makes the AOC alternative more attractive, assuming that the cost will go down as the volumes go up. • AOC is a promising solution for chip-to-chip interconnect owing to its potential advantages of low latency, high bandwidth, high density, and low power consumption. AOC with embedded E/O conversion will have a minimal impact on system design, manufacturing process, and test. • The volumes required for computational systems chip-to-chip interconnect are very high and will require AOC solutions developed specifically for this application: 1. Maximum cable length is 1 m or less 2. Low-cost E/O conversion devices and manufacturing processes 3. Very low-power operation (5 pJ/bit or lower per terminal assuming very short interconnect between AOC terminal and host SoC) 4. Very low-power-down mode per individual link and fast wake-up • AOC adoption for chip-to-chip interconnect will depend on collaboration among all parties (SoC/system development, AOC designers, passives, and E/O conversion device development) to set a common roadmap and a high-level specification.

16 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 | Thank You!

Trademark Attribution AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks of their respective owners.

17 | Compute Communication Interfaces and AOC – Petre Popescu | MIT Microphotonics Center April 5, 2011 |