Silicon for next generation computing systems

O / I

l

a

ic t

p O Tutorial given at the European Conference on Optical Communications, September 22, 2008

http://www.ecoc2008.org/documents/SC2_Vlasov.pdf Yurii Vlasov IBM Research Division

© 2008 IBM Corporation TJ Watson Research Center, Yorktown Heights, NY www.research..com / photonics Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon : – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

2 ECOC September 2008 www.research.ibm.com / photonics September 2008 Hierarchy of interconnects in HPC systems Backplanes

Blades Server Server Cabinet I/O Cabinet

Server Server Cabinet Switch Cabinet Cabinet Server Server Cabinet Cabinet Switch Cabinet Server … Server … Cabinet… Cabinet

Shelves . Multiple blades on shelf interconnected through an electrical backplane . Optical interconnects between shelves – Interconnects moderated by the switch . Many Tb/sec off node card and growing (20-50% CGR)

3 ECOC September 2008 www.research.ibm.com / photonics September 2008 Moore’s law in HPCS

100 PB/s Biggest machine 1 PB/s

10 TB/s

100 GB/s

After Jack Dongarra, top500.org 209 of the top 500 supercomputers have been built by IBM Sum total performance 14.03596 PF

4 ECOC September 2008 www.research.ibm.com / photonics September 2008 Roadrunner - #1 in the world

19,872 processors, 1 PFlops Triblade: 2 Cells Expansion module 2 Opterons

Area: • 296 racks 3.9 MW Power: 0.35 GF/Watt Cluster of 18 Connected Units (CU) 104 TB aggregate memory

InfiniBand 4x DDR fat-tree fabric 2-stage fat-tree; all-optical cables Full bi-section BW within each CU: 384 GB/s Half bi-section BW among CUs: 3.45 TB/s

5 ECOC September 2008 www.research.ibm.com / photonics September 2008 Optical interconnects between racks and switches

Infiniband active optical cables one of 26 switches

From Ken Koch (LANL) presentation at SC2008 conference

6 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

7 ECOC September 2008 www.research.ibm.com / photonics September 2008 Rack-to-Rack: Parallel Optics in Supercomputers Length: ~100m # links: ~5-10K Price: few$ per Gbps BW: ~10Gbps/link Reliability! Power: ~50mW/Gb/s/link

MareNostrum (Barcelona) 62TFlops MareNostrum central switch racks: About 5000 fiber cables About 1700 fiber cables/rack today

8 ECOC September 2008 www.research.ibm.com / photonics September 2008 Examples of silicon photonics integration Luxtera

4 TRx @ 10Gbps: CMOS integrated: Si modulators+Ge detectors; Mux, VOA, CMOS drivers+TIA/LA; CDR, ADC, monitors, etc. Co-packaged lasers and fiber couplers A.Narasimha et al, OFC 2008, paper a1451_1 2.3W total  ~50pJ/bit

1 TRx @ 10Gbps: CMOS integrated: Si modulators, CMOS drivers+amplifiers; CDR etc. Co-packaged detectors, lasers and fiber couplers 0.4W total  ~40pJ/bit

Can Si Photonics bring $/Gbps down?

9 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

10 ECOC September 2008 www.research.ibm.com / photonics September 2008 Optics between cards: Optical backplane

Length: ~50cm # links: ~10K Price: <1$ per Gbps BW: ~10Gbps/link Reliability!! Power: ~10mW/Gb/s/link . Electronics – Fast CMOS designs with low power consumption . Opto-electronics – Separate from processor to avoid heat/lifetime issues – VCSEL/PD: 2D-arrays (cost) . Optical channel – Fibers, fiber flexes – PCB embedded waveguides . Coupling – Passive alignment and positioning Courtesy of IBM ZRL – Christoph Berger, Bert Jan Offrein, Martin Schmatz

11 ECOC September 2008 www.research.ibm.com / photonics September 2008 Multilayer optical waveguides in PCB

Light 4x12 waveguides on 250μmx250μm grid incoupling (proof of feasibility)

Light outcoupling at end facet 250 m

250 m

4x12 waveguides

Courtesy of IBM ZRL – Roger Dangel, Folkert Horst, Bert Offrein

12 ECOC September 2008 www.research.ibm.com / photonics September 2008 Chip-to-chip link. Length: ~1cm # links: ~100K Price: <0.1$ per Gbps BW: ~1Tbps/link Reliability !!! Power: <10mW/Gb/s/link IBM Terabus project

Optomodule Transceiver IC OE SLC Lens Array Optocard 16 Channels TRX1 ←→ TRX2 TRX1 Waveguides in PCB TRX2

2 17mm 240 Gb/s bi-directional 9 mW/Gb/s per unidirectional link Courtesy of IBM Terabus team – Jeff Kash, Clint Schow, Fuad Doany et al

13 ECOC September 2008 www.research.ibm.com / photonics September 2008 Penetration of optics into HPC on-chip

al rack back plane module on-board ci er m WW volume in 2006 m co C HP

Single HPC machine will contain a similar number of parallel optical channels as currently exists today in all telecommunications links worldwide All future dates and specifications are estimations only. Subject to change without notice. Courtesy of M.Taubenblatt (IBM)

14 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

15 ECOC September 2008 www.research.ibm.com / photonics September 2008 On-chip optical interconnects

Length: ~0.1-0.3cm Price: <<0.01$ per Gbps # links: ~100K Reliability !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! BW: ~1Tbps/link Power: <1mW/Gb/s/link

Footprint Power Bandwidth Loss Latency

Cell processor (IBM/Toshiba/Sony) 9 cores, 256 GFlops,

16 ECOC September 2008 www.research.ibm.com / photonics September 2008 Si photonics vs Si nanophotonics Rack-to-rack; board-to-board On-chip Replace Cu with fibers Goal: Replace Cu with Si Main driver: cost/bit Main driver: power/bit ~10 Tx on a single chip ~1000 Tx on a single chip >40Gbps aggregate >1Tbs aggregate is a must 1. Leverage CMOS fab as much as 1. Integration within high-performance possible to reduce the cost/bit CMOS stack requires aggressive 2. Since high-performance CMOS is not scaling of footprints required, previous gen CMOS fab is 2. Advanced CMOS is a must (e.g. 45nm) preferred (e.g. 130nm) 3. In 3D CMOS stack can provide 3. No change in server architecture significant architectural advantages. 4. Footprint, power, loss is not as 4. Cost is not as important; relevant important – relevant comparison with comparison with CMOS Cu BEOL Cu cables wiring

5. Direct competition with other 5. Direct competition with low power technologies (e.g. InP) BEOL Cu

17 ECOC September 2008 www.research.ibm.com / photonics September 2008 On-chip global interconnects For 130nm, 1.2V BGA Logic: ~0.1-1pJ/bit Interconnects: on-chip: 3pJ/bit (3mW/Gbps) off-chip: 40-60pJ/bit (40mw/Gbps)

M1 pitch scaling RC delay is increasing exponentially Severe bandwidth limitations In a link with repeaters: power/Gbps/length Energy/bit scaling slows Bandwidth needs growth faster for multicore processors Exponentially growing power consumption BGA pitch scaling is limited – off-chip bandwidth bottleneck

After ITRS 2007

18 ECOC September 2008 www.research.ibm.com / photonics September 2008 1000

10

0.1 Biggest machine

Processor Interconnect bandwidth (TB/s) 0.5B/FLOP

19 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

20 ECOC September 2008 www.research.ibm.com / photonics September 2008 Silicon nanophotonic waveguides 1 meter waveguide with 0.3dB/cm

S.Spector et al, IPR, paper IThE5 (2004)

Ultra-high optical confinement 1.3Tbps through a single waveguide mode x-section 0.1m2 With 2um BOX losses ~1-2dB/cm Sharp bend (R~3um) loss <0.001dB/turn

Chromatic dispersion (103 of D in fibers)

Nonlinear optical effects (SPM, XPM, FWM, solitons,etc.) B.Lee et al, IEEE PTL, 20, 398 (2008)

21 ECOC September 2008 www.research.ibm.com / photonics September 2008 Silicon nanophotonic waveguides Photonic wire devices

Ultra-high optical confinement mode x-section 0.1m2 Record low loss 1.7dB/cm

Ultra-compact devices – CMOS scale ! •Sharp bends •Splitters •Crossings •Couplers •Dense waveguide arrays •Etc.

22 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

23 ECOC September 2008 www.research.ibm.com / photonics September 2008 Vision for 22nm CMOS (circa 2018) - 10 TFLOPs on a 3D chip

s l

a c i n f 36 “Cell” chip (~300 cores) g f i a O s / r I l t l

a l ca ic t i t p ca p i O t o p

p o System level study: i

h p c i IBM, Columbia, Cornell, UCSB - f h f c O - n Co-PIs: O Jeff Kash (IBM) Photonic Plane Memory Plane Keren Bergman (Columbia) Logic Plane Yurii Vlasov (IBM)

Logic plane ~300 cores Photonic layer is not only Memory plane ~30GB eDRAM connecting various cores, Photonic plane On-Chip Optical Network >70Tbps optical on-chip but also routes the traffic >70Tbps optical off-chip

All future dates and specifications are estimations only. Subject to change without notice.

24 ECOC September 2008 www.research.ibm.com / photonics September 2008 On-chip optical network

N modulators at Switch fabric

different wavelengths WDM

Detector

Modulator WDM WDM bit-parallel message N parallel channels N parallel channels

Message is encoded in N WDM channels Deserializer Serializer In a bit-parallel manner

Requires switching all the WDM traffic at Message Message once 1Tbps aggregate BW 1Tbps aggregate BW No optical buffering. Switching of the whole message once per Core 1 communication. Resembles circuit Core N switch network.

25 ECOC September 2008 www.research.ibm.com / photonics September 2008 Network on a chip: electronics vs photonics

75Tb/s off-chip & on chip switching

RX RX RX RX TX RX TX RX TX TX TX TX

• Electronics • Optics: Power is bandwidth x length dependent Power independent of bitrate and length On chip: buffer, receive and re-transmit Modulate/receive ultra-high bandwidth data every single bit at every switch stream once – no re-transmit (~15x communications power savings) Off-chip: even more power hungry (50) Off-chip and on-chip power and bandwidth and bandwidth limited by BGA count ______are equivalent Electronic network ~500W (~40x chip interconnect power savings) Broadband switch fabric is nearly free in power______dissipation  highly scalable Photonic network <80W

Power efficient computing. More FLOPs per Watt

All specifications are estimations only. Subject to change without notice.

26 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

27 ECOC September 2008 www.research.ibm.com / photonics September 2008 Wavelength multiplexers AWG Number of channels MMI Xtalk  Lattice filters Footprint Ring resonator filters Insertion loss Echelle grating Temperature tolerances Fabrication tolerances non-IBM -processor @3.3 GHz, 1.4 Vdd Si thermooptic coefficient 70oC n/dT=2*10-4 K-1

±15°C

25oC H. F. Hamann,et al, IEEE Journ. Solid-State Circuits, 42, 56 (2007).

28 ECOC September 2008 www.research.ibm.com / photonics September 2008 Wavelength multiplexers F. Horst, F.Xia, W.Green, S.Assefa, Y.-H. Kim, L. Sekaric, Y. Vlasov

 Ultra-compact (200x200um2)  Tolerant to on-chip “hot spots” ±15°C temperature variations  Low-loss <2dB insertion loss  FEOL CMOS integration  Small cross-talk 20dB

F.Xia et al OFC 2007 F. Xia et al Optics Express 15, 17106 (2007).

F. Xia et al Opt. Express, 15, 11934 (2007)

F. Horst et al SPIE Europe March 2008

29 ECOC September 2008 www.research.ibm.com / photonics September 2008 Cascaded Lattice filters

3 cascaded MZI filters

. 400x300um . Cross-talk 15db (13dB with 2 outliers) . Insertion loss 1.5dB

Phase error due to variations of the waveguide width of the order of just a few nm

F.Horst et al , SPIE Europe, Feb. 2008

30 ECOC September 2008 www.research.ibm.com / photonics September 2008 Echelle grating No active tuning. As-fabricated!

“safe” design

. 650x170um . Cross-talk 19db . Insertion loss 3dB

“extreme” design

. 250x200um . Cross-talk 17db . Insertion loss 4dB F.Horst et al , SPIE Europe, Feb. 2008 31 ECOC September 2008 www.research.ibm.com / photonics September 2008 Ring resonator WDM filters

40x70um (6 channels, 400GHz) Cross-talk 30db Insertion loss 1.3dB F.Xia et al Optics Express (2007). ±150C

 2 nm wide error-free operating window @ 10 Gbps  Temperature independent operation ±15°C Y. Vlasov, W. M. J. Green, and F. Xia, Nature Photonics 2008

32 ECOC September 2008 www.research.ibm.com / photonics September 2008 Prospects for ultimate integrated CMOS photonics

Die-to-die variation of a single ring Wafer-to-wafer variation of all rings

WIDTH GAP 193nm litho WIDTH GAP

AVERAGE 0.4802 0.1986 AVERAGE 0.4843 0.1960 3SIGMA 0.0077 0.0090 3SIGMA 0.0053 0.0087 2um RANGE 0.0065 0.011 RANGE 0.0168 0.0122

MEASURED 22 24 MEASURED 346 140

ITRS CMOS tolerances for different technologies CMOS generation Litho Total tolerance 3 130nm 248nm, OPC 40nm 65nm 193nm, OPC 35nm 45nm 193nm, OPC 16nm 22nm EUV 1.2nm Scaling of Si CMOS is done – performance will only increase with new generations of CMOS

33 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

34 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

35 ECOC September 2008 www.research.ibm.com / photonics September 2008 Modulator with external CW laser

36 ECOC September 2008 www.research.ibm.com / photonics September 2008 Broadband MZI modulator . Ultra-compact cross-section: ~0.1 m2 . Simultaneous localization of injected free Vb V carriers and optical mode profile: ac 1. Large change in carrier density during each ON- OFF modulation cycle 2. Nearly complete carrier/optical mode overlap . Ultra-compact bend radius < 5 m – Device footprint limited only by length of phase shifter

W. M. J. Green, et al Optics Express Dec. 2007

37 ECOC September 2008 www.research.ibm.com / photonics September 2008 Fast operation 5 Gbps: LMZM = 100 m

 Broadband, non-resonant modulator  High speed operation: 10 Gbps  Ultra-compact footprint: ~ 0.001 mm2  10-100X smaller than previous MZIs  Only 5x larger than microring modulator

38 ECOC September 2008 www.research.ibm.com / photonics September 2008 Low RF power

DC power 287uW RF power 51mW

Low RF power: 51 mW or 5 pJ/bit

10 Gbps: LMZM = 200 m

39 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

40 ECOC September 2008 www.research.ibm.com / photonics September 2008 Broadband Deflection Switch

Design Metrics: . High throughput – 1Tbps . Low latency – few ns . Ultra-compact footprint < 0.001 mm2 . Low crosstalk < -25 dB . High extinction ratio > 10 dB . Temperature independent operation ±15°C

41 ECOC September 2008 www.research.ibm.com / photonics September 2008 Switching Characteristics

INPUT THRU

Laser spot

DROP

Free carrier injection into central ring: . Drop port transmission suppressed by >15dB . Through port transmission increases to zero loss

Y. Vlasov, W. M. J. Green, and F. Xia, Nature Photonics 2, pg. 242-246 (2008).

42 ECOC September 2008 www.research.ibm.com / photonics September 2008 Multi-Channel 40 Gbps Operation

OFF - Drop OFF - Thru ON - Drop

 Wavelength-insensitive operation  Up to 360 Gbps throughput: 9 x 40 Gbps ~1 Tbps by reducing FSR

43 ECOC September 2008 www.research.ibm.com / photonics September 2008 Broadband High-Throughput 360 Gbps Switch

Cascaded switch fabric

 Error-free (BER > 10-12) operation @ 40 Gbps  Power penalty < 0.3 dB in switch ON state

44 ECOC September 2008 www.research.ibm.com / photonics September 2008 Electrical injection

Photolithography

CMOS integration

Cascaded switch fabric

45 ECOC September 2008 www.research.ibm.com / photonics September 2008 Outline . Hierarchy of interconnects in HPC . Active cables for rack-to-rack communications . Board level interconnects . On-chip optical interconnects – CMOS integration challenges – Photonic network on a chip . Silicon nanophotonics: – WDM – Light sources – Modulators – Switches – Detectors . Conclusions

DISCLAIMER. The views expressed in this document are those of the author and do not necessarily represent. the views of IBM Corporation.

46 ECOC September 2008 www.research.ibm.com / photonics September 2008 Challenges for on-chip nanophotonics

Current Gen. CMOS +1 Gen. CMOS CMOS scaling

Area 2X smaller Power per circuit 2.8X lower Photonics scaling On-chip networks Tele- and Data-com

Best available integrated photonics Area 10-20X (either InP or CMOS Si/Ge) Power 30-100X

BW ~40-1000Gbps Power ~ 50pJ/bit Area ~0.5cm2 Equivalent to 5-10 generations of CMOS

47 ECOC September 2008 www.research.ibm.com / photonics September 2008 Summary

O / I

l

a

ic t

p

O

. Requirements from system level should define the requirements and performance metrics for individual devices . These requirements are very tough and almost impossible to meet . Nevertheless at the moment it looks that there is a room for innovation

48 ECOC September 2008 www.research.ibm.com / photonics September 2008