Energy Considerations in Data Transmission and Switching

Rod Tucker

Department of Electrical & Electronic Engineering University of Melbourne Summary

• Big‐picture view of data and energy in network & DCs • Snapshot of 2017 and projections for 2021 • Key energy bottlenecks – Access networks – Servers in data centers – Data storage – Switches and routers – Global optical transmission • Gap between technology limits and reality The Internet Storage Servers Switches Router Routers etc Global Data Core Network Base Center Station Switch Data Center Data Networks Center Metro/Edge Access Network Network Data Center Content Distribution OLT ONU Data Networks Router Center This talk Fiber Optical TX/RX Router Data usage in 2017 ‐ 1.5 Zettabytes >15 ? 12 Zettabytes/year 200 Tb/s 3200 Tb/s 800 Tb/s (40 %) 1.5x1021 bytes/year* Global 500 Tb/s (avg.) Data Core Network 135 kb/s (avg.) Center

Data Center Data Networks Center Metro/Edge Access Network Network Data Center 300 Tb/s Content Distribution 3.7 billion users Data Networks (50 % of world pop.) Center 300 Tb/s (60 %)

Sources: Cisco, VNI*, 2016‐2021, (2017); Google; Facebook; Akamai 2021 ‐ Twice as much data as 2017 > 40 ? 25 Zettabytes/year 300 Tb/s 6.4 Pb/s 1.6 Pb/s (30 %) 3.3x1021 bytes/year* Global 1Pb/s (avg.) Data Core Network 220 kb/s (avg.) Center

Data Center Data Networks Center Metro/Edge Access Network Network Data Center 600 Tb/s Content Distribution 4.5 billion users Data Networks (60 % of world pop.) Center 700 Tb/s (70 %)

Sources: Cisco, VNI*, 2016‐2021, (2017); Google; Facebook; Akamai Data, energy and efficiency

Data (~ 25 % p.a. Increase) Index Net increase in energy consumption

1 Time

Energy intensity 1/ɳ (Joules per bit)

Energy efficiency of network equipment (bits per Joule) Network energy consumption

Inventory‐based estimate

Van Heddeghem et al., Computer. Comms., 2014 Power consumption of the global Internet

1012 Global electricity supply ~ 4 % ~ 3 % ~ 2% Access Network 1011 Network Total

1010 Data Centers*

Power Consumption (W) 1 Nuclear power 109 station

2010 2015 2020 Year *Scaled to global Sources: Tucker, OFC 2011 (updated), Van Heddeghem et al., 2014 ( ), Shehabi et al*., 2016 8 Power consumption of the Internet (2017) 5x109 W 5x1010 W Global 10 Data Core Network 6x10 W Center

Data Data Center Networks Metro/Edge Center Access 33 W/user Network Network Data Data centers: Content Center 14 W/user Distribution 3.6 billion Network: Data Networks users 19 W/user Center ? W

1.2x1011 W Energy per customer bit ~ = 240 μJ 180 μJ (2021) 500x1012 b/s 4 kb/J Routers

Cisco

Juniper

10 Router energy per bit Router Energy Efficiency 10000

Cisco AGS

30% p.a. improvement 1000

Wellfleet BCN

100 Cisco GSR 12000 15% p.a. improvement (nJ per bit) Cisco GSR 12000b

Cisco CRS‐1 Avici TSR

Energy Consumption Cisco CRS‐3

nano‐Joules per bit 10 ALU7750 CRS‐X

Cisco NCS 6000 Juniper T4000 1 1985 1990 1995 2000 2005 2010 2015 Year

Sources: Nielsen, 2010, Cisco and Juniper Data Sheets, 2015 11 High‐end core router

Line Card Buffer

Forwarding Engine Routing O/E Switch Fabric Tables

I/O Line Card Buffer Power supply inefficiency Buffer Forwarding Engine Routing O/E Switch Engine Fans and Control blowers Control Plane Data Plane

Energy/bit 0.1 nJ 0.5 nJ 0.1 nJ 0.6 nJ 0.1 nJ 0.1 nJ Fraction of Total

Software‐Defined Optical Switching? Networking? Source: G. Epps, Cisco, 2015 12 Some (ns) switch fabrics m x m Benes SOA Gate Array CMOS 1 1 2 2 3 144 x 144 @11 Gb/s SOA Gate 3‐dB Coupler m m Optical Dilated ring resonator switch 1 1

2 2

Wavength CMOS m Converters driver m

Optical Energy per bit per crosspoint

Energy per Bit per Crosspoint (J) 10 10 10 10 ‐10 ‐11 ‐9 ‐8 10 * VSC3144 Ring resonator Bit Rate per Wavelength (Gb/s) Semiconductor optical Wavelength routed (AWG) amplifier array Broadcom 256x256 Switch Fabric 100 Source: Tucker, OFC, 2012; CMOS (Including SERDES) Stone, Yesterday 1000 Optical transmission

> 100 million km Transmission energy consumption 10 5 Marconi Wireless Trans‐Atlantic 10 3 Telegraphy Fessenden Trans‐Atlantic Coax 10 Optical + Regen NY ‐ Paris First Trans‐ Atlantic Optical + EDFA 10 ‐1 Newhaven ‐ Azores TAT‐1 ‐3 10 Key West ‐ Havana TAT‐3 TAT‐8 10 ‐5 TAT‐5

Energy/Bit/1000 km (J) TAT‐10 ‐7 ~15% improvement p.a. TAT‐9 10 TAT‐11 TAT‐12/13 10 ‐9 Terrestrial systems Short‐range 100 Gb/s link 10 ‐11 1840 1860 1880 1900 1920 1940 1960 1980 2000 2020 2040 Year

Source: Tucker, JSTQE 2011 16 Gap between limits and reality 10‐4 ↓ 15 % per annum 10‐5 “Reality”

10‐6 Transmission 10‐7 Global Routers B x 103 10‐8 Technology “Limits” Network X ‐9 X

Energy per Bit (J) 10 Transmission 10‐10 A x 102

‐11 Devices 10 Crosspoint Sub‐systems 10‐12 2010 2015 2020 2025 Year

Source: Tucker, JSTQE 2011. (adapted) 17 Gap between limits and reality (A)

• Transmission: ‐ SERDES, Clock recovery, DSP, amplifiers, hop count, protection and restoration, etc

• Routers and Switches: ‐ Routing plane, switch controllers, buffers, hop count, protection and restoration, etc.

18 Gap between limits and reality (B)

Protocols, DSP, device efficiency, Real interconnects, system margins and penalties, etc

PTotal Ideal

Poverhead Power consumption Inefficiency Traffic (b/s) P function Overhead Function Subsystem

Management and power Energy per bit Ideal overheads, interconnects, etc. Traffic (b/s)

19 Diurnal traffic variations Italy North America

* Real Busy‐hour traffic growth: 51% p.a.

Average traffic growth: 32% p.a. Ideal * Cisco, VNI*, 2016‐2021, (2017) Power consumption

20 Data centres

• Servers • Switches and routers • Interconnects • Data Storage

http://www.google.com/about/datacenters/gallery/#/all Zoom in to server chip

Intel 14‐nm chip • Energy dominated by 1 μm ‐ wires at high speed ‐ leakage currents at low speed

Wire M12 Energy Power per op.

M9 leakage M1 Clock/data rate

• Chip‐level optical interconnects ‐ limited to upper layer(s) ‐ alleviate the pin‐out problem Chip energy 10‐10 12‐m 10‐11 PMOS SiGe VCSEL 50 Gb/s* 10‐12 FinFET RX 64 Gb/s** 10‐13 (1‐m)3 laser*** ‐14 10 Total CMOS chip energy including wires ↓ 25 % p.a. 10‐15 45 130 32 22 ‐16 CMOS gate 10 90 11 energy 65 ‐17 45 (100‐nm3) laser*** 10 32 22 Link goal

Energy per Operation (J) ‐18 10 Feature 11 size in nm 10‐19

10‐20 1970 2000 2010 2020 2030 1980 1990 Year Sources ITRS ’97‐’10 Roadmaps; Hinton et al., JSTQE 2008; Tucker, JSTQE, 2010, Miller, OFC, 2016*** https://www.e3s‐center.org/ ; Belfiore, JSSC, 2017* ; Cevrero, ISSCC, 2017** Zoom out to data centers (2017) 5x109 W

3200 Tb/s

2x1010 W

25+ Gb/s 100+ Gb/s

Servers: 2x1010 W Energy per bit = = 6.3 μJ Energy per customer bit = 40 μJ 3200 Tb/s Network:

Energy per network bit = 1.5 μJ Energy per customer bit = 10 μJ

24 Gap between limits and reality – Data Center 10‐4 ↓ 15 % per annum Current 10‐5 Trends

10‐6

10‐7 Switches Data Technology “Limits” Centers 10‐8

‐9 Links Energy per Bit (J) 10 x 106

‐10 10 Crosspoint 10‐11 Devices 10‐12 Links Sub‐systems

2010 2015Year 2020 2025

25 Take home • Energy is an important issue – Network consumption: 19 W per user – Data centers: 14 W per user • Access networks dominate, followed by data centers • Technology improvements barely keeping up with data growth • Potential for 3+ orders of magnitude improvement – Device level – Circuit level – System level