Thermal Management Roadmap Cooling Electronic Products from Hand-HeldDevices to

Richard C. Chu - IBM

May 16, 2003, Rohsenow Symosium Cambridge, MA ThermalThermal ManagementManagement TechnicalTechnical WorkingWorking GroupGroup (TWG)(TWG)

Richard C. Chu, IBM (Chair) Jogenda Joshi (Co-chair)

Avi Bar-Cohen, U. of Maryland Gregory M. Chrysler, Intel Darvin Edwards, TI Suresh Garimella, Purdue U.

Magnus Herrlin, Telcordia Larry Mok, IBM

Donald Price, Raytheon Bahgat Sammakia, SUNY-Binghamton Roger Schmidt, IBM Lian-Tuu Yeh, Boeing

Dedicated to the Advancement of North American Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal ManagementManagement RoadmapRoadmap -- ScopeScope

- Introduction • Overview • Review of thermal design requirements • Review of thermal design requirement matrix • Review of current product cooling designs • Review of cooling technologies • Review of advanced cooling technology development activities - Outline of Thermal Technology Needs • High performance product sector • Cost performance product sector • Telecommunications product sector • Hand held product sector • Harsh environment (automotive) product sector • Harsh environment (military) product sector - Summary and Conclusions - Future Cooling Technologies & Strategy Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” OverviewOverview

Thermal management will play a pivotal role in the coming decade for all types of electronic products. Increased heat fluxes at all levels of packaging from chip to system to facility pose a major cooling challenge. To meet the challenge significant cooling technology enhancements will be needed in each of the following areas:

l Thermal interfaces l Heat spreading l l Indirect and direct l Immersion cooling l cooling l l Equipment-facility interface

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal DesignDesign RequirementsRequirements (Traditional)(Traditional)

Design for Performance Design for Reliability Design for Serviceability Design for Extensibility Design for Minimal Cost Design for Minimal Impact on User

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal DesignDesign RequirementsRequirements (New)(New)

Design for improved coolability at the package level via optimized internal paths. Design for direct air cooling at the product level via enhanced process over the packages. Design for special cooling needs at the module level via spot cooling devices attached to the packages. Design for low temperature applications - subambient to cryogenic. Design for low cost via Computer Aided Thermal Engineering (CATE) and improved manufacturability.

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal ManagementManagement RequirementsRequirements MatrixMatrix

- State-of-Art -Electrics - Likely Air CooliConducng (IndtiironectD Liriquid)ect LiquidHeat Pipe Thermo Low TempeNewrature CooTechnologyling(see list) Needed

PC/Handheld/Wearable Workstations Mid-Size Computers Storage Subsystems Large Scale Computers Super Computers

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Microprocessor Power Dissipation Trends

ITRS 2002 Roadmap "First Introduction" 60 7% CGR

50

40 Intel Power4(1.3GP (1.3 GHz) GHz) Pentium 4 McKinley zSeries 30 Sun UltraSPARC Freeway Pentium 3 20 i & pSeries

10 Chip Heat Flux (Watts/sq cm) 0 1996 1998 2000 2002 2004 2006 Year of Announcement

International Technology Roadmap for Semiconductors 2002 Update

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Module Heat Flux Explosion

15

Bipolar IBM ES9000

CMOS 2

10

Fujitsu VP 2000 IBM GP IBM 3090S TCM IBM RY5 NTT Steam Iron 5 Fujitsu M780 IBM RY7 5 W/cm2 Pulsar IBM 3090 TCM Module Heat Flux (Watts/cm ) IBM RY6 CDC Cyber 205 IBM RY4 IBM 3081 TCM IBM 4381 IBM RY3 Fujitsu M380 NEC LCM Merced Vacuum IBM 370 IBM 3033 Tube IBM 360 Hitachi S-810 Pentium II (DSIP) Honeywell DPS-88 0 1950 1960 1970 1980 1990 2000 2010 Year of Announcement Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” System Heat Density Trends

10,000 100,000

8,000 80,000 2 2 6,000 60,000 4,000 40,000 watts / m watts /ft - - 2,000 20,000

1,000 Communication Equipment (frames) 10,000 800 8,000 600 6,000

400 Servers & DASD (tall racks) 4,000

Workstations 200 2,000

Tapes 100 1,000 80 800 60 Heat Load per Machine Floor Area Heat Load per Machine Floor Area 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010

Year of Product Announcement The Uptime Institute, 2000.

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Projected Chip Heat Flux and Cooling Technology Limits

275 T = 85 o C 250 chip

T = 25 o C 225 inlet

200 2

175 Future 150 Cooling (TBD)

125

100 High Performance Conduction Cooling Chip Heat Flux ( W/cm ) 75 Indirect Water or Immersion 50 Cooling Air Cooling 25 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Year Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Current Product Cooling Designs

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM Thinkpad T23 Air Flow Layout

Inlet / Outlet zoom

Outlet FAN BAY

Intake

HDD

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM Thinkpad T24 Push Fan

Top

CPU

Bottom

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Hitachi Water Cooling Laptop (Prototype Model)

Reservoir Tank

Flexible Tube

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 (with extra rack)

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 (continued)

24" System Rack, 42U

Bulk Power Subsystem (8U)

Processors 8-32 SMP's Clock speed 64-bit 1.1 to 1.3 GHz Main memory 8 GB to 256 GB OS images 1-16 Central Electronics Complex Memory bandwidth 205 GB/sec (CEC) (17U) I/O bandwidth 16 GB/sec Internal storage 4.66 TB - 8 drawers (with extra rack) PCI adaptors up to 160 Media Drawer (required) (1U) PCI hot-plug slots yes PCI bus recovery yes Up to 4 Optional internal PCI bus deallocate yes batteries (2U each) Battery backup yes Primary I/O Drawer (4U)

Optional I/O Drawer (4U)

Optional I/O Drawer 4U)

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 (continued)

Local stiffener, backside, 4 position pSeries 681 MCM-L3 L3 Heatsink / Insulator Opposed, laminated Module Asm. spring plates. 4 per MCM Power Distribution Board Fixed stop, actuation Mayfly board screw - 8 Way MCM Mayfly Drive screw until Socket and L3 screw head meets Module Heatsink/Module Global Stiffener springplate. Asm. Load Post Heatsink Mayfly Board Load Column MCM Key Guide 0 3 Front side Stiffeners 1 Mayfly board

Insulator 2 Interposer

Load posts

Local stiffener

Fixed stop actuation screw, drive screw until screw head meets spring plate.

Laminated spring plate, asm. MCM power ~ 624 watts Processor - 2 cores - 156 watts max MCM heatsink / module assembly Processor 415 sq mm Tj ~ 105C Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 MCM

Aluminum Air Cooled - 68 fins - fin width = 0.6 mm - fin spacing = 0.73 mm

Shroud with gasket Cap (Lid)

4 Chip MCM w/ SiC Thermal Spreaders Cap / Heat Sink thermal interface - Powerstrate 60 - Al foil with phase change material on both sides

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Server (with extra rack)

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Server (continued)

Power

Modular Cooling Units (MRUs)

Refrigerant (R134A) Lines

Evaporator (MCM behind)

Memory Cards

Blowers

I/O Cage

Front Side Back Side Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Server Evaporator

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Modular Refrigeration Unit

Compressor Filter Drier

Blower

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Large IBM Servers

IBM z900 IBM pSeries S/390 Mainframe Regatta-H 13KW Dual Bulk Power: 20KW Dual Bulk Power:

CPU Cage: Modular CPU Cooling Units: * 32 way SMP @ 1.3GHz * Cools CPU Chips to 0C Tj * Advanced Refrigeration * Fully Redundant CPU Cage: * 20 way SMP @ 0.8GHz

CPU Cage Cooling Blowers: 4 Memory Books: * High Pressure & Flow * 24GB / Memory Book * Intelligent Variable Speed

Removable Media Drawer: CPU Cage Cooling Blowers: * High Pressure & Flow * Intelligent Variable Speed I/O and Storage Drawer: I/O Cage: * 28 I/O Book Slots

High Density Single Frame Systems

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries Server Model z900

Bulk Power Supplies

Modular Refrigeration Units(MRU)

Evaporator/MCM Outlet Inlet Oil Interface Outlet Inlet Oil Interface Evaporator DeCap

Blowers Grease Chip Evaporator DeCap C4s Al/Cu Hat Grease Chip C-Ring Al/Cu Hat C4s Thin Film Input/Output Ceramic Substrate Base PlateC-Ring

Thin Film Ceramic Substrate Base Plate

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM’s xSeries Model 335

Power Supply

PCI-X Slots Fans

Xeon Second Processor 4 DIMM processor slots spot

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM’s zSeries Model z990: T-Rex

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Cooling

Top View CRAC - Computer Room Air Conditioner

CRAC

CRAC CRAC

Facilities Chilled Water Side View Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Super Computer System Summary (studied)

Water Air Number Number Power (1) Flow Flow of of (W) Rate (1) Rate (1) Modules Processors (gpm) (cfm)

Module n/a 28 40 n/a 6

1,440 1 216 Board 36 1,008 (2,000) (1.5) (300) 5,760 4 864 Cabinet 144 4,032 (8,000) (6) (1,200) 1,474,560 1,024 221,184 System 36,864 1,032,192 (2,048,000) (1,536) (307,200)

Note: 1) Top number pertains to the processors; bottom numbers (in parentheses) pertain to the total package Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Super Computer System Cooling Module (studied)

Al/ Cu Cold Plate

Al Hat

Al Piston

cap Square Header (if necessary) substrate

board

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Super Computer Cooling System (studied)

MER

Pump Building Return

Supply

Heat Exchanger

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM Prototype Data-Storage System: Storage-Array Brick

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Cooling Technologies

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Vapor Chamber Heat Spreader

Heat sink

Air flow

Vapor chamber Chips(s)

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Examples of Heat Pipes Used in Electronics Cooling

Q

Q

Air Flow Q Air Flow

Q

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Example of a Large Air-Cooled Heat Sink for A High Performance Processor Module

Air flow

Q

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Closed Loop Water Cooling System With Heat Rejection to Air

Air to water Filter

Pump Air flow

Water reservoir Cold plate

Electronic module

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Closed Loop Water Cooling System With Heat Rejection to Air

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Closed Loop Water Cooling System With Heat Rejection to Air

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Liquid Jet Impingement Cooling

Outlet Inlet

Cooling cap

Chip Substrate

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Liquid Spray Cooling

Outlet Inlet

Cooling cap Board

Substrate Chip

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Refrigeration Loop and Components for Cooling a High Performance Processor

Condenser Filter/Drier

TX Valve

DC Rotary Compressor Hot Gas Bypass Valve

Accumulator Evaporator

Flexible Flexible Hose Hose Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Cooling Enhancement of an Electronic Module With a Thermoelectric Cooler

Heat sink

N P N P N P N P N P N Thermoelectric Epoxy P interfaces module Cap

Thermal paste Chip Substrate

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Advanced Cooling Technology Development Activities

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Microfabrication Alliance (Georgia Tech/Maryland/Sandia/HP/Thermacore) Two-Phase Thermosyphon Test Vehicle

Demonstrated for 85 W Intel Pentium 4 Processor in 2001.

“Heat Out of Small Packages”, Y. Joshi, Mechanical Engineering, Vol. 123, pp. 56-58, Dec. 2001. Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Spray Cooling (Carnegie Mellon University)

http://www.darpa.mil/MTO/HERETIC/projects/2.html

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Droplet Atomization and Microjets (Georgia Tech)

http://www.darpa.mil/MTO/HERETIC/projects/4.html

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Thermoelectric Coolers for Lasers (JPL)

http://www.darpa.mil/MTO/HERETIC/projects/5.html Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Thermoacoustic (Rockwell)

http://www.darpa.mil/MTO/HERETIC/projects/6.html

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Electrokinetic Pumped Loops (Stanford)

http://www.darpa.mil/MTO/HERETIC/projects/7.html

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Microjets With Liquid/Vapor Phase Change (UCLA)

http://www.darpa.mil/MTO/HERETIC/projects/10.html Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Soild State Thermionic Coolers (UCSB)

http://www.darpa.mil/MTO/HERETIC/projects/11.html

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” SummarySummary andand ConclusionsConclusions

CMOS will continue to be the pervasive semiconductor technology for both memory and logic. Chip sizes will increase but with a higher corresponding increase in circuit density resulting in higher heat flux. All new electronic products will most likely be air-cooled, including most computers, for the next few years. Portable (laptop) computers will need enhanced cooling technology in the near future despite the emphasis on low power dissipation. Power of hand held devices is not increasing with time. Battery life poses major restrictions on power dissipation and most applications do not require any thermal management. High heat flux cooling capability is required for all high performance electronics. High interface material is needed for heat sink applications. Low temperature cooling may get “hot” in the near future. Supercomputers with highly parallel scalable design may require new cooling systems when node power exceeds current levels, or the number of nodes continues to increase significantly, resulting in a large system load “explosion”. Cost will be a significant challenge for all future thermal designs and the speed to accomplish new designs will be vital to their success. Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Future Cooling Technologies and Strategy

Enhanced Air Cooling Technology and System - High performance heat sink - Mini air movers for local enhancement - Higher pressure air movers and higher volume air flow systems - Highly parallel flow distribution system - Active redundancy with control Other Candidate Cooling Technologies - Direct liquid cooling technology - for high performance applications - and vapor chamber cooling technology - for special situations - Thermoelectric cooling technology - for special situations - Thermal interface enhancement technology - Self-contained, low cost liquid cooling technology - Low temperature cooling technology - for performance enhancement Strategy for the Future - Explore all options - Establish a closer working relationship with vendors - Pool resources to fund cooling technology development - Get university/research labs involved Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Grand Challenges for Electronic Cooling Technology in the Coming Decade

Low cost, high performance, direct immersion cooling technology

Low cost, high performance thermal interface (10X) technology

Low cost, high performance cold plate (5X) technology

Low cost, high performance heat sink (5X) technology

Low cost and low noise (2X), high performance (2X) air/liquid moving device technology

Low cost, high performance, scalable cooling system

Low cost, high performance, future data center cooling concept

Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”