Thermal Management Roadmap Cooling Electronic Products from Hand-HeldDevices to Supercomputers
Richard C. Chu - IBM
May 16, 2003, Rohsenow Symosium Cambridge, MA ThermalThermal ManagementManagement TechnicalTechnical WorkingWorking GroupGroup (TWG)(TWG)
Richard C. Chu, IBM (Chair) Jogenda Joshi (Co-chair)
Avi Bar-Cohen, U. of Maryland Gregory M. Chrysler, Intel Darvin Edwards, TI Suresh Garimella, Purdue U.
Magnus Herrlin, Telcordia Larry Mok, IBM
Donald Price, Raytheon Bahgat Sammakia, SUNY-Binghamton Roger Schmidt, IBM Lian-Tuu Yeh, Boeing
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal ManagementManagement RoadmapRoadmap -- ScopeScope
- Introduction • Overview • Review of thermal design requirements • Review of thermal design requirement matrix • Review of current product cooling designs • Review of cooling technologies • Review of advanced cooling technology development activities - Outline of Thermal Technology Needs • High performance product sector • Cost performance product sector • Telecommunications product sector • Hand held product sector • Harsh environment (automotive) product sector • Harsh environment (military) product sector - Summary and Conclusions - Future Cooling Technologies & Strategy Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” OverviewOverview
Thermal management will play a pivotal role in the coming decade for all types of electronic products. Increased heat fluxes at all levels of packaging from chip to system to facility pose a major cooling challenge. To meet the challenge significant cooling technology enhancements will be needed in each of the following areas:
l Thermal interfaces l Heat spreading l Air cooling l Indirect and direct water cooling l Immersion cooling l Refrigeration cooling l Thermoelectric cooling l Equipment-facility interface
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal DesignDesign RequirementsRequirements (Traditional)(Traditional)
Design for Performance Design for Reliability Design for Serviceability Design for Extensibility Design for Minimal Cost Design for Minimal Impact on User
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal DesignDesign RequirementsRequirements (New)(New)
Design for improved coolability at the package level via optimized internal thermal conduction paths. Design for direct air cooling at the product level via enhanced convection process over the packages. Design for special cooling needs at the module level via spot cooling devices attached to the packages. Design for low temperature applications - subambient to cryogenic. Design for low cost via Computer Aided Thermal Engineering (CATE) and improved manufacturability.
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” ThermalThermal ManagementManagement RequirementsRequirements MatrixMatrix
- State-of-Art -Electrics - Likely Air CooliConducng (IndtiironectD Liriquid)ect LiquidHeat Pipe Thermo Low TempeNewrature CooTechnologyling(see list) Needed
PC/Handheld/Wearable Workstations Mid-Size Computers Storage Subsystems Large Scale Computers Super Computers
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Microprocessor Power Dissipation Trends
ITRS 2002 Roadmap "First Introduction" 60 7% CGR
50
40 Intel Power4(1.3GP (1.3 GHz) GHz) Pentium 4 McKinley zSeries 30 Sun UltraSPARC Freeway Pentium 3 20 i & pSeries
10 Chip Heat Flux (Watts/sq cm) 0 1996 1998 2000 2002 2004 2006 Year of Announcement
International Technology Roadmap for Semiconductors 2002 Update
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Module Heat Flux Explosion
15
Bipolar IBM ES9000
CMOS 2
10
Fujitsu VP 2000 IBM GP IBM 3090S TCM IBM RY5 NTT Steam Iron 5 Fujitsu M780 IBM RY7 5 W/cm2 Pulsar IBM 3090 TCM Module Heat Flux (Watts/cm ) IBM RY6 CDC Cyber 205 IBM RY4 IBM 3081 TCM IBM 4381 IBM RY3 Fujitsu M380 NEC LCM Merced Vacuum IBM 370 IBM 3033 Tube IBM 360 Hitachi S-810 Pentium II (DSIP) Honeywell DPS-88 0 1950 1960 1970 1980 1990 2000 2010 Year of Announcement Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” System Heat Density Trends
10,000 100,000
8,000 80,000 2 2 6,000 60,000 4,000 40,000 watts / m watts /ft - - 2,000 20,000
1,000 Communication Equipment (frames) 10,000 800 8,000 600 6,000
400 Servers & DASD (tall racks) 4,000
Workstations 200 2,000
Tapes 100 1,000 80 800 60 Heat Load per Machine Floor Area Heat Load per Machine Floor Area 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010
Year of Product Announcement The Uptime Institute, 2000.
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Projected Chip Heat Flux and Cooling Technology Limits
275 T = 85 o C 250 chip
T = 25 o C 225 inlet
200 2
175 Future 150 Cooling (TBD)
125
100 High Performance Conduction Cooling Chip Heat Flux ( W/cm ) 75 Indirect Water or Immersion 50 Cooling Air Cooling 25 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Current Product Cooling Designs
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM Thinkpad T23 Air Flow Layout
Inlet / Outlet zoom
Outlet FAN BAY
Intake
HDD
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM Thinkpad T24 Push Fan
Top
CPU
Bottom
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Hitachi Water Cooling Laptop (Prototype Model)
Reservoir Tank
Flexible Tube
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 (with extra rack)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 (continued)
24" System Rack, 42U
Bulk Power Subsystem (8U)
Processors 8-32 SMP's Clock speed 64-bit 1.1 to 1.3 GHz Main memory 8 GB to 256 GB OS images 1-16 Central Electronics Complex Memory bandwidth 205 GB/sec (CEC) (17U) I/O bandwidth 16 GB/sec Internal storage 4.66 TB - 8 drawers (with extra rack) PCI adaptors up to 160 Media Drawer (required) (1U) PCI hot-plug slots yes PCI bus recovery yes Up to 4 Optional internal PCI bus deallocate yes batteries (2U each) Battery backup yes Primary I/O Drawer (4U)
Optional I/O Drawer (4U)
Optional I/O Drawer 4U)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 (continued)
Local stiffener, backside, 4 position pSeries 681 MCM-L3 L3 Heatsink / Insulator Opposed, laminated Module Asm. spring plates. 4 per MCM Power Distribution Board Fixed stop, actuation Mayfly board screw - 8 Way MCM Mayfly Drive screw until Socket and L3 screw head meets Module Heatsink/Module Global Stiffener springplate. Asm. Load Post Heatsink Mayfly Board Load Column MCM Key Guide 0 3 Front side Stiffeners 1 Mayfly board
Insulator 2 Interposer
Load posts
Local stiffener
Fixed stop actuation screw, drive screw until screw head meets spring plate.
Laminated spring plate, asm. MCM power ~ 624 watts Processor - 2 cores - 156 watts max MCM heatsink / module assembly Processor 415 sq mm Tj ~ 105C Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM pSeries 690 MCM
Aluminum Air Cooled Heat Sink - 68 fins - fin width = 0.6 mm - fin spacing = 0.73 mm
Shroud with gasket Copper Cap (Lid)
4 Chip MCM w/ SiC Thermal Spreaders Cap / Heat Sink thermal interface - Powerstrate 60 - Al foil with phase change material on both sides
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Server (with extra rack)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Server (continued)
Power
Modular Cooling Units (MRUs)
Refrigerant (R134A) Lines
Evaporator (MCM behind)
Memory Cards
Blowers
I/O Cage
Front Side Back Side Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Server Evaporator
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries 900 Modular Refrigeration Unit
Compressor Filter Drier
Blower
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Large IBM Servers
IBM z900 IBM pSeries S/390 Mainframe Regatta-H 13KW Dual Bulk Power: 20KW Dual Bulk Power:
CPU Cage: Modular CPU Cooling Units: * 32 way SMP @ 1.3GHz * Cools CPU Chips to 0C Tj * Advanced Refrigeration * Fully Redundant CPU Cage: * 20 way SMP @ 0.8GHz
CPU Cage Cooling Blowers: 4 Memory Books: * High Pressure & Flow * 24GB / Memory Book * Intelligent Variable Speed
Removable Media Drawer: CPU Cage Cooling Blowers: * High Pressure & Flow * Intelligent Variable Speed I/O and Storage Drawer: I/O Cage: * 28 I/O Book Slots
High Density Single Frame Systems
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM zSeries Server Model z900
Bulk Power Supplies
Modular Refrigeration Units(MRU)
Evaporator/MCM Outlet Inlet Oil Interface Outlet Inlet Oil Interface Evaporator DeCap
Blowers Grease Chip Evaporator DeCap C4s Al/Cu Hat Grease Chip C-Ring Al/Cu Hat C4s Thin Film Input/Output Ceramic Substrate Base PlateC-Ring
Thin Film Ceramic Substrate Base Plate
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM’s xSeries Model 335
Power Supply
PCI-X Slots Fans
Xeon Second Processor 4 DIMM processor slots spot
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM’s zSeries Model z990: T-Rex
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Data Center Cooling
Top View CRAC - Computer Room Air Conditioner
CRAC
CRAC CRAC
Facilities Chilled Water Side View Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Super Computer System Summary (studied)
Water Air Number Number Power (1) Flow Flow of of (W) Rate (1) Rate (1) Modules Processors (gpm) (cfm)
Module n/a 28 40 n/a 6
1,440 1 216 Board 36 1,008 (2,000) (1.5) (300) 5,760 4 864 Cabinet 144 4,032 (8,000) (6) (1,200) 1,474,560 1,024 221,184 System 36,864 1,032,192 (2,048,000) (1,536) (307,200)
Note: 1) Top number pertains to the processors; bottom numbers (in parentheses) pertain to the total package Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Super Computer System Cooling Module (studied)
Al/ Cu Cold Plate
Al Hat
Al Piston
cap Square Header (if necessary) substrate
board
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Super Computer Cooling System (studied)
MER
Pump Building Coolant Return
Supply
Heat Exchanger
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” IBM Prototype Data-Storage System: Storage-Array Brick
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Cooling Technologies
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Vapor Chamber Heat Spreader
Heat sink
Air flow
Vapor chamber Chips(s)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Examples of Heat Pipes Used in Electronics Cooling
Q
Q
Air Flow Q Air Flow
Q
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Example of a Large Air-Cooled Heat Sink for A High Performance Processor Module
Air flow
Q
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Closed Loop Water Cooling System With Heat Rejection to Air
Air to water heat exchanger Filter
Pump Air flow
Water reservoir Cold plate
Electronic module
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Closed Loop Water Cooling System With Heat Rejection to Air
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Closed Loop Water Cooling System With Heat Rejection to Air
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Liquid Jet Impingement Cooling
Outlet Inlet
Cooling cap
Chip Substrate
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Liquid Spray Cooling
Outlet Inlet
Cooling cap Board
Substrate Chip
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Refrigeration Loop and Components for Cooling a High Performance Processor
Condenser Filter/Drier
TX Valve
DC Rotary Compressor Hot Gas Bypass Valve
Accumulator Evaporator
Flexible Flexible Hose Hose Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Cooling Enhancement of an Electronic Module With a Thermoelectric Cooler
Heat sink
N P N P N P N P N P N Thermoelectric Epoxy P interfaces module Cap
Thermal paste Chip Substrate
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Advanced Cooling Technology Development Activities
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Microfabrication Alliance (Georgia Tech/Maryland/Sandia/HP/Thermacore) Two-Phase Thermosyphon Test Vehicle
Demonstrated for 85 W Intel Pentium 4 Processor in 2001.
“Heat Out of Small Packages”, Y. Joshi, Mechanical Engineering, Vol. 123, pp. 56-58, Dec. 2001. Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Spray Cooling (Carnegie Mellon University)
http://www.darpa.mil/MTO/HERETIC/projects/2.html
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Droplet Atomization and Microjets (Georgia Tech)
http://www.darpa.mil/MTO/HERETIC/projects/4.html
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Thermoelectric Coolers for Lasers (JPL)
http://www.darpa.mil/MTO/HERETIC/projects/5.html Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Thermoacoustic Refrigerators (Rockwell)
http://www.darpa.mil/MTO/HERETIC/projects/6.html
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Electrokinetic Pumped Loops (Stanford)
http://www.darpa.mil/MTO/HERETIC/projects/7.html
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Microjets With Liquid/Vapor Phase Change (UCLA)
http://www.darpa.mil/MTO/HERETIC/projects/10.html Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Recent Research (DARPA HERETIC) Soild State Thermionic Coolers (UCSB)
http://www.darpa.mil/MTO/HERETIC/projects/11.html
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” SummarySummary andand ConclusionsConclusions
CMOS will continue to be the pervasive semiconductor technology for both memory and logic. Chip sizes will increase but with a higher corresponding increase in circuit density resulting in higher heat flux. All new electronic products will most likely be air-cooled, including most computers, for the next few years. Portable (laptop) computers will need enhanced cooling technology in the near future despite the emphasis on low power dissipation. Power of hand held devices is not increasing with time. Battery life poses major restrictions on power dissipation and most applications do not require any thermal management. High heat flux cooling capability is required for all high performance electronics. High thermal conductivity interface material is needed for heat sink applications. Low temperature cooling may get “hot” in the near future. Supercomputers with highly parallel scalable design may require new cooling systems when node power exceeds current levels, or the number of nodes continues to increase significantly, resulting in a large system load “explosion”. Cost will be a significant challenge for all future thermal designs and the speed to accomplish new designs will be vital to their success. Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Future Cooling Technologies and Strategy
Enhanced Air Cooling Technology and System - High performance heat sink - Mini air movers for local enhancement - Higher pressure air movers and higher volume air flow systems - Highly parallel flow distribution system - Active redundancy with control Other Candidate Cooling Technologies - Direct liquid cooling technology - for high performance applications - Heat pipe and vapor chamber cooling technology - for special situations - Thermoelectric cooling technology - for special situations - Thermal interface enhancement technology - Self-contained, low cost liquid cooling technology - Low temperature cooling technology - for performance enhancement Strategy for the Future - Explore all options - Establish a closer working relationship with vendors - Pool resources to fund cooling technology development - Get university/research labs involved Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution” Grand Challenges for Electronic Cooling Technology in the Coming Decade
Low cost, high performance, direct immersion cooling technology
Low cost, high performance thermal interface (10X) technology
Low cost, high performance cold plate (5X) technology
Low cost, high performance heat sink (5X) technology
Low cost and low noise (2X), high performance (2X) air/liquid moving device technology
Low cost, high performance, scalable cooling system
Low cost, high performance, future data center cooling concept
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”