Summer 2019 - Assessing the Facilities and Systems Supporting HPC at ORNL Jim Rogers Director, Computing and Facilities National Center for Computational Sciences Oak Ridge National Laboratory ORNL is managed by UT-Battelle, LLC for the US Department of Energy ORNL Leadership Class Systems 2004 - 2018 27 PF 2012 2.5 Cray XK7 1 PF Titan PF 263 2009 TF 2008 Cray XT5 62 Cray XT5 Jaguar 54 2008 25 TF TF Jaguar 18.5 TF Cray XT4 TF 2006 2007 Jaguar Cray XT3 Cray XT4 2004 2005 Jaguar From 2004 – 2018, HPC systems relied Cray X1E Cray XT3 Jaguar Phoenix Jaguar on chiller-based cooling (5.5°C supply) with annualized PUEs to ~1.4 2 ORNL’s Transition to Warmer Facility Supply Temperatures Frontier: Custom mechanical Summit: A combination of packaging is >95% room- Titan: Refrigerant-based direct on-package cooling neutral with a 30°C supply. per-rack cooling with and RDHX with 21°C supply • ~100% Evaporative Cooling, direct rejection of heat to is > 95% room-neutral. with supplemental HVAC for cold 5.5°C water • Above dewpoint parasitic loads • Below dewpoint • Contribution by chillers • 100% use of chillers ~20% of the hours of the year ~1.5 EF 200 2021/2022 PF Frontier 27 PF 2018/2019 2012 IBM Cray XK7 Summit Titan 3 GTC'154 Session S5566 GPU Errors on HPC Systems Oak Ridge National Laboratory’s Cray XK7 Titan Operational from November 2012 through August 1 2019 18,688 compute nodes – 1 AMD Opteron + 1 NVIDIA Kepler/node 27PF (peak); 17.59PF (HPL); 9.5MW (peak) Delivered > 27B compute hours over its life Titan entered as the #1 supercomputer in the world in November 2012, and was still #12 on the Top500 list at the time of its decommissioning 4 Perspective on Titan as an Air-cooled Supercomputer ORNL’s Cray XK7 “Titan” Wait. What? • 200 cabinets, each with a 3,000 CFM fan • 600,000 CFM (air volume) Titan moves as much air as a • Dry air has a density of 13.076 cubic feet per pound long-range Airbus A340 at • Titan moves 600,000/13.076/60 -> 765 lbs/sec cruising altitude. Airbus A340 • At takeoff, each of (4) engines generates 140kN of force and consumes 1000 lbs/sec of air • At cruise, the A340’s engines each produce ~29kN and consumes ~200 lbs/sec of air. 5 Motivation for “Warmer” Cooling Solutions Serving HPC Centers • Reduced Cost, Both CAPEX and OPEX – Reduce or eliminate the need for traditional chillers. • No chillers, no ozone-depleting refrigerants (GREEN) – Oak Ridge calculates an annualized PUE for air--cooled devices of no better than 1.4 Oak Ridge, TN (ASHRAE Zone 4A – Mixed Humid) – HPC power budgets continue to grow – Summit has a design point for >12MW (HPC-only). Minimizing PUE/ITUE is critical to the budget. • Easier, more reliable design – Design is reduced to pumps, evaporative cooling, heat exchangers. – Traditional chilled water may not be necessary at all (NREL, NCAR, et al) 6 OLCF Facilities Supporting Summit • Titan – 9MW @ heavy load • Sitting on 250 pounds/ft2 raised floor • Uses 42F water and special CDUs (XDPs) • Summit – 256 compute cabinets on-slab • 100% room-neutral design uses RDHX • 20MW warm-water cooling plant using centralized CDU/secondary loop 7 21°C/20MW/7700 ton Facility System Design System • Summit – Demand: 3-10MW; • Secondary Loop – Supply 3300GPM (12,500 liters/min) @ 21°C; Return @ 29-33°C – CPUs and GPUs use cold plates – DIMMs and parasitic loads use RDHX – Storage and Network use RDHX 8 Cooling Design Existing chilled water cooling loop Primary Loop uses Evaporative Cooling Towers (~80% of the hours of the year) When the MTW RETURN is above the 21C setpoint, use a second set of Trim HX (with 5.5C on the other side) to drive MTW to the 21C setpoint. The need for the trim-loop is about 20% of the hours in the year, and can ramp 0-100% to meet the setpoint back to Summit New primary cooling loop 9 Benefits of Warm Water + Operating Dashboards • Warm Water allows annualized PUE of 1.1 – ~$1M cost per MW-year for consumption on Summit; – ~$100k cost per MW-year for waste heat management • Integration with PLC allows us to tune water flow – Better delta(t); less pumping energy • Integration with IBM’s OpenBMC allows us to protect these 40k components from inadequate flow across the cold plates • Integration with the scheduler allows us to correlate power and temperature data with individual applications. 10 • Additional data streams to be added- most from the Facility PLC Comparing Energy Performance - Titan to Summit • Demonstrated Performance on Titan (Oct 2012) • 17.59 PF 8.9MW peak 8.3MW average • Demonstrated Performance on Summit (Oct 2018) • 143.5 PF 11,065kW peak 9,783kW average 8.16x performance increase 17.9% avg. power increase Titan: ~2.1 GFLOPs/Watt Summit: 14.66 GFLOPs/Watt >7x increase in energy efficiency 11 12 13 14 Summit MTW Cooling Loads and Temperatures HPL Run 5/24/19 Duration: 5:48 PUE during HPL Run = 1.081 100 14000 95 90 12000 85 10000 80 IT Load follows F) ° 75 a traditional HPL profile 8000 70 Supply temperature to Summit stayed kW 65 OAWBT remained Total power load rock-solid at 70F. 6000 60 at/above the on the energy supply target, Temperature( 55 plant reflects affecting ECT storage and other 4000 50 items A small portion of the total load 45 required the use of 2000 CHW (trim RDHX) 40 35 0 MTW kW (cooling) CHW kW (cooling) MTW Return Temp MTW Supply Temp 23:51:00 K10023:58:30 00:06:00Avg00:13:30 Space00:21:00 00:28:30 00:36:00 Temp00:43:30 00:51:00 00:58:30 01:06:00 01:13:30 01:21:00 01:28:30 Outdoor01:36:00 01:43:30 01:51:00 Air01:58:30 Wet02:06:00 02:13:30Bulb02:21:00 Temp02:28:30 02:36:00 02:43:30 02:51:00 02:58:30 03:06:00 IT kW03:13:30 03:21:00 03:28:30 03:36:00 03:43:30 03:51:00 03:58:30 04:06:00 04:13:30 04:21:00 04:28:30 04:36:00 04:43:30 04:51:00 04:58:30 05:06:00 05:13:30 05:21:00 05:28:30 05:36:00 15 Time 16 Idle: 2.97MW kW 10000 12000 10000 12000 2000 4000 6000 8000 2000 4000 6000 8000 0 0 1 239215 477429 643 11,065 kW 11,065 Max power power Max 715 measurements 151,284 seconds; 21,612 857 953 1071 12851191 14991429 17131667 11,065 kW 11,065 19271905 kW 9,783 power Average power Max 21432141 23812355 26192569 28572783 2997 3095 3211 34253333 – 36393571 38533809 40674047 42854281 45234495 – 47614709 49994923 5137 5237 5351 – 55655475 57795713 59935951 62076189 64276421 66656635 69036849 71417063 7277 OLCF 7379 OLCF 7491 77057617 79197855 8093 8133 10 10 - - 4 Summit HPL Power Measurements 4 Summit HPLPower 83478331 Measurements 4 Summit HPLPower - 85698561 - 25 88078775 25 8989 - 9045 - 18 92839203 18 9417 9521 Measurement (1/second) Measurement Measurement (1/second) Measurement 9631 - 98459759 - 03:31:48 to 09:32:00 03:31:48 to 100599997 09:32:00 03:31:48 to 1027310235 1048710473 1071110701 1094910915 11129 TotalLoad IT 11187 PUE TotalMech Load 1142511343 11557 11663 11771 1198511901 1219912139 1241312377 1262712615 1285312841 1309113055 1332913269 1356713483 13697 13805 13911 1412514043 1433914281 1455314519 1476714757 1499514981 1523315195 1547115409 1570915623 15837 1605115947 1626516185 1647916423 58,730 kW 58,730 1669316661 1.0246 kW 1,449 1689916907 kW 9,783 power Average 1713717121 1737517335 1761317549 1785117763 17977 1819118089 1840518327 18565 18619 - 18803 hours 18833 - 1904119047 hours 1927919261 1951719475 1975519689 19903 19993 20117 2033120231 2054520469 – 2075920707 2097320945 2118321187 2142121401 0 10000 20000 30000 40000 50000 60000 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 Accumulated kW-hoursAxis Title 17 10000 12000 2000 4000 6000 8000 kW 0 10000 12000 2000 4000 6000 8000 1 0 214 1 427 215 429 640 643 853 857 1066 1071 1285 1279 1499 1492 1713 1705 1927 2141 1918 2355 2131 2569 2344 122.3PF 2783June2018 2997 2557 3211 2770 3425 2983 3639 3853 3196 4067 3409 4281 3622 4495 4709 3835 4923 4048 5137 4261 5351 5565 4474 5779 4687 5993 4900 6207 6421 5113 6635 5326 6849 7063 5539 7277 5752 7491 5965 7705 7919 OLCF 6178 8133 6391 8347 10 - 8561 Measurements 4 Summit HPL Power 6604 - 8775 25 6817 8989 - 7030 9203 18 9417 7243 - Measurement (1/second) Measurement 9631 03:31:48 to 09:32:00 03:31:48 to 7456 9845 143.5PF 2018 Oct 7669 10059 7882 10273 10487 8095 10701 8308 10915 8521 11129 11343 8734 11557 8947 11771 9160 11985 12199 9373 12413 9586 12627 9799 12841 13055 10012 13269 10225 13483 10438 13697 13911 10651 14125 10864 14339 14553 11077 14767 11290 14981 11503 15195 15409 11716 15623 11929 15837 12142 16051 16265 12355 16479 12568 16693 12781 16907 17121 12994 17335 13207 17549 13420 17763 17977 13633 18191 13846 18405 14059 18619 18833 14272 19047 14485 19261 14698 19475 19689 14911 19903 15124 20117 15337 20331 20545 15550 20759 15763 20973 21187 15976 21401 16189 Cooling System Performance - PUE 18 20°C 21°C 22°C 24°C 25°C 26°C 27°C 19 23°C 26°C 19°C 20 Caution – Secondary (Closed) Loop Concerns Worsen as Temperatures Rise… Allowable Wetted Materials Accelerated Corrosion • Lead-free copper alloys with less than 15% zinc.