ANALYST REPORT National Security Agency Doing More with Less: Cooling Computers with Immersion Pays Off Republished by GRC — 2018 Author: David Prucnal, PE Original Publication: The Next Wave | Vol. 20 | No. 2 | 2013 YEARS 11525 Stonehollow Drive, Suite A-150, Austin, TX 78758 512.692.8003 • [email protected] • grcooling.com GRC • 512.692.8003 • [email protected] • grcooling.com Page 1 DOING MORE WITH LESS: COOLING COMPUTERS WITH IMMERSION PAYS OFF David Prucnal, PE A consequence of doing useful work with Immersion cooling works by directly immersing computers is the production of heat. Every watt IT equipment into a bath of cooling fluid. The of energy that goes into a computer is converted National Security Agency’s Laboratory for to a watt of heat that needs to be removed, or Physical Sciences (LPS) acquired and installed else the computer will melt, burst into flames, an oil-immersion cooling system in 2012 and or meet some other undesirable end. Most has evaluated its pros and cons. Cooling computer systems in data centers are cooled with computer equipment by using oil immersion air conditioning, while some high-performance can substantially reduce cooling costs; in fact, systems use contained liquid cooling systems this method has the potential to cut in half the where cooling fluid is typically piped into a cold construction costs of future data centers. plate or some other heat exchanger. Network servers are submerged into a tank of mineral oil. (Photo used with permission from Green Revolution Cooling: grcooling.com.) GRC • 512.692.8003 • [email protected] • grcooling.com Page 1 FEATURE The fundamentalTHE FUNDAMENTAL problem PROBLEM Operating expense: kW/ton BeforeBefore ggettingetting in intoto t hthee det detailsails of ofimmer immersionsion co ocooling,ling, Energy used to make Heat let’slet’s ttalkalk aaboutbout t hthee p rproductionoduction o fof he heatat by byco mcomputersputers and circulate cold air and the challenge of effectively moving that heat from and the challenge of effectively moving that heat from a data center to the atmosphere or somewhere else wherea data centhe theater to canthe abetmosp reused.here or somewhere else where the heat can be reused. Compute Power Cooling In order for computers to do useful work, they require node energy.In or derThe f oefficiencyr computers ofto thedo us workeful wthatork, they they do can Capital expense: bereq measureduire energ yas. Th thee eratiofficienc of ythe of numberthe work of th operationsat they do Infrastructure required can be measured as the ratio of the number of opera- to make and circulate that they perform to the amount of energy that cold air theytions consume. that they pThereerform are to quitethe a ma ofewunt metricsof energ yused that to Useful computation measurethey con sumecomputer. Ther eenergy are qui teefficiency, a few metrics but us edthe to most OPS/W basicmeasur is eoperations computer perener wattgy effi (OPS/W).ciency, bu Optimizingt the most this metricbasic is has opera beention sthe per topic watt (Oof PS/W).many PhDOptimizin thesesg tandhis Fig.FIGURE 1. There 1. Ther aree artwoe t whalveso halv esto tothe the computer computer power power efficiency will continue to be the subject of future dissertations. problem: efficiency of the actual computation (green sector) and metric has been the topic of many PhD theses and will efficiencyefficiency pr ofoblem: the cooling efficienc infrastructurey of the ac tual(blue c omputationsector). (green Over the years, there has been progress against this continue to be the subject of future dissertations. Over sector) and efficiency of the cooling infrastructure (blue sector). metric, but that progress has slowed because much of the years, there has been progress against this metric, the low-hanging fruit has been harvested and some of of energy used to expel the heat that the computer but that progress has slowed because much of the the key drivers, Moore’s Law and Denard scaling, have generatesenergy. He byre, consumington refers to energy an amo (seeunt ofiguref air co 1).ndi- low-hanging fruit has been harvested and some of the approached the limits of their benefit. Improvements tioning; hence, kW/ton has to do with the amount of key drivers, Moore's Law and Denard scaling, have ap- In fact, many traditional data centers consume as to the OPS/W metric can still be made, but they energy used to expel the heat that the computer gener- usuallyproach edcome the atlimi thets oexpensef their b eofne performance.fit. Improvements to much energy expelling heat as they do performing ates by consuming energy (see figure 1). the OPS/W metric can still be made, but they usually useful computation. This is reflected in a common The problem is not unlike miles per gallon for cars. dataIn centerfact, ma metricny tradi calledtional powerdata cen usageters co effectivenessnsume as Thecome internal at the exp combustionense of per fengineorman ceis. well understood (PUE),much ener whichgy expin itsellin simplestg heat as form they dois theperf ratioormin ofg the andTh hase p beenroblem optimized is not unlik to thee miles nth pdegree.er gallo Forn fo ar cagivenrs. power coming into a data center to the power used useful computation. This is reflected in a common engine,The int ecarrnal weight, combustio and nfrontal engine area, is w ethell under gas mileagestood is to run the computers inside. A data center with a data center metric called power usage effectiveness essentiallyand has be fixed.en opt imizedThe only to twayhe ntoth improve degree. Ftheor amiles given per PUE of 2.0 uses as much power to support cooling, gallon is to reduce the performance or exploit external (PUE), which in its simplest form is the ratio of the engine, car weight, and frontal area, the gas mileage lighting, and miscellaneous loads as it does powering benefits. In other words, drive slower, accelerate less, power coming into a data center to the power used to is essentially fixed. The only way to improve the miles the computers. Of these other loads, cooling is by far drift down hills, find a tailwind, etc. Even after doing all therun dominantthe comp ucomponent.ters inside. A So, da anotherta center way with to a improvePUE per gallon is to reduce the performance or exploit of these things, the improvement in gas mileage is only dataof 2.0 center uses as efficiency much pow eris ttoo suimprovepport co coolingoling, ligefficiency.ht- external benefits. In other words, drive slower, accel- marginal. So it is, too, with computers. Processor clock Theing, abestnd mis casecella scenarioneous loads would as itbe do esto pachieveowering a t hPUEe erate less, drift down hills, find a tailwind, etc. Even frequencies and voltages can be reduced, sleep modes ofco m1.0.pu Oneters. Ofway th estoe achieveother loads, this cowouldoling beis b yto fa buildr the a canafter be do used,ing all memory of these taccesseshings, the and imp communicationsrovement in gas datadomina centernt co minp oan enlocationt. So, a nwhereother wathey toenvironmental improve canmile beage juggledis only ma tor gamortizeinal. So i t theiris, too energy, with co costs,mput -but conditions allow for free cooling. Some commercial data center efficiency is to improve cooling efficiency. eveners. P withrocess allor ofclo this,ck fr eqtheuencies improvement and voltag ines OPS/W can be is companies have taken this approach and built data The best case scenario would be to achieve a PUE of limited.reduced , sleep modes can be used, memory accesses centers in northern latitudes with walls that can be opened1.0. One towa ylet to inac hieoutsideve th isair w otould cool be t othe bu ildcomputers a data Aan dnatural comm unicaconsequencetions can ofbe jdoinguggled usefulto amo rworktize t hwitheir whencenter the in aoutside locatio ntemperature where the en andvir ohumiditynmental arecondi- within computersenergy costs, is btheut e vproductionen with all ooff thheat.is, th eEvery impr owattve- of allowabletions allow limits. for fr eeHowever, cooling. Sforome those commer of cialus whocom parea- energyment in that OPS/W goes is intolimi tead. computer is converted into tiednies hato vthee ta kmid-en th Atlanticis approac regionh and wherebuilt da summersta centers arein a watt of heat that needs to be removed from the typically hot and humid, year-round free cooling is not computer,A natural or co elsenseq ituence will omelt,f doin burstg usef intoul w oflames,rk with or northern latitudes with walls that can be opened to let a viable option. How can data centers in this type of meetcomp usometers is otherthe p roundesirableduction of heend.at. E Anothervery watt metric,of en- in outside air to cool the computers when the outside environment improve their kW/ton and PUE? whichergy t huntilat go recentlyes into a wascom lessputer researched is converte thand int oOPS/W, a watt temperature and humidity are within allowable limits. iso fkilowatts heat that perneeds ton t o(kW/ton), be remov whiched from has th enothing compu ttoer ,do However, for those of us who are tied to the mid- HOW COMPUTERS ARE COOLED withor els thee it weightwill mel oft, btheurs tcomputer into flames, system or meet that so isme using Atlantic region where summers are typically hot and upother the undesira energy.b leHere, end .ton Ano refersther metr to anic, wamounthich un oftil air Therehumid ,are ye armany-round different free coolin waysg is no thatt a viacomputersble option. are conditioning;recently was less hence, rese kW/tonarched thashan to O doPS/W with, is the kilo amountwatts keptHow coolcan da int adata cent ecentersrs in this today; type o fhowever, environmen thet most per ton (kW/ton), which has nothing to do with the improve their kW/ton and PUE? weight of the computer system that is using up the GRC • 512.692.8003 • [email protected] • grcooling.com Page 2 The Next Wave | Vol.