Oak Ridge Leadership Computing Facility Annual Report 2010–2011 Contents Science at the OLCF 2 Best at the Petascale, First to the Exascale 4 In a League of Its Own 8 4.5 Billion Hours and Counting 12 Exploring the Magnetic Personalities of Stars 14 Earthquake Simulation Rocks Southern California 16 OLCF Vis Team Helps Simulate Japanese Crisis 17 Supercomputers Assist Cleanup of Decades-Old Nuclear Waste 18 Supercomputers Simulate the Molecular Machines that Replicate and Repair DNA 20 The Problem with Cellulosic Ethanol 22 Industrial Partnerships Driving Development

Road to Exascale 26 Titan

Inside the OLCF 32 OLCF Resources Pave the Way 36 Say Hello to ADIOS 1.2 38 World-Class Systems Deserve World-Class People 40 Education, Outreach, and Training 43 High School Students Build Their Own Supercomputer—Almost—at ORNL 44 High-Impact Publications

Creative Director: Jayson Hines The research and activities described in this report were Writers: Gregory Scott Jones, Dawn Levy, Leo Williams, performed using resources of the Oak Ridge Leadership Caitlin Rockett, Wes Wade, and Eric Gedenk Computing Facility at Oak Ridge National Laboratory, which is Graphic Design and Layout: Jason B. Smith supported by the Office of Science of the U.S. Department of Graphic Support: Jane Parrott and Andy Sproles Energy under Contract No. DE-AC0500OR22725 Photography: Jason Richards Additional images: iStockphoto On the cover: Leadership-class molecular dynamics simulation of the James J. Hack, Director plant components lignin and cellulose. This 3.3 million-atom National Center for Computational Sciences simulation was performed on 30,000 cores of the Jaguar XT5 supercomputer and investigated lignin precipitation on Arthur S. Bland, Project Director cellulose fibrils, a process that poses a significant obstacle Oak Ridge Leadership Computing Facility to economically viable bioethanol production. Simulation by Jeremy Smith, ORNL. Visualization by Mike Matheson, ORNL. Oak Ridge Leadership Computing Facility P.O. Box 2008, Oak Ridge, TN 37831-6008 TEL: 865-241-6536 FAX: 865-241-2850 EMAIL: [email protected] URL: www.olcf.ornl.gov

Introduction

The Oak Ridge Leadership Computing Facility (OLCF) was established at Oak Ridge National Laboratory (ORNL) in 2004 with the mission of standing up a supercomputer 100 times more powerful than the leading systems of the day.

The facility delivered on that promise four years later, in 2008, when its Cray XT Jaguar system ran the first scientific applications to exceed 1,000 trillion calculations a second (1 petaflop). In the meantime four other applications have topped a petaflop, all on Jaguar. To date, no other supercomputer has reached this level of performance with real scientific applications.

With 224,256 processing cores delivering a peak performance of more than 2.3 petaflops, Jaguar gives the world’s most advanced computational researchers an opportunity to tackle problems that would be unthinkable on other systems. The OLCF welcomes investigators from universities, government agencies, and industry who are prepared to perform breakthrough research in climate, materials, alternative energy sources and energy storage, chemistry, nuclear physics, astrophysics, quantum mechanics, and the gamut of scientific inquiry. Because it is a unique resource, the OLCF focuses on the most ambitious research projects—projects that provide important new knowledge or enable important new technologies.

Looking to the future, the facility is moving forward with a roadmap that by 2018 will deliver an exascale supercomputer—one able to deliver 1 million trillion calculations each second. Along the way, the OLCF will stand up systems of 20, 100, and 250 petaflops. 2 SCIENCE AT THE OLCF T Oak Ridge Leadership ComputingFacility on Jaguar. temperature on these systems.of application This reached 1.84petaflops application that analyzes systems magnetic and, in particular, the effect ORNL,Eisenbach of holdstherecord to date with WL-LSMS, an high-temperature superconductors. Another team, led by Markus Centre—andDCA++,with colleagues the had first which models theSwiss NationalThomas Schulthess—head of Supercomputing date, five scientific applications havepetaflop, 1 topped all on Jaguar. simulations earlier becomes clearerpicture of andmore detailed. To thephysicalof systems describe. they As aresult, thehazy, general Faster simulations more gather dataand, thereby, provide greater detail applicationspetaflop andtheGordon Bell Prize. Consider Jaguar’s two measures of success asascientific workhorse: applications into age. the petascale ithasbrought because real, apart Jaguar working itself scientific sets supercomputer isnotinitsranking, but inthework itcanaccomplish. computing, but itdoesn’t to tell thestory. begin ascientific value The of States. inaworld run good asvolatile isavery This ashigh-performance 2010, the most powerful and it is still supercomputer in the United powerful systems. It ayear spent thelistin2009and atthetop of the semiannual Top500six iterations of the world’s List of most Jaguar among hasbeen theworld’s top three supercomputers through Jaguar,with 2.3petaflops. which isnow capable of more powerful supercomputers thantheleading theday. of It succeeded Created seven years ago, its mission was to stand up a system 100 times challenges that could not otherwise be attempted. be challenges thatcould nototherwise computational science, of boundaries tackling and overcoming he OakRidge Leadership Computing exists Facility to pushthe First totheExascale Best atthePetascale, Watts nuclear Bar power plant. 3½gigawatts,be or about three thenearby thepower times output of thatamountwould 500times electricity; draws about7megawatts of it would require asmuch 500times power. current The Jaguar system more than100million. It would require asmuch 500times space. And as Jaguardesign would contain as 500 manytimes processing cores— power requirements. An exascale supercomputer on relying the same approach,This however, among hasitslimits; them are high space and cores, Jaguar has224,000. home computers, but ahome computer while may have two processing central processing units, or CPUs. areThese the same chips that run Jaguar reached by thousandsof thepetascale connecting hundreds of well asfor us. well asspeed, challenges for bring ourusers as andthis change will representTitan will amajor change from Jaguar inconfiguration as moretimes powerful thanJaguar, able to reach from 10to 20petaflops. Titan, asmuch be next upandrunning year as10 should be andwill We to expect achieve theexascale by 2018. Ournext system, called asanexaflop. known otherwise That isa one followed by 18 zeroes. supercomputer calculationseach second, able totrillion amillion perform more excited to pushingtoward be thenext major milestone: a We are proud to have theway led into age. thepetascale We are even Tech. byled Georgia George Biros of system. plasmainthecirculatory with camefrom effort This ateam simulated 200 million realistic red blood cells and their interaction in 2008 and Eisenbach in 2009. application in 2010 winning The Prizewinners for thepastthree years on Jaguar, ran including Schulthess eachMachinery year to recognize theworld’s pre-eminent application. GordonThe Prize Bell comes from theComputing Association of for ComputingandComputationalSciences Jeff Nichols, Associate LaboratoryDirector

First totheExascale Best atthePetascale, GPUs andother accelerators for theforeseeable future, notjust at beyond Titan. world’s The mostpowerful supercomputers use will applicable be insystems well will news good isthattheir efforts The possible.the mostparallelism researchers workingbe with to their code rewrite inorder to exploit parallelism.to thisnew level take of As advantage aresult, of we will their applications must many rewrite that they inthefact be of users will anew system andmakes reality.of itapractical challenge The for our hardwaresupporting andsoftware thattakes thetheoretical potential challengeThe for us is that we must continue to revolutionize the blistering speed. them with hundreds, to simplecalculationsandprocess ittheability divide giving two, four, computing or eight cores, processor agraphics canhave and pushing it to a new level. While processor a traditional may have computers, approach taking the parallel inherent in a system like Jaguar more recently have effectively very used been to high-performance speed decade ago to improve rendering for graphics computer displays. GPUs CPUstraditional processing andgraphics units(GPUs), developed a To overcome obstacles, theseobvious useacombination Titan of will major milestone:asupercomputerableto perform amilliontrillioncalculationseach second, otherwiseknownasanexaflop. We areproudtohaveled thewayinto the petascaleage.We are evenmore excited tobepushingtowardthenext there. Wherever theforefront scientific supercomputing of goes,be will we Administration’s mostpowerful supercomputer, Gaea. the Nationalalso houseand support Oceanic and Atmospheric also themostpowerful academic supercomputer inthecountry. We supercomputer, theUniversity Tennessee’s of Kraken system, which is theNationalhouse andsupport Science Foundation’s mostpowerful a lab for computing, reason.good and with In to addition Jaguar, we competitivevery field for years to come.become ORNLhas as known Nevertheless, theOLCF maintainleadership inthis andORNLwill represented inthetop andseven 10spots inthetop 20. supercomputers inJapan andChina. In fact, there are four countries computing. top The two on spots theTop500 Listnow to belong high-performance not alone be in pushing of ORNL will the boundaries the exascale. incorporatedbe at the exascale. We hope to scale this solution up to prototype, thehardware many incorporating of approaches thatwill than Titan, 200petaflops.be an This of systemwill exascale in the range by more about2018.magnitude powerfulbe an orderwill first The of We envision two systems beyond Titan to achieve exascale performance parallelism, but once do, they for pay years their will off to efforts come. havewill their fields their adapt to of codes to thisnew level of ORNL but around the world. Researchers want who to push the limits Annual Report 2010–2011

3 SCIENCE AT THE OLCF 4 SCIENCE AT THE OLCF In aLeagueofItsOwn J Oak Ridge Leadership ComputingFacility barrier.petaflop second scientificapplications istheroster thathave of broken the Prize, which recognizes the world’s foremost scientific application. The two measures. with illustrated best firstThe is the annual Gordon Bell Jaguar’s prominence attheforefront scientific computingbe can of it’s aunique scientific instrument.” like theLarge Hadron Collider inEurope or theHubble Space Telescope; cluster—thea workstation or a departmental machine isreally more it feels like computing“Though on any computer—for modern example sciencedirector attheOakRidge Leadership of Computing Facility. “Jaguar remains aunique resource,” noted Bronson Messer, acting anything comparable to Jaguar’s record scientific productivity. of itsown.Jaguar isinaleagueof No other system intheworld has research, serious In you of fact, focuson performance intheservice if the listdoes notevaluate applications thatproduce scientific results. it reliesBecause on the benchmarking High Performance Linpack test, But theTop500 story. Listdoes nottell thewhole Jaguar isnotinthetop 10. toneed go back to June 2006inorder to finda Top500which Listin supercomputing. brutal competition at the pinnacle of In fact, you most powerful systems, which is impressive youwhen consider the When itcomestorealscience,Jaguarhasnopeer It the lastthree hasspent years ranked among theworld’s three ful supercomputers. aguar has an had impressive among the world’srun most power - indispensible in applications ranging fromresonance indispensible inapplications magnetic ranging imaging Superconducting materials are immensely both useful—being 1.64petaflops. performance of that, at the time, 180,000-plushad processor cores and a peak to break the petaflop barrier, reachingpetaflops 1.35 on a Jaguar system DCA++, astheapplication isknown, wasthefirst scientific application ORNL and the Swiss SchulthessNational of Supercomputing Centre. brought inthe2008Gordon Prize Bell for ateam by led Thomas An application that simulates superconducting copper-oxide materials Gordon BellPrize2008:High-temperaturesuperconductors phenomenon reacts.” actually this particular that universe. You can parameterschange and you various can howsee before now. And, you because are simulation, doing itvia you control phenomena thatyou know happen but were unable to insimulations see “As you increase resolution you physical the see development of experiment. anumerical run speeds, atsustained petaflop applications thatcanrun you’re able to a way to confirm physical intuition,” Messer noted. “But now,with computational science, of thehistory “For simulation mostof hasbeen electronics. which broke of all new groundto from infields to biology seismology past three years on Jaguar, also ran as did many finalists for the prize, them were on Jaguar.of running all Gordon Prize for Bell the winners Five applications have greater than1 petaflop; sustained performance of superconductors. Gordon BellPrizeforitssimulations ofhigh-temperature Supercomputing Centre,center, wasawarded the2008 A teamledby Thomas SchulthessoftheSwissNational kept notbe that need socold. superconductivity and, possibly, new superconducting design materials in order to better uncover high-temperature thesecrets of superconducting. simulates DCA++ cuprate materials atom by atom kept be still wellbelow minus Fahrenheit 200degrees to be the mostadvanced superconductors discovered sofar, yet must they must kept be cold. very Copper-oxide materialsascuprates known are systems—and immensely challenging, transportation they because machines to the Large Hadron Collider levitation to magnetic principles, quantum-mechanical exploration of the energy containedprinciples, theenergy quantum-mechanical exploration of Aprà and colleagues achieved 1.39 petaflops on Jaguar in afirst- Another ORNL-led team was a finalist for the 2009 prize. Edoardo for stronger, more stable magnets. that have not yet produced—thebeen application is useful in the search materials—even specific materials revealing properties of the magnetic particular, temperature on thosesystems. of theeffect By accurately application—WL-LSMS—analyzesTheir systems and, magnetic in 2.33petaflops. of to 224,000 processors upgraded that been had performance and a peak 223,000used processors to reach 1.84petaflops on a Jaguar system went 2009prize The to ORNL’s Markus Eisenbach andcolleagues, who Gordon BellPrize2009:Magneticsystems tool for seismic wave simulation.” calculation onsystem theORNLpetaflop that powerfulenables a new solittleeffort,”system with noted Carrington. wasalandmark “This “It’s amazingthatacode thiscomplex couldto ported such be alarge years to come. computational wellfor scientists bodes groundbreaking science inthe traveling theseleading through theearth. experience The of the used system to shatter its own record in seismicmodeling waves Supercomputing Diego theSan byof led Carrington Center Laura electronic structure.conducts calculations of first-principles A team Gordon Prize Bell for algorithmic innovation an application with that Wang Wang Lawrence of Berkeley National won aspecial Laboratory Two other 2008finalists advantage Jaguar.took of A team led by Lin- Annual Report 2010–2011

5 SCIENCE AT THE OLCF 6 SCIENCE AT THE OLCF available to other researchers, able to be usethesehighly will who technique known as coupled cluster. make Theteam will its results aquantum water chemistry by meansof of electronic structure application 223,200processingThe used cores to accurately study the Northwest National Laboratory. application known as NWChem, which was developed at Pacific waterin clusters molecules. of team The acomputational used chemistry minimizing approximations assumptions. or simplifying Schrödinger equation from for electronicfirst systemsprinciples while toand scaled Jaguar thefull system thatsolves inamethod the University Tennessee–Knoxville. of The team reached 1.3 petaflops and Anton Kozhevnikov ETHZurich and of Adolfo G. the Eguiluz of An honorablemention inthe2010competition went to Schulthess cells have through plasma. addition, it included the long-distance communication that red blood andchange shape,simulated cells thatwiggle like awater balloon. In cells. While earlier simulations red spherical were cells, blood of MoBo cells blood beingsimulated, those of the realism of but alsoof because team’sThe the number work wasaleapforward notonly of because system.circulatory plasmainthe red200 million cells with blood andtheir interaction 700 simulates flow. blood The 196,000 team used processors run at to MoBo,colleagues took with theprize afluid dynamics application that finalists,winner. including the Georgia Tech’s George Biros and Jaguar wasprominent for again the2010prize, six four hosting of Gordon BellPrize2010:Redbloodcellsinplasma materials codeusefulinanalyzingmagneticsystems. Gordon BellPrizeforitsworkonJaguarusingWL-LSMS,a A teamledbyORNL’sMarkusEisenbach,left,wonthe2009 accurate dataasinputs to theirsimulations. own Oak Ridge Leadership ComputingFacility trillion calculations a second,trillion or 700 teraflops, simulating The simulation needed the power of Jaguar simulationThe thepower needed for two of reasons, principal the earth’s 500mileslong, crust 250mileswide, and 50milesdeep. went on to assesstheshakingproduced by thisrupture on achunk of California andcontinued to nearly theMexican border. simulation The SCECteamThe simulated inCentral a340-mile rupture thatbegan spreadas they through theregion. travels along thefaultandearthquake waves andresultant shaking professor Kim Olsen. application The calculates therupture both asit developed by team State member Diego andSan University geophysics for earthquake onbased method calculating anumerical waves originally AWP-ODC,called for Anelastic Wave Propagation–Olsen-Day-Cui, teamThe anearthquake used wave propagation application program large-scale seismic simulation.previous the simulation reached 220 teraflops, any more than thetwice of speed 125,000-square-mile swathacross California. Southern Known asM8, Jaguar’s processing cores to simulate 8quake amagnitude shakinga focusing on thestate’s San Andreas Fault. team of The all nearly used the world’sperform mostadvanced earthquake simulation to date, CaliforniaSouthern Earthquake Center (SCEC), Jaguar which used to A third finalistwas a team led Thomasby Jordan, the director of without getting bogged down. bogged getting without processors thesimulation thousandsof allowing to on hundreds run of simulation’s to ability reallocate computing tasksamong processors, asimulation.and dynamic areas of In particular, improve they the or AMR, which focusesacomputer’s power on themostimportant advance algorithms The atechnique asadaptive known meshrefinement, and reached 175teraflops. magazine. On Jaguar the team scaled the algorithms to the entire system Ranger system on itaspot thecover earned the August of 27Science tectonic plates. Its work on theTexas Advanced Computing Center’s hot, of rock viscous and falling thatisamaindriver for themotionof convectivewithin theearth’s flows mantle, the rising multimillion-year teamThe conducting has been groundbreaking simulations of supercomputer’sa power. advance an innovative algorithms that selectively collection of focuses Technology,California Institute of andRice University Jaguar used to Finally, researchers from theUniversity Texas–Austin of (UT), the simulations ateven frequencies. higher so M8represents amilestone toward similar thelarger goalof to strong ground motionsusewaves intheir analyses, upto 10hertz the other hand, responses analyze who building engineers structural frequency requires a16-fold increase incomputational resources. On Diego’s Supercomputing Diego San Center, each doubling inwave Yifeng Cui, acomputational scientistattheUniversity California–San of calculate earthquake waves above 1hertz. According to team member No able thisscalehasbeen to directly earthquake simulation previous of 2 hertz, or 2cycles second, per to resorting approximation. without theseismic waves,of which thesimulation wasable to calculate upto intodivided 436billion 40-cubic-meter cells. Second wasthefrequency studied, theregionbeing First wasthesize of which thesimulation according to architect SCECinformationtechnology PhilipMaechling. smallest possible scale. possible smallest travel electrons asthey atthe through electronic devices of journey than 220,000processor cores andreached 1.03petaflops the to model Purdueof University. Klimeck andPurdue’s Mathieu Luisier more used petaflop A fifthapplicationcame from aled team by Gerhard Klimeck and Aprá’s team’s water molecules. simulation of systems, Eisenbach’s team’s systems, magnetic analysis of prize-winning superconductors and solution to the Schrödinger equation for electronic Schulthess’s teams’ high-temperature simulation of prize-winning or finalistsintheGordon eitherbeen winners Bell competition: theapplications thathaveFour of broken have barrier thepetaflop At thepetascale:Modelingjourneyofelectrons four-orders-of-magnitude reduction intheproblem size.” to cells. 10,000-fold reduction inthenumber of isalmosta This believe that’s abig accomplishment. We have a5,000- on theorder of towe ran over efficiency, parallel 200,000cores high very with we noted UTcomputational geoscientistOmar Ghattas, “so that thefact “Algorithms inthepasthave scaling evento difficulty had 10,000 cores,” the AMR. Jaguar usingonly thesystem’s 0.05percent of resources on specifically teamThe wasable to overcome thisobstacle, thesimulations on with resources andprogressively lessto calculations. do theactual usingprogressively of asystem’s therisk morerun of power to reallocate therefore, reallocated be among theprocessors. As aresult, simulations the cells inthoseregionsbecome larger andwork andsmaller must, become more asimulation, or lessactive thecourse during of meaning scaling thetechnique for large supercomputers. Different regions While AMR isnotanew approach, researchers have great difficulty had simulating bloodflow. In 2010,ateamledbyGeorgia Tech’s GeorgeBiros,thirdfromleft,wontheGordonBellPrizeforitsworkonJaguar atoms’ outermost shells—from their fundamental properties. These in the system—valence particles most important electrons located on NEMO). of team The calculates nameisananagram the whose effort as Nanoelectric Modeling (NEMO)3DandOMEN(amore recent teamThe ispursuing thiswork on Jaguar two applications, with known atomicLeo Williams nuclei andelectronsby on theinner shells.— the applications approximate particles—the less critical the behavior of are theelectronssystem. that flow inandout of Ontheother hand, Image courtesyGerhardKlimeck,PurdueUniversity. as theymovefromthesourcetodrainoftransistor. energy byemittingphononsorcrystalvibrations(greenstars) tal) andenergy(vertical).Electrons(filledbluecircles)lose sistor. The currentisplottedasafunctionofposition(horizon Illustration ofelectron-phononscatteringinananowiretran Annual Report 2010–2011

- -

7 SCIENCE AT THE OLCF 8 SCIENCE AT THE OLCF C Groundbreaking computational research, however, relies on ground- Oak Ridge Leadership ComputingFacility computing 135,000dual-core power of laptops. reach asmuch as557teraflops. haveTogether they theroughly 2.33petaflops,Jaguarwhile performance of hasapeak Intrepid can Intrepid, ranked number 15. three intheworld, andthe Argonne Leadership Computing Facility’s Science systems,Office of theOLCF’s Jaguar,ranked currently number the physical sciences. INCITE researchers work on two world-class Science,Office of the United States’ research largest supporter of in resourcesThe Energy’s are provided of by (DOE’s) theDepartment theseunique resources. makeis thatthey themostof theworld’sof mostpowerful supercomputers. only The requirement advanced computational researchers receive on substantial time some Through INCITE, asitismore commonly known, theworld’s most comes program andExperiment in. Theory That’s where the Innovative and Novel Computational Impact on canget,they could but they often usemore—sometimes much more. breaking computing power. thecomputing Scientistsuseall resources INCITE givesresearchersunprecedentedcomputingpower 4.5 BILLIONHOURSANDCOUNTING crannies of thenanoscale, inbetween. of andeverywhere crannies theuniverse,new knowledge atthelimitsof and thenooks omputational researchers are anambitiousbunch, hunting for challenges. or itmay choose instead to consult itsliaison only on specific with may askitsliaison to join theresearch team member, asafull-fledged code indesigning andoptimizingitfor Jaguar.experts A large project mathematics, numerical analysis, computer science—but are they also liaisons areintheir fields—chemistry, experts physics, astrophysics, supercomputerskilled andcomputational experts scientists. OLCF system, highly of as well as world-class systems support and a staff it with America’s mostpowerful supercomputer, theCray XT5 Jaguar OLCFThe isdeeply committed to theINCITE program, supporting INCITE atthepetascale approaches.”to analytic complex systemsof which are simply too hard or sofarare notamenable donebe exists—simulationseither experimentally or for which no theory aswell.support It gives to ustheability simulate which cannot things through simulation, science. is the third of pillar It a staple is actually andtheory.experiment In fact, computation, high-end especially “It isoften saidthatscience on isbased two pillars,” hesaid, “namely talk to theCouncil on Competitiveness. for science. explained thereasoning Orbach INCITEina2007 behind was created in2003by Raymond Orbach, then DOE’s undersecretary Now managedby thetwo computing facilities, theINCITEprogram experts fromexperts national laboratories, universities, and industry. Winners in an annual for participate call proposals, proposals with reviewed by Researchers interested in doing work on theINCITEsupercomputers atmosphere thatforms. ” don’t you’re if born get notable to resolve theshearlayer inthe resolution regionalclimate where models you get hurricanes. They simulations,” hesaid. “A example good are therecent, very-high- phenomena that you know are there but weren’t present in earlier “As you increase resolution you physical the see development of time, but now we have awave applications.” of simulations.predictions with We’ve thatfor along examples had of “That’s anew thing, to some extent—to able to be make quantitative theoretical result. out that you would have confidence comparing to anor experimental resolved highly very really physical good with models. You numbers get at theOLCF. meansthey’re“This able to experiment— anumerical run huge computations,” saidBronson Messer, science director acting of “Applications are speeds atsustained petaflop able thatcanrun to do knowledge and advance technology. to researchers important is critically speed toThis needing improve thefive applications of reported to have brokenpetaflop barrier. the Jaguar, created supercomputing, to atthevanguard be of hosted each research need. thetools scientistsget they andhelp ensure helps support This thattheworld’s mostambitiouscomputational INCITE Hours Allocated toOLCF processor hours from INCITEover theyears. government have will laboratories allotted been more than4.5billion up,By 2011wraps thetime researchers from academia, industry, and The OLCFdoesitspart hours,27 million one receiving with more hours. than110million received 1.7billion processor atotal of hours. averaged allocations Those Scientific Computing Center. In contrast, the57 awardees in2011 Lawrence Berkeley National Laboratory’s National Research Energy three projects received just under processor 5million hours total from modestly by today’s off INCITE started standards. In itsfirst year, 2004, processor hours itwould have to day for run 12½years. andnight 224,000 processor hours. For adual-core laptop to reach 224,000 Jaguar’s of hour usingall 224,000processing cores, inturn, would be A processor hour, then, houron one processing isasingle core. A single have tens thousands. to hundreds of your laptop may have two or even four processing cores, supercomputers use processors much like those found on home computers. But while hour. Many theworld’s of mostpowerful systems, including Jaguar, allocates computing INCITEprogram The resources by theprocessor systems—about 45,000processing cores inJaguar’s case. INCITE projects are expected to use 20percent the INCITE or more of and their potential to advance science andtechnology. In general, are picked to thesupercomputers for both their ability make themostof Annual Report 2010–2011

9 SCIENCE AT THE OLCF 10 SCIENCE AT THE OLCF • • • • research: energy, projects thegamutThose of run technology, andbasicscience 178 yearly totaling allocations 2.5billion processor hours. totaling processor 950million hours. told,All theOLCF hascontributed five projects. By 2010theOLCF with allocations 45 washosting projects joined the program joinedin 2006,the program providing 3.5 processormillion hours to have thosehoursMost willl come from of theOLCF. first facility The Oak Ridge Leadership ComputingFacility other phenomena. better understand catalysis, corrosion, growth, crystal andmany haveAnd they on molecules surfaces to simulated thebinding of temperature superconductors andsolarcells atthenanoscale. charge carriers. havemechanical states They of high- modeled mechanical forces at work in electronic and devices the quantum atoms andmolecules.the scaleof have They thequantum modeled Materials scientistshave examinedat electronics andtechnology membranes. fusion. haveAnd they lookedproteins attherole of inbiological DNA. have They looked proteins attherole of inmembrane material genetic andtechniquesthe replication for sequencing of andpoplar trees.plants such asswitchgrass have They looked at Biologists tothat are many immensely important processes. industrial coal. And have they taken a close at the look chemical catalysts relatively for theproductiontechnology from of clean energy internal combustion engines. have They simulated gasifier haveThey worked to understand combustion theprocess in of Chemists have simulated more radically powerful carbatteries. commercial explosives. andstoring have transporting they simulatedof thesafety have nuclear reactors. simulatedof andoperation thedesign And researchersEngineering aircraft. have of thedesign They modeled have from ethanol simulated woody theproduction of Visualization courtesyJamison Daniel,ORNL. configuration oftheCCSMCommunity AtmosphericModel. of thetotalcolumnwatervaporfromahigh-resolution image showsthesimulatedmonthly-averageddistribution hydrological cycleandtheplanetaryradiationbudget. This climate systemisessentialtoanaccuratetreatmentofthe The propersimulationofwatervapordistributioninthe 2011 State of theUnion2011 State of Address. was promising to from enough acallout get President inthe Obama is,This course, of awoefully incomplete list. Another INCITEproject • • • feasibility of storing carbon dioxide storing carbon deep underground. of feasibility examined ingroundwater contaminant migration andthe building and emergency safety preparedness. And have they forecasting. have They simulated earthquakes in order to improve past. have They inorder simulated hurricanes to improve and predictive for models climate inthefuture andinto thefar Earth astrophysicalthe behavior of plasmasaround thesolarsystem. determine the galaxy. the size and shapeof And have they examined to matter undetectable nearly dark year gathering simulations of thesun.or more have themassof times They 13billion-plus- run scales.fantastic have They stars simulatedeight the explosion of Astrophysicists have INCITEto used explore theuniverse at wave theplasma. at radio of heating eye to thefusion confinementreaction. of have And they looked turbulence. an have They theplasmawith examined theedgeof electrons. have They by simulatedlosscaused plasma energy ions and at the muchturbulence scale smaller of at the scale of thedecade.by theend of researchers These have on focused plasma be theworld’sbe largest fusion reactor operation to andisset begin Fusion researchers have simulated ITER, which, built, when will and climate scientistshave developed ever-more refined make sure do.—Leo they Williamsby scientific supercomputers.be there will INCITE to forward to access getting to thenext generationof Researchers andmany inthesefields others look • • • able such to be systems. do with understands itwill what computational science community thatalready resourcesThese arrive none too soonfor a will thedecade. by theend of an exascale system more 500times powerful thanJaguar asitmovesthe program beyond toward thepetascale able to be contributeORNL will progressively more to and another in2011. 18million processor 8million hours of in2010 allocations with unprecedented simulationsperform of detail on Jaguar INCITE gives Evans andcolleagues theresources to the neutrons responsible for nuclear fission. radiation’s source, photons, itbe whether electrons, or the any stronglywith thatdoes notinteract radiation inafluid, particles individual to simulate of distribution equation,Boltzmann thestatistical which describes by ORNL physicist Thomas Evans. It solves the applicationThe they’re Denovo, usingiscalled created organizations. laboratories, universities, private industry, andother Water Reactors, national anORNL-led collaborationof the Consortium for Advanced Light Simulation of our nuclear facilities.”out of “They,” in this context, is “they’re usingsupercomputers to alotmore get power “At OakRidge National Laboratory,” thepresident said, Nuclear simulationgetsattention of attributes, remain unaffected. will of few industries reactions, properties, electrical or any combination maximized for strength, resilience, chemical When chemists are able to new materials design element factories. theuniverse’sthe secrets of mostproductive massive stars into space, have will they uncovered manner inwhich collapsing iron cores blow When astrophysicists have worked out the precise wean usfrom ourreliance on foreign oil. step toward producing low-cost thathelps ethanol sugars atthenanoscale, have will they taken alarge into itscomponent switchgrass breakdown of principles, theenzymatic nanoscalesimulations of When chemists are able to accurate, perform first- Annual Report 2010–2011 11 SCIENCE AT THE OLCF 12 SCIENCE AT THE OLCF M Oak Ridge Leadership ComputingFacility the greatest unsolvedone of problems in astrophysics. But perhaps The evolution these and elemental nature fountains of a ismystery, still collapse supernovas (CCSNs), or stars greater than8solarmasses. oneAnd day nostars do itbetter become thanthosethatwill core- by Crosby, Stills, Nash, and Young. Even thehippies know it. we are golden, we are billion-year-old carbon” goes thesong “Woodstock” ourselves some star’s istheresult of demise. violent “We are stardust, life aswe know it:of hydrogen, oxygen, carbon, etc. Everything including And, eventually, explode, they littering the universe the elements with unfathomable strength. by constantly changing magnetic fields of almostby constantly fields of magnetic changing churn,they turn, they creating thewhile hostage andheld all assive stars are inherently creatures—they violent burn, stellar giants evolvestellar giants into elemental firecrackers. fields, researchers could ultimatelygreat explain a deal as whyto these themysteryIn surrounding thestars’ untangling powerful magnetic Astrophysical Journal. Its findings were publishedinthe 20,April 2010,The issue of CCSNs.fundamental fields of questions surrounding the magnetic simulations allowed theteam themostof to address some of thesemassivehow theexplosions impact stars. of they A recent suite researchers wantto know are fields how created thesemagnetic and Essentially, Endeve, said Eirik the team’s author lead of latest paper, Jaguar. of help the CCSNexplosions with closergetting of to explaining theorigins not unsolved for long. A team byled ORNL’s Tony Mezzacappa is California–Davis. Image courtesyKwan-LiuMa,Universityof out complexflowpatternsinthesimulation. the SASI. The fluidvelocity streamlinestrace supernova shockwaveduringtheoperationof Volume renderingoffluidflowsbelowthe similar to the expansion of aspring. similar to theexpansion of which canalso stretch and strengthen the progenitor’s fields, magnetic theSASI event,another during susceptible ishighly to turbulence, the SASI, counter-rotating when one layers against thestarrub of the team’sAnother major finding of simulations isthatshearflow from fields. magnetic mode drive rotation, apulsar’s itcanalsodetermine thestrength of mode,created thespiral via andsecond, that notonly canthespiral rotation asakey thatserves driver asupernova’s behind is magnetism new revelationThis meanstwo to things astronomers: first, thatany star spinning, but alsoyank andstretch as fields well. itsmagnetic enters theinterior,boundary creating vorticesthe thatnotonly start supernova explosion comes to ahalt, matter outside theshockwave (SASI).shock instability As theexpanding shockwave the driving supernova’s core inaphase asthestandingaccretion known stalls modeoccursthe spiral shock when wave expanding from a Nature of 2007 edition , demonstrated thataphenomenon as known supernovae” intheaccretionan instability shock of intheJanuary simulations, which were theresults publishedin of “Pulsar spinsfrom pulsar spins is likewise responsible for fields. their magnetic The earlier the latesttogether simulations with reveals behind that the culprit Interestingly, this finding builds on the team’s previous work,which driver.could aprimary be generation mechanism, contradicting accepted that rotation theory theseprogenitorsthe simulations of revealed arobust magnetic-field- Supernova progenitors tend to slower-rotating be stars. Nevertheless, star rotates fields. thestronger itsmagnetic apulsar’sin determining thestrength of faster field—the a magnetic rotate. asthey lighthouses rotation This to abig be factor isthought emitted by apulsar, produced brightness by similarto thevarying radiation stars their get namefrom of pulsingbeam theseemingly stellar community. magnetized, These highly rotating rapidly neutron when it comes to fields, magnetic pulsars are the top players in the Collapsed supernova remnants are commonly aspulsars, known and Rotation isn’teverything itcomeswhen to how operate. thesestellar marvels supercomputer. process The revealed have thatwe still much to learn hours on Jaguar, of millions using tens of theUnited States’ fastest simulated asupernova progenitor, or astarinits pre-supernova phase, fields,In to aneffort locate themagnetic thesource the team of mechanism, contradictingacceptedtheory Simulations oftheseprogenitorsrevealed that rotationcouldbeaprimarydriver. a robustmagnetic-field-generation conductive fluids. After theinitial setting conditions, the ran team equations, electrically or equations the properties that describe of magnetohydrodynamic simulationsThe solved essentially of aseries counterpart, which assumed CCSNswere spherical. perfectly andmakes fewer andgravity approximationstransport thanitsearlier Florida Atlantic University, features anovel approach to neutrino Budiardja, and Tony Mezzacappa at ORNL and Pedro Marronetti at under development by Cardall, Christian Endeve, Eirik Reuben to studytheprogenitor’s the evolution of fields. magnetic GenASiS, teamThe theGeneral used Astrophysical Simulation System (GenASiS) The GenASiSofmagneticfields field to pulsarstrengths,” themagnetic magnifying saidEndeve. supernova progenitor, mechanism thisturbulent iscapable of similar field what to we thinkisin a amagnetic with that starting flow-induced turbulence roiling the inner the star.core of “We found efficientlycan be generated by a somewhatunexpected source: shear twoThese taken findings fields together show thatCCSNmagnetic of ouruniverse.—of by GregoryScottJones levels, science one step bringing closer to afundamental understanding able to investigate stellar cataclysms atunprecedented these important CCSNs.mysteries of As GenASiS continues to evolve, be theteam will latestThis discovery isjust one more step toward unraveling the require Jaguar’s unprecedented computing power. Endeve, to need employ itwill anadaptive mesh. nodoubt And itwill CCSNs. However, to make such a powerful code economical, said gravity,relativistic give itaneven which will more realistic picture of and next incorporate team sophisticatedThe transport will neutrino Messer,” hesaid. Rossespecially Toedte, andthegroup’s liaison to theOLCF, Bronson “We have alsoreceived from help team, thevisualization alotof team’s research, data. asone produces model terabytes of hundreds of center’s High Performance Storage System to the important is very here facilities [OLCF]“The are excellent,” saidEndeve, thatthe adding Jaguar’s queue scheduling isa policy “great advantage.” job sizes isthesamedue to totime solution for thesehugely varying inaCCSN. activity He thatthe themagnetic thatthefact added of are upto scaled 64,000cores, theteam amore giving realistic picture becomes more turbulence andother factors, chaotic with thesimulations at 4,000 processing The model starts cores, said Endeve, but as the star interesting. occurred. However, uptheresolution, ramped asthey got things werethey atlower run resolutions, activity littlesignificant but very takingmoremodels thanamonth to complete. Initially, saidEndeve, several atlow models resolutions, andhigh thehighest-resolution with Annual Report 2010–2011 13 SCIENCE AT THE OLCF 14 SCIENCE AT THE OLCF C Oak Ridge Leadership ComputingFacility and 50milesdeep. theearth’srupture on achunk of 500mileslong, crust 250miles wide, level. simulation The went on to assesstheshakingproduced by this community theSalton on theeastern Sea, shore of below 225feet sea car accident—and continued southeast to Bombay Beach—a desert Luis Countybest known for Obispo its proximity to James Dean’s fatal Andreas Fault nearCholame—an unincorporated community inSan SCECteamThe simulated on theSan a340-milerupture thatbegan United States, Greenland, eastern Russia. chunk andahefty of Ocean, the North while American Plate includes the the remainder of California,California andBaja aswellHawaii the Pacific andmostof North tectonicAmerican plates. Pacific The Plate includes asliver of San The betweenAndreas thePacific Fault theboundary forms and for notonly themostlikely, but alsotheworst case.” wereis thatthey wantto ready they thequestion be changing because and said, ‘Tell uswhat’s theworst thatcould happen.’ My understanding we’re going to have to face.’ But after changed Katrina thequestion they ‘What’s likely to happen? Give usthe most probable scenarios that Hurricane Katrina,” Maechling noted. “Before Katrinawere they asking, go back to achange from emergency management after organizations these investigations,“Some of the oneespecially we just did on Jaguar, that could plausibly California. hitSouthern thelargest quakesthe center itisone of chose8because magnitude According to architect SCECinformationtechnology PhilipMaechling, last year. in 1906—oraspowerful 30times asthequake thatdevastated Haiti aspowerful again asthetemblor thatdestroyedhalf Francisco San Known asM8, theSCECproject simulated a6-minute earthquake application. Prize, awarded to theworld’s mostadvanced scientific computing sixfinalists for the2010Gordon theteam earned aslotasoneBell of on Californiatheregion.Southern andassessitsimpact simulation The simulate quake amagnitude-8 shakinga125,000-square-mile area of Earthquake Center (SCEC) director Thomas Jordan Jaguarused to California anditsSan Andreas Fault. A team by led California Southern anearthquake ever on asupercomputer performed of focuseson It shouldcome asnosurprise, then, thatthemostadvanced simulation often major, quakes. potentially devastating two major tectonic plates to andissubject relatively frequent, takesalifornia seriously. earthquakes very state The straddles 2 hertz, or 2cycles second, per to resorting approximation. without theseismic waves,of which thesimulation wasable to calculate upto intodivided 435billion 40-cubic-meter cells. Second wasthefrequency studied, theregionbeing First wasthesize of which thesimulation The M8 simulation required Jaguar for two reasons, Maechling noted. California’s next big shakeup. inevitable planners working to minimize brought thedevastation on by Southern not only to seismologists, but also to building and emergencydesigners As aresult, provided information valuable be by theM8project will chunkThat ishome to one people—about in15 20million Americans. in structural engineering andbuilding design.” engineering in structural And purposes, thatacceleration isreally especially for needed alotof unless we go to frequencies—2 or preferably higher hertz even higher. higher-frequency concept. We won’t able to be capture thatacceleration thepointor thegroundisreally location but a theacceleration of able to be we capture might thedisplacement or even thevelocities, up anddown, the point. or theacceleration of With thelow frequency side to side atapoint, thepointfrom side to or of thevelocity side or “For example, you cancharacterize thewaves by thedisplacement from thewaves,”parameters of noted SDSUgeophysics professor Kim Olsen. “Going allow to frequencies usto thesehigher capture will alotmore frequencies. a milestone toward similar simulations ateven the larger higher goal of motions usewaves intheir analyses, upto 10hertz soM8represents responses to buildinganalyzing engineers structural strong ground 16-fold increase incomputational resources. Ontheother hand, California–San Diego,of each doubling inwave frequency requires a Yifeng Supercomputing Diego theSan Cui of Center attheUniversity earthquake waves above 1hertz. According to computational scientist No able thisscalehasbeen to directly earthquake calculate simulation of sloshing backandforthinthetub. That’s an analogytowhathappensseismic Imagine abathtubfullofwater. Ifyou kick thebathtub,waterwillstart waves inasedimentarybasin. geology. Decades of oil exploration mean that the geology of theareageology. of oil exploration meanthatthegeology of Decades SouthernCalifornia’sthree-dimensional (3D) modelof complex thesimulation anaccurate regionisthatitused within locations all One reason theM8simulation produced accurate ground motions for waves andresultant spread shakingasthey through the region. calculated the rupture as it traveled along the fault, and the earthquake difference equations defined on astaggeredgrid, theapplication waves developed by Olsen. originally partial acoupled Solving system of Day-Cui—based for onearthquake amethod calculating numerical AWP-ODC—forcalled program Anelastic WavePropagation–Olsen- simulation. It anearthquake used wave propagation application any completed previously of thespeed large-scale seismictwice processing cores. simulation The reached 220teraflops, more than Jaguar’s of all nearly for 24hours andtakingadvantage of 224,000-plus The project conducted its first Jaguar simulation in 2010,April running Image courtesyGeoffrey Ely, UniversityofSouthernCalifornia. Peak horizontalgroundvelocitiesderivedfromtheM8simulation. extended.” waves tend to thewaves amplified, get of and the duration tends to get sediments in these basins can exceed a kilometer. In those areas the Bernardino,” Olsen explained, “and relatively thethickness of loose basins such asthosebelow Los Angeles, Fernando, San andSan waves“The are stronger generally inpromulgating through sedimentary geology. of forms populated areas have shakinglonger andharder thehabit of thanother Unfortunately, the layersand underlie most sedimentvalleys that fill of makes a profound difference in the way an earthquake plays out. haveEarthquake experts discovered the geology that the nature of available for theregion. thoroughlyhas been analyzed, are models andaccurate 3Dgeological

Annual Report 2010–2011 15 SCIENCE AT THE OLCF 16 SCIENCE AT THE OLCF S From there provided itcanbe to Japanese scientistsandofficials. Steven command Secretary Chu.sends themto upthechain of Energy resultsThe are forwarded to ORNL Director Thom Mason, then who Light Waterof Reactors. those developed by the ORNL-led for Consortium Advanced Simulation the OLCF’s simulations, codes, which have of including avariety used computer-aided isamixof from effort The anddatagleaned design arebundles theconsequences. to where mitigating iscritical arethe bundles put into atdifferent thepool times, which knowing differentdecay rod fuel pool. ratefuel of inthespent bundles Because According to Pugmire, themajor issuesisthe studied being one of rods fuel spent within. site, the frombehaviorto the the of reactor fly-bys building itself of andDavid Pugmire,Daniel theplant of are aspects simulating various and around theplant, two OLCF specialists, visualization Jamison In in order isoccurring exactly what to give of theORNLteam aview the Japanese government information. with nuclear andcomputational specialists to scientistsinaneffort provide OLCF VisTeamHelpsSimulateJapaneseCrisis The teamThe hascompared results from data theM8simulation with to happens to what seismic waves basin.” inasedimentary inthetub. sloshingback andforth start the water will That’s ananalogy water,” of abathtub“Imagine full he said. you kick“If the bathtub, on. thanmoving rather back andforth bounce populated regionstend earthquake to waves, trap which then In fact, hesaid, unconsolidated thesegeologically andoften highly for future earthquakes. byused thestate’s buildingandemergency designers agencies to prepare contributeIn these simulationssignificantly time to theinformation will to scientific question. address thisimportant attemptAdditional SCECwill M8rupture scenarios within planned and may account strong for theespecially ground shakinginthebasins. “supershear” rupture creates awave analogous effect to asonic boom wave a particular knownthan of as the theshear speed wave. This simulation includes long stretches inwhich theground ruptures faster earthquake.amplification inaspecific Inparticular, henoted, the not reflect caused effects by complex source propagation and 3D basin discrepancy isreadily explained by that averaged the fact records do than predicted by theaverage datarecords, noted Olsen, but that Angeles and Ventura Los (northwestAngeles) of larger wasgenerally available data. basinssuch asLos ground The motioninsedimentary (i.e., the basins)matches wellwith areas very other thansedimentary particular, theaverage shakingseen intheM8simulation on rock sites averaged from many real earthquakes pleased. andisgenerally In Oak Ridge Leadership ComputingFacility the crisis, employing thelaboratory’s talented unique mixof Dai-Ichi nuclear plantinMarch 2011, simulating of ORNLbegan oon after Japan’s theearthquake andtsunami crippled Fukushima —by GregoryScottJones Nichols,said Jeff ORNL’s associate labdirector for scientific computing. up to thebigger do machines),” some large-scale simulations (on some of computing resources, that may the not case be for long. “We’re working And thesimulations while are currently done being on ORNL smaller around theworld.”—Leo Williamsby aboutearthquakes herelearn applicable shouldbe to earthquakes understand the seismic hazard California.in Southern we things The earthquake processes of models andthen to to usethesemodels better “We’re doing simulations in order to develop and validate predictive atrisk. people are alotof region, meters. ground motionsensors andGPSstrain Plus there the well desert—is exposed. There’s instrumentation in the a lot of he said. “We’ve got abig plate boundary. it’s lotof geology—a The in you’re“If earthquakes, studying California spot,” Southern isagood the globe, notjust inCalifornia. atearthquake-prone looking scientists andother experts regionsacross According to Maechling, ultimately useful to be thisknowledge will howgive theground uncertain motionis.” them anidea of the mean. I think that’s a lot more valuable for engineers. will That epicentral locations and slip on the fault, around and find the variation thesedifferent realizations, thepointsof of all including different motions,” explained Olsen. “We take theaverage ground motionsfor ground thefaultintoon we thesamestretchanensemble what call of of into thespentfuel pool. three racks. The rodsarecoloredbythedatethey placed pool fromabove. The spentfuelrodsareplacedinto one of Rendering oftheFukushimareactor buildingspentfuelrod “Basically, we combine different thesameearthquake realizations of

T Site’s 300 Area. ImagecourtesyPeterC.Lichtner, LANL. Jaguar tomodelthedistributionofuraniumatHanford Peter LichtnerandcolleaguesrunthePFLOTRANcodeon and DOE. Ecology, of Department theEnvironmental Protection Agency (EPA), stems from a 1989Tri-Party Agreement involving the Washington research,This among thelatest incleanup attheHanford efforts Site, simply dischargedhad been to pondsandtrenches, Lichtner said. tanks. thegroundwater But now theuranium penetrating andriver storedhas been atHanford since 1940s, theearly mostly inunderground Columbia River for decades. Waste from nuclear weapons production HanfordThe plumegroundwater polluting hasbeen andthenearby at OakRidge andother contamination.” areas dealing with other sites as well, not only at Hanford, but also around the country— too. And wefrom what learn thissite we able to shouldbe apply to could apply to other sites along theColumbia River thatare contaminated Research and Advanced Scientific Computing Research. “The results receives and Environmental funding fromBiological offices theDOE of remediating thesite, it’s if even feasible,” saidLichtner, project whose project’s“The results could one certainlyhelp decide how to go about contaminant remediation. the Columbia River, may aidstakeholders options for inweighing uranium, which propertiesof hasinfiltrated underground migration at theHanford Site’s 300 Area. theA better understanding of anunderground waste uranium three-dimensional plume of model Los Alamosof National (LANL) isusingJaguar Laboratory to build a waste grounds in the country. A research team byled Peter C. Lichtner the Manhattan Project—is among themostcontaminated nuclear Supercomputers AssistCleanupofDecades-OldNuclearWaste Supercomputers AssistCleanupofDecades-OldNuclearWaste nearly five locations for decades, three primary and was one of forslugs nuclear weapons, actedasawaste for storage facility he Hanford Site in Washington state—which produced fuel EPA has seta contaminant 30 micrograms perliter. limit of The uranium doses cancauseboneor liverin high cancer damage. or kidney The classified asaheavy-metal radiation hazardone.rather thana Ingestion particles, external exposure isnotdeemed arisk. In fact, is uranium As decays uranium itemits alpha particles. skinblocks alpha Because theColumbiayards River. west of Hanford’s 300 Area, aroughly 1.5square milesite approximately 100 uranium, copper, and sodium aluminate. The uranium plume is in and arequantities spread throughout severalthe site, types including HanfordThe Site covers land. 586square milesof Contaminants of Uncovering theunseen Champaign. Svyatskiy LANL, of and Al Valocchi theUniversity of Illinois, Urbana- of ArgonneSmith of National Laboratory, Dave Moulton andDaniil National Laboratory, Bobby ORNL; PhilipandRichard Barry Mills of Lichtner’s collaborators include Pacific Glenn Hammond of Northwest to Hanford’s remediation. timely —by Wes Wade 11 hours by using more than 4,000 processors. Such is crucial speed takemight astheplume migrates. team The simulated 1year inonly 28 composing theplume require to themodel account for more than uraniumand additionalThe chemicalcompounds properties of the river’s plume. themigrating with interaction at 1hourintervals, realistic of models which allowed of construction lossfromthe uranium theplume andthe fluxinto the Columbia River cells, 5.5×0.5cubic meters each.grid of team The calculated yards andwassimulated control 2million usingnearly volumes, or gravel, silts. andfine-grained The plume measures 984×1,42222 movement soluble sand, depleted through asoil uranium mixture of of fluidthrough porous theflowmedia, describe of inthiscasethe codeThe on more run has been than130,000 processors Jaguar to of SciDAC-2,called which aimsto advance computing atthepetascale. flow through soil usingPFLOTRAN, a code developed under a project depleted uranium massively teamThe performed simulations of parallel uncertain,very saidLichtner, isto goal whose decrease thisuncertainty. uranium. Yet research further until isconducted, thesenumbers remain Columbia River each year from theestimatedsource 55to 83tons of tests indicate leach uranium into thatfrom the 55to of 110pounds the plume into theriver. Initial simulation results field coupled with A challenge for Lichtner’s from teamuranium isto predict thelossof according to field tests. contaminating Hanford’s 300 Area exceeds thislimitby four times, million degrees of freedom—the number of actions these compounds actions freedom—the number of of degrees million Annual Report 2010–2011 17 SCIENCE AT THE OLCF 18 SCIENCE AT THE OLCF I Oak Ridge Leadership ComputingFacility the open clamp and the clamp loaderof then encircles primer-template clamp’s three subunits, separate at one juncture. A complex made up Protein structures, secondary sheets, or beta the atthejunctures of from energy activate adenosine its (ATP).openingtriphosphate with or PCNA) onto DNA. clamp The first loader binds to the clamp to sliding clampa doughnut-shaped (proliferating cell nuclear antigen, researchers revealed that a clamp loader (replication factor C) places howSo does machine themolecular work inaeukaryote? The their hosts.of their clamp own loaders but instead co-opt thereplication machinery but itworks alittledifferently. Viruses, on theother hand, do nothave compartmentalized, alsohave machine amolecular to clamps, load membrane. Prokaryotes, such asbacteria, genes whose are not material genetic isenclosedand other whose inanuclear organisms on cyclefocused theclamp-loading ineukaryotes—or plants, animals, Foundation’s Center for Physics. Theoretical Biological project The by theHoward Hughes Medical Institute andtheNational Science degenerative diseases,” saidIvanov, investigation whose wassupported incancer and thisintegrity andthelossof integrity genetic of research“This on understanding hasdirectbasis themolecular bearing , inspireSociety anew approach for attacking diverse diseases. in theMay 10, 2010, theJournal the of issueAmerican of Chemical thatenable gene repairenzymes or replication. findings, Their published slidingclamps,called are onto loaded DNA andcoordinate strands Jaguarused to elucidate themechanism by proteins, which accessory and J. Andrew McCammon the University of California–San Diego of StateGeorgia University, John Tainer the Research Scripps of Institute, the world’s fastest supercomputers have revealed. Ivaylo Ivanov of That’s groundbreaking what biochemical simulations on one run of DNA replication, modification, and repair happen inasimilarway. repairs you until are done. composite material to theholes. fill glidegash making along the You and close. Then you applying move andbegin thegash to thefarend of which you open, slide onto atether connecting you to thespace station, Supercomputers SimulatetheMolecularMachines repairing thedamage. Your aclamp, spacesuit with isequipped the spaceinto station, the side of and you have taskedbeen with youmagine are anastronaut. space junk hascut agash A piece of That ReplicateandRepairDNA These sliding clamps and clamp loaders are part of thereplisome—the slidingclampsThese andclamp of loaders are part replication andrepair. slide along the DNA and coordinate strand for needed enzymes another. Again activated by ATP, the clamp closes and is now free to DNA, in inone regionandsingle-stranded which isdouble-stranded Georgia StateUniversity. The clamp-loadingcycle.Image courtesyIvayloIvanov, allowed theresearchers to NAMD, run dynamics amolecular code. supercomputing through of An theINCITEprogram time allocation computing.” system time, inareasonable amountof we access needed to large-scale moresystem than300,000atoms. of To make progress simulating the constituent clamp/clamp assembly—required loader parts—the a system to understand,” explained Ivanov. its “Simulating just afew of and dynamic, interchanging with parts. It’s anincredibly challenging cell division. materialgenetic during replisome complex isvery “The the responsible for machinery thefaithfulduplicationmolecular of PCNA are modifications the in replication determining thefatekey of by excessive replication andunchecked cell growth, such ascancer. thatthisclamp implicated hasbeen indiseasesaccompaniedsurprising Given PCNA of inreplication thedual function andrepair, itisnot astemplatesthat act for makingnew DNA. DNAin which double-stranded prongs gives to single-stranded rise processes by players recruitingcrucial to thereplication fork, astructure DNA the clamp is to of the rolemodification orchestrate of a variety themselves to theclamp to their functions. perform In this capacity as cell cycle checkpoint inhibitors or DNA repair enzymes, attach the cellular response to damage. genetic proteins, A number of such In DNA repair, theslidingclamp asthemaster serves coordinator of them.together thousandsof iteratively four basesto DNA have one add they of until strands strung DNA andmakes sure replication continues uninterrupted. Polymerases replication, strand the of asitprevents from off falling polymerases increases significantly the efficiency polymerases sliding clamps with DNA from strand anexisting DNA template. the of association The a new catalyzemessage. the formation of polymerases called Enzymes carry, base they of so bases are determinethe type what the genetic buildingnucleotides. blocks called made of Nucleotides differ only in In DNA material genetic replication of theclamp slides along astrand Master coordinator taken years. usually run, simulation continuously asingle would running have total computing time. Using machine on which NAMD thekindof is XT4 andXT5 components in2009and2010, takingafew months of workThe consumed more processor than2million hours on theJaguar State University, andMikeMatheson,ORNL. (spheres). ImagecourtesyIvayloIvanov, Georgia open slidingclamp(showningray)andcounterions orange, andred)isdepictedincomplexwitharing- loader (withitssubunitsshowninblue,green,yellow, in bothDNA replicationandrepair. Heretheclamp clamps ontoDNA.Slidingplayvitalroles molecular machinerythatopensandloadssliding OLCF supercomputersilluminatetheworkingsof want to demystify the entire cycle.— clamp-loading Levy by Dawn the researchers, fueled by enthusiasm at the therapeutic prospects, DNAthe cell to upondetection arrest division of damage. Ultimately, alternative clamps such asaPCNA-related protein complex thatsignals In the future Ivanov and hisstudy colleagues mechanisms will of unwanted side effects. produceinhibiting clampmight loading withouteffects therapeutic in processes increased DNA with replication, such ascancer, DNAinterfering with replication in normal human cells. Similarly, DNA indiseasessuch aschickenpox, herpes, and AIDS without to theclamp could loader selectively viral inhibit replication of differences exist, in the ways specifically ATP is used. targeted Drugs life share manyof architectural features, mechanistic significant plants, andanimals. clamp loaders fromAlthough different kingdoms exploit differences among organisms as diverse as viruses, bacteria, Improved thereplisome may understanding of make to itpossible diseases such as cancer.” may allow“That to targets new drug developed be for hyperproliferative different components this complex within machinery,” Ivanov said. just focusingon polymerase,“Instead we caninterfere of many with thereplisome,a better understanding of however, may shiftthespotlight. MostDNA studies of replication have on focused polymerases. Gaining (biomarker) for cancer. Therefore, PCNA and prognostic tool asadiagnostic used hasbeen fork and, ultimately, tumor both progression andtreatment outcome. Annual Report 2010–2011 19 SCIENCE AT THE OLCF 20 SCIENCE AT THE OLCF L Oak Ridge Leadership ComputingFacility from 1toranging 1,000angstroms. Its results because are important on Jaguar and neutron scattering—to resolve lignin’s at scales structure Smith’s team employed ORNL’s two of signature strengths—simulation E. Physical Review journal molecules, individual scale of was publishedJune 15, 2011, inthe theseclumps isrough andfolded,the surface of evento the magnified atom). acarbon of team’s The thanthewidth smaller conclusion, that clumps down ameter, to 1angstrom (equal to a10billionth of or quest for cheaper biofuels by lignin revealing thesurface of structure A team by led ORNL’s Jeremy Smith hastaken asubstantialstep inthe But cellulosic for distill ethanol, thosewho it’s just abother. lighter andmore efficient. fuel future itmay fiberthatmakes become thecarbon trucks cars and that complicatesnext-generationbiofuels Simulation providesaclose-uplookatthemolecule with CellulosicEthanol The Problem keep thestalkandbranches standing andstrong. And inthe thefiberthat keeps of you happy andhealthy. In aplantithelps handy inmany isvery ignin ways. In your itprovides diet much producing food. not take uplandandresources go toward thatwould otherwise does notrequireswitchgrass constant care and attention, anditdoes wetregions both anddry. Unlike biofuel crops traditional such ascorn, the country, itthrives any innearly soil, anditappearsto happy be in renewable substitute for gasoline. It already grows throughout wild ethanol, asasourceSwitchgrass hasmany of theprimary virtues The valueofswitchgrass is soeffective thateven expensive pretreatments failto neutralize it. environments. When theseplantsare inbiofuels, used however, lignin attack hardy helps live of plantssuch range asswitchgrass inawide oxygen, andcarbon. to itsability protect In thewild cellulose from large, isavery hydrogen, complex very itself made upof molecule Lignin sugars that will eventually fermented. be sugars thatwill preventing from enzymes breaking down cellulose into molecules the cellulosic to isamajor impediment ethanol, theproductionlignin of very well.very Ontheone hand, neutron scattering could notreveal the biomass. In fact, said Petridis, the two reinforce methods one another supercomputer simulations andneutron scattering to of thestructure Smith’s project is thefirst to apply both molecular dynamics as much surfaceitwere area asitwould smooth. if surfacea smooth would. In fact, droplet a lignin has about 3½ times than upinthelignin togives farmore caught get enzymes opportunity folded). broccoli. 2.6 is similar to that of valueThe of roughnessThis asurface from andranges to smooth) 3(very 2(very of irregularity 2.6. theroughness or dimension isameasure surface The fractal of about folded, ishighly aggregates dimension of asurface fractal with neutrons andsupercomputing—confirmed lignin thatthe surface of femtosecond, asecond. of two or The 1thousandtrillionth methods— a steps intime atoms—here of andwater—typically of lignin NAMD usesNewton’s asystem motionto law calculate themotionof of methods. 10 to 100angstroms allowed theteam to validate results between resolve from thestructure 100angstroms down to 1. overlap The from (for Not just Another Molecular Jaguar used program) Dynamics to down to 10angstroms. dynamicsA molecular NAMD application called High Flux Isotope Reactor to resolve from structure the lignin 1,000 According to Petridis, theteam neutron used ORNL’s scattering with pretreatment process andfindmore successful enzymes. the aggregates, to be a more design the better effectivewill able they Smith’sof team’s study. better The thatscientistsare able to understand thatareenzymes to added themixture. are aggregates These thefocus onaggregates thecellulose, droplets the thatcancapture upto half washawaypectin thepretreatment, with re-forms thelignin into Even refuses then thelignin to cooperate. While hemicellulose and more accessible to theenzymes. above Fahrenheit 300degrees inadilute to acid make thecellulose hydrolyzed. After it is ground into pieces, small is heated the switchgrass must pretreated be so wellthatground-up switchgrass before itis thegame.enzymes, does thisjob takingthem essentially out Lignin of and thecellulose, aphysical forming barrier. Second, itbinds to passing blocks hydrolysisLignin intwo ways. First, between itgets theenzymes onemolecules by one. as hydrolysis, move in which the glucoseenzymes along and snip off to fermented, be mustbe first they broken ina down process known together. glucose strung molecules sugar of In order for thesesugars another large molecule, hundreds to which ismadethousands upof hemicellulose, theseiscellulose, of andpectin. mostimportant The Switchgrass contains four major components: cellulose, lignin, recalcitrance—resistance degradation.” the plants againstenzymatic of What we’re to do isunderstand thephysical biomass trying basisof computational physicist atORNL, “so itisnoteasyto make thefuels. attack,” enzymatic against explained team member Loukas Petridis, a “Nature has evolved sophisticated a very mechanism to protect plants fermentation. environments makes itstubbornly resistant to breakdown and Yet thehardiness to thatallows thrive switchgrass ininhospitable enzymes and lignin are andlignin likely eachenzymes other andattach. to attract fit on thelignin, might enzymes but ithasnot yet whether the told us the cellulose nor inother theenzymes; words, itcantell uswhere the For example, thisproject andincluded only focused on lignin neither economically cellulosic production, ethanol viable much work remains. While thisresearch step toward isanimportant developing efficient, to study lignocellulose.” was thefindingnew, but these techniques were used for time thefirst droplets thelignin over length scales. of structure various Not only 10 or even 100angstroms. iswhy This we have able been to study the length scales—from 1angstrom or even subangstrom theway all to “On the other hand, molecular dynamics simulations can go to smaller up to 1,000angstroms. can probe length scales, for example, from 10angstromstheway all techniques gives you lignin. amultiscale picture Neutron of scattering donebefore. ithasnotbeen all on lignins of combination The of “When you neutrons atthecombination look andsimulation, of first most powerful supercomputer. not cover scales even on Jaguar, of range the full the United States’ research interesting questions hasalotof to answer.”Leo Williams —by role in biomass recalcitrance shows youbiomass that the field of thedroplets“Not thatplay of thestructure knowing such animportant itdelvesbecause into areas thatcould explored fully notbe before. researchThe promises enlightening, to very be Petridis said, especially computerof time.” course,and thelignin. work Of andrequire thatwould alotof be alot we’d like to atinthefuture look between theenzyme istheinteractions droplets,and thelignin which we didn’t have now. until Onething theenzymes, both molecular-level which we have of description already, “To understand how thishappens, you anatomic-level need or exploited. investigation into why that’s thecaseandhow thesedifferences canbe provided lignins by thislatest researchof opens thedoor to further areenzymes more likely to bind thanothers. to lignin understanding The Petridis noted. For example, showed experiments earlier thatsome production ways, more efficient andless of expensivevariety ina researchThis andsimilarprojects have thepotential to make bioethanol Chemical Simulation). dynamics GROMACS application called (for Groningen Machine for 3.3 million-atom system, are another molecular done being with andcellulose.include lignin both latest The simulations, on a Moving forward, theteam ispursuing even larger simulations that structure at the smallest scales. atthesmallest structure Ontheother hand, simulation could Annual Report 2010–2011 21 SCIENCE AT THE OLCF 22 SCIENCE AT THE OLCF A Oak Ridge Leadership ComputingFacility (CO Ramgen Power Systems dioxide carbon aproduct isdesigning to bury to turbines power machines airandwater.generation of inboth And (GE)ispromoting renewable byElectric working energy on the next fluid possible.dynamics themost to model efficient airplanes General lost. be Company Boeing otherwise The isemploying computational waste heatthatwould them to allowing in of the form capture energy materials into cars’by thermoelectric incorporating exhaust systems, MotorsGeneral (GM)isworking to automobile boost efficiency fuel upward. exponentially, andthe development ispointed curve exponentially never-before-seen challenges—that is HPC. Supercomputing has grown fromquestions arising thefast-paced world—continually presenting andgovernmentcollaboration between industry works to answer the efficiency. to from energy forward inareas security ranging addition,In technologies thatmay nothave otherwise—pushing possible been industry.American create cutting-edge industry helps partnership The researchers thebrightest together some in of isbringing program The speed. Program, them the resources giving to drive innovation at blazing through ORNL’s High-Performance Industrial Computing Partnership two years, theOLCF hasopened itsdoors to companiesAmerican has become success. anessentialingredient for industrial For thelast increasingly complex problems, computing high-performance (HPC) In anincreasingly competitive international economy thatpresents And thecompany isnotalone. and development. to research go from traditional concept of the time to product inhalf OLCFand efficient, with thehelp of andcomputing resources itisable mind, isBMI workingCorporation to make shipping more affordable 2 ) underground, the atmosphere. keeping itout of lower the country’s bybill $5 energy billion. With this goal in sound like abig leap, but applied to large, itmay Class 8trucks 3-mile-per-gallon improvement efficiency in fuel may not Industrial Partnerships Driving Development components. A trailerequippedwithBMICorp.SmartTruck UnderTray With from help theOLCF, thesecompanies are creating of afleet Company to improve the the efficiencycountry’s of biggestgas guzzlers. National Aeronautics andSpace Administration, and theBoeing its sister company SmartTruck researchers andwith from ORNL, the Corporation, inGreenville, based South Carolina, collaborated with Take forcompany transportation, example. services BMI Engineering many isbecoming imperative. industries resources sustain acontinually growing population, innovation in sustainable energy. worldThe isrealizing of theimportance As limited Shaping theshippingindustry fields.own more andmore able to be companies change intheir thegame will technology,changing HPCPartnership theIndustrial Program, andwith next-generation technologies.American agame- called HPChasbeen is a collection of polycarbonate forms strategically placed underneath polycarbonate strategically forms is acollection of runs. fuel-efficiency and closed-track The resulting UnderTray system NASA’s Boeingandcollaboration with with Kennedy Space Center, supercomputer simulation on Jaguar, aerodynamics testing in company’sThe were R&Defforts affair, aneclectic incorporating calculationsareality. impossible making previously With Jaguar, however, BMIsuddenly access had to 224,000cores, simulate the more complex fluid being generated.dynamic models cluster 96processing with cores. With thatsystem BMIwasunable to aerodynamic on early itran when it learned simulations on acomputer companyThe asupercomputer thepower needed of to do thejob, as freight transport. UnderTray system a project iswill the firstchange BMI hopesphase of add-on to a improvepractical efficiency.The company’s SmartTruck In to aneffort reduce tractor-trailer emissions, out BMIset to design you do itright,” saidBMIfounder andCEOMike Henderson. aerodynamic—just atplanes. look You if trucks cando thesamewith aerodynamic. But we’ve we shown canmake big systems very has not muchbeen “There investigation into how to make trucks CO of pounds emit of millions traverse more than150,000milesinayear. Unfortunately, also they consumer75 percent products of intheUnited year; States some every tractor-trailers. Class 8 monsters of fleet These 18-wheel deliver roughly For thelasttwo years, though, on BMIhasfocused America’s massive its hydrogen concept cars. profiles, andhelped Ford Motor Company improveaerodynamics for liner,jet worked NASCAR vehicles’ with Series on theSprint aerodynamic simulations to improve the MD80 commercial the cruising abilities of Aerion to thefirst supersonic Corporation design business jet,ran in the air, or on the interstate. company The the collaborated with tobeen improve aerodynamics andefficiency,racetrack,be it onthe hasalwaysBMI’s industry thetransportation inworking goal with 2 intheprocess. than acar. theproject would to be more make of goal trucks aerodynamic trailer aerodynamic vehicle optimum efficiency,” fuel with hesaid.eventualThe “We hope to ourattention soonturn to creating abrand-new, highly to new vehicles for improved designed efficiencyperformance. and Henderson andSmartTruck hopesthis isonly the beginning canlook for PepsiCo,trucks Frito-Lay, Con-way, andSwift Transportation. 2010’s top 20new products, andBMIhasalready outfitting begun Heavy TruckingDuty UnderTray systems on theproduction linesin18months. Jaguar, of help however, were they able to put thefirst SmartTruck and verificationbefore a product was ready for manufacture. With the Researchers anticipated had itwould take rigorous testing 3½years of intodumped theenvironment.” that’s carbon efficiency we fuel andcutdrag boost theamount of being totrailers make them more aerodynamic,” Henderson said. “By reducing “Our andfortrucks existing firstadd-on was parts goal to design couldthey implement inareasonable andatareasonable time cost. roving Classthan 1.3million 8trucks from coast to coast technology add-ons that could by installed be purchasers on site, the more giving using over points. grid 100million research The team looked for realistic simulateresistance wind and trailer the truck on trucks, half modeling theFUN3Dfluid BMI used dynamics code developed by NASA to by reducing each truck’s yearly intake fuel by 4,500gallons. 2007). It could also cut annual costsfuel by about $13,500 truck per CO of UnderTray couldtons 16.4million technology to lead areduction of Considering travel thatClass 8trucks over year, 130billion milesper in efficiency from today’sadvanced most trailers. over trucks getting the9mile–per-gallon 33percent mark—a increase and on the sides of Class 8 truck trailers. Class 8truck BMIbelieves itiscloseand on thesidesto of 2 in the atmosphere (about half of China’s of intheatmosphere (abouthalf total emissions in magazine citedmagazine theUnderTray system asone of Annual Report 2010–2011 23 SCIENCE AT THE OLCF 24 SCIENCE AT THE OLCF Oak Ridge Leadership ComputingFacility more from information with which to learn. results, thescientific ultimatelymorecommunity giving simulations up thesecodes to achieve faster, more detailed, more physically accurate as inBoeing’s next-generation aircraft. team The isworking to scale codes, advancement supporting inthegreater HPCcommunity aswell flight,affects are they also improving computational fluid dynamics to predict where, try airplanes—and when, and to drag degree what As researchers Boeing simulate reversers—which thrust slow down Jaguar towhere find help thoseimprovements maybe. savings, intoalso translate gigantic Company andtheBoeing isusing Improved efficiency inthe flight commercial could aviation industry for testing. actual simulation,via soonly themostpromising materials are manufactured researchmaterials it hopes further allow itto thermoelectric will design material containing lead, antimony, silver, and tellurium. Ultimately, power source.primary Yang’s group isfocusingon LAST, athermoelectric system,positioning thatwould functions nolongerthevehicle’s drain power systems water such pump, as the electric lights, radio, and global directly into thatheatenergy electricity.of electricity, The inturn, can materials thatmay convertto nanoscalesimulations of perform some waste heat, but ateam by led GMresearcher Jihui Yang isusingJaguar generated by to theenergy carloses60percent itsengine of A typical research, but on scale. asmaller Like BMI, GMaimsto through improvetransportation theefficiency of design. compressor could propulsion appliedalso be in flight gas turbine and officer, technology and chief its rotor-disk of saidthatthe design make compression carbon affordable. Shawn Lawlor, Ramgen’s founder CO of pounds spare theatmosphere of millions company’s supersonic shock compression would technology notonly CO capture andbury focusing on shockwave compression, thatmay technology one day Such Ramgen Power isthecasewith Systems, company a Seattle-based many businesses. Some have achieved groundbreaking results on Jaguar. HPC’s to ability produce results quickly has piqued the interest of Partnerships forprogress that are takingsteps to decrease footprint. their carbon andrecognizing emissionsgreenhouse gas by companies certifying and recently theSmartWay began program, to aneffort reduce freight vehicles. In addition, theEPA thefreight industry with partnered 7to 12percent,increases of the state’s easily meeting requirements for by 5percent. Tests thecurrent UnderTray of system have already shown vehicles to aerodynamic add elements to improve efficiency their fuel In 2009, the California Air Resources requiringClass all Board began 2 to help in the fight against global warming.The against to inthefight help 2 over but time also 8 8 alternative powering ourlives, meansof from make they everything By fossil andworking fuels down slowing toward ourconsumption of improving projectsThese share security. all of thegoal energy American simulating improved-efficiency, low-pressure turbines. experiments.running current Their work focuseson accurately a big picture and beforerefine aspects producing and prototypes Advanced Technologies. allows This researchers to as their design see dock,” saidMike Idelchik, GEGlobal Research’s vice-president of “We are able to optimize an entire from turbine nozzle to exhaust stemming from turbines. wind blades, turbine aerodynamic efficiency butof alsotheassociated noise researchers giving are dataon turbines notonly capable of the wind pure computation. guesswork without Simulations for next-generation researchersallowing to make decisions thatwould design have been at toa time a turbine system-level full simulations, of at portions looking power. Via Jaguar, able GEhasbeen to take simulations from turbine create new technologies, and improve ways upon traditional to generate its Global Research to conceptual arm turn sources energy into realities, GE, thelargest multinationalintheworld, corporations one of isusing on theground. turbines applied to wind more efficient for planesintheair,be can andthesame principle Work on fluid dynamics helps propellerbecome wings blades and the field of computation inthe process.— the field of Gedenk by Eric company working to aid thecountry’s mission andadvance energy consumption andefficiency.The OLCF with any poised is to partner theOLCFcollaborating with to advance theirandimprove fields energy partners companiesThese industrial were thefirst generation of new ones. surface andinclude plansfor fine-tuning their products or developing fossil fuels. Many theseresearch of projects are just scratching the daily commutes to onless turning reliant the television upon imported ORNL. Visualization byMikeMatheson, colored bymachnumber. the complexreflectedstructures cell two-bodysimulationshowing high-resolution resultofabillion- equipment. This imageisthe that offered byconventional at asignificantlylowercostthan will achievecarbonsequestration Jaguar tosimulateequipmentthat Ramgen PowerSystemsisusing Annual Report 2010–2011 25 SCIENCE AT THE OLCF 26 ROAD TO EXASCALE O Oak Ridge Leadership ComputingFacility for crunching numbers.energy-efficient technology familiar AMD Opteron CPUs GPUs, alongside general-purpose amore the overall machine.efficiency of Specifically, feature will the Titan each toallowing doitdoes best, what thus increasing the power and heterogeneous processors, architecture of couple different will types cores, CPUs, upthenumber just essentially of ramping Titan’s Whereas recent have gains achieved been by more adding homogenous increase inthenumber and, likely, processing cores. of thetypes powerful nodesthantoday’s systems. amarked achieved be via will This exascale computing power anarchitecture rests with that utilizes more It’s recently become clear that America’s achieving chance best of isnecessary.strategy Introducing Titan. continue to increase in power and speed, a revolutionary change in quickly reaching their maximum potential, computers and if are to Well, those days are over. Conventional processing architectures are what’sof andhow ahead to best prepare for it. hardwareand giving andsoftware acomfortable designers knowledge 2 to 2½years, electronic devices regularly increasing theperformance of For more than50years computers have roughly every doubled inspeed OLCF’s next-generationsystemtolighttheway to exascale As on any great journey, have thefamiliarwill to left behind. be petaflops), theOLCF be walkinginuncharted territory.will n theroad to exascale systems (i.e., systems able to reach 1,000 Messer. architecture,” saidthe OLCF’s science, director acting of Bronson believe thatwe are takingtheconcrete first step towardsviable exascale a asuccessful, of part competitiveas avital technological future. “We istoAmerica continue to theway lead inHPC, increasingly recognized Nothing easy, be about Titan will if necessary be will but theeffort computing thosethatmaximize power be datalocality. will science. codes The Titan’s take that best advantage of enormous most computationally large-scale computational expensive of aspect the future to need minimize communication between processors, the have to change the way program. they Specifically, theapplications of savings, software andcomputational designers scientistsare going to But it’s aboutthehardware. notall To achieve therequired power efficiency. Titan isthefirst step metric. thislandmark toward achieving Jaguarroughly what twice usesnow, a500-fold increase in power increase indelivered apower performance but within envelope that’s exascale computing ultimateThe is goal of to achieve a thousandfold in theworld. a major expense you’re when themostpowerful dealing with machines in computing power increase only aslight in power with consumption, computational power, and(2)itaccomplishes anexponential increase twofold: increasing way (1)itseems toof themoststraightforward be argumentsThe for Titan’s heterogeneous, multi-core architecture are words, theGPUs marshaled to themostintensive be perform will In Titan, soldiers, be will they and the CPUs the generals. In other GPUs many performing are different capable of atonce. operations conventional CPUs, serially, operations which perform or one-by-one, GPUs of liesin their massively beauty The nature. parallel Unlike time. researchers to achieve faster breakthroughs more via simulations over science, Titan’s isvital, time and theintroduction of GPUs allow will indaysrun on Titan. When itcomes to productive computational means simulations thatwould take weeks or months on Jaguar might afford anenormousincreaseperformance, infloating-point which application accelerators, spawned industry, from game thevideo will theGPUs. changer maingame The istheintroduction of These thesame, look different,” somewhat be will but theguts will saidMesser. a system andefficiently thatmust rapidly deployed. be cabinets “The Jaguar’s existing XT architecture. This heritageis a great advantage for Despite itsrevolutionary promise, Titan isin many ways evolved from Hardware have will else to change.just abouteverything thehardware. with It starts While retain Titan thesameoverall will cooling andpower infrastructure, mostlikely make will That Titan thefastest machine intheworld. faster10 times thanJaguar, groundbreaking performing while science. Titan isto aimof The achieve 20petaflops, just underpeak speed or a terms of storage andfileinput/output. terms of given moment than Jaguar, havethe new system will to keep up in to Spider. producing more capable of be Titan dataatany Because will storage of petabytes andtens bandwidth of second per of gigabytes of system. In order to Titan, play with ball the OLCF hundreds add will machine. Perhaps first and foremost among theseisthe Spider file ecosystem, or theattendant hardware andsoftware surrounding the As Titan isdeployed, some alsobe changes there inthecomputational will HPC. of as communicationthemostexpensive between nodesisone of elements the for need time-consuming “handshaking.” This improvement is key, one processorallowing to another processor share datawith without increases thecomputer’s to ability do one-sided communication, Gemini is Titan’s second major breakthrough. Essentially, Gemini dubbed Gemini. replaced be chipsand theSeastar by will anew interconnect chip removewill one Opteron from each nodeandreplace aGPU, itwith nodeisconnected Cray’severy via custom Seastar router chips. Titan on thecurrent Jaguar XT architecture hastwo Opteron CPUs, and Specifically, differTitan and will as follows:Jaguar Each compute node scientificapplications. various the AMD Opteronsresponsible be for will “command andcontrol” in tocrunchnumbers, their ability rapidly of while calculations because Annual Report 2010–2011 27 ROAD TO EXASCALE 28 ROAD TO EXASCALE Oak Ridge Leadership ComputingFacility andsent to divided are theGPUs further for increased performance. executionof onprocessors; local andvector level, inwhich thethreads each taskfrom into workload isdivided thedistributed discrete threads nodes; threaddomain among anumber level, isdivided of inwhich parallelism:three distributed, levels of in which atask or physical software challenge on Titan primary The is to uncover and exploit Software it runs. matter how powerful Titan becomes, it’s only astheprograms asgood But thehardware thechallenge to isjust come.one dimension of No future of, once place again thenationasleader andwill in, HPC. tribulations, mostlikely but intheend themachine represent will the taken together. As Titan is deployed and no doubt trials there be will changes,These incremental, though represent leapwhen anenormous computing challenges. will enable Titan’s userstoanswernext-generation Gemini interconnect’sexcellentinternodescalability, components. This computeunit,combinedwiththe run applicationswitheitherscalaroraccelerator power-efficiency ofacceleration,andflexibilityto result isahybridunitwiththeintranodescalability, NVIDIA’s Tesla X2090many-coreaccelerator. The 16-core OpteronTM6200Seriesprocessorand hybrid computing. The nodeabovecombines AMD’s Titan’s architecturerepresentsthelatestinadaptive adding up to plenty of physics points. up toadding of grid at individual plenty On current system,the Earth e.g., theatmosphere, theocean, andtheSun’s radiation, each contains numerous components various to of variables describe points,At these grid each however, of there is a lot going on; for instance, cells, into eachcell grid on living processor. anindividual the Earth with heavily parallelism, on distributed thesurface dividing essentially of Take climate simulation, for instance. complex These exercises rely theGPUsthe real power of andspeed isfound. the GPUsthe use of through effective vector-level programming, where parallelism, i.e., the vector. Ultimately, Titan’s potential lies in optimizing mostimmediateThe software challenges reside inthethird layer of environment. and Pthreads, standards forthreads creating inashared parallel memory like processing methods shared OpenMP memory via in the form of Further, Jaguar allows ways to parallelism expose thethread level of work over processors, lotsof Jaguar’s one of assets. best efficiently.These same writers alsoallowtools applicationtodistribute Arrays allow applications to various communicate between processors parallelism. Tools like theMessage Passing Interface (MPI) andGlobal Currently, the software on Jaguar distributed does a great job of avoid this model. CUDA isalmostthere, saidMesser, hovering between aslow-levelis known programming, and the OLCF isdoing to its best their architecture.of Programming thatdemands hardware knowledge it could before awhile be thecommunityasoundunderstanding absorbs knowledge, thiskindof possess GPUs andbecause are relatively new theGPUs of structure andthesurrounding hardware. Few programmers CUDAeffective useof requires thephysical anin-depth knowledge of codes that have with particularly evolved over years. Unfortunately, plug CUDA into their existing codes is an enormous convenience, don’t they That have to anentirely learn new language andcansimply and scientistsalike. extension, andnot anew language, thingfor good programmers isavery extension from GPU-maker NVIDIA. that Thefact it’s simply a language discovering ways to do just that. current The champ is CUDA, alanguage After anderror, much trial thesupercomputing community isnow said. availablesimulations layer whereparallelism isexploited,” every of he “Ultimately we hope theapplications on full-scale Titan be will long on wallclock times Jaguar days GPUs. to with ahandfulof Messer, adding that researchers at ORNL are already reducing months- reach ittook because months to out been of previously had run, said package on Jaguarresearcher chemistry that useanatmospheric might GPUs means even more optimized simulations. For example, a On Titan, vector-like availability the additional of the via parallelism more sophisticated approximations, resulting in greater overall accuracy. thesimulations, theuse themof of upandallowing speeding fidelity each calculation. Tapping this unrealized can increase the parallelism pointyou thatateachhave grid fact for multiple layers parallelism of However, tackled atthethread canalsobe they level by exploiting the point.grid systems thesecalculationsare usually done serially, one-by-one, ateach accurate aspossible. Finding errors isessentialto that simulation ensuring results are as supercomputers programs, mission-critical it’s with atitsmostcomplex. Software is never simple, and whenit comes to world-class debuggers andperformance analysis tools. When for success: itcomes critical be to tools Titan, will of two types Tools That’s where thenext challenge lies: tools. to know broke what andexactly where itbroke. complex with anew platform be Titan hardware, will difficult be itwill the software to experiments. used numerical canbe perform Because problemsto aconsistent thatmust andbugs solved be of barrage before becoming more efficient and realistic.This constant development leads leadership-class like platforms Jaguar are necessarily always changing, development, saidMesser. on cutting-edgeapplications The running scientific software is that of it is A hallmark constantly under their maximum potential on Titan. may GPUarchitecture nolonger to theminutiae need of learn to reach is an enormous breakthrough for one simple reason: Programmers performance numbers that are equivalent to hand-coded CUDA. This internal boilers, and combustionthe efficiency engines of has gotten code developed National atSandia (SNL)thatisimproving Laboratories As atestament to thiswork, Cray’s compiler on S3D, acombustion ontime Titan. decisions and researchershelp get the most from their codes and their allowcompiler thecompiler technologies thatwill to make better as possible. Currently, Cray, PGI, andCAPSare working on new language extensionpartnership involves assmall makingthispenalty are notoptimaldecision makers. thework on thenew of brunt The performance fromthe highest an application. Compilers, out, it turns the compiler ismakingthedecisions, theuser isn’t necessarily guaranteed course,Of directives present challenges aswell, saidMesser. Because across HPC benefitting thefield. andinthelong platforms run of to their codes run on avariety GPUs them, ignored on be a machine without will thus users allowing computer language such asbinary), are done right. In other words, the language suchinto aprogramming another asFortran that transforms thedirectives,if inthecompiler or instructions (acomputer program much low-level programming, is that the work code on any will platform ahigher-level great benefit The extension,of besides not as requiring how to best accomplish runtime. thetaskduring computer theuser wantsto what happen andthecomputer decide will hardware knowledge. In essence, thisnew extension tell the will one that operates atlevel, a higher i.e., one that doesn’t require as much likelihood in all joint another be product language extension will but atacompiler-basedlooking approach software. to thiskindof The (PGI), andCAPS, multi-core provider aleading of software tools, are theOLCF’sthat several of partners, including Cray, Portland The Group “We doing this,” ahigher-level need of description saidMesser, adding and low-level. high-level, amachine’s inwhich lessknowledge of hardware isrequired, future,” saidOLCF Director Buddy Bland. contribute immensely to science and America’s competitive technological Titan, theOLCF insimulation, on embark anew era will one sure to “Titan isthefirst step towards ensuring America’s exascale future. With and implementing theOLCF’s future planandphilosophy. program absolutely be performance to analysis crucial will realizing its potential Titan istoIf themonster be machine envisioned, and debugging heterogeneous architecture. optimize their applications themostperformance andget from Titan’s the applications inprogress, Vampir programmers andresearchers lets code andallowing researchersidentify inefficientof sections view to toused measure performance anddiscover bottlenecks. By to helping Technical University Dresden on its Vampir package, tool amodeling On theperformance analysis front, theOLCF isalsoworking with partnership.Allinea is amonumental task, head-on by andone met thatisbeing theOLCF/ out howfiguring a to debuggerprogram accommodateto their presence programs, to include GPU debugging. Because GPUs are so new, product, MPI aleadingtool MP and Open for parallel debugging OLCFThe isworking closely with toAllinea extend its DDTdebugging • • codes fromcritical domain sciences: various were considered, but thislistwaseventually six pared down to of aset from current andformer INCITEawardees. Initially 50applications OLCF aswellfromApplication Requirements responses and Strategy from research done for the2009OLCF for Exascale: Preparing report to itsmixed take advantage full architecture. of listwasgleaned This were Titan—the to thevanguards be first of codes that adapted be would In 2009theOLCF candidate applications compiling that began alistof utilizingitsinnovativecapable of computing environment. architecture applications alone isnoHPCgame-changer without resource long before thehardware waspurchased. Titan’s novel 2011, itsnext but leadership theOLCF of preparing for thearrival began Titan are firstThe thousandnodes of scheduled inlatefor installation Application Readiness combined in 2008by ORNLto Markus calculate Eisenbach of temperatures,all notjust absolute zero. two The were methods method, which guides calculationsto explore system behavior at for electron behavior. second The istheMonte Carlo Wang-Landau to solve equation,theory theDirac wave arelativistic equation the lowest temperature possible functional by density applying scattered electrons at of scattering, thejourneys which describes uses two methods. The first is locally self-consistent multiple motors. found inelectric magnets disks andthepermanent It materials—such as those foundin magnetic in computer hard WL-LSMS calculates between electrons theinteractions andatoms engine.stabilization diesel inadirect-injection physics therepresentation fine-grained some of relevant of to lifted small autoigniting hydrocarbonof flames, jet for allowing Chen Jaguar used to create theworld’s first fully resolved simulation simulation code combustion.that models In 2009, a team byled S3D, developed by Jacqueline SNL, Chen of isadirect numerical Annual Report 2010–2011 29 ROAD TO EXASCALE 30 ROAD TO EXASCALE Oak Ridge Leadership ComputingFacility • • • • Preparing forExascale:Six CriticalCodes large-scale CO Stability andviabilityof PFLOTRAN systems. in nanoscalematerialsand statistics, andfluctuations Role ofmaterialdisorder, WL-LSMS groundwater transport. tion; predictivecontainment atomic, meso, or continuum scales. atoms or, more generically, simulator particle at the as a parallel a classical dynamics molecular code to used thatcanbe model Simulator, researchers was developed by agroup of atSNL. It is LAMMPS, theLarge-scale Atomic/Molecular Massively Parallel regularly employed by theclimate community. modeling tonecessary couple this core to physics packages—such as CAM— for HOMMEto a useful tool be scientists, for atmospheric itis equationsfluid andthermodynamic on resolved scales. In order Environment, dynamical core, isanatmospheric which solves and climate research. HOMME, the HighOrder Method Modeling Atmosphere Model) atmosphere is a global formodel weather conditions.simulate atmospheric global CAM(Community Cam-SE represents two thatwork models inconjunction to multi-step approaches nuclear to high-fidelity reactor simulations. Developed by ORNL’s Tom Evans, Denovo allows consistent fully National Laboratory, simulates groundwater contamination flows. PFLOTRAN, developed by Peter C. Los LichtnerAlamos of on Jaguar. barrier the petascale parameters. combined The thefirst codecodes wasoneto of break materials atafinite withoutadjustable magnetic temperature 2 sequestra-

burn moreefficiently? eration ofdiesel/biofuels How willthenextgen S3D - die on thevine,” said Messer, adding, “we to had ensure that our changes “We to had make sure we made changes to thecodes thatwon’t just teams theworld over platforms. on various mostimportant,and perhaps theseapplications are by used research Titan isinproduction—any changes made must flexible. be Second, arethey under constant developmentcontinue to andwill after be adaptations. First, these applications because are current INCITE codes, acknowledged intended some fundamental principles to optimize their Before on thesesixcodes, work began thedevelopment teams Guiding principles GPUs. each accelerated codebeing thatare via of capable of identify parts domains, problema single from theteams varying have worked to the floor, researchers doing canstart real science,” said Messer. Using isto maingoal thesesixcodes get “The ready Titan sothatwhen hits inTitan.used individuals from NVIDIA, the GPUs supplying be which will to be representatives from hardware developer Cray; andone or more from people theapplications’of development teams; one or more Scientific Computing with the Groupcode;who isfamiliar a number to work on each. teams These included one liaison from theOLCF’s Once theapplications were chosen, were experts assembled teams of tion scenarios. adaptation and-mitiga specific climate-change- Answers questionsabout CAM-SE - from theland’ssurface. concentration originating Denovo of theatmosphericCO Simulated timeevolution LAMMPS tions. and technologyapplica variety ofnuclearenergy that canbeusedina transport calculations High-fidelity radiation 2 -

take them. size, or they’ve where simply thecode reached base can thelimit of their codespredictions anddon’t with have to increase theproblem solved. Users have reached thepointwhere canmake they quantitative theproblem increasingto associatedthesize to with thelabel of be that research teams have weak scaling,” out of run saidMesser, referring applications attheexascale. “What we’ve discovered is atthepetascale twoThese discoveries are thekey to enabling strong scalingof languages suchprogramming asC, C++, andFortran. GPU to effectively CUDA, code with of avariety which works with requiresThis thedevelopers to know alotaboutthehardware the of Second, the teams have how learned to marshal data for the GPUs. can be “fed” to theseravenous GPUs. examined thecodes lineby lineto datathat identify large chunks of but require dataatonce. huge amountsof Developers have carefully voracious can’t animals—they little sated be with “bites” information, of work for theteams. GPUs have to fedcorrectly, information be like so that it can used be efficiently), that a fact is creating the most difficult (thewaystructures informationisstored andorganized inacomputer First, for application adaptation GPUarchitectures changes data developmentThe teams have from plenty their work learned thus far. work done by theseteams won’t time. survive compiled thatcanbe repository on any architecture; otherwise, the Messer. It’s essentialfor developers to able to be a check acode out of on Titanruns andanother version architectures,” for traditional said ‘production trunk’ thesecodes—there all won’t of one be version that “We’ve made changes that we’re sure are going to remain the within systems CPU-structured sincetraditional adapted. being which hasexperienced atwofold increase on inperformance speed performance on non-GPU architectures. example isDenovo,A prime only made thecodes on hybrid functional systems, helped but actually platforms.” teams The discovered not their modifications thatsome of at least the very do are whilethey no harm runningon other, non-GPU “Titan isthefirststeptowardsensuring America’s exascalefuture.With Titan, immensely toscienceand America’s in simulation,onesuretocontribute the OLCFwillembarkonanewera competitive technologicalfuture,” —OLCF DirectorBuddyBland.

America cancontinueAmerica to solve theworld’s greatest scientific challenges its reputation astheworld leader insupercomputing. With Titan, be TitanAmerica’s will first step toward will cement the exascale and unthinkable just a few years ago. Simply put, standing up and operating able tobe exploit theGPUs to simulation achieve virtually alevel of Ultimately, saidMesser, on Titan theapplications running will of all compilers directives-based for usewith codes. compilers Cray and also with directly to Cray, adapt PGI, and CAPS the OLCF’s Application Performance Tools Group on tools and skip thedirectives. OLCF code development teams are working with GPUs, have thecode will commands thattell themachine to simply GPUs.the code via directives acode with on If amachine runs without order thecompiler to figure out how to execute of complexportions for code adaptation. placed be Commands in applications that will code development work toward teams will adirectives-based approach As HPCadopts hybrid architectures andmoves closer to theexascale, Messer explained. “We this attheexascale.” expect time,” sophisticated simulations amountof inashorter number of one simulationby GregoryScottJones atatime.— andCaitlinRockett “It’s now strong scaling—producing become aquestion of agreater Annual Report 2010–2011 31 ROAD TO EXASCALE 32 INSIDE THE OLCF I Oak Ridge Leadership ComputingFacility The JaguarThe XT5 spreads out over 5,000square feet, 200cabinets with current 2.33petaflops. incarnation, capable of Jaguar 119 teraflops has grown fromto its of speed peak aninitial Jaguar revsup space. expanses of problems, from matter the smallestbuilding to blocks the of vast computing machine intheworld, impossible tackling previously supercomputing, Jaguar remains themostproductive scientific Today, astheOLCF to looks new horizons intheworld of Cray JaguarXT5supercomputer Linux modified The platform removes processes unnecessary from Cray’sEach node runs theSuSE version system. Linux operating of directed andinput/output (I/O)nodes. by 256service Jaguar second, per travel at240gigabytes inandout of databeing with theentire memory. systemgiving 300terabytes of Information can housing 18,688compute memory, nodes. of Each nodehas16gigabytes became oneofthemostpowerfulsupercomputersinworld. for science. Jaguar,The machine,dubbed sprang upandquickly n 2008, the OLCF started building the world’s most powerful computer RESOURCES analyze, andmanage their projects. availability additional This meantthatusers more had to time compute, Jaguarscheduled wasexceeded availability of by 10percent in2010. And notonly isJaguar efficient, itisalso reliable. In fact, the expected kilowattmillion hours annually. heat astheairenters andexits. efficient This systemcooling saves 2.5 ECOphlex conditions air as it is drawn into the machine removing while high-temperature inautomobile refrigerantused airconditioners, Cray theECOphlex cooling designed technology. Using R-134a, a squareper foot, necessitating aliquid cooling system to dissipate heat. theseprocesses give of All Jaguar 1,750watts power apeak of density performance tools. I/O, mathematicallibraries, compilers, debuggers, andother connects Jaguar’s system services, networking software, communications, each nodeandminimizes interruptions. Linux This environment PAVE THEWAY user survey conducteduser survey to users’ gauge experiences, below 5, 3.5. theOLCFfacility noone ranking 4.3out of with averaged of agrade User were queries answered inlessthanthree days thetime, 87percent anaverage 31minutes. of with response of time In theOLCF annual 2010OLCFUserCommunitybySponsor • • • “At thehuman(support)andtechnical(software,admin)level,OLCFisafirst-rateinstitution.” phone conferencecallsandon-sitemeetings.” ments providedbysubjectmatterexpertsandtheproactiveapproachofreachingouttousersviatele be verycustomerorientedandworkshardtomaximizetheexperience.Iappreciatecom “Project staff experienceswhencontactingOLCFsupporthavebeenverypositive.Supportstaff seemsto tion withmultiplesupercomputercenters.” “The helpservicesprovidedbytheOLCFarebestIhaveeverexperiencedinoveradecadeofinterac User feedback: 4.28 in2009,and4.312010. scale of1 =Very Dissatisfiedto5= Very Satisfied. Theresultswere3.7in2006,4.12007,4.22008, Overall satisfactionwiththeOLCFhasseenayear-to-yearincreaseinmeanofresponseson increase from4.28in2009. OLCF overall.Onthescaleof1=Very Dissatisfiedto5= Very Satisfied,themeanratingwas4.31,aslight Overall ratingsfortheOLCFwerepositive,as90%reportedbeing“satisfied”or“verysatisfied”with 2010 OLCFUserSurveyHighlights OLCF computing systems. improve their applications on current andemerging theperformance of toolsperformance-characterization research that help teams access and provides tools, new modeling users with languages, middlewear, and and cybersecurity. Finally, the Application Performance Tools Group 24 hours a day, 7 days a week, providing administration, configuration, GroupComputing Operations monitors systems thehigh-performance systems, and archival storage infrastructure. The High Performance GroupIntegration ensures thatusers have up-to-date networks, file OLCF systems access, policies, andprocedures. Technology The technical to support research teams, documentationand generating on for OLCFsupport users, creating accounts for news users, providing UserThe Assistance andOutreach Group represents thefront lineof and workflow, as addressas well any computational issues that arise. research. andoptimize groups code design Liaisons their assigned help fromof theScientific Computing with thefield Groupwho isfamiliar Each research on substantialtime Jaguar teamaliaison with isassigned once have runs completed. been optimization before andanalysis to simulation runs data visualization throughout help specialized their allocation—from codewith comprises fivegroups that provide high-performance computer users research themostfrom teams glean their simulations. OLCF The resources, professionals skilled a network but also of with help who provides world-class thisdiverse users not only computational with of set Over 1,000 users from currently 58 countries use Jaguar. OLCF The The OLCFusercommunity Annual Report 2010–2011 - - - 33 INSIDE THE OLCF 34 INSIDE THE OLCF (approximately 45,000cores theresource on Jaguar) of for single machine for production simulations,usingatleast20percent typically allocation, aproject must aleadership-class demonstrate effective useof leadership-classimpossible without systems. To secure an INCITE high-impact, research grand-challenge be thatwould otherwise approximately theavailable 60percent time. of facilitates program The Most projects receive on time Jaguar through INCITE, which distributes Computing Challenge (ALCC), andDirector’s Discretionary. Office of Advanced Scientific Computing Research Leadership Access to Jaguar isavailable through three programs: INCITE, theDOE Oak Ridge Leadership ComputingFacility 2010 Allocation HoursonJaguarXT5 2010 INCITE Allocation HoursonJaguarXT5 byDiscipline time. around the world are to eligible apply for Director’s Discretionary hours. theINCITEand As with ALCC programs, researchers from awards processor 1million on theorderDiscretionary areof typically theleadership similararchitecture. machines or asystemof of Director’s proposals must include out on datafromone benchmarking carried out benchmarking,can alsorequest since to time INCITE carry work, inpreparation for afuture INCITEsubmittal. typically Researchers order leadership computing with to andother development started get provides a way for teams to request access to leadership systems in throughis allocated theDirector’s program. Discretionary program This The finalpercent 10 time available of on the leadership-class machines processor hours on Jaguar in2011. leadership-machine users. The ALCC awarded program over 368 million also awarded to projects thatextend possible thecommunity of alternativesclean energy or climate simulation. ALCC are allocations directly related to thedepartment’s mission inareas energy such as emphasis on high-risk, simulations high-payoff infields particular ontime Jaguar. ALCC interest special to projects seeks of DOE, with The ALCC, managed by DOE, allocates up to the 30 availablepercent of are thelargest awards ever made under theINCITEprogram. project and include one allocation topping 110 million processor hours, in2011,Allocations which average processor 27million hours per an average project receives more processor than20million hours. technological breakthroughs. Awards from project to vary project, but how theproposed work may enable scientificdiscovery or facilitate factor in award decisions, soitisessentialfor researchers to express potential for impact. potential The isthepredominant for impact projects for the team’s both computational readiness andthe research’s production runs. INCITEissues annual for calls proposals andexamines was essential to achieving petaflop petaflop was essentialtocomputing. achieving incorporate InfiniBand networking on a Cray XT system. This upgrade Lustre filesystem, andstorage.The OLCF wasthefirst institution to many platforms, cluster including the visualization for analysis, the and enables users to move data throughout large amounts the center’s of standard networking for systemThe high-performance istheindustry which nodesare individual connected to multiple network switches. is aswitched-fabric, communication high-performance system in In 2007theOLCF adopted InfiniBand fabric center-wide. InfiniBand 18,000terabytesnearly in2010. on thecenter’s resources—HPSS grew from 1,000terabytes in2006to resourcestape to theever-growing handle dataproduced mountain of and redundancy. members are constantly Staff more adding disk and toserver theplatform andswitched Linux, improving performance to transferred tapes.data isgradually OLCF The haschanged theHPSS System dataonto datamovers, disksby (HPSS)writes high-speed the storage systems in the world. After the High Performance Storage their datasecurely,of theOLCF thelargest archival maintainsone of datageneratedresearch. during collections of To users help keep track OLCF users require storage for high-speed and retrieval the large Data management Petabytes Total Amount ofDataStored(inpetabytes) HPSS: 2006-2010

to become abackbone network for theESNet network by next year. ResearchEnergy Scientific Computing Center.This network poised is namely the OLCF, Argonne Leadership Computing Facility, and National Science network for Office connectingprototype centers, DOE of funded by the RecoveryAmerican andReinvestment Act—to build a centerthe The with Advanced haspartnered Networking Initiative— accesshigh-speed to research other DOE and other facilities networks. community’sDOE internet network. primary This gives the OLCF centerThe usestwo 10-gigabit-per-second connectors to ESNet, the second. per 240gigabytes of a blazing-fast bandwidth development cluster, andthecenter’s dedicated servers—at GridFTP Jaguar, of the XT5 partition cluster, theLENSvisualization theSmoky 2005, andnow Spider filesystem isthemainoperational connecting computing into platforms file aunified system. in Thebegan project technological integration. Spider organizes datafrom themultiple Lustre-basedThe filesystem, Spider, theOLCF’s liesatthe center of institutions. around thecenter, but alsobetween theOLCF andother research delivery isimperative. notonly isneeded datatransfer High-speed are spread around the world, meaning that accurate, data high-speed from oneto location another quickly and efficiently. Many OLCF teams AMD Opteron processors andtwo NVIDIA8800GTXGPUs. dedicatedown Lustre filesystem, andeach nodehousestwo dual-core pixels—a pixels. 35million total of 15-node The everest cluster has its 27-projectorThe Powerwall displays images at11,520-by-3,072 fostering scientificdiscovery. istoIts channel purpose datagenerated on Jaguar results, into visual memory,of andone NVIDIATesla memory. of 4gigabytes GPUwith GeForce processing 768megabytes 8800GTXgraphics unit(GPU)with memory,AMD Opteron of processors 64gigabytes with one NVIDIA 32-node cluster, each node containing with four quad-core, 2.3 gigahertz EVEREST andtheLENScluster thefacility. supporting LENSisa Researchers usingOLCF automatically facilities receive access to simulations. of view at theOLCF. gives facility researchers visualization The asharp, detailed screen, dataanalysis toolsvisualization themostimportant isone of and Technology (EVEREST), its30-foot-wide, with 10-foot-tall Exploratory The Visualization Environment for Research inScience Visualization Data analysis Another major challenge to isthe need move data large amountsof Annual Report 2010–2011 35 INSIDE THE OLCF 36 INSIDE THE OLCF T Oak Ridge Leadership ComputingFacility automatically retrieve theaverage value, minimum, maximum, and datainthe output.for self-describing For example, users can file or M-files,transparently. Version 1.2 also features support further switch from on P-processors running to P-files—or andwriting one leadership-class systems. And now, users on run large who systems can itintoand aggregates larger pieces for maximum performance on ADIOS alsofeatures datato subfiles acustom thatwrites I/Omethod (API) directly to time. interactively run during new construct variables For instance, users can now use the application programming interface supercomputing leading of platforms. improvements doubtless aidresearchers thatwill advantage intakingfull and combustion. new version The features some interesting researchersADIOS hashelped make infusion, huge strides astrophysics, Say Helloto on thespot. thesamedatasimultaneously cansee andcollaborate geographically expertise. Inareas other of words, anddegrees researchers separated it allows collaboration between simulation scientists from different computational andanalysis elements. insteadvariables of And finally, low-level technical detailsfrom theusers, theusers to allowing ponder a simulation from any computer on any browser. Second, ithides the tool availablesingle thatprovides access into thestatus of andinsight First, theweb. thesimulation via itallows monitoring of It istheonly packageThe offers three major benefits for computational scientists. leadership systems continue data. to generate of petabytes such as file systems and directories, an increasingly complex area as “science” HPC simulated, being of theintricacies thanlearn rather Ultimately, theDashboard aboutthe allows thescientiststo worry displaying data just a minute or two the simulations behind themselves. “window” into simulations shows running results occur, almostasthey University Utah, of North Carolina State University, andORNL, this Developed by theScientific Computing and InstituteImaging atthe time. scientiststoallowing monitor and analyze their simulations inreal- Simulation Monitoring (eSiMon) Dashboard version 1.0to thepublic, Researchers from theOLCF have recently released theElectronic Computational scientistshave anew weapon attheir disposal. eSiMon simulationtool on thewall. software sousers may their datato see users helping their get images Powerwall. team The visualization assistsusers inanything from writing Chromium with toin partnership deliver andDMXto OpenGL the VisIt, EnSight, POV-Ray, AVS/Express, ParaView, andIDL, which work software, are of OLCF versed including staff inavariety visualization Developed by a partnership led byDeveloped led by theOLCF’s apartnership Scott Klasky, computational science’s one of of mosteffective I/Otools. he year 2010saw thereleaseADIOS 1.2, of thelatest incarnation source I/Operformance library. said theOLCF’s Scott Klasky, developer aleading ADIOS, of anopen- running simulation,” monitoring their automatically thebenefit get of interfaceprogramming into an sothat ADIOS method ADIOS users “We are currently the eSiMon working on integrating application tools. into external software anduser-customized analysis andvisualization and raw dataonto machines. local Future versions include hooks will processedpan capabilities, of anddownloading datalineage viewing, annotate movies. Otherfeatures include zoom/ vector with graphics Researchers canalsotake electronic notes on thesimulation aswell andamore inclusiveportal anddiverse user base. thesystem createscomplexity of alighterandmore accessible web images andvideo.and retrieve publication-quality Hiding the will be conducted.— be by GregoryScottJoneswill embody amajor innovationthey intheway computational science advances significant toward more efficient simulations. Takentogether, Taken separately, ADIOS’s of all improvements individual represent changes theuser smooth experience.” maintaining100percentwhile backward compatibility. Some additional enhancements application thatcanuse broaden theclassesADIOS of community,” saidteam member Jay Lofstead. “The AMR-focused long-standing requests from asmall, but vocal,ouruser of part the API to introductionThe calls of replaceaddresses theXMLfile focus for this release“The is broader compatibility and user convenience. evenallowing faster I/O. finally, version 1.2 features methods, some new asynchronous transport computationalstandard for arrays all at negligible deviation cost. And Furthermore, easyclicking researchers anddragging, via cangenerate The eSiMonDashboard. 1.2 with the memory per node per increasing from thememory 16towith 32gigabytes. Jaguar’s Cray’s newest XK6nodes. increase from Each nodewill 12to 16cores, this year, OLCF plans to replace the node boards in each Jaguar of with the Top500 the world’s Listof mostpowerful supercomputers. Late Jaguar, theCray XT5 thatcurrently holdsthenumber three on spot Titan’s theOLCF’s with begins actually story current leadership machine, Titan’s firststeps second. calculationsper quadrillion Titan,called 10–20 amachine expected to reach of speed apeak data for inOLCF’s the first landmark road to the exascale—a Cray EX6 the new 32-cabinetfilesystem, store will which up petabytes of to 10 convert them to smaller, sustain usable power will levels. electricity The remote units, distribution which take voltage high andcurrent and power reconfigured be unitsand distribution will of usingaseries According to Jim Rogers, OLCF operations, director of this switchboard volt amperes), or watts. 2million 2.5mVA from thatcarries atransformer redirects electricity (million switchboard that thismachine removalThe frees upanelectrical of XT4 to theOLCF belonging wasdecommissioned on March 1, 2011. ORNL’ssecond floor of Computational Sciences Building. An 84-cabinet old hardware to theCray go; thisisthecasewith XT4 located on the data. innew hardware of Bringing petabytes often meansit’s for time to compute onesecond calculationsper quintillion andstore tens of power bigger, faster machines, machines thatare especially expected With computing, themostfundamental question ishow to efficiently computational science possible. thatmake needs infrastructure HPC hasitsown leadership-class as roads, water supplies, andpowerenable human civilization, grids resourceful toolsthechallenges arethatmust met. be just some of Just hardware,specialized developing innovative software, and designing roadThe to exascale obstacles—engineering computing with isbeset Providing power —by CaitlinRockett system isprojectedupgraded to accessible be to users all by 2013. speed. After staggered and acceptance upgrades tests, Titan’s fully 20 petaflops,which isfive to timesfasterten than Jaguar’s peak current Titan’s isexpected to performance speed peak clock inbetween 10and NVIDIA application accelerators to the1,000-nodesystem inlate 2012. Titan’s to build will on upgrades Jaguar configuration initial adding by CPUstraditional andGPUs. Titan—the OLCF’s first architecturehybrid system, both exploiting for users asatestto serve partition optimize their applications forwill a more stable, fault-tolerant network. Ultimately this1,000-nodesystem computation, double available for larger memory problems, and present Jaguar’s providewill new configuration more nodehours for operations. providing for one-sided communications andatomic memory Gemini network, increasing bandwidth, decreasing latency, and communicate one another, with replaced be Cray’s will with latest Seastar interconnect, thesystem’s which allows each of nodes to Annual Report 2010–2011

37 INSIDE THE OLCF 38 INSIDE THE OLCF T Operations Jim RogersOperations manages day-to-day computing systems. And Director of Messer guides theresearch teams thatusethe Boudwin. Acting Science Director Bronson of are aided by Director Deputy Kathlyn theOLCF of supercomputers.upgrades They and installation Buddy Bland supervises thecenter, of vision Project while Director team. Director Jim Hack manages theoverall processThe isguided by ourmanagement future research. current both and to of the quality guarantee continue unabated, theOLCF works tirelessly toneeded ensureadvances thatscientific unique resources to thelong-term planning involved unique simulations inrunning on breakthroughs. From theday-to-day challenges OLCF staffensurethatoursystemsareindeedhigh-performance Oak Ridge Leadership ComputingFacility DESERVE WORLD-CLASS PEOPLE WORLD-CLASS SYSTEMS potential and deliver regular scientific world-class systems reach their full he OLCF exists to ensure thatour the Scientific Computing Group, or SciComp. closest working liaisons in relationship with Our scientific researchers may have their and infrastructure. andplanningforoperations future systems at [email protected]. leader Ricky A. Kendall for more information and streamline their work. Contact group flow themto help their insights deepen experts researchersvisualization and work- find will on specific challenges. SciCompwhere isalso choose instead to consult itsliaison only with team member, asafull-fledged or itmay project may askitsliaison to join theresearch optimizing itfor theOLCF systems. A large code areindesigning and they alsoexperts analysis,numerical computer science—but chemistry, physics, astrophysics, mathematics, liaisonsThe areintheir fields— experts articles inoutside publications.articles create They website, regular newsletters andreports, and andaccomplishmentsgoals through theOLCF communicateOur outreach staff thefacility’s computing resources.use of toeducation facilitate opportunities efficient User Assistance and alsooffers training community through weekly email notifications. and relates to information theuser important issues. group The alsoprovides documentation codes,compiling of andaccess andfilesystem include debugging, optimization, and performance computing challenges that high- of range themhelping through a wide provide day-to-day to researchers, support public. Ouruser assistance professionals accomplishments to our sponsors and the and communicates OLCF and goals provides professional consultation to ourusers UserThe Assistance andOutreach Group

DESERVE WORLD-CLASS PEOPLE WORLD-CLASS SYSTEMS

Group members anticipate problems before tools todiagnostic continually monitor them. anduses are and upgraded they when installed and cybersecurity. group The alsotests systems administration, configuration management, days ayear and are responsible for systems 24hours aday, seven days aweek, 365 supercomputers. Group members monitor Jaguarsystems aswellwith and other OLCF running, theinfrastructure working with Group keeps theOLCF leadership systems HighPerformanceThe Computing Operations group leader, at [email protected]. and Outreach Group, contact Ashley D. Barker, concerninginformation theUser Assistance display attheOLCF andother venues. For slideshows, andother materials for public facilities. alsocreate They posters, videos, scientific research and on OLCF systems and on focused andother articles highlights TechInt works to develop tools such as computing resources continue to scale up, OLCF’s network. area-wide As OLCF measures for and implement the cybersecurity datatransfer working to increase of thespeed OLCF’s HPSSstorage system andisconstantly areas they adopted. TechInt co-developed the technologies andtools into theinfrastructure toprogramming new seamlessly integrate technologiesandprovidesemerging system systems. group The researches andevaluates intoinfrastructure the OLCF computing networks, filesystems, andarchival storage is responsible the for and integrating updating TechnologyThe Group Integration (TechInt) contact Ann Baker at [email protected]. policy.cybersecurity For more information, also ensures systems that all conform to ORNL components that are near failure. group The those problems arise, identify andthey advantage of theleadership-class systems.advantage of analysis tools thatallow users to take full compilers, debuggers, andperformance- group leader, [email protected]. more information, contact Rich Graham, large-scale systems,very such as Jaguar. For research applications are when they on run for on issues thatarise focuses primarily tools.performance-characterization group The modeling tools, languages, middleware, and vendorscontacts with new for thepurchase of computing systems. group The also manages applications on OLCF current andemerging and improve their theperformance of software tools researchers thathelp of access researches, tracks, andpurchases range awide The Application Performance Tools Group more [email protected]. information Contact Galen Shipman, group leader, for Annual Report 2010–2011 39 INSIDE THE OLCF 40 INSIDE THE OLCF Oak Ridge Leadership ComputingFacility models. and an introduction to astrophysics and molecular dynamics computer data,topics models, programming including of parallel visualization second year, thecourse gives current HPC students abroad of overview online HPCclasses for Morehouse of a series College students. In its resources, Whitten educate alsohelps future users. Whitten co-teaches In to addition teaching users how to employ thecenter’s leadership their codes.users to billion-way with harness parallelism their codes, within parallelism thelong-term while isto goal prepare educate users abouthow to employ GPUs to expose new levels of and the applications we use to do science.” is goal to short-term The therefore we have to change the way we think the about hardwareboth systems,” heexplained. “GPUs are theway changing we do business— ever. “In thepastourresources were easily homogenous programmable and education for current andpotential users than ismore important explains that the move toward exascale computing means that training Bobby Whitten the OLCF of User Assistance and Outreach Group future researcherseducation of andcomputational scientists. through itssuperlative resources staff, andskilled but alsothrough the OLCFThe to seeks strengthen thescientific not onlycommunity Education andOutreach Education, Outreach,andTraining Directorate. Projects from ranged climate research to performance fromprimarily theComputing andComputational Sciences (CCS) students were engaged in leading-edge research projects mentors with studentsto from graduate post theUniversity California, of Berkeley, Science (RAMS) program. From freshmen at the University Tennessee of through a summer internship in the Research Alliance in Math and atthe Oak Ridge and facilities Nationalopportunities Laboratory the continent andPuerto theunique Rico took advantage of Twenty four students from colleges eighteen anduniversities across The Research Alliance inMathandScience andbenefitsfrompublic thatultimately supports thescience conducted. Speakers Bureau, astrong the supports connection with thefacility 950 in2010), to andspeeches outside groups through theORNL community at large. Throughuser meetings, public tours (more than OLCFThe asaliaison between itsuser alsoacts community andthe results theOLCF from useof facilities. from around theworld andavehicle for researchers to present scientific isagreat way series The for ORNL researchers colleagues to with interact OLCF and/or useOLCF andcollaborateinteract with staff resources. month, investigators in principal bringing and domain who specialists computing. Thecenter to schedule tries at least one seminar each anemphasis computing onexascale petascale with andthefuture of OLCFThe alsooffered several seminars in2010covering HPCtopics, cluster would have ranked been fastest intheworld. to computers rank on theTop500 List), anddetermine their when theHigh-Performancetest via itsspeed Linpack code (which isused werethey to given collaboratively theopportunity build asupercomputer, in the two-weekstudents participated ARC in 2010, program where school students. high and minority options to Ten underprivileged the developmentAppalachian of region by collegeoffering and career theeducational isto goal and business whose people support the Appalachian Regional Commission (ARC), educators abody of The yearThe 2010 marked the thirdthe OLCF’s year of with partnership visualization softwarevisualization for developed DOE analyzing andwithin research results. workshop This introduces users to VisIt, ascientific For many analyzing HPC projects, of aspect is a crucial visualization June 4, 2010 Visualization with storage by 2015. exabytes of managing capable of requirements,and scalability andalsodeveloped aplanfor asystem architecture to HighProductivity Computing meet Systems performance Participants were to theproposed given review Lustre theopportunity multiplea systembandwidth. terabytes second per capable of of to issues anddevelop identify scalability arealistic roadmap to deliver workshopThis brought together key Lustre developers andengineers next-generation Lustre-based of the design storage systems. Sun Microsystemswith and Cray to host this two-day workshop on at increasing speed. To future meet dataneeds, theOLCF collaborated concurrently expand data provided to increasing handle amounts of As HPCmoves closer to exascale capabilities, storage systems must May 19and20, 2010 Lustre Scalability Workshops and isadministered through theCCSDirectorate Office. sponsored by the Office of Advanced Scientific Computing Research increasing workforceterm goal of diversity. RAMSThe is program projects andencouragement to the long- advanced seek with degrees engineering, andmathematicsdisciplinesthrough hands-on research mentor underrepresented populations in science, technology, Computing conference. RAMSaimsto The program identify and in theDiscovery 2010Scientific participated through Advanced LCF-led workshop “Crash Course inSupercomputing,’ afew while and other on-going research areas. inthe Several students participated tuning to data analytics totuning to sensors to dataanalytics robotics to applied mathematics Workshop VisIt run efficientlyrun on architectureshybrid is a forgoal key the HPC With exascale computing on the horizon, existing codes adapting to August 3–6, 2010 SciApps programming, compiling, aprogram. andrunning course, concepts learned thesenewly students together put each by of libraries.thetwo-day programming By parallel leading theclose of topics such asparallelization, aswellMPI andOpenMP, thetwo commands and VI editor commands. Day two covered more advanced environment. makefiles, Itthebasicsof alsotaught common UNIX students totaught program, compile, code andrun inaUNIX to computenecessary on leadership-class systems like Jaguar. Day one programming parallel courseThis students thebasicsof taught June 17and18, 2010 Crash Course inSupercomputing sessions included ORNL’s working with cluster, visualization LENS. introduce to class participants VisIt in an interactive setting. Active between lecture discussions and hands-on exercises and isto designed extreme-scalevisualizing simulation datasets. class The alternates Kothe. Computing Group Leader Ricky Kendall Science andDirector Doug of and Reinvestment Act, SciApps 2010wasco-hosted by OLCF Scientific a foundation for exascale research. Funded by the RecoveryAmerican applications on HPCsystems leading toward looking while building practices,best andknowledge abouthow to sustain large-scale Applications (SciApps) Conference and Workshop to share experience, researchers70 interdisciplinary gathered atORNLfor theScientific application requirements andscientificmission goals. Approximately development, and into obtain insight near-term and medium-term class computing users, explore to opportunities strengthen application venue to facilitate among interactions current andpotential leadership- community. thisworkshop wasto offer of goal The across-disciplinary Annual Report 2010–2011 41 INSIDE THE OLCF 42 INSIDE THE OLCF user views, opinions, andideas to OLCF management. asthevoice theusers, act councilelected member will of presenting OLCF Users’ Council on alsomet thisday to anew chair. elect This groupsbut available also tosupport the various to them at OLCF. The OLCF users were introduced Jaguar not only to system,the upgraded on Jaguar allocations granted throughbeen theINCITEprogram. New The OLCF Users Meeting focused on the research teams have who May 12, 2010 OLCF Users Meeting event. Representatives from INCITE, OLCF, and ALCF were present at the answers to questions about the proposal process and review for INCITE. prospectiveboth to specific get andreturning users theopportunity (ALCF) for an “INCITE Proposal Writing Lecture/Webinar” to provide OLCFThe the teamed upwith Argonne Leadership Computing Facility January 24, 2011 Proposal conference concluded aroundtable with discussion by led Kendall. aswellsustaining climatepractices science’s research. petascale The resources. Onthefinalday, topics included software engineering resource toward capabilitiesandlooking thepotential for future problems. Day two’s sustaining thepresent talksdescribed computing which supercomputing andaccuracywith speed applications solve Baker for adiscussionprogram led testing abouttheJoule the metric exascale computing.of Computational scientistRebecca Hartman- Project thearchitecture Director Buddy of Bland provided asummary the OLCF, theworkshopDay included of one anoverview of and OLCF Oak Ridge Leadership ComputingFacility Writing Webinars to resolve any issue.—by CaitlinRockett codes and work NCCS, one-on-one with NICS, members and Cray staff toallowed access participants Jaguar and/or Kraken usingtheir own to 100,000or more cores. Hands-on sessions throughout theworkshop debuggers on theXT5, scaling anddeveloping applications capable of NICS, covering andCray staff key topics like XT5 architecture, using Sciences’ (NICS’s) Kraken. The workshop featured lectures from NCCS, ORNL: theOLCF’s Jaguar andtheNational Institute for Computational returning, theCray andpotential users with XT5 systems located at CrayThe XT5 Hex-core workshop new, focuseson familiarizing May 10–12, 2010 XT5 Hex-Core Workshop L install acommunicationinstall protocol, which formats messages into a personal accounts to onlog to their cluster, were they how taught to determine the processors whether were indeed connected. After creating connect the CPUs to testsEthernet cables and oneran toanother via After course acrash incomputer hardware, how students learned to thatmakethings asupercomputer work. studentsThrough theprogram received the afoundation inmany of world—but it successfully software the high-performance ran installed. down thehall—Jaguar,right which isranked number three inthe students’ cluster did not compute nearly asfast asthe beefed-up cluster another tomachine, operate Mac asasingle miniCPUs. out of The computer cluster, computers or group of communicating one with from universities, students the high-school went to work building a did.that they With ORNLstaff, of thehelp collaborators, andinterns courseThe wastitled “Build aSupercomputer—Well Almost.” And theOLCFist atORNLandfacilitator of program. weeks,”their for own acouple of saidBobby Whitten,- anHPCspecial to get HPCbasics, learn “They andit’s achance for them to live on theregion. of from backgrounds andparts various and Mathematics, took place July 12–23. OLCFThe hosted 10 students camp, between a partnership ORNL and the ARC Institute for Science theworld’sfrom some of computing leading experts. summer The Appalachia gathered atORNLthissummer for interactive training However, year, for thethird straight students and teachers from around Supercomputer—Almost—at ORNL High SchoolStudentsBuildTheirOwn ARC studentsatORNL workingonaclusterofMacminis. envision summer vacation. network is probably not the way school students most high howearning to linkmultiple processors andbuild acomputer by Eric Gedenk by Eric biomechanical engineering.— first year, 2008, to Cornell isheading off University inthefall to study Whitten hisstudents from happily notes thatoneprogram’s of correct answer. the problem and of collaborated aspects towardon specific afaster to solvetry theproblem individually, theother students while worked problem. Onestudent waschosen to operate asasingle “node” and another to quickly solve problems, students were given alengthy math For example, to show how multiple processors communicate one with otherwise. topicsillustrate related to HPCthestudents may nothave gotten exercises working asagroup, notonly to theidea of instill but alsoto thecourse.presentation attheend of Instructors team-building used communication,” hesaid, thatstudents noting to had give anoral well. “[Students] work on team building, collaboration, and students receive, to improve tries theprogram in other areas skills as Whitten explained to thatinaddition thetechnical knowledge the I cannow on ameanEthernet cable!” crimp noted,playfully “A over thematerial went straight my lotof head, but networking,” Poole said. Participating student Kenziah Terefenko thefield.of “We andthen fromwent intobeginning started the very computing knowledge andasmall base principles solid foundation of worked the with ARC students, would felt they benefitfrom having a Stephanie Poole, anORNLintern andstudent atPellissippi State who “I learned so much“I learned in my here. time short I only wish their experiences andsuggestdescribe improvements. Students out anonymous filled comment sheets to isasaint.”“Jerry 1sand0s”made of comments to such light-hearted as is” and code—what acomputer“binary understands, from ranged notebook paper cluster aBeowolf “what the2weeks. during learned had Answers on scribbled collaborator, the had students that they list 10 things Pellissippi State Community CollegeandORNL Sherrod,Course Jerry instructor aprofessor at supercomputers.which of istheforte large-scale computations, solving software capable of Finally, high-performance the students and ran installed common language computers cansend andreceive. longer, by 2weeks!” anextra to thepoint:went straight to needs program be “This we more had time,” said. one entry Another suggestion Annual Report 2010–2011 43 INSIDE THE OLCF 44 INSIDE THE OLCF Oak Ridge Leadership ComputingFacility L08703. Be Avoided by Mitigation?” Geophysical Research 36(2009): Letters Lawrence, L. Buja, and W.G. Strand. “How Much Climate Change Can Washington, W.M., R. Knutt, G.A. Meehl, H.Y. Teng, C. Tebaldi, D. Framework.” Monthly Weather 138, no.Review 8(2010): 3333–3341. Element Atmospheric Model Using aFully Implicit Solution Evans, K.J., M.A. Taylor, and J.B. Drake. “Accuracy a Spectral Analysis of Climate 12838–12844. Graphene.” aMetastable of HydrophobicObservation Wetting Two-Layer Ice on Smith, Z. Dohnalek, and B.D. Kay. “No Confinement Needed: Kimmel, G.A., J. Matthiesen, M. Baer, C.J. Mundy, N.G. Petrik, R.S. B114, no.Physical Chemistry 21(2010): 7245–7249. Interface: A Comprehensive Molecular Study.” Dynamics Journal of Vibrational Sulfur Sum-Frequency Dioxide of atthe Spectra Air/Water Baer, M., C.J. Mundy, T.M. Chang, F.M. Tao, andL.X. Dang. “Interpreting Chemistry 132, no. Society Chemical 21(2010): 7372–7378. Factor CPromotes Eukaryotic Clamp-Loading.” Journal the of American Proliferating State NuclearOpened Cell of Antigen by Replication Tainer, J.A., J.A. McCammon, andI. Ivanov. theRing- “Recognition of 5, no. 10(2009): 2798–2808. Petascale Supercomputer.” andComputation Journal Chemical of Theory Multimillion-Atom Molecular Biological Dynamics Simulation on a Schulz, R., B. Lindner, L. Petridis, and J.C. Smith. “Scaling of Letters Are by aSmall Effective Described Charge Density.” Physical Review Wedemann, andR. Seidel. “DNA-DNA Interactions inTight Supercoils Maffeo, C., R. Schopflin, H. Brutzer, R. Stehr, A. Aksimentiev, G. Biology 310–377. Plasmas.” and Fluid Turbulent Cascades inMagnetized Weakly Collisional Howes, E. Quataert, and T. Tatsuno. “Astrophysical Gyrokinetics: Kinetic Schekochihin, A.A., S.C. Cowley, W. Dorland, G.W. Hammett, G.G. 869–872. Supernovae from Broken Symmetries.” Nature 460, no. 7257(2009): Kasen, D., F.K. Ropke, andS.E. Woosley. Type Diversity of “The Ia Astrophysical Journal 713, no. 2(2010): 1219–1243. by Fields Magnetic Accretion theStationary of Shock Instability.” Endeve, E., C.Y. Cardall, R.D. Budiardja, and A. Mezzacappa. “Generation Astrophysics High-Impact Publications 105, no. 15(2010): 158101. Astrophysical Journal Supplement Series Journal 131, the of American Society no. Chemical 35(2009): 182, no. 1 (2009):

Nearby Cosmic-Ray Accelerators andInterstellar Medium Turbulence Malkov, M.A., P.H. Diamond, L.O. Drury, andR.Z. Sagdeev. “Probing Fusion no. 19(2010): 199701. Overdoped Cuprate Superconductors.’” Letters Physical Review HubbardReveal theSingle-Band theInapplicability of Model to Phillips, P., andM. Jarrell. “Comment on ‘X-ray Absorption Spectra Research 49, Chemistry no.Engineering 21(2010): 10767–10779. Resolution Coal Injection inaGasifier.” Simulations of & Industrial Li, T.W., A. Gel, M. Syamlal, C. Guenther, andS. Pannala. “High- Parametrization.” B81, Physicalno. Review 1(2010): 125202. Suitable for [110]UniaxialBinding Theory Strain Applied to aSilicon Induced, Off-Diagonal, Same-Atom Parameters Tight- inEmpirical Boykin, T.B., M. Luisier, M. Salmani-Jelodar, and G. Klimeck. “Strain- Engineering Properties of PbTe,Properties of PbSe, andPbS: First-Principles Study.” Physical Zhang, Y., X.Z. Ke, C.F. Chen, J. Yang, andP.R.C. Kent. “Thermodynamic 17 (2009): 175508. Metal Atoms Skutterudites.” inFilled 102, Letters no. Physical Review Wang, Y.G., X.F. Xu, andJ.H. Yang. Misch-“Resonant of Oscillation 104, Letters no.Physical Review 24(2010): 247001. Interplay between Inhomogeneities andtheSuperconducting State.” Hubbard Model Charge-Density-Wave Stripelike with Modulations: Cluster Quantum Monte aTwo-Dimensional Carlo Simulations of Maier, T.A., G. Alvarez, M. Summers, andT.C. Schulthess. “Dynamic Materials no. 7249(2009): 978–U102. and Uplift by Warming Heterogeneous of Lithosphere.” Roy, M., T.H. Jordan, andJ. Pederson. “Colorado Plateau Magmatism Geosciences 6 (2010): 065002. Plasma.” aCylindrical Laboratory of 104, Letters no. Physical Review and J.H. Yu. Rotation“Intrinsic from aResidual Stress attheBoundary Yan, Z., M. Xu, P.H. Diamond, C. Holland, S.H. Muller, G.R. Tynan, (2009): 085004. Modes inCollisionless Plasmas.” 103, Letters no. Physical Review 8 Xiao, Y., and Z.H. Lin. “Turbulent Transport Trapped-Electron of 750–761. MILAGROwith Hot Spots.” Astrophysical Journal 721, no. 1(2010): Review B80, no.Review 2(2009): 024304. Nature 459, 105,