NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER 2003 ANNUAL REPORT

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER 2003 ANNUAL REPORT

This work was supported by the Director, Office of Science, Office of Advanced Scientific Computing Research of the U.S. Department of Energy under Contract No. DE-AC 03-76SF00098.

LBNL-54267, January 2004

TABLE OF CONTENTS

THE YEAR IN PERSPECTIVE...... 2 RESEARCH NEWS ...... 47

ADVANCES IN COMPUTATIONAL SCIENCE . . . 5 Accelerator Physics ...... 47

Astrophysics: Computing the Big Picture . . . . 7 Applied Mathematics ...... 48 Universe: The Movie ...... 8 Astrophysics ...... 50 Simulation Matches Historic Chemical Sciences ...... 52 Gamma-Ray Burst...... 11 Climate Change Research...... 55 Nearby Supernova Factory Churns Computer Science ...... 56 Out Discoveries ...... 14 Fusion Energy Sciences ...... 57 Fusion Energy: Improved Algorithms Shift High Energy Physics...... 62 Plasma Simulations into High Gear...... 17 Materials Sciences ...... 63 Full Waves, Faster Runs for AORSA . . . 18 Medical and Life Sciences...... 66 SuperLU Solver Supercharges Nuclear Physics ...... 68 NIMROD ...... 20 AMR Methods Accelerate THE NERSC CENTER...... 71 MHD Simulations ...... 22 Clients, Sponsors, and Advisors ...... 71 Biology: Deciphering Nature’s Capability and Scalability ...... 73 Assembly Manuals ...... 25 Comprehensive Scientific Support...... 78 ProteinShop: Solving Protein Structures from Scratch ...... 26 Connectivity...... 81 Mapping the Universe of Protein Advanced Development ...... 83 Structures ...... 29 Appendix A: NERSC Policy Board...... 87 DOE Facilty Plan Sees Major Role for Computing...... 32 Appendix B: NERSC Computational Review Panel ...... 88 Nanoscience: Modeling a Realm Where Everything Is Different ...... 33 Appendix C: NERSC Users Group Making Fuel Cells More Efficient Executive Committee...... 90 and Affordable...... 34 Appendix D: Supercomputing Allocations Climate Modeling: Predicting the Results Committee ...... 91 of a Global Experiment...... 37 Appendix E: Office of Advanced Scientific Fingerprints in the Sky ...... 38 Computing Research ...... 92 Applied Mathematics: The Bridge between Appendix F: Advanced Scientific Computing Physical Reality and Simulations...... 41 Advisory Committee ...... 93 Algorithms in the Service of Science . . 42 Appendix G: Organizational Acronyms ...... 94

THE YEAR IN PERSPECTIVE

Supercomputing continues to evolve at a rapid rate, and events at NERSC in 2003 demonstrate this quite clearly. For a few months we could claim to run the largest unclassified system in the U.S. But as we all know, these superlatives don’t last. What does last are the scientific accomplish- ments obtained using large systems.

It is with great satisfaction that we write this introduction to a report that describes some truly astounding com- putational results that were obtained with the help of NERSC in 2003. In July 2003 Dr. Raymond Orbach, Director of the DOE Office of Science, while testifying about high-end computing before the U.S. Congress, explicitly mentioned NERSC’s role in the discovery that the expansion of the Universe is accelerating, as well as the recent U.S. decision to rejoin the interna- tional collaboration to build the ITER fusion reactor. “What changed,” he said, “were simulations that showed that the new ITER design will in fact be capable of achieving and sustain- ing burning plasma.” From accom-

Horst D. Simon

2 THE YEAR IN PERSPECTIVE

plishments such as these, it is clear came online one month earlier than For many users, Seaborg became the that we really have entered the new planned. The first applications test first platform on which they could era where computational science is an results on the upgraded Seaborg were explore calculations using several equal partner with theory and exper- very impressive: not only could users thousand processors. iment in making scientific progress. quickly scale several applications to New opportunities for computa- At NERSC the biggest change in 4,096 processors, but they were also tional science breakthroughs will 2003 was the upgrade of our current able to achieve very high sustained hopefully arise in 2004 through the Seaborg platform from 5 to 10 Tflop/s performance rates, more than 60% new Innovative and Novel Compu- peak performance. This created one of peak in a few cases. The speed of tational Impact on Theory and of the largest systems in the U.S. the upgrade process and the new Experiment (INCITE) program at dedicated to basic computational applications results are a testimony NERSC. The goal of the program is science. It now gives NERSC users to the expert staff at NERSC. We to support a small number of com- around-the-clock access to an unprece- have again demonstrated our capa- putationally intensive, large-scale dented 6,556 processors, coupled bility to quickly field and use a new research projects that can make high- with one of the largest memory sys- system productively. impact scientific advances through tems anywhere. DOE-supported The expanded computational the use of a substantial allocation of computational scientists continue to power of Seaborg made it possible computer time and data storage at have access to one of best possible to initiate a number of new projects. NERSC. We just completed the resources to further the DOE mis- In the first half of 2003, NERSC selection of the first three INCITE sion in the basic sciences. started the scalability program, which projects, and we hope to announce The upgrade to 10 Tflop/s went gave users the opportunity to evalu- breakthrough results from these very smoothly, and the new system ate the scaling of their applications. projects in next year’s annual report.

William T. C. Kramer

THE YEAR IN PERSPECTIVE 3 One of the signposts of change are thrilled that the DOE computa- in 2002 was the launch of the Earth tional science community, in partic- Simulator system in Japan, which ular our NERSC users, are ready to reenergized the high-end computing take the lead in moving to the next community in the U.S., resulting in level of computational science. a number of new activities in 2003 One small step towards this bright that made a profound impact on com- future will happen at NERSC in putational science and high perform- the next few months. We will have ance computing. Both the High End the opportunity to acquire a new Computing Revitalization Task Force computing system that will take over (HECRTF) report and the Science- some of the existing user workload Based Case for Large-Scale Simulation from Seaborg, making even more (SCaLeS) report demonstrated that time available on Seaborg for large- the high-end computing community scale capability computing. This will in the U.S. is well positioned for be just an intermediate step towards future developments and growth in our next big acquisition with NERSC supercomputing. The SCaLeS report 5. We are looking forward to another gives us a glimpse of what simulations year of both scientific and computing are possible at sustained speeds in accomplishments at NERSC. As the range of tens of teraflop/s, and always, this progress would not be the HECRTF report lays out the possible without the NERSC staff, critical research issues that we need who continue to tirelessly dedicate to address in order to reach petascale their time, skill, and effort to make computing performance. This is excit- NERSC the best scientific comput- ing news, because we are thinking ing resource in the world. Our special big again in supercomputing. We thanks to all of you.

Horst D. Simon NERSC Center Division Director

William T. C. Kramer Division Deputy and NERSC Center General Manager

4 THE YEAR IN PERSPECTIVE

ADVANCES IN COMPUTATIONAL SCIENCE

ADVANCES IN COMPUTATIONAL SCIENCE 5

ASTROPHYSICS COMPUTING THE BIG PICTURE

From ancient times, people have tried to interpret what they see in the night sky as a way of understanding their place in the cosmos. Today astrophysicists rely on high-end computers, networks, and data storage to manage and make sense of the massive amounts of astronomical data being produced by advanced instruments. Computational simulations performed on NERSC computers are answering questions as general as how structure developed in the Universe, and as specific as how supernovae produce gamma-ray bursts. NERSC’s real- time integration of data gathering with data analysis is enabling collaborators like the Nearby Supernova Factory to step up the pace of discovery. The projects described in this section demonstrate how modern instruments and computers have ushered in a golden age of astropysics.

ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics 7

UNIVERSE: THE MOVIE

When we look up into a clear night sky, we’re looking into the past. Even with the naked eye, we can see the Andromeda galaxy as it was 2.3 million years ago. Earth- and space-based telescopes provide us with a plethora of snapshots of the history of the Universe. But none of these snapshots show how the Universe or any of the objects within it evolved.

“Turning the snapshots into movies match recent observations. His by gravitational attraction, with is the job of theorists,” says Joel research group uses NERSC’s small objects merging in a continu- Primack, Professor of Physics at the Seaborg computer to create simula- ous hierarchy to form more and University of California, Santa tions of galaxy formation. Their more massive objects. Cruz. “With computational simula- work here has two major thrusts: However, “objects” in this con- tions, we can run the history of simulations of large-scale structure text refers not only to ordinary stars, galaxies, or the whole formation in the Universe, and sim- “baryonic” matter (composed of Universe forward or backward, test- ulations of galaxy interactions. protons and neutrons), but also to ing our theories and finding expla- How did the Universe evolve cold dark matter—cold because it nations for the wealth of new data from a generally smooth initial state moves slowly, and dark because it we’re getting from instruments on after the Big Bang (as seen in the cannot be observed by its electro- the ground and especially in space.” cosmic microwave background radi- magnetic radiation. In fact, the lat- Primack is one of the origina- ation) to the chunky texture that we est simulation results1 suggest that tors and developers of the theory of see today, with clusters, filaments, galaxy clustering is primarily a cold dark matter, which has and sheets of galaxies? That is the result of the formation, merging, become the standard theory of question of large-scale structure for- and evolution of dark matter structure formation in the Universe mation. In the cold dark matter the- “halos.” because its predictions generally ory, structure grows hierarchically The concept of dark matter

8 ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics

halos came from the realization around 100,000 halos. These mod- FIGURE 1 Simulated distribution of dark matter that the dark matter in a galaxy els can be used to predict the types particles (points) and halos (circles) at extends well beyond the visible stars and relative quantities of galaxies, as redshift 3 (left) and redshift 0 (right). and gas, surrounding the visible well as how they interact. Figure 1 (Higher redshift signifies greater dis- matter in a much larger halo; the shows the simulated distribution of tance and age.) Image: Kravtsov et al., astro-ph/0308519. outer parts of galaxies are essentially dark matter particles and halos at all dark matter. Scientists came to two different stages of structure for- this conclusion because of the way mation. Interestingly, there are stars and galaxies move, and more subhalos in this model than because of a phenomenon called there are visible objects in corre- gravitational lensing: when light sponding observational data. passes through the vicinity of a Observations from the DEEP galaxy, the gravity of the dark matter (Deep Extragalactic Evolutionary halo bends the light to different Probe) and ESO/VLT (European wavelengths, acting like imperfec- Southern Observatory/Very Large tions in the glass of a lens. Telescope) sky surveys should soon In the view of Primack and his confirm or refute the predictions of colleagues, the large-scale structure this model. of the Universe is determined pri- Elliptical galaxies, galactic marily by the invisible evolution bulges, and a significant fraction of and interactions of massive halos all the stars in the Universe are and subhalos, with ordinary matter thought to be shaped largely by being swept along in the gravita- galaxy interactions and collisions, 1 A.V. Kravtsov,A.A. Berlind, R. H.Wechsler, tional ebbs and flows. They have with dark matter still playing the A.A. Klypin, S. Gottlöber, B.Allgood, and used Seaborg to create the highest- major role but with bright matter J. R. Primack,“The dark side of the halo resolution simulations of dark mat- interactions adding some details. occupation distribution,”Astrophysics ter halos to date, producing 10 tera- For the first half of the Universe’s Journal (submitted); astro-ph/0308519 bytes of output and models with history, stars were formed mostly in (2003).

ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics 9

spectacular “starburst” events result- merger, and found that previous going a remarkable evolution, driv- ing from the compression of gases studies had overestimated the en mainly by the wonderful wealth by the gravitational tides of interact- amount of starburst. Their results of new data,” he says. “The ques- ing galaxies. A starburst is a relative- showed that galaxy mergers produce tions that were asked from the ly brief event on a cosmic scale, large outflows of gas that inhibit star beginning of the 20th century are lasting perhaps 100 million years, formation for a while after the now all being answered. New ques- while the merger of two galaxies merger, an effect that they are tions arise, of course, but there’s a might take 2 to 3 billion years. The adding to their models of galaxy for- good chance that these answers are Hubble space telescope in 1996 mation. definitive.” captured spectacular images of two Like many physicists and colliding galaxies and the resulting astronomers, Primack believes that Research funding: HEP,NSF,NASA starburst (Figure 2). we are now living in the golden age (Organizational acronyms are spelled out By creating galaxy interaction of cosmology. “Cosmology is under- in Appendix G) simulations (Figure 3) and compar- ing them with images and data from a variety of telescopes and spectral bands, Primack’s research team tries to understand the evolu- tion of galaxies, including phenom- ena such as starbursts. They recent- ly focused their attention on how much gas is converted to stars dur- ing the course of a major galaxy

FIGURE 2 The Hubble telescope uncovered over 1,000 bright, young star clusters bursting to life in the heart of a pair of colliding galaxies. The picture in the upper left provides a ground-based telescopic view of the two galaxies, called the Antennae because a pair of long tails of luminous matter, formed by the gravita- tional tidal forces of their encounter, ments of dark dust. A wide band of resembles an insect’s antennae. The chaotic dust stretches between the galaxies are located 63 million light- cores of the two galaxies. The sweeping years away in the southern constellation spiral-like patterns, traced by bright Corvus. Hubble’s close-up view pro- blue star clusters, are the result of a vides a detailed look at the “fireworks” firestorm of star birth that was triggered in the center of the collision. The by the collision. Image: Brad Whitmore respective cores of the twin galaxies are (Space Telescope Science Institute) and the orange blobs, crisscrossed by fila- NASA.

10 ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics

FIGURE 3 Snapshots from a simulation of two galaxies merging. In this simulation, the galaxies do not crash head-on; their cores sideswipe each other and spin wildly apart before their gravi- ty pulls them back together. Image: Thomas J. Cox, University of California, Santa Cruz.

SIMULATION MATCHES HISTORIC GAMMA-RAY BURST

After three decades of scientific head-scratching, the origins of at least some gamma-ray bursts (GRBs) are being revealed, thanks to a new generation of orbiting detectors, fast responses from ground- based robotic telescopes, and a new generation of computers and astrophysics software.

A GRB detected on March 29, if we could see gamma rays with domly from every direction. They 2003 has provided enough informa- our eyes, about once a day, some- do not repeat themselves. And they tion to eliminate all but one of the where in the world, people would last from only a few milliseconds to theoretical explanations of its ori- see a spectacular flash, a hundred a few minutes—astronomers consid- gin. Computational simulations million times brighter than a super- er a GRB long if it lasts longer than based on that model were already nova. These gamma-ray bursts, first 2 seconds. being developed at the NERSC discovered in the 1960s by satellites With data so hard to pin down, Center at when the discovery was looking for violations of the scientists for a long time could only made. Nuclear Test Ban Treaty, are the speculate about what causes GRBs. Gamma radiation from outer most energetic events in the Speculations ranged from the space is blocked by the Earth’s Universe, but they are also among intriguing (comet/anti-comet anni- atmosphere. But if it were not, and the most elusive. GRBs appear ran- hilation) to the far-fetched (inter-

ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics 11

stellar warfare). By 1993, 135 differ- 1997 astronomers discovered that allowed astronomers to study the ent theories on the origin of GRBs long GRBs leave an afterglow of event in unprecedented detail; they had been published in scientific lower-energy light, such as X rays or even joked about its casting shad- journals. But in recent years, three visible light, that may linger for ows. The Hubble Space Telescope theoretical models emerged as front- months, allowing researchers to pin- will be able to study the afterglow runners—one involving colliding point where the GRB originated for another year, and radio tele- neutron stars, and the other two and providing important clues scopes may be able to track it involving supernovae, the “supra- about the event that produced it. longer. nova” and “collapsar” models. The presence of iron in the after- The first 3D computational sim- In the collapsar model (intro- glow light strongly suggested star ulations of jet formation and break- duced in 1993 by Stan Woosley, explosions. In 1998, the supernova out in the collapsar model (Figures Professor and Chair of Astronomy theories got an additional boost 4 and 5) were already being con- and Astrophysics at the University of when a GRB and a supernova ducted on Seaborg by Woosley and California, Santa Cruz), the iron appeared in the same vicinity at Weiqun Zhang, a Ph.D. candidate core of an aging star runs out of roughly the same time; but the data at UC Santa Cruz, under the spon- fuel for nuclear fusion and collapses was inconclusive and some scien- sorship of the DOE Office of of its own weight, creating a black tists remained skeptical about the Science’s SciDAC Supernova hole or a dense neutron star. connection. Several similar sugges- Science Center (http://www.super- Material trying to fall onto this tive but inconclusive events in sub- sci.org/), one of two SciDAC object forms a hot swirling disk and sequent years divided astronomers (Scientific Discovery through a narrow jet, which shoots out of into two camps on the issue of asso- Advanced Computing) programs the star in less than ten seconds at ciating GRBs with supernovae. focusing on developing new com- nearly the speed of light. When this The skepticism was dispelled on putational methods for understand- jet erupts into interstellar space, it March 29, 2003, when HETE ing supernovae. The goal of the creates a “fireball” of gamma rays— detected an unusually bright and Supernova Science Center is to dis- the highest energy, shortest wave- close GRB—only 2.6 billion light cover the explosion mechanism of length form of electromagnetic years from Earth instead of the typi- supernovae through numerical sim- radiation. The rest of the star cal 10 billion. The discovery trig- ulation. explodes as a supernova, but that gered a swarm of observations that Zhang, Woosley, and colleagues event is eclipsed by the brighter found the unmistakable spectral sig- had been modeling jet formation GRB. nature of a supernova in the after- after core collapse in both two and In the late 1990s, GRB research glow, as reported in the June 19, three dimensions for a few years took a great leap forward when a 2003 issue of Nature.2 This event, when observations from new generation of orbiting detectors named GRB030329 after its detec- GRB030329 validated their work were launched, including the tion date, was dubbed the “Rosetta and made it newsworthy. “We Italian-Dutch satellite BeppoSAX stone” of GRBs because it conclu- weren’t utterly surprised,” Woosley and NASA’s High-Energy Transient sively established that at least some said, “because the evidence associ- Explorer (HETE). These satellites long GRBs come from supernovae, ating GRBs with supernovae had were designed to provide quick and it singled out the collapsar been accumulating for several notification of GRB discoveries for model as the one that best fits the years. But we weren’t totally confi- immediate follow-up observations data collected so far. The March 29 dent, either. When the light spectrum by ground-based instruments. In burst’s afterglow was so bright that it from the March 29 burst confirmed

2 J. Hjorth, J. Sollerman, P.Møller, J. P.U. Fynbo, S. E.Woosley, et al.,“A very energetic supernova associated with the γ-ray burst of 29 March 2003,”Nature 423, 847 (2003).

12 ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics

FIGURE 4 that it came from a supernova, that GRB observations—as well as simu- This image from a computer simulation felt good.” lating the full star explosion that of the beginning of a gamma-ray burst shows the jet 9 seconds after its cre- Woosley emphasized, however, accompanies the GRB and the core ation at the center of a Wolf-Rayet star that “the data is still way ahead of collapse that precedes it. The proj- by the newly formed, accreting black the theory.” The computational ect will be challenging, even on a hole within. The jet is now just erupting through the surface of the star, which simulations are still incomplete, computer as fast as Seaborg. has a radius comparable to that of the and the resolution needs to be “This does not mean that the sun. Blue represents regions of low improved. Upcoming work will gamma-ray burst mystery is solved,” mass concentration, red is denser, and include higher-resolution 3D stud- Woosley added. “We are confident yellow denser still. Note the blue and red striations behind the head of the jet. ies of jet stability—which will help that long bursts involve a core col- These are bounded by internal shocks. scientists interpret the differences in lapse, probably creating a black Image: Weiqun Zhang and Stan Woosley.

FIGURE 5 This image from a computer simulation shows the distribution of relativistic particles (moving near light speed) in the jet as it breaks out of the star. Yellow and orange are very high energy and will ultimately make a gamma-ray burst, but only for an observer looking along the jet (± about 5 degrees). Note also the presence of some small amounts of energy in mildly relativistic matter (blue) at larger angles off the jet. These will produce X-ray flashes that may be seen much more frequently. Image: Weiqun Zhang and Stan Woosley.

ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics 13

hole. We have convinced most because supernovae are currently understanding of the physical skeptics. We cannot reach any con- thought to be the source of all the processes involved. NERSC’s IBM clusion yet, however, on what caus- elements heavier than iron. supercomputer is currently being es short gamma-ray bursts.” Because supernovae cannot be used by several research groups Aside from explaining the mys- recreated in a laboratory, numerical studying supernovae. terious long GRBs, the study of simulation is the only tool available supernovae will fill an essential gap for interpreting observational data Research funding: NP,SciDAC in our knowledge of the Universe, and for developing a detailed

NEARBY SUPERNOVA FACTORY CHURNS OUT DISCOVERIES

When the Supernova Cosmology Project got up to speed in the mid-1990s, its team of astrophysicists, led by Saul Perlmutter of Lawrence Berkeley National Laboratory, were looking forward to finally getting decent measurements of how much the expansion of the Universe was slowing down.

The team had figured out how to their high-quality measurements all energy, the unknown force driving accelerate the discovery of Type Ia right, but the data showed that the the expansion, which accounts for supernovae by systematically expansion of the Universe was about two-thirds of the total energy searching the same patches of sky at speeding up, not slowing down as density in the Universe. To find out intervals, then subtracting the everyone had assumed. The data what dark energy is, scientists need images from one another, following were analyzed on NERSC comput- more detailed measurements of the up with more detailed measure- ers using different sets of cosmologi- Universe’s expansion history. Thus ments of the remaining bright spots. cal assumptions, but the results they need even more observations By comparing the distance of these were inescapable. And Australia’s of Type Ia supernovae, and they exploding stars (which all have the High-z Supernova Search Team, need to reduce some remaining same intrinsic brightness) with the working independently, had uncertainty about the uniform redshifts of their home galaxies, reached the same conclusion. brightness of these exploding stars. researchers could calculate how fast So in 1998, the two teams of Reducing the uncertainty and the Universe was expanding at dif- researchers made the startling improving the calibration of Type ferent times in its history. announcement that the expansion Ia supernovae are among the chief As so often happens in science of the Universe is accelerating. goals of the Nearby Supernova and in life, things did not turn out Before long a new term entered the Factory (SNfactory), a spinoff of the as expected. The researchers got vocabulary of astrophysics: dark Supernova Cosmology Project, led

14 ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics

by Berkeley Lab physicist Greg Aldering. At present, astronomers can determine the distance to a well-studied Type Ia supernova with an accuracy of 5 percent, an accu- racy the SNfactory expects to improve. “Our motive is to establish a much better sample of nearby Type Ia supernovae, against which the brightness of distant supernovae can be compared to obtain relative distances,” says Aldering. What researchers want to meas- ure, says Aldering, also includes “the intrinsic colors of a Type Ia at every stage, so we’ll know the effects of intervening dust. And we want to know what difference the ‘metallicity’ of the home galaxy million images and archived 6 tera- FIGURE 6 makes—that is, the presence of ele- bytes (trillion bytes) of compressed A special link in the High Performance Wireless Research and Education ments heavier than helium.” data at NERSC—one of the few Network (HPWREN) transmits images To accomplish these goals, the centers with an archive large from the Near Earth Asteroid Tracking SNfactory needs to discover and enough to store this much data. program (NEAT) at Mount Palomar to make detailed observations of 300 “The SNfactory owes much of its NERSC for storage and processing for the SNfactory. to 600 low-redshift supernovae, success to NERSC’s ability to store many more than have been studied and process the vast amounts of so far. Automation and tight coordi- data that flow in virtually every nation of the search and follow-up night,” says Aldering. stages are absolutely essential to The SNfactory’s custom-devel- achieving these numbers. During oped data pipeline software, devel- its first year of operation, the oped by UC Berkeley graduate stu- SNfactory found 34 supernovae. dent Michael Wood-Vasey, manages “This is the best performance ever up to 50 gigabytes (billion bytes) of for a ‘rookie’ supernova search,” data each night from wide-field Aldering says. “In our second year, cameras built and operated by the we have shown we can discover Jet Propulsion Laboratory’s Near supernovae at the rate of nine a Earth Asteroid Tracking program month, a rate other searches have (NEAT). NEAT uses remote tele- reached only after years of trying.” scopes at Mount Palomar This remarkable discovery rate Observatory in Southern California is made possible by a high-speed and at the U.S. Air Force’s Maui data link, custom data pipeline soft- Space Surveillance System on ware, and NERSC’s data handling Mount Haleakala. and storage capacity. So far the With the help of the High SNfactory has processed a quarter- Performance Wireless Research and

ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics 15

Education Network (HPWREN) that can sift through billions of around a supernova—in this case, program at the San Diego objects. This analysis is done using SN 2002ic, discovered near maxi- Supercomputer Center (SDSC), NERSC’s PDSF Linux cluster. mum light.3 Researchers have been the SNfactory was able to establish Eventually the pipeline will looking for hydrogen to discrimi- a custom-built, high-speed link with automate the entire discovery and nate between different types of pos- Mount Palomar (Figure 6). Existing confirmation process. Once a sible progenitor systems to Type Ia links to Maui and to SDSC supernova is discovered from the supernovae. Large amounts of through the DOE’s Energy Palomar or Maui images, follow-up hydrogen around SN 2002ic sug- Sciences Network (ESnet) com- observations will be obtained via gest that the progenitor system con- plete the connection to NERSC. remote control of a custom-built tained a massive asymptotic giant NEAT sends images of about dual-beam optical spectrograph branch star that lost several solar 500 square degrees of the sky to (being completed by SNfactory col- masses of gas in a “stellar wind” in NERSC each night. The data laborators in France) mounted on the millennia leading up to the pipeline software automatically the University of Hawaii’s 88-inch Type Ia explosion. Discoveries like archives these in NERSC’s High telescope on Mauna Kea. The this, which provide a more detailed Performance Storage System Hawaii observations will be shipped understanding of the physics of (HPSS). NEAT’s telescopes revisit by Internet for image processing at supernovae, will improve the use- the same regions of the sky roughly a supercomputing center in France fulness of supernovae as distance every six days during a typical 18- and then sent to NERSC for analysis. markers on the journey back day observing period. When a A recent major discovery made through the history of the cosmos. supernova appears in one of those possible by the SNfactory was the galaxies, the SNfactory can find it first detection of hydrogen in the Research funding: HEP,LBNL, FBF,CNRS, using image subtraction software form of circumstellar material IN2P3, INSU, PNC

3 M. Hamuy, M. M. Phillips, N. B. Suntzeff, J. Maza, L. E. González, M. Roth, K. Krisciunas, N. Morrell, E. M. Green, S. E. Persson, and P.J. McCarthy,“An asymptotic-giant-branch star in the progenitor system of a type Ia supernova,”Nature 424, 651 (2003).

16 ADVANCES IN COMPUTATIONAL SCIENCE / Astrophysics

FUSION ENERGY IMPROVED ALGORITHMS SHIFT PLASMA SIMULATIONS INTO HIGH GEAR

“The computer literally is providing a new window through which we can observe the natural world in exquisite detail.”1

Nowhere is that statement more true than in the field of plasma science, where for nearly three decades, computational simulations have contributed as much as experiments to the advancement of knowledge. The simulation of a magnetic fusion confinement system involves modeling of the core and edge plasmas, as well as the plasma-wall interactions. In each region of the plasma, turbulence can cause anomalies in the transport of the ionic species in the plasma; there can also be abrupt changes in the form of the plasma caused by large-scale instabilities. Computational modeling of these key processes requires large- scale simulations covering a broad range of space and time scales.

DOE’s SciDAC program has brought together plasma physicists, computer sci- entists, and mathematicians to develop more comprehensive models and more powerful and efficient algorithms that enable plasma simulation codes to take full advantage of the most advanced terascale computers, such as NERSC’s Seaborg IBM system. Dramatic improvements in three of these codes are described in this section.

1 J. S. Langer, chair of the July 1998 National Workshop on Advanced Scientific Computing.

ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy 17

FULL WAVES, FASTER RUNS FOR AORSA

DOE’s magnetic fusion energy research program aims to produce energy by the same process that takes place in the sun.

However, on Earth a fusion reac- Multidimensional Fusion Plasmas” exist with very different wavelengths tion requires that the fuel be heated is to develop the computational and polarizations. A wave launched to hundreds of millions of degrees, capability to understand the physi- into the plasma can in a short dis- even hotter than in the sun. At cal behavior of waves in fusion plas- tance completely change its charac- these astronomical temperatures, mas and to model these wave phe- ter to another type of wave, a the fuel atoms are torn apart into nomena accurately enough that process called mode conversion. To their constituent electrons and their effect on other plasma process- study these effects, the computer nuclei, forming a state of matter es, such as stability and transport of model must have very high resolu- called plasma. particles and energy, can be under- tion to see the small-scale structures One of the ways to get the fuel stood. that develop, which means that very this hot is to use intense electro- According to project leader Don large computers are needed to solve magnetic waves, much as a Batchelor of Oak Ridge National for the very large number of microwave oven is used to heat Laboratory, “Because the particles unknowns in the equations. Also, food. Waves are also used for other are so hot, they move at speeds the computers must be extremely important purposes such as to drive almost the speed of light and can fast in order to obtain the solutions electric currents affecting the plas- travel a distance comparable to a in a reasonable time.” ma magnetic configuration, to force wavelength in the time of a few Until now researchers wishing mass flow of the plasma, affecting oscillations of the wave. This to calculate the effects of waves in its stability, and for other plasma motion makes it difficult to calcu- plasma have been forced by compu- control tasks. Therefore, it is impor- late how the plasma particles will tational feasibility to make a diffi- tant to have a good theoretical respond to the waves and how cult choice between restricting con- understanding of wave behavior much heating or electric current sideration to a single dimension or and to be able to calculate it accu- they will produce.” simplifying the physics model. The rately. The goal of the SciDAC “Another challenge,” Batchelor first choice, treating the plasma project “Terascale Computation of adds, “is that at a given frequency, geometry as one-dimensional, is Wave-Plasma Interactions in several kinds of plasma waves can akin to tunnel vision. The wave is

18 ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy

computed along a single line processes about 220,000 coupled 40,000. The new reduced system through the plasma, but one does complex equations that require required only 26 GB of memory and not get a picture of what is occurring about 788 GB of memory. AORSA- was completed in about 13 minutes, in the whole plasma cross-section. 3D took about 358 minutes to run with a 100-fold speedup in the lin- The second choice eliminates from on 1,936 processors of NERSC’s ear solver and a 27-fold speedup in consideration many of the most Seaborg computer. The efficient overall runtime. The efficiency important wave processes for today’s computation achieved over 60 per- gained from the real space formula- fusion experiments, which require cent of available peak performance tion of AORSA-3D allows higher high frequencies and can have very (over 1.9 teraflop/s or 1,900 billion resolution and more design analysis short wavelengths in some regions operations per second). for future experiments such as the of the device. A new formulation of AORSA- Quasi-Poloidal Stellarator. In a partnership between plas- 3D transforms the linear system “Funding from the SciDAC ma physicists and computer scien- from a Fourier basis to a representa- project has enabled us to assemble tists, this project has increased the tion in the real configuration space. a team from multiple institutions to resolution and speed of plasma This alternative representation pres- work on a unified set of goals in a wave solvers to the point that it is ents new opportunities to reduce way that was not possible before possible for the first time to study the overall number of equations SciDAC,” Batchelor says. “Working the mode conversion process in the used. In one example of 34 × 34 × with computer scientists funded to complicated geometry and the full 64 modes, the number of equations work directly with our project has scale of real fusion devices. was reduced by more than a factor made it feasible to study new regimes Collaborators have developed new of 5—from 220,000 to about of wave physics which previously wave solvers in 2D and 3D called

All-Orders Spectral Algorithm (a) Real (E⊥)(b) Real (E||) (AORSA) solvers based on a more E parallel for C-Mod D (3He, H) general formulation of the physics He3 H 0.3 0.2 called an integral equation. It is now possible to compute plasma 0.2 waves across an entire plasma cross- 0.1 section with no restriction on wave- 0.1 length or frequency (Figure 1). In this approach the limit on attain- 0 0 Z (m) able resolution comes only from the size and speed of the available –0.1 computer. –0.1 AORSA uses a fully spectral rep- –0.2 resentation of the wave field in a –0.3 –0.2 Cartesian coordinate system. All modes in the spectral representa- 0.4 0.5 0.6 0.7 0.8 0.9 0.575 0.625 0.675 0.725 0.60 0.65 0.70 tion are coupled, and so computing R (m) R (m) the solution requires calculating and inverting a very large dense FIGURE 1 Mode conversion in a plasma cross sec- × matrix. For example, with 272 tion computed with AORSA. From E. F. 272 Fourier modes in two dimen- Jaeger et al., “Sheared poloidal flow sions or 34 × 34 × 64 modes in driven by mode conversion in tokamak plasmas,” Phys. Rev. Lett. 90, 195001 three dimensions, the system (2003).

ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy 19

could only be treated in one dimen- maticians,” he added, “we are problems. And we are making sci- sion or with approximate tech- developing new mathematical rep- entific progress on problems of niques, and to carry out multiple resentations for the wave fields that importance to the fusion program code runs at high resolution for sci- are more data efficient than the tra- by applying the codes we have entific case studies where before it ditional spectral techniques presently developed and improved.” was only possible to perform a sin- employed and offer promise to sub- gle run at minimal resolution.” stantially reduce the size of the com- Research funding: FES, SciDAC, ORNL “Working with applied mathe- putational load to solve plasma wave

SUPERLU SOLVER SUPERCHARGES NIMROD

Developers of the NIMROD code, which is used to simulate fusion reactor plasmas, collaborated with members of the SciDAC Terascale Optimal PDE Simulations Center to implement the SuperLU linear solver software within NIMROD.

As a result, NIMROD runs four to formulations (see below). step in a simulation. The stiffness five times faster for cutting-edge By applying modern computa- inherent in the physical system leads simulations of nonlinear macro- tional techniques to the solution of to matrices that are ill-conditioned, scopic electromagnetic dynamics— extended magnetohydrodynamics since rapid wave-like responses pro- with a corresponding increase in (MHD) equations, the NIMROD vide global communication within a scientific productivity. team is providing the fusion com- single time-step. The preconditioned The NIMROD project, funded munity with a flexible, sophisticated conjugate gradient (CG) solver that by the DOE Office of Fusion tool which can lead to improved has been used was the most compu- Energy Sciences and the SciDAC understanding of these phenomena, tationally demanding part of the Center for Extended Magneto- ultimately leading to a better ap- algorithm, so Carl Sovinec of the hydrodynamic Modeling (CEMM), proach to harnessing fusion energy. NIMROD team consulted with the is developing a modern computer Since the beginning of the project, SciDAC Terascale Optimal PDE code suitable for the study of long- the NIMROD code has been devel- Simulations (TOPS) Center to find wavelength, low-frequency, non- oped for massively parallel compu- a replacement. linear phenomena in fusion reactor tation, enabling it to take full Conversations with David Keyes plasmas. These phenomena involve advantage of the most powerful of Old Dominion University, Dinesh large-scale changes in the shape computers to solve some of the Kaushik of Argonne National and motion of the plasma and largest problems in fusion. The Laboratory, and Sherry Li and severely constrain the operation of team’s primary high-end computing Esmond Ng of Lawrence Berkeley fusion experiments. CEMM also resource is Seaborg at NERSC. National Laboratory pointed to supports a complementary plasma NIMROD’s algorithm requires SuperLU as a possibly more effi- simulation code, AMRMHD, solution of several large sparse cient matrix solver for NIMROD. which uses different mathematical matrices in parallel at every time SuperLU is a library of software for

20 ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy

FIGURE 2 Full 3D numerical simulation of plasma particle drift orbits in a tokamak. From solving nonsymmetric sparse linear sional, nonlinear tokamak simula- Charlson C. Kim, Scott E. Parker, and systems in parallel using direct tions (Figure 2), which require the NIMROD team, “Kinetic particles in methods. supercomputing resources, NIM- the NIMROD fluid code,” 2003 Sherwood Fusion Theory Conference, Corpus It took less than a month to ROD with SuperLU runs four to Christi, Texas. implement, test, and release five times faster than before. This SuperLU into the full NIMROD improved efficiency yields four to production code, and the perform- five times more physics results for ance improvements were dramatic. the same amount of time on the For two-dimensional linear calcula- computer—a major improvement tions of MHD instabilities, NIMROD in scientific productivity. runs 100 times faster with SuperLU The nonlinear simulations accu- than it does with the CG solver. mulate relatively small changes in While linear calculations do not the matrix elements at each time require supercomputer resources, step, but there are no changes to they are extremely useful for pre- the sparsity pattern. The NIMROD tions, but it is still very significant. liminary explorations that help implementation allows SuperLU to The net result is that computa- determine which cases require in- reuse its factors repeatedly until the tionally demanding simulations of depth nonlinear simulations. For matrix elements accumulate a sig- macroscopic MHD instabilities in linear problems, this performance nificant change and refactoring tokamak plasmas, which until now improvement makes NIMROD becomes necessary. Refactoring were considered too difficult, have competitive with special-purpose makes the performance improve- become routine. linear codes. ment less dramatic in nonlinear For cutting-edge three-dimen- simulations than in linear calcula- Research funding: FES, SciDAC

ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy 21

AMR METHODS ACCELERATE MHD SIMULATIONS

The annual Supercomputing conference held every November is well known as a sort of international watering hole, where many of the world’s leading experts in high-performance computing gather for a week to take stock of the competition, exchange ideas, and make new connections.

At SC2000 in Dallas, Phil Colella, informal collaboration which was oped in conjunction with Princeton head of the Applied Numerical soon formalized under the auspices researcher Ravi Samtaney and Algorithms Group at Lawrence of SciDAC—Jardin is principal Berkeley Lab researcher Terry Berkeley National Laboratory, and investigator for the Center for Ligocki, is already producing new Steve Jardin, co-leader of the Extended Magnetohydrodynamic physics results as well. It powered Computational Plasma Physics Modeling (CEMM), while Colella the first simulation demonstrating Group at Princeton Plasma Physics is PI for the Applied Partial that the presence of a magnetic Laboratory, were both scheduled to Differential Equations Center field will suppress the growth of the give talks in the Berkeley Lab (APDEC). Jardin’s group was able Richtmyer-Meshkov instability booth. Colella discussed “Adaptive to incorporate the CHOMBO adap- when a shock wave interacts with a mesh refinement research and soft- tive mesh refinement (AMR) code contact discontinuity separating ion- ware at NERSC,” where Jardin has developed by Colella’s group into a ized gases of different densities. The conducted his scientific computing new fusion simulation code, which upper and lower images in Figure 3 for years. Jardin gave a presentation is now called the Princeton contrast the interface without on “A Parallel Resistive MHD AMRMHD code. “Using the AMR (upper) and with (lower) the mag- Program with Application to code resulted in a 30 times netic field. In the presence of the Magnetic Reconnection.” improvement over what we would field, the vorticity generated at the The scientist and the mathe- have had with a uniform mesh code interface is transported away by the matician got to talking between of the highest resolution,” Jardin fast and slow MHD shocks, remov- their presentations and one thing says. ing the driver of the instability. led to another. They began an The AMRMHD code, devel- Results are shown for an effective

22 ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy

FIGURE 3 a1 Stabilization of Richtmyer-Meshkov instability by a magnetic

t = t1 B = 0 field. From R. Samtaney, “Suppression of the Richtmyer- Meshkov instability in the presence of a magnetic field,” Princeton Plasma Physics Laboratory report PPPL-3794 (submitted to Phys. Fluids). a2

B ≠ 0

b1 t = t 2 B = 0 1.669 1.713 1.732

b2

B ≠ 0 1.760 1.876 1.942

c1

t = t3 B = 0

1.978 2.253 2.593 FIGURE 4 c2 Current bunching and ejection during magnetic reconnec- tion. From R. Samtaney, S. Jardin, P. Colella, and T. Ligocki, B ≠ 0 “Numerical simulations of magnetic reconnection using AMR,” Sherwood Fusion Theory Conference, Rochester, NY, April 2002. mesh of 16,384 × 2,048 points collaborating with other SciDAC that M3D was already using to call which took approximately 150 software centers in addition to the PETSc solvers. The combined hours to run on 64 processors of APDEC. For example, in a collabo- PETSc-Hypre solver library allows Seaborg—25 times faster than a ration that predated SciDAC, the M3D to solve its linear systems two non-AMR code. group developing the M3D code to three time faster than before. Another new physical effect dis- was using PETSc, a portable toolkit According to Jardin, fusion plas- covered by the AMRMHD code is of sparse solvers distributed as part ma models are continually being current bunching and ejection dur- of the ACTS Collection of DOE- improved both by more complete ing magnetic reconnection (Figure developed software tools. Also in the descriptions of the physical process- 4). Magnetic reconnection refers to ACTS Collection is Hypre, a library es, and by more efficient algo- the breaking and reconnecting of of preconditioners that can be used rithms, such as those provided by oppositely directed magnetic field in conjunction with PETSc. Under PETSc and CHOMBO. Advances lines in a plasma. In the process, SciDAC, the Terascale Optimal such as these have complemented magnetic field energy is converted to PDE Solvers (TOPS) Center increases in computer hardware plasma kinetic and thermal energy. worked with CEMM to add Hypre speeds to provide a capability today The CEMM project has been underneath the same code interface that is vastly improved over what

ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy 23

s 6 10 lop FIGURE 5

gaf Micro-turbulence “Effective speed” increases in fusion Global MHD

gi effective speed codes came from both faster hardware 5 effective speed 10 and improved algorithms. Image: Steve ent Jardin. val Full Earth Improved Simulator electron Improved qui 4 10 (Japan) models linear solvers d in e d in ee 3 Higher-order 10 Delta-f, elements d sp magnetic Semi- ine 1000 NERSC coordinates implicit 2 SP3 processors sta 10 (typical)

su Gyro- Effective speed

ive kinetics from hardware 1 improvements ect 10 16 processor alone Cray C90 Eff Partially- implicit 0 Cray YMP 10 1970 1980 1990 2000 2010 Calendar Ye ar

was possible 30 years ago (Figure 5). microscopic turbulent transport in and to the proposed burning plasma This rate of increase of effective the smaller fusion experiments that experiments such as ITER [the capability is essential to meet the exist today, at least for short times,” International Thermonuclear anticipated modeling demands of Jardin says. “Anticipated increases Experimental Reactor].” fusion energy research, Jardin says. in both hardware and algorithms “Presently, we can apply our during the next five to ten years most complete computational mod- will enable application of even Research funding: FES, SciDAC els to realistically simulate both more advanced models to the For further discussion of AMR in nonlinear macroscopic stability and largest present-day experiments computational science, see page 42.

24 ADVANCES IN COMPUTATIONAL SCIENCE / Fusion Energy

BIOLOGY DECIPHERING NATURE’S ASSEMBLY MANUALS

Now that a working draft of the human genome has been created, scientists are busy identifying the genes within the sequences of those 3 billion DNA base pairs of nucleotides. The complete genomes of many other organisms—from microbes to plants to animals—are also available, with new genomes being completed every week. Scientists are now facing the opportunity to understand not just the human body but the fundamental processes of life itself.

Once a gene’s boundaries have been established, the next steps are to deter- mine the structures of the proteins encoded in the gene, then to determine the functions of those proteins. NERSC users and collaborators are developing innovative computational tools to help determine protein structures and func- tions, as described in this section. These tools bring us closer to the goal of utilizing genomic discoveries to improve human health and to protect and restore the environment.

ADVANCES IN COMPUTATIONAL SCIENCE / Biology 25

PROTEINSHOP: SOLVING PROTEIN STRUCTURES FROM SCRATCH

ProteinShop, a computer visualization tool for manipulating protein structures, is helping biologists close in on one of their cherished goals: completely determining an unknown protein’s shape from its gene sequence.

Silvia Crivelli of the Visualization gene sequences gave rise to the colleagues, including Crivelli from Group in Berkeley Lab’s Compu- unique CASP scientific competi- Berkeley Lab and Bobby Schnabel, tational Research Division says a tion. Teams of biologists and com- Richard Byrd, and Betty Eskow major step forward came when “we puter scientists from around the from the University of Colorado. copied concepts from robotics. world try to beat one another to the Instead of depending heavily on When you move a robot’s arm, you fastest and most accurate predic- knowledge of protein folds (3D sets move all the joints, like your real tions of the structures of recently of structures) already stored in the arm.” After a year of work on found but as-yet unpublished pro- Protein Data Bank (PDB), the ProteinShop, Crivelli says, “we were teins. The number of competitors Global Optimization Method uses able to apply the same mathematical has more than quintupled, from initial protein configurations provid- techniques to protein structures.” fewer than three dozen groups in ed by ProteinShop, which was cre- It’s just one of the components 1994 to 187 in 2002’s CASP5. ated by Wes Bethel and Crivelli of the ProteinShop vehicle uses to At CASP5, a team led by Teresa Berkeley Lab along with Nelson help competitors in the hottest race Head-Gordon of Berkeley Lab’s Max of Lawrence Livermore in biological computation—the Physical Biosciences Division, who National Laboratory and the “Grand Prix of bioinformatics” is also Assistant Professor of Bio- University of California at Davis, known as CASP, the Critical engineering at the University of and UC Davis’s Bernd Hamann Assessment of Structure Prediction. California at Berkeley, employed and Oliver Kreylos. In 1994, the challenge of predicting the Global Optimization Method With a ProteinShop jumpstart, finished protein structures from developed by Head-Gordon and her Head-Gordon’s CASP5 team was

26 ADVANCES IN COMPUTATIONAL SCIENCE / Biology

a. b. c.

d. e. f.

FIGURE1 sheet, forming hydrogen bonds between e. Now the user selects the central helix SETUP PHASE OF THE GLOBAL selected amino acid residues from each and manipulates it to reduce interfer- OPTIMIZATION METHOD. strand. In the yellow coil region, inverse ence (where the orange spheres mark a. ProteinShop reads an unknown pro- kinematics calculates the correct atom collisions). Meanwhile, inverse tein’s string of amino acids and dihedral angles as the strands are kinematics calculates the angles in assembles the predicted secondary bent together. the flanking coil regions, and structures: in this case, “uncoiled” c. To assemble a second beta sheet, ProteinShop keeps all structures coils alternate with beta strands and ProteinShop semiautomatically aligns intact while the helix is moved. a central alpha helix. Even for large hydrogen bonds in the third and fourth f. The final 3D protein structure, ready proteins, this step is virtually instan- beta strands. for optimization. taneous. d. The two partial sheets are pulled to- Images: Oliver Kreylos b. The user tells ProteinShop to align gether and stabilized by hydrogen bonds two beta strands into a partial beta between beta strands one and four.

able to predict the configurations of reduce attractive and repulsive plus those that form the flat strands 20 new or difficult protein folds forces among the linked amino that arrange themselves side by side in ranging from 53 to 417 amino acids acids. The result is local regions beta sheets. This secondary-structure in length, which, says Crivelli, “is where the energy required to main- information can be stored in data great for a method that doesn’t use tain the structure is minimized. banks and applied to unknown pro- much knowledge from the PDB.” The secondary structures swiftly tein folds. A protein’s shape is what deter- fold up into a 3D tertiary structure; But the so-called coil regions mines its function. Yet a one-dimen- for most proteins this represents the that link secondary structures in an sional string of amino acid residues, “global” minimum energy for the unknown protein are not so easily as specified by a gene’s coding structure as a whole. (Some compli- predicted. The more amino acids in sequences, does not on the face of it cated proteins like hemoglobin, the protein, the more local energy reveal much about 3D shapes. assembled from distinct components, minima there are, and the harder it Swimming in a watery environment, have a quaternary structure as well.) becomes to calculate the global the amino-acid string—the protein’s By studying thousands of known energy minimum. primary structure—quickly twists proteins, biologists have learned to The Global Optimization into familiar shapes known as alpha recognize sequences of amino acids Method tackles the problem in two helices and beta sheets, striving to that are likely to form alpha helices, distinct phases. During the setup

ADVANCES IN COMPUTATIONAL SCIENCE / Biology 27

phase (Figure 1a–f), the program at each joint, contortions that don’t of signals. In one mode, little generates likely secondary structures break limbs or penetrate bodies can orange spheres pop up when the from a given sequence of amino acids, be predicted. user has forced atoms to collide: the which are combined into several “The difference is that with a more energy-expensive the colli- approximate configurations. During robot you have maybe 10 or 20 sion, the bigger the orange sphere. the next phase, the overall energy of joints, but in a coil we often have Manipulation can also be simplified the structure is reduced step by step. long regions, 80 amino acids,” by hand-selecting residues for auto- Global optimizations are performed Crivelli says, “and we want all of the matic bonding. on subsets of the angles among the dihedral angles among them to move The second phase of the Global amino acid residues (dihedral angles) in a concerted way.” In ProteinShop, Optimization Method—seeking the which are predicted to make up coil the secondary structures and coils global energy minimum—remains a regions. Eventually, because the are built up by adding amino acids computational challenge. The basic energy function is not perfect, sev- to the structure one at a time, treat- approach to finding the optimal eral tertiary structures are proposed. ing each as a jointed segment of a energy for a tertiary structure is to Before ProteinShop, the setup flexible structure. Within seconds a repeatedly tweak the energy budgets phase of the Global Optimization “geometry generator” module incor- of many smaller regions, incorporat- Method was immensely time-con- porates predicted secondary struc- ing each improvement until there is suming. Because a protein’s primary tures, or fragments of them, in the no further change. This “physics- structure contains dozens or hun- string. “The whole thing looks like based” method depends only on dreds of acid residues, thus thou- you could just move it around like energy calculations, not on knowl- sands of atoms, the initial phase spaghetti,” says Crivelli. “But before edge of existing folds. required computer runs of hours or we incorporated reverse kinematics, In practice, to do such calcula- days before settling on secondary if you tried to move a protein con- tions for every part of a structure structures. Moreover, while the pre- figuration, it broke.” would take much too long. The diction of alpha helices was straight- Now the process works fast sophisticated procedures developed forward, beta sheets were hard to enough to be truly interactive, by Head-Gordon’s team optimize assemble from beta strands, and allowing the user to alter the dihedral only randomly sampled, small their configurations were less certain. angles between individual amino regions. Thus it’s not certain that In CASP4, in 2000, the team pre- acids. In difficult cases like the the end result is the true global dicted the structures of only eight assembly of beta strands into sheets, optimum. Even so, global optimiza- folds, the longest containing 240 the user can manipulate the confor- tion of an unknown protein of mod- amino acids. mation to achieve the best, least erate size may require weeks of The Visualization Group devel- energetic fit. Moreover, the user computer time. oped ProteinShop in preparation for can play with the entire “preconfig- “We are working out new CASP5. ProteinShop incorporates uration,” dragging whole secondary methodology that will combine our the mathematical technique called assemblies into new relationships physics-based approach with knowl- inverse kinematics—well known not without breaking previous structures. edge-based methods,” says Crivelli. only in robotics but also in video “Once we have the alpha and beta “By recognizing structures and frag- gaming. Inverse kinematics is used structures, we want to leave them ments that are known to work, we to predict the overall movements of alone,” Crivelli says. “Mainly we won’t have to calculate every angle structures consisting of jointed seg- work on the coil regions.” from scratch. The tool will be highly ments, for example the fingers, arms, The ProteinShop program interactive, displaying energies and and shoulders of a robot or an ani- allows for various styles of illustrat- saving minima as the user finds them. mated figure. By taking into account ing the fold and its parts, and it It will be organized as a guided the degrees of freedom permissible helps out the user by sending plenty search through a dynamically evolving

28 ADVANCES IN COMPUTATIONAL SCIENCE / Biology

tree, basing new structures on previ- high-performance parallel-process- combination of local and global ous ones that have been shown to ing codes. The results will have optimization approaches, such as work for the fold.” implications for scientific problems transportation and scheduling, To enable this high degree of far beyond the successful solution structural optimization, chip design, interaction, still higher performance of unknown protein structures. and machine learning. will be required from the Global Other complex optimization prob- Optimization Method’s already lems might be amenable to this Research funding: BER,ASCR-MICS, NSF

MAPPING THE UNIVERSE OF PROTEIN STRUCTURES

The protein universe, the vast assemblage of biological molecules that are the building blocks of living cells and control the chemical processes which make those cells work, has finally been mapped.

α/β Researchers at Berkeley Lab and the University of California at Berkeley have created the first 3D global map of the protein structure universe (Figure 2). This map provides important insight into the evolution and demo- graphics of protein structures and may help scientists identify the functions of newly discovered proteins. Sung-Hou Kim, a chemist with a β joint appointment at Berkeley Lab and UC Berkeley, led the develop- ment of this map. An internationally α+β recognized authority on protein structures, he expressed surprise at α how closely the map, which is based solely on empirical data and a math- FIGURE 2 This 3D map of the protein structure universe shows the distribution in space of the ematical formula, mirrored the widely 500 most common protein fold families as represented by spheres. The spheres, used Structural Classification System which are colored according to classification, reveal four distinct classes: α (red), β of Proteins (SCOP), which is based (yellow), α+β (blue), and α/β (cyan). It can be seen that the α, β, and α+β classes share a common area of origin, while the α/β class of protein structures is shown to on the visual observations of scien- be a late bloomer on the evolutionary flagpole. From J. Hou, G. E. Sims, C. Zhang, tists who have been solving protein and S.-H. Kim, “A global representation of the protein fold space,” PNAS 100, 2386 structures. (2003).

ADVANCES IN COMPUTATIONAL SCIENCE / Biology 29

“Our map shows that protein With the completion of a work- tein folds are discovered each year, folds are broadly grouped into four ing draft of the human genome, in despite the fact that the number of different classes that correspond to which scientists determined the protein structures determined annu- the four classes of protein structures sequences of the 3 billion DNA ally is increasing exponentially,” defined by SCOP,” Kim says. “Some bases that make up the human Kim says. “This and other observa- have argued that there are really genome, the big push now is to tions strongly suggest that the total only three classes of protein fold identify coding genes and the number of protein folds is dramati- structures, but now we can mathe- molecular and cellular functions cally smaller than the number of matically prove there are four.” of the proteins associated with genes.” Protein folds are recurring struc- them. Coding genes are DNA The rationale behind this idea is tural motifs or “domains” that sequences that translate into that through the eons, proteins have underlie all protein architecture. sequences of amino acids, which selectively evolved into the architec- Since architecture and function go RNA assembles into proteins. tural structures best suited to do hand-in-hand for proteins, solving The prevailing method for pre- their specific jobs. These structures what a protein’s structure looks like dicting the function of a newly dis- essentially stay the same for proteins is a big step towards knowing what covered protein is to compare the from all three kingdoms of life— that protein does. sequence of its amino acids to the bacteria, archaea, and eukarya— The 3D map created by Kim amino acid sequences of proteins even though the DNA sequences along with Jingtong Hou, Gregory whose functions have already been encoding for a specific type of protein Sims, and Chao Zhang, shows the identified. A major problem with can vary wildly from the genome of distribution in space of the 500 relying exclusively on this approach one organism to another, and some- most common protein fold families is that while proteins in different times even within the same organism. as represented by points which are organisms may have similar struc- In the map created by Kim and spatially separated in proportion to ture and function, the sequences of his colleagues, elongated groups of their structural dissimilarities. The their amino acids may be dramati- fold distributions approximately cor- distribution of these points reveals a cally different. “This is because pro- responding to the four SCOP struc- high level of organization in the fold tein structure and function are much tural classifications can be clearly structures of the protein universe more conserved through evolution seen. These classifications, which and shows how these structures have than genetically based amino acid are based on secondary structural evolved over time, growing increas- sequences,” Kim says. compositions and topology, are the ingly larger and more complex. Kim has been a leading advocate “alpha” helices, “beta” strands, and “When the structure of a new for grouping proteins into classes on two mixes of helices and strands, one protein is first solved, we can place the basis of their fold structures and called “alpha plus beta” and the it in the appropriate location on the using these structural similarities to other “alpha slash beta.” The map map and immediately know who its help predict individual protein reveals that the first three groups neighbors are and its evolutionary functions. While the protein uni- share a common area of origin, pos- history, which can help us predict verse may encompass as many as a sibly corresponding to small primor- what its function may be,” Kim says. trillion different kinds of proteins on dial proteins, while the “alpha slash “This map provides us with a con- Earth, most structural biologists beta” class of proteins does not ceptual framework to organize all agree there are probably only about emerge until much later in time. protein structures and functions and ten thousand distinctly different “It is conceivable that, of the have that information readily avail- types of folds. primordial peptides, those contain- able in one place.” “A smaller number of new pro- ing fragments with high helix

30 ADVANCES IN COMPUTATIONAL SCIENCE / Biology

and/or strand propensity found their the map developed by Kim and his For the next phase of this way to fold into small alpha, beta, colleagues holds promise for a num- research, Kim and his colleagues and alpha-plus-beta structures,” ber of areas of biology and biomed- plan to tap into NERSC’s super- Kim says. “The alpha-slash-beta fold ical research, including the design computers to add the rest of the structures do not appear until pro- of more effective pharmaceutical some 20,000 and counting known teins of sufficient size rose through drugs that have fewer side effects. protein structures to their map. evolution and the formation of “This map can be used to help They also plan to set up a Web site supersecondary structural units design a drug to act on a specific where researchers can submit for became possible.” protein and to identify which other inclusion new protein structures Since understanding the molec- proteins with similar structures they have solved. ular functions of proteins is key to might also be affected by the drug,” understanding cellular functions, Kim says. Research funding: BER, NIH, NSF

ADVANCES IN COMPUTATIONAL SCIENCE / Biology 31

DOE FACILITY PLAN SEES MAJOR ROLE FOR COMPUTING

society,” said Abraham. “With this nersc.gov/news/HECRTF-V4- plan our goal is to keep the United 2003.pdf). States at the scientific forefront.” An upgrade of the NERSC facil- Priority 2 in the plan is an Ultra- ity is tied for Priority 7. This upgrade Scale Scientific Computing will ensure that NERSC, DOE’s Capability (USSCC), to be located premier scientific computing facility at multiple sites, which would for unclassified research, continues increase by a factor of 100 the com- to provide high-performance com- puting capability available to support puting resources to support the open scientific research. Most exist- requirements of scientific discovery. ing supercomputers have been NERSC will continue to pro- designed with the consumer market vide the core scientific computing in mind; USSCC’s new configura- needed by the research community, tions will develop computing capa- and will complement the “grand bility specifically designed for science challenge” approach pursued under and industrial applications. These the USSCC, according to the facility facilities will operate like Office of plan. Reaffirming the goals of the Science light sources: available to NERSC Strategic Proposal for all, subject to proposal peer review. 2002–2006, the plan describes the In a speech at the National The USSCC will involve long- NERSC upgrade as using Grid Press Club on November 10, 2003, term relationships with U.S. com- technology to deploy a capability U.S. Energy Secretary Spencer puter vendors and an integrated designed to meet the needs of an Abraham outlined the Department program of hardware and software integrated science environment of Energy’s Office of Science 20-year investments designed to optimize combining experiment, simulation, science facility plan, a roadmap for computer performance for scientific and theory by facilitating access to future scientific facilities to support and commercial problems by creat- computing and data resources, as the department’s basic science and ing a better balance of memory size, well as to large DOE experimental research missions. The plan priori- processor speed, and interconnection instruments. NERSC will concen- tizes new, major scientific facilities rates. A very similar strategy was trate its resources on supporting sci- and upgrades to current facilities. spelled out in the white paper that entific challenge teams, with the “This plan will be the corner- NERSC together with seven other goal of bridging the software gap stone for the future of critical fields DOE labs submitted to the High between currently achievable and of science in America. These facili- End Computing Revitalization Task peak performance on the new tera- ties will revolutionize science—and Force in June 2003 (see http://www. scale platforms.

32 ADVANCES IN COMPUTATIONAL SCIENCE / DOE Facility Plan

NANOSCIENCE MODELING A REALM WHERE EVERTHING IS DIFFERENT

The nanoscale—between 10 and 100 billionths of a meter—is the scale at which the fundamental properties of materials and systems are established. Melting temperature, magnetic properties, charge capacity, and even color are dictated by the arrangement of nanoscale structures. The realm of molecular biology also operates largely at the nanoscale.

While much is known about the physical properties and behavior of isolated molecules and bulk materials, the properties of matter at the nanoscale cannot necessarily be predicted from those observed at larger or smaller scales. Nanoscale structures exhibit important differences that cannot be explained by traditional models and theories.

Research at the nanoscale aims to provide a fundamental understanding of the unique properties and phenomena that emerge at this length scale as well as the ability to tailor those properties and phenomena to produce new materials and technologies in fields as diverse as electronics, biotechnology, medicine, trans- portation, agriculture, environment, and national security. But a quantitative understanding of nanoscale systems requires development of new computational methods. This section focuses on one application central to DOE’s mission: pro- moting clean and efficient energy production.

ADVANCES IN COMPUTATIONAL SCIENCE / Nanoscience 33

MAKING FUEL CELLS MORE EFFICIENT AND AFFORDABLE

Fuel cells sound like an ideal power source—they are efficient and quiet, have few moving parts, run on hydrogen, and emit only water vapor.

It’s no wonder that President Bush circuit, creating a usable electric Finding more efficient and proposed new research into hydro- current. Meanwhile, on the cath- affordable catalysts is one of the gen-powered automobiles in his ode side, oxygen molecules (O2) are goals of a research group led by 2003 State of the Union address. being separated by the catalyst into Perla Balbuena, Associate Professor But for this dream to become reali- separate atoms in a process called of Chemical Engineering at the ty, scientists and engineers need to oxygen reduction. The negatively University of South Carolina. Using make fuel cells less expensive. charged oxygen atoms (adsorbed on computational chemistry and While there are several types of the catalyst) attract the positively physics techniques, they are trying fuel cells, proton exchange mem- charged hydrogen ions through the to predict the properties of elec- brane (PEM) cells are the leading PEM, and together they combine trodes consisting of bimetallic and candidates for powering light-duty with the returning electrons to form trimetallic nanoparticles embedded vehicles and buildings and for water molecules (H2O). in a polymer system and dispersed replacing batteries in portable elec- While splitting the hydrogen on a carbon substrate. They are tronic devices. The PEM is a poly- molecules is relatively easy for plat- working to identify the atomic distri- mer electrolyte that allows hydrogen inum-alloy catalysts, splitting the bution, vibrational modes, and dif- ions (protons) to pass through it. As oxygen molecules is more difficult. fusion coefficients of a variety of hydrogen (H2) is fed to the anode Although platinum catalysts are the alloy nanoparticles. side of the fuel cell, a catalyst (usu- best known for this task, they are “The rationale behind the use ally platinum or a platinum-rutheni- still not efficient enough for wide- of alloy nanoparticles,” Balbuena um alloy) encourages the hydrogen spread commercial use in fuel cells, says, “is that each of the elements, atoms to separate into electrons and and their high price is one of the or particular combinations of them, protons. The electrons travel from reasons why fuel cells are not yet may catalyze different aspects of the anode to cathode along an external economically competitive. reaction. The large surface-to-volume

34 ADVANCES IN COMPUTATIONAL SCIENCE / Nanoscience

area of nanoparticles may also con- tribute to their effectiveness.” Experimental data indicates that alloys can be as effective or better than platinum catalysts, but nobody fully understands why this is so. Little is known about the physical, chemical, and electrical properties of alloy nanoparticles, and in par- ticular about the atomic distribu- tion of the various elements on the catalyst surface, which is difficult to determine experimentally. Compu- tational studies play an essential role in answering these questions. For example, Figure 1 shows a test simulation of a system involving only platinum nanoparticles on a carbon substrate and surrounded by The researchers concluded that FIGURE 1 Platinum nanoparticles (gold-colored a hydrated polymer. The goal of this cobalt, nickel, and chromium make atoms) are deposited on a graphite sup- simulation was to test the stability of platinum-based complexes more port. The long chains are Nafion frag- the nano-size catalyst in such an reactive because they enable a high- ments composed of fluorine (green), environment—that is, how the poly- er number of unpaired electrons. carbon (blue), oxygen (red), and sulfur (yellow) atoms; some of these chains mer groups, water, and substrate Their computations yielded exten- become adsorbed over the metallic influence the shape and mobility of sive new information that would be clusters. The background contains water the nanoparticles, and how they very hard to obtain experimentally, molecules (red for oxygen and white for hydrogen). Nafion, widely used for pro- influence the exposed surfaces where such as showing for the first time ton exchange membranes, has both the oxygen molecules will be adsor- how alloying results in changes in hydrophilic and hydrophobic chemical bed and dissociated into oxygen atoms. the density of electronic states, and groups, so some degree of water segre- gation is observed. Image: Eduardo J. Balbuena and her colleagues how those changes influence the Lamas. recently developed a new computa- oxygen reduction process. One of tional procedure to investigate the the questions they addressed was behavior of active catalytic sites, whether nickel, cobalt, and chromi- including the effect of the bulk um might become oxidized and act environment on the reaction. They as “sacrificial sites” where other demonstrated this procedure in sim- species can adsorb, leaving the plat- ulations of oxygen dissociation on inum sites available for oxygen dis- platinum-based clusters alloyed with sociation. Their study shows that cobalt, nickel, or chromium, and this may be the case for nickel, but embedded in a platinum matrix. To that alternative mechanisms are determine the optimal alloy, they possible, especially for cobalt and studied a variety of compositions chromium, which instead of being and geometries for alloy ensembles just sacrificial sites, act as active embedded into the bulk metal sites where oxygen dissociates, in (Figure 2). some cases more efficiently than on

ADVANCES IN COMPUTATIONAL SCIENCE / Nanoscience 35

platinum sites. They found that the of tens of nanometers, as well as most effective active sites that pro- developing a full understanding of mote oxygen dissociation are the role of the rest of the environ-

O2CoPt, O2Co2Pt, O2CrPt, and ment (polymer, water, and sub-

O2Cr2Pt, while ensembles involving strate) on the outcome of the reac- nickel atoms are as catalytically tion. The ultimate goal of these effective as pure platinum. studies is to enable the design of The Balbuena group’s research new catalysts from first principles is now focusing on the time evolu- rather than trial-and-error experi- tion of the complete reaction on the ments, reducing the cost of catalysts best catalytic ensembles, including for fuel cells through the use of the combination of oxygen atoms optimum alloy combinations requir- with protons leading to water forma- ing only minimal amounts of tion. Future work will address the expensive materials. possibilities of reducing further the catalyst size, nowadays in the order Research funding: BES

FIGURE 2 Oxygen molecules (red) adsorb on a bimetallic surface of platinum (purple) and cobalt (green). The oxygen mole- cules most frequently adsorb on “bridge” sites composed of two cobalt atoms or one cobalt and one platinum atom. The oxygen molecules will subse- quently dissociate into individual atoms. From P. B. Balbuena, D. Altomare, L. Agapito, and J. M. Seminario, “Theoretical analysis of oxygen adsorp- tion on Pt-based clusters alloyed with Co, Ni, or Cr embedded in a Pt matrix,” J. Phys. Chem. B (in press).

36 ADVANCES IN COMPUTATIONAL SCIENCE / Nanoscience

CLIMATE MODELING PREDICTING THE RESULTS OF A GLOBAL EXPERIMENT

Ever since the beginning of the industrial era, when humans began dramatically increasing the amount of carbon dioxide in the atmosphere by burning large quantities of fossil fuels, we have unintentionally been conducting a massive experiment on the Earth’s climate. Unfortunately, this uncontrolled experiment does not meet the criteria of the scientific method—we do not have a parallel “control” planet Earth where we can observe how the climate would have evolved without the addition of CO2 and other greenhouse gases. One of the goals of computational climate modeling is to provide such control scenarios so that human contributions to climate change can be quantified. This section describes how climate researchers studied the rising boundary between the tro- posphere and stratosphere and determined that human activity is the principal cause.

ADVANCES IN COMPUTATIONAL SCIENCE / Climate Modeling 37

FINGERPRINTS IN THE SKY

For a physicist and atmospheric scientist, Ben Santer of Lawrence Livermore National Laboratory has an unusual specialty—he is a fingerprint expert.

Like a police detective, Santer looks cations of this “fingerprinting” tech- driven by the warming of the tropo- for unique identifying marks that nique recently answered the ques- sphere by greenhouse gases and the can be used to determine responsi- tion of why the tropopause has been cooling of the stratosphere by ozone bility. But instead of looking for rising for more than two decades. depletion.1 But to quantify the criminals, Santer is looking for spe- The tropopause is the boundary influences (or “forcings” in climate cific, quantitative evidence of between the lowest layer of the jargon) even further, they consid- human influence on changes in the atmosphere—the turbulently mixed ered three anthropogenic forcings— Earth’s climate. The “fingerprints” troposphere—and the more stable well-mixed greenhouse gases, sul- he tries to match are patterns of data stratosphere. The “anvil” often seen fate aerosols, and tropospheric and from observations of the atmosphere at the top of thunderclouds is a visi- stratospheric ozone—as well as two and from computer simulations. ble marker of the tropopause, which natural forcings—changes in solar “We examine the output from the lies roughly 10 miles above the irradiance and volcanic aerosols— computer models and compare this Earth’s surface at the equator and 5 all of which are likely to influence with observations,” Santer says. “In miles above the poles. tropopause height.2 the model world (unlike the real The average height of the The observational data were world!) we have the luxury of being tropopause rose about 200 meters compared with seven simulation able to change one factor at a time, between 1979 and 1999. In their experiments using the Parallel while holding the others constant. first comparison of observed data Climate Model (PCM). In the first This allows us to isolate and quanti- with computer models, Santer and five experiments, only one forcing fy the effect of individual factors.” his colleagues concluded that the at a time was changed. In the sixth One of the most dramatic appli- increase in tropopause height was experiment, all five forcings were

1 B. D. Santer, R. Sausen,T. M. L.Wigley, J. S. Boyle, K.AchutaRao, C. Doutriaux, J. E. Hansen, G.A. Meehl, E. Roeckner, R. Ruedy, G. Schmidt, and K. E.Taylor,“Behavior of tropopause height and atmospheric temperature in models, reanalyses, and observations: Decadal changes,”J. Geophys. Res. 108, 4002 (2003).

38 ADVANCES IN COMPUTATIONAL SCIENCE / Climate Modeling

se ght rea FIGURE 1 Hei –5 A Combined forcings inc Effect of different forcings on tropopause height. Image: B.D. Santer, y (hPa) M. F. Wehner, T. M. L. Wigley et al.

0 se mal ght

SV rea 5 ALL El Chichón Hei

Ano NCEP Agung ERA dec Santa Maria Pinatubo 10 –6 B Individual forcings –4 –2 0 G (well-mixed GHGs) 2 A (sulfate aerosols) O (ozone) 4 y (hPa) S (solar) ALL 6 V (volcanos) SUM

mal 8 1900 1920 1940 1960 1980 2000 Ano included simultaneously. The sev- ing the stratosphere, but this effect enth experiment used only the two is short-lived. The analysis con- natural forcings. To improve the cludes that about 80% of the rise in reliability of the results, each experi- tropopause height is due to human ment was performed four times activity. The model-predicted fin- using different initial conditions from gerprint of tropopause height a control run. This methodology changes was statistically detectable allowed the researchers to estimate in two different observational data the contribution of each forcing to sets, NCEP and ERA (Figure 2). overall changes in atmospheric tem- Michael Wehner of Berkeley perature and tropopause height. Lab’s Computational Research The results of the analysis Division, a co-author of the Science showed that in the mid-1980s, the paper, points out that it would have combined five forcings (ALL) been very difficult to estimate the diverged from the natural forcings relative contributions of different (SV), showing the dominance of forcing mechanisms without the human influences (Figure 1A). Of very large ensembles of climate the five forcings, greenhouse gases model experiments conducted by (G) and ozone (O) played the DOE-funded researchers at the biggest role (Figure 1B). Major vol- National Center for Atmospheric canic eruptions can be seen to Research and made possible by lower tropopause height sharply by NERSC and other DOE and NSF cooling the troposphere and warm- supercomputing facilities. Wehner

2 B. D. Santer, M. F.Wehner,T. M. L.Wigley, R. Sausen, G.A. Meehl, K. E.Taylor, C.Ammann, J.Arblaster,W. M.Washington, J. S. Boyle, and W. Brüggemann,“Contributions of anthro- pogenic and natural forcing to recent tropopause height changes,”Science 301, 479 (2003).

ADVANCES IN COMPUTATIONAL SCIENCE / Climate Modeling 39

manages the largest single collec- ANCEP linear trend (1979–1999) B ERA linear trend (1979–1993) tion of publicly available climate model output, the DOE Coupled Climate Model Data Archive (http://www.nersc.gov/projects/gcm_ data/), which is stored in NERSC’s HPSS system. Linear trend (hPa/decade) Linear trend (hPa/decade) –5 –3 –1 1 3 5 –5 –3 –1 1 3 5 Looking at the tropopause data –6 –4 –2 0 2 4 6 –6 –4 –2 0 2 4 6 as part of the bigger climate picture, C PMC ALL linear trend (1979–1999) D PMC ALL EOF 1 (mean included) Santer said, “Tropopause height is an integrated indicator of human- induced climate change. It reflects global-scale changes in the temper- ature structure of the atmosphere. It is also consistent with results from Linear trend (hPa/decade) EOF loading other studies that have identified –5 –3 –1 1 3 5 –1.5 –0.9 –0.3 0.3 0.9 1.5 anthropogenic fingerprints in a –6 –4 –2 0 2 4 6 –1.8 –1.2 –0.6 0 0.6 1.2 1.8 range of different climate variables, E PMC ALL EOF 1 (mean removed) such as ocean heat content, surface temperature, sea-level pressure pat- terns, and Northern Hemisphere FIGURE 2 sea-ice extent.” Linear trends over time (A,B,C) and fin- gerprints (D,E) of tropopause pressure “The challenge for the future,” (inversely related to tropopause height). Santer added, “is to take a closer Image: B.D. Santer, M. F. Wehner, T. M. L. EOF loading –2.5 –1.5 –0.5 0.5 1.5 2.5 Wigley et al. look at the internal consistency of –3 –2 –1 0 1 2 3 all these climate change variables. Our present work suggests that this story is a consistent one, but there are lots of details that need to be worked out.” Santer received the Department of Energy’s 2002 Ernest Orlando Lawrence Award for “his seminal and continuing contributions to our understanding of the effects of human activities and natural phenomena on the Earth’s climate system.”

Research funding: BER, NOAA

40 ADVANCES IN COMPUTATIONAL SCIENCE / Climate Modeling

APPLIED MATHEMATICS THE BRIDGE BETWEEN PHYSICAL REALITY AND SIMULATIONS

“Mathematics is the bridge between physical reality and computer simulations of physical reality. The starting point for a computer simulation is a mathemati- cal model, expressed in terms of a system of equations and based on scientific understanding of the problem being investigated, such as the knowledge of forces acting on a particle or a parcel of fluid.” 1

One of the challenges facing many model developers is managing model com- plexity. Many physical systems have processes that operate on length and time scales that vary over many orders of magnitude. These systems require multi- scale modeling, which represents physical interactions on multiple scales so that the results of interest can be recovered without the unaffordable expense of representing all behaviors at uniformly fine scales. This section describes the award-winning work of two research groups specializing in multiscale modeling.

1 “A Science-Based Case for Large-Scale Simulation,”report to the DOE Office of Science, July 30, 2003, p. 31.

ADVANCES IN COMPUTATIONAL SCIENCE / Applied Mathematics 41

ALGORITHMS IN THE SERVICE OF SCIENCE

John Bell and Phillip Colella, applied mathematicians at Lawrence Berkeley National Laboratory and long-time NERSC users, were named co-recipients of the 2003 SIAM/ACM Prize in Computational Science and Engineering, awarded by the Society for Industrial and Applied Mathematics (SIAM) and the Association for Computing Machinery (ACM).

According to SIAM President Mac porous media, and combustion. applying it in areas such as beam Hyman, the prize “is awarded in the Bell is head of the Center for dynamics in accelerator designs, gas area of computational science in Computational Sciences and jets in laser-driven plasma-wakefield recognition of outstanding contribu- Engineering (CCSE), while Colella accelerators, simulations of fusion tions to the development and use of heads the Applied Numerical reactors, and biological cell modeling. mathematical and computational Algorithms Group (ANAG), both In order to deal with problems tools and methods for the solution within Berkeley Lab’s Computa- in such diverse and complex sci- of science and engineering prob- tional Research Division. Both ences, CCSE and ANAG create lems.” This is the first year the prize groups are participants in the customized software components has been awarded. SciDAC Applied Partial Differential that are able to translate scientific Algorithms developed by Bell, Equations Center (APDEC). ideas and empirical insights into Colella and their research groups at In 2003, CCSE has been mod- precise, 3D models that can be rep- Berkeley Lab are used for studying eling turbulent reacting flow in resented on a computer. This complex problems arising in fluid methane flames, and applying some involves computational research mechanics and computational of the information gleaned in those that bridges the gap between mathe- physics. The mathematical con- experiments toward supernova matical, physical and computation- cepts, models, and methods they research and modeling. ANAG has al sciences. By breaking up each have developed have been applied in been involved in developing more problem into its fundamental math- such diverse areas as shock physics, sophisticated geometric tools for its ematical parts, and designing opti- turbulence, astrophysics, flow in Chombo software toolkit, and mal algorithms to handle each

42 ADVANCES IN COMPUTATIONAL SCIENCE / Applied Mathematics

piece, they are able to model a wide models that can be used to analyze creates a problem size that demands range of scientific phenomena. flame properties and characteristics some hefty computing power. In describing the cell modeling of turbulence. Even with a simple In order to create the flame work he recently started with fuel like methane, there are some visualizations, computations were Berkeley Lab biologists, Colella 20 chemical species and 84 reac- performed on 2,048 processors of explained that “in order to get bet- tions involved; the reaction zone, NERSC’s IBM SP, “Seaborg.” ter predictive models, you have to where the adaptive mesh is concen- CCSE has created its own data keep trying things out. The process trated, occurs in an area that is 150 analysis framework with specific becomes a very experimental, microns, while the whole grid con- algorithms to reformulate equations empirical activity, and that’s why sumes 12 centimeters. All of this and accelerate the computations. the software tools are so important here.” The software tools are based on adaptive mesh refinement (AMR), a grid-based system that uses partial differential equations to describe information in the most efficient manner possible—applying a fine mesh, and maximum computing power, to the areas of most interest, and a coarse mesh to areas where data is not as significant. AMR is particularly useful in problems where there is a wide range of time scales, and adaptive methods must be used to track movements in the mesh over time. One of Bell and CCSE’s main projects in 2003 has been creating 3D simulations of turbulent methane combustion with swirling flow, which has produced the first detailed simulations in this area of combustion science (Figure 1). In FIGURE 1 Instantaneous flame surface from a tur- the course of the project, two things bulent premixed combustion simulation. have been of primary interest: how CCSE carried out the calculation using turbulence in the fuel stream affects a low Mach number flow model with an AMR solution algorithm. The coarsest, the chemistry, and how emissions base grid is invisible but fills the out- are formed and released. By study- lined cube. This cutaway view displays ing the reacting flow of turbulent as green and pink boxes the containers flames, CCSE has been able to that hold two successive levels of refined computational cells. These containers develop computational models that are dynamically reconfigured to follow dissect the detailed chemical reac- the evolving flame surface, and are dis- tions occurring in the flame. The tributed over the parallel processors of the supercomputer. Image: Marc Day. group has also developed predictive

ADVANCES IN COMPUTATIONAL SCIENCE / Applied Mathematics 43

The result is a computational cost researchers to frame and calculate of cells is governed by a complex that is reduced by a factor of 10,000 the problems. network of reacting chemical compared with a visualization using To a great degree, CCSE and species that are transported within a a standard uniform-grid approach. ANAG rely on their own computa- cell and among cellular communi- While a fundamental goal of tional scientists to apply the soft- ties. ANAG is working with the the project is a better understanding ware to new projects and problem Arkin Laboratory in Berkeley Lab’s of turbulence in flames, it could sets. But one of their goals has been Physical Biosciences Division and also have practical ramifications, as to develop software components UC Berkeley’s Bioengineering and in engineering new designs for that other scientists can utilize more Chemistry departments to develop more efficient turbines, furnaces, easily. the software infrastructure to turn incinerators, and other equipment “We really want to support this data into quantitative and pre- held to strict emission require- geometry and we need to support lots dictive models. These models will ments. A side benefit of the project of different applications in different contribute to a deeper understand- is that it is stretching the group’s ways,” said Colella. “So how can we ing of cellular regulation and devel- software to the limit, putting new re-factor the software to do that? For opment, more efficient discovery of demands on the algorithmic and one, we’re learning a lot about soft- cellular networks, and a better abili- geometric elements of the toolset. ware engineering and managing the ty to engineer or control cells for “The methane flame happens to development process, using a ‘spiral medical, agricultural, or industrial be the one application that encom- design’ approach. When we add a purposes. passes most of the pieces of the soft- new capability and develop applica- The California Department of ware. They are all folded into this tions in it, we get it out there for Water Resources has also asked for application,” said Marc Day, a staff review. As we get feedback, we ANAG’s help in creating high-quali- scientist at CCSE. “You have to make changes to the software and ty simulation software for modeling have applied math and computing repeat the process over again, so it water flows from the Sacramento hardware, together with good com- becomes an ongoing spiral.” Delta to the San Francisco Bay putational science, in order to Another one of the group’s goals (Figure 2). These models are valu- accomplish what we have. Without in software development is to create able for researchers interested in any one of those pieces, it doesn’t algorithmic tools for doing compu- better understanding fresh water work.” tations in complex geometries. So if flows, and other environmental and The methane flame project falls a problem demands hyperbolic, water quality issues. into a class of problems called elliptic, and parabolic solver com- One of the reasons these mathe- “interface dynamics,” where some- ponents, each of those pieces can matical software tools are so unique thing happens to cause an instabili- be integrated while preserving the and powerful is that they can be ty in an interface between two flu- integrity of the model as a whole. applied to a wide array of scientific ids of different chemicals and densi- Both groups’ software develop- research, in a remarkably flexible ty. This instability causes the inter- ment is an ongoing process, and fashion. face to grow and change form. The new capabilities are constantly After analyzing the results of the modeling goal then becomes to being added to make them more methane flame project, for instance, track the evolution of the new inter- flexible and applicable in broader CCSE scientists started to apply some face. AMR is able to tackle these arenas of science. of this newfound knowledge toward types of problems, where sophisti- Computational biology is one of supernova research. Scientists have cated and powerful tools are needed the major growth areas, said long been interested in fluid to scale and quantify the complicat- Colella. Current research is creat- dynamics and energy transfers in ed energy dynamics. Equally impor- ing enormous amounts of data on supernovae, and trying to determine tant is a skilled, dedicated group of how the development and behavior how and why they burn out. Using

44 ADVANCES IN COMPUTATIONAL SCIENCE / Applied Mathematics

FIGURE 2. One of the goals of the ANAG software algorithms developed during its flame combustion turn into a nuclear effort is generating grids for complex turbulence project, CCSE created flame. So we were able to make sig- geometries. The images here show a cut-cell grid generated to compute flow 3D models of supernovae that shed nificant progress over a short period in the San Francisco Bay. At grid cells new light on this problem, and have of time in the area of supernova intersecting the bottom boundary, the produced insights into the various physics because we already had this numerical representation of the bound- ary is colored by the z-coordinate (i.e., stages of supernova explosions. stuff laying around.” depth) of the boundary in each cell. The “You wouldn’t think that super- All of this makes you wonder left image is of the entire bay, the right novae have anything in common what else CCSE and ANAG have one of the area around the Golden Gate. with Bunsen burners, but algorith- lying around, and where it might Image: Mike Barad (UC Davis) and Peter Schwartz (Berkeley Lab). mically they were very similar,” said lead them next. Day. “There was just a small num- ber of pieces that we had to change Research funding:ASCR-MICS, SciDAC, out in order to make the chemical NASA, CDWR

ADVANCES IN COMPUTATIONAL SCIENCE / Applied Mathematics 45

RESEARCH NEWS This section of the Annual Report provides a repre- sentative sampling of recent discoveries made with the help of NERSC’s computing, storage, networking, and intellectual resources. News items are organized by science category, with funding organizations indi- cated by acronyms in blue type. Organizational acronyms are spelled out in Appendix G.

ACCELERATOR PHYSICS

0.23 (a) OPTIMIZING LUMINOSITY IN ELECTRON CLOUDS AND )

HIGH-ENERGY COLLIDERS PARTICLE ACCELERATOR p

PERFORMANCE ω The beam-beam interaction puts a z(c/ strong limit on the performance of Electron clouds have been shown to –0.23 most high-energy colliders. The be associated with limitations in –0.00375y(c/ωp) 0.00375 electromagnetic fields generated by particle-accelerator performance in one beam focus or defocus the beam several of the world’s largest circular 04Density (cm–3) ×109 moving in the opposite direction, proton and positron machines. 0.23 (b)

reducing luminosity and sometimes Rumolo et al. have studied the ) p

causing beam blowup. Qiang et al. interaction between a low-density ω

performed the first-ever million- electron cloud in a circular particle z(c/ particle, million-turn, “strong-strong” accelerator with a circulating –0.23 (i.e., self-consistent) simulation of charged particle beam. The particle –0.00375y(c/ωp) 0.00375 colliding beams (in the Large Hadron beam’s space charge attracts the

Collider). Instead of a conventional cloud, enhancing the cloud density –4×107 Density (cm–3) 0 particle-mesh code, they used a near the beam axis (Figure 1). This method in which the computational enhanced charge and the image FIGURE 1 mesh covers only the largest of the charges associated with the cloud (a) Positively charged beam density. (b) The corresponding cloud density. This two colliding beams, allowing them charge and the conducting wall of figure shows cloud compression on the to study long-range parasitic colli- the accelerator may have important axis of the beam. sions accurately and efficiently. consequences for the dynamics of the beam propagation. J. Qiang, M. Furman, and R. Ryne,“Strong- strong beam-beam simulation using a G. Rumolo,A. Z. Ghalam,T. Katsouleas, C. K. cloud effects on beam evolution in a cir- Green function approach,”Phys. Rev. ST Huang,V. K. Decyk, C. Ren,W. B. Mori, F. cular accelerator,”Phys. Rev. ST AB 6, AB 5, 104402 (2002). HEP,SciDAC Zimmermann, and F.Ruggiero,“Electron 081002 (2003). HEP,SciDAC, NSF

RESEARCH NEWS 47

DEVELOPING FUTURE trum and for the three main com- PLASMA-BASED ponents of the secondary yield, ACCELERATORS namely, the true secondary, redif- fused, and backscattered components. High-gradient acceleration of both positrons and electrons is a prereq- M.T. F.Pivi and M.A. Furman,“Electron uisite condition to the successful cloud development in the Proton Storage development of a plasma-based Ring and in the Spallation Neutron linear collider. Blue et al. provided Source,”Phys. Rev. ST AB 6, 034201 simulation support for the E-162 (2003). HEP,SNS Plasma Wakefield Accelerator experiment at the Stanford Linear Accelerator Center. This work was APPLIED MATHEMATICS successful at demonstrating the high- gradient acceleration of positrons in a meter-scale plasma for the first HIGH-PERFORMANCE OPTIMIZING MEMORY time. Excellent agreement was SPARSE-MATRIX HIERARCHY found between the experimental ALGORITHMS The BeBOP toolkit automatically results and those from 3D particle- Sparse-matrix problems are at the tunes scientific codes by applying in-cell simulations for both energy heart of many scientific and engi- statistical techniques to produce gain and loss. neering applications of importance highly tuned numerical subroutines. B. E. Blue, C. E. Clayton, C. L. O’Connell, to DOE and the nation. Some Vuduc et al. used the BeBOP tool F.-J. Decker, M. J. Hogan, C. Huang, applications are fusion energy, SPARSITY to investigate memory R. Iverson, C. Joshi,T. C. Katsouleas, accelerator physics, structural hierarchy issues with such sparse W. Lu, K.A. Marsh,W. B. Mori, P.Muggli, R. dynamics, computational chem- matrix kernels as Ax (matrix-vector Siemann, and D.Walz,“Plasma-wakefield istry, and groundwater simulations. products) and AT Ax (normal equa- acceleration of an intense positron Teranishi et al. studied the effect of tion products). They evaluated opti- beam,”Phys. Rev. Lett. 90, 214801 data mapping on the performance of mizations on a benchmark set of 44 (2003). HEP,SciDAC, NSF triangular solutions. They showed matrices and four platforms, show- that a careful choice of data map- ing speedups of up to a factor of 4.2. ELECTRON CLOUD ping, coupled with the use of selec- R.Vuduc,A. Gyulassy, J. Demmel, and K. DEVELOPMENT IN PROTON tive inversion, can significantly Yelick,“Memory hierarchy optimizations STORAGE RINGS reduce the amount of communica- and performance bounds for sparse AT tion required in the solution of a Pivi and Furman have simulated Ax.”U.C. Berkeley Technical Report sparse triangular linear system. electron cloud development in the UCB/CS-03-1232, February 2003.ASCR- Proton Storage Ring (PSR) at Los K.Teranishi, P.Raghavan, and E. G. Ng, MICS, SciDAC, NSF,Intel Alamos National Laboratory and “Reducing communication costs for par- the Spallation Neutron Source allel sparse triangular solution,”in MODELING THE COMPONENT (SNS) under construction at Oak Proceedings of the SC2002 Conference STRUCTURES OF FLAMES Ridge National Laboratory. Their (2002).ASCR-MICS, SciDAC, NSF Combustion in turbulent flows may simulation provides a detailed take the form of a thin flame description of the secondary electron wrapped around vortical structures. emission process, including a refined Marzouk et al. used the flame- model for the emitted energy spec- embedding approach to decouple

48 RESEARCH NEWS

computations of the “outer” non- IMPROVING THE ACCURACY J. Glimm, X.-L. Li,Y.-J. Liu, Z. L. Xu, and N. reacting flow and the combustion OF FRONT TRACKING IN Zhao,“Conservative front tracking with zone by breaking the flame surface FLUID MIXING improved accuracy,”SIAM J. Sci. Comp. (in into a number of elemental flames, press).ASCR-MICS, FES,ARO, NSF,LANL, Acceleration-driven fluid-mixing each incorporating the local impact NSFC instabilities play important roles in of unsteady flow-flame interaction inertially confined nuclear fusion, (Figure 2). Results show that using PREDICTING THE as well as in stockpile stewardship. the flame leading-edge strain rate ENERGIES OF QUANTUM Turbulent mixing is a difficult and gives reasonable agreement in cases WELL STATES centrally important issue for fluid of low-strain-rate and weak-strain-rate dynamics that affects such questions Understanding the electronic struc- gradient within the flame structure. as the rate of heat transfer by the tures of quantum-well states This agreement deteriorates sub- Gulf Stream, resistance of pipes to (QWSs) in metallic thin films can stantially when both are high. fluid flow, combustion rates in lay the groundwork for designing Agreement between the 1D model automotive engines, and the late- new materials with predictable and the 2D simulation improves time evolution of a supernova. To properties. An et al. compared the greatly when the actual strain rate provide a better understanding of experimental electronic properties at the reaction zone of the 1D these instabilities, Glimm et al. of QWSs in a copper film grown on flame is made to match that of the have developed a new, fully conser- a cobalt substrate with ab initio sim- 2D flame. vative front tracking algorithm that ulations. The calculations confirm Y. M. Marzouk,A. F.Ghoniem, and H. N. results in one-order-of-accuracy the concept of the quantization Najm,“Toward a flame embedding model improvement over most finite differ- condition inherent in the phase- for turbulent combustion simulation,”AIAA ence simulations and shows good accumulation model (PAM) to Journal 41, 641 (2003).ASCR-MICS agreement with experimental results. predict the energies of QWSs as a

FIGURE 2 Premixed flame interaction with a counter-rotating vortex pair. The vertical right-hand edge of each frame is the centerline of the vortex pair, only half of which is shown. Colored contours indi- cate gas temperature, with darker shad- ing corresponding to burned combus- tion products. Solid/dashed contours delineate levels of positive/negative 0.0 ms 2.0 ms 4.0 ms 6.0 ms 8.0 ms vorticity.

RESEARCH NEWS 49

function of their thickness, and to ASTROPHYSICS provide new insight into their nature. The results also show that band structures and reflection phases CALCULATING ELECTRON the problem. Klein et al. have simu- obtained from either experiment or CAPTURE RATES DURING lated the gravitational collapse and ab initio theory can quantitatively CORE COLLAPSE fragmentation of marginally stable predict QWS energies within the turbulent molecular cloud cores Supernova simulations to date have PAM model. assumed that during core collapse, J. M.An, D. Raczkowski,Y. Z.Wu, C.Y.Won, electron captures occur dominantly L.W.Wang,A. Canning, M.A.Van Hove, E. on free protons, while captures on Rotenberg, and Z. Q. Qiu,“The quantization heavy nuclei are Pauli blocked and condition of quantum-well states in are ignored. Langanke et al. have Cu/Co(001),”Phys. Rev. B 68, 045419 calculated rates for electron capture (2003).ASCR-MICS, NSF on nuclei with mass numbers A = 65–112 for the temperatures and REDUCED-DENSITY densities appropriate for core col- MATRICES FOR ELECTRONIC lapse. They found that these rates STRUCTURE CALCULATIONS are large enough so that, in contrast to previous assumptions, electron Accurate first-principles electronic capture on nuclei dominates over structure calculations are of special capture on free protons. This result importance for spectroscopy and for leads to significant changes in core chemical-reaction dynamics. collapse simulations. Reduced-density matrices (RDM) that have linear scaling for large N K. Langanke, G. Martinez-Pinedo, J. M. offer a first-principles alternative to Sampaio, D. J. Dean,W. R. Hix, O. E. B. density-functional and semi-empiri- Messer,A. Mezzacappa, M. Liebendorfer, cal calculations for large systems. H.-Th. Janka, and M. Rampp,“Electron Zhao et al. reformulated the existing capture rates on nuclei and implications RDM method from a primal to a dual for stellar core collapse,”Phys. Rev. Lett. formulation, substantially reducing 90, 241102 (2003). NP,SciDAC, DRA, the computational cost. They com- JIHIR, MCYT, NASA, NSF,PECASE puted the ground state energy and dipole moment of 47 small mole- SIMULATING BINARY STAR cules and molecular ions, both FORMATION open and closed shells, achieving Developing a comprehensive theory results more accurate than those of star formation remains one of the from other approximations. most elusive goals of theoretical Z. Zhao, B. Braams, M. Fukuda, M. Overton, astrophysics, partly because gravita- and J. Percus,“The reduced density matrix tional collapse depends on initial method for electronic structure calcula- conditions within giant molecular tions and the role of three-index repre- cloud cores that only recently have sentability,”J. Chem. Phys. (submitted, been observed with sufficient accu- 2003).ASCR-MICS, NSF racy to permit a realistic attack on

50 RESEARCH NEWS and follo mass fragments as they interact with the radiation of the protostars form- ing at their centers. Their results show excellent agreement with observations of real cores, including their aspect ratios, rotation rates, linewidth-size relations, and forma- tion of binary systems (Figure 3).

R. I. Klein, R.T. Fisher, M. R. Krumholz, and C. F.McKee,“Recent advances in the col- lapse and fragmentation of turbulent molecular cloud cores,”RevMexAA 15, 92 (2003). NP, NASA

LOOKING DOWN THE HOLE IN A SUPERNOVA

Type Ia supernovae are born from a white dwarf accreting material from a nearby companion star. Soon after the white dwarf explodes, the ejected material runs over the companion star, and in the interaction a sub- stantial asymmetry can be imprinted on the supernova ejecta. Kasen et al. performed the first-ever calcula- tion of a 3D atmosphere model of a Type Ia colliding with its companion star. They explored the observable consequences of ejecta-hole asym- metry, using the simulation results to explain the diversity of observed supernovae.

D. Kasen, P.Nugent, R.Thomas, and L. Wang,“Filling a hole in our understanding of Type Ia supernovae,”Astrophysical Journal (in press). NP,NASA.

FIGURE 3 Simulation of the birth of a binary star system. Each member of the binary appears to be surrounded by a disk, with one component of the binary show- ing a two-arm spiral. The binary appears to form inside the structure of a fila- ment in the cloud core.

PROBING THE CHEMICAL E. Baron, P.Nugent, D. Branch, P. experiment and found that the EVOLUTION OF THE Hauschildt, M.Turatto, and E. Cappellaro, effect on CMB power spectrum UNIVERSE “Determination of primordial metallicity observations is negligible. and mixing in the Type IIP supernova Type II supernovae have a very large 1993W,”Astrophysical Journal 586, 1199 A. H. Jaffe,A. Balbi, J. R. Bond, J. Borrill, spread in their intrinsic brightness— (2003). NP,IBM SUR, NASA, NSF P.G. Ferreira, D. Finkbeiner, S. Hanany, greater than a factor of 500—because A.T. Lee, B. Rabii, P.L. Richards, G. F. of their diverse progenitors. Despite CHECKING THE FORE- Smoot, R. Stompor, C. D.Winant, and this diversity, Baron et al. have GROUND OF THE COSMIC J. H. P.Wu,“Determining foreground con- shown that their atmospheres can MICROWAVE BACKGROUND tamination in CMB observations: Diffuse be well understood with detailed galactic emission in the MAXIMA-I field,” synthetic spectral modeling that can Observations of the cosmic Astrophysical Journal (in press); astro- determine the stellar compositions, microwave background (CMB) can ph/0301077. HEP,NASA, NSF degree of mixing, and kinetic ener- be contaminated by diffuse fore- gy of the explosion. These results ground emission from sources such SIMULATING BLACK make Type II supernovae attractive as galactic dust and synchrotron HOLE MERGERS AND probes of chemical evolution in the radiation, but an international team GRAVITATIONAL WAVES Universe and potentially useful of researchers has developed a tech- independent checks on the cosmo- nique for quantifying this effect. Simulations of the gravitational waves logical results derived from Type 1a They applied this technique to resulting from the collision of two supernovae. CMB data from the MAXIMA-1 black holes will provide patterns

RESEARCH NEWS 51

that new gravitational wave detec- CHEMICAL SCIENCES tors can look for as they attempt to verify the predictions of General Relativity. A research team led by CALCULATING THE INSTA- prodigious basis sets and high-order Edward Seidel of the Albert BILITY OF NITROGEN SALTS correlation treatments for accurate Einstein Institute/Max Planck predictions. Kenny et al. calculated Institute for Gravitation Physics in Can the combination of a stable molecular partial-wave expansions Potsdam, Germany, has determined polynitrogen cation, such as N5+, for the T-shaped and linear forms of effective computational and theo- with a stable polynitrogen anion, SiC2 using the MP2-R12/A method. retical (gauge) parameters needed such as N3–, lead to a stable ionic Their results are the first of sufficient to carry out binary black hole evolu- nitrogen allotrope, such as N5+N3–? accuracy to call for revisitation of tions for time scales never before Dixon et al. used high-level ab ini- the spectroscopically determined achieved. Another research group, led tio molecular orbital theory to con- barrier of 5.4 ± 0.6 kcal mol–1, mov- by Carlos Lousto of the University of clusively determine whether the lat- ing it upward to 6.3 kcal mol–1. tice energies of N +N – and N +N – Texas at Brownsville, has achieved a 5 3 5 5 J. P.Kenny,W. D.Allen, and H. F.Schaefer, are sufficient to stabilize them as full numerical computation of the “Complete basis set limit studies of con- solids. Their calculations showed gravitational radiation generated by ventional and R12 correlation methods: that neither salt can be stabilized the evolution of a series of binary The silicon dicarbide (SiC ) barrier to lin- and that both should decompose 2 black hole configurations with earity,”J. Chem. Phys. 118, 7353 (2003). spontaneously into N radicals and aligned and counter-aligned spins. 3 BES, SciDAC N2. This conclusion was experimen- M.Alcubierre, B. Bruegmann, P.Diener, M. tally confirmed for the N5+N3– salt. Koppitz, D. Pollney, E. Seidel, and R. HOW ELECTRON IMPACT Takahashi,“Gauge conditions for long- EXCITES A MOLECULE term numerical black hole evolutions with- D.A. Dixon, D. Feller, K. O. Christe,W.W. A major goal of studying electron- out excision,”Phys.Rev.D 67, 084023 Wilson,A.Vij,V.Vij, H. D. Brooke Jenkins, R. molecule collisions is to explore the (2003). NP,AEI, EU, LSU M. Olson, and M. S. Gordon,“Enthalpies of mechanisms that control the flow of formation of gas phase N3, N3–, N5+, and J. Baker, M. Campanelli, C. O. Lousto, and energy from electronic to nuclear N5– from ab initio molecular orbital theory, R.Takahashi,“The final plunge of spin- degrees of freedom. McCurdy et al. stability predictions for N5+N3– and N5+N5–, ning binary black holes,”Phys.Rev.D have completed the first calculation and experimental evidence for the insta- (submitted); astro-ph/0305287 (2003). on vibrational excitation of a mole- bility of N5+N3–,”J.Am. Chem. Soc. (in NP,NASA, NRC, NSF cule (CO ) via coupled resonant press). BES, BER, SciDAC, DARPA,AFOSR, 2 states (Figure 4). The multidimen- NSF sional nuclear dynamics on these coupled surfaces was necessary to A MORE ACCURATE explain quantitatively, for the first CALCULATION OF time, the experimental observations SILICON DICARBIDE on this system. These methods have Unlike many typical chemical sys- been applied in calculations on tems, in which errors due to incom- electron-impact dissociation of plete treatment of the one-electron water, which is one of the principal basis and electron correlation are bal- mechanisms suspected to initiate anced and largely cancelled at modest radiation damage of biological tis- levels of theory, silicon dicarbide has sue in environments with ionizing a barrier to linearity that requires both radiation.

52 RESEARCH NEWS

2 2 A1 B1 Re(Eres) Re(Eres) 0.3 0.45 0.4 0.25 0.35 0.2 0.3 0.15 0.25 0.1 0.2 0.05 0.15 0 0.1

3.8 3.8 4 2 4 2 4. 4.4 4. 4 2 C-O 4.6 8 2 C-O4. 4.6 4. 5 40 4.8 5.2 10 20 30 5 (bo 5.4 (bo 5.25.4 5.6 -40 -30 -20 -10 gle 5.6 -40 -30 -20 gle hr) Ben0 d an hr) -10Ben0 d 10an 20 30 40

2 2 A1 B1 2 Im(Eres) 2 Im(Eres) 0.18 0.25 0.16 0.14 0.2 0.12 0.1 0.15 0.08 0.06 0.1 0.04 0.05 0.02 0 0

3.8 3.8 4 4 4.2 2 4.4 4. 4.4 2 C-O 4.6 8 2 C-O 4.6 8 4. 5 40 4. 5 5.2 10 20 30 (bo 5.4 (bo 5.25.4 5.6 -40 -30 -20 -10 gle 5.6 -40 -30 -20 -10 gle hr) Ben0 d an hr) Ben0 d10 an 20 30 40

FIGURE 4 Complex potential surfaces for both components of the resonance. Top row, real part of resonance energies; bottom row, widths. Energies are in hartrees, and the bend angle, defined as π minus the O—C—O bond angle, is in degrees.

C.W. McCurdy,W.A. Isaacs, H.-D. Meyer, and diameters in which they pulled Cummings,“Oscillatory behavior of double and T. N. Rescigno,“Resonant vibrational the inner nano-tube out of the outer nanotubes under extension:A simple excitation of CO2 by electron impact: nanotube to several different dis- nanoscale damped spring,”Nano Letters Nuclear dynamics on the coupled compo- tances and allowed the nanotube to 3, 1001 (2003). BES, NSF 2 nents of the Πu resonance,”Phys. Rev.A retract and subsequently oscillate. 67, 042708 (2003). BES They found that the oscillation is SOLVING THE SCHRÖDINGER damped (hence acting like a nano EQUATION FOR LARGE DESIGNING NANO-SCALE shock absorber) and can be predict- SYSTEMS SHOCK ABSORBERS ed by a very simple mechanical Exact solutions of the Schrödinger model that takes into account the Carbon nanotubes are the funda- equation within a given one-particle sliding friction between the sur- mental building blocks in many basis set, lie at the heart of quantum faces. The oscillation was around 1 nanosystems. Rivera et al. performed chemistry, providing invaluable GHz, consistent with prior experi- molecular dynamics computational benchmarks by which the accuracy mental and theoretical calculations. “experiments” on double-walled of more approximate methods may carbon nanotubes of varying lengths J. L. Rivera, C. McCabe, and P.T. be judged. Chan and Head-Gordon

RESEARCH NEWS 53

used a newly developed density OXYGEN DISSOCIATION ON out quantum dynamical simulations matrix renormalization group algo- COPPER STEPS for complex molecular systems. The rithm to compute exact solutions of efficiency of the new formulation Copper and copper alloy catalysts are the Schrödinger equation for water was demonstrated by converged at the heart of important industrial at two geometries in a basis of 41 quantum dynamical simulations for processes, and copper also plays an orbitals, demonstrating that it is systems with a few hundred to a important role in the microelectron- now possible to obtain numerically thousand degrees of freedom. ics industry. Dissociation of dioxy- exact solutions for systems consider- gen (O2) is the first step of oxygen H.Wang and M.Thoss,“Multilayer formu- ably larger than can be contemplat- chemistry on the surface of metals lation of the multiconfiguration time- ed using traditional full configura- such as copper. Xu and Mavrikakis dependent Hartree theory,”J. Chem. Phys. tion interaction algorithms. studied the adsorption and dissocia- 119, 1289 (2003). BES, NSF,DAAD G. K. L. Chan and M. Head-Gordon,“Exact tion of dioxygen on copper steps solution (within a triple zeta, double polar- using periodic self-consistent density ADDING HYDROGEN ization basis set) of the electronic functional theory calculations, and TO LEAN PREMIXED Schrödinger equation for water,”J. Chem. found that the adsorption of both NATURAL GAS Phys. 118, 8551 (2003). BES atomic and molecular oxygen is Hawkes and Chen performed direct enhanced on the stepped surface. numerical simulation of lean pre- QUANTUM MONTE CARLO Y. Xu and M. Mavrikakis,“The adsorption mixed methane-air and hydrogen- STUDY OF THE ELECTRONIC and dissociation of O2 molecular precur- methane-air flames to understand EXCITED STATES sors on Cu:The effect of steps,”Surface how the addition of hydrogen can Accurate computational predictions Science 538, 219 (2003). BES, DOD, NSF, affect flame stability and pollutant of molecular electronic excited 3M formation in gas turbines. They states have proved more difficult to found a greater rate of heat release obtain than ground states. Ethylene A NEW METHOD FOR WAVE- in the enriched flame, which is is the prototypical π-electron system PACKET PROPOGATION explained by a difference in flame whose photochemical behavior is areas and influences of turbulent Wang and Thoss have presented a important in chemistry, biology, stretch on the local burning rates. method for wave-packet propagation and technology. El Akramine et al. They found that there may be a pol- that is based rigorously on a varia- carried out a theoretical study of the lutant tradeoff between NO and tional principle and is also sufficiently transition between the ground state CO emissions when using hydrogen- efficient to be applicable to complex and the lowest triplet state of ethyl- enriched fuels. dynamical problems with up to a few ene using the diffusion Monte hundred degrees of freedom. This E. R. Hawkes and J. H. Chen,“Turbulent Carlo method. The ground state method can treat substantially more stretch effects on hydrogen enriched lean atomization energy and heat of for- physical degrees of freedom than the premixed methane-air flames,”3rd Joint mation were consistent with experi- original method, and thus signifi- Meeting of the U.S. Sections of the Combus- mental results, and the triplet-state cantly enhances the ability to carry tion Institute, Chicago, IL (2003). BES atomization energy and heat of for- mation were also predicted.

O. El Akramine,A. C. Kollias, and W.A. Lester, Jr.,“Quantum Monte Carlo study of the singlet-triplet transition in ethylene,” J. Chem. Phys. 119, 1483 (2003). BES

54 RESEARCH NEWS

CLIMATE CHANGE RESEARCH

A MILLENNIUM OF experiments represents the most model the interaction between coarse- CLIMATE CHANGE extensive set of 20th century forcing scale and fine-scale phenomena. climate simulations ever attempted. The method implies improved cli- Researchers at the National Center mate and turbulence simulations by for Atmospheric Research continued A. Dai,A. Hu, G.A. Meehl,W. M.Washington, allowing increased resolution in a their benchmark 1,000-year Com- and W. G. Strand,“North Atlantic Ocean few dynamically significant areas, munity Climate System Model circulation changes in a millennial control such as fronts or regions of dense (CCSM) control run and extended run and projected future climates,”Journal topography. Similarly, regional cli- the Parallel Climate Model (PCM) of Climate (in press). BER, SciDAC, NSF mate simulations may be improved control run to nearly 1,500 years by allowing regional resolution to be (Figure 5). Science experiments with MULTIRESOLUTION incorporated within a global model the PCM included historical 20th CLIMATE MODELING in a two-way interactive fashion. century climate simulations using various combinations of specified Baer and colleagues have developed F.Baer and J. J.Tribbia,“Sensitivity of greenhouse gases, atmospheric sul- the Spectral Element Atmospheric atmospheric prediction models to ampli- fate concentrations, solar and vol- Model (SEAM) to incorporate local tude and phase,”Tellus (submitted, 2003). canic forcing, and tropospheric and mesh refinement within a global BER, SciDAC stratospheric ozone. This set of model. This makes it feasible to

Ye ar 75-84, 170–199, Spin-up Run, Top 100m 20. cm/s a PCM Control Run, Top 100m 20. cm/s c 65°N 65°N

60°N 60°N

55°N 55°N

50°N 50°N 20 0 45°N 45°N –20 40°N 40°N –40 –60 35°N 35°N –80 60°W50°W40°W30°W20°W 10°W0°E 60°W50°W40°W30°W20°W 10°W0°E –100 Ye ar 80-84, 1070–1099, –120 PCM Control Run, Top 100m 20. cm/s b PCM Control Run, Top 100m 20. cm/s d 65°N 65°N –140 –160 60°N 60°N –180

55°N 55°N

50°N 50°N

45°N 45°N

40°N 40°N

35°N 35°N 60°W50°W40°W30°W20°W 10°W0°E 60°W50°W40°W30°W20°W 10°W0°E

FIGURE 5 Sea surface height (color, in cm relative to a motionless ocean) and top 100m- averaged ocean currents (arrows) in the North Atlantic Ocean simulated by the PCM (a) at the end of the ocean-sea ice alone run (right before the coupling), and during the control run at (b) years 80–84, (c) 170–199, and (d) 1070–1099.

RESEARCH NEWS 55

SORTING OUT THE SOURCES transport model. They found that al. have obtained the first results from OF TROPOSPHERIC OZONE climate forcing by soot and smoke a fully coupled, non-flux-corrected, was not as large as previously multi-century climate-carbon simu- Ozone in the troposphere is both claimed, because the adjustment of lation. They have performed three formed in situ and transported from the atmospheric temperature and complete simulations: a control case the stratosphere, which has much clouds to aerosols tends to decrease with no emissions, a standard case, higher ozone levels. The importance the forcing, particularly if the and a case where the uptake of CO of each source may vary both region- 2 aerosols reside at higher levels with- by vegetation is limited. The limited ally and seasonally. Atherton et al. in the atmosphere. case shows that substantially more analyzed the results of a simulation CO stays in the atmosphere when in Phoenix from April 1 through J. E. Penner, S.Y. Zhang, and C. C. Chuang, 2 carbon uptake by vegetation is October 31, 2001. Their results “Soot and smoke aerosol may not warm reduced, thus giving a range of showed that there are periods of climate,”J. Geophys. Res., in press uncertainty in current carbon-climate time during the “ozone season” in (2003). BER modeling. which higher levels of surface ozone may be due to transport of S.Thompson, B. Govindasamy, J. Milovich, CARBON-CYCLE FEEDBACK stratospheric ozone from above, A.A. Mirin, and M.Wickett,“A comprehen- IN CLIMATE CHANGE rather than created by ground-level sive GCM-based climate and carbon cycle (and energy-related) processes. Taking into account the feedback model:Transient simulations to year 2100 effects of climate change on absorp- using prescribed greenhouse gas emis- C.Atherton, D. Bergmann, P.Cameron- tion of carbon by the ocean and ter- sions,”Amer. Geophys. Union Mtg., San Smith, P.Connell, D. Rotman, and J. restrial biosphere will allow better Francisco, CA, December 2002; Lawrence Tannahill,“Large scale atmospheric predictions of future climate; however, Livermore National Laboratory technical chemistry simulations for 2001:An analy- these effects are ignored in present abstract UCRL-JC-149761. BER, LLNL sis of ozone and other species in central U.S. climate models. Thompson et Arizona,”American Meteorological Society 83rd Annual Meeting, Long Beach, California, February 2003. BER

DO SOOT AND SMOKE COMPUTER SCIENCE CONTRIBUTE TO CLIMATE CHANGE? REDUCING THE applied mathematics, and other dis- Soot and smoke contain black car- PERFORMANCE GAP ciplines, significantly improving the bon which absorbs solar radiation. performance and scalability of the These aerosols have been thought The SciDAC Performance Eval- codes. For example, improved com- to enhance climate warming by uation Research Center (PERC) is piler annotation technology (Active absorbing radiation that might oth- working toward reducing the gap Harmony) achieved a 15% reduc- erwise be scattered and by reducing between peak and sustained per- tion in run time on the POP ocean overall cloudiness. Penner et al. formance on high-end computers by modeling code. evaluated forcing and changes in developing hardware and software atmospheric temperature and tools to monitor, model, and control D. H. Bailey, B. de Supinski, J. Dongarra, et clouds associated with both direct performance. In the past year, PERC al.,“Performance technologies for peta- and indirect effects of soot and has analyzed and optimized appli- scale systems,”http://perc.nersc.gov/ smoke using an atmospheric-climate cation codes in astrophysics, lattice docs/PERC-white-aes15.doc.ASCR-MICS, model coupled to their chemical- gauge theory, climate modeling, SciDAC

56 RESEARCH NEWS

INTEGRATING THE GLOBUS nonlinear preconditioner; they collaborators are investigating UPC, TOOLKIT WITH FIREWALLS showed numerically that the new a “partitioned global address space” method converges well even for some language. They developed and The Globus Toolkit is a collection of difficult problems, such as high Rey- released an open-source version of clients and services that enable Grid nolds number flows, where a tradi- the full UPC compiler, which trans- computing. Each component of the tional inexact Newton method fails. lates UPC to C with calls to a newly toolkit has its own network traffic developed UPC runtime layer. The characteristics that must be consid- X.-C. Cai and D. E. Keyes,“Nonlinearly pre- open-source compiler has low over- ered when deploying the toolkit at a conditioned inexact Newton algorithms,” head for basic operations on shared site with a firewall. Members of the SIAM J. Sci. Comp. 24, 183 (2002).ASCR- data; it is competitive with, and Globus Project and the DOE Science MICS, SciDAC, NSF,NASA sometimes faster than, the commer- Grid engineering team have devel- cial compiler. oped requirements and guidance DEVELOPING AN ALTERNA- for firewall administrators, describ- TIVE PROGRAMMING MODEL W. Chen, D. Bonachea, J. Duell, P. ing the network traffic generated by Husbands, C. Iancu, and K.Yelick,“A per- While MPI is currently the de facto the Globus Toolkit services, how formance analysis of the Berkeley UPC standard for programming super- the Toolkit’s use of the ephemeral compiler,”17th Annual International computers, several federal agencies port range can be controlled, and Conference on Supercomputing (ICS), are investing in alternative models to firewall requirements for client and 2003.ASCR-MICS, DOD, NSF simplify programming. Yelick and server sites. NERSC provided a fire- wall testbed that was used for verify- ing much of the information in this document.

V.Welch et al.,“Globus Toolkit Firewall Requirements Version 5,”July 22, 2003, http://www.globus.org/security/firewalls/. SciDAC FUSION ENERGY SCIENCES

PRECONDITIONING INEXACT NEWTON GEODESIC ACOUSTIC MODES and the corresponding simulations ALGORITHMS IN PLASMA TURBULENCE provide compelling evidence that geodesic acoustic modes are active in Inexact Newton algorithms are Researchers in the SciDAC Plasma the turbulence in the outer regions commonly used for solving large Microturbulence Project have observed of tokamak plasmas and play an sparse nonlinear system of equa- a coherent oscillation in the poloidal integral role in affecting and medi- tions F(u*) = 0. Even with global flow of density fluctuations that shows ating the fully saturated state of tur- strategies such as line search or trust remarkable similarity to predicted bulence and resulting transport. region, the methods often stagnate characteristics of geodesic acoustic at local minima of ||F ||, especially modes with beam emission spec- G. R. McKee, R. J. Fonck, M. Jakubowski, K. for problems with unbalanced non- troscopy in DIII-D plasmas. Two H. Burrell, K. Hallatschek, R.A. Moyer, D. L. linearities, because the methods do methods of analyzing the data, one Rudakov,W. Nevins, G. D. Porter, P.Schoch, not have built-in machinery to deal based on wavelets and one based on and X. Xu,“Experimental characterization with the unbalanced nonlinearities. time-resolved cross-correlation, reveal of coherent, radially-sheared zonal flows Cai and Keyes studied a nonlinear similar features in the resulting flow in the DIII-D tokamak,”Phys. Plasmas 10, additive Schwarz-based parallel field. These experimental observations 1712 (2003). FES, SciDAC

RESEARCH NEWS 57

COMPUTATIONAL ATOMIC enter the stability analysis through ration-average time-dependent PHYSICS FOR FUSION changes introduced in the plasma close-coupling calculations. For the ENERGY compressibility. remaining oxygen ions, the experi- mental measurements are in good Colgan and Pindzola have applied P.Zhu,A. Bhattacharjee, and Z.W. Ma,“Hall agreement with time-independent the time-dependent close-coupling magnetohydrodynamic ballooning instabil- distorted-wave calculations. As theory to the study of the electron- ity in the magnetotail,”Phys. Plasmas 10, expected, the accuracy of the per- impact ionization of helium from 249 (2003). FES, SciDAC, NSF turbative distorted-wave calculations the excited (1s2s) configuration. improves with increasing ion charge. They made the calculations in an SIZE SCALING OF effort to resolve the discrepancy TURBULENT TRANSPORT S. D. Loch, J. P.Colgan, M. S. Pindzola,W. between theoretical calculations Westermann, F.Scheuermann, K.Aichele, Lin et al. have developed a nonlinear and existing experimental measure- D. Hathiramani, and E. Salzborn,“Electron- model for turbulence radial spreading ments for electron scattering from impact ionization of O q+ ions for q = based on the modified porous-medium the metastable states of helium. The 1–4,”Phys. Rev.A 67, 042714 (2003). equation. The model offers a pheno- relative difference between the non- FES, SciDAC, DFG menological understanding of the perturbative close-coupling and the transition from Bohm to gyro-Bohm perturbative distorted-wave results SIMULATION OF INTENSE scaling when the parameter (minor grew larger as the principal quantum BEAMS FOR HEAVY-ION radius/gyroradius) is larger than 500. number was increased. This differ- FUSION They also estimated the role of the ence has important implications in trapped electron nonlinearity in A multibeamlet approach to a high- the collisional-radiative modeling of zonal flow generation in trapped- current-ion injector, whereby a large many astrophysical and laboratory electron-mode turbulence in the number of beamlets are accelerated plasmas that use perturbation theory. context of parametric instability theory. and then merged to form a single J. P.Colgan and M. S. Pindzola,“Time- beam, offers a number of potential Z. Lin,T. S. Hahm, S. Ethier,W.W. Lee, J. dependent close-coupling studies of the advantages over a monolithic single- Lewandowski, G. Rewoldt,W. M.Tang,W. X. electron-impact ionization of excited-state beam injector. These advantages Wang, L. Chen, and P.H. Diamond,“Size helium,”Phys. Rev.A 66, 062707 (2002). include a smaller transverse foot- scaling of turbulent transport in tokamak FES, SciDAC print, more control over the shaping plasmas,” Proceedings of 19th IAEA Fusion and aiming of the beam, and more Energy Conference, Lyon, France (2002). BALLOONING INSTABILITY IN flexibility in the choice of ion sources. FES, SciDac THE EARTH’S MAGNETOTAIL A potential drawback, however, is a larger emittance. Grote et al. are The Earth’s magnetotail, the main ELECTRON-IMPACT IONIZA- studying the merging of beamlets source of the polar aurora, is the part TION OF OXYGEN IONS (Figure 6) and how this merging of the magnetosphere that is pushed determines emittance. Their results away from the sun by the solar wind. Loch et al. made experimental suggest that a multibeamlet injector The ballooning instability of the measurements of the ionization can be built with a normalized magnetotail interests researchers cross section for Oq+, where q = 1–4, emittance less than 1 πmm mrad. because of its possible relevance to performed with a crossed-beams the dynamics of magnetic substorms. apparatus, and compared them with D. P.Grote, E. Henestroza, and J.W. Kwan, Zhu et al. studied this ballooning theoretical calculations. For O+, the “Design and simulation of a multibeamlet instability within the framework of experimental measurements are in injector for a high-current accelerator,’’ Hall magnetohydrodynamics, which very good agreement with configu- Phys. Rev. ST AB 6, 014202 (2003). FES

58 RESEARCH NEWS

FIGURE 6 The merging of the beamlets creates complex space-charge waves that prop- agate across the beam and mix. This fig- ure shows contours of density at several locations in configuration space (x-y) Y X' and in phase space (x-x′). They show that the individuality of the beamlets is quickly lost, leaving a nearly uniform beam. The z is the distance from the end of the preaccelerator column.

z = 0.5 m

MODELING THE PLASMA- DENSITY LIMIT

Understanding the plasma-density Y X' limit is crucial for projecting the performance of future fusion reac- tors. Xu et al. have developed a model for the density limit in a

z = 1.9 m tokamak plasma based on the edge simulation results from BOUT and UEDGE codes. Simulations of tur- bulence in tokamak boundary plas- mas show that turbulent fluctuation levels and transport increase with Y X' collisionality. The simulations also show that it is easier to reach the density limit as the density increases while holding pressure constant

z = 4.1 m than holding temperature constant.

XXX. Q. Xu,W. M. Nevins,T. D. Rognlien, R. H. Bulmer, M. Greenwald,A. Mahdavi, L. D. SIMULATING THE the neighborhood of a sunspot Pearlstein, and P. Snyder,“Transitions of THERMAL STRUCTURE group in 3D for the first time. They turbulence in plasma density limits,”Phys. OF SOLAR STORMS found that the strong, structured Plasmas 10, 1773 (2003). FES, MIT,GA magnetic field from the sunspots Solar storms can potentially disrupt guides the heat flow and plasma MICROSTRUCTURE the operation of orbiting satellites, flow, resulting in fine structures in EVOLUTION IN endanger astronauts, and cause fail- temperature and density because of IRRADIATED MATERIALS ure of long-distance power grids on the complicated field-line topology. land. Understanding the physical The promise of fusion as a viable mechanisms that originate these Y. Mok, R. Lionello, Z. Mikic, and J. Linker, energy source depends on the devel- storms is of fundamental impor- “Calculating the thermal structure of solar opment of structural materials capa- tance if they are to be forecasted. active regions in three dimensions,”Astro- ble of sustaining operation in harsh Mok et al. simulated the thermal phys. J. (submitted, 2003). FES, NSF,NASA radiation conditions. The structure structure of the solar atmosphere in and mobility of self-interstitial atom

RESEARCH NEWS 59

(SIA) clusters has profound signifi- STABILIZING MICRO- density gradient. This outflow ulti- cance on the microstructural evo- TURBULENCE IN PLASMAS mately balances the inward Ware lution of irradiated materials. pinch, leading to a steady state. Bourdelle et al. have shown that Marian et al. have studied the effect microturbulence can be stabilized D. R. Ernst et al.,“Role of trapped electron of substitutional copper solute atoms in the presence of steep temperature mode turbulence in density profile control on the mobility of self-interstitial and density profiles. High values of with ICRH in Alcator C-Mod,”International clusters in Fe-Cu alloys. The effect |β′| have a stabilizing influence on Sherwood Fusion Theory Conference, of this oversized substitutional solute drift modes. This may form the basis Corpus Christi,Texas,April 28–30, 2003. is to enhance the three-dimensional for a positive feedback loop in which FES character of small-cluster diffusion high core β values lead to improved and to affect general cluster diffu- confinement, and to further increase DESIGNING COMPACT sion properties. in β. In high-β spherical tokamak TOKAMAK-STELLARATOR J. Marian, B. D.Wirth,A. Caro, B. Sadigh, G. plasmas, high |β′| rather than low HYBRIDS R. Odette, J. M. Perlado, and T. Diaz de la aspect ratio is a source of stabiliza- Ware et al. described for the first Rubia,“Dynamics of self-interstitial cluster tion. Therefore, the effect of high time the complete physics properties migration in pure alpha-Fe and Fe-Cu |β′| should be stabilizing in the plas- of compact, drift-optimized tokamak- alloys,”Phys. Rev. B 65,144102 (2002). FES mas of the National Spherical Torus stellarator hybrids with a high-shear Experiment. tokamak-like rotational transform DESIGNING THE NATIONAL C. Bourdelle,W. Dorland, X. Garbet, G.W. profile and |B| that is quasipoloidally COMPACT STELLERATOR Hammett, M. Kotschenreuther, G. Rewoldt, symmetric (Figure 7). The rotational EXPERIMENT and E. J. Synakowski,“Stabilizing impact of transform in these hybrid configura- The U.S. fusion community is con- high gradient of β on microturbulence,” tions is produced primarily by a self- structing a compact, moderate-cost Phys. Plasmas 10, 2881 (2003). FES consistent bootstrap current. The experiment, the National Compact nonaxisymmetric components of |B| Stellerator Experiment (NCSX), to TRAPPED ELECTRON MODE yield a bootstrap current lower than investigate 3D confinement and sta- TURBULENCE that in an axisymmetric device, which bility. The configurations being leads to good magnetohydrodynamic studied are compact, are passively Gyrokinetic simulations of turbulent stability properties without the need stable to known instabilities, and particle transport in the Alcator C- for a conducting wall. The neoclas- are designed to have a “hidden” Mod by Ernst et al. have led to an sical confinement improves with symmetry to give good transport understanding of the observed spon- increasing β, leading to a new form properties. In addition, because of taneous particle transport barrier of configurational invariance in their enhanced stability, they formation, and its control using radio- these stellarators. should be free of disruptions. These frequency heating in the core. As the A. S.Ware, S. P.Hirshman, D.A. Spong, L.A. configurations are designed to com- density gradient steepens, nonreso- Berry,A. J. Deisher, G.Y. Fu, J. F.Lyon, and bine the best features of the toka- nant, collisionless, trapped electron R. Sanchez,“High-β equilibria of drift- mak and the stellerator. The NCSX modes (TEMs) are driven unstable. optimized compact stellarators,”Phys. Rev. has achieved a successful conceptual Gyrokinetic stability analysis shows Lett. 89, 125003 (2002). FES design review. that the modes are driven by the density gradient, have wavelengths UNDERSTANDING NEO- G. H. Neilson and the NCSX Team,“Physics around twice the ion gyroradius, and CLASSICAL TEARING MODES considerations in the design of NCSX,” are partially damped by collisions. Proceedings of 19th IAEA Fusion Energy After the TEM onset, the turbulent Tearing modes are magnetic islands Conference, Lyon, France (2002). FES outflow strongly increases with the formed by the topological rearrange-

60 RESEARCH NEWS

FIGURE 7 (a) Last closed flux surface for two compact stellarator configurations: (a) QPS3, with three field periods, A = 3.7, β = 15%; and (b) QPS2, with two field periods, A = 2.7, β = 5%. Color indicates the variation in |B|.

SIMULATING PSEUDO- CHAOTIC DYNAMICS

Nonlinear dynamics and chaos

1.75 simulations are important for understanding the microscale of

1.50 plasma turbulence, as well as for understanding general nonlinear

la dynamical effects. A family of ran- 1.25 Te s dom systems with zero Lyapunov | in in |

(b) B | exponent is called pseudochaos. 1.00 Zaslavsky and Edelman have shown how the fractional kinetic equation 0.75 can be introduced for pseudochaos and how the main critical exponents of fractional kinetics can be evaluated from the dynamics. Pseudochaotic dynamics and fractional kinetics can be applied to the behavior of streamlines or magnetic field lines.

G. M. Zaslavsky and M. Edelman, “Pseudochaos,’’ in Perspectives and Problems in Nonlinear Science, edited by E. Kaplan, J. E. Marsden, and K. R. ment of magnetic field lines obvious ideal mode, causing a seed Sreenivasan (Springer-Verlag, New York, through reconnection. The preven- island. Based on theoretical and 2003), pp. 421–443. FES, ONR, NSF tion of neoclassical tearing modes experimental results, Brennan et (NTMs) in tokamak plasmas is a al. have proposed and tested a new TURBULENT TRANSPORT major challenge because they can mechanism for tearing-mode onset WITH KINETIC ELECTRONS degrade plasma confinement. Ideal that explains these puzzling observa- modes can seed NTMs through tions. Chen et al. have developed a new forced reconnection, yet in saw- electromagnetic kinetic electron- toothing discharges it is not well D. P.Brennan, R. J. La Haye,A. D.Turnbull, simulation model in 3D toroidal flux- understood why a particular saw- M. S. Chu,T. H. Jensen, L. L. Lao,T. C. Luce, tube geometry that uses a general- tooth crash seeds an NTM after sev- P.A. Politzer, E. J. Strait, S. E. Kruger, and ized split-weight scheme, where the eral preceding sawteeth did not. D. D. Schnack,“A mechanism for tearing adiabatic part is adjustable. This Also, tearing modes sometimes onset near ideal stability boundaries,” model includes electron–ion colli- appear and grow without an Phys. Plasmas 10, 1643 (2003). FES sional effects. For H-mode parame-

RESEARCH NEWS 61

HIGH ENERGY PHYSICS

ters, the nonadiabatic effects of LATTICE QCD CONFRONTS Pierro,A. El-Khadra,A. S. Kronfeld, P.B. kinetic electrons increase linear EXPERIMENT Mackenzie, D. Menscher, and J. Simone, growth rates of the ion-temperature- “High-precision lattice QCD confronts gradient-driven (ITG) modes, main- For almost thirty years precise experiment,”Phys. Rev. Lett. (submitted, ly due to trapped-electron drive. numerical studies of nonperturbative 2003), hep-lat/0304004. HEP,NSF,PPARC The ion heat transport is also QCD, formulated on a space-time increased from that obtained with lattice, have been stymied by an TWO-COLOR QCD WITH adiabatic electrons. The linear inability to include the effects of FOUR CONTINUUM behavior of the zonal flow is not sig- realistic quark vacuum polarization. FLAVORS nificantly affected by kinetic elec- The MILC, HPQCD, UKQCD, Recently there has been a re-evalua- trons. The ion heat transport and Fermilab Lattice collaborations tion of the old idea that quark pairs decreases to below the adiabatic have presented detailed evidence of might condense, giving rise to a tran- electron level when finite plasma β a breakthrough that may now permit sition to a color superconducting state is included due to finite-β stabiliza- a wide variety of high-precision, non- at high baryon-number density. Kogut tion of the ITG modes. perturbative QCD calculations, et al. approached this question by including high-precision B and D Y. Chen, S. E. Parker, B. I. Cohen,A. M. examining the spectrum of two-color meson decay constants, mixing Dimits,W. M. Nevins, D. Shumaker,V. K. lattice QCD with four continuum amplitudes, and semi-leptonic form Decyk, and J. N. Leboeuf,“Simulations of flavors at a finite chemical potential factors—all quantities of great impor- turbulent transport with kinetic electrons (µ) for quark-number, on a 123 × 24 tance in current experimental work and electromagnetic effects,”Nucl. Fusion lattice. They found evidence that the on heavy-quark physics. The break- 43, 1121 (2003). FES, NASA system undergoes a transition to a through comes from a new discretiza- state with a diquark condensate, which tion for light quarks: Symanzik- spontaneously breaks quark number improved staggered quarks. The at µ = m /2, and that this transition researchers compared a wide variety π is mean field in nature. They then of nonperturbative calculations in examined the three states that would QCD with experiment, and found be Goldstone bosons at µ = 0 for zero agreement to within statistical and Dirac and Majorana quark masses. systematic errors of 3% or less.

C.T. H. Davies, E. Follana,A. Gray, G. P. J. B. Kogut, D.Toublan, and D. K. Sinclair, Lepage, Q. Mason, M. Nobes, J. Shigemitsu, “The pseudo-Goldstone spectrum of 2- H. D.Trottier, M.Wingate, C.Aubin, C. color QCD at finite density,”Argonne Bernard,T. Burch, C. DeTar, S. Gottlieb, E. National Laboratory report ANL-HEP-PR- B. Gregry, U. M. Heller, J. E. Hetrick, J. 03-022 (May 2003), hep-lat/0305003. Osborn, R. Sugar, D.Toussaint, M. Di HEP,NSF,HFS

62 RESEARCH NEWS

MATERIALS SCIENCES

BIMETALLIC SURFACE 5'C–3'G + SEGREGATION OF TOTA (I) G–C NANOPARTICLES A–T

To better understand catalysts in fuel cells, Wang et al. used a Monte Carlo code built around the embed- ded atom method to investigate the

segregation of platinum atoms on the TOTA+ region TOTA+ region removed surfaces of platinum-nickel nano- from above particles. They used four kinds of nanoparticle shapes (cube, tetrahe- dron, octahedron, and cubo-octahe- dron) terminated by {111} and {100} facets to examine the extent of Pt segregation. Regardless of the shape and composition of the nanoparticles, the Pt atoms segregated preferentially

to the facet sites, less to edge sites, and TOTA+ and water regions removed least to vertex sites in the outermost atomic layer of the nanoparticles.

G.Wang, M.A.Van Hove, and P.N. Ross, “Segregations in Pt-Ni catalyst nanoparti- cles,”Surface Science (submitted, 2003). BES, SciDAC

ELECTRONIC INTERACTION OF TOTA+ WITH DNA FIGURE 8 Investigating the electronic interac- Polarization-charge (δρ) isosurfaces of backbone, water, and counterion + tions of compounds intercalated in calculated for the hydrated TOTA (I) 3-bp interactions, which shift the energy duplex DNA segments. Magenta regions DNA is important because of the levels of the base and the intercalat- correspond to those where excess elec- biological and medical roles played tronic charge is induced by the interca- ed TOTA+ orbitals significantly. by intercalators. Reynisson et al. lation process, and yellow regions denote The calculations also showed that those where depletion of charge ensued. made first-principles quantum the inserted TOTA+ strongly polar- The upper panel shows δρ for the entire mechanical calculations of trioxatri- hydrated TOTA+(I) 3-bp duplex segment; izes the intercalation cavity, where a + angulenium ion (TOTA+) binding in the middle (right) panel, the TOTA sheet of excess electron density sur- intercalator region is excluded; and in to duplex DNA by intercalation. + rounds the TOTA+ (Figure 8). the bottom panel, both the TOTA inter- The electronic structure calcula- calator and hydration shell regions are tions revealed no meaningful J. Reynisson, G. B. Schuster, S. B. excluded. The latter illustrates clearly the electronic polarization that is charge transfer from DNA to Howerton, L. D.Williams, R. N. Barnett, C. L. induced by the intercalation of the + TOTA , but showed the importance Cleveland, U. Landman, N. Harrit, and J. B. TOTA cation.

RESEARCH NEWS 63

Chaires,“Intercalation of trioxatriangu- bonding to the metal. As a result, the pentacene for applications in elec- leneium ion with DNA: Binding, electron formation of a capped single-walled tronic devices. Tiago et al. have transfer, x-ray crystallography, and elec- nanotube is overwhelmingly favored studied the optical and electronic tronic structure,”J.Am. Chem. Soc. 125, compared to any structure with dan- properties of crystalline pentacene, 2072 (2002). BES, NSF,NCI gling bonds (such as graphite sheets) using a first-principles Green’s func- or to a fullerene. tion approach. They investigated the ELECTRONIC STRUCTURE OF role of polymorphism on the elec- X. Fan, R. Buczko,A.A. Puretzky, D. B. C MONOLAYERS tronic energy gap and linear optical 60 Geohegan, J.Y. Howe, S.T. Pantelides, and spectrum by studying two different Exhibiting properties such as high S. J. Pennycook,“Nucleation of single- crystalline phases: the solution-phase transition temperature supercon- walled nanotubes,”Phys. Rev. Lett. 90, structure and the vapor-phase struc- ductivity and antiferromagnetism, 145501 (2003). BES, ORNL, NSF,MEVU ture. Charge-transfer excitons were C -based fullerides are ideal model 60 found to dominate the optical spec- compounds for exploring key con- INVESTIGATING THE CON- trum. Excitons with sizable binding ceptual issues in strongly correlated NECTION BETWEEN ORBITAL energies were predicted for both physics. Yang et al. studied the AND SPIN ORDERING phases. structure and electronic structure of Medvedeva et al. studied the complex electron-doped C monolayers. The 60 nature of the fundamental connection M. L.Tiago, J. E. Northrup, and S. G. Louie, overall shape of the calculated band between orbital ordering phenomena “Ab initio calculation of the electronic structure showed good agreement and spin ordering alignment on a and optical properties of solid pentacene,” with the high-resolution angle- microscopic scale using their recently Phys. Rev. B 67, 115212 (2003). BES, NSF resolved photoemission experiments, developed average spin state calcula- indicating a reduction of the occupied tion scheme. This scheme allowed RELATIVITY AND THE bandwidth by about 40 percent due them to treat a paramagnetic state and STABILITY OF INTER- to electron-phonon interaction. to describe successfully the experi- METALLIC COMPOUNDS W. L.Yang,V. Brouet, X. J. Zhou, H. J. Choi, mental magnetic and orbital phase Why are some intermetallic com- S. G. Louie, M. L. Cohen, S.A. Kellar, P.V. diagrams of LaMnO , KCuF , 3 3 pounds stable while others are not? Bogdanov,A. Lanzara,A. Goldoni, F. La Sr MnO , and LaSr Mn O . 0.5 1.5 4 2 2 7 Wang and Zunger showed that the Parmigiani, Z. Hussain, and Z.-X. Shen, For the first time, they investigated existence of stable, ordered 3d-5d “Band structure and Fermi surface of from first principles the stability of intermetallics CuAu and NiPt, as electron-doped C60 monolayers,”Science charge and orbital ordering upon opposed to the unstable 3d-4d iso- 300, 303 (2003). BES, ONR, NSF onset of a magnetic transition in valent analogs CuAg and NiPd, La Sr MnO and LaSr Mn O . 0.5 1.5 4 2 2 7 results from relativity. First, in NUCLEATION OF SINGLE- J. E. Medvedeva, M.A. Korotin,V. I.Anisimov, shrinking the equilibrium volume WALLED CARBON and A. J. Freeman,“Charge and orbital of the 5d element, relativity reduces NANOTUBES ordering melting in layered LaSr2Mn2O7” the atomic size mismatch with The nucleation pathway for single- (in preparation, 2003). BES respect to the 3d element, thus low- walled carbon nanotubes has been ering the elastic packing strain. investigated using first-principles CALCULATING THE PROPER- Second, in lowering the energy of density-functional calculations. Fan TIES OF AN ORGANIC the bonding 6s,p bands and raising et al. have shown that incorporation SEMICONDUCTOR the energy of the 5d band, relativity of pentagons at an early stage of the enhances (diminishes) the occupa- nanotube growth reduces the num- There has been increasing interest in tion of the bonding (antibonding) ber of dangling bonds and facilitates organic semiconductors such as bands. Raising the energy of the 5d

64 RESEARCH NEWS

band also brings it closer to the one monolayer (ML). Using first prin- MOLECULAR-DYNAMICS energy of the 3d band, improving ciples calculations, Chan et al. have SIMULATIONS OF KINETIC the 3d-5d bonding. determined the atomic configuration, COEFFICIENTS energy stability, and electronic struc- L. G.Wang and A. Zunger,“Why are the 3d- Asta et al. have demonstrated a new ture of the different phases of Pb/Si 5d compounds CuAu and NiPt stable, equilibrium molecular-dynamics sim- (111). Their calculations will help whereas the 3d-4d compounds CuAg and ulation technique for calculating the experimenters identify the atomic NiPd are not,”Phys. Rev. B 67, 092103 kinetic coefficient (i.e., the propor- model and domain wall arrangement (2003). BES tionality constant between interface for the controversial structures of velocity and undercooling) of solid- Pb/Si(111) at the Pb coverage of 1 liquid interfaces in elemental metals. ELECTRONIC STRUCTURES to 4/3 ML. AND PROPERTIES OF This work represents a necessary first VITAMIN B12 T.-L. Chan, C. Z.Wang, M. Hupalo, M. C. step towards studying dynamic prop- Tringides, Z.-Y. Lu, and K. M. Ho,“First- erties of alloy solid-liquid interfaces Methodologies developed for the principles studies of structures and sta- near equilibrium. materials sciences are being extended bilities of Pb/Si(111),”Phys. Rev. B 68, M.Asta, D.Y. Sun, and J. J. Hoyt,“Role of to the study of biomaterials and 045410 (2003). BES atomistic simulation in the modeling of biomolecules. Kurmaev et al. have solidification microstructure,”in NATO-ASI calculated the electronic structure GROWTH AND FORMATION Proceedings: Thermodynamics, Microstruc- of the vitamin B12 molecule and its OF SILICON NANOCLUSTERS ture and Plasticity, ed. by A. Finel (in co-enzymes: cyanocobalamin, Silicon nanoclusters have been the press). BES methylcobalamin, and adenosyl focus of intense theoretical and cobalamin. For the first time, all experimental interest, because the SOLVING THE MYSTERY OF the side chains in these complex combination of the unique optical EXCHANGE BIAS molecules were included in the ab properties of silicon quantum dots and Exchange bias, a shift in magnetiza- initio calculation. their compatibility with existing sili- tion that occurs when two different E. Z. Kurmaev,A. Moewes, L. Ouyang, L. con-based and nanobiological tech- kinds of magnet come into contact, Randaccio, P. Rulis,W.Y. Ching, M. Bach, nologies demonstrates great promise is used in numerous electronic and M. Neumann,“The electronic struc- for numerous applications. Draeger applications such as magnetic ture and chemical bonding of vitamin et al., using density-functional and multilayer storage and read head B12,”Europhys. Lett. 62, 582 (2003). BES, quantum Monte Carlo calculations, devices, but the effect is still not NSF,EU, RFBR, NSERC, NEDO, MIUR, CMR have demonstrated that the optical properly understood. Canning et al. gap is significantly reduced by the have performed first-principles spin- DENSITY FUNCTIONAL presence of oxygen and fluorine dynamics simulations of the magnetic CALCULATION OF LEAD atoms on the surface. This is an structure of iron-manganese/cobalt ON SILICON important result for experimentalists, (FeMn/Co) interfaces (Figure 9). as oxygen is a common contaminant Lead and silicon can form a well- These quantum mechanical simula- in the procedure for synthesizing defined interface (Pb/Si) which is tions, involving 2,016-atom super- nanoparticles. ideal for studying two-dimensional cell models, reveal details of the ori- behavior. But despite extensive work E. Draeger, J. C. Grossman,A. J.Williamson, entational configuration of the mag- in the literature, the structure of the and G. Galli,“Influence of synthesis condi- netic moments at the interface that different Pb phases formed on tions on the structural and optical proper- are unobtainable by any other Si(111) is still not clear, especially ties of passivated silicon nanoclusters,” means, offering important clues to when the Pb coverage is more than Phys. Rev. Lett. 90, 167402 (2003). BES solve the mystery of exchange bias.

RESEARCH NEWS 65

FIGURE 9 Visualization of the exchange-bias sys- tem initial state, viewed from the direc- tion of the y-axis. The arrows represent the orientation of the magnetic moments. The gold balls represent the Fe atoms, the grey ones the Mn atoms, while the green and pink balls stand for the relaxed and unrelaxed Co atoms. On the left, the 3Q antiferromagnetic ordering 3Q is schematically displayed within the conventional fcc structure.

A. Canning, B. Ujfalussy,T. C. Schulthess, UNDERSTANDING the first time, an interplay between X.-G. Zhang,W.A. Shelton, D. M. C. ATOMIC ORDERING OF ordering and surface reconstruction Nicholson, G. M. Stocks,Y.Wang, and T. SEMICONDUCTOR ALLOYS during the growth was proposed to Dirks,“Parallel multi-teraflops studies of explain the experimental observations. It has long been understood that the magnetic structure of FeMn alloys,” The model is qualitatively different atomic ordering, widely observed in Proc. Int. Parallel and Distributed Processing from the widely accepted surface certain epitaxially grown semicon- Symposium (IPDPS’03,April 2003, Nice, reconstruction- and dimerization- ductor alloys, is driven by surface France), p. 256b. BES,ASCR-MICS induced ordering models that explain thermodynamics and/or by growth only the in-plane 2D patterns. kinetics, but not by bulk thermody- FAST LARGE-SCALE namics. Batyrev et al. have proposed I. G. Batyrev,A. G. Norman, S. B. Zhang, and NANOSTRUCTURE a microscopic growth model for the S.-H.Wei,“Growth model for atomic order- CALCULATIONS ordering of gallium arsenide anti- ing:The case for quadruple-period order- monide (GaAsSb) to account for the ing in GaAsSb alloys,”Phys. Rev. Lett. 90, Fast first-principles calculation of observed 3D ordered structure. For 026102 (2003). BES thousand-atom systems is a chal- lenge in computational nanoscience. Li and Wang have found a method MEDICAL AND LIFE SCIENCES to solve this challenge, using the charge-patching method for the charge density and an idealized COMPUTATIONAL MODELING inhibitors is complicated by the fact surface passivation. It takes only OF PROTEIN SWITCHES that human cells contain ~500 dif- about one hour on 64 processors to ferent protein kinases, all of which calculate a thousand-atom system The Src-family kinases and their catalyze the same chemical reac- with first-principles accuracy. close relatives, such as Abl, function tion and contain highly conserved Thousand-atom quantum dots for as molecular switches and are active sites. Because crystal struc- various materials were calculated important targets for therapeutic tures provide only static views of for the first time. intervention; for example, the c-Abl certain states of each kinase, Nagar J. B. Li and L.W.Wang,“First principle inhibitor Gleevec is used to treat et al. used molecular dynamics sim- thousand atom quantum dot calculations,” chronic myolegenous leukemia. ulations to help understand specific Phys. Rev. Lett. (submitted, 2003). BES The development of specific switching mechanisms. The simula-

66 RESEARCH NEWS

A N-lobe

SH3 αC Tyr 245 N PD166326 SH2- Kinase Tyr 412 SH3-2 linker Activation connector loop SH2αA SH2 αE αF

αI αH C-lobe I-I' loop αI' Myristate C SH2- B Kinase linker

N-lobe FIGURE 10 SH3 (A) Ribbon and surface representations of c-Abl. N (B) Superposition of the structure of c-Src.

SH3-2 connector

SH2

C Tyr 527 C-lobe C c-Abl c-Src tions confirmed that the regulatory THE IMPACT OF of nuclear magnetic resonance mechanism of the proto-oncogenic CARCINOGENS ON DNA spectra for [POB]dG were used Abl protein shares an important fea- STRUCTURE as inputs to the DUPLEX code, ture with the regulatory mechanism which produced a minimum energy Understanding the 3D structure of the related Src-family tyrosine structure that agreed with the of DNA that has been modified by kinases (Figure 10). This result was NMR data. chemical carcinogens is important verified by in vitro experiments. in understanding the molecular basis B. Nagar, O. Hantschel, M.A.Young, K. of cancer and chemical toxicology. L.A. Peterson, C.Vu, B. E. Hingerty, S. Scheffzek, D.Veach,W. Bornmann, B. Peterson et al. studied [POB]dG, Broyde, and M. Cosman,“Solution structure Clarkson, G. Superti-Furga, and J. Kuriyan, which is produced from metaboli- of an O6-[4-oxo-4-(3-Pyridyl)butyl]guanine “Structural basis for the autoinhibition of cally activated tobacco-specific adduct in an 11mer DNA duplex: Evidence c-Abl tyrosine kinase,”Cell 112, 859 nitrosomines. This compound for formation of a base triplex,”Biochemistry (2003). BER, HFSP,EMBL, EMBO,AF,GNMF, modifies the base guanine, posing a ASAP,DOI: 10.1021/bi035217v (2003). CAG, DRCRF,BWF significant cancer risk. The results BER, NIH, ORNL

RESEARCH NEWS 67

MODELING ENZYME STRUC- INTERACTIONS BETWEEN building pyrimidine nucleobase of TURE AND KINETICS AMINO ACIDS AND RNA. They demonstrated that the NUCLEOBASES most stable complexes between Although information on the func- uracil and glycine are formed when tion of enzymes has been obtained There is a paucity of theoretical the carboxylic group of glycine is experimentally, there is still an information about interactions bound through two hydrogen bonds essential missing link between the between amino acids and nucle- to uracil. structures and biomolecular dynam- obases. Dabkowska et al. made elec- ics and function. Yang et al. studied tronic-structure calculations con- I. Dabkowska, J. Rak, and M. Gutowski, the binding change mechanism of cerning the simplest amino acid- “Computational study of hydrogen-bonded

F1-ATPase, the enzyme responsible nucleobase complex, i.e., the dimer complexes between the most stable tau- for most of the ATP synthesis in liv- of glycine and uracil. Glycine is the tomers of glycine and uracil,”J. Phys. ing systems. By computing the smallest amino acid, and uracil is a Chem.A 106, 7423 (2002). BER, KBN chemical potential, they identified

βTP as the tight binding site for ATP and βDP as the loose site. The half NUCLEAR PHYSICS closed site was identified as the binding site for the solution ADP QUANTUM MONTE CARLO and Pi in ATP synthesis; it is differ- Detector (KamLAND) is to search ent from the empty binding site for STUDIES OF FERMI GASES for the oscillation of antineutrinos emitted from distant power reactors. ATP hydrolysis. Based on this result, Determining the properties of Fermi KamLAND’s first results indicate a consistent structural and kinetic gases is an intriguing topic for many- that it is measuring only 61% of the model for the binding change body physics, with applications to reactor anti-neutrino flux expected mechanism is being developed for phenomena such as neutron stars. in the no-oscillation hypothesis. F1-ATPase. Carlson et al. conducted quantum This result (using CPT invariance) W.Yang,Y. Q. Gao, Q. Cui, J. P.Ma, and M. Monte Carlo calculations of super- excludes all solutions to the solar- Karplus,“The missing link between ther- fluid Fermi gases with short-range neutrino problem except for the modynamics and structure in F -ATPase,” two-body attractive interactions with 1 “large mixing angle” (LMA) region PNAS 100, 874 (2003). BER, NIH infinite scattering length. The energy (Figure 11). of such gases is estimated to be (0.44 ± 0.01) times that of the non- K. Eguchi et al. (KamLAND Collaboration), interacting gas, and their pairing gap “First results from KamLAND: Evidence for is approximately twice the energy reactor antineutrino disappearance,”Phys. per particle. Rev. Lett. 90, 021802 (2003). NP, MEXT

J. Carlson, S.-Y. Chang,V. R. Pandharipande, and K. E. Schmidt,“Superfluid Fermi gases EXCITED BARYON PHYSICS with large scattering length,”Phys. Rev. FROM LATTICE QCD Lett. (in press), arXiv:physics/0303094 Unraveling the nature of Roper res- (2003). NP,NSF onance (the first excited state of the nucleon with the same quantum DATA ANALYSIS AND number) has a direct bearing on SIMULATIONS FOR KAMLAND understanding the quark structure The primary goal of the Kamioka and chiral dynamics of baryons, one Liquid Scintillator Anti-Neutrino of the primary missions at laboratories

68 RESEARCH NEWS

FIGURE 11 which directly reflect the presence The neutrino oscil- of deformation. Alexandrou et al. lation parameter region for two- subsequently calculated this form neutrino mixing of factor using lattice QCD. The cal- solar neutrino culated magnetic dipole form factor experiments. The solid circle shows and electric quadrupole amplitude the best fit to the were consistent with experimental KamLAND data. results, but systematic errors due to lattice artifacts prevented a determi- nation of the Coulomb quadrupole form factor. Further study of these lattice artifacts is needed for better control of systematic errors.

C.Alexandrou, P.de Forcrand,T. Lippert, H. Neff, J.W. Negele, K. Schilling, and A. Tsapalis,“N to ∆ electromagnetic transition such as Jefferson Lab. Dong et al. of ~225% and 185% in the predict- form factor from lattice QCD,”hep- lat/0307018 (2003). NP,ESOP,UC have calculated the correct level- ed atomization energy for HsO4 and ordering between the Roper (1440) OsO4, respectively. (2) Mulliken HFB CONTINUUM PROBLEM resonance and the S11 (1535) reso- population analysis of the Dirac-Fock nance for the first time on the lattice. self-consistent field wave functions, SOLVED FOR DEFORMED This result verifies that the Roper in contrast to the nonrelativistic NUCLEI state is a radial excitation of the Hartree-Fock wave function, predicts One of the fundamental questions nucleon with three valence quarks, the HsO4 to be less volatile than the of nuclear structure physics is, What and indicates that spontaneously lighter congener OsO4, and this are the limits of nuclear stability? broken chiral symmetry dictates the prediction is in accord with the first Terán et al. have solved, for the first dynamics of light quarks in the experimental work on hassium. time, the Hartree-Fock-Bogoliubov nucleon. G. L. Malli,“Dramatic relativistic effects in (HFB) continuum problem in coor- S. J. Dong,T. Draper, I. Horvath, F.X. Lee, K. atomization energy and volatility of the dinate space for deformed nuclei in F.Liu, N. Mathur, and J. B. Zhang,“Roper superheavy hassium tetroxide and OsO4,” two spatial dimensions without any J. Chem. Phys. 117,10441 (2002). NP approximations. The novel feature resonance and S11 (1535) in lattice QCD,” Phys. Rev. Lett. (submitted, 2003), hep- of the new Vanderbilt HFB code is ph/0306199. NP EXPLORING NUCLEON that it takes into account high-energy STRUCTURE USING continuum states with an equivalent LATTICE QCD ELECTRONIC STRUCTURE OF single-particle energy of 60 MeV or SUPERHEAVY ELEMENTS more. In the past, this has only An important question about the been possible in 1D calculations for nucleon and its excited state, ∆, is Malli investigated the effects of rela- spherical nuclei. tivity on the electronic structure whether they are spherical or and bonding in the tetroxides of the deformed. Recent experiments have E.Terán,V. E. Oberacker, and A. S. Umar, superheavy element hassium and its accurately measured the electric “Axially symmetric Hartree-Fock- lighter congener osmium, with two and Coulomb quadrupole and mag- Bogoliubov calculations for nuclei near major conclusions: (1) Relativistic netic dipole multipoles of the the drip-lines,”Phys. Rev. C 67, 064314 effects lead to a dramatic increase nucleon-to-∆ transition form factor, (2003). NP

RESEARCH NEWS 69

MEASURING THE FATE OF um as a function of the size of the burn steadily for billions of years. PARTON FRAGMENT JETS overlapping system. The back-to- Wiringa and Pieper demonstrated back correlations were reduced con- that the binding energies, excitation In collisions of heavy nuclei at high siderably in the most central Au + structure, and relative stability of light energies, a new state of matter con- Au collisions, indicating substantial nuclei, including the opening of the sisting of deconfined quarks and interaction as the hard-scattered A = 5 and 8 mass gaps, are crucially gluons at high density is expected. partons or their fragmentation prod- dependent on the complicated Large transverse momentum (pT) ucts traversed the medium. structure of the nuclear force. They partons in the high-density system calculated the energy spectra of light result from the initial hard scattering C.Adler et al. (STAR Collaboration),“Disap- nuclei using a variety of nuclear- of nucleon constituents. After a hard pearance of back-to-back high p hadron T force models ranging from very sim- scattering, the parton fragments to correlations in central Au + Au collisions — ple to fully realistic, and they observed create a high-energy cluster (jet) of at √s = 200 GeV,”Phys. Rev. Lett. 90, NN how features of the experimental particles. The STAR Collaboration 082302 (2003). NP,HEP,NSF,BMBF, spectrum evolve with the sophistica- measured two-hadron angular cor- IN2P3, EPSRC, FAPESP,RMST, MEC, NNSFC tion of the force. They found that relations at large transverse momen- the absence of stable five- and tum for p + p and Au + Au collisions. CALCULATING THE ENERGY eight-body nuclei depends crucially These measurements provided the SPECTRA OF LIGHT NUCLEI on the spin, isospin, and tensor most direct evidence for production components of the nuclear force. of jets in high-energy nucleus- The absence of stable five- or eight- nucleus collisions and allowed the body nuclei is crucial to both pri- R. B.Wiringa and S. C. Pieper,“Evolution of first measurements of the fate of mordial and stellar nucleosynthesis, nuclear spectra with nuclear forces,”Phys. back-to-back jets in the dense medi- enabling stars such as our sun to Rev. Lett. 89, 182501 (2002). NP

70 RESEARCH NEWS

NERSC CENTER

CLIENTS, SPONSORS, AND ADVISORS

258,700 677,611 737,322 800,365 1,140,145 Universities DOE Labs 1,164,898 1,694,737 35% 55% 14,264,870 1,893,419

3,252,543

6,287,364

Industries 6% Other Labs 8,309,044 10,104,475 4%

Lawrence Berkeley Brookhaven Lawrence Livermore Sandia FIGURE 1 Oak Ridge National Renewable Energy Princeton Plasma Physics Los Alamos NERSC usage by institution type, FY03 Argonne Ames Pacific Northwest Other Labs (6) National Center for Atmospheric Research

FIGURE 2 DOE and other ferderal laboratory usage at NERSC, FY03

NERSC served 2,323 scientists organizations that used large alloca- awarded to research groups, regard- throughout the United States in FY tions of computer time. Computa- less of their source of funding, based 2003. These researchers work in tional science conducted at NERSC on an annual review of proposals. DOE laboratories, universities, covers the entire range of scientific As proposals are submitted, they are industry, and other Federal agencies. disciplines, but is focused on research subjected to peer review by the Figure 1 shows the proportion of that supports the DOE’s mission and Computational Review Panel NERSC usage by each type of insti- scientific goals, as shown in Figure 4. (CORP, Appendix B) to evaluate the tution, while Figures 2 and 3 show Allocations of computer time computational approach and the laboratory, university, and other and archival storage at NERSC are readiness of the applicant’s codes to

NERSC CENTER 71

1% 3,798,873 3% 6,972,216 5% 7% 32% 523,491 3,534,760 546,798 560,041 7% 608,885 3,408,282 668,295 705,452 7% 786,129 2,667,480 1,259,662 1,401,660 11% 1,483,836 2,161,381 1,576,950 16% 1,610,968 2,095,420 11% 2,039,613

University of Arizona Massachusetts Institute Fusion Energy Chemistry Science Applications of Technology Nuclear Physics Accelerator Physics International Corp. University of California, High Energy Physics Computer Science and University of California, Los Angeles Materials Science Mathematics Santa Cruz University of Texas Climate and Environmental Life Sciences University of California, University of Wisconsin Science Earth and Engineering Berkeley New York University Sciences Auburn University Harvard University General Atomics Georgia Institute of Technology University of Kentucky Vanderbilt University FIGURE 4. Max Planck Institute Northwestern University NERSC usage by scientific discipline, FY03 University of Maryland George Washington University Others (50)

FIGURE 3 Academic and private laboratory usage at NERSC, FY03

fully utilize the computing resources Experiment (INCITE) awarded al oversight for the NERSC Center: being requested. DOE Office of 10% of NERSC resources to three the NERSC Policy Board Science program managers and the computationally intensive research (Appendix A) advises the Berkeley Supercomputing Allocations Com- projects that were judged capable of Lab Director on the policies that mittee (SAC, see Appendix D) making high-impact scientific determine the impact and perform- review and make award decisions for advances through the use of a large ance of the NERSC Center, and all “base award” requests, reflecting allocation of computer time and the NERSC Users Group their mission priorities. SciDAC data storage. The three projects are: (Appendix C) advises the NERSC awards are made by the SciDAC Thermonuclear Supernovae: Stellar management and provides feedback Resource Allocation Management Explosions in Three Dimensions; from the user community. DOE Team, while allocations for small Fluid Turbulence and Mixing at program management is provided startup projects are determined by High Reynolds Number; and by the Office of Advanced NERSC. Quantum Monte Carlo Study of Scientific Computing Research For FY 2004, a new program Photoprotection via Carotenoids in (Appendix E), with advice from the entitled Innovative and Novel Com- Photosynthetic Centers. Advanced Scientific Computing putational Impact on Theory and Two other groups provide gener- Advisory Committee (Appendix F).

72 NERSC CENTER

CAPABILITY AND SCALABILITY

The past year was marked by a system that would have the most cost- dramatic increase in NERSC’s effective impact on DOE science,” computational capability, resulting Kramer said when the decision was in an impressive upsurge in the pro- announced. The new contract ductivity of many research projects. includes five years of support for the But in traditional NERSC fashion, combined system, named “Seaborg.” these changes took place almost The new Seaborg has 416 16- seamlessly, with little disruption to CPU Power 3+ SMP nodes (6,656 users’ daily work. processors) with a peak performance In 2002, after requesting propos- of 10 teraflop/s, which made it the als to replace NERSC’s soon-to-be most powerful computer for unclas- decommissioned Cray systems, the sified research in the United States, procurement team, lead by Deputy with the largest memory capacity, at Director Bill Kramer, made the the time of its installation. The sys- decision to increase the capability of tem has 7.8 terabytes of aggregate the existing IBM SP Power 3 system, memory and a Global Parallel File rather than purchase an entirely System with 44 terabytes of storage. new one. This expansion gave the A team of NERSC staff (Figure user community an unexpected 1) worked for six months to config- increase of 30 million MPP hours ure, install, and test the system. The in FY 2003, and doubled the hours system was available more than 98 that were allocated for FY 2004. percent of the time during testing, “The bottom line was choosing a and benchmarks ran at 72 percent

FIGURE 1 The Seaborg expansion procurement and installation teams doubled NERSC’s com- puting resources 20 months earlier than expected and for 75% of the cost expected, without interrupting service to the research community.

NERSC CENTER 73

140,000 95% FIGURE 2 130,000 90% NERSC strives to maintain a high rate 85% rs of processor utilization, balancing this 120,000 80% 110,000 goal with good job turnaround times.

Hou 100,000 90,000 sor 80,000

ces 70,000 60,000 Pro 50,000 40,000 30,000 20,000 10,000 0 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

Number of Performance of peak speed, much higher than Project processors (% of peak) that achieved on similar parallel sys- Electromagnetic Wave-Plasma Interactions 1,936 68% tems. The team resolved major con- Cosmic Microwave Background Data Analysis 4,096 50% figuration issues, such as whether to operate Seaborg as one or two sys- Terascale Simulations of Supernovae 2,048 43% tems (they decided on one), and Quantum Chromodynamics at High Temperatures 1,024 13% brought it into production status on TABLE 1 March 4, 2003, one month earlier Performance Results for Science-of-Scale Applications than planned. The expanded system quickly achieved the high availability and utilization rates that NERSC survey of user programs that run on time in FY 2003 and 50% in FY demands of itself (Figure 2). a large number of nodes, as well as 2004 be used by jobs that require Initial results from the expanded programs that run on a smaller 512 or more processors. Job sizes for Seaborg showed some scientific number of nodes but account for a FY 2003 are shown in Figure 3. applications running at up to 68% of significant fraction of time used on The scaling report notes that as the system’s theoretical peak speed, the system. The report “IBM SP Seaborg doubled in size, the number compared with the 5–10% of peak Parallel Scaling Overview” (http:// of jobs increased correspondingly, performance typical for scientific www.nersc.gov/computers/SP/ and over time the number of jobs applications running on massively scaling) discusses issues such as con- then decreased as the parallelism of parallel or cluster architectures. straints to scaling, SMP scaling, MPI jobs increased (Figure 4). The wait- Performance results for four science- scaling, and parallel I/O scaling. ing time for large-concurrency jobs of-scale applications are summarized The report offers recommendations also changed in accordance with in Table 1. (More details are avail- for choosing the level of parallelism changes in queueing policies able at http://www.nersc.gov/news/ best suited to the characteristics of (Figure 5). newNERSCresults050703.pdf.) the code. Another measure of Seaborg’s To learn about how Seaborg Improving the scalability of performance is Sustained System performs as a production environ- users’ codes helps NERSC achieve Performance (SSP). The SSP metric ment for a larger sample of scientific goals set by the DOE Office of was developed at NERSC as a com- applications, NERSC conducted a Science: that 25% of computing posite performance measurement of

74 NERSC CENTER

140,000 the floating-point operation count 130,000 by the sum of observed elapsed 1-63 Processors 120,000 times of the applications. Finally, 64-255 Processors rs 110,000 256-511 Processors the SSP is calculated by multiply- 100,000 512-1023 Processors 1024-2047 Processors ing the computational rate by the Hou 90,000 >2047 Processors number of processors in the system. sor 80,000 The SSP is used as the primary 70,000

ces performance figure for NERSC’s 60,000 contract with IBM. It is periodically

Pro 50,000 reevaluated to track performance 40,000 and monitor changes in the system. 30,000 For example, the performance 20,000 10,000 impact of a compiler upgrade is 0 readily detected with a change in Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep SSP. The expansion of Seaborg more than doubled the SSP, from FIGURE 3 Job sizes on Seaborg over the course of FY 2003. DOE’s goal for NERSC is to 657 to 1357. increase the percentage of jobs that use 512 or more processors. During the acceptance period for the Seaborg expansion, there codes from five scientific applica- tem performance in a way that is were reports of new nodes perform- tions in the NERSC workload: relevant to scientific computing. ing slightly better than old nodes, fusion energy, material sciences, To obtain the SSP metric, a which was puzzling since the hard- cosmology, climate modeling, and floating-point operation count is ware is identical. All of these reports quantum chromodynamics. Thus, obtained from microprocessor hard- involved parallel MPI codes; no SSP encompasses a range of algo- ware counters for the five applica- performance differences were rithms and computational tech- tions. A computational rate per detected on serial codes. The per- niques and manages to quantify sys- processor is calculated by dividing formance differences appeared to

FIGURE 4 Number and size of jobs run on Seaborg in May 2002 (left) and May 2003 (right). The horizontal axis is time. The colored rectangles are jobs, with start and stop time depicted by width, and level of parallelism (number of nodes) depicted by height. Colors signify different users. The rectangles are arranged vertically to avoid overlapping.

NERSC CENTER 75

FIGURE 5 Queue waiting time for jobs run on Seaborg in May 2002 (left) and May 2003 (right). Jobs are depicted as in Figure 3 except for colors. Here the color range signifies waiting time, with blue depicting a shorter wait and red a longer wait.

800 depend on the concurrency and the amount of synchronization in the MPI calls used in the code, but periodic observations and testing 600 provided inconsistent results, and at times no difference could be meas- ured. So in October 2003, groups of

abytes 400 old and new nodes were set aside Ter for in-depth testing. A critical insight occurred when the control workstation happened to 200 be inoperable during the node test- ing. During that time, the perform-

0 0 ance of the old nodes improved to 1 /04 1 0 1 0 0 1 1 0 200 0 match the new nodes. NERSC staff 1 2000/ /1 2003/ 999/ 1 1 2000/04 2002/ 2003/04 998/ 999/04 discovered that the control work- 1 1 200 2002/04 station ran certain commands from Year and Month its problem management subsystem FIGURE 6 up to 27 times more often on old Cumulative storage by month, 1998–2003. nodes than on new nodes. Once the problem was pinpointed, it was quickly resolved by deleting four In addition to the Seaborg nuclear physics communities, was deactivated problem definitions. expansion, NERSC also upgraded expanded to 412 nodes with a disk Resolving this issue improved the its PDSF Linux cluster and High capacity of 135 terabytes (TB). New synchronization of MPI codes at Performance Storage System (HPSS). technologies incorporated this year high concurrency, resulting in faster The PDSF, which supports include Opteron chips and 10 TB and more consistent run times for large data capacity serial applica- of configurable network attached most NERSC users. tions for the high energy and storage (NAS).

76 NERSC CENTER

At NERSC, the data in storage 100 doubles almost every year (Figure 6). As of 2003, NERSC stores approxi- mately 850 TB of data (30 million 80 files) and handles between 3 and 6 TB of I/O per day (Figure 7). To keep 60 up with this constantly increasing abytes) workload, new technology is imple- mented as it becomes available. I/O (Ter 40 NERSC currently has two HPSS systems, one used primarily for user files and another used primarily for 20 system backups. The maximum the- oretical capacity of NERSC’s archive 10 /04 10 0 / 1 0 is 8.8 petabytes, the buffer (disk) 0 1 /1 0 200 0 /04 1 2000 /1 2003/ 999/ 2000/04 1 2003/04 cache is 35 TB, and the theoretical 999/04 1 2002 1998/ 1 200 2002 transfer rate is 2.8 gigabytes per sec- Year and Month ond. Implementation of a four-way FIGURE 7 stripe this year increased client data I/O by month, 1998–2003. transfers to 800 megabytes per second.

NERSC CENTER 77

COMPREHENSIVE SCIENTIFIC SUPPORT

Many of the important break- for some projects; and CVS servers throughs in computational science and support for community code are expected to come from large, management. multidisciplinary, multi-institutional Providing this level of support collaborations working with advanced sometimes requires dogged determi- codes and large datasets, such as the nation. That was the case when the SciDAC and INCITE collaborations. Center for Extended MHD Model- These teams are in the best position ing, a SciDAC project, reported to take advantage of terascale com- that one of their plasma simulation puters and petascale storage, and codes was hanging up intermittently. NERSC provides its highest level of NERSC consultant Mike Stewart support to these researchers. This (Figure 1) tested the code on support includes specialized con- Seaborg in both heavily and lightly sulting support; special service coor- loaded scenarios, at various compil- dination for queues, throughput, er optimization levels, on different increased limits, etc.; specialized combinations of nodes, and on algorithmic support; special soft- NERSC’s SP test system. He sub- ware support; visualization support; mitted the bug to IBM as a “severity conference and workshop support; 2” problem; but after a week passed Web sever and Web content support with no response, he asked the local IBM staff to investigate. They put Mike in touch with an IBM AIX developer, at whose request he ran FIGURE 2 more test cases. Mike even wrote a Richard Gerber has made significant contributions to several research proj- FORTRAN code to mimic the C ects by improving code performance test case, and it displayed the same and helping scientists use NERSC problem. resources effectively. Using TotalView debugging software, Mike finally pinpointed the location of the bug in an MPI released a revised MPI runtime file read routine. With this new library that allowed the plasma code information, IBM assigned the to run without interruption. Without problem to an MPI developer, who Mike’s persistence, the researchers’ worked with Mike for several hours work might have been delayed for by phone to fix the bug. IBM then months. NERSC staff are not content to wait for users to report problems. Consultant Richard Gerber (Figure 2) has taken extra initiative to FIGURE 1 strengthen support for SciDAC and NERSC consultant Mike Stewart’s per- other strategic projects by staying in sistence paid off in the timely fix of an obscure software bug. touch with principal investigators

78 NERSC CENTER

and by attending project meetings, Stewart found two obscure refer- tage of (Figure 3). David Skinner where he discusses the state of their ences to this function in IBM docu- analyzed these complicated scripts research efforts and ways in which mentation that pointed the way to a and with great insight was able to NERSC can contribute to their solution. Richard used this informa- engineer this process for older com- success. From these discussions, tion to conduct some tests, and pilers as well. Majdi Baddourah built Richard has undertaken a number found that a runtime setting that extensive test cases and showed of successful projects to improve controls the number of threads used ingenuity in resolving the problems code performance and to resolve in the intrinsic function can increase these test cases exposed. David Paul problems, such as finding a way to code performance by a factor of 2 integrated the multiple compilers install a logistical networking node when running 16 tasks on one node. into the system administration struc- that meets NERSC security require- Using this setting, the beam model- ture. Together they made the use of ments. ing code is now performing better multiple compilers fail-safe for users, One particularly difficult problem than ever. and they gave helpful feedback to involved the SciDAC Accelerator With the variety of codes that IBM on improvements in compiler Science and Technology project’s run on NERSC systems, NERSC support. 3D beam modeling code, which tries to provide users with flexibility To maintain quality services and was scaling poorly on the expanded and a variety of options, avoiding a a high level of user satisfaction, Seaborg. Richard’s initial analysis “one size fits all” mentality. For NERSC conducts an annual user showed that the code’s runtime per- years NERSC and other high per- survey. In 2003, 326 users responded, formance depended strongly on the formance computing (HPC) sites the highest response level in the number of tasks per node, not on have been urging IBM to support survey’s six year history. Overall sat- the number of nodes used. He then multiple versions of compilers, isfaction with NERSC was 6.37 on pinpointed the problem as the since not all codes get the best per- a 7-point satisfaction scale. Areas default threading strategy used by formance from the same compiler with the highest user satisfaction IBM to implement the FORTRAN version. When IBM provided scripts (6.5–6.6) were HPSS reliability, random number intrinsic function, to install versions newer than the timely consulting response, techni- and he enlisted the help of other default compiler—but not older cal advice, HPSS uptime, and local NERSC consultants to find a solu- versions—NERSC staff saw an area network. Areas with the lowest tion. Jonathan Carter and Mike opportunity they could take advan- user satisfaction (4.7–5.0) were Access Grid classes, visualization software and services, and training. In answer to the question “What does NERSC do well?” respondents pointed to NERSC’s good hardware management practices, user support

FIGURE 3 David Skinner, David Paul, and Majdi Baddourah (not pictured) helped NERSC users get the best performance out of their codes by implementing support for multiple compilers.

NERSC CENTER 79

and responsive staff, documenta- During the past year, NERSC tion, and job scheduling and instituted several changes based on throughput. The question “What the results of the 2002 survey: should NERSC do differently?” • Run-time limits were extended so prompted concerns about favoring that jobs could run longer, espe- large jobs at the expense of smaller cially jobs running on more than ones, as well as requests for more 32 nodes. resources devoted to interactive computing and debugging, more • Interactive jobs were given a higher compute power overall, different priority than debugging jobs. architectures, mid-range computing • Two new queue classes were support, vector architectures, better added to facilitate pre- and post- documentation, and more training. processing and data transfers to Complete survey results are avail- HPSS. able at http://hpcf.nersc.gov/about/ survey/2003/first.php. • Software was enhanced to make the SP environment easier to use.

• The NERSC user web site (http://hpcf.nersc.gov) was restruc- tured with new navigation links that make finding information faster and easier.

• The presentation of SP documen- tation was streamlined.

• More training was provided on performance analysis, optimiza- tion, and debugging.

• More information on initial account setup was added to the New User Guide, which was also reformatted for ease of use.

80 NERSC CENTER

CONNECTIVITY

As a national facility whose GridFTP data transfer functionality, SoBig, Klez, Slammer, Blaster, and users routinely transfer massive including Seaborg, HPSS, PDSF, Bugbear were wreaking havoc quantities of data, NERSC needs and the math and visualization across the Internet. In the case of fast networks as much as fast com- servers. Globus libraries have been SoBig, over one million computers puters. DOE’s Energy Sciences installed on Seaborg’s interactive were infected within the first 24 Network (ESnet) provides high per- nodes and will be running on all hours and over 200 million comput- formance connections between nodes in early 2004, when the ers were infected within a week. DOE sites as well as links to other Globus Gatekeeper interface will Nevertheless, NERSC had no networks on the Internet. This year also be installed, providing basic major security incidents, thanks to NERSC upgraded its border router data and job submission services. its alert security staff and intrusion connection to ESnet to OC-48 (2.4 A Grid interface to the NERSC detection by Bro, which is updated Gb/s). Two one-gigabit Ethernets Information Management (NIM) regularly to defend against emerg- channeled together link the router system will make it easier for users ing threats. In addition, NERSC’s with NERSC’s internal network. to get Grid authentication certifi- server team upgraded the email The router itself was upgraded to cates. The active intrusion-detec- server and added a spam filter. Jumbo Frames capability, enabling tion program Bro is being modified When DOE officials realized NERSC to send 9,000-byte data to monitor Grid traffic. And a Grid- that they were unable to categorize packets across the Internet instead enabled Web portal interface called of the previous 1,500-byte packets. VisPortal is being developed to FIGURE 1 Integration of NERSC’s com- deliver visualization services to Grid Eli Dart, Scott Campbell, Brent Draney, and Howard Walter developed a method- puting and storage systems into the collaborators (see page 85 below). ology to categorize network traffic that DOE Science Grid is almost com- In the past year, viruses and has been adopted across the DOE com- plete. All production systems have worms with names like Nimda, plex.

NERSC CENTER 81

FIGURE 2 This team won top honors for the high- est performing application in the SC2002 Bandwidth Challenge competi- and measure different types of net- traffic is clearly driven by science. tion, moving data at a peak speed of 16.8 work traffic on ESnet, NERSC’s Although the Berkeley Lab/ GB/s. The team included staff from networking and security group, led by NERSC Bandwidth Challenge team Berkeley Lab’s Computational Research Howard Walter, stepped in to meet (Figure 2) retired from the competi- (CRD), Information Technologies and Services (ITSD), and NERSC Center the challenge. In less than a month, tion after winning three years in a divisions. Shown here are: (back row) Al using network monitoring data from row at the annual SC conference Early (ITSD), Mike Bennett (ITSD), Bro, routers, and databases, Brent on high performance computing Brian Tierney (CRD), John Christman (ITSD); (middle row) Jason Lee (CRD), Draney, Eli Dart, and Scott Campbell and networking, NERSC’s network- Wes Bethel (CRD), Cary Whitney produced data that clearly catego- ing and security staff continued to (NERSC); (front row) John Shalf (CRD) rized more than 95% of all network play a major role at SC2003 in and Shane Canon (NERSC). Chip Smith and Eli Dart of NERSC are not shown. traffic (Figure 1). The categories they Phoenix, helping to create an infra- developed—bulk data, database, structure providing the 7,500 atten- Grid, interactive, mail, system serv- dees with high-speed wired and been grabbed. Another screen ices, and Web—are now in use at wireless connections. Bro was used showed the number of complete and all DOE laboratories and facilities to monitor the conference network incomplete network connections to monitor and measure network for security breaches, and passwords emanating from the conference. traffic. The most important out- sent unsecurely over the network The strong interest in Bro was come of this effort was the demon- were displayed on a large plasma demonstrated by a standing-room- stration that ESnet is very different screen in the exhibits hall. The dis- only turnout for a presentation on Bro from commercial Internet service play continually attracted viewers, in the Berkeley Lab booth shortly providers, and that DOE network looking to see if their passwords had before the end of the conference.

82 NERSC CENTER

ADVANCED DEVELOPMENT

In the field of high-performance in which teams of scientific applica- tion code for the Parallel FTP computing, there is a saying that if tions specialists and computer sci- (PFTP) file transfer protocol; this you are standing still, you are really entists would work with computer enhancement to PFTP is now being falling behind. On the other hand, architects from major U.S. vendors distributed by IBM. The second just grabbing the newest technology to create hardware and software contribution, in collaboration with in the hope of keeping ahead is environments that will allow scien- Argonne National Laboratory, is a not necessarily a prudent course. tists to extract the maximum per- testbed for the Globus/Grid inter- Knowing what is looming just over formance and capability from the face with HPSS. the horizon—and helping shape it hardware. The paper included a NERSC is also collaborating before it comes completely into conceptual design for a system with the DOE’s Joint Genome view—has long been a hallmark of called Blue Planet that would Institute (JGI) on a project to opti- the NERSC staff. enhance IBM’s Power series archi- mize genomic data storage for wide By working in close partnership tecture for scientific computing. accessibility. This project will dis- with leading developers of high- The first implementation of this tribute and enhance access to data performance computing systems and strategy in 2003 involved a series of generated at JGI’s Production Gen- software, NERSC staff share their meetings between NERSC, IBM, ome Facility (PGF), and will archive collective expertise and experience and Lawrence Livermore National the data on NERSC’s HPSS. The to help ensure that new products Laboratory (LLNL). IBM had won PGF is one of the world’s largest can meet the demands of computa- a contract to build the ASCI Purple public DNA sequencing facilities, tional scientists. Vendors benefit system at LLNL, the world’s first producing 2 million trace data files from this relationship in that supercomputer capable of up to each month (25 KB each), 100 NERSC serves as a real-world testbed 100 teraflop/s, powered by 12,544 assembled projects per month for innovative ideas and technolo- Power 5 processors. LLNL and IBM (50–250 MB), and several very large gies. This section presents current scientists thought that they might assembled projects per year (~50 examples of how NERSC is working enhance the performance of ASCI GB). NERSC storage will make this in partnership to advance the state Purple by incorporating key ele- data more quickly and easily avail- of scientific computing. ments of the Blue Planet system able to genome researchers. Storage architecture, particularly the node will be configured to hold more data CREATING SCIENCE-DRIVEN configuration, and so NERSC staff online and intelligently move the COMPUTER ARCHITECTURE joined in discussions to work out data to enable high-speed access. the details. The resulting architec- NERSC and JGI staff will work In 2002 Lawrence Berkeley and ture enhancements are expected to closely to improve the organization Argonne national laboratories, in benefit all scientific users of future and clustering of DNA trace data so close collaboration with IBM, pro- IBM Power series systems. that portions of an assembled posed a strategy to create a new class dataset can be downloaded instead of computational capability that is MAKING ARCHIVES MORE of the entire project, which could optimal for science. The white paper ACCESSIBLE contain millions of trace files. The “Creating Science-Driven Computer Web interface being developed will Architecture: A New Path to Scien- As a long-time collaborator in for the first time correlate the DNA tific Leadership” (http://www. HPSS development, NERSC made trace data with portions of the nersc.gov/news/blueplanet.html) two major contributions this year. assembled sequence, simplifying envisioned development partnerships The first was the Grid authentica- the selection of trace data, reducing

NERSC CENTER 83

the amount of data that must be of NERSC’s high-performance pro- The major enabling components handled, and enhancing researchers’ duction computational systems. of this envisioned environment are ability to generate partial compar- GUPFS will provide unified file a high-performance shared-disk file isons. Cached Web servers and file namespace for these systems and will system and a cost-effective, high systems will extend the archive to be integrated with HPSS. Storage performance storage-area network the wide area network, distributing servers, accessing the consolidated (SAN). These emerging technolo- data across multiple platforms based storage through the GUPFS shared- gies, while evolving rapidly, are not on data characteristics, then unify- disk file systems, will provide hierar- targeted towards the needs of high- ing data access through one inter- chical storage management, backup, performance scientific computing. face. When this project is complet- and archival services. An additional The GUPFS project intends to ed, its storage and data movement goal is to distribute GUPFS-based encourage the development of techniques, together with its data file systems to geographically remote these technologies to support HPC interfaces, could serve as a model facilities as native file systems over needs through collaborations with for other read-only data repositories. the DOE Science Grid. other institutions and vendors, and When fully implemented, through active development. SHAPING THE FUTURE OF GUPFS will eliminate unnecessary During the first three years of FILE STORAGE data replication, simplify the user the project, the GUPFS team environment, provide better distri- (Figure 1) tested and evaluated In 2001, NERSC began a project bution of storage resources, and per- shared-disk file systems, SAN tech- to develop a system to streamline its mit the management of storage as a nologies, and other components of file storage system. In a typical HPC separate entity while minimizing the GUPFS environment. This environment, each large computa- impacts on the computational systems. investigation included open source tional system has its own local disk along with access to additional net- work-attached storage and archival storage servers. Such an environ- ment prevents the consolidation of storage between systems, thus limit- ing the amount of working storage available on each system to its local disk capacity. The result is an un- necessary replication of files on multiple systems, an increased workload on users to manage their files, and a burden on the infra- structure to support file transfers between the various systems. NERSC is developing a solu- tion using existing and emerging technologies to overcome these inefficiencies. The Global Unified Parallel File System (GUPFS) proj- FIGURE 1 ect aims to provide a scalable, high- Rei Lee, Cary Whitney, Will Baird, and Greg Butler, along with (not shown) Mike Welcome (Computational Research Division) and Craig Tull, are working to develop performance, high-bandwidth, a high performance file storage system that is easy to use and independent of com- shared-disk file system for use by all puting platforms.

84 NERSC CENTER

and commercial shared-disk file sys- can also automate complex work- will change its configuration tools to tems, new SAN fabric technologies flows like the distributed generation make it easier to install the system as they became available, SAN and of MPEG movies or the scheduling on existing hardware configurations. file system distribution over the of file transfers. The VisPortal devel- Performance benchmarks showed wide-area network, HPSS integra- opers are currently working with that Unlimited Linux compares tion, and file system performance NERSC users to improve the work- favorably to the IBM system software and scaling. flow management and robustness of that runs on Alvarez. In 2003, following extended the prototype system. Cray Inc. was given access to trials with multiple storage vendors, Alvarez to test the operating system NERSC selected the YottaYotta under development for the Red A TESTBED FOR CLUSTER NetStorager system to address the Storm supercomputer being devel- SOFTWARE scalability and performance demands oped by Cray Inc. and DOE’s of the GUPFS project. A major goal Three years ago, Berkeley Lab Sandia National Laboratories for the is to provide thousands of heteroge- purchased a 174-processor Linux National Nuclear Security Admin- neous hosts with predictable and cluster, named Alvarez in honor of istration’s Advanced Simulation and scalable performance to a common Berkeley Lab Nobel Laureate Luis Computing (ASCI) program. The pool of storage resources with no Alvarez, to provide a system testbed first round involved testing system increase in operational costs or for NERSC and other staff to evalu- software and administration, and complexity. ate cluster architectures as an option system scalability. The Red Storm for procurement of large production Linux-based software allows simula- computing systems. Recently the tion of multiple virtual processors ENABLING GRID-BASED system has also proven useful for per physical processor, and simula- VISUALIZATION vendors looking to test and develop tions of up to 1,000 processors were The Berkeley Lab/NERSC visu- software for large-scale systems, run on Alvarez. A set of scripts for alization group is developing a Web such as Unlimited Linux and oper- system configuration and easier job portal interface called VisPortal that ating system components for Red launch were developed, and experi- will deliver Grid-based visualization Storm. ments provided some initial data on and data analysis capabilities from Unlimited Scale, Inc. (USI) used launch time for small and large exe- an easy-to-use, centralized point of Alvarez to test a beta version of Un- cutable size programs. The team access. Using standard Globus/Grid limited Linux, an operating system found and fixed approximately 20 middleware and off-the-shelf Web developed to provide production system bugs, from the portals layer automation, VisPortal hides the capabilities on commodity proces- up through MPI and system launch. underlying complexity of resource sors and interconnects that match In the second stage of testing, selection and application management or exceed those of the Cray T3E Cray’s I/O team is working with among heterogeneous distributed supercomputer, once the workhorse NERSC to use the GUPFS testbed computing and storage resources. of the NERSC Center. USI asked platform, and GUPFS joined From a single access point, the user NERSC to assess Unlimited Linux’s together with Alvarez, to do scala- can browse through the data, wher- operational and administrative func- bility testing of two potential Red ever it is located, access computing tionality, performance, desired Storm file systems, PVFS and resources, and launch all the com- features and enhancements, and Lustre. They will also collaborate ponents of a distributed visualization user functionality. A few minor with NERSC staff who are working application without having to know problems were discovered in adapt- on global file systems that will even- all the details of how the various ing Unlimited Linux to the Alvarez tually be deployed within the systems are configured. The portal network address scheme, and USI NERSC Center.

NERSC CENTER 85

APPENDIX A NERSC POLICY BOARD

ROBERT J. GOLDSTON DANIEL A. REED Princeton Plasma Physics University of North Carolina Laboratory ROBERT D. RYNE STEPHEN C. JARDIN (ex officio, NERSC Users (ex officio, NERSC Group Chair) Computational Review Panel) Lawrence Berkeley National Princeton Plasma Physics Laboratory Laboratory TETSUYA SATO SIDNEY KARIN Earth Simulator Center/Japan University of California, San Marine Science and Technology Diego Center

WILLIAM J. MADIA STEPHEN L. SQUIRES Battelle Memorial Institute Hewlett-Packard

PAUL C. MESSINA MICHAEL S. WITHERELL Argonne National Laboratory Fermi National Accelerator Laboratory ALBERT NARATH Lockheed-Martin Corporation (retired)

APPENDICES 87

APPENDIX B NERSC COMPUTATIONAL REVIEW PANEL (CORP)

COMPUTATIONAL SCIENCE FUSION ENERGY Ralph Byers Stephen Jardin University of Kansas HIGH ENERGY PHYSICS Princeton Plasma Physics Robert Sugar Marc Day Laboratory University of California, Santa Lawrence Berkeley National Barbara Donald Spong Laboratory Oak Ridge National Laboratory ACCELERATOR PHYSICS Eric de Sturler Robert Ryne CLIMATE RESEARCH AND University of Illinois at Urbana- Lawrence Berkeley National ENVIRONMENTAL SCIENCES Champaign Laboratory Ferdinand Baer Tony Drummond University of Maryland NUCLEAR AND Lawrence Berkeley National ASTROPHYSICS Douglas Rotman Laboratory Douglas Swesty Lawrence Livermore National Joseph Grcar State University of New York at Laboratory Lawrence Berkeley National Stony Brook LIFE SCIENCES Laboratory CHEMISTRY Brian Hingerty Van Henson Robert Harrison Oak Ridge National Laboratory Lawrence Livermore National Oak Ridge National Laboratory COMPUTER SCIENCE AND Laboratory MATERIAL SCIENCES APPLIED MATHEMATICS Ilse Ipsen David Turner David Bailey North Carolina State University Ames Laboratory Lawrence Berkeley National Laboratory Andrew Knyazev Michael Weinert University of Colorado at Denver Brookhaven National Laboratory APPLIED MATHEMATICS Robert Ward EARTH AND ENGINEERING Jesse Barlow University of Tennessee SCIENCES Pennsylvania State University David Alumbaugh Chao Yang University of Wisconsin Christopher Beattie Lawrence Berkeley National Virginia Polytechnic Institute and Laboratory State University

88 APPENDICES COMPUTER SCIENCE Allan Snavely CHEMISTRY San Diego Supercomputer Jonathan Carter Jim Ahrens Center David Skinner Los Alamos National Laboratory Erich Strohmaier MATERIAL SCIENCES Robert Armstrong Lawrence Berkeley National Andrew Canning Sandia National Laboratories Laboratory Lin-Wang Wang David Bernholdt Rajeev Thakur EARTH AND ENGINEERING Syracuse University Argonne National Laboratory SCIENCES Brett Bode Osni Marques Mary Zosel Ames Laboratory Lawrence Livermore National FUSION ENERGY Sung-Eun Choi Laboratory Thomas DeBoni Los Alamos National Laboratory Jodi Lamoureux NERSC STAFF REVIEWERS Erik DeBenedictis CLIMATE RESEARCH AND Sandia National Laboratories HIGH ENERGY AND NUCLEAR ENVIRONMENTAL SCIENCES PHYSICS Chris Ding William Gropp Iwona Sakrejda Helen He Argonne National Laboratory LATTICE QCD Michael Wehner James Kohl Parry Husbands LIFE SCIENCES Oak Ridge National Laboratory ACCELERATOR PHYSICS Adrian Wong John Mellor-Crummey Andreas Adelmann Jonathan Carter Rice University ASTROPHYSICS APPLIED MATHEMATICS AND COMPUTER SCIENCE Shirley Moore Julian Borrill University of Tennessee Peter Nugent Esmond Ng Robert Numrich University of Minnesota

APPENDICES 89

APPENDIX C NERSC USERS GROUP EXECUTIVE COMMITTEE

MEMBERS AT LARGE OFFICE OF BASIC ENERGY Carl Sovinec,** University of SCIENCES Wisconsin, Madison Ahmet Y. Aydemir,* University of Texas at Austin Feng Liu,** University of Utah Donald A. Spong,* Oak Ridge National Laboratory David J. Dean, Oak Ridge National G. Malcolm Stocks, Oak Ridge Laboratory National Laboratory

Stephen C. Jardin, Princeton Theresa L. Windus, Pacific OFFICE OF HIGH ENERGY AND NUCLEAR PHYSICS Plasma Physics Laboratory Northwest National Laboratory

Ricky A. Kendall, Ames Laboratory, Kwok Ko, Stanford Linear Accelerator Center Iowa State University OFFICE OF BIOLOGICAL AND ENVIRONMENTAL RESEARCH Gregory Kilcup,* The Ohio State Douglas L. Olson, Lawrence University Carmen M. Benkovitz,* Berkeley National Laboratory Brookhaven National Laboratory Donald Sinclair,** Argonne Robert D. Ryne (NUG Chair), National Laboratory Michael Colvin,** University of Lawrence Berkeley National California, Merced, and Laboratory Donald A. Spong,** Oak Ridge Lawrence Livermore National National Laboratory Laboratory John A. Taylor, Argonne National * Outgoing members Douglas Rotman (NUG Vice Laboratory Chair), Lawrence Livermore ** Incoming members National Laboratory

OFFICE OF ADVANCED Vincent Wayland, National Center SCIENTIFIC COMPUTING for Atmospheric Research RESEARCH

David Bernholdt,** Oak Ridge National Laboratory OFFICE OF FUSION ENERGY SCIENCES Ahmed Ghoniem,* Massachusetts Institute of Technology Stephane Ethier,** Princeton Plasma Physics Laboratory David Keyes, Old Dominion University Alex Friedman, Lawrence Livermore and Lawrence Xiaolin Li,* State University of Berkeley National Laboratories New York, Stony Brook Jean-Noel Leboeuf,* University of David Turner,** Ames Laboratory California, Los Angeles

90 APPENDICES APPENDIX D SUPERCOMPUTING ALLOCATIONS COMMITTEE

The Supercomputing Allocations Committee (SAC) is responsible for setting the policies associated with the utilization of computing, data stor- age, and other auxiliary services available to DOE Office of Science (SC) researchers and otherwise coordinating SC’s computational projects. The Committee sets the distribution of NERSC and other available Office of Advanced Scientific Computing Research resources for scientific comput- ing applications every year.

COMMITTEE CHAIRMAN OFFICE OF BIOLOGICAL AND ENVIRONMENTAL RESEARCH David Goodwin Michael Riches

OFFICE OF ADVANCED SCIENTIFIC COMPUTING OFFICE OF FUSION ENERGY RESEARCH SCIENCES

Daniel Hitchcock Michael Crisp

OFFICE OF BASIC ENERGY OFFICE OF HIGH ENERGY SCIENCES PHYSICS AND OFFICE OF NUCLEAR PHYSICS Robert Astheimer David Goodwin

APPENDICES 91

APPENDIX E OFFICE OF ADVANCED SCIENTIFIC COMPUTING RESEARCH

Mathematical, Information, and Computational Sciences Division

The primary mission of the Lab thanks the program managers Advanced Scientific Computing with direct responsibility for the Research (ASCR) program, which NERSC program and the MICS is carried out by the Mathematical, research projects described in this Information, and Computational report: Sciences (MICS) subprogram, is to discover, develop, and deploy the C. EDWARD OLIVER GARY M. JOHNSON computational and networking tools Associate Director, Office of Advanced Computing Research that enable researchers in the scien- Advanced Scientific Computing Testbeds tific disciplines to analyze, model, Research, and Acting MICS simulate, and predict complex phe- THOMAS D. NDOUSSE- Director nomena important to the Depart- FETTER ment of Energy. To accomplish this ALAN J. LAUB AND Networking mission, the program fosters and WILLIAM H. KIRCHOFF supports fundamental research in CHARLES H. ROMINE SciDAC advanced scientific computing— Applied Mathematics and Statistics applied mathematics, computer sci- WALTER M. POLANSKY ence, and networking—and oper- MARY ANNE SCOTT Acting MICS Director ates supercomputer, networking, (until March 2003) Collaboratory Research and related facilities. In fulfilling this primary mission, the ASCR pro- WILLIAM H. (BUFF) MINER, JR. GEORGE R. SEWERYNIAK gram supports the Office of Science NERSC and Applications Program ESnet Strategic Plan’s goal of providing Manager extraordinary tools for extraordinary LINDA TWENTY science as well as building the foun- DANIEL HITCHCOCK ASCR Program Analyst dation for the research in support of ASCR Senior Technical Advisor the other goals of the strategic plan. JOHN R. VAN ROSENDALE In the course of accomplishing this FREDERICK C. JOHNSON Computer Science Research mission, the research programs of Computer Science Research ASCR have played a critical role in the evolution of high performance computing and networks. Berkeley

92 APPENDICES

APPENDIX F ADVANCED SCIENTIFIC COMPUTING ADVISORY COMMITTEE

The Advanced Scientific Com- computing to other scientific disci- puting Advisory Committee (ASCAC) plines, and maintaining appropriate provides valuable, independent balance among elements of the advice to the Department of Energy program. The Committee formally on a variety of complex scientific reports to the Director, Office of and technical issues related to its Science. The Committee primarily Advanced Scientific Computing includes representatives of universi- Research program. ASCAC’s rec- ties, national laboratories, and ommendations include advice on industries involved in advanced long-range plans, priorities, and computing research. Particular strategies to address more effective- attention is paid to obtaining a ly the scientific aspects of advanced diverse membership with a balance scientific computing including the among scientific disciplines, institu- relationship of advanced scientific tions, and geographic regions.

CO-CHAIRS Roscoe C. Giles Thomas A. Manteuffel Center for Computational Department of Applied Margaret H. Wright Science Mathematics Courant Institute of Boston University University of Colorado at Mathematical Sciences Boulder New York University James J. Hack National Center for Karen R. Sollins John W. D. Connolly Atmospheric Research Massachusetts Institute of Center for Computational Technology Sciences Helene E. Kulsrud University of Kentucky Center for Communications Ellen B. Stechel Research Ford Motor Company

MEMBERS William A. Lester, Jr. Stephen Wolff Department of Chemistry CISCO Systems Jill P. Dahlburg University of California, Berkeley Naval Research Laboratory Gregory J. McRae David J. Galas Department of Chemical Keck Graduate Institute Engineering Massachusetts Institute of Technology

APPENDICES 93

APPENDIX G ORGANIZATIONAL ACRONYMS

DOE U.S. Department of Energy SC Office of Science ASCR-MICS Office of Advanced Scientific Computing Research, Mathematical, Information, and Computational Sciences Division BER Office of Biological and Environmental Research BES Office of Basic Energy Sciences FES Office of Fusion Energy Sciences HEP Office of High Energy Physics NP Office of Nuclear Physics SciDAC Scientific Discovery through Advanced Computing Program

3M 3M Company AEI Albert Einstein Institute/Max Planck Institute for Gravitation Physics (Germany) AF Aventis Foundation AFOSR Air Force Office of Scientific Research ARO Army Research Office BMBF Bundesministerium für Bildung und Forschung (Germany) BWF Burroughs Wellcome Fund CAG Cellzome AG CDWR California Department of Water Resources CNR Consiglio Nazionale delle Ricerche (Italy) CNRS Centre National de la Recherche Scientifique (France) DAAD German Academic Exchange Service DARPA Defense Advanced Research Projects Agency DOD Department of Defense DRCRF Damon Runyon Cancer Research Foundation DFG Deutsche Forschungsgemeinschaft

94 APPENDICES

APPENDIX G

DRA Danish Research Agency EMBL European Molecular Biology Laboratory EMBO European Molecular Biology Organization EPSRC Engineering and Physical Sciences Research Council (UK) ESOP European Electron Scattering Off Confined Partons Network EU European Union FAPESP Fundação de Amparo à Pesquisa do Estado de São Paulo, Brazil FBF France-Berkeley Fund GA General Atomics GNMF German National Merit Foundation HFS Holderbank Foundation Switzerland HFSP Human Frontier Science Program IBM SUR IBM Shared University Research Grant IN2P3 Institut National de la Physique Nucléaire et de la Physique des Particules (France) INSU Institut National des Sciences de l’Univers (France) Intel Intel Corporation JIHIR Joint Institute for Heavy Ion Research KBN Polish State Committee for Science Research LANL Los Alamos National Laboratory LBNL Lawrence Berkeley National Laboratory LLNL Lawrence Livermore National Laboratory LSU Louisiana State University MCYT Spanish Ministry of Science and Technology MEC Ministry of Education of China MEVU William A. and Nancy F. McMinn Endowment at Vanderbilt University MEXT Japan Ministry of Education, Culture, Sports, Science, and Technology

APPENDICES 95

APPENDIX G

MIT Massachusetts Institute of Technology MIUR Ministero dell’Istruzione, dell’Università e della Ricerca (Italy) NASA National Aeronautics and Space Administration NCI National Cancer Institute NEDO New Energy Development Organization (Japan) NIH National Institutes of Health NNSFC National Natural Science Foundation of China NOAA National Oceanographic and Atmospheric Administration NRC National Research Council NSERC Natural Sciences and Engineering Research Council of Canada NSF National Science Foundation NSFC National Science Foundation of China ONR Office of Naval Research ORNL Oak Ridge National Laboratory PECASE Presidential Early Career Award for Scientists and Engineers PNC Programme National de Cosmology (France) PPARC Particle Physics and Astronomy Research Council (UK) RFBR Russian Foundation for Basic Research RMST Russian Ministry of Science and Technology SNS Spallation Neutron Source Project UC University of Cyprus

96 APPENDICES Published by the Berkeley Lab Technical and Electronic Information Department in collaboration with NERSC researchers and staff. JO#8798

EDITOR AND PRINCIPAL WRITER: John Hules CONTRIBUTING WRITERS: Rich Albert, Jon Bashor, Christopher Jones, Paul Preuss, Lynn Yarris

DISCLAIMER

This document was prepared as an account of work sponsored by the United States Government. While this document is believed to contain correct information, neither the United States Government nor any agency thereof, nor The Regents of the University of California, nor any of their employees, makes any warranty, express or implied, or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or The Regents of the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, or The Regents of the University of California.

Ernest Orlando Lawrence Berkeley National Laboratory is an equal opportunity employer.