Scientific Discovery

A progress report on the U.S. Department SCIENTIFIC DISCOVERY of Energy's Scientific Discovery through Advanced Computing (SciDAC) Program Cover: When the cores of massive stars collapse to form black holes and accretion disks, relativistic jets known as gamma ray bursts pass through and then explode the star. SciDAC’s Supernova Science Center used simulations like the one of the cover to test and confirm theories about the origin of gamma-ray bursts.

This work was supported by the Director, Office of Science, Office of Advanced Scientific Computing Research of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. A progress report on the U.S. Department SCIENTIFIC DISCOVERY of Energy's Scientific Discovery through Advanced Computing (SciDAC) Program 2 EXECUTIVE SUMMARY

laboratories and universities across the country. The SciDAC Program consisted of three research components: • Creating a new generation of scientific simulation codes to take full advantage of the powerful computing capabilities of terascale supercomputers Executive Summary • Creating the mathematical and computing systems software to enable these scientific simulation Greetings: scientists created the first labora- codes to effectively and efficiently tory-scale flame simulation in use terascale computers Five years ago, the U.S. Department three dimensions, an achievement • Creating a distributed science of Energy Office of Science launched which will likely help improve software infrastructure to enable an innovative software development efficiency and reduce pollution. scientists to effectively collaborate program with a straightforward name • Looking toward future energy in managing, disseminating and — Scientific Discovery through resources, magnetic fusion analyzing large datasets from Advanced Computing, or SciDAC. researchers, applied mathemati- large-scale computer simulations The goal was to develop scientific cians and computer scientists and scientific experiments and applications to effectively take advan- have worked together to success- observations. tage of the terascale supercomputers fully carry out advanced simula- Not only did the researchers work (capable of performing trillions of tions on the most powerful modern in teams to complete their objectives, calculations per second) then becom- supercomputing platforms. They but the teams also collaborated with ing widely available. discovered the favorable result one another to help each other suc- From the most massive explo- that for the larger reactor-scale ceed. In addition to advancing the sions in our universe to our planet’s plasmas of the future such as overall field of scientific computing changing climate, from developing ITER (the planned international by developing applications which can future energy sources to understand- fusion experiment), heat losses be used by other researchers, these ing the behavior of the tiniest parti- caused by plasma turbulence do teams of experts have set the stage cle, the SciDAC program has lived not continue to follow the empiri- for even more accomplishments. up to its name by helping scientists cal trand of increasing with the This report will describe many of make important discoveries in many size of the system. the successes of the first five years of scientific areas. • To make the most of existing par- SciDAC, but the full story is probably Here are some examples of the ticle accelerators and reduce the too extensive to capture — many of achievements resulting from SciDAC: cost of building future accelerators, the projects have already spurred • For the first time, astrophysicists teams developed new methods researchers in related fields to use the created fully resolved simulations for simulating improvements. In applications and methods developed of the turbulent nuclear combus- addition to helping us understand under SciDAC to accelerate their tion in Type 1a supernovae, the most basic building blocks of own research, triggering even more exploding stars which are critical matter, accelerators make possible breakthroughs. to better understanding the nuclear medicine. While SciDAC may have started nature of our universe. • Physicists studying the Standard out as a specific program, Scientific • Using climate modeling tools Model of particle interactions Discovery through Advanced developed and improved under have, after 30 years of trying, been Computing has become a powerful SciDAC, climate change scientists able to model the full spectrum of concept for addressing some of the in the United States are making particles known as hadrons at the biggest challenges facing our nation the largest contribution of global highest level of accuracy ever. The and our world. climate modeling data to the results could help lead to a deeper world’s leading body on climate understanding of the fundamental studies, the Intergovernmental laws of physics. Michael Strayer Panel on Climate Change. These and other impressive sci- • To better understand combustion, entific breakthroughs are the results SciDAC Program Director which provides 80 percent of the of hundreds of researchers working U.S. Department of Energy energy used in the United States, in multidisciplinary teams at national Office of Science

TABLE OF CONTENTS

Table of Contents

Introduction: Scientific Discovery through Advanced Computing...... 6

Scientific Discovery...... 8 Supernovae Science: Earth-based Combustion Simulation Tools Yield Insights into Supernovae ...... 8 From Soundwaves to Supernova: SciDAC Provides Time, Resources to Help Astrophysicists Simulate a New Model ...... 11 Insight into Supernovae: SciDAC’s Terascale Supernova Intiative Discovers New Models for Stellar Explosions, Spinning Pulsars...... 14 CCSM: Advanced Simulation Models for Studying Global Climate Change ...... 18 Plasma Physics: Terascale Simulations Predict Favorable Confinement Trends for Reactor-Scale Plasmas ...... 22 Fueling the Future: Simulations Examine the Behavior of Frozen Fuel Pellets in Fusion Reactors ...... 26 Burning Questions: New 3D Simulations Are Closing in on the Holy Grail of Combustion Science ...... 30 Pushing the Frontiers of Chemistry: Dirac’s Vision Realized ...... 34 Accelerators: Extraordinary Tools for Extraordinary Science...... 36 Lattice QCD: Improved Formulations, Algorithms and Computers Result in Accurate Predictions of Strong Interaction Physics ...... 40

Scientific Challenge Codes: Designing Scientific Software for Next-Generation Supercomputers ...... 43 Basic Energy Sciences: Understanding Energy at the Atomic Level ...... 43 Biological and Environmental Research: Advanced Software for Studying Global Climate Change...... 46 Fueling the Future: Plasma Physics and Fusion Energy ...... 52 High Energy and Nuclear Physics: Accelerating Discovery from Subatomic Particles to Supernovae ...... 58

Building a Better Software Infrastructure for Scientific Computing ...... 67

Applied Mathematics ISICs ...... 67 Computer Science ISICs 71

Creating an Advanced Infrastructure for Increased Collaboration...... 76

National Collaboratory Projects...... 76 Middleware Projects 81 Networking Projects ...... 82

List of Publications ...... 86

5 INTRODUCTION

Today, almost five years after it was announced, the SciDAC pro- Scientific Discovery gram has lived up to its name and goal of advancing scientific discovery through Advanced through advanced computing. Teaming Up for Better Science In the 1930s, physicist Ernest Computing Orlando Lawrence pioneered a new approach to scientific research. Rather than working alone in their labs, physicists, engineers, chemists Over the past 50 years, computers have transformed nearly and other scientists were brought together to form multidisciplinary every aspect of our lives, from entertainment to education, teams which could bring a range of medicine to manufacturing, research to recreation. Many of knowledge and expertise to bear on these innovations are made possible by relentless efforts by solving scientific challenges. This “big science” approach formed the basis scientists and engineers to improve the capabilities of high for the national laboratories operated performance computers, then develop computational tools to by the Department of Energy. Since the 1950s, scientists at the take full advantage of those capabilities. DOE national labs have increasingly employed high performance computers in their research. As a result, As these computer systems opment of a new generation of tools computational science has joined become larger and more powerful, and technologies for scientific com- experimental science and theoretical they allow researchers to create more puting. SciDAC projects were selected science as one of the key methods of detailed models and simulations, to capitalize on the proven success of scientific discovery. enabling the design of more complex multidisciplinary scientific teams. In With SciDAC, this approach has products, ranging from specialized all, 51 projects were announced involv- come full circle, with mutlidiscipli- medicines to fuel-efficient airplanes. ing collaborations among 13 DOE nary teams from various research Critical to the success of these efforts national laboratories and more than institutions working together to is an every-increasing understanding 50 colleges and universities. develop new methods for scientific of the complex scientific processes SciDAC is an integrated program discovery, while also developing and principles in areas such as that has created a new generation of technologies to advance collaborative physics, chemistry and biology. scientific simulation codes. The codes science. Over the past five years, teams of are being created to take full advan- And as with other research scientists and engineers at national tage of the extraordinary computing accomplishments, the results are laboratories operated by the U.S. capabilities of today’s terascale com- being shared through articles pub- Department of Energy, along with puters (computers capable of doing lished in scientific and technical jour- researchers at universities around the trillions of calculations per second) to nals. Additionally, the tools and tech- country, have been part of a concert- address ever larger, more complex niques developed under the SciDAC ed program to accelerate both the problems. The program also includes program are freely available to other performance of high performance research on improved mathematical scientists working on similar chal- computers and the computing alloca- and computing systems software that lenges. tions used to advance our scientific will allow these codes to use modern knowledge in areas of critical impor- parallel computers effectively and tance to the nation and the world. efficiently. Additionally, the program A Full Spectrum of Scientific Launched in 2001, this broad- is developing “collaboratory” software Discovery based program is called Scientific to enable geographically separated The SciDAC program was creat- Discovery through Advanced scientists to effectively work together ed to advance scientific research in Computing, or SciDAC for short. as a team, to control scientific instru- all mission areas of the Department The $57 million-per-year program ments remotely and to share data of Energy’s Office of Science. From was designed to accelerate the devel- more readily. looking back into the origins of our

6 INTRODUCTION

universe to predicting global climate change, from researching how to Computing then and now burn fuels more efficiently and clean- ly to developing environmentally While the achievements of the SciDAC and economically sustainable energy teams are impressive on their own, they also sources, from investigating the behavior of the smallest particles to demonstrate just how far scientific computing learning how to combine atoms to has come in the past 20 years. create new nanotechnologies, computational science advanced under Back in 1986, a state-of-the-art supercom- SciDAC affects our lives on many puter was a Cray-2, which had a peak speed of levels. Today, almost five years after it 2 gigaflops (2 billion calculations) per second was announced, the SciDAC pro- and a then-phenomenal 2 gigabytes of memory. gram has lived up to its name and goal of advancing scientific discovery Such machines were rare, located only at a lim- through advanced computing. This ited number of research institutions, and access progress report highlights a number of the scientific achievements made was strictly controlled. Scientific programs possible by SciDAC and also looks at were typically written by individuals, who the computing methods and infrastructure created under the program coded everything from scratch. Tuning the which are now driving computation- code to improve performance was done by al science advances at research institutions around the world. hand – there were no tools freely available, no The achievements range from means of creating visualizations. The lucky sci- understanding massive exploding stars known as supernovae to study- entists were able to submit jobs remotely using ing the tiniest particles which com- a 9600 baud modem. bine to form all the matter in the universe. Other major accomplish- Today, inexpensive highly parallel com- ments include better methods for modity clusters provide teraflops-level per- studying global climate, developing new sources of energy, improving formance (trillions of calculations per second) combustion to reduce pollution and and run on open source software. Extensive designing future facilities for scientific research. libraries of tools for high performance scientific computing are widely available — and expanding, thanks to SciDAC. Computing grids offer remote access at 10 gigabits per second, allowing hundreds or thousands of researchers to use a single supercomputer. Scientists can do code development on desktop computers which have processors faster than the Cray-2 and offer more memory. Once an application runs, powerful visualization tools provide detailed images which can be manipulated, studied and compared to experimental results.

7 Supernovae Science: Earth-Based Combustion Simulation Tools Yield Insights into Supernovae

In the far reaches of space, a small star about the size of Earth simmers away, building up the heat and pressure necessary to trigger a massive thermonuclear explosion. FIGURE 1. This series shows the development of a Rayleigh-Taylor unstable flame. The interface between the fuel and ash is visualized The buoyant ash rises upward as the fuel falls downward in the strong gravitational field, wrinkling the flame front. This increases the surface area of the flame and accelerates the burning. At late times, the flame has become fully turbulent.

The star, known as a white dwarf, as a supernova. Under SciDAC, teams Computational scientists tackle consists of oxygen and carbon. At its of researchers worked to develop such problems by dividing them into center, the star is about two billion computational simulations to try to small cells, each with different values times denser than water. Already 40 understand the processes. It turns out for density, pressure, velocity and percent more massive than our Sun, that an algorithm developed to simu- other conditions. The conditions are the star continues to gain mass as a late a flame from a Bunsen burner in then evolved one time step at a time, companion star dumps material onto a laboratory is well suited — with some using equations which say how things the surface of the white dwarf. adaptation — to studying the flame like mass, momentum and energy As this weight is added, the inte- front moving through a white dwarf. change over time, until a final time is rior of the star compresses, further The SciDAC program brought reached. The astrophysicists had been heating the star. This causes the car- together members of the Supernova using a code that included modeling bon nuclei to fuse together, creating Science Center project and LBNL the effects of sound waves on the heavier nuclei. The heating process, mathematicians and computational flame front, which limited them to much like a pot of water simmering scientists associated with the Applied very small time steps. So, modeling a on the stove, continues for more than Partial Differential Equations problem over a long period of time 100 years, with plumes of hot material Integrated Software Infrastructure exceeded the availability of computing moving through the star by convection Center (ISIC). “We had two groups resources. The CCSE code allowed and distributing the energy. in very different fields,” said Michael the astrophysicists to filter out the Eventually, the nuclear burning Zingale, a professor at the State sound waves, which aren’t important occurs at such a rate that convection University of New York–Stony Brook in this case, and take much bigger cannot carry away all the energy that and member of the Supernovae time steps, making more efficient use is generated and the heat builds dra- Science Center. “We met with these of their supercomputing allocations. matically. At this point, a burning mathematicians who had developed The net result, according to thermonuclear flame front moves codes to study combustion and Zingale, is that the project team was quickly out from the core of the star, through SciDAC, we started using able to carry out studies which had burning all the fuel in a matter of their tools to study astrophysics.” never been possible before. Their seconds. The result is a Type Ia The adaptive mesh refinement findings have been reported in a supernova, an exploding star which is (AMR) combustion codes developed series of papers published in scientific one of the brightest objects in the at the Center for Computational journals such as Astrophysical universe. Because of their brightness, Sciences and Engineering (CCSE) at Journal. supernovae are of great interest to Lawrence Berkeley National While observational astronomers astrophysicists studying the origin, Laboratory are very effective for are familiar with the massive explo- age and size of our universe. modeling slow-moving burning sion which characterizes a Type Ia Although astronomers have been fronts. The behavior of the flame supernova, one of the unknowns is observing and recording supernovae front leading up to a supernova is where the flame starts, or if it starts for more than 1,000 years, they still very similar, and the scientists in more than one location. When the don’t understand the exact mechanisms worked together to adapt the codes flame front starts out, it is moving at which cause a white dwarf to explode to astrophysical environments. about one one-thousandth of the SCIENTIFIC DISCOVERY

speed of sound. As it burns more fuel, Astrophysicists had predicted that the flame front speeds up. By the time as the flame front burns further out the star explodes, the front must be from the center, the lower density of moving at half the speed of sound. the star would cause it to burn less The trick is, how does it accelerate to vigorously and become more unstable that speed? due to increased turbulence and mix- One answer explored by the ing of fuel and ash. This represents a SciDAC team is the effect of a wrin- different mode of combustion known kled flame. As the flame becomes as a “distributed burning regime.” wrinkled, it has more surface area, Scientists believe that such combus- which means it burns more fuel, tion occurs in the late stage of a which in turn accelerates the flame supernova explosion. front. The AMR codes were used to Using the AMR combustion codes model small areas of the flame front and running simulations for 300,000 in very high resolution. processor hours at DOE’s National As the flame burns, it leaves in its Energy Research Scientific Computing wake “ash,” which through intense Center, the Supernova Science Center heat and pressure is fused into nickel. team was able to create the first-ever This hot ash — measuring several bil- three-dimensional simulations of such lion degrees Kelvin — is less dense an event. The results are feeding into than the fuel. The fuel ahead of the the astrophysics community’s knowl- front is relatively cooler, at about 100 edge base. million degrees Kelvin, and denser “While other astrophysicists are than the ash. These conditions make modeling entire stars at a different the flame front unstable. This instabil- scale, they need to know what is ity, known as the Rayleigh-Taylor going on at scales their models can’t instability, causes the flame to wrinkle. resolve — and we’re providing that As the flame starts out from the model for them,” Zingale said. center and burns through most of the “Although it takes a large amount of star, the front between the fuel and computing time, it is possible now the ash is sharp and the front is called thanks to these codes. Before our a “flamelet.” SciDAC partnership, it was impossible.”

FIGURE 2. This series of images show the growth of a buoyant reactive bubble. At the density simulated, the rise velocity is greater than the laminar flame velocity, and the bubble distorts significantly. The vortical motions transform it into a torus. Calculations of reactive rising bubbles can be used to gain more insight into the early stages of flame propagation in Type Ia supernovae.

10 SCIENTIFIC DISCOVERY

the physics of all the interactions in the star. From Soundwaves to One thing that is known is that supernovae produce neutrinos, particles with very little mass which travel through space and everything in their Supernova: path. Neutrinos carry energy from the deep interior of the star, which is SciDAC Provides Time, Resources to Help being shaken around like a jar of supersonic salad dressing, and deposit Astrophysicists Simulate a New Model the energy on the outer region. One theory holds that if the neutrinos deposit enough energy throughout Once every 30 to 50 years — and then just for a few the star, this may trigger the explosion. milliseconds — an exploding star known as a core- To study this, a group led by Adam Burrows, professor of astrono- collapse supernova is the brightest object in the uni- my at the University of Arizona and a verse, brighter than the optical light of all other stars member of the SciDAC Supernova Science Center, developed codes for combined. simulating the behavior of a supernovae core in two dimensions. While a 3D version of the code would be Supernovae have been documented only because of their central role in optimum, it would take at least five for 1,000 years and astrophysicists astronomy and nucleosynthesis, but more years to develop and would know a lot about how they form, because a full understanding of super- require up to 300 times as much what happens during the explosion novae may lead to a better under- computing time. As it was, the group and what’s left afterward. But for the standing of basic physics. ran 1.5 million hours of calculations past 40 years, one problem has The massive stars which become at DOE’s National Energy Research dogged astrophysicists — what is the supernovae feature a number of con- Scientific Computing Center. mechanism that actually triggers the ditions which are also found on Earth But the two-dimensional model is massive explosion? Under SciDAC, a — winds, fluid flows, heating and suitable for Burrows’ work, and the number of computational projects cooling. But whereas a hurricane may instabilities his group is interested in were established to simulate plausible blast along at 150 kilometers an hour studying can be seen in 2D. What explanations. One group created simon earth, the winds on these stars they found was that there is a big ulations showing a new mechanism rage at tens of kilometers per second. overturning motion in the core, for core-collapse supernovae. This The temperature of the star is 100 which leads to wobbling, which in model indicates that the core of the million times that of Earth, and the turn creates sound waves. These star generates sound waves which in density of the object can be 14 orders waves then carry energy away from turn become shock waves powerful of magnitude more dense than lead. the core, depositing it farther out near enough to trigger the explosion. In short, studying supernovae is a the mantle. Understanding these explosions very complicated problem. Key to According to Burrows, these and their signals is important, not finding an answer is understanding oscillations could provide the “power

11 SCIENTIFIC DISCOVERY

source” which puts the star over the edge and causes it to explode. To imagine what such a scenario would look like, think of a pond into which rocks are thrown, causing waves to ripple out. Now think of the pond as a sphere, with the waves moving throughout the sphere. As the waves move from the denser core to the less dense mantle, they speed up. According to the model, they begin to crack like a bullwhip, which creates shockwaves. It is these shockwaves, Burrows believes, which could trigger the explosion. So, what led the team to this model? They came up with the idea FIGURE 1. A 2D rendition of the entropy field of the early blast in the inner 500 km of an by following the pulsar — the neutron exploding supernova. Velocity vectors depict the direction and magnitude of the local flow. The bunching of the arrows indicates the crests of the sound waves that are escalating into star which is the remains of a superno- shock waves. These waves are propagating outward, carrying energy from the core to the va. They wanted to explore the origin mantle and helping it to explode. The purple dot is the protoneutron star, and the purple streams crashing in on it are the accretion funnels. (All images courtesy of Adam Burrows, of the high speed which pulsars seem University of Arizona) to be born with and this led them to create a code that allowed the core to move. However, when they implemented this code, the core not only recoiled, but oscillated, and generated sound waves. The possible explosive effect of the oscillations had not been considered before because previous simulations of the conditions inside the core used smaller time steps, which consumed more computing resources. With this limitation, the simulation ran their course before the onset of oscillations. With SciDAC support, however, Burrows’ team was able to develop new codes with larger time steps, allowing them to model the oscillations for the first time. Calling the simulation a “real

12 SCIENTIFIC DISCOVERY

FIGURE 2. These shells are isodensity contours, colored according to entropy values. The red (blast area) indicates regions of high entropy, and the orange outer regions have low entropy. In the image on the left, matter is accreting from the top onto the protoneutron star in the center (the orange dot that looks like the uvula in the throat). The image is oriented so that the anisotropic explosion is emerging down and towards the viewer. The scale is roughly 5000 km. The image on the right shows the same explosion later in time and at a different orien- tation with respect to the viewer.

FIGURE 3. Another isodensity shell colored with entropy, showing simultaneous accretion on the top and explosion on the bottom. The inner green region is the blast, and the outer orange region is the unshocked material that is falling in. The purple dot is the newly numerical challenge,” Burrows said formed neutron star, which is accumulat- the resulting approach “liberated the ing mass through the accretion funnels (in orange). inner core to allow it to execute its natural multidimensional motion.” This motion led to the excitation of the core, causing the oscillations at a distinct frequency. SciDAC made the work possible, he said, by providing support over five years — enough time to develop and test the code — and the computing resources to run the simulations. The results look promising, but as is often the case, more research is needed before a definitive mechanism for triggering a supernova is determined. The group published a paper on their research in the Astrophysical Journal. Neutrino transfer is included as a central theme in a 2D multi-group, multi-neutrino, flux-limited transport scheme. It is approximate but has the important components, and is the only truly 2D neutrino code with results published in the archival literature. “The problem isn’t solved,” Burrows said. “In fact, it’s just beginning.”

13 Insight into Supernovae: SciDAC’s Terascale Supernova Initiative Discovers New Models for Stellar Explosions, Spinning Pulsars FIGURE 1: This image provides a snapshot of the angular momentum of the layered fluid flow in the stellar core below the supernova shock wave (the outer surface) during a core collapse supernova explosion. Pink depicts a significant flow rotating in one direction directly below the shock wave. Gold depicts a deeper flow directly above the proto-neutron star surface, moving in the opposite direction. The supernova shock wave instability (SASI) leads to such counter rotating flows and is important in powering the supernova, and may be responsible for the spin of pulsars (rotating neutron stars). The inner flow (in green) spins up the proto-neutron star. These three-dimensional simulations were performed by John Blondin (NCSU) under the auspices of the Terascale Supernova Initiative, led by Tony Mezzacappa (ORNL). The visualization was performed The massive stellar explosions known as by Kwan-Liu Ma (University of California, Davis). core-collapse supernovae are not only some of the brightest and most powerful phenomena in the universe, but are also the Supernova science has advanced dramatically with the advent of pow- source of many of the elements which erful telescopes and other instruments for observing and measuring the make up our universe — from the iron in deaths of these stars, which are 10 our blood cells to the planet we live on to times more massive than our sun, or more. At the same time, increasingly the solar systems visible in the night sky. accurate applications for simulating supernovae have been developed to take advantage of the growing power compressed ball. This results in a Initiative (TSI) was launched, con- of bigger and faster supercomputers. shock wave which propagates from sisting of astrophysicists, nuclear Scientists now know that these the core to the outermost layers, physicists, computer scientists, math- massive stars are layered like onions. causing the star to explode. ematicians and networking engineers Around the iron core are layer after Despite these gains in understand- at national laboratories and universi- layer of lighter elements. As the stellar ing, however, a key question remains ties around the country. Their singular core becomes more massive, gravity unanswered: What triggers the forces focus was to understand how these causes the core to collapse to a cer- which lead a star to explode? Under massive stars die. tain point at which it rebounds like a SciDAC, the Terascale Supernova “When these stars die in stellar SCIENTIFIC DISCOVERY

explosions known as core-collapse to add them again later. They were tems at Oak Ridge gave Blondin the supernovae, they produce many of the interested in learning how the core computing horsepower he needed to elements in the universe. In fact, they flow behaves during the explosion as run his simulations in 3D. are arguably the single most impor- one goes from two spatial dimensions But scaling up from two to three tant source of elements,” said Tony to three. They first ran the code in dimensions is not just a matter of Mezzacappa, an astrophysicist at two dimensions, then in 3D. When running on a larger computer. It is Oak Ridge National Laboratory and they ran the simulations, they started also a matter of developing more principal investigator for the TSI. to see instability in the shock wave. detailed codes. “Learning what triggers supernovae But, Mezzacappa said, they initially In this case, neutrinos are central is tantamount to understanding how believed that the physics in their to the dynamics of core-collapse we came to be in the universe.” code should not have led to that supernovae. The explosion releases Ascertaining the explosion mech- behavior. Their first reaction was that 1053 ergs of radiation, almost all of it anism is one of the most important the code contained an error, but fur- in the form of neutrinos. The explo- questions in physics. It is a complex, ther study led them into a new area sion energy (the kinetic energy of the multi-physics problem involving fluid of supernova research. ejected material) is only 1051 ergs. flow, instability, and turbulence, stel- Despite decades of supernova This intense emission of radiation lar rotation, nuclear physics, particle theory focused on the formation and plays a key role in powering the physics, radiation transport and mag- propagation of the shock wave, appar- supernova, heating the interior mate- netic fields. ently no one considered whether the rial below the shock wave and adding “Much of modern physics comes shock wave was stable, according to energy to drive the shock wave out- to play a role in a core-collapse super- Mezzacappa. The TSI team’s research ward. But this radiation transport — nova,” Mezzacappa says. “The cosmos led them to conclude that the shock the production, movement and inter- is very much a physics laboratory.” wave is unstable; and if it is perturbed, action of the neutrinos — is one of Once the explosion mechanism is it grows in an unbounded, nonlinear the most difficult things to simulate accurately predicted, scientists will be fashion. As it spreads, it becomes computationally. It requires that the able to predict all the other associated more and more distorted. The TSI supercomputer solve a huge number byproducts and phenomena, such as group found that the instability of of algebraic equations. Applied math- the synthesis of elements. The key to the shock wave could help contribute ematicians on the TSI team developed solving this question depends on devel- to the explosion mechanism. For methods for solving the equations on oping more detailed computer mod- example, a distorted shock wave terascale supercomputers. els, such as those advanced under could explain why supernovae “This is another example of how SciDAC, and detailed observations of explode asymmetrically. SciDAC makes a world of difference core-collapse supernovae, which only Under further study, the TSI — we could not have done these simu- occur about twice every century in our researchers found that as they added lations without the help of the applied galaxy (although many such super- more physics back into their code, mathematicians,” Mezzacappa said. novae are observed each year from the instability did not go away, giving “SciDAC really enabled us to think outside of our galaxy). As models are them more confidence in their find- about this problem in all of its com- developed and improved, their accu- ings. Calling this new discovery the plexity in earnest for the first time.” racy can be tested by comparing the Stationary Accretion Shock Instability, But this also led to a new chal- results with the observed data. By or SASI, the team published their find- lenge. Suddenly, the team was faced adjusting any parameters of the model ings, which were then corroborated with managing massive amounts of (of course, the goal in developing by other supernova researchers. As a data with no infrastructure for analyz- sophisticated models is to minimize result, Mezzacappa said, the work ing or visualizing these terabytes of the number of free parameters), sci- has fundamentally changed scientists’ data. “We didn’t know what we had,” entists can generate results which are thinking about the explosion phenom- Mezzacappa said. closer and closer to the actual data. enon. And, he adds, it probably would The TSI team partnered with Because the simulations are com- not have happened without the mul- another SciDAC project led by Micah putationally intensive, Mezzacappa and tidisciplinary approach fostered by Beck of the University of Tennessee, John Blondin, a professor at North SciDAC. which developed a new data transfer Carolina State University, started by For starters, SciDAC provided solution known as logistical network- developing codes to look at the fluid the team with access to some of the ing. Logistical networking software flow at the core of a supernova. To world’s fastest supercomputers — a tools allow users to create local stor- focus on this one area, they removed resource otherwise unavailable to age “depots” or utilize shared storage the other physics components from researchers like Blondin. This access depots deployed worldwide to easily their code, knowing they would have to the Cray X1E and Cray XT3 sys- accomplish long-haul data transfers,

16 SCIENTIFIC DISCOVERY

temporary storage of large datasets possible in the two-dimensional sim- as you can spin a bicycle tire by (on the order of terabytes) and pre- ulations. Once a core-collapse super- swiping your hand across the tread. positioning of data for fast on-demand nova explodes, what remains is known The team computed their spin pre- delivery. as a neutron star. In some cases, this dictions and found that the results The group provided the hardware neutron star spins and emits radia- were within the observed range of and software needed to create a pri- tion as light and is called a pulsar. pulsars spins. vate network linking Oak Ridge and What is not known, however, is what “This was another breakthrough North Carolina State, allowing the causes a neutron star to get spun up. made possible by SciDAC,” data to be moved to the university. A previous theory held that neu- Mezzacappa said. “It gives us new There, Blondin used a visualization tron star spin is generated when the possibilities for explaining the spins cluster to analyze and visualize the stellar core spins up as it collapses of pulsars.” data, allowing the team to pursue the during the supernova, much like an The key to their results in both 3D science. The TSI network now ice skater who increases his rotational cases, Mezzacappa noted, was scaling spans the U.S., linking the two original speed by pulling his arms close to his their application to simulate complex sites with facilities like the National body. But this simple model cannot phenomena in 3D, rather than two Energy Research Scientific Computing explain all the observed characteris- dimensions. “In other scientific arti- Center in California to the State tics of pulsars and at the same time cles, supernova researchers have University of New York at Stony explain the predicted characteristics written that they don’t see much dif- Brook. As the TSI project demon- of stars at the onset of collapse. ference between simulations in 2D strated the benefits of logistical net- In TSI’s 3D models, the SASI and 3D, but now we know they are working, researchers in the fusion produces two counter-rotating flows very different, and have shown how and combustion communities also between the proto-neutron star and this difference is related to the super- built their own logistical networks. the propagating shock wave. As the nova mechanism and byproducts,” Once the data from the 3D simu- innermost flow settles onto the proto- Mezzacappa said. “Mother Nature lations could be analyzed, the team neutron star, it imparts angular clearly works in a 3D universe.” made another discovery that was not momentum and causes it to spin, just

17 CCSM: Advanced Simulation Models for Studying Global Climate Change

One of the most widely discussed scientific questions of the past 20 years is the issue of global climate change. But it’s not just a matter of science. The question of global climate change also involves economic, environmental, social and political implications. In short, it’s a complex question with no easy answers. But detailed climate modeling tools are helping researchers gain a better understanding of how

human activities are influenc- FIGURE 1. Surface temperature (in ºC) averaged over December of year 9 of the fully coupled chemistry simulation. Vectors represent the surface wind. ing our climate. To try to ensure that decision- makers around the world have access 33 to the most accurate information available, the Intergovernmental

30 Panel on Climate Change (IPCC) was established in 1988 under the auspices of the World Meterological 27 Organization and the United Nations Environment Program. The role of 24 the IPCC is to assess on a comprehensive, objective, open and transpar-

21 ent basis the scientific, technical and socio-economic information relevant to understanding the scientific basis 18 of risk of human-induced climate change, its potential impacts and 15 options for adaptation and mitigation. In accordance with its mandate,

12 the major activity of the IPCC is to prepare at regular intervals comprehensive and up-to-date assessments 9 of relevant for the understanding of human induced climate change, poten- 6 tial impacts of climate change and options for mitigation and adaptation.

3 The First Assessment Report was completed in 1990, the Second Assessment Report in 1995 and the 0 Third Assessment Report in 2001. The Fourth Assessment Report is –3 scheduled to be completed in 2007. SciDAC-sponsored research has

–6 enabled the United States climate modeling community to make significant contributions to this report. In –9 fact, the United States is the largest contributor of climate modeling data –12 to the report, as compared to the 2001 report in which no U.S.-gener-

–15 ated data was included. Two large multi-laboratory SciDAC projects are directly relevant –18 to the activities of the IPCC. The first, entitled “Collaborative Design and –21 Development of the Community Climate System Model for Terascale

–24 Computers,” has made important software contributions to the recently released third version of the Com- –27 munity Climate System Model (CCSM3.0), developed by the –30 National Center for Atmospheric Research, DOE laboratories and the academic community. The second project, entitled “Earth System Grid SCIENTIFIC DISCOVERY

II: Turning Climate Datasets into ran extensive simulations using the Community Resources,” aims to facili- CCSM and another climate modeling tate the distribution of the copious application, PCM, to project climate amounts of data produced by coupled change through the end of the 21st climate model integrations to the gen- century. eral scientific community. The results, reported in the March A key chapter in the Fourth 18, 2005 issue of Science magazine, Assessment Report will look at Global indicate that “even if the concentrations of greenhouse gases in the atmosphere had been stabilized in the year 2000, we are already committed Annual Precipitation (mm/day) March Snow Water (m) to further global warming of about Observed Observed 49N another half degree and an additional 48N 320 percent sea level rise caused by 47N 46N 2 thermal expansion by the end of the 5 45N 1 2 21st century. … At any given point in 44N 0.5 43N 1 0.2 time, even if concentrations are stabi- 42N 0.5 0.1 lized, there is a commitment to future 41N 0.2 40N 0.05 0.1 climate changes that will be greater 39N 0.02 38N than those we have already observed.” 37N As one of many research organi- 36N zations from around the world providing input for the IPCC report, the Simulated 1980-2000 Simulated 1980-2000 49N DOE Office of Science is committed 48N to providing data that is as scientifi- 47N 46N 2 cally accurate as possible. This com- 5 45N 1 mitment to scientific validity is a key 44N 2 0.5 43N 1 factor behind the SciDAC program to 0.2 42N 0.5 0.1 refine and improve climate models. 41N 0.2 40N 0.05 In order to bring the best science 0.1 39N 0.02 38N to bear on future climate prediction, 37N modelers must first try to accurately 36N simulate the past. The record of historical climate change is well docu- Simulated 1980-2100 Simulated 1980-2100 mented by weather observations and 49N 48N measurements starting around 1890. 47N So, after what is known about the 46N 2 5 45N 1 physics and mathematics of atmos- 2 44N 0.5 43N 1 pheric and ocean flows is incorporat- 0.2 42N 0.5 0.1 ed in a simulation model running on 41N 0.2 40N 0.05 one of DOE’s supercomputers, the 0.1 39N 0.02 model is tested by “predicting” the cli- 38N 37N mate from 1890 to the present. 36N 124W 120W 116W 112W 108W 124W 120W 116W 112W 108W The input to such a model is not 122W 118W 114W 110W 106W 122W 118W 114W 110W 106W the answer, but rather a set of data describing known climate conditions — what the atmospheric concentrations of greenhouse gases were, what solar fluctuations occurred, and what

FIGURE 2. Annual precipitation and March Climate Projections, which will exam- volcanic eruptions took place. While snow water from an IPCC simulation using ine climate change to the year 2100 past climate models were not able to the subgrid orography scheme in Ghan and and beyond. Gerald Meehl, one of replicate the historical record with its Shippert (2005). two lead authors of the chapter and a natural and induced variability, the scientist at the National Center for present generation of coupled ocean, Atmospheric Research in Colorado, atmosphere, sea ice, and land compo-

20 SCIENTIFIC DISCOVERY

nents do a remarkably good job of (which has not been ensemble-inte- over 200 papers are in press referenc- predicting the past century’s climate. grated). ing this data. (You can see a list at This accuracy gives confidence that This increase in production is sci- http://www-pcmdi.llnl.gov/ipcc/ when the same model is used to gen- entifically important in many ways. diagnostic_subprojects.php.) erate predictions into the future based The need for large numbers of indi- When researchers encounter dis- on various emission scenarios, the vidual simulations is especially agreement among models, scientific results will have a strong scientific important when attempting to char- debates and discussions ensue about basis and are more than scientific acterize the uncertainty of climate the physical mechanisms governing opinions of what might happen. change. For instance, in looking at the dynamics of climate. This debate The Community Climate System recent climate change (over the last is typically vigorous and points in Model, CCSM3, is centered at century), running more statistically directions that models can be NCAR, arguably a world leader in independent simulations of the 20th improved. To take advantage of such the field of coupled climate models. century allows researchers to produce discussions and to improve the valid- This model continues to be devel- a much better defined pattern of cli- ity of climate models, DOE sponsors oped jointly by the National Science mate changes. To model future cli- the Program for Model Intercompar- Foundation and DOE to capture the mate change, scientists will run multi- ison and Diagnosis (PCMDI), which best scientific understanding of cli- ple simulations with a given level of places the scientific basis of climate mate. In addition to being used for greenhouse gas emissions to quantify modeling front and center in its climate change studies, the CCSM is the range of possible outcomes. And work. Such rigorous peer review and being used to study climate-related when researchers want to study pos- discussion is critical to ensuring that questions ranging from ancient cli- sible outcomes for a range of emis- government officials have access to mates to the physics of clouds. sions, they need to run even more the most scientifically valid informa- However, the DOE national labora- simulations. tion when making decisions regard- tories and the National Center for Running simulations for extended ing climate change. Atmospheric Research periodically times are also important to help This SciDAC project is a collaboration use the model for assessment purposes researchers detect and take into between six DOE National Laboratories such as the IPCC project. account conditions which result from (ANL, LANL, LBNL, LLNL, ORNL, The SciDAC CCSM Consortium flaws in the climate model. Such PNNL) and NCAR. The lead investigators developing the model has also made flaws cause “drift,” or an unsubstanti- representing these institutions are R. Jacob, P. Jones, C. Ding, P. Cameron-Smith, J. sure that it runs effectively on the most ated change in temperature. While Drake, S. Ghan, W. Collins, W. Washington powerful supercomputers at DOE’s drift-free control runs covering sever- and P. Gent. leading computing centers — the al hundred years are useful for study- Leadership Class Facility (LCF) in ing climate change over a few Tennessee and the National Energy decades, examining climate change Research Scientific Computing of the entire 20th century requires a Center (NERSC) in California. DOE control run of at least 1,000 years to contributed significant allocations of avoid misinterpretation of the result- supercomputer time to the interna- ing climate model. In summary, large tional efforts (along with NCAR and ensembles and long control runs are the Japanese Earth Simulator Center) necessary to better understand the toward the completion of the IPCC clear patterns of climate change. runs with CCSM3. Through SciDAC, DOE also The fruit of these labors is sub- sponsors the technology being used stantial. Enabled by these sizable to make available the results to scien- allocations, the CCSM3 is the largest tists worldwide. The Earth System single contributor to the IPCC Grid (ESG) project collects all the Assessment Report 4 database. Not data from all the runs of models in only is the model represented in more the U.S., as well as from other coun- transient scenarios than any other tries, and makes the data accessible model, more statistically independent to scientists throughout the world. realizations of each of these scenarios This allows scientists in the interna- have been integrated than with any tional climate research community to other model. Also, the resolution of analyze more extensive datasets, the atmosphere component is compare results and document differ- exceeded by only one other model ences in model predictions. In fact,

21 Plasma Physics Terascale Simulations Predict Favorable Confinement Trend for Reactor-Scale Plasmas

Most people do not think about turbulence very often, except when they are flying and the captain turns on the “Fasten Seat Belts” sign. The kind of turbulence that may cause problems for airplane passengers involves

swirls and eddies that are a great deal larger In fusion research, all of the conditions necessary to keep a plasma than the aircraft. But in fusion plasmas, much dense and hot long enough to under- smaller-scale turbulence, called microturbu- go fusion are referred to as confinement. The retention of heat, called lence, can cause serious problems—specifical- energy confinement, can be threat- ened by microturbulence, which can ly, instabilities and heat loss that could stop make particles drift across, rather the fusion reaction. than along with, the plasma flow. At the core of a fusion reactor such as a tokamak, the temperatures and densities are higher than at the outside edges. As with weather, when there FIGURE 1. Cutaway illustration of the ITER tokamak.

be built at Cadarache in southern France, is one of the highest strategic priorities of the DOE Office of Science. Underlining America’s commitment to ITER, U.S. Energy Secretary Samuel Bodman stated on June 28, 2005, “Plentiful, reliable energy is critical to worldwide economic development. Fusion technologies have the potential to transform how energy is produced and provide significant amounts of safe, environmentally friendly power in the future. The ITER project will make this vision a reality.” ITER is expected to produce 500 million thermal watts of fusion power—10 times more power than is needed to heat the plasma—when it reaches full operation around the year 2016. As the world’s first production-scale fusion reactor (Figure 1), ITER will help answer questions about the most efficient ways to configure and operate future commercial reactors. The growth of the microinstabili- ties that lead to turbulent transport has been extensively studied over the years, not only because understanding this process is an important practical problem, but also because it is a true scientific grand challenge which is particularly well suited to be addressed by modern terascale computational resources. And the latest are two regions with different tem- the size of the plasma increases, the news, confirmed by multiple simula- peratures and densities, the area relative level of heat loss from turbu- tions using different codes, is good: between is subject to turbulence. In a lent transport also increases. Because in reactors the size of ITER, heat tokamak, turbulence can allow the size (and therefore cost) of a losses caused by plasma turbulence charged particles in the plasma to fusion experiment is determined no longer follow the empirical trend move toward the outer edges of the largely by the balance between fusion of increasing with the size of the reactor rather than fusing with other self-heating and turbulent transport plasma. Instead, the rate of heat loss particles in the core. If enough parti- losses, understanding this process is levels off and stabilizes. cles drift away, the plasma loses tem- of utmost importance for the design Progress in understanding plasma perature and the fusion reaction can- and operation of fusion devices such ion dynamics has been impressive. not be sustained. as the multi-billion-dollar ITER proj- For example, studies show that elec- One troublesome result of toka- ect. ITER (Latin for “the way”), a trostatic ion temperature gradient mak experiments to date is that as multinational tokamak experiment to (ITG) driven turbulence can be sup- SCIENTIFIC DISCOVERY

pressed by self- For the reactor-scale plasmas of the With flow generated zonal future, these simulations suggest that flows within the the relative level of heat loss driven plasma. The sup- by electrostatic ITG turbulence does pression of turbu- not increase with plasma size (Figure lence is caused 3). This transition from “Bohm” (lin- by a shearing ear) scaling to “gyro-Bohm” (quadrat- action that ic) scaling is a positive trend, because destroys the fin- simple empirical extrapolation of the Without flow ger-like density smaller system findings would pro- contours which duce much more pessimistic predic- promote thermal tions for energy confinement. Since transport. This neither experiments nor theory and dynamic process simulations have previously been able is depicted by the to explore such trends in an ITER- sequences shown sized plasma, these results represent a in Figure 2, significant scientific discovery enabled obtained using by the SciDAC program. the Gyrokinetic Exploration of the underlying FIGURE 2. Turbulence reduction via sheared Toroidal Code (GTC) developed by causes for the transition in the rate of plasma flow compared to case with flow suppressed. the SciDAC-funded Gyrokinetic heat loss that simulations show Particle Simulation Center (GPSC). around the 400 gyroradii range has The lower panels show the nonlinear inspired the development of new non- evolution of the turbulence in the linear theoretical models based on the absence of flow, while the upper pan- spreading of turbulence. Although els illustrate the turbulence decorrela- this predicted trend is a very favorable tion caused by the self-generated one, the fidelity of the analysis needs “E×B” flow, which arises from crossed to be further examined by investigat- electric and magnetic fields. ing additional physics effects, such as This kinetic electromagnetic dynamics, simulation which might alter the present predic- 1.5 is a good tions. Formula example of The excellent scaling of the GTC 1.0 the effec- code provides strong encouragement

/ i tive use of that future simulations will be able to

χ Simulation 0.5 powerful capture the additional complexity and supercom- lead to greater scientific discoveries. puters (in GTC is currently involved in bench- 00 200 400 600 800 1000 ρ a/ i this case, mark tests on a variety of supercom- the 10 ter- puting platforms—including the Earth aflop/s Simulator in Japan, the Cray X1E and FIGURE 3. Full torus particle-in-cell gyrokinetic simulations (GTC) of turbulent trans- Seaborg IBM SP at NERSC). Typical XT line at Oak Ridge National port scaling. (Left) The granular structures global particle-in-cell simulations of Laboratory, the IBM Power SP line, represent the scales of the turbulence in a this type have used one billion parti- and the IBM Blue Gene line—and typical plasma which need to be included in realistic plasma simulations. (Right) The cles with 125 million grid points over exhibits scaling properties which look horizontal axis expresses the plasma size, 7000 time steps to produce significant to be readily extensible to the peta- and the point at 1000 represents ITER’s size. The vertical axis represents the ther- physics results. Simulations of this scale regime. GTC’s performance and mal diffusion, or heat loss. size would not be feasible on smaller scaling on a variety of leading plat- computers. forms is illustrated in Figure 4. Large-scale simulations have also The GS2 and GYRO codes devel- explored key consequences of scaling oped by the Plasma Microturbulence up from present-day experimental Project (the SciDAC predecessor to fusion devices (around 3 meters radius the current GPSC) have also con- for the largest existing machines) to tributed productively to the interpre- ITER-sized reactors (about 6 meters). tation of turbulence-driven transport

24 SCIENTIFIC DISCOVERY

trends observed in experiments. These Compute Power of the Gyrokinetic Toroidal Code two SciDAC projects have involved Number of particles (in million) moved 1 step in 1 second researchers from Lawrence Livermore 10000 National Laboratory; Princeton Plasma Physics Laboratory (PPPL); Columbia University; the University Latest vector optimizations Not of California at Los Angeles, Irvine, tested on Earth Simulator 1000 and Davis; the University of Colorado; the University of Maryland; the University of Tennessee at Knoxville; Oak Ridge National Laboratory; and General Atomics. 100 CRAY XIE Bill Tang, Chief Scientist at PPPL, NEC SX-8 Earth Simulator (05) is encouraged by the progress in com- CRAY XI putational fusion research. “SciDAC CRAY XT3 (Jaguar) has contributed strongly to the accel- Jacquard 10 Thunder

Compute Power (millions of particles) erated development of computational Blue Gene/L tools and techniques needed to devel- Seaborg (MPI) op predictive models for the analysis Seaborg (MPI + OMP) and design of magnetically confined 64 128 256 512 1024 2048 4096 81926 16384 plasmas,” Tang commented. “Unravel- Number of processors ing the complex behavior of strongly S. Ethier, PPPL, Nov. 2005 nonlinear plasma systems under realistic conditions is a key component of Spring 2006, p. 8): “At the speeds that FIGURE 4. Scaling study of the GTC code the next frontier of computational we are talking about [50 teraflop/s], on multiple high performance computing plasma physics in general and fusion the electrons in a fusion device can be platforms. research in particular. Accelerated considered as real point particles and progress in the development of the do not have to be treated in mean needed codes with higher physics field approximations. So for the first fidelity has been greatly aided by the time at 50 TF, one will be able to do interdisciplinary alliances championed simulations of high-density, high-tem- by the SciDAC Program, together with perature plasmas that were never pos- necessary access to the tremendous sible before. This will have a signifi- increase in compute cycles enabled by cant impact on the treatment of these the rapid advances in supercomputer highly nonlinear systems. These sys- technology.” tems are not subject to analytic exam- Ray Orbach, Director of the DOE ination and there may be instabilities Office of Science, expressed his hope that no one has thought about. This for high-end computing’s future con- has already been found to be of pro- tributions to fusion energy in an inter- found importance for ITER [and] for view in SciDAC Review (Number 1, fusion science in general.”

25 HFS Pellet Injection: Fueling the Future R. Samtaney Imagine that there is a large Simulations Examine the Behavior of Frozen Fuel chamber in hell that’s shaped like a doughnut, and that you’d like to Pellets in Fusion Reactors shoot a series of hailstones into that chamber so that as they melt, the water vapor penetrates as deeply as possible and disperses as evenly as possible throughout the chamber. Setting aside for a moment the question of why you would want to do that, consider the multitude of how questions: Should you shoot in the hailstones from outside the ring of the doughnut or from inside the hole? What size hailstones should you use? At what speed and angle should they enter the chamber? This strange scenario is actually an analogy for one of the questions facing fusion energy researchers: how to refuel a tokamak. A tokamak is a machine that produces a toroidal (doughnut-shaped) magnetic field. In What happens when you shoot one of the that field, two isotopes of hydrogen — deuterium and tritium — are heated coldest materials into the hottest environ- to about 100 million degrees Celsius (more than six times hotter than the ment on earth? The answer may help solve interior of the sun), stripping the the world’s energy crisis. electrons from the nuclei. The magnetic field makes the electrically charged particles follow spiral paths around the magnetic field lines, so that they spin around the torus in a T = 2000 T = 32000 T = 82000

T = 2000 T = 32000 T = 82000

FIGURE 1. These simulations show the results of pellet injection into a tokamak fusion reactor from the low-field-side (LFS) and high-field-side (HFS). The top row shows a time sequence of the density in LFS injection while the bottom panel shows density evolution in HFS injection. The dominant motion of the ablated pellet mass is along field lines accompanied by transport of material across flux surfaces towards the low field side. This observation is qualitatively consistent with experimental observations leading to the conclusion that HFS pellet injection is a more efficient refueling technique than LFS injection. The MHD instabilities which cause the pellet material to move towards the low field side are currently under investigation.

fairly uniform flow and interact with tokamak experiment to be built at In the past, progress in develop- each other, not with the walls of the Cadarache in southern France. ITER, ing an efficient refueling strategy for tokamak. When the hydrogen ions one of the highest strategic priorities ITER has required lengthy and (nuclei) collide at high speeds, they of the DOE Office of Science, is expensive experiments. But thanks to fuse, releasing energy. If the fusion expected to produce 500 million a three-year, SciDAC-funded collabo- reaction can be sustained long thermal watts of fusion power — 10 ration between the Computational enough that the amount of energy times more power than is needed to Plasma Physics Theory Group at released exceeds the amount needed heat the plasma — when it reaches Princeton Plasma Physics Laboratory to heat the plasma, researchers will full operation around the year 2016. and the Advanced Numerical have reached their goal: a viable As the world’s first production-scale Algorithms Group at Lawrence energy source from abundant fuel fusion reactor, ITER will help answer Berkeley National Laboratory, com- that produces no greenhouse gases questions about the most efficient puter codes have now reproduced and no long-lived radioactive ways to configure and operate future some key experimental findings, byproducts. commercial reactors. resulting in significant progress High-speed injection of frozen However, designing a pellet injec- toward the scientific goal of using hydrogen pellets is an experimentally tion system that can effectively deliv- simulations to predict the results of proven method of refueling a toka- er fuel to the interior of ITER repre- pellet injection in tokamaks. mak. These pellets are about the size sents a special challenge because of “To understand refueling by pel- of small hailstones (3–6 mm) and its unprecedented large size and high let injection, we need to understand have a temperature of about 10 temperatures. Experiments have two phases of the physical process,” degrees Celsius above absolute zero. shown that a pellet’s penetration dis- said Ravi Samtaney, the Princeton The goal is to have these pellets pen- tance into the plasma depends researcher who is leading the code etrate as deeply as possible into the strongly on how the injector is ori- development effort. “The first phase plasma so that the fuel disperses ented in relation to the torus. For is the transition from a frozen pellet evenly. example, an “inside launch” (from to gaseous hydrogen, and the second Pellet injection will be the pri- inside the torus ring) results in better phase is the distribution of that gas mary fueling method used in ITER fuel distribution than an “outside in the existing plasma.” (Latin for “the way”), a multinational launch” (from outside the ring). The first phase is fairly well SCIENTIFIC DISCOVERY

understood from experiments and quickly heats up to form a local nomena that Samtaney and his col- theoretical studies. In this phase, region of high pressure, higher than leagues want to quantify and exam- called ablation, the outer layer of the can be stably confined by the local ine in detail. frozen hydrogen pellet is quickly magnetic field. A form of “local insta- Figure 3 shows the fuel distribu- heated, transforming it from a solid bility” (like a mini-tornado) then tion phase as simulated by Samtaney into an expanding cloud of dense develops, causing the high-density and his colleagues in the first detailed hydrogen gas surrounding the pellet. region to rapidly move across, rather 3D calculations of pellet injection. This gas quickly heats up, is ionized, than along, the field lines and flux The inside launch (top row) distrib- and merges into the plasma. As abla- surfaces — a motion referred to as utes the fuel in the central region of tion continues, the pellet shrinks until “anomalous” because it deviates from the plasma, as desired, while the out- all of it has been gasified and ionized. the large-scale motion of the plasma. side launch (bottom row) disperses The second phase — the distribu- Fortunately, researchers have dis- the fuel near the plasma boundary, as tion of the hydrogen gas in the plasma covered that they can use this insta- shown in experiments. — is less well understood. Ideally, the bility to their advantage by injecting Simulating pellet injection in 3D injected fuel would simply follow the the pellet from inside the torus ring, is difficult because the physical magnetic field lines and the “flux sur- because from this starting point, the processes span several decades of faces” that they define, maintaining a anomalous motion brings the fuel time and space scales. The large dis- stable and uniform plasma pressure. pellet closer to the center of the plasma, parity between pellet size and tokamak But experiments have shown that the where it does the most good. This size, the large density differences high-density region around the pellet anomalous motion is one of the phe- between the pellet ablation cloud and

FIGURE 2. This illustration shows how computational problems are solved by dividing them into smaller pieces by covering them with a mesh. In this case, the fuel pellet for a fusion reactor is buried within the finest mesh which occupies less than 0.015 percent of the volume of the coarsest mesh. Using a technique known as adaptive mesh refinement, scientists can focus the power of a supercomputer on the most interesting part of a problem, such as the fuel pellet in a hot plasma.

28 SCIENTIFIC DISCOVERY

Field data coloring legend Field data coloring legend Field data coloring legend 6.00 6.00 6.00

4.67 4.67 4.67

3.35 3.35 3.35

2.02 2.02 2.02

0.700 0.700 0.700 Field data coloring legend Field data coloring legend Field data coloring legend 6.00 6.00 6.00

4.67 4.67 4.67

3.35 3.35 3.35

2.02 2.02 2.02

0.700 0.700 0.700

FIGURE 3. Time sequence of 2D slices from a 3D simulation of the injection of a fuel pellet into a the ambient plasma, and the long- methodology, Samtaney’s work on tokamak plasma. Injection from outside the torus distance effects of electron heat pellet injection is only beginning. (bottom row, injection from right) results in the transport all pose severe numerical “The results presented in this paper pellet stalling and fuel being dispersed near the plasma boundary. Injection from inside the torus challenges. To overcome these diffi- did not include all the detailed physi- (top row, injection from left) achieves fuel distribu- culties, Samtaney and his collabora- cal processes which we’re starting to tion in the hot central region as desired. tors used an algorithmic method incorporate, along with more realistic called adaptive mesh refinement physical parameters,” he said. “For (AMR), which incorporates a range example, we plan to develop models of scales that change dynamically as that incorporate the perpendicular the calculation progresses. AMR transport of the ablated mass. We eventually become part of a compre- allowed this simulation to run more also want to investigate other launch hensive predictive capability for than a hundred times faster than a locations. And, of course, we’ll have ITER, which its supporters hope will uniform-mesh simulation. to validate all those results against bring fusion energy within reach as a While these first calculations rep- existing experiments.” commercial source of electrical resent an important advance in This pellet injection model will power.

29 Burning Questions New 3D Simulations Are Closing in on the Holy Grail of Combustion Science: Turbulence–Chemistry Interactions

Controlling fire to provide heat and light was one of humankind’s first great achievements, and the basic chemistry of combustion — what goes in and what comes out — was established long ago. But a complete quantitative understanding of what happens during the combustion process has remained as elusive as the ever-changing shape of a flame.

SCIENTIFIC DISCOVERY

Even a simple fuel flame equations some 30 years later

like methane (CH4), using Cray-1 supercomputers. Those the principal compo- calculations, which are routine on nent of natural gas, personal computers today, enabled a burns in a complex renaissance in combustion science by sequence of steps. allowing chemists to observe the Oxygen atoms gradu- interrelationships among the many ally replace other hypothesized reaction processes in atoms in the hydrocar- the flame. bon molecules, ulti- Simulating three-dimensional tur- mately leaving carbon bulent flames took 20 more years of dioxide and water. advances in applied mathematics, Methane oxidation computer science, and computer involves about 20 hardware, particularly massively par- chemical species for allel systems. But the effort has been releasing energy, as worth it. New 3D simulations are well as many minor beginning to provide the kind of species that can detailed information about the struc- become pollutants. ture and dynamics of turbulent flames Turbulence can distort that will be needed to design new the distribution of low-emission, fuel-efficient combus- species and the redis- tion systems. tribution of thermal The first 3D simulation of a labo- energy which are ratory-scale turbulent flame from first required to keep the principles — the result of a SciDAC- flame burning. These funded collaboration between compu- turbulence–chemistry tational and experimental scientists at interactions can cause Berkeley Lab — was featured on the the flame to burn cover of the July 19, 2005 Proceedings FIGURE 1. The calculated surface of a turbulent premixed laboratory methane flame. faster or slower and to of the National Academy of Sciences create more or less (Figure 1). The article, written by pollution. John Bell, Marc Day, Ian Shepherd, The holy grail of Matthew Johnson, Robert Cheng, combustion science Joseph Grcar, Vincent Beckner, and has been to observe Michael Lijewski, describes the simu- these turbulence–chemistry effects. lation of “a laboratory-scale turbulent Over the past few decades amazing rod-stabilized premixed methane V- progress has been made in observa- flame.” This simulation was unprece- tional techniques, including the use of dented in several aspects — the num- lasers to excite molecules of a given ber of chemical species included, the species and produce a picture of the number of chemical reactions mod- chemical distribution. But laser imag- eled, and the overall size of the flame. ing is limited in the species and con- This simulation employed a differ- centrations that can be reliably ent mathematical approach than has observed, and it is difficult to obtain typically been used for combustion. simultaneous images to correlate dif- Most combustion simulations ferent chemical species. designed for basic research use com- Because observing the details of pressible flow equations that include combustion is so difficult, progress in sound waves, and are calculated with combustion science has largely coin- small time steps on very fine, uniform cided with advances in scientific com- spatial grids — all of which makes puting. For example, while basic con- them very computationally expensive. cepts for solving one-dimensional flat Because of limited computer time, flames originated in the 1950s, it only such simulations often have been became possible to solve the 1D restricted to only two dimensions, to

32 SCIENTIFIC DISCOVERY

scales less than a centimeter, or to just enough to be compared directly with a few carbon species and reactions. experimental diagnostics. In contrast, the Center for Comp- The simulation captured with utational Sciences and Engineering remarkable fidelity some major fea- (CCSE), under Bell’s leadership, has tures of the experimental data, such as developed an algorithmic approach flame-generated outward deflection in that combines low Mach-number the unburned gases, inward flow con- equations, which remove sound vergence, and a centerline flow accel- waves from the computation, with eration in the burned gases (Figure 2). adaptive mesh refinement, which The simulation results were found to bridges the wide range of spatial match the experimental results within scales relevant to a laboratory experi- a few percent. This agreement direct- ment. This combined methodology ly validated both the computational strips away relatively unimportant method and the chemical model of aspects of the simulation and focuses hydrocarbon reaction and transport computing resources on the most kinetics in a turbulent flame. important processes, thus slashing the The results demonstrate that it is computational cost of combustion possible to simulate a laboratory-scale simulations by a factor of 10,000. flame in three dimensions without Using this approach, the CCSE having to sacrifice a realistic represen- team has modeled turbulence and tur- tation of chemical and transport bulence–chemistry interactions for a processes. This advance has the three-dimensional flame about 12 cm potential to greatly increase our (4.7 in.) high, including 20 chemical understanding of how fuels behave in species and 84 fundamental chemical the complicated environments inside reactions. The simulation was realistic turbulent flames.

1 cm

FIGURE 2. Left: A typical centerline slice of the methane concentration obtained from the simulation. Right: Experi-mentally, the instantaneous flame location is determined by using the large differences in Mie scattering intensities from the reactants and products to clearly outline the flame. The wrinkling of the flame in the computation and the experiment is of similar size and structure.

33 Pushing the Frontiers of Chemistry Dirac’s Vision Realized

In most first-year chemistry classes at universities, students begin by learning how atoms of carbon and other elements form bonds with other atoms to form molecules, which in turn combine to form ourselves, our planet and our universe.

Paul Dirac, one of the founders of Understanding the properties calculations. Adding each extra elec- quantum mechanics. and behavior of molecules, or better tron then entails about a million yet, being able to predict and con- times more work, a condition known trol the behavior, is the driving force as exponential scaling, so it is no behind modern chemistry. The wonder that the largest exact calcu- development of the theory of quan- lations still involve no more than studying molecular systems in a field tum mechanics in the 1920’s made it about 15 electrons. Even with antici- such as nanotechnology, they are inter- possible that all the properties of pated supercomputing advances ested in systems with at least a few molecules could be predicted. One over the next decade, only one or hundred electrons. its founders, Paul Dirac, wrote in maybe two more electrons could be Realizing that a brute-force 1929 that “the difficulty is only that treated this way. approach would not succeed, the the exact application of these laws Quantum chemistry, however, Advanced Methods for Electronic leads to equations much too compli- provides scientists with the models Structure: Local Coupled Cluster cated to be soluble.” to approximate these values without Theory project was designed to devel- Almost 80 years later this is still doing all the calculations. But there is op new methods which strike novel true, even using the most powerful a tradeoff, since researchers need to compromises between accuracy and supercomputers. For example, a sci- balance accuracy against feasibility. feasibility. The goal was to extend cou- entist could solve for the behavior of A very simple model can be applied pled cluster electronic structure theory a molecule with one electron moving to a giant molecular system, but the to larger molecules. This was achieved in one dimension on a grid of roughly results will be uselessly inaccurate. by developing both new theory and 100 points spread along that line. As noted, a very accurate model, new high-performance algorithms. The Since electrons in free space move in due to its complexity, may only be resulting method scales much better (at three dimensions, not one, the scien- feasible for systems of up to 15 elec- the 3rd power) than the old codes (to tist would need 1003 (one million) trons. However, when scientists are the 6th power), and can therefore be LUMO 0.17 e–

singlet diradical with the characteristics of broken bonds. The species was found computationally to have around 17 percent broken bond character, rather than the much higher fraction believed by experimentalists. This explained the stability of the material, and the origin of the 17 percent result could be understood from the role of neighboring groups, which shows the ability to tune the reactivity of the molecule by chemical design.

HUMO 0.83 e– Project leader Martin Head- Gordon notes that the algorithms are currently not suitable for all problems, but that the team is looking to develop them into general purpose codes, and is working on improved successors. He predicts that the project’s work will make its way into the mainstream within five years, bringing new computational chemistry capabilities to many of the 40,000 chemists who use such applications. Already, project members are getting inquiries from experimental chemists asking if they can use their codes to study specific molecules. This has led to collaborations investigating the diradical character FIGURE 1. Graphical representation of the highest occupied molecular orbital (HOMO) and the lowest unoccupied level (LUMO) for the boron-phosphorus material. Instead of being fully empty, the of what may be the world’s longest LUMO contains 0.17 electrons, corresponding to 17 percent broken bond character. carbon-carbon bonds as well as work in progress on triple bonds between heavier elements such as tin and ger- applied to much larger molecules In particular, scientists have long manium. Chemists are interested in using existing supercomputers or been interested in studying molecules whether these heavy elements can even commodity computers. in which the chemical bonds are form triple bonds the way carbon One advantage the new method breaking, as opposed to the more does. Initial results indicate they do, has over older methods is that it can common states where the molecule but the behavior of the resulting mol- be applied to systems where the elec- is either stable, with the bonds intact, ecules is very different. trons are highly correlated. In other or unstable, in which the bonds are Computation plays a key part in words, the motions of the two elec- broken. In bond-breaking, the elec- interpreting what experimentalists trons are closely tied to each other, trons are strongly correlated. find and also in predicting what their much like two ballroom dancers or Scientists are interested in studying experiments will produce, according figure skaters who go through their molecules as they are on their way to to Head-Gordon, who has a joint motions almost like a single unit. breaking into fragments because this appointment as a chemistry professor Such a condition is mathematically may give insights that allow the at the University of California, much harder to describe than sys- design of useful materials in areas Berkeley, and DOE’s Lawrence tems with weaker correlation, which ranging from molecular electronics Berkeley National Laboratory. “High can be imagined as two friends danc- and spintronics to self-assembly of performance computing, together ing separately at a rock concert. functional nanostructures. with new theory and algorithms, Being able to treat highly correlated The project team studied a allows us to push the frontiers of electrons reliably allows scientists to boron-phosphorus based material chemistry and go where previous study interesting molecules which that caused excitement when it was generations of chemists have not would otherwise be difficult. reported as the first indefinitely stable been able to go,” Head-Gordon said. Given the importance of particle accelerators to the United States’ scientific, industrial and economic competi- tiveness, bringing the most advanced high performance computing tools to bear on accelerator design, optimization and operation is in the national interest. Within the DOE Office of Science, particle accelerators have enabled remarkable scientific discoveries and important technological advances that span several programs. In the High Energy Physics and Nuclear Physics programs, experiments associated with high-energy accelerators have led to important discoveries about elementary particles and the fundamental forces of nature, quark dynamics, and nuclear structure. In the Basic Energy Sciences and the Biological and Environmental Research programs, experiments with synchrotron light Accelerators: sources and spallation neutron sources have been crucial to advances in the materials, chemical and biological sci- Extraordinary Tools for Extraordinary Science ences. In the Fusion Energy Sciences program, great strides have been made in developing heavy-ion particle accelerators as drivers for high energy density physics research and ultimately inertial fusion energy. The importance of accelerators to the Office of Science mission is evident from an examination of the DOE “Facilities for the Future of Science: A Twenty-Year Outlook.” Of the 28 facilities listed, 14 involve new or Particle accelerators are some of the upgraded accelerator facilities. The SciDAC Accelerator Science and most powerful experimental tools avail- Technology (AST) modeling project was able for basic science, providing a national research and development effort aimed at establishing a compre- researchers with insight into the basic hensive terascale simulation environment needed to solve the most challenging building blocks of matter and enabling problems in 21st century accelerator sci- some of the most remarkable discoveries ence and technology. The AST project had three focus areas: computational of the 20th century. They have led to beam dynamics, computational electromagnetics, and modeling advanced accel- substantial advances in applied science erator concepts. The tools developed and technology, such as nuclear medi- under this program are now being used by accelerator physicists and engineers cine, and are being studied for potential across the country to solve the most challenging problems in accelerator application to problems related to energy design, analysis, and optimization. The and the environment. following examples of scientific and engineering accomplishments achieved by the AST project illustrate how the package for state-of-the-art simulation soon-to-be-operating Large Hadron project software is already improving of linear and circular accelerators Collider (LHC) at CERN. In the these extraordinary tools for extraor- with a fully three-dimensional treat- future, BeamBeam3D is expected to dinary science. ment of space charge. Space-charge be used to model beam-beam effects calculations are computationally in the proposed International Linear intensive, typically requiring the use Collider. Driving Discovery in of parallel computers. BeamBeam3D simulations of Synergia was designed to be dis- RHIC are relevant to both RHIC Existing Accelerators tributable to the particle accelerator operations and to the future operation community. Since compiling hybrid of the LHC (due to the similar beam- DOE operates some of the world’s code can be a complicated task which beam conditions). In regard to RHIC, most productive particle accelerators, is further complicated by the diverse the code was used to study beam- and one component of the AST proj- set of existing parallel computing beam effects in three areas. The first ect focused on developing scientific environments, Synergia includes a are the coherent beam-beam effects. codes to help researchers get even “build” system that allows it to be One problem is that beam-beam more science out of these facilities. compiled and run on various plat- effects in hadron colliders, such as As part of the SciDAC Accelerator forms without requiring the user to the one at RHIC, can create an oscil- Science and Technology project, the modify the code. lation under which the beams become Fermilab Computational Accelerator unstable; this instability represents a Physics group has developed the roadblock to increasing the luminosity. Synergia framework. Integrating and When Beams Collide Using BeamBeam3D, SciDAC extending existing accelerator physics researchers modeled the coherent codes in combination with new codes While some accelerator research beam-beam effects first during colli- developed by the group, Synergia is involves firing a beam at a stationary sions of single bunches of particles, designed to be a general-purpose target, other accelerators are used to then during multiple bunch collisions. framework with an interface that is generate particle beams which are In the latter case, each beam has accessible to accelerator physicists targeted to collide with each other, three bunches that are coupled to who are not experts in simulation. resulting in millions of particles flying three bunches in the opposite beam In recent years, accurate model- out from the collision. High intensity, at four interaction points. Using the ing of beam dynamics in high-current, tightly focused beams result in high BeamBeam3D code running on high low-energy accelerators has become “luminosity,” a parameter which is performance computers, project necessary because of new machines proportional to the number of events members were able to calculate at under consideration for future appli- seen in an accelerator’s detectors. But which point the beams would become cations, such as the High Energy high luminosity also leads to increased unstable. However, the group also Physics neutrino program, and the beam disruption due to “beam-beam” found that by appropriately arrang- need to optimize the performance of effects, which consequently lowers ing the accelerator machine lattice existing machines, such as the the luminosity. parameters of the two rings at RHIC Spallation Neutron Source and the To better study these effects, a and generating sufficient tune separa- Fermilab Booster. These machines parallel three-dimensional, particle-in- tion between the two beams, oscilla- are characterized by high currents cell code called BeamBeam3D was tion could be controlled to the extent and require excellent control of beam created to model beam-beam effects that there is no risk of instability. losses. A common problem in accel- of colliding beams at high energy ring Second, using BeamBeam3D, the erators is that the particles can stray colliders. The resulting information can team studied emittance growth (a from the beam core, creating what is now be used to help accelerator oper- reduction in beam quality that affects known as a “halo,” which can lead to ators determine the optimum param- the luminosity) in beams which are beam loss. Understanding how the eters for colliders to maximize lumi- offset from one another. The team accelerator design and various physi- nosity and hence maximize scientific found that, while static offsets do not cal phemonena (such as the space- output. Under SciDAC, BeamBeam3D cause significant emittance growth charge effect) affect this halo forma- was used to model colliding beams in over short time periods, the impact of tion is an essential component of several DOE accelerators including offsets is much larger when they are accelerator modeling. the Fermilab Tevatron, the Relativistic time-modulated (due, for example, to Several computer simulations of Heavy Ion Collider (RHIC) at mechanical vibrations in the final- space-charge effects in circular accel- Brookhaven National Laboratory, focus magnets.) This finding provides erators using particle-in-cell techniques the PEP-II B-factory at Stanford a potential mechanism to account for have been developed. Synergia is a Linear Accelerator Center, and the the extra emittance growth observed during the machine operation. ing ten billion electrons or Lastly, the team used BeamBeam3D positrons is accelerated down to study long-range beam-beam effects the linac by superconducting at RHIC. Simulations performed at accelerating cavities, which give the energy level at which particles are them more and more energy till injected into the accelerator showed they meet in an intense crossfire a strong sensitivity of the long-range of collisions at the interaction beam-beam effects to the machine point. At these energies, tunes; this sensitivity was also observed researchers anticipate significant in the experiments. The team subse- discoveries that will lead to a quently modeled the long-range beam- radically new understanding of beam effects at the higher energy what the universe is made of levels at which the particles collide. and how it works. The energy Such simulations are providing of the ILC beams can be adjust- insight and a means to numerically ed to home in on elementary explore the parameter space for the particle processes of interest.

future wire beam-beam compensa- A critical component of the FIGURE 1. Using software developed under SciDAC, accelerator tion experiment planned at RHIC. ILC linac is the superconducting scientists developed this model of the new low-loss cavity for the radio frequency (SRF) accelerat- proposed International Linear Collider. The different colors illustrate how various sections were modeled in parallel on multiple ing cavity. The design, called processors. The end perpectives show the geometry details in Shaping the Future of TESLA, was created in the early the couplers which could not previously be modeled because of 1990s and has been the focus of the disparate length scales. Accelerator Design more than a decade of experimental R&D effort. Recently, a for preserving low emittance of parti- Although their scientific value is new cavity design has been proposed cles from the beams, which can reduce substantial, accelerators are very expen- which has a higher accelerating gra- the beam quality. sive and difficult to design, build and dient and 20 percent less cryogenics Previously, evaluating HOM operate. As a result, the scientific loss over the TESLA design – an damping in an SRF cavity took years community is now looking at inter- important consideration as the cavities to complete either experimentally or national collaboration to develop the must be cooled to extremely low tem- numerically. Today, using the tools next generation of accelerators. One peratures (2 degrees Kelvin, or about developed under SciDAC, HOM cal- prime example is the International –456 degrees Fahrenheit) to maintain culations can be done in a matter of Linear Collider (ILC), the highest superconductivity. weeks with the new parallel solver priority future project in the world- Accompanying intense experi- running on NERSC’s IBM SP3 and wide high energy physics community mental efforts at the KEK accelerator NCCS’s Cray X1E supercomputers. for probing into the fundamental center in Japan and Jefferson Lab in In addition, the use of tetrahedral nature of matter. Virginia, the development of this grid and higher order basis functions Presently, a large team of acceler- low-loss (LL) cavity has been greatly has enabled solutions in the complex ator physicists and engineers from facilitated by the new parallel electro- cavity geometry to be obtained with Europe, Asia and North America is magnetic codes developed at the unprecedented accuracy. As a result, working on the design of this tera- Stanford Linear Accelerator Center costly and time-consuming prototyp- electronvolt-scale particle accelerator (SLAC) under the AST project. In ing by the trial-and-error approach is under a unified framework, called the collaboration with the SciDAC avoided as the LL cavity can be Global Design Effort (GDE), to make Terascale Optimal PDE Solvers computationally designed as close to the most of limited R&D resources. Integrated Software Infrastructure optimal as possible. Under the AST project, modeling Center (TOPS ISIC), the SLAC team tools were developed and used to created a nonlinear 3D parallel finite study the effectiveness of proposed element eigensolver with which, for Advanced Accelerators: designs and look for better solutions. first time, one can directly solve for In the ILC design, two facing lin- the damped dipole modes in the 3D Smaller, Faster, Futuristic ear accelerators (linacs), each 20 kilo- LL cavity, complete with higher- meters long, hurl beams of electrons order mode (HOM) couplers. HOM DOE is also looking at future- and positrons toward each other at damping is essential for stable trans- generation accelerators, in which nearly the speed of light. Each port of “long bunch trains” of particles lasers and plasmas would be used nanometer-scale size beam contain- as they race down the ILC linacs and instead of electromagnetic cavities to

38 SCIENTIFIC DISCOVERY

One approach being explored oped with SciDAC support, on is a plasma-wakefield accelerator, supercomputers at DOE’s NERSC. in which a drive beam — either an This allowed scientists to see details intense particle beam or laser pulse of the evolution of the experiment, — is sent through a uniform plasma. including the laser pulse breakup and This creates a space-charge wake the injection of particles into the on which a trailing beam of parti- laser-plasma accelerator. This allows cles can surf. SciDAC investigators them to understand how the injec- in the Acceler-ator Science and tion and acceleration occur in detail Technology (AST) project have so that the experiment’s designers developed a suite of particle-in- can figure out how to optimize the cell (PIC) codes to model such process. devices. The results of the experiment and SciDAC support has enabled simulations were published as the the development of four independ- cover article in the Sept. 30, 2004 ent, high-fidelity, particle-in-cell issue of Nature (Figure 2). (PIC) codes: OSIRIS (fully explicit Great progress has also been PIC), VORPAL (fully explicit PIC made in the development of reduced plus ponderomotive guiding cen- description models for laser/plasma ter), QuickPIC (quasi-static PIC simulation. One such example, plus ponderomotive guiding cen- QuickPIC, has been shown for some ter), and UPIC (a framework for problems to provide answers as accu- rapid construction of new codes rate as OSIRIS but with two to three such as QuickPIC). Results from orders of magnitude less computa- these codes were included in tion time. Before the SciDAC effort, a FIGURE 2. SciDAC codes were used to analyze the three articles in Nature and eight full-scale simulation of a 1 TeV after- experiments in two of three featured articles on compact particle accelerators in the September in Physical Review Letters. For burner would have required 5 million 30, 2004 issue of Nature. The cover image was example, members of the SciDAC node hours on a supercomputer (and created using the VORPAL code developed by AST project have performed thus was not done); after SciDAC, Tech-X Corp. partly with SciDAC support. large-scale simulations using the using QuickPIC, the simulation was codes OSIRIS and VORPAL to done in 5,000 node hours on a clus- help interpret laser- and plasma- ter system. accelerate particles, which could wakefield accelerator experiments result in accelerators with 1,000 times and to gain insight into the accelera- the performance of current technology. tion mechanism. The challenge is to control these Researchers at Lawrence high-gradient systems and then to Berkeley National Laboratory took a string them together. Such technolo- giant step toward realizing the prom- gies would enable the development of ise of laser wakefield acceleration, by ultra-compact accelerators, which guiding and controlling extremely would be measured in meters, not intense laser beams over greater dis- kilometers. However, experiments to tances than ever before to produce date have yielded acceleration over high-quality, energetic electron very short distances (millimeters to beams. The experimental results centimeters) resulting in beams of were analyzed by running the VOR- modest quality. PAL plasma simulation code, devel-

39 The Standard Model has been enormously successful in explaining a wealth of data produced in accelerator and cosmic ray experiments over the past twenty-five years. However, our knowledge of the Standard Model is incomplete because it has been difficult to extract many of the most interesting predictions of QCD, those that depend on the strong coupling domain of the theory. The only way to extract these predictions from first principles and with controlled errors is through large-scale numerical simulations. These simulations are needed to obtain a quantitative understanding of the physical phe- Lattice QCD nomena controlled by the strong interactions, to determine a number of the basic parameters of the Improved Formulations, Algorithms and Standard Model, and to make precise tests of the Standard Model’s range Computers Result in Accurate Predictions of validity. Despite the many successes of of Strong Interaction Physics the Standard Model, it is believed that to understand physics at the shortest distances or highest energies, a more general theory, which unifies all four of the fundamental forces of nature, will be required. However, to determine where the Standard Model breaks down and new physics is required, one must first know what the Standard Model predicts. Numerical simulations of QCD The long-term goals of high energy and address problems that are at the heart of the Department of Energy’s nuclear physicists are to identify the funda- large experimental programs in high energy and nuclear physics. Among mental building blocks of matter and to the major goals are (1) to calculate determine the interactions among them that the effects of strong interactions on weak interaction processes to an give rise to the physical world we observe. accuracy needed to make precise tests of the Standard Model; (2) to Major progress towards these goals has determine the properties of strongly been made through the development of interacting matter under extreme conditions, such as those that existed the Standard Model of high energy physics. in the very early development of the universe and are created today in rel- The Standard Model consists of two quan- ativistic heavy ion collisions; and (3) tum field theories: the Weinberg-Salam to calculate the masses of strongly interacting particles and obtain a Theory of the electromagnetic and weak quantitative understanding of their internal structure. interactions, and quantum chromodynamics According to QCD, the funda- (QCD), the theory of the strong interactions. SCIENTIFIC DISCOVERY

mental building blocks of strongly interacting matter are quarks. Quenched approximation Full QCD Interactions among quarks are mediated by gluons, in a manner analo- fπ fπ gous to the way photons mediate the fΚ fΚ electromagnetic interactions. QCD 3M – M 3M – M interactions are so strong that one Ξ Ν Ξ Ν 2M – M 2M – M does not directly observe quarks and Βμ Τ Βμ Τ ψ ψ gluons under ordinary laboratory (1P – 1S) (1P – 1S) conditions. Instead, one observes γ(1D – 1S) γ(1D – 1S) strongly interacting particles, such as γ(2P – 1S) γ(2P – 1S) protons and neutrons, which are γ(3S – 1S) γ(3S – 1S) bound states of quarks and gluons. γ(1P – 1S) γ(1P – 1S) QCD was initially formulated in the 1% errs 1% errs four-dimensional space-time continu- 0.8 1.0 1.2 1.4 0.8 1.0 1.2 1.4 um; however, to carry out numerical LatticeQCD/Expít LatticeQCD/Exp’t simulations, one must reformulate the theory on a discrete four-dimen- FIGURE 1. The ratio of several quantities calculated in lattice QCD to their experimental values. The sional lattice—hence the name lattice panel on the left shows results from the quenched approximation, and that on the right from full QCD. QCD. Major progress has been made in physics community, it has been fea- quark and a charmed anti-quark, was the numerical study of QCD during tured in Fermi News Today, Physics first observed in 1998, but its mass the course of the SciDAC program Today, and in a cover article in the was only poorly measured. With the through the introduction of improved CERN Courier. aid of SciDAC-funded prototype formulations of QCD on the lattice, A number of other important val- clusters at Fermilab, the mass of the coupled with improved algorithms idations of lattice QCD methods have Bc meson was calculated to be 6304 and major advances in the capabilities also been enabled by the SciDAC ± 20 MeV, a dramatic improvement of high performance computers. These Program. They include the determi- in accuracy over previous lattice cal- developments have allowed physicists nation of the strong interaction cou- culations. Soon after this result was to fully include the effects arising pling constant in agreement with but made public, the CDF experiment at from the polarization of the QCD with somewhat smaller errors than the Fermilab’s Tevatron finished a new, ground state due to the creation and world average from a number of dif- precise measurement of the mass: annihilation of quark-antiquark pairs. ferent experimental determinations; 6287 ± 5 MeV, confirming the pre- The inclusion of vacuum polar- the determination of the nucleon axial diction from lattice QCD. The fine ization effects has been the greatest charge, which governs the β decay of agreement was covered in the New challenge to performing accurate a free neutron into a proton, electron, Scientist, The Scotsman newspaper, numerical calculations of QCD. and neutrino; and the calculation of and a “News and Views” article in Their importance is illustrated in the Cabibbo-Kobayashi-Maskawa Nature. The success of the lattice cal-

Figure 1, where lattice QCD calcula- (CKM) matrix element Vus to an culation was named one of the top tions of the masses of a number of accuracy comparable with experiment. physics stories of 2005 by Physics strongly interacting particles, and the (The CKM matrix describes how News Update, which described it as decay constants of the π and K mesons, quarks couple to the weak interac- “the best-yet prediction of hadron are compared with their experimental tions; its elements are fundamental masses using lattice QCD.” values. In each case, agreement with parameters of the Standard Model.) One of the major objectives of experiment was found within statisti- During the course of the SciDAC the field of lattice QCD is to deter- cal and systematic errors of 3% or Program, the lattice QCD communi- mine the decay properties of pseu- less when vacuum polarizations were ty moved from the validation of doscalar mesons with one light and included (right panel), but this was techniques, through the calculation one heavy quark. Strong interaction not the case when the uncontrolled of quantities that are well known effects in leptonic decays are charac- approximation of ignoring vacuum experimentally, to the successful pre- terized by decay constants, while in polarization was made (left panel). diction of quantities that had not yet semileptonic decays they are charac- This work was described in a “News been measured. One important terized by various form factors. The and Views” article in Nature, as well example was the prediction of the decay constants and form factors for as in a “News Focus” article in mass of the Bc meson. This exotic B and Bs mesons, which contain Science. Within the high energy particle, which consists of a bottom heavy b quarks, play a critical role in

41 SCIENTIFIC DISCOVERY

The first lattice QCD results for q2 /m2 max Ds* the leptonic decay constants of the D 2.5 and Ds mesons that fully took into account vacuum polarization effects were announced in June 2005. Within 2 a few days of these results being made public, the CLEO-c Collaboration announced its experimental result for the decay constant of the D meson; 1.5 and in April 2006 the BaBar Colla-

)

2 boration announced its determination

f of the decay constant of the Ds meson. 1 In both cases the experiments confirmed the lattice QCD calculations with comparable errors. The lattice and experimental results for the decay 0.5 Experiment [Belle, hep-ex/0510003] of the D meson were the subjects of Lattice QCD [Fermilab/MILC, hep-ph/0408306) cover articles in the CERN Courier and the New Scientist. 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 Finally, the form factors that char- D Klv acterize the decay of a D meson into 2 a K meson and leptons were predicted FIGURE 2. The semileptonic form factor f + (q ) for the decay of a D meson into a K meson, a lepton, and a neutrino, as a function of the momentum transfer to the leptons q2. The orange curve is the with lattice QCD in work that was lattice result, and the blue points are the experimental results of the Belle Collaboration. also enabled by the SciDAC Program. The results were subsequently confirmed in experiments by the Focus and Belle collaborations. The lattice tests of the Standard Model that are contain heavy c quarks, are being results are compared with experi- currently a major focus of the experi- measured to high accuracy by the mental ones from Belle in Figure 2. mental program in high energy physics. CLEO-c Collaboration. Since the lat- The excellent agreement between These quantities are very difficult to tice techniques for studying mesons theory and experiment provides one measure experimentally, so accurate with c and b quarks are identical, more piece of evidence that lattice lattice calculations of them would be these experiments provide an excel- QCD calculations are able to make of great importance. On the other lent opportunity to validate the lat- accurate predictions of strong inter- hand, the decay constants and form tice approach being used in the study action physics. This work too was

factors of D and Ds mesons, which of D and B decays. featured in Fermi News Today.

42 SOFTWARE FOR SCIENCE

lation codes as well as on the mathematical and computing systems Scientific Challenge software that underlie these codes. In the following pages, descriptions of many of the software projects will illustrate how SciDAC has Codes: Designing supported software development to benefit the offices of Basic Energy Research, Biological and Environ- Scientific Software for mental Research, Fusion Energy Sciences and High Energy and Nuclear Physics. Next-Generation Basic Energy Sciences: Understanding Energy Supercomputers at the Atomic Level As one of the world’s leading For more than 40 years, the power of computer processors has sponsors of basic scientific research, doubled about every 18 months, following a pattern known as the Department of Energy has long supported investigations into how Moore’s Law. This constant increase has resulted in desktop atoms interact, how they form mol- computers which today are more powerful than the super- ecules and how groups of atoms and molecules react with one computers of yesteryear. Today’s supercomputers, at the same another. Understanding the forces time, offer computing power which enables researchers to at work among the most basic building blocks of matter can pro- develop increasingly complex and accurate simulations to vide greater insight into how energy address the most challenging scientific problems. However, is released, how waste products are generated during chemical process- the software applications for studying these problems have es and how new materials can be not kept pace with the advances in computational hardware. developed. Within the Office of Science, the Basic Energy Sciences (BES) Throughout the Office of the most powerful supercomputers, program supports such fundamental Science research program are major and to run as efficiently as possible research in materials sciences and scientific challenges that can best be to make the most effective use of engineering, chemistry, geosciences addressed through advances in sci- those systems. and molecular biosciences. Basic entific supercomputing. These chal- This is a daunting problem. research supported by the BES pro- lenges including designing materials Current advances in computing gram touches virtually every aspect with selected properties, under- technology are typicaly driven by of energy resources, production, standing and predicting global cli- market forces in the commercial conversion, efficiency, and waste mate change, understanding and sector, resulting in systems designed mitigation. Under the SciDAC pro- controlling plasma turbulence for for commerce, not scientific com- gram, a number of projects were fusion energy, designing new parti- puting. Harnessing commercial funded to support computational cle accelerators, and exploring basic computing technology for scientific chemistry. questions in astrophysics. research poses problems unlike those Research in chemistry leads to A key component of the encountered in previous supercom- advances such as efficient combus- SciDAC program is the development puters, both in magnitude as well as tion systems with reduced emissions of scientific challenge codes — new in kind. This problem will only be of pollutants, new processes for applications aimed at addressing solved by increased investments in converting solar energy, improved key research areas and designed to computer software — in research catalysts for the producing fuels and take advantage of the capabilities of and development on scientific simu- chemicals and better methods for

43 SOFTWARE FOR SCIENCE

environmental remediation and waste management. Over the past 50 years, molecular theory and modeling have advanced from merely helping explain the properties of molecules to the point where they provide exact predictive tools for describing the chemical reactions of three- and four-atom systems, the starting point for many of these chemical processes. This is increasingly important as researchers seek to understand more complex molecules and processes such as combustion, which involves complex interactions of chemistry with fluid dynamics. The advances in computational chemistry in recent years in providing accurate descriptions of increas- FIGURE 1: Simulations of turbulent nonpremixed ethylene-air flames reveal detailed structure of dynamics of the soot formation process. Shown from left to right are instantaneous images of the ingly complex systems have come vorticity, temperature, and soot volume fraction. as much from improvements in theory and software as from improved Two of the projects focused on (DNS) to enable the highest accura- computational hardware. However, improving computational modeling cy and allow scientists to study the the computational requirements of combustion, which is key to 80 fine-scale physics found in turbulent often outweigh the scientific gains. percent of the energy production in reacting flows. For example, electronic structure the United States. Developing new Under SciDAC, the simulation theory provides the framework for methods to make combustion more code named S3D has been enhanced modeling complex molecular-level efficient and cleaner will benefit the with many new numerical algorithms systems (from atoms to thousands economy, the environment and the and physical models to provide pre- of atoms) with increasing accuracy. quality of life. With such a promi- dictive capabilities for many of the But accurate electronic structure nent role in energy production and detailed processes which occur dur- theories have computational costs use, combustion has long been an ing combustion, including thermal that rise with the 6th power (or important research focus for the radiation, soot dynamics, spray higher) of molecular size, so that 10 DOE. injection and evaporation, and times the computing resources But fully understanding combus- flame-wall interaction. The S3D translates into treating a system less tion is an extremely complex prob- code was used to perform detailed than 1.5 times bigger. Predictive lem involving turbulent flows, many three-dimensional combustion sim- modeling of such processes is cur- chemical species and a continuing ulations of flames in which fuel and rently beyond the capabilities of series of interdependent chemical oxygen are not premixed. This existing computational resources processes and reactions. As comput- research could have applications in and computational methods. ers have become more powerful, such areas as jet aircraft engines, Under the SciDAC program, more detailed combustion simula- where fuel and oxidizers are not BES supported a number of proj- tions can be created to help scien- premixed for safety reasons, and in ects aimed at developing computa- tists better understand the process. direct-injection internal combustion tional approaches to solving prob- The Terascale High-Fidelity engines. Under certain conditions, lems in the modeling of chemical Simulations of Turbulent Combus- this type of combustion can sudden- processes that exceed current com- tion with Detailed Chemistry proj- ly and unexpectedly become extin- putational capabilities. These project is a multi-university collabora- guished, and this project is expected ects were selected to increase the tive effort to develop a high-fidelity to lead to a better understanding of accuracy of models, increase the turbulent reacting flow simulation this problem, as well as re-ignition size of systems which could be sim- capability that can take advantage of extinguished flames. The team ulated and expand the capabilities of the most powerful supercomput- demonstrated the advanced DNS of key applications already being ers available. The approach is based capability by undertaking several used in the research community. on direct numerical simulation laboratory-scale simulations to high-

44 SOFTWARE FOR SCIENCE

light fundamental aspects of turbu- chemical analysis algorithms and speed, scalability and accuracy of lence-chemistry interaction occur- software that enable both the electronic structure applications, ring in many practical combustion extraction of enhanced physical had two thrusts. systems. understanding from reacting flow (1) The Multiresolution Principal Investigators: Hong G. Im, computations, and the development Quantum Chemistry — Guaranteed University of Michigan; Arnaud Trouvé, of chemical models of reduced Precision and Speed component University of Maryland; Christopher J. complexity. The overall construc- worked to increase the accuracy of Rutland, University of Wisconsin; and tion is being implemented in the chemistry codes as they are scaled up Jacqueline H. Chen, Sandia National context of the common component to run on larger systems, while also Laboratories architecture (CCA) framework. The reducing the amount of processing The Computational Facility for assembled software will be applied time needed to run the calclulations. Reacting Flow Science project is to targeted reacting flow problems, Additionally, computational chem- aimed at advancing the state of the in two and three dimensions, and istry is expected to benefit from the art in the understanding and predic- validated with respect to reacting resulting computational framework tion of chemical reaction processes, flow databases at the Combustion that is substantially simpler than such as combustion, and their inter- Research Facility of Sandia National conventional atomic orbital meth- actions with fluid flow. Such detailed Laboratories. ods, that has robust guarantees of reacting flow computations are Principal Investigator: Habib Najm, Sandia both speed and precision, and that computationally intensive, yet they National Laboratories is applicable to large systems. In provide information that is neces- contrast, current conventional Another set of projects focused methods are severely limited in on improving applications for mod- both the attainable precision and eling electronic structure. Electronic size of system that may be studied. structure calculations, followed by The project achieved its goals for dynamical and other types of mole- effective one-electron theories and cular simulations, are recognized as is studying many-body theories. a cornerstone for the understanding The electronic structure methods and successful modeling of chemical should be valuable in many disci- processes and properties that are plines, and the underlying numerical relevant to combustion, catalysis, methods are broadly applicable. photochemistry, and photobiology. The Advanced Methods for Principal Investigator: Robert Harrison, Oak Ridge National Laboratory Electronic Structure project advanced the capabilities of quantum chem- (2) The component to develop ical methods to describe efficiently Advanced Methods for Electronic and with controllable accuracy the Structure: Local Coupled Cluster electronic structure, statistical Theory was designed to extend FIGURE 2. Reaction-diffusion high-order AMR mechanics, and dynamics of atoms, coupled cluster (CC) electronic computations of the propagation of random premixed hydrogen-oxygen ignition kernels molecules and clusters. The project, structure theory to larger molecules in two dimensions. The temperature field is which had goals of improving the (“coupled cluster” is a numerical shown, indicating cold reactants (blue) and technique used for describing many- hot combustion products (red), separated by the propagating flame fronts. body systems). This method was achieved by developing novel “fast” coupled cluster algorithms that are sary for the understanding and pre- much more scalable with respect to diction of turbulent combustion. system size to more fully realize the The project’s approach towards potential of high performance com- achieving this goal includes two puting for treating larger molecular broad focus areas. The first involves systems. CC methods are well developing high-accuracy adaptive established as the wave-function- mesh refinement (AMR) algorithms based electronic structure method and implementing them in a flexible of choice. However, even the sim- software toolkit for reacting flow plest CC method, incorporating just computations. The second involves correlations between pairs of elec- FIGURE 3. Molecular orbital of the benzene trons (double substitutions), still the development of advanced dimer with the adaptive grid and an isosurface. requires computational costs that

45 SOFTWARE FOR SCIENCE

scale with the 6th power of the size face. The first goal is to provide bet- of the molecule. This research ter thermochemistry for larger sys- Biological and focused on defining new and pow- tems and improve the performance erful “local correlation” models that of related components in the Environmental reduce scaling of coupled cluster Columbus computational chemistry calculations, and developing effec- software system. The second goal is Research: Advanced tive algorithms for their implemen- to develop highly parallelized quan- tation. tum dynamics and quantum kinetics Software for Studying software. Lastly, common compo- Principal Investigator: Martin Head- Global Climate Change Gordon, Lawrence Berkeley National nent architecture techniques will be Laboratory used to integrate kinetics and elec- In many fields of scientific tronic structure software into a The project for Advancing research, scientists conduct experi- package that will allow users to Multi-Reference Methods in ments to test theories, then analyze compute kinetics information just Electronic Structure Theory the results, and use the information by specifying the reactants. The researched and developed new mod- to continue refining their theories infrastructure program’s focus is on els for investigating detailed mecha- and/or experiments as they gain preconditioners (tools which nisms of chemical reactions for com- additional insight. In the study of improve convergence rates of cer- plex systems, including biomolecu- global climate change, however, the tain mathematical methods) and on lar processes, such as the interaction “experiment” takes place day after potential energy surface fitting of protein molecules. Specifically, day, year after year, century after schemes that reduce the number of the project worked on the develop- century, with the entire planet and electronic structure calculations ment of highly scalable electronic its atmosphere as the “laboratory.” necessary for accurate kinetics and structure codes that are capable of Climate change literally affects each dynamics. The infrastructure soft- predicting potential energy surfaces and every person on Earth and ware has benefited from Integrated of very high accuracy, which is because any measures taken to Software Infra-structure Centers important since potential energy address the issue will have far- (ISIC) support. surfaces determine how atoms and reaching social, economic and polit- chemical bonds rearrange during Principal Investigators: Albert F. Wagner ical implications, it’s critical that we chemical reactions. The ability to (2001-03), Ron Shepard (2004-05), have confidence that the decisions Argonne National Laboratory study extended systems containing being made are based on the best tens to thousands of atoms with The Theoretical Chemical possible information. reasonable accuracy is of paramount Dynamics Studies of Elementary Scientists have thought for more importance. The development of Combustion Reactions project than 100 years that increasing methods to adequately treat such involved modeling the dynamics of atmospheric concentrations of car- systems requires highly scalable, chemical reactions of large poly- bon dioxide (CO2) and other green- highly correlated electronic struc- atomic molecules and radicals house-gases from human activity ture methods interfaced with classi- important in combustion research, would cause the atmospheric layer cal methods and mesoscale codes. using a combination of theories and closest to the Earth’s surface to warm by several degrees. Because of Principal Investigator: Mark S. Gordon, methods. These computationally Ames Laboratory challenging studies will test the the complex feedbacks within the accuracy of statistical theories for Earth system, precisely estimating The Advanced Software for the predicting reaction rates, which the magnitude and rate of the Calculation of Thermochemistry, essentially neglect dynamical effects. warming, or understanding its Kinetics and Dynamics project This research is expected help effects on other aspects of climate, consisted of two integrated programs extend theoretical chemical dynam- such as precipitation, is a daunting to develop both scalable kinetics/ ics to the treatment of complex scientific challenge. Thanks to a dynamics software and infrastruc- chemical reactions of large poly- wealth of new observational data ture software. The overall thrust is atomic molecules. and advances in computing technol- to develop software to efficiently ogy, current climate models are able provide reliable thermochemistry Principal Investigator: Donald L. Thompson, University of Missouri to reproduce the global average (heat-related chemical reactions), temperature trends observed over kinetics, and dynamics for large the twentieth century and provide molecular systems. Kinetics/dynam- evidence of the effect of human ics studies compute how molecules activity on today’s climate. move over a potential energy sur- The Climate Change Research

46 SOFTWARE FOR SCIENCE

R15 T42 T63

T85 T106 T170

FIGURE 4. As supercomputers have become more powerful, climate researchers have developed codes with finer and finer resolution, allowing the models to incorporate more details which affect climate. The resulting climate models are more accurate. Images by Gary Strand, NCAR

Division in the DOE Office of nental-scale features of the observed techniques for studying and model- Science was established “… to climate, the models still cannot sim- ing climate change, focusing on the advance climate change science and ulate, and therefore cannot predict, numerical and computational improve climate change projections climate changes with the level of aspects of climate modeling. In par- using state-of-the-science coupled regional spatial accuracy desired for ticular, SciDAC funding has enabled climate models, on time scales of a complete understanding of the DOE researchers to participate with decades to centuries and space causes and effects of climate change. NSF and NASA researchers over an scales of regional to global.” As climate science advances and extended period in the design and Fortunately, the growing power new knowledge is gained, there is a implementation of new climate and capabilities of high-perform- demand to incorporate even more modeling capabilities. There are ance computers are giving climate factors into the models and to three major pieces, two of which researchers more accurate tools for improve acknowledged shortcom- pursued research in the academic analyzing and predicting climate ings in existing models. These arena that was longer term and change. Climate, whether in our demands, which make the models more basic. neighborhood, our region or conti- more accurate, will unfortunately nent, is determined by many factors, overwhelm even the most opti- some local, others global. Geography, mistic projections of computer A National Consortium to air temperature, wind patterns, power increases. So, a balance must Advance Global Climate ocean temperature and currents and be struck between the costs, bene- Modeling sea ice are among the forces that fits and tradeoffs required to allo- shape our climate. As a result, com- cate scarce computer and human The major SciDAC effort in cli- prehensive climate models which resources to determine which mate modeling was a multi-discipli- couple together simulations of dif- improvements to include in the next nary project to accelerate develop- ferent processes are among the generation of climate models. ment of the Community Climate most complex and sophisticated Under SciDAC, teams of climate System Model (CCSM), a computer computer codes in existence. While researchers worked to develop model of the Earth’s climate that they can simulate most of the conti- improved computational tools and combines component models for

47 SOFTWARE FOR SCIENCE

150

130

110

-10

-30

-50

-70

-90

-110

-130

-150

2 FIGURE 5. Using the Community Climate System Model, climate scientists ran a fully coupled simulation of the global carbon cycle to study how CO is either taken up or released by the ocean. Understanding this transfer is important because a significant fraction of anthropogenic CO2 emissions are currently absorbed by the ocean, thus slowing down the accumulation of CO2 in the atmosphere. This image shows the average exchange (in tonnes of carbon per square kilometer per year) in the last month (December) of a nine-year simulation. Areas with positive values (shown ranging from yellow to pink) indicate where CO2 is being taken up by the ocean, and negative values (shown in green ranging to purple) indicate where CO2is being released. When the CO2 in the ocean is out of equilibrium with that in the atmosphere, CO2 is exchanged between the two. Nonequilibrium can occur due to the temperature of the water, with the Northern Hemisphere (colder water in December) able to absorb more CO2 than the relatively warm Southern Ocean. Biological activity can also alter the equilibrium when microscopic plants use CO2 in the water for photosynthesis, which is then replaced by absorption from the atmosphere. The yellow and orange areas in the Southern Ocean demonstrate this phenomenon, showing uptake resulting from a seasonal phytoplankton bloom.

the atmosphere, ocean, land and sea portability across the wide variety decomposition schemes to enable ice. The CCSM is developed by of computer architectures required tuning for each computational plat- researchers from NSF, DOE, NASA for climate assessment simulations. form. The sea ice and land models and NOAA laboratories, as well as The second goal was to accelerate were restructured to improve per- universities. As one of the leading the introduction of new numerical formance, particularly for vector climate models in the United States, algorithms and new physical computers, and new software was CCSM has a large user base of sev- processes within CCSM models. developed to improve the coupling eral hundred climate scientists. The Under SciDAC, a consortium of of the four components into a fully CCSM community contributed a six national laboratories and resear- coupled model. large number of climate change chers from the National Center for These changes enabled the simulation results for the periodic Atmospheric Research (NCAR) and largest ensemble of simulations — Intergovernmental Panel on Climate NOAA worked together on a vari- 10,000 years worth — of any model- Change (IPCC) climate assessment ety of improvements to the CCSM. ing group in the world for the report. The Collaborative Design In the early years of the five-year recent IPCC assessment. These sim- and Development of the Com- project, attention was focused on ulations were performed at relative- munity Climate System Model for improving the performance and ly high resolution, generating more Terascale Computers project had portability of CCSM on the wide than 110 terabytes of climate model two goals. variety of vector and scalar comput- data, which were distributed inter- The first goal was to improve ers available to the community. nationally via the SciDAC-funded software design and engineering of The atmosphere and ocean Earth System Grid. This important the CCSM and its component models were improved with the contribution to the international cli- models and improve performance introduction of new flexible data mate research effort was made pos-

48 SOFTWARE FOR SCIENCE

sible by the critical software engi- ecosystems. The prototype model ocean components and alternative neering work performed by the developed under SciDAC will form numerical methods. SciDAC team members on all the the basis for future work on a com- The project, A Geodesic Climate components of CCSM. prehensive Earth system model and Model with Quasi-Lagrangian In addition to software expert- help the climate community enter a Vertical Coordinates, was created ise, the SciDAC CCSM consortium new phase of climate change to develop a new, comprehensive also contributed new model algo- research. model of the Earth's climate system rithms and new scientific capabilities. Principal Investigators: John Drake, Oak that includes model components for The consortium contributed to the Ridge National Laboratory, Phil Jones. Los the atmosphere, ocean, sea-ice, and introduction of a new finite-volume Alamos National Laboratory land surface, along with a model method for simulating atmospheric coupler to physically link the com- dynamics and developed new alterna- A Brand New Model ponents. A multi-institutional col- tive schemes for ocean models as well. Under SciDAC, DOE’s Office of laboration of universities and gov- Details of the consortium’s soft- Biological and Environmental ernment laboratories in an integrat- ware engineering of the CCSM Research pursued the novel concept ed program of climate science, were reported in articles featured in of a five-year cooperative agreement applied mathematics and computa- a special issue on climate modeling with one or two universities to build tional science, this project built of the International Journal of High the prototype climate model of the upon capabilities in climate model- Performance Computing and Appli- future. The concept was to develop ing, advanced mathematical cations (Vol. 19, No. 3, 2005). In the last years of the project, the focus of the SciDAC consortium was the development of new capabilities for simulating the carbon and sulfur cycles as part of the climate system. Until recently, most climate change scenarios specify a concentration of greenhouse gases and atmospheric aerosols. By adding the biological and chemical processes that govern the absorption and emission of greenhouse gases, better simulations of how the Earth system responds to human- caused emissions are possible (see Figure 1). A prototype carbon-climate-biogeochemistry model was assembled as a demonstration of the readiness to undertake coupled Earth system simulation at this dramatically new level of complexity. The new model included a comprehensive atmospheric chemistry formulation as well as land and FIGURE 6. By coupling different climate modeling components, climate researchers can take advan- ocean ecosystem models. In this tage of advancements in the separate codes to improve overall accuracy. first step towards a comprehensive a new climate model without the research, and high-end computer Earth system model, CO2 fluxes legacies of past approaches, which architectures to provide useful pro- were exchanged between compo- may be scientifically or computa- jections of climate variability and nents (see Figure 5) and the oceanic tionally outdated. Only one quali- change at regional to global scales. flux of dimethyl sulfide (DMS) was fied proposal was submitted, that The project team developed com- used in the atmospheric model to from a group centered at Colorado ponents for modeling the atmos- create sulfate aerosols. These State University, which is develop- phere, oceans and sea ice, and a aerosols then interacted and affect- ing a model using a radically differ- coupler component to links the ed the physical climate system, the ent grid design, alternative formula- physical model components. For oceanic carbon cycle and terrestrial tions for both the atmosphere and example, the coupler computes the

49 SOFTWARE FOR SCIENCE

latent heat flux exchanged between applying advanced computational conditions, understanding smaller the ocean surface and atmosphere. and statistical methods to help cli- phenomena such as tropical storms This physical coupling enables the mate models better predict these require 1-km resolution. The Con- coupled system to evolve in a effects. The computational aspect of tinuous Dynamic Grid Adap-tation coherent manner. As the pieces the project was aimed at developing in a Global Atmospheric Model is come together the integrated sys- efficient multi-level methods to sim- aimed at providing a capability to tem is being thoroughly tested. ulate these ocean flows and study break down the larger grid structure Principal Investigator: David A. Randall, their dependence on physically rele- to target areas of interest. While cli- Colorado State University vant parameters. The oceanographic mate models are not expected to and climate work consists in apply- provide a uniform 1-km resolution ing these methods to study the within the next 15 years, this grid University-Led Climate bifurcations in the wind-driven cir- adaptation capability is a promising Modeling Projects culation and their relevance to the approach to providing such resolu- flows observed at present and those tion for specific conditions. SciDAC supported two rounds that might occur in a warmer cli- of university grants to individual Principal Investigators: J.M. Prusa and W.J. mate. Both aspects of the work are researchers to address the 5- to 15- Gutowski, Iowa State University crucial for the efficient treatment of year needs and opportunities to large-scale, eddy-resolving numeri- While much discussion of climate advance climate modeling. These cal simulations of the oceans and an change focuses on the global scale, grants supported research into new increased understanding of climate there is also increasing demand for methodologies and numerical meth- change. climate modeling on the regional or ods. It is this basic research that mesoscale covering areas from 50 to explores new ideas and concepts Principal Investigators: Michael Ghil, several hundred miles in size. The that tie climate science and compu- UCLA; and Roger Temam, Indiana University project to develop Decadal Climate tational science together as both Studies with Enhanced Variable fields advance. With the advent of more power- and Uniform Resolution GCMs A major factor in climate change ful supercomputers and modeling Using Advanced Numerical Tech- between decades is the effect of applications, climate models, resolu- niques focused on using developed winds on ocean circulation. The tion of the globe has become increas- and evolving state-of-the-art general project on Predictive Understanding ingly precise — to the extent that circulation models (GCMs) with of the Oceans: Wind-Driven some models can focus on an area as enhanced variable and uniform res- Circulation on Interdecadal Time small as 10 kilometers square. While olution to run on parallel-processing Scales was aimed at developing and this is useful for large-scale climate terascale supercomputers. The major accomplishment of the project was completion of the international SGMIP-1 (Stretched-Grid Model Intercomparison Project, phase-1), which allows smaller regional models to be computed as part of global modeling systems. The “stretched grid” approach provides an efficient down-scaling to mesoscales over the area of interest and computes the interactions of this area with the larger model. This capability is expected to improve our understanding of climate effects on floods, droughts, and monsoons. Collaboration with J. Côté of MSC/RPN and his group is a strong integral part of the joint effort. The companion study with the members of another SciDAC group, F. Baer and J. Tribbia, is devoted to devel- FIGURE 7. Adapted grid with block-data structure. oping a stretched-grid GCM using the advanced spectral-element tech-

50 SOFTWARE FOR SCIENCE

nique with variable resolution, and ance in the climate modeling com- ments (a method of approximating the NCAR CAM physics. munity. The project to develop A mathematical solutions). The Earth’s Principal Investigator: Michael S. Fox- Vertical Structure Module for spherical domain is tiled with spec- Rabinovitz, University of Maryland Isopycnal Ocean Circulation tral elements that can be arbitrarily Models aimed at providing a mod- sized to meet local scaling require- The project for Development of ule for representing key processes in ments, allowing the model to create an Atmospheric Climate Model with such a model and organizing the predictions over a range of scales on Self-Adapting Grid and Physics vertical structure. The module can the entire global domain without was funded to develop adaptive grid then be inserted in the ‘dynamic user involvement in the computa- techniques for future climate model core’ of existing models and used by tional process. The method also and weather predictions. This the modeling community. takes optimum advantage of state- approach will lead to new insights of-the-art MPP by minimizing com- into small-scale and large-scale flow Principal Investigator: Kirk Bryan, Princeton University munication among elements and interactions that cannot be modeled thereby amongst processors. This by current uniform-grid simulations. The advent of terascale super- procedure has yielded dramatic This project will result in a climate computers has advanced climate speedup, making the production of model that self-adjusts the horizon- modeling by allowing the use of multiple realizations more feasible. tal grid resolution and the complex- higher-resolution atmospheric and In addition to serving as a research ity of the physics module to the ocean models. However, more dra- and training tool, the model is atmospheric flow conditions. Using matic improvements are likely expected to provide more accurate an approach called adaptive mesh through development of improved climate predictions. refinement to model atmospheric descriptions of physical processes motion will improve horizontal res- treated by the models, especially Principal Investigator: Ferdinand Baer, University of Maryland olution in a limited region without those that can be observationally requiring a fine grid resolution characterized in much finer details The Decadal Variability in the throughout the entire model domain. than are conventionally included in Coupled Ocean-Atmosphere System Therefore, the model domain to be the climate models. The project for project researched the slowly chang- resolved with higher resolution is Improving the Processes of Land- ing circulations in the oceans and kept at a minimum, greatly reducing Atmosphere Interaction in CCSM their influence on long-term global computer memory and speed require- 2.0 at High Resolution has con- climate variability. There were two ments. Several tests show that these tributed to this goal by advancing main themes in the project. The first modeling procedures are stable and the treatment of land with much concerned the decadal variability of accurate. improved details in the climatically upper ocean circulation and its role Principal Investigator: Joyce E. Penner, most important processes, using pri- in the long period fluctuations of the University of Michigan marily the Community Land Model coupled ocean/atmosphere system. (CLM). The second was the role of mesoscale The credibility of ocean models eddies — the oceanic flows on scales for climate research depends on their Principal Investigator: Robert E. Dickinson, Georgia Institute of Technology of 10 to 100 km — in the large-scale ability to simulate the observed state heat and energy budget of the ocean. and natural variations of the oceans As part of the effort to develop The model was developed and test- as indicated by actual measurements. climate models for making reliable ed on a smaller computer and will Existing models, which can be used climate predictions, one need is for be adapted for use on larger mas- for simulating the long time scales an efficient and accurate method for sively parallel supercomputers. of climate change of the order of producing regional climate predic- centuries, still do not provide a very tions and the development of com- Principal Investigator: Paola Cessi, Scripps Institution of Oceanography – University satisfactory treatment of key climat- puting methodology which uses the of California, San Diego ic processes such as water mass for- latest in computing hardware (mas- mation in the subpolar oceans. Ocean sively parallel processing or MPP) A project called Towards the models based on Cartesian coordi- most efficiently and economically, Prediction of Decadal to Multi- nates have been well tested and to produce the best prediction Century Processes in a High- their drawbacks are well known. results with minimal expenditure of Throughput Climate System Models based on a moving vertical resources. To meet this goal, the Model pursued interdisciplinary coordinate have the potential to Multi-Resolution Climate Modeling research to simulate decadal to provide a much more accurate sim- project developed a Spectral Element multi-century global variability and ulation, but are not “mature’” enough Atmospheric Model (SEAM), a fairly change in an earth system model at present to gain widespread accept- recent concept using spectral ele- that couples climate to the terrestri-

51 SOFTWARE FOR SCIENCE

nario, in which the concentration of GCM CO2 in the atmosphere was dou- ATMOSPHERE bled, the model showed that vegetation increased the uptake of CO2 by

NCO2 H2O GAP DYNAMICS: months/years 48 percent and that surface temper- MORTALITY, FIRE atures in some regions increased by (Moorcroft, et al., 2001) up to 2° C due to stomatal closure. GROWTH (Friend, in prep.) days Principal Investigator: Nancy Y. Kiang, ALBEDO (NI-Meister, et al., 1999) Columbia University CONDUCTANCE/ minutes PHOTOSYNTHESIS Gaining a better understanding (Friend & Klang, in prep.; of the relationship between climate Kull & Krult, 1999) change and the transport of water LEAF AREA vapor and chemicals in the atmos- NITROGEN phere was the aim of the Modeling RADIATION and Analysis of the Earth’s Hydro- logic Cycle project. This research addresses fundamental issues underlying the understanding and modeling of hydrologic processes, includ- SOIL ing surface evaporation, long-range CARBON transport of water, and the release SOIL SOIL of latent heating through evapora- NITROGEN BIOCHEMISTRY tion. These processes play a central months/years role in climate. The goals of the SOIL project are to advance climate MOISTURE HYDROLOGY change modeling by developing a hybrid isentropic model (in which the flow variables change gradually) FIGURE 8. Diagram of the dynamic vegetation embedded within the atmospheric and land surface for global and regional climate sim- components of a GCM. ulations, to advance the understanding of physical processes involving al ecosystem. The primary tool will energy, water, carbon dioxide and water substances and the transport be a high-throughput climate- nitrogen over land surfaces. The of trace constituents, and to exam- ecosystem model called FOAM that aim of the project for Modeling ine the limits of global and regional enables rapid turnaround on long Dynamic Vegetation for Decadal climate predictability. climate runs. FOAM is used to con- to Century Climate Change Studies Principal Investigator: Donald R. Johnson, tinue the study of the mechanisms is to develop, evaluate, utilize and University of Wisconsin – Madison of decadal climate variability, the make available a model of vegeta- dynamics of global warming, and tion/soil dynamics to improve the the interaction of climate and the ability of general circulation models land biosphere. The project also (GCMs) to make predictions of Fueling the Future: linked climate scientists with eco- future climate change. The model is system modelers in building a fully being developed by combining Plasma Physics and coupled ocean-atmosphere-land treatments of carbon and nitrogen biosphere model, which enabled the fluxes, and vegetation community Fusion Energy study of the interaction of land veg- dynamics, as a standalone module Plasmas, or very hot ionized etation changes and climate changes within the NASA Goddard Institute gases, make up more than 99 per- for past, present, and future scenarios. for Space Studies GCM. This model cent of the visible universe. In fact, will be a tool for answering questions Principal Investigator: Zhengyu Liu, the earth’s sun and the stars we see University of Wisconsin-Madison about past climate and vegetation at night are masses of plasma burn- distributions, as well as for predicting at millions of degrees Celsius. Vegetation is an important coming global changes due to rising Inside these stars, the incredibly ponent of the global climate system atmospheric CO2 in the coming high temperatures and pressures through its control of the fluxes of decades and century. In one sce- result in atoms fusing together and

52 SOFTWARE FOR SCIENCE

releasing more energy. For much of the plasma and hence to develop a understanding of the many complex the 20th century, scientists have successful fusion reactor. Achieving physical phenomena in the universe, investigated whether similar condi- such a goal is a long-term objective which we are only starting to tions could be created and used on and many steps are needed before understand. Being able to capture Earth as source of energy. efficient fusion reactors can be and reproduce the phenomenon of Over the last 30 years, comput- designed and built. A number of fusion on Earth would solve the ing has evolved as a powerful tool smaller experimental reactors have world’s energy problems in an eco- in scientific research, including been built in the U.S. and around nomically and environmentally sus- fusion. High performance comput- the world. These facilities are tainable manner. ers are now routinely being used to expensive to build and operate, but Like other fields, fusion science advance our understanding of fusion give scientists valuable information has developed subfields such as energy, with the goal of harnessing for future designs. plasma microturbulence theory, the same power source of the sun in Computational physics research magnetohydrodynamics (MHD) specially built reactors to provide has also helped give fusion scientists and transport theory to study the clean and almost unlimited energy. important insights into many aspects phenomena which occur, and com- But just as fusion energy is filled of plasma confinement, reactor fuel- putational science has greatly with huge potential, high performing and the effects of turbulence on advanced research in these areas. ance computing also presents plasmas. In fact, turbulence has Under SciDAC, multi-institution daunting scientific challenges. emerged as one of the biggest chal- projects were created to advance Most of DOE’s current research, lenges in magnetic fusion — as the understanding in specific areas to both in experimental facilities and in plasmas are heated and confined, help advance fusion research computational science, focuses on turbulence occurs and introduces around the world. magnetic fusion research. One instabilities, which disrupt the con- approach centers on reactors with finement. If the plasmas cannot be doughnut-shaped chambers called confined, the plasmas come into Center for Extended tokamaks which would be used to contact with the reactor walls, los- Magnetohydrodynamic heat ionized gas to about 100 mil- ing temperature and making fusion Modeling lion degrees centigrade, then use impossible. powerful magnetic fields to confine Under the SciDAC program, The Center for Extended and compress the plasma until computational scientists, applied Magnetohydrodynamic Modeling hydrogen atoms fuse together and mathematicians and fusion researchers project was established to enable a release helium and energy. This sus- worked in teams to develop new realistic assessment of the mecha- tained fusion reaction is known as a tools and techniques for advancing nisms leading to disruptive and “burning plasma.” As envisioned, fusion research, then using these other stability limits in the present such reactors would generate more methods to run more detailed simu- and next generation of fusion energy than they consume. Sustained lations. The result is ever increasing devices. Rather than starting from fusion power generation requires knowledge and understanding about scratch, the project built on the that we understand the detailed the forces at work within a fusion work of fusion research teams physics of the fluid of ions and elec- reactor. This information is being which developed the NIMROD trons, or plasma, in a fusion reactor. directly applied to the design and and M3D codes for magnetohy- With recent advances in super- construction of ITER, a multina- drodyamic (MHD) modeling. This computer hardware and numerical tional facility to be built in France. discipline studies the dynamics of algorithm efficiency, large-scale Since construction costs for electrically conducting fluids such as computational modeling can play ITER have been estimated at $12 plasmas. The goal was to develop an important role in the design and billion and once completed, operat- these codes to enable a realistic analysis of fusion devices. Among ing costs could be up to $1 million assessment of the mechanisms lead- the nations engaged in developing per day, it’s critical that the newest, ing to disruptive and other stability magnetic confinement fusion, the most detailed research findings be limits in the present and next gener- U.S. remains the world leader in applied before the system is com- ation of fusion devices. The main plasma simulation. pleted. Not only will this help mini- work involves extending and Computational techniques for mize the number of costly modifica- improving the realism of the leading simulating the growth and satura- tions, but it will also mean a greater 3D nonlinear magneto-fluid based tion of turbulent instabilities in this likelihood of success. models of hot, magnetized fusion plasma are critical for developing The results can also be applied plasmas, increasing their efficiency, this understanding of how to control on a broader scale to increase our and using this improved capability

53 SOFTWARE FOR SCIENCE

was able to achieve a series of scien- (a) 1 tific milestones in 2005 and 2006.

0.5 These modes shed thermal energy from the edge of the confinement z 0 region and, in their most virulent –0.5 form, could overheat the walls near the plasma, and may also affect the –1 core plasma. Using the codes advan- 0 0.5 1 1.5 2 2.5 3 3.5 ced under this project, the center r team was able to create more exten- (b) sive simulations of the effects of 1 ELMs than previously possible. The 0.5 evolution of the temperature during FIGURE 10. The (turbulent) electrostatic poten- the nonlinear evolution of an ELM tial from a GYRO simulation of plasma micro-

z 0 turbulence in the DIII-D tokamak. in shot 113207 is shown in Figure 9, –0.5 illustrating the formation of finger- theoretical interest are easily imple-

–1 like structures near the plasma edge mented in simulations, while similar with increasingly fine structure as measurements in the laboratory are 0 0.5 1 1.5 2 2.5 3 3.5 the calculation progresses. difficult or prohibitively expensive. r The project also made a number The development of tools in this (c) of modifications to the M3D code project will significantly advance the 1 to improve the accurate representa- interpretation of experimental con- 0.5 tion of a number of conditions in finement data and will be used to

z 0 tokamaks. test theoretical ideas about electrostatic and electromagnetic turbulence. –0.5 Principal Investigator: Steve Jardin, Princeton Plasma Physics Laboratory By fulfilling the objective of enabling –1 direct comparisons between theory and experiment, direct numerical 0 0.5 1 1.5 2 2.5 3 3.5 simulation will lead to improve- r The Plasma Microturbulence ments in confinement theory and Project increased confidence in theoretical FIGURE 9. Evolution of the temperature during the nonlinear evolution of an edge localized A key goal of magnetic fusion confinement predictions. The mode in shot 113207. programs worldwide is the con- Plasma Microturbulence Project struction and operation of a burning (PMP) is addressing this opportuni- to pioneer new terascale simulations plasma experiment. The perform- ty through a program of code of unprecedented realism and reso- ance of such an experiment is deter- development, code validation, and lution. As a result, scientists gained mined by the rate at which energy expansion of the user community. new insights into low frequency, is transported out of the hot core The project has implemented a long-wavelength nonlinear dynamics (where fusion reactions take place) number of code improvements for in hot magnetized plasmas, some of to the colder edge plasma (which is modeling different geometries and the most critical and complex phe- in contact with material surfaces). multi-species plasma models to sim- nomena in plasma and fusion science. The dominant mechanism for this ulate both the turbulent electric and The underlying models are validated transport of thermal energy is plas- magnetic fields in a realistic equilib- through comparisons with experi- ma microturbulence. rium geometry (see Figure 10). The mental results and other fusion codes. The development of terascale project’s global codes are able to An example of an important supercomputers and of efficient sim- simulate plasmas as large as those in fusion problem that is being addressed ulation algorithms provides a new present experiments, and the even by the center is the onset and non- means of studying plasma microtur- larger plasmas foreseen in burning linear evolution of “edge localized bulence — direct numerical simulation. plasma experiments. modes” (ELMs) and their effect on Direct numerical simulation comple- The project’s algorithms scale plasma confinement and the reactor ments analytic theory by extending nearly linearly with processor num- walls. By using the two MHD its reach beyond simplified limits. ber to ~1000 processors, which is codes and data from the DIII-D Simulation complements experiment important as fusion codes are signif- fusion device operated by General because non-perturbative diagnostics icant users of supercomputing Atomics in California, the center measuring quantities of immediate resources at DOE centers. To help

54 SOFTWARE FOR SCIENCE

the fusion research community ben- the necessary simulations are large. ticles are transported through the efit from this investment in the GS2 The ITER will have a major radius plasma by the turbulence. Experi- and GYRO codes, the project has of 6.2 meters and a minor radius of ments show that the particles are engaged researchers in helping to two meters, and a correspondingly transported quite differently than validate the codes against experi- large confined plasma must be sim- theory suggests. One of the objec- mental results. Additionally, project ulated. Second, the temperature tives of the center’s simulations is to members have conducted training inside ITER will be higher than any bridge this gap between experiment workshops and presented talks on other fusion reactor. This higher and theory. the codes at key fusion research temperature means that new types In the simulations, particles will meetings. of physics will be encountered. As a fly around in the plasma according Principal Investigator: Bill Nevins, result, new numerical simulation to Newton’s laws of motion, although Lawrence Livermore National Laboratory models must be developed. in a much smaller number than will The new codes must also be actually be present in the ITER created to perform as efficiently as plasma. Algorithms will be devel- Center for Gyrokinetic Particle possible so that the larger-scale sim- oped to follow the path of each Simulations of Turbulent ulations can be run on the nation’s simulated particle as it interacts most powerful supercomputers. with other particles and is affected Transport in Burning Plasmas The major challenge for the by turbulence transport. The results The Center for Gyrokinetic project is to use simulations to bet- of the simulations will then be com- Particle Simulations of Turbulent ter understand and minimize the pared with experimental results. Transport in Burning Plasmas con- problem of turbulence in the reac- To model the particles, the sortium was formed in 2004 to tor. At the core of the reactor, the group is using the the gyrokinetic develop codes to simulate turbulent temperatures are at their highest. At particle-in-cell (PIC) method, first transport of particles and energy, and the outside edges, the temperatures developed in the early 1980s and improve the confinement of burning are lower. As with weather, when widely adopted in the 1990s. Gyro- plasmas in fusion reactors. In partic- there are two regions with different kinetic refers to the motion of the ular, the project is aimed at devel- temperatures, the area between is particles in a strong magnetic field. oping the capabilities for simulating subject to turbulence. This turbu- In such a field, particles spiral along, burning plasma experiments at the lence provides a means for the with electrons spinning in one direc- scale of ITER, the international charged particles in the plasma to tion and ions in the opposite direc- thermonuclear experimental reactor move toward the outer edges of the tions. By simplifying the spiraling which is expected to be the first reactor rather than fusing with motion as a charged ring, researchers fusion reactor capable of sustaining other particles. If enough particles are able to increase the time steps a burning plasma when the machine (and their energy) come into con- by several orders of magnitude goes on line in about 10 years. tact with the reactor wall, the parti- without sacrificing scientific validity. Although fusion scientists have cles lose temperature and the fusion The project team also collabo- been creating simulation codes to reaction cannot be sustained. rated with Terascale Optimal PDE study fusion for more than 30 years, Scientists understand that the Simulations Integrated Software ITER presents two unique chal- difference in temperatures as well as Infrastructure Center (TOPS ISICs) lenges, according to Wei-li Lee, densities is what causes the turbu- established under SciDAC. By inte- head of the center. lence. But what is still not fully grating other software libraries for First, the size of the reactor and understood is the rate at which par- solving the equations describing the interaction between the particles, the project team was able to solve a number of physics problems and integrate more realistic physics modules in their simulations. The result is that the simulations are expected to be able to model ITER- scale plasmas by solving a system of equations with millions of unknowns by following billions of particles in the computer. FIGURE 11. Potential contours of microturbulence for a magnetically confined plasma. The finger-like Principal Investigator: W.W. Lee, Princeton perturbations (streamers) stretch along the weak field side of the poloidal plane as they follow the magnetic field lines around the torus. Plasma Physics Laboratory

55 SOFTWARE FOR SCIENCE

Understanding Magnetic computer scientists, which will take full advantage of national terascale Reconnection in Plasmas computational facilities to address The Center for Magnetic the present and future needs for Reconnection Studies (CMRS) is a atomic-scale information for fusion multi-university consortium dedicated and other plasma environments. to physical problems involving mag- Especially in regard to the needs netic reconnection in fusion, space, of fusion, atomic and molecular col- and astrophysical plasmas. Under- lisions significantly influence the standing magnetic reconnection is transport and energy balance in one of the principal challenges in regions crucial for the next step of plasma physics. Reconnection is a magnetic confinement fusion devel- process by which magnetic fields opment, the divertor and edge of reconfigure themselves, releasing FIGURE 13. A simulation of the so-called kink- tokamaks. For example, the micro- tearing instability in tokamaks that can cause energy that can be converted to internal current disruption and core tempera- scopic modeling of turbulence and particle energies. ture collapse. The two intertwining helical transport in magnetic fusion edge The goal of the Magnetic flux tubes are formed as a result of reconnection, and their 2D projections show mag- plasmas relies heavily on an accu- Reconnection project was to pro- netic island structure. rate knowledge of the underlying duce a unique high performance atomic processes, such as elastic code to study magnetic reconnection scattering, vibrational energy transfer, in astrophysical plasmas, in smaller- project team has applied the MRC mutual neutralization, and dissocia- scale laboratory experiments and in to a wide spectrum of physical tive recombination. The transport fusion devices. The principal com- problems: internal disruption (Figure and atomic conversion of radiation putational product of the CMRS — 13) and error-field induced islands is also at the heart of inertial fusion the Magnetic Reconnection Code in tokamaks, storms in the magne- experiments, where a vast array of (MRC) — is a state-of-the-art, large- tosphere, solar/stellar flares, and electron and photon collision pro- scale MHD code for carrying out vortex singularity formation in fluids. cesses, as well as of radiative and magnetic reconnection research electron cascade relaxations, is Principal Investigator: Amitava Bhattacharjee, with accuracy and completeness in University of New Hampshire required to model, diagnose, guide, 2D and 3D. The MRC is massively and understand experiments. In parallel and modular, has the flexi- addition, most magnetic fusion diag- bility to change algorithms when Terascale Computational Atomic nostics involve interpretation of necessary, and uses adaptive mesh observations of the conversion of refinement (AMR) (Figure 12). The Physics for the Edge Region in Controlled Fusion Plasmas FIGURE 14. Accurate modeling of plasmas Atomic physics plays a central requires accurate knowledge of atomic role in many of the high temperature processes, such as scattering. This series is from a simulation capturing all scattering and high density plasmas found in processes for the scattering of a wavepack magnetic and inertial confinement from a helium atom. fusion experiments, which are crucial to our national energy and defense interests, as well as in technological plasmas important to the U.S. economic base. In turn, the development of the necessary atomic physics knowledge depends on advances in both experimental and FIGURE 12. AMR allows the MRC code to place refined numerical grids automatically computational approaches. The where the fine spatial structures require Terascale Computational Atomic them, as shown in the figure where sharp spatial gradients at the center are resolved Physics for the Edge Region in by placing refined grids (see inset for zoom). Controlled Fusion Plasmas project The MRC code has the capability to refine was established to develop a new not only in space but also in time, which is more efficient since one does not need to generation of scientific simulation take unnecessarily small time steps in regions codes, through a broad ranging col- where the fields are smooth. laboration of atomic physicists and

56 SOFTWARE FOR SCIENCE

fast electron energy to atomic ion mak, scientists are studying different used to design and build fusion light emission, such as in the well approaches to heating. One method reactors, including ITER. During known charge exchange recombina- involves radio waves, similar to the the first three years of the project, tion spectroscopy. Development of microwaves used to heat food. As the main focus was on wave inter- new scientific simulation codes to the waves propagate through the action and conversion. What address these needs will also benefit plasma, they interact with the ions researchers discovered was that research involving a huge range of spinning in the plasma. When the under certain conditions, the rela- technical, astrophysical, and atmos- ions’ gyro-frequency resonates at tively long waves propagated from pheric plasmas that also depend on the same frequency as the wave, the an antenna inside the tokamak accurate and large scale databases of ions spin up to a very high energy, could be converted to much shorter atomic processes. further heating the plasma. waves, change characteristics and Many of the project’s atomic The goal of Numerical Comp- become much shorter in length. collision codes are being imple- utation of Wave-Plasma Interact- This process is known as “mode mented on supercomputers at DOE ions in Multi-Dimensional Systems conversion.” One form of the shorter computing centers in California and is to create two-dimensional and wave, known as an “ion cyclotron Tennessee. Results of these code three-dimensional simulations of wave,” interacts with both the ions developments will be disseminated the interactions between the plasma and the electrons in the plasma. openly to the relevant plasma sci- and the waves to make accurate The project is using two plasma ence and atomic physics communi- predictions of the processes. The modeling codes — AORSA and ties, and will enable new regimes of results of these simulations will be TORIC — to study the effects of the computation by taking advantage of terascale and successor facilities. All RF accelerated beam ions of the atomic collision codes have important applications in a variety of research areas outside controlled fusion plasma science. Principal Investigator: Michael Pindzola, Injected beam ions Auburn University

Minor radius Numerical Computation of Vperp Wave-Plasma Interactions in Multi-Dimensional Systems Thermal ions In order to achieve the extremely Vpar high temperatures — six times hotter than the core of the sun — needed FIGURE 15. Under SciDAC, fusion energy researchers were allocated time on the Cray XT-3 massively parallel supercomputer “Jaguar” at Oak Ridge National Laboratory. By using thousands of proces- to drive fusion reactions in a toka- sors, the team was able for the first time to run this calculation of how non-thermal ions are distributed in a heated fusion plasma.

57 SOFTWARE FOR SCIENCE

ion cyclotron wave. One effect achieved through the scaling of sim- Advanced Computing for 21st appears to be that the wave drives ulation codes and the availability of flows in the plasma which help terascale computing systems. Century Accelerator Science & break up turbulence, thereby Principal Investigator: Don Batchelor, Oak Technology improving confinement of the plasma. Ridge National Laboratory (2001-05), Paul In summer 2005, the AORSA Bonoli, Massachusetts Institute of Accelerators underpin many of global-wave solver was ported to Technology, (2006-present) the research efforts of the Office of the new Cray XT3 supercomputer Science and physics research around at Oak Ridge National Laboratory. the world. From biology to medicine, The code demonstrated excellent from materials to metallurgy, from scaling to thousands of processors. High Energy and elementary particles to the cosmos, Preliminary calculations using 4,096 accelerators provide the microscopic processors have allowed the first Nuclear Physics: information that forms the basis for simulations of mode conversion in scientific understanding and applica- ITER. Mode conversion from the Accelerating Discovery tions. Though tremendous progress fast wave to the ion cyclotron wave has been made, our present theory has been identified in ITER using from Subatomic of the physical world is not com- mixtures of deuterium, tritium and plete. By tapping the capabilities of helium-3 at 53 MHz. Particles to Supernovae the different types of accelerators, A second focus of the project including colliders, spallation neu- has been on how energies are dis- The Office of Science supports a tron sources and light sources, sci- tributed in the plasma. The simplest program of research into the funda- entists are making advances in type of distribution is known as a mental nature of matter and energy many areas of fundamental science. Maxwellian distribution, which is through the offices of High Energy Much of our knowledge of the used to calculate how energy is dis- Physics and Nuclear Physics. In car- fundamental nature of matter results tributed in all gases, such as the air rying out this mission, the Office of from probing it with directed beams in a room. This distribution function Science: of particles such as electrons, pro- was previously the only method in • builds and operates large, world- tons, neutrons, heavy ions and pho- which fusion researchers could cal- class charged-particle accelerator tons. The resulting ability to “see” culate wave propagation and facilities for the nation and for the building blocks of matter has absorption. But, when plasmas are the international scientific had an immense impact on society heated to very high temperatures, a research community; and our standard of living. Over the long tail of particles can form, • builds detectors and instruments last century, particle accelerators which represents a non-Maxwellian designed to answer fundamental have changed the way we look at distribution. Using another code questions about the nature of nature and the universe we live in called CQL3D, project team mem- matter and energy; and and have become an integral part of bers were able to calculate the new • carries out a program of scientific the nation’s technical infrastructure. energy distributions. The code was research based on experimental Today, particle accelerators are also installed on the Cray XT3 data, theoretical studies, and sci- essential tools of modern science supercomputer along with the entific simulation. and technology. AORSA code, and both were pro- For example, about 10,000 can- grammed to automatically iterate The scale of research ranges cer patients are treated every day in with one another. So, once the new from the search for subatomic parti- the United States with electron energy distributions were calculated, cles, which are one of the most beams from linear accelerators. this data was fed back into the basic building blocks of matter, to Accelerators produce short-lived AORSA code to recalculate the studying supernovae, massive radioisotopes that are used in over heating of the plasma. Once the exploding stars which also serve as 10 million diagnostic medical proce- reheating was simulated, the an astrophysical laboratory in which dures and 100 million laboratory CQL3D code then calculated the unique conditions exist that are not tests every year in the United States. energy distribution. The iterations achievable on Earth. Under the The SciDAC Accelerator were repeated until a steady state SciDAC program, projects were Science and Technology (AST) was reached in the simulation. launched to advance research in the modeling project was a national The project results, which make areas of accelerator science and sim- research and development effort key contributions to understanding ulation, quantum chromodynamics aimed at establishing a comprehen- processes in hot plasmas, were and supernovae science. sive terascale simulation environ-

58 SOFTWARE FOR SCIENCE

ment needed to solve the most at SLAC. Neutron sources include to help improve the design for more challenging problems in 21st century the Spallation Neutron Source at efficient acceleration. High-resolution, accelerator science and technology. ORNL and the Los Alamos Neutron system-scale simulation utilizing The AST project had three focus Science Center at Los Alamos terascale computers such as the areas: computational beam dynam- National Laboratory in New Mexico. IBM SP supercomputer at NERSC, ics, computational electromagnetics, While accelerators can be of differ- has been made possible by SciDAC- and modeling advanced accelerator ent configurations with different supported code development efforts concepts. The newly developed capabilities, most use radio frequen- and collaborations with the SciDAC tools are now being used by accel- cies (RF) to accelerate the particles. ISICs. This team-oriented approach erator physicists and engineers In order to experimentally pursue the to computing is beginning to make across the country to solve the most quest for the grand unified theory a qualitative difference in the R&D challenging problems in accelerator ever more powerful accelerators are of major DOE accelerators, existing design, analysis, and optimization. needed. Such massive facilities are or planned. An accelerator generates and costly to build and require special- Beam Dynamics: Maximizing collects billions of elementary parti- ized infrastructure. cles, such as electrons, positrons or The long-term future of experi- Scientific Productivity protons, into a limited space, then mental high-energy physics research The AST project has had a accelerates these particles in a beam using accelerators depends on the major impact on computational toward a target — another beam of successful development of new beam dynamics and the design of particles or devices for producing acceleration methods which can particle accelerators. Thanks to radiation or other particles. In the accelerate particles to even higher SciDAC, accelerator design calcula- process of acceleration, the energy energy levels per acceleration length tions that were once thought of every particle in the beam is (known as the accelerating gradi- impossible are now carried out rou- increased tremendously. In order to ent). In the past, this was achieved tinely. SciDAC accelerator modeling advance elementary particle physics by building a longer accelerator, an codes are being used to get the into regions beyond our present approach which faces practical lim- most science out of existing facili- knowledge, accelerators with larger its in space and cost. ties, to produce optimal designs for final beam energies approaching the Since building such accelerators future facilities, and to explore tera electron volt (TeV) scale are is time-consuming and expensive, advanced accelerator concepts that required. scientists are using computer simu- may hold the key to qualitatively The Department of Energy lations to increasing our under- new ways of accelerating charged operates some of the world’s most standing of the physics involved and particle beams. Here are some high- powerful accelerators, including a three-kilometer-long linear accelera-

0.35 tor at the Stanford Linear Accelerator 0.30 0.25 0.20 y 0.15 –0.15 Center (SLAC) in California, the z 0.10 –0.10 0.05 –0.05 0.00 –0.10 –0.05 Tandem Linac Accelerator System –0.15 0.00 –0.20

–0.25 0.05 at Argonne National Laboratory –0.30 0.10 –0.35 0.15 (ANL) in Illinois, the Tevatron at 0.15 Fermilab in Illinois, the Relativistic 0.15 0.10 Heavy Ion Collider (RHIC) at 10 0.05 Brookhaven National Laboratory 0. (BNL) in New York, the Continuous 0.05 0.00 –0.05 x Electron Beam Accelerator Facility 0.00 x in Virginia and the Holifield Radio- –0.10 –0.05 active Ion Beam Facility at Oak –0.15 0

0.30. –0.10 3 0 5 Ridge National Laboratory (ORNL) 0.20. 2 0 5 0 . 1 –0.15 0 .1 in Tennessee. DOE-operated light 0. 5 . 0 0 –0.10 – 0 5 0 – 0 –0.05 0 – 0 . sources are the Advanced Light 0.0 05 – 5 0.15. y –0. 0 10 z 0.0 – .20 0.30 Source at Lawrence Berkeley 0.10 –0. 5 25 0.1 National Laboratory in California, 35 the Advanced Photon Source at ANL, the National Synchrotron Light Source at BNL and the Stanford FIGURE 16. This simulation of the Fermi Booster, a synchrotron accelerator ring, was created using Synchrotron Radiation Laboratory the Synergia software developed under SciDAC.

59 SOFTWARE FOR SCIENCE

lights from the AST project Beam and the proposed ILC and RIA. Dynamics focus area in regard to Central to these efforts is a suite algorithm development, software of 3D parallel electromagnetic design development, and applications. codes: Omega3P, Tau3P, Track3P, Particle simulation methods S3P and T3P. Collaborations with have been among the most success- SciDAC ISICs led to a number of ful and widely used methods in improvements in the codes which computational beam physics, plas- FIGURE 17. (Left) A model of RIA’s hybrid RFQ have not only increased the speed ma physics and astrophysics. Under and (Right) an enlarged portion of the corre- and accuracy of the codes up to sponding Omega3P mesh. the AST project a comprehensive, tenfold, but have expanded their state-of-the-art set of parallel parti- capabilities for solving more compli- cle-in-cell (PIC) capabilities has ISICs and with other DOE-support- cated problems of importance to been developed, including: ed activities. In some cases, the use present and future accelerators. The • IMPACT: An integrated suite of of advanced algorithms has led to codes have been used to advance a codes originally developed to applications running up to 100 number of DOE accelerator proj- model high intensity ion linacs, times faster. ects, including the ILC, PEP-II, IMPACT’s functionality has SciDAC AST beam dynamics NLC, RIA and LCSS. been greatly enhanced so that it codes have been applied to several For example, the AST electro- is now able to model high important projects within the DOE magnetic codes were applied to the brightness electron beam Office of Science. Examples include: PEP-II facility, which now operates dynamics, ion beam dynamics • existing colliders, including the with twice the luminosity of the and multi-species transport Tevatron, RHIC and PEP-II original design and is aiming for through a wide variety of sys- • future colliders such as the LHC another twofold increase. However, tems. currently under construction beam heating in the interaction region • BeamBeam3D: A code for mod- • proposed linear colliders such as (IR) could limit the accelerator from eling beam-beam effects in col- the International Linear Collider meeting that goal. The Tau3P code liders. This code contains multi- (ILC) has been used to calculate the heat ple models and multiple collision • high intensity machines, includ- load in the present design. Higher geometries and has been used to ing the Fermilab booster and the currents and shorter bunches of par- model the Tevatron, Positron- Spallation Neutron Source ticles will require modifications to Electron Project (PEP)-II, (SNS) ring under construction the IR to reduce the increased heat- Relativistic Heavy Ion Collider • linacs for radioactive ion beams, ing. T3P will be able to model the (RHIC), and Large Hadron such as the proposed Rare entire IR geometry and to simulate Collider (LHC) accelerators. Isotope Accelerator (RIA) the actual curved beam paths in any • MaryLie/IMPACT: A code that • electron linacs for fourth-genera- new IR design. combines the high-order optics tion light sources, such as the The proposed RIA, which is modeling capabilities of the Linac Coherent Light Source ranked third in the Office of Science’s MaryLie Lie algebraic beam now under construction. 20-year Science Facility Plan, requires transport code with the parallel design of a variety of radio frequency PIC capabilities of IMPACT. It is Electromagnetic Systems quadropole (RFQ) cavities (Figure 17) used to model space-charge Simulation: Prototyping for its low-frequency linacs. Due to effects in large circular machines through Computation lack of accurate predictions, tuners such as the ILC damping rings. are designed to cover frequency • Synergia: A parallel beam The AST project has supported deviations of about 1 percent. With dynamics simulation framework, a well-integrated, multi-institutional, Omega3P, frequency accuracy of 0.1 Synergia combines multiple multi-disciplinary team to focus on percent can be reached, significantly functionality, such as the space- the large-scale simulations necessary reducing the number of tuners and charge capabilities of IMPACT for the design and optimization of their tuning range. Parallel comput- and the high-order optics capa- electromagnetic systems essential to ing will play an important role in bilities of MXYZPLT, along with accelerator facilities. As a result, an prototyping RIA’s linacs and help a “humane” user interface and increasing number of challenging reduce their costs. standard problem description. design and analysis problems are being Crucial to the development of solved through large-scale simula- the AST project’s codes has been tions, benefiting such projects as the the collaboration with the SciDAC PEP-II, Next Linear Collider (NLC),

60 SOFTWARE FOR SCIENCE

Modeling Advanced Accelerators: sure creates a space-charge wake on acceleration while the code was Miniaturizing Accelerators which a trailing beam of particles can constructed rapidly using the UPIC from Kilometers to Meters surf. To model such devices accu- Framework. In some cases, QuickPIC rately usually requires following the can completely reproduce the results The long-term future of experi- trajectories of individual plasma par- from previous algorithms with a fac- mental high-energy physics research ticles. Therefore, the software tools tor of 50 to 500 savings in computer using accelerators is partly depend- developed fully or partially under this resources. It is being used to study ent on the successful development project — OSIRIS, VORPAL, OOPIC, beam-based science in regimes not of novel ultra high-gradient acceler- QuickPIC and UPIC — rely on the accessible before. ation methods. New acceleration PIC techniques. Rapidly adding realism into the techniques using lasers and plasmas Among the major accomplish- computer models: The large elec- have already been shown to exhibit ments are: tromagnetic fields from the intense gradients (or acceleration) and The development of UPIC: drive beams can field ionize a gas, focusing forces more than 1000 Using highly optimized legacy code, forming a plasma. Early SciDAC times greater than conventional a modern framework for writing all research using two-dimensional codes technology. The challenge is to con- types of parallelized PIC codes revealed that this self-consistent for- trol these high-gradient systems and including electrostatic, gyrokinetic, mation of the plasma from the drive then to string them together. Such electromagnetic, and quasi-static beam needs to be included in many technologies would not only enable has been developed. The UPIC cases. Via the SciDAC team approach, a cost-effective path to the outermost Framework has obtained 30 percent ionization models have been added reaches of the high-energy frontier, of peak speed on a single processor and benchmarked against each but could also lead to the develop- and 80 percent efficiency on well other in the fully three-dimensional ment of ultra-compact accelerators. over 1,000 processors. It is used PIC codes, VORPAL and OSIRIS. Such compact accelerators could across both the accelerator and Extending the plasma codes to benefit science, industry and medi- fusion SciDAC projects. model the electron-cloud instability: cine by shrinking large facilities to a The rapid construction of The electron cloud instability is one much reduced size and allowing QuickPIC: The development of of the major obstacles for obtaining them to be built nearer research QuickPIC is a success story for the the design luminosity in circular organizations, high-tech businesses rapid construction of a new code accelerators, storage rings and and medical centers. using reusable parallel code via a damping rings. A set of modules has Under the AST Project, the SciDAC team approach. The basic been written for QuickPIC that Advanced Accelerator effort has equations and algorithms were models circular orbits under the emphasized developing a suite of developed from a deep understand- influence of external focusing ele- fully parallel 3D electromagnetic ing of the underlying physics involved ments. This new software tool has PIC codes. The codes have been in plasma and/or laser wakefield already modeled 100,000 km of beam benchmarked against each other and their underlying algorithms, as well as against experiments. The resulting codes provide more realistic models, which are being applied 15 to advanced accelerators as well as more mainstream problems in 10 accelerator physics. Furthermore, the effort has included running 5 these codes to plan and interpret experiments and to study the key 0 15 30 physics that must be understood 40 10 before a 100+ GeV collider based 20 5 on plasma techniques can be ] p 10 designed and tested. ω 0 / 0 c In some advanced accelerator x1 [ concepts a drive beam, either an intense particle beam or laser pulse, is sent through a uniform plasma. The space charge or radiation pres- FIGURE 18. 3D afterburner simulation results.

61 SOFTWARE FOR SCIENCE

propagation of the SPS machine at CERN. It is a major improvement over previously existing tools for modeling electron cloud interactions. Applying the suite of codes to discover new accelerator science: The suite of PIC codes has been used to model particle beam accelerator experiments at the Stanford Linear Accelerator Center and the laser-plasma experiments at the L’OASIS lab at Lawrence Berkeley National Laboratory. They have also been used to study key physics satellite. But earth-bound supercom- a field that couples astrophysics, issues related to the afterburner puters are also proving to be power- particle physics, nuclear physics, concept, in which the energy of an ful tools for studying the forces behind turbulence theory, combustion theory existing beam from an accelerator is the most powerful explosions in the and radiation transport. The project doubled with gradients near 10 universe. To help understand the team also worked to develop and GeV/m. An example is shown in conditions which lead to supernovae modify the necessary computer codes Figure 18 where the beam and plas- explosions, the SciDAC program to incorporate the physics efficiently ma density is shown from a three- launched two projects — the Super- and do exploratory calculations. dimensional simulation for the after- nova Science Center and the Terascale Examples of these alliances are burner, including field ionization. Supernova Initiative. the chemical combustion groups at Lawrence Berkeley and Sandia Principal Investigators: Robert Ryne, The Supernova Science Center Lawrence Berkeley National Laboratory, National Laboratories; the NSF's and Kwok Ko, Stanford Linear Accelerator The Supernova Science Center Joint Institute for Nuclear Astro- Center (SNSC) was established with the physics (JINA); and radiation trans- objective of using numerical simula- port and nuclear physics experts at tions to gain a full understanding of Los Alamos and Lawrence Livermore Supernovae Science: Getting how supernovae of all types explode National Laboratories. With LBNL, the Inside Story on How Stars and how the elements have been the team applied codes previously created in nature. These computa- optimized to study chemical com- Explode tional results are then compared bustion on large parallel computers While supernovae are awesome with astronomical observations, to novel problems in nuclear comin their own right — these exploding including the abundance patterns of bustion in white dwarf stars, carry- stars expire in flashes brighter than elements seen in our own sun. ing out for the first time fully whole galaxies — they can also pro- The explosions both of massive resolved studies of the flame in both vide critical information about our stars as “core-collapse” supernovae the “flamlet” and “distributed” universe. Because of their regular and of white dwarfs as “thermonu- regimes. brightness, Type Ia supernovae are clear” supernovae (also called Type The project team worked with used as astronomical “standard can- Ia) pose problems in computational JINA to develop a standardized dles” for cosmic distances and the astrophysics that have challenged library of nuclear data for applica- expansion of the universe. Super- the community for decades. Core- tion to the study of nucleosynthesis novae are also helping scientists collapse supernovae require at least in stars, supernovae and X-ray gain a better understanding of the a two- (and ultimately three-) bursts on neutron stars. dark energy thought to make up 75 dimensional treatment of multi- The SNSC is also tapping the percent of the universe. And finally, energy-group neutrinos coupled to expertise of the SciDAC ISICSs. For supernovae are also the source of multi-dimensional hydrodynamics. example, the TOPS ISIC is working the heavy elements in our universe, Type Ia explosions are simulation with members of the team to pro- as well as prolific producers of neu- problems in turbulent (nuclear) duce more efficient solvers, with a trinos, the most elusive particles so combustion. goal of increasing the performance far discovered. During its first five years, the of the multi-dimensional codes by a The best perspective from SNSC focused on forging the inter- factor of 10. which to observe supernovae is disciplinary collaborations necessary Other SNSC achievements from a telescope mounted on a to address cutting-edge problems in include:

62 SOFTWARE FOR SCIENCE

• Carrying out the first 3D, full-star simulation of core collapse and explosion. The project also carried out simulations in 2D with multi-group neutrino transport of the collapse, bounce, and early post-bounce evolution. Recently a new supernova mechanism was discovered, powered by neutron star vibrations. • Developing and using 3D adaptive mesh relativistic hydrocodes to model the production of gamma-ray bursts in massive stars, which led to the prediction FIGURE 19. Snapshots of a thermonuclear supernova simulation with asymmetric ignition. After of a new kind of high energy igniting a sub-sonic thermonuclear flame slightly off-center of the white dwarf progenitor star, it transient – cosmological x-ray propagates towards the surface. The interaction of buoyancy and flame propagation changes the flashes, which were later discov- morphology of the initially spherical burning bubble (see 1st snapshot, where the flame corresponds to the blue isosurface and the white dwarf star is indicated by the volume rendering of the logarithm ered. (An image from this model of the density) to a toroidal shape perturbed by instabilities (2nd snapshot, cf. Zingale et al., SciDAC appears on the cover of this 2005 proceedings). The flame interacts with turbulence and is accelerated, but due to the asymmetric report.) ignition configuration, only a small part of the star is burned. Thus the white dwarf remains gravitationally bound and when the ash bubble reaches its surface it starts to sweep around it (3rd snap- • Carrying out the world's best shot), as also noticed by the ASC Flash center at the University of Chicago. The question addressed simulations of Type I X-ray it the present study was whether the collision of ash on the far side of the star's surface (4th snapshot) would compress fuel in this region strong enough to trigger a spontaneous detonation (“GCD scenario”, bursts on neutron stars. The Plewa et al., 2004). Three-dimensional simulations like those shown in the images disfavor this possibility. nuclear physics of these explosions is a primary science goal of The Terascale Supernova els for core collapse supernovae and DOE’s proposed Rare Isotope Initiative: Shedding New Light enabling technologies in radiation Accelerator. on Exploding Stars transport, radiation hydrodynamics, • Calculating the ignition of the nuclear structure, linear systems and nuclear runaway in a white Established to “shed new light eigenvalue solution and collabora- dwarf star becoming a Type Ia on exploding stars,” SciDAC’s tive visualization. These calcula- supernova, which showed that Terascale Supernova Initiative tions, when carried out on terascale the ignition occurs off-center. (TSI) is a multidisciplinary collabo- machines, will provide important Other research has shown that ration which will use modeling of theoretical insights and support for the resulting supernova depends integrated complex systems to search the large experimental efforts in critically upon whether it is for the explosion mechanism of high energy and nuclear physics. ignited centrally or at a point off core-collapse supernovae – one of A key focus of the TSI project is center. This issue of how the the most important and challenging understanding what causes the core white dwarf ignites has become problems in nuclear astrophysics. of a large star to collapse. When the the most debated and interesting The project team, which includes star runs out of fuel, pressures in the issue in Type Ia supernova mod- scientists, computer scientists and core build and fuse elements together, eling today. mathematicians, will develop mod- leading to accelerating combustion.

63 SOFTWARE FOR SCIENCE

Entropy 1.00 1.65 2.30 2.94 3.59 4.24 4.89 5.53 6.18

26.7

23.3

20.0

16.7

13.3

10.0

6.67

3.33

0.00

3.33

6.67

10.0

13.3

16.7

20.0

23.3

26.7 0.00 3.33 6.6710.0 13.3 16.7 20.0 23.3 26.7 (km)

FIGURE 21. A closeup look at a proto-neutron star model, shown at about 32 ms following FIGURE 20. This image illustrates the stable rotational flow found in recent 3D simulations of the core bounce, as calculated by the 2D radiation- SASI, computed on the Cray X1 at NCCS. The streamlines show the inward flow of the core being hydrodynamic code, V2D. The distance scale deflected by the nascent supernova shock and ultimately wrapping around the inner core. is given in kilometers. This view focuses on the inner 25 km of a model that extends nearly At the end, only iron is left at the showing the flow in a stellar explo- 6000 km into the oxygen shell. The figure is core and fusion stops. As the iron sion developing into a strong, stable, color mapped by entropy. Velocity direction is shown using an LEA texture map. Convective core grows, gravitational forces rotational flow (streamlines motion can be seen to encompass the entire eventually cause the core to wrapped around the inner core). inner core. However, even at this early stage implode, or collapse, leading to an The flow deposits enough angular of evolution, vigorous motion has ceased, leaving this region moving at only a few per- explosion. This explosion creates momentum on the inner core to cent of the sound speed. both heavy elements and neutrinos. produce a core spinning within a Once the core can no longer col- period of only a few milliseconds. lapse any more, it “bounces” back This new ingredient of core-col- rotational instability. Finally, the and sends a shock wave through lapse supernovae, which can only accretion of the rotational flow onto the external layers of the star. If the be modeled in full 3D simulations, the central core can spin up the shock wave kept going, it would may have a dramatic impact on both proto-neutron star to periods of lead to the supernova blast. But, for the supernova mechanism itself and order tens of milliseconds, consis- reasons still not understood, the on the neutron star left behind by tent with the inferred rotational shock wave stalls before it can trig- the supernova event. The strong period of radio pulsars at birth. ger the explosion in current simula- rotational flow below the supernova Other team members were able tions. TSI researchers are formulat- shock leads to an asymmetric densi- to carry out the most physically ing new ways to examine critical ty distribution that can dramatically detailed 2D core-collapse supernova events in the supernova core within alter the propagation of neutrinos, simulations to date using precondi- the first second after the core which may in turn enhance the effi- tioners, which are approximating bounces. One theory is that neutri- ciency of a neutrino-driven shock. techniques for speeding up linear no heating helps re-energize the This rotational flow will also pro- solvers. Earlier simulations used a stalled shock. vide a significant source of energy gray approximation, which assumes The TSI project created a series for amplifying any pre-existing mag- a shape of the energy distribution of 3D hydrodynamic simulations netic fields through the magneto- spectrum for neutrinos. In contrast,

64 SOFTWARE FOR SCIENCE

the TSI simulations using the V2D be significant for the core collapse at reaching these goals. Remarkable code computed the energy spectra supernova explosion problem, for progress has been made through the of the neutrinos, which change with associated nucleosynthesis and for a development of the Standard Model time and with position as the super- neutrino signal detection. Although of high energy and nuclear physics, nova evolves. That’s far more realistic, some experimental results indicate which provides fundamental theories but requires extensive computing that neutrinos would not be able to of the strong, electromagnetic and power. With SciDAC providing access transform to the point where they weak interactions. This progress has to terascale supercomputers, the could affect the shock wave, the been recognized through the award team was able to solve the problem. TSI team has mapped out scenarios of Nobel Prizes in Physics for the Using V2D, initial studies have in which the behavior of neutrinos development of each of the compo- shown a number of new results and could in principle affect the dynam- nents of the Standard Model: the confirmed some results reported by ics and nucleosynsthesis deep inside unified theory of weak and electro- others. These include a much earlier the supernova environment. magnetic interactions in 1979, and onset of post-bounce radiative-hydro- QCD, the theory of the strong dynamic instabilities and develop- National Computational interactions, in 1999 and 2004. ment of a hydrodynamically unstable However, our understanding of the region that quickly grows to engulf Infrastructure for Lattice Gauge Standard Model is incomplete the entire region inside the core- Theory: Accomplishments and because it has proven extremely dif- bounce shock wave, including both ficult to determine many of the neutrino-transparent regions as well Opportunities most interesting predictions of as the proto-neutron star where mate- The long-term goals of high QCD, those that involve the strong rial is opaque to neutrinos. The most energy and nuclear physicists are to coupling regime of the theory. To detailed simulations have proceeded identify the fundamental building do so requires large scale numerical to about 35 milliseconds following blocks of matter, and to determine simulations within the framework of core bounce, and the team is work- the interactions among them that lattice gauge theory. ing to extend these simulations to lead to the physical world we observe. The scientific objectives of lattice significant fractions of a second and The fundamental theory of the strong QCD simulations are to understand beyond. interactions between elementary the physical phenomena encom- Interaction between astrophysi- particles is known as quantum chro- passed by QCD, and to make pre- cists and nuclear physicists on the modynamics, or QCD. The strong cise calculations of the theory’s pre- TSI team resulted in the first stellar force is one of four fundamental dictions. Lattice QCD simulations core collapse simulations to imple- forces in nature, and provides the are necessary to solve fundamental ment a sophisticated model for the force to bind quarks to construct problems in high energy and nuclear stellar core nuclei. The increased protons and neutrons, which account physics that are at the heart of DOE’s sophistication in the nuclear models for about 98 percent of the matter large experimental efforts in these led to significant quantitative changes in nature. fields. Major goals of the experi- in the supernova models, demon- The objective of the SciDAC mental programs in high energy strating that both the microphysics National Computational Infrastruc- and nuclear physics on which lattice and the macrophysics of core col- ture for Lattice Gauge Theory QCD simulations can have an lapse supernovae must be computed project is to construct the computa- important impact are to: (1) verify with care. The significance of this tional infrastructure needed to study the Standard Model or discover its work led to the publication of two QCD. Nearly all high energy and limits; (2) understand the internal associated Physical Review Letters, nuclear physicists in the United structure of nucleons and other one focusing on the astrophysics States working on the numerical strongly interacting particles and (3) and one focusing on the nuclear study of QCD are involved in this determine the properties of strongly physics. The work has also motivated project, and the infrastructure created interacting matter under extreme nuclear experiments to measure will be available to all. The project conditions, such as those that existed nuclear cross-sections as a way to includes the development of com- immediately after the Big Bang and validate the nuclear models used in munity software for the effective use are produced today in relativistic supernova simulations. of terascale computers, and research heavy–ion experiments. Lattice In the last few years experiments and development of specialized QCD calculations are essential to and observations have revealed that computers for the study of QCD. research in all of these areas. neutrinos have mass and can change The Department of Energy sup- The numerical study of QCD their flavors (electron, muon and tau). ports major experimental, theoretical requires very large computational This development could turn out to and computational programs aimed resources, and has been recognized as

65 SOFTWARE FOR SCIENCE

one of the grand challenges of com- commodity clusters optimized for work has taken place at FNAL and putational science. The advent of QCD, and commercial supercom- JLab. The objective has been to pro- terascale computing, coupled with puters. Design goals included enabling vide computing platforms to test the recent improvements in the formula- users to quickly adapt codes to new QCD API, and to determine optimal tion of QCD on the lattice, provide architectures, easily develop new configurations for the terascale clus- an unprecedented opportunity to applications and preserve their large ters planned for FY 2006 and beyond. make major advances in QCD calcu- investment in existing codes. The clusters that have been built lations. The infrastructure created The QCD API is an example of are being used to carry out impor- under this SciDAC grant will play a an application-specific code base serv- tant research in QCD. The bottle- critical role in enabling the U.S. lattice ing a national research community. neck in QCD calculations on clusters, QCD community to take advantage It exploits the special features of QCD as on commercial supercomputers, of these opportunities. In particular, calculations that make them particu- is data movement, not CPU per- the hardware research and develop- larly well suited to massively parallel formance. QCD calculations take ment work provides the groundwork computers. All the fundamental com- place on four-dimensional space- for the construction of dedicated ponents have been implemented and time grids, or lattices. To update computers for the study of QCD, are in use on the QCDOC hardware variables on a lattice site, one only and the community software will at BNL, and on both the switched needs data from that site and a few enable highly efficient use of these and mesh architecture Pentium 4 neighboring ones. The standard computers and the custom designed clusters at Fermi National Accelerator strategy is to assign identical sub- 12,228-node QCDOC computer Labor-atory (FNAL) and Thomas lattices to each processor. Then, one recently constructed at Brookhaven Jefferson National Accelerator can update lattice points on the National Laboratory (BNL). Facility (JLab). The software code interior of the sub-lattices, for which Software Development and documentation are publicly all the relevant neighbors are on the available via the Internet. same processor, while data is being The project’s software effort has Hardware Research and collected from a few neighboring created a unified programming envi- processors to update lattice sites on Development ronment, the QCD Application the sub-lattice boundaries. This Programming Interface (QCD API), The second major activity under strategy leads to perfect load bal- that enables members of the U.S. the Lattice QCD SciDAC grant has ancing among the processors, and, lattice gauge theory community to been the design, construction and if the computer and code are prop- achieve high efficiency on terascale operation of commodity clusters opti- erly tuned, to overlap of computa- computers, including the QCDOC, mized for the study of QCD. This tion and communications.

FIGURE 22. Under SciDAC’s Lattice QCD project, two computer clusters were developed for use by the lattice gauge theory community. The system on the left is the 6n cluster at DOE’s Jefferson Lab. At the right is the QCDOC computer, constructed at Brookhaven National Laboratory by a group centered at Columbia University. The RBRC computer on the right side of the photo was funded by the Riken Research Institute in Japan. Together, the two systems have a peak speed of 20 teraflop/s.

66 SCIENTIFIC SOFTWARE INFRASTRUCTURE

try to approach the problems from different perspectives to create Building a Better broadly applicable solutions. In many cases, computing tools originally developed and refined for one type or problem were found to be Software Infrastructure useful in solving problems in other scientific areas, too. As a result of these focused efforts, scientists can for Scientific Computing now draw on a wide range of tools and techniques to ensure their codes perform efficiently on super- Infrastructure usually brings to mind images of physical struc- computers of varying sizes, with different architectures and at various tures such as roads, bridges, power grids and utilities. But computing centers. computers also rely on an infrastructure of processors, soft- An important emphasis of SciDAC is that the ISIC teams ware, interconnects, memory and storage, all working together worked with other projects to to process data and accomplish specific goals. While personal enable the effective use of terascale systems by SciDAC applications. computers arrive ready to use with easy-to-install software, Specifically, the ISICS addressed scientists who rely on supercomputers have typically devel- the need for: • new algorithms which scale to oped their own software, often building on codes that were parallel systems having thou- developed 10 or 20 years ago. sands of processors • methodology for achieving As supercomputers become bogs down. Because these “legacy portability and interoperability more powerful and faster, computa- codes” often represent significant of complex high performance tional scientists are seeking to adapt intellectual investment aimed at scientific software packages their codes to utilize this computing solving very specific problems, sci- • operating systems tools and sup- power to develop ever more entists are reluctant to start anew in port for the effective manage- detailed simulations of complex developing new codes. Finally, even ment of terascale and beyond problems. Other scientists are using when applications can be adapted systems, and supercomputing centers to store, to run more efficiently on one com- • effective tools for feature identifi- analyze and share data from large puter architecture, they may not cation, data management and experimental facilities, such as parti- achieve similar performance on visualization of petabyte-scale cle colliders or telescopes searching another type of supercomputer, an scientific datasets. deep space for supernovae. issue of growing importance as But scaling up applications to supercomputing centers increasingly The result is a comprehensive, run on the newest terascale super- share their computers to meet the integrated, scalable, and robust high computers, which can perform tens growing demand for greater com- performance software infrastructure of trillions of calculations per sec- puting capability. Understanding for developing scientific applica- ond, presents numerous challenges. computer performance and devel- tions, improving the performance of First, many of these scientific appli- oping tools to enhance this perform- existing applications on different cations were written by researchers ance enables scientists to scale up supercomputers and giving scien- over the course of years or decades their codes to study problems in tists new techniques for sharing and for older computers with tens or greater detail for better understanding. analyzing data. hundreds of processors and may To help bridge the gap between not be easily adapted to run on these legacy codes and terascale Applied Mathematics thousands or tens of thousands of computing systems, SciDAC estab- processors. Second, in some cases, lished seven Integrated Software ISICs scaling an existing application to Infrastructure Centers, or ISICs, run on more processors can actually which brought together experts Three applied mathematics slow down the run time as the from government computing cen- ISICs were created to develop algo- communication between processors ters, scientific disciplines and indus- rithms, methods and mathematical

67 SCIENTIFIC SOFTWARE INFRASTRUCTURE

libraries that are fully scalable to project has been to develop a high namic (MHD) modeling for fusion many thousands of processors with performance algorithmic and soft- energy reactors called tokamaks, full performance portability. ware framework for multiscale and modeling particle accelerators. problems based on the use of block- In combustion, the center devel- structured adaptive mesh refine- oped new, more efficient simulation An Algorithmic and Software ment (AMR) for representing multi- methods for solving chemically Framework for Applied PDEs ple scales. In this approach, the reacting fluid flow problems. These (APDEC) physical variables are discretized on algorithms and software have been a spatial grid consisting of nested used to study a variety of problems Many important DOE applica- rectangles of varying resolution, in the combustion of hydrocarbon tions can be described mathemati- organized into blocks. This hierar- fuels, as well as providing the start- cally as solutions to partial differen- chical discretization of space can ing point for the development of a tial equations (PDEs) at multiple adapt to changes in the solution to new capability for simulating scales. For example, combustion for maintain a uniform level of accuracy nuclear burning in Type 1a super- energy production and transporta- throughout the simulation, leading novae. tion is dominated by the interaction to a reduction in the time-to-solu- In the area of fusion research, of fluid dynamics and chemistry in tion by orders of magnitude com- the center developed AMR codes localized flame fronts. Fueling of pared to traditional fixed-grid calcu- to investigate a variety of scientific magnetic fusion devices involves the lations with the same accuracy. In problems, including tokamak fueling dispersion of material from small short, AMR serves as a computa- (Figure 1) and magnetic reconnec- injected fuel pellets. The successful tional microscope, allowing scien- tion, which can help in the design design of high-intensity particle tists to focus their computing of fusion reactors for future energy accelerators relies on the ability to resources on the most interesting production. accurately predict the space-charge aspects of a problem. The resulting In the area of accelerator mod- fields of localized beams in order to algorithms have enabled important eling, two new capabilities were control the beams, preserve the scientific discoveries in a number of developed: an AMR-particle-in-cell beam emittance and minimize parti- disciplines. (AMR-PIC) algorithm that has been cle loss. The APDEC center has been incorporated into the MLI and The goal of the Algorithmic involved with the development of Impact beam dynamics codes and Software Framework for applications in the areas of combus- (Figure 2); and a prototype capability Applied PDEs (APDEC) ISIC tion simulations, magnetohydrody- for use in simulating gas dynamics

FIGURE 1. AMR simulation of a fuel pellet being injected into a FIGURE 2. AMR particle-in-cell calculation, showing the electrostatic field induced by tokamak. Image shows the pressure along a slice as well as the particles (in collaboration with the SciDAC AST project). the outlines of the various refined grid patches (in collaboration with the SciDAC CEMM project).

68 SCIENTIFIC SOFTWARE INFRASTRUCTURE

can be calculated much more quickly. A preliminary implementation of this algorithm has obtained 85 percent efficiency on 1,024 processors, with less than 4 percent of the execution time spent in communications. The APDEC adaptive calculation used 3 billion grid points, as compared to the 2 trillion grid points that would have been required using a uniform grid algorithm at the same resolution, leading to a reduction in computing time of at least two orders of magnitude. This solver is the basis for FIGURE 3. Axisymmetric gas jet expanding into a vacuum, with the axis of symmetry along the bottom of the figure, in a laser-driven plasma-wakefield accelerator (in collaboration with the SciDAC the AMR-PIC method developed AST project). for the beam dynamics codes men- tioned above. problems in jets and capillary tubes For AMR algorithms for simple Principal Investigator: Phil Colella, Lawrence Berkeley National Laboratory arising in the design of plasma- compressible flow models, the wakefield accelerators (Figure 3). APDEC project has measured 75 To manage these and other percent efficiency up to 1,024 complex AMR algorithms and soft- processors on IBM SP and HP Terascale Optimal PDE Solvers ware, the APDEC team used a col- Alpha systems. For incompressible (TOPS) ISIC lection of libraries written in a com- flow, they have observed 75 percent In many areas of science, physi- bination of programming languages. efficiency up to 256 processors. For cal experimentation is impossible, This software architecture, which low-Mach-number combustion, such as with cosmology; dangerous, maps the mathematical structure of production calculations on IBM SP as with manipulating the climate; or the algorithm space onto a hierar- and SGI systems are typically done simply expensive, as with fusion chy of software layers, enables the using 128–512 processors, and the reactor design. Large-scale simula- codes to be more easily adapted efficiency of the APDEC algorithms tions, validated by comparison with and reused across multiple applica- has enabled significant scientific related experiments in well-under- tions. The project team has also achievements in combustion and stood laboratory contexts, are used developed new grid-generation astrophysics problems. by scientists to gain insight and tools for calculations using mathe- For many problems, such as confirmation of existing theories in matical representations from combustion applications, a principal such areas, without benefit of full microscopy, medical image data barrier to scaling past a few hun- experimental verification. But today’s and geophysical data. dred processors is the performance high-end computers, such as those Since the APDEC was estab- of iterative methods for solving at DOE’s supercomputing centers, lished, a guiding principle has been elliptic equations. To address this are one-of-a-kind, and come with- to develop a software infrastructure problem, the APDEC Center is out all of the scientific software that performs as efficiently as possi- developing a new class of AMR libraries that scientists expect to find ble. By creating algorithms to effi- solvers in three dimensions. The on desktop workstations. ciently use the number of available goal is to create a solver with a far The Terascale Optimal PDE processors and combining these lower communications-to-computa- Solvers (TOPS) ISIC was created with tools for load balancing, the tion ratio than traditional iterative to develop and implement algo- result is software which has a com- methods, which means the solution putational cost per grid point and scalability comparable to that of uniform-grid calculations using the same algorithms, giving scientists more detailed results without requiring additional computing resources. FIGURE 4. Model of the ILC low-loss cavity

69 SCIENTIFIC SOFTWARE INFRASTRUCTURE

rithms and support scientific investi- problems. One well-known example Finally, TOPS software is taught gations performed by DOE-spon- of this approach is called the multi- in many courses in the U.S. and sored researchers. These simulations grid method, which solves a prob- abroad, and forms a core of the often involve the solution of partial lem by tackling a coarse approxima- annual training workshop spon- differential equations (PDEs) on tion and then using the solution to sored by DOE to introduce terascale computers. The TOPS generate the solution of a better researchers to the Advanced Center researched, developed and approximation, and so on. It can be CompuTational Software (ACTS) deployed an integrated toolkit of shown that the performance of Collection. TOPS software is also open-source, optimal complexity multigrid method is optimal for cer- regularly featured in short courses solvers for the nonlinear partial dif- tain classes of problems. The under- at professional meetings. ferential equations that arise in many lying philosophy of this and other Principal Investigator: David Keyes, DOE application areas, including similar approaches is to make the Cornell University fusion, accelerator design, global cli- majority of progress towards a high mate change and reactive chemistry. quality result through less complex The algorithms created as part of intermediate steps. The Terascale Simulation Tools this project were also designed to The efforts defined for TOPS reduce current computational bot- and its collaborations with other and Technology (TSTT) Center tlenecks by orders of magnitude on projects have been chosen to revo- Terascale computing provides terascale computers, enabling scien- lutionize large-scale simulation an unprecedented opportunity to tific simulation on a scale heretofore through incorporation of existing achieve numerical simulations at impossible. and new optimal algorithms and levels of detail and accuracy previ- Nonlinear PDEs give mathemat- code interoperability. TOPS pro- ously unattainable. DOE scientists ical expression to many core DOE vides support for the software pack- in many different application areas mission applications. PDE simula- ages Hypre, PARPACK, PETSc, can reach new levels of understand- tion codes require implicit solvers ScaLAPACK, Sundials, SuperLU ing through the use of high-fidelity for the multirate, multiscale, multicom- and TAO, some of which are in the calculations based on multiple cou- ponent, multiphysics phenomena of hands of thousands of users, who pled physical processes and multiple hydrodynamics, electromagnetism, have created a valuable experience interacting physical scales. The best chemical reaction and radiation base on thousands of different com- way to achieve detailed and accurate transport. Currently, such problems puter systems. simulations in many application typically reach into the tens of mil- Software developed and sup- areas, and frequently the only way to lions of unknowns — and this size is ported by the TOPS project is being obtain useful answers, is to use adap- expected to increase 100-fold in just used by scientists around the globe. tive, composite, hybrid approaches. a few years. In the past few years, researchers However, the best strategy for a Unfortunately, the algorithms outside the SciDAC community particular simulation is not always traditionally used to solve PDEs authored more than two dozen sci- clear and the only way to find the were designed to address smaller entific papers reporting results best method is to experiment with problems and become much less achieved using TOPS software. The various options. This is both time- efficient as the size of the system papers cover many disciplines, from consuming and difficult due to the being studied increases. This creates astronomy to chemistry to materials lack of easy-to-apply, interoperable a double jeopardy for applications, science to nanotechnology to optics. meshing, discretization and adaptive particularly in the case when the TOPS solver software has also technologies. algorithms are based on iterations — been incorporated into numerous The Terascale Simulation Tools as it takes more computing resources other packages, some commercial and Technologies (TSTT) Center to solve each step of the problem, and some freely available. Among was established to address the tech- and the number of steps also goes the widely distributed packages nical and human barriers preventing up. Fortunately, the physical origin maintained outside of the SciDAC the effective use of powerful adap- of PDE problems often allows them program that employ or interface to tive, composite, and hybrid meth- to be addressed by using a sequence TOPS software “under the hood” ods. The primary objective was to of approximations, each of which is are Dspice, EMSolve, FEMLAB, develop technologies that enable smaller than the one before. The FIDAP, Global Arrays, HP application scientists to easily use solutions to the approximations, Mathematical Library, libMesh, multiple mesh and discretization which may be obtained more effi- Magpar, Mathematica, NIKE, strategies within a single simulation ciently, are combined judiciously to Prometheus, SCIRun, SLEPc, Snark on terascale computers. A key provide the solution to the original and Trilinos. aspect of the project is to develop

70 SCIENTIFIC SOFTWARE INFRASTRUCTURE

services that can be used interopera- and the TOPS ISIC to create an adaptive mesh refinement with front bly to enable mesh generation for automatic design optimization loop tracking and used this technology representing complex and possibly to provide automatic tuning of accel- to develop a new capability for evolving domains, high-order dis- erator geometries to significantly diesel engine design. This effort has cretization techniques for improved increase the speed and decrease the emphasized the modeling of the numerical solutions, and adaptive cost by which new accelerators can instability and breakup of a diesel strategies for automatically optimiz- be designed. jet into spray, thought to be the first ing the mesh to follow moving In the field of plasma physics, such effort to provide a predictive fronts or to capture important solu- TSTT researchers helped the Center capability in this arena. tion features. for Extended MHD Modeling Principal Investigators: Jim Glimm, State The center encapsulated its (CEMM) solve a number of prob- University of New York — Stony Brook; research into software components lems, particularly the case of aniso- Lori Freitag Diachin, Lawrence Livermore with well-defined interfaces that tropic diffusion, after determining National Laboratory enable different mesh types, dis- that using fewer higher-order ele- cretization strategies and adaptive ments significantly decreased the techniques to interoperate in a “plug total solution time needed to obtain Computer Science ISICs and play” fashion. All software was a given accuracy. This demonstra- designed for terascale computing tion resulted in a new effort by sci- The software infrastructure environments with particular entists at Princeton Plasma Physics vision of SciDAC is for a compre- emphasis on scalable algorithms for Laboratory (PPPL) to develop fifth- hensive, portable, and fully integrated hybrid, adaptive computations and order finite elements for their pri- suite of systems software and tools single processor performance opti- mary application code, M3D. TSTT for the effective management and mization. researchers also worked with scien- utilization of terascale computation- To ensure the relevance of the tists at PPPL to insert adaptive al resources by SciDAC applica- project’s research and software devel- mesh refinement technologies and tions. The following four Computer opments to the SciDAC goals, the error estimators directly into the Science ISIC activities addressed TSTT team collaborated closely with new high-order M3D code. critical issues in high performance both SciDAC application researchers TSTT researchers are collabo- component software technology, and other ISICs. Specifically, TSTT rating with climate modeling scien- large-scale scientific data manage- technologies were integrated into tists to develop enhanced mesh ment, understanding application/ fusion, accelerator design, climate generation and discretization capa- architecture relationships for modeling and biology applications to bilities for anisotropic planar and improved sustained performance, both provide near-term benefits for geodesic surface meshes. The initial and scalable system software tools those efforts as well as receive the mesh is adapted and optimized to for improved management and utili- feedback for further improvements. capture land surface orographic or ty of systems with thousands of Working with the Accelerator topographic height fields. This tech- processors. Modeling project, TSTT researchers nology has improved the predica- generated high-quality meshes from tion of rainfall, snowfall and cloud CAD (computer-aided design) mod- cover in regional weather models in Center for Component els for accurate modeling of the prototype simulations. Technology for Terascale interaction region of the PEP II TSTT researchers are collabo- accelerator. This led to the first suc- rating with computational biologists Simulation Software cessful simulation with a transit to develop the Virtual Microbial The Center for Component beam, using the Tau3P accelerator Cell Simulator (VMCS), which is Technology for Terascale Simu- modeling application. This success targeting DOE bioremediation lation Software (CCTTSS) was led to a decision to continue to use problems to clean up heavy metal created to help accelerate computa- Tau3P for PEP-II computations for waste using microbes. Methods tional science by bringing a “plug reaching higher luminosities. Similar developed by TSTT were used to and play” style of programming to mesh generation efforts by TSTT study how microbe communities high performance computing. researchers were used to analyze aggregate in certain environments, Through a programming model the wakefield in the advanced providing new insight into the called the Common Component Damped Detuned Structure and behavior of these microbes. Architecture (CCA), scientists can verify important performance char- In partnership with combustion dramatically reduce the time and acteristics of the system. scientists, TSTT researchers have effort required to compose inde- TSTT also partnered with SLAC developed a new tool that combines pendently created software libraries

71 SCIENTIFIC SOFTWARE INFRASTRUCTURE

into new terascale applications. This ous service capabilities, tools for System Performance: Science and approach has already been adopted mesh management, discretization, Engineering ISIC, also known as by researchers in the application linear algebra, integration, optimiza- the Performance ISIC or PERC, areas of combustion, quantum tion, parallel data description, paral- focused on developing tools and chemistry and climate modeling, lel data redistribution, visualization, techniques to help scientists deter- with new efforts beginning in fusion and performance evaluation. The mine how they can best execute a and nanoscale simulations. CCTTSS is also collaborating with specific application on a given com- While the CCA approach has the APDEC, TSTT and TOPS puter platform. The research was been more common in industry, the SciDAC centers to define common aimed at determining what level of goal of the CCTTSS was to bring a interfaces for mesh-based scientific achievable performance is realistic, similar approach to scientific com- data management as well as linear, how scientific applications can be puting. Unlike more narrowly nonlinear, and optimization solvers. accelerated toward these levels, and focused commercial applications, SciDAC funding has accelerated how this information can drive the the scientific CCA effort faced such both CCA technology development design of future applications and challenges as maintaining high per- and the insertion of this technology high-performance computing systems. formance, working with a broad into massively parallel scientific appli- In addition to helping scientists spectrum of scientific programming cations, including major SciDAC better understand performance languages and computer architec- applications in quantum chemistry behavior of their codes, the major tures, and preserving DOE invest- and combustion. Additionally, numer- improvements to computer perform- ments in legacy codes. ous CCA-compliant application ance modeling technology made by An example of the CCA components have been developed PERC researchers are also helping approach in action is a prototype and deployed, and the underlying supercomputer centers to better application being developed within infrastructure is maturing and estab- select systems, thereby ensuring the Community Climate System lishing itself in the scientific com- higher scientific productivity on these Model (CCSM) project. The CCA munity. publicly funded systems. Such super- is used at the level of system inte- Principal Investigator: Rob Armstrong, computer procurements run to mil- gration to connect skeleton compo- Sandia National Laboratories lions of dollars, and PERC research nents for the atmosphere, ocean, sea is being used by centers operated by ice, land surface, river routing, and DOE, the National Science Found- flux coupler. Prototype applications High-End Computer System ation and the Department of Defense within the Earth System Modeling Performance: Science and around the country. PERC researchers Framework (ESMF), the infrastruc- have also collaborated with several ture targeted for the future CCSM, Engineering ISIC SciDAC scientific application efforts, also employ the CCA, including As supercomputers become ever resulting is major speedups for sev- reusable components for visualiza- more powerful tools of scientific dis- eral high-profile computer codes tion and connectivity. covery, demand for access to such used in these projects. In addition, CCTTSS researchers resources also increases. As a result, For the Community Climate are collaborating closely with appli- many scientists receive less time on System Model (CCSM), one of the cation scientists to create high- supercomputers than they would like. United States’ primary applications performance simulations in quan- Therefore, it’s critical that they make for studying climate change, PERC tum chemistry and combustion. the most of their time allocations. researchers helped determine opti- Moreover, new externally funded One way to achieve this optimal mal algorithmic settings for the projects incorporate CCA concepts efficiency is to analyze the perform- Community Atmosphere Model in applications involving nanotech- ance of both scientific codes and (CAM), a key component of the nology, fusion, and underground the computer architectures on CCSM, when running the Intergov- transport modeling, and proposals which the codes are run. The result- ernmental Panel on Climate Change have been recently submitted ing improvements can be dramatic. scenario runs on the IBM p690 involving biotechnology and fusion. In one instance, an astrophysics code cluster computer at ORNL, acceler- As a part of its mission, the which had been running at about 10 ating the completion of this mile- CCTTSS has developed production percent of a supercomputer’s theo- stone. Similar studies are ongoing components that are used in scien- retical peak speed (which is typical) on the SGI Altix, Cray XD1, and tific applications as well as proto- was able to run at more than 50 Cray XT3 supercomputers. The type components that aid in teach- percent efficiency after careful per- resulting data from these runs will ing CCA concepts. These freely formance analysis and tuning. contribute to an international report available components include vari- SciDAC’s High-End Computer on global climate change.

72 SCIENTIFIC SOFTWARE INFRASTRUCTURE

Working with the Plasma Omega3P, where most of this time try and collectively defining stan- Microturbulence Project (PMP), is due to a complex and redundant dardized interfaces between system PERC researchers applied Active computation. By exchanging this components. At the same time, this Harmony, a software tool support- redundant computation for cached group is producing a fully integrated ing distributed execution of compu- lookup to pre-computed data, suite of systems software components tational objects, to GS2, a gyroki- researchers were able to achieve 4.5 that can be used by the nation’s netic turbulence simulator used by times speedups of the post-process- largest scientific computing centers. the fusion energy research commu- ing phase, and 1.5 time speedups of The Scalable Systems Software nity. The result was a 2.3 to 3.4 a full Omega3P run. suite is being designed to support times speedup of GS2 for a com- In addition to working directly computers that scale to very large mon configuration used in produc- with scientists on the performance of physical sizes without requiring that tion runs. codes, PERC researchers have made the number of support staff scale PERC researchers worked with major improvements in the usability along with the machine. But this the Terascale Supernovae Initiative and effectiveness of several perform- research goes beyond just creating a (TSI) project to port and optimize ance tools, enabling researchers to collection of separate scalable com- the EVH1 hydrodynamics code on analyze their codes more easily. The ponents. By defining a software the Cray X1 supercomputer, achiev- result is an integrated suite of meas- architecture and the interfaces ing excellent performance for large urement, analysis and optimization between system components, the problems. The EVH1 performance tools to simplify the collection and Scalable Systems Software research is analysis was completed for up to analysis of performance data and creating an interoperable framework 256 processors on all current target help users in optimizing their codes. for the components. This makes it platforms. Principal Investigator: David Bailey, much easier and cost effective for Working with two other ISICs, Lawrence Berkeley National Laboratory supercomputer centers to adapt, the Terascale PDE Simulations update, and maintain the components (TOPS) and Terascale Simulation in order to keep up with new hard- Tools and Technology (TSTT) Scalable Systems Software for ware and software. A well-defined centers, PERC researchers analyzed interface allows a site to replace or the performance of a TOPS-TSTT Terascale Computer Centers customize individual components as mesh smoothing application and As terascale supercomputers needed. Defining the interfaces found that the sparse matrix-vector become the standard environment between components across the multiply achieves 90 percent of the for scientific computing, the nation’s entire system software architecture peak performance imposed by the premiere scientific computing cen- provides an integrating force between memory bandwidth limit. ters are at a crossroads. Having built the system components as a whole PERC researchers collaborated the operations of their systems and improves the long-term usability closely with SciDAC’s Lattice Gauge around home-grown systems soft- and manageability of terascale sys- Theory project, conducting per- ware, system administrators and tems at supercomputer centers formance analyses of the MILC managers now face the prospect of across the country. (MIMD Lattice Computation) rewriting their software, as these The Scalable Systems Software application on Pentium-3 Linux incompatible, ad hoc sets of systems project is a catalyst for fundamentally clusters. The PERC team collected tools were not designed to scale to changing the way future high-end detailed performance data on differ- the multi-teraflop systems that are systems software is developed and ent aspects of the code and identi- being installed in these centers distributed. The project is expected fied the key fragments that affect today. One solution would be for to reduce facility management costs the code performance, then present- each computer center to take its by reducing the need to support ed their findings to the lattice gauge home-grown software and rewrite it home-grown software while making theory community. to be scalable. But this would incur higher quality systems tools avail- In the Accelerator Science and a tremendous duplication of effort able. The project will also facilitate Technology program, the PERC and delay the availability of terascale more effective use of machines by team has shown how to improve computers for scientific discovery. scientific applications by providing the performance of the post-pro- The goal of the SciDAC’s scalable job launch, standardized job cessing phase within the Omega3P Scalable Systems Software project monitoring and management soft- application at the Stanford Linear is to provide a more timely and ware, and allocation tools for the Accelerator Center. Post-processing cost-effective solution by pulling cost-effective management and uti- can consume 40 percent or more of together representatives from the lization of terascale and petascale total execution time in a full run of major computer centers and indus- computer resources.

73 SCIENTIFIC SOFTWARE INFRASTRUCTURE

than 3,000/month) now use the system software interfaces developed in this ISIC. Prominent users Resource & Queue include 75 of the world’s top 100 Allocation Management Management supercomputers and commercial Accounting & User industries such as Amazon.com and Management Ford Motor Co. Principal Investigator: Al Geist, Oak Ridge Fault Tolerance National Laboratory

Security Checkpoint Restart The Scientific Data Management Center System Scientific exploration and dis- Monitoring covery typically takes place in two System Build & Configure phases: generating or collecting data and then analyzing the data. In the Job Management data collection/generation phase, large volumes of data are generated by simulation programs running on supercomputers or collected from FIGURE 5. The mission of the Scalable Systems Software center is the development of an integrated suite of systems software and tools for the effective management and utilization of terascale com- experiments. As experimental facili- putational resources, particularly those at the DOE facilities. ties and computers have become larger and more powerful, the amount Here are some of the highlights released at the SC2005 conference of resulting data threatens to over- of the project: — the leading international meeting whelm scientists. Without effective Designed modular architecture: of the supercomputing community tools for data collection and analysis, A critical component of the pro- — and will be followed by quarterly scientists can find themselves ject’s software design is its modular- updates for the following year. spending more time on data man- ity. The ability to plug and play Production users: Full system agement than on scientific discovery. components is important because of software suites have been put into The Scientific Data Manage- the diversity in both high-end sys- production on clusters at Ames ment Center (SDM) was estab- tems and the individual site man- Laboratory and Argonne National lished to focus on the application of agement policies. Laboratory. Pacific Northwest known and emerging data manage- Defined XML-based interfaces National Laboratory and the ment technologies to scientific that are independent of language National Center for Supercomputer applications. The center’s goals are and wire protocol: The interfaces Applications have adopted one or to integrate and deploy software- between all the components have more components from the suite, based solutions to the efficient and been fully documented and made and the project team is in discussion effective management of large vol- publicly available, thereby allowing with Department of Defense sites umes of data generated by scientific others to write replacement compo- about use of some of the system applications. The purpose is not only nents (or wrap their existing com- software components. to achieve efficient storage and access ponents) as needed. The reference Adoption of application pro- to the data using specialized index- implementation has a mixture of gramming interface (API): The ing, compression, and parallel storage components written in a variety of suite’s scheduler component is the and access technology, but also to common programming languages, widely used Maui Scheduler. The enhance the effective use of the sci- which allows a great deal of flexibil- public Maui release (as well as the entists’ time by eliminating unpro- ity to the component author and commercial Moab scheduler) has ductive simulations, by providing allows the same interface to work been updated to use the public specialized data-mining techniques, on a wide range of hardware archi- XML interfaces and has added new by streamlining time-consuming tectures and facility polices. capabilities for fairness, higher sys- tasks, and by automating the scien- Reference implementation tem utilization, and improved tists’ workflows. released: Version 1.0 of the fully response time. All new Maui and The center’s approach is to pro- integrated systems software suite was Moab installations worldwide (more vide an integrated scientific data

74 SCIENTIFIC SOFTWARE INFRASTRUCTURE

management framework where learned from that information. assembly of modules using process components can be chosen by sci- Because of the large volume of data, automation technologies. entists and applied to their specific it is also useful to perform analysis as Since it was established, SDM domains. By overcoming the data the data are generated. For example, has adopted, improved and applied management bottlenecks and a scientist running a thousand-time- various data management technolo- unnecessary information-technology step, 3D simulation can benefit from gies to several scientific application overhead through the use of this analyzing the data generated by the areas. By working with scientists, integrated framework, scientists are individual steps in order to steer the the SDM team not only learned the freed to concentrate on their sci- simulation, saving unnecessary com- important aspects of the data man- ence and achieve new scientific putation, and accelerating the discov- agement problems from the scien- insights. ery process. However, this requires tists’ point of view, but also provid- Today’s computer simulations sophisticated workflow tools, as ed solutions that led to actual and large-scale experiments can well as efficient dataflow capabilities results. The successful results generate data at the terabyte level, to move large volumes of data achieved so far include: equivalent to about 5 percent of all between the analysis components. • More than a tenfold speedup in the books in the Library of Congress. To enable this, the center uses an writing and reading NetCDF Keeping up with such a torrent of integrated framework that provides files was achieved by developing data equires efficient parallel data a scientific workflow capability, sup- Parallel NetCDF software on systems. In order to make use of ports data mining and analysis tools, top of the MPI-IO. such amounts of data, it is necessary and accelerates storage access and • An improved version of PVFS is to have efficient indexes and effective data searching. This framework now offered by cluster vendors, analysis tools to find and focus on facilitates hiding the details of the including Dell, HP, Atipa, and the information that can be extracted underlying parallel and indexing Cray. from the data, and the knowledge technology, and streamlining the • A method for the correct classification of orbits in puncture plots from the National Compact Stellarator eXperiment (NCSX) at PPPL was developed by converting the data into polar coordinates and fitting second-order polynomials to the data points. • A new specialized method for indexing using bitmaps was used to provide flexible efficient search over billions of collisions (events) in high energy physics applications. A paper on the system received best paper award at the 2004 International Supercomputing Conference. • The development of a parallel version of the popular statistical package R is being applied to fusion, geographic information systems and proteomics (biochemistry) applications. • A scientific workflow system was developed and applied to the analysis of microarray data, as well as astrophysics applications. The system greatly increased the efficiency of run-

FIGURE 6. The STAR Experiment at Brookhaven National Laboratory generates petabytes of data ning repetitive processes. resulting from the collisions of millions of particles. The Scientific Data Management ISIC developed software to make it easier for scientists to transfer, search and analyze the data from such Principal Investigator: Arie Shoshani, large-scale experiments. Lawrence Berkeley National Laboratory

75 ADVANCED COLLABORATIVE INFRASTRUCTURE

structure linking various systems with unique security and access proce- Creating an Advanced dures, sometimes across international boundaries, is a daunting task. Grid technology is evolving to provide the services and infrastruc- Infrastructure for ture needed for building “virtual” systems and organizations. A Grid- based infrastructure provides a way Increased Collaboration to use and manage widely distributed computing and data resources to support science. The result is a A hallmark of many DOE research efforts is collaboration standard, large-scale computing, data, instrument, and collaboration among scientists from different disciplines. The multidiscipli- environment for science that spans nary approach is the driving force behind the “big science” many different projects, institutions, and countries. breakthroughs achieved at DOE national laboratories and other research institutions. By creating teams comprising National Collaboratory physicists, chemists, engineers, applied mathematicians, biolo- Projects gists, earth scientists, computer scientists and others, even the most challenging scientific problems can be defined, analyzed DOE’s investment in National Collaboratories includes four and addressed more effectively than if tackled by one or two SciDAC projects focused on the scientists working on their own. goal of creating collaboratory software environments to enable geographically separated scientists to With the advent of high-perform- grams are producing data libraries effectively work together as a team ance computers and networks, such at the petabyte level, the equivalent and to facilitate remote access to teams can now be assembled virtu- of 500 billion standard-size docu- both facilities and data. ally, with a multitude of research ment pages, or 100 times the data talents brought to bear on scientific volume of the U.S. Library of problems of national and global Congress. Tools which enable the The DOE Science Grid importance. DOE scientists and easy transfer of huge datasets are A significant portion of DOE engineers have been developing essential to the success of these science is already, or is rapidly technologies in support of these research efforts. becoming, a distributed endeavor “collaboratories.” The goal is to Under SciDAC, a number of involving many collaborators who make collaboration among scientists collaboratory infrastructure projects are frequently working at institutions at geographically dispersed sites as were established to advance collab- across the country or around the effective as if they were working at orative research. These projects world. Such collaborations typically the same location. A key aspect of incorporated technologies from dis- have rapidly increasing needs for collaboratories is providing seamless tributed systems often referred to as high-speed data transfer and high- and secure access to experimental “Grids.” Built upon networks like performance computing. More and facilities and computing resources at the Internet, a computing Grid is a more, these requirements must be sites around the country. service for sharing computer power addressed with computing, data, As these collaborative experi- and data storage capacity over the ments, whether a particle accelerator Internet, resulting in large, distrib- in New York or a massive telescope uted computational resources. DOE atop a volcano in Hawaii, come on is expanding the concept of Grids line, scientists will be inundated with beyond computers to integrate massive amounts of data which must access to experimental facilities and be shared, analyzed, archived and data archives around the world. retrieved by multiple users from While the idea is straightforward, multiple sites. Large research pro- developing a unified, secure infra-

76 ADVANCED COLLABORATIVE INFRASTRUCTURE

and instrument resources that are based method of identifying and The National Fusion often more widely distributed than authenticating users, which leads the collaborators. Therefore, devel- to a “single sign-on” so that any Collaboratory oping an infrastructure that supports system on the Grid can accept a widely distributed computing and uniform user identity for author- data resources is critical to DOE’s ization. This work is currently leading-edge science. used by DOE’s Particle Physics Such infrastructures have Data Grid, Earth Systems Grid, Developing a reliable energy become known as Grids, which and Fusion Grid projects. system that is economically and provide secure and reliable access to • A resource monitoring and environmentally sustainable is the these critical resources. While the debugging infrastructure that long-term goal of DOE’s Fusion individual institutions have their facilitates managing this widely Energy Science (FES) research pro- own policies and procedures for distributed system and the build- gram. In the U.S., FES experimental providing access to users, successful ing of high performance distrib- research is centered at three large deployment of a Grid requires that uted science applications. facilities with a replacement value of a common approach be implement- • Development and deployment over $1 billion. As these experi- ed so that when an authorized user partnerships established with ments have increased in size and logs into a Grid, all appropriate several key vendors. complexity, there has been a con- resources can be easily accessed. • Use of the Grid infrastructure by current growth in the number and Under SciDAC, the DOE Science applications from several disci- importance of collaborations among Grid was developed and deployed plines — computational chemistry, large groups at the experimental across the DOE research complex groundwater transport, climate sites and smaller groups located to provide persistent Grid services modeling, bioinformatics, etc. nationwide. to advanced scientific applications • In collaboration with DOE’s The next generation fusion and problem-solving frameworks. Energy Sciences Network experimental device is the ITER By reducing barriers to the use of (ESnet), the design and deploy- reactor, an international collabora- remote resources, it has made sig- ment of the DOE Grids tion with researchers from China, nificant contributions to SciDAC Certification Authority (Grid Europe, India, Japan, Korea, Russia and the infrastructure required for authentication infrastructure) and the U.S. ITER will be a burning the next generation of science. The that provides Grid identity cer- fusion plasma magnetic confinement DOE Science Grid testbed was ini- tificates to users, systems, and experiment located in France. With tially deployed among five project services, based on a common, ITER located outside the U.S., the sites: the National Energy Research science collaboration oriented ability to maximize its value to the Scientific Computing Center and policy that is defined and U.S. program is closely linked to the Argonne, Lawrence Berkeley, Oak administered by the subscribing development and deployment of Ridge and Pacific Northwest national virtual organizations. collaborative technology. As a result laboratories. of the highly collaborative present and future nature of FES research, In addition to the construction Principal Investigator: William Johnston, of a Grid across five major DOE Lawrence Berkeley National Laboratory the community is facing new and facilities with an initial complement unique challenges. of computing and data resources, The National Fusion Collabora- other accomplishments include: tory established under SciDAC • Integration into the Grid of the unites fusion and computer science large-scale computing and data researchers to directly address these storage systems at NERSC, challenges by creating and deploy- DOE’s Office of Science flagship ing collaborative software tools. In supercomputing center. particular, the project has developed • Design and deployment of a and deployed a national FES Grid Grid security infrastructure that (FusionGrid) that is a system for is facilitating collaboration secure sharing of computation, visu- between U.S. and European alization and data resources over high energy physics projects, the Internet. This represents a fun- and within the U.S. magnetic damental paradigm shift for the fusion community. This infra- fusion community where data, structure provides a global, policy- analysis and simulation codes, and

77 ADVANCED COLLABORATIVE INFRASTRUCTURE

FIGURE 2. FusionGrid services are being used to collaborate in real time by fusion scientists in the U.S. and abroad. Pictured in the fore- ground is Dr. deGrassie (San Diego) leading the JET experiment (England), and on the screen is the JET control room and various JET data traces being discussed in real time.

enhance collaboration within the control rooms of fusion experiments. Shared display walls have been installed in the control rooms of the three large U.S. fusion experiments for enhanced large-group collabora- FIGURE 1. The control rooms of NSTX (a), DIII-D (b), and C-Mod (c) with shared display walls being tion. The project has created and used to enhance collocated collaboration. On the DIII-D shared display is also video from remote deployed unique software that pres- collaborators in Europe who were participating in that day’s experiment. ents the scientists with a multi-user environment allowing them to simul- visualization tools are now thought runs and has supported analysis of taneously share data to the large of as network services. In this new fusion devices on three continents. display and simultaneously interact paradigm, access to resources (data, TRANSP was also used to demon- with the display to edit, arrange, codes, visualization tools) is separated strate the feasibility of between-shot and highlight information. For col- from their implementation, freeing analysis on FusionGrid. located scientists, the ability to pub- the researcher from needing to know Fusion scientists also use the lish and share a scientific graph on about software implementation FusionGrid as an advanced collabo- the display wall results in easier and details and allowing a sharper focus rative environment service that pro- broader discussion compared to the on the physics. The long-term goal vides computer-mediated communi- old model of a few individuals gath- of FusionGrid is to allow scientists cations techniques to enhance work ering around a small computer at remote sites to participate as fully environments and to enable increased monitor. For remote scientists, the in experiments and computational productivity for collaborative work. ability of the shared display to activities as if they were working on In January 2004 a fusion scientist in accept video combined with audio site, thereby creating a unified virtual San Diego remotely led an experiment has allowed them to interact and organization of the geographically on the largest European magnetic even lead an experiment. dispersed U.S. magnetic fusion com- fusion experiment, located in the Taken as a whole, the technolo- munity — more than 1,000 researchers United Kingdom, using FusionGrid’s gies being deployed by the National from over 40 institutions. remote collaboration services. These Fusion Collaboratory project are Fusion scientists use the Fusion- services included multiple video further facilitating data sharing and Grid to provide wide access to the images and a unified audio stream decision-making by both collocated complex physics codes and associated along with real-time secure access and remote scientists. With the eyes data used for simulating and analyz- to data. With the introduction of of the worldwide fusion community ing magnetic fusion experiments. FusionGrid’s remote collaboration shifting towards France for the next The first FusionGrid service came services, remote participation in generation device, ITER, such capa- online in fall 2002. Since then, the fusion experiments is becoming bility becomes increasingly impor- service — a transport code known as more commonplace. tant for scientific success. TRANSP — has been used by scien- The National Fusion Collabora- Principal Investigator: David Schissel, tists to produce over 6,000 code tory project is also working to General Atomics

78 ADVANCED COLLABORATIVE INFRASTRUCTURE

The Particle Physics Data Grid specific end-to-end systems, through computing using Grid technolo- partnering with the National Science gies. Up to 20 sites around the The Department of Energy’s Foundation’s iVDGL and GriPhyN world now receive and analyze national laboratories are home to projects and working with European data from the D0 experiment, some of the world’s leading facilities counterparts such as the European which has led to new discover- for particle physics experiments. Data Grid and Enabling Grids for ies, such as results showing the Such experiments typically generate EsciencE, PPDG has helped pave the mixing of B mesons, mesons millions of particles which must be s way for a worldwide multi-science that contain a “beauty quark” extensively analyzed and filtered to production Grid infrastructure. In and a “strange quark.” The find and measure the exotic parti- the U.S., PPDG has played a leader- PPDG has contributed to such cles that herald new discoveries. ship role in creating first the Grid3 successes by improving a num- The Particle Physics Data Grid and now the Open Science Grid ber of the software applications (PPDG) collaboratory was launched infrastructure, which provides a used to transfer and analyze as a collaboration between physi- common software installation, oper- data, more than doubling the cists and computer scientists to ational procedures and sharing of efficiency of these operations. adapt and extend Grid technologies services for 50 laboratory and uni- CDF seamlessly supports simu- and the physicists’ applications to versity sites. lation and analysis jobs at up to build distributed systems to handle The PPDG-supported improve- 10 sites in Europe and the U.S. the storage, retrieval and analysis of ments in data handling include a • The BaBar experiment at the the particle physics data at DOE’s tenfold increase in the amount of Stanford Linear Accelerator most critical research facilities. sustained data distribution, a 50 per- Center routinely moves all the PPDG has been instrumental in cent reduction in the effort required data collected from the accelera- the significant progress made over to distribute data and analysis among tor to computing centers in the past five years in deploying five institutions, and up to a 50 per- France, Italy and England. The Grid-enabled, end-to-end capabili- cent increase in available resources Storage Resource Broker was ties into the production data pro- due to Grid-enabled sharing. And extended to provide the distrib- cessing infrastructure of the partici- even though funding for the PPDG uted catalogs describing and pating experiments, and in making has concluded, the collaborations managing the data. These cata- extended and robust Grid technolo- established under the program con- logs are necessary to manage gies generally available. Also, PPDG tinue to boost scientific productivity. the location and selection infor- has successfully demonstrated the Significant scientific results and mation for the multipetabyte effectiveness of close collaboration benefits that the technology devel- distributed datasets. between end users and the develop- opments and collaborations in PPDG • The Large Hadron Collider ers of new distributed technologies. have enabled include: (LHC) experiments are partici- The experiments involved span • The shortened turnaround time pating in a worldwide Grid proj- those already in steady data-taking for analysis of the STAR nuclear ect to provide global distribution mode — where existing production physics experiment data pro- and analysis of data from CERN systems were carefully extended duced early results in the first in Switzerland. The U.S. ATLAS step by step to take advantage of direct measurement of particles and U.S. CMS collaborations are Grid technologies — to those devel- known as “open charm” pro- developing and testing their oping new global systems for detector duction at the Relativistic Heavy physics algorithms and distrib- commissioning and the acquisition Ion Collider. This was enabled uted infrastructure in prepara- of data. through the experiment’s col- tion for the onslaught of tens of As a result of PPDG and other laboration with the Storage petabytes a year of data to be Grid projects in the U.S. and Europe, Resource Management group, analyzed by 2008. At the end of particle physics collaborations now which for the past few years 2005, ATLAS produced more include Grid-distributed computing has provided sustained transfer than one million fully simulated as an integral part of their data pro- of more than 5 terabytes a week events on the Open Science Grid cessing models. By collaboratively between Brookhaven National with 20 different physics sam- deploying their developed technolo- Laboratory in New York and ples to test the data distribution gies into significant production use, Lawrence Berkeley National and analysis capabilities. CMS is the computer science groups have Laboratory in California. running “Data Challenges” to extended their technologies and • All results from the Fermilab distribute data from the CERN made them generally available to Tevatron experiments, CDF and accelerator location to the global benefit other research communities. D0, now rely on distributed sites. Data is transferred from In addition to the experiment-

79 ADVANCED COLLABORATIVE INFRASTRUCTURE

the challenge of enabling manage- 4200 4000 Total jobs per VO ment, discovery, access and analysis 3800 of these enormous and extremely 3600 GIN.GGF. 3400 ATLAS ORG MINIBOONE important data assets. For the mod- 3200 AUGER GLOW MIPP 3000 eling teams, the daily management CDF GRASE MIS 2800 and tracking of their data is already 2600 CDMS GRIDEX NANOHUB 2400 proving to be a significant problem. 2200 CMS GROW OSG 2000 DOSAR ILC SDSS Then there are climate researchers 1800 working at institutions and universi- Running jobs 1600 DZERO IVDGL STAR 1400 FERMILAB KTEV THEORY ties across the U.S. whose ability to 1200 FERMILAB- 1000 TEST LIGO ZEUS discover and use the data is 800 GADU 600 extremely limited. That’s today, and 400 the problem is rapidly escalating. 200 0 As computers become more Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep 2005 2006 powerful and can run the models GMT time faster and with more complex details, such as finer geographic res- FIGURE 3: Monitored Jobs on Open Science Grid olution, the amount of data will increase significantly. Much of the current modeling activity is focused the U.S. Tier-1 Center to sites The Earth System Grid II: upon simulations aimed at the around the world as part of the upcoming Intergovernmental Panel data distribution and analysis Turning Climate Model on Climate Change (IPCC) assess- system. Peak transfer rates of ~5 Datasets into Community ment, and these simulations have Gbps are reached. Resources twice the horizontal resolution of • The Open Science Grid Consor- the models that have been run for tium provides a common infra- One of the most important — and the past several years, which structure, validated software challenging — scientific problems increases the resulting data volume stack and ongoing operations which requires extensive computa- fourfold. Figure 4 depicts new work and collaboration which all the tional simulations is global climate representing yet another doubling groups in PPDG use and to change. The United States’ climate of the resolution. which PPDG has made signifi- research community has created the All of this adds up to an enor- cant contributions. The use of Community Climate System Model mous increase in the volume and Open Science Grid has increased (CCSM), one of the world’s leading complexity of data that will be pro- steadily since its launch in July climate modeling codes. By chang- duced. Moving that data will 2005, and it is hoped to contin- ing various conditions, such as the become increasingly costly, which ue with this work to benefit all emission levels of greenhouse gases, could discourage the sharing of data the PPDG collaborators over scientists can model different cli- among the community of climate the next few years. Ideally, the mate change scenarios. Because researchers. significant progress made by running such simulations is compu- The heart of ESG is a simple, PPDG towards a usable, global tationally demanding, the code is elegant, and powerful Web portal computing infrastructure sup- only being run at a few DOE com- that allows scientists to register, porting large institutions can be puting facilities and the National search, browse, and acquire the data extended to benefit smaller Center for Atmospheric Research. they need. This portal was demon- research programs and institu- The resulting data archive, distrib- strated at the Supercomputing 2003 tions as well. uted over several sites, currently contains upwards of 100 terabytes conference on high performance Principal Investigators: Richard Mount, of simulation data and continues to computing. The portal incorporates Stanford Linear Accelerator Center; grow to petabytes by the end of the new capabilities for cataloging Harvey Newman, California Institute of datasets, registering new users and Technology; Miron Livny, University of decade. Making the data available Wisconsin Madison; Ruth Pordes, Fermi to a broad range of researchers for interoperability between different National Accelerator Laboratory (added analysis is key to a comprehensive mass storage architectures. 2003). study of climate change. One of ESG’s strategies is to SciDAC’s Earth System Grid dramatically reduce the amount of (ESG) is a collaborative interdisci- data that needs to be moved over plinary project aimed at addressing the network, and the project team

80 ADVANCED COLLABORATIVE INFRASTRUCTURE

the access rights of the participant and the policy of the resource. While mechanisms for authentication and authorization have been defined, the issues of distribution, dynamics and scale complicate their application to collaborative environments. Additionally, sites providing the resources for a collaboration often have overruling security mechanisms and policies in place which must be taken into account, rather than replaced by the collaboration. The project team has instantiated its research results into the Globus FIGURE 4: From the atmospheric component of the CCSM, this visualization shows precipitable ® water from a high-resolution experimental simulation (T170 resolution, about 70 km). Toolkit’s widely used Grid Security Infrastructure. Since the Globus Tool- kit is already adopted by many sci- has done groundbreaking work in collaboration, what roles they play, ence projects — the Particle Physics developing generalized remote data what types of activities they are Data Grid, Earth Systems Grid, DOE access capabilities. This involves entitled to perform, what communi- Science Grid and other non-DOE heavy collaboration with the SciDAC ty resources are available to mem- Grid activities like NSF TeraGrid, DataGrid Toolkit project, as well as bers of the collaboration and what NASA IPG and the European Data joint work with the community are the policies set down by the Grid — the project’s mechanisms for OpenDAP project. owners of those resources. security and authentication can be In summer 2004, ESG began The Security and Policy for easily used by these scientists. serving IPCC and other model data Group Collaboration project was Principal Investigator: Stephen Tuecke, to a global community in close part- created to develop scalable, secure Argonne National Laboratory nership with the IPCC effort and the and usable methods and tools for World Meteorological Organization. defining and maintaining member- ship, rights, and roles in group col- Principal Investigator: Ian Foster, Argonne Middleware Projects National Laboratory laborations. Such collaborations share common characteristics: Two projects are conducting • The participants, as well as the research and development that will Security and Policy for Group resources, are distributed both address individual technology ele- geographically and organization- ments to enable universal, ubiquitous, Collaboration ally. easy access to remote resources and Today, scientific advances are • Collaborations can scale in size to contribute to the ease with which rarely the result of an individual from a few individuals to thou- distributed teams work together. toiling in isolation, but rather the sands of participants, and mem- Enabling high performance for sci- result of collaboration. Such collab- bership may very dynamic. entific applications is especially orations are using their combined • Collaborations may span areas important. talents to address scientific prob- of expertise, with members filling lems central to DOE’s mission in different roles within the collab- areas such as particle physics exper- oration. Middleware Technology to iments, global climate change and • The work of the team is enabled Support Science Portals: fusion science. While considerable by providing team members work has been done on tools to with access to a variety of com- Gateways to the Grid help perform the work of a collabo- munity resources, including While extensive networks such ration, little attention has been paid computers, storage systems, as the Internet have been in exis- to creating mechanisms for estab- datasets, applications, and tools. tence for decades, it took the organ- lishing and maintaining the struc- Central to this problem of struc- ization of the World Wide Web and ture of such collaborative projects. ture is determining the identity of the creation of easy-to-use Web This structure includes methods for both participants and resources and, browsers to make the Internet the identifying who are members of the based on this identity, determining dynamic information resource that

81 ADVANCED COLLABORATIVE INFRASTRUCTURE

it is today. In much the same way, the Advanced Photon Source or ect have become the de facto stan- networked Grids linking computing, earthquake engineering shake tables, dard for data management and, in data and experimental resources for must be visualized in real time so fact, are opening up new realms of scientific research have the potential that a scientist can adjust the exper- scientific discovery. to greatly accelerate collaborative iment while it is running. For example, participants in the scientific discovery. In order for these applications Large Hadron Collider experiment, Under SciDAC, the Middleware to be feasible, infrastructure must be which will come on line in 2007 in Technology to Support Science Por- in place to support efficient, high Switzerland, are conducting “data tals project was created to develop performance, reliable, secure, and challenges” to ensure that the data a “Science Portal” environment policy aware management of large- coming out of the experiment can which would allow scientists to pro- scale data movement. The SciDAC be accommodated. In a 2005 chal- gram, access and execute distributed DataGrid Middleware project has lenge, a continuous average data Grid applications from a conven- provided tools in three primary areas flow of 600 megabytes per second tional Web browser and other tools in support of this goal. GridFTP, was sustained over 10 days, for a on their desktop computers. Portals developed primarily at Argonne total transfer of 500 terabytes of allow each user to configure their National Laboratory, provides a data – all enabled by GridFTP. own problem-solving environment secure, robust, high performance trans- Moving the same amount of data for managing their work. The goal port mechanism that is recognized over a typical household network is to allow the scientist to focus as the de facto standard for trans- connection (at 512 kilobits per sec- completely on the science by mak- port on the Grid. The Globus repli- ond) would take about 250 years. ing the Grid a transparent extension ca tools, developed primarily at the Principal Investigators: Ian Foster, Argonne of the user’s desktop computing University of Southern California’s National Laboratory; Carl Kesselman, environment. Information Sciences Institute, pro- University of Southern California The first step in this project was vide tracking of replicated data sets Information Sciences Institute; Miron to create a software platform for and provide efficiency in the selec- Livny, University of Wisconsin-Madison building the portal and which could tion of data sets to access. The be used as a testbed. Then the proj- Condor team at the University of ect team partnered with applications Wisconsin is providing storage researchers to get feedback on how resource management, particularly Networking Projects well the applications worked when space reservation tools. Together, As high-speed computer net- accessed by portal users. For appli- these three components provide the works play an increasingly impor- cations, the project partnered with basis for DataGrid applications. tant role in large-scale scientific several National Science Foundation These DataGrid tools are being research, scientists are becoming (NSF) projects. The team also worked used by other SciDAC projects. The more dependent on reliable and with the Global Grid Forum, with Earth System Grid and the Particle robust network connections, NSF funding, to refine the research Physics Data Grid use GridFTP whether for transferring experimen- portal technology for production use. servers to stage input data and move tal data, running applications on Principal Investigator: Dennis Gannon, results to mass storage systems. They distributed supercomputers or col- Indiana University also employ the project’s first gener- laborating across the country or ation replica catalog to determine around the globe. But as anyone the best location from which to store who has logged onto the Internet DataGrid Middleware: Enabling and/or retrieve data. The Laser can attest, network connections can Interferometer Gravitational Wave Big Science on Big Data unexpectedly break or bog down at Observatory project has moved any time. To help keep such system Many of the most challenging over 50 terabytes of data and has a problems from disrupting scientific scientific problems being addressed replica location service with over 3 research, SciDAC funded three by DOE researchers require access million logical files and over 30 mil- research projects to develop better to tremendous quantities of data as lion physical filenames. The Grid3 methods for analyzing and address- well as computational resources. project, part of the Grid Physics ing network performance. Physicists around the world cooper- Network, moves over 4 terabytes of ate in the analysis of petabytes of data a day. GridFTP is now a accelerator data. Climate modelers Global Grid Forum standard and compare massive climate simulation there is a replication working group outputs. Output from multi-million working on standards as well. The dollar online instruments, such as tools developed as part of this proj-

82 ADVANCED COLLABORATIVE INFRASTRUCTURE

INCITE: Monitoring Logistical Networking: A New Mbps between Oak Ridge National Laboratory (ORNL) in Tennessee Performance from the Outside Approach for Data Delivery and North Carolina State University Looking In The project for Optimizing (NCSU). A new, private LN infra- Performance and Enhancing The INCITE (InterNet Control structure is in place, designed specif- Functionality of Distributed and Inference Tools at the Edge) ically for TSI and other SciDAC Applications using Logistical project was aimed at developing projects. Depots at ORNL, San Diego Networking explored the use of a tools and services to improve the Supercomputing Center, SUNY- new technique to provide fast, effi- performance of applications running Stony Brook and NCSU provide 8 cient and reliable data delivery to on distributed computing systems terabytes of storage and form the support the high-performance appli- linked by computational grids. network’s backbone. cations used by SciDAC collabora- Although such applications are The second area of focus is sup- tors. Logistical networking (LN) is a complex and difficult to analyze, port for SciDAC’s fusion energy knowledge of the internal network traffic conditions and services could ORNL FIGURE 5: The interworking optimize the overall performance. 1 DEPOTS components of the Logistical Runtime System Tools (LoRS Without special-purpose network UPLOAD moves a file from Tools) in the high perform- support (at every router), the only a local disk to IBP depots, ance distribution of data returning an exNode, which between IBP depots at alternative is to indirectly infer LOCAL DISK contains file metadata. 1 AT ORNL Terascale Supernova dynamic network characteristics Initiative sites at ORNL and from edge-based network measure- NCSU. ments. Therefore, this project 2 NCSU researched and developed network AUGMENT transfers tools and services for analyzing, data to other locations, UTK whichadds a replica to modeling, and characterizing high- ORNL the exNode. 2 performance network end-to-end performance based solely on edge- NCSU 3 based measurements made at the DEPOTSS hosts and routers at the edges of the DOWNLOAD by the client network, rather than midpoints on retrieves data from the ”nearest” depots. LOCAL DISK the network. Specifically, the goal 3 AT NCSU was to develop multi-scalable online tools to characterize and map network performance as a function new way of synthesizing network- of space, time, applications, proto- researchers. Typical fusion plasma ing and storage to create a commu- cols, and services for end-to-end experiments require real-time feed- nication infrastructure that provides measurement, prediction, and net- back for rapid tuning of experimen- superior control of data movement work diagnosis. tal parameters, meaning data must and management for distributed Two of the tools developed by be analyzed during the 15-minute applications based on shared network the project have been incorporated intervals between plasma-generating storage. LN software tools allow into a toolkit used to identify per- pulses in experimental reactors. Such users to create local storage “depots” formance problems at a number of rapid assimilation of data is achieved or utilize shared storage depots major networks and research sites. by a geographically dispersed research deployed worldwide to easily accom- INCITE users include: Globus, team, a challenge for which LN plish long-haul data transfers, tem- SciDAC’s Supernova Science Center technologies are well suited. porary storage of large datasets (on and Particle Physics Data Grid Pilot, Principal Investigator: Micah Beck, the order of terabytes), pre-position- Scientific Workspaces of the Future, University of Tennessee ing of data for fast on-demand deliv- TeraGrid, Transpac at Indiana ery, and high performance content University, San Diego Supercom- distribution such as streaming video. puting Center, Telecordia, CAIDA, Estimating Bandwidth to Find A primary research drive of the Autopilot, TAU and the European project was to work with SciDAC’s the Fastest, Most Reliable Path GridLab project. Terascale Supernova Initiative (TSI) The Bandwidth Estimation: Principal Investigator: Richard Baraniuk, group, which uses LN to share mas- Measurement Methodologies and Rice University sive datasets at speeds up to 220 Applications project focused on the

83 ADVANCED COLLABORATIVE INFRASTRUCTURE

research, development and deploy- urement methodologies and tools formance of applications, for ment of scalable and accurate band- for various bandwidth-related met- dynamic path selection and traffic width estimation tools for high- rics, such as per-hop capacity, end- engineering. capacity network links. Such tools to-end capacity, and end-to-end As part of the project, Georgia would allow applications to adapt available bandwidth. Capacity is the Tech researchers developed two to changing network conditions, maximum throughput that the path bandwidth estimation tools, finding the fastest, most reliable net- can provide to an application when Pathrate (which measures capacity) work path. This adaptive ability there is no competing traffic. and Pathload (which measures would benefit a large class of data- Available bandwidth, on the other availability), which were released in intensive and distributed scientific hand, is the maximum throughput January 2004. Pathrate and Pathload applications. However, previous that the path can provide to an have been downloaded by more tools and methodologies for meas- application given the path’s current than 2,000 users around the world. uring network capacity, available traffic load. Measuring capacity is Principal Investigators: K. C. Klaffy, San bandwidth and throughput have crucial for debugging, calibrating Diego Supercomputer Center; been largely ineffective across real and managing a path, while measur- Constantinos Davrolis, Georgia Institute of Internet infrastructures. ing available bandwidth is impor- Technology The goal was to develop meas- tant for predicting end-to-end per-

LIST OF PUBLICATIONS

Z. Cai, T. Manteuffel, S. McCormick and J. Ruge, “First-order System LL* (FOSLL*): Scalar Elliptic Partial Differential Equations,” SIAM Journal on Numerical Analysis, 39:1418-1445, 2001. K. Campbell, “Functional Sensitivity Analysis for Computer Model Output,” Proc. of the Seventh Army Conference on Statistics, Santa Fe, NM, Oct. 2001. H. Chen, et al., “Calibrating Scalable Multi- List of Publications Projector Displays Using Camera Homography Trees,” Proc. of Computer Vision and Pattern Recognition, 2001. H. Chen, et al., “Data Distribution Strategies for High-Resolution Displays,” Computers & Graphics, 25(5):811-818, 2001. T.-Y. Chen, “Preconditioning sparse matrices for computing eigenvalues and computing lin- S. Benson, L. Curfman McInnes, and J. ear systems of equations,” PhD Dissertation, 2001 More, “A Case Study in the Performance and UC Berkeley. 2001. Scalability of Optimization Algorithms,” Y. Chen, et al., “Software Environments for ACM Transactions of Mathematical Cluster-based Display Systems,” Proc. of the K. Abazajian, G. M. Fuller, M. Patel, Software 27:361-376, 2001. First IEEE/ACM International “Sterile neutrino hot, warm, and cold dark J. M. Blondin, R. A. Chevalier, D. M. Symposium on Cluster Computing and matter,” Physical Review D, 64, 023501, Frierson, “Pulsar Wind Nebulae in Evolved the Grid, 2001. 2001. Supernova Remnants,” Astrophysical B. Cheng, J. Glimm, X. L. Li D. H. Sharp, Journal Letters, 563, 806-815, 2001. P. R. Amestoy, I. S. Duff, J.-Y. L’Excellent “Subgrid Models and DNS Studies of Fluid and X.S. Li, “Analysis and Comparison of J. M. Blondin, K. J. Borkowski, S. P. Mixing,” Proceedings of the 7th Two General Sparse Solvers for Distributed Reynolds, “Dynamics of Fe Bubbles in Young International Conference on the Physics Memory Computers,” ACM Transactions Supernova Remnants,” Astrophysical of Compressible Turbulent Mixing, E. on Mathematical Software, 27:388-421, Journal Letters, 557, 782-791, 2001. Meshkov, Y. Yanilkin, V. Zhmailo, eds., p. 2001. 385-390, 2001. M. Borland, H. Braun, S. Doebert, L. S. F. Ashby, M. J. Holst, T. A. Manteuffel Groening, and A. Kabel, “Recent E. Chow, “Parallel Implementation and and P. E. Saylor, “The Role of the Inner Experiments on the Effect of Coherent Practical Use of Sparse Approximate Inverses Product in Stopping Criteria for Conjugate Synchrotron Radiation on the Electron Beam With A Priori Sparsity Patterns,” Gradient Iterations,” B.I.T., Vol 41, No. 1, of CTF II,” Proc. of the 2001 Particle International Journal of High Performance 2001. Accelerator Conference, Chicago, Ill., June Computing Appl., 15, pp. 56-74, 2001. 2001. N. R. Badnell, and D. C. Griffin, “Electron- S. Chu, and S. Elliott, “Latitude versus 20+ impact excitation of Fe , including n=4 lev- M. Brezina, A. Cleary, R. Falgout, V. depth simulation of ecodynamics and dis- els,” Journal of Physics B, 34 681, 2001. Henson, J. Jones, T. Manteuffel, S. solved gas chemistry relationships in the cen- McCormick, and J. Ruge, 2001, “Algebraic tral Pacific,” Journal of Atmospheric N. R. Badnell, D. C. Griffin, and D. M. multigrid based on element interpolation Chemistry, 40(3), 305, 2001. Mitnik, “Electron-impact excitation of (AMGe),” SIAM Journal of Scientific Fe21+, including n=4 levels,” Journal of D. D. Clayton, E. A.-N. Deneault, B. S. Computing, 22:1570-1592. Physics B, 34, 5071, 2001. Meyer, “Condensation of Carbon in S. W. Bruenn, K. R. De Nisco, A. Radioactive Supernova Gas,” Astrophysical M. Baertschy and X. S. Li, “Solution of a Mezzacappa, “General Relativistic Effects in Journal Letters, 562, 480-493, 2001. Three-Body Problem in Quantum Mechanics the Core Collapse Supernova Mechanism,” Using Sparse Linear Algebra on Parallel J. P. Colgan, M. S. Pindzola, D. M. Mitnik, Astrophysical Journal Letters, 560, 326- Computers,” Proc. of the SC2001 confer- D. C. Griffin, and I. Bray, “Benchmark non- 338, 2001. ence, Denver, Colorado, November 2001. perturbative calculations for the electron- D. Bruhwiler, R. Giacone, J. Cary, J. impact ionization of Li(2s) and Li(2p),” M. Baker, “Cluster Computing White Paper,” Verboncoeur, P. Mardahl, E. Esarey, W. Physical Review Letters, 87, 213201, International Journal of High Performance Leemans, B. Shadwick, “Particle-in-Cell 2001. Computing Applications – special issue Simulations of Plasma Accelerators and Cluster Computing White Paper, Vol. 15, J. P. Colgan, M. S. Pindzola, and F. J. Electron-Neutral Collisions,” PRST-AB, 4, No. 2, Summer 2001. Robicheaux, “Fully quantal (Á, 2e) calcula- 101302, 2001.

86 LIST OF PUBLICATIONS

tions for absolute differential cross sections of P. Heggernes, S. Eisenstat, G. Kumfert Particle Accelerator Conference, Chicago, helium,” Journal of Physics B, 34, L457, and A. Pothen, “The Computational Ill., June 2001. 2001. Complexity of the Minimum Degree K. Keahey, T. Fredian, Q. Peng, D. P. Algorithm,” Proc. of the Nordic Computer V. K. Decyk and D. E. Dauger, Schissel, M. Thompson, I. Foster, M. Science Conference (NIK), 2001. “Supercomputing for the Masses: A Parallel Greenwald, and D. McCune, Macintosh Cluster,” Proc. 4th Intl. Conf. on W. D. Henshaw, “An Algorithm for “Computational Grids in Action,” The Parallel Processing and Applied Projecting Points onto a Patched CAD National Fusion Collaboratory Future Mathematics, Naleczow, Poland, Model,” Proc. of the 10th International Generation Computer System, 2001. September, 2001, Springer-Verlag Lecture Meshing Roundtable, October 2001. S.-D. Kim, T. Manteuffel and S. Notes in Computer Science 2328, ed. by I. Hofmann, O. Boine-Frankenheim, G. McCormick, “First-order system least R. Wryrzykowski, J. Dongerra, M. Franchetti, J. Qiang, R. Ryne, D. Jeon, J. squares (FOSLS) for spatial linear elasticity: Paprzycki, and J. Wasniewski Springer- Wei, “Emittance Coupling in High Intensity pure traction,” SIAM Journal on Numerical Verlag, Berlin, 2002. Beams Applied to the SNS Linac,” Proc. of Analysis, 38:1454-1482, 2001. C. H. Q. Ding and Y. He, “Ghost Cell the 2001 Particle Accelerator Conference, A. Kuprat, A. Khamayseh, “Volume Expansion Method for Reducing Communi- p. 2902, Chicago, Ill., June 2001. Conserving Smoothing for Piecewise Linear cations in Solving PDE Problems,” Proc of I. Hofmann, J. Qiang, R. D. Ryne and R. Curves, Surfaces, and Triple Lines,” Journal SC2001, Denver, Coloradao, Nov. 2001. W. Garnett, “Collective Resonance Model of of Computational Physics, 172, p. 99-118, F. Dobrian, “External Memory Algorithms Energy Exchange in 3D Nonequipartitioned 2001. for Factoring Sparse Matrices,” PhD Disser- Beams,” Physical Review Letters 86, C.-L. Lappen and D. A. Randall, “Toward tation, Old Dominion University, 2001. 2313–2316, 2001. a Unified Parameterization of the Boundary F. Dobrian and A. Pothen, 2001, “The D. J. Holmgren, P. Mackenzie, D. Layer and Moist Convection. Part III: Design of I/O-Efficient Sparse Direct Petravick, R. Rechenmacher and J. Simulations of Clear and Cloudy Solvers,” Proc. of the SC 2001 conference, Simone, “Lattice QCD production on a com- Convection,” Journal of Atmospheric Denver, Colorado, 2001. modity cluster at Fermilab,” Computing in Sciences., 58, 2052, 2001. High-Energy Physics and Nuclear, 2, V. Eijkhout, “Automatic Determination of C.-L. Lappen and D. A. Randall “Toward Beijing, China, Sept. 2001. Matrix Blocks,” Lapack Working note 151, a Unified Parameterization of the Boundary Parallel Programming, 2001. C. Huang, V. Decyk, S. Wang, E. S. Dodd, Layer and Moist Convection. Part II: Lateral C. Ren, W. B. Mori, T. Katsouleas, and T. Mass Exchanges and Subplume-Scale A. Gabric, W. Gregg, R. G. Najjar, D. J. Antonsen Jr., “QuickPIC: A Parallelized Fluxes,” Journal of Atmospheric Sciences., Erickson III and P. Matrai, “Modeling the Quasi-Static PIC Code for Modeling Plasma 58, 2037, 2001. biogeochemical cycle of DMS in the upper Wakefield Acceleration,” Proc. of the 2001 ocean: A review,” Chemosphere - Global C.-L. Lappen and D. A. Randall, “Toward Particle Accelerator Conference, Chicago, Change Science 3, 377, 2001. a Unified Parameterization of the Boundary Ill., June 2001. Layer and Moist Convection. Part I: A New F. Gerigk, M. Vretenar, and R. D. Ryne, E.-J. Im and K. A. Yelick, Paper: Type of Mass-Flux Model,” Journal of “Design of the Superconducting Section of the “Optimizing Sparse Matrix-Vector Multipli- Atmospheric Science, 58, 2021, 2001. SPL Linac at CERN,” Proc. PAC 2001, p. cation for Register Reuse,” International 3909. J. Larson, R. Jacob, I. Foster, J. Guo, “The Conference on Computational Science, Model Coupling Toolkit,” Proc. of the. 2001 J. R. Gilbert, X. S. Li, E. G. Ng and B. W. San Francisco, Calif., May 2001 International Conference on Peyton, “Computing Row and Column V. Ivanov, A. Krasnykh, “Confined Flow Computational Science, eds. V. N. Counts for Sparse QR and LU Factorization,” Multiple Beam Guns for High Power RF Alexandrov, J. J. Dongarra, C. J. K. Tan, BIT 41:693-710, 2001. Applications,” Proc. of the 2nd IEEE Springer-Verlag, 2001. J. Glimm, X. L. Li, Y. J. Liu, N. Zhao, International Vacuum Electronics S. Lee, T. Katsouleas, R. G. Hemker, E. S. “Conservative Front Tracking and Level Set Conference, Noordwijk, Netherlands, Dodd, and W. B. Mori, “Plasma-Wakefield Algorithms,” Proc. National Academy of April 2001. Acceleration of a Positron Beam,” Physical Sci., p. 14198-14201, 2001. V. Ivanov, A. Krasnykh, “3D Method for Review E, 64, No. 4, 04550, 2001. D. C. Griffin, D. M. Mitnik, J. P. Colgan the Design of Multi or Sheet Beam RF Z. Li, N. Folwell, G. Golub, A. Guetz, B. and M. S. Pindzola, “Electron-impact exci- Sources,” Proc. of the 2001 Particle McCandless, C.-K. Ng, Y. Sun, M. Wolf, tation of lithium,” Physical Review A, 64, Accelerator Conference, Chicago, Ill., R. Yu and K. Ko, “Parallel Computations of 032718, 2001. June 2001. Wakefields in Linear Colliders and Storage D. C. Griffin, D. M. Mitnik and N. R. A. Kabel, “Coherent Synchrotron Radiation Rings,” Proc. of the 2001 Particle Badnell, “Electron-impact excitation of Ne+,” Calculations Using TRAFIC4: Multi-proces- Accelerator Conference, Chicago, Ill., Journal of Physics B, 34, 4401 2001. sor Simulations and Optics Scans,” Proc. of June 2001. the 2001 Particle Accelerator Conference, E. D. Held, J. D. Callen, C. C. Hegna, and Z. Li, K. L. Bane, R. H. Miller, T. O. Chicago, Ill., June 2001. C. R. Sovinec, “Conductive electron heat flow Raubenheimer, J. Wang, R. D. Ruth, along magnetic field lines,” Physics of A. Kabel, “Quantum Corrections to “Traveling Wave Structure Optimization for Plasmas 8, 1171, 2001. Intrabeam Scattering,” Proc. of the 2001 the NLC,” Proc. of the 2001 Particle

87 LIST OF PUBLICATIONS

Accelerator Conference, Chicago, Ill., Proc. of 10th Annual Meshing Round Distributed Computing Practices – special June 2001. Table, Newport Beach, Calif., October issue, Nova Science Publishers, Inc., ISSN 2001. 1097-2803,Volume 4, Number 4, Dec. Z. Li, P. Emma and T. Raubenheimer, 2001. “The NLC L-band Bunch Compressor,” J. Pruet, K. Abazajian, G. M. Fuller, “New Proc. of the 2001 Particle Accelerator connection between central engine weak S. L. Scott, M. Brim and T. Mattson, Conference, Chicago, Ill., June 2001. physics and the dynamics of gamma-ray burst “OSCAR: Open Source Cluster Application fireballs,” Physical Review D, 64, 063002, Resources,” 2001 Linux Symposium, Z. Li, N. Folwell, G. Golub, A. Guetz, B. 2001. Ottawa, Canada, July 2001. McCandless, C.-K. Ng, Y. Sun, M. Wolf, R. Yu and K. Ko, “Parallel Computations Of J. Pruet, G. M. Fuller, C. Y. Cardall, “On S. L. Scott, M. Brim, G.A. Geist, B. Wakefields In Linear Colliders And Storage Steady State Neutrino-heated Ultrarelativistic Luethke and J. Schwidder, “M3C: Rings,” Proc. of the 2001 Particle Winds from Compact Objects,” Managing and Monitoring Multiple Accelerator Conference, Chicago, Ill., Astrophysical Journal Letters, 561, 957- Clusters,” CCGrid 2001 – IEEE June 2001. 963, 2001. International Symposium on Cluster Computing and the Grid, Brisbane, Z. Liu, et al., “Progressive View-Dependent S. C. Pryor, R. J. Barthelmie, J. T. Schoof, Australia, May 2001. Isosurface Propagation,” Proc. of the IEEE L. L. Sorensen, D. J. Erickson III, TCVG Symposium on Visualization, “Implications of heterogeneous chemistry for M. Seth, K. G. Dyall, R. Shepard, and A. 2001. nitrogen deposition to marine ecosystems: Wagner, “The Calculation of f-f Spectra of Observations and modeling,” International Lanthanide and Actinide Ions by the W. P. Lysenko et al., “Characterizing Journal of Environmental Pollution, MCDF-CI Method,” Journal of Physics B: Proton Beam of 6.7 MeV LEDA RFQ by Water, Air and Soil Pollution, Focus 1, 99, At. Molecular and Optical Physics 34, Fitting Wire-Scanner Profiles to 3-D 2001. 2383-2406 2001. Nonlinear Simulations,” Proc. of the 2001 Particle Accelerator Conference, p. 3051, J. Qiang and R. Ryne, “Parallel 3D Poisson R. Shepard, A. F. Wagner, J. L. Tilson and Chicago, Ill., June 2001. Solver for a Charged Beam in a Conducting M. Minkoff, “The Subspace Projected Pipe,” Computer Physics Approximate Matrix (SPAM) Modification B. S. Meyer, S. S. Gupta, L.-S. The, S. Communications 138, 18, 2001. of the Davidson Method,” Journal of Long, “Long-Lived Nuclear Isomers and Computational Physics 172, 472-514. Short-Lived Radioactivities,” Meteoritics & J. Qiang, R. Ryne, B. Blind, J. Billen, T. 2001. Planetary Science (Supp.) 36, 134, 2001. Bhatia, R. Garnett, G. Neuschaefer, H. Takeda, “High Resolution Parallel Particle- A. Snavely, “Modeling Application D. M. Mitnik, D. C. Griffin and N. R. In-Cell Simulation of Beam Dynamics in the Performance by Convolving Machine Badnell, “Electron-impact excitation of SNS Linac,” Nuclear Instrument and Signatures with Application Profiles,” IEEE Ne5+,” Journal of Physics B, 34, 4455, Methods Physics, Res. A, 457, 1, 2001. 4th Annual Workshop on Workload 2001. Characterization, Austin, Texas, Dec. J. Qiang, I. Hofmann, and R. D. Ryne, P. Muggli, et al., “Collective refraction of a 2001. “Cross-Plane Resonance: A Mechanism for beam of electrons at a plasma-gas interface,” Very Large Amplitude Halo Formation,” C. R. Sovinec, J. M. Finn, and D. del- Physical Review Letters Special Topics- Proc. of the 2001 Particle Accelerator Castillo-Negrete, “Formation and sustain- Accelerators & Beams, Vol. 4, No. 9, pp. Conference, p. 1735, Chicago, Ill., June ment of spheromaks in the resistive MHD 091301-091303, Sept. 2001. 2001. model,” Physics of Plasmas 8, 475, 2001. S. Nath et al., “Comparison of Linac J. Qiang and R. D. Ryne, “A Layer-Based L. Sugiyama, W. Park, H. R. Strauss, et al., Simulation Codes,” Proc. of the 2001 Object-Oriented Parallel Framwork for Beam “Studies of spherical tori, stellarators, and Particle Accelerator Conference, p. 264, Dynamics Studies,” Proc. of the 2001 anisotropic pressure with the M3D code,” Chicago, Ill., June 2001. Particle Accelerator Conference, p. 3060, Nuclear Fusion 41, 739, 2001. J. Neiplocha, H. E. Trease, B. J. Palmer, D. Chicago, Ill., June 2001. K. Teranishi, P. Raghavan and E. G. Ng, R. Rector, “Building an Application J. F. Remacle, M. S. Shephard, J. E. “Scalable Preconditioning Using Incomplete Domain Specific Programming Framework Flaherty, O. Klass, “Parallel Algorithm ori- Factors,” Proc. of the Tenth SIAM for Computational Fluid Dynamics ented mesh database,” Proc. 10th Conference on Parallel Processing for Calculations on Parallel Computers,” Tenth International Meshing Roundtable, p. Scientific Computing, 2001. SIAM Conference on Parallel Processing 341-349, Newport Beach, Calif., October for Scientific Computing, March 2001. T. A. Thompson, A. Burrows and B. S. 2001. Meyer, “The Physics of Proto-Neutron Star E. Parker, B. R. de Supinski and D. R. Samanta, et al., “Parallel Rendering with Winds: Implications for r-Process Quinlan, “Measuring the Regularity of K-Way Replication,” Proc. of the IEEE Nucleosynthesis,” Astrophysical Journal, Array References,” Los Alamos Computer Symposium on Parallel and Large-Data 562:887-908, Dec. 2001. Science Institute Symposium (LACSI Visualization and Graphics, 2001. 2001), Los Alamos, NM, Oct. 2001. P. Vanek, M. Brezina and J. Mandel, S. L. Scott, M. Brim, R. Flanery, A. Geist “Convergence analysis of algebraic multigrid N. A. Peterson, K. K. Chand, “Detecting and B. Luethke, “Cluster Command & based on smoothed aggregation,” Numerical translation errors in CAD surfaces and Control (C3) Tools Suite,” Parallel and Mathematics 88, 559—579, 2001. preparing geometries for mesh generation,”

88 LIST OF PUBLICATIONS

R. Vuduc, J. Demmel and J. Bilmes, V. Akcelik, G. Biros and O. Ghattas, 2002, Induced Fluorescence Imaging of Nitric Oxide “Statistical Models for Automatic “Parallel Multiscale Gauss-Newton-Krylov in an NH3-seeded non-premixed methane/air Performance Tuning,” International Journal Methods for Inverse Wave Propagation,” flame,” Proc. of the Combustion Institute, of High Performance Computing Proc. of the IEEE/ACM SC2002 29:2195, 2002. Applications, 2001. Conference, Baltimore, Md., Nov. 2002. G. J. O. Beran, S. R. Gwaltney, and M. T. P. Wangler et al., “Experimental Study of W. Allcock, J. Bester, J. Bresnahan, I. Head-Gordon, “Can coupled cluster singles Proton-Beam Halo Induced by Beam Foster, J. Gawor, J. Insley, J. Link, M. and doubles be approximated by a valence Mismatch in LEDA,” Proc. of the 2001 Papka, “GridMapper: A Tool for Visualizing active space model?,” Journal of Chemistry Particle Accelerator Conference, p. 2923, the Behavior of Large-Scale Distributed Physics. 117, 3040-3048, 2002. Chicago, Ill., June 2001. Systems,” Proc. of the 11th IEEE C. Bernard, et al. (The MILC International Symposium on High A. D. Whiteford, N. R. Badnell, C. P. Collaboration), “Lattice Calculation of Performance Distributed Computing Ballance, M. G. O’Mullane, H. P. Heavy-Light Decay Constants with Two HPDC-11, 179-188, 2002. Summers and A. L. Thomas, “A radiation- Flavors of Dynamical Quarks,” Physical damped R-matrix approach to the electron- C. K. Allen, K. C. D. Chan, P. L. Review D 66, 094501, 2002. impact excitation of helium-like ions for diag- Colestock, K. R. Crandall, R. W. Garnett, L. A. Berry, E. F. D’Azevedo, E. F. Jaeger, nostic application to fusion and astrophysical J. D. Gilpatrick, W. Lysenko, J. Qiang, J. D. B. Batchelor and M. D. Carter, “All- plasmas,” Journal of Physics B, 34, 3179, D. Schneider, M. E. Shulze, R. L. Orders, Full-Wave RF Computations Using 2001. Sheffield, H. V. Smith, and T. P. Wangler, Localized Basis Functions,” 29th EPS “Beam-Halo Measurements in High-Current K. Wu, E. J. Otoo, and A. Shoshani, “A Conference on Plasma Phys. and Contr. Proton Beams,” Physical Review Letters, performance comparison of bitmap indexes,” Fusion, ECA Vol. 26B, P-4.109, June 89, 214802, Nov. 2002. Proc. of the 2001 ACM CIKM 2002. International Conference on Information J. Amundson and P. Spentzouris, “A G. Biros and O. Ghattas, “Inexactness and Knowledge Management, Atlanta, Hybrid, Parallel 3D Space Charge Code with Issues in Lagrange-Newton-Krylov-Schur Georgia, USA, Nov. 2001. Circular Machine Modeling Capabilities,” Methods for PDE-Constrained Proc. of the International Computational Optimization,” Lecture Notes in Accelerator Physics Conference 2002 Computational Science and Engineering, (ICAP2002), East Lansing, Michigan, 30, Springer-Verlag, 2002. Oct. 2002. D. L. Brown, L. Freitag, J. Gilman, C. P. Ballance, N. R. Badnell, and K. A. “Creating interoperable meshing and dis- Berrington, “Electron-impact excitation of 2002 cretization software: the Terascale Simulation H-like Fe at high temperatures,” Journal of Tools and Technology Center,” Proc.of the Physics B, 35, 1095, 2002. Eighth International Conference on J. Abate, S. Benson, L. Grignon, P. G. P. Balls, P. Colella, “A finite difference Numerical Grid Generation in Hovland, L. McInnes, and B. Norris, domain decomposition method using local cor- Computational Field Simulations, “Integrating Automatic Differentiation with rections for the colution of Poissons’s equa- Honolulu, Hawaii , June 2002. Object-Oriented Toolkits for High tion,” Journal of Computational Physics, Performance Scientific Computing,” From X.-C. Cai and D. E. Keyes, “Nonlinearly 180, 25, 2002. Simulation to Optimization, G. Corliss, C. Preconditioned Inexact Newton Algorithms,” Faure, A. Griewank, L. Hascoet and U. D. B. Batchelor, L. A. Berry, M. D. SIAM Journal of Scientific Computing, Naumann, eds, p 173-178, Springer-Verlag, Carter, E. F. Jaeger, C. K. Phillips, R. 24:183-200, 2002. New-York, 2002. Dumont, P. T. Bonoli, D. N. Smithe, R. X.-C. Cai, D. E. Keyes and L. W. Harvey, D. A. D’Ippolito, J. R. Myra, A. Adelmann, “Space Charge Studies at Marcinkowski, “Nonlinear Additive E. D’Azevedo, L. Gray and T. Kaplan, PSI,” Proc. 20th ICFA Advanced Beam Schwarz Preconditioners and Applications in “Tera-scale Computation of Wave-Plasma Dynamics Workshop on High Brightness Computational Fluid Dynamics,” Interactions in Multidimensional Fusion Hadron Beams (ICFA-HB 2002), p. 316, International Journal of Numerical Plasmas,” Plasma Physics and Controlled Batavia, Ill. April 2002. Methods in Fluids, 40:1463-1470, 2002. Nuclear Fusion Research, 2002, Proc. of A. Adelmann and D. Feichtinger, “Generic the 19th International Conference, Lyon, X.-C. Cai, D. E. Keyes and D. P. Young, Large Scale 3D Visualization of Accelerators France, International Atomic Energy “A Nonlinearly Additive Schwarz and Beam Lines,” Proc. ICCS 2002, p. 362, Agency, Vienna, 2002. Preconditioned Inexact Newton Method for 2002. Shocked Duct Flow,” Proc. of the 13th J. B. Bell, M. S. Day and J. F. Grcar, International Conference on Domain S. Adjerid, K. D. Devine, J. E. Flaherty, L. “Numerical Simulation of Premixed Decomposition Methods,” (N. Debit et Krivodonova, “A posteriori error estimation Turbulent Methane Combustion,” Proc. of al., eds.) CIMNE pp. 345-352, 2002. for discontinuous Galerkin solutions of hyper- the Combustion Institute, 29:1987, 2002. bolic problems,” Computer Methods in M. D. Carter, F. W. Baity, Jr., G. C. J. B. Bell, M. S. Day, J. F. Grcar, W. G. Applied Mechanics and Engineering, 191, Barber, R. H. Goulding, Y. Mori, D. O. Bessler, C. Shultz, P. Glarborg and A. D. p. 1997-1112, 2002. Sparks, K. F. White, E. F. Jaeger, F. R. Jensen, “Detailed Modeling and Laser- Chang-Diaz, and J. P. Squire, “Comparing

89 LIST OF PUBLICATIONS

Experiments with Modeling for Light Ion J. P. Colgan and M. S. Pindzola, “(Á, 2e) Huang, E. Dodd, B. E. Blue., C. E. Helicon Plasma Sources,” Physics of total and differential cross section calculations Clayton, C. Joshi, S. Wang, F-J. Decker, Plasmas 9, 5097, 2002. for helium at various excess energies,” M. J. Hogan, R. H. Iverson, C. O’Connell, Physical Review A, 65, 032729, 2002. P. Raimondi, and D. Walz, “3-D L. Chatterjee, M. R. Strayer, J. S. Wu, Simulation of Plasma Wakefield Acceleration “Pair production by neutrino J. P. Colgan and M. S. Pindzola, “Core- with Non-Idealized Plasmas and Beams,” bremsstrahlung,” Physical Review D, 65, excited resonance enhancement in the two- Advanced Accelerator Concepts, Tenth 057302, 2002. photon complete fragmentation of helium,” Workshop, eds. C. E. Clayton and P. Physical Review Letters, 88, 173002, H. Chen, et al., “Scalable Alignment of Muggli, AIP Conf. Proc. No. 647, p. 592- 2002. Large-Format Multi-Projector Displays Using 599. 2002. Camera Homography Trees,” Proc. of IEEE J. P. Colgan and M. S. Pindzola, “Double L. Derose, K. Ekanadham, J. K. Visualization (VIZ), October 2002. photoionization of beryllium,” Physical Hollingsworth and S. Sbaraglia, “SIGMA: Review A, 65, 022709, 2002. H. Chen, et al., “Memory Performance A Simulator to Guide Memory Analysis,” Optimizations for Real-Time Software J. P. Colgan and M. S. Pindzola, “Time- Proc. of the SC 2002 conference, HDTV Decoding,” Proc. of the IEEE dependent close-coupling studies of the elec- Baltimore, Md., Nov. 2002. International Conference on Multimedia tron-impact ionization of excited-state D. A. Dimitrov, D. L. Bruhwiler, W. and Expo, August 2002. Helium,” Physical Review A, 66, 062707, Leemans, E. Esarey, P. Catravas, C. Toth, 2002. H. Chen, et al., “A Parallel Ultra-High B. Shadwick, J. R. Cary, and R. Giacone, Resolution MPEG-2 Video Decoder for PC J. P. Colgan, M. S. Pindzola and F. J. “Simulations of Laser Propagation and Cluster Based Tiled Display System,” Proc. Robicheaux, “A Fourier transform method of Ionization in l’OASIS Experiments,” Proc. of International Parallel and Distributed calculating total cross sections using the time- Advanced, Accel. Concepts Workshop, p. Processing Symposium, 2002. dependent close-coupling theory,” Physical 192, Mandalay Beach, CA, 2002. Review A, 66, 012718, 2002. H. Chen, et al., “Experiences with D. A. Dimitrov, D. L. Bruhwiler, W. P. Scalability of Display Walls,” Proc. of the J. H. Cooley, T. M. Antonsen Jr., C. Leemans, E. Esarey, P. Catravas, C. Toth, 7th Annual Immersive Projection Huang, V. Decyk, S. Wang, E. S. Dodd, C. B. A. Shadwick, J. R. Cary and R. E. Technology Symposium, 2002. Ren, W. B. Mori, and T. Katsouleas, Giacone, “Simulations of Laser Propagation “Further Developments for a Particle-in-Cell and Ionization in l’OASIS Experiments,” E. Chow, T. A. Manteuffel, C. Tong, B. K. Code for Efficiently Modeling Wakefield Proc. Advanced Accel. Concepts Wallin, “Algebraic elimination of slide sur- Acceleration Schemes,” Advanced Workshop, AIP Conf. Proc. 647, eds. C. face constraints in implicit structural analy- Accelerator Concepts, Tenth Workshop, E. Clayton and P. Muggli, AIP, New sis,” International Journal for Numerical eds. C. E. Clayton and P. Muggli, AIP York, 2002. Methods in Eng., 01, 1-21, 2002. Conf. Proc. No. 647, p. 232-239, 2002. E. S. Dodd, R. G. Hemker, C.-K.Huang, S. Chu, S. Elliott and M. Maltrud, “Whole J. J. Cowan, C. Sneden, S. Burles, I. I. S. Wang, C. Ren, W. B. Mori, S. Lee and ocean fine resolution carbon management sim- Ivans, T. C. Beers, J. W. Truran, J. E. T. Katsouleas, “Hosing and Sloshing of ulations systems,” EOS, Transactions, Lawler, F. Primas, G. M. Fuller, B. Short-Pulse Ge-V-Class Wakefield Drivers,” American Geophysical Union 2002 Pfeiffer, K.-L. Kratz, “The Chemical Physical Review Letters, 88, 125001, Spring Meeting, 83(19), S125, 2002. Composition and Age of the Metal-poor Halo March 2002. C. E. Clayton, B. E. Blue, E. S. Dodd, C. Star BD +17°3248,” Astrophysical Journal E. D. Dolan and J. J. More, “Benchmarking Joshi, K. A. Marsh, W. B. Mori, S. Wang, Letters, 572, 861-879, 2002. Optimization Software with Performance P. Catravas, S. Chattopadhyay, E. Esarey, T. Cristian, I.-H. Chung and J. K. Profiles,” Mathematical Programming, W. P. Leemans, R. Assmann, F. J. Decker, Hollingsworth, “Active Harmony: Towards 91:201-213, 2002. M. J. Hogan, R. Iverson, P. Raimondi, R. Automated Performance Tuning,” Proc. of H. Siemann, D. Walz, T. Katsouleas, S. D. Dolgov, et al. (The LHPC Collabora- the SC 2002 conference, Baltimore, Md., Lee and P. Muggli, “Transverse Envelope tion):, “Moments of nucleon light cone quark Nov. 2002. Dynamics of a 28.5-GeV Electron Beam in a distributions calculated in full lattice QCD,” Long Plasma,” Phys. Rev. Lett., 88, 15 p. S. Deng, F. Tsung, S. Lee, W. Lu, W. B. Physical Review D 66, 034506, 2002. 154801/1-4, April 2002. Mori, T. Katsouleas, P. Muggli, B. E. S. C. Doney and M. W. Hecht, “Antarctic Blue., C. E. Clayton, C. O’Connell, E. D. D. Clayton, B. S. Meyer, L.-S. The, M. Bottom Water Formation and Deep Water Dodd, F. J. Decker, C. Huang, M. J. F. El Eid, “Iron Implantation in Presolar Chlorofluorocarbon Distributions in a Global Hogan, R. Hemker, R. H. Iverson, C. Supernova Grains,” Astrophysical Journal Ocean Climate Model,” Journal of Physics, Joshi, C. Ren, P. Raimondi, S. Wang and Letters, 578, L83-L86, 2002. Oceanography 32, 1642, 2002. D. Walz, “Modeling of Ionization Physics E. Colby, V. Ivanov, Z. Li, C. Limborg, with the PIC Code OSIRIS,” Advanced R. J. Dumont, C. K. Phillips and D. N. “Simulation Issues for RF Photoinjector,” Accelerator Concepts, Tenth Workshop, Smithe, “ICRF wave propagation and Proc. of the International Computational eds. C. E. Clayton and P. Muggli, AIP absorption in plasmas with non-thermal pop- Accelerator Physics Conference 2002 Conf. Proc. No. 647, p. 219-223, 2002. ulations,” 29th EPS Conference on Plasma (ICAP2002), East Lansing, Michigan, Phys. and Contr. Fusion, ECA Vol. 26B, S. Deng, T. Katsouleas, S. Lee, P. Muggli, Oct. 2002. P-5.051, Montreux, Switzerland, June W. B. Mori, R. Hemker, C. Ren, C. 2002.

90 LIST OF PUBLICATIONS

D. J. Erickson, III and J. L. Hernandez, “A J. R. Cary, and R. Giacone, “Semi-analyti- Factorization with Static Pivoting,” Proc. of global, high resolution, satellite-based model of cal 6D Space Charge Model for Dense the IEEE/ACM SC2002 Conference, air-sea isoprene flux,” Gas Transfer at Electron Bunches with Large Energy Baltimore, Md., Nov. 2002. Water Surfaces, ed. by M. A. Donelan, W. Spreads,” Proc. of the Advanced W. Gropp, L. McInnes and B. Smith, M. Drennan, E. S. Saltzman, and R. Accelerator Concepts Workshop “Chapter 19 of CRPC Handbook of Parallel Wanninkhof, American Geophysical Mandalay Beach, Calif., 2002. Computing,” Morgan Kaufmann, 2002. Union Monograph 127, 312, p. 312, 2002. G. M. Fuller, “The nuclear physics and P. T. Haertel and D. A. Randall, “Could a R. Evard, D. Narayan, J. P. Navarro, and astrophysics of the new neutrino physics,” pile of slippery sacks behave like an ocean?” D. Nurmi, “Clusters as large-scale develop- AIP Conf. Proc. 610: Nuclear Physics in Monthly Weather Review, 130, 2975, ment facilities,” Proc of the 4th IEEE the 21st Century, p. 231-246, 2002. 2002. International Conference on Cluster A. Gebremedhin, F. Manne, and A. Computing (CLUSTER02), p. 54, IEEE A. Heger, S. E.Woosley, T. Rauscher, R. Pothen, “Parallel Distance- k Coloring Computer Society, 2002. D. Horman, and M. M. Boyes, “Massive Algorithms for Numerical Optimization,” star evolution: nucleosynthesis and nuclear R. D. Falgout and U. M. Yang, “hypre: a Lecture Notes in Computer Science, reaction rate uncertainties,” New Library of High Performance 2400, p. 912-921, Springer-Verlag, 2002. Astronomy Review, 46:463-468, July Preconditioners,” in Computational Science E. George, J. Glimm, X. L. Li, A. 2002. - ICCS 2002 Part III, P. M. A. Sloot, C .J. Marchese, Z. L. Xu, “A Comparison of K. Tan. J. J. Dongarra, and A. G. Hoekstra, R. P. Heikes, “A comparison of vertical coor- Experimental, Theoretical and Numerical eds., Lecture Notes in Computer Science, dinate systems for numerical modeling of the Simulation Rayleigh-Taylor Mixing Rates,” 2331, p. 632-641, Springer-Verlag, 2002. general circulation of the atmosphere,” Ph.D. Proc. National Academy of Science, 59, thesis, Colorado State University, 2002. J. E. Flaherty, L. Krivodonova, J. F. p. 2587-2592, 2002. Remacle, M. S. Shephard, “Aspects of dis- W. D. Henshaw, “Generating Composite S. J. Ghan, X. Bian, A. G. Hunt and A. continuous Galerkin methods for hyperbolic Overlapping Grids on CAD Geometries,” Coleman, “The thermodynamic influence of conservation laws,” Finite Elements in Numerical Grid Generation in Compu- subgrid orography in a global climate model,” Analysis and Design, 38:889-908, 2002. tational Field Simulations, June 2002. Climate Dynamics, 20, 31-44, 2002. J. E. Flaherty, L. Krivodonova, J. F. W. D. Henshaw, “An Algorithm for A. Z. Ghalam, T. Katsouleas, S. Lee, W. Remacle, M. S. Shephard, “High-order Projecting Points onto a Patched CAD B. Mori, C. Huang, V. Decyk and C. Ren, adaptive and parallel discontinuous Galerkin Model,” Engineering with Computers, 18, “Simulation of Electron-Cloud Instability in methods for hyperbolic systems,” WCCM V: p. 265-273, 2002. Circular Accelerators using Plasma Models,” Fifth World Congress on Computational Advanced Accelerator Concepts, Tenth V. E. Henson and U. M. Yang, Mechanics, Vienna Univ. of Tech., Workshop, eds. C. E. Clayton and P. “BoomerAMG: a Parallel Algebraic Vienna, Austria, p. 1:144, 2002. Muggli, AIP Conf. Proc. No. 647, p. 224j- Multigrid Solver and Preconditioner,” R. A. Fonseca, et al., “OSIRIS: A 3D, fully 231, 2002. Applied Numerical Mathematics, 41, p. relativistic PIC code for modeling plasma 155-177, 2002. O. Ghattas and L. T. Biegler, “Parallel based accelerators,” Lect. Notes Comput. Algorithms for Large-Scale Simulation-based J. J. Heys, C. G. DeGroff, T. Manteuffel, S. Sci, 2331, 3 p. 342-351. 2002. Optimization,” iModeling and Simulation- McCormick and W. W. Orlando, “First- T. W. Fredian and J. A. Stillerman, Based Life Cycle Engineering, Spon order system least squares for elastohydrody- “MDSplus, Current Developments and Press, London, 2002. namics with application to flow in compliant Future Directions,”The 3rd IAEA blood vessels,” Biomed. Sci. Instrum. 38, p. T. A. Gianakon, S. E. Kruger, and C. C. Technical Meeting on Control, Data 277-282, 2002. Hegna, “Heuristic closures for numerical sim- Acquisition, and Remote Participation for ulation of neoclassical tearing modes,” M. J. Hogan, C. D. Barnes, C. E. Clayton, Fusion Research, Fusion Engineering and Physics of Plasmas 9, 536, 2002. C. O’Connell, F. J. Decker, S. Deng, P. Design, 60, 229-233, 2002. Emma, C. Huang, R. Iverson, D. K. J. Glimm, X. L. Li, A. Lin, “Nonuniform L. Freitag, T. Leurent, P. Knupp, D. Johnson, C. Joshi, T. Katsouleas, P. approach to terminal velocity for single mode Melander, “MESQUITE Design: Issues in Krejcik, W. Lu, K. A. Marsh, W. B. Mori, Rayleigh-Taylor instability,” Acta the Development of a Mesh Quality P. Muggli, R. H. Siemann and D. Walz, Mathematicae Applicatae Sinica, 18, p. 1- Improvement Toolkit”, Proceedings of the “Acceleration and Focusing of Electrons and 8, 2002. 8th Intl. Conference on Numerical Grid Positrons using a 30 GeV Drive Beam,” Generation in Computational Field J. Glimm, X. L. Li, Y. J. Liu, “Conservative Advanced Accelerator Concepts, Tenth Simulations, Honolulu, p159-168, 2002. Front Tracking in One Dimension,” Fluid Workshop, eds. C. E. Clayton and P. Flow and Transport in Porous Media: Muggli, AIP Conf. Proc. No. 647, pp. 3- C. L. Fryer and M. S. Warren, “Modeling Mathematical and Numerical Treatment, 10, 2002. Core-Collapse Supernovae in Three Contemporary Mathematics, Z. X. Chen, Dimensions,” Astrophysical Journal P. Hovland, B. Norris and B. Smith, R. Ewing, eds., American Mathematical Letters, 574:L65-L68, July 2002. “Making Automatic Differentiation truly Society, 295, p. 253-264, 2002. Automatic: Coupling PETSc with ADIC,” G. Fubiani, G. Dugan, W. Leemans, E. L. Griogori and X. S. Li, “A New Proc. of ICCS 2002, Amsterdam, Esarey, P. Catravas, C. Toth, B. Shadwick, Scheduling Algorithm for Parallel Sparse LU Netherlands, 2002.

91 LIST OF PUBLICATIONS

E. C. Hunke and J. K. Dukowicz, “The A. Kabel, “Parallel Simulation Algorithms Kapur, M. Martin, B. Thompson, T. Tung Elastic-Viscous-Plastic Sea Ice Dynamics for the Strong-Strong Beam-Beam and D. Yoo, “Design, Implementation and Model in General Orthogonal Curvilinear Interaction,” Proc. of the International Testing of Extended and Mixed Precision Coordinates on a Sphere—Incorporation of Computational Accelerator Physics BLAS,” ACM Transactions of Metric Terms,” Monthly Weather Review, Conference 2002 (ICAP2002), East Mathematical Software 22:152-205, 2002. 130, 1848, 2002. Lansing, Michigan, Oct. 2002. M. Liebendorfer, O. E. B. Messer, A. C. Joshi, B. Blue, C. E. Clayton, E. Dodd, T. Katsouleas, A. Z. Ghalam, S. Lee, W. Mezzacappa, W. R. Hix, F.-K. C. Huang, K. A. Marsh, W. B. Mori, S. B. Mori, C. Huang, V. Decyk and C. Ren, Thielemann, K. Langanke, “The importance Wang, J. J. Hogan, C. O’Connell, R. “Plasma Modeling of Wakefields in Electron of neutrino opacities for the accretion luminos- Siemann, D. Watz, P. Muggli, T. Clouds,” ECLOUD’02 Mini-Workshop on ity in spherically symmetric supernova mod- Katsouleas and S. Lee, “High Energy Electron-Cloud Simulations for Proton els,” Proc. of the 11th Workshop on Density Plasma Science with an and Positron Beams, CERN, Geneva, Nuclear Astrophysics, p. 126-131, Max- Ultrarelativistic Electron Beam,” Physics of Switzerland, 2002. Planck-Institut fuer Astrophysik, Plasmas, Vol. 9, No. 5, p. 1845-1855, May Germany, 2002. K. Keahey and V. Welch, “Fine-Grain 2002. Authorization for Resource Management in Z. Liu, et al., “Improving Progressive View- J. Ivanic and K. Ruedenberg, “Deadwood the Grid Environment,” Proc. of the 3rd Dependent Isosurface Propagation,” in Configuration Spaces. II. SD and SDTQ International Workshop on Grid Computers and Graphics, 26 (2): 209- Spaces,” Theoretical Chemistry Accounts, Computing, 2002. 218, 2002. 107, 220-228, 2002. D. E. Keyes, “Terascale Implicit Methods for S. D. Loch, M. S. Pindzola and J. P. V. Ivanov, N. Folwell, G. Golub, A. Guetz, Partial Differential Equations,” The Barrett Colgan, “Electron-impact ionization of all Z. Li, W. Mi, C.-K. Ng, G. Schussman, Y. Lectures, University of Tennessee ionization stages of krypton,” Physical Sun, J. Tran, M. Weiner, M. Wolf and K. Mathematics Department, 2001, Review A, 66, 052708, 2002. Ko, “Computational Challenge In Large Contemporary Mathematics 306:29-84, C.-D. Lu and D. A. Reed, “Compact Scale Accelerator Modeling,” ICCM-2002, AMS, Providence, RI, 2002. Application Signatures for Parallel and Novosibirsk, Russia, June 2002. D. E. Keyes, P. D. Hovland, L. C. Distributed Scientific Codes,” Proc. of the V. Ivanov, R. Ives, A. Krasnykh, G. McInnes and W. Samyono, “Using SC 2002 conference, Baltimore, Md., Miram, M. Read, “An Electron Gun for a Automatic Differentiation for Second-Order Nov. 2002. Sheet Beam Klystron,” IVEC-2002, Matrix-free Methods in PDE-constrained X. Luo, M. S. Shephard, J. F. Remacle, R. Monterey, Calif., April 2002. Optimization,” Automatic Differentiation M. Bara, M. W. Beall, B. A. Szab, “p- of Algorithms: From Simulation to V. Ivanov, G. Schussman, M. Weiner, Version Mesh Generation Issues,” 11th Optimization (G. Corliss et al., eds.), “Particle Tracking Algorithms for 3D Parallel International Meshing Roundtable, 2002. Springer-Verlag, p 35-50, 2002. Codes in Frequency and Time Domains,” X. Luo, M. S. Shephard, J. F. Remacle, Proc. of 18th Annual Review of Progress A. Khamayseh, A. Kuprat, “Hybrid Curve “The Influence of Geometric Approximation in Applied Computational Point Distribution Algorithms,” SIAM on the Accuracy of High Order Methods,” Electromagnetics, ACES-2002, Monterey, Journal on Scientific Computing, 23, p. Proc. 8th International Conference on Calif., March 2002. 1464-1484, 2002. Numerical Grid Generation in E. F. Jaeger, L. A. Berry, E. D’Azevedo, D. A. Klawonn, O. B. Widlund and M. Computational Field Simulations, Miss. B. Batchelor, M. D. Carter, K. F. White Dryja, “Dual-Primal FETI Methods for State University, 2002. and H. Weitzner, “Advances in Full-Wave Three-dimensional Elliptic Problems with W. P. Lysenko, R. W. Garnett, J. D. Modeling of Radio Frequency Heated, Heterogeneous Coefficients,” SIAM Journal Gilpatrick, J. Qiang, L. J. Rybarcyk, R. D. Multidimensional Plasmas,” Physics of on Numerical Analysis, 40:159-179, 2002. Ryne, J. D. Schneider, H. V. Smith, L. M. Plasmas 9, 1873, 2002. K. Ko, “High Performance Computing in Young, and M. E. Schulze, “High Order G. C. Jordan, B. S. Meyer, “Sensitivity of Accelerator Physics,” Proc. of 18th Annual Beam Features and Fitting Quadrupole Scan Isotope Yields to Reaction Rates in the Alpha Review of Progress in Applied Data to Particle Code Model,” Proc. of the Rich Freezeout,” Proc. of the Workshop on Computational Electromagnetics, ACES- International Computational Accelerator Astrophysics, Symmetries, and Applied 2002, Monterey, Calif., March 2002. Physics Conference 2002 (ICAP2002), Physics at Spallation Neutron Sources East Lansing, Michigan, Oct. 2002. S. Lee, T. Katsouleas, P. Muggli, W. B. (ASAP 2002), p. 42. Oak Ridge, Tenn. Mori, C. Joshi, R. Hemker, E. S. Dodd, C. K.-L. Ma, G. Schussman, B. Wilson, K. March 2002. E. Clayton, K. A. Marsh, B. Blue, S. Ko, J. Qiang and R. Ryne, “Advanced A. Kabel, “A Parallel Code for Lifetime Wang, R. Assmann, F. J. Decker, M. Visualization Technology for Terascale Calculations in P/P-Storage Rings in the Hogan, R. Iverson, and D. Walz, “Energy Particle Accelerator Simulations,” Proc. of Presence of Parasitic Beam-Beam Crossings,” Doubler for a Linear Collider,” Physical the IEEE/ACM SC2002 Conference, Proc. of the International Computational Review Special Topics-Accelerators And Baltimore, Md., Nov. 2002. Accelerator Physics Conference 2002 Beams, 5, 1, January 2002. J. L. McClean, P.-M. Poulain, J. W. Pelton, (ICAP2002), East Lansing, Michigan, X. Li, J. Demmel, D. Bailey, G. Henry, Y. and M. E. Maltrud, “Eulerian and Oct. 2002. Hida, J. Iskandar, W. Kahan, S. Kang, A. Lagrangian statistics from surface drifters

92 LIST OF PUBLICATIONS

and a high-resolution POP simulation in the C.-K. Ng, L. Ge, A. Guetz, V. Ivanov, Z. J. Qiang, M. Furman and R. Ryne, “Strong- North Atlantic,” Journal of Physical Li, G. Schussmann, Y. Sun, M. Weiner, M. Strong Beam-Beam Simulation Using a Oceanography, 32, 2472, 2002. Wolf, “Numerical Studies of Field Gradient Green Function Approach,” Phys. Rev. ST and Dark Currents in SLAC Structures,” Accel. Beams, 5, 104402, Oct. 2002. C. W. McCurdy, P. Cummings, E. Stechel, Proc. of the International Computational B. Hendrickson and D. E. Keyes, “Theory J. Qiang and R. Ryne, “Parallel Beam Accelerator Physics Conference 2002 and Modeling in Nanoscience,” whitepaper Dynamics Simulation of Linear Accelerators,” (ICAP2002), East Lansing, Michigan, commissioned by the U.S. DOE Offices Proceedings of the 2002 Applied Oct. 2002. of Basic Energy Sciences and Advanced Computational Electromagnetics Society Scientific Computing, 2002. C.-K. Ng et. al.,“Modeling HOM Heating in (ACES’2002) Conference, March 2002. the PEP-II IR with Omega3P,” Proc. of the C. L. Mendes and D. A. Reed, D. A. Randall, T. D. Ringler, R. P. Heikes, International Computational Accelerator “Monitoring Large Systems via Statistical P. Jones and J. Baumgardner, “Climate Physics Conference 2002 (ICAP2002), Sampling,” LACSI Symposium, Santa Fe, modeling with spherical geodesic grids,” East Lansing, Michigan, Oct. 2002. Oct. 2002. Computing in Science and Engr., 4, 32- B. Norris, S. Balay, S. Benson, L. Freitag, 41. 2002. B. S. Meyer, “r-Process Nucleosynthesis with- P. Hovland, L. McInnes, and B. Smith, out Excess Neutrons,” Physical Review T. Rauscher, A. Heger, R. D. Hoffman “Parallel Components for PDEs and Letters, 89, 231101, 2002. and S. E. Woosley, “Nucleosynthesis in Optimization: Some Issues and Experiences,” Massive Stars with Improved Nuclear and D. M. Mitnik, D. C. Griffin, and M. S. Parallel Computing 28:1811-1831, 2002. Stellar Physics,” American Physics Journal, Pindzola, “Time-dependent close-coupling cal- C. O’Connell et al., “Dynamic Focusing of 576:323-348, Sept. 2002. culations of dielectronic capture in He,” an Electron Beam through a Long Plasma,” Physical Review Letters, 88, 173004, J. F. Remacle, O. Klass, J. E. Flaherty, M. Phys. Rev. STAB, 5, 121301, 2002. 2002. S. Shephard, “A parallel algorithm oriented R. J. Oglesby, S. Marshall, D. J. Erickson mesh database,” Engineering with J. J. More, “Automatic Differentiation Tools III, J. O. Roads and F. R. Robertson, Computers, 18, 274,2002. in Optimization Software,” in Automatic “Thresholds in atmosphere-soil moisture inter- Differentiation 2000: From Simulation to J. F. Remacle, M. S. Shepard, J. E. actions: Results from climate model studies,” Optimization, G. Corliss, C. Faure, A. Flaherty, “Some issues on distributed mesh Journal of Geophysical Research,. 107, Griewank, L. Hascoet, and U. Naumann, representations,” Proc. 8th International 2002. eds, p 25-34, Springer-Verlag, New-York, Conference on Numerical Grid 2002. L. F. Pavarino and O. B. Widlund, Generation in Computational Field “Balancing Neumann-Neumann methods for Simulations, Miss. State University, 2002. W. B. Mori, “Recent Advances and Some incompressible Stokes equations,” Comm. Results in Plasma-Based Accelerator J. F. Remacle, O. Klass, M. S. Shephard, Pure Appl. Math. 55:302-335, 2002. Modeling,” Advanced Accelerator “Trellis: A framework for adaptive numerical Concepts, Tenth Workshop, eds. C. E. M. S. Pindzola, “Proton-impact excitation of analysis based on multiparadigm program- Clayton and P. Muggli, AIP Conf. Proc. laser-excited lithium atoms,” Physical ming in C++,” WCCM V: Fifth World No. 647, p. 11-28, 2002. Review A, 66, 032716, 2002. Congress on Computational Mechanics, Vienna Univ. of Tech, Vienna, Austria, p. P. Muggli, “Transverse Envelope Dynamics P. Ploumans, G. S. Winckelmans, J. K. 1-164, 2002. of a 28.5-GeV Electron Beam in a Long Salmon, A. Leonard, and M. S. Warren, Plasma,” Physical Review Letters, Vol. 88, “Vortex Methods for High-Resolution J. F. Remacle, X. Li, N. Chevaugeon, M. No. 15 pp. 154801/1-4, April 2002. Simulation of Three-Dimensional Blu® Body S. Shepard, “Transient mesh adaptation Flows; Application to the Sphere at Re=300, using conforming and non-conforming mesh C. Naleway, M. Seth, R. Shepard, A. F. 500 and 1000,” Journal of Computational modifications,” 11th International Meshing Wagner, J. L. Tilson, W. C. Ermler and S. Physics, 178:427-463, 2002. Roundtable, 2002. R. Brozell, “An Ab Initio Study of the Ionization Potentials and the f-f Spectroscopy J. Pruet and N. Dalal, “Implications of T. D. Ringler and D. A. Randall, “The of Europium Atoms and Ions,” Journal of Neutron Decoupling in Short Gamma-Ray ZM Grid: An Alternative to the Z Grid,” Chemistry Physics. 116, 5481-5493. 2002. Bursts,” ApJ, 573:770-777, July 2002. Monthly Weather Review, 130, 1411–1422, 2002. J. P. Navarro, R. Evard, D. Nurmi, and N. J. Pruet, S. Guiles, G. M. Fuller, “Light- Desai, “Scalable cluster administration - Element Synthesis in High-Entropy T. D. Ringler, and D. A. Randall, “A {Chiba City I} approach and lessons learned,” Relativistic Flows Associated with Gamma- Potential Enstrophy and Energy Conserving Proc of the 4th IEEE International Ray Bursts,” Astrophysical Journal Letters, Numerical Scheme for Solution of the Conference on Cluster Computing 580, 368-373, 2002. Shallow-Water Equations on a Geodesic (CLUSTER02), p. 54, IEEE Computer Grid,” Monthly Weather Review, 130, J. Qiang, P. L. Colestock, D. Gilpatrick, H. Society, 2002. 1397–1410, 2002. V. Smith, T. P. Wangler and M. E. J.W. Negele, “Understanding parton distri- Schulze, “Macroparticle Simulation Studies F. J. Robicheaux and J. D. Hanson, butions from lattice QCD: Present limitations of a Proton Beam Halo Experiment,” Phys. “Simulation of the expansion of an ultracold and future promise,” Nuclear Physics A Rev. ST Accel. Beams, vol 5, 124201, neutral plasma,” Physical Review Letters, 711, 281, 2002. 2002. 88, 055002. 2002.

93 LIST OF PUBLICATIONS

J. M. Sampaio, K. Langanke, G. Martinez- Ottawa Linux Symposium (OLS 2002), Accelerator Structure Design,” Proc. of the Pinedo, D. J. Dean, “Neutral-current neutri- Ottawa, Canada, July 2002. International Computational Accelerator no reactions in the supernova environment,” Physics Conference 2002 (ICAP2002), S. L. Scott, J. L. Mugler and T. Naughton, Physics Letters B, 529, 19-25. 2002. East Lansing, Michigan, Oct. 2002. “User Interfaces for Clustering Tools,” Ottawa D. P. Schissel, M. J. Greenwald, and W .E. Linux Symposium (OLS 2002), Ottawa, K. Teranishi, P. Raghavan, and E. G. Ng, Johnston, “Fusion Energy Sciences Canada, July 2002. “A New Data-Mapping Scheme for Latency- Networking: A Discussion of Future Tolerant Distributed Sparse Triangular S. L. Scott, B. Luethke, “C3 Power Tools Requirements,” DOE High Performance Solution,” Proc. of the IEEE/ACM Commodity, High-Performance Cluster Network Planning Workshop, 2002. SC2002 Conference, Baltimore, Md., Nov. Computing Technologies and Applications,” 2002. D. P. Schissel, “An Advanced Collaborative Sixth World Multiconference on Environment to Enhance Magnetic Fusion Systemics, Cybernetics, and Informatics K. Teranishi, P. Raghavan, and E. G. Ng, Research, for the National Fusion (ISAS SCI2002), Orlando, Fla., July 2002. “Reducing Communication Costs for Parallel Collaboratory Project,” Proceedings of the Sparse Triangular Solution,” Proc. of the S. L. Scott, T. Naughton, B. Barrett, J. Workshop on Advanced Collaborative IEEE/ACM SC2002 Conference, Squyres, A. Lumsdain and Y. C. Fang, Environments, 2002. Baltimore, Md., Nov. 2002. “The Penguin in the Pail – OSCAR Cluster D. P. Schissel, et al., “Data Management, Installation Tool Commodity, High- F.-K. Thielemann, D. Argast, F. Brachwitz, Code Deployment, and Scientific Performance Cluster Computing Technologies G. Martinez-Pinedo, T. Rauscher, M. Visualization to Enhance Scientific Discovery and Applications,” Sixth World Liebendoerfer, A. Mezzacappa, P. Hoflich, in Fusion Research through Advanced Multiconference on Systemics, K. Nomoto, “Nucleosynthesis and Stellar Computing,” The 3rd IAEA Technical Cybernetics, and Informatics (ISAS Evolution,” Astrophysics and Space Meeting on Control, Data Acquisition, SCI2002), Orlando, Fla., July 2002. Science, 281, 25-37, 2002. and Remote Participation for Fusion B. A. Shadwick, G. M. Tarkenton, E. H. F.-K Thielemann,. D. Argast, F. Research. Fusion Engineering and Design Esarey and W. P. Leemans, “Fluid simula- Brachwitz, G. Martinez-Pinedo, T. 60, 481-486, 2002. tions of intense laser-plasma interactions,” Rauscher, M. Liebendoerfer, A. K. L. Schuchardt, J. D. Myers, E. G. IEEE Trans. Plasma Sci. 30, 38, 2002. Mezzacappa, P. Hoflich, K. Iwamoto, K. Stephan, 2002, “A web-based data architec- Nomoto, “Nucleosynthesis in Supernovae. M. S. Shephard, X. Luo, J. F. Remacle, ture for problem-solving environments: appli- (I),” ASP Conf. Ser. 253: Chemical “Meshing for p-version finite element meth- cation of distributed authoring and versioning Enrichment of Intracluster and ods,” WCCM V: Fifth World Congress on the extensible computational chemistry envi- Intergalactic Medium, p. 205, 2002. Computational Mechanics, Vienna Univ. ronment,” Cluster Computing 5(3):287- of Tech, Vienna, Austria, p. 1-464, 2002. F.-K. Thielemann, P. Hauser, E. Kolbe, G. 296, 2002. Martinez-Pinedo, I. Panov, T. Rauscher, A. Snavely, et al., “A Framework for D. R. Schultz, C. O. Reinhold, P. S. Krstic K.-L. Kratz, B. Pfeiffer, S. Rosswog, M. Application Performance Modeling and and M. R. Strayer, “Ejected-electron spec- Liebendoerfer, A. Mezzacappa, “Heavy Prediction,” Proc. of the SC 2002 confer- trum in low-energy proton-hydrogen colli- Elements and Age Determinations,” Space ence, Baltimore, Md., Nov. 2002. sions,” Physical Review A, 65, 052722, Science Reviews, 100, 277-296, 2002. 2002. P. Spentzouris, J. Amundson, J. Lackey, L. J. L. Tilson, C. Naleway, M. Seth, R. Spentzouris, R. Tomlin, “Space Charge G. Schussman and K.-L. Ma, “Scalable Shepard, A. F. Wagner, and W. C. Ermler, Studies and Comparison with Simulations Self-Orienting Surfaces: A Compact, Texture- “An Ab Initio Study of the f-f Spectroscopy of Using the FNAL Booster,” Proc. of the Enhanced Representation for Interactive Americium+3,” Journal of Chemistry International Computational Accelerator Visualization Of 3D Vector Fields,” Proc. of Physics. 116, 5494-5502, 2002. Physics Conference 2002 (ICAP2002), the Pacific Graphics 2002 Conference, East Lansing, Michigan, Oct. 2002. S. J. Thomas, J. M. Dennis, H. M. Tufo, P. Beijing, China, Oct. 2002. F. Fischer, “A Schwarz preconditioner for the N. Sullivan, A. D. Jensen, P. Glarborg, M. G. Schussman and K.-L. Ma, “Scalable cubed-sphere,” Proc. of the Copper S. Day, J. F. Grcar, J. B. Bell, C. J. Pope, Self-Orienting Surfaces: A Compact Texture- Mountain Conference on Iterative and R. J. Kee, “Ammonia Conversion and Enhanced Representation for Interactive Methods, 2002. NOx Formation in Laminar Coflowing Visualization Of 3d Vector Fields,” Proc. of Nonpremixed Methane-Air Flames,” H. Trease, L. Trease, J. Fowler, R. Corley, the 10th Pacific Conference on Computer Combustion and Flame, 131(3):285, 2002. C. Timchalk, K. Minard, D. Rommeriem, Graphics and Applications, IEEE “A Case Study: Extraction, Image Computer Society, Washington, D.C., Y. Sun, N. Folwell, Z. Li and G. H. Golub, Reconstruction, And Mesh Generation From Oct. 2002. “High Precision Accelerator Cavity Design NMR Volume Image Data From F344 Rats Using the Parallel Eigensolver Omega3p,” S. L. Scott, B. Luethke and T. Naughton, For Computational Biology Applications,” Proc. of 18th Annual Review of Progress “C3 Power Tools – The Next Generation,” 8th International Conference on Grid in Applied Computational 4th DAPSYS Conference, Linz, Austria, Generation and Computational Physics, Electromagnetics, ACES-2002, Monterey, Sept.–Oct. 2002. 2002. Calif., March 2002. S. L. Scott, B. Luethke, “HPC Federated F. S. Tsung., R. G. Hemker, C. Ren, W. B. Y. Sun, G. Golub and K. Ko, “Solving the Cluster Administration with C3 v3.0,” Mori, L. O. Silva and T. Katsouleas, Complex Symmetric Eigenproblem in

94 LIST OF PUBLICATIONS

“Generation of Ultra-Intense, Single-Cycle ray Emission from Betatron Motion in a P. H. Worley, T. H. Dunigan, Jr., M. R. Laser Pulse using Photon Deceleration,” Plasma Wiggler,” Physical Review Letters, Fahey, J. B. White III, A. S. Bland, “Early Proceedings of the National Academy of Vol. 88, No. 13, p. 135004/1-4, April Evaluation of the IBM p690,” Proc. of the Sciences., Vol. 99, pp. 29-32, 2002. 2002. IEEE/ACM SC2002 Conference, Baltimore, Md., Nov. 2002. T. Van Voorhis and M. Head-Gordon, A. Wetzels, A. Gurtler, L. D. Noordam, F. “Implementation of generalized valence-bond Robicheaux, C. Dinu, H. G. Muller, M. J. K. Wu, J. Ekow, E. J. Otoo, and A. inspired coupled cluster theories,” Journal of J. Vrakking and W. J. van der Zande, Shoshani, “Compressing bitmap indexes for Chemical Physics, 117, 9190-9201, 2002. “Rydberg state ionization by half-cycle-pulse faster search operations,” Proc. of excitation: strong kicks create slow electrons,” SSDBM’02, p. 99, Edinburgh, Scotland, J. L. Vay, P. Colella, P.McCorquodale, B. Physical Review Letters, 89, 273003. 2002. Van Straalen, A. Friedman, D.P. Grote, 2002. “Mesh refinement for particle-in-cell plasma M. Zingale, L. J. Dursi, J. ZuHone, A. C. simulations: Applications to and benefits for A. D. Whiteford, N. R. Badnell, C. P. Calder, B. Fryxell, T. Plewa, J. W. Truran, heavy ion fusion,” Laser and Particle Ballance, S. D. Loch, M. G. O’Mullane A. Caceres, K. Olson, P. M. Ricker, K. Beams, 20, 4, p. 569-575, 2002. and H. P. Summers, “Excitation of Ar15+ Riley, R. Rosner, A. Siegel, F. X. Timmes and Fe23+ for diagnostic application to and N. Vladimirova, “Mapping Initial J. S. Vetter and P. Worley, “Asserting fusion and astrophysical plasmas,” Journal of Hydrostatic Models in Godunov Codes,” Performance Expectations,” Proc. of the SC Physics B, 35, 3729, 2002. Astrophysical Journal Supplement, 2002 conference, Baltimore, Md., Nov. 143:539-565, Dec. 2002. 2002. B. Wilson, K.-L. Ma and P. McCormick, “A Hardware-Assisted Hybrid Rendering J. S. Vetter and A. Yoo, “An Empirical Technique for Interactive Volume Performance Evaluation of Scalable Scientific Visualization,” 2002 Symposium on Applications,” Proc. of the SC 2002 con- Volume Visualization and Graphics, ference, Baltimore, Md., Nov. 2002. Boston, Mass., Oct. 2002. J. S. Vetter, “Dynamic Statistical Profiling of B. Wilson, K.-L. Ma, J. Qiang, and R. 2003 Communication Activity in Distributed Ryne, “Interactive Visualization of Particle Applications,” Proc. of SIGMETRICS Beams for Accelerator Design,” In S. I. Abarzhi, J. Glimm, A. D. Lin, 2002: Joint International Conference on Proceedings of the Workshop on Particle “Rayleigh-Taylor instability for fluids with a Measurement and Modeling of Computer Accelerator Science and Technology, finite density contrast,” Phys. Fluids, 15, p. Systems, ACM, 2002. ICCS 2002, Amsterdam, Netherlands, 2190-2197, 2003. J. S. Vetter and F. Mueller, “Communication April 2002. S. I. Abarzhi, J. Glimm, K. Nishihara, Characteristics of Large-Scale Scientific M. Wingate, et al. (The HPQCD “Rayleigh-Taylor instability and Richtmyer- Applications for Contemporary Cluster Collaboration), “Heavy Light Physics Using Meshkov instabilities for fluids with a finite Architectures,” Proc. of the International NRQCD/Staggered Actions,” Nuclear density contrast,” Phys. Lett. A, 11, p. 1-7, Parallel and Distributed Processing Physics B (Proc. Suppl.) 106, 379, 2002. 2003. Symposium, 2002. M. Wolf, A. Guetz and C.-K. Ng, “Modeling M. Adams, M. Brezina, J. Hu, and R. R. Vuduc, J. W. Demmel, K. A. Yelick, S. Large Accelerator Structures with the Tuminaro, “Parallel multigrid smoothing: Kamil, R. Nishtala and B. Lee, Parallel Field Solver Tau3P,” Proc. of 18th polynomial versus Gauss-Seidel,” Journal of “Performance Optimizations and Bounds for Annual Review of Progress in Applied Computational Physics, 188, p. 593-610, Sparse Matrix-Vector Multiply,” Proc. of the Computational Electromagnetics, ACES- 2003. IEEE/ACM SC2002 Conference, 2002, Monterey, Calif., March 2002. Baltimore, Md., Nov. 2002. V. Akcelik, J. Bielak, G. Biros, I. C. S. Woodward, K. E. Grant and R. Epanomeritakis, A. Fernandez, O. R. Vuduc, S. Kamil, J. Hsu, R. Nishtala, J. Maxwell, “Applications of Sensitivity Ghattas, E. J. Kim, D. O’Hallaron and T. W. Demmel and K. A. Yelick, “Automatic Analysis to Uncertainty Quantification for Tu, 2003, “High-resolution Forward and Performance Tuning and Analysis of Sparse Variably Saturated Flow,” Computational Inverse Earthquake Modeling on Terascale Triangular Solve,” ICS 2002: Workshop on Methods in Water Resources, S.M. Computers,” Proc. of the ACM/IEEE Performance Optimization via High- Hassanizadeh, R. J. Schotting, W. G. SC03 conference, Nov. 2003. Level Languages and Libraries, 2002. Gray, and G. F. Pinder, eds., vol.1, pp. 73- C. Alexandrou et al., “Calculation of the N G. Wallace, “Display Wall Cluster 80, Elsevier, Amsterdam, 2002. to Delta electromagnetic transition matrix ele- Management,” Proc. of the IEEE S. E. Woosley, A. Heger, and T. A. ment,” Nuclear Physics B (Proc. Suppl.) Visualization 2002 Workshop on Weaver, “The evolution and explosion of 119, 413, 2003. Commodity-Based Visualization Clusters, massive stars,” Reviews of Modern Oct. 2002. P. R. Amestoy, I. S. Duff, J.-Y. L’Excellent Physics, 74:1015-1071, Nov. 2002. and X. S. Li, “Impact of the Implementation S. Wang, C. E. Clayton, B. E. Blue, E. S. P. H. Worley, “Scaling the Unscalable: A of MPI Point-to-Point Communications on Dodd, K. A. Marsh, W. B. Mori, C. Joshi, Case Study on the AlphaServer SC,” Proc. of the Performance of Two General Sparse P. Muggli, T. Katsouleas, F. J. Decker, M. the IEEE/ACM SC2002 Conference, Solvers,” Parallel Computing, 29:833-849, J. Hogan, R. H. Iverson, P. Raimondi, D. Baltimore, Md., Nov. 2002. 2003. Watz, R. Siemann, and R. Assmann, “X-

95 LIST OF PUBLICATIONS

J. Amundson, J. Lackey, P. Spentzouris, Roundtable, Sandia National S. Bhowmick, L. McInnes, B. Norris, and G. Jungman, L. Spentzouris, “Calibration Laboratories, p. 33-42, 2003. P. Raghavan, 2003, “The Role of Multi- of the Fermilab Booster ionization profile Method Linear Solvers in PDE-based J. B. Bell, M. S. Day, J. F. Grcar, M. J. monitor,” Physics Review ST Accel. Simulations,” Proc. of the 2003 Lijewski, M. Johnson, R. K. Cheng and I. Beams. 6, 102801, 2003. International Conference on G. Shepherd, “Numerical Simulation of a Computational Science and its J. Amundson, P. Spentzouris, “FNAL Premixed Turbulent V-Flame,” 19th Applications, Montreal, Canada, Lecture Booster: Experiment and Modeling,” Proc. of International Colloquium on the Notes in Computer Science, 2677, p. 828- IEEE Particle Accelerator Conference Dynamics of Explosions and Reactive 839, V. Kumar, M. L. Gavrilova, C. J. K. 2003, Portland, Ore., May 2003. Systems, July-Aug. 2003. Tan, and P. L’Ecuyer. eds, May 2003. J. Amundson, P. Spentzouris, “SYNER- J. B. Bell, M. S. Day, A. S. Almgren, R. K. L. Biegler, O. Ghattas, M. Heinkenschloss GIA: A Hybrid, Parallel Beam Dynamics Cheng, and I. G. Shepherd, “Numerical and B. van Bloemen Waanders, editors, Code with 3D Space Charge,” Proc. of Simulation of Premixed Turbulent Methane “PDE-Constrained Optimization: State-of- IEEE Particle Accelerator Conference Combustion,” Proc. of the Second MIT the-Art,” Lecture Notes in Computational 2003, Portland, Ore., May 2003. Conference on Computational Fluid and Science and Engineering, 30, Springer- Solid Mechanics, June 2003. C. Aubin, et al. (The MILC Verlag, 2003. Collaboration), “Chiral logs with staggered J. B. Bell, M. S. Day, J. F. Grcar, and M. J. J, M. Blondin, “The Inherent Asymmetry of fermions,” Nuclear Physics B (Proc. Lijewski, “Analysis of carbon chemistry in Core-Collapse Supernovae,” ASP Conf. Ser. Suppl.), 119, 233, 2003. numerical simulations of vortex flame interac- 293: 3D Stellar Evolution, p. 290, 2003. tions,” 19th International Colloquium on N. R. Badnell, D. C. Griffin, and D. M. the Dynamics of Explosions and Reactive J. M. Blondin, A. Mezzacappa, C. Mitnik, “Electron-impact excitation of B+ Systems, July-Aug. 2003. DeMarino, “Stability of Standing Accretion using the R-matrix with pseudo-states Shocks, with an Eye toward Core-Collapse method,” Journal of Physics B, 1337 2003. C. Bernard, et al. (The MILC Supernovae,” Astrophysical Journal Collaboration), “Topological Susceptibility N. R. Badnell, M. G. O’Mullane, H. P. Letters, 584, 971-980, 2003. with the Improved Asqtad Action,” Phys. Summers, Z. Altun, M. A. Bautista, J. P. Rev. D 68, 114501, 2003. B. E. Blue et al., “Plasma-Wakefield Colgan, T. W. Gorczyca, D. M. Mitnik, Acceleration of an Intense Positron Beam,” M. S. Pindzola, and O. Zatsarinny, C. Bernard, et al. (The MILC Phys. Rev. Lett. 90, 214801, 2003. “Dielectronic recombination data for dynamic Collaboration), “Light hadron properties finite-density plasmas: I. Goals and methodol- with improved staggered quarks,” Nuclear P. A. Boyle, C. Jung and T. Wettig, “The ogy,” Astronomy and Astrophysics, 406, Physics B (Proc. Suppl.), 119, 257, 2003. QCDOC supercomputer: Hardware, soft- 1151, 2003. ware, and performance,” Econf C0303241, C. Bernard, et al. (The MILC THIT003, 2003. A. B. Balantekin, G. M. Fuller, “Supernova Collaboration), “Topological susceptibility neutrino nucleus astrophysics,” Journal of with the improved Asqtad action,” Nuclear M. Branstetter, and D. J. Erickson III, Physics G, 29, 2513-2522, 2003. Physics B (Proc. Suppl.), 119, 991, 2003. “Continental runoff dynamics in the CCSM2.0,” Journal of Geophysical C. P. Ballance, N. R. Badnell and E. S. C. Bernard, et al. (The MILC Research 108(D17), 4550, 2003. Symth, “A pseudo-state sensitivity study on Collaboration), “Static hybrid quarkonium hydrogenic ions,” Journal of Physics B, 36. potential with improved staggered quarks,” D. P. Brennan, R. J. La Haye, A. D. 3707. 2003. Nuclear Physics B (Proc. Suppl.), 119, Turnbull, et al., “A mechanism for tearing 598, 2003. onset near ideal stability boundaries,” C. P. Ballance, N. R. Badnell, D. C. Physics of Plasmas 10 1643, 2003. Griffin, S. D. Loch and D. M. Mitnik, C. Bernard, et al. (The MILC “The effects of coupling to the target continu- Collaboration), “High temperature QCD J. A. Breslau, S. C. Jardin and W. Park, um on the electronimpact excitation of Li+,” with three flavors of improved staggered “Simulation studies of the role of reconnection Journal of Physics B, 36, 235, 2003. quarks,” Nuclear Physics B (Proc. Suppl.), in the ‘current hole’ experiments in the joint 119, 523, 2003. European torus,” Physics of Plasmas 10 G. T. Balls, S. B. Baden, P. Colella, 1665, 2003. “SCALLOP: A highly scalable parallel C. Bernard, et al. (The MILC Poisson solver in three dimensions,” Proc. of Collaboration), “Exotic hybrid mesons from M. Brewer, L. Diachin, T. Leurent, D. ACM/IEEE SC03 conference, Phoenix, improved Kogut-Susskind fermions,” Nuclear Melander, “The Mesquite Mesh Quality Arizona, November, 2003. Physics B (Proc. Suppl.), 119, 260, 2003. Improvement Toolkit,” Proc. of the 12th International Meshing Roundtable, Santa M. W. Beall, J. Walsh, M. S. Shephard, “A C. Bernard, et al. (The MILC Fe NM, p. 239-250, 2003. comparison of techniques for geometry access Collaboration), “Heavy–light meson decay related to mesh generation,” Proc. 7th US constants with Nf =3,” Nuclear Physics B P. N. Brown, B. Lee, T. A. Manteuffel, “A Nat. Congress on Comp. Mech., p. 62, (Proc. Suppl.), 119, 613, 2003. moment-parity multigrid preconditioner for 2003. the first-order least-squares formulation of the C. Bernard, et al. (The MILC Collabora- Blotzmann transport equations,” SIAM M. W. Beall, J. Walsh, M. S. Shephard., tion), “Lattice calculation of 1?+ hybrid Journal of Scientific Computing, 25, 2, p. “Accessing CAD geometry for mesh genera- mesons with improved Kogut-Susskind fermi- 513-533, 2003. tion,” 12th International Meshing ons,” Physical Review D 68, 074505, 2003.

96 LIST OF PUBLICATIONS

P. N. Brown, P. Vassilevski, and C. S. Fusion Science and Technology 43(1), face ocean nitrogen, iron, sulfur cycling,” Woodward, “On Mesh-Independent 2003. Chemosphere, 50, 223, 2003. Convergence of an Inexact Newton-Multigrid J. R. Cary, R. E. Giacone, C. Nieter, D. L. S. Chu, S. Elliott, M. Maltrud, F. Chai and Algorithm,” SIAM Journal of Scientific Bruhwiler, E. Esarey, G. Fubiani and W. F. Chavez, “Studies of the consequences of Computing, 25, 2, p. 570-590, 2003. P. Leemans, “All-Optical Beamlet Train ocean carbon sequestration using Los Alamos D. L. Bruhwiler, D. A. Dimitrov, J. R. Generation,” Proc. of IEEE Particle ocean models,” Proceedings of the DOE Cary, E. Esarey, W. P. Leemans and R. E. Accelerator Conference 2003, Portland, 2nd Annual Conference on Carbon Giacone, “Particle-in-cell simulations of tun- Ore., May 2003. Sequestration, Alexandria, Va., May 2003. neling ionization effects in plasma-based G. K. L. Chan and M. Head-Gordon, D. L. Cler, N. Chevaugeon, M. S. accelerators,” Physics of Plasmas 10, p. “Exact solution (in a triple zeta, double Shephard, J. E. Flaherty, J. F. Remacle, 2022, 2003. polarization basis) of the Schrödinger elec- “CFD application to gun muzzle blast - A D. L. Bruhwiler, D. A. Dimitrov, J. R. tronic equation for water,” Journal of validation case study,” 41st AIAA Cary, E. Esarey and W. P. Leemans, Chemical Physics, 118, 8851-8854, 2003. Aerospace Sciences Meeting and Exhibit, “Simulation of ionization effects for high-den- Jan. 2003. B. Cheng, J. Glimm, H. Jin, D. H. Sharp, sity positron drivers in future plasma wake- “Theoretical methods for the determination of A. Codd, T. Manteuffel and S. field experiments,” Proc. of IEEE Particle mixing,” Laser and Particle Beams, 21, p. McCormick, “Multilevel first-order system Accelerator Conference 2003, Portland, 429-436, 2003. least squares for nonlinear elliptic partial dif- Ore., May 2003. ferential equations,” SIAM Journal on N. Chevaugeon, J. E. Flaherty, L. N. Bucciantini, J. M. Blondin, L. Del Numerical Analysis, 41, p. 2197-2209, Krivodonova, J. F. Remacle, M. S. Zanna, E. Amato, “Spherically symmetric 2003. Shephard, “Adaptive and parallel discontin- relativistic MHD simulations of pulsar wind uous Galerkin method for hyperbolic conser- A. Codd, T. Manteuffel, S, McCormick nebulae in supernova remnants,” Astronomy vation law,” Proc. 7th US Nat. Congress and J. Ruge, “Multilevel first-order system & Astrophysics, 405, 617-626, 2003. on Comp. Mech., p. 448, 2003. least squares for elliptic grid generation,” T. Burch and D. Toussaint (The MILC SIAM Journal on Numerical Analysis, 41, N. Chevaugeon, J. F. Remacle, J. E. Collaboration), “Hybrid configuration con- pp. 2210-2232, 2003. Flaherty, D. Cler, M. S. Shephard, tent of heavy S-wave mesons,” Physical “Application of discontinuous Galerkin J. P. Colgan, M. S. Pindzola, A. D. Review D, 68, 094504, 2003. method. Large scale problem: Cannon blast Whiteford, and N. R. Badnell, R. Butler, D. Narayan, A. Lusk and E. modeling,” Proc. 7th US Nat. Congress on “Dielectronic recombination data for dynamic Lusk, “The process management component of Comp. Mech., p. 470, 2003. finite-density plasmas: III. The Beryllium iso- a scalable system software environment,” electronic sequence,” Astronomy and Y. Choi, S. Elliott, D. R. Blake, F. S. Proc. of the 5th IEEE International Astrophysics, 412, 597, 2003. Rowland, et al., “PCA survey of whole air Conference on Cluster Computing hydrocarbon data from the second airborne J. P. Colgan and M. S. Pindzola, “Total and (CLUSTER03), Kowloon, Hong Kong, BIBLE expedition,” Journal of Geophysical differential cross section calculations for the Dec. 2003. Research, 108, 10, 2003. double photoionization of the helium 1x2s1,3 L. Bytautas, J. Ivanic, K. Ruedenberg, S states,” Physical Review A, 67, 012711, T. Chartier, R. D. Falgout, V. E. Henson, “Split-Localized Orbitals Can Yield Stronger 2003. J. Jones, T. Manteuffel, S. McCormick, J. Configuration Interaction Convergence than Ruge and P. S. Vassilevski, “Spectral J. P. Colgan, M. S. Pindzola, and F. J. Natural Orbitals,” Journal of Chemical AMGe (ÚAMGe),” SIAM Journal of Robicheaux, “Time-dependent studies of sin- Physics, 119, 8217, 2003. Scientific Computing, 25, p. 1-26, 2003. gle and multiple photoionization of H2+,” C. Cardall, A. Mezzacappa, “Conservative Physical Review A, 68, 063413, 2003. E. Chow, “An Unstructured Multigrid formulations of general relativistic kinetic the- Method Based on Geometric Smoothness,” J. P. Colgan, M. S. Pindzola, and M. C. ory,” Physical Review D, 68, 023006, Numer. Linear Algebra Appl., 10, p. 401- Witthoeft, “Time-dependent electron-impact 2003. 421, 2003. scattering from He+ using variable lattice L. Carrington, A. Snavely, N. Wolter, and spacings,” Physical Review A, 032713, E. Chow, T. A. Manteuffel, C. Tong and X. Gao, “A Performance Prediction 2003. B. K. Wallin, “Algebraic Elimination of Slide Framework for Scientific Applications,” Surface Constraints in Implicit Structural J. P. Colgan, S. D. Loch, M. S. Pindzola, ICCS Workshop on Performance Analysis,” Intern. Journal on Numerical C. P. Ballance, and D. C. Griffin, Modeling and Analysis (PMA03), Analysis,. Meth. Engrg., 57, p. 1129-1144, “Electron-impact ionization of all ionization Melbourne, Australia, June 2003. 2003. stages of Beryllium,” Physical Review A, M. D. Carter, F. W. Baity Jr., G. C. Barber, 032712, 2003. E. Chow and P. Vassilevski, “Multilevel R. H. Goulding, Y. Mori, D. O. Sparks, K. Block Factorizations in Generalized D. Datta, C. Picu, M. S. Shephard, F. White, E. F. Jaeger, F. R. Chang-Diaz Hierarachical Bases,” Numer. Linear “Multiscale simulation of defects using multi- and J. P. Squire, “Comparing Experiments Algebra Appl., 10, p. 105-127, 2003. grid atomistic continuum method,” Proc. 7th with Modeling to Optimize Light Ion US Nat. Congress on Comp. Mech., p. Helicon Plasma Sources for VASIMR,” Proc. S. Chu, S. M. Elliott, and M. E. Maltrud, 814, 2003. of Open Systems, Korea, Transactions of “Global eddy permitting simulations of sur-

97 LIST OF PUBLICATIONS

C. Davies, et al. (The HPQCD R. J. Dumont, C. K. Phillips, and D. N. P. Fischer, F. Hecht, Y. Maday, “A Parareal Collaboration), “The Determination of Smithe, “Effects of Non-Maxwellian Plasma in Time Semi-implicit Approximation of the Alpha(s) from Lattice QCD with 2+1 Species on ICRF Propagation and Navier-Stokes Equations,” Proc. Of the 15th Flavors of Dynamical Quarks,” Nuclear Absorption,” Radio Frequency Power in Int. Conf. on Domain Decomposition Physics B (Proc. Suppl.), 119, 595, 2003. Plasmas, Proc. 15th Topical Conf., Moran, Methods, Berlin, 2003. Wyoming, 2003, AIP, N. Y., 2003. J. Demmel and Y. Hida, “Accurate Floating P. F. Fischer, J. W. Lottes, “Hybrid Point Summation,” SIAM Journal of T. H. Dunigan, Jr., M. R. Fahey, J. B. Schwarz-Multigrid Methods for the Spectral Scientific Computing, 25, 4, p. White III, P. H. Worley, “Early Evaluation Element Method: Extensions to Navier- 1214–1248, 2003. of the Cray X1,” Proc. of the ACM/IEEE Stokes,” Proc. Of the 15th Int. Conf. on SC03 conference, Phoenix, Ariz., Nov. Domain Decomposition Methods, Berlin, S. Dey, L. Couchman, D. Datta, “Error 2003. 2003. and convergence of p-refinement schemes for elastodynamics and coupled elasto-acoustics,” L. J. Dursi, M. Zingale, A. C. Calder, B. N. Folwell, P. Knupp “Increasing Tau3P Proc. 7th US Nat. Congress on Comp. Fryxell, F. X. Timmes, N. Vladimirova, R. Abort-Time via Mesh Quality Improvement,” Mech., , p. 517, 2003. Rosner, A. Caceres, D. Q. Lamb, K. Proceedings of the 12th International Olson, P. M. Ricker, K. Riley, A. Siegel, Meshing Roundtable, Santa Fe NM, p. M. di Pierro et al., “Charmonium with three and J. W. Truran, “The Response of Model 379-392, 2003. flavors of dynamical quarks,” Nuclear and Astrophysical Thermonuclear Flames to Physics B (Proc. Suppl.) 119, 586, 2003. N. Folwell, P. Knupp and M. Brewer, Curvature and Stretch,” ApJ, 595:955-979, “Increasing the TAU3P Abort-time via Mesh J. Dongarra, K. London, S. Moore, P. Oct. 2003. Quality Improvement,” Proc. of the 12th Mucci, D. Terpstra, H. You, and M. Zhou, S. Dutta, E. George, J. Glimm, X. L. Li, A. International Meshing Roundtable, Sandia “Experiences and Lessons Learned with a Marchese, Z. L. Xu, Y. M. Zhang, J. W. National Laboratories, Santa Fe, NM, Portable Interface to Hardware Performance Grove, D. H. Sharp, “Numerical methods Sept. 2003. Counters,” PADTAD Workshop, IPDPS for the determination of mixing,” Laser and 2003, Nice, France, April 2003. N. Folwell, P. Knupp, K. Ko, “Mesh Particle Beams, 21, p. 437-442, 2003. Quality Improvement for Accelerator Design,” J. Dongarra and V. Eijkhout, “Self-adapting S. Dutta, J. Glimm, J. W. Grove, D. H. SIAM Conference on Computational Numerical Software and Automatic Tuning Sharp, Y. Zhang, “A fast algorithm for mov- Science and Engineering, San Diego, of Heuristics,” ICCS 2003, Workshop on ing interface problems,” International Calif., Feb. 2003. Adaptive Algorithms, Melbourne, Conference on Computational Science Australia, June 2003. C. L. Fryer and P. Meszaros, “Neutrino- and Its Applications (ICCSA 2003), driven Explosions in Gamma-Ray Bursts J. Dongarra, A. Malony, S. Moore, P. Lecture Notes in Computer Science 2668, and Hypernovae,” ApJ Letters, 588:L25- Mucci, and S. Shende, “Performance V. Kumar et al.., ed., Springer-Verlag, L28, May 2003. Instrumentation and Measurement for Berlin/Heidelberg, p. 782-790, 2003. Terascale Systems,” ICCS 2003, Terascale G. Fubiani, E. Esarey, W. Leemans, J. S. Elliott and S. Chu, “Comment on ‘ocean Systems Workshop, Melbourne, Australia, Qiang, R. D. Ryne and G. Dugan, fertilization experiments initiate large scale June 2003. “Numerical Studies of Laser-Plasma phytoplankton blooms,’” Geophysical Produced Electron Beam Transport,” Proc. of P. Dreher, et al. (The SESAM Research Letters, 30, 1274, 2003. IEEE Particle Accelerator Conference Collaboration), “Continuum extrapolation of S. Elliott, and H. Hanson, “Syndication of 2003, Portland, Ore., May 2003. moments of nucleon quark distributions in full the Earth system,” Environmental Science QCD,” Nuclear Physics B (Proc. Suppl.), G. M. Fuller, A. Kusenko, I. Mocioiu, S. and Policy, 6, 457, 2003. 119, 392, 2003. Pascoli, “Pulsar kicks from a dark-matter S. Elliott, D. R. Blake, F. S. Rowland, et sterile neutrino,” Physical Review D, 68, M. Dryja and O. Widlund, “A Generalized al., “Whole air sampling as a window on 103002, 2003. FETI-DP Method for a Mortar Pacific biogeochemistry,” Journal of Discretization of Elliptic Problems,” Proc of E. Gabriel, G. Fagg and J. Dongarra, Geophysical Research, 108(D3), 8407, the 14th International Symposium on “Evaluating the Performance of MPI-2 2003. Domain Decomposition Methods, Dynamic Communicators and One-sided Cocoyoc, Mexico, Jan. 2002, I. Herrera, D. J. Erickson III, J. Hernandez, P. Communication,” EuroPVM/MPI 2003, D. E. Keyes, O. B. Widlund and R. Yates Ginoux, W. Gregg, C. McClain, and J. Venice, Italy, Sept. 2003. eds., 2003. Christian, “Atmospheric Iron Delivery and Z. Gan, Y. Alexeev, R. A. Kendall, and M. surface ocean biological activity in the R. J. Dumont, C. K. Phillips, A. Pletzer S. Gordon, “The Parallel Implementation of Southern Ocean and Patagonian region,” and D.N. Smithe, “Effects of Non- a Full CI Program,” Journal of Chemical Geophysical Research Letter, 30(12), Maxwellian Plasma Species on ICRF Wave Physics, 119, 47, 2003. 1609, 2003. Propagation and Absorption in Toroidal X. Gao and A. Snavely, “Exploiting Magnetic Confinement Devices,” invited J. Fetter, G. C. McLaughlin, A. B. Stability to Reduce Time-Space Cost for paper in the 10th European Fusion Balantekin, G. M. Fuller, “Active-sterile neu- Memory Tracing,” ICCS Workshop on Theory Conference, Helsinki, Finland, trino conversion: consequences for the r- Performance Modeling and Analysis Sept. 2003. process and supernova neutrino detection,” (PMA03), Melbourne, Australia, June Astroparticle Physics, 18, 433-448, 2003. 2003.

98 LIST OF PUBLICATIONS

E. George, J. Glimm, J. W. Grove, X. L. Computational Fluid and Solid Sampaio, K. Langanke, D. J. Dean, G. Li, Y. J. Liu, Z. L. Xu, N. Zhao, Mechanics, 1:1951-1955, K. J. Bathe, ed., Martinez-Pinedo, “Consequences of Nuclear “Simplification, conservation and adaptivity Elsevier, 2003. Electron Capture in Core Collapse in the front tracking method,” in Hyperbolic Supernovae,” Physical Review Letters, 91, A. Gray, et al. (The HPQCD Problems: Theory, Numerics, 201102, 2003. Collaboration), “The Upsilon Spectrum from Applications, T. Hou, E. Tadmor, eds., Lattice QCD with 2+1 Flavors of W. R. Hix, A. Mezzacappa, O. E. B. Springer Verlag, Berlin and New York, p. Dynamical Quarks,” Nuclear Physics B Messer, S. W. Bruenn, “Supernova science 175-184200, 2003. (Proc. Suppl.), 119, 592, 2003. at spallation neutron sources,” Journal of F. X. Giraldo, J. B. Perot and P. F. Fischer, Physics G, 29, 2523-2542, 2003. J. F. Grcar, M. S. Day and J. B. Bell, “A spectral element semi-Lagrangian (SESL) “Conditional and opposed reaction path dia- I. Hofmann, G. Franchetti, O. Boine- method for the spherical shallow water equa- grams for the analysis of fluid-chemistry Frankenheim, J. Qiang, and R. D. Ryne, tions,” J. of Comp. Phys., 190 2, p. 623- interactions,” 19th International “Space-Charge Resonances in Two- and 650, 2003. Colloquium on the Dynamics of Three-Dimensional Anisotropic Beams,” J. Glimm, S. Hou, Y. Lee, D. Sharp, K. Ye, Explosions and Reactive Systems, July- Physics Rev. ST Accel. Beams, 6, 024202, “Solution error models for uncertainty quan- Aug. 2003. 2003. tification,” Contemporary Mathematics, D. C. Griffin, J. P. Colgan, S. D. Loch, M. I. Hofmann, G. Franchetti, J. Qiang and 327, p. 115-140, 2003. S. Pindzola, and C. P. Ballance “Electron- R. D. Ryne, “Self-Consistency and Coherent J. Glimm, H. Jin, M. Laforest, F. impact excitation of beryllium and its ions,” Effects in Nonlinear Resonances,” Proc. of Tangerman, Y. Zhang, “A two pressure Physical Review A, 68, 062705, 2003. the 29th ICFA Advanced Beam numerical model of two fluid mixing,” SIAM Dynamics Workshop on Beam Halo A. T. Hagler, et al. (The LHPC J. Multiscale Model. Simul., 1, p. 458-484, Dynamics, Diagnostics, and Collimation Collaboration), “Moments of nucleon gener- 2003. (HALO’03), Montauk, NY, May 2003. alized parton distributions in lattice QCD,” J. Glimm, Y. Lee, D. H. Sharp, K. Q. Ye, Physical Review D 68, 034505, 2003. M. J. Hogan et al., “Ultrarelativistic- “Prediction using numerical simulations, A Positron-Beam Transport through Meter-Scale J. C. Hayes, M. L. Norman, “Beyond Flux- Bayesian framework for uncertainty quantifi- Plasmas,” Phys. Rev. Lett. 90, 205002, limited Diffusion: Parallel Algorithms for cation and its statistical challenge,” in 2003. Multidimensional Radiation Proceedings of the Fourth International Hydrodynamics,” Astrophysical Journal P. Hovland, K. Keahey, L. C. McInnes, B. Symposium on Uncertainty Modeling Supplement Series, 147, 197-220, 2003. Norris, L. Freitag and P. Raghavan, “A and Analysis (ISUMA 2003), B. M. Quality-of-Service Architecture for High- Ayyub, N. O. Attoh-Okine, eds., IEEE M. Head-Gordon, T. Van Voorhis, G. Performance Numerical Components,” Computer Society, 2003. Beran and B. Dunietz, “Local Correlation Proceedings of QoSCBSE2003, Toulouse, Models,” Computational Science: ICCS J. Glimm, X. L. Li, Y. J. Liu, Z. L. Xu, N. France, June 2003. 2003, Part IV, Lecture Notes in Zhao, “Conservative front tracking with Computer Science, Vol. 2660, p. 96-102, C. Hsu, “Effects of Block Size on the Block improved accuracy,” SIAM J. Numerical 2003. Lanczos Algorithm,” Senior Thesis, U.C. Analysis, 41, p. 1926-1947, 2003. Berkeley Dept. of Mathematics, June A. Heger, C. L. Fryer, S. E. Woosley, N. J. Glimm, X. L. Li, W. Oh, A. Marchese, 2003. Langer, and D. H. Hartmann, “How M.-N. Kim, R. Samulyak, C. Tzanos, “Jet Massive Single Stars End Their Life,” ApJ, P. Hu, N. Chevaugeon, J. E. Flaherty, M. breakup and spray formation in a diesel 591:288-300, July 2003. S. Shephard, “Modelizations of moving engine,” Proc. of the Second MIT object in a fluid flow using level set,” Proc. Conference on Computational Fluid and E. D. Held, “Unified form for parallel ion 7th US Nat. Congress on Comp. Mech., Solid Mechanics, Cambridge, Mass., 2003. viscocus stress in magnetized plasmas,” p. 288, 2003. Physics of Plasmas, 10, 4708, 2003. P. Goldfeld, 2003, “Balancing Neumann- A. L. Hungerford, C. L. Fryer, and M. S. Neumann for (In)Compressible Linear E. D. Held, J. D. Callen, C. C. Hegna, and Warren, “Gamma-Ray Lines from Elasticity and (Generalized) Stokes - Parallel C. R. Sovinec, “Conductive electron heat Asymmetric Supernovae,” ApJ, 594:390- Implementation,” Proc of the 14th flow along an inhomogeneous magnetic field,” 403, Sept. 2003. International Symposium on Domain Physics of Plasmas 10, 3933, 2003. Decomposition Methods, Cocoyoc, M. Ikegami, et al., “Beam Commissioning of B. Hientzsch, 2003, “Fast Solvers and Mexico, Jan. 2002, I. Herrera, D. E. Keyes, the J-PARC Linac Medium Energy Beam Schwarz Preconditioners for Spectral Nedelec O. B. Widlund and R. Yates, eds., 2003. Transport at KEK,” Proc. of IEEE Particle Elements for a Model Problem in H(curl),” Accelerator Conference 2003, Portland, P. Goldfeld, L. F. Pavarino and O. B. Proc of the 14th International Ore., May 2003. Widlund, “Balancing Neumann-Neumann Symposium on Domain Decomposition Preconditioners for Mixed Approximations of Methods, Cocoyoc, Mexico, Jan. 2002, I. M. Ikegami, T. Kato, Z. Igarashi, A. Ueno, Heterogenous Problems in Linear Elasticity,” Herrera, D. E. Keyes, O. B. Widlund and Y. Kondo, J. Qiang, and R. Ryne, Numer. Math., 95, 283-324, 2003. R. Yates, eds., 2003. “Comparison of Particle Simulation with J- PARC Linac MEBT Beam Test Results,” D. A. Goussis, M. Valorani, F. Creta, and W. R. Hix, O. E. B. Messer, A. Proc. of the 29th ICFA Advanced Beam H. N. Najm, “Inertial Manifolds with CSP,” Mezzacappa, M. Liebendoerfer, J. Dynamics Workshop on Beam Halo

99 LIST OF PUBLICATIONS

Dynamics, Diagnostics, and Collimation 2003, Portland, Ore., May 2003. 25, p. 682-700, 2003. (HALO’03), Montauk, NY, May 2003. G. C. Jordan, S. S. Gupta, B.S. Meyer, S. Lefantzi and J. Ray, “A Component-based J. Ivanic and K. Ruedenberg, “A M C S C F “Nuclear reactions important in alpha-rich Scientific Toolkit for Reacting Flows,” Method for Ground and Excited States freeze-outs,” Physical Review C, 68, Computational Fluid and Solid Based on Full Optimizations of Successive 065801, 2003. Mechanics, 2:1401-1405, K. J. Bathe, Ed., Jacobi Rotations,” Journal of Elsevier, 2003. A. Kabel, “Maxwell-Lorentz Equations in Computational Chemistry, 24, 1250-1262, General Frenet-Serret Coordinates,” Proc. of S. Lefantzi, J. Ray and H. Najm, “Using 2003. IEEE Particle Accelerator Conference the Common Component Architecture to V. Ivanov, A. Krasnykh, G. Scheitrum, D. 2003, Portland, Ore., May 2003. Design High Performance Scientific Sprehn, L. Ives, G. Miram, “3D Modeling Simulation Codes,” Proc. of the Parallel and A. Kabel, “Parallel Simulation Algorithms Activity for Novel High Power Electron Distributed Processing Symposium, June for the Three-dimensional Strong-Strong Guns at SLAC,” Proc. of IEEE Particle 2003. Beam-Beam Interaction,” Proc. of IEEE Accelerator Conference 2003, Portland, Particle Accelerator Conference 2003, J. Li, “A Dual-Primal FETI Method for Ore., May 2003. Portland, Ore., May 2003. Solving Stokes/Navier-Stokes Equations,” V. Ivanov, C. Adolphsen, N. Folwell, L. Proc of the 14th International A. Kabel, “Particle Tracking and Bunch Ge, A. Guetz, Z. Li, C.-K. Ng, G. Symposium on Domain Decomposition Population in TRAFIC4 2.0,” Proc. of Schussman, J.W. Wang, M. Weiner, M. Methods, Cocoyoc, Mexico, Jan. 2002, I. IEEE Particle Accelerator Conference Wolf, K. Ko, “Simulating Accelerator Herrera, D. E. Keyes, O. B. Widlund and 2003, Portland, Ore., May 2003. Structure Operation at High Power,” Proc. R. Yates, eds., 2003. of IEEE Particle Accelerator Conference A. Kabel, Y. Cai, B. Erdelyi, T. Sen, and L. Li, J. Glimm, X. L. Li, “All isomorphic 2003, Portland, Ore., May 2003. M. Xiao, “A Parallel Code for Lifetime distinct cases for multi-component interfaces Simulations in Hadron Storage Rings in the V. Ivanov, C. Adolphsen, N. Folwell et al., in a block,” J. Comp. Appl. Mathematics, Presence of Parasitic Weak-strong Beam-beam “Dark Current Simulation and Rise Time 152, p. 263-276, 2003. Interactions,” Proc. of IEEE Particle Effects at High Power,” Workshop on High Accelerator Conference 2003, Portland, X. S. Li and J. W. Demmel, Gradient RF, Chicago, Ill., Oct. 2003. Ore., May 2003. “SuperLU_DIST: A Scalable Distributed- E. F. Jaeger, L. A. Berry, J. R. Myra, D. B. memory Sparse Direct Solver for K. Keahey, “Fine Grain Authorization Batchelor, E. D’Azevedo, P. T. Bonoli, C. Unsymmetric Linear Systems,” ACM Policies in the Grid: Design and K. Phillips, D. N. Smithe, D. A. Transactions on Math. Software, 29:110- Implementation,” Proc. of the Middleware D’Ippolito, M. D. Carter, R. J. Dumont, J. 140, 2003. 2003 Conference, 2003. C. Wright, R. W. Harvey, “Sheared X. Li, M. S. Shephard, M. W. Beall, Poloidal Flow Driven by Mode Conversion in E. Kim, J. Bielak, and O. Ghattas, “Large- “General mesh adaptation using mesh modifi- Tokamak Plasmas,” Physical Review scale Northridge Earthquake simulation cations,” Proc. 7th US Nat. Congress on Letters, 90, 195001, 2003. using octree-based multiresolution mesh Comp. Mech., p. 17, 2003. method,” Proc. of 16th ASCE Engineering A. Jensen, V. Ivanov, A. Krasnykh, G. Mechanics Conference, Seattle, Wash., X. Li, M. S. Shephard, M. W. Beall, Scheitrum, “An Improved Version of July 2003. “Accounting for curved domains in mesh TOPAZ3D,” Proc. of IEEE Particle adaptation,” International Journal for Accelerator Conference 2003, Portland, L. Krivodonova, J. E. Flaherty, “Error esti- Numerical Methods in Engineering, 58, Ore., May 2003. mation for discontinuous Galerkin solutions of p. 246-276, 2003. multidimensional hyperbolic problems,” N. A. C. Johnston, F. S. Rowland, S. Advances in Computational Mathematics, Z. Li, C. Adolphsen, D. L. Burke, V. Elliott, K. S. Lackner, et al., “Chemical 19, 57, 2003. Dolgashev, R. M. Jones, R. H. Miller, C. transport modeling of CO2 sinks,” Energy Nantista, C.-K. Ng, T. O. Raubenheimer, Conversion and Management, 44, 681, K. Langanke, D. J. Dean, W. Nazarewicz, R. D. Ruth, P. T. Tenenbaum, J. W. Wang, 2003. “Shell model Monte Carlo studies of nuclei in N. Toge, T. Higo, “Optimization of the X- the A~80 mass region,” Nuclear Physics A, J. E. Jones, P. S. Vassilevski, and C. S. band Structure for the JLC/NLC,” Proc. of 728, 109-117, 2003. Woodward, “Nonlinear Schwarz-FAS IEEE Particle Accelerator Conference Methods for Unstructured Finite Element K. Langanke, G. Martinez-Pinedo, J. M. 2003, Portland, Ore., May 2003. Problems,” Proc. of the Second M.I.T. Sampaio, D. J. Dean, W. R. Hix, O. E. B. Z. Li, R. Johnson, S. R. Smith, T. Naito, J. Conference on Computational Fluid and Messer, A. Mezzacappa, M. Liebendorfer, Rifkin, “Cavity BPM with Dipole-mode- Solid Mechanics, Cambridge, Mass., June H.-T. Janka, M. Rampp, “Electron Capture selective coupler,” Proc. of IEEE Particle 2003. Rates on Nuclei and Implications for Stellar Accelerator Conference 2003, Portland, Core Collapse,” Physical Review Letters, R. M. Jones, Z. Li, R. H. Miller, T. O. Ore., May 2003. 90, 241102, 2003. Raubenheimer, “Optimized Wakefield M. Liebendoerfer, A. Mezzacappa, O. E. Suppression & Emittance Dilution-Imposed B. S. Lee McCormick, B. Philip, and D. B. Messer, G. Martinez-Pinedo, W. R. Alignment Tolerances in X-Band Accelerating Quinlan, “Asynchronous fast adaptive com- Hix, F.-K Thielemann, “The neutrino signal Structures for the JLC/NLC,” Proc. of posite-grid methods: numerical results,” in stellar core collapse and postbounce evolu- IEEE Particle Accelerator Conference SIAM Journal of Scientific Computing,

100 LIST OF PUBLICATIONS

tion,” Nuclear Physics A, 719, 144-152, mates for first-order system least squares supernovae,” Nuclear Physics, A. 718, 449- 2003. (FOSLS),” Journal on Numerical 451, 2003. Analysis, Math., 11, p. 163-177, 2003. Y. Lin et al., “Ion Cyclotron Range of O. E. B. Messer, M. Liebendorfer, W. R. Frequencies Mode Conversion Electron J. Marathe, F. Mueller, T. Mohan, B. R. de Hix, A. Mezzacappa, S. W. Bruenn, “The Heating in Deuterium-Hydrogen Plasmas in Supinski, S.A. McKee and A. Yoo, “MET- Impact of Improved Weak Interaction Physics the Alcator C-Mod Tokamak,” Plasma Phys. RIC: Tracking Down Inefficiencies in the in Core-Collapse Supernova Simulations,” Control. Fusion 45, (6) 1013, 2003. Memory Hierarchy Via Binary Rewriting,” From Twilight to Highlight: The Physics International Symposium on Code of Supernovae, W. Hillebrandt, D. Y. Lin, S. J. Wukitch, P. T. Bonoli, A. Generation and Optimization 2003, Leibundgut, eds., p. 70, Springer-Verlag, Mazurenko, E. Nelson-Melby, M. San Francisco, Calif., March 2003. 2003. Porkolab, J. C. Wright, I. H. Hutchinson, E. S. Marmar, D. Mossessian, S. Wolfe, C. E. S. Marmar, B. Bai, R. L. Boivin, P. T. B. S. Meyer, “Neutrinos, supernovae, molyb- K. Phillips, G. Schilling, and P. Phillips, Bonoli, C. Boswell, R. Bravenec, B. denum, and extinct 92Nb,” Nuclear Physics “Study of Ion Cyclotron Range of Frequencies Carreras, D. Ernst, C. Fiore, S. A, 719, 13-20, 2003. in the Alcator C-Mod Tokamak,” Radio Gangadhara, K. Gentle, J. Goetz, R. A. Mezzacappa, “The Road to Prediction: Frequency Power in Plasmas, Proc. 15th Granetz, M. Greenwald, K. Hallatschek, Toward Gravitational Wave Signatures from Topical Conf., Moran, Wyoming, 2003. J. Hastie, J. Hosea, A. Hubbard, J. W. Core Collapse Supernovae,” AIP Conf. Proc. Hughes, I. Hutchinson, Y. In, J. Irby, T. S. D. Loch, J. P. Colgan, and M. S. 686: The Astrophysics of Gravitational Jennings, D. Kopon, G. Kramer, B. Pindzola, M. Westermann, F. Wave Sources, p. 29-39, 2003. LaBombard, W. D. Lee, Y. Lin, B. Scheuermann, K. Aichele, D. Lipschultz, J. Liptac, A. Lynn, K. Marr, R. A. Mezzacappa, J. M. Blondin, “A New Hathiramani, and E. Salzborn, “Electron- Maqueda, E. Melby, D. Mikkelsen, D. Twist on the Core Collapse Supernova impact ionization of Oq+ ions for q=1-4,” Mossessian, R. Nazikian, W. M. Nevins, Mechanism?” in From Twilight to Physical Review A, 67, 042714, 2003. R. Parker, T. S. Pedersen, C. K. Phillips, P. Highlight: The Physics of Supernovae, W. Y. Luo, I. Malik, Z. Li, M S. Shephard, Phillips, C. S. Pitcher, M. Porkolab, J. Hillebrandt, D. Leibundgut, eds., p. 63, “Adaptive mesh refinement in accelerator Ramos, M. Redi, J. Rice, B. N. Rogers, W. Springer-Verlag, 2003. modeling with OMEGA3P,” Proc. 7th US L. Rowan, M. Sampsell, G. Schilling, S. D. Mika, P. Myers, A. Majorell, L. Jiang, Nat. Congress on Comp. Mech., p. 50, Scott, J. Snipes, P. Snyder, D. Stotler, G. S. Kocak, J. Wan, M. S. Shephard, “Alloy 2003. Taylor, J. L. Terry, H. Wilson, J. R. 718 extrusion modeling - unstable flow and Wilson, S. M. Wolfe, S. Wukitch, X. Q. W. P. Lysenko, J. Qiang, R. W. Garnett, T. mesh enrichment,” Proc. 7th US Nat. Xu, B. Youngblood, H. Yuh, K. P. Wangler, “Challenge of Benchmarking Congress on Comp. Mech., p. 342, 2003. Zhurovich and S. Zweben, “Overview of Simulation Codes for the LANL Beam-Halo recent Alcator C-Mod research,” Nucl. A. J. Miller, M. A. Alexander, G. J. Boer, Experiment,” Proc. of IEEE Particle Fusion 43, 1610, 2003. F. Chai, K. Denman, D. J. Erickson III, R. Accelerator Conference 2003, Portland, Frouin, A. Gabric, E. Laws, M. Lewis, Z. Ore., May 2003. G. Martinez-Pinedo, D. J. Dean, K. Liu, R. Murtugudde, S. Nakamoto, D. J. Langanke, J. Sampaio, “Neutrino-nucleus K.-L. Ma, “Visualizing Time-Varying Volume Neilson, J. R. Norris, C. Ohlmann, I. interactions in core-collapse supernova,” Data,” IEEE Computing in Science and Perry, N. Schneider, K. Shell and A. Nuclear Physics A, 718, 452-454, 2003. Engineering,” 5(2), p. 34, March/April Timmermann, “Potential feedbacks between 2003. W. Maslowski, and W. H. Lipscomb, Pacific Ocean ecosystems and interdecadel cli- “High-resolution simulations of Arctic sea ice mate variations,” Bulletin of the American M. Macri, S. De, M. S. Shephard, during 1979-93,” Polar Research, 22, 67, Meteorology, Soc. 84, 617, 2003. “Hierarchical tree-based discretization for the 2003. method of spheres,” Computers and G. H. Miller, “An iterative Riemann solver Structures, 81(8-11), p. 789-803, 2003. L. McInnes, B. Norris, S. Bhowmick, and for systems of hyperbolic conservation laws, P. Raghavan, 2002, “Adaptive Sparse Linear with application to hyperelastic solid mechan- G. G. Maisuradze, and D. L. Thompson, Solvers for Implicit CFD Using Newton- ics,” Journal of Computational Physics “Interpolating Moving Least-Squares Method Krylov Algorithms,” Proc. of the Second 193, p. 198-225, 2003. for Fitting Potential Energy Surfaces: MIT Conference on Computational Fluid Illustrative Approaches and Applications,” D. M. Mitnik, D. C. Griffin, C. P. and Solid Mechanics, Boston, Mass., June Journal of Physical Chemistry A 107, Ballance, and N. R. Badnell, “An R-matrix 2003. 7118-7124, 2003. with pseudo-states calculation of electron- D. J. McGillicuddy, L. A. Anderson, S. C impact excitation in C2+,” Journal of G. G. Maisuradze, D. L. Thompson, A. F. Doney and M. E. Maltrud, “Eddy-driven Physics B, 36, 717, 2003. Wagner, and M. Minkoff, “Interpolating sources and sinks of nutrients in the upper Moving Least-Squares Method for Fitting T. Mohan, B. R. de Supinski, S. A. ocean: results from a 0.1 degree resolution Potential Energy Surfaces: Detailed Analysis McKee, F. Mueller, A. Yoo, M. Schulz, model of the North Atlantic,” Global of One Dimensional Applications,” Journal “Identifying and Exploiting Spatial Biogeochemical Cycles, 17, 1035, 2003. of Chemical Physics 119, 10002-10014, Regularity in Data Memory References,” 2003. O. E. B. Messer, W. R. Hix, M. Proc. of ACM/IEEE SC03 conference, Liebendorfer, A. Mezzacappa, “Electron Phoenix, Ariz., Nov. 2003. T. Manteuffel, S. McCormick, and C. capture and shock formation in core-collapse Pflaum, “Improved discretization error esti-

101 LIST OF PUBLICATIONS

J. Mueller, S. Nagrath, X. Li, J. E. Jansen, processes in proton collisions with lithium Conference, Austria, Lecture Notes in M. S. Shephard, “Efficient computational atoms,” Physical Review A, 68, 013404, Computer Science, 2789, p. 214-223, methods for the investigation of vascular dis- 2003. Springer-Verlag, Aug. 2003. ease,” Proc. 7th US Nat. Congress on M. S. Pindzola and F. Texier, “Collective D. Quinlan, M. Schordan, B. Miller and Comp. Mech., p. 195, 2003. modes of ground and dark-soliton states of M. Kowarschik, “Parallel Object-Oriented J. D. Myers, C. Pancerella, C. Lansing, K. Bose-Einstein condensates in anisotropic Framework Optimization,” Special Issue of L. Schuchardt and B. Didier, “Multi-scale traps,” Journal of Physics B, 36, 1783, Concurrency: Practice and Experience, science: Supporting emerging practice with 2003. 16, 2-3, p. 293-302, 2003. semantically-derived provenance,” Proc. of A. Pochinsky, et al., “Large scale commodity D. Quinlan, M. Schordan, Q. Yi, and B. R. the Workshop on Semantic Web clusters for lattice QCD,” Nuclear Physics B de Supinski, “Semantic-driven Technologies for Searching and (Proc. Suppl.) 119, 1044, 2003. Parallelization of Loops Operating on User- Retrieving Scientific Data, Sanibel Island, Defined Containers,” 16th Annual Fla., Oct. 2003. J. Pruet, “Neutrinos from the Propagation of Workshop on Languages and Compilers a Relativistic Jet through a Star,” ApJ, J. D. Myers, A. Chappell, M. Elder, A. for Parallel Computing, College Station, 591:1104-1109, July 2003. Geist, J. Schwidder, “Re-integrating the Texas, Oct. 2003. research record,” IEEE Computing in J. Pruet and G. M. Fuller, “Estimates of D. Quinlan, M. Schordan, Qing Li, and B. Science and Engineering, 5(3):44-50, 2003. Stellar Weak Interaction Rates for Nuclei in R. de Supinski, “A C++ Infrastructure for the Mass Range A=65-80,” Astrophysical D. Narayan, R. Bradshaw, R. Evard and Automatic Introduction and Translation of Journal Supplement, 149:189-203, Nov. A. Lusk, “Bcfg: a configuration management OpenMP Directives,” Proc. of the 2003. tool for heterogeneous environments,” Proc. of Workshop on Open MP Applications and the 5th IEEE International Conference J. Pruet, S. E. Woosley, and R. D. Tools (WOMPAT’03), Lecture Notes in on Cluster Computing (CLUSTER03), Hoffman, “Nucleosynthesis in Gamma-Ray Computer Science, 2716, p. 13-25, Kowloon, Hong Kong, Dec. 2003. Burst Accretion Disks,” ApJ, 586:1254- Springer-Verlag, June 2003. 1261, April 2003. E. Nelson-Melby, M. Porkolab, P. T. P. Raghavan, K. Teranishi, and E. G. Ng, Bonoli, Y. Lin, A. Mazurenko, and S. J. J. Qiang and R. W. Garnett, “Smooth “A Latency Tolerant Hybrid Sparse Solver Wukitch, “Experimental Observations of Approximation with Acceleration in an Using Incomplete Cholesky Factorization,” Mode-Converted Ion Cyclotron Waves in a Alternating Phase Focusing Superconducting Numerical Linear Algebra with Tokamak Plasma by Phase Contrast Linac,” Nuclear Instruments and Methods Applications 10, p. 541-560, 2003. Imaging,” Physical Review Letters, 90, in Physics Res. A 496, 33, 2003. S. Ratkovic, S. Iyer Dutta, M. Prakash, 155004, 2003. J. Qiang, M. Furman and R. Ryne, “Differential neutrino rates and emissivities R. M. Olson, M. W. Schmidt, M. S. “Parallel Particle-In-Cell Simulation of from the plasma process in astrophysical sys- Gordon and A. P. Rendell, “Enabling the Colliding Beams in High Energy tems,” Physical Review D, 67, 123002, Efficient Use of SMP Clusters: The Accelerators,” Proc. of the ACM/IEEE 2003. GAMESS/DDI Model,” Proc. of the SC03 conference, Phoenix, Ariz., Nov. D. A. Reed, C. L. Mendes and C. Lu, ACM/IEE SC03 conference, Phoenix, 2003. “Intelligent Application Tuning and Ariz., Nov. 2003. J. Qiang, M. Furman, R. D. Ryne, W. Adaptation,” Chapter in The Grid: E. Ong, J. Larson and R. Jacob, “A Real Fischer, T. Sen, M. Xiao, “Parallel Strong- Blueprint for a New Computing Application of the Model Coupling Toolkit,” Strong/Strong-Weak Simulations of Beam- Infrastructure, 2nd edition, I. Foster & C. Proc. of the 2002 International Beam Interactions in Hadron Accelerators,” Kesselman eds., Morgan Kaufmann, Nov. Conference on Computational Science, Proc. of the 29th ICFA Advanced Beam 2003. eds. P. Sloot, C. J. K. Tan, J. J. Dongarra, Dynamics Workshop on Beam Halo D. A. Reed, C. Lu, C. L. Mendes, “Big A. Hoekstra, Springer-Verlag, 2003. Dynamics, Diagnostics, and Collimation Systems and Big Reliability Challenges,” (HALO’03), Montauk, NY, May 2003. C. M. Pancerella, J. Hewson, W. Koegler Proc. of Parallel Computing 2003, D. Leahy, M. Lee, L. Rahn, C. Yang, J. D. J. Qiang, R. D. Ryne, and I. Hofmann, Dresden, Germany, Sept. 2003. Myers, D. Didier, R. McCoy, “Metadata in “Space-Charge Driven Emittance Growth in J. F. Remacle, J. E. Flaherty, M. S. the collaboratory for multi-scale chemical sci- a 3D Mismatched Anisotropic Beam,” Proc. Shephard, “An adaptive discontinuous ence,” Proc, of the 2003 Dublin Core of IEEE Particle Accelerator Conference Galerkin technique with an orthogonal basis Conference: Supporting Communities of 2003, Portland, Ore., May 2003. applied compressible flow problems,” SIAM Discourse and Practice-Metadata J. Qiang, R. D. Ryne, T. Sen, M. Xiao, Review, 45(1), p. 53-72, 2003. Research and Applications, Seattle, “Macroparticle Simulations of Antiproton Wash., Sept.-Oct. 2003. J. F. Remacle, M. S. Shephard, “An algo- Lifetime at 150 GeV in the Tevatron,” Proc. rithm oriented mesh database,” W. Park, J. Breslau, J. Chen, et al,, of IEEE Particle Accelerator Conference International Journal for Numerical “Nonlinear simulation studies of tokamaks 2003, Portland, Ore., May 2003. Methods in Engineering, 58, p. 349-374, and STs,” Nuclear Fusion 43, 483, 2003. D. Quinlan and M. Schordan, “A Source-to- 2003. M. S. Pindzola, T. Minami, and D. R. Source Architecture for User-Defined J. Rikovska Stone, J. C. Miller, R. Schultz, “Laser-modified charge-transfer Optimizations,” Joint Modular Languages Koncewicz, P. D. Stevenson, M. R. Strayer,

102 LIST OF PUBLICATIONS

“Nuclear matter and neutron-star properties “Observation of trielectronic recombination in [Computer] Scientific Community,” Proc. of calculated with the Skyrme interaction,” Be-like Cl ions,” Physical Review Letters, the 17th Annual International Symposium Physical Review C, 68, 034324, 2003. 91, 043001, 2003. on High Performance Computing Systems and Applications and the 1st F. J. Robicheaux and J. D. Hanson, M. Schordan and D. Quinlan, “A Source-to- Annual OSCAR Symposium, Quebec, “Simulated expansion of an ultracold neutral Source Architecture for User-Defined Canada, May 2003. plasma,” Physics of Plasmas, 10, 2217, Optimizations,” Proc, of the Joint Modular 2003. Languages Conference (JMLC’03), S. L. Scott, B. Luethke, “Scalable C3 Power Lecture Notes in Computer Science, 2789, Tools,” Cluster World Conference and T. M. Rogers, G. A. Glatzmaier, and S. E. p. 214-223, Springer-Verlag, June 2003. Expo (CWCE 2003) and 4th Linux Woosley, “Simulations of two-dimensional Cluster Institute Conference (LCI), San turbulent convection in a density-stratified D. R. Schultz and P. S. Krstic, “Ionization Jose, Calif., June 2003. fluid,” Physical Review E, 67(2):026315, of Helium by antiprotons: Fully correlated, Feb. 2003. four-dimensional lattice approach,” Physical S. L. Scott, J. Mugler, T. Naughton, B. Review A, 022712, 2003. Barrett, A. Lumsdaine, J. Squyres, B. G. Rosa, E. Lum, K.-L. Ma, and K. Ono, Ligneris, F. Giraldeau and C. “An Interactive Volume Visualization System G. L. Schussman, “Interactive And Leangsuksun, “OSCAR Clusters,” Proc. of for Transient Flow Analysis,” Proc. of the Perceptually Enhanced Visualization Of the Ottawa Linux Symposium (OLS’03), Volume Graphics 2003 Workshop, Tokyo, Large, Complex Line-Based Datasets,” Ph.D. Ottawa, Canada, July 2003. Japan, July 2003. thesis, University of California Davis. Dec. 2003. S. L. Scott, C. Leangsuksun, L. Shen, T. A. L. Rosenberg, J. E. Menard, J. R. Liu and H. Song, “Dependability Prediction Wilson, S. Medley, R. Andre, D. Darrow, S. L. Scott, B. Luethke and J. Mugler, of High Availability OSCAR Cluster R. Dumont, B. P. LeBlanc, C. K. Phillips, “Cluster & Grid Tools: C3 Power Tools with Server,” 2003 International Conference on M. Redi, T. K. Mau, E. F. Jaeger, P. M. C2G,” Newsletter of the IEEE Task Force Parallel and Distributed Processing Ryan, D. W. Swain, R. W. Harvey, J. on Cluster Computing, 4, 2, March/April Techniques and Applications Egedal, and the NSTX Team, 2003. (PDPTA’03), Las Vegas, Nevada, June “Characterization of Fast Ion Absorption of S. L. Scott, C. Leangsuksun, L Chen and 2003. the High Harmonic Fast Wave in the T. Lui, “Building High Availability and National Spherical Torus Experiment,” S. L. Scott, I. Haddad and C. Performance Clusters with HA-OSCAR Radio Frequency Power in Plasmas, Proc. Leangsuksun, “HA-OSCAR: The birth of a Toolkits,” 2003 High Availability and 15th Topical Conf., Moran, Wyoming, Highly Available OSCAR,” Linux Journal, Performance Workshop, Santa Fe, NM, AIP, N. Y., 185, 2003. 2003, 115, Seattle, Wash., Nov. 2003. Oct. 2003. G. Rumolo, A. Z. Ghalam, et al., “Electron S. L. Scott, T. Naughton, Y. C. Fang, P. S. L. Scott, C. Leangsuksun, L. Shen, T. Cloud Effects on Beam Evolution in a Pfeiffer, B. Ligneris, and C. Leangsuksun, Liu and H. Song, “Availability Prediction Circular Accelerator,” Phys. Rev. STAB, 6, “The OSCAR Toolkit: Current and Future and Modeling of High Availability OSCAR 081002, 2003. Developments,” Dell Power Solution maga- Cluster,” Proc. of the 5th IEEE zine, Nov. 2003. J. M. Sampaio, K. Langanke, G. Martinez- International Conference on Cluster Pinedo, E. Kolbe, D. J. Dean, “Electron Computing (CLUSTER03), Kowloon, S. L. Scott, T. Naughton, B. Barrett, J. capture rates for core collapse supernovae,” Hong Kong, Dec. 2003. Squyres, A. Lumsdaine, Y. C. Fang, and V. Nuclear Physics A, 718, 440-442, 2003. Mashayekhi “Looking Inside the OSCAR S. L. Scott, B. Luethke and T. Naughton, Cluster Toolkit,” Dell Power Solution mag- R. Samtaney, “Suppression of the Richtmyer- “OSCAR Cluster Administration With C3,” azine, Nov. 2003. Meshkov instability in the presence of a mag- Proc. of the 17th Annual International netic field,” Physics of Fluids 15, L53-56, Symposium on High Performance A. Snavely, L. Carrington, N. Wolter, C. 2003. Computing Systems and Applications and Lee, J. Labarta, J. Gimenez, and P. Jones, the 1st Annual OSCAR Symposium, “Performance Modeling of HPC Applications, S. Sankaran, J. M. Squyres, B. Barrett, A. Quebec, Canada, May 2003. Mini-symposium on Performance Modeling,” Lumsdaine, J. Duell, P. Hargrove and E. Proc. of Parallel Computing 2003, Roman, “The LAM/MPI Checkpoint/Restart S. L. Scott, C. Leangsuksun, L. Shen, H. Dresden, Germany, Sept. 2003. Framework: System-Initiated Checkpointing,” Song and I. Haddad, “The Modeling and LACSI Symposium, Santa Fe, NM, Oct. Dependability Analysis of High Availability C. Sneden, J. J. Cowan, J. E. Lawler, I. I. 2003. OSCAR Cluster System,” Proc. of the 17th Ivans, S. Burles, T. C. Beers, F. Primas, V. Annual International Symposium on Hill, J. W. Truran, G. M. Fuller, B. Pfeiffer, D. P. Schissel, “National Fusion High Performance Computing Systems K.-L. Kratz, “The Extremely Metal-poor, Collaboratory - Advancing the Science of and Applications and the 1st Annual Neutron Capture-rich Star CS 22892-052: A Fusion Research,” Proc. of the 2003 DOE OSCAR Symposium, Quebec, Canada, Comprehensive Abundance Analysis,” SciDAC PI Meeting, 2003. May 2003. Astrophysical Journal Letters, 591, 936- M. Schnell, G. Gwinner, N. R. Badnell, 953, 2003. S. L. Scott, T. Naughton, B. Ligneris and M. E. Bannister, S. Bohm, J. P. Colgan, S. N. Gorsuch, “Open Source Cluster C. R. Sovinec, T. A. Gianakon, E. D. Kieslich, S. D. Loch, D. Mitnik, A. Muller, Application Resources (OSCAR): Design, Held, S. E. Kruger, D. D. Schnack and the M. S. Pindzola, S. Schippers, D. Schwalm, Implementation and Interest for the NIMROD Team, “NIMROD: a computa- W. Shi, A. Wolf and S. G. Zhou,

103 LIST OF PUBLICATIONS

tional laboratory for studying nonlinear F.-K. Thielemann, D. Argast, F. Desktop,” Proc. of the IEEE International fusion magnetohydrodynamics,” Physics of Brachwitz, W. R. Hix, P. Hoflich, M. Workshop on Projector-Camera Systems Plasmas 10, 1727, 2003. Liebendorfer, G. Martinez-Pinedo, A. (PROCAMS), Oct. 2003. Mezzacappa, K. Nomoto, I. Panov, I. C. R. Sovinec, B. I. Cohen, G. A. Cone, E. G. Wallace, et al., “Color Gamut Matching “Supernova Nucleosynthesis and Galactic B. Hooper, and H. S. McLean, “Numerical for Tiled Display Walls,” Proc. of the Evolution,” From Twilight to Highlight: investigation of transients in the SSPX spher- Immersive Projection Technology The Physics of Supernovae, W. omak,” Physical Review Letters 94, Workshop (IPT), May 2003. Hillebrandt, D. Leibundgut, eds., p. 331, 035003, 2003. Springer-Verlag, 2003. J. Wan, S. Kocak, M. S. Shephard, P. Spentzouris, J. Amundson, “FNAL “Automated adaptive forming simulations,” S. J. Thomas, J. M. Dennis, H. M. Tufo, P. Booster: experiment and modeling,” Proc. of 12th International Meshing Roundtable, F. Fischer, “A Schwarz Preconditioner for IEEE Particle Accelerator Conference Sandia National Laboratories, p. 323-334, the Cubed-Sphere,” SIAM J. Sci. Comp., 25, 2003, Portland, Ore., May 2003. 2003. p. 442-453, 2003. N. Stojic, J. W. Davenport, M. Komelj, J. J. Wan, S. Kocak, M. S. Shephard, M. Thompson, “Fine-grained Authorization Glimm, “Surface magnetic moment in\alpha- “Controlled mesh enrichments for evolving for Job and Resource Management using uranium by density-functional theory,” Phys. geometries,” Proc. 7th US Nat. Congress Akenti and Globus,” Proceedings of the Rev. B, 68, 094407, 2003. on Comp. Mech., p. 42, 2003. 2003 Conference for Computing in High P. H. Stoltz, M. A. Furman, J. -L. Vay, A. Energy and Nuclear Physics, 2003. T. P. Wangler, K. R. Crandall, R. W. W. Molvik, R. H. Cohen, “Numerical Garnett, D. Gorelov, P. Ostroumov, J. T. A. Thompson, A. Burrows, and P. A. Simulation of the Generation of Gecondary Qiang, R. Ryne, R. York, “Advanced Beam- Pinto, “Shock Breakout in Core-Collapse Electrons in the High Current Experiment,” Dynamics Simulation Tools for the RIA Supernovae and Its Neutrino Signature,” Physics Rev. ST Accel. Beams. Vol 6, Driver Linac, Part I: Low Energy Beam Astrophysical Journal, 592:434-456, July 054701. 2003. Transport and Radiofrequency Quadrupole,” 2003. Proc. RIA R&D Workshop, Washington, A. Stompel, K.-L. Ma, J. Bielak, O. C. Tse, S. T. Teoh, and K.-L. Ma, “A n D.C., Aug. 2003. Ghattas and E. Kim, “Visualizing Large- Interactive Isosurface Visualization Cluster,” Scale Earthquake Simulations,” Proc. of the A. Wetzels, A. Gurtler, F. Rosca-Pruna, S. Proc. of High Performance Computing ACM/IEEE SC03 conference, Phoenix, Zamith, M. J. J. Vrakking, F. Robicheaux 2003 Symposium, Orlando, Fla., March Ariz., Nov. 2003. and W. J. van der Zande, “Two-dimensional 2003. momentum imaging of Rydberg states using A. Stompel, K.-L. Ma, E. Lum, J. Ahrens, M. Valorani, F. Creta and D. A. Goussis, half-cycle pulse ionization and velocity map and J. Patchett, “SLIC: Scheduled Linear “Local and Global Manifolds in Stiff imaging,” Physical Review A, 68, 041401, Image Compositing for Parallel Vollume Reaction-Diffusion Systems,” Computational 2003. Rendering,” Proc. of the 2003 Symposium Fluid and Solid Mechanics 2003, 1548- on Parallel Visualization and Graphics, M. Wingate, et al. (The HPQCD 1551, K. J. Bathe, ed., Elsevier, 2003. Seattle, WA, Oct. 2003. Collaboration), “Bs Mesons Using Staggered M. Valorani, H. N. Najm and D. A. Light Quarks,” Nuclear Physics B (Proc. D. W. Swain, M. D. Carter, J. R. Wilson, Goussis, “CSP Analysis of a Transient Suppl.) 119, 604, 2003. P. M. Ryan, J. B. Wilgen, J. Hosea and A. Flame-Vortex Interaction: Time Scales and Rosenberg, “Loading and Asymmetry M. C. Witthoeft, J. P. Colgan, and M. S. Manifolds,” Combustion and Flame, Measurements and Modeling for the National Pindzola, “Electron-impact excitation of Li 134(1-2):35-53, July 2003. Spherical Torus Experiment Ion Cyclotron to high principal quantum number,” Physical Range of Frequencies System,” Fusion J. S. Vetter and F. Mueller, “Communication Review A, 68, 022711, 2003. Science and Technology, 43, 503, 2003. Characteristics of Large-Scale Scientific F. Wolf, B. Mohr, “Automatic performance Applications for Contemporary Cluster K. Takahashi, K. Sato, A. Burrows, and T. analysis of hybrid MPI/OpenMP applica- Architectures,” Journal of Parallel and Thompson, “Supernova neutrinos, neutrino tions,” Journal of Systems Architecture, Distributed Computing, 63, p. 853-865, oscillations, and the mass of the progenitor Special Issue: Evolutions in Parallel 2003. star,” Phys. Rev. D, 68:113009, 2003. Distributed and Network-Based R. Vuduc, “Automatic Performance Tuning of Processing, A. Clematis, D. D’Agostino, T. J. Tautges, “MOAB-SD: Integrated Sparse Matrix Kernels,” Ph.D. thesis, U.C. eds., Elsevier, 49(10-11), p. 421-439, Nov. Structured and Unstructured Mesh Berkeley Dept. of Computer Science, 2003. Representation,” 6th U.S. Congress on Dec. 2003. Computational Mechanics, Trends in P. H. Worley, T. H. Dunigan, “Early Unstructured Mesh Generation, R. Vuduc, A. Gyulassy, J. W. Demmel and Evaluation of the Cray X1 at Oak Ridge Albuquerque, NM, July 2003. K. A. Yelick, “Memory Hierarchy National Laboratory,” Proc. of the 45th Optimizations and Performance Bounds for Cray User Group Conference, Columbus, F.-K. Thielemann, D. Argast, F. Brachwitz, Sparse ATA*x,” Proc. of ICCS 2003: Ohio, May 2003. W. R. Hix, P. Hoflich, M. Liebendorfer, G. Workshop on Parallel Linear Algebra, Martinez-Pinedo, A. Mezzacappa, I. J. C. Wright, P. T. Bonoli, M. Brambilla, F. Melbourne, Australia, June 2003. Panov, T. Rauscher, “Nuclear cross sections, Meo, E. D’Azevedo, D. B. Batchelor, E. F. nuclear structure and stellar nucleosynthesis,” G. Wallace, et al., “DeskAlign: Jaeger, L. A. Berry, C. K. Phillips and A. Nuclear Physics A, 718, 139-146, 2003. Automatically Aligning a Tiled Windows Pletzer, “Full Wave Simulations of Fast

104 LIST OF PUBLICATIONS

Wave Mode Conversion and Lower Hybrid User Group Conference, Knoxville, TN, mination of the strange and light quark mass- Wave Propagation in Tokamaks,” Bull. Am. May 2004. es from full lattice QCD,” Physical Review Phys. Soc. 48, 127, 2003. D, 70, 031504(R), 2004. V. Akcelik, J. Bielak, G. Biros, I. K. Wu, W. Koegler, J. Chen and A. Epanomeritakis, O. Ghattas, L. Kallivokas C. Aubin, et al. (The MILC Shoshani, “Using bitmap index for interac- and E. Kim, “An Online Framework for Collaboration), “Light hadrons with tive exploration of large datasets,” Proc. of Inversion-Based 3D Site Characterization,” improved staggered quarks: approaching the SSDBM 2003, p. 65, Cambridge, Mass, Lecture Notes in Computer Science, continuum limit,” Physical Review D 70, USA, 2003. 3038, p. 717, Springer-Verlag, 2004. 094505, 2004. K. Wu, W.-M. Zhang, A. Sim, J. Gu and M. A. Albao, D.-J. Liu, C. H. Choi, M. S. C. Aubin, et al. (The MILC A. Shoshani, “Grid collector: An event cata- Gordon and J. W. Evans, “Atomistic model- Collaboration), “Light pseudoscalar decay log with automated file management,” Proc. ing of morphological evolution during simul- constants, quark masses, and low energy con- of IEEE Nuclear Science Symposium taneous etching and oxidation of Si(100),” stants from three-flavor lattice QCD,” 2003, 2003. Surface Science 555, 51, 2004. Physical Review D, 70, 114501, 2004. R. G. Zepp, T. V. Callaghan and D. J. A. Alexakis, A. C. Calder, A. Heger, E. F. C. Aubin, et al. (The MILC Erickson III, “Interactive effects of ozone Brown, L. J. Dursi, J. W. Truran, R. Rosner, Collaboration), “Pion and kaon physics with depletion and climate change on biogeochemi- D. Q. Lamb, F. X. Timmes, B. Fryxell, M. improved staggered quarks,” Nuclear cal cycles,” Journal of Photochemistry, Zingale, P. M. Ricker and K. Olson, “On Physics B (Proc. Suppl.), 129-130, 227, Photobiol. 2, 51, 2003. Heavy Element Enrichment in Classical 2004. Novae,” ApJ, 602:931-937, Feb. 2004. O. Zatsarinny, T. W. Gorczyca, K. T. T. Austin, T. Manteuffel, and S. Korista, N. R. Badnell and D. W. Savin, C. Alexandru, et al., “N to Delta electro- McCormick, “A robust approach to mini- “Dielectronic recombination data for dynamic magnetic transition form factors from lattice mizing H(div) dominated functionals in an finite-density plasmas: II. The Oxygen isoelec- QCD,” Physical Review D, 69, 114506, H1-conforming finite element space,” Journal tronic sequence,” Astronomy and 2004. on Numerical Analysis, Lin. Alg. App. 11, Astrophysics, 412, 587, 2003. p. 115, 2004. C. Alexandrou, et al., “gamma N – Delta M. Zeghal, U. El Shamy, M. S. Shephard, transition form factors in quenched and N(F) I. Baraffe, A. Heger, and S. E. Woosley, R. Dobry, J. Fish, T. Abdounm, “Micro- = 2 QCD,” Nuclear Physics (Proc. Suppl.) “Stability of Supernova Ia Progenitors against mechanical analysis of saturated granular 129-130, 302, 2004. Radial Oscillations,” ApJ, 615:378-382, soils,” Journal of Multiscale Computational Nov. 2004. J. Amundson, P. Spentzouris, “Emittance Engineering, 1(4), p. 441-460, 2003. dilution and halo creation during the first S. Basak, et al. (The LHPC W. Zhang, S. E. Woosley, and A. I. milliseconds after injection at the Fermilab Collaboration), “Baryon operators and spec- MacFadyen, “Relativistic Jets in Booster,” Proc. 33rd ICFA Advanced troscopy in lattice QCD,” Nuclear Physics Collapsars,” Astrophysical Journal, Beam Dynamics Workshop on High (Proc. Suppl.) 129-130, 186, 2004. 586:356-371, March 2003. Intensity & High Brightness Hadron S. Basak, et al. (The LHPC Beams, Bensheim, Germany, Oct. 2004. Collaboration), “Mass spectrum of N* and J. Amundson, P. Spentzouris, J. Qiang, R. source optimization,” Nuclear Physics D. Ryne, A. Adelmann, “A test suite of (Proc. Suppl.) 129-130, 209, 2004. space-charge problems for code benchmark- D. B. Batchelor, E. F. Jaeger, L. A. Berry, ing,” 9th European Particle Accelerator E. D’Azevedo, M. D. Carter, R. W. Conference (EPAC 2004), Lucerne, 2004 Harvey, R. J. Dumont, J. R. Myra, A. A. Switzerland, July 2004. D’Ippolito, D. N. Smithe, C. K. Phillips, P. A. Andrady, P. J. Aucamo, A. F. Bais, C. T. Bonoli and J. C. Wright, “Integrated A. Adelmann, J. Amundson, P. L. Ballare, L. O. Bjorn, J. F. Bornman, M. modeling of wave-plasma interactions in Spentzouris, J. Qiang, R. D. Ryne, “A Test Caldwell, A. P. Cullen, D. J. Erickson III, fusion systems,” Theory of Fusion Plasmas, Suite of Space-charge Problems for Code F. R. de Gruijl, D.-P. Hader, M. ilyas, G. Proc. of the Joint Varenna-Lausanne Benchmarking,” Proc. of EPAC 2004, Kulandaivelu, H. D. Kumar, J. Longstreth, International Workshop Varenna, Italy, Lucerne, Switzerland, p. 1942, 2004. R. L. McKenzie, M. Norval, H. H. Aug.-Sept. 2004. P. K. Agarwal, R. A. Alexander, E. Apra, Redhwi, R. C. Smith, K. R. Solomon, Y. M. W. Beall, J. Walsh, M. S. Shephard, “A S. Balay, A. S. Bland, J. Colgan, E. F. Takizawa, X. Tang, A. H. Teramura, A. comparison of techniques for geometry access D’Azevedo, J. J. Dongarra, T. H. Dunigan, Torikai, J. C. van der Leun, S. Wilson, R. related to mesh generation,” Engineering Jr., M. R. Fahey, R. A. Fahey, A. Geist, M. C. Worrest, and R. G. Zepp, with Computers, 20(3), p. 210-221, 2004. Gordon, R. J. Harrison, D. Kaushik, M. “Environmental effects of ozone depletion and Krishnakumar, P. Lusczek, A. its interactions with climate change: Progress J. B. Bell, M. S. Day, C. A. Rendleman, S. Mezzacappa, J. A. Nichols, J. Nieplocha, Report 2003,” Photochemical and E. Woosley, and M. A. Zingale, “Direct L. Oliker, T. Packwood, M. S. Pindzola, T. Photobiological Sciences. 3, 1-5, 2004. Numerical Simulations of Type Ia C. Shulthess, J. S. Vetter, J. B. White III, T. Supernovae Flames II: The Rayleigh-Taylor C. Aubin, et al. (The HPQCD, MILC L. Windus, P. H. Worley, T. Zacharia, Instability,” Astrophysical Journal, 608, and UKQCD Collaborations), “First deter- “ORNL Cray X1,” Proc. of the 46th Cray 883, 2004.

105 LIST OF PUBLICATIONS

J. B. Bell, M. S. Day, C. A. Rendleman, S. B. Billeter, C. DeTar, and J. Osborn, Ricker, F. X. Timmes, M. Zingale and K. E. Woosley, and M. A. Zingale, “Direct “Topological susceptibility in staggered fermi- Olson, “Issues with Validating an Numerical Simulations of Type Ia on chiral perturbation theory,” Physical Astrophysical Simulation Code,” Computing Supernovae Flames I: The Landau-Darrieus Review D, 70, 077502, 2004. in Science and Engineering, 6(5):10, 2004. Instability,” Astrophysical Journal, 606, L. Bonaventura, and T. D. Ringler, C. Y. Cardall, A. O. Razoumov, E. 1029, 2004. “Analysis of discrete shallow-water models on Endeve, A. Mezzacappa, “The Long Term: J. B. Bell, M. S. Day, C. A. Rendleman, S. geodesic Delaunay grids with C-type stagger- Six-dimensional Core-collapse Supernova E. Woosley, and M. A. Zingale, “Adaptive ing,” Monthly Weather Review, 133, Models,” in Open Issues in Core Collapse low Mach number simulations of nuclear 2351, 2004. Supernova Theory, Proc. of the Institute flame microphysics,” Journal of for Nuclear Theory, Vol. 14, p. 196, P. T. Bonoli et al., “Full-wave Computational Physics, 195:677-694, University of Washington, Seattle, Wash., Electromagnetic Field Simulations in the April 2004. June 2004. Lower-Hybrid Range of Frequencies,” Bull. J. B. Bell, M. S. Day, J. F. Grcar, and M. J. American Phys. Soc. 49, 325, RI13, 2004. G. R. Carr, I. L. Carpenter, M. J. Cordery, Lijewski, “Stochastic Algorithms for the J. B. Drake, M. W. Ham, F. M. Hoffman M. Brezina, R. Falgout, S. MacLachlan, T. Analysis of Numerical Flame Simulations,” and P. H. Worley, “Porting and Performance Manteuffel, S. McCormick and J. Ruge, Journal of Computational Physics, 202, of the Community Climate System Model “Adaptive Smoothed Aggregation (_SA),” 262, 2004. (CCSM3) on the Cray X1,” Proc. of the SIAM Journal of Scientific Computing, 11th Workshop on High Performance S. Belviso, L. Bopp, C. Moulin, J. C. Orr, 25, 1896-1920, 2004. Computing in Meteorology, Reading, T. R. Anderson, O. Aumont, S. Chu, S. S. W. Bruenn, “Issues with Core-Collapse UK, Oct. 2004. Elliott, M. E. Maltrud and R. Simó, Supernova Progenitor Models,” in Open “Comparison of global climatological maps of L. Carrington, N. Wolter, A. Snavely and Issues in Core Collapse Supernova sea surface dimethylsulfide,” Global C. B. Lee, “Applying an Automated Theory, Proc. of the Institute for Nuclear Biogeochemical Cycles, 18(3), GB3013, Framework to Produce Accurate Blind Theory, Vol. 14, p. 99, University of 2004. Performance Predictions of Full-Scale HPC Washington, Seattle, Wash., June 2004. Applications,” DOD User Group S. J. Benson, L. Curfman McInnes, J. J. N. Bucciantini, R. Bandiera, J. M. Blondin, Conference 2004, Williamsburg, Va., June Moré and J. Sarich, “Scalable Algorithms in E. Amato, L. Del Zanna, “The effects of 2004. Optimization: Computational Experiments,” spin-down on the structure and evolution of AIAA Multidisciplinary Analysis and Y. Chen, et al., “Chiral Logs in Quenched pulsar wind nebulae,” Astronomy & Optimization Conference, 2004. QCD,” Physical Review D, 70, 034502, Astrophysics, 422, 609-619, 2004. 2004. G. J. O. Beran and M. Head-Gordon, A. Burrows, S. Reddy, and T. A. “Extracting dominant pair correlations from Z. Chen, J. Dongarra, P. Luszczek, and K. Thompson, “Neutrino Opacities in Nuclear many-body wavefunctions,” Journal of Roche, “LAPACK for Clusters Project: An Matter,” Nuclear Physics A, in press, 2004. Chemical Physics, 121, 78-88, 2004. Example of Self Adapting Numerical J. R. Burruss, et al., “Remote Computing on Software,” Hawaii International C. Bernard, et al. (The MILC the National Fusion Grid,” The 4th IAEA Conference on System Sciences (HICSS- Collaboration), “Quark Loop Effects in Technical Meeting on Control, Data 37), Jan. 2004. Semileptonic Form Factors for Heavy-Light Acquisition, and Remote Participation for Mesons,” Nuclear Physics B (Proc. Suppl.), S. Chu, M. Maltrud, S. Elliott, J. L. Fusion Research. Fusion Engineering and 129-130, 364, 2004. Hernandez and D. J. Erickson III, Design 71, 251-255, 2004. “Ecodynamic and eddy resolving dimethyl C. Bernard, et al. (The MILC L. Bytautas and K. Ruedenberg, sulfide simulations in a global ocean biogeo- Collaboration), “The Phase Diagram of “Correlation Energy Extrapolation Through chemistry/circulation model,” Earth High Temperature QCD with Three Flavors Intrinsic Scaling. I,” Journal of Chemical Interactions, 8 (11), 1, 2004. of Improved Staggered Quarks,” Nuclear Physics, 121, 10905, 2004. Physics B (Proc. Suppl.), 129-130, 626, S. Chu, S. Elliott, M. E. Maltrud, F. 2004. L. Bytautas and K. Ruedenberg, Chavez and F. Chai, “Ecodynamics of mul- “Correlation Energy Extrapolation Through tiple Southern Ocean iron enrichments simu- C. Bernard, et al. (The MILC Intrinsic Scaling. II,” Journal of Chemical lated in a global, eddy permitting GCM,” in Collaboration), “Excited States in Staggered Physics, 121, 10919, 2004. Environmental Sciences and Meson Propagators,” Nuclear Physics B Environmental Computing Vol. II, The (Proc. Suppl.), 129-130, 230, 2004. L. Bytautas and K. Ruedenberg, EnviroComp Institute, 2004. “Correlation Energy Extrapolation Through S. Bhowmick, P. Raghavan, L. McInnes, Intrinsic Scaling. III. Compact S. Chu, S. Elliott, M. E. Maltrud, F. and B. Norris, “Faster PDE-based Wavefunctions,” Journal of Chemical Chavez, and F. Chai, “Southern Ocean Iron Simulations Using Robust Composite Linear Physics, 121, 10852, 2004. Enrichments Simulated in a global, eddy per- Solvers,” Future Generation Computer mitting GCM,” in Environmental Sciences Systems, The International Journal of A. C. Calder, L. J. Dursi, B. Fryxell, T. and Environmental Computing Vol. II, ed. Grid Computing: Theory, Methods and Plewa, V. G. Weirs, T. Dupont, H. F. P. Zannetti, The EnviroComp Institute, Applications, 20, p. 373-387, 2004. Robey, R. P. Drake, B. A. Remington, G. 2004. Dimonte, J. Hayes, J. M. Stone, P. M.

106 LIST OF PUBLICATIONS

I.-H. Chung, and J. K. Hollingsworth, H. de Sterck, T. Manteuffel, S. resonances,” Nuclear Physics A, 737, 167, “Using Information from Prior Runs to McCormick, and L. Olson, “Least-squares 2004. Improve Automated Tuning Systems,” Proc. finite element methods and algebraic multi- S. Elliott, S. Chu, M. Maltrud and A. of ACM/IEEE SC2004 conference, grid solvers for linear hyperbolic PDEs,” McPherson, “Animation of global marine Pittsburgh, Penn., Nov. 2004. SIAM Journal of Scientific Computing, chlorophyll distributions from fine grid bio- 26, 31-54, 2004. I.-H. Chung, and J. K. Hollingsworth, geochemistry/transport modeling,” in “Automated Cluster-Based Web Service S. Deng et al., “Plasma wakefield accelera- Environmental Sciences and Performance Tuning,” Proceedings of IEEE tion in self-ionized gas or plasmas,” Physical Environmental Computing Vol. II (Edited Conference on High Performance Distri- Review E 68, 047401, 2004. by P. Zannetti), The EnviroComp buted Computing (HPDC), June 2004. Institute, 2004. Y. Deng, J. Glimm, J. Davenport, X. Cai, J. P. Colgan, M. S. Pindzola, and N. R. E. Santos, “Performance models on QCDOC S. Ethier and Z. Lin, “Porting the 3D gyro- Badnell, “Dielectronic recombination data for for molecular dynamics with coulomb poten- kinetic particle-in-cell code GTC to the NEC dynamic finite-density plasmas: V. The tials,” International Journal of High SX-6 vector architecture: perspectives and Lithium isoelectronic sequence,” Astronomy Performance Computing Applications, challenges,” Computer Physics and Astrophysics, 417, 1183, 2004. 18, p. 183-198, 2004. Communications, 164, 456, 2004. J. P. Colgan and M. S. Pindzola, “Double M. di Pierro et al., “Properties of charmoni- R. D. Falgout and P. S. Vassilevski, “On photoionization of helium at high photon um in lattice QCD with 2+1 flavors of Generalizing the AMG Framework,” SIAM energies,” Journal of Physics B, 37, 1153, improved staggered sea quarks,” Nuclear Journal on Numerical Analysis, 42, p. 2004. Physics (Proc. Suppl.) 129-130, 340. 2004. 1669-1693, 2004. D. Crawford, K.-L. Ma, M.-Y. Huang, S. M. di Pierro, et al., “Ds spectrum and lep- D. G. Fedorov, R. M. Olson, K. Kitaura, Klasky and S. Ethier, “Visualizing tonic decays with Fermilab heavy quarks and M. S. Gordon, and S. Koseki, “A new hier- Gyrokinetic Simulations,” Proc. of the IEEE improved staggered light quarks,” Nuclear archical parallelization scheme: Generalized Visualization 2004 conference, Austin, Physics (Proc. Suppl.) 129-130, 328, 2004. distributed data interface (GDDI), and an Texas, Oct. 2004. application to the fragment molecular orbital J. B. Drake, P. H. Worley, I. Carpenter, M. method (FMO),” Journal of Computational T. L. Dahlgren, P. T. Devanbu, “Adaptable Cordery, “Experience with the Full CCSM,” Chemistry, 25, 872, 2004. Assertion Checking for Scientific Software Proc. of the 46th Cray User Group Components,” Proc. of the First Interna- Conference, Knoxville, TN, May 2004. S. Flanagan, et al., “A General Purpose Data tional Workshop on Software Engineering Analysis Monitoring System with Case M. Dryja, A. Gantner, O. Widlund, and B. for High Performance Computing System Studies from the National Fusion Grid and I. Wohlmuth, “Multilevel Additive Schwarz Applications, Edinburgh, Scotland, UK, p the DIII-D MDSplus between Pulse Analysis Preconditioner for Nonconforming Mortar 64-69, May 2004. System,” 4th IAEA Technical Meeting on Finite Element Methods,” Journal on Control, Data Acquisition, and Remote D. Datta, R. C. Picu, M. S. Shephard, Numerical Analysis, Math., 12, 1, 23-38, Participation for Fusion Research, Fusion “Composite Grid Atomistic Continuum 2004. Engineering and Design 71, 263-267, Method: An adaptive approach to bridge T. H. Dunigan, Jr., J. S. Vetter, and P. H. 2004. continuum with atomistic analysis,” Journal Worley, “Performance Evaluation of the of Multiscale Computational Engineering, E. Follana, et al. (The HPQCD Cray X1 Distributed Shared Memory 2(3), p. 401-420, 2004. Collaboration), “Improvement and taste Architecture,” Proc. of IEEE Hot symmetry breaking for staggered quarks,” C. T. H. Davies, et al., “High-Precision Interconnects, 2004. Nuclear Physics (Proc. Suppl.) 129-130, Lattice QCD Confronts Experiment,” S. Dutta, J. Glimm, J. W. Grove, D. H. 384, 2004. Physical Review Letters, 92, 022001, 2004. Sharp, Y. Zhang, “Error comparison in N. Folwell, L. Ge, A. Guetz, V. Ivanov, et T. A. Davis, J. R. Gilbert, S. I. Larimore, tracked and untracked spherical simulations,” al., “Electromagnetic System Simulation – and E. G. Ng, “A column approximate mini- Computers and Mathematics with Prototyping Through Computation,” SciDAC mum degree ordering algorithm,” ACM Applications, 48, p. 1733-1747, 2004. PI meeting, 2004. Transactions of Mathematical Software S. Dutta, J. Glimm, J. W. Grove, D. H. 30, p. 353-376, 2004. C. L. Fryer, “Neutron Star Kicks from Sharp, Y. Zhang, “Spherical Richtmyer- Asymmetric Collapse,” ApJ Letters, T. A. Davis, J. R. Gilbert, S. I. Larimore, Meshkov instability for axisymmetric flow,” 601:L175-L178, Feb. 2004. and E. G. Ng, “Algorithm 836: COLAMD, Mathematics and Computers in a column approximate minimum degree Simulations, 65, p. 417-430, 2004. C. L. Fryer, D. E. Holz, and S. A. Hughes, ordering algorithm,” ACM Transactions of “Gravitational Waves from Stellar Collapse: S. I. Dutta, S. Ratkovic, M. Prakash, Mathematical Software 30, p. 377-380, Correlations to Explosion Asymmetries,” ApJ, “Photoneutrino process in astrophysical sys- 2004. 609:288-300, July 2004. tems,” Physical Review D, 69, 023005, D. J. Dean, “Intersections of Nuclear Physics 2004. C. L. Fryer and M. S. Warren, “The and Astrophysics,” in The Future Collapse of Rotating Massive Stars in Three R. G. Edwards, G. Fleming, D. G. Astronuclear Physics, A. Jorissen, S. Dimensions,” American Physics Journal, Richards, H. R. Fiebig and The LHPC Goriely, M. Rayet, L. Siers, J. Boffin, eds., 601:391-404, Jan. 2004. Collaboration, “Lattice QCD and nucleon EAS Publications Series, p. 175-189, 2004.

107 LIST OF PUBLICATIONS

L. Ge, L.-Q. Lee, Z. Li, C.-K. Ng, K. Ko, Y. Guo, A. Kawano, D. L. Thompson, A. compliant blood flow with FOSLS,” Biomed Y. Luo and M. Shephard, “Adaptive Mesh F. Wagner and M. Minkoff, “Interpolating Sci Instrum. 40, p. 193-9, 2004. Refinement For High Accuracy Wall Loss Moving Least-Squares Methods for Fitting J. Heys, T. Manteuffel, S. McCormick, Determination In Accelerating Cavity Potential Energy Surfaces: Applications to and L. Olson, “Algebraic multigrid (AMG) Design,” Proc. of the Eleventh Biennial Classical Dynamics Calculations,” Journal of for higher-order finite elements,” Journal of IEEE Conference on Electromagnetic Chemical Physics, 121, 5091-5097, 2004. Computational Physics 195, p. 560-575, Field Computation, Seoul, Korea, June A. Gurtler, F. Robicheaux, W. J. van der 2004. 2004. Zande and L. D. Noordam, “Asymmetry in J. Heys, T. Manteuffel, S. McCormick, L. Ge, L. Lee, L. Zenghai, L. Ng, C. Ko, the strong field ionization of Rydberg atoms and J. Ruge, “First-order system least squares K., Y. Luo, M. S. Shephard, “Adaptive by few-cycle pulses,” Physical Review (FOSLS) for coupled fluid-elasticity prob- Mesh Refinement for High Accuracy Wall Letters, 033002, 2004. lems,” Journal of Computational Physics Loss Determination in Accelerating Cavity A. Gurtler, F. Robicheaux, M. J. J. 195, p. 560-575, 2004. Design,” IEEE Conf. on Electromagnetic Vrakking, W. J. van der Zande and L. D. Field Computations, June 2004. D. Higdon, M. Kennedy, J. C. Cavendish, Noordam, “Carrier phase dependence in the J. A. Cafeo, R. D. Ryne, “Combining Field C. G. R. Geddes, C. Toth, J. van Tilborg, ionization of Rydberg atoms by short radio- Data and Computer Simulations for E. Esarey, C. B. Schroeder, D. Bruhwiler, frequency pulses: a model system for high Calibration and Prediction,” SIAM Journal C. Nieter, J. Cary and W. P. Leemans, order harmonic generation,” Physical on Scientific Computing, 26, 2, p. 448- “High-quality electron beams from a laser Review Letters, 92, 063901, 2004. 466, March 2004. wakefield accelerator using plasma-channel S. W. Hadley, D. J. Erickson III, J. L. guiding,” Nature 431, p.538-541, 2004. W. R. Hix, O. E. B. Messer, A. Hernandez and S. Thompson, “Future Mezzacappa, “The interplay between J. Glimm, J. W. Grove, Y. Kang, T. Lee, X. U.S. energy use for 2000-2025 as computed nuclear electron capture and fluid dynamics Li, D. H. Sharp, Y. Yu, K. Ye, M. Zhao, with temperatures from a global climate pre- in core collapse supernovae,” in Cosmic “Statistical riemann problems and a composi- diction model and energy demand model,” explosions in Three Dimensions, P. tion law for errors in numerical solutions of Proc. of the 24th Annual North American Hoflich, P. Kumar, J. C. Wheeler, eds., p. shock physics problems,” SIAM Journal on Conference of the USAEE/IAEE, United 307, Cambridge University Press, 2004. Scientific Computing, 26, p. 666-697, States Association for Energy Economics, 2004. Cleveland, OH, 2004. F. M. Hoffman, M. Vertenstein, H. Kitabata, J. B. White III, P. H. Worley, J. J. Glimm, S. Hou, Y. Lee, D. Sharp, K. Ye, P. T. Haertel, D. A. Randall and T. G. B. Drake, M. Cordery, “Adventures in “Sources of uncertainty and error in the simu- Jensen, “Toward a Lagrangian ocean model: Vectorizing the Community Land Model,” lation of flow in porous media,” Comp. and Simulating upwelling in a large lake using Proc. of the 46th Cray User Group Applied Mathematics, 23, p. 109-120, slippery sacks,” Monthly Weather Review, Conference, Knoxville, TN, May 2004. 2004. 132, 66, 2004. I. Hofmann, G. Franchetti, J. Qiang, R. D. J. Glimm, H. Jin, Y. Zhang, “Front tracking A. T. Hagler, et al. (The LHPC Ryne, “Dynamical Effects of the Montague for multiphase fluid mixing,” Collaboration), “Transverse structure of Resonance,” Proc. EPAC 2004, Lucerne, Computational Methods in Multiphase nucleon parton distributions from lattice Switzerland, p. 1960, 2004. Flow II, A. A. Mammoli, C. A. Brebbia, QCD,” Physical Review Letters, 93, eds., p. 13-22, WIT Press, Southampton, 112001, 2004. I. Hofmann, G. Franchetti, M. UK, 2004. Giovannozzi, M. Martini, E. Metral, J. W. C. Haxton, “Supernova Neutrino- Qiang, R. D. Ryne, “Simulation aspects of J. N. Goswami, K. K. Marhas, M. Nucleus Physics and the r-Process,” The r- the code benchmarking based on the CERN- Chaussidon, M. Gounelle, B. Meyer, Process: The Astrophysical Origin of the PS Montague-resonance experiment,” Proc. “Origin of Short-lived Radionuclides in the Heavy Elements and Related Rare 33rd ICFA Advanced Beam Dynamics Early Solar System,” Workshop on Isotope Accelerator Physics, Proc. of the Workshop on High Intensity & High Chondrites and the Protoplanetary Disk, First Argonne/MSU/JINA/INT R/A Brightness Hadron Beams, Bensheim, p. 9120, 2004. Workshop, p. 73, World Scientific, 2004. Germany, Oct. 2004. J. F. Grcar, P. Glarborg, J. B. Bell, M. S. Y. He and C. Ding, “MPI and OpenMP D. J. Holmgren, P. Mackenzie, A. Singh Day, A. Loren, and A. D. Jenson, “Effects Paradigms on Cluster of SMP Architectures: and J. Somine, “Lattice QCD clusters at of Mixing on Ammonia Oxidation in the Vacancy Tracking Algorithm for Multi- Fermilab,” Computing in High-Energy Combustion Environments at Intermediate Dimensional Array Transpose,” Journal of Physics (CHEP ’04), Interlaken, Temperatures,” Proceedings of the Parallel and Distributed Computing Switzerland, Sept. 2004. Combustion Institute, 30:1193, 2004. Practice, 2, 5, 2004. U. Hwang, J. M. Laming, C. Badenes, F. L. Grigori and X. S. Li, “Performance E. D. Held, J. D. Callen, C. C. Hegna, C. Berendse, J. Blondin, D. Cio, T. DeLaney, Analysis of Parallel Right-looking Sparse LU R. Sovinec, T. A. Gianakon, S. E. Kruger, D. Dewey, R. Fesen, K. A. Flanagan, C. L. Factorization on Two-dimensional Grids of “Nonlocal closures for plasma fluid simula- Fryer, P. Ghavamian, J. P. Hughes, J. A. Processors,” Proceedings of the PARA'04 tions,” Physics of Plasmas 11, 2419, 2004. Morse, P. P. Plucinsky, R. Petre, M. Pohl, Workshop on State-of-the art in Scientific J. J. Heys, C. G. DeGroff, T. Manteuffel, S. L. Rudnick, R. Sankrit, P. O. Slane, R. K. Computing, Springer Lecture Notes in McCormick, and H. Tufo, “Modeling 3-d Smith, J. Vink, and J. S. Warren, “A Million Computer Science, 2004.

108 LIST OF PUBLICATIONS

Second Chandra View of Cassiopeia A.,” ApJ Astrophysical Origin of the Heavy Processing Symposium, Santa Fe, NM, Letters, 615, L117-L120, Nov. 2004. Elements and Related Rare Isotope April 2004. Accelerator Physics, Proc. of the First A. V. Ilin, F. R. Chang-Diaz, J. P. Squire, T. Lee, Y. Yu, M. Zhao, J. Glimm, X. Li, Argonne/MSU/JINA/INT R/A A. G. Tarditi, M. D. Carter, “Improved sim- K. Ye, “Error analysis of composite shock Workshop, p. 157, World Scientific, 2004. ulation of the ICRF waves in the VASIMR interaction problems,” Proc. of PMC04 plasma,” Proceedings of the 18th R. Katz, M. Knepley and B. Smith, Conference, 2004. International conference on the numerical “Developing a Geodynamics Simulation with W. W. Lee, “Theoretical and numerical Simulation of Plasmas, Computer Physics PETSc,” in Numerical Solution of Partial properties of a gyrokinetic plasma: implica- Communications 164, 251, 2004. Differential Equations on Parallel tions on transport scale simulation,” Computers, A. M. Bruaset, P. Bjorstad, E.-J. Im, I. Bustany, C. Ashcraft, J. Computer Physics Communications, 164, and A. Tveito, eds., p. 413, Springer- Demmel and K. Yelick, “Toward automatic 244, 2004. Verlag, 2004. performance tuning of matrix triple products W. W. Lee, “Steady State Global based on matrix structure,” PARA'04 S. R. Kawa, D. J. Erickson III, S. Pawson, Simulations of Microturbulence,” American Workshop on State-of-the-art in Scientific and Z. Zhu, “Global CO2 transport simula- Physical Society Division of Plasma Computing, Copenhagen, Denmark, June tions using meteorological data from the Physics, Savannah, Ga., Nov. 2004. 2004. NASA data assimilation system,” Journal of Geophysical Research, 109. D18, 2004. J. Li and O. Widlund, “FETI--DP, BDDC, E.-J. Im, K. A. Yelick and R. Vuduc, and Block Cholesky Methods,” International “SPARSITY: An Optimization Framework K. Keahey et al., “Agreement Based Journal on Numerical Analysis, Methods for Sparse Matrix Kernels,” International Interactions for Experimental Science,” Proc. Engineering, 2004. Journal of High Performance Computing of Europar 2004 conference, 2004. Applications, 18 (1), p. 135-158, Feb. N. Li, Y. Saad and E. Chow, “Crout K. Keahey, et al., “Grids for Experimental 2004. Versions of ILU for General Sparse Matrices,” Science: The Virtual Control Room,” Proc. of SIAM Journal of Scientific Computing, J. P. Iorio, P. B. Duffy, B. Govindasamy, S. the CLADE 2004 Workshop, 2004. 25, p. 716-728, 2004. L. Thompson, M. Khairoutdinov and D. K. Keahey, et al., “Fine-Grained A. Randall, “Effects of model resolution and X. Li, J. F. Remacle, N. Chevaugeon, M. Authorization for Job Execution in the Grid: subgrid-scale physics on the simulation of S. Shephard, “Anisotropic Mesh Gradation Design and Implementation,” Concurrency daily precipitation in the continental United Control,” 13th International Meshing and Computation: Practice and States,” Climate Dynamics, 23, 243, 2004. Roundtable, Sandia National Experience 16 (5), 477-488, 2004. Laboratories, p. 401-412, 2004. L. Ives, T. Bui, W. Vogler, A. Bauer, M. J. P. Kenny, S. J. Benson, Y. Alexeev, J. Shephard, “Operation and performance of a Z. Li, N. T. Folwell, L. Ge, A. Guetz, V. Sarich, C. L. Janssen, L. Curfman 3D finite element charged particle code with Ivanov, M. Kowalski, L.-Q. Lee, C.-K. Ng, McInnes, M. Krishnan, J. Nieplocha, E. adaptive meshing Source,” 5th IEEE Int. G. Schussman, R. Uplenchwar, M. Wolf Jurrus, C. Fahlstrom and T. L. Windus, Vacuum Electronics Conf. (IVEC 2004), and K. Ko, “X-Band Linear Collider R&D “Component-Based Integration of Chemistry p. 316-317, 2004. In Accelerating Structures Through Advanced and Optimization Software,” Journal of Computing,” Proc. of the 9th European C. Jablonowski, “Adaptive grids in weather Computational Chemistry, 25, 1717-1725, Particle Accelerator Conference, Lucerne, and climate modeling,” University of 2004. Switzerland, July 2004. Michigan thesis, 2004. M. E. Kowalski, C.-K. Ng, L.-Q. Lee, Z. Z. Li, N. Folwell, L. Ge, A. Guetz, V. C. Jablonowski, M. Herzog, J. E. Penner, Li and K. Ko, “Time-Domain Calculation of Ivanov, M. Kowalski, L.-Q. Lee, C.-K. Ng, R. Oehmke, Q. F. Stout, B. van Leer, Guided Electromagnetic Wave Propagation G. Schussman, R. Uplenchwar, M. Wolf, “Adaptive grids for weather and climate mod- on Unstructured Grids,” U. S. National L. Xiao and K. Ko, “High Performance els,” ECMWF Seminar Proceedings on Committee of the International Union of Computing In Accelerator Structure Design Recent Developments in Numerical Radio Science (URSI) National Radio And Analysis,” Proc. of the 8th Methods for Atmospheric and Ocean Science Meeting, Boulder, Colo., Jan. International Computational Accelerator Modelling, Reading, UK, Sept. 2004. 2004. Physics Conference, St. Petersburg, S. C. Jardin, “A triangular finite element B. C. Lee, R. Vuduc, J. Demmel and K. Russia, June 2004. with first-derivative continuity applied to Yelick, “Performance Models for Evaluation M. Liebendoerfer, O. E. B. Messer, A. fusion MHD applications,” Journal of and Automatic Tuning of Symmetric Sparse Mezzacappa, S. W. Bruenn, C. Y. Cardall, Computational Physics 200, 133, 2004. Matrix-Vector Multiply,” International F.-K. Thielemann, “A Finite Difference Conference on Parallel Processing, G. C. Jordan, B. S. Meyer, “Nucleosynthesis Representation of Neutrino Radiation Montreal, Quebec, Canada, Aug. 2004. in Fast Expansions of High-Entropy, Proton- Hydrodynamics in Spherically Symmetric rich Matter,” Astrophysical Journal L.-Q. Lee, L. Ge, M. Kowalski, Z. Li, C.- General Relativistic Spacetime,” Letters, 617, L131-L134, 2004. K. Ng, G. Schussman, M. Wolf and K. Ko, Astrophysical Journal Supplement Series, “Solving Large Sparse Linear Systems In 150, 263-316, 2004. G. C. Jordan, B. S. Meyer, “Nuclear End-To-End Accelerator Structure Reaction Rates and the Production of Light Y. Lin, S. Wukitch. P. Bonoli, E. Nelson- Simulations,” Proc. of the 18th p-Process Isotopes in Fast Expansions of Melby, M. Porkolab, J. C. Wright, N. International Parallel and Distributed Proton-Rich Matter,” The r-Process: The Basse, A. E. Hubbard, J. Irby, L. Lin, E. S.

109 LIST OF PUBLICATIONS

Marmar, A. Mazurenko, D. Mossessian, Hansen and C. Johnston, eds., Academic Intensity & High Brightness Hadron A. Parisot, J. Rice, S. Wolfe, C. K. Phillips, Press, Oct. 2004. Beams, Bensheim, Germany, Oct., 2004. G. Schilling, J. R. Wilson, P. Phillips and H. MacMillan, T. Manteuffel and S. B. S. Meyer, “Synthesis of Short-lived A. Lynn, “Investigation of Ion Cyclotron McCormick, “First-order system least Radioactivities in a Massive Star,” ASP Range of Frequencies Mode Conversion at the squares and electrical impedance tomogra- Conf. Ser. 341: Chondrites and the Ion-Ion Hybrid Layer in Alcator C-Mod,” phy,” SIAM Journal on Numerical Protoplanetary Disk, p. 515, 2005. Physics of Plasmas 11 (5) 2466-2472, Analysis, 42, p. 461-483, 2004. 2004. B. S. Meyer, “Online Tools for Astronomy G. Mahinthakumar, M. Sayeed, J. Blondin, and Cosmochemistry,” 36th Annual Lunar Z. Lin and T. S. Hahm, “Turbulence spread- P. Worley, W. Hix and A. Mezzacappa, and Planetary Science Conference, p. ing and transport scaling in global gyrokinetic “Performance Evaluation and Modeling of a 1457, 2005. particle simulations,” Physics of Plasmas, Parallel Astrophysics Application,” Proc. of 11, 1099, 2004. B. S. Meyer, “Nucleosynthesis of Short-lived the High Performance Computing Radioactivities in Massive Stars,” Z. Lin et al., “Electron Thermal Transport in Symposium 2004, The Society for Workshop on Chondrites and the Tokamaks: ETG or TEM Turbulence?” Modeling and Simulation International, Protoplanetary Disk, p. 9089, 2004. International Atomic Energy Agency Arlington, Va., 2004. Conference, IAEA-CN/TH/8-4, P. A. Milne, A. L. Hungerford, C. L. G. G. Maisuradze, A. Kawano, D. L. Vilamoura, Portugal, 2004. Fryer, T. M. Evans, T. J. Urbatsch, S. E. Thompson, A. F. Wagner, and M. Boggs, J. Isern, E. Bravo, A. Hirschmann, W. Lipscomb and T. D. Ringler, “An incre- Minkoff, “Interpolating Moving Least- S. Kumagai, P. A. Pinto, and L.-S. The, mental remapping transport scheme on a Squares Methods for Fitting Potential Energy “Unified One-Dimensional Simulations of spherical geodesic grid,” Monthly Weather Surfaces: Analysis of an Application to a Six- Gamma-Ray Line Emission from Type Ia Review, 133, 2335, 2004. Dimensional System,” Journal of Chemical Supernovae,” Astrophysical Journal, Physics, 121, 10329-10338, 2004. E. Livne, A. Burrows, R. Walder, I. 613:1101-1119, Oct. 2004. Lichtenstadt and T. A. Thompson, “Two- G. Martinez-Pinedo, K. Langanke, J. M. P. Mucci, J. Dongarra, R. Kufrin, S. dimensional, Time-dependent, Multigroup, Sampaio, D. J. Dean, W. R. Hix, O. E. B. Moore, F. Song, F. Wolf, “Automating the Multiangle Radiation Hydrodynamics Test Messer, A. Mezzacappa, M. Liebendorfer, Large-Scale Collection and Analysis of Simulation in the Core-Collapse Supernova H.-T. Janka, M. Rampp, “Weak Interaction Performance,” Proc. of the 5th LCI Context,” Astrophysical Journal, 609:277- Processes in Core-Collapse Supernovae,” IAU International Conference on Linux 287, July 2004. Colloq. 192: Cosmic Explosions, On the Clusters: The HPC Revolution, Austin, 10th Anniversary of SN1993J, p. 321, S. D. Loch, C. J. Fontes, J. P. Colgan, M. Texas, May 2004. 2005. S. Pindzola, C. P. Ballance, D. C. Griffin, P. Muggli, B. E. Blue, C. E. Clayton, S. M. G. O’Mullane and H. P. Summers, “A N. Mathur, et al., “A Study of Pentaquarks Deng, F.-J. Decker, M. J. Hogan, C. collisional-radiative study of lithium plas- on the Lattice with Overlap Fermions,” Huang, R. Iverson, C. Joshi, T. C. mas,” Physical Review E, 69, 066405, Physical Review D, 70, 074508, 2004. Katsouleas, S. Lee, W. Lu, K. A. Marsh, 2004. B. S. Lee McCormick, B. Philip, and D. W. B. Mori, C. L. O'Connell, P. Raimondi, C. Lu and D. Reed, “Assessing Fault Quinlan, “Asynchronous fast adaptive com- R. Siemann, and D. Walz, “Meter-Scale Sensitivity in MPI Applications,” Proc. of posite-grid methods for elliptic problems: theo- Plasma-Wakefield Accelerator Driven by a the ACM/IEEE SC2004 conference, retical foundations,” SIAM Journal on Matched Electron Beam,” Physical Review Pittsburgh, Penn., Nov. 2004. Numerical Analysis, 42, p. 130-152, 2004. Special Topics-Accelerators and Beams 93, 014802, 2004. X. J. Luo, M. S. Shephard, R. M. O’Bara, P. McCorquodale, P. Colella, D. P. Grote, R. Nastasia, M. W. Beall, “Automatic p-ver- J.-L. Vay, “A node-centered local refinement J. W. Murphy, A. Burrows and A. Heger, sion mesh generation for curved domains,” algorithm for Poisson's equation in complex “Pulsational Analysis of the Cores of Massive Book of Abstracts: ICOSAHOM 4004, geometries,” Journal of Computational Stars and Its Relevance to Pulsar Kicks,” Brown University, June, 2004. Physics, 201, p. 34-60, 2004. Astrophysical Journal, 615:460-474, Nov. 2004. X. J. Luo, M. S. Shephard, R. M. O’Bara, E. Métral, C. Carli, M. Giovannozzi, M. R. Nastasia, M. W. Beall., “Automatic p-ver- Martini, R. R. Steerenberg, G. Franchetti, J. D. Myers, T. C. Allison, S. Bittner, D. sion mesh generation for curved domains,” I. Hofmann, J. Qiang, R. D. Ryne, Didier, M. Frenklach, W. H. Green, Jr., Y.- Engineering with Computers, 20(3), p. “Intensity Dependent Emittance Transfer L. Ho, J. Hewson, W. Koegler, C. 273-285, 2004. Studies at the CERN Proton Synchrotron,” Lansing, D. Leahy, M. Lee, R. McCoy, Proc. EPAC 2004, Lucerne, Switzerland, M. Minkoff, S. Nijsure, G. von Laszewski, K.-L. Ma and E. Lum, “Techniques for p. 1894, 2004. D. Montoya, L. Oluwole, C. Pancerella, Visualizing Time-Varying Volume Data,” in R. Pinzon, W. Pitz, L. A. Rahn, B. Ruscic, Visualization Handbook, C. Hansen and E. Métral, M. Giovannozzi, M. Martini, R. K. Schuchardt, E. Stephan, A. Wagner, T. C. Johnston, eds., Academic Press, Oct. R. Steerenberg, G. Franchetti, I. Windus, C. Yang, “A collaborative informat- 2004. Hofmann, J. Qiang, R. D. Ryne, “Space- ics infrastructure for multi-scale science,” Charge Experiments at the CERN Proton K.-L. Ma, G. Schussman and B. Wilson, Challenges of Large Applications in Synchrotron,” Proc. 33rd ICFA Advanced “Visualization for Computational Accelerator Distributed Environments (CLADE) Beam Dynamics Workshop on High Physics,” in Visualization Handbook, C. Workshop, Honolulu, Hawaii, June 2004.

110 LIST OF PUBLICATIONS

J. R. Myra, L. A. Berry, D. A. D’Ippolito, circular colliders,” Physical Review Letters J. Qiang, R. D. Ryne, I. Hofmann, “Space- E. F. Jaeger, “Nonlinear Fluxes and Forces 92, 214801, 2004. charge driven emittance growth in a 3D mis- from Radio-Frequency Waves with matched anisotropic beam,” Physical Review K. Ohmi, M. Tawada, Y. Cai, S. Kamada, Application to Driven Flows in Tokamaks,” Letters, 92, 174801, 2004. K. Oide, and J. Qiang, “Luminosity limit Physics of Plasmas 11, 1786, 2004. due to the beam-beam interactions with or J. Qiang, “Halo formation due to beam-beam D. Narayan, R. Bradshaw, A. Lusk and E. without crossing angle,” Physical Review interactions of beams optically mismatched at Lusk, “MPI cluster system software,” ST Accel. Beams, vol 7, 104401, 2004. injection,” Physical Review ST Accel. Recent Advances in Parallel Virtual Beams, vol 7, 031001, 2004. C. D. Ott, A. Burrows, E. Livne and R. Machine and Message Passing Interface, Walder, “Gravitational Waves from J. Qiang, “Strong-strong beam-beam simula- Lecture Notes in Computer Science, Axisymmetric, Rotating Stellar Core tion on parallel computer,” ICFA Beam 3241, p. 277-286. Springer-Verlag, 2004. Collapse,” Astrophysical Journal, 600:834- Dynamics Newsletter, No. 34, p.19-26, D. Narayan, R. Bradshaw, A. Lusk, E. 864, Jan. 2004. 2004. Lusk and R. Butler, “Component-based clus- S. Ovtchinnikov and X.-C. Cai, “One-level D. Quinlan, M. Schordan, Q. Yi and A. ter systems software architecture: A case Newton-Krylov-Schwarz algorithm for Saebjornsen, “Classification and Utilization study,” Proc. of the 6th IEEE unsteady nonlinear radiation diffusion prob- of Abstractions for Optimization,” First International Conference on Cluster lem,” Numerical Linear Alg. Applic., 11, International Symposium on Leveraging Computing (CLUSTER04), p. 319-326, 867-881, 2004. Applications of Formal Methods, Paphos, IEEE Computer Society, 2004. Cyprus, Oct. 2004. P. P. Pebay, H. N. Najm, and J. Pousin, “A J. W. Negele, et al. (The LHPC Non-Split Projection Strategy for Low Mach D. A. Randall, “The new CSU dynamical Collaboration), “Insight into nucleon struc- Number Reacting Flow,” International core,” Seminar on Recent Developments in ture from generalized parton distributions,” Journal of Multiscale Computational Eng., Numerical Methods of Atmosphere and Nuclear Physics (Proc. Suppl.) 129-130, 2(3):445-460, Aug. 2004. Ocean Modeling,” European Centre for 910, 2004. Mid-Range Weather Forecasting, UK, P. Petreczky and K. Petrov, “Free energy of J. W. Negele, et al., “Insight into nucleon Sept. 2004. a static quark anti-quark pair and the renor- structure from lattice calculations of moments malized Polyakov loop in three flavor QCD,” D. A. Randall and W. H. Schubert, of parton and generalized parton distribu- Physical Review D, 70, 054503, 2004. “Dreams of a stratocumulus sleeper,” tions,” Nuclear Physics (Proc. Suppl.) 128, Atmospheric Turbulence and Mesoscale 170, 2004. M. Prakash, et al., “Neutrino Processes in Meteorology, E. Fedorovich, R. Rotuno Supernovae and Neutron Stars in Their J. W. Negele, “What we are learning about and G. Stevens, eds., p. 71, Cambridge Infancy and Old Age,” in Compact Stars: the quark structure of hadrons from lattice University Press, 2004. The Quest for New States of Dense QCD,” 3rd International Symposium on Matter, Proc. of the KIAS-APTCP K. Robinson, L. J. Dursi, P. M. Ricker, R. the Gerasimov-Drell-Hearn Sum Rule International Symposium on Astro- Rosner, A. C. Calder, M. Zingale, J. W. and its Extensions (GDH 2004), Norfolk, Hadron Physics, Seoul, Korea, p. 476, Truran, T. Linde, A. Caceres, B. Fryxell, Virginia, June 2004. 2004. K. Olson, K. Riley, A. Siegel and N. C.-K. Ng, N. Folwell, A. Guetz, V. Ivanov, Vladimirova, “Morphology of Rising J. Pruet, R. Surman and G. C. L.-Q. Lee, Z. Li, G. Schussman and K. Hydrodynamic and Magnetohydrodynamic McLaughlin, “On the Contribution of Ko, “Simulating Dark Current In Nlc Bubbles from Numerical Simulations,” Gamma-Ray Bursts to the Galactic Inventory Structures,” Proc. of the 8th International Astrophysical Journal, 601:621-643, Feb. of Some Intermediate-Mass Nuclei,” Computational Accelerator Physics 2004. Astrophysical Journal Letters, 602:L101- Conference, St. Petersburg, Russia, June L104, Feb. 2004. A. L. Rosenberg, J. E. Menard, J. R. 2004. Wilson, S. Medley, R. Andre, D. Darrow, J. Pruet, T. A. Thompson and R. D. M. A. Nobes and H. D. Trottier, “Progress R. Dumont, B. P. LeBlanc, C. K. Phillips, Hoffman, “Nucleosynthesis in Outflows from in automated perturbation theory for heavy M. Redi, T. K. Mau, E. F. Jaeger, P. M. the Inner Regions of Collapsars,” quark physics,” Nuclear Physics (Proc. Ryan, D. W. Swain, R. W. Harvey, J. Astrophysical Journal, 606:1006-1018, Suppl.) 129-130, 355, 2004. Egedal and the NSTX Team, “Fast Ion May 2004. Absorption of the High Harmonic Fast Wave B. Norris, J. Ray, R. C. Armstrong, L. C. J. Qiang, M. Furman and R. Ryne, “A in the National Spherical Torus Experiment,” McInnes, D. E. Bernholdt, W. R. Elwasif, Parallel Particle-In-Cell Model for Beam- Physics of Plasmas 11, 2441, 2004. A. D. Malony and S. Shende, Beam Interactions in High Energy Ring “Computational quality of service for scientif- B. Ruscic, “Active Thermochemical Table,” in Colliders,” Journal of Computational ic components,” Proc. of the International 2005 Yearbook of Science and Physics vol. 198, 278, 2004. Symposium on Component-Based Technology, an annual update to the Software Engineering (CBSE7), J. Qiang and R. Gluckstern, “Three- McGraw-Hill Encyclopedia of Science Edinburgh, Scotland, May 2004. Dimensional Poisson Solver for a Charged and Technology, McGraw-Hill, New Beam with Large Aspect Ratio in a York, 2004. K. Ohmi, M. Tawada, Y. Cai, S. Kamada, Conducting Pipe,” Computer Physics K. Oide, J. Qiang, “Beam-beam limit in e+e- Communications, 160, 120, 2004.

111 LIST OF PUBLICATIONS

B. Ruscic, R. E. Pinzon, M. L. Morton, G. Annual OSCAR Symposium, Winnipeg, F. D. Swesty, E. S. Myra, “Advances in von Laszewski, S. Bittner, S. G. Nijsure, Manitoba Canada, May 2004. Multi-Dimensional Simulation of Core- K. A. Amin, M. Minkoff, A. F. Wagner. Collapse Supernovae,” in Open Issues in S. L. Scott, I. Haddad and C. 2004, “Introduction to active thermochemical Core Collapse Supernova Theory, Proc. Leangsuksun, “HA-OSCAR: Towards tables: Several “key” enthalpies of formation of the Institute for Nuclear Theory, Vol. Highly Available Linux Clusters,” Linux revisited,” Journal of Physical Chemistry 14, p. 176, University of Washington, World Magazine, 2, 3, March 2004. A, 108:9979-9997, 2004. Seattle, Wash., June 2004. M. S. Shephard, M. W. Beall, R. M. R. S. Samtaney, S. Jardin, P. Colella, D. F. T. J. Tautges, “MOAB-SD: Integrated O’Bara, B. E. Webster, “Toward simulation- Martin, “3D adaptive mesh refinement simu- Structured and Unstructured Mesh based design,” Finite Elements in Analysis lations of pellet injection in tokamaks,” Representation,” Engineering With and Design, 40, p. 1575-1598, 2004. Computer Physics Communications 164 Computers, 20:286-293, 2004. 1-3 220, 2004. M. S. Shephard, M. W. Beall, B. E. S. Thompson, B. Govindasamy, A. Mirin, Webster, “Supporting Simulations to Guide A. R. Sanderson, et al., “Display of Vector K. Caldeira, C. Delire, J. Milovich, M. Engineering Design,” AIAA-2004-4448, Fields Using a Reaction-Diffusion Model,” Wickett and D. J. Erickson III, AIAA, Richmond, Va., 2004. Proc. of the 2004 IEEE Visualization “Quantifying the effects of CO2-fertilized Conference, 115-122, 2004. J. Shigemitsu, et al. (The HPQCD collab- vegetation on future global climate and car- oration), “Heavy-light Meson Semileptonic bon dynamics,” Geophysical Research D. P. Schissel, “The National Fusion Decays with Staggered Light Quarks,” Letters, 31, L23211, 2004. Collaboratory Project,” Proc. of the 2004 Nuclear Physics (Proc. Suppl.) 129-130, DOE SciDAC PI Meeting, 2004. M. M. Tikir, J. K. Hollingsworth, “Using 331, 2004. Hardware Counters to Automatically D. P. Schissel, “The National Fusion L. O. Silva et al., “Proton shock acceleration Improve Memory Performance,” Proc. of Collaboratory Project: Applying Grid in laser-plasma interactions,” Physical ACM/IEEE SC2004 conference, Technology for Magnetic Fusion Research,” Review Letters, 92, 015002, 2004. Pittsburgh, Penn., Nov. 2004. Proc. of the Workshop on Case Studies on Grid Applications at GGF10, 2004. F. Song, F. Wolf, N. Bhatia, J. Dongarra, S. J. L. Tilson, C. Naleway, M. Seth, R. Moore, “An Algebra for Cross-Experiment Shepard, A. F. Wagner, and W. C. Ermler, D. P. Schissel, et al., “Building the U.S. Performance Analysis,” 2004 International “An Ab Initio Study of AmCl+: f-f National Fusion Grid: Results from the Conference on Parallel Processing (ICCP- Spectroscopy and Chemical Binding,” Journal National Fusion Collaboratory Project,” The 04), Montreal, Quebec, Canada, Aug. of Chemical Physics, 121, 5661-5675, 4th IAEA Technical Meeting on Control, 2004. 2004. Data Acquisition, and Remote Participation for Fusion Research. Fusion C. R. Sovinec, A. H. Glasser, D. C. A. Toselli and O. Widlund, “Domain Engineering and Design 71, 245-250, Barnes, T. A. Gianakon, R. A. Nebel, S. E. Decomposition Methods - Algorithms and 2004. Kruger, D. D. Schnack, S. J. Plimpton, A. Theory,” Springer Series in Computational Tarditi, M. S. Chu and the NIMROD Mathematics, 34, Oct. 2004. W. Schroers, et al. (The LHPC Team, “Nonlinear magnetohydrodynamics Collaboration), “Moments of nucleon spin- H. D. Trottier, “Higher-order perturbation with high-order finite elements,” Journal Of dependent generalized parton distributions,” theory for highly-improved actions,” Nuclear Computational Physics 195, 355, 2004. Nuclear Physics (Proc. Suppl.) 129-130, Physics (Proc. Suppl.) 129-130, 142, 2004. 907, 2004. H. R. Strauss, A. Pletzer, W. Park, S. F. S. Tsung et al., “Near-GeV-energy laser- Jardin, J. Breslau, and L. Sugiyama, W. H. Schubert, “Generalization of Ertel's wakefield acceleration of self-injected electrons “MHD simulations with resistive wall and potential vorticity to a moist atmosphere,” in a centimeter-scale channel” Physical magnetic separatrix,” Computer Physics Meteorologische Zeitschrift, 13, (Hans Review Letters 93, 185002, 2004. Communications 164, 1-3 40-45, 2004. Ertel Special Issue), 465, 2004. S. Vadlamani, S. E. Parker, Y. Chen, “The H. R. Strauss, L. Sugiyama, G. Y. Fu, W. G. Schussman and K.-L. Ma, “Anisotropic Particle-Continuum method: an algorithmic Park, J. Breslau, “Simulation of two fluid Volume Rendering for Extremely Dense, Thin unification of particle-in-cell and continuum and energetic particle effects in stellarators,” Line Data,” Proc. of the IEEE methods,” Computer Physics Nuclear Fusion 44, 9, 1008-1014, 2004. Visualization 2004 Conference, Austin, Communications, 164, 209, 2004. Texas, Oct. 2004. M. M. Strout and P. D. Hovland, “Metrics J.-L. Vay, P. Colella, J. W. Kwan, P. and Models for Reordering Transformations,” S. L. Scott, C. B. Leangsuksun, T. Roa, J. McCorquodale, D. B. Serafini, A. Proc. of the Second ACM SIGPLAN Xu, and R. Libby, “Experiments and Friedman, D. P. Grote, G. Westenskow, J. Workshop on Memory System Analysis towards effective cluster manage- C. Adam, A. Héron and I. Haber, Performance (MSP), June 2004. ment system,” 2004 High Availability and “Application of adaptive mesh refinement to Performance Workshop (HAPCW), Santa M. M. Strout and P. D. Hovland, “Metrics particle-in-cell simulations of plasmas and Fe, NM, Oct. 2004. and Models for Reordering Transformations,” beams,” Physics of Plasmas 11, 2928, 2004. Proc. of the Second ACM SIGPLAN S. L. Scott, C. Leangsuksun, T. Lui and R. J-L. Vay, P. Colella, A. Friedman, D. P. Workshop on Memory System Libby, “HA-OSCAR Release 1.0 Beta: Grote, P. McCorquodale, D. B. Serafini, Performance (MSP), June 2004. Unleashing HA-Beowulf,” Proc.of 2nd “Implementations of mesh refinement schemes

112 LIST OF PUBLICATIONS

for particle-in-cell plasma simulations,” Astrophysical Journal Supplement, U. M. Yang, “On the Use of Relaxation Computer Physics Communications 164, 151:75-102, March 2004. Parameters in Hybrid Smoothers,” Numer. p. 297-305, 2004. Linear Algebra Appl., 11, p. 155-172, S. E. Woosley, S. Wunsch and M. Kuhlen, 2004. M. Vertenstein, K. Oleson, S. Levis, and F. “Carbon Ignition in Type Ia Supernovae: An M. Hoffman, “Community Land Model Analytic Model,” Astrophysical Journal, Q. Yi, K. Kennedy, H. You, K. Seymour, Version 3.0 (CLM3.0) User's Guide,” 607:921-930, June 2004. and J. Dongarra, “Automatic Blocking of QR National Center for Atmospheric and LU Factorizations for Locality,” Second P. H. Worley, J. Levesque, “The Research, June 2004. ACM SIGPLAN Workshop on Memory Performance Evolution of the Parallel Ocean System Performance, Washington, D.C., R. Vuduc, J. W. Demmel and J. A. Bilmes, Program on the Cray X1,” Proc. of the 46th June 2004. “Statistical Models for Empirical Search- Cray User Group Conference, Knoxville, Based Performance Tuning,” International TN, May 2004. Q. Yi and D. Quinlan, “Applying Loop Journal of High Performance Computing Optimizations to Object-Oriented J. C. Wright, P. T. Bonoli, M. Brambilla, F. Applications, 18 (1), p. 65-94, February Abstractions Through General Classification Meo, E. D’Azevedo, D. B. Batchelor, E. F. 2004. of Array Semantics,” 17th International Jaeger, L. A. Berry, C. K. Phillips and A. Workshop on Languages and Compilers G. Wallace, et al., “Automatic Alignment of Pletzer, “Full Wave Simulations of Fast for Parallel Computing, West Lafayette, Tiled Displays for a Desktop Environment,” Wave Mode Conversion and Lower Hybrid Ind., Sept. 2004. Journal of Software. 15(12): 1776-1786, Wave Propagation in Tokamaks,” Physics of Dec. 2004. Plasmas 11, 2473, 2004. H. Yu, K.-L. Ma and J. Welling, “A Parallel Visualization Pipeline for Terascale W. X. Wang, et al., “Global delta-f particle J. C. Wright, P. T. Bonoli, E. D'Azevedo Earthquake Simulations,” Proc. of the simulation of neoclassical transport and and M. Brambilla, “Ultrahigh resolution ACM/IEEE SC04 conference, ambipolar electric field in general geometry,” simulations of mode converted ion cyclotron Pittsburgh, Penn., Nov. 2004. Computer Physics Communications, 164, waves and lower hybrid waves,” Proc. of 178, 2004. the 18th International Conference on the H. Yu, K.-L. Ma and J. Welling, “I/O Numerical Simulation of Plasmas, Strategies for Parallel Rendering of Large T. P. Wangler, C. K. Allen, K. C. D. Chan, Computer Physics Communications 164, Time-Varying Volume Data,” Proc. of the P. L. Colestock, K. R. Crandall, R. W. 330, 2004. Eurographics/ACM SIGGRAPH Garnett, J. D. Gilpatrick, W. Lysenko, J. Symposium on Parallel Graphics and Qiang, J. D. Schneider, M. E. Schulze, R. K. Wu, J. Ekow, E. J. Otoo, and A. Visualization, June 2004. L. Sheffield, H. V. Smith, “Beam-Halo in Shoshani, “On the performance of bitmap Mismatched Proton Beams,” Instrument indices for high cardinality attributes,” Proc. O. Zatsarinny, T. W. Gorczyca, K. T. Methods Physics Res. A 519, 425, 2004. of the Thirtieth International Conference Korista, N. R. Badnell and D. W. Savin, on Very Large Data Bases, Toronto, “Dielectronic recombination data for dynamic J. Wei, A. Fedotov, W. Fischer, N. Canada, Morgan Kaufmann, Sept. 2004. finite-density plasmas: IV. The Carbon iso- Malitsky, G. Parzen, J. Qiang, “Intra-beam electronic sequence,” Astronomy and Scattering Theory and RHIC Experiments,” K. Wu, W.-M. Zhang, V. Perevoztchikov, Astrophysics, 417, 1173, 2004. Proc. 33rd ICFA Advanced Beam J. Lauret and A. Shoshani, “The grid collec- Dynamics Workshop on High Intensity & tor: Using an event catalog to speed up user H. Zhang, et al., “Providing Data Transfer High Brightness Hadron Beams, analysis in distributed environment,” Proc. with QoS as Agreement-Based Service,” Bensheim, Germany, Oct. 2004. of Computing in High Energy and Proc. of the IEEE International Nuclear Physics (CHEP), Interlaken, Conference on Services Computing, 2004. M. Wingate, et al. (The HPQCD collabo- Switzerland, Sept. 2004. ration), “Progress Calculating Decay W. Zhang, S. E. Woosley and A. Heger, Constants with NRQCD and AsqTad S. Wunsch and S. E. Woosley, “Convection “The Propagation and Eruption of actions,” Nuclear Physics (Proc. Suppl.) and Core-Center Ignition in Type Ia Relativistic Jets from the Stellar Progenitors 129-130, 325, 2004. Supernovae,” Astrophysical Journal, of Gamma-Ray Bursts,” Astrophysical 616:1102-1108, Dec. 2004. Journal, 608:365-377, June 2004. M. Wingate, et al. (The HPQCD Collaboration), “The Bs and Ds decay con- D. Xu and B. Bode, “Performance Study stants in 3 flavor lattice QCD,” Physical and Dynamic Optimization Design for Review Letters 92, 162001, 2004. Thread Pool Systems,” Proc. of the International Conference on Computing, F. Wolf, B. Mohr, J. Dongarra, S. Moore, Communications and Control “Efficient Pattern Search in Large Traces Technologies (CCCT’2004), Austin, through Successive Refinement,” Proc. of 2005-2006 Texas, Aug. 2004. Euro-Par 2004, Pisa, Italy, Aug.-Sept. 2004. C. Yang, W. Gao, Z. Bai, X. Li, L. Lee, P. G. Abla, et al., “Shared Display Wall Based Husbands and E. Ng., “Algebraic Sub-struc- S. E. Woosley, A. Heger, A. Cumming, R. Collaboration Environment in the Control turing for Large-scale Electromagnetic D. Hoffman, J. Pruet, T. Rauscher, J. L. Room of the DIII-D National Fusion Applications,” PARA'04 Workshop on Fisker, H. Schatz, B. A. Brown and M. Facility,” Fifth Annual Workshop on State-of-the-art in Scientific Computing, Wiescher, “Models for Type I X-Ray Bursts Advanced Collaborative Environments Copenhagen, Denmark, June 2004. with Improved Nuclear Physics,” (WACE2005), 2005.

113 LIST OF PUBLICATIONS

G. Abla, et al., “Advanced Tools for Volume Data,” Proc. of the International A. Andrady, P. J. Aucamo, A. F. Bais, C. Enhancing Control Room Collaborations,” Workshop on Volume Graphics 2005, L. Ballare, L. O. Bjorn, J. F. Bornman, M. The 5th IAEA Technical Meeting on Stony Brook, NY, June 2005. Caldwell, A. P. Cullen, D. J. Erickson III, Control, Data Acquisition, and Remote F. R. de Gruijl, D.-P. Hader, M. ilyas, G. M. A. Albao, D.-J. Liu, C. H. Choi, M. S. Participation for Fusion Research, 2005. Kulandaivelu, H. D. Kumar, J. Longstreth, Gordon and J. W. Evans, “Competitive R. L. McKenzie, M. Norval, H. H. Mark Adams, “Ax=b: The Link Between Etching and Oxidation of Vicinal Si(100) Redhwi, R. C. Smith, K. R. Solomon, Y. Gyrokinetic Particle Simulations of Turbulent Surfaces,” MRS Proceedings Vol. 859E, Takizawa, X. Tang, A. H. Teramura, A. Transport in Burning Plasmas and Micro-FE JJ3.6., 2005. Torikai, J. C. van der Leun, S. Wilson, R. Analysis of Whole Vertebral Bodies in M. A. Albao, M. M. R. Evans, J. Nogami, C. Worrest and R. G. Zepp, Orthopaedic Biomechanics,” invited talk, D. Zorn, M. S. Gordon and J. W. Evans, “Environmental effects of ozone depletion and SciDAC Conference, San Francisco, Calif., “Atomistic processes mediating growth of one- its interactions with climate change: Progress June 2005. dimensional Ga rows on Si(100),” Physical Report 2004,” Photochem. Photobiol. Sci., C. M. Aikens, G. D. Fletcher, M. W. Review B 72, 035426, 2005. 2005. Schmidt and M. S. Gordon, “Scalable M. A. Albao, D.-J. Liu, M. S. Gordon and D. J. Antonio et al. (The RBC and Implementation Of Analytic Gradients For J. W. Evans, “Simultaneous Etching and UKQCD Collaborations), “Baryons in 2+1 Second-Order Z-Averaged Perturbation Oxidation of Vicinal Si(100) Surfaces: flavour domain wall QCD,” Proc. of XXIII Theory Using The Distributed Data Atomistic Lattice-Gas Modeling of International Symposium on Lattice Field Interface,” Journal of Chemical Physics., Morphological Evolution,” Physical Review Theory (Lattice 2005), 098, 2005. 124, 014107, 2006. B 72, 195420, 2005. C. Aubin, et al. (The MILC V. Akcelik, J. Bielak, I. Epanomeritakis, O. A. Alexandru, et al., “Progress on a Collaboration), “Properties of light quarks Ghattas and E. Kim, “High fidelity forward Canonical Finite Density Algorithm,” Nuclear from lattice QCD simulations,” Journal of and inverse earthquake modeling in large Physics B (Proc. Suppl.) 140, 517, 2005. Physics: Conference Proceedings, 16, 160, basins,” Proc. of the Sixth European 2005. Conference on Structural Dynamics C. Alexandru, et al. (The LHPC EURODYN'05, Paris, France, Sept. 2005. Collaboration), “First principles calculations C. Aubin, et al. (The Fermilab Lattice, of nucleon and pion form factors: MILC, and HPQCD Collaborations), V. Akcelik, G. Biros, A. Dragenescu, J. Understanding the building blocks of nuclear “Semileptonic decays of D mesons in three-fla- Hill, O. Ghattas and B. van Bloemen matter from lattice QCD,” Journal of vor lattice QCD,” Physical Review Letters Waanders, “Dynamic data-driven inversion Physics Conf. Ser. 16, 174, 2005. 94, 011601, 2005. for terascale simulations: Real-time identifica- tion of airborne contaminants,” Proc. of, C. Alexandrou, et al., “Momentum depend- C. Aubin, et al. (The Fermilab Lattice, ACM/IEEE SC2005 conference, Seattle, ence of the N to Delta transition form fac- MILC, and HPQCD Collaborations), Wash., Nov. 2005. tors,” Nuclear Physics B (Proc. Suppl.) “Charmed meson decay constants in three fla- 140, 293, 2005. vor lattice QCD,” Physical Review Letters V. Akcelik, G. Biros, O. Ghattas, D. 95, 122002, 2005. Keyes, K. Ko, L.-Q. Lee and E. Ng, I. F. Allison et al. (The HPQCD “Adjoint Methods For Electromagnetic Shape Collaboration), “Mass of the B/c meson in C. Aubin, et al. (The MILC Optimization Of The Low-Loss Cavity For three-flavor lattice QCD,” Physical Review Collaboration), “Topological susceptibility The International Linear Collider,” Proc. of Letters 94, 172001, 2005. with three flavors of staggered quarks,” SciDAC 2005 Conference, San Francisco, Nuclear Physics. B (Proc. Suppl.) 140, I. F. Allison et al. (The HPQCD Calif., June 2005. 600, 2005. Collaboration), “A precise determination of V. Akcelik, G. Biros, O. Ghattas, D. the B/c mass from dynamical lattice QCD,” C. Aubin, et al. (The MILC Keyes, K. Ko, L.-Q. Lee and E. G. Ng, Nuclear Physics (Proc. Suppl.) 140, 440, Collaboration), “Results for light pseu- “Adjoint Methods for Electromagnetic Shape 2005. doscalars from three-flavor simulations,” Optimization of the Low-Loss Cavity for the Nuclear Physics. B (Proc. Suppl.) 140, A. S. Almgren, J. B. Bell, C. A. International Linear Collider,” Proc. of 231, 2005. Rendleman, and M. Zingale, “Low Mach SciDAC 2005, San Francisco, Calif., June Number Modeling of Type Ia Supernovae,” C. Aubin, et al. (The MILC 2005. ApJ, 637, 922, 2006. Collaboration), “The scaling dimension of V. Akcelik, G. Biros, O. Ghattas, J. Hill, D. low lying Dirac eigenmodes and of the topo- J. Amundson, P. Spentzouris,“Space charge Keyes, and B. van Bloeman Waanders, logical charge density,” Nuclear Physics. B experiments and simulation in the Fermilab “Parallel PDE constrained optimization,” in (Proc. Suppl.) 140, 626, 2005. Booster,” Proc. of Particle Accelerator Frontiers of Parallel Computing, M. Conference (PAC 05), Knoxville, Tenn., B. Bader, O. Ghattas, B. van Bloemen Heroux, P. Raghaven, and H. Simon, eds, May 2005. Waanders and K. Willcox, “An optimiza- SIAM, to appear 2006. tion framework for goal-oriented model-based J. Amundson, P. Spentzouris, J.Qiang and H. Akiba, K.-L. Ma, and J. Clyne, “End-to- reduction of large-scale systems,” 44th IEEE R. Ryne, “Synergia: An accelerator modeling end Data Reduction and Hardware Conference on Decision and Control, tool with 3-D space charge,” Journal of Accelerated Rendering Techniques for Seville, Spain, Dec. 2005. Computational Physics vol. 211, 229, Visualizing Time-Varying Non-uniform Grid 2006.

114 LIST OF PUBLICATIONS

D. H. Bailey and A. S. Snavely, G. J. O. Beran, B. Austin, A. Sodt, and M. equation,” J. Comput. Phys. 209, p. 1-18, “Performance Modeling: Understanding the Head-Gordon, “Unrestricted perfect pairing: 2005. Present and Predicting the Future,” Proc. of The simplest wave-function-based model N. Bhatia, F. Song, F. Wolf, J. Dongarra, SIAM PP04, 2005. chemistry beyond mean field,” Journal of B. Mohr, S. Moore, “Automatic Physical Chemistry A, 109, 9183-9192, A. H. Baker, E. R. Jessup, and T. A. Experimental Analysis of Communication 2005. Manteuffel, “A technique for accelerating the Patterns in Virtual Topologies,” Proc. of the convergence of GMRES,” SIAM J. Mat. C. Bernard, et al. (The MILC International Conference on Parallel Anal. & App., 26, 4, p. 962, 2005. Collaboration), “The locality of the fourth Processing (ICPP-05), Oslo, Norway, June root of staggered fermion determinant in the 2005. S. Basak, et al., “Combining quark and link interacting case,” XXIII International smearing to improve extended baryon opera- B. Bhattacharjee, P. Lemonidis, W. H. Symposium on Lattice Field Theory tors,” Proc. of XXIII International Green, and P. I. Barton, “Global Solution of (Lattice 2005), 299, 2005. Symposium on Lattice Field Theory Semi-Infinite Programs,” Mathematical (Lattice 2005), 076, 2005. C. Bernard, et al. (The MILC Programming, 103, 283, 2005. Collaboration), “More evidence of localiza- S. Basak, et al. (The LHPC G. Biros and O. Ghattas, “Parallel tion in the low-lying Dirac spectrum,” XXIII Collaboration), “Baryonic sources using irre- Lagrange-Newton-Krylov-Schur Methods for International Symposium on Lattice Field ducible representations of the double-covered PDE-Constrained Optimization. Part I: The Theory (Lattice 2005), 299, 2005. octahedral group,” Nuclear Physics. (Proc. Krylov-Schur Solver,” SIAM Journal on Suppl.) 140, 281, 2005. C. Bernard, et al. (The MILC Scientific Computing, 27, 687, 2005. Collaboration), “The Equation of State for S. Basak, et al. (The LHPC G. Biros and O. Ghattas, “Parallel QCD with 2+1 Flavors of Quarks,” Collaboration), “Analysis of N* spectra using Lagrange-Newton-Krylov-Schur Methods for Proceedings of Science, Lattice 2005, 156, matrices of correlation functions based on irre- PDE-Constrained Optimization. Part II: The 2005. ducible baryon operators,” Nuclear Physics Lagrange-Newton Solver, and its Application (Proc. Suppl.) 140, 278, 2005. C. Bernard, et al. (The MILC to Optimal Control of Steady Viscous Flows,” Collaboration), “Update on pi and K SIAM Journal on Scientific Computing, S. Basak, et al. (The LHPC Physics,” XXIII International Symposium 27, 714, 2005. Collaboration), “Group-theoretical construc- on Lattice Field Theory (Lattice 2005), tion of extended baryon operators,” Nuclear B. Bistrovic, et al. (The LHPC 025, 2005. Physics. (Proc. Suppl.) 140, 287, 2005. Collaboration), “Understanding hadron C. Bernard, et al. (TheMILC structure from lattice QCD in the SciDAC S. Basak, et al. (The LHPC Collaboration), “QCDThermodynamics with era,” Journal of Physics Conf. Ser. 16, 150, Collaboration), “Group-theoretical construc- Three Flavors of Improved Staggered 2005. tion of extended baryon operators in lattice Quarks,” Physical Review D, 71, 034504, QCD,” Physical Review D, 72, 094506, J. M. Blondin, “Discovering new dynamics of 2005. 2005. core-collapse supernova shock waves,” Journal C. Bernard, et al. (The MILC of Physics Conference Series, 16, 370-379, S. Basak, et al. (The LHPC Collaboration), “Heavy-light decay con- 2005. Collaboration), “Clebsch-Gordan construc- stants using clover valence quarks and three tion of lattice interpolating fields for excited J. M. Blondin, A. Mezzacappa, “The flavors of dynamical improved staggered baryons,” Physical Review D, 72, 074501, Spherical Accretion Shock Instability in the quarks,” Nuclear Physics. B (Proc. Suppl.) 2005. Linear Regime,” ArXiv Astrophysics e- 140, 449, 2005. prints, July 2005. A. C. Bauer, M. S. Shephard, M., C. Bernard, et al. (The MILC Nuggehally, “Design overview of the TSTT T. Blum, “Lattice calculation of the lowest- Collaboration), “Three Flavor QCD at integrated field approximation for numerical order hadronic contribution to the muon High Temperatures,” Nuclear Physics B simulation (FANS) library,” Proc. 8th US anomalous magnetic moment,” Nuclear (Proc. Suppl.) 140, 538, 2005. Nat. Congress on Comp. Mech., 2005. Physics B (Proc. Suppl.) 140, 311, 2005. M. Berndt, T. Manteuffel, and S. J. B. Bell, M. S. Day, J. F. Grcar, and M. J. T. Blum and P. Petreczky, “Cutoff effects in McCormick, and G. Starke, “Analysis of Lijewski, “Active Control for Statistically meson spectral functions,” Nuclear Physics B first-order system least squares (FOSLS) for Stationary Turbulent Premixed Flame (Proc. Suppl.) 140, 553, 2005. elliptic problems with discontinuous coeffi- Simulations,” Communications in Applied cients: Part I,” SIAM Journal on Numerical F.D.R. Bonnet, et al. (The LHPC Mathematics and Computational Science, Analysis, 43, p. 386-408, 2005. Collaboration), “Lattice computations of the 1, 29, 2006. pion form factor,” Phys. Rev. D 72, 054506, M. Berndt, T. Manteuffel and S. J. B. Bell, M. S. Day, I. G. Shepherd, M. 2005. McCormick, “Analysis of first-order system Johnson, R. K. Cheng, J. F. Grcar, V. E. least squares (FOSLS) for elliptic problems P. A. Boyle, et al., “Overview of the Beckner, and M. J. Lijewski, “Numerical with discontinuous coefficients: Part II,” QCDSP and QCDOC computers,” IBM Simulation of a Laboratory-Scale Turbulent SIAM Journal on Numerical Analysis, 43, Research Journal 49, 351, 2005. V- flame,” Proceedings of the National p. 409-436, 2005. Academy of Sciences, 102 (29) 10006, P. A. Boyle, et al., “QCDOC: Project status 2005. M. Barad, P. Colella, “A fourth-order accu- and first results,” Journal of Physics Conf. rate local refinement method for Poisson's Ser. 16, 129, 2005.

115 LIST OF PUBLICATIONS

P. A. Boyle, et al., “Localisation and chiral L. Bytautas and K. Ruedenberg, L. Chen, F. Zonca, and Z. Lin, “Nonlinear symmetry in 2+1 flavour domain wall “Correlation Energy Extrapolation through Toroidal Mode Coupling: A New Paradigm QCD,” Proc. of XXIII International Intrinsic Scaling. IV. Accurate Binding for Drift Wave Turbulence in Toroidal Symposium on Lattice Field Theory Energies of the Homonuclear Diatomic Plasmas,” Plasma Phys. Contr. Fusion 47, (Lattice 2005), 141, 2005. Molecules Carbon, Nitrogen, Oxygen and B71, 2005. Fluorine,” Journal of Chemical Physics, P. A. Boyle, et al., “The QCDOC project,” M. Cheng, et al. (The RBC-Bielefeld 122, 154110, 2005. Nuclear Physics (Proc. Suppl.) 140, 169, Collaboration), “Scaling test of the P4- 2005. X.-C. Cai, L. Marcinkowski, and P. improved staggered fermion action,” Proc. of Vassilevski, “An element agglomeration XXIII International Symposium on J. Brannick, M. Brezina, S. MacLachlan, nonlinear additive Schwarz precondi- Lattice Field Theory (Lattice 2005), 045, T. Manteuffel, S. McCormick, and J. tioned Newton method for unstructured 2005. Ruge, “An energy-based AMG coarsening finite element problems,” Applications of strategy,” Journal on Numerical Analysis,. N. Chevaugeon, P. Hu, X. Li, J. Xin, D. Mathematics, 50, p. 247-275, 2005. Lin. Alg. Appl., 13, 133, 2006. Cler, J. E. Flaherty, M. S. Shephard, C. Y. Cardall, E. J. Lentz, A. Mezzacappa, “Discontinuous Galerkin methods applied to D. P. Brennan, S. E. Kruger, T. A. “Conservative special relativistic radiative shock and blast problems,” Journal of Gianakon, and D. D. Schnack, “A catego- transfer for multidimensional astrophysical Scientific Computing, 22, p. 227-243, rization of tearing mode onset in tokamaks simulations: Motivation and elaboration,” 2005. via nonlinear simulation,” Nuclear Fusion Physical Review D, 72, 043007, 2005. 45. 1178, 2005. E. Chow, “An Aggregation Multilevel C. Y. Cardall, A. O. Razoumov, E. Method Using Smooth Error Vectors,” SIAM M. Brezina, R. Falgout, S. MacLachlan, T. Endeve, E. J. Lentz, A. Mezzacappa, Journal of Scientific Computing, 27, 1727, Manteuffel, S. McCormick and J. Ruge, “Toward five-dimensional core-collapse super- 2006. “Adaptive Smoothed Aggregation (_SA),” nova simulations,” Journal of Physics SIAM Review: SIGEST, 47, p. 317, 2005. M. Choi, V. S. Chan, V. Tang, P. Bonoli, Conference Series, 16, 390-394, 2005. R. I. Pinsker, “Monte-Carlo Orbit/Full Wave M. Brezina, R. Falgout, S. MacLachlan, T. G. R. Carr, M. J. Cordery, J. B. Drake, M. Simulation of Fast Alfvén (FW) Damping Manteuffel, S. McCormick and J. Ruge, W. Ham, F. M. Hoffman and P. H. on Resonant Ions in Tokamaks,” Radio “Adaptive Algebraic Multigrid,” SIAM Worley, “Porting and Performance of the Frequency Power in Plasmas, Proc. 16th Journal of Scientific Computing, 27, 1534, Community Climate System Model Topical Conf., Park City, Utah, 2005. 2006. (CCSM3) on the Cray X1,” Proc. of the M. A. Clark, P. de Forcrand and A. D. M. Brezina, C. Tong, R. Becker, “Parallel 47th Cray User Group Conference, Kennedy, “Algorithm shootout: R versus Algebraic Multigrids for Structural Albuquerque, NM, May 2005. RHMC,” Proc. of XXIII International Mechanics,” SIAM Journal of Scientific L. Carrington, M. Laurenzano, A. Symposium on Lattice Field Theory Computing, to appear. Snavely, R. Campbell and L. Davis, “How (Lattice 2005), 115, 2005. R. C. Brower, H. Neff, and K. Orginos, well can simple metrics represent the perform- C. S. Co, A. Friedman, D. P. Grote, J.- “Moebius fermions: Improved domain wall ance of HPC applications?” Proc. of the L.Vay, W. Bethel, K. I. Joy, “Interactive chiral fermions,” Nuclear Physics. (Proc. ACM/IEEE SC05 conference, Seattle, Methods for Exploring Particle Simulation Suppl.) 140, 686, 2005. Wash., Nov. 2005. Data,” Proc. of EuroVis 2005 conference, J. R. Burruss, et al., “Security on the U. S. T. Chartier, R. Falgout, V. E. Henson, J. 2005. Fusion Grid,” The 5th IAEA Technical Jones, T. Manteuffel, S. McCormick, J. B. Cohen, E. B. Hooper, R. H. Cohen, D. Meeting on Control, Data Acquisition, Ruge and P. Vassilevski, “Spectral element N. Hill, H. S. McLean, R. D. Wood, S. and Remote Participation for Fusion agglomerate AMGe,” Lecture Notes in Woodruff, C. R. Sovinec, and G. A. Cone, Research, 2005. Computational Science and Engineering, “Simulation of spheromak evolution and ener- to appear. J. R. Burruss, et al., “Simplifying FusionGrid gy confinement,” Physics of Plasmas, 12, Security,” Proc. Challenges of Large J. Chen, S. C. Jardin, and H. R. Strauss, 056106, 2005. Applications in Distributed “Solving anisotropic transport equations on S. Cohen, “Preliminary Study of BK on 2+1 Environments, 95-105, 2005. misaligned grids,” Lecture Notes In flavor DWF lattices from QCDOC,” Proc. Computer Science, 3516, 1076-1079, J. R. Burruss, “The Resource Oriented of XXIII International Symposium on 2005. Authorization Manager (ROAM),” Proc. Lattice Field Theory (Lattice 2005), 346, 14th IEEE International Symposium on J. Chen, J. Breslau, G. Fu, S. Jardin, W. 2005. High Performance Distributed Park, “Symmetric solution in M3D,” P. Colella, D. T. Graves, B. J. Keen, D. Computing, 308-309, 2005. Computer Physics Communications 164, Modiano, “A cartesian grid embedded 1-3, 468-471, 2005. J. R. Burruss, et al., “Developments in boundary method for hyperbolic conservation Remote Collaboration and Computation,” H. Chen, et al., “Memory Performance laws,” Journal of Computational Physics, The 14th ANS Topical Meeting on the Optimizations for Real-Time Software 211, p. 347-366, 2006. Technology of Fusion Energy, Fusion HDTV Decoding,” The Journal of VLSI I. R. Corey, J. R. Johnson and J. S. Vetter, Science and Technology, 47, 3, 814, 2005. Signal Processing, 21, 2, 193-207, 2005. “Performance Assertions and Sensitivity

116 LIST OF PUBLICATIONS

Analysis: A Technique for Developing System Administration Conference (LISA R. J. Dumont, C. K. Phillips and D. N. Adaptive Grid Applications,” Proceedings XIX), San Diego, Calif., Dec. 2005. Smithe, “Effects of non-Maxwellian species of the Eleventh IEEE International on ion cyclotron wave propagation and N. Desai, A. Lusk, R. Bradsha, and E. Symposium on High Performance absorption in magnetically confined plasmas,” Lusk, “{MPISH}: A parallel shell for {MPI} Distributed Computing, (HPDC), Physics of Plasmas 12, 042508, 2005. programs,” Proc. of the 1st Workshop on Edinburgh, Scotland, July 2006. System Management Tools for Large- R. G. Edwards, et al. (The LHPC A. P. Craig, R. L. Jacob, B. Kauffman, T. Scale Parallel Systems (IPDPS '05), Collaboration), “Hadron structure with Bettge, J. Larson, E. Ong, C. Ding, and Y. Denver, Colo., 2005. light dynamical quarks,” Proc. of XXIII He, “CPL6: The New Extensible, High- International Symposium on Lattice Field Z. M. Djurisic, D. Amusin, T. Bereknyei, Performance Parallel Coupler for the Theory (Lattice 2005), 056, 2005. T. C. Allison, M. Frenklach, “Reporting of Community Climate System Model,” experimental data for development and vali- R. G. Edwards et al. (The LHPC and International Journal of High dation of chemical kinetic models,” Fall UKQCD Collaborations), “The Chroma Performance Computing Applications, Meeting of the Western States Section of Software System for Lattice QCD,” Nuclear 19, 3, p. 309, Aug. 2005. the Combustion Institute, Stanford, CA, Physics (Proc. Suppl.) 140, 832, 2005. R. K. Crockett, P. Colella, R. Fisher, R. I. Oct. 2005. S. Ethier, “Gyrokinetic Particle-in-Cell Klein, C. F. McKee, “An unsplit, cell-cen- J. B. Drake, P. W. Jones, G. R. Carr Jr., Simulations of Plasma Microturbulence on tered Godunov method for ideal MHD,” J. “Overview of the Software Design of the Advanced Computing Platforms,” keynote Comput. Phys. 203, p. 422-448, 2005. Community Climate System Model,” talk, SciDAC Conference, San Francisco C.T.H. Davies, G. P. Lepage, F. International Journal of High 2005. Niedermayer and D. Toussaint, (The Performance Computational Application, J. W. Evans, “Kinetic Monte Carlo HPQCD, MILC and UKQCD 19, 177, 2005. Simulation of Non-Equilibrium Lattice-Gas Collaborations), “The Quenched Continuum T. Draper, et al., “Improved Measure of Models: Basic and Refined Algorithms Limit,” Nuclear Physics B (Proc. Suppl.) Local Chirality,” Nuclear Physics B (Proc. applied to Surface Adsorption Processes,” in 140, 261, 2005. Suppl.) 140, 623, 2005. Handbook of Materials Modeling, Part A, M. S. Day, J. B. Bell, J. F. Grcar, and V. E. Ch. 5.12, ed. S. Yip, Springer, Berlin, T. Draper, et al., “Locality and Scaling of Beckner, “Numerical Control of 3D 2005. Quenched Overlap Fermions,” Draper, et al., Turbulent Premixed Flame Simulations,” Proc. of XXIII International Symposium M. R. Fahey, S. Alam, T. H. Dunigan, Jr., 20th International Colloquium on the on Lattice Field Theory (Lattice 2005), J. S. Vetter and P. H. Worley, “Early Dynamics of Explosions and Reactive 120, 2005. Evaluation of the Cray XD1,” Proc. of the Systems, July-Aug, 2005. 47th Cray User Group Conference, H. Duan, G. M. Fuller, Y.-Z. Qian, H. de Sterck, T. Manteuffel, S. Albuquerque, NM, May 2005. “Collective Neutrino Flavor Transformation McCormick and L. Olson, “Numerical con- In Supernovae,” ArXiv Astrophysics e- R. Falgout, J. Brannick, M. Brezina, T. servation properties of H(div)-conforming prints, Nov. 2005. Manteuffel, and S. McCormick, “New least-squares finite element methods for the multigrid solver advances in TOPS,” Burgers equation,” SIAM Journal of J. Dudek, R. G. Edwards and D. G. Proceedings of SciDAC 2005, Journal of Scientific Computing, 26, p. 1573-1597, Richards, “Radiative transitions in charmo- Physics: Conference Series, Institute of 2005. nium,” Proc. of XXIII International Physics, San Francisco, Calif., June 2005. Symposium on Lattice Field Theory H. de Sterck, R. Markel and R. Knight, (Lattice 2005), 210, 2005. R. D. Falgout, J. E. Jones, and U. M. Yang, “TaskSpaces: a software framework for paral- “The Design and Implementation of hypre, a lel bioinformatics on computational grids,” in T. J. Dudley, R. M. Olson, M. W. Library of Parallel High Performance Parallel Computing in Bioinformatics and Schmidt, and M .S. Gordon, “Parallel Preconditioners,” chapter in Numerical Computational Biology, A. Zomaya, edi- Coupled Perturbed CASSCF Equations and Solution of Partial Differential Equations tor, J. Wiley and Sons, 2005. Analytic CASSCF Second Derivatives,” on Parallel Computers, A. M. Bruaset, P. Journal of Computational Chemistry, 27, J. Demmel, J. Dongarra, V. Eijkhout, E. Bjørstad, and A. Tveito, eds., Springer- 352, 2006. Fuentes, A. Petitet, R. Vuduc, R. C. Verlag, 2006. Whaley, and K. Yelick, “Self-Adapting T. H. Dunigan, Jr., J. S. Vetter and P. H. R. D. Falgout, J. E. Jones, and U. M. Yang, Linear Algebra Algorithms and Software,” Worley, “Performance Evaluation of the SGI “Conceptual Interfaces in hypre,” Future Proc. of the IEEE, Special Issue on Altix 3700,” Proc. of the 2005 Generation Computer Systems, Special Program Generation, Optimization, and International Conference on Parallel Issue on PDE software, 22, p. 239-251, Platform Adaptation, 93(2), Feb. 2005. Processing, Oslo, Norway, June 2005. 2005. N. Desai, R. Bradshaw, S. Matott, S. T. H. Dunigan, Jr., J. S. Vetter, J. B. White R. D. Falgout, J. E. Jones and U. M. Yang, Bittner, S. Coghlan, R. Evard, C. III, and P. H. Worley, “Performance “Pursuing Scalability for hypre's Conceptual Luenighoener, T. Leggett, J. P. Navarro, Evaluation of the Cray X1 Distributed Interfaces,” ACM Transactions of G. Rackow, C. Stacey, and T. Stacey, “A Shared-Memory Architecture,” IEEE Micro, Mathematical Softw., 31, p. 326-350, case study in configuration management tool 25(1), p. 30, Jan./Feb. 2005. 2005. deployment,” Proc. of the Nineteenth

117 LIST OF PUBLICATIONS

R. D. Falgout, P. S. Vassilevski, and L. T. G. M. Fuller, Y.-Z. Qian, “Simultaneous by dephasing and beam loading in channeled Zikatanov, “On Two-Grid Convergence Flavor Transformation of Neutrinos and and unchanneled laser plasma accelerators,” Estimates,” Numer. Linear Algebra Appl., Antineutrinos with Dominant Potentials from Physics of Plasmas, 12, 5, p. 56709, May 12, p. 471-494, 2005. Neutrino-Neutrino Forward Scattering,” 2005. ArXiv Astrophysics e-prints, May 2005. D.G. Fedorov, K. Kitaura, H. Li, J. H. S. J. Ghan, and T. Shippert, “Load balanc- Jernsen, and M. S. Gordon, “The polariz- E. Gamiz, et al. (The HPQCD ing and scalability of a subgrid orography able continuum model (PCM) interfaced Collaboration), “Dynamical Determination scheme in a global climate model,” with the fragment molecular orbital method of BK from Improved Staggered Quarks,” International Journal of High Performance (FMO),” Journal of Computational Proc. of XXIII International Symposium Computer Applications., 19, 237, 2005. Chemistry, in press. on Lattice Field Theory (Lattice 2005), J. Glimm, X. L. Li, Z. L. Xu, “Front track- 47 2005. J. Feng, W. Wan, J. Qiang, A. Bartelt, A. ing algorithm using adaptively refined mesh- Comin, A. Scholl, J. Byrd, R. Falcone, G. E. Gamiz, et al. (The HPQCD es,” Proc. of the 2003 Chicago Workshop Huang, A. MacPhee, J. Nasiatka, K. Collaboration), “BK from Improved on Adaptive Mesh Refinement Methods, Opachich, D. Weinstein, T. Young and H. Staggered Quarks,” Nuclear Physics (Proc. Lecture Notes in Computational Science A. Padmore, “An Ultra-Fast X-Ray Streak Suppl.) 140, 353, 2005. and Engineering, 41, 83-90, 2005. Camera for the Study of Magnetication W. Gao, X. S. Li, C. Yang, Z. Bai, M. S. Gordon and M. W. Schmidt, Dynamics,” Proceedings of SPIE, 2005. “Performance Evaluation of a Multilevel “Advances in Electronic Structure Theory: G. Fleming, et al. (The LHPC Sub-structuring Method for Sparse GAMESS a Decade Later,” Theory and Collaboration), “Pion form factor using Eigenvalue Problems,” Proc. of the 16th Applications of Computational domain wall valence and asqtad sea quarks,” International Conference on Domain Chemistry, Ch. 41, C. E. Dykstra, G. Nuclear Physics (Proc. Suppl.) 140, 302 Decomposition Methods, 2005. Frenking, K. S. Kim, G. E. Scuseria, Eds., 2005. Elsevier, 2005. X. Gao, B. Simon, and A. Snavely, G. Folatelli, C. Contreras, M. M. Phillips, “ALITER: An Asynchronous Lightweight S. Gottlieb, et al., “Onium masses with three S. E. Woosley, S. Blinnikov, N. Morrell, Instrumentation Tool for Event Recording,” flavors of dynamical quarks,” Proc. of XXIII N. B. Suntzeff, B. L. Lee, M. Hamuy, S. Workshop on Binary Instrumentation and International Symposium on Lattice Field Gonzalez, W. Krzeminski, M. Roth, W. Applications, St. Louis, Mo. Sept. 2005. Theory (Lattice 2005), 203, 2005. Li, A. V. Filippenko, W. L. Freedman, B. X. Gao, M. Laurenzano, B. Simon, and A. D. A. Goussis and M. Valorani, “An effi- F. Madore, S. E. Persson, D. Murphy, S. Snavely, “Reducing Overheads for Acquiring cient iterative algorithm for the approxima- Boissier, G. Galaz, L. Gonzalez, P. J. Dynamic Traces,” International Symposium tion of the fast and slow dynamics of stiff sys- McCarthy, A. McWilliam and W. Pych, on Workload Characterization (ISWC05), tems,” Journal of Computational Physics, “SN 2005bf: A Transition Event Between Austin, Texas, Sept. 2005. 214, 316, 2006. Type Ib/c Supernovae and Gamma Ray Bursts,” ArXiv Astrophysics e-prints, Sept. R. Garnett, J. Billen, T. Wangler, P. D. A. Goussis, and M. Valorani, F. Creta 2005. Ostroumov, J. Qiang, R. Ryne, R. York, K. and H. N. Najm, “Reactive and Crandall, “Advanced Beam-Dynamics Reactive/Diffusive Time Scales in Stiff I. Foster, “Service-Oriented Science,” Science Simulation Tools for RIA,” Proc. PAC2005, Reaction-Diffusion Systems,” Prog. in Magazine, 308, 5723, 814-817, May 2005. Knoxville, Tenn., May 2005. Computational Fluid Dynamics, 5(6):316- N. Fout, H. Akiba, K.-L. Ma, A. Lefohn, 326, Sept. 2005. C. Gatti-Bono, P. Colella, “An anelastic all- and J. Kniss, “High Quality Rendering of speed projection method for gravitationally A. Gray, et al. (The HPQCD Compressed Volume Data Formats,” Proc. of stratified flows,” Journal of Computational Collaboration), “The B meson decay con- the Eurographics/IEEE-VGTC Physics, in press. stant from unquenched lattice QCD,” Symposium on Visualization (EuroVis Physical Review Letters 95, 212001, 2005), Leeds, UK, June 2005. L. Ge, L.-Q. Lee, Z. Li, C.-K. Ng, G. 2005. Schussman, L. Xiao and K. Ko, “Advanced N. Fout, K.-L. Ma, and J. Ahrens, “Time- Eigensolver For Electromagnetic Modeling Of A. Gray, et al. (The HPQCD Varying Multivariate Volume Data Accelerator Cavities,” The 15th Conference Collaboration), “The Upsilon spectrum and Reduction,” Proc. of the ACM Symposium on the Computation of Electromagnetic m(b) from full lattice QCD,” Physical on Applied Computing 2005, p. 1224- Fields, Shenyang, China, June 2005. Review D, 72, 094507, 2005. 1230, Santa Fe, NM, March 2005. L. Ge, L.-Q. Lee, Z. Li, C.-K. Ng, K. Ko, A. Gray, et al. (The HPQCD C. L. Fryer and A. Heger, “Binary Merger E. Seol, A. C. Bauer and M. Shephard, Collaboration), “B Leptonic Decays and Progenitors for Gamma-Ray Bursts and “Adaptive Mesh Refinement In Accelerator B_B Mixing with 2+1 Flavors of Hypernovae,” ApJ, 623:302-313, April Cavity Design,” SIAM Conference on Dynamical Quarks,” Nuclear Physics 2005. Computational Science & Engineering, (Proc. Suppl.) 140, 446, 2005. C. L. Fryer, G. Rockefeller, A. Orlando, Fla., Feb. 2005. J. F. Grcar, M. S. Day, and J. B. Bell, “A Hungerford, and F. Melia, “The Sgr B2 X- C. G. R. Geddes, C. Toth, J. van Tilborg, Taxonomy of Integral Reaction Path ray Echo of the Galactic Center Supernova E. Esarey, C. B. Schroeder, D. Bruhwiler, Analysis,” Combust. Theory Modelling, to Explosion that Produced Sgr A East,” ArXiv C. Nieter, J. Cary, and W. P. Leemans, appear. Astrophysics e-prints, June 2005. “Production of high quality electron bunches

118 LIST OF PUBLICATIONS

C. M. Greenfield, “Advanced Tokamak Woosley, “Neutrino nucleosynthesis,” Joshi, T. Katsouleas, P. Krejcik, W. Lu, K. Research at the DIII-D National Fusion Physics Letters B, 606, 258-264, Jan. 2005. A. Marsh, W. B. Mori, P. Muggli, C. L. Facility in Support of ITER,” Proceedings O’Connell, E. Oz, R. H. Siemann and D. A. Heger, S. E. Woosley and H. C. Spruit, of SciDAC 2005, Journal of Physics: Walz, “Multi-GeV Energy Gain in a “Presupernova Evolution of Differentially Conference Series, Institute of Physics, Plasma-Wakefield Accelerator,” Physical Rotating Massive Stars Including Magnetic San Francisco, Calif., June 2005. Review Letters 95, 054802, 2005. Fields,” ApJ, 626:350-363, June 2005. M. Greenwald, D. P. Schissel, J. R. J. Hollingsworth, A. Snavely, S. Sbaraglia, J. J. Heys, C. G. DeGroff, T. Manteuffel, Burruss, T. Fredian, J. Lister and J. and K. Ekanadham, “EMPS: An and S. McCormick, “First-order system least Stillerman, “Visions for Data Management Environment for Memory Performance squares (FOSLS) for modeling blood flow,” and Remote Collaboration on ITER,” Proc. Studies,” IPDPS 2005, Next Generation Med. Eng. Phys., to appear. of ICALEPCS05, 2005. Software Workshop, to appear. A. C. Hindmarsh, P. N. Brown, K. E. E. Gulez, et al. (The HPQCD D. J. Holmgren, “PC clusters for lattice Grant, S. L. Lee, R. Serban, D. E. Collaboration), “B Semileptonic Decays QCD,” Nuclear Physics (Proc. Suppl.) Shumaker, and C. S. Woodward, “SUN- with 2+1 Dynamical Quark Flavors,” Proc. 140, 183, 2005. DIALS: Suite of Nonlinear and of XXIII International Symposium on Differential/Algebraic Equation Solvers,” D. J. Holmgren, “U.S. Lattice Clusters and Lattice Field Theory (Lattice 2005), 220, ACM Transactions on Mathematical the USQCD Project,” Proc. of XXIII 2005. Software, 31(3), p. 363-396, 2005. International Symposium on Lattice Field O. D. Gurcan, P. H. Diamond, T. S. Theory (Lattice 2005), 105, 2005. W. R. Hix, O. E. B. Messer, A. Hahm and Z. Lin, “Dynamics of Turbulence Mezzacappa, M. Liebendoerfer, D. J. C. Homescu, L. Petzold, and R. Serban, Spreading in Magnetically Confined Dean, K. Langanke, J. Sampaio, G. “Error estimation for reduced order models of Plasmas,” Physics of Plasmas, 12, 032303, Martinez-Pinedo, “Terascale input physics: dynamical systems,” SIAM Journal on 2005. the role of nuclear electron capture in core col- Numerical Analysis, 43, 1693, 2005. A. T. Hagler, et al. (The LHPC lapse supernovae,” Journal of Physics E. B. Hooper, R. A. Kopriva, B. I. Cohen, Collaboration), “Helicity dependent and Conference Series, 16, 400-404, 2005. D. N. Hill, H. S. McLean, R. D. Wood, S. independent generalized parton distributions W. R. Hix, B. S. Meyer, “Thermonuclear Woodruff and C. R. Sovinec, “A magnetic of the nucleon in lattice QCD,” European Kinetics in Astrophysics,” Nuclear Physics meconnection event in a driven spheromak,” Physics Journal, A24S1, 29, 2005. A, in press, Sept. 2005. Physics of Plasmas 12, 092503, 2005. T. S. Hahm, P. H. Diamond, Z. Lin, G. F. M. Hoffman, W. W. Hargrove, D. J. I. Horvath, et al., “Inherently Global Nature Rewoldt, O. Gurcan, and S. Ethier, “On Erickson III and W. Oglesby, “Using clus- of Topological Charge Fluctuations in QCD,” the Dynamics of Edge-Core Coupling,” tered climate regimes to analyze and compare Physics Letters B, 612, 21, 2005. Physics of Plasmas, 12, 090903, 2005. predictions from fully coupled general circula- I. Horvath, et al., “The Negativity of the K. Hashimoto, T. Izubuchi and J. Noaki, tion models,” Earth Interactions, 9, 1, 2005. Overlap-Based Topological Charge Density “The static quark potential in 2+1 flavour F. M. Hoffman, M. Vertenstein, H. Correlator in Pure-Glue QCD and the Non- domain wall QCD from QCDOC,” Proc. of Kitabata, and J. B. White III. “Vectorizing Integrable Nature of its Contact Part,” XXIII International Symposium on the Community Land Model (CLM),” Physics Letters B, 617, 49, 2005. Lattice Field Theory (Lattice 2005), 093, International Journal of High 2005. M. Huang, and B. Bode, “A Performance Performance Computing Applications, Comparison of Tree and Ring Topologies in J. C. Hayes, S. W. Bruenn, “Toward 2-D 19(3), 247, Aug. 2005. Distributed Systems,” Parallel and and 3-D simulations of core-collapse super- F. M. Hoffman, W. W. Hargrove, D. J. Distributed Scientific and Engineering novae with magnetic fields,” Journal of Erickson, and R. J. Oglesby, “Using Computing PDSEC-05, Denver, Colo., Physics Conference Series, 16, 395-399, Clustered Climate Regimes to Analyze and April 2005. 2005. Compare Predictions from Fully Coupled A. L. Hungerford, C. L. Fryer, and G. M. Y. He and C. Ding, “Coupling Multi- General Circulation Models,” Earth Rockefeller, “Gamma-Rays from Single Component Models with MPH on Interactions, 9(10), 1-27, Aug. 2005. Lobe Supernova Explosions,” ArXiv Distributed Memory Computer I. Hofmann, G. Franchetti, A. U. Luccio, Astrophysics e-prints, July 2005. Architectures,” International Journal of M. Giovannozzi, E. Métral, J. F. High Performance Computing F.-N. Hwang, and X.-C. Cai, “Parallel fully Amundson, P. Spentzouris, S. Machida, J. Applications, 19, 3, p. 329, 2005. coupled Schwarz preconditioners for saddle Qiang, R. Ryne, S. M. Cousineau, J. A. point problems,” Electronic Transactions M. Head-Gordon, G. J. O. Beran, A. Sodt Holmes, F. Jones, “Benchmarking of on Numerical Analysis, 2005. and Y. Jung, “Fast electronic structure meth- Simulation Codes Based on the Montague ods for strongly correlated molecular systems,” Resonance in the CERN-PS,” Proc. F.-N. Hwang, and X.-C. Cai, “A parallel Journal of Physics: Conf. Ser. 16, 233-242, PAC2005, Knoxville, Tenn., May 2005. nonlinear additive Schwarz preconditioned 2005. inexact Newton algorithm for incompressible M. J. Hogan, C. D. Barnes, C. E. Clayton, Navier-Stokes equations,” Journal of A. Heger, E. Kolbe, W. C. Haxton, K. F. J. Decker, S. Deng, P. Emma, C. Computational Physics, 204, p. 666-691, Langanke, G. Martinez-Pinedo and S. E. Huang, R. H. Iverson, D. K. Johnson, C. 2005.

119 LIST OF PUBLICATIONS

F.-N. Hwang, and X.-C. Cai, “A combined D. J. Kerbyson and P. W. Jones, “A simulations of DIII-D plasmas with the linear and nonlinear preconditioning tech- Performance Model of the Parallel Ocean NIMROD code,” Computer Physics nique for incompressible Navier-Stokes equa- Program,” International Journal of High Communications 164, 1-3, 34, 2005. tions,” J. Dongarra, K. Madsen, and J. Performance Computing Applications, S. E. Kruger, D. D. Schnack and C. R. Wasniewski, eds., Lecture Notes in 19, 261, 2005. Sovinec, “Dynamics of the major disruption Computer Science, 3732, p. 313-322, C. C. Kim, C. R. Sovinec, S. E. Parker and of a DIII-D plasma,” Physics of Plasmas Springer-Verlag, 2005. the NIMROD Team, “Hybrid kinetic- 12, 056113, 2005. E. Ipek, B. R. de Supinski, M. Schulz and MHD simulations in general geometry,” M. Kuhlen, S. E. Woosley and G. A. S. A. McKee, “An Approach to Performance Computer Physics Communications 164, Glatzmaier, “Carbon Ignition in Type Ia Prediction for Parallel Applications,” Euro- 1-3, 448, 2005. Supernovae: II. A Three-Dimensional Par 2005, Lisbon, Portugal, Sept. 2005. S. Klasky, “Data Management on the Fusion Numerical Model,” ApJ 640, 407, 2006. R. Jacob, J. Larson, and E. Ong, “MxN Computational Pipeline,” invited talk, J.-F. Lamarque, J. T. Kiehl, P. G. Hess, W. Communication and Parallel Interpolation in SciDAC Conference, San Francisco, Calif., D. Collins, L. K. Emmons, P. Ginoux, C. CCSM3 Using the Model Coupling Toolkit,” June 2005. Luo and X. X. Tie, “Response of a coupled International Journal of High A. Klawonn, O. Rheinbach and O. chemistry-climate model to changes in aerosol Performance Computational Applications, Widlund, “Some Computational Results for emissions: Global impact on the hydrological 19(3), 293, 2005. Dual-Primal FETI Methods for Elliptic cycle and the tropospheric burdens of OH, E. F. Jaeger, “Self-consistent full-wave and Problems in 3D,” Proc. of the 15th ozone and NOx,” Geophysical Research Fokker-Planck calculations for ion cyclotron International Symposium on Domain Letter, 32, 16, L16809, 2005. heating in non-Maxwellian plasmas,” Bull. Decomposition Methods, Berlin, C.-L. Lappen and D. A. Randall, “Using American Physics Society 50, 257, 2005. Germany, July 2003. idealized coherent structures to parameterize E. F. Jaeger, L. A. Berry, R. W. Harvey, et A. Klawonn and O. Widlund, “Selecting momentum fluxes in a PBL mass-flux al., “Self-Consistent Full-Wave / Fokker- Constraints in Dual-Primal FETI Methods model,” Journal of Atmospheric Sciences, Planck Calculations for Ion Cyclotron for Elasticity in Three Dimensions,” Proc. of 62, 2829, 2005. Heating in non-Maxwellian Plasmas,” Radio the 15th International Symposium on J. Larson, R. Jacob and E. Ong, “The Frequency Power in Plasmas, Proc. 16th Domain Decomposition Methods, Berlin, Model Coupling Toolkit: A New Fortran90 Topical Conf., Park City, Utah, 2005. Germany, July 2003. Toolkit for Building Multi-Physics Parallel S. C. Jardin, J. A. Breslau, “Implicit solution K. Ko, N. Folwell, L. Ge, A. Guetz, L.-Q. Coupled Models,” International Journal of of the four-field extended-magnetohydrody- Lee, Z. Li, C.-K. Ng, E. Prudencio, G. High Performance Computing namic equations using high-order high-conti- Schussman, R. Uplenchwar and L. Xiao, Applications, 19(3), 277-292, 2005. nuity finite elements,” Physics of Plasmas “Advances in Electromagnetic Modeling M. Laurenzano, B. Simon, A. Snavely and 12, 10, 2005. through High Performance Computing,” M. Gunn, “Low Cost Trace-driven Memory Proc. of 12th International Workshop on J.E. Jones and B. Lee, “A Multigrid Method Simulation Using SimPoint,” Workshop on RF Superconductivity (SRF 2005), for Variable Coefficient Maxwell's Binary Instrumentation and Applications, Cornell, New York, July 2005. Equations,” SIAM Journal on Scientific St. Louis, Mo., Sept. 2005. Computing, 27, 168, 2006. K. Ko, N. Folwell, L. Ge, A. Guetz, V. I. Lee, P. Raghavan, and E. G. Ng, Ivanov, A. Kabel, M. Kowalski, L. Lee, Z. P. W. Jones, P. H. Worley, Y. Yoshida, J. B. “Effective Preconditioning Through Ordering Li, C. Ng, E. Prudencio, G. Schussman, R. White III and J. Levesque, “Practical per- Interleaved with Incomplete Factorization,” Uplenchwar, L. Xiao, E. Ng, W. Gao, X. formance portability in the Parallel Ocean SIAM J. Matrix Anal. Appls., 27, 106, Li, C. Yang, P. Husbands, A. Pinar, D. Program (POP),” Concurrency 2006. Bailey, D. Gunter, L. Diachin, D. Brown, Computation-Practice Experiment & D. Quinlan, R. Vuduc, P. Knupp, K. J. C. Lee, H. N. Najm, S. Lefantzi, J. Ray, Experience, 17, 1317, 2005. Devine, J. Kraftcheck, O. Ghattas, V. M. Frenklach, M. Valorani and D. A. C. Jung, “Thermodynamics using p4- Akcelik, D. Keyes, M. Shephard, E. Seol, Goussis, “On chain branching and its role in improved staggered fermion action on A. Brewer, G. Golub, K. Ma, H. Yu, Z. homogeneous ignition and premixed flame QCDOC,” Proc. of XXIII International Bai, B. Liao, T. Tautges and H. Kim, propagation,” Computational Fluid and Symposium on Lattice Field Theory “Impact of SciDAC on accelerator projects Solid Mechanics 2005, K. J. Bathe, ed., p. (Lattice 2005), 150, 2005. across the Office of Science through electro- 717, Elsevier, 2005. magnetic modeling,” Proc. of SciDAC 2005 A. Juodagalvis, D. J. Dean, “Gamow-Teller L.-Q. Lee, Z. Bai, W. Gao, L. Ge, Z. Li, meeting, San Francisco, Calif., June 2005. GT+ distributions in nuclei with mass A = C.-K. Ng, C. Yang, E. Ng and K. Ko, 90-97,” Physical Review C, 72, 024306, S. Kronfeld, et al. (The Fermilab Lattice “Nonlinear Eigenvalue Problem in 2005. and MILC Collaborations), “Predictions Accelerator Cavity Design,” SIAM Annual from Lattice QCD,” Proc. of XXIII Meeting, New Orleans, La., July 2005. A. Juodagalvis, K. Langanke, G. Martinez- International Symposium on Lattice Field Pinedo, W. R. Hix, D. J. Dean, J. M. L.-Q. Lee, Z. Bai, W. Gao, L. Ge, P. Theory (Lattice 2005), 206, 2005. Sampaio, “Neutral-current neutrino-nucleus Husbands, M. Kowalski, X. Li, Z. Li, C.- cross sections for A~50-65 nuclei,” Nuclear S. E. Kruger, C. R. Sovinec, D. D. K. Ng, C. Yang, E. Ng and K. Ko, Physics A, 747, 87-108, 2005. Schnack and E. D. Held, “Free-boundary “Modeling Rf Cavities With External

120 LIST OF PUBLICATIONS

Coupling,” SIAM Conference on Z. Li, A. Kabel, L. Ge, R. Lee, C.-K. Ng, Approach to System Software, Tools and Computational Science & Engineering, G. Schussman, L. Xiao and K. Ko, Libraries for Clusters,” International Orlando, Fla., Feb. 2005. “Electromagnetic Modeling For The ILC,” Journal of High Performance Computing International Linear Collider Physics and Applications (to appear 2006). L.-Q. Lee, L. Ge, Z. Li, C. Ng, G. Detector Workshop and 2nd ILC acceler- Schussman, K. Ko, E. Ng, W. Gao, P. K. L. Ma, E. B. Lum, H. Yu, H. Akiba, M. ator Workshop, Snowmass, Colo., Aug. Husbands, X. Li, C. Yang, V. Akcelik, O. Y. Huang, M.-Y. Huang, Y. Wang and G. 2005. Ghattas, D. Keyes, T. Tautges, H. Kim, J. Schussman, “Scientific Discovery through Craftcheck, P. Knupp, L. Freitag Diachin, M. Liebendoerfer, M. Rampp, H.-T. Janka, Advanced Visualization,” Journal of K.-L. Ma, Z. Bai and G. Golub, A. Mezzacappa, “Supernova Simulations Physics, 16, DOE SciDAC 2005 “Achievements in ISICs/SAPP collaborations with Boltzmann Neutrino Transport: A Conference, San Francisco, Calif., June for electromagnetic modeling of accelerators,” Comparison of Methods,” Astrophysical 2005. Proc. of SciDAC 2005 meeting, San Journal Letters, 620, 840-860, 2005. Q. Mason et al. (The HPQCD Francisco, Calif., June 2005. H.-W. Lin and N. H. Christ, “Non-pertur- Collaboration), “Accurate determinations of W. P. Leemans, E. Esarey, J. van Tilborg, batively determined relativistic heavy quark alpha(s) from realistic lattice QCD,” P. A. Michel, C. B. Schroeder, C. Toth, C. action,” Proc. of XXIII International Physical Review Letters 95, 052002, G. R. Geddes and B. A. Shadwick, Symposium on Lattice Field Theory 2005. “Radiation from laser accelerated electron (Lattice 2005), 225, 2005. T. Manteuffel, S. McCormick, J. Ruge and bunches: coherent terahertz and femtosecond M.-F. Lin, “Probing the chiral limit ofMpi J. G. Schmidt, “First-order system LL* x-rays,” IEEE Trans. Plasma Sci., 33, 1, and fpi _in 2+1 flavor QCD with domain (FOSLL*) for general scalar elliptic prob- 0093-3813, Feb. 2005. wall fermions from QCDOC,” Proc. of lems in the plane,” SIAM Journal on S. Lefantzi, J. Ray, C. A. Kennedy and H. XXIII International Symposium on Numerical Analysis, 43, 209, 2005. N. Najm, “A Component-based Toolkit for Lattice Field Theory (Lattice 2005), 94, T. Manteuffel, S. McCormick, O. Röhrle Reacting Flows with Higher Order Spatial 2005. and J. Ruge, “Projection multilevel methods Discretizations on Structured Adaptively Y. Lin, S. Wukitch, A. Parisot, J. C. for quasilinear elliptic partial differential Refined Meshes,” Prog. in Computational Wright, N. Basse, P. Bonoli, E. Edlund, L. equations: numerical results,” SIAM Journal Fluid Dynamics, 5(6), 298-315, Sept. 2005. Lin, M. Porkolab, G. Schilling and P. on Numerical Analysis, 44, 120, 2006. J. L. V. Lewandowski, “Low-noise collision Phillips, “Observation and modeling of ion T. Manteuffel, S. McCormick and O. operators for particle-in-cell simulations,” cyclotron range of frequencies waves in the Röhrle, “Projection multilevel methods for Physics of Plasmas, 12, 052322, 2005. mode conversion region of Alcator C-Mod,” quasilinear elliptic partial differential equa- Plasma Phys. and Control. Fusion 47, J. L. V. Lewandowski, “Global particle-in- tions: theoretical results,” SIAM Journal on 1207, 2005. cell simulations of microturbulence with kinet- Numerical Analysis, 44, 139, 2006. ic electrons,” invited talk, APS/DPP meet- Y. Lin, X. Wang, Z. Lin, and L. Chen, “A J. Marathe, F. Mueller and B. R. de ing, Denver, Colo., 2005. Gyrokinetic Electron and Fully Kinetic Ion Supinski, “A Hybrid Hardware/Software Plasma Simulation Model,” Plasma Phys. H. Li, and M. S. Gordon, “Gradients of the Approach to Efficiently Determine Cache Contr. Fusion 47, 657, 2005. Exchange-Repulsion Energy in the Effective Coherence Bottlenecks,” Proc. of the Fragment Potential Method,” Theoretical Z. Lin, L. Chen, and F. Zonca, “Role of International Conference on Chemistry Accounts, in press. Nonlinear Toroidal Coupling in Electron Supercomputing, Heidelberg, Germany, Temperature Gradient Turbulence,” Physics June 2005. K. Li, et al., “Dynamic Scalable of Plasmas, 12, 056125, 2005. Visualization for Collaborative Scientific D. F. Martin, P. Colella, M. Anghel, F. Applications,” Proceedings of the Next D.-J. Liu and J. W. Evans, “From Atomic Alexander, “Adaptive mesh refinement for Generation Software Workshop, Denver Scale Ordering to Mesoscale Spatial Patterns multiscale nonequilibrium physics,” Colo., April 2005. in Surface Reactions: A Heterogeneous Computers in Science and Engineering, 7, Coupled Lattice-Gas (HCLG) Simulation 3, p. 24-31, 2005. X. Li, M. S. Shepard, M. W., Beall, “3-D Approach,” SIAM Multiscale Modeling Anisotropic Mesh Adaptation by Mesh N. Mathur, et al., “Roper Resonance and and Simulation, 4, 424, 2005. Modifications,” Computer Methods in S11(1535) from Lattice QCD,” Physics Applied Mechanics and Engineering, K. F. Liu and N. Mathur, “A Review of Letters B, 605, 137, 2005. 194(48-49), p. 4915-4950, 2005. Pentaquark Calculations on the Lattice,” S. McCormick, “Projection multilevel meth- Proc. of 2005 Int. Conf. on QCD and X. S. Li, “An Overview of SuperLU: ods for quasilinear elliptic partial differential Hadron Physics, 2005. Algorithms, Implementation, and User equations: V-cycle theory,” SIAM J. Mult. Interface,” ACM Trans. on Math. X. Luo, L. Yin, M. S. Shephard, R. M. Modeling Sim., 4, 133, 2006. Software, 31, 3, 2005. O’Bara, M. W. Beall, “An automatic adap- P. McCorquodale, P. Colella, G. Balls, S. tive directional variable p-version method for X. S. Li, W. Gao, P. J. R. Husbands, C. B. Baden, “A scalable parallel Poisson solver 3-D curved domains,” Proc. 8th US Nat. Yang and E. G. Ng, “The Roles of Sparse with infinite domain boundary conditions,” Congress on Comp. Mech., 2005. Direct Methods in Large-Scale Simulations,” Proc. of the 7th Workshop on High Proc. of SciDAC 2005 meeting, San E. Lusk, N. Desai, R. Bradshaw, A. Lusk Performance Scientific and Engineering Francisco, Calif., June 2005. and R. Butler, “An Interoperability Computing, Oslo, Norway, June 2005.

121 LIST OF PUBLICATIONS

D. M. Medvedev, E. M. Goldfield and S. D. Carter, “Nonlinear IRCR-Plasma O. O. Oluwole and W. H. Green, K. Gray, “An OpenMP/MPI Approach to Interactions,” Radio Frequency Power in “Rigorous Error Control in Reacting Flow the Parallelization of Iterative Four-Atom Plasmas, Proc. 16th Topical Conf., Park Simulations Using Reduced Chemistry Quantum Mechanics,” Computer Physics City, Utah, 2005. Models,” Computational Fluid and Solid Comm. 166, 94-108, 2005. Mechanics: Proceedings of the Third H. N. Najm, J. C. Lee, M. Valorani, D. A. MIT Conference on Computational Fluid D. M. Medvedev, S. K. Gray, A. F. Goussis and M. Frenklach, “Adaptive and Solid Mechanics, ed. by K. J. Bathe, Wagner, M. Minkoff and R. Shepard, chemical model reduction,” Journal of Elsevier, 2005. “Advanced Software for the Calculation of Physics: Conference Series, 16, 101-106, Thermochemistry, Kinetics, and Dynamics,” June 2005. C. D. Ott, S. Ou, J. E. Tohline and A. Journal of Physics: Conference Series 16, Burrows, “One-armed Spiral Instability in a D. Narayan, “Bcfg2: A Pay As You Go 247-251, 2005. Low-T/|W| Postbounce Supernova Core,” Approach to Configuration Complexity,” ApJ. Letters, 625:L119-L122, June 2005. J. E. Menard, R. E. Bell, E. D. Proc. of the 2005 Australian UNIX Users Fredrickson, W. Park, et al., “Internal kink Group, Sydney, Australia, Oct. 2005. C. Ozen, D. J. Dean, “Shell Model Monte mode dynamics in high-beta NSTX plasmas,” Carlo method in the pn-formalism and appli- D. Narayan, E. Lusk and R. Bradshaw, Nuclear Fusion 45 (7), 539-556, 2005. cations to the Zr and Mo isotopes,” ArXiv “{MPISH2}: Unix integration for {MPI} pro- Nuclear Theory e-prints, Aug. 2005. A. Mezzacappa, “Ascertaining the Road grams,” Proc, of EuroPVM-MPI 2005, Ahead,” Annual Review of Nuclear and Sept. 2005. J. E. Penner, M. Herzog, C. Jablonowski, Particle Science, 55, 467-515, 2005. B. van Leer, R. C. Oehmke, Q. F. Stout H. Nassar, M. Paul, I. Ahmand, J. Ben- and K. G. Powell, “Development of an B. Mohr, A. Khnal, M.-A. Hermanns, F. Dov, J. Caggiano, S. Ghelberg, S. Goriely, atmospheric climate model with self-adapting Wolf, “Performance Analysis of One-sided J. P. Greene, M. Hass, A. Heger, A. Heinz, grid and physics,” Journal of Physics: Communication Mechanisms,” Mini- D. J. Henderson, R. V. F. Janssens, C. L. Conference Series, 16, 353, 2005. Symposium “Tools Support for Parallel Jiang, Y. Kashiv, B. S. Nara Singh, A. Programming,” Proc. of Parallel Compu- Ofan, R. C. Pardo, T. Pennington, K. E. P. Petreczky, K. Petrov, D. Teaney and A. ting (ParCo), Malaga, Spain, Sept. 2005. Rehm, G. Savard, R. Scott and R. Velytsky, “Heavy quark diffusion and lattice Vondrasek, “The 40Ca(_,_)44Ti reaction in correlators,” Proc. of XXIII International S. Moore, F. Wolf, J. Dongarra, S. Shende, the energy regime of supernova nucleosynthe- Symposium on Lattice Field Theory A. Malony, B. Mohr, “A Scalable Approach sis,” Physical Review Letters, 96, 041102, (Lattice 2005), 185, 2005. to MPI Application Performance Analysis,” 2006. Proc. of the 12th European Parallel K. Petrov, A. Jakovac, P. Petreczky and A. Virtual Machine and Message Passing Y. Nishimura, Z. Lin, J. L. V. Velytsky, “Bottomonia correlators and spec- Interface Conference, Sept. 2005. Lewandowski and S. Ethier, “A Finite tral functions at zero and finite temperature,” Element Poisson Solver for Global Proc. of XXIII International Symposium S. Moore, F. Wolf, J. Dongarra, B. Mohr, Gyrokinetic Particle Simulations in a Global on Lattice Field Theory (Lattice 2005), “Improving Time to Solution with Automated Field Aligned Mesh,” Journal of 153, 2005. Performance Analysis,” Second Workshop Computation & Physics, 214, 657, 2006. on Productivity and Performance in J. Petrovic, N. Langer, S.-C. Yoon and A. High-End Computing, 11th International B. Norris, L. McInnes, and I. Veljkovic, Heger, “Which massive stars are gamma-ray Symposium on High Performance “Computational quality of service in parallel burst progenitors?” Astronomy & Computer Architecture (HPCA-2005), CFD,” Proc. of the 17th International Astrophysics, 435:247-259, May 2005. San Francisco, Calif., Feb. 2005. Conference on Parallel Computational L. R. Petzold, Y. Cao, S. Li and R. Serban, Fluid Dynamics, University of Maryland, J. Mueller, O. Sahni, K. E. Jansen, M. S. “Sensitivity Analysis of Differential-Algebraic College Park, Md., May 2005. Shephard, “An anisotropic mesh adaption Equations and Partial Differential procedure for pulsatile blood flow simulation,” M. Nuggehally, M. Picu, C. Shephard, M. Equations,” Proc. of the Chemical Process Proc. 8th US Nat. Congress on Comp. S. Shephard, “Adaptive model selection pro- Control Conference, Lake Louise, Mech., 2005. cedure in multiscale problems,” Proc. 8th US Alberta, Canada, Jan. 2006. Nat. Congress on Comp. Mech., 2005. J. Mueller, O. Sahni, K. E. Jansen, M. S. J. Pjesivac-Grbovic, T. Angskun, G. Shephard, C. A. Taylor, “A tool for the effi- M. Okamoto, et al. (The Fermilab Lattice, Bosilca, G. Fagg, E. Gabriel, J. Dongarra, cient FE-simulation of cardio-vascular flow,” HPQCD and MILC Collaborations), “Performance Analysis of MPI Collective Proc. 2nd NAFEMS CFD Seminar: “Semileptonic D _ _/K and B _ _ /D decays Operations,” 4th International Workshop Simulation of Complex Flows (CFD), in 2+1 flavor lattice QCD,” Nuclear on Performance Modeling, Evaluation, April 2005. Physics B (Proc. Suppl.) 140, 461, 2005. and Optmization of Parallel and Distributed Systems (PMEO-PDS 2005), B. Mustapha, V. N. Aseev, P. Ostroumov, L. Oliker, J. Carter, M. Wehner, A. Denver, Colo., April 2005. J. Qiang, R. Ryne, “RIA Beam Dynamics: Canning, S. Ethier, A. Mirin, G. Bala, D. Comparing TRACK to IMPACT,” Proc. of Parks, P. Worley, S. Kitawaki and Y. A. Pochinsky, “Domain Wall Fermion Particle Accelerator Conference (PAC Tsuda, “Leading Computational Methods on Inverter on Pentium 4,” Nuclear Physics B 05), Knoxville, Tenn., May 2005. Scalar and Vector HEC Platforms,” Proc. of (Proc. Suppl.) 140, 859, 2005. the ACM/IEEE SC05 conference, J. R. Myra, D. A. D’Ippolito, D. A. E. Prudencio, R. Byrd, and X.-C. Cai, Seattle, Wash., Nov. 2005. Russell, L. A. Berry, E. F. Jaeger and M. “Parallel full space SQP Lagrange-Newton-

122 LIST OF PUBLICATIONS

Krylov-Schwarz algorithms for PDE- Future Generation Computer Systems, F. Stanton, P. G. Szalay, P. R. constrained optimization problems,” SIAM 22, 293,2006. Westmoreland, F. Zabel, T. Berces, Journal of Scientific Computing, 27, 130, “IUPAC critical evaluation of thermochemi- J. F. Remacle, S. S. Frazao, M. S. 2006. cal properties of selected radicals: Part I,” Shephard, M. S. Flaherty, “An Adaptive Journal of Physical and Chemical J. Pruet, S. E. Woosley, R. Buras, H.-T. Discretization of Shallow Water Equations Reference Data, 34(2), 573-656, 2005. Janka and R. D. Hoffman, “Nucleosynthesis based on Discontinuous Galerkin Methods,” in the Hot Convective Bubble in Core- International Journal for Numerical B. Ruscic, R. E Pinzon, G. Von Collapse Supernovae,” ApJ, 623:325-336, Methods in Fluids, 62(7), pp. 899-923, Laszewski, D. Kodeboyina, A. Burcat, D. April 2005. 2005. Leahy, D. Montoya, A. F. Wagner, “Active thermochemical tables: Thermochemistry for J. Qiang, “Terascale Beam-Beam Simulations J. F. Remacle, X. Li, M. S. Shephard, J. E. the 21st Century,” Journal of Physics: for Tevatron, RHIC and LHC,” Proc. Flaherty, “Anisotropic Adaptive Simulation Conference Series, 16:561-570, 2005. PAC2005, Knoxville, Tenn., May 2005. of Transient Flows using Discontinuous Galerkin Methods,” International Journal R. Ryne, D. Abell, A. Adelmann, J. J. Qiang, M. A. Furman, R. D. Ryne, W. for Numerical Methods in Engineering, Amundson, C. Bohn, J. Cary, P. Colella, Fischer, K. Ohmi, “Recent advaces of 62, pp. 899-923, 2005. D. Dechow, V. Decyk, A. Dragt, R. strong-strong beam-beam simulation,” Gerber, S. Habib, D. Higdon, T. Nuclear Instruments & Methods in D. B. Renner, et al. (The LHPC Katsouleas, K.-L. Ma, P. McCorquodale, Physics Research A, 558, 351, 2006. Collaboration), “Hadronic physics with D. Mihalcea, C. Mitchell, W. Mori, C. T. domain-wall valence and improved staggered J. Qiang, S. M. Lidia, R. Ryne, C. Mottershead, F. Neri, I. Pogorelov, J. sea quarks,” Nuclear Physics. (Proc. Limborg-Deprey, “A 3D Parallel Beam Qiang, R. Samulyak, D. Serafini, J. Shalf, Suppl.) 140, 255, 2005. Dynamics Code for Modeling High C. Siegerist, P. Spentzouris, P. Stoltz, B. Brightness Beams in Photoinjectors,” Proc. G. Rewoldt, “Particle-In-Cell simulations of Terzic, M. Venturini, P. Walstrom, PAC2005, Knoxville, Tenn., May 2005. electron transport,” invited talk, SciDAC “SciDAC Advances and Applications in Conference, San Francisco, Calif., June Computational Beam Dynamics,” J. Qiang, D. Leitner, D. Todd, “A Parallel 2005. Proceeding of SciDAC 2005, June 26-30, 3D Model for the Multi-Species Low Energy San Francisco, Calif., June 2005. Beam Transport System of the RIA Prototype D. G. Richards, “Lattice studies of baryons,” ECR Ion Source VENUS,” Proc. PAC2005, Journal of Physics Conf. Ser. 9, 238, 2005. M. J. Savage, “Nuclear Physics and QCD,” Knoxville, Tenn., May 2005. Proc. of XXIII International Symposium G. Rockefeller, C. L. Fryer, F. K. Baganoff on Lattice Field Theory (Lattice 2005), P. Raghavan, M. J. Irwin, L. C. McInnes and F. Melia, “The X-ray Ridge Surrounding 020, 2005. and B. Norris, “Adaptive software for scien- Sgr A* at the Galactic Center,” ArXiv tific computing: Co-managing quality-per- Astrophysics e-prints, June 2005. E. Scannapieco, P. Madau, S. Woosley, A. formance-power tradeoffs,” Proc. of the Heger and A. Ferrara, “The Detectability of G. Rockefeller, C. L. Fryer, and F. Melia, IEEE International Parallel & Distributed Pair-Production Supernovae at z < 6,” “Spin-Induced Disk Precession in Sagittarius Processing Symposium 2005, IEEE Astrophysical Journal, 633, 1031, 2005. A*,” ArXiv Astrophysics e-prints, July Computer Society Press, 2005. 2005. D. P. Schissel, “Advances in Remote J. J. Ramos, “Fluid formalism for collisionless Participation for Fusion Experiments,” Proc. F. K. Roepke, W. Hillebrandt, J. C. magnetized plasmas,” Physics of Plasmas of the 21st IEEE/NPSS Symposium on Niemeyer, and S. E. Woosley, “Multi-spot 12, 052102, 2005. Fusion Engineering, 2005. ignition in type Ia supernova models,” ArXiv J. J. Ramos, “General expression of the gyro- Astrophysics e-prints, Oct. 2005. D. P. Schissel, et al., “The Collaborative viscous force,” Physics of Plasmas 12, Tokamak Control Room,” The 5th IAEA T. M. Rogers and G. A. Glatzmaier, 112301, 2005. Technical Meeting on Control, Data “Penetrative Convection within the Anelastic Acquisition, and Remote Participation for N. S. Rao, S. M. Carter, Q. Wu, W. R. Approximation,” ApJ, 620:432-441, Feb. Fusion Research, 2005. Wing, M. Zhu, A. Mezzacappa, M. 2005. Veeraraghavan, J. M. Blondin, “Networking D. P. Schissel, “Advances in the Real-Time D. Rotem, K. Stockinger, and K. Wu, for large-scale science: infrastructure, provi- Interpretation of Fusion Experiments,” “Optimizing candidate check costs for bitmap sioning, transport and application mapping,” Proceedings of SciDAC 2005, Journal of indices,” Proc. of CIKM 2005, ACM Press, Journal of Physics Conference Series, 16, Physics: Conference Series, Institute of 2005. 541-545, 2005. Physics, San Francisco, Calif., June 2005. D. Rotem, K. Stockinger, and K. Wu, J. Ray, C. Kennedy, J. Steensland and H. D. P. Schissel, et al., “Grid Computing and “Optimizing i/o costs of multi-dimensional Najm, “Advanced algorithms for computa- Collaboration Technology in Support of queries using bitmap indices,” in DEXA, tions on block-structured adaptively refined Fusion Energy Sciences,” Physics of Plasmas Lecture Notes in Computer Science meshes,” Journal of Physics: Conference 12, 058104, 2005. 3588, p. 220. Springer-Verlag, 2005. Series, 16:113-118, June 2005. D. P. Schissel, et al., “Advances in Remote B. Ruscic, J. E. Boggs, A. Burcat, A. G. D. A. Reed, C. Lu and C. L. Mendes, Participation for Fusion Experiments,” The Csaszar, J. Demaison, R. Janosche, J. M. “Reliability Challenges in Large Systems,” 23rd Symposium on Fusion Technology, L. Martin, M. L. Morton, M. J. Rossi, J. 2005.

123 LIST OF PUBLICATIONS

W. J. Schroeder, M. S. Shephard, M. S. Shephard, A. C. Bauer, E. S. Seol, J. cient bitmap indices to accelerate scientific “Computational Visualization,” Encyclopedia Wa., “Construction of Adaptive Simulation visualization,” Proc. of 2005 Scientific and of Computational Mechanics, E. Stein, R. Loops,” Proc. 8th US Nat. Congress on Statistical Database Management DeBorst, T.J.R. Hughes, eds., Wiley, 2005. Comp. Mech., 2005. Conference, p. 35, 2005. K. Schuchardt, O. Oluwole, W. Pitz, L. A. M. S. Shephard, J. E. Flaherty, K. E. K. Stockinger, J. Shalf, E. W. Bethel and Rahn, J. William, H. Green, D. Leahy, C. Jansen, X. Li, X. j. Luo, N. Chevaugeon, J. K. Wu, “Query-driven visualization of large Pancerella, M. Sjöberg and J. Dec, “New F. Remacle, M. W. Beall, R. M. O’Bara, data sets,” IEEE Visualization 2005, Approaches for Collaborative Sharing of “Adaptive mesh generation for curved Minneapolis, MN, Oct. 2005. Chemical Model Data and Analysis Tools,” domains,” Applied Numerical K. Stockinger, K. Wu, R. Brun and P. 2005 Joint Meeting of the US Sections of Mathematics, 53(2-3) p. 251-271, 2005. Canal, “Bitmap indices for fast end-user the Combustion Institute, 2005. R. Shepard, M. Minkoff, and Y. Zhou, physics analysis in Root,” Nuclear K. Schuchardt, O. Oluwole, W. Pitz, L. A. “Software for Computing Eigenvalue Bounds Instruments and Methods in Physics Rahn, J. William, H. Green, D. Leahy, C. for Iterative Subspace Methods,” Computer Research Section A, 539, 99, 2006. Pancerella, M. Sjöberg, J. Dec, Physics Comm. 170, 109-114, 2005. K. Stockinger, K. Wu, S. Campbell, S. “Development of the RIOT web service and M. S. Shephard, E. Seol, J. Wan, A. C. Lau, M. Fisk, E. Gavrilov, A. Kent, C. E. information technologies to enable mechanism Bauer, “Component-based adaptive control Davis, R. Olinger, R. Young, J. Prewett, P. reduction for HCCI simulations,” Journal of procedures,” Proc. from the Conference on Weber, T. P. Caudell, E. W. Bethel and S. Physics: Conference Series, 16:107-112, Analysis, Modeling and Computation of Smith, “Network traffic analysis with query 2005. PDE and Multiphase Flow, Kluwer driven visualization,” Proc. of the M. Schulz, D. Ahn, A. Bernat, B. R. de Academic Press, 2005. ACM/IEEE SC05 conference, Seattle, Supinski, S. Ko, G. Lee and B. Rountree, Wash., Nov. 2005. J. Shigemitsu, et al. (The HPQCD “Scalable Dynamic Binary Instrumentation Collaboration), “Semileptonic B Decays P. Strack and A. Burrows, “Generalized for BlueGene/L,” Workshop on Binary with Nf = 2+1 Dynamical Quarks,” Boltzmann formalism for oscillating neutri- Instrumentation and Applications, Sept. Nuclear Physics (Proc. Suppl.) 140, 464, nos,” Physics Rev. D, 71(9), 093004, May 2005. 2005. 2005. P. Schwartz, D. Adalsteinsson, P. Colella, J. N. Simone, et al. (The Fermilab Lattice, L. Sugiyama, W. Park, H. R. Strauss, G. A. P. Arkin, M. Onsum, “Numerical compu- HPQCD and MILC Collaborations), Fu, J. Breslau, J. Chen and S. Klasky, tation of diffusion on a surface,” Proc. Nat. “Leptonic decay constants fDs and fD in “Plasmas beyond MHD: two-fluids and sym- Acad. Sci., 102, p. 11151-11156, 2005. three flavor lattice QCD,” Nuclear Physics metry breaking,” Journal of Physics: P. Schwartz, M. Barad, P. Colella, T. J. B (Proc. Suppl.) 140, 443, 2005. Conference Series 16, 54-58, 2005. Ligocki, “A Cartesian grid embedded bound- A. Socrates, O. Blaes, A. Hungerford, and F. D. Swesty, E. S. Myra, “Multigroup mod- ary method for the heat equation and C. L. Fryer, “The Neutrino Bubble els of the convective epoch in core collapse Poisson's equation in three dimensions,” Instability: A Mechanism for Generating supernovae,” Journal of Physics Conference Journal of Computational Physics 211, p. Pulsar Kicks,” Astrophysical Journal, Series, 16, 380-389, 2005. 531-550, 2006. 632:531-562, Oct. 2005. S. Tamhankar, et al., “Charmonium E. S. Seol, M. S. Shephard, “A Flexible C. R. Sovinec, and C. C. Kim, D. D. Spectrum from Quenched QCD with Distributed Mesh Data Structure to Support Schnack, A. Y. Pankin, S. E. Kruger, E. D. Overlap Fermions,” Nuclear Physics B Parallel Adaptive Analysis,” Proc. 8th US Held, D. P. Brennan, D. C. Barnes, X. S. (Proc. Suppl.), 140, 434, 2005. Nat. Congress on Comp. Mech., 2005. Li, D. K. Kaushik, S. C. Jardin and the T. A. Thompson, E. Quataert, and A. R. Serban and A. C. Hindmarsh, NIMROD Team, “Nonlinear Burrows, “Viscosity and Rotation in Core- “CVODES: the Sensitivity-Enabled ODE Magnetohydrodynamic (MHD) Simulations Collapse Supernovae,” Astrophysical Solver in SUNDIALS,” Proc. of the 5th using High-Order Finite Elements,” SciDAC Journal, 620:861-877, Feb. 2005. International Conference on Multibody 2005 meeting, Journal of Physics: Systems, Nonlinear Dynamics and Conference Series, 16, p. 25-34, San D. Trebotich, G. H. Miller, P. Colella, D. Control, Long Beach, Calif., Sept. 2005. Francisco, Calif., June 2005. T. Graves, D. F. Martin, P. O. Schwartz, “A tightly coupled particle-fluid model for B. A. Shadwick, G. M. Tarkenton, E. C. Stan, “The mean meridional circulation: DNA-laden flows in complex microscale Esarey and C. B. Schroeder, “Fluid and A new potential-vorticity, potential-tempera- geometries,” Proc. of the Third MIT Vlasov models of low temperature, collisionless ture perspective,” Ph.D. thesis, Colorado Conference on Computational Fluid and laser-plasma interactions,” Physics of State University. 2005. Solid Mechanics, June 2005. Plasmas, 12, 5, p. 56710-18, May 2005. A. W. Steiner, M. Prakash, J. M. Lattimer, D. Trebotich, P. Colella, G. H. Miller, “A S. Shende, A. Malony, A. Morris, F. Wolf, P. J. Ellis, “Isospin asymmetry in nuclei and stable and convergent scheme for viscoelastic “Performance Profiling Overhead neutron stars,” Physics Reports, 411, 325- flow in contraction channels,” J. Comput. Compensation for MPI Programs,” Proc. of 375, 2005. Phys. 205, p. 315-342, 2005. the 12th European Parallel Virtual K. Stockinger, J. Shalf, E. W. Bethel and Machine and Message Passing Interface D. Todd, D. Leitner, C. Lyneis, J. Qiang, K. Wu, “Dex: Increasing the capability of sci- Conference, Sept. 2005. D. Grote, “Particle-in-Cell Simulations of the entific data analysis pipelines by using effi-

124 LIST OF PUBLICATIONS

VENUS Ion Beam Transport System,” Proc. Group Conference, Albuquerque, NM, B. S. White, S. A. McKee, B. R. de PAC2005, Knoxville, Tenn., May 2005. May 2005. Supinski, B. Miller, D. Quinlan and M. Schulz, “Improving the Computational D. Toussaint and C. Davies (MILC and R. Vuduc, J. Demmel and K. Yelick, Intensity of Unstructured Mesh Applications,” UKQCD Collaborations), “The Ω_ and the “OSKI: A library of automatically tuned Proceedings of the International strange quark mass,” Nuclear Physics B sparse matrix kernels,” Proc. of SciDAC Conference on Supercomputing, (Proc. Suppl.), 140, 234, 2005. 2005, Journal of Physics: Conference Heidelberg, Germany, June 2005. Series, San Francisco, Calif., June 2005. T. Tu, D. O'Hallaron, and O. Ghattas, B. Willems, M. Henninger, T. Levin, N. “Scalable parallel octree meshing for terascale R. Vuduc, and H.-J. Moon, “Fast sparse Ivanova, V. Kalogera, K. McGhee, F. X. applications,” Proceedings of the matrix-vector multiplication by exploiting Timmes, and C. L. Fryer, “Understanding ACM/IEEE SC05 conference, Seattle, variable block structure,” Proc. of the Compact Object Formation and Natal Kicks. Wash., November 2005. International Conference on High- I. Calculation Methods and the Case of GRO Performance Computing and R. Tweedie, “Light meson masses and decay J1655-40,” Astrophysical Journal, 625, Communications, Sorrento, Italy, Sept. constants in a 2+1 flavour domain wall 324-346, May 2005. 2005. QCD,” Proc. of XXIII International F. Wolf, A. Malony, S. Shende, A. Morris, Symposium on Lattice Field Theory G. Wallace, et al., “Tools and Applications “Trace-Based Parallel Performance Overhead (Lattice 2005), 080, 2005. for Large-Scale Display Walls,” IEEE Compensation,” Proc. of the International Computer Graphics & Applications, F. Y. Tzeng and K. L. Ma, “Intelligent Conference on High Performance Special Issue on Large Displays, 25, 24, Feature Extraction and Tracking for Large- Computing and Communications July/Aug. 2005. Scale 4D Flow Simulations,” Proc. of the (HPCC), Sorrento, Italy, Sept.2005. ACM/IEEE SC05 conference, Seattle, R. Walder, A. Burrows, C. D. Ott, E. S. E. Woosley and T. Janka, “The Physics of Wash., Nov. 2005. Livne, I. Lichtenstadt and M. Jarrah, Core-Collapse Supernovae,” Nature Physics, “Anisotropies in the Neutrino Fluxes and M. Valorani, F. Creta, D. A. Goussis, H. 1, 147, 2005. Heating Profiles in Two-dimensional, Time- N. Najm, and J. C. Lee, “Chemical kinetics dependent, Multigroup Radiation P. H. Worley, S. Alam, T. H. Dunigan, Jr., mechanism simplification via CSP,” Hydrodynamics Simulations of Rotating M. R. Fahey and J. S. Vetter, “Comparative Computational Fluid and Solid Core-Collapse Supernovae,” Astrophysical Analysis of Interprocess Communication on Mechanics 2005, K. J. Bathe, ed., p. 900, Journal, 626:317-332, June 2005. the X1, XD1, and XT3,” Proc. of the 47th Elsevier, 2005. Cray User Group Conference, W. Wan, Y. Chen, S. E. Parker, M. Valorani, D. A. Goussis, F. F. Creta Albuquerque, NM, May 2005. “Gyrokinetic delta f simulation of the colli- and H. N. Najm, “Higher Order Corrections sionless and semicollisional tearing mode P. H. Worley and J. B. Drake, “Performance in the Approximation of Low Dimensional instability,” Physics of Plasmas 12, 012311, Portability in the Physical Parameterizations Manifolds and the Construction of Simplified 2005. of the Community Atmospheric Model,” Problems with the CSP Method,” Journal of International Journal for High Computational Physics, 209(2), 754-786, J. Wan, M. S. Shephard, “Contact Surface Performance Computer Applications, Nov. 2005. Control in Forming Simulations,” Proc. 8th 19(3), p. 187, Aug. 2005. US Nat. Congress on Comp. Mech., J-L. Vay, P. Colella, P. McCorquodale, D. 2005. J. C Wright, L. A. Berry, P. T. Bonoli, D. Serafini, V. Straalen, A. Friedman, D. P. B. Batchelor, E. F. Jaeger, M. D. Carter, E. Grote, J-C. Adam, A. Héron, “Application W. X. Wang, “Neoclassical and turbulent D’Azevedo, C. K. Phillips, H. Okuda, R. of Adaptive Mesh Refinement to Particle-In- transport in shaped toroidal plasmas,” invit- W. Harvey, D. N. Smithe, J. R. Myra, D. Cell Simulations of Plasmas and Beams,” ed talk, APS/DPP meeting, Denver, A. D’Ippolito, M. Brambilla and R. J. Workshop on Multiscale Processes in Colo., 2005. Dumont, “Nonthermal particle and full- Fusion Plasmas, IPAM, Los Angeles, J. Weinberg, M. O. MCracken, A. Snavely, wave diffraction effects on heating and cur- Calif., Jan. 2005. and E. Strohmaier, “Quantifying Locality rent drive in the ICRF and LHRF regimes,” J.-L. Vay, M. Furman, R. Cohen, A. in the Memory Access Patterns of HPC Nuclear Fusion 45, 1411, 2005. Friedman, D. Grote, “Initial self-consistent Applications,” Proc. of the ACM/IEEE J. C. Wright, P. T. Bonoli, M. Brambilla, E. 3-D electron-cloud simulations of LHC beam SC05 conference, Seattle, Wash., Nov. D’Azevedo, L. A. Berry, D. B. Batchelor, with the code WARP+POSINST,” Proc. 2005. E. F. Jaeger, M. D. Carter, C. K. Phillips, PAC2005, Knoxville, Tenn., May 2005. V. Wheatley, D. I. Pullin and R. H. Okuda, R. W. Harvey, J. R. Myra, D. J. Vetter, B. R. de Supinski, J. May, L. Samtaney, “On the regular refraction of a D’Ippolito and D. N. Smithe, “Full-wave Kissel and S. Vaidya, “Evaluating High MHD shock at a density interface,” Journal Electromagnetic Field Simulation of Lower Performance Computers,” in Concurrency: of Fluid Mechanics, 522, 179-214, 2005. Hybrid Waves in Tokamaks,” Radio Practice and Experience, 17, 1239, 2005. Frequency Power in Plasmas, Proc. 16th V. Wheatley, D. I. Pullin, and R. Topical Conf., Park City, Utah, 2005. J. S. Vetter, S. R. Alam, T. H. Dunigan, Jr., Samtaney, and R. Samtaney, “Stability of M. R. Fahey, P. C. Roth and P. H. Worley, an impulsively accelerated density interface in K. Wu, J. Ekow, E. J. Otoo, and A. “Early Evaluation of the Cray XT3 at MHD,” Physical Review Letters 95, 12, Shoshani, “An efficient compression scheme ORNL,” Proc. of the 47th Cray User 2005. for bitmap indices,” ACM Transactions on

125 LIST OF PUBLICATIONS

Database Systems, to appear. 2005 meeting, Journal of Physics: P. A. Young, C. Meakin, D. Arnett, and C. Conference Series, 16, p. 425-434, San L. Fryer, “The Impact of Hydrodynamic K. Wu, J. Gu, J. Lauret, A. M. Poskanzer, Francisco, Calif., June 2005. Mixing on Supernova Progenitors,” A. Shoshani, A. Sim, and W. M. Zhang, Astrophysical Journal Letters, 629:L101- “Grid collector: Facilitating efficient selective C. Yang, W. Gao, Z. Bai, X. Li, L.-Q. Lee, L104, Aug. 2005. access from data grids,” Proc. of P. Husbands, and E. Ng, “An Algebraic International Supercomputer Conference, Sub-Structuring for Large-Scale Eigenvalue Y. Zhang, X. S. Li, and O. Marques, Heidelberg, Germany, June 2005. Calculation,” SIAM Journal on Scientific “Towards an Automatic and Application- Computing, 27, 873, 2005. Based Eigensolver Selection,” LACSI B. Wylie, B. Mohr, F. Wolf, “Holistic hard- Symposium 2005, Santa Fe, NM, 87501, ware counter performance analysis of parallel U. M. Yang, “Parallel Algebraic Multigrid Oct. 2005. programs short version:,” Proceedings of Methods - High Performance Preconditioners,” Parallel Computing 2005 (ParCo 2005), chapter in Numerical Solution of Partial Y. Zhou, R. Shepard and M. Minkoff, Malaga, Spain, Sept. 2005. Differential Equations on Parallel “Computing Eigenvalue Bounds for Iterative Computers, A. M. Bruaset, P. Bjørstad, Subspace Methods,” Computer Physics A. Yamaguchi, “Chiral condensate and spec- and A. Tveito, eds., Springer-Verlag, 2006. Comm. 167, 90-102, 2005. tral flows on 2+1 flavours Domain Wall QCD from QCDOC,” Proc. of XXIII L. Yin, X. Luo, M. S. Shephard, M. Zingale, S. E. Woosley, C. A. International Symposium on Lattice Field “Identifying and meshing thin section of 3-d Rendleman, M. S. Day, and J. B. Bell, Theory (Lattice 2005), 135, 2005. curved domains,” Proc. 14th International “Three-dimensional Numerical Simulations of Meshing Roundtable, p. 33-54, Springer, Rayleigh-Taylor Unstable Flames in Type Ia T. Yamaguchi, “Entraining cloud-topped 2005. Supernovae,” Astrophysical Journal, 632, boundary layers,” M.S. Thesis, Colorado 1021-1034, Oct. 2005. State University, 2005. H. Yu and K.-L. Ma, “A Study of I/O Techniques for Parallel Visualization,” C. Yang, “Solving Large-scale Eigenvalue Journal of Parallel Computing, 31, 2, p. Problems in SciDAC Applications,” SciDAC 167-183, Feb. 2005.

126 This report was researched, written and edited by Jon Bashor and John Hules of the LBNL Computing Sciences Communications Group, and designed by the Creative Services Office at LBNL. JO11877 LBNL-61963

DISCLAIMER This document was prepared as an account of work sponsored by the United States Government. While this document is believed to contain correct information, neither the United States Government nor any agency thereof, nor The Regents of the University of California, nor any of their employees, makes any warranty, express or implied, or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorse- ment, recommendation, or favoring by the United States Government or any agency thereof, or The Regents of the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, or The Regents of the University of California.

Ernest Orlando Lawrence Berkeley National Laboratory is an equal opportunity employer.