Modeling and Simulation at the Exascale for Energy and the Environment

Co-Chairs: Horst Simon Lawrence Berkeley National Laboratory April 17–18, 2007 3 Thomas Zacharia Oak Ridge National Laboratory May 17–18, 2007 Rick Stevens Argonne National Laboratory E May 31–June 1, 2007 On the cover:

Teraflops Research Chip - Intel Corporation

Wind Farm - Randy Hayes, Bureau of Land Management

Blue Marble - Reto Stöckli, NASA Goddard Space Flight Center Image Modeling and Simulation at the Exascale for Energy and the Environment

Report on the Advanced Scientific Computing Research Town Hall Meetings on Simulation and Modeling at the Exascale for Energy, Ecological Sustainability and Global Security (E3)

Co-Chairs: Lawrence Berkeley National Laboratory: Horst Simon Oak Ridge National Laboratory: Thomas Zacharia Argonne National Laboratory: Rick Stevens Office of Advanced Scientific Computing Research Contact: Michael Strayer

Special Assistance Technical: Lawrence Berkeley National Laboratory: Deb Agarwal, David Bailey, John Bell, Wes Bethel, Julian Borrill, Phil Colella, William Collins, Nikos Kyrpides, Victor Markowitz, Juan Meza, Norman Miller, Peter Nugent, Leonid Oliker, Arie Shoshani, Erich Strohmaier, Brian Tierney, Lin-Wang Wang, Michael Wehner

Oak Ridge National Laboratory: Eduardo D’Azevedo, David Bernholdt, John Drake, David Erickson, George Fann, James Hack, Victor Hazlewood, Al Geist, Igor Jouline, Douglas Kothe, Bronson Messer, Anthony Mezzacappa, Jeff Nichols, Stephen Poole, B. (Rad) Radhakrishnan, Nagiza Samatova, Srdjan Simunovic, Scott Studham, Jeffrey Vetter, Gilbert Weigand

Argonne National Laboratory: Raymond Bair, Pete Beckman, Charles Catlett, Robert Edwards, Paul Fisher, Ed Frank, Ian Foster, William Gropp, Ahmed Hassanein, Mark Hereld, Paul Hovland, Robert Jacob, Kenneth Kemner, Veerabhadra Kotamarthi, Don Lamb, Ewing Lusk, Jorge Moré, Lois McInnes, Folker Meyer, Boyanna Norris, David Nowak, Michael Papka, Robert Ross

Administrative: Lawrence Berkeley National Laboratory: Jon Bashor, Yeen Markin Oak Ridge National Laboratory: Linda Malone, Debbie McCoy Argonne National Laboratory: Kathy Dibennardi, Janet Werner, Cheryl Zidel

Publication: Oak Ridge National Laboratory: Creative Media Argonne National Laboratory: Joseph Insley

Editorial: Oak Ridge National Laboratory: Agatha Bardoel, Bonnie Nestor Argonne National Laboratory: Gail Pieper

This report is available on the web at http://www.sc.doe.gov/ascr/ProgramDocuments/ProgDocs.html

Modeling and Simulation at the Exascale for Energy and the Environment Contents

Executive Summary��vii

Introduction �� 1

Section 1: Climate �� 5

Section 2: Energy

2A: Combustion �� 19

2B: Nuclear Fusion �� 27

2C: Solar�� 33

2D: Nuclear Fission �� 37

Section 3: Biology �� 49

Section 4: Socioeconomic Modeling�� 57

Section 5: Astrophysics �� 73

Section 6: Math and Algorithms �� 85

Section 7: Software �� 101

Section 8: Hardware �� 109

Section 9: Cyberinfrastructure �� 117

Appendix A: Initiative Summary �� 129

Appendix B: Agendas �� 135

Appendix C: Attendees �� 141

Appendix D: Abbreviations and Terminology �� 153

Appendix E: References �� 157

Executive Summary

Exascale Town Hall Meetings Letter Report

Lawrence Berkeley, Oak Ridge, and Argonne national laboratories convened three town hall meetings aimed at collecting community input on the prospects of a proposed new Department of Energy (DOE) initiative entitled Simulation and Modeling at the Exascale for Energy and the Environment, or E3 for short.

The goal of the town hall meetings was to engage the computational science community in a series of broad and open discussions about the potential benefits of advanced computing at the exascale (1018 operations per second) on “global” challenge problems in the areas of energy, the environment, and basic science.

Approximately 450 researchers from universities, national laboratories, and U.S. companies participated at the three meetings held in April, May, and June 2007.

In addition to the scientific and engineering challenges and opportunities, the meetings also addressed needed advances in computer science and software technology, large-scale hardware, applied mathematics, and cyberinfrastructure and cyber security.

Here we summarize the major conclusions of the town hall meetings. Feasibility of Exascale Systems General-purpose exascale computer systems are expected to be technologically feasible within the next 15 years. These systems will likely have between 10 million and 100 million processing elements or cores. The major U.S. vendors of large-scale systems and processors (e.g., IBM, Intel, Cray, AMD) are in general agreement that these systems will push the envelope of a number of important technologies, including processor architecture, scale of multicore integration (perhaps into the range of 1000 cores per chip or beyond), power management, and packaging. The projected exascale systems themselves will have part counts comparable to those of today’s largest systems (or slightly larger). Detailed cost studies have not been done, but the consensus is that costs will be comparable to those of the largest systems being contemplated today ($100 million to $200 million per system).

Significant challenges arise in accomplishing exascale computing, in areas that include architecture, scale, power, reliability, cost, and packaging. A major source of uncertainty is how quickly the general marketplace will be able to adopt highly parallel, single-chip, multicore systems in normal information technology (IT) products. The current belief is that the broad market is not likely to be able to adopt multicore systems at the 1000-processor level without a substantial revolution in software and programming techniques for the hundreds of thousands of programmers who work in industry and do not yet have adequate parallel programming skills.

Extrapolation of current hardware trends suggests that exacale systems could be available in the marketplace by approximately 2022 via a “business as usual” scenario. With the appropriate level of investments, it may be possible to accelerate the availability by up to five years, to approximately 2017.

vii Executive Summary

Exascale systems will also require substantial investments in input/output (I/O) and storage research. The current trends in disk drives and other storage technologies are optimized for the consumer market and may not have the optimal ratios of capacity to bandwidth needed for large-scale systems.

Power efficiency is also expected to be a major problem, with the goal of an exaflops system at less than 20 MW sustained power consumption perhaps achievable. Driving the earlier availability of the systems will compromise the power efficiencies to some degree.

We note that Japan has outlined in its current petascale initiative a rough roadmap to the exascale that proceeds via three systems: a 10 petaflops (PF) system in ~2012, a 100 PF system in ~2017, and a 1000 PF system in the ~2022 timeframe. It appears possible for a U.S. computing program to maintain leadership during the next decade in this area – but only if increased investments are started immediately and are sustained over the long term. Science and Engineering Opportunities The three town hall meetings examined a range of applications that would be materially transformed by the availability of exascale systems. We highlight here several significant opportunities in the areas of energy, climate, socioeconomics, biology, and astrophysics. Energy Energy research offers significant opportunities to exploit computing at the exascale, in order to advance our understanding of basic processes in areas such as combustion, which would naturally lead to a design capability for improving the efficient use of liquid fuels, whether from fossil sources or renewable sources. First-principles computational design and optimization of catalysts will become possible at the exascale, as will de novo design of biologically mediated pathways for energy conversion.

Access to exascale systems and the appropriate applications codes could have a dramatic impact on nuclear fission reactor design and optimization and would help accelerate understanding of key plasma physics phenomena in fusion science critical to getting the most from the U.S. investment in ITER.

Exascale systems should also enable a major paradigm shift in the use of large-scale optimization techniques to search for near-optimal solutions to engineering problems. Many energy and industrial problems are amenable to such an approach, in which many petascale instances of the problem are run simultaneously under the control of a global optimization procedure that can focus the search on parameters that produce an optimal outcome. Environment Three broad areas relating to the environment were discussed: climate modeling; integrated energy, economics, and environmental modeling; and multiscale biological modeling from molecules to ecosystems.

Climate modeling. As the most mature of the three environmental application areas, climate modeling is expected to make good use of exascale systems. The impact of these systems will be threefold. First, they will enable the development of much higher resolution models that will advance our understanding of local impacts of climate change; second, they will enable the dramatic improvement of physical, chemical, and biological process representations in the climate models, which will more accurately reflect the real climate system; and third, they will enable a thorough exploration of the parameters that give rise to uncertainty in climate models via large-scale ensemble computations. Significant investments will be needed, however, to port and improve climate models for exascale architectures, including the explicit targeting of multicore in next-generation models and the development of an integrated climate research computing environment that will link climate modelers with climate data sources, collaborators, and university and laboratory resources.

Integrating energy, socioeconomics, and environmental modeling. There exists a considerable opportunity to couple detailed computer models of energy utilization and production with geospatialized socioeconomic models and, in turn, to couple these to an Earth systems model that captures the feedbacks to and from the environment from human activities. This integrated modeling suite would enable fundamental research into strategies for sustainable global economic development and would lead to exploration of alternative development paths and their impacts on global energy security. viii Modeling and Simulation at the Exascale for Energy and the Environment Multiscale biological modeling. Large-scale computing is starting to have an impact in the biological sciences. Exascale computing will enable computational biologists to begin to build models that can bridge the space-time parameters that characterize important biological processes, including models of diverse microbial ecosystems from which we may gain considerable new biotechnology (bioenergy, carbon sequestration, environmental technology, and industrial processes). Bridging the scales from the molecular to the ecosystem offers many challenges for model developers, but it also provides many opportunities for coupling research in high-throughput genomics, proteomics, and bioinformatics to applications via exascale computing. This activity is key, for example, to accelerating the computing vision of programs such as DOE’s Genomics:GTL initiative. Astrophysics Simulation opportunities in astrophysics include large-scale structure formation, galaxy formation, stellar evolution, supernovae, and compact objects. For example, in the Type Ia supernova problem, exascale computing will enable simulations with resolutions down to the Gibson scale (the length scale at which turbulent motion is effectively smoothed by the propagation of the nuclear flame) with definitive prescriptions for nuclear energy release and the associated nucleosynthesis.

Core collapse supernovae simulations with the spatial resolution required to properly model critical aspects of the explosion dynamics (e.g., the evolution of the stellar core magnetic fields and their role in generating the supernova) will require much higher resolution than today’s terascale codes. These codes, in turn, will require exascale computing, particularly if a number of simulations are to be performed across the range of stellar progenitors and input physics. One such simulation is expected to take ~8 weeks, assuming 20% efficiency on an exaflops machine.

Advances in these areas will require the adaptation of existing, and in some cases the development of new solution algorithms for the underlying partial differential equations governing the evolution of these astrophysical systems and the codes that execute them, as well as the optimization of both as they advance to the exascale. Computer Science and Applied Mathematics To realize science at the exascale will require a concerted effort to couple advances in algorithms, programming models, operating systems, filesystems, I/O environments, and data analysis tools. In fact, exascale systems are likely to be so demanding that they will drive new working relationships between the disciplinary scientists and the computer science and mathematics communities.

Of great interest are methods that will enable the power of exascale computing to advance the use of mathematical optimization in many areas of science and engineering. Examples include the use of ensembles and “outer loop” optimization to iterate design parameters of new nuclear reactor designs that would simultaneously improve safety margins and lower cost, or to explore the parameter space of technology choices and how they might impact global energy security strategies.

Specific challenges that need to be overcome include development of scalable operating system services that can manage 10 million to 100 million cores, scalable programming models and tools that will enable developers to express orders of magnitude more concurrency in applications, and data storage environments that can scale to exabytes of capacity and sustained transfer speeds of terabytes per second. While reaching the needed scaling and performance goals will be a challenge, the community believes that it is possible and achievable on a schedule that would not limit the prospect of accelerating availability of exascale systems to 2017. Cyberinfrastructure and Cyber Security Large-scale computing resources are only a part of the overall computing environment needed to advance science. This environment also includes high-performance networking, mid-range and smaller clusters, visualization engines, large- scale data archives, a variety of data sources and instrumentation including emerging sensor networks, and the tens of thousands of workstations that enable access to and are the primary development machines. Complementing the hardware and networking is a vast software ecosystem that connects resources and enables them to work as part of a whole, spanning networking software, databases, security, and hundreds of domain-specific tools. This overall collection, commonly referred to as “cyberinfrastructure,” will require investments to fully exploit the power and promise of exascale computing. The quality and robustness of the cyberinfrastructure will impact the productivity of the exascale computing resources. Additional

ix Executive Summary investments in cyberinfrastructure and cyber security are needed to ensure that large-scale systems will be productive and secure. While extreme-scale systems are sometimes the targets of security attacks, they are generally well protected by layers of infrastructure. On the other hand, the general cyberinfrastructure that provides the rich computing environment surrounding the extreme-scale systems is often vulnerable. Clearly, we must be wise in our development of exascale systems to make balanced investments in the security of the overall scientific computing environment. Conclusions The broad computational science community has a golden opportunity to accelerate the availability of usable exascale systems. To take full advantage of this opportunity to deliver exascale computing by 2017 will require an integrated program of investments in hardware and software research and development, (R&D). Also required will be a tight coupling to a selected set of science communities and the associated applied mathematics R&D. In some cases, such as astrophysics and climate, the communities are well on the way to exploiting petascale systems. In other cases, such as socioeconomics and multiscale biology, there is great opportunity for acceleration. Computational science and engineering opportunities in energy are wide and deep and have an enormous potential impact on advancing energy technology and fundamental science. If acceleration is to be achieved—and there is every reason to both desire it and believe that it can be accomplished—then every minute will count, and even modest investments early in the cycle (e.g., 2008 and 2009) could have dramatic benefit and will reduce uncertainties moving ahead.

Rick Stevens, Argonne National Laboratory Thomas Zacharia, Oak Ridge National Labortory Horst Simon, Lawrence Berkeley National Laboratory

x Introduction

The U.S. Department of Energy (DOE) Office of Advanced offer the potential to unravel scientific mysteries that we have Scientific Computing Research (OASCR) has proposed a not yet been able to address. Examples relevant to DOE mis- 10-year initiative on Simulation and Modeling at the Exascale sions include: for Energy, Ecological Sustainability, and Global Security (E3). This initiative, which is aligned with the strategic theme • Resolving clouds, forecasting weather and extreme of scientific discovery and innovation in DOE’s Strategic events, and providing quantitative mitigation strategies Plan, is designed to focus the computational science experi- • Understanding high-temperature superconductivity ence gained over the past ten years on the opportunities that will be introduced with exascale computing to revolutionize • Developing clean and efficient combustion systems for our approaches to global challenges in energy, environmen- diesel and alternative fuels tal sustainability, and security. A summary of the E3 initiative is presented in Appendix A. • Developing a detailed understanding of cellulase enzyme mechanisms and creating more efficient enzymes for cel- Planned petascale and potential exascale systems provide an lulose degradation through protein engineering unprecedented opportunity to attack these global challenges through modeling and simulation. In combination with theory • Understanding the interaction of radiation with materials and experiment, computation has become a critical tool for • Advancing magnetic fusion through predictive capabili- understanding the behavior of the fundamental components ties with core-edge coupling, realistic mass ratios, and of nature and for exploring complex systems with billions of validated turbulence models for ITER components, including humans. Computing has already been used in partnership with theory and experiment to attack such • Explaining and predicting core-collapse supernovae and problems as the time evolution of atmospheric CO2 concen- putting theories of general relativity, dense equation of trations originating from the land surface, the activity of the state (EOS), and stellar evolution to the test cellulase enzyme on a time scale of 50 to 100 nanoseconds (ns), the stabilization of lifted flames in diesel engines and Equally important, leading the development, acquisition, and gas turbine combustors, and the behavior of superheated ionic deployment of exascale systems has the potential to make gases in plasmas. U.S. industry more competitive and to enable the solution of problems of national importance. In response to the OASCR The deployment in this decade of several systems with peak initiative, Argonne National Laboratory (ANL), Lawrence performance in the range of 1018 operations per second (peta- Berkeley National Laboratory (LBNL), and Oak Ridge flops), enabling simulations sustaining hundreds of teraflops, National Laboratory (ORNL) organized a community input should be followed in the next decade by systems with peak process in the form of three town hall meetings (see Table I.1). performance in the exaflops range and simulations sustaining The agendas of these meetings are provided in Appendix B. a hundred or more petaflops. Exascale computers will have About 450 participants, listed in Appendix C, attended these processing capability similar to that of the human brain and three town hall meetings and contributed to this report.

Location Date Web site Lawrence Berkeley National Laboratory April 17–18, 2007 http://hpcrd.lbl.gov/E3SGS/main.html Oak Ridge National Laboratory May 17–18, 2007 http://computing.ornl.gov/workshops/town_hall/index.shtml Argonne National Laboratory May 31–June 1, 2007 https://www.cls.anl.gov/events/workshops/townhall07/index.php Table I.1 Town hall meetings on Modeling and Simulation at the Exascale for Energy and the Environment

1 Introduction

The goals of the town hall meetings were to integrate increasingly complex astrophysical measurements with simulations of the growth and evolution of • to gather community input for possible DOE research ini- structure in the universe, linking the known laws of mi- tiatives in high-performance computing (HPC), computer crophysics to the macro world. science, computational science and advanced mathematics, and to examine how these capabilities could be ap- The four technical areas address the development and deploy- plied to global challenge problems; ment of the tools needed to deliver scientific discovery at the exascale: • to examine the prospects for dramatically broadening the reach of HPC to new disciplines, including areas such as • Math and Algorithms. Advancing mathematical and al- predictive modeling in biology and ecology, integrative gorithmic foundations to support scientific computing in modeling in earth and economics sciences, and bottom‑up emerging disciplines such as molecular self-assembly, sys- design for energy and advanced technologies; tems biology, behavior of complex systems, agent-based modeling, and evolutionary and adaptive computing. • to identify emerging domains of computation and computational science that could have dramatic impacts on • Software. Integrating large, complex, and possibly dis- economic development, such as agent-based simulation, tributed software systems with components derived self-assembly, and self-organization; from multiple application domains and with distributed data gathering and analysis tools. • to outline the challenges and opportunities for exascale- capable systems, ultralow-power architectures, and ubiq- • Hardware. Driving innovation at the frontiers of com- uitous multicore technologies (including software); and puter architecture and information technology, preparing the way for the ubiquitous adoption of parallel comput- • to identify opportunities for end-to-end investment in new ing, power-efficient systems, and the software and archi- computational science problem areas (including valida- tectures needed for a decade of increased capabilities, tion and verification). and accelerating the development of special-purpose Each town hall meeting was a day and a half in length and devices with the potential to change the simulation para- combined invited plenary talks and parallel breakout sessions. digm for certain science disciplines. Breakout sessions at each meeting were organized and facili- • Cyberinfrastructure. Developing tools and methods to tated by a team of leading experts with representation from protect the distributed information technology infra- each of the three laboratories. At all three meetings, breakout structure by ensuring network security, preventing dis- sessions were focused on five application areas and four tech- ruption of our communications infrastructure, and de- nical areas. The application areas and their central goals were fending distributed systems against attacks. as follows: • Climate. Improve our understanding of complex biogeo- Each breakout session was tasked with addressing eight chemical (C, N, P, etc.) cycles that underpin global eco- charge questions: systems functions and control the sustainability of life on • What (in broad brush) is feasible or plausible to accom- Earth. plish in 5–10 years? • Energy. Develop and optimize new pathways for renewable energy production and development of long-term, • What are the major challenges in the area? secure nuclear energy sources through computational • What is the state of the art in the area? nanoscience and physics-based engineering models. • Biology. Enhance our understanding of the roles and • How would we accelerate development? functions of microbial life on Earth, and adapt these capa- • What are expected outcomes and impact of acceleration bilities for human use, through bioinformatics and com- or increased investment (i.e., what problems would we putational biology. aim to solve or events we would cause to occur)? • Socioeconomics. Develop integrated modeling environments for coupling the wealth of observational data and • What scale of investment would be needed to accom- complex models to economic, energy, and resource mod- plish the outcome? els that incorporate the human dynamic, enabling large- • What are the major risks? scale global change analysis. • Astrophysics. Develop a “cosmic simulator” capability • What and who are missing?

2 Modeling and Simulation at the Exascale for Energy and the Environment This report provides detailed answers to these questions hardware and software research and development (R&D), for each breakout topic; the major challenges for each carried out in partnership with key science communities and application and technical area are summarized in Table I.2. accompanied by applied mathematics R&D, can be expected The consensus at the town hall meetings was that all of these to produce the transformational science and disruptive challenges, while formidable, can be overcome if action is technologies needed to successfully attack global challenges taken immediately to accelerate the availability of usable in energy, the environment, and basic science. exascale systems. An integrated program of investments in

Major challenges in exascale computing Topic Major challenges

Climate v Integrating high-resolution Earth system models with massive assimilation of satellite and other data v Detailed modeling of controlled and modified ecosystems to fit the environmental envelope in which future climate changes will occur v Development of process-scale mechanistic models for biogeochemical, hydroecological, cloud microphysical, and aerosol processes v Rational design and analysis of computer experiments to navigate very large parameter space with very large outputs

Energy v Combustion: – Predictive simulation capabilities that can accurately model combustion in new high-temperature, low-emission regimes – Robust and reliable ignition and combustion models for next-generation engines and power plants – Multiscale formulations that can exploit the specialized structure of typical combustion applications – Scalable algorithms for multiphysics reacting-flow problems – Improved discretization procedures – Management of software complexity – New tools for data management and information v Nuclear fusion: – Accelerated development of computational tools and techniques to extend the scientific understanding needed to develop predictive models – Advanced computations (in tandem with experiment and theory) to deliver the new science and technology needed to achieve continuous power with higher Q in a device similar to ITER in size and magnetic field v Solar energy: – Exploration of huge parameter spaces – Identification of the best materials and designs for device improvement and optimization, either through direct numerical material-by-design searches or through new understanding of fundamental processes in nanosystems v Nuclear fission: – Identification of fuel cycles that reduce generation of high-level radioactive waste – Reducing the time required for fuel development and qualification – New tools for assessing life-cycle performance, addressing safety concerns, and predicting fuel rod behavior in accidents – Accurate predictions of the behavior of transuranic fuel

Biology v Model-driven high-throughput experimental data generation v Improving model development by incorporating genome-scale metabolic networks, regulatory networks, signaling and developmental pathways, microbial ecosystems, and complex biogeochemical interactions v New bioinformatics techniques to address the integration of genomics, proteomics, metagenomics, and structural data to screen for novel protein function discovery v Molecular modeling techniques that can address multiscale challenges

Table I.2 Major challenges in exascale computing

3 Introduction

Topic Major challenges Socioeconomic v Comprehensive suite of validated models of unprecedented geospatial and temporal detail modeling v Comprehensive error analysis v Leverage of state-of-the-art climate modeling activities (to include economic prediction models under alternative climate regimes, supported by basic research into spatial statistics, modeling of social processes, relevant micro-activity and biosphere coupling issues, and relevant mathematical challenges, such as multiscale modeling) v Novel, robust numerical techniques and HPC approaches to deal with the expected orders-of-magnitude increase in model complexity v Assembly and quality control of extensive data collections

Astrophysics v Simulation of the formation of large-scale structures to understand the nature of dark energy v Detailed simulations of the formation of galaxies to compare with observational data, requiring dynamic ranges of order 10,000 in space and time v Full-scale simulation with validation quality of the helium shell flash convection zone in stars v Supernova models that include realistic nucleosynthesis studies v Accurate descriptions of binary systems (e.g., a black hole and a neutron star or two neutron stars) Technical areas Math and v Systematic approach for quantifying, estimating, and controlling the uncertainty algorithms caused by (e.g.) reduced models, uncertain parameters, or discretization error v Robust and reliable optimization techniques that exploit evolving architectures and are easy to use v Appropriate algorithms for novel optimization paradigms that can be implemented only at the exascale (e.g., hierarchical optimization problems over multiple time stages) v Handling of problems with hundreds of thousands of discrete parameters. v AMR for efficient solution of linear and nonlinear systems of partial differential equations (PDEs) v Dynamic load balancing v New data representations, data handling algorithms, efficient implementations of data analysis algorithms on HPC platforms, and representations of analysis results for massive data sets Software v Development and formal verification tools integrated with exascale programming models v New fault tolerance paradigms v Application development tools, runtime steering, post-analysis, and visualization v New approaches to handle the entire data life-cycle of exascale simulations (effective formats for storing and managing scientific data, automatically capturing provenance, seamlessly integrating data into scientists’ workflow) Hardware v Performance per watt v Large-scale integration (packaging 10M to 100M cores with their associated memories and interconnects) v Integrated hardware- and software-based fault management v Integrated programming models

Cyber v Scalable and flexible resources for representing information and reducing infrastructure information overload v Federated approach to: – Authentication and authorization – Creation and management of virtual organizations v Higher performance tools and techniques for data management and movement v Security products v Tools and techniques for system configuration, verification, troubleshooting, and management v Framework and semantics for integrating information in individual cyber security component systems for situational awareness, anomaly detection, and intrusion response v Data transfer tools that provide dedicated channels for control communication and graded levels of control

Table I.2 Major challenges in exascale computing

4 1 Climate

How can we improve our understanding of ing, modeling capability, and computational complex biogeochemical cycles that under- infrastructure from that represented in the pin global ecosystem functions and control current studies of global average quantities the sustainability of life on Earth? The ur- [National Research Council 2001]. Achiev- gency of developing a science of global eco- ing this capability may be impossible without systems is common for several key questions. a concentrated effort over the next 5–10 years The U.S. Climate Change Science Program to develop the detailed process models that and the Intergovernmental Panel on Climate are demanded by increased spatial resolution Change (IPCC) have concluded that climate and driven by societal needs. We believe that change will accelerate rapidly during the 21st accelerating scientific development, through century unless there are dramatic reductions targeted attack and application of exascale in greenhouse emissions [Alley et al. 2007]. simulation, is the best way to make a differ- Assessments by the IPCC and the Military ence in the limited time while key decisions Advisory Board [CNA Corporation 2007] must be made. suggest that global warming could have serious implications for the natural and social We have identified ten key scientific ques- fabric in many parts of the world. Fortunately, tions that address major unsolved issues and sensible policies to reduce greenhouse emis- represent targets of opportunity for computa- sions could be formulated by using reliable tionally intensive simulation. Resolution of climate forecasts and developing next-gener- these questions will yield a much better, more ation Earth system models (ESMs), including defensible scientific grounding for policy and processes and mechanisms to represent the political discourse. Dramatic advances in the most likely mitigation strategies that depend science will be required, however, to provide on ecological and biological process over robust answers and quantitative estimates of land, over oceans, and below the ground. uncertainty. After describing the urgent questions, we discuss the scientific advances that To develop the necessary forecasts, scientists are necessary to address these issues. Com- must address two major challenges. First, mon to many of the scientific advances are how well can we forecast with increased cer- more accurate multiscale models that inte- tainty the committed climate change over the grate the physical, chemical, and biological next few decades resulting from historical processes in the climate system. emissions? Second, how well can we forecast longer-term climate change (Figure 1.1), in- A new type of multidisciplinary research pro- cluding interactions and feedbacks between gram is urgently required to advance the state all components of the ESM and at spatial of our knowledge, to address the attendant scales of relevance to communities? The scientific challenges, and to project the future second question is particularly difficult to of Earth’s environment from the local to the answer given our rather limited and rudimen- global scale. Such a program is beyond the tary knowledge of biogeochemical cycles and scope of the current limited, piecemeal ap- A new type of interdisciplinary feedbacks. proach to climate modeling adopted by sever- research is needed to project al U.S. agencies. Required over the next 5–10 the future of the Earth’s Meeting these challenges will require a quali- years are focused and well-scoped invest- environment from the local to tatively different level of scientific understand- ments in rapidly developing process-scale the global scale.

5 Section 1: Climate

1. Urgent Earth System Questions

Each of the ten questions in this section present significant scientific challenges. In some cases, these challenges can be overcome by basic research into processes, better observation networks, deeper theoretical understanding, and more advanced modeling approaches. In all cases, the path will be more direct and progress accelerated if we can take advantage of petascale and exascale computational power. As the demand is amplified for accurate and reliable predictions of the causes and effects of climate change, the best approach that scientists can take is to continue the development of comprehensive ESMs that can be used as scientific tools Figure 1.1 Relation between weather prediction and climate change studies. Climate to determine the safe concentration levels prediction covers time scales of months to decades and has great relevance to threat for CO2 and other greenhouse gases in the assessment in hydropower, ecosystems, and energy sectors. (Image courtesy of atmosphere. While the scientific community Kevin Trenberth [NCAR]). is engaged in expanding our theoretical knowledge and improving our observational science related to biological and ecological depth, we are committed to exploiting our processes of the Earth system; new method- new understanding to address the societal ologies and software tools that can integrate challenges posed by climate change. This these branches of Earth system science with initiative will allow the science community to the existing ESMs; and significantly larger accelerate these efforts. computing resources, such as those proposed by the exascale program. The opportunity to 1.1 Development of Carbon positively affect the outcome of the current Sequestration Process Models global change debate is restricted by the current inability of the models to address these Decisions on land use and carbon capture and regional and local-scale impacts effectively. storage technologies will have to be made over Significant investment over the next few years the next few decades. Can we develop new and can lead to a quantitative impact on this pro- coupled models representing the microbial, cess. Climate science is largely data limited, ecological, and physiological processes for and the success of the research is contingent methods that are currently under consider- on basic measurements and observations nec- ation in oceans, land, and subsurface? essary to validate, verify, and constrain ESMs. Quantifying the uncertainties in predictions is If we had five years to come up with ase- expected to require a new level of integration questration strategy, we would have to use between modeling and observational science. reduced-form models. Major factors are ne- New mathematical methods and algorithmic glected in such models; more detailed models techniques will also be required to address clearly are desired to take account of carbon the fundamental challenges of multiscale and allocation under large perturbations, of plant multiphysics coupling. Even with exascale mortality, of species migration, and of change computing, approximations and assumptions rates. The ability to do detailed and opera- must be made. Computing power has been tional forecasting of the carbon cycle and Quantifying the uncertainties in and will continue to be a key factor in making climate in the 20- to 50-year range would predictions will require a new these advances possible. level of integration between put sequestration strategies on firm scientific modeling and observational ground and be a valuable tool in helping soci- science.

6 Modeling and Simulation at the Exascale for Energy and the Environment ety adapt to decadal climate change and sea- needed is a more rigorous systems approach, Models development is critical: sonal transients. utilizing control theory for systems of partial In many cases we do not know differential equations (PDEs). Such a theory even the sign of the carbon The predictability of short-term carbon– will allow model developers to determine flux signal for many parts of the earth. climate models must be rigorously assessed whether important factors and processes are through evaluation with historical data sets. A missing from current models and to pinpoint significant emphasis of this theme will be to model components that contain errors when incorporate measurements and observations compared with the historical climate record. (Figure 1.2) to develop more mechanistic- based models of the various ecological and Abrupt climate change, and the potential for biological processes at scales ranging from rapid shifts from one climate equilibrium state a single tree or plant to scales of ecological to another, could be understood in the context systems. For example, starting with the year of this system theory. The ability to develop 1870 (preindustrial conditions), modeled car- accurate models that incorporate multiscale bon budgets driven by land use change and phenomena from process studies would be increasing atmospheric concentration can be greatly advanced by such a theory. performed with some certainty. Since predictability in the decadal range is expected to be 1.3 Probability of Extreme Weather low (based on theoretical results, Figure 1.3), Events and Significant Shifts in data assimilation techniques for carbon will Regional Climates be required to constrain these hindcast predictions. Because of slow decomposition of As climate change accelerates, questions frozen soils in high latitudes, carbon storage arise regarding the frequency and occurrence in soil and litter is greater in this part of the of extreme weather shifts in regional climate global ecosystem, almost twice as concentrat- patterns. How can the climate models be ed in the boreal and tundra regions as in tem- adapted to meet these challenges? perate regions. With significant changes to the precipitation in high latitudes (up to 20% The study of extreme events using ESMs is increase for IPCC scenarios), the possibility just beginning, but this is just the kind of infor- of abrupt changes and release of large stored mation that affects community needs. Some carbon pools needs to be investigated. This of the largest impacts of climate change are effort requires model development and data collection to understand the processes that form the foundation for further model development work and ultimate integration into an ESM. We do not know the sign of the carbon flux signal for many parts of the Earth under climate change scenarios.

1.2 Characterizing and Bounding the Coupled Earth System

A systematic understanding of the mathematics of the climate system has yet to be discovered. Can we develop a theory of internal and forced modes of multiphysics, multiscale components that interact as a coupled system?

The paleo record provides evidence that the carbon cycle is well bounded, especially in the interglacial periods, but there are no asymp- Figure 1.2 The terrestrial biosphere as a consumer and producer of chemicals in the atmosphere. As chemicals cycle through plants, soil, and atmosphere, the long- totic analyses of the cycle based on the math- term feedbacks affect where they are stored. Nutrients such as nitrogen stimulate ematics of carbon–climate models. Clearly plant growth.

7 Section 1: Climate

associated with changes in relatively rare but scientific issues that introduce first-order extreme localized phenomena, such as more uncertainties in climate forecasts. Since sta- intense hurricanes, violent rainstorms, flash tistics are important for extreme events, a floods, and heat waves, as well as low-fre- computational challenge arises in character- quency extremes such as droughts. More tem- izing the tails of the distributions where ex- poral and spatial specificity at scales relevant treme events occur. for agriculture, industry, and society is not yet feasible from a computational viewpoint. With petascale computing, horizontal reso- The ability of existing models to accurately lution can be increased to the 10- to 25-km simulate extreme temperature and precipita- scale, permitting reasonable simulation of tion events is severely limited by horizontal tropical cyclones. Moderate increases in en- resolution constraints. Furthermore, since ex- semble size, from the current state of the art treme events are rare by definition, adequate of 10 realizations to about 50 realizations, statistical characterization of the tails of the should also be possible. Exascale computing distribution of weather events is required to will permit a combination of further resolu- make quantitative statements about changes tion increases to better resolve individual in their behavior. It is likely that downscaling storms and increased ensemble size to better methods will still be needed to reach the local capture extreme value statistics. This will of- scale, even with exascale computing power. fer a great advance in characterizing the un- An important part of this challenge involves certainty of climate models and provide the engaging stakeholders to iteratively define the impacts community with reliable expecta- interface and the important metadata needed tions of models. to interpret or interpolate between analysis tools, such as global information system 1.4 Sustainability of the Tropical (GIS) collections or GoogleEarth. How do Rain Forest we tell what is important? Can priorities be model-based? Precipitation in the tropics is a leading-order factor governing the carbon cycle. What are Using models, we should be able to iden- the magnitude and stability of the carbon– tify key triggering mechanisms for extreme climate feedback for tropical ecosystems? weather and climate events and identify open The Amazon rain forest plays a pivotal role in the climate and, in particular, the carbon cycle. If this ecosystem were to collapse, a large amount of carbon from decaying plants would be released into the atmosphere Since climate change simulations indicate that precipitation will decrease in the tropics under warming scenarios and that less snowmelt will feed the Amazon River basin, a drier Amazon could represent a positive feedback for global warming. Attempts to forecast this situation highlight the possibility of large changes in the next 50 years.

New methods and models are urgently needed, with increased details and significantly more species diversity of plant and microbial life. Also needed are mechanistic process- based models of below- and above-ground Figure 1.3 Predictability of weather and climate models: high on the short time ecology. scales and in the long, asymptotic scales. Since many of the questions to be an- swered are targeted to the 20–50 year range, the ability of models to provide reliable forecasts will be challenged. Image courtesy of Kevin Trenberth (NCAR).

8 Modeling and Simulation at the Exascale for Energy and the Environment

1.5 Stability of the Polar Caps ter redistribution beneath ice streams, and Exascale simulation is a key and Greenland and Antarctic Ice their tributaries, and flow acceleration and tool in predicting the course of Sheets divergence into an ice shelf. Successive de- sheet ice dynamics. velopment will couple this ice sheet simulator Melting or breakup of either the Greenland to models of the dynamic earth, atmosphere, ice sheet or significant portions of the Antarc- and ocean for predictions of future sea-level tic ice sheet could cause a sea level rise of 6 m. change. Systematic methods of constraining What is the likelihood of this happening and the model against the available datasets us- on what time scale? ing inverse modeling have very high computational demands but will help yield robust The past 15 years has seen an unanticipated results. acceleration of ice flow into the ocean from individual catchments of the Greenland and 1.6 Release of Methane Hydrates Antarctic ice sheets. If these accelerations are sustained, even larger portions of the polar As warming occurs in the oceans and on land, ice sheets could become vulnerable to mass frozen deposits of methane will be released wasting, with potentially grave consequences into the atmosphere. What is the potential in for society. Sea level rise is likely the great- the next 20–50 years for a sudden increase in est uncertainty in evaluating climate change warming as a result of melting of Arctic and impacts. ocean-shelf deposits of methane hydrates?

Currently we cannot exclude the possibility Historic records need to be further quantified, of a >1 m cumulative sea level rise over the and new observations are needed to quantify next century because of partial collapse of the the potential for a sudden release and posi- major ice sheets, in addition to the 0.5-m rise tive feedback. High-latitude peatland regions expected due to thermal expansion. This prob- are rapidly warming, and as permafrost melts, lem is coupled not only to the climate system the shift from anaerobic to aerobic condi- but also to energy and population security. tions will need to be part of ESM land sur- Densely populated coastal areas would be face schemes. Ocean-shelf methane hydrate severely impacted. Critical infrastructure— deposits similarly will release methane as a including oil refineries (80% of U.S. refining threshold temperature is exceeded. Models capacity is at ≤ 1.5 m above sea level), nucle- of methane hydrate deposits need to be de- ar power stations, ports, and industrial facili- veloped and coupled to both the land surface ties—is often concentrated around coastlines. process models and deep ocean circulation Coastal biomes, many of which are implicitly models. part of coastal infrastructure designs, may be severely impacted worldwide. 1.7 Sustainability of Sea Life

Exascale simulation is a key tool in predicting The increasing concentration of atmospheric the likely course of ice sheet dynamics. Pre- CO2 is changing the pH of the ocean. What viously unanticipated complexity in ice sheet level of change will trigger coral reef col- dynamics is emerging, and new observational lapse, impacts on fisheries, megafauna techniques are providing a wealth of data on changes, and a change in the ability of ocean past and present controls on ice sheet change. organisms to take up CO2? A new dynamic ice sheet model must incorporate new processes, including ice-stream flow The ocean removes a large percentage of CO2 regime change, production and transport of from the atmosphere. The increased levels of basal fluids, surface-melt to bed lubrication, CO2 in the ocean reduce the carbonate ions fracturing, grounding line physics, and ice- available for producing calcium carbonate shelf/ocean interactions. Such a model must shells. The result is that the skeletal growth represent scales appropriate for slow creep rates of corals and some plankton can slow, deformation and fabric formation within the and in extreme cases some shells may even vast ice sheet interior, fast plug flow and wa- dissolve. Research is needed to understand

9 Section 1: Climate

Competition over water the ocean’s role as a “sink” for CO2 and to at these scales for precipitation and hydrol- resources threatens security determine the potential impacts on the ocean ogy is essential to effectively address these is- and political stability in many food web. sues over the next couple of decades. Model areas of the world: New models development focused on methodologies for are needed for predicting river 1.8 Sustainability and Agricultural flow and underground water. dealing with the stochastic nature of precipi- Ecosystems tation is urgently needed, as is development of more hydrological basin-scale based ap- The shift from agricultural or forest produc- proaches for predicting river flow and un- tion to production of crops for biofuel could derground water resources. New approaches be a significant change in land use patterns with remotely sensed GRACE (Gravity Re- in 10–20 years. How might this change the covery and Climate Experiment) water levels climate’s hydrologic cycle and affect the and data assimilation techniques will reduce potential for carbon sequestration of the these groundwater model uncertainties. biosphere? 1.10 Dynamical Linking of Biofuels offer an attractive alternative fuel Socioeconomic and Climate

source that reduces the net emissions of CO2 Responses into the atmosphere and enhances our inde- pendence from declining and potentially un- At present, there exist one-way and loosely stable sources of petroleum. Development coupled flows of information between physi- of other fuel sources also looks promising. cally based ESM and socioeconomic model The benefits and drawbacks need to be con- interfaces. How can we ensure two-way dy- sidered, however, in the context of a carbon namic feedback between these models? sequestration strategy, pollution control, and the hydrological cycle. To this end detailed Current models are moving from prescribed models of agricultural ecosystems are needed emissions scenarios to dynamic emission at the level of each crop and associated land scenarios, and ESMs are beginning to include use (see Figure 1.4). Existing models are fair- dynamic feedback from impacts models [Fos- ly simple and parametric [Ma, Shaffer, and ter 2007]. Bringing the needed feedbacks into Ahuja 2001]. A significant investment and a coupled ESM requires a methodology for larger computational resources are urgently incorporating social response as a function of needed to advance the development of these climate into the emissions scenarios, incorpo- models and their integration into an ESM. ration of detailed local-to-regional socioeconomic phenomena, and decision-game theory 1.9 Changes in Precipitation mapped into these concepts. Patterns and Hydrology 2. State of the Art Regional scale shifts in climate patterns could stress surface and groundwater resources The science surrounding the biogeochemi- and lead to a disruption of current levels of cal coupling of climate has become central agriculture production and overall econom- to answering these questions as we learn ic sustainability. What is the extent of these more about how the coupled carbon cycle changes, where do they occur, how often, and has changed in the fossil record, how it is what do we need in the ESM to predict these changing now, and how it might change in re- changes with some certainty? sponse to global climate change. Addressing the science issues will require new observa- The intelligence community is calling cli- tions and methods of analysis, new theoreti- mate change a serious threat to global secu- cal understanding of the carbon cycle, and rity. Competition over water resources under new models of the Earth system that include stress and regional scale shifts in precipita- the interactions between human society and tion patterns could further affect security and the environment. These models play pivotal political stability in many areas of the world. roles in interpreting the paleoclimate records, An increase in confidence in ESM predictions in synthesizing and integrating measurements

10 Modeling and Simulation at the Exascale for Energy and the Environment to study the current carbon cycle, and in pro- jecting the future responses of human society and the natural world to evolving climate regimes.

One of the most promising pathways to improving our understanding has been to develop models that represent the complexity of interactions in the Earth system as accurately as possible. Over the past 30 years, these models have advanced considerably in spatial and temporal resolution and in the representation of key processes. However, forecasts of environmental and societal responses to climate change remain highly uncertain. The principal challenges are quantifying the sources of uncertainty, reducing the level of uncertainty at all scales using observations and fundamental theory, and understanding the natural and an- Figure 1.4 Biofuel production will entail land use changes that interact with the cli- thropogenic feedbacks in the climate system. mate system. National, regional, and local impacts could be modeled in an exascale New, multidisciplinary teams of physical, Earth system model. biological, and social scientists could accelerate progress on these challenges with trans- many factors, but one of the most important formational levels of computing. is the large range of projections for tropical precipitation. This illustrates that better un- The carbon cycle has been characterized derstanding of biogeochemical cycles is, to a by using observations from ships, land sur- large degree, contingent on better understand- face sites, and aircraft. The amounts of CO2 ing and simulation of the physical climate. in and exchanges of CO2 among the atmo- Systematic error reduction of the physical sphere, ocean, and land have been estimated climate system needs to progress concurrent to the first order. Representations of the car- with the advancement of ESMs with biogeo- bon cycle have been introduced into a first chemical processes and ultimately socioeco- generation of ESMs. In contrast to earlier nomic/energy and emissions feedbacks. In atmosphere-ocean general circulation models addition, the simulated carbon cycle tends to (AOGCMs), ESMs can simulate the coupled amplify global warming, although this ampli- physical, chemical, and biogeochemical state fication is also quite uncertain. The feedback of the Earth system. Modern AOGCMs oper- is caused by changes in the terrestrial carbon ate on terascale systems, realistically repro- cycle that are difficult to test empirically with duce the historical record of global warming, our limited observational network and limited and consistently attribute this warming to process models. The feedbacks could become human-induced changes in atmospheric important and could therefore confound ef- chemistry. forts to mitigate climate change in the latter part of the 21st century. One method of assessing our state of understanding is to compare the process- and sys- Human society has been measurably perturb- tem-level simulations from multiple ESMs for ing the natural carbon cycle since the mid- a single scenario for anthropogenic emissions 18th century. Thanks to comprehensive data in hindcast and forecast modes for integration on the production and use of fossil fuels, we periods ranging from seasons to centuries. can quantify the emission of CO2 from these Exascale computing will Recent studies indicate that the simulated fuels and its disposition throughout the cli- enable scientists to reduce the carbon cycle interacts with climate change mate system. It is unclear, however, whether uncertainty in models of natural to increase, not decrease, the uncertainty in we have socioeconomic models capable of and anthropogenic feedbacks in these forecasts. This uncertainty is caused by hindcasting or predicting emissions of CO2 the climate system.

11 Section 1: Climate

with sufficient accuracy for policy formation. in assimilation systems for chemical and bio- This lack of certainty arises especially be- geochemical observations from in situ and cause these models can be evaluated only by satellite platforms. Also required will be con- using historical data, whereas the economic siderably more advanced models to under- transformations required to mitigate climate stand the error characteristics of the analysis change are without historical precedent. system.

In summary, significant advances in under- 3.2 Process-Level Modeling of standing biogeochemical cycles will follow Biogeochemical Cycles from

• integration of models and observations of Simulation of biogeochemical cycles requires the carbon cycle, detailed understanding of terrestrial and oceanic ecosystems; the exchange of organic and • process-level modeling of biogeochemi- inorganic carbon compounds with other parts cal cycles across space and time scales, of the climate system; and the fluxes of energy, water, and chemical compounds (e.g., • accurate models of the coupled physical nutrients) that affect these ecosystems. The and biogeochemical system, and critical nutrient cycles for ocean and land • robust economic models hindcasting and ecosystems span time scales ranging from a predicting climate-changing pollutants few days (e.g., nitrogen) to over 1000 years (e.g., iron). Modeling over these large time Attaining these new capabilities requires new scales to fully evaluate the couplings between approaches that extend across the traditional biogeochemical cycles and ecology will be a disciplines of geophysics, biology, and ecol- significant computational challenge. The spa- ogy. Major advances are needed in observa- tial heterogeneity in the biosphere is a funda- tional, theoretical, and computational studies mental issue overlying much of this science. of our environment. New models are needed to develop sensible volume/area/mass-averaged and mass-con- 3. Major Challenges serving idealizations that preserve the heterogeneity of the process and still allow for Three major technical challenges face scien- a degree of conceptualization. Other major tists in understanding biogeochemical cycles. challenges are the sophistication of the ecological representations, the effects of high- 3.1 Integration of Models and frequency spatial and temporal variability on Observations of the Carbon Cycle the carbon cycle (e.g., fronts and eddies), and Meteorological and oceanic analyses have be- the behavior of the biogeochemical cycles in come an important tool for studying the mean coastal zones [Doney 2004]. state and variability of the current physical climate. Such analyses are constructed by The ecosystem representations tend to be using a model that is adjusted by incorporat- formulated as paradigms of ecological func- ing observations during its integration. These tions. The field certainly needs more mecha- analyses have proved particularly useful for nistic models of these ecosystems constructed understanding the relationship between ob- at the level of individual organisms. It also servations and the underlying dynamics of needs much more detailed understanding of the climate system. It would be especially the nutrient networks and how these networks valuable to have a comparable analysis of affect the carbon cycle. The effects of sharp biogeochemical cycles that could relate local gradients or rapid changes in the physical and global biogeochemical processes. environment of the components of the car- Coupling of biogeochemical bon cycle are not well understood. With the cycles with ocean and land No extant analyses encompass the physical, advent of ultrahigh-resolution ESMs over ecosystems requires simulation chemical, and biogeochemical processes in the next decade, scientists should be able to over time sales from a few days the climate system. Development of these probe the effects of rapid variability on scales to thousands of years. analyses will require significant investment much smaller than the mesoscale.

12 Modeling and Simulation at the Exascale for Energy and the Environment

Moreover, the biogeochemistry in coastal At a minimum, enhanced computing ca- Global models of the Earth zones has not been adequately studied. These pability should make it possible to replace system are irreplaceable tools regions have been challenging to simulate in parameterizations selectively with computa- for studying past, present, and global models with insufficient resolution to tionally intensive representations at the limit future climate. resolve the coastal regions, the discharges of of our present theoretical understanding. For river sediments into the regions, and other re- example, with exascale computing it may lated features. be possible to replace conventional parameterizations of the carbon cycle (over limited 3.3 Accurate Models of domains) with mechanistic models that repre- the Coupled Physical and sent individual organisms. It should also be- Biogeochemical System come possible to replace conventional cloud parameterizations with models of cloud for- Global models of the Earth system are irre- mation based on the fundamental physics of placeable tools for studying the past, pres- condensation. ent, and future climate. The accuracy of these models can, for some processes, be Ocean models will make better use of the determined through comparisons with fun- placement of grid points through unstructured damental theory, with observations, or with and adaptive mesh technologies that allow benchmark computational models. For many for eddy-resolving simulations with dynamic processes—for example, the formation and coasts, sea-level rise, detailed boundary cur- evolution of clouds and convection—no rents, and refinement of critical areas of the practical fundamental theory exists. These bathymetry such as sills and overflows. De- processes are represented in AOGCMs and tails of tidal mixing—as tides move over ice ESMs by using simplified statistical repre- melts and enhance melting—will couple with sentations, or parameterizations. There is also sea and land ice sheet models for accurate pre- no mathematical theory for the derivation of diction of sea-level rise. At the petascale, we parameterizations from either observations or will simulate for centuries; at the exascale, the benchmark computational models. As a re- millennial time scales of the deep circulation sult, the parameterizations in ESMs represent will be simulated. A seamless suite of climate a primary source of uncertainty, both in the prediction capability would be a potential aim reliability of the models as predictive tools of the ESM in the future. and in the fidelity of models to the actual processes in nature. This uncertainty is mani- 4. Feasible Objectives over fest in the uncertainties regarding the sign of the Next Decade cloud feedbacks on climate change, the sign of convective feedbacks on water vapor, and Despite the significant challenges outlined so forth. above, we are confident that significant progress can be made in biogeochemical simula- While it is relatively easy to evaluate the tion within the next 10 years. simulations produced by using parameterizations from observations and benchmark cal- 4.1 Integrated Models and culations, it has proved extremely difficult to Measurements of Biogeochemical determine how to improve the parameteriza- Cycles tions based on these evaluations. It has also proved difficult to quantify accuracy—in a Integration of models and observations of basic sense, it is not clear what level of ac- the Earth system appears feasible in the next curacy is attainable. It is well known that 5–10 years. The integration should include weather cannot be predicted accurately be- new measurements of the carbon cycle from yond roughly one week because of the fun- planned deployments of automated ocean- damental sensitivity of fluid evolution to the sondes and aerosondes and from new satel- initial conditions of the fluid. However, there lites such as the Orbiting Carbon Observatory is no analogous theory for seasonal, interan- and Earthcare. These new observational data nual, decadal, or centennial prediction. streams will give total column CO2 measure-

13 Section 1: Climate

A hierarchy of models is needed to represent the diversity and ments. They will enable studies critical for the model is doing the right thing for the right heterogeneity of ecological detection and attribution of changes in the reasons. processes in agricultural carbon cycle, such as the characterization of systems. the natural variability in the coupled carbon 4.2 Development of Next- cycle, the response of biogeochemical sourc- Generation Ecological Models es and sinks to natural variability in physical climate, and the ways in which natural dis- Ecological models representing the diversity turbances such as fires and volcanoes perturb of plant life are under development [Stich biogeochemical cycles. 2003]. Most of these models are highly parametric, are primarily based on individual data The coupling of ocean pH with atmospheric sets, and tend to be site specific. The recent

CO2 will allow a closer examination of the generation of dynamic vegetation models has ability of sea life to adapt to and tolerate cli- started taking a more holistic approach to rep- mate change. The complexity of the ocean resenting this diversity and heterogeneity by ecosystem suggests that the carbon cycle is adopting macroscale aggregation based on only the first step in coupling the terrestrial plant functional types. However, agricultural biosphere with climate. For example, iso- ecosystems either are not present or find lim- prene emissions from Pacific Ocean algae ited roles in these models. Since agricultural appear to have an effect on cloud forma- ecosystems are among the largest terrestrial tion. Models of secondary organic aerosols ecosystems, better representation of these (SOAs), when compared with the best field systems in terms of individual crops and cli- measurements, underestimate SOAs by a mate zones is needed. factor of 10. The treatment of cloud aerosol interactions, and particulates in general, will One possible solution is to build a hierarchy be important for predicting radiation changes of models that can represent the diversity and as well as nutrient cycles for land and ocean heterogeneity of the ecological processes ecosystems. These treatments require devel- found in agricultural systems. Complexity opment of much better microphysical models could range from an agricultural crop mon- of multicomponent aerosols for the full mul- oculture to a diverse native prairie, and from tidecade range of particle sizes observed in these models one could develop reduced- the atmosphere. Physically based and com- form models with higher levels of abstrac- putationally demanding models based on the tion. These reduced-form models could be aerosol general dynamic equation should be functionally similar to the current genera- reconsidered. A comprehensive methodol- tion of dynamic vegetation models (DVMs) ogy for generating SOAs from the poten- with capability to both affect and respond to tially numerous organic compounds in the the dynamics of the more detailed models. atmosphere from anthropogenic and biogenic Individual-based or agent-based modeling emissions should be developed. This method approaches could be targeted for develop- will help in delineating the differences being these detailed models. Development of tween the various degrees of biomass in a mechanistic process-based models would be burning plume, a cause for much uncertainty needed for below-ground soil and microbe for calculating radiative forcing in the current processes, in addition to the physiology of and models. Cloud-resolving models at very high competition among plant functional groups. spatial resolution will need cloud condensa- Approaches such as genomic typing that are tion nuclei and droplet activation models that under consideration for representing micro- go beyond the current parametric representa- bial life would be evaluated and targeted for tion and that can account for multicomponent further development. If implemented, such a aerosols with surfactants and with inert and modeling approach would enhance our ability hydrophilic particle nuclei. The impacts of to plan for mitigation strategies such as car- biomass burning on air quality give urgency bon sequestration that involve the biosphere to addressing the scientific challenge of un- and land processes. derstanding these processes and ensuring that

14 Modeling and Simulation at the Exascale for Energy and the Environment 4.3 Better Theory for and mitigation of climate change requires ESMs Quantification of Uncertainty that have been designed from the outset to couple to models for ecology, biology, society, Formulating a firm theoretical foundation for and the economy. The design and exploitation uncertainty quantification will require major of these models would be greatly enhanced new approaches to error attribution and new by direct collaborations between the climate developments in the mathematical theory for community and ecologists, biologists, social complex model systems. The motivation for scientists, and experts in public policy. realizing such a foundation follows from the challenge to develop demonstrably more ac- The design of mitigation strategies that adapt curate models. If we understand the sources to the changing climate and our understanding of uncertainty, we may be able to make mod- of those changes requires new combinations els so good in particular areas that we are at of econometrics and game theory. The cli- the limits of what we can learn from observa- mate community should collaborate directly tion. Conversely, where models are uncertain, with mathematical economists to incorporate we may be able to suggest observations or ex- and study the behavior of interactive mitiga- periments that would significantly add to our tion modules in ESMs. knowledge of the climate system. Topics to be addressed by the development The propagation of uncertainty through a teams include the following: coupled model is particularly challenging be- • High-resolution ESMs with massive as- cause nonlinear and non-normal interactions similation of satellite and other data can amplify the forced response of a system. New systematic theories about multiscale, • Hierarchical unit testable models with re- multiphysics couplings are needed to better quirements for accuracy in the ESMs quantify relationships. Such theories will be important as ESM results are used to couple • Detailed modeling of controlled and with economic and impact models. The sci- modified ecosystems to fit the environ- ence of the coupling and the quantification mental envelope in which future climate of uncertainties through coupled systems are changes will occur necessary groundwork to support complex • Greater scalability and identification of decisions that will be made over the next few greater degrees of parallelism decades. • Process-scale mechanistic models for 5. Accelerating Development biogeochemical, ecological, and aerosol Advances in our understanding require im- processes proved observations, theory, and computa- 5.2 Applied Mathematics and tionally based models of the climate system. Computer Science Collaborations Here we focus on three areas that will accelerate such development: model development The climate community needs to force much teams, close interaction with applied math- closer collaboration with applied mathema- ematicians, and high-end simulation. In each ticians to address the complexity of climate case the emphasis is on collaboration with models. Such collaborations could be useful ecologists and biologists, social scientists and in theoretical studies of climate models as dy- economists, and applied mathematicians and namical systems, new approaches to quantify systems experts. and reduce uncertainty, new methods to synthesize models and data, and techniques to 5.1 Focused Model Development parameterize very complicated processes. Formulating a firm theoretical Teams with Dedicated Resources foundation for uncertainty Two cross-cutting developments are critically quantification requires At present, climate modelers develop ESMs needed: (1) new applications of new algo- major new approaches to error attribution and in the in a mode suitable for large scientific enter- rithms in the physical climate model and (2) prises. However, assessing the impacts and mathematical theory for new software architectures and rapid devel- complex model system.

15 Section 1: Climate

opment environments to facilitate code refor- • Rational design and analysis of computer mulation and refactoring. experiments to navigate very large parameter space with very large outputs 5.3 High-End Simulation Capability • Advances in analysis tools with paral- lelized capabilities, and the ability to Individual ESMs in the next IPCC assess- explore the full climate solution space ment will produce on the order of a petabyte using climate experiments based on data of output. Data volume of this magnitude is mining, objective and repeatable metrics already taxing traditional (and usually serial) (e.g., Taylor diagrams), and expert pat- analysis techniques and database systems. tern recognition and learning capabilities The new class of ESMs for the environment and society could produce truly prodigious • Increased computational capacity and ca- amounts of model data. Extraction of infor- pability with dedicated cycles for large mation critical for impact studies (e.g., sys- climate change studies tematic shifts in precipitation extremes and natural modes of variability) will require new Given the urgency of finding answers to key approaches in data archiving, data mining, questions, and the added complexity of the and feature extraction. Figure 1.5 shows the modeling enterprise in the exascale environ- balance of modeling investments that result ment, the staffing and training of the next from the availability of terascale, petascale, generation of Earth and computational scien- and exascale computers. Factors such as tists are limiting factors. Exascale machines model complexity are traded for resolution, will require a new level of engagement from given limited computational resources. the algorithmic point of view. In Table 1.3, the shift in algorithmic focus is shown. Mul- Specific needs as we move toward the exas- tiscale problems are forcing us to consider cale include the following: many new algorithmic approaches. Com- puter hardware components and the underlying operating systems will also be subject to unprecedented demands. The climate community will have to form even closer collaborations and alliances with hardware vendors and systems developers to address these major issues. 6. Expected Outcomes

The most important outcome of accelerated development and understanding will be quicker answers to the key questions that the climate science community is being asked. At the same time, we will be building momentum for stronger leadership and depth in climate research and creating the ability to produce reliable climate forecasts.

These activities should enable the development of entirely new methods for the attribution of errors in environmental simulation and understanding. These methods will be based on a hierarchy of ESMs running at var- Figure 1.5 Investment of exascale and petascale computational resources in sever- ious ultrahigh resolutions combined with a al aspects of a simulation: spatial resolution, simulation complexity, ensemble size, hierarchy of process-level models of varying etc. Each red pentagon represents a balanced investment at a compute scale. complexity. It should become feasible to sep-

16 Modeling and Simulation at the Exascale for Energy and the Environment

Code Structured Unstructured FFT Dense Sparse Particles Monte Data Agents grids grids linear linear Carlo assimilation algebra algebra CAM X X X X X X POP X X X X CLM X X X X CICE X X X X X

Table 1.1 The “seven dwarfs” extended for atmosphere, ocean, land, and sea ice models. New developments are highlighted. arate the uncertainties in the model into terms using massive ensembles of ESMs with per- associated with the frontiers of understand- turbed physics coupled to models for ecology, ing at the process and observational level. At biology, and society. The expected outcome present, the errors are frequently caused by will be twofold: the relatively simple reductions of these processes and observations incorporated in cur- • Detailed and comprehensive informa- rent AOGCMs. The accelerated development tion regarding the probabilistic risks of should make it feasible to separate model er- climate change for the environment and ror into three categories: society

• Asymptotic process uncertainties — er- • Better engagement of stakeholder com- rors remaining in the limit of the greatest munities to define information interfaces process fidelity (e.g., incorporation of full- that inform decision processes complexity cloud models) based on fundamental theory that can be constrained 7. Major Risks by observations. These uncertainties are also caused by the interactions of errors The major risks to this enterprise are similar among process representations. to those that have historically impeded better understanding of climate and biogeochemical • Asymptotic scale uncertainties — errors cycles. in the mean state and uncertainties in its high-order statistics (e.g., extremes) re- Lack of sufficient data to constrain key cli- maining in the limit of highest possible mate processes. One major risk is the lack spatial and temporal resolution. These are of sufficient observational data to develop, due to couplings between the processes, test, and evaluate new process models. For state, and dynamics out of the reach of example, there are essentially no routine ob- modern observational systems. servations of the vertical velocity of the atmosphere, and the observations that are available • Asymptotic state uncertainties — errors are collected at isolated surface sites. The ab- in the constituents of the system (the sence of data on vertical velocities has made mixture of condensed and gaseous spe- it much harder to understand and model the cies) remaining in the limit of the most interactions of aerosols and clouds, the dy- detailed possible constituent treatments. namics of the boundary layer, and the vertical Representative treatments include master mixing of chemical compounds. chemical mechanisms and aerosol modules that track huge ensembles of indi- Slow reduction of model uncertainty due to vidual aerosol particles. highly intractable or more complicated climate processes. It may prove quite difficult It should also become feasible to attribute un- to reduce the uncertainties in hindcasting, certainties in studies of impacts, adaptation, forecasting, or present-day simulation of the and mitigation to uncertainties in observa- coupled climate system. For example, the tion, theory, and computation. For example, range of equilibrium climate sensitivity has this type of attribution will be facilitated by remained essentially unchanged over the past

17 Section 1: Climate

Exascale simulation offers the 30 years of model development. While the References promise of reducing the range transformation of computing over this time of equilibrium climate sensitivity, R. Alley, T. Berntsen, N. L. Bindoff, Z. Chen, A. has enabled the development of much more Chidthaisong, P. Friedlingstein, J. Gregory, G. which has remained essentially realistic climate models, estimates of sensitiv- unchanged for 30 years. Hegerl, M. Heimann, B. Hewitson et al. (2007), ity from ensembles of AOGCMs have still not Climate change 2007: The physical science basis, converged. Although this lack of convergence IPCC, Working Group 1 for the Fourth Assess is usually attributed to the rapid increase in ment, WMO. process complexity, it is caused primarily CNA Corporation (2007), National security and the by uncertainties in basic processes that have threat of climate change. National Research Coun- been studied intensively for decades, includ- cil (2001), Improving the effectiveness of U.S. cli- ing cloud evolution and convection. mate modeling, National Academies Press.

Absence of adequate diagnostic frameworks S. Doney, ed. (2004), Ocean carbon and climate connecting forcing, response, and initial change: An implementation strategy for U.S. conditions. The climate community has not ocean carbon research, U.S. Carbon Cycle Sci developed methods to link errors in climate ence Scientific Steering Group. simulation to errors in process representation, Ian Foster, ed. (2007), Exascale global socioeco- forcing, or initial conditions. The major dif- nomic modeling enabling comprehensive climate ficulty is in attributing error in highly non- change impact and response analysis, DOE. linear systems with huge numbers of degrees of freedom (e.g., AOGCMs). Conversely, the L. Ma, M. J. Shaffer and L. R. Ahuja (2001), Ap absence of a basic theory of error attribution plication of RZWQM for soil nitrogen manage complicates efforts to understand the connec- ment, pp. 265–301 in M. J. Shaffer, L. Ma and S. tions between process-level realism and the Hansen, eds., Modeling Carbon and Nitrogen Dy namics for Soil Management, Lewis Publ., Boca fidelity of the entire model system. This risk Raton, Florida. is also related to the lack of proper diagnostic tools suitable for analysis of complex ESMs. S. Stich, B. Smith, C. I. Prentice, A. Arneth, A. Bondeau, W. Cramer, J. Kaplan, S. Levis, W. Overly complicated models. The rapid de- Lucht, M. Sykes, K. Thonicke and S. Venevsky velopment of ESMs could produce models (2003), Evaluation of ecosystem dynamics, plant that are too complicated or too expensive for geography, and terrestrial carbon cycling in the adoption by the academic, impacts, or mitiga- LPJ dynamic global vegetation model, Glob. tion communities. Change Biol. 9, 161–185.

18 A Energy: 2 Combustion

Combustion currently provides 85% of U.S. unburned fuel, and large-amplitude oscilla- energy needs. Furthermore, because of large tions in pressure that can result in poor com- infrastructure costs, combustion will continue bustion efficiency, toxic emissions, or even to be the predominant source of energy for the mechanical damage to turbo machinery. A near and middle term. However, environmen- fundamental understanding of the dynamics tal, economic, energy security, and American of premixed flame propagation and structure competetiveness concerns, coupled with the for a variety of different fuels is needed to specter of a diminishing supply of oil, are advance combustion modeling capability and driving a shift toward alternative fuel sources. thereby achieve the engineering design goals New combustion systems are needed to utilize for new power plants. these fuels with high efficiency while meeting stringent requirements on emissions. For Transportation is the second largest consum- example, new power plant concepts based on er of energy in the United States, responsible clean coal technologies, such as FutureGen for 60% of our nation’s use of petroleum, an [DOE Office of Fossil Energy 2004], require amount equivalent to all of the oil imported novel combustion systems that can burn ei- into the United States. Virtually all transpor- ther hydrogen or syngas. For transportation, tation energy today comes from petroleum. new engine concepts will be needed to reduce The nature of transportation technologies pro- emissions while simultaneously shifting to vides opportunities for significant (25–50%) operate with alternative fuel sources. These improvements in efficiency through strate- new sources—whether oil shale, oil sands, gic technical investments in both advanced biodiesel, or ethanol—all have physical and fuels and new low-temperature engine con- chemical properties that vary significantly cepts [DOE Office of Basic Energy Sciences from traditional fuels. 2006]. Such enhanced efficiency will aid in energy conservation and minimize environ- 1. State of the Art mental impact. Methods involving low-temperature compression ignition (LTC) engines, Land-based stationary gas turbines constitute such as homogeneous charge compression a significant portion of the power generation ignition, offer diesel-like efficiency with the industry. As part of the overall system design environmental acceptance of current gaso- for next-generation power plants, there is line-fueled cars. These engines operate under considerable interest in developing clean and high-pressure, low-temperature, dilute, and efficient burners for turbines that can- oper fuel-lean, oxygen-rich conditions compared ate with a variety of fuels such as hydrogen, to current designs. These new concepts rely syngas, and ethanol. Concepts based on lean on subtle control mechanisms that require a premixed burners have the potential to meet fundamental understanding of combustion these requirements because of their high ther- science in these relatively uncharted regimes To achieve the design mal efficiency and low emissions of NOx due of combustion for their optimal implemen- goals associated with lean to lower post-flame gas temperatures. How- tation. Although LTC-based designs have combustion, researchers need ever, combustion in this regime occurs near shown promise in reducing energy consump- a fundamental understanding of the lean flammability limit, making the flame tion, pollutant emissions, and greenhouse the dynamics of premixed flame susceptible to local extinction, emissions of gas emissions, the combination of unex- propagation.

19 Section 2A: Energy: Combustion

nal combustion engines and power plants will operate in nonconventional, mixed-mode, turbulent combustion under previously unexplored aero-thermo-chemical regimes. Com- pared to current devices, combustion in these next-generation devices is likely to be characterized by higher pressures, lower temperatures, higher levels of dilution, and excess air (fuel-lean). In this environment, near-limit combustion sensitivities are amplified—for example, ignition, flammability, and extinction (see Figure 2A.1). These near-limit flame characteristics not only govern efficiency, combustion stability, and emissions but also determine the very existence of combustion in many situations.

Combustion processes in these environments, combined with new physical and chemical Figure 2A.1 Panel (a) Experimental measurement of the hydroxyl radical OH using fuel properties associated with non-petro- planar laser-induced fluoresence and showing local extinction of a premixed hydrogen flame at ultralean conditions. Recent advances in simulation have made it pos- leum-based fuels, result in complex interac- sible to capture this phenomenon in idealized simulations. Panel (b) Slice through a tions that are unknown even at a fundamental three-dimensional simulation that captures the local extinction phenomena. level. These unknown parameters place new Panel (c) View of the flame surface in the simulation. demands on simulation and severely restrict our ability to predict the behavior of these plored thermodynamic environments and new systems from first principles and our ability to chemical fuel properties results in complex optimize them. There is an urgent demand for interactions among multiphase fluid mechan- high-fidelity simulation approaches that cap- ics, thermodynamic properties, and chemical ture these aero-thermo-chemical interactions kinetics—the so-called aero-thermo-chemical and, in particular, capture and distinguish interactions—that are not understood even at a the effects of variations in fuel composition. fundamental level. Future combustion technologies will require an unprecedented level of fundamental un- These new design concepts for both power derstanding to develop a new generation of generation and transportation will operate in predictive models that can accurately repre- combustion regimes that are not well under- sent the controlling combustion processes for stood. Effective design of these systems will evolving fuel sources. require new computational tools that provide unprecedented levels of chemical and fluid To achieve these goals, we need a deeper sci- dynamical fidelity. Current engineering prac- entific understanding of the combustion pro- tice is based on relatively simple models for cesses and advanced modeling technologies turbulence combined with phenomenological to encapsulate that understanding in engi- models for the interaction of flames with the neering design codes. Theory and experiment underlying turbulent flow. Design computa- alone cannot address these issues. Theory tions are often restricted to axisymmetric cannot provide detailed flame structure or flows or relatively coarse three-dimensional the progression of ignition in complex fuels, (3D) models with low-fidelity approxima- while experimental diagnostics provide only tions to the chemical kinetics. Although these a limited picture of flame dynamics and igni- approaches have proven extremely effective tion limits. For example, advanced nonintru- for traditional combustor design, a dramatic sive laser diagnostics are extreme constraints improvement in fidelity will be required to at high pressure because of constraints on model the next generation of combustion de- optical access and inherent limitations in the vices. Next-generation, alternative-fuel inter- spectroscopy. Numerical simulation, working

20 Modeling and Simulation at the Exascale for Energy and the Environment in concert with theory and experiment, has the of the underlying physical processes in engi- potential to address the interplay of fluid me- neering design calculations. chanics, chemistry, and heat transfer needed to address key combustion design issues. Both types of simulations must be time- dependent and include accurate representa- Recent developments in numerical method- tions of underlying physical processes such ology for combustion simulations in combi- as chemical kinetics and transport. Our cur- nation with new high-performance parallel rent ability to perform these types of simula- computing architectures have enabled dra- tions relies on both high-performance parallel matic improvements in our ability to simulate machines and new algorithmic technologies. reacting flow phenomena. We are now able to New approaches based on high-resolution simulate realistic laboratory-scale gas-phase discretization approaches, adaptive mesh re- turbulent flames with high-fidelity models for finement, and multiscale formulations have chemistry and transport. This type of simula- led to significant improvements in the types tion, performed without incorporating explicit of problems that can be simulated. Over the turbulence modeling assumptions, is referred next decade, we anticipate that continued to as direct numerical simulation (DNS); see improvements in algorithm technology will Figures 2A.2–2A.3. DNS tools are currently enable scientists to model new classes of being extended to treat multiphase flows and problems with increased physical and geo- radiative heat transfer. metric complexity.

Scientists are also developing a new generation of engineering combustion design codes based on the concept of large eddy simulation (LES); see Figure 2A.4. This approach provides a more accurate model for turbulent flow than previous approaches and makes it possible to include detailed chemical kinetics models in engineering simulations. 2. Advances in the Next Decade High-fidelity simulations of combustion phenomena based on DNS and LES approaches require high-resolution simulation of turbulent, reacting flows in three dimensions. Such simulations have benefited enormously from sustained growth in high-performance computing. DNS simulations are one of the key tools needed to study fundamental observations of the fine-scale turbulence-chemistry interactions in combustion; however, they are currently limited by computer power to moderate turbulence intensities and to relatively simple laboratory configurations. LES approaches provide a direct treatment of the large-scale flow dynamics, but physical modeling of the unresolved subgrid scales is re- Figure 2A.2 Instantaneous image of the hydroperoxy radical (HO2), a good marker quired. LES can be applied to both laboratory for ignition, in a lifted turbulent H2/O2 jet flame at Re = 11,000 from a DNS simula- and practical configurations and has the po- tion. The simulation had 1 billion grids and transported 14 variables requiring 2.5 tential to include high-fidelity representations million CPU hours on the Cray XT3 at ORNL. The stabilization mechanism of this lifted flame is due to autoignition upstream of the high-temperature flame base.

21 Section 2A: Energy: Combustion

The other key factor that will influence com- 3. Major Challenges bustion over the next decade is the continued development of new experimental diagnostic The next generation of combustion devices procedures. New laser diagnostic procedures will need to operate at high efficiencies and are making it possible to probe turbulent low emissions with fuels such as hydrogen, flames experimentally in ways that elucidate syngas, or biofuels. These devices will need the turbulent flame structure in much greater to operate in new combustion regimes that detail than has previously been possible. For are fundamentally different from current en- example, scientists can now measure time- gineering practice. Successful development resolved velocity fields and flame locations of these new combustors will require new to capture the interaction of the flame with predictive simulation capabilities that can turbulence. However, new quantitative diag- accurately model combustion in these new nostic methods are needed at all scales, from regimes. Advances in combustion simulation individual reactive encounters, to controlled face a number of technological barriers. molecular ensembles, to in situ combustion One important scientific challenge is to de- chambers. Especially important will be the velop robust and reliable ignition and com- development of diagnostics applicable at high bustion models adapted to the wide range of pressure, where spectral broadening interferes combustion regimes observed in next-gen- with current optical diagnostic techniques. eration engines and power plants, including In situ techniques are particularly challeng- propagation-controlled combustion, mixing- ing. At high pressures, spatial gradients are controlled combustion, kinetically controlled very steep, and new diagnostics need to be combustion, and combined mixed modes of developed to resolve the spatial structure of combustion. Progress here depends on hav- the reaction fronts. Diffraction limitations ing kinetics and thermodynamic models for may require new methods to capture these realistic fuel compositions at high pressures gradients. Propagating optical beams through and low temperatures. Also needed are new high-pressure turbulent media and boundary strategies for chemical mechanism develop- layers is also extremely challenging. As these ment and reduction that are both accurate and measurement techniques are improved, we computationally tractable in multiscale simu- will have much better characterization of de- lations of combustion. tailed flame and ignition behavior, which will provide the data needed for accurate valida- Although simulation methodologies are avail- tion of new simulation capabilities, particu- able for many of the computational problems larly when operating at pressure. identified, additional development is required to harness the power of new computer architectures for these problems. For example, multiscale formulations that can exploit the specialized structure of typical combustion applications are needed. Another critical area of research is scalable algorithms for multiphysics reacting-flow problems. Particular issues in this area include the development of scalable solver techniques for variable coefficient and nonlinear implicit systems and the development of improved load-balancing strategies for heterogeneous physics. Substantial increases in capability are also needed, requiring development of improved discretization procedures that not only provide better representations of the basic physi- Figure 2A.3 New simulation approaches have made it possible to simulate labo- cal processes but also improve the coupling ratory-scale premixed flames with detailed chemistry and transport. Two examples are (a) a V-flame and (b) a slot Bunsen flame. The flame surfaces are colored by between these processes. curvature to emphasize the wrinkling of the flame surface by turbulence.

22 Modeling and Simulation at the Exascale for Energy and the Environment

Another issue facing combustion science is management of software complexity. The simplest simulations are multiphysics algorithms that incorporate fluid mechanics, chemical kinetics, and transport. More complex problems will also require algorithms that treat particles, radiation, and multiphase processes and interfaces. Adaptive mesh and multiscale methodologies are often required to solve problems with the necessary fidelity. Additional issues are raised by the geometric requirements of realistic combustion devices that typically involve complex and, in the case of engines, moving geometries.

A major issue underlying the development of new combustion simulation methodologies is data management. The core simulation methodology discussed here will generate data at enormous rates. This volume of data poses two challenges. First, the data must be archived and software facilities created that allow simulation datasets to be accessed by the larger combustion community. This situation raises issues involving raw storage, software for data extraction, and data security. Second, for the simulations to have impact on design issues, we need to develop tools for extracting knowledge from simulation data and for encapsulat- Figure 2A.4 New approaches to LES are enabling high-fidelity simulations of real- ing that data in engineering models that can be istic combustion devices such as the high-swirl laboratory-scale annular combustor directly used for design optimization. shown in the left frame. The middle image shows the instantaneous velocity field. The image on the right shows the time-averaged velocity. Combustion science and the supporting simulations rely on a diverse set of chemi- with expertise in different aspects of the com- cal inputs and models, and also produce new bustion problem. data and models. New data informatics approaches are needed to provide for the rapid Software development for combustion must collaborative development and exchange of provide not only the support needed to fa- chemical mechanisms, thermodynamics data, cilitate implementation of the numerical al- validated model descriptions, and annotated gorithms, but also the flexibility to integrate experimental data that will support the com- different physics modules without loss of per- putational studies. formance. Users must be able to incorporate different chemistry and transport packages 4. Accelerating Development as required for different applications. At the same time, the overall implementation frame- Developing predictive simulations tools for work must support the programming models designing new combustion systems involves needed to exploit large numbers of processors the integration of activities across a range of while insulating the application scientist from disciplines. Effectively harnessing the com- the details of a particular architecture. pute power of exascale computers to solve key combustion design issues will require a Applied mathematicians will need to develop collaboration of computer scientists, applied new discretization procedures and associated mathematicians, and combustion scientists solvers. There will be a pacing need to extend

23 Section 2A: Energy: Combustion

methodologies to more complex physical sys- bustion regimes that are not understood at a tems while at the same time developing the fundamental level implies that exascale com- new algorithmic approaches needed to ef- puting will play a deciding role in whether we fectively solve the resulting systems on com- are able to design these types of systems. puters with large numbers of processors. A particularly challenging issue is to develop 6. Required Investment high-fidelity discretization approaches for moving geometries that do not sacrifice com- Developing the computational tools needed to putational efficiency. design next-generation combustion devices is not simply a question of building an exascale Successful application of the software tools simulation capability. Meeting the require- to the design of new combustion systems will ments for designing new systems will neces- also require expertise from the combustion sitate a family of new simulation codes with community in a range of topics. In addition capabilities ranging from fundamental DNS to traditional roles in design and analysis of studies to high-fidelity engineering design computational experiments, software devel- codes. In addition, capitalizing on the simu- opment will require expertise in experiment, lations will require a significant investment theory, and basic chemistry and transport. As in software for managing and analyzing the the simulation tools are developed and vali- simulation data. dated, they will provide drivers for improvements in the fundamental chemistry and 7. Major Risks transport that determine the flame properties and the formation of pollutants. Establishing Several components are key to the develop- these types of linkages will require improved ment of exascale simulation tools for de- approaches for uncertainty quantification that signing the next generation of combustion can be integrated into the validation process. systems. One is the management and analy- As we begin to explore new combustion re- sis of simulation data. Combustion simula- gimes, experimentalists will need to work tions at the exascale will generate data at an with computational scientists to define appro- enormous rate. This data must be archived, priate benchmarks for validating the software. and tools need to be developed to manipulate Similarly, theorists will need to be involved the data and extract fundamental knowledge in fundamental studies to provide the exper- about flame structure that can be used in the tise needed to develop predictive engineering engineering design process. models from more fundamental computational flame studies. Achieving this goal will also require that several facets of combustion science be brought 5. Expected Outcomes together. Experimentalists, computational scientists, and chemists will need to work The development of predictive exascale com- together synergistically to design validation bustion simulation methodology and the as- experiments, assess simulation studies and sociated supporting software infrastructure relate uncertainty in chemical behavior to the will have enormous impact on the develop- overall fidelity of simulations. Theorists will ment of next-generation combustion sys- need to work closely with experimentalist and tems. With these tools it will be possible to computational scientists to develop new mod- optimize the design of lean, premixed turbine eling paradigms that can be incorporated into combustors for stationary power generation. new engineering design tools. This new level We will be able to simulate advanced engine of collaboration between traditionally dispa- concepts across a range of operating condi- rate activities within the combustion commu- tions and predict engine efficiency and emis- nity will be a key component to successfully A family of new simulation codes is needed, with sions. Moreover, we will be able to evaluate harnessing the power of exascale computing capabilities ranging from new biofuels from the perspective of both to solve major problems in combustion. DNS studies to high-fidelity combustion efficiency and pollutants. The engineering design codes. requirement that new systems work in com-

24 Modeling and Simulation at the Exascale for Energy and the Environment References

DOE Office of Basic Energy Sciences 2006. Basic Research Needs for Clean and Efficient Combus tion of 21st Century Transportation Fuels (http:// www.sc.doe.gov/bes/reports/files/CTF_rpt.pdf).

DOE Office of Fossil Energy 2004. FutureGen In- tegrated Hydrogen, Electric Power Production and Carbon Sequestration Research Initiative (http:// www.fossil.energy.gov/programs/powersystems-/ futuregen/futuregen_report_march_04.pdf).

25 Section 2A: Energy: Combustion B Energy: 2 Nuclear Fusion

Nuclear fusion, the power source of the sun worldwide, supported by advanced computa- and other stars, occurs when certain isotopes tion, indicate that ITER is likely to achieve its of the lightest element, hydrogen, combine to design performance. Indeed, temperatures in make helium. At the very high temperatures existing experiments have already exceeded (100 million degrees Centigrade) needed for what is needed for ITER. fusion reactions to occur, the electrons of atoms become separated from the nuclei. The While many of the technologies used in ITER resulting ionized gas is known as a plasma. will be the same as those required in an ac- Often referred to as “the fourth state of matter” tual demonstration power plant, further sci- plasmas comprise over 99% of the visible mat- ence and technology advances are needed to ter in the universe. They are rich in complex, achieve the demonstration power plant goal collective phenomena and are the subject of of 2500 MW of continuous power with a gain major areas of research including plasma as- of 25 in a device of similar size and magnetic trophysics and fusion energy science. field. Accordingly, strong R&D programs are needed to harvest the scientific knowledge The development of fusion as a secure and reli- from ITER and leverage its results. Advanced able energy system that is environmentally and computations in tandem with experiment and economically sustainable is a formidable sci- theory are essential. In particular, acceler- entific and technological challenge facing the ated development of computational tools and world in the 21st century. In addition to being techniques is needed in order to develop pre- an attractive, long-term source of energy, fu- dictive models that can prove superior to ex- sion can have a major impact on climate change trapolations of experimental results. Essential

challenges, since it does not release CO2. to such development is access to leadership- class computing resources—both petascale Progress achieved in fusion energy to date—10 and the projected exascale systems—that al- million watts (MW) of power sustained for low simulations of increasingly complex phe- 1 second with a gain (the ratio of the fusion nomena with greater physics fidelity. power to the external heating power) of order unity—has led to the ITER project, an inter- 1. State of the Art national burning plasma experiment supported by seven partners (including the United Significant recent progress in particle and States) that represent over half of the world’s fluid simulations of fine-scale turbulence and population. ITER is designed to use magnetic in large-scale dynamics of magnetically con- fields to contain a plasma that will produce fined plasmas has been enabled by access to 500 MW of heat from fusion reactions for terascale supercomputers and by innovative over 400 seconds with gain exceeding 10, analytic and computational methods for de- thereby demonstrating the scientific and tech- veloping reduced descriptions of physics phe- nical feasibility of magnetic fusion energy at nomena spanning a huge range in time and a cost of about $10 billion. It is a dramatic space scales. In particular, the plasma science step forward in that the fusion fuel will be community has developed advanced codes Strong R&D programs will be sustained at high temperature by the fusion for which computer runtime and problem size needed to harvest the scientfic reactions themselves. Data from experiments scale well with the number of processors on knowledge from ITER.

27 Section 2B: Energy: Nuclear Fusion

leadership-class computers. For example, as illustrated in Figure 2B.1, the Gyrokinetic Toroidal Code (GTC) [Lin et al. 1998; Lin et al. 2000] scales well on virtually all of the current leadership-class facilities worldwide, using Message Passing Interface (MPI) and Open MPI [Ethier 2007; Oliker et al., 2007; Ethier et al. 2005, Oliker et al. 2005, Oliker et al. 2004]. As indicated, GTC scales to over 32,000 processors on Blue Gene (BG) Wat- son with better than 95% efficiency on the second core and achieves 96% efficiency on over 10,000 dual-core Opteron processors on the Cray XT3.

In common with general PIC approaches, the gyrokinetic (GK) PIC method [Lee 1983; Lee 1987] consists of moving particles along the Figure 2B.1 Scaling of fusion turbulence codes on various supercomputers characteristics of the governing equation— (courtesy of S. Ethier). here the 5D GK equation. The equation increases in complexity because the particles massively parallel machines (MPPs). A good are subjected to forces from an externally example is the effective use of the full power imposed (equilibrium) magnetic field and of multi-teraflops MPPs to produce 3D, gen- from internal electromagnetic fields gener- eral geometry, nonlinear particle simulations ated by the charged particles. A grid is used that have enhanced scientific understanding to map the charge density at each point due of the nature of plasma turbulence in fusion- to the particles in the vicinity. This is called grade high-temperature plasmas confined in the “scatter” phase of the PIC simulation. The a doughnut-shaped (toroidal) configuration. Maxwell equations governing the fields (e.g., These calculations, which typically involve the Poisson equation in electrostatic simula- billions of particles for thousands of timesteps, tions) are then solved for the forces, which are would not have been possible without access then gathered back to the particles’ positions to powerful present-generation MPP platforms during the “gather” phase of the simulation. together with modern diagnostic and visualiza- This information is then used for advancing tion capabilities to help interpret the results. the particles by solving the equations of motion, or “push” phase of the simulation. Realistic modeling of turbulence-driven heat, particles, and momentum losses is of interest The original parallel scheme implemented in for burning plasma laboratory experiments as GTC consists of a 1D domain decomposition well as for astrophysics and space and solar in the toroidal direction and a particle distribu- physics in natural environments [Tang and tion within these domains [Ethier et al. 2005]. Chan, 2005]. Accelerated progress on this Each process is in charge of a domain and a critical issue is especially important for ITER fraction of the number of particles in that do- because the size and cost of a fusion reactor main. Interprocess communications are han- are determined by the balance between such dled with MPI calls. Particles move from one loss processes and the self-heating rates of domain to another while they travel around the actual fusion reactions [Batchelor et al. the torus. Only nearest-neighbor communica- 2007]. Computational modeling and simulation in a circular fashion is used to move the tion are essential in dealing with such chal- particles between domains or processors. lenges because of the huge range of temporal and spatial scales involved. Existing parti- This method scales extremely well to a cle-in-cell (PIC) techniques have demon- large number of processors but eventually is strated excellent scaling on current terascale dominated by communications as more par-

28 Modeling and Simulation at the Exascale for Energy and the Environment ticles move in and out of the shrinking do- leadership-class platforms provides great en- mains at each timestep. This limit is never couragement for being able to use petascale reached, however, because the number of (and eventually exascale) resources to incor- domains is actually determined by the long- porate the highest physics fidelity. In general, wavelength physics that we are studying. A new insights gained from advanced simula- toroidal grid with more than 64 or 128 planes, tions provide great encouragement for being or grid points, introduces waves of shorter able to include increasingly realistic dynam- wavelengths in the system. These waves are ics to enable deeper physics understanding damped by a physical collisionless damping of plasmas in both natural and laboratory process known as Landau damping. Using a environments. higher toroidal resolution leaves the results unchanged; hence, GTC generally uses 64 2. Advances in the Next planes for production simulations. This ap- Decade proach has limitations, however, because the local grid is replicated on each MPI process A computational initiative called the Fusion within a domain, leading to high memory re- Simulation Project, led by DOE’s Office of quirements for simulating large devices. Fusion Energy Sciences with collaborative support from OASCR, is being developed Initial tests on the dual-core BG/L system with the primary objective of producing a were conducted in coprocessor mode, with world-leading predictive integrated plas- one BG/L core used for computation and the ma simulation capability that is important second dedicated to communication [Ethier to ITER and relevant to major current and et al. 2007]. Additional tests were then con- planned toroidal fusion devices. This initia- ducted in virtual mode node, with both cores tive will involve the development over the participating in both computation and com- next decade of advanced software designed munication. Results showed a per-core ef- to use leadership-class computers (at the ficiency of over 96%. These results indicate petascale and beyond) for carrying out un- that indirect addressing of the gather-scatter precedented multiscale physics simulations PIC algorithm is limited more by memory to provide information vital to delivering a latency than by memory bandwidth, as the realistic integrated fusion simulation model- dynamic random access memory (DRAM) ing tool. Modules with much improved phys- bandwidth is shared between the two cores. ics fidelity will enable integrated modeling of fusion plasmas in which the simultaneous Motivated by the strong shift to multicore interactions of multiple physical processes environments extending well beyond the are treated in a self-consistent manner. The quad-core level, researchers are focusing on associated comprehensive modeling capabil- improving current programming frameworks ity will be developed in close consultation for GTC, such as systematically testing a two- with experimental researchers and validated level hybrid programming method. In mak- against experimental data from tokamaks ing full use of the multiple cores on a node, around the world. Since each long-pulse shot scientists are currently constrained to an MPI in ITER is expected to cost over $1 million, process on each core. Since some arrays get this new capability promises to be a most replicated on all these processes, the memory valuable tool for discharge scenario modeling limit will be reached for the larger problem and for the design of control techniques under sizes of interest. If the hybrid programming burning plasma conditions. method proves successful in initial benchmark- ing studies on modest multicore machines in The following are examples of expected ad- Princeton, the plan is to test it on the quad-core vances needed to enable a comprehensive in- Delivery of a realistic, integrated leadership-class systems, such as Blue Gene/P tegrated modeling capability. modeling tool for multiscale at ANL and the Cray XT4 at ORNL. physics simulations will require • Coupling of state-of-the-art codes for the advanced software capable of The excellent scaling of fusion turbulence plasma core and the plasma edge region running on DOE’s leadership- codes such as GTC on the most advanced class computers.

29 Section 2B: Energy: Nuclear Fusion

• Coupling of state-of-the-art codes for be necessary to consider algorithms and magnetohydrodynamics (MHD) and aux- programming techniques to construct paral- iliary heating of the plasma via radio fre- lel programs where the code that runs on an quency (RF) waves individual CPU chip uses the multithreaded, shared-memory programming model and • Development of more realistic reduced uses the message-passing programming models based on results obtained from model (such as MPI or UPC) to communicate DNS-type major codes that use petascale among the CPU chips in a parallel computer. capabilities The associated transition is to multicore pro- • Development of advanced workflow gramming characterized by very low latency management needed for code coupling but limited bandwidth to main memory. In order to achieve the desired accelerated prog- 3. Major Challenges ress, several developments must be made:

In order to achieve accelerated scientific dis- • Compilers to decompose the code run- covery in fusion energy science (as well as ning on a single node into fine-grained many other applications domains), new meth- computation tasks to utilize the collection ods must be develoed to effectively utilize the of cores on a single chip dramatically increased parallel computing power that is expected within the next decade • Highly efficient runtime systems to as the number of cores per chip continues to schedule fine-grained tasks to optimize increase. for available parallelisms and to maximize on-chip cache locality to overcome Examples of outstanding challenges in the fu- off-chip memory latency and bandwidth sion energy science application area include constrains the following. • Hybrid programming frameworks for • Efficient scaling of MHD codes beyond MPI and UPC by incorporating the terascale levels to enable higher-resolu- compilation and runtime systems with tion simulations with associated greater existing MPI and UPC programming physics fidelity environments. • Efficient extension of global PIC codes into fully electromagnetic regimes to cap- 5. Expected Outcomes ture the fine-scale dynamics that not only If proper investments in research efforts are is relevant to transport but also helps to made, specific state-of-the-art codes for fu- verify the physics fidelity of MHD codes sion energy science (such as GTC) can be in the long-mean-free-path regimes ap- transformed to versions using a hybrid pro- propriate for fusion reactors gramming framework and then systemati- • Data management techniques to help de- cally exercised to demonstrate and to test the velop (and debug) advanced integrated effectiveness of this proposed paradigm. If codes such methods for optimally utilizing multicore systems prove effective, then such codes • Innovative data analysis and visualization can realize their high potential for achieving methods to deal with increasingly huge new scientific discoveries with exascale com- amounts of data generated in simulations puting power. at the petascale and beyond Other expected outcomes if dedicated efforts 4. Accelerating Development are properly supported include the following. Future multicore processors As the future multicore processor chips are will require new programming • Clear demonstration of the ability to ef- likely to support coherent shared memory techniques, such as a hybrid fectively integrate (or at least couple) framework incorporating both and parallel computers are likely to support advanced codes to deliver new physics MPI and UPC. only message passing among nodes, it may insights

30 Modeling and Simulation at the Exascale for Energy and the Environment

• Significant progress in the ability to reli- ing resources at the petascale range and be- Hybrid programming ably predict the important edge-localized yond to address ITER burning-plasma issues. frameworks promise enormous mode (ELM) behavior at the plasma Even more powerful exascale platforms will benefits, including greater periphery be needed to meet the future challenges of de- physics fidelity and more reliable, predictive capability. signing a demonstration fusion reactor. With • Significant improvement in the phys- ITER and leadership-class computing being ics fidelity of “full-device modeling” of two of the most prominent missions of the ITER, along with significant progress in DOE Office of Science, full-device integrated achieving reliable predictive capability modeling, which can achieve the highest pos- for ITER sible physics fidelity, is a worthy exascale-relevant project for producing a world-leading, 6. Required Investment realistic predictive capability for fusion. This should prove to be of major benefit to U.S. The new Fusion Simulation Project will re- strategic considerations for energy, ecologi- quire on the order of $25 million/year over the cal sustainability, and global security. course of the next 15 years and more. In addition, research progress enabled by ultrascale References compute power will demand much greater D. A. Batchelor, M. Beck, A. Becoulet, R. V. computer time. For example, a single GTC Budny, C. S. Chang, P. H. Diamond, J. Q. Dong, simulation carried out at present to investigate G. Y. Fu, A. Fukuyama, T. S. Hahm, D. E. Keyes, the long-time evolution of turbulent transport Y. Kishimoto, S. Klasky, L. L. Lao, K. Li, Z. Lin, B. Ludaescher, J. Manickam, N. Nakajima, T. requires around 100,000 cores for 240 hours, Ozeki, N. Podhorszki, W. M. Tang, M. A. Vouk, or 24 million CPU hours. Since the current R. E. Waltz, S. J. Wang, H. R. Wilson, X. Q. Xu, version of the leading plasma edge code XGC M. Yagi and F. Zonca (2007), Simulation of fu sion plasmas: Current status and future direction. requires roughly the same amount of time, ac- Plasma Sci. Technol. 9:312–387. tual coupled simulations of the core and edge regions could demand approximately 50 mil- S. Ethier, W. M. Tang and Z. Lin (2005), Gyro kinetic particle-in-cell simulations of plasma mi lion CPU hours. If additional dynamics (such croturbulence on advanced computing platforms. as the modeling of RF auxiliary heating) are J. Phys. Conf. Series 16, 1–15. included, then computational resources at the S. Ethier, W. M. Tang, R. Walkup and L. Oliker exascale will be essential. (2007), Large scale gyrokinetic particle simula tion of microturbulence in magnetically confined 7. Major Risks fusion plasmas, IBM J. Res. Dev., accepted for publication. Without the dedicated investments described W. W. Lee (1983), Gyrokinetic approach in par in the preceding sections, the leading fusion ticle simulation. Phys. Fluids 26(2) 556–562. codes (especially the MHD codes, with their W. W. Lee (1987), Gyrokinetic particle simulation scaling challenges) run the risk of not being model. J. Comput. Phys. 72, 243–269. able to effectively utilize the large number of Z. Lin, T. S. Hahm, W. W. Lee, W. M. Tang and R. B. White (1998), Turbulent transport reduction processors at the exascale. The development by zonal flows: Massively parallel simulations. of effective mathematical algorithms for inte- Science 281, 1835–1837. gration and coupling is very difficult and may Z. Lin, T. S. Hahm, W. W. Lee, W. M. Tang and be hard to achieve within the next decade. R. B. White (2000), Gyrokinetic simulations in Moreover, if the fusion energy science appli- general geometry and applications to collisional damping of zonal flows. Phys. Plasmas 7(5) cations are able to effectively utilize only a 1857–1862. small fraction of the cores on a CPU, major Leonid Oliker, Andrew Canning, Jonathan Carter, efforts will be needed to develop innovative John Shalf and Stephane Ethier (2004), Scientific methods for per-processor performance. computations on modern parallel vector systems. In Proc. SC2004, Pittsburgh, PA.. 8. Benefits L. Oliker, A. Canning, J. Carter, C. Iancu, M. Likewski, S. Kamil, J. Shalf, H. Shan, E. Stroh maier, S. Ethier, and T. Goodale (2007), Scientific Reliable full-device modeling capabilities in application performance on candidate petaScale fusion energy sciences will demand comput- platforms. In Proc. IPDPS’07, Long Beach, CA.

31 Section 2B: Energy: Nuclear Fusion

L. Oliker, J. Carter, M. Wehner, A. Canning, S. Ethier, B. Govindasamy, A. Mirin, D. Parks, P. Worley, S. Kitawaki, Y. Tsuda (2005), Leading computational methods on scalar and vector HEC platforms. In Proc. SC2005, Seattle, WA. W.M. Tang and V. S. Chan (2005), Advances and challenges in computational plasma science. Plasma Phys. Control. Fusion 47(2) R1–R34.

32 C Energy: 2 Solar

Solar energy, either in the form of photovol- 2. Major Challenges taic or solar chemical fuel generation, can be the ultimate renewable energy solution for In computational materials science and nano- our energy/global warming crisis. Key scien- science, three major challenges exist. tific challenges and research directions that • Developing appropriate numerical ap- will enable efficient and economical use of proximations and models for accurately the abundant solar resource have been identi- calculating the corresponding physics fied (DOE Office of Basic Energy Sciences properties 2005); the need for new computational and modeling tools to meet the challenges of solar • Integrating the diverse models and com- energy research is widely recognized. putational approaches and programs used to calculate different parts and aspects 1. State of the Art of a complex system, hence enabling More than 30 years were needed for the rela- the simulation of the whole system and tively simple thin-film crystal/multicrystal process Si solar cell to reach its current efficiency of • Calculating the large-scale systems (nano- 24%. In order to develop next-generation so- systems containing tens of thousands of lar cells based on new materials and nanosci- atoms) dynamically for a long period of ence fast enough to reduce the global warming time (nanoseconds or microseconds) crisis, a different paradigm of research is essential. Exascale computation can change the The computational physics and chemis- way the research is done—either through a try communities have been addressing the direct numerical material-by-design search or first challenge since the invention of quan- by enabling a better understanding of the fun- tum mechanics. Although the many-body damental processes in nanosystems that are Schrödinger’s equation is well known and critical for solar energy applications. exactly describes almost all phenomena in materials science, the direct accurate solu- Investigations include finding the right mation of that equation is almost impossible. terials for hydrogen storage; identifying the The reason is that a system with N electrons most reliable and efficient catalysts for water must be described by a many-body wave- dissociation in hydrogen production; deter- function in N-dimensional space. That makes mining an inexpensive, environmentally be- the needed numerical coefficients scale asN N. nign, and stable material for efficient solar cell A direct solution of such a problem might application; understanding the photo-electron be possible only with a quantum computer. process in a nanosystem; and hence helping The most common approximation is to de- to design an efficient nanostructure solar cell. scribe the many-body wavefunction and the In all of these areas, the possible exploratory Schrödinger’s equation with an N single- parameter spaces are huge. This situation on particle wavefunction. This is exemplified by Developing next-generation the one hand provides ample opportunity and the currently popular density functional theory solar cells based on new potential for device improvement, but on the (DFT), where a direction calculation scales as materials and nanoscience other hand presents a tremendous challenge N3, instead of NN. requires a new paradigm to find the best material and design. exploiting exascale computing.

33 Section 2C: Energy: Solar

insight into the pathways in water splitting. Understanding these critical phenomena will help scientists tremendously in rationally designing new solar cells and solar chemical cells (see Figure 2c. 1). 3. Advances in the Next Decade DFT can describe many properties accurately, including atomic structures and bind- ing energies. Thus, it is useful in the search for hydrogen storage materials and catalysts. Figure 2c. 1 A core/shell coaxial cable structure can be useful for future solar cell applications by separating the photon-generated electrons from the holes. Here the Although it gives the wrong band gap, with electron (green) and hole (cyan) states are shown in nano-coaxial wires, with (a) some corrections it still can be used to study GaN(core)-GaP(shell) and (b) GaP(core)-GaN(shell). Fast and accurate calcula- optical properties and electron-phonon inter- tions including the electron-hole correlation effects, the carry transports, the carrier actions and hence also for solar cell simula- cooling, and other dynamical effects are critical for solar cell simulations. tions. Other methods, such as the coupled cluster method in quantum chemistry and the The second challenge focuses on software de- GW method in materials science, can provide velopment and integration. It also presents a more accurate calculations for chemical bind- computational hardware requirement more on ing and band structures, respectively. the capacity side than on the capability side. At the least, an integrated but heterogeneous New codes will also be needed, and these and computing platform with different emphases existing codes must be integrated for maxi- on speed, memory size, input/output (I/O), mum usefulness. Specifically envisioned in and communication may be needed, rather the next decade is a new community code that than a single homogeneous system. Most ex- is both highly flexible and modular, enabling perimentally measured physical properties different research groups to contribute differ- (e.g., solar cell efficiency and a nanosystem ent modules. This community code must be synthesis) are results of the combination of more than a collection of codes as in NW- different physical processes. At present, we Chem, and it should go beyond the modula- have different methods and codes to calculate tion, exemplified in the ABINIT project. each individual property and process. But we lack a flexible tool to integrate these proper- Another key factor in advances in the next ties and processes or to easily replace one decade is the availability of exascale com- model or algorithm in one part of a calcula- puters. Computation can potentially provide tion by another model. Clearly needed is the a direct way to reveal the secrets of diverse development of a common framework and a processes involved in nanoscale interactions. common community code. Indeed, computation is key because probing some of these processes experimentally is ex- The third challenge calls for both algorithm tremely difficult. Because of the O(N3) scal- development and exascale computers. Many ing for DFT, even with petascale computers of the critical processes in solar energy appli- one can probably calculate only the electronic cations are poorly understood. For example, structures of a system with 50,000 atoms for a it is not clear how a nanocontact between given atomic configuration. If many timesteps metal and semiconductor works. Also un- are needed to simulate a dynamic process, this known is the mode of hole carrier transport direct approach will become unfeasible. Thus, in a solar cell using organic materials. How linear-scaling (to the size of the system) ap- a photon-generated exciton dissociates itself proaches become necessary. Fortunately, as a into an electron and hole in a nanosystem is result of the near-sighted feature of quantum another critical issue that needs to be better mechanical effects, such linear-scaling algo- understood. Moreover, researchers have little rithms based on domain decomposition are

34 Modeling and Simulation at the Exascale for Energy and the Environment possible and effective for the large systems Earth Simulator) based on the direct TDDFT With linear-scaling algorithms discussed here. formalism. Thus, there remains a gap of about and exascale computing, 105 in time scale and 102 in size scale. Both scientists will be able to algorithm development (both in linear scal- simulate a whole nanostructure 4. Accelerating Development device. ing and in accelerating the dynamics) and With linear-scaling algorithms and exascale exascale computers are needed to close this computing, we should be able to simulate a gap. But the benefit will be tremendous for whole nanostructure device—from photon understanding the photon-electron process absorption and exciton generation, to exciton in solar-cell-related applications. The lack of dissociation and carrier collection in a nano- such understanding is the current bottleneck size solar cell. This simulation can be done in developing more efficient nanostructure by following a time-dependent Schrödinger’s solar cells. equation (e.g., time-dependent DFT, TD- DFT) and at the same time doing Newtonian 6. Required Investment dynamics for the atoms. Although some uncertainties remain about how to do this ex- Traditionally, code developments have not actly (for quantum state collapsing), such a been supported by either OASCR or DOE’s simultaneous dynamical simulation for the Office of Basic Energy Sciences (BES). Rath- electrons and atoms will help to reveal the er, they are often supported within an individ- electron-phonon interaction, the nonradiative ual investigator’s projects as a side product carrier decay, and cooling, thereby helping for studying a specific physical phenomenon. scientists to understand carrier transport and Hence, community code development and collection. Such simulations can also help us integration have been severely limited. Argu- to understand the electrocatalysis of water ably, more recent federal efforts such as the splitting and to figure out the dynamic path- Scientific Discovery through Advanced Com- way of the water splitting process. puting (SciDAC) projects have attempted to address this limitation, but currently these ef- 5. Expected Outcomes forts are either too general on the computer science side or too narrow in specific phys- Carrying out simulations for experimental- ics-oriented projects. Further investment is sized nanosystems is a tremendous computa- needed to ensure that nanoscience and mate- tional challenge. In addition to the problem of rials science research are supported at a level size scale is the long time scale. The typical needed to solve key problems in solar energy carrier dynamics takes tens of nanoseconds, simulation. while the typical time step needed for a TD- DFT is usually in the order of 10-3 femtosec- Reference onds (Fs), which is a thousand times shorter than a time step for atomic molecular dynam- DOE office of Basic Energy Sciences 2005. Basic ics due to the small mass of an electron. Thus, Research Needs for Solar Energy utilization: the number of the timesteps is on the order Report of the Basic Energy Sciences Workshop of 1010. Currently, one can do tens of fem- on Solar Energy Utilization, April 18-21, 2005. tosecond simulations for small systems con- (http://www.sc.doe.gov/bes/reports/files/SEU_ taining about a hundred atoms (e.g., on the rpt.pdf).

35 Section 2C: Energy: Solar D Energy: 2 Nuclear Fission

Commercial nuclear power plants (NPPs) • Establish supply arrangements among generate approximately 22% of the electricity nations to provide reliable fuel services produced in the United States. There is grow- worldwide for generating nuclear energy, ing interest in operating the existing fleet of by providing nuclear fuel and taking back NPPs beyond their original design lifetimes, spent fuel for recycling, without spread- in constructing new NPPs, and in developing ing enrichment and reprocessing technol- and deploying advanced nuclear energy sys- ogies; tems to meet the rising demand for carbon- free energy; this situation presents significant • Develop, demonstrate, and deploy ad- opportunities for the application of petascale vanced, proliferation resistant nuclear and exascale computing. The new GNEP power reactors appropriate for the power (Global Nuclear Energy Partnership) Program grids of developing countries and re- [DOE Office of Nuclear Energy, Science, and gions; Technology, 2006] seeks to bring about wide- • In cooperation with the IAEA (Interna- scale use of nuclear energy while at the same tional Atomic Energy Agency), develop time decreasing the risk of nuclear weapons enhanced nuclear safeguards to effective- proliferation and effectively addressing the ly and efficiently monitor nuclear mate- challenge of nuclear waste disposal. GNEP rials and facilities, to ensure commercial aims to advance the nonproliferation and na- nuclear energy systems are used only for tional security interests of the United States peaceful purposes; by reinforcing its nonproliferation policies and reducing the spread of enrichment and A recent BES report [DOE Office of Basic reprocessing technologies, and eventually Energy Sciences 2006] reviewed the status eliminating excess civilian plutonium stocks and basic science challenges, opportunities, that have accumulated. The stated goals of and research needs for advanced nuclear en- GNEP are: ergy systems, with specific attention to the role of predictive modeling and simulation • Expand nuclear power to help meet grow- (M&S) in addressing the difficulties posed ing energy demand in an environmentally by the radioactive materials and harsh envi- sustainable manner; ronments found in these systems. The con- • Develop, demonstrate, and deploy ad- clusions drawn in this report were similar to vanced technologies for recycling spent those of the town hall meetings: nuclear fuel that do not separate pluto- • Computational M&S offers the opportu- nium, with the goal over time of ceasing nity to accelerate nuclear energy develop- separation of plutonium and eventually ment by simulating complex systems to eliminating excess stocks of civilian plutonium and drawing down existing stocks evaluate options and predict performance, of civilian spent fuel; thus narrowing the technology path and optimizing testing requirements. • Develop, demonstrate, and deploy ad- Nuclear power plants are vanced reactors that consume transuranic • Today’s high-performance computational becoming increasingly attractive elements from recycled spent fuel; systems are capable of modeling complete because of the possibility of offering carbon-free energy.

37 Section 2D: Energy: Nuclear Fission

reactor systems and related technologies; Material Property Determination: Basic the availability of exascale systems will material properties is the area with the most enable high-fidelity M&S that can further potential for progress with the greatest improve the performance of existing reac- return on investment. Properties including tors and have a significant positive impact nuclear (neutron and gamma reactions), on both the design and the operation of fu- thermophysical (e.g. thermal conductivity, ture reactors. phase diagrams), mechanical (e.g. tensile properties, fracture toughness) and chemical Simulation has the potential for addressing (e. g. corrosion rates) have to be determined the critical needs of advanced nuclear energy under static and dynamic conditions. The systems by providing the tools necessary for fuel process selection is an example that safety assessments, design activities, cost, illustrates the possible gain. In the design and risk reduction. One can, for example, of a reactor, fuel definition along with the imagine virtual prototyping of reactor cores choice of coolant, is always the first step that yielding data that leads to more accurate then determines the subsequent components identification of design margins, allows early of the system. The traditional approach experimentation with novel design concepts, requires fabrication of samples or pins of and ultimately significantly reduces plant cer- the new fuel, measurements of physical and tification timelines. In other areas, such as ad- mechanical properties, and finally neutron vanced fuel fabrication, atomistic fuel simu- exposure to high fluence under relevant lations could ultimately make it possible to operating conditions (e.g. temperature, stress target a small subset of promising candidate constraints, interaction with coolant, etc.) with fuel types for further experimentation, great- subsequent characterization. This approach ly reducing the number of experiments to be requires a great expense of money and time performed. A simulation-based methodology (several years). In some cases, the fuel form is within reach with exascale computers. may have become obsolete or irrelevant as a result of programmatic considerations 1. State of the Art by the time the experimental evaluation is complete. A similarly long process is required Modeling and simulation (M&S) has always for the structural materials involved in the been an integral part of nuclear engineering fuel pin cladding and other critical in-core analysis, safety, and design. Computational components. analyses have been used to predict detailed quantities that could not be readily measured Spent Fuel Reprocessing: Reprocessing was in situ, for example, the aging of structures, abandoned in the 1970s as an option within power distributions in cores, transient safety the current nuclear fuel cycle, so there is great behavior, etc. Existing (legacy) tools for mate- opportunity for development of advanced pro- rial property determination, spent fuel repro- cesses. Current reprocessing models provide cessing, fuel performance, reactor safety and only qualitative descriptions of process behav- design, and waste forms and storage based on ior. Empirical models of chemical behavior for a large experimental database will be insuffi- major components are used to provide overall cient. Experimental testing will be extremely descriptions of various reprocessing strategies. expensive, protracted, and in some cases un- These empirical models are based on bench- feasible. Furthermore, the existing experi- top experiments, and usually assume chemical mental database is insufficiently documented equilibrium conditions are met instantly. Even and often has inadequate precision to support then, current models are unable to answer a modern validation process. Complementing many questions involving phase equilibria, or replacing testing with high-fidelity comput- such as precipitation from solution or deter- er simulation will make it possible to collect mining oxidation states, where multiple pos- simulated data that can, in conjunction with sibilities exist. Very few reaction rate constants a sound experimental validation program, be are known, and wherever transient conditions used to understand fundamental processes are simulated, they are usually just assumed that affect facility efficiency, safety, and cost. or selected heuristically.

38 Modeling and Simulation at the Exascale for Energy and the Environment Fuel development and performance evalu- “reactor core codes” have been in existence Fuel performance simulation ation is currently an empirical process that for decades, but need improved physical, tools could reduce by a factor takes decades. For acceptance, new fuels numerical, and geometric fidelity. Codes of 3 the time needed to qualify must be fabricated, placed in a test reactor developed at the time used lumped parameter new fuels. over multiple cycles, tested under multiple models for predictions of neutronics, accident scenarios, undergo post-irradiation thermal hydraulics, and structural mechanics examinations, and finally be placed in an quantities. These simple codes were calibrated operational reactor for several cycles. Fuel against a very large experimental database, performance simulation tools can help to ac- developed over the years for specific projects celerate current fast and thermal reactor fuel by calibrated against principally integral data. qualification practices by helping decrease the Existing reactor core codes, for example, time required to qualify new fuels, with the employ the traditional single-channel or goal being a reduction by a factor of three in sub-channel approach to model reactor core the current 10- to 15-year qualification time. heat transfer. Traditionally, separate thermal The tools must accommodate all relevant fuel hydraulic code systems are used to execute types in both normal operating (quasi steady design and safety calculations. state) and transient conditions. Neutronics modeling has traditionally relied Current state-of-the-art models of nuclear on both stochastic (Monte Carlo) simulations fuel performance empirically capture vari- and deterministic transport and diffusion ous phenomena that occur in a fuel rod dur- theory approaches. Monte Carlo techniques ing nuclear reactor operation. Currently, these incorporate the basic physics at the level of models are mostly limited to axi-symmetric stochastic particle tracking with the general geometries. In this simple axi-symmetric system geometry and material cross sections setting, the empirical models capture power governing the particle track histories. Monte generation, dynamic fuel/cladding gap, ther- Carlo offers the strong conceptual advantage mal analysis solution, simplified neutronics, of keeping a close (essentially exact) fission product generation and propagation in correspondence between computational a fuel rod, localized chemical state, fuel/clad and physical geometric and cross section swelling/creep, fuel cracking, fuel contact and energy dependence models. Nevertheless, mechanical interaction with the clad, and clad Monte Carlo can become computationally interaction with adjacent structures like grid impractical for several different classes of spacers. A simple thermal hydraulics model problems. These include calculations of is generally assumed for flow channel heat small reactivity coefficients, some types of transfer. Because current models are based on sensitivity/uncertainty propagation studies, empirical curve fits to fuel behavior in com- time-dependent solutions, and some types of mon nuclear reactor environments, they can- burn up calculations. not be trusted to predict the behavior of fuels under conditions outside their narrow range of Structural mechanics software development calibration. Transient fuel performance simu- has been driven by a breadth of applications lators are employed to predict the rod thermal/ over the last 30 years which include automo- mechanical behavior under design basis active, aerospace, national defense, civil infra- cidents (DBAs), which for the light water re- structure, and, in the 1970’s and 80’s, nuclear actor (LWR) fuel rod is a reactivity insertion reactor technology. These developments have accident (RIA). Survival of the cladding and led to a number of finite element-based com- maintenance of the core coolability are the puter programs that have relatively mature el- primary issues for these scenarios. The time ement technologies and solution algorithms. frame for these codes is on the order of mil- Existing software that has application rel- liseconds to seconds rather than the days to evance in the nuclear fuel cycle area can gen- months for the quasi-steady-state simulators. erally be divided into three categories: linear finite element programs, implicit time integra- Reactor safety and design simulation tools tion nonlinear finite element programs, and require models for thermal hydraulics, explicit time integration nonlinear programs. neutronics, and structural mechanics. Such

39 Section 2D: Energy: Nuclear Fission

2. Major Challenges proach for integrating the various multiscale components, i.e. when to use parameter pass- Significant exascale simulation challenges ing and when models should be more tightly are present in each component of the fuel coupled. cycle. Opportunities exist in the areas of material property determination, spent fuel Spent Fuel Reprocessing. Modeling of a re- reprocessing, fuel performance, thermal and processing plant involves many complicated fast reactors, and waste forms and storage, steps, each of which requires knowledge of to name a few. These are briefly discussed in several areas of physics, chemistry, or engi- the following. neering. Fuel disassembly involves mechanical processes (chopping, filtering) and/or chemi- Material Property Determination. With mod- cal dissolution in strong acid. The fuel solution ern methods and powerful computing tools, is then passed through many stages of solvent one can foresee the opportunity to employ extraction in order to separate several fission computational simulations to advance the product and actinide streams. Several sepa- evaluation and selection of advanced fuels rate solvent extraction processes are required and structural materials. For example, first to accomplish this separation, each using dif- principles methods can now realistically be ferent additives and components in the organ- used to determine fundamental material prop- ic phase, as well as different acid concentra- erties and support the development of new in- tions in the aqueous phase. Safety consider- teratomic potentials that can be used in Mo- ations require: monitoring of volatile fission lecular Dynamics (MD) simulations involv- product and organic gaseous releases, careful ing millions of atoms. These MD simulations evaluation of component inventories in each can be used to study defect properties and stage and even in piping, to avoid costly shut- to determine parameters such as the atomis- downs, or repairs which must be performed tic reaction rates that are required in coarser remotely, strict attention to nuclear criticality scale simulations of degradation mechanisms safety in actinide solutions with widely vary- that take place over long times. These param- ing component inventories, control systems eters can be employed in reaction rate theory which are based on realistic models of pro- or Kinetic Monte Carlo (KMC) models of cesses, not generalizations or even intelligent microstructural evolution. MD-based dislo- assumptions. The hazards of plant operation cation dynamics simulations can also provide involve radioactive materials, toxic materi- the fundamental dislocation-defect interac- als, strong acids, highly flammable materials, tion parameters required for continuum 3D and highly volatile materials. Thus, detailed Dislocation Dynamics (DD) simulations. accounting of all components through all pro- The 3D DD simulations can be used to ob- cess stages is of utmost importance. tain needed information on the constitutive behavior of the materials which is required To support both detailed design and safe op- for use in macroscopic methods such as finite eration, the improvement of reprocessing element models. Taken together, this family models requires improved chemistry model- of multiscale simulations can provide pre- ing, including both equilibria and kinetics. dictions from the atomistic and microscopic New fuel materials and requirements arising through to the mesoscopic and macroscopic from nonproliferation concerns demand the level. An initial goal should be to establish use of modern sophisticated modeling tools the required degree of accuracy and practical for the design and optimization of a process limitations at each level of simulation (e.g. consisting of several major steps, each of ab initio, atomistic, mesoscale, continuum). which presents its own chemical and physi- This will provide a basis for predicting the cal complexity. These steps include fuel dis- expected impact of an advanced simulation solution in acid, dissolved fuel treatment in program to reduce both the absolute develop- a series of solvent extraction processes, and ment time and the related uncertainties as part fabrication into fuel or waste forms. None of of the overall fuel and materials development these steps is adequately simulated in a pro- effort. The initial technical objectives should duction sense, and some require experimenta- include a strategy for determining the best ap- tion to understand or confirm their chemical

40 Modeling and Simulation at the Exascale for Energy and the Environment behavior. Advanced simulation can be used to or concurrent calculations, but they must be help understand and optimize these process- designed to function within larger systems of es, as well as integrate their behavior into the codes. overall model in areas such as: The simulation challenge is to couple these • Unit operations of chemical separations specific computational models into aco- and materials processes based on first herent package that is considered a unified principles: extraction, evaporation, dis- simulation with a distributed, modular mod- solution, etc. eling structure that allows interactive selection of specific models at levels of detail and • Three-dimensional, transient behavior of breadths of scope required for analysis. The multiphase, multi-species reactive pro- model structure must support three levels of cesses coupled engineering detail:

• Turbulent, multiphase fluid flow and in- • A discrete event model that provides terfacial phenomena throughput analysis, scheduling impacts, • Predicting physical/chemical properties output compositions, and serves as the backbone to call supporting simulations • Development of separating agents/pro- at other levels of detail not captured in cesses with functionalities specified by: discrete event simulations. affinity for target species; viability of synthesis; interaction with other materi- • Chemical process models to represent als; chemical and radiolytic stability; etc. modular, exchangeable simulations of specific unit operations or groups of op- • Basic knowledge for process develop- erations. This allows a balance between ment, including: equilibrium partition- simulation speed and required levels of ing in complex mixtures, transport and rigor. kinetics in non-equilibrium, multiphase systems, and improved process model • Specialized models to address specific solution algorithms. behavior accurately, including phase equilibrium calculations, computational Added value can come from the optimization fluid dynamics, specialized data retrieval, of the fuel cycle as a whole, the possibility of detailed adsorption models, and even de- detecting diversions, criticality problems, or tail as fine as computational chemistry if possible effluent composition deviations out- necessary. side specifications. Fuel Performance. Nuclear fuel assemblies In addition to improved chemistry, additional must perform in aggressive environments elements of reprocessing systems must be characterized by stress, heat, corrosion, and modeled which are currently not even consid- irradiation, all of which lead to progressive ered. Fluid dynamics must be considered in degradation of the mechanical and physical piping as well as in process equipment. Effects properties of fuel cladding materials and other of control systems on component inventories, structural components. and vice versa, are necessary to adequately understand inventories throughout the plant. The process by which nuclear fuel undergoes Interfaces with nuclear criticality calculations change in a nuclear reactor core is inherently are important for both design and safe op- multiscale. Materials issues to be considered eration. Balance-of-plant modeling (such as include thermal and mechanical properties, volatile releases to the environment) must be swelling, microstructural phase changes and included. As reprocessing gains in popular- crack formation, as well as the effects of ity, new and improved processes will almost point defects and fission products. Reliable certainly develop. Therefore, it is essential predictions of fuel behavior are essential to that computer codes be flexible, adaptable, preventing fuel failures and to improving the and modular. They must not only be compat- economics of reactor operations. ible with other codes which perform related

41 Section 2D: Energy: Nuclear Fission

Fuel performance modeling places an • Able to predict the spatial distribution and emphasis on the detailed understanding of the chemical composition of separate metal thermal, mechanical, physical and chemical and oxide phases; processes governing fuel rod behavior during normal reactor operation and under accident • Able to predict fission product conditions. Fuel rod performance codes are used concentrations at the fuel-to-cladding gap extensively in research, by fuel vendors, and by and chemical interactions with the clad; licensing authorities for the prediction of fuel • Able to predict (via the transport models) and cladding performance. The simulation tool the accumulation of volatile species (noble should consist of a clearly defined mechanical- gases, Am, Cs, iodine) in the fuel pin free mathematical framework into which physical volume (especially the plenum); models can be easily incorporated. Besides its flexibility for fuel rod design, the code will be • Can allow for chemical composition utilized for a wide range of different situations, feedback into the fuel physical properties; as given in experiments, under normal, off- normal and reactor accident conditions. The • Can allow for independent updating with time scale of the problems may range from most recent models and experimental milliseconds to years. All important physical results; and models are included in the fuel performance • Can be directly validated using simple code: i.e. models for thermal and irradiation- experiments and PIE results. induced densification of fuel, fuel swelling due to solid and gaseous fission products, fuel creep To meet these requirements, a high-resolution and plasticity, pellet cracking and relocation, 3D spatial representation of the fuel and fission gas release, oxygen and plutonium cladding that allows for non-symmetrical redistribution within the fuel, volume changes power generation and clad/fluid boundary during phase transitions, formation and closure conditions will be needed. In particular, of center void and treatment of axial forces resolving the thermo-mechanical response (between the fuel and cladding), cladding of the fuel and cladding on a sub-pellet level creep and cladding/coolant interactions (such will be paramount. Coupling with other as oxidation), etc. Additionally, the code must neutronics packages will also be necessitated, have access to a comprehensive material as will reaction-diffusion chemistry for fuels, database for oxide, mixed oxide, carbide, cladding, coolant, and plenum gases. Direct nitride, and inert matrix fuels, Zircaloy (and coupling with databases of thermo-mechanical advanced zirconium alloys) and steel claddings and chemical properties and models of (with the capability to add new fuel forms irradiation effects on the properties will also and advanced claddings). Also, interfaces (or be desirable. subcoding) must be available for thermal/ hydraulic and neutronics feedback to the fuel The drivers for exascale platform requirements performance models. in fuel performance can be quantified. For example, a high-resolution simulation of Fuel rod simulators must be able to predict a bundle of 40 fuel rods (~300 million the thermal-mechanical-chemical response elements with a 10-μ scale size), with only of the fuel rod throughout its irradiation life- thermomechanics and no coupling to other time, including fuel rod failure mechanisms. multiphysics models, is estimated to require Requirements for fuel performance simulation about a half day on a 20 PF platform. A more tools include rigorous simulation of a single fuel pellet (~1 billion elements with a 1 μ scale size), again • Must be coupled with the nuclide inventory, with only thermomechanics, would require which is required for the determination of approximately one day on a 1 PF platform. the chemical/phase state of the fuel; Incorporating all relevant additional physics, • Able to predict chemical species evolution such as neutronics, fluid flow, and fundamental and transport (in addition to FP); materials science, will easily multiply these requirements by a factor of 1000, bringing

42 Modeling and Simulation at the Exascale for Energy and the Environment high-fidelity fuel performance simulation assess the efficiency and safety of all nuclear requirements to the exascale. systems and to identify areas where existing experimental data are insufficient. In addition, Reactor Safety and Design. Thermal hydraulic improvements in modeling capabilities can computational tools will be required to perform reduce unnecessarily conservative margins both design and safety analysis, allowing to ensure economical operation. A complete reactor and subsystem designs to incorporate a neutronics capability requires a tight coupling “safe from birth” philosophy. Long term focus of several independent components that have is on the development of a next generation traditionally been developed in a layered ap- thermal hydraulic computational code that proach. The standard components of a com- provides a completely integrated design and plete neutronics suite of simulation codes in- safety analysis architecture, that operates clude several computational tools and drivers, on leadership computing platforms, that along with the ability to propagate uncertain- seamlessly interfaces with the multiple physics ties in the basic nuclear data and biases of each necessary to perform high fidelity reactor and computational tool through the analysis. process analyses (neutronics, structural, fuels, chemistry, etc.), and that is experimentally The key components of a complete neutron- validated to perform licensing analysis. ics capability include: processing of data from Design and safety engineers will form the user evaluated nuclear data files to provide accu- base for these codes, and will greatly benefit rate, problem-dependent cross sections that ac- from faster, more integrated, and higher count for material temperature and resonance resolution thermal hydraulic analysis systems. shielding; radiation transport methods (Monte Expected code capabilities required to support Carlo and deterministic) for steady-state and the design and licensing of the experimental quasi-static time-dependent simulation of test facilities include: transient single and neutrons and photons within a system and the two phase fluid flow analysis, the capability determination of the criticality condition; com- to quantify the effect of both code and input plete tracking of isotopic inventory changes uncertainty levels on design and safety limits, due to fuel depletion, actinide transmutation, the capability to interface with neutronics and fission product buildup and decay, and asso- structural (and/or other as necessary) analysis ciated radiation source terms; and integration codes, the ability to perform multidimensional of neutronics components with other physics thermal hydraulic analysis for selected packages, such as thermal-fluid dynamics, components or component regions, and the fuel performance, chemistry, and structural verification, validation, and documentation mechanics. This comprehensive suite of sim- packages required for licensing. Longer term ulation tools is required to provide analysis requirements include the ability to analyze for the design, construction, and operation of multiple coolants ranging from gases to liquid near-term facilities (short- and intermediate- metals, the ability to use unstructured grids for term objectives) and must consist of the cur- three-dimensional thermal hydraulic analysis rent best-in-class, qualified simulation tools. of any component, additional multi-physics However, advanced optimization techniques, interfaces including fuel analysis, coolant improved solution fidelity, and multi-physics chemistry, etc, the capability of executing coupling will provide substantial impact on the these codes on exascale platforms, and a code viability of commercial-scale facilities. structure that allows the methodology to be physically validated. For structural mechanics, there is a compelling need for development and implementation of Detailed neutronics analyses required to sup- advanced material constitutive models that port the design, licensing, and operation of can accurately represent the time dependent many of the components of the fuel cycle range behavior of materials in extreme environments. from the physics analysis of reactors, critical- These should address the effects of high Difficult analyses such as ity safety of the separations and fuel fabrica- radiation levels, extreme temperatures, and the structural performance tion facilities, and radiation shielding and dose chemical interactions on material behavior. of the system under seismic assessment. Accurate neutronics methods will Advanced materials models should account loads are now possible with be used by facility designers and regulators to for the fully 3D, multi-axial states of stress advanced simulations.

43 Section 2D: Energy: Nuclear Fission

both at low strain rates (normal operations), include the configuration of tunnels within the and at high strain rates (accident scenarios). mountain, the configuration of waste packages The development of macroscopic, continuum- within each tunnel, and the waste loading based phenomenological models must per package. Difficult analyses such as the progress in parallel to fundamental materials structural performance of the system under science research aimed at understanding seismic loads, the interactions of incoming microscopic material behavior in extreme water with the drift walls including capillary environments. Recent developments in solid/ forces, and fracture flow and transport, are all structural mechanics have moved towards a amenable to advanced simulation that analysts merging of capabilities from traditional solids would not have considered even a decade hydro-type codes, which have pushed the ago, because the computer power required to computational technologies for representing execute such analyses was far beyond what extreme deformations and flow of materials, was available. with traditional structural type elements. Such codes have been developed in frameworks 3. Advances in the Next that prevent mesh tangling at extreme ranges Decade of response. This has begun to open up substantially the types of problems that can The stated long-term simulation goal for be modeled for extremely nonlinear accident nuclear energy via a GNEP-based Program scenarios. It would be very desirable to move [DOE Office of Nuclear Energy, Science, and towards a single program that can solve Technology, 2006] is the development of an multiple problems associated with slow (static architectural model that will facilitate the pre- and quasi-dynamic) phenomena associated dictive modeling of the entire fuel cycle from with operations, slow accidental events like mining through final disposition of waste ma- earthquakes, and also accurately simulate terial, taking into account interacting factors extreme accidents associated with very rapid such as market forces, socio-political effects, transients such as pipe breaks. and technology risk. This architecture must incorporate a comprehensive suite of simu- Waste Forms and Storage. Different incoming lation tools for the design, analysis and en- waste streams will likely be sent to a geologic gineering of next-generation nuclear energy repository relative to the waste streams (mostly systems with enhanced safety, reduced en- commercial LWR spent fuel) currently planned. vironmental impact, optimal deployment of This new waste steam, consisting mainly facilities, and reduced construction cost. The of fission-product wastes from recycling, scope of these tools is daunting: will ultimately require that the repository be analyzed for the purposes of regulatory • Integrated 3D reactor core simulations compliance, and perhaps have its design with rigorous propagation of uncertainty updated to take advantage of the significantly reduced loadings (heat loading, mass loading, • Coupled thermal hydraulic and primary and much shorter overall half lives) that the loop simulation new waste stream will represent. If spent fuel in • Advanced fuel design and performance the current incoming waste stream is replaced by reprocessed fission-product waste in glass • Fuel behavior engineering (or another solid waste form), the opportunity arises for a redesign of the repository, where • Advanced secondary loop and balance of multi-faceted simulations can more effectively plant engineering and analysis capture the major processes for a suite of new • Advanced fuel cycle design design concepts. This will allow optimization of the safety (represented by dose to a human • Separations facility engineering optimi- receptor on the surface in the distant future), zation the cost, and the repository volume within Yucca Mountain. For a new design, the • Repository design including seismic, opportunities for improvement are many. They geological, chemical, and thermal modeling and simulation

44 Modeling and Simulation at the Exascale for Energy and the Environment • Overall nuclear energy systems model 4. Accelerating Development development suitable for alternative economic analysis. Modeling and simulation of advanced nuclear fuel cycles will require a hierarchy of mod- More broadly, exascale M&S objectives have els of vastly different physical systems across the potential for a significant impact on nu- a wide range of space-time scales, from de- clear facility design and operations: tailed molecular dynamics of new materials to systems level simulation of the entire cy- • to study fuel cycle and resource issues cle. The final goal will be optimization in the (once-through, plutonium recycle, actinide presence of modeling and input uncertainty in burning, waste disposal, etc.); order to design safe, reliable, economical, and • to develop and optimize nuclear plant and socially acceptable end-to-end solutions for facility designs (systems design, physical nuclear energy production. While there have layout, materials requirements, cost, been many advances in fundamental enabling economics); technologies in mathematics and computer science in the past, additional research and • to demonstrate fundamental safety issues development will undoubtedly be required to (defensive systems, passive safety, tackle a problem of this scale. At each level, establishing the licensing basis); new enabling technologies will be required to enhance predictive capability, understand and • to avoid (reduce) costly prototype propagate uncertainties, model unresolved construction and operations (critical physics, and couple multiple physical models assemblies, prototypes, intermediate-scale at a wide range of space-time scales. Like- plants); wise, new research and development is re- • to accelerate optimum fuel selection (fuel quired to analyze, visualize, and optimize the irradiation is almost certainly required); results of such large simulations, and to do so and in a way that is useful to designers and decision makers, who must be fully aware of the • to characterize SNF-related requirements limitations of the computational predictions (on-site storage and criticality, shipping cask and the uncertainties inherent in the simulat- designs, and repository requirements);. ed results, due to the inevitable uncertainties of input parameters and modeling assump- • optimization of performance in mixed tions. Associated with this is the stringent utility electrical grids (cycle length, need to establish careful protocols for simu- fuel resource requirements, economics, lation code verification and validation. These outages; analysis of hundreds of thousands tools must be uniformly accessible within an of core loading options); integrated computational environment that • demonstration of cycle-specific safety reduces time-to-simulation, provides com- requirements (static and transient patible geometry representations, and allows calculations: tens of thousands of coupled for a hierarchy of model fidelity running on neutronics/thermal-hydraulic/systems workstations to state-of-the-art parallel com- computations); puter architectures.

• support for reactor operators (optimize Multiphysics Coupling. Predictive simulation startup and power maneuvers: thousands of each process within the fuel cycle requires of calculations per cycle); accurate solutions to multiple, simultaneous nonlinear physical processes. Traditional • on-line monitoring functions (real-time simulation approaches to this problem involve surveillance of safety margins: thousands segmented solution techniques whereby the of calculations per cycle); and simultaneous physics are assumed to proceed in a sequential, loosely-coupled manner. Such • operator training (real-time simulation on a solution approach is not nonlinearly consis- full-scope simulators). tent, is prone to numerical errors (particularly

45 Section 2D: Energy: Nuclear Fission

sensitive to time step size), and in some cases Optimization and Confidence Analysis. does not even converge (exhibits zeroth order Sensitivity analysis is used to determine the errors). Such an approach is in general not pre- change in a computed result due to a change in dictive. Fully-coupled, nonlinearly-consistent some input parameter used in the calculation. multi-physics time integration algorithms Sensitivity and uncertainty (S/U) analysis is solve this problem. Multi-physics coupling ef- important not only for performing methods/ forts will also provide common computational data V&V studies, but also for certifying tools and interface structures, through execut- that a proposed system design satisfies all ables and/or modules, to unite the varied tem- performance and safety specifications. This is poral and spatial discretizations of each phys- especially significant because the new reactor ics package to provide a consistent basis for and fuel processing/reprocessing systems analysis and information propagation. These have not been previously characterized by products and interfaces will be used by facil- measurements. Specifically, S/U methods can ity designers to integrate multiple state-of-the- provide the following types of information: art physics simulation packages to provide a generalized coupling. This work will not only • Improved physical insight into the under- pay off in more accurate, predictive simulation lying phenomena governing the system results, but in many cases result in increased of interest, by indicating relationships efficiency (time to solution), by virtue of being between variables. This is often useful for able to take a larger integration time step per guiding design modifications and inter- given level of desired temporal accuracy. preting hypothetical accident scenarios.

The algorithmic and software coupling of mul- • Realistic, best-estimate design margins tiple physics modules and codes will provide that can reduce the tolerances obtained analysis capabilities for the design, construc- from bounding-analysis. tion, and operation of near-term facilities and • Quantitative ranking of important model- must integrate the current best-in-class, quali- ing/data parameters that impact the cal- fied simulation tools. However, utilizing high- culated results. This analysis can identify performance computational systems, high- major sources of uncertainties and can fidelity simulation packages may be tightly- determine the required measurement ac- coupled to provide a fully-integrated facility curacy in the input data necessary to simulation that will provide substantial impact achieve a desired accuracy in the com- on the viability of commercial-scale facilities. puted results. Associated strategies: • Rapid (but approximate) evaluation of • Utilizing and developing advanced com- how design perturbations affect computed putational algorithms for fully-coupled, output parameters. This can be coupled to nonlinearly-consistent multi-physics time a system optimization algorithm. integration techniques; Although developed most extensively • Providing the improved algorithms via for criticality safety and reactor physics standardized software interfaces for the analysis, sensitivity techniques can also communication of information from each be applied to other types of calculations, physics package; including shielding evaluations for reactors, • Developing generalized interpolation, reprocessing, and transportation systems, integration, extrapolation routines to en- fuel depletion studies of actinide burning, sure compatibility of solution with differ- core lifetime, and proliferation indicators, ent geometric representations and time- ex-core fuel cycle parameters such as source scales; term activities/decay heat, and safety analysis involving reactor kinetics with thermal • Integrating tightly-coupled physics pack- hydraulics feedback. ages into a unified module to provide an efficient solution on high-performance computing architectures;

46 Modeling and Simulation at the Exascale for Energy and the Environment 5. Risks simulations using first principles (some codes are adequate, while others need Nuclear energy does not have the same drivers, work); and the integration of multiple from an M&S perspective, as nuclear weap- codes. Exascale computing is expected ons, climate change, or the aircraft industry (to to reduce R&D cost and time, improve name a few). Unless M&S activities are driven and accelerate the design process, support by requirements set by industry and regulatory process scaleup, lower facility costs, and agencies, approaches in which “better” can be- provide opportunities for major change. come the enemy of “good enough” may pre- The biggest payoff for M&S in reprocess- vail and defocus efforts. The oft-quoted state- ing, however, is expected to result from ment that “all models are wrong, some models advances in the modeling of waste forms, are useful; the only way to tell the difference is which should make it possible to delay or to get some data” applies to the role of M&S avoid the construction of additional re- in the nuclear fuel cycle. V&V activities will positories for SNF. be crucial. • The application of exascale computing to 6. Expected Outcomes fuel performance should reduce the time needed for fuel development and quali- Enhanced use of M&S for nuclear energy fication and lead to reliable assessments will lead to improvements in knowledge and of life cycle performance, predictions of reduction of uncertainties that will produce fuel rod behavior in a design basis ac- cost savings in current reactor operations and cident (DBA), and predictions of tran- substantially reduce the cost of future reactors. suranic (TRU) fuel behavior. Such improvements could also provide a basis for innovative designs that will reduce the • Benefits of exascale M&S for reactor need for excessively conservative (and costly) analysis and design include eliminating system specifications, improve efficiency and unrealistic assumptions that drive to more performance, enhance safety and reliability, conservative designs and thus higher cost, and extend operating lifetimes. helping to achieve higher power efficiencies, reducing learning curves to efficient For the existing fleet of NPPs, simulation operation, improving safety posture, can reduce margins, thereby increasing optimizing the design of the power grid performance. For example, for 100 reactors and the fuel cycle, and supporting better with an operating cost of $1 million/day, a 5% (more efficient) operations, including in- power increase results in an annual return of line monitoring and operator training. $2 billion, for 20+ years. For advanced new reactors, optimized designs can be developed References without any “learning curve” (thus increasing years in operation). For the current fleet, for DOE Office of Basic Energy Sciences (2006), example, the difference between 30 years at Basic Research Needs for Advanced Nuclear 60% capacity and 10 years at 90% capacity can Energy Systems: Report of the Basic Energy Sciences Workshop on Basic Research Needs be viewed as unrealized potential equating to for Advanced Nuclear Energy Systems, July $328 billion (in 2007 dollars), over 30 years. 31–August 3, 2006 (http://www.sc.doe.gov/bes/ As an example, for the three areas of reactor reports/files/ANES_rpt.pdf). and fuel cycle technology, the principal DOE Office of Nuclear Energy, Science, and benefits are as follows. Technology (2006), Global Nuclear Energy Part- nership Technology Development Plan, Global • Exascale M&S could have a substantial Nuclear Energy Partnership Technology Devel- impact on spent fuel reprocessing in ar- opment Program Integration Office, January 9, eas such as the simulation of separations; 2007. the development of new separation agents, addressing whether these agents can be “engineered” to provide the desired results; the performance of full-scale plant

47 Section 2D: Energy: Nuclear Fission

48 3 Biology

Microbial life is responsible for all major pro- As microbial sequencing continues to expo- cesses on Earth, from growth of corn in a field nentially produce the blueprint of cultivated to deposition of carbon under the oceans. Un- microbial life, comparative computational derstanding the role of microbes in these pro- genomics tools are enhancing our ability to cesses is the first step in manipulating them, model individual proteins, groups of proteins, enhancing the positive aspects of microbial bi- and microbial cells. These analyses are help- ology to enhance all life on earth. Our current ing to unravel novel metabolic and regulatory understanding of microbial life comes almost pathways that have not been characterized entirely from experimental observations. Bi- previously. Moreover, the recent advent of ology must capitalize on emerging technolo- environmental sequencing—understanding gies to combine computational analyses and the molecular makeup of all organisms in modeling into a greater understanding of mi- an environment simultaneously—has revo- crobial systems and their interactions within lutionized our comprehension of microbial a community. Such understanding will pave diversity on Earth. To this wealth of infor- the way for the new scientific frontiers of syn- mation, high-throughput experimental capa- thetic biology and will be central to ecologi- bilities, such as mass spectroscopy and chip cal sustainability in the future. technologies, are providing information that adds meaning to the sequence. Biological experimentation has revealed the intricate interplay between proteins and their HPC in biology is accelerating the transition ligands: crystal structures and genetic tests from hypothesis-driven to design-driven re- have demonstrated atomic and molecular- search at all scales. Computational simulation level interactions driving reactions that raise of biological systems is beginning to drive the or raze complex macromolecules essential for direction of biological experimentation and life. Within cells, biopolymers and molecular the generation of insights. Computational bi- complexes are constructed through regulated ologists thus are proving to be at the vanguard pathways. As we enhance our rudimentary of the revolution in biological sciences. understanding of the connections between the systems that form a cell, and their tem- 1. State of the Art poral and spatial separation, we move toward modeling whole microbial cells. Modeling Microbial life affects every known physical steady-state cellular growth will expand and and geochemical process on the planet. If we adapt into real-time continuous culture mod- are to manipulate and control the pathways eling. In nature, cells do not live in isolation. and mechanisms, we must understand the Combinations of microbial cells form com- roles of microbes in these critical processes. munities, and coordinated microbial actions affect local and global environments. Determining how a protein is folded in vivo Understanding microbial and then how that protein binds its ligand systems will pave the way The genomics period started in the mid-1990s or substrate is central to understanding the for new scientific frontiers of with the first complete microbial genome se- function of that protein. For the majority of synthetic biology and will be quences, and progress towards more rapid and proteins that are identified, the ligands are central to ensuring ecological cheaper sequencing has continued unabated. unknown or assigned based on sequence or sustainability in the future. Section 3: Biology

With exascale computing, structural similarity to other proteins. There- the complexity biological experimentation, to scientists can run fore, the modeling process may begin with predict functions, and to design experiments models that fully represent docking or protein:ligand interaction studies that can confirm or refute those predictions. the complex, spatially to identify suitable substrates from the entire With the advent of technology and price heterogeneous, complex chemical universe. A subset of these points that enable complete sequencing of all multiscale, multimodel searches is critical to drug discovery—finding cultivated microbial life, these computations processes of microbial systems. enzyme inhibitors that block specific func- will become ever more complex. These three tions. Searching through portions of chemi- intertwined problems—alignment, phyloge- cal space and comparing those 3D chemical netic trees, and structure predictions—are at structures with 3D protein structures requires the heart of protein function prediction and massively parallel computations. However, currently operate at the terascale. For exam- once a protein:ligand interaction has been ple, there are approximately 10,000 proteins demonstrated computationally, genetical- in families. To generate 100 trees per family, ly, and structurally, modeling the temporal at approximately 1 day per tree, requires 106 variations in specific atom-level interactions CPU-days with current computational plat- remains beyond the realm of current compu- forms. These phylogenetic trees will improve tational capacity. the design of structure prediction and testing models that will in turn lead to enhance- New techniques are providing high-through- ments in the phylogenetic methods. Advances put structure prediction and characterization in algorithms for string matching, similarity [Goens et al. 2004,] but exascale computing searching, and identification of similar pro- is required to predict structures for all identi- teins that are required to reduce the com- fied proteins. The complexity of this problem putational load for these approaches all is reduced through identification of “protein depend on high-performance, integer-based families” —groups of similar (but not iden- computations. tical) proteins that are common to different microorganisms. Clustering, similarity, align- Systems biology aims to develop validated ment, and maximum likelihood trees and phy- capabilities for simulating cells as spatially logenetic methods will be leveraged to reduce extended mechanical and chemical systems in a way that accurately represents processes such as cell growth, metabolism, locomo- tion, and sensing. Initially this work will be developed in the much simpler, and better understood, bacterial realm rather than at the level of complex multicellular systems. Cur- rently, the state-of-the-art methods for modeling bacteria are flux-balance analysis and regulatory and signal transduction network analysis. Metabolic pathways can be extract- ed automatically from an annotated genome, and derivation of stoichiometric metabolic networks suitable for flux balance methods is becoming automated. More realistic models of bacterial growth in defined conditions will soon replace these simplistic models of cells in a single state. Two developments render such an effort feasible with exascale computing: (1) first principles–based algorithmic Figure 3.1 Central metabolism in all organisms consists of a set of interconnected approaches to fully represent the complex pathways. Although the components of most of the pathways are known, model- spatially heterogeneous, multiscale, and mul- ing microbial metabolism, signal transduction, and regulation currently can handle steady-state processes. Exascale modeling will allow real-time fluxes to be simu- timodel processes characteristic of microbial lated, leading to design driven experimentation. (Image from: http://en.wikipedia. modeling; and (2) the wealth of new experi- org/wiki/Metabolic_pathway) mental techniques that can provide the basis

50 Modeling and Simulation at the Exascale for Energy and the Environment into ecosystems and environments. Develop- for validating the models generated, including ment of tools for simulation and modeling high-throughput methods for acquiring ge- of microbial life using exascale computing nomics and proteomics data, and high-resolu- will impact virtually every aspect of human tion imaging techniques that can, for example, interaction with nature. Tools developed for track the locations of individual molecules in exascale computing models of microbial life a cell. The development of simulation models at each scale will be used to simulate the ef- will provide a theoretical framework that will fects of perturbations. These models will transform these massive data streams into provide unparalleled ability to produce hy- design-driven research priorities. potheses and to design experiments from the Unlike other modeling efforts, microbial- computational framework, to test the models sized multiscale models may enable design- through interactions with biologists, and to driven research that can be rapidly tested by refine the models based on the results of the experimental biologists. This combination of experiments. modeling, prediction, and testing will gener- Models and simulations will be generated ate a positive feedback loop, continually en- at the molecular dynamics, systems biology, hancing the models that are generated. This and ecosystem levels. A challenge will be to new paradigm, with simulation driving the integrate these models from different scales hypotheses to be tested by the biologists, will into unified systems that will stimulate quan- place computational biology firmly at the titative biology. The exascale computational helm of biological discoveries. framework will accelerate the transition of As combinations of microbial models are computing in biology to design-driven re- combined into communities, it will be pos- search. Enhancing our understanding and sible, initially at an entirely different scale, modeling of microbial life will benefit a broad to model whole communities not only of mi- range of application areas: crobes but of their associated Eukarya: plants • Energy. Microbial modeling at the exas- and animals. To understand and manipulate cale will identify new proteins, pathways, carbon fluxes in terrestrial or aquatic envi- subsystems, and manipulations that will ronments, to enhance energy production or be used as biocatalysts and to derive bio- carbon sequestration, will require not only fuels, and to accelerate the development understanding individual isolated organisms of biofuel cells and direct solar energy but also modeling the entire balance and flow cells. of carbon, nitrogen, phosphorus, and oxygen through the life cycle of the ecosystem. New • Environmental remediation. Microbial models will be needed that extend to micro- modeling at the exascale will provide the bial eukaryotes such as the fungi most critical ability to engineer individual organisms for nitrogen fixation in the rhizosphere. In ad- to more effectively transform unwanted dition, microbial ecology will need to learn compounds to harmless metabolites, as from traditional macro ecology and models of well as the ability to engineer multispe- different ecosystems. Studies in environmen- cies communities for efficient functional tal microbiology are just beginning to unearth purposes through design-driven research. the fundamental links between micro- and macroecology that will be critical to under- • Industrial-scale microbiology. Microbial standing microbes’ roles in the world. modeling at the exascale will produce recipes of mutations that transform wild 2. Advances in the Next microbial strains to production strains Decade (for production of fine chemicals, phar- Development of tools for maceuticals, and next–generation green modeling microbial life at the Microbial life is so pervasive, and so essen- feedstocks). Complete metabolic and exascale will tial for human life and sustainability, that functional models, design-driven predic- provide unparalleled ability we need to understand the connections from tions, and experimental confirmation pro- to simulate the effects of individual proteins through whole cells and perturbations on microbial systems.

51 Section 3: Biology

Simulating 100 ms of a viding positive feedback are all required systems. For example, determination of the protein folding is expected to to ensure high-quality products. structures of diverse proteins, including tra- require 3 years on a petaflops ditionally intractable structures such as mem- architecture; with exascale • Carbon sequestration. Microbial model- brane proteins, is critical for understanding computing, scientists will be ing at the exascale will provide the abil- the flow of compounds within and between able to simulate the entire ity to manipulate microbes to sequester cells, and essential for protein-scale model- self-assembly process in carbon from the environment via miner- ing components. Increased genome sequenc- microseconds. alization. Without understanding the very ing capacity—initially sufficient to handle nature and forces driving the fixation of sequencing all known microbial genomes— CO , we will not be able to reduce the 2 is essential to capture the diversity of life on concentrations of atmospheric green- earth. However, at projected sequencing rates house gases. of hundreds to thousands of genomes per year, • Sustainability. Microbial modeling at the DNA availability, logistics, and manual cura- exascale will yield models of complex, tion of sequenced genomes are likely to pose productive, and sustainable environments, more of a hurdle than sequencing capacity it- like the prairie grass that does not require self. As we move toward exascale computing, exogenous fertilization. These ecological technological advances will likely be capable models provide the key to sustainability of delivering a whole genome as a routine mi- and security into the future. crobial assay, a fundamental requirement for understanding ecosystems-level biology. Enhancements in our understanding of biology at each scale—from atomic, through The biotechnological advances will demand genomic and cellular, to ecosystems—are similar computational advances. Current driving the need for high-performance bio- biomolecular modeling and simulation logical computing. These approaches and capabilities used for self-assembly in molecular algorithms bring their own challenges and biology rely heavily on coarse-grained problems, in particular the need for multi- techniques and empirical approximations for scale modeling. However, computing at the particle-particle interactions. The simulation exascale level will provide unique opportuni- is also necessarily limited in length, and ties to reveal intimate knowledge about the current scales do not capture the long smallest, and yet the most important, mem- time periods of the self-assembly process. bers of the biosphere. For example, the IBM Blue Gene project estimates that to simulate 100 microseconds 3. Major Challenges of a protein folding will require about 3 years of computation on petaflops architecture. The technological challenges for microbial With exascale computing, the entire self- multiscale modeling are vast, but not insur- assembly process will be simulated over mountable. The central technological chal- micro- or millisecond time scales, leading lenge is to advance modeling to the point to new scientific insights and design-driven where it can produce a steady flow of predic- research. Combining higher-fidelity and tions that are fed into wet-lab operations for longer-time simulations will enable research functional characterization. These character- into the parameter space that affects individual izations will feed back to the modeling to pro- protein:protein and protein:ligand interactions vide near real-time assessment of the quality (temperature, pressure, different connection of the model. In essence, this approach will and surface capping molecules, etc.) or even construct a growing body of models that are the use of optimization procedures to design continuously calibrated against the parallel new structures with desired properties. The growth in phenotypic data. challenges for other types of computational biology (such as homology searching) are Not all the challenges of moving into the ex- even more overwhelming as a result of high ascale computational era are technological, I/O and memory requirements and traditional however. Significant biological hurdles must dependence on shared-memory resources. be overcome before exascale computing can be efficiently applied to multiscale microbial

52 Modeling and Simulation at the Exascale for Energy and the Environment

Figure 3.2 Microbes are the original carbon se- questerers. Microbial cells (Emiliana huxleyi; A) are approximately 5 µ across, yet when they A. bloom in the ocean the vast numbers of cells are visible from space (B; an E. huxleyi bloom C. B. off the southwest coast of England). Over millen- nia, these microbial blooms are deposited on the ocean floor, storing carbon as calcite, and hence removing CO2 from the atmosphere (C: The white cliffs of Dover were created by microbes, especially cocolithophors). Design-driven biology based on exascale computing could identify how to enhance the growth of cocolithophores, increasing deposition of carbon on the ocean floor. (Images A and B from http://en.wikipedia. org/wiki/Emiliana_huxleyi. Image C from http:// en.wikipedia.org/wiki/White_cliffs_of_dover.)

Improving the consistency and accuracy of design-driven outcomes of such a synthesis protein functional annotations through the might be. elucidation of metabolic pathways or processes as subsystems that cover almost all of the 4. Accelerating Development machinery embodied in the newly sequenced genomes will be central to unraveling the me- Parallel advances in biological and compu- tabolism within each microbe and the roles of tational sciences will be required to realize these microbes in their environments. These multiscale microbial modeling at the exas- improved annotations will also feed into sys- cale. Advances will be required at each in- tems biology approaches to understanding the dividual layer from atomic protein models holistic nature of the cell. However, model- through whole cell modeling to ecosystem- ing and simulation provide only a glimpse of scale models. a narrow local view of each process without interaction between modalities and scales. Biomolecular dynamical models need to The approach to addressing all of these chal- adapt to varying parameter space, local varia- lenges is development of exascale algorith- tions of constituents, and effects of nearby mic and software tools for representing the proteins. Entirely new models that can lever- various macroscopic subsystems, combined age exascale computing to realize micro- or with the application of these tools in various millisecond-scale protein simulations are es- combinations to simulate specific problems in sential for this modeling to succeed beyond systems biology. The design of the tools and the nanosecond scale. their application requires both biologists and mathematicians, with theory and experiments Technologists are driving the sequencing and informing the design of models for specific annotation of thousands of single-celled or- problems and the factorization of the algo- ganisms (including Archaea, Bacteria, and rithmic tools required into reusable software Eukarya). By the time exascale computing components. The feedback between biologi- arrives, the majority of diverse microbial cal experimentation and mathematical tool organisms will be completely sequenced. development is an ongoing process that will Improvements to terascale and petascale yield robust representations of systems biol- integer-based computations will be essen- Exascale algorithms and ogy across multiple specific problems. For tial to ensure that computational efficiency software tools will be critical example, we have neither the understanding scales with technological efficiency. The fed- for representing macroscopic of how to attack the hybridization of stochas- eration of molecular databases will result in subsystems, a key step to tic and deterministic algorithms in a single kernel databases that are well annotated and understanding the holistic nature of the cell. simulation, nor any ideal what the biological maintained and support massive amounts of

53 Section 3: Biology

New biomolecular models must sequence data. These databases will have to – Protein structure prediction and be able to leverage exascale address data centralization versus distribution classification computing to realize microscale issues common to other data-intensive scientific or millisecond scale protein endeavors. However, these databases, together – Prediction of interacting protein simulations. with an enhanced computational framework partners that accelerates the modeling and characteriza- – Prediction of protein-protein tion of heretofore-unknown biochemistry, will complexes arise simultaneously with our genome-level understanding of microbial cells. – Refinement of function prediction (e.g., specifics of substrates the The new computational framework for sim- protein may bind) ulating complete cells will need to incorporate protein functions from these federated – Prediction of structure-function databases, regulation and cellular states (exchanges caused by single amino perimentally tested by microarrays), and acid (aa) changes or indels [e.g., compound concentrations and stoichiometries nonsynonymous single nucleotide required to model flows through individual polymorphisms (SNPs)] cells, groups of cells, and complex ecosystems. In addition to these separate develop- • Predictive capabilities of metabolic ments, each of which will occur in parallel, a models. Automatic generation, testing, complex integration layer that transcends the and verification of models for keystone separate domains will also be needed. This species for given environments and for multiscale modeling will be essential for un- the 50 species with the most wet-lab derstanding how proteins interact with other data will lead to the framework for high- proteins and/or their ligands in one cell and throughput generation of accurate flux- how these interactions might affect the me- based models for all sequenced microbes. tabolism of a cell elsewhere in the ecosystem. These models will predict the state of the This new approach to computing in biology cells and the transitions between states, will be the driving force behind the deter- while allowing comparison of those pre- mination of the role of proteins, pathways, dictions to measured variables to assess microbes, and communities in complex bio- the efficacy of the model. logical processes. • Modeling and simulation of less complex microbial communities. Several 5. Expected Outcomes low-complexity microbial systems have The generation of predictive capabilities is at been well studied at the molecular and se- the heart of exascale microbial modeling and quence level. These include the acid mine will drive the transition to design-driven bio- drainage system, the enhanced biological logical sciences. However, existing research phosphate removal system, the Soudan infrastructure is not able to support the tran- mine, and the solar saltern crystallizer sition to exascale computing. Therefore, ad- ponds. Modeling these environments will ditional investments are needed in key target move us towards understanding complex areas to ensure that computing in biology is environments at the exascale. appropriately situated to take advantage of the exascale computing opportunities. • Algorithms, architectures, and methods for integrating networks. The integration • Novel molecular dynamics algorithms of transcriptome, metabolome, genome, for microsecond simulations of proteins and other data requires new algorithms and ligand interactions. Enhancing the that will enable the adoption of exascale algorithms used for molecular dynamics technology by computational biologists. approaches will include extending them Some of these algorithms may require out to microsecond or millisecond times- novel computer architectures that go be- cales for all proteins encoded by all mi- yond the limits of floating-point-centric, crobes (bacteria and viruses) sequenced:

54 Modeling and Simulation at the Exascale for Energy and the Environment The generation of predictive memory-limited cluster computing that A related risk is the failure to lower the barrier capaabilities is at the heart of are suitable for the physical sciences. for access to exascale computers. Accessibil- exaascale microbial modeling. ity of simulation software for microbiologists 6. Required Investment is a constant problem. Failure to address this problem will impede the adoption of exascale In order to enable exascale systems biology, computing. funding must be available for generation of experimental data (both large-scale data and References painstaking measurements of critical param- S. S. Goens, S. Botero, A. Zemla, C. E. Zhou and eters), development of experimental meth- M. L. Perdue (2004), J. Gen. Virol. 85, 3195. ods to measure biochemical parameters in a high-throughput fashion, data standardization E. Klipp, W. Liebermeister, A. Helbig, A. Kow [Klipp et al. 2007], and database integration, ald and J. Schaber (2007), Nature Biotechnol. 25, as well as experimental validation and verifi- 390. cation of model predictions.

7. Major Risks

A major risk is that biologists are not sufficiently trained to leverage high-performance computational methods and resources. A critical component of facilitating exascale microbial modeling is to cross-train microbiologists with state-of-the-art modeling tools and applications.

55 Section 3: Biology Socioeconomic 4 Modeling

Decades of research have deepened scientific while bottom-up economic and energy mod- understanding of the impact of greenhouse els describe, for example, the greenhouse gas gases on climate, as documented, for example, emissions of different industry sectors and in the assessment reports of the Intergovern- mitigation costs. However, computational mental Panel on Climate Change (IPCC). Cen- limitations have prevented any existing model tral to this research are Earth system models from including substantial regional and sec- (ESMs) incorporating detailed representations toral disaggregation, a dynamic treatment of of the atmosphere, ocean, cryosphere, and bio- world economic development and industrial- sphere and their interactions, largely through ization, and detailed accounting for processes chemical systems. Model predictions are now such as technological innovation, industrial considered to be reasonably reliable on global competition, population changes, and migra- and continental scales, and research is now tion. Other than emission scenario drivers being directed towards improving predictive for climate models, feedbacks from human capability at regional and subregional scales. activities to climate change are generally not addressed, and it has not been feasible to ex- As the scientific community’s confidence in plore the uncertainty range within climate and these model results grows, discussion of im- economic model domains with Monte Carlo pacts and of potential adaptation and mitiga- or more general Bayesian methods. The lack tion strategies also grows. Modeling tools can of computational power has also limited the play a vital role in developing the scientific ability to apply the best statistical techniques knowledge base required to understand these to existing data. issues. In developing such tools, we must recognize that the impact of climate change The emergence of petascale and (within the on society and the ultimate effectiveness of next ten years) exascale computers makes it specific responses depend critically on human possible, in principle, to attempt a detailed actors, who will determine, for example, how and fully integrated treatment of these diverse energy supply and demand evolve over time factors. By allowing for far more detailed and how and when different “solutions” are treatments of the various components and deployed and applied. Hence, we must model feedbacks among these different components, human responses if we are to understand the and issues of uncertainty and risk, exascale likely effectiveness and impacts of different computers have the potential to transform un- responses and thus help to sustain a prosper- derstanding of socioeconomic-environmental ous and secure society. interactions.

Integrated modeling of the social, economic, Such detailed and integrated models can al- Integrated modeling of and environmental system with an extensive low for the quantitative study of key ques- the social, economic, and treatment of couplings among these differ- tions, including the following: environmental system with an ent elements and consequent nonlinearities extensive treatment of coupling and uncertainties is a scientific and compu- • How will climate change impact energy of these different elements and tational grand challenge. Existing integrated demand and prices? consequent nonlinearities and models incorporate treatments of economic uncertainties is a scientific and impacts and impacts on human well-being, computational grand challenge.

57 Section 4: Socioeconomic Modeling

• How will nonlinearities, thresholds, and work that incorporates detailed treatments feedbacks in the coupled climate-eco- of the various components listed above and nomic-energy system impact both cli- that allows for computational investigations mate and energy supply? of both individual elements of the socioeconomic-environmental system and the entire • How will different adaptation and miti- coupled system. The program must address gation strategies affect energy supply and a wide range of methodological and compu- demand, the overall economy, the envi- tational problems, including data manage- ronment, individual products and servic- ment and acquisition, parameter estimation, es, public health, and the vulnerability of technology characterization and forecasting, the U.S. economy and infrastructure? socioeconomic model specifications, regional and subregional disaggregations appropriate • How can computational approaches help for ESMs, and the need to quantify the de- us to identify and visualize good strate- gree of uncertainty in forecasts. This system gies for R&D, policy formulation, and will capture feedbacks between socioeco- technology adoption under conditions of nomic systems and climate and will allow for future uncertainty, but with future infor- quantitative evaluation of potential strategies mation feedback opportunity? designed to reduce greenhouse gas emissions with minimal societal costs. • How are answers to these questions influenced by other processes, such as In a series of staged R&D steps, the program population growth and demographic can: change; economic growth and development, particularly in Asia; immigration; • Enhance existing socioeconomic models and technological change (such as tele- to provide far greater geographical and conferencing technology)—all of which temporal resolution of important process- will affect transportation and land use es, interactions, and feedbacks. patterns, human behavior, and business conditions? • Incorporate important socioeconomic elements previously treated as exogeneous, • How will the success of potential solu- such as economic development, exchange tions be affected by technical issues vs rates, and foreign industrialization. other social, political, regulatory, and market factors, such as barriers to entry • Invest in multidisciplinary tool develop- for new technology providers? ment including algorithms, validation, data analysis techniques, and uncertainty DOE’s leadership role in high-end computing analysis. and its strong interest and expertise in climate change make it natural for DOE to take a • Extend large-scale socioeconomic mod- leadership role in an R&D program aimed at eling to new domains, such as analysis of building detailed, integrated socioeconomic- market barriers to new technologies and environmental models designed for exascale health impacts of climate change. computers. Building on DOE expertise in climate and energy system modeling, and bring- • Integrate existing and new data sources to ing to bear the latest methods in economics, allow for rigorous model validation. quantitative techniques in behavior and decision theory, and HPC tools, such a program • Integrate global ESMs with global socio- With expertise in high-end computing and climate change, can develop tools to deepen the understand- economic models that highlight human DOE is in an excellent position ing of technical, economic, and social issues and geophysical interactions. to lead the development that underpin the climate change challenge. of detailed, integrated, In the following, we describe this need and socioeconomic-environmental Such a program should aim to create a high- our approach in more detail. We also mention models for exascale computers. performance, high-fidelity modeling frame- other potentially important applications of the

58 Modeling and Simulation at the Exascale for Energy and the Environment proposed socioeconomic model, which can be HUMAN ACTIVITY (EPPA) both coupled with ESMs and used in complex national and/or regional economic human agriculture, development, emissions, land use health computational settings such as game theory forestry, e ects bio-energy, and stochastic dynamic programming ecosystem land productivity climate/ energy CO2, CH4, CO, use change demand N2O, NOx, trace gas SOx, NH3, 1. Importance of Global uxes CFCs, HFCs, hydrology/ sea EXAMPLES OF water (CO2, CH4, PFCs, SF6, N O) level MODEL OUTPUTS Socioeconomic Modeling supply 2 VOCs, BC, etc. change and policy GDP growth, constraints energy use, Earth system modeling has progressed to a policy costs, agriculture and point where there is considerable confidence coupled ocean, atmosphere, and land EARTH SYSTEM health impacts... in predictions of continental- and global- global mean ATMOSPHERE URBAN and latitudinal scale climate changes over the next 100 years 2-Dimensional Chemical temperature and & Dynamical Processes Air Pollution Processes precipitation, [IPCC 2007]. We are thus understandably in- solar sea level rise, forcing sunlight, water cycles, energy & momentum transfers, sea-ice cover, terested in determining likely impacts of cli- air & sea temperatures, CO2, CH4, N2O, nutrients, greenhouse gas mate change on ecosystems, human society, pollutants, soil properties, surface albedo, concentrations, sea-ice coverage, ocean CO2 uptake, air pollution levels... human health [McMichael et al. 2003] and land CO2 uptake, vegetation change... soil and vegetative carbon, net primary well-being, the economy, and national secu- volcanic OCEAN LAND productivity, rity and in understanding the effectiveness of forcing 3-Dimensional Dynamics, Water & Energy Budgets trace gas emissions Biological, Chemical & (CLM) from ecosystems, potential adaptation and mitigation strategies. Ice Processes Biogeochemical Processes permafrost area... (MITgcm) (TEM & NEM)

One set of questions that we must answer to Figure 4.1 Earth system–human system interactions, as captured in the MIT address these issues concerns the geographi- modeling system [Sokolov et al. 2005]. cal and temporal distribution of climate change. For example, will climate change in- ductivity of land used to grow biofuels, as crease or decrease the level, variability, and a result of climate change, or by changes timing of rainfall and temperature in agricul- in energy demand due to changes in tem- tural regions? Will it increase or decrease the perature patterns. frequency of hurricanes over low-lying coastal regions or extend the scope of vulnerable • Reductions in greenhouse gases may de- regions? To answer these and other related pend on the emergence and adoption of questions, DOE has defined as a primary goal new energy production, distribution, and over the next five to seven years the improve- consumption technologies. Thus, we may ment of the performance of ESMs on regional ask what factors affect the emergence and subregional scales. of new technologies and, for potential climate change solutions, how their suc- A second set of questions, equally central to cess will be affected by technical or other understanding climate change impacts and factors, such as availability of capital for responses, concerns climate system-human these ventures and obstacles to entry for system interactions (see Figure 4.1). Such new providers. interactions can be both important and complex, as the following examples demonstrate. • The nature and pace of climate change response may depend on international • Biofuels have the potential to reduce agreements. In such agreements, there both overall greenhouse gas emissions may be winners and losers. We may want and dependence on foreign oil. On the to quantify the benefits or costs of alter- other hand, the production of biofuels native responses to different countries, can increase both water consumption and states, and industries and study the inter- food prices, with implications for human actions that may occur between different societies—and, if grown on previously parties. Such studies can contribute to the unfarmed land, greenhouse gas emissions design of negotiation strategies and inter- [Sustainable Bioenergy 2007]. The ulti- national emission treaties, including pen- mate cost-effectiveness of biofuels may alties for violators. also be influenced by changes in the pro-

59 Section 4: Socioeconomic Modeling

Proposed responses to climate change range from controlling greenhouse gas emissions to mitigation and adaptation. In order to curb the increase in greenhouse gases, strategies such as carbon taxes, cap and trade, alternative fuels, voluntary agreements to reduce emissions, regulations including building codes and mandatory energy performance standards for equipment, and stricter emission standards for industry and transportation are under consideration. On a larger scale, we may ask how different responses affect energy supply and demand, the overall economy, the environment, demand for individual products and Figure 4.2 Aggregated world regions used in the World Energy Outlook (WEO) services, public health, and the vulnerability 2006 study [IEA 2007]. of the U.S. economy and infrastructure. In answering these questions, we must consider Answers to such questions may be sensitive feedbacks from human responses to the cli- to changes in population distribution, em- mate system through, for example, changes ployment patterns, income growth and prices, in greenhouse gas emissions and surface al- demand for resources, and human prefer- bedos due to land use changes, which may in ences. Thus, we need to understand trends turn be affected by the adoption (or not) of in population growth, demographic change, low-carbon energy systems—developed, per- economic growth and development (particu- haps, as a result of investment in R&D pro- larly in Asia), technological innovation, and grams designed to position us with options behavioral shifts such as increasing consumer for the future. preferences for particular energy-intensive services and pressure on companies to reduce Detailed global-scale socioeconomic models their environmental footprint. capable of capturing some or all of these processes will provide invaluable input to scien- Socioeconomic modeling seeks to construct tific understanding and decision making. In quantitative descriptions of these aspects of some cases, these models may serve simply human systems. Climate models are based on to show that a particular response may not our understanding of the physics, chemistry, work as expected. In other cases, they may ecology, and biology of the Earth system and provide data that can guide the design of ef- are often based on well-established laws of fective responses. nature. Human motivations and behaviors are complex, and thus socioeconomic Because the models required to study socio- models are frequently more approximate. economic aspects of climate change must In contrast to physical systems, diverse necessarily encompass many aspects of hu- humans, corporate organizations, and other man behavior, they can be expected to have institutions are active, inventive, adaptive, important applications to numerous prob- and forward-looking in making decisions. lems besides climate change, such as the In many areas of economics and the social following: sciences, however, there exist well-developed theory, substantial amounts of data, and • Managing natural resources: water, for- substantial experience with computational ests, biodiversity, ecological systems, methods. Interdisciplinary work is also land use Because human motivations and behavior are complex, moving forward, although it has far to go. In socioeconomic models particular, socioeconomic models are not yet • Addressing issues of energy security of climate change effects well integrated with biophysical models. and exhaustible resources: oil, gas, coal, necessarily will involve uranium approximations.

60 Modeling and Simulation at the Exascale for Energy and the Environment

• Optimizing the supply and delivery of are computationally extremely challenging Many socioeconomic models energy services even in isolation (e.g., when using game theo- currently are single dimensional, ry solution concepts, market equilibrium con- sacrificing detail or ignoring • Promoting economic development, in- cepts that require the finding of fixed points, certain feedback effects; linking come growth, poverty reduction or stochastic dynamic programming formu- such models with climate lations) and are not necessarily well under- models will involve major • Improving industrial productivity and computational challenges. competitiveness stood. This situation, as well as a general lack of access to high-end computers within the • Investing in human capital: education, socioeconomic modeling community, has led health care, job opportunities, social to integrated models being limited in various security regards. They may sacrifice sectoral detail (e.g., see Figure 4.2), avoid consideration of • Making tax policy more efficient and certain subsystems, assume myopic behavior, equitable ignore certain feedback effects and the notion • Promoting common human values such of an overall dynamic equilibrium, or fail to as protecting the earth’s environment adequately address issues of uncertainty. In short, they may be theoretical models focused 2. State of the Art on understanding a single dimension. The use of computation-intensive model- To go from simpler, single-dimensional mod- ing to study climate system-human system els to elaborate multidimensional models interactions is far from new. For several de- will require both adding detail and consis- cades, researchers have developed integrated tently specifying coupling among different models for this purpose, on a variety of time systems. scales and geographical scales [Edmonds et Linking climate models to socioeconomic al. 1997; IMAGE team 2001; Sokolov et al. models is similar to what was done in the 2005]. Such integrated models couple dif- 1980s for the Acid Precipitation Assessment, ferent types of (sub)model, such as climate in which a socioeconomic model set was (atmosphere, ocean, cryosphere, biosphere, assembled and linked to large-scale atmo- etc.); land use, vegetation and ecology (ur- spheric models and used to address key ques- ban, agriculture, forests, biomass, wildlife tions of interest [NAPAP 1990]. The climate habitat); environmental and resource chang- assessment would presumably be on a much es; and socioeconomic (greenhouse gas emis- grander scale. sions, health, political stability, feedbacks on economic activities and production costs). For socioeconomic systems, the various parts of the economy operate within a global, mac- Concurrently, researchers in economics and ro environment. At the same time, the macro the social sciences have been applying com- economy is made up of the sum of its parts. putation to issues such as energy system op- In order to help understand the behavior of timization, investment analysis, improving the integrated socioeconomy and to estimate human institutions such as our tax system and impacts due to changing conditions, large- conditions for economic and social develop- scale computational modeling frameworks ment, game theory strategies, and hedging can be used. Over the past 25 years, econo- uncertainties [Amman, Kendrick, and Rust mists, systems analysts, and quantitative so- 1996; Judd 1998]. The time scales involved cial scientists have developed and expanded are typically in the range of 5–40 years, de- their models and computational techniques to pending on the decision problem, and the unit solve specifications with many diverse- ele of analysis may be country, major region, in- ments, including; dustry sector, or income group.

In principle, integrated models can (and argu- • various household consumer groups; ably should) incorporate detailed descriptions of all aspects of the social and economic sub- • industrial sectors and business services; systems. However, many of these problems

61 Section 4: Socioeconomic Modeling

damages, and promote technology and investment.

Computational models are typically structured as hierarchies to be able to aggregate consistently from the micro to the macro and then to disaggregate income formation and macro environmental conditions to households and firms. Computational methods systematically calculate prices of goods and services up the levels of a hierarchy to final demands and allocate various quantities down the hierarchy, taking into account price, income, and other elasticities. The levels of the hierarchy include opportunities to improve economic efficiency through trade and substitution and penetration of advanced technologies. These production hierarchies are created for each sector in each region.

The structure of these socioeconomic models provides “hooks,” or connection points, for the uptake of low-carbon resources and technologies such as biofuels and biorefining Figure 4.3 Models included in the IMAGE 2.2 integrated modeling system [IMAGE or energy efficiency. The comprehensiveness 2007]. of the model allows estimation of the relative market roles that different technologies may • conventional and low-carbon or high- play (which may depend on uncertain fac- efficiency technologies and production tors), and their end-to-end evaluation (life cy- activities; cle analysis). Constraints may be added to the model to represent security considerations. • electricity, petroleum, gas, biofuels, solar, nuclear, and hydrogen energy forms; Applications for such a model include R&D investments under uncertainty (a current • implications of carbon constraints on DOE-identified need); economy-to-climate- transportation and goods distribution; to-economy feedbacks; making strategic investments to position ourselves to maintain • environmental emissions and impacts; future options [Dixit and Pindyck 1992]; analysis of the interconnected issues of world • developing supply and demand optimiza- oil security; optimizing energy demand, sup- tions for sectoral resources; ply, and infrastructure choices; analyzing the impacts of carbon caps and taxes within the • land, water, other resources, and climate existing tax system structure; and more in- feedback; tegration with issues related to climate, such as economic development, international in- • regional and country divisions, including terests, security, transportation, logistics net- policy variations; works, health, and vegetation/ecology.

• taxation systems and financial markets; Figure 4.1 illustrates one model of the climate and system/ecological system/socioeconomic system designed for integrated assessments and • public policy instruments to protect pri- policy evaluation. Another model, IMAGE, is vate property, mitigate environmental shown in Figure 4.3 to illustrate the range of

62 Modeling and Simulation at the Exascale for Energy and the Environment issues that may be included in such models. to better represent the substitution of capital A popular trend in model IMAGE, which was developed in the Neth- for energy in end-use energy-intensive appli- integration is called “hybrid erlands, provides its detailed global land cations. Labor can also be disaggregated into modeling,” in which physical use database and its calculation of land use skill levels and occupation groups. Electricity energy forms, resources, emissions, and technologies are changes over a spatial grid. However, many can be disaggregated into peak, shoulder, and integrated into socioeconomic of its other modules lack detailed represen- intermediate demands. Other forms of pur- models. tations and supporting data. IMAGE is one chased energy include natural gas, petroleum of 19 models that participated in Stanford products, and hydrogen. University’s Energy Modeling Forum study in climate change policy and assessment Cultural values and income, taking into ac- (EMF-21) [EMF Study-21 2006]. The AIM count income distribution over population model from Japan, which also participated in groups in the countries of the world, affect the EMF-21 study, also has strong integrated residential household expenditure patterns. assessment capabilities. For example, in both developed and developing countries, major energy-intensive pur- Much detailed work is under way, particularly chases include light-duty cars and trucks. in the United States, on land use and agricul- Thus, a socioeconomic model must track the ture, including biocrops. The Polysis [English stock, new sales, fuel demands, and carbon et al. 2006] and FASOM models [Murray et emissions of different types and sizes of light- al. 2005] are leading examples, but a num- duty vehicles. Fortunately, computer code ex- ber of midwestern universities have major ists to represent these aspects of society and research and modeling efforts, financed by the economy, and this code is easily scalable. the U.S. Department of Agriculture. Polysis Similarly, greenhouse gas emissions and pe- models the production of about 20 commer- troleum fuel use by aircraft and heavy trucks cial crops by county in each state and groups and other forms of freight transportation are land resources into 6 productivity categories. large and increasing, both absolutely and as Its results are being used by the Energy In- a share of the total. Hence, the modeling of formation Administration (EIA) in preparing transportation and fuel technologies is of high Annual Energy Outlooks for biomass produc- importance [Wang 2001]. tion. These results are also used by national laboratories, including Oak Ridge, Argonne, This integration of physical forms of energy, and the National Energy Technology Labo- resources, emissions, and specific technolo- ratory. The Climate Change Division at the gies into socioeconomic models has become U.S. Environmental Protection Agency uses known as “hybrid modeling” because of its the FASOM model for biofuels studies and soil carbon sequestration assessments.

Most climate-socioeconomic models have certain common ingredients. Goods and services are produced to meet human needs. The value of these goods and services sums to gross domestic product. The output of each production sector is based on a production technology employing labor, capital, energy, and, for agriculture sectors, land. Sec- tors trade materials and semi-finished goods among themselves. For example, car manu- facturers purchase tires from the rubber sector. Some of this trade crosses international boundaries. Industrial capital stock is disaggregated in some models, such as the Argonne All Modular Industry Growth Assessment (AMIGA) model [Hanson and Laitner 2006], Figure 4.4 Typical configuration of a hybrid energy-economic model.

63 Section 4: Socioeconomic Modeling

Exascale computing allows combination of physical and observation- and fuels, energy consumption and related us to be far more ambitious in based statistical models. Figure 4.4, which greenhouse gas emissions, technologies conceiving issue-oriented Earth illustrates a typical configuration for a hybrid adopted, sector outputs and final demands system-human system models energy-economic model, shows anthropogen- for goods and services, distribution of that can provide insights for policy makers. ic greenhouse gas emissions as an input to a income, macroeconomic variables, resource climate model and feedback from climate to management, and other measures of welfare certain economic sectors. or quality of life in societies.

Climate and climate-related changes then be- 3. Need for Exascale come inputs to analyzing these problems. As Computing these decisions unfold over many years, there will also be feedback to the climate system, For many decades, researchers investigating such as economic decisions that influence climate change impacts have been forced by greenhouse gas emissions. Two examples profound limitations in both computational of problems in this category are planning capacities and data to grossly simplify their for a low-carbon energy system (including models and analyses. As a result, they have vehicles), taking into account technological mostly ignored issues such as the following: synergies (e.g., biomass and petroleum-based fuels), and planning an R&D program that • integration of diverse complex systems positions us with options for the future. and their relationships;

A hybrid energy-economic model, with simi- • incorporation of (often only partially un- lar features to those shown in Figure 4.4, derstood) nonlinearities and thresholds; has been used by the International Energy Agency (IEA) to prepare the World Energy • full representation of feedback effects; Outlook (WEO) for 2006 [IEA 2007]. Run- and ning this model shows a difference between a business-as-usual growth scenario and an • meaningful analysis, reduction, and treat- alternative scenario that included policies to ment of uncertainties promote renewable energy, energy efficiency, and nuclear power. The WEO also highlights Exascale computing allows us to think differ- the human dimension: about a billion people ently about what is possible. While any useful in the world still lack access to electricity. model must incorporate simplifications, the This kind of energy poverty illustrates the in- anticipated availability of exascale comput- terplay between economic development, so- ing encourages us to be far more ambitious cial goals, and environmental goals. in conceiving issue-oriented Earth system– human system models that shed insights on For WEO 2006, the world was divided into the complex realities that policymakers must 22 country regions (Figure 4.2). Interfacing deal with. with climate models will require much greater geographical and spatial disaggregation. Part First, significant progress can be achieved of this disaggregation can be accomplished by in sectoral detail. Many dynamic economic using modern spatial statistics [Stein 1999]. models developed for climate studies are based on the computable general equilibrium Socioeconomic models have been used ex- (CGE) approach, the dynamic optimization tensively to generate sets of greenhouse gas approach, or the overlapping generations emission scenarios to drive climate models (OLG) approach, and most aim to model the [IPCC 2007]. A new round of global scenario dynamic paths of economic growth—but sac- simulations is starting in preparation for fu- rifice sectoral detail. Some existing models ture IPCC Assessment Reports. describe energy-related sectors at regional and national scales (e.g., the U.S. National Outputs of most interest from the Energy Modeling System, NEMS [National socioeconomic model include prices of energy Energy Modeling System 2003]) or global

64 Modeling and Simulation at the Exascale for Energy and the Environment scale (e.g., the AMIGA model [Hanson and Laitner 2006; Laitner and Hanson 2006]). However, such models typically do not model economic dynamics in a consistent, modern manner. Most climate change issues require both sectoral detail and careful dynamic modeling.

Each of the three approaches used in modern economics has distinct advantages. Climate change issues challenge us to synthesize a new set of model frameworks suitable for advanced computational architectures. For example, when using the OLG framework, an addition will be needed to address the inter- Figure 4.5 Changes in heating and cooling end use and primary energy under cli- generational time scales involved in issues re- mate change [Hadley et al. 2006]. lated to climate change, and greater sectoral detail will be necessary to produce reliable rate of adoption of new technologies. The de- results. Consistently coupling sophisticated termination of the many parameters in these sector models such as NEMS with new-gen- models presents us with many challenging eration dynamic CGE models will greatly im- computational tasks. We must develop suit- prove the quality of climate change research. able estimation methods with new types of more comprehensive data collection and cali- Increased sectoral detail is required for cou- bration. Because of uncertainty in parameter pling with climate models. Existing socioeco- values, we must also develop efficient meth- nomic models are driven with simple climate ods, such as the use of low discrepancy sets or models with limited geographic and temporal sparse grids, to explore the variables used to resolution. But understanding of both climate generate these parameters. This uncertainty change impacts and feedbacks between so- will be even greater when we couple a socioeconomic systems and the Earth system cioeconomic model with an ESM that has its requires the treatment of finer-grained in- own uncertainties. Current methods for ad- teractions. In one early example involving a dressing large parametric system uncertain- one-way coupling of a global climate model ties, such as Monte Carlo and more general and NEMS, energy demands for increased Bayesian methods, may need to be revisited cooling were found to outweigh savings due and adapted for this potentially large-scale to reduced heating in the United States (Fig- problem. This work must be performed at the ure 4.5) [Hadley et al. 2006]. component, subsystem, set of subsystems, Increased detail is also required for the global and full system levels. water cycle. As climate changes, so do the Progress is also possible in the treatment of distribution, intensity, and usability of pre- social and political inputs. These inputs are cipitation and groundwater [Held and Soden modeled as offline forcings in the current 2006]. Such changes have significant socio- generation of integrated models. Although economic impacts and feedbacks, not only some demographic and population-based on agriculture but also migration, wages, studies have generated numerical model- industrialization, and prices. Simulation of ing techniques for predicting future popula- the global and regional hydrologic cycle and tion growth and human migration patterns, associated economic/management decision– much of this work is still ad hoc and in need making processes over the next 25–100 years of greater rigor. It will be in our interest to represents a major challenge for socioeco- work toward developing more quantitative nomic modeling. approaches, as in economics, and to collabo- Exascale computing challenges scientists to synthesize a new Larger computers will also advance socio- rate with and strengthen institutes such as the set of climate change models economic models that describe, for example, International Institute for Applied Systems Analysis. It may also be useful to collabo- with greater sectoral detail and consumer preferences, technology, and the intergenerational time scales.

65 Section 4: Socioeconomic Modeling

rate on setting up a similar institute dedicated plex computational problems can be tackled to quantitative sociological modeling in the that call for exascale computing. The follow- United States. ing are examples:

A fourth area of opportunity concerns uncer- • Possible strategic behavior of countries, tainty and risk. Some climate change data is or coalitions of countries, to improve known with relatively high confidence (e.g., their positions relative to other countries average global warming), while other data is far less certain (e.g., regional climate change, • Calculation of payoff matrices for dif- frequency of extreme events). We need to ferent countries under different policy study how uncertainty in the climate system approaches, taking into account specific propagates through economic models, both climate-related risk exposure of different to enable quantification of uncertainty when countries reporting on outcomes and to enable evalua- • Design of incentive systems to encourage tion of risk. Economic cycles are also subject cooperative behavior to shocks. It is important to design mitigation strategies and policies that are robust under • Computation of general equilibrium economic shocks and avoid costly policy conditions changes. Researchers at the University of Chicago are undertaking computationally • Solution of stochastic dynamic program- intensive research attempting to quantify the ming problems that recognize that soci- long-run risks associated with coupled cli- eties can wait for feedback information mate-socioeconomic models. In their model, in order to make better decisions in the Mathematical issues such as the researchers let household investors and future about greenhouse gas abatement convergence become important firms base their expectations about climate- measures and other mitigation and adap- as model complexity increases. related risks, and who can most efficiently tation decisions bear these risks, on actual risks calculated by • Representation of noncompetitive mar- the climate system model. ket situations such as Middle East oil A fifth area is the integrated computation of production energy-economy hybrid models, in which • Optimal technology deployment with detailed technology models are embedded in learning rates and stochastic outcomes key areas such as power generation, petro- (these are typically two point bound- leum refining, combined heat and power, and ary value problems and may use opti- vehicle stocks. mal control methods with deterministic A sixth area concerns solution techniques. conditions) For example, NEMS is solved by using a • Other problem formulations involving Gauss-Seidel process that is not guaranteed to complex human behavior converge. Mathematical issues such as convergence become more important as model 4. Major Challenges complexity increases [Judd 1998]. Alterna- tive solution techniques with better numerical Tackling the problems outlined in the preced- properties are frequently more computation- ing section will require significant advances, ally demanding. not only in economics, the social sciences, natural sciences, and computational sciences, A seventh area in which increased computa- but also in cross-discipline interaction meth- tional power can enable new approaches is odologies and institutional arrangements. in the calculation of large ensembles of in- These advances will be essential if we are to tegrated climate-socioeconomic model runs, make required progress at the interface be- The increased power of for sensitivity studies and to calculate prob- tween technological advances, cultural shifts exascale computing can enable abilities of extreme events. new approaches for calculating because of such advances, and the response the probability of extreme Given the availability of detailed global so- of these changes to macro-economic sensi- climate events. cioeconomic models, a range of other com- tivities and climate feedbacks in general.

66 Modeling and Simulation at the Exascale for Energy and the Environment

Many climate change impacts and adaptive this problem. Furthermore, recent results in Exascale computing power responses will apply at the regional or sub- approximation theory can be used to guide us will enable modelers of regional scale. Thus, reductions in climate in using exascale computing power to search socioeconomic processes to model uncertainty, as targeted by DOE’s for efficient methods. replace “computationally light” Earth system modeling program for the next methods of data analysis, five to seven years, will be crucial for efforts Systems are so complex that mathematical producing far superior empirical aimed at studying climate system–human analysis of convergence properties is impos- results. system interactions. sible—we will be coupling Navier-Stokes in ocean and atmosphere with ice, physics, Socioeconomic models such as those dis- and economics. We need to develop methods cussed earlier become large-scale, nonlinear for quantifying uncertainty in both data and systems when accounting for resource supply models. functions or constraints, production-possibility frontiers involving joint production, non- Progress is also required in various areas of convexities in the development and adoption socioeconomic modeling. One major chal- of new generations of energy demand and lenge, which has been the focus of intense ef- supply technologies, and social behavior pat- fort over the past decade, partly in response to tern responses. Progress is required in nonlin- climate change concerns, is capturing induced ear optimization techniques, as suggested by technical change in economic growth models. this class of problems. This is referred to as the endogenous technical change problem. We know that technology We have referred already to the need to ob- advance is not entirely accidental. Instead the tain new types of quantitative and qualitative need, or demand, for some technological so- data as well as to assemble and use the data lution increases the likelihood of it occurring. that currently exists effectively. New meth- Economists, historians, and others are study- ods will be required for acquiring, accessing, ing how successful technologies have gone evaluating, and integrating these data. We through stages of discovery, innovation, early need efficient search and summary technolo- adoption by particular groups, and ultimately gies; the ability to draw on data for estima- to market transformation. Empirical evidence tion, including data coded and compatible continues to be collected on these stages of with geographic information system (GIS) innovation. Innovation results in more avail- technology; and the ability to compare actual able technology options and ultimately shifts and simulated data rapidly. Again, a problem sectors’ production possibility frontiers in facing modelers of socioeconomic processes economic models. is the computational burden of applying the best statistical methods to existing data. This Once we have an economic model specifica- problem has led econometricians to search for tion of induced technical change, which may “computationally light” methods. The best push the boundaries of performance potential, statistical methods applied to large data sets we may need to verify that physical laws of will require exascale computing power and nature are not violated (e.g., mass and energy present novel challenges in the utilization of balance, the second law of thermodynamics). large–scale computer architectures—but will In other words, do engineering configurations produce far superior empirical results. exist consistent with the economists’ models of technology advance? One approach to dealing with uncertainty is to perform multiple ensemble runs (param- Modern finance can also assist with economic eter sweeps) with various combinations of models to represent the willingness of own- the uncertain parameters. Since the space of ers of buildings and production facilities to parameters will be of high dimension, we will retrofit/replace processes with new ultraclean have to address the challenges of designing investments. Modern finance has developed efficient parameter sweep methods for high- an understanding how investment decisions dimensional spaces. Recent advances in ap- tend to be made. proximation theory and data mining methods, On the human dimension, there will be a real such as sparse grids, offer new approaches to challenge to understand, model, and manage

67 Section 4: Socioeconomic Modeling

rapid change that leads not only to major win- We envision the following specific tasks: ners but also to extreme losses. Under historically normal rates of growth, markets have • Construct a comprehensive suite of managed change reasonably well. Over time, models of unprecedented geospatial and “all ships tend to rise”—though not without temporal detail, with comprehensive er- increased inequality and persistent poverty ror analysis on the representation, some on a relative, if not absolute, basis. These based on existing models, some entirely adjustments occur in the context of the usual new. economic forces: comparative advantage, • Leverage state-of-the-art climate model- specialization of production, gains-from- ing activities (e.g., SciDAC) to include trade, and the resulting distribution of income economic prediction models under alter- over nations, population groups, sectors, and native climate regimes. owners of capital/investors. However, it may be possible (or even likely) in a world expe- • Perform basic research into such founda- riencing significant climate change to have a tional issues as spatial statistics, modeling shift in the ordering of fundamental econom- of social processes, relevant micro-activ- ic forces, such as comparative advantages in ity and biosphere coupling issues, and certain production activities, resulting in rap- relevant mathematical challenges, such id and notable losers (e.g., coal mining, island as multiscale modeling. states, hurricane–prone regions, agricultural areas that turn arid, and emergence of certain • Assemble and perform quality control of groups less able to adapt). extensive data collections—much from existing sources, but also much from new 5. Accelerating Development and unconventional sources. A focused DOE R&D program can achieve • Perform comprehensive and detailed significant advances in the state of the art in validation of both individual models and modeling the human elements of the climate large model systems. system. Building on DOE expertise in climate • Develop novel, robust numerical tech- and energy system modeling, and bringing to niques and high-performance comput- bear the latest methods in economics, quan- ing approaches to deal with the expected titative techniques in behavior and decision orders-of-magnitude increase in model theory, and modeling and HPC tools, such a complexity. program can develop a deepened understanding of the technical, economic, political, and • Conduct a wide range of application social issues that underpin the climate change studies aimed at both validation and challenge. application.

Achieving this goal will require a sustained, • Develop education programs aimed at large-scale program aimed at applying com- training the next generation of computa- putational science methods, with the objec- tional economists and other social scien- tive of creating, within a decade, the tools tists, including not only formal training and methodologies needed for the quantita- programs but also web-based modeling tive study of questions introduced earlier. and simulation tools that allow wide- This program should aim to create a highly spread access to the new models and their integrated, time-dependent modeling system results. that encompasses detailed micro socioeco- A focused R&D program in nomic data and models in a comprehensive, Partnerships with international scientists and socioeconomic modeling must organizations will be important to develop include basic research in integrated global framework. This system the global-scale models and obtain the data foundational issues such as will allow for treatment of socio-demograph- spatial statistics, multiscale ic groups and the marketplace, including needed for realistic assessment. modeling, and coupling producers, consumers, and intermediaries, in of micro-activity and the unprecedented detail. biosphere.

68 Modeling and Simulation at the Exascale for Energy and the Environment

Figure 4.6 illustrates some of the advances that we expect as we move to future petascale and exascale computers.

Further study is required to determine the best organizational structure for such a program. We outline here one approach that we feel holds promise. In this approach, we define, as one dimension of the organizational matrix, five teams, each focusing on solving problems and issues in a specific area (and also interacting and sharing information and progress):

1. Modern economic computation approaches, involving greatly expanded micro databases to capture diversity Figure 4.6 Anticipated model improvements resulting from a major DOE program in in decision-making agents and strategic detailed global socioeconomic modeling. incentives that different country players may exercise. 6. Expected Outcomes 2. Consistent socioeconomic modeling with many regions/sectors and with technol- A focused DOE program in this area is ex- ogy rich specifications that capture tech- pected to achieve a significant improvement nology-system efficiencies, synergies, in understanding important questions relating and diversities and represent least-cost to socioeconomic aspects of climate change. dynamic transition pathways to a low- This improved understanding can improve carbon, adaptive global economy the effectiveness of U.S. and international policy responses to climate change, the ability 3. Social science modeling to capture the of U.S. industry to engage effectively in de- human dimension of climate-induced veloping and “productizing” climate change changes in greenhouse gas emissions, solutions in the areas of low-carbon energy mitigation, and adaptation supply and demand, and the ability of the United States to minimize the vulnerability of 4. Integrated assessment, bringing in cli- its economy and energy infrastructure. mate and natural resource models (water, agriculture, land use, forest changes, DOE efforts in this area will also have a biodiversity, health and disease) and their significant impact on the development of coupling to socioeconomic models computational tools, expertise, and understanding in economics and social sciences as 5. Advanced optimization and computa- a whole. Other anticipated results include the tion methods needed for integrated as- following: sessment, solving large-scale nonlinear systems, and addressing approaches for accounting for uncertainty • Socioeconomic models will become major users of DOE supercomputers, both As progress is made in all five of these areas, individually and within integrated mod- and as the various modeling teams better un- els linking socioeconomic models and derstand how to consistently couple their mod- ESMs. eling areas together, we will move to a common large-scale integrated model system. • DOE efforts in the simulation of energy production, distribution, and use will be accomplished via exascale computational platforms.

69 Section 4: Socioeconomic Modeling

A pilot study would help • Various DOE-centric economic climate- a range of plausible alternative scenarios demonstrate the viability of energy scenarios will be provided to and search for policies that are robustly merging socioeconomic theory policy makers. successful. Furthermore, any analysis must within Earth system model consider how human institutions and agents frameworks and of simulating will react to those uncertainties. testable hindcast economic • DOE will become a place to which other responses to climate variations. parts of the government go to increase As with any cutting-edge simulation, there is understanding of these issues and rel- a need to clearly portray the uncertainties and evant policy information. associated levels of credibility. A pilot phase would help to demonstrate the viability of 7. Required Investment merging socioeconomic theory within ESM The creation of an integrated global socio- frameworks, simulating testable hindcast economic model, the validation of model economic responses to climate variations, components, and applications to the under- and applying testable hypotheses on the ex- standing of human actors and their coupling pected outcomes from a new human/climate to the climate system involve a massive grand Earth system approach. challenge, requiring an interdisciplinary team The most compelling reason to create and nur- of arguably unprecedented scale and scope. ture a program such as that described here is A rough estimate of the required resources the risk of not acting. The challenge of merging would be $100M/year over 10 years for hu- the diverse economic and geophysical models man resources—in addition to the hardware is significant but must be addressed if we are to resources needed for storage and simulation. understand and predict future socioeconomic This work would be organized and divided systems in a warming world. among several institutions, representing the References multiple disciplines and expertise involved. Regular communication, coordination, and H. M. Amman, D. A. Kendrick and J. Rust, eds. publication of interim results would be (1996), Handbook of Computational Economics, required. vol. 1. Elsevier.

Fortunately, there already exist years of accu- A. Dixit and R. Pindyck (1992), Investment Under mulated research experience and preliminary Uncertainty. MIT Press. model development activities upon which to J. Edmonds, M. Wise, H. Pitcher, R. Richels, T. draw. Wigley and C. MacCracken (1997), An integrated assessment of climate change and the accelerated 8. Major Risks introduction of advanced energy technologies: An application of MiniCAM 1.0, Mitigation and The proposed activity represents a signifi- Adaptation Strategies for Global Change 1(4) cant departure from current practice in the 311–339. climate change and socioeconomic modeling communities. EMF Study-21 (2006), Special issue on multi- greenhouse gas mitigation and climate policy, Perhaps the biggest challenge is that of EMF Study 21. The Energy Journal. uncertainty. The ESMs used to quantify B. C. English, D. G. De La Toore Ugarte, K. climate change benefit from large quantities Jenson, C. Hellwinckel, J. Menard, B. Wilson, of data and (mostly) well understood physical R. Roberts and M. Welsh (2006), 25% renewable laws. Yet despite enormous complexity and energy for the United States by 2025: Agricultural large amounts of computing power, their and economic impacts. Department of Agricultur results exhibit considerable uncertainty, al Economics, University of Tennessee. particularly when it comes to regional effects. Human factors are in many regards less well S. W. Hadley, D. J. Erickson III, J. L. Hernandez, C. T. Broniak and T. J. Blasing (2006), Responses understood and are heavily parameterized of energy use to climate change: A climate model with empirical relationships. In both cases, ing study. Geophys. Res. Lett., 33 (L17703). the only responsible approach is to examine

70 Modeling and Simulation at the Exascale for Energy and the Environment D. A. Hanson and J. A. Laitner (2006), Technol B. C. Murray, A. J. Sommer, B. Depro, B. L. ogy policy and world greenhouse gas emissions in Sohngen, B. A. McCarl, D. Gillig, B. D. Angelo the AMIGA modeling system, The Energy Jour and K. Andrasko (2005), Greenhouse gas mitiga nal (Special Issue on Multi-Greenhouse Gas Miti tion potential in US forestry and agriculture. EPA gation and Climate Policy). Report 430-R-05-006.

I. Held and B. Soden (2006), Robust responses of NAPAP Integrated Assessment Report. National the hydrological cycle to global warming. Journal Acid Precipitation Assessment Program (NA of Climate 19:5686–5699. PAP), Office of the Director, 1990.

IEA (2007), International Energy Agency. World The National Energy Modeling System: An over Energy Outlook, www.worldenergyoutlook.org. view (2003). Energy Information Administration, U.S. Department of Energy. IMAGE 2.2 Model Flow Diagram (2007), www. mnp.nl/image/model_details. A. P. Sokolov, C. A. Schlosser, S. Dutkiewicz, S. Paltsev, D. W. Kicklighter, H. D. Jacoby, R. G. IMAGE team (2001), The IMAGE 2.2 implemen Prinn, C. E. Forest, J. Reilly, C. Wang, B. Felzer, tation of the SRES scenarios: A comprehensive M. C. Sarofim, J. Scott, P. H. Stone, J. M. Melillo analysis of emissions, climate change and impacts and J. Cohen (2005), The MIT Integrated Global in the 21st century, RIVM CD-ROM publication System Model (IGSM) version 2: Model descrip 481508018, National Institute for Public Health tion and baseline evaluation. MIT. and the Environment, Bilthoven, the Netherlands. M. L. Stein (1999), Statistical Interpolation of IPCC (2007), Intergovernmental Panel on Climate Spatial Data: Some Theory for Kriging. Springer, Change (IPCC) 4th assessment report, www.ipcc. New York. ch, 2007. Sustainable Bioenergy: A Framework for Deci K. Judd (1998), Numerical Methods in Econom sion Makers (2006), UN Energy. ics, MIT Press. M. Q. Wang (2001), Development and use of J. A. Laitner and D. A. Hanson (2006), Modeling GREET 1.6 fuel-cycle model for transportation detailed energy-efficiency technologies and tech fuels and vehicle technologies (2001), Center nology policies within a CGE framework, The En for Transportation Research, Argonne National ergy Journal (Special Issue on Hybrid Modeling Laboratory. of Energy-Environmental Policies: Reconciling Bottom-up and Top-down).

A. J. McMichael, D. H. Campbell-Lendrum, C. F. Corvalán, K. L. Ebi, A. K. Githeko, J. D., Scheraga and A. Woodward, eds. (2003), Climate change and human health: Risks and Responses. World Health Organization, Geneva.

71 Section 4: Socioeconomic Modeling 5 Astrophysics

To understand our universe and our place in it energy is the vacuum energy field responsible will require that we understand the universe on for the accelerating rate of cosmic expansion. all scales, from the evolution of the universe Understanding the nature of dark energy has as a whole and the formation of its largest- been declared the most important and funda- scale structures to the formation and evolution mental problem in the physical sciences. of compact objects, only one one-hundredth the size of Earth but with mass greater than Funding agencies are currently considering our own sun. The largest scale structures in several major ventures—the NASA/DOE/ the universe and compact objects differ in NSF Joint Dark Energy Mission (JDEM), the size by 22 orders of magnitude. Understand- Large Synoptic Survey Telescope (LSST), ing the universe on each scale presents exas- and the Square Kilometer Array (SKA), each cale computing challenges; while exascale in the range of $0.3 billion to $1 billion—as computing will allow increased connections well as a variety of lower-cost pathfinder mis- to be made between scales (e.g., connections sions. The generation of mock catalogs from between large-scale structure formation and petascale cosmological simulations will be galaxy formation), beyond exascale comput- used to demonstrate the feasibility of these ing lies the challenge of tying these scales missions. If one or more of the missions are together more broadly in order to develop a funded, exascale simulations will be needed seamless description that can take us through in the next decade to help pull out the dark the history of the universe, from the Big Bang energy parameters from the observations. to the development of conscious life peering 1.2 Galaxies back on this unfolding history. From collections of galaxies we move to the 1. State of the Art scales of individual galaxies and their interactions. One of the most fundamental questions Our discussion of the state of the art in simu- about the universe—What is the nature of its lation of the universe covers all scales: from major constituents?—remains shrouded in large-scale structure to galaxies to stars and mystery. Visible matter is only 4% of the con- finally to compact objects. tent of the universe. This situation poses great 1.1 Large-Scale Structure challenges for the computations required to constrain dark matter and dark energy prop- We begin with the challenge to simulate and erties from the observed distribution of gal- understand the formation of the largest struc- axies. Simulations to follow the formation tures in the universe, the lacelike structures of our galaxy in sufficient detail to compare that thread the universe and are composed of with observational data from the James Webb clusters and “superclusters” (clusters of clus- Space Telescope (JWST) and LSST will re- ters) of galaxies (see Figures 5.1 nd 5.2). The quire dynamic ranges of order 10,000 in space formation of these structures in the early uni- and time. A simulation with enough resolu- verse is inexorably tied to the evolution of the tion to accurately model the visible properties Understanding the universe — from the smallest objects to universe as a whole, which in turn depends on of individual galaxies and yet contain a fair those 22 orders of magnitude the nature of the mysterious dark energy that sample of the universe suitable for compari- larger — requires exascale permeates it [Hu and Dodelson, 2002]. Dark computing, and more.

73 Section 5: Astrophysics

son with large surveys will require exaflop- AGN, the fact that AGNs are rare today is interpreted to mean that most black holes are no longer accreting material. This interpretation further suggests that AGN activity may be closely tied to the assembly of galaxies. AGNs within clusters of galaxies provide a particularly revealing look at how AGNs interact with their host environments. Such AGNs create enormous cavities ~30,000 light years in size.

Computational study of the AGN-environment interaction is at present limited to simulations that cover only a small portion of the dynamical range and crudely model the effects that arise outside that range. Questions that we would like to answer include the following: What turns AGNs on and off? How do AGNs affect the physical state of the intracluster medium? What role do AGNs play in galaxy and cluster assembly? How do the central black holes form in the first place?

The convergence of these observational data- Figure 5.1 Structure formation in the Santa Fe Light Cone [Hallman et al. 2007]; 100 square degrees of sky incorporating 700 megaparsecs were simulated with the sets at the same time that exascale computing ENZO code. makes it possible to elaborate our theoretical predictions for this problem promises to significantly advance our understanding of the weeks of computing. astrophysical questions posed above.

An understanding of individual galaxies can- 1.3 Stars not be obtained, of course, without an understanding of their cores. About 1% of galaxies We owe our existence to stars for a number of in the local universe host powerful central ra- reasons, not the least of which is that our own diation sources known as active galactic nu- sun is a star. The elements necessary for life clei (AGNs). These sources produce emission are made in stars during their lifetimes and across a wide range of photon energies (from during their deaths as supernova explosions. radio to X-rays) and are usually quite vari- It is through violent phenomena such as dra- able in their output. They are often associated matic stellar outflows during a star’s life or with bidirectional outflows of material mov- supernovae that these elements are dispersed ing at relativistic velocities. AGNs appear to into the interstellar medium, peppering the have been much more common several bil- soup from which our solar system formed billion years ago, when galaxy formation was at lions of years ago. its peak. In particular, half of the elements heavier than The most widely accepted model for AGNs iron are made in stars through what is called proposes that they are powered by accreting the s-process (slow neutron capture process). supermassive (more than 100 million solar These elements are then injected into the in- masses) black holes surrounded by obscur- terstellar medium through the expulsion of ing tori of dust and gas [Lynden-Bell 1969]. the outer layers of the star, forming planetary Since spiral galaxy rotation curves suggest nebulae. Both the production and transport of that most galaxies harbor such black holes these heavy elements are sensitive to 3D ef- at their centers, whether or not they host an fects, including the turbulent mixing of stably

74 Modeling and Simulation at the Exascale for Energy and the Environment stratified material into the convection zone, stellar explosions known as core-collapse the dredge-up of processed material into the supernovae [Mezzacappa 2005; Janka et al. envelope, and the global convection in the 2006]. Such supernovae are an important outer envelope of the star and the expulsion link in our chain of origin from the Big Bang of this envelope. to the present day. Understanding how they occur is key to understanding how we came The cause of the repeated envelope expulsion to be in the universe. They are the dominant in stars is thought to be the periodically oc- source of elements in the periodic table be- curring flashes of the helium-burning shell tween oxygen and iron, and there is growing within the central, Earth-sized core of the evidence that they are indeed responsible for star. These can be likened to a global storm, producing half of the elements heavier than and the full sequence of helium shell flashes iron (the other half coming from the s-process can be likened to stellar climate. With petas- in stars, discussed above). Moreover, they are cale computing, high-quality simulations of the most energetic explosions in the universe, portions of the space-time domain for one and there is now an indisputable connection such helium shell flash could be performed. between “peculiar” hyper-energetic core- With exascale computing, we may be able to collapse supernovae, also known as “hyper- simulate the entire helium shell flash, with novae,” and one of two classes of gamma its 2-year duration, using validated statistical ray bursts (GRBs) in the universe [Woosley models of phenomena operating on smaller and Bloom 2006]. Both phenomena occur length and time scales. Exascale computing under a common umbrella of massive stellar will allow full-scale simulation with valida- core-collapse. And core-collapse supernovae tion quality for the entire helium shell con- are among the events expected to produce vection zone for time scales of hours. gravitational waves—ripples in the fabric of

Aside from understanding the complete phe- nomenology of thermonuclear supernovae as a goal in itself, the importance of a complete picture of the explosion mechanism extends to a fundamental question in cosmology as well: What are the nature and effect of dark energy? Only when we fully understand the explosion mechanism can we validate the use of Type Ia supernovae as “standard candles” for use in measuring the size and expansion history of the universe. Current Type Ia surveys rely on purely empirical corrections to determine the intrinsic brightness of observed supernovae. As observations extend to higher and higher redshifts, we must understand how evolutionary effects (e.g., metallicity) might contribute to the intrinsic brightness of the events. This level of calibration is required to make use of data to be obtained by JDEM, which is designed to detect thousands of Type Ia supernovae and thereby put real constraints on the equation of state of the mysterious dark energy that appears to make up 70% of our universe. Figure 5.2 Simulated X-ray observation of a simulated merger between clusters Massive stars more than ten times the mass having a mass ratio of 1:3. Contours indicate X-ray surface brightness, while the of our sun evolve for millions of years and colors indicate X-ray temperature, with blue indicating cooler gas and red indicating then die in a matter of hours in spectacular hotter gas. (P. Ricker)

75 Section 5: Astrophysics

Figure 5.3 Development of the computationally discovered (SciDAC Terascale Supernova Initia- tive) instability of the core-collapse supernova shock wave, shown in a snapshot from a 3D simulation by John Blondin (NCSU) and Anthony Mez- zacappa (ORNL). The visualization was performed by Kwan-Liu Ma (UCD). The instability leads to growing deformations (away from spherical) of the shock wave, which is represented by the surface in this image. The deformations in turn lead to circulating flow below it. Two strong, counter- rotating flows are formed. Streamlines in this image highlight one flow moving clockwise just beneath the shock surface and a second, deeper flow moving counter clockwise just above the pro- to-neutron star surface. Moreover, the inner flow is capable of spinning up the proto-neutron star, perhaps explaining the origin of pulsars [Blondin and Mezzacappa 2007]. The spins generated in these simulations are consistent with observations of young pulsars.

space—in the galaxy, the detection of which ements in supernova ejecta, data which must will mark a historic event. be explained by explosion models.

The temperatures to which the core-collapse Observations of gamma-ray lines are an supernova shock wave (see Figure 5.3) heats excellent way to determine not just the el- the ejecta from such supernovae make ul- emental but also the isotopic production of traviolet observations a very useful way of supernovae. They are therefore of particular studying the enriched composition of recent interest, both to supernova theory in general ejecta. Examples include observations per- and to supernova nucleosynthesis. Observa- formed in the past with the International Ul- tions such as the Compton Gamma-Ray Ob- traviolet Explorer (IUE), more recently with servatory (CGRO) measurement of the ratio the Hubble Space Telescope (HST), or cur- of 57Ni to 56Ni in SN1987A or the detection of rently with the Far Ultraviolet Spectroscopic 44Ti in the Vela and Cassiopeia A SN remnants Explorer (FUSE). Indeed one objective of the probe the deepest layers of the supernova FUSE mission is “studies of nova and super- ejecta, just above the mass cut. Only the neu- nova explosions and their remnants, to test trino and gravitational wave signals, which theories of heavy element nucleosynthesis follow the evolution of the proto-neutron star, and study how supernova shock waves heat provide information from deeper within the the interstellar gas.” With missions such as explosion. Therefore, gamma-ray line obser- NASA’s SWIFT GRB Mission, these observations, past (CGRO) and future (Integral, vations can be extended to include the X-ray the International Gamma-Ray Astrophysics region. Furthermore, observations of super- Laboratory), coupled with supernova models nova remnants, such as the Chandra X-Ray that include realistic nucleosynthesis studies, Observatory observations of Cassiopeia A will greatly constrain explosion models. and SN1987A, also provide detailed data on the composition and distribution of heavy el-

76 Modeling and Simulation at the Exascale for Energy and the Environment

1.4 Compact Objects rent matter simulations in numerical relativity are beginning to add microphysics such as Neutron stars and stellar mass black holes are realistic equations of state, magnetic fields, the remnants of the death throes of massive photon and neutrino radiation transport, and stars, discussed above. They are among the nuclear networks. Exascale computers will be smallest objects in the universe but are ex- needed to describe with accuracy systems as amples of nature at its most extreme. They physics-rich as BHNS and BNS. These simu- are important in their own right, as key com- lations are essential for the understanding of ponents of some astrophysical systems, such matter at extreme densities. as binary systems comprising a star and neutron star or two neutron stars, and as labora- 2. Major Challenges tories for fundamental physics. Neutron stars are approximately 10–15 km in radius, with Given the need to resolve all relevant scales a mass approximately one to two times the in the astrophysical environments described mass of our sun. in section 1 and the ability to do so with the promise of exascale computing, given the in- We stand at the threshold of the birth of a new creasing use of adaptive mesh refinement in type of astronomy: gravitational wave astron- these simulations, and given the generally omy. Observatories such as the NSF-funded long run times anticipated to cover the evolu- Laser Interferometric Gravitational Wave tion of the above systems in both space and Observatory (LIGO) are poring through the time, two issues emerge: sky in search of gravitational waves from astrophysical sources. Three key sources have • An increased need for dynamic load bal- been identified: the merger of two neutron ancing and smart algorithms to provide stars in orbit around one another, the merger this capability of two black holes in orbit around one another, and core-collapse supernova explosions. • An increased need for fault tolerance, All three are expected to be seen by LIGO for that is, for both efficient parallel algo- galactic events. rithms that minimize the time to solution and thereby mitigate this issue and fault- In the past few years, the community has tolerant solution algorithms. produced the first stable and accurate simulations of binary black holes (BBHs) [Baker et The data anticipated from both observational al. 2006; Campanelli et al. 2006]. However, facilities and computational simulations will the complexity of these simulations increases be enormous. For example, 100 PB of data with the addition of matter (i.e., nonvacuum are expected from the LSST, and simulations scenarios). Binaries composed of a black hole of core-collapse supernovae at the exascale and a neutron star (BHNS) or two neutron are expected to produce ~100 PB of data per stars (BNS) are also of great interest and the simulation over a period of a few months. subject of active study (see Figure 5.4). Cur- The analysis and visualization of these data,

Figure 5.4 Snapshots of the space-time evolution of a neutron star orbiting a massive black hole [Sopuerta, Sperhake, and Laguna 2006].

77 Section 5: Astrophysics

with an eye toward scientific discovery, will 3. Advances in the Next present great challenges. Data analysis algo- Decade rithms will need to share the same efficiency and scalability of the algorithms used in the The computational astrophysics community original numerical simulations that produced has a significant tradition in computing at the simulation data, and given the size of the scale and in using existing architectures to do data sets, the analyses themselves will like- so. Given this history, the overriding sense ly require exascale computing resources, or of the community is that existing approaches at least dedicated computing resources with will scale to the exascale and that exciting significant computational capabilities. The advances in science will be feasible within infrastructure surrounding the exascale plat- the next decade in each of the four areas dis- forms must provide sufficient capabilities for cussed in Section 1: large-scale structure, gal- data archiving, data access, networking, and axies, stars, and compact objects. visualization for both local and wide area networks. This will further accentuate the need 3.1 Large-Scale Structure for fault-tolerant approaches to data storage and access (e.g., multisource, multistream At present, the only means we have to inves- approaches), latency-tolerant approaches to tigate dark energy is to use the new observa- remote visualization, and the like. tional surveys of the universe on a vast scale. The interagency (DOE/NSF/NASA) Dark As the data volume from observational facili- Energy Task Force (DETF) has identified ties expands, the possibilities of integrating four promising observational techniques for “sky truth” into simulations become more and measuring the so-called dark energy equation more appealing. Nevertheless, the consensus of state (the relationship between important suggests that the more likely mode of inclu- quantities that characterize the dark energy, sion will be more widespread construction such as pressure and energy density) [Albrecht of artificial observations—that is, simulation et al. 2006]. Three of the four will require the data convolved with instrument character- guidance and interpretation of cosmological istics to produce a representative notion of simulations of the large-scale distribution of what the simulated object would “look like” galaxies and galaxy clusters of high precision to an observer. This construction has two im- (1%), high physics fidelity, and performed on mediate consequences. First, the data volume an unprecedented scale (a good fraction of the from the simulations actually tends to expand observable universe). In essence, if we are to markedly when subjected to this type of a understand our accelerating universe, we will posteriori analysis, in contrast to more tra- first have to faithfully simulate a good frac- ditional forms of data analysis (often termed tion of it on a computer. While this sounds “reduction”) that produce a more compact set preposterous, it is in fact what is needed to of data. Second, this wedding of data and ex- complement and to derive the benefit from perimental uncertainty is the nexus in which the observational surveys that will likely be validation is ultimately performed for astro- moving forward. physical simulation. As it is the observations that will tell us whether our simulations solve 3.2 Galaxies the proper set of equations (i.e., the equations that describe nature in the settings we investi- JWST is the designated successor to the HST gate), having a complete view of the interplay as a “great observatory.” It was the astronom- of observer and observed phenomena is cru- ical community’s top priority in the last dec- cial to actually realizing the goal of validated adal survey and has an estimated cost of $4.5 astrophysical simulation. billion. It is specifically designed for understanding the formation of the first stars and If we are to understand our galaxies, measuring the geometry of the uni- accelerating universe, we must verse and the distribution of dark matter, and first faithfully simulate a good investigating the evolution of galaxies. Com- fraction of it computationally. parison with theoretical models will require

78 Modeling and Simulation at the Exascale for Energy and the Environment simulations with enough fidelity to match the evolved neighbor. This increase in mass can Three-dimensional simulations JWST observations of the earliest galaxies as lead to an uncontrolled thermonuclear explo- of neutrino transport equations they form to the objects they become today. sion, completely disrupting the white dwarf in already require petascale a cataclysm visible across most of the observ- computing; six-dimensional simulations, needed to The LSST is another national priority, with able universe. The details of the explosion accurately model supernova an estimated cost of $300 million. It will con- mechanism of these thermonuclear supernovae (identified with the spectroscopic Type explosions, will require exascale strain the nature of dark energy by following computing. the evolution of structure, and the nature of Ia label) are still an unsolved problem. The dark matter by mapping strong gravitational interplay of rapid nuclear burning and strong lensing in clusters of galaxies. gravity in the interior of the white dwarf results in a combustion problem that requires Both of these observational programs require the resolution of flamelets roughly the width theoretical predictions about the clustering of of a finger while simulating [Gamezo et al. dark matter and the distribution of galaxies in 2005; Calder et al. 2007] an object the size of order to fully constrain the dark energy/dark the Earth. Ultimately, such simulations must matter models. also include a detailed understanding of the nuclear species formed in the event and their For the timescale on which exascale computing spectroscopic signatures. Exascale computing is expected to become a reality (2015–2020), will enable simulations with resolutions down several exciting observational developments to the Gibson scale (the length scale where promise to dramatically enhance our knowl- turbulent motion is effectively smoothed by edge of the structure of AGNs and the proper- the propagation of the nuclear flame) with ties of their environments. In the X-ray, the definitive prescriptions for nuclear energy re- Constellation-X telescopes will provide spec- lease and the associated nucleosynthesis. tra and images of AGNs and cluster cores that will allow us to study the dynamics of Core-collapse supernovae are driven in part, the innermost accretion disks of AGNs and or perhaps largely, by an intense flux of radia- the structure of the intracluster medium. In tion in the form of nearly massless particles the radio, SKA and the Enhanced Very Large known as neutrinos that emerges from the Array (EVLA) will produce sensitive maps proto-neutron star at the center of the explo- of radio emission from the enormous cavi- sion. The enormous energy contained in the ties produced in clusters with AGNs, provid- neutrino radiation field must be modeled ac- ing detailed information about the relativistic curately in order to model the much less ener- plasmas and magnetic fields that fill them. getic phenomenon of the supernova explosion In the optical, LSST will produce numerous itself. This modeling in turn will require the (more than 10 million objects) samples of solution of the six-dimensional (6D) neutrino AGNs, stretching through most of the epoch transport equations, the solution of which of galaxy formation and telling us how the will give the distribution of neutrinos in angle AGN-environment interaction has shaped the of propagation (two dimensions) and energy luminosity and variability of AGNs during (one dimension) at each 3D spatial location. this critical period. The coming generations Petascale platforms are already required for of very large optical telescopes, such as the 3D simulations with multiangle, multienergy Giant Magellan Telescope (GMT), will also neutrino transport at moderate resolution. make it possible to image the central regions Simulations with the spatial resolution re- of nearby AGNs with a resolution hitherto in- quired to properly model other critical aspects conceivable. of the explosion dynamics—for example, the evolution of the stellar core magnetic fields 3.3 Stars and their role in generating the supernova— will require much higher resolution, which Stars like our own sun end their lives as white in turn will require exascale computing, par- dwarfs: compact remnants held up against ticularly if a number of simulations are to be their own self-gravity by electron degeneracy performed across the range of stellar progeni- pressure. If the white dwarf is in a binary, the tors and input physics. One such simulation companion star can shed mass onto its more

79 Section 5: Astrophysics

is expected to take ~8 weeks, assuming 20% holes found at the center of galaxies, where efficiency on an exaflops machine. the inspiral dynamics are bound to be affected by the presence of accretion disks and galactic In the future, the enhanced spectral resolu- environments. These simulations will (one tion and throughput of missions like Constel- hopes) shed light on the origin and dynamics lation-X will provide data from more distant of galactic jets and quasar evolution. (and therefore more numerous) supernovae. Infrared observations of heavy element abun- Among the driving forces behind simulations dances, like those performed in the past with of compact object binaries are the new gen- the Infrared Space Observatory (ISO) and in eration of laser-interferometric gravitational the present with the Spitzer Observatory, and wave observatories such as LIGO, VIRGO, those possible with future spectroscopic mis- and LISA and future missions such as NASA’s sions like JWST, can report the temporal his- Black Hole Finder Probe and Imager, and the tory of element production from supernovae. associated large investment that has been These observations, coupled with theoretical made. One expected outcome of a simulation study of the ways the earliest supernovae differ is the computation of the gravitational waves from those of the present epoch, are important produced during the binary merger. Without to understanding how the first stars formed and the gravitational wave “templates” from nu- how they changed over time into the objects merical relativity, detection and characteriza- recognized in the present universe. tion of gravitational radiation sources will be extremely difficult. The coincidence of several GRBs with core- collapse supernovae has opened a new chap- In addition, binaries with at least one neu- ter in the study of supernovae. Time-sensitive tron star are the most likely engine of detections, such as the HETE-II observations short-duration GRBs (the other class of GRBs of GRB030329 (which led to the discovery in the universe; long-duration GRBs are asso- of Supernova 2003dh), and Swift’s detection ciated with core-collapse supernovae as dis- of GRB060218, as well as those that may be cussed above). Their modeling is crucial for provided in the future by missions like the the interpretation of data from current mis- Energetic X-ray Imaging Survey Telescope sions such as Swift and future probes such as (EXIST), will enhance the opportunity for si- NASA’s Gamma-ray Large Area Space Tele- multaneous detailed multiband observations scope (GLAST). from missions such as Integral, Chandra, and HST. These will shed light on the GRB/su- 4. Accelerating Development pernova connection by greatly improving the quality and quantity of data available on these Exascale simulation of the universe on all events. scales will require a variety of simulations that will be performed using a variety of codes 3.4 Compact Objects founded on different numerical methods. The community has produced codes that solve While simulations of BBHs are progressively the equations of N-body dynamics, hydrody- becoming more accurate and efficient, some namics, MHD, radiation transport, radiation aspects of these calculations are outside the hydrodynamics, radiation MHD, and both realm of current computational resources. Newtonian and general relativistic gravity. The advent of exascale computing resources The underlying methods have thus far proven will allow a wider coverage of BBH param- robust in scaling to present-day architectures. eter space, determined by the masses, spins, The underlying assumption, and hope, of the and orbit eccentricities. It will also allow for computational astrophysics community is that the simulation of BBHs with extreme mass these methods can be extended to the exascale Modeling of long-duration ratios (smaller than 1/20). without significant modification. Of course, gamma-ray bursts is crucial for the community is aware that, in some cases, interpreting data from NASA The addition of matter will also improve the new algorithms for the underlying PDEs will missions. realism of simulations involving supermassive be needed. Moreover, modifications will

80 Modeling and Simulation at the Exascale for Energy and the Environment have to be made to accommodate the increas- Area Science ing numbers of cores on each socket and the changes in the demand for memory bandwidth Large-Scale Simulations of the large-scale distribution of galaxies and associated with this new feature, but there are Structure Formation galaxy clusters over a large fraction of the observable no obvious bottlenecks at present that suggest universe with 1% precision, required of the observational that an entirely new set of codes will have to program proposed by the DOE/NASA/NSF-sponsored be deployed. The community also recognizes Dark Energy Task Force. that this hope is predicated upon the hope that the architectures at the exascale will not be Galaxy Formation Simulations of galaxy formation with sufficient resolution dramatically different either from those used to predict the observed properties of individual galaxies now or from those that will be used in the in a volume containing a sufficient fraction of the ob- near term at the petascale. Thus, of primary servable universe to compare with large-scale surveys. importance to the computational astrophys- Simulations of the formation of the Milky Way galaxy with ics community will be collaboration with the sufficient precision to compare with data from JWST and applied mathematics community with an eye LSST. toward porting existing methods and codes to exascale platforms.

The ability to perform simulations at much Stellar Evolution Simulations of the entire stellar envelope in AGB stars, higher resolution will be of great benefit to responsible for the supply of half of the heavy elements the computational astrophysics community. (elements above iron) in nature. Nonetheless, given the multiscale nature of the astrophysical systems studied by this com- Supernovae Definitive 3D multiphysics simulations of core-collapse munity, adaptive mesh refinement (AMR), supernovae, the dominant source of elements between which is becoming more prevalent now, will oxygen and iron and the half of the heavy elements not no doubt see increased use in the future. AMR produced in AGB stars. can be performed in different ways (e.g., grid-based or cell-by-cell refinement), each having its pros and cons. Thus, we will have Compact Objects Definitive simulations of binaries involving two neutron to consider both the scaling of our numerical stars, or one black hole and one neutron star, which are solution methods, as discussed above, and the among the leading candidates for the production of gravi- scaling of our approach to AMR. The primary tational waves in our galaxy. issue with regard to AMR and parallel computing is, of course, load balancing. Thus, the development of methods for dynamic load Table 5.1 Examples of exascale-computing-enabled science balancing across large numbers of processors will be an increasingly important need of the mentioned above to follow the evolution of computational astrophysics community, re- astrophysical systems. The confluence of gardless of the approach to AMR. higher grid resolution, or a larger number of particles, and longer physical simulation Exascale computing will also provide the times will push explicit methods to their lim- ability to simulate astrophysical systems for its. In some instances, explicit methods will significantly longer physical evolution time no longer be viable. Thus, the development scales. In short, we can expect significantly of implicit (in time) methods for the solution longer run times at the exascale. The devel- of some of the underlying PDEs listed above opment of fault-tolerant solution algorithms will be required. and efficient parallel solution algorithms that minimize the time to solution will be required The computational astrophysics community to mitigate the need for fault tolerance. is already using implicit (in time) methods to solve the systems of equations governing Longer run times will also present a very radiation transport in astrophysical systems. different challenge. In many instances, ex- Such methods lead to a set of underlying plicit (in time) methods are used in the codes large, sparse linear systems of equations,

81 Section 5: Astrophysics

the context of the science delineated above? Proposed Observatory Estimated Cost “Launch” Date First and foremost, exascale computing will JWST $4.6B 2013 enable 3D modeling across the phenomena we have discussed. At present, the ability to LISA $1.7B 2015 perform complete simulations in three spatial Constellation-X $1.7B 2017 dimensions is not a given. For example, no 3D multiphysics simulations of core- LSST $300M 2017 (ground-based) collapse supernovae have been performed JDEM >$0.6B 2015 to date. The current state of the art remains at two spatial dimensions. Second, exascale Table 5.2 Scheduled investments in observatories during the next decade computing will mean higher grid resolution which at present are solved with both matrix- or an increased number of particles. Third, based and matrix-free implementations of exascale computing will enable the study of Newton-Krylov (NK) methods. These linear larger physical volumes — for example, the systems can be ill conditioned, which has mo- entire sky rather than a slice of the sky in tivated a need for higher-precision arithmetic simulations of large-scale structure formation and for the development of the associated so- or the entire convective stellar layer in AGB lution algorithms. stars vs only a portion of it. Finally, exascale computing will enable complete multiphysics Throughout the scales of the universe, gravity simulations. Almost without exception, is (of course) prevalent in astrophysical sys- simulations performed to date across the tems. The gravitational field, in conjunction suite of areas discussed above have left out a with the energy (mass) content that defines significant physical component. it, is among the major components of most astrophysical systems/models. In the New- Our ultimate goal is to understand the formation tonian case, the gravitational field is deter- and evolution of the major constituents of our mined by solving the Poisson equation. In the universe, from the largest to the smallest of general relativistic case, the complete system scales, from the universe as a whole to the of Einstein’s equations (or a good approxi- clusters of galaxies making up its large-scale mation thereof) needs to be solved. In both structure to individual galaxies and stars to instances, scalable methods for the solution compact objects such as neutron stars and of elliptic systems of equations on exascale black holes, and, when possible, to understand platforms will be needed. the connection between phenomena at different scales. 5. Expected Outcomes 6. Required Investment Table 5.1 lists examples of astrophysical science that will be enabled by the advent of ex- As shown by a few representative examples ascale computing. in Table 5.2, over the next decade significant investments will be made to develop and Computational astrophysics has a long and deploy the next-generation ground- and space- storied tradition of driving advances both in based observatories that will provide more computing, per se, and in algorithmic devel- complete and more precise observations of the opment. For these reasons and others, com- universe across the electromagnetic spectrum, putational astrophysics has also proven to be and in other forms of radiation such as a fertile training ground for computational gravitational waves and neutrinos, in regions scientists who ultimately spend most of their currently observed. Equally important, theses professional lives in other disciplines. The instruments will allow us to peer farther into need for broadly trained computational scien- space and farther back in time to regions of the tists will only increase as the ambitious aims universe and its history heretofore unexplored. Exascale computing will enable outlined here are undertaken. The success of these new observatories, and study of the entire sky, rather the benefit of the significant investments that than merely a slice of the sky at What will exascale computing mean to the will be made to make these observatories a a time. computational astrophysics community in reality, will depend on the development of

82 Modeling and Simulation at the Exascale for Energy and the Environment more complete and more precise simulations of the Dark Energy Task Force, arXiv.org:astro- Without the development of of phenomena in the cosmos. Without a ph/0609591. exascale platforms and without significant investment in computational the associated development John G. Baker, Joan Centrella, Dae-Il Choi, astrophysics and the development of the of applied mathematics Michael Koppitz and James van Meter (2006), and computer science, the computational platforms, applied mathematics, Gravitational wave extraction from an inspiraling benefits of investment in and computer science on which simulations by configuration of merging black holes. Phys. Rev. new observations will not be the computational astrophysics community Letters 96:111102. will depend, the benefit of investments in new realized. observatories will not be harvested. J. M. Blondin and A. Mezzacappa (2007), Pulsar spins from an instability in the accretion shock of 7. Major Risks supernovae, Nature 445:58–60. The computational astrophysical community A. C. Calder, D. M. Townsley, I. R. Seitenzahl, F. Peng, O. E. B. Messer, N. Vladimirova, E. F. has identified the following key risks in em- Brown, J. W. Truran and D. Q. Lamb (2007), bracing exascale computing: Capturing the fire: flame energetics and neutroni zation for Type Ia supernova simulations, ApJ • The simulations proposed above will put 656:313–332. severe constraints on the memory per processor, memory bandwidth, and total M. Campanelli, C. O. Lousto, P. Marronetti and memory of the new exascale machines. Y. Zlochower (2006), Accurate evolutions of or This situation is largely due to the mul- biting black-hole binaries without excision, Phys. tidimensional (even beyond three spatial Rev. Letters 96: 111101. dimensions, in order to include radiation V. N. Gamezo, A. M. Khokhlov and E. S. Oran angles and energies) and multiphysics na- (2005), Three-dimensional delayed-detonation ture of the phenomena being simulated. model of Type Ia supernovae, ApJ 623:337–346.

• An astrophysical system often has an in- E. J. Hallman, B. W. O’Shea, J. O. Burns, M. herent physical parallelism. Ideally, ar- L. Norman, R. Harkness and R. Wagner (2007), chitectures would be designed to exploit The Santa Fe Light Cone Simulation Project, I: such parallelism by performing the asso- Confusion and the WHIM in upcoming Sun ciated work on a single socket rather than yaev-Zel’dovich effect surveys, ArXiv e-prints across sockets, but this would require 704:0704.2607. sufficient socket memory. Wayne Hu and Scott Dodelson (2002), Cosmic microwave background anisotropies, Ann. Rev. • The increased number of cores per socket Astronomy and Astrophysics 40:171. envisioned for the exascale will likely overwhelm the memory bandwidth to the Hans-Thomas Janka, K. Langanke, A. Marek, G. shared socket memory available to these Martinez-Pinedo and B. Mueller (2007), Theory of cores. core-collapse supernovae, Phys. Rept. 442:38–74.

• Some proposed simulations will be mem- D. Lynden-Bell (1969), Galactic nuclei as col ory bound. For example, core-collapse lapsed old quasars, Nature 223:690. supernova simulations at the exascale will Anthony Mezzacappa (2005), Ascertaining the require approximately 1 exabyte per flop. core-collapse supernova mechanism: The state of A more complete response to the issue of the art and the road ahead, Ann. Rev. Nucl. Part. risk will evolve as detailed knowledge of Sci. 55, 467–515 (2005). exascale architectures comes to light. Carlos F. Sopuerta, Ulrich Sperhake and Pablo References Laguna (2006), Hydro-without-hydro framework for simulations of black hole-neutron star bina Andreas Albrecht, Gary Bernstein, Robert Cahn, ries, Classical and Quantum Gravity 23:S579. Wendy L. Freedman, Jacqueline Hewitt, Wayne S. E. Woosley and J. S. Bloom (2006), The su Hu, John Huth, Marc Kamionkowski, Edward pernova – gamma-ray burst connection, Ann. Rev. W. Kolb, Lloyd Knox, John C. Mather, Suzanne Astronomy and Astrophysics 44:507. Staggs and Nicholas B. Suntzeff (2006), Report

83 Section 5: Astrophysics Math and 6 Algorithms

Advanced and improved simulations for cli- tions will be developed for the coupled mate, renewable energy, nuclear energy, as- systems. In many cases, existing models trophysics, biotechnology, and nanoscience and codes will be extended in terms of (to name a few areas in the DOE portfolio) scalability, and new mathematical ap- demand significant advances in mathematical proaches (general and domain-specific) methods, scalable algorithms, and their imple- to model coupling will be developed. mentations. The required advances are driven New implicit approaches for dealing with by the increased complexity of the problems, long time-scale coupled simulations will involving multiple and coupled physics mod- also be pursued. els, high dimensionality described by large numbers of equations, and huge time and spa- • Uncertainty: To add rigor to exascale tial scales—from nano to global and even to simulation results, we must develop a astronomical scales. systematic approach for quantifying, estimating and controlling the uncertainty These advances will represent a fundamental, caused, for example, by reduced models, exciting, and powerful shift in the compu- uncertain parameters, or discretization tational science modus operandi, enabling a move beyond simply solving larger problems at higher resolutions to providing new capabilities for optimizing and quantifying uncertain systems and unknowns and for assessing risk.

The results of these simulations and their interpretation will help guide high-impact policy and scientific decisions with broad social, engineering, and ecological consequences. 1. Advances in the Next Decade As depicted in Figure 6.1, the needs of exascale applications will be addressed by advances in four major and interlinked areas: coupled models, uncertainty, optimization, and data.

• Coupled Models: Most exascale applications will involve multiple models (PDE or data-based) for different phenomena or at different scales of the same phenomenon. In some cases, new models and corresponding scalable implementa- Figure 6.1 E3 applications characteristics and math and algorithms needs.

85 Section 6: Math and Algorithms

• Large, Messy, and Noisy Datasets: Ad- vances in technology have enabled the production of massive volumes of data through observations and simulations in many applications such as biology, high-energy physics, and astrophysics. To effectively utilize this flood, we will develop new data representations, data- handling algorithms, efficient implementations of data analysis algorithms on high-performance computing platforms, and representations of analysis results.

Exascale computing presents an opportunity not just for quantitative but also for qualitative changes in the role of computation in scientific discovery and in high-consequence decision support. Applications for exascale systems will demand rigorous V&V methodologies with quantified uncertainties that are presented in a manner that enables simulation- based critical decisions and optimization. Ex- ascale computing opens many opportunities that will help drive mathematical and algorithmic research, making possible the design Figure 6.2 A cross section of a small portion of an Escherichia coli cell, an example of safe, reliable, economical, and socially ac- of a complex system that might one day be the subject of an ab initio simulation. The cell wall, with two concentric membranes studded with transmembrane proteins, is ceptable solutions for energy, climate, and the shown in green. A large flagellar motor crosses the entire wall, turning the flagellum environment. The E3 initiative must ensure that extends upwards from the surface. The cytoplasmic area is colored blue and that critical applied mathematics, algorithms, purple. The large purple molecules are ribosomes; the small, L-shaped maroon and software challenges are addressed. molecules are tRNA; and the white strands are mRNA. Enzymes are shown in blue. The nucleoid region is shown in yellow and orange, with the long DNA circle shown in yellow, wrapped around HU protein (bacterial nucleosomes). In the center of the For computational biology, sustainability, nucleoid region shown here, one might find a replication fork, with DNA polymerase and global security applications, HPC sys- (in red-orange) replicating new DNA. Image courtesy of David S. Goodsell. tems must place greater emphasis on scalable shared-memory performance, memory la- error. Complex, fully coupled multiphys- tency and bandwidth, and integer operations, ics and multiscale applications require in- in contrast to the current focus that rests al- trusive tools that automatically construct most exclusively on floating-point perfor- representations of uncertainty, handle the mance. Bioinformatics and computational uncertainty propagation and coupling ef- biology, genomics, and medical applications fects, and provide sharp estimates of the may differ significantly from the current HPC uncertainty of key merit criteria. workloads in that the data structures are often irregular (based on strings, trees, graphs, and • Optimization: Large-scale design prob- networks), without the high degree of spatial lems demand scalable algorithms for and temporal locality seen in physics-based continuous nonlinear optimization. Also simulations using regular matrices. We also needed are parallel branch-and-cut meth- see a growing need for database research in ods for linear and nonlinear optimization areas of probabilistic queries and queries by problems with discrete variables, as well structure, such as DNA sequence data, pro- as more sophisticated parallel methods for tein structure, and phylogenetic graphs or solving stochastic optimization problems. networks. We envision that new exascale algorithms will require tight integration of computation with database operations and

86 Modeling and Simulation at the Exascale for Energy and the Environment queries, as well as the ability to handle new standing of the “teleconnections” in model types of queries such as combined structural, output that depend on the dynamical system ethno-botanical, socio-geographic, phylo- under study; such dependence information is geographical queries. critical to studying extremes and extremes of uncertainty. 2. Major Challenges Optimization. The design of complex systems Exascale computing has the transformational and physical processes that maximize some power to allow scientists to move beyond performance measure can be expressed as op- simulation of complex systems and to con- timization problems [Nocedal 2006]. Optimi- sider a paradigm shift that will enable them to zation holds the promise of better designs and address more challenging questions such as a more efficient use of natural resources. The design and the quantification of uncertainty. key challenge is the development of robust This paradigm shift, together with the mas- optimization techniques that are reliable and sive increase in computing power offered by exploit the evolving new computer architec- exascale computing, will create a set of com- tures. These techniques must be easy to use, putational and mathematical hurdles that must in order to ensure their widespread acceptance be overcome. Four broad areas have been within the scientific community. An additional identified that encapsulate the application challenge is the determination of appropriate needs: uncertainty quantification, optimiza- algorithms for novel optimization paradigms tion, PDEs, and the management of massive that could not have been approached on a sets of data. system scale before the advent of exascale architectures. Such novel paradigms include Uncertainty Quantification. The quantifica- hierarchical optimization problems over mul- tion of uncertainty answers such fundamental tiple time stages and problems with chance questions as simulation code V&V against constraints that better reflect licensing and reality. It is important to establish mathemati- operational requirements. A third challenge cally firm foundations for these fundamental that will arise with the use of exascale ar- questions and to develop computational tools chitectures is the handling of problems with within an integrated computational environ- hundreds of thousands of discrete parameters. ment that reduces time to simulation, provides Moreover, as scientists move increasingly compatible geometry representations, allows from simulation to design, they will need to for a hierarchy of model fidelities running on address the solution of complex, nonlinear a range of architectures from workstations optimization problems—possibly involving to state-of-the-art parallel computers, and trade-offs between multiple objectives and includes a rich suite of postprocessing capa- discrete choices. A portfolio of techniques bilities. Typically, scientists have assumed in- will be needed, ranging from more sophisti- dependence between various sources of input cated PDE-constrained optimization methods uncertainty. While this assumption simplifies to new, rigorous techniques that exploit a hi- the analysis, it does not represent reality, and erarchy of physical models. there exists a need to support higher-dimensional input densities to model true model PDEs. Solving linear and nonlinear systems uncertainty. New methodology is needed to of PDEs is at the heart of many DOE appli- systematically explore the input uncertainty cations, such as accelerator modeling, astro- space for an optimal characterization of the physics, nanoscience, and combustion. As the In enabling scientists to move output uncertainty space. Typically, research- focus of scientists extends from simulation to from simulation to design, exascale computing will also ers have developed techniques principally for optimization and uncertainty quantification, demand novel paradigms, such optimizing physical experiments; these tech- the solution of linear and nonlinear systems increases in importance. The efficient solu- as hierarchical optimization niques must be adapted for computational over multiple time stages tion of PDEs requires AMR to provide mesh- experiments, leading to uncertainty quantifi- and problems with chance cation for thousands or millions of parame- quality-preserving automatic adaptation for constraints that better reflect ters. The development of these mathematical different types of complex geometries, while licensing and operational foundations and tools is critical to the under- allowing the addition of domain-specific in- requirements.

87 Section 6: Math and Algorithms

terpolation strategies. Dynamic load balanc- 3.1 Solvers ing is especially challenging at the exascale level, leading to hard combinatorial problems The dominant computational solution strat- [Streensland 2002]. Many complex physical egy over the past 30 years has been the use systems, such as astrophysics and climate of first-order-accurate operator-splitting, modeling, face bottlenecks in time step re- semi-implicit and explicit time integration strictions because of higher spatial resolu- methods, and decoupled nonlinear solution tions, which must be overcome to ensure strategies. Such methods have not provided convergence for reasonable time steps. the stability properties needed to perform accurate simulations over the dynamical time Massive Datasets. The handling and manage- scales of interest. In most cases, numerical ment of increasingly large and heterogeneous errors and means for controlling such errors sets of data will create new challenges for are understood heuristically at best. In addi- scientists and computer architectures. Data tion, the impact of these choices is difficult to can be the result of an exascale simulation assess and control. For this reason, solutions that must be postprocessed for human inter- for these complex systems can be fragile and pretation, or it can form the input to complex exhibit nonintuitive instabilities, or they may problems via data assimilation. Browsing or simply be stable in a crude sense but contain looking at data is no longer possible as we significant long-time integration error. near a petabyte. To visualize 1% of 1 petabyte at 10 MB/s takes 35 workdays. There is an Direct methods for solving sparse linear equa- enormous need for methods to dynamically tions are used in many applications, such as analyze, organize, and present data by vari- the inversion operator for the shift-and-invert ability of interest. For example, how do we algorithms for high-accuracy eigencompu- organize ocean eddies in a climate simulation tations, solution of coarse-grid problems as so that by viewing a very small set of them part of a multigrid solver, and subdomain so- we have a good idea about all the types of lutions in domain decomposition methods, as eddies present in the entire petabyte dataset? well as in cases where the linear systems are What about particles in a fusion simulation? extremely ill-conditioned [Li 2006]. Iterative Solutions to these problems will likely come methods are required for most 3D and cou- from dynamically considering high-dimen- pled physics problems, however, since direct sional probability distributions of quantities methods become computationally infeasible of interest. This approach requires new con- because of memory and time requirements tributions from mathematics, probability, and [Eijkhout 1998]. statistics. Besides being large, data is often noisy, or inaccurate, thus creating additional The numerical optimization community has challenges. For example, we may be interest- focused principally on the development of se- ed in estimating the response of a nonlinear rial algorithms for solving linear, quadratic, dynamical system that is polluted by noise. and nonlinear optimization problems with How can we detect the real signal? Answers continuous variables or a mixture of continu- to such questions require new mathematical ous and discrete variables. Techniques for theory and computational tools to classify solving linear and quadratic programs can be shapes in terms of stochastic models and to split into active-set and interior-point meth- deal with both local and long-range correla- ods, while algorithms for solving nonlinear tions between features. programming problems are generally based on solving a sequence of linear or quadratic 3. State of the Art approximations or interior-point methods [Nocedal 2006]. Parallel implementations In this section, we examine the current state have been written for some problem classes of the art in six areas that will require signifi- by parallelizing the linear algebra [Gondzio Dynamic load balancing is cant mathematical and algorithmic advances 2003] or exploiting the problem structure by especially challenging at the in order to meet the emerging needs of exas- using a decomposition method, for example. exascale level, leading to hard cale applications. Preconditioned Krylov methods and approxi- combinatorial problems. mate solutions to the systems of equations

88 Modeling and Simulation at the Exascale for Energy and the Environment are not well understood from a theoretical perspective, especially for problems with inequality constraints, but they have been used successfully in some contexts. Algo- rithms for optimization problems with discrete variables solve a sequence of linear and nonlinear relaxations in a branch-and-cut framework [Nemhauser 1988]. These methods are augmented by a number of heuristics Figure 6.3 Error of a multiscale model reduction approach for electronic structure for finding an initial feasible point, selecting density functional theory (DFT) energy minimization for a string of hydrogen atoms. the branching variables, and determining the cuts to add. Parallelization of these methods step 2 when several models with such un- is achieved by solving multiple relaxations in certainty representation are coupled. This is parallel. Optimization problems with stochas- a major bottleneck in expanding the reach of tic variables have also been studied, in which such methods and is responsible for the fact one typically samples the random variables that parametric uncertainty quantification for and solves the resulting deterministic prob- PDE applications has been limited to tens of lem. The problems for the realizations can be parameters for problems whose state space is solved in parallel. For large parameter spaces, on the order of millions in magnitude. the sampling method employed becomes very important for obtaining reasonable results. 3.3. Adaptive Mesh Refinement

3.2 Uncertainty Quantification AMR is not widespread in scientific simulation today but is slowly gaining acceptance. A complete answer to the uncertainty ques- It is used within both structured and unstruc- tion in computational experiments involves tured mesh contexts. Block-structured AMR three distinct steps: (1) representation of in- combines the efficiency of having resolution put uncertainty [Klir 1994], (2) propagation only where needed with the simple array- of uncertainty through a simulation model based storage and mathematical operations [Christianson 2005], and (3) representation of structured meshes. Most AMR computa- and calculation of output uncertainty. The tions are performed using packages such as overall approach is guided by the initial rep- Chombo, GrACE, or Paramesh, since such resentation of the uncertainty. For pure sto- computations require sophisticated numeri- chastic representations, the state of the art cal, algorithmic, and software constructs to for step 1 includes Gaussian randomness of run efficiently on parallel machines. Most inputs and parameters, used in conjunction discretization techniques (finite differences with perturbative techniques for step 2 and and finite volumes) on block-structured AMR with Monte Carlo for step 3. Such representa- are second-order accurate, although there tions have been carried to dimensions of the have been recent extensions to fourth-order random vector as high as millions. Nonethe- numerical schemes. The use of AMR within less, further progress in this direction is hin- unstructured meshes, and especially finite- dered by the severe nonlinearities exhibited element calculations, is far more advanced. by most models of interest or by the failure Simulations with both automatic resolution of the data to support efficient Gaussian ap- and discretization order refinement (hp-re- proximation. A relatively recent development finement) are conducted today. is the use of stochastic finite elements to rep- Stochastic finite-element methods can accommodate resent uncertainty in inputs and parameters as AMR necessitates load balancing on parallel severe nonlinearities, but an element in a linear space at step 1. This computers. Several models for partitioners propagating uncertainties has the promise of very compact and efficient and load balancers have been applied success- through coupled models representations of the uncertainty. Moreover, fully to unstructured problems [Hendrickson presents a major bottleneck the approach has been able to accommodate 1995; Pilkington 1994; Simon 1991]. The to expanding the range of quite severe nonlinearities. Nonetheless, to best-known model is graph partitioning, such methods to exascale date we have no efficient way of achieving where data/work is represented by graph ver- architectures.

89 Section 6: Math and Algorithms

algorithms (e.g., bi-level, knapsack) take a middle-ground approach. Furthermore, many algorithms have tunable parameters to trade off load balance with communication costs, given the character of a particular machine. Determining optimal values for these parameters is difficult, however, and doubly so for time-evolving simulations on adaptive meshes.

3.4 High-Dimensional Spaces

In genomics applications involving mutation analysis, parameter spaces may exceed 5,000 dimensions and may take only discrete values. Yet state-of-the-art algorithms, such as entro- py-based methods, have not been successful for problems exceeding 100 dimensions.

In the simulation of a fission reactor, the situation is even more challenging. Some parameters are outside the user’s control, such as scattering, absorption, and fission cross sec- Figure 6.4 Left: Temperature (color map) and heat release contours for a partially tions; these parameters suffer from a larger premixed unsteady methane jet flame. Right: The corresponding AMR mesh. degree of uncertainty than do others. For a moderately high number of energy groups, tices. Hypergraph partitioning approaches the number of such parameters can easily sur- improve on the graph model by representing pass 10,000. Yet the state of the art currently data dependences within sets of vertices, thus is limited to parameter spaces of perhaps 20 allowing nonsymmetric and nonsquare data to 30 dimensions. dependences to be modeled. In general, hypergraph algorithms produce decompositions A complicating factor in the topic of high- with lower communication volume (vis-à-vis dimensional parameter spaces is the fact that graph partitioners) while supporting a larger such problems can also have a huge state range of applications. space. In a fission reactor, for example, the state space is composed of an ensemble of the Scientific simulations that use block-struc- neutron flux, temperature, and coolant veloc- tured adaptive meshes use mainly geomet- ity distributions in a 3D configuration with ric models [Patra 1995], which implicitly sizes on the order of meters in any direction assume that objects that are physically near and physics that must be represented at scales each other depend on each other. Geometric far below the millimeter range. methods are very fast and scalable compared to graph and hypergraph algorithms. Current 3.5 Data Analysis techniques perform domain decomposition in The current state of data analysis lags far be- such a way that the subdomain per processor hind our ability to produce simulation data or can be represented as a collection of nonover- record observational data. A particular gap lapping rectangular boxes. Two extreme ob- exists in the mathematics needed to bring jectives can be adopted in partitioning such analysis and estimation methodology into meshes: (1) reduce load imbalance, even at a data-parallel environment. Parallel linear higher communication costs (patch-based algebra methods go a long way toward en- partitioning), and (2) reduce communication abling data-parallel analysis, but they do not costs, at the price of increased load imbal- solve it, just as they would not solve a cli- ance and synchronization costs. Successful

90 Modeling and Simulation at the Exascale for Energy and the Environment mate simulation problem. For example, the preconditioners, eigensolvers, algorithms for standard principal component analysis com- mesh generation and adaptation, and ordinary putation does not become data-parallel with a differential equation (ODE) / differential al- parallel singular value decomposition (SVD) gebraic equation (DAE) integrators. solver, even though the SVD is the core computation in that analysis. Data-parallel solu- 4.1 Solvers tions for applications on exascale resources In order to achieve accurate, stable, efficient will require new mathematics that considers and scalable predictive simulations for multi- an entire estimation problem for develop- ple-time-scale systems with implicit methods, ing scalable data-parallel algorithms in data many advances in numerical methods and analysis. computational science are required.

3.6 High-Precision Arithmetic • Development, demonstration, and com- No HPC vendor currently offers hardware sup- prehensive evaluation of stable, accu- port for 128-bit floating-point arithmetic, and rate, efficient, and scalable fully implicit there is little prospect that this situation will methods coupled with uncertainty quan- change within the next few years. Some For- tification techniques (deterministic and tran compilers support the REAL*16 datatype probabilistic) and with estimation and in software. Unfortunately, such facilities are control of long-time integration error for not provided in all compilers—not in the GNU large-scale, complex multiple-time-scale compilers, for instance—and even when pres- applications. Verification and well-char- ent, they are usually very slow, often 50 to 100 acterized prototype problems for multi- times slower than conventional 64-bit floating- ple-time-scale multiphysics systems must point arithmetic. Few scientists are willing to be developed. experiment with such a facility when the per- • Deterministic uncertainty quantification The current state of data formance penalty is so great. tools based on sensitivity and adjoint- analysis lags far behind our The alternative is to use independently written based techniques for data, integration, and ability to produce simulation software libraries, such as those mentioned model error estimation and control must data or to record observational data. by Bailey [2005]. With such software, one be further developed and demonstrated. specifies which variables and arrays are to Adjoint methods have shown promise for be treated as high precision by means of spe- estimating and controlling data, model, cial type statements. Then when one of these and long-time integration error. In adjoint- variables or arrays appears in an arithmetic based methods, advances are required statement or argument, the proper library to limit solution storage requirements, routines are automatically called via opera- memory usage, parallel communication, tor overloading. Such facilities have several and cost of the adjoint solve. There is a drawbacks: (1) at present they are available significant need for the development of only for Fortran and C++; (2) it is necessary public-domain software to enable adjoint to make alterations (mostly minor) to one’s methods for time-dependent problems. In source code; (3) certain subexpressions may addition, adjoint techniques for hyperbolic be performed only to conventional precision; systems are critically required. and (4) slowdown is typically a factor of 5 for • Probabilistic approaches based on sam- double-double arithmetic, 25 for quad-dou- pling methods (e.g., Monte Carlo) and ble, and even higher if transcendental func- direct methods (e.g., polynomial chaos) tion references are involved. require advances in algorithms and computationally efficient implementations 4. Accelerating Development for transient simulations. Hybrid deter- The movement to multiscale, multiphysics ministic/probabilistic approaches must Few compilers or software simulations necessitates advances in several be developed and studied in the context libraries can handle high- fundamental classes of numerical algorithms, of complex systems. precision arithmetic, and those including linear solvers, nonlinear solvers, that do so are often extremely slow.

91 Section 6: Math and Algorithms

Figure 6.5 A sequence of solutions and their deviation from the optimal solution for a bound constrained minimization problem. The objective is the surface with minimal area that satisfies Dirichlet boundary conditions and is constrained to lie above a solid plate.

• NK and Jacobian free NK (JFNK) tech- • The System of Systems (SOS) approach niques in complex large-scale applica- offers promise in building alternative tions must be significantly extended. physical models based on the integration Needed extensions include automated of a smaller, well-understood set of known generation of adjoint models and auto- physical models into a larger model. This matic differentiation technologies to sup- approach may prove useful, for instance, port NK and JFNK methods in complex in exploring change in global tempera- applications. ture based on smaller local models (e.g., climate models for small regions or series • NK methods require algorithmic and soft- of physics-based PDE models). ware developments in physics-based and approximate block factorization precon- Research in linear and nonlinear solvers re- ditioners for coupled elliptic, parabolic, mains a critical focus area because the solvers and hyperbolic systems. Critical subcom- provide the foundation for more advanced so- ponent physics include incompressible lution methods. In fact, as modeling becomes and compressible flow, transport-reaction more sophisticated and increasingly includes systems, coupled porous media flow, optimization, uncertainty quantification, per- nonlinear elasticity, fluid-structure inter- turbation analysis, and more, the speed and action, electromagnetics, and ideal and robustness of the linear and nonlinear solvers resistive MDH. will directly determine the scope of feasible problems to be solved. • Efficient and scalable physics-based preconditioners require subblock solvers • The performance of direct methods re- based on multilevel (multigrid and alge- lies heavily on the ordering of rows and braic multigrid) methods for scalar and columns of the matrix, and research in vector systems with strong anisotropic effective orderings is still important. Fur- effects and large-scale variations in coef- thermore, for extremely large systems, ficients. Advances in algebraic multigrid out-of-core implementations will be methods for compatible physics-based needed to overcome the memory bottle- discretizations are also required. neck. The efficiency of such out-of-core

92 Modeling and Simulation at the Exascale for Energy and the Environment

solvers will need to be revisited. Efficient the best modeling and solution techniques for and scalable algorithms for sparse indefi- optimization problems. nite systems, particularly those from optimization, must be investigated. Moreover, • Modeling systems that allow applica- further work is needed for fully distrib- tion scientists to express their models uted direct sparse solvers, where all data in a natural, domain-specific format and objects are distributed across the parallel that couple applications to solvers. This machine and efficient parallel algorithms effort will involve the development of are used for all phases of the solver. new tools that support PDE constraints, chance constraints, and simulation-based • Block iterative methods, which can effec- applications. tively solve multiple simultaneous right- hand sides, represent an area of growing • Scalable parallel solvers for optimization importance in order to exploit the growing problems with PDE constraints. One ap- number of such problems. Furthermore, proach is to extend current nonlinear op- block iterative methods have attractive timization techniques to allow for inexact machine performance characteristics that subsystem solves. will become even more important on future architectures. • New methods for hierarchical and simulation-based design problems that exploit • Convergence of iterative methods is exascale architectures. Such methods strongly affected by the quality of pre- should include constraints, allow for in- conditioners. Advanced efforts in pre- exact simulations, and provide support conditioning are focused on exploiting for multiscale optimization. problem characteristics via segregation of variables for coupled systems, multilevel • Efficient parallel branch-and-cut solv- Block iterative methods have attractive machine performance methods for keeping complexity costs ers for nonlinear discrete optimization characteristics that will low, and nesting of iterative methods via problems that enable the solution of ap- become important on exascale inner-outer techniques. However, even plications with hundred of thousands of architectures. these techniques still rely on basic pre- discrete parameters. conditioners (incomplete factorizations, Jacobi, Gauss-Seidel, etc.) and iterative 4.2 Uncertainty Quantification, methods as building blocks. Therefore, Validation, and Verification any improvements in basic preconditioners, such as better reorderings, reduced Effective, scalable algorithms for uncertainty error in incomplete factors, or emerging quantification and optimization will likely be approaches such as support graph tech- intrusive, requiring extensive changes to ex- niques, can have a broad impact. isting application codes, the underlying algorithms, and the solvers required for efficient • Progress in nonlinear solvers continues to solution that will drive the need for improved be important, especially for tightly cou- iterative solvers and preconditioners. While pled and highly nonlinear systems where the advantages of intrusive methods have continuation methods can be essential to been well documented, their use has been getting a converged solution. Work in limited by the substantial changes and related graph coloring is an important related solver advances required. E3 should promote problem for efficient Hessian and Jaco- next-generation applications that support bian computations, as is automatic differ- fully intrusive uncertainty quantification and entiation as a means of obtaining accurate optimization algorithms. To achieve this ob- and flexible operator formulations. jective, advances in numerical methods and computational science are required: Advances in preconditioners, Optimization research should focus on the such as better reorderings, following four areas; a complementary effort • Verification. Address the mathemati- can have a broad impact on the should involve educating scientists to select cal challenges of error estimation for performance of iterative solvers.

93 Section 6: Math and Algorithms

complex, coupled, multiscale, and mult- computation, scientific application do- iphysics calculations with possibly noisy main experts, probability and statistics, and discontinuous data and engage the and decision theory. community to improve the overall quality of computational testing, including 4.3 Adaptive Mesh Refinement the design, documentation, and repository of useful and influential benchmark AMR at the exascale requires significant ad- problems. vances in the following areas, some of which overlap with the combinatorial algorithms re- • Validation. Focusing on the selection search discussed in Section 4.5. and design of appropriate experimental benchmarks as well as the mathematical • Exascale simulations will exhibit large and computational challenges of validat- length-scale heterogeneities within coming systems of coupled simulations that plex geometries [Chandra 2007]. Main- may be used well outside their validation taining resolution and mesh quality will regime. require new algorithms as well as hp-refinement techniques [Patra 1995]. • Uncertainty quantification. Formulate mathematical foundations, scalable algo- • AMR meshes change with time and must rithms, and high-quality software imple- be rebalanced. The availability of a port- mentations for stochastic PDEs, sampling folio of partitioning strategies will enable methods, polynomial chaos expansions, the selection of the optimal rebalancing uncertainty propagation, adjoint-based strategy for a given problem or computa- sensitivity methods, Bayesian inference tion stage [Streensland 2002]. strategies, and possibilistic and fuzzy inference strategies. Also needed are ef- • High-order methods can reduce the de- fective models to deal with information gree of refinement required, leading to presented with imprecise probability, smaller meshes and more efficient solu- nonmodel uncertainty mitigation, and tions. While AMR finite-element calcu- experimental characterization of input lations have successfully incorporated uncertainties. high-order elements, block-structured methods are limited to second order. The • Decision making. Explore the role of primary challenge is creation of tractable uncertainty quantification for effective stencils at coarse-fine interfaces for im- communication of computational results plementing conservation laws. in complex decision environments, including within optimization. A major • The development of efficient and rigorous remaining mathematical challenge is error metrics for refinement is an impor- how to properly formalize confidence in tant challenge. Indirect techniques that computational simulations to best sup- estimate the spectral content of the solu- port decision-making under uncertainty tion locally (and thus recommend a level and imprecise information. Uncertainty of resolution) provide a trade-off between quantification must be tightly coupled efficiency and mathematical rigor. with development and execution of sophisticated mathematical methods for • Block-structured AMR meshes are usu- V&V, advanced analysis, and visualiza- ally restricted to simple geometries. Map- tion methods for uncertain data. ping such meshes to smooth geometries automatically is an important challenge • Community awareness. Develop a re- that will enable new applications to ben- Adaptive mesh refinement search training regime for computational efit from AMR [Schwartz 2006]. at the exascale requires science V&V and uncertainty quantifica- development of efficient and tion. Such a regime will certainly be highly • Efficient algorithms must be developed to rigorous error metrics. interdisciplinary, involving mathematics, unravel grid hierarchy into a sparse ma-

94 Modeling and Simulation at the Exascale for Energy and the Environment

trix form (and vice versa) to allow new solvers such as Krylov solvers to be used instead of traditional geometric multigrid methods.

• AMR will need adaptive partitioning to scale to exascale levels. Rather than con- figuring a partitioner at the beginning of the run, a “control system” approach to partitioning may have to be adopted.

• AMR in turn will benefit from advances made on discrete algorithms to address new multicore architectures, data layout, and processor interconnectivity, leading to multiple competing objectives.

4.4 High-Dimensional Spaces

Calculations have been done for resolutions that are far coarser than what will be required by exascale applications, but current and expected advances in scalable algorithms for PDE software make it likely that the computation of state variables will be achievable on exascale computers. This is not the case Figure 6.6 Temperature map and mesh refinement from a H2-air mixture ignited for increasing the parameter space, where the by random high-temperature kernels, after ~90 µs of simulation time. curse of dimensionality applies to most obvious algorithms, including possibly random may be better suited for large-scale applica- sampling (since the variance itself may be un- tions, such as robust optimization and the use bounded with the increase in the size of the of chance constraints; or from radically new parameter space, even if the error decrease ideas brought about by increased focus in this is dependent on the number of samples and crucial direction. not on the dimension of the space). There- fore, significant research must be undertaken 4.5 Combinatorial and Discrete to develop new algorithms that make use of Algorithms problem structure for breaking the curse of dimensionality. Combinatorial and discrete algorithms have long played an important role in many appli- Random sampling tends to be the most com- cations of scientific computing, such as sparse monly used technique to approach this prob- matrix computations and parallel computing, lem; however, the points need not be chosen to enhance the performance of numerical by a stochastic algorithm. Indeed, recent algorithms. Emerging applications such as approaches that seem promising choose de- computational biology, scientific data min- terministic points. Progress may come from ing, and complex systems bring combinato- the refinement of ideas currently at the fore- rial algorithms to the fore as an integral part front of this research, such as higher-order of computational sciences. We must develop sampling methods (e.g., randomized Monte highly scalable kernels for discrete math- Carlo sampling), structured stochastic finite- ematics to support such applications. These element approaches, sparse grids, and hierar- kernels should be developed independently chical models for importance sampling and from the application use and should be lev- preconditioning; from the identification of eraged by a wide array of code bases on the new assessment and design paradigms that petascale, exascale, and beyond. Systems that

95 Section 6: Math and Algorithms

Figure 6.7 A graph layout generated by LGL [Adai et al. 2004] showing a portion of the minimum spanning protein homology tree with over 300,000 proteins. An edge is colored blue if it connects 2 proteins from the same species, and red if it connects 2 proteins from 2 different species. If that information is not available, the edges are colored based on layout hierarchy. They remain white if there is no species information available. Image courtesy of Alex Adai and Edward Marcotte.

will benefit from improved combinatorial al- 4.6 High-Precision Arithmetic gorithms include the following: Some, and possibly many, exascale applica- • The unit commitment problem for power tions will require high-precision arithmetic systems. Optimal power flow is a chal- facility, yet there is little prospect for vendor lenging combinatorial problem, and support. While some software packages are power system analysis for dynamic secu- available, they have shortcomings. What is rity assessment is vital. The solution of needed is a simple-to-use facility that infal- realistic-sized models involves discrete libly converts large application programs for parameters on an unprecedented scale. high precision, yet results in only a modest inflation in run time. Such a facility is pos- • Bioinformatics models of DNA, RNA, sible but not yet available. and proteins as sequences over small al- phabets. Searches for specific structures, Beyond the immediate needs of usable high- gene regulatory networks, protein inter- precision software, better understanding is action networks, and metabolic networks needed of situations that can lead to numerical all give rise to combinatorial problems. difficulties in large computations. Also needed are tools that can quickly detect whether and • Efficient utilization of underlying com- where an application program is experiencing putational infrastructure and intercon- difficulties in this area. Along this line, cur- nection networks. The increasing gap rent research in numerical analysis, particu- between CPU and memory performance larly in the area of interval arithmetic, may be argues for the design of new algorithms of use in the HPC world, in that it may pro- and data structures and for data reorga- vide a means to determine whether and where nization to improve locality at memory, a code is experiencing numerical difficulties, cache, and register levels. and to certify that the final results are within a certain specified tolerance of their correct • Utilization of observational devices such values [Hayes 2003]. as telescopes. Projects now share telescope time with interleaved schedules. 4.7 Data Analysis Yet experiments are designed for longer terms (a complete supernova observa- Solutions for estimation problems based on tion, for example, requires observations a mixture of integer (categorical) and real Interval arithmetic may help over a 60-day period), making scheduling values are needed for biological data in par- researchers determine where a crucial challenge. ticular. Many solutions exist (most based on a code is having numerical the likelihood principle), but they are com- difficulty and to certify final plex, and their implementations are serial, so results. they typically do not scale to even terabyte

96 Modeling and Simulation at the Exascale for Energy and the Environment datasets. The mathematics of many estima- the nature and volume of communication Domain decomposition tion problems must be reorganized so that the between agents techniques for parallel agent- estimation can be performed in a data-parallel based models must be environment of the future petascale and exas- formulated to address exascale 5. Expected Outcomes application needs. cale environments. A focused program to address the qualita- 4.8 Agent-Based Modeling tively and quantitatively different challenges of next-generation scientific applications, Agent-based modeling [Woolridge 2002; tailored to encourage and support close col- North and Macal 2007] must be advanced laborations between applied mathematicians, along several directions before it can present computer scientists, application scientists, and a viable approach for addressing exascale ap- hardware vendors, will result in the ability to plication needs: tackle emerging complex, coupled problems in climate, biology, energy, the environment, • Distributed query resolution to allow and other global problems. agents to flexibly and repeatedly find other agents and recognize affordances for in- 6. Required Investment teraction in a dynamic environment with a continually and endogenously evolving Sufficient and timely investment in new structure (e.g., nonreified networks) mathematical approaches and algorithms is crucial for transforming computation into • Situational activation of agents based on a powerful tool for scientific discovery and contextual factors and associated real- high-consequence decision support. The ef- location to workings sets of processors fort will require multidisciplinary teams to with appropriate interprocessor locality advance the state of the art in each of the key areas laid out in Section 4. Significant energy • Efficient implementation of periodic must be expended on designing, developing, fine-grained interactions between agents testing, and deploying the integrated frame- where the payoffs from the interplay are works and tools required to ensure effective endogenously defined as a function of the use of the software in the major algorithm ongoing interactions themselves, such classes. Some effort must also be dedicated that players are free to enter and leave the to maintaining collaborations that will sup- interactions at idiosyncratic times port cross-cutting aspects of the development in mathematics and algorithms with the de- • Distributed time scheduling at a lev- velopment in the application areas and in the el of parallelism beyond the current software, hardware, and cyberinfrastructure approaches development teams. We anticipate that this will require between $100M and $200M over • Extremely high volume data warehous- the next decade. More finely tuned projec- ing to allow efficient exploration of huge tions will become possible as the envisioned numbers of large model runs efforts advance through planning and ramp- up stages. • Efficient directed sweeps across huge model parameter spaces with appropriate 7. Major Risks adaptation as results are discovered The principal risks and downside conse- • Domain decomposition techniques for quences linked to R&D in mathematics and parallel agent-based simulations where algorithms are as follows: the computational load per agent is variable, in time for the same agent, as well • If advances in theoretical performance as from agent to agent, and where the of new hardware architectures are not geographical locality has no relation to matched with significant R&D in mathematics and algorithms, then real perfor-

97 Breakout Group 6: Math and Algorithms

mance gains are unlikely to be realized. utility of results from large and complex The investment in hardware advances simulations. will be rendered ineffective for lack of • If we do not design our algorithms and scalable algorithms. interface layers in collaboration with the hardware and software developers, it will • If significant advances in methods for be extremely difficult to realize the goal managing complexity in models and of developing scalable implementations. computational simulations fail to materi- This situation will be increasingly rel- alize, then the opportunity to expand our evant as raw performance results from simulation capability into a number of combinations of parallelism through core critical application areas will be lost. We count and inclusion of special-purpose will be left with a dwindling cadre of ap- acceleration hardware. plications that can benefit from advances on the hardware front. • If we do not increase the use of good software engineering practices, the com- • If we fail to develop new mathematical plexities of developing and maintaining methods, data structures, and algorithms our software will overtake the energies for integrating the growing volume of expended in mathematics and algorithms data coming from our experiments into R&D. our simulations, we will not be able to extract sufficient benefit from such • If we fail to monitor and maintain an ap- development. propriate balance between precision calculation and robust solutions as guiding • If we inadequately connect the develop- metrics of our algorithm design, we run ment described in this report to the ap- the risk of becoming needlessly and over- plication developers’ needs, we will whelmingly burdened with the demand for generate tools and methods that are not precision, which will result in delayed de- widely accepted and therefore miss the ployment, incomplete risk assessment, or mark on broad benefits. outright failure. Aggressive R&D in mathematical methods • If R&D is not supported with long-term and scalable algorithms is key to successful investment, short-term gains will fail development and deployment of future appli- to mature and evolve with the chang- cations relying on computational simulation, ing needs of the applications and with data assimilation, and analytics. It will be a the changing details of the hardware ar- critical component of an effort that will result chitectures. The benefit of initial invest- in quantitatively and qualitatively new and ments will be lost. powerful scientific methods.

• If we fail to build robust abstraction lay- References ers insulating the application developer from the details of the hardware archi- A. T. Adai, S. V. Date, S. Wieland and E. M. tecture and the algorithm, the anticipated Marcotte (2004), LGL: Creating a map of protein complexities of expressing and managing function with an algorithm for visualizing very large biological networks, J. Mol. Biol. computations at the exascale will stifle 340(1):179-190. development of applications. D. A. Bader (2004), Computational biology and • If we do not develop invasive implemen- high-performance computing, special issue on The complexities of managing tations for some of the more complex and bioinformatics, Communications of the ACM computation at the exascale will onerous aspects described above, partic- 47(11):34–41. require robust abstraction layers ularly optimization and uncertainty, they that insulate the application will generally go unused, thereby limit- D. A. Bader, A. Snavely and G. Jacobs (2006), NSF developer from the details of workshop report on petascale computing in the ing the kinds of applications that will be both the hardware and the biological sciences, August 29–30, Arlington, VA. algorithms. developed, as well as casting doubt on the

98 Modeling and Simulation at the Exascale for Energy and the Environment D. H. Bailey (2005), High-precision arithmetic in rithms for parallel SAMR applications, Technical scientific computation, Computing in Science and Report 2006-047, Uppsala University, Dept. of Engrg., May–June, 54–61. Information Technology.

M. Barad and P. Colella (2005), A fourth-order G. Karypis and V. Kumar (1998), A fast and high accurate local refinement method for Poisson’s quality multilevel scheme for partitioning irregular equation, J. Comp. Physics 209(1) 1–18. graphs, SIAM J. Sci. Computing 20(1) 359–392.

M. J. Berger and S. H. Bokhari (1987), A parti G. Karypis and V. Kumar (1997), A coarse-grain tioning strategy for nonuniform problems on mul parallel multilevel k-way partitioning algorithm, tiprocessors, IEEE Trans. Computers C-36(5): in Proc. 8th SIAM Conf. Parallel Processing for 570–580. Scientific Computing.

J. Birge and F. Louveaux (1997), Introduction to G. J. Klir (1994), The many faces of uncertainty, Stochastic Programming, Springer. pp. 3–19 in B. M Ayyub and M. M. Gupta, eds., Uncertainty Modeling and Analysis: Theory and U. Catalyurek and C. Aykanat (1996), Decom Applications, Elsevier Science. posing irregularly sparse matrices for parallel ma trix-vector multiplications, Lecture Notes in Com X. Li (2006), Direct solvers for sparse matrices, puter Science 1117: 75–86. http://crd.lbl.gov/~xiaoye/SuperLU/SparseDi rectSurvey.pdf, September. U. Catalyurek and C. Aykanat (1999), Hyper graph-partitioning-based decomposition for par G. Nemhauser and L. Wolsey (1988), Integer and allel sparse-matrix vector multiplication, IEEE Combinatorial Optimization, John Wiley & Sons. Trans. Parallel Dist. Systems 10(7): 673–693. J. Nocedal and S. Wright (2006), Numerical Op nd S. Chandra, M. Parashar and J. Ray (2007), Ana timization, 2 ed., Springer. lyzing the impact of computational heterogeneity on runtime performance of parallel scientific com M. J. North and C. M. Macal (2007), Managing ponents, in Proc. 15th High Performance Comput Business Complexity: Discovering Strategic Solu ing Symposium (HPC-07), SCS Spring Simula tions with Agent-Based Modeling and Simulation, tion Multiconference, Norfolk, VA. Oxford.

B. Christianson and M. Cox (2005), Automatic A. Patra and J. T. Oden (1995), Problem de propagation of uncertainties, pp. 47–58 in M. composition strategies for adaptive hp finite element Buecker, G. Corliss, P. Hovland, U. Naumann, and methods, Computing Systems in Eng. 6(2) 97–109. B. Norris, eds., Automatic Differentiation: Ap J. R. Pilkington and S. B. Baden (1994), Parti plications, Theory, and Implementations. Lecture tioning with spacefilling curves, CSE Technical Notes in Computational Science and Engineering Report CS94-349 Dept. Computer Science and vol. 50, Springer. Engineering, University of California – San Diego. K. Devine, E. Boman, R. Heaphy, R. Bisseling J. Ray, C. A. Kennedy, S. Lefantzi and H. N. Najm and U. Catalyurek (2006), Parallel hypergraph (2007), Using high-order methods on adaptively partitioning for scientific computing, in Proc. IP refined block-structured meshes derivatives, in DPS 2006. terpolations, and filters, SIAM J. Sci. Computing V. Eijkhout (1998), Overview of iterative linear 29(1):139–181. system solver packages, NHSE Review 3(1). P. Schwartz, M. Barad, P. Colella and T. Ligocki J. Gondzio and R. Sarkissian (2003), “Parallel (2006), A Cartesian grid embedded boundary Interior-Point Solver for Structured Linear Pro method for the heat equation and Poisson’s equa grams.” Mathematical Programming 96: 561-584. tion in three dimensions, J. Comput. Phys. 211(2) 531–550. B. Hayes (2003), A lucid interval, American Sci entist 91(6) 484–488. H. D. Simon (1991), Partitioning of unstructured problems for parallel processing, Computing Sys B. Hendrickson and R. Leland (1995), A multi tems in Engrg. 2: 135–148. level algorithm for partitioning graphs. in Proc. Supercomputing ‘95. J. Steensland (2002), Efficient partitioning of dy namic structured grid hierarchies, Uppsala Uni H. Johansson and J. Steensland (2006), A per versity Library, Uppsala, Sweden. formance characterization of load balancing alog

99 Section 6: Math and Algorithms

J. Steensland and J. Ray (2003), A heuristic re- mapping algorithm reducing inter-level com munication in SAMR applications, in Proc. 15th IASTED International Conference on Parallel and Distributed Computing and Systems 2003 (PDCS03).

R. W. Walters and L. Huyse (2002), Uncertainty analysis for fluid mechanics with applications. Technical report, NASA, Feb.

M. Woolridge (2002), An Introduction to Multi- Agent Systems. Wiley & Sons.

100 7 Software

Solutions to the major software, algorithm, • integrate knowledge discovery into the and data challenges in newly emerging pro- entire software life-cycle, from discov- gramming environments are needed to make ering bugs to monitoring and steering it possible for applications to scale up to the simulations to discovering new scientific required levels of parallelism and integrate results hidden in huge volumes of data; technologies into complex coupled systems and for real-world multidisciplinary modeling and simulation. Developing these solutions will • develop new approaches to handling the likely involve a shift from current static ap- entire data life-cycle of exascale simula- proaches for application development and exe- tions, from inputs that may be dynamic, cution to a combination of new software tools, live feeds, to distributed data analysis, algorithms, and dynamically adaptive meth- and long-term archival. ods. Additionally, we must bring together new developments in system software, data man- 1. Advances in the Next agement, analysis, and visualization to allow Decade disparate data sources (both simulation and real-world) to be managed in order to guide The infrastructure must be designed in a way research and to directly advance science. that allows it to adapt to future trends and remain relevant, correct, and high performance Computer vendors have little incentive to as systems evolve to exascale and beyond. tackle these challenges because the commer- The characteristics of these exascale systems cial market at this scale is relatively small. with millions of CPUs require the develop- For this reason, government support for the ment of new ideas and approaches. necessary R&D in exascale software is essential. It is both feasible and necessary to build on newly emerging programming environment In particular, significant new efforts are (interpreted broadly) technologies to develop needed to the infrastructure that will allow applications to scale up to the required levels of parallelism • fundamentally change how applications and integrate into complex coupled systems are built and maintained; for real-world multidisciplinary modeling and simulation. Considerable investment has been • improve scientists’ and administrators’ made in the past few years in new languages productivity when working with exascale designed to improve productivity for program- systems; mers. Examples are the Partitioned Global Address Space Languages and the High-Pro- • improve the robustness and reliability of ductivity Computing System (HPCS) lan- Exascale systems with systems and applications through fault guages. In 5–10 years, these efforts should millions of CPUs demand new tolerance, validation, and verification of have a chance to coalesce into one or two approaches to address the software components; standard, widely supported languages. challenges of the data tsunami, system reliability, and memory hierarchies.

101 Section 7: Software

braries, in some cases discipline- or even application-specific, which specify results to be obtained with less attention to the details of the computation than is currently necessary. Implementation of such libraries and languages will require lower-level programming models and tools that permit execution on a wide range of hardware and exploit the capabilities of exascale architectures.

• Rapid, modular construction of new applications from existing suites of interop- erable components. Scientific software components with well-defined interfaces, recently being prototyped but not yet widely adopted, have the potential to greatly increase code reuse, thus short- ening development times and increasing Figure 7.1 Scientific workflow diagram software reliability.

It is plausible to bring together new develop- • Coupling of multiple applications into ev- ments in system software, data management, er-larger applications through automated analysis, and visualization to allow disparate workflows. Single large runs remain an and voluminous data sources (both simula- important class of large-scale computation and real-world) to be managed in order tions, but many applications need param- to guide research and to directly advance sci- eter studies consisting of large numbers ence. of coordinated sets of runs, each perhaps consisting of a pipeline of computation For these developments to be useful at full and analyses. High-level, standard lan- scale, however, a more flexible and dynamic guages for coordinating such families of resource management capability is needed executions will enable scientists to focus throughout the computing environment to al- on science rather than “run management.” low simultaneous integration of computing, (See Figure 7.1.) analysis, visualization, and live data during a simulation. • Dynamic data storage and management. In order to support exascale data generation, The vision for the next decade is to have a data storage will fundamentally change. totally integrated approach to how applica- Much of the data will be archived. For ar- tions are built, modified, updated, and used chival storage to be energy efficient, yet in other applications. In this development available on demand, tools will be needed environment, the tools will interoperate and to manage the movement of data auto- assist the scientists in writing, debugging, matically across a storage hierarchy. Fre- tuning, and maintaining their codes. Such an quently used data will be moved to highly environment will support fundamentally new parallel dynamic storage, while archived approaches: data will reside in powered-down storage or passive storage devices. Furthermore, • New ways of specifying computations. algorithms for automatically tracking and Software components, Scientists must be freed from the de- removing unused data from dynamic stor- recently prototyped but not yet tails of managing data movement among age will be essential to minimize storage widely adopted, will shorten memory systems and synchronizing ac- costs. Collections of datasets will be or- development time and increase cess to shared memory among threads of ganized as directories. Such abstraction software reliability. control. They will need languages and li- will fundamentally change the way the

102 Modeling and Simulation at the Exascale for Energy and the Environment

I/O is expressed by applications and will Memory will increase in relative cost, and it The shift to multicore exascale involve a storage management layer that will be harder and harder to maintain the de- systems will require applications maps datasets into physical devices with- sired byte-to-flop ratio—in absolute capacity to exploit million-way out affecting the applications. (flops/s per byte) and bandwidth terms (flops parallelism. per byte). Achieving this vision will require fostering long-term, sustained, community-wide ac- Hence, applications will have to be redesigned tivity in evolving code suites. Large-scale to make better user of limited memory. This applications, like large-scale computers them- may mean more out-of-core solutions (pushing selves, require the support of multiple special- on the I/O bottleneck) or algorithms with ists within a single community. Indeed, the better storage characteristics. Additionally, community of computer vendors, application applications will have to deal with increasing scientists, and computer scientists, together hierarchies of memory (and indeed storage). with the hardware and software that they both There are now often five levels of direct- develop and use, forms an integrated, interde- access memory common (register sets, three pendent ecosystem. levels of cache, and main memory). In the future there may be more levels or more (or 2. Major Challenges less) sharing of these levels within an SMP.

Five major challenges must be addressed in 2.3 Increasing Parallelism order to realize efficient and effective exascale computing. In today’s environment, code development usually takes place assuming a homogenous 2.1 Improving Programmability run-time environment, with parallelization done manually by each code developer. At Exascale computer architectures will require the scale where applications need to make use radical changes to the software used to op- of millions of heterogeneous processes, dis- erate them and the applications that run on covering the opportunities for parallelization them. The shift from faster processors to mul- becomes much more difficult and requires a ticore processors is as disruptive to software set of tools that can automate the paralleliza- as the shift from vector to distributed memory tion of the trivially parallelizable segments supercomputers 15 years ago. That change of code, and aid the application developer required complete restructuring of scientific in finding less obvious opportunities. This application codes, which took years of effort. task is even more daunting when consider- The shift to multicore exascale systems will ing future multicore architectures, since the require applications to exploit million-way parallelization algorithms have to take into parallelism and significant reductions in the account in-core parallelism vs between-cores bandwidth and amount of memory available parallelism. These tools are also needed for to millions of CPUs. This “scalability chal- developing simulation code that will run well lenge” affects all aspects of the use of HPC. on heterogeneous hardware, with the com- It is critical that work begin today if the soft- piler automating as much of this as possible ware ecosystem is to be ready for the arrival and providing code-restructuring assistance of exascale systems in the coming decade. where automation is not possible.

2.2 Building New Applications 2.4 Handling the Data Tsunami

The transition from frequency-based scal- The data tsunami includes dealing with the ing to core-based scaling also means that the volume, different formats, transfer rates, community will face a memory crisis. This is analysis, and visualization of massive (poten- not just a bandwidth and latency issue that has tially distributed) data sets. been faced as memory gets farther away from CPU operations (at least in terms of clocks), Exascale applications running on as many but also an overall memory capacity problem. as a million processors are likely to generate

103 Section 7: Software

Exascale applications will data at a rate of several terabytes per second theme in this report—that existing methods generate several terabytes of (even assuming only a few megabytes per will not scale to meet the challenges of exas- data per second, presenting processor). Because it is not practical to store cale systems and data—holds true in the area enormous challenges in storage raw data generated at such a rate, dynamic re- of knowledge discovery. Existing approaches and management. duction of the data by summarization, subset for knowledge discovery will not evolve to selection, and more sophisticated dynamic the exascale. Failure to address the issues of pattern identification methods will be neces- knowledge discovery in the exascale ecosys- sary. The reduced data volume must be stored tem will have a profound adverse impact on at the same rate that data are generated, in or- all science programs. der for the exascale computation to progress without interruption. Knowledge discovery applications tend to be more complex than simulation codes in many This requirement presents new challenges in respects: they often have graphical user in- orchestrating data movement from the com- terfaces; they interact with distributed computation machines to local and remote storage puting and data storage resources; they rely systems. It will no longer be possible to store on a comparatively deep “software stack”; all the data locally and then distribute them they are used interactively and on parallel as secondary tasks. Data distribution will resources; they place great demands on sys- have to be integrated into the data generation tem resources—consuming vast amounts of phase. Here again, managing the dataflow us- memory, I/O bandwidth and processor cy- ing well-coordinated workflow engines will cles; and they generate many different forms be required as part of the software infrastruc- of output (movies, images, derived quantities, ture that runs the simulations. new datasets, etc.). The same arguments for software engineering, performance analysis, The issue of large-scale data movement will and optimization tools that apply to simula- become more acute as very large datasets or tion codes also apply to knowledge discovery subsets are shared by large scientific com- applications. munities. This will require the replication or movement of large volumes of data between As with other technology areas in this report, production and analysis machines, often algorithms and approaches for knowledge dis- across the wide area. While networking tech- covery in use today are not expected to scale nology is greatly improving with the introduc- into the exascale regime. Existing approaches tion of optical connectivity, the transmission will fail because of the sheer volume of data, of large volumes of data will inevitably en- the complexity of data to be processed, and counter transient failures, and automatic re- the growing impedance mismatch between covery tools will be necessary. size/complexity and human ability to understand and interact with knowledge discov- Another fundamental requirement is the au- ery infrastructure. A number of different, yet tomatic allocation, use, and release of storage complementary, approaches to address these space. Replicated data cannot be left in stor- problems will require exploration: (1) the abil- age devices unchecked, or storage systems ity to visualize and analyze results at coarse will fill and become clogged. A new paradigm and fine resolutions to support the natural in- of attaching a lifetime to replicated datasets, vestigatory process that relies on context/fo- and automatic management of data whose cus interaction; (2) better visual data analysis lifetime expires, will be essential. algorithms for characterizing and presenting uncertainty; (3) integration of visual data pre- 2.5 Accelerating Knowledge sentation and data analysis techniques (e.g., Discovery clustering, classification, statistical analysis, and representation) to aid in accelerating One of the principal bottlenecks in contem- knowledge discovery; (4) greater emphasis porary science is the process of discovering on the human-computer interface to increase knowledge and testing hypotheses in the pres- the efficacy of visual presentation motifs and ence of a growing deluge of data. A recurring interactive knowledge discovery interaction

104 Modeling and Simulation at the Exascale for Energy and the Environment models; (5) context-centric interfaces to sim- together with traditional sequential languages plify use of complex software infrastructure; (C, Fortran, C++). New architectures with and (6) rethinking the design and implemen- many cores per chip are expected to prevent tation of fundamental knowledge discovery this approach from exploiting exascale hard- algorithms and software infrastructure to ef- ware. Thus new approaches are needed. For fectively leverage exascale platforms. example, today’s software is largely MPI- based, with some global view techniques 3. State of the Art such as Unified Parallel C (UPC) and Co- Array Fortran (CAF). In order to facilitate the Devising ways to address the challenges use of extreme-scale resources, new “hybrid” identified in Section 2 requires a thorough programming models, or more radical ap- understanding of the state of the art. In this proaches as represented by the project initi- section we review the state of the art in sev- ated by the DARPA HPCS program, must be eral key areas. explored.

Knowledge Discovery. Algorithms exist for Data Management. Keeping track of data is analyzing terabytes of static data stored in a already a daunting task. The meaning of the single location, but very few analysis algo- data, referred to as metadata, requires precise rithms can handle a dynamic dataset distrib- capture of how the data was generated and uted across sites or streamed in live. Feature the scientific interpretation of each data item. detection is primitive or nonexistent in many Furthermore, many scientific datasets are science domains. Human interaction through generated from other datasets, or perhaps a visualization is today’s norm. In addition, dif- combination of datasets. This requires the ca- ferences in the rate of increase of performance pability of tracking the history, or provenance, in computer technology (such as memory and of the data. Today, such tools are provided in processors) and in storage technology (such ad hoc manner; some metadata is collected as magnetic disk and tape) will result in the in various forms of notebooks, some in da- need to achieve unprecedented levels of par- tabases, and some embedded as headers of allelism to enable the storage systems to keep files. In the exascale regime, the automation up with the computers. of this task is essential because of the sheer volume of the data and the accelerated rate of Application Building. As computational ca- their production. Standard metadata models pabilities have grown, so have the resolution and tools will have to be developed, as well as and complexity of the simulation models. tools to automatically capture the metadata as Today’s large simulation codes incorporate the datasets are generated. Furthermore, the multidiscipline, multiphysics, multiple time data models need to support standard ontol- scales, and multiple solution methods. They ogy for each scientific domain and allow for represent years of development by teams of dynamic evolution of such standards. programmers and scientists and can include millions of lines of code. As we make the leap Data Organization. Datasets are organized to exascale computation, the cost to update, and stored today as collections of files. The recode, and incorporate more advanced mod- sizes of files are often dictated by the storage els into the simulations is an order of magni- systems. For example, current mass storage tude higher than the cost of the supercomputer systems that use tape storage prefer certain hardware. In order to contain these costs, the fairly large size files, a fact that forces sci- exascale software ecosystem must support entists to either aggregate smaller files into more efficient program development. larger files or partition very large datasets into multiple files. This approach does not scale Visualization methods will be Programming Models. Current large-scale and is irrelevant to the scientists. Dealing critical for handling the growing applications have made the transition from with the volume of data generated by exas- impedance mismatch between shared-memory vector architectures to 100-TF cale machines will require support for data- the size/complexity of data and distributed-memory machines by using the sets regardless of their size. a person’s ability to understand message-passing programming model (MPI) and interact with that data.

105 Section 7: Software

Debugging Tools. An integral part of applica- will need to be developed and integrated into tion development includes verifying that code both existing and new applications. runs as expected. Current tools, limited mostly to debuggers, have not yet caught up with 4. Accelerating Development the needs of terascale computing, with many application developers for today’s large sys- Focused investments in the major software, tems falling back to a very inefficient method algorithm, and data challenges are needed to of debugging—dumping user-inserted debug scale and tune applications to the exascale. code to output files. With the vast increase Several flagship projects of a truly interdis- of process count going to exascale systems, ciplinary nature that address key national searching manually for a single anomalous problems should be selected and supported process among the millions of running pro- to motivate, guide, and validate the computer cesses and threads is not tenable. Moreover, science research results. Possible examples today’s debuggers are not scalable to a thou- are modeling the social impact of climate sand processors. Scientists will need to have change, modeling the global nuclear mate- their applications run on millions of proces- rials cycle, or modeling regional social and sors and require the tools to meet the overall economic futures at a new level of detail and simulation requirements. reliability.

Fault Recovery. Modern PCs may run for Software development requires a different us- weeks without needing rebooting. Today’s age model from the “production” running of supercomputers often run for only a few days applications and is often delayed by lack of before rebooting, because of their complex- access to scalable resources dedicated to the ity and their thousands of processors. Exas- development process. cale systems will be even more complex and have millions of processors. The scale of the Creating an exascale software ecosystem systems means that component failure is the entails more than just coming up with novel norm, not the exception. This requires a ma- solutions. It includes educating scientists on jor shift from today’s software infrastructure. how to use the solutions, both new tools and Every part of the exascale software ecosys- new approaches, and demonstrating why us- tem must be able to cope with frequent faults; ing these solutions is to their advantage. It otherwise applications will not be able to run includes making sure that the solutions are to completion. The system software must be hardened to production quality so that they designed to detect and adapt to frequent fail- can be integrated into the software suites of ure of hardware and software components. On the nation’s supercomputer centers. It in- today’s supercomputers every failure, even cludes making pieces available as they are those that are reconfigured around, kills the completed, rather than waiting until every- application running on the affected resources. thing is done. And it includes helping users These applications have to be restarted from integrate these pieces into existing codes so the beginning or from their last checkpoint. that science teams can benefit in the near term The checkpoint/restart technique will not be and build up trust in the solutions being pro- an effective way to utilize exascale systems, vided for the exascale software ecosystem. because checkpointing stresses the I/O system and restarting kills 999,999 running tasks be- To help applications make effective use of cause one fails in a million-task application. exascale systems, we need to go well beyond With the potential for exascale systems to the current state of affairs in the performance undergo constant failures somewhere across analysis process. Today’s tools are limited Searching manually for a single the system, application software will not be in scope, capability, and scalability, and anomalous process among able to rely on checkpoint/restart to cope with feedback to the application is manual. The the millions of running process faults since a new fault is likely to occur be- overhead associated with current measure- threads on exascale computers fore the application can be restarted. For ex- ment techniques is too intrusive at this size in simply not tenable: new tools ascale systems, new fault tolerance paradigms and may skew analysis so much as to render are essential. any analysis ineffective. Therefore, we need

106 Modeling and Simulation at the Exascale for Energy and the Environment to develop scalable and less intrusive meth- search) and analysis. No “one size fits all” Probability says that exascale ods of collecting performance data, develop formula will meet the needs of all of science. systems will spontaneously knowledge discovery methods for extracting Because of the diversity and complexity of change answers to wrong key performance features, and provide assis- knowledge discovery technologies, a broad values occasionally. Formal verification methods are needed tance in feeding the results of these analyses technical portfolio of projects will help to to ensure that calculations are back to the code transformation. maximize the likelihood of success. done correctly. For simulation codes to be able to run cor- 5. Expected Outcomes rectly as well as effectively, in a reasonable time frame, an investment in developing for- Successful development of the flagship ap- mal verification methods is essential. The plications identified earlier would be of real scale and complexity of the science problems benefit to the government and society. Simu- enabled by exascale systems require new lation could inform policy development in a techniques for making sure that the calcula- number of diverse areas, including emergen- tions are done correctly. Cosmic rays have cy planning, health system improvement, and been shown to change the memory values in management of climate change. a supercomputer, causing wrong answers to be calculated. In another supercomputer it The development of predictive power in was discovered that in one in a billion data validated models connected to real-time data transfers, correct values sent to another pro- would also provide economic value to busi- cessor would emerge from the wire as wrong nesses that must respond to rapid changes in values. Probability says that exascale systems the operating environment, such as airlines, with memories bigger than any computer on power companies, and emergency manage- the planet and the ability to calculate and send ment operations. trillions of bits of data per second will spontaneously change answers to the wrong values But without doubt the major impact of this occasionally. One approach is to incorporate initiative is a software infrastructure that en- algorithms into the system software and have ables new applications beyond the initial flag- the application software catch and correct any ship applications. This software infrastructure spontaneous mistakes caused by phenomena will enable a phase transition in large-scale such as cosmic rays. scientific simulation and modeling.

Investment in the reliability and robustness 6. Required Investment of exascale systems for running large simulations is critical to the effective use of these As noted in Section 4, focused investments in systems. New paradigms must be developed the major software, algorithm, and date chal- for handling faults within both the system lenges are needed. An investment of $100M software and user applications. Equally im- over five years would make a real impact on portant are new approaches for integrating these challenges. The exact details of its dis- detection algorithms in both the hardware and tribution would need to be worked out to re- software and new techniques to help simula- spond to realistic proposals. tions adapt to faults. 7. Major Risks Knowledge discovery R&D programs are crucial to scientific discovery at the exascale One major risk is a disruptive shift in hard- Many potential approaches merit exploration. ware technology in the next 5–10 years that One is to include some knowledge discovery facilitates a complete change in the approach software in the simulation itself. This ap- to data analysis, programmability, and inter- proach could have the benefit of reducing the active computing. Another risk is the sheer I/O load for the simulation. Another approach complexity of the software systems eliciting entails a closer coupling between knowledge unexpected problems at the scale of the com- discovery and related technologies, such as puters and volume of data produced by simu- data management (I/O, movement, index/ lations and experiment in this time frame.

107 Section 7: Software

The risk in allowing current trends to contin- J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur, ue slowly is that while scientists will envision W. Gropp, R. Latham, A. Siegel, B. Gallagher, and breakthrough computations and the requisite M. Zingale (2003), Parallel netCDF: A high-per hardware will be available, the software in- formance scientific I/O interface, inProc. SC2003, frastructure for programming, executing, and Phoenix, AZ. understanding the results of these exascale computations will be inadequate to the task. E. Lusk and Kathy Yelick (2007), Languages for high-productivity computing: The DARPA HPCS References language project, Parallel Processing Letters 17(1), March. Ray Bair, Lori Diachin, Stephen Kent, George Michaels, Anthony Mezzacappa, Richard Mount, J. Parker, T. Engelsiepen, R. Ross, R. Thakur, Ruth Pordes, Larry Rahn, Arie Shoshani, Rick R. Latham, and W. Gropp (2006), High-perfor Stevens, and Dean Williams (2003), Planning mance file I/O for the BlueGene/L supercomputer, ASCR/Office of Science data-management in Proc. 12th International Symposium on High- strategy. http://www-conf.slac.stanford.edu/ Performance Computer Architecture (HPCA-12). dmw2004/docs/DM-strategy-final.doc. R. Ross, Jose Moreira, Kim Cupps, and Wayne J. Duell, P. Hargrove and E. Roman (2002), The Pfeiffer (2006), Parallel I/O on the IBM Blue Design and Implementation of Berkeley Lab’s Gene /L system, BlueGene Consortium Quarterly Linux Checkpoint/Restart, Berkeley Lab Techni Newsletter, February. cal Report LBNL-54941. Rob Ross, Evan Felix, Bill Loewe, Lee Ward, Gary R. L. Graham, S.-E. Choi, D. J. Daniel, N. N. Grider and Rob Hill (2005), HPC file systems and Desai, R. G. Minnich, C. E. Rasmussen, L. D. scalable I/O: Suggested research and development Risinger and M. W. Sukalksi (2003), A network- topics for the fiscal 2005–2009 time frame, ftp:// failure-tolerant message-passing system for teras ftp.lanl.gov/public/ggrider/HEC-IWG-FS-IO- cale clusters, Int. J. Parallel Programming 31(4) Workshop-08-15-2005/FileSystems-DTS-SIO- 285-303. FY05-FY09-R&D-topics-final.pdf.

W. Gropp and Ewing Lusk (2004), Fault tolerance Douglas Thain, Todd Tannenbaum and Miron in Message Passing Interface programs, Int. J. Livny (2005), Distributed computing in practice: High Perform. Comput. Appl. 18(3):363-372. The Condor experience, Concurrency – Practice and Experience 17(2–4) 323-356.

108 8 Hardware

Exascale computing requires innovation at factor in these systems’ increasing capabili- the frontiers of computer architecture and in- ties. This trend is sure to continue in the near formation technology. Major challenges exist future. The current rate of growth points to- in providing sufficient compute performance ward a concurrency level for the high end of per watt, memory performance, interconnect 107 cores by 2015 and 108 cores by 2020. An performance, and I/O and persistent storage additional switch from multicore to many- performance and reliability. In addition, evi- core technology during this time frame could dence suggests that both software and algo- easily increase these levels by an order of rithms will need to be developed in concert magnitude, from 108 to 109 cores. with hardware to ensure that both legacy and new exascale applications will be able to Many of these system trends (and their conse- make effective use of the advanced systems. quences) can be explored further by examin- In addition, since such systems are likely to ing the growth trends for critical performance have 10 million to 100 million separate pro- factors for the commodity component tech- cessing elements and 10 to 100 petabytes (PB) nologies. Based on historical trends, in 2015, of memory, fault detection and handling by one potential exascale system could require hardware, software, and algorithms will be 1.3M PEs, where each PE generates 768 gi- essential. With sufficient investment, -par gaflops, derived from 64 cores, where each ticularly in ongoing point-design studies and core produces 4 flops per cycle and operates hardware-assisted simulation, exascale sys- at 3 GHz. Assuming that the PEs would re- tems can become an effective resource for quire 100 W each, the resulting exascale sys- attacking complex science and engineering tem alone would require 130 MW of power. problems. Assuming 1 GB per core, the resulting exascale system would have 83 PB of memory, 1. Advances in the Next composed of about 5 million memory parts. Decade The bisection bandwidth and scale of the interconnect network will depend on the topol- To understand that challenges of constructing ogy and the switch/router configurations, but an exascale system, one must first understand they will also be limited by the signaling and where current technology trends are leading bandwidth rates. Also, the scale of the entire the community. While there is always risk in system will be limited by the physical length extrapolation, the 14-year history of the Top of cables in the entire system. Storage sys- 500 data (Figure 8.1) suggests that without tems will continue dramatic increases in ca- further technology acceleration the first sys- pacity (approximately 50% compound annual tem exceeding 1 exaflop LINPACK perfor- growth rate), while disk bandwidth and seek mance can be expected by 2019. Based on latencies will improve only slightly, because past programs, it is therefore believable that a of their mechanical characteristics. An exas- Exascale systems are likely concerted, sufficiently funded, multiyear re- cale system capable of checkpointing 50% of to have up to 100 petabytes search program should be able to accelerate its physical memory in 10 minutes would re- of memory, enabling new the availability of exaflop technology by four quire approximately 250,000 disks. applications but also requiring years, to the 2015 timeframe. Concurrency major innovations in software in existing systems continues to be a primary and algorithms.

109 Section 8: Hardware

several years. A different semi-customized architecture for climate computing [Wehner et al. 2007] shows promise of enabling kilometer-scale climate modeling decades before it would be possible through the conventional Moore’s law technology improvements.

Three designs, summarized in Table 8.1, illustrate some of the challenges for exascale computing. New processor and system architectures that are optimized for better power efficiency will be necessary if total system Figure 8.1 Actual and projected Top 500 performance. power is to be kept under 30 MW. Since much of the performance will come from orders of The slowing pace of commodity micropro- magnitude more processor cores than in the cessor performance improvements, combined current systems, innovations in hardware, with ever-increasing chip power demands, has software, and algorithms will be needed to become of utmost concern to computational make effective use of these systems. scientists. As a result, the HPC community is examining other approaches that address While most of the above discussion has fo- the limitations of conventional large-scale cused on the processing elements, a capabil- computing systems in the coming decade. ity platform must also provide access to a Already, commodity processors for graph- sufficiently large memory hierarchy, an effec- ics and computer games achieve far higher tive interconnect between the processors, rea- computational peaks per socket and per watt sonable I/O capabilities, and networking to than conventional processors (this includes the outside. For example, in order to achieve both GPGPUs and the Sony/Toshiba/IBM sustained performance in the 10–20% range Cell processor). These processors have many on applications such as computational fluid drawbacks for use in scientific computing, dynamics (direct numerical simulation), including single-precision arithmetic, but high-energy physics (quantum chromody- could easily evolve over the next decade to namics) and computational biology (molecu- provide suitable computing platforms. Em- lar dynamics), our in-depth understanding of bedded processors also provide very high ef- today’s algorithmic and subsequent commu- ficiencies in terms of operations per watt; in nication requirements drive a system balance fact, the processor in IBM’s Blue Gene ma- requiring injection bandwidth on the order of chines (the world’s fastest computer in 2007) 150 GB/s, order 1.5 PB/s global bandwidth, is a System on Chip (SOC) design employing and order 500 nanoseconds latency. These conventional PPC440 embedded processor drive the technology requirements in the ar- cores. Similarly, the SiCortex system em- eas of processor, interconnect, I/O, and stor- ploys SOCs with MIPS/embedded processor age design. The applications will also require cores to achieve much lower power utiliza- 0.5–2 bytes/flop of memory bandwidth, and tion for sustained performance [Reilly et al. we should assume at least 2–4 GB of memory 2006]. Like BG/L, however, this approach per CPU core. accelerates the march toward massive concurrency and motivates fundamental advanc- 2. Major Challenges es in applied mathematics to develop scalable algorithms. Other recent examples of special- Architecture developers face a number of ized systems include the MD-Grape system challenges. These involve two broad ques- in Japan, developed for less than $10M. A tions: How should one engineer the major semi-customized architecture for molecu- system components, and what are the impli- lar dynamics under development at D.E. cations of these designs on software and ap- Shaw (Shaw et al. 2007) promises to speed plication development? Several issues must high-fidelity protein-folding simulations by be addressed:

110 Modeling and Simulation at the Exascale for Energy and the Environment

Peak/ Peak/ Example Ops/ Freq Cores/ Total Power socket Sockets system system cycle [GHz] socket cores [MW] [TF/s] [EF/s] A 4 3.0 64 0.768 1300k 85M 1.0 130

B 8 16.0 128 16.0 120k 15M 2.0 60 - 80

C 8 1.5 512 6.1 200k 100M 1.8 20 - 40

Table 8.1 Some key parameters of exascale systems based on rough projections from current technologies.

• Sufficient effective interconnect perfor- less-conventional architectures (these mance, particularly latency, bandwidth, would help open up the space of hard- and cross-section bandwidth ware approaches)

• I/O (persistent store and file systems), • Acceleration of applications and algo- particularly data density, transfer band- rithms (especially new ones) to petaflops width, and fault management now to prepare for exaflops

• Memory (main system memory), particu- • Programming languages and enviro- larly cost and size, power efficiency, ac- nments cess latency, bandwidth, and number of hierarchy levels – PGAS, domain-specific, auto-tuner, hierarchical programming models • Power consumption everywhere (built on current models)

• Processor architecture and implementa- – Interaction with hardware (e.g., us- tion, particularly latency and bandwidth er-managed caches, remote atomic management, concurrency levels, hetero- updates) geneous designs, and fault frequencies – Performance modeling and debugging • Software and algorithms, particularly Productivity as they provide workarounds and alternative approaches to deal with latency, – System software, operating system bandwidth, memory hierarchy, faults, (e.g., memory management) heterogeneity, and highly increased con- Because of cost and power limitation, an ex- currency levels ascale system by the year 2015 will likely In addition, a number of crosscutting issues not be a general-purpose system in the cur- need to be addressed. rent sense. The design of such a system will require some compromises with respect to • Better characterization of algorithm re- size of system resources such as memory or quirements with respect to system ratios disk space and with respect to system ratios such as cross-section bandwidth to link band- • New algorithms to match system ratios, width to memory bandwidth. A sufficient un- particularly between disk and main mem- derstanding of the requirements of exascale ory and the bandwidth of the interconnect applications is needed to guide the choices to the flops rate of the processor necessary for these compromises, so that the • New algorithms and software to detect final design supports a reasonably large frac- and handle faults tion of all potential applications.

• New approaches to algorithms and soft- Future storage architectures will have to be ware for specialized/disruptive processor designed to be an integral part of the over- architectures; that is, better methods for all system. Currently I/O is an afterthought, moving applications to GPGPUs, PIMs, which is then “bolted” onto a system. The FPGAs, heterogeneous systems, or other bandwidth requirements for systems of this size will prohibit this behavior in the future

111 Section 8: Hardware

for balanced systems. Also needed are the tem simulation and as platforms for early development of new storage semantics that application development and performance enable high performance and scalability, the evaluation (e.g., Berkeley RAMP [Kras- integration of database concepts (relational nov et al. 2007] and emulators for com- and object models) into the notions of file mercial FPGA-ASIC design flows from systems and storage models, and the creation companies such as Altera [Blyler 2005]). of effective associative access methods for integrated memory/data systems. • Quantum computing technology solves a set of problems that are orthogonal to the 3. State of the Art approach taken by existing computational technology. Therefore, quantum comput- Today’s state of the art is reflected in several ing offers an opportunity to expand the approaches for petascale systems: scope of problems that can be addressed with computing as a complementary ap- • Strongly increased concurrency levels to- proach to computing rather than supplant- gether with reduced clock-rate to manage ing current computational methods. There power consumption; several specialized has been quite a bit of progress in the ba- system networks (including collective sic and underlying technologies, but to networks); commodity processing-core date nothing has been realized beyond the architectures, but integrated in custom basic hardware layer and very small (<16 chips (IBM BG/C, L, P, Q; SiCortex qbit) systems. Maintaining quantum en- (www.sicortex.com); DoD-funded pro- tanglement for larger-scale hardware has cessors). These changes in system design proven difficult, but start-up companies require a heavy investment in adapting such as D-Wave [Maxcer 2007] have tak- the software infrastructure. The ability of en the first steps toward commercializa- rapid integration of commodity cores in tion of quantum computing technology. custom processor chips is opening up a A breakthrough in quantum computing large potential for semi-specialized sci- would imply an enormous software in- entific computing systems with high per- vestment for anything but critical hand- formance and power efficiency. written or hand-tuned applications.

• Modified instruction sets and memory Today’s software for high-end machines is architectures that attempt to better match largely based on message passing, sometimes current memory hierarchies and in some using a thread model for each MPI process on cases to increase power efficiency (IBM/ a shared-memory node. As machines grow in Sony/Toshiba Cell; stream processors; scale and complexity, techniques to make the PIM; Cray Black Widow and Cray El- most effective use of network, memory, and dorado/SUN Niagara; GPGPU). These processor resources will also become increas- approaches require even heavier invest- ingly important. Programming models that ment in the entire software development rely on one-sided communication or global chain. address space support have demonstrated advantages for productivity and performance, • Commodity processors plus special- but they are most effective when used with ized networks [Cray XT/Cascade, IBM proper hardware and operating system sup- PERCS [Elnozahy 2006] (HPCS)] follow port. Global view languages, such as UPC established engineering models but cur- and CAF, are seeing some use and are likely rently face increasing power efficiency to be important in the next 5–10 years (see, problems. for example, Coarfa et al. 2006).

Balanced exascale systems • Reconfigurable processors and FPGAs Many feel that new programming models and will require a new approach to are still plagued with a very limited soft- languages are needed for machines at this I/O — which currently is simply ware development environment. They do scale. The same view was held for petascale “bolted” onto a system. show potential for rapid large-scale sys- machines, yet clever use of hierarchical ap-

112 Modeling and Simulation at the Exascale for Energy and the Environment proaches and good algorithms have allowed Technology Issue Addressed the simple message-passing approach to succeed with the current near-petascale sys- Optimizing the use of die Power efficiency, sustained performance tems. The DARPA HPCS program [Johnson space for CPU 2007] has fostered the development of three Optical network Interconnect performance, system scalability new languages: Chapel, Fortress, and X10. Optical in/out of processor Interconnect performance, power efficiency Exploration of these languages is important, particularly as they promise enhanced pro- 3D chips; integrated memory/ Memory performance, packaging density, scal- processor able system hierarchy ductivity and could help increase the number Faster development of cus- Technology acceleration, memory performance, of application areas that made effective use of tomized processors system scalability high-end computing. We note, however, that Hardware accelerated system Software and algorithm development, perfor- it typically takes 10 years for a language to verification (e.g., RAMP) mance modeling and predictions, accelerated reach the level of maturity required by most system design applications. NAND Flash, MRAM, other Scalable system hierarchy, power efficiency, I/O non-volatile memory performance Many promising technical developments can Myriad approaches to power Power efficiency address these issues. A sampling is shown in efficiency Table 8.2. Table 8.2 Promising technologies for exascale computing. 4. Accelerating Development detailed analysis and guidance about program directions at all stages of this Accelerating the availability of exascale com- process. puting requires concerted efforts in several areas. Early evaluations of different potential • Establish ongoing research for early sim- technologies are key for technology accelera- ulation and modeling of future hardware tion. These include evaluations of current and and the impact of promising or disruptive new hardware, software, and application de- technologies. This will require modeling signs. The results of such evaluations must be the elements of a system as well as mod- presented and shared openly. Only by itera- eling entire systems. Existing simulation tively improving on one another’s research efforts need further support to extend can the high-performance computing com- their capabilities for the required scale munity as a whole make the most rapid prog- and to be available in the future whenever ress toward a common goal. In particular, the new technologies are proposed. Promis- following steps can be taken to accelerate ing approaches for different levels of development. simulation include FPGA-based systems (e.g., Berkeley RAMP) and highly paral- • Develop technology assessments (5 years lel simulation software on conventional out) and “point design studies” (10 years systems. out) that investigate in more detail the specific challenges in different design • Develop new algorithms and supporting approaches, including all-commodity software. The rapidly increasing concur- and partial custom designs. These design rency levels and the increasing complexi- studies should represent a continuous ef- ty of system hierarchies pose tremendous fort to understand future technologies and challenges to application and basic soft- their impact and should follow a common ware developers. These challenges need evaluation methodology, which needs to to be addressed as soon as possible to be established. They should evaluate the give these communities time to have their impact of specific hardware components software ready as soon as exascale hard- on operating systems, algorithm designs, ware becomes available. and application performance as well as their impact on the application develop- • Conduct point design studies and early ment process in general. These design application development. These ar- studies are essential to provide more eas each require advanced system and

113 Section 8: Hardware

performance modeling and evaluation able vendors to shift some development methodologies. Only new evaluation risk to early systems revenue. methodologies based on application requirement can provide accurate feed- • Find the optimal point between fully gen- back about utility of new technologies. eral (one solution that does nothing well) Advanced performance modeling and and fully specialized (solves one prob- prediction methodologies are necessary lem but does not help with any others) to aid evaluation of future systems and in hardware, software, and algorithms. to guide application development. These This technique has proven itself already evaluation and modeling methodologies in many cases. Additional funding to en- have to be combined with simulators, hance an already planned product line both cycle accurate and system accurate, allows the government to optimize its for hardware, software, and application funding strategy and fits well with the development and assessments. proposed solution technologies. In some sense, algorithms are already here, and • Invest in truly innovative hardware (with the hardware is moving in this direction a 5- to 10-year development horizon). (e.g., Eldorado); software, particularly Such research can be facilitated through languages, needs to take this approach research partnerships between laborato- with domain-specific approaches. ries, universities, and end users, along with the vendors who would be the pri- Efforts should embrace open source for all mary developers of full systems. Re- aspects (not just software) to allow the maxi- search finding should be committed to a mum leverage from each advance. Multiple broad set of paths with checkpoints and prototypes should be built and evaluated; costs incentives for acceleration of technology can be managed by building slices of poten- into production systems. Currently such tial systems. A framework that follows efforts efforts remain unfunded from the govern- from the development of a point design to a ment perspective in various technologies. full system without multiple points of evalu- Without funding, necessary technologies ation will be essential in guiding the develop- will not be available in time. ment of an exascale system. • Invest in hardware emulation technolo- 5. Expected Outcomes gies that make research into innovative hardware design more cost-effective for Increased investment could have a positive academic researchers Such environments impact—substantial investments would steer also accelerate the feedback loop between innovation in hardware, software, systems de- hardware and software development. sign, and the necessary infrastructures, while smaller investments will not achieve the critical • Commit to building and experimenting mass needed to effect the necessary changes. with small-scale systems for node architecture changes and massively paral- Accelerating the development of the building lel (but lower-performance) systems for blocks for exascale systems would have dra- scaling studies. Such early prototypes matic commercial spin-offs outside the scope should complement modeling and simu- of the initiative, into areas such as medical im- lation efforts. They allow more precise age processing, intelligent sensors, and more technology assessments and will be es- capable robotics, including the prospect of sential for facilitating early software and fully automated, fully capable vehicles and air- application developments. craft. The improvement in the ability to broadly develop applications for parallel systems • Provide significant incentives for the ear- will have an overall boosting effect on the U.S. ly participation of applications in design software development enterprise, which is cur- Multiple prototypes should be and evaluation efforts. Commit to a de- rently the most productive by far. It would also built and evaluated to guide ployment timetable and roadmap that en- help to provide a high-value trajectory for U.S. development of exascale software development, as the lower-value ele- systems.

114 Modeling and Simulation at the Exascale for Energy and the Environment ments are more easily migrated offshore. This matically alternative technologies (quantum, True innovation demands that would reaffirm the lead once held by the Unit- biological, nano, combination, etc.) hardware architects, software ed States in many technological areas where developers, and users all be we are no longer considered a competitor. Failing to couple hardware, software, and engaged from the start. applications. It is critical that the coupling of 6. Required Investment the critical elements for success happen early and often in this program. The hardware archi- At a minimum the program of R&D on the tects, the operating and tool developers, and hardware-related and associated systems- the scientific users must all be engaged from related technologies would need to ramp the beginning to provide the critical feedback to approximately $100M–$150M per year. loops that are needed for true innovation. Early progress could be made for less than that in launching the innovation efforts. The References deployment of large-scale systems from each J. Blyler (2005), Navigating the Silicon Jun generation of development would require ap- gle: FPGA or ASIS, Chip Design, June/July, proximately another $150M per year (assum- http://www.chipdesignmag.com/display. ing three rotating deployments of $50M per php?articleId=115&issueId=11 year each). C. Coarfa, Y. Dotsenko, J. Mellor-Crummey, F. 7. Major Risks Cantonnet, T. El-Ghazawi, A. Mohanty and Y. Yao (2006), An evaluation of global address space lan- We have identified four major areas of poten- guages: Co-array Fortran and Unified Parallel C, tial risk in establishing an exascale hardware in Proc. Principles and Practice of Parallel Pro- initiative. gramming (PPoPP), New York.

Failing to establish vendor buy-in. Industry is M. Elnozahy (2006), IBM has its PERCS, HP CWire 15(14), April. http://www.hpcwire.com/ likely to be a major participant in development; hpc/614724.html. in addition, industry will need to develop and bring these systems to market. Industry must F. Johnson (2006), DARPA HPCS Program, Sci believe there is a long-term federal commitment DAC Review 2, Fall, http://www.scidacreview. to deployment to be able to provide substantial org/0602/html/news2.html cost-match during the development. The A. Krasnov, Andrew Schultz, John Wawrzynek, hardware development program should include Greg Gibeling and Pierre-Yves Droz (2007), both large and small companies and also a RAMP Blue: A message-passing manycore sys range of technologies from somewhat risky to tem in FPGAs, in Proc. FPL 2007 - International bleeding edge. Some of these are already being Conference on Field Programmable Logic and discussed, have vendor acknowledgment, and Applications, Amsterdam. are looking for funding agents. They could be brought on line in less than 12 months. C. Maxcer (2007), D-Wave claims quantum com puting breakthrough, TechNewsWorld, www.tech Not aiming high enough. Research must be newsworld.com/story/55801.html. aimed at the long term to achieve transforma- M. Reilly, L. C. Stewart, J. Leonard and D. Gingold tional impact, and it needs to be stably support- (2006), SiCortex technical summary, www.sicor- ed to achieve the innovative breakthroughs we tex.com/whitepapers/sicortex-tech_summary.pdf. are looking for. Focus on near-term advances will stifle innovation. We also must not under- D. E. Shaw et al. (2007), Anton, a special-purpose machine for molecular dynamics simulations, in estimate the scale of the potential impact of Proc. 34th Annual International Symposium on the combined exponentials of improvements Computer Architecture, San Diego, pp. 1-12. in processor performance, storage capacity, networking, and user interfaces. The rate of M. Wehner, L. Oklier, and J. Shalf (2007), To improvement of these four factors will mean wards ultra-high performance resolution models that things will be dramatically different of Climate and Weather, Intl. J. High-Performance 10–15 years from now. We must also have Comp. Apps., to appear. a component of the program looking at dra-

115 Section 8: Hardware 9 Cyberinfrastructure

Today’s cyberinfrastructure is a loose cou- the researchers building on top of it. Ensuring pling of libraries, tools, and policies that sup- such a cyberinfrastructure, and addressing ports researchers by enabling the application the related cyber security issues, will require of multiple, distributed resources to a scien- considerable investment to support future tific problem. Even for the most basic work- research. Areas to be investigated include flow, such as collecting data, performing a workflow management; collaboration frame- high-performance computation, and analyzing works and techniques; data management and the result, it is rare that all of the necessary peo- movement of exascale datasets and data col- ple, computing, storage, and analysis systems lections; authorization and authentication are within the same location. Beyond the rela- for flexible interdisciplinary computational tively simple problem of moving data across science teams (“virtual organizations”); and networks is the much more challenging issue cyberinfrastructure management tools, tech- of managing secure access across multiple or- niques, and methodologies to understand the ganizational boundaries. Today’s cyberinfra- performance of the infrastructure and to pro- structure is already showing signs of weakness tect it from attack, disruption, and data loss. from the standpoint of technical capabilities (data movement and analysis, for example) in 1. Advances in the Next the transition from terascale to petascale sci- Decade ence, suggesting that either our approach must be radically adjusted or we must accelerate the The challenges associated with cyberinfra- development of solutions to these challenges, structure are driven by the increased size and or both. Beyond technical challenges associ- complexity of applications and datasets; the ated with speed and size of resources, we see need to combine computational, experimen- distributed science teams growing larger. This tal, information, and analysis resources to situation, combined with the distributed nature support the scientific workflow; and the in- of the resources, presents a fundamental dif- creasingly distributed and multidisciplinary ficulty with respect to authorization and ac- nature of the teams that will be tackling cess. Traditional HPC has been a client-server these problems. game, with a set of one-to-one trust relationships between independent computing centers Five critical cyberinfrastructure areas have and individual users. As we move toward ex- been identified as necessary and feasible to ascale science, these one-to-one relationships support exascale science: workflow manage- transform into many-to-many matrices, with ment, collaboration frameworks and tech- distributed multidisciplinary teams working niques, data management and movement, together to harness resources from multiple cyberinfrastructure management tools, and Unlike traditional high- service providers. cyber security. Success of these teams in performance computing, working together and harnessing an inherent- exascale science will Cyberinfrastructure provides the foundation ly distributed set of resources will have major involve many-to-many trust for a wide class of users, from scientists to impact on the world we live in and America’s relationships, with distributed system administrators to end users. This role in such a world. multidisciplinary teams foundation must be well defined, secure, per- harnessing resources from sistent, robust, and increasingly transparent to multiple service providers.

117 Section 9: Cyberinfrastructure

Workflow management. Exascale applica- Google Gears does for web applications). tions will generate vast amounts of data. The Such a process should happen in as natural process of analyzing that data will require an environment as possible, so that scientists correspondingly vast computational power. perceive practically no change in their Equally important, the systems, toolkits, and personal working environment, yet interact user interfaces through which scientists will with and benefit from the current shared explore and analyze the data must be engi- state of the collaboration (see, e.g., [Stevens, neered for automated end-to-end data trans- Papka, and Disz 2003]). port, resource management, and security and integrity. Workflows comprise the computa- Data management and movement. Exascale tions and data analysis tasks that are composed science will generate data at rapidly increas- of a (potentially vast and complex) sequence ing rates, causing both short- and long-term of related but distinct jobs, including one or challenges that data management and move- more data preprocessing steps (e.g., data as- ment will need to address. Researchers today similation and cleaning), followed by a series are already creating terascale and petascale of computational steps related by complex datasets and discovering that they are spend- dependences, followed by postanalysis. Be- ing a significant portion of their time man- cause the engineering of such workflow sys- aging data rather than performing scientific tems at the exascale is a daunting problem in investigation. Data management and move- both architecture and software development ment tools for exascale datasets and data col- and quality assurance, it will be vital for work- lections must provide capabilities for data flow components such as job management, virtualization [Nefedova et al. 2006], man- dataset or data collection management, and agement of properties and attributes indepen- account and access control management to be dent of the underlying storage system, access shared and used across many diverse science control, provenance metadata, data integrity, communities (for recent work on community and data security. accounts, see [Welch et al. 2006]). Cyberinfrastructure management tools. The Collaboration frameworks and techniques. computational systems that will enable exas- Scientific pursuits have become increasingly cale science will be orders of magnitude more compute-intensive, collaborative, and complex than today’s resources. Performance geographically dispersed; exascale science tuning, troubleshooting, and systems man- will continue this trend with unprecedented agement capabilities on today’s terascale sys- distributed collaborations involving scientists, tems are barely keeping up with adequately their data, and the compute resources. Tools monitoring, reporting usage, and providing currently available for collaborative science troubleshooting capabilities for high-perfor- center largely on interactive presentations mance networks, storage area networks, HPC using video- and web-conferencing for systems made up of hundreds or thousands of synchronous collaboration and email, wikis, nodes, and archival systems made up of mul- and blogs for asynchronous collaboration. tiple petabytes of storage resources. Tools The need for collaborative technologies is that support local and remote job monitoring, evident in the recent explosion of tools for web system monitoring, configuration manage- collaboration, integration of collaboration ment, and troubleshooting are necessary for into standard business tools, and the success the well-being of these expensive resources. of commercial web-conferencing offerings. Many of the systems at DOE supercomputer Scientists should have a current image of the centers and data centers are unique resources state of their collaboration available to them that expand far beyond the development en- in their personal working environment (e.g., vironments provided by the vendors for their laptop, desktop) at all times. To the extent system management and performance tool Workflow management at possible, they should be able to interact with developers. Often, system owners develop the exascale is a daunting their local state when they are offline, and the tools for these specialized systems either problem in both architecture synchronize this state with the appropriate on their own or in cooperation with grid proj- and software development. group state when they return online (e.g., as ects. These tools, whether developed by the

118 Modeling and Simulation at the Exascale for Energy and the Environment system owners or the vendors, are not capable of scaling to exascale resources. R&D is re- ITER quired to develop new methodologies, tech- Tokamak Device niques, and tools capable of managing these complex systems. In addition, coupling this effort with the data management and movement tools developed for exascale datasets and data collections will prove useful for dealing with the huge quantities of environmental, systems events, and job management data that will be generated by these systems.

Cyber security. Currently every site performs some level of security monitoring using a combination of network and host intrusion detection systems (IDSs), intrusion preven- Fusion Facility tion systems (IPSs), network flow monitor- Control Room ing, vulnerability scanning, and so on. It is essential that the network and host monitoring mechanisms, such as IDS, IPS, and firewalls, Figure 9.1 Large distributed teams collaborate to run experiments at fusion scale to support the dynamic and high-speed facilities. Remote participation and collaboration will become even more critical to the U.S. fusion science community, because the next-generation fusion device, the networking environment that will be required international ITER project, will not be located in the United States. for exascale computing. For large facilities, 100 Gbps networks will be common in 5 to 10 years. Today’s commercial security offer- Collaboration frameworks and techniques. ings show no signs of being able to keep up. Scientific collaborations increasingly involve more people, more computers, and larger data- 2. Major Challenges sets collected across greater distances than ever before. The challenge with the greatest The scientific and technological efforts pro- user visibility is representing this information posed as part of this initiative will pose major and resource overload in a natural, usable, col- challenges for cyberinfrastructure and cyber laborative manner so that it is understandable security. R&D timelines will require 5 to 10 and accessible to the researchers involved. years in order to ensure effective scaling and efficient use of systems at the exascale. Collaboration-based authentication and authorization. The cross-site and international Workflow management. Significant improve- nature of DOE Office of Science collabora- ment in workflow management at the exascale tions demands a well-managed, scalable, will require research into the identification of flexible, and federated approach to authenti- common workflow requirements and prob- cation and authorization and to the creation lems in the different science disciplines and and management of the virtual organizations communities of users. Research will be re- that manage collaboration resources. Current quired in order to understand fundamental approaches are disparate and impose a high workflow issues across science disciplines overhead on scientists and security profes- and to identify effective solutions (for recent sionals, without producing a high assurance efforts, see von Laszewski et al. 2007; Zhao, that proper authentication of user identity has Wilde, and Foster 2007]. The development been achieved. There is inadequate support for of tools and methods that follow recognized the specialized needs of distributed collabora- standards and find widespread adoption will tions, such as dynamic assignment and man- be a challenge, if we are to prevent waste agement of attributes such as roles and group of scarce resources and avoid reinvention of membership to individuals and resources in a solutions to common problems that may al- virtual organization. Interorganizational trust ready be solved.

119 Section 9: Cyberinfrastructure

We envision a cybersecurity is fundamental to the operation of virtual situational awareness (SitAware) capability situational awareness capability, organizations. to provide analysts with accurate, terse sum- “SitAware,” that provides maries of attack traffic, organized to highlight accurate summaries of attack Data management and movement. Manage- the most prominent facets. These summaries traffic and analyses that help in deploying countermeasures. ment of large datasets and data collections would include the essential elements that de- related to scientific research and related ap- fine the pattern of the attack traffic so they plications have presented significant chal- can be easily shared with and understood lenges at the terascale and petascale [Allcock by other sites without disclosing private in- et al. 2001]. This situation is expected to be formation. SitAware should also supplement even more unwieldy at the exascale. Data these reports with drill-down analysis to fa- management and movement tools and tech- cilitate countermeasure deployment and fo- niques must be developed for data centers rensic study. A critical feature of a SitAware and archives, portals, and intersite and intra- capability would be an overall view across site file transfers. multiple sites of a collaboration or virtual organization. What is needed is a distributed Cyberinfrastructure management. Exascale cooperative security monitoring framework systems are expected to be increasingly that can combine security-related data from complex, comprising thousands or even many sites. This will allow independent sites millions of components. The configuration, to extend their sense of the Internet’s activ- verification, troubleshooting, and management ity beyond their local viewpoints. Even more of such complex systems, as well as the important, advanced capabilities in this area development of the tools necessary to perform will facilitate collaborative threat response, these functions on the often one-of-a-kind including cross-site notification of and re- resources, will be a significant challenge. sponse to security events anywhere in the collaboration environment. Cyber security. IDSs, firewalls, vulnerability scanners, and other components that make up Since exascale compute resources are ex- the cyber security infrastructure of the expect- tremely valuable and a tempting target of ed state-of-the-art exascale resources will not attacks, we must develop advanced security have commercial products available that scale data analysis capabilities to facilitate collab- to line rates or capacities for months or even orative threat response. These capabilities are years after the exascale resources are deployed. essential for such crucial tasks as cross-site The challenge will be to provide adequate se- notification of security events and subsequent curity functionality at the exascale with open response to these events. Related research source or locally developed tools. Investment topics include audit frameworks, data-mining in cyber security tools that can be shared by algorithms for security logs, data visualization the laboratories seems a plausible option. techniques for exploration of log data, anomaly detection techniques to help identify suspi- Situational awareness, anomaly detection, cious activity, data anonymization techniques and response. A key challenge to cyber secu- to allow cross-site sharing of log data, inci- rity methodologies and tools of the future will dent profiling techniques, and ontology-based be the creation of a framework and semantics forensic analysis of security data and logs. for integrating information in the individual cyber security component systems for situ- “Sandbox” Technologies: Many DOE Office ational awareness, anomaly detection, and of Science HPC systems have changed from intrusion response. Current technologies are “low” security categorization to “moderate” segregated and unaware of the related infor- security categorization as the resource has be- mation available in their audit trails. Auto- come unique and desirable by industry. This mated, intelligent tools are needed to detect trend is expected to continue. The current anomalies in the behavior of users, jobs and DOE policy is to certify systems to the highest processes, applications, and services that level of risk for the system and implement the scale from system, department, and enterprise corresponding security control level, forcing to multiple sites. We envision a cybersecurity all who desire to use the resource to conform,

120 Modeling and Simulation at the Exascale for Energy and the Environment regardless of the risk level of their individual alysts to focus on the control channels where work. These types of tight controls present risks are greatest. Conversely, point-to-point real barriers to the science community inter- networking may allow bulk data transfers ested in participation in open science. The from trusted system to trusted system, regard- concept of a “sandbox” is introduced to pro- less of the size of the transaction. The ease of vide a potential solution. Sandboxes are de- information exchange is important to the open fined, bounded virtual environments in which science community. Investment in network virtual organizations (VOs) collaborate and monitoring for performance and anomaly de- share information. Each VO, regardless of tection also has the potential for significant re- the heterogeneity and physical locations of its ward with regard to identification of potential members, has its own sandbox and is protect- attacks, by facilitating cyber security analysis ed based on its unique information sensitivity in order to better characterize behaviors to de- and specific constraints (e.g., confidentiality, tect anomalies for the exascale environment. integrity, availability, risk). Application of a “sandbox” model allows independent certi- 3. State of the Art fication, such that a “moderate” system can provide both “moderate” and “low” sand- In this section, we review the state of the art boxes. Policy issues related to sandbox tech- in each of the five areas discussed above— nology must be included in any investigation workflow management, collaboration frame- to address the potential DOE policy change works and techniques, data management and from certification of systems to the sandbox movement, cyberinfrastructure management level. Within a “low” security sandbox, the tools, and cyber security — to provide an un- level of rigor of the controls would be relaxed derstanding of where the cyberinfrastructure as compared to higher-security sandboxes. is today and what is needed to meet the chal- Basically, the sandbox model moves the pe- lenges raised by exascale applications. rimeter from the outside (machine level) to the inside (sandbox level). The model makes Workflow management. Workflow systems the security problem scalable. today are evolving at a promising rate but show no signs yet of coalescing around a Dedicated network channels. In the exascale compact set of solutions. Progress is being environment, dataflow will grow significant- made in industry on workflow models and ly, and effective use and analysis of the cy- mechanisms for service-oriented architectures berinfrastructure are crucial. Cyber security (such as BPEL) and architectures for specific analysis requirements do not necessarily scale environments such as the Microsoft Windows up from the terascale environment to the ex- workflow framework. In the realm of cyber- ascale environment. Protection of interactive/ infrastructure, good progress is being made control sessions at exascale is similar to what in terascale distributed environments such as it was at terascale. Yet, the sheer increase in TeraGrid [Catlett et al. 2007] and Open Sci- the bulk data transfer for an exascale environ- ence Grid. Similar progress is evident in appli- ment presents significant challenges for cyber cation-domain (job-oriented) systems that are security analysis. Currently few protocols ex- resource cognizant and are starting to make ist for bulk data transfers. While some do use transparent to scientists the steps involved in specific ports, these are not widely deployed. transporting, cataloging, and replicating data Tools for data transfer must be developed that between stages of a workflow spread across use dedicated channels to separate data from distributed parallel computing sites. Prog- control communication and facilitate the ap- ress in these systems is also being made in plication of graded levels of control for exas- the integration of support for data provenance cale. Control sessions represent the primary tracking into the workflow system itself, so The “sandbox” model — in threat. Segregation of the data flow is one that scientists who use such systems to pro- which each virtual organization strategy to allow application of appropriate cess vast datasets can gain the added benefit is protected based on its controls commensurate to security risks. Sys- of transparently recording details of the data specific constraints — makes tem-to-system data channels may require less derivation or analysis process for later audit- the security manageable at the stringent security, allowing cyber security an- ing and discovery. exascale.

121 Section 9: Cyberinfrastructure

movement tools will become critical for data virtualization, management of properties and attributes independent of the underlying storage system, access control, provenance metadata, data integrity, and security.

Cyberinfrastructure management tools. Cur- rently, cyberinfrastructure management tools for the largest HPC systems are mostly collections of scripts and tools not well suited to managing these systems. Nagios, a commonly used monitoring tool, unfortunately does not scale well. Most sites that use Nagios end up running multiple instances of the tool, tenuously fitted together with local scripts to provide a monitoring system that provides Figure 9.2 The Access Grid enables collaborations between multiple groups of geo- minimal capability. Many additional scripts graphically distributed researchers. In addition to multiple audio and video streams and tools are needed to fill in the missing from each location, participants can share presentations, data, visualizations, and other applications through persistent virtual venues. elements, resulting in ad hoc local management environments. To some extent, many Collaboration frameworks and techniques. sites cover the basics and ignore the rest, with Collaboration is already a fundamental com- performance frequently being neglected. Yet ponent of science, with teams often spread much could be done to provide robust sys- across the country or extending around the tems functioning at peak performance—if the world. These collaborations require both tools were available. The IBM Blue Gene/L synchronous and asynchronous tools and RAS management system is currently one of communication. Asynchronous collaboration the best tools for gathering system and job infrastructure today includes the use of wikis, events. It has many shortcomings, some of blogs, and other emerging social network- which will be addressed with the next-gener- ing tools, whereas synchronous collaboration ation Blue Gene system. Unfortunately, some infrastructure includes context- and loca- of the largest issues—for example, the inabil- tion-aware, persistent visualization and col- ity to correlate events between the compute laboration environments (see, e.g., Stevens components (i.e., Blue Gene/L hardware) and 2003]. These environments can seamlessly no–compute components (i.e., external data display a multitude of information from both storage systems and networks), and mecha- local and remote sources, but often at the cost nisms for dealing with the large volumes of of lowering the experience to the lowest com- events—remain unsolved. Work is under way mon denominator. This means that the col- at the Center for the Improvement of Fault laboration capabilities of today do not create Tolerance in Systems (CIFTS) to build a fault- a strong sense of presence, with the remote tolerant backplane to alleviate the difficulties participants not feeling they are full partici- of fault prediction, notification, management, pants in the experience. In addition, tool de- and recovery. This project, funded by DOE, velopers have no straightforward method for has the potential to move the field closer to constructing tools and applications that can what is needed for petascale architectures, but support collaborative work. much will remain to be done to provide a production solution at the exascale. Neither the Data management and movement. As not- Blue Gene RAS system nor the CIFTS work ed earlier, many researchers who are creat- addresses performance tuning, reporting us- Environments that create a ing terascale and petascale datasets find that age, or configuration management. Tools strong sense of presence with they spend a significant portion of their time such as bcfg2 [Desai et al. 2003] or cfengine remote participants will be managing data rather than scientific investi- provide reasonable solutions for systems con- critical for effective collaboration gation. Exascale science will generate data at sisting of as many as 10,000 nodes, and these at the exascale. ever-increasing rates. Data management and tools will remain adequate for the configura-

122 Modeling and Simulation at the Exascale for Energy and the Environment tion management of support systems such as ture R&D if we are to meet the needs of ex- Current cyberinfrastructure file servers. They do not, however, address ascale science. tools can handle systems the configuration of machines with 100,000 of thousands of nodes, but nodes or other, nontraditional environments Workflow management. To maximize the new solutions are needed for machines with 100,000 nodes. such as virtual machines. The IBM Blue Gene productivity of scientists and extract the max- and other systems have, for the most part, imum amount of knowledge and discovery avoided the configuration management prob- from exascale science, a focused effort on lem by running a small, lightweight operating workflow management and the underlying system rather than a full system (such as Li- tools and libraries is required; such an effort nux), loading it over an out-of-band network should include the following: as needed. This situation has prevented many applications from utilizing these systems to • Development of transparent and highly the fullest capability. As we move to provid- optimized data transport between exas- ing a full operating system, the problem of cale workflow stages configuration management for the petascale and beyond will become a serious problem. • Automation of the complex policy-driven and congestion-sensitive scheduling de- Cyber security. Currently no single technol- cisions that need to be made to keep an ogy provides complete cyber security. Most exascale complex operating at an accept- organizations protect their information tech- able level of utilization nology resources through a defense-in-depth approach that covers network, host, and ap- • Extensions to workflow models that can plication security technologies; provides cyber take advantage of dynamic exploratory security awareness, skills development, and models enabled by vast computing re- training to staff and users; and details cyber sources (e.g., exploring parallel paths and security policy and standards. Tools employed branching a new, large-scale workflow by many organizations to implement a de- from each viable result) fense-in-depth approach include commercial firewall products and virtual private networks, • Development of common workflow lan- open source network intrusion detection tools guages and pluggable implementations such as Snort or Bro, syslog and tripwire that allow scientists to specify their pro- host intrusion detection, Nessus vulnerability cesses in a manner independent of the di- scanning, Websense web filtering, cfengine verse architectures that may evolve, even or SMS configuration management, and com- within a single exascale complex mercial anti-virus protection. Security tools often lag significantly in their availability on Collaboration frameworks and techniques. early high-performance network and com- Highly productive exascale science will de- putational systems. Sites generally have to mand seamless access to scientists, data, and use open source tools, such as Bro, to be able computational resources. The goal is to en- to perform security functions at top perfor- able scientists to ascertain at a glance the sta- mance speeds. The network and host moni- tus of the elements of collaborations that are toring mechanisms, such as IDSs, IPSs, and relevant to their work at any moment and to firewalls, do not adequately scale to support share their work products with their collabo- the dynamic and high-speed networking envi- rators synchronously and asynchronously. ronments that are delivered to early adopters Achieving this goal will involve the devel- of high-performance computing sites. opment of a library of collaborative software components, including the following: 4. Accelerating Development • Components to enable remote participants Few resources are currently directed to or to hear and see each other comfortably planned for cyberinfrastructure for exascale and interact naturally. These components computers. Clearly, we need to establish a should be as easy and reliable to use as plan designed to accelerate cyberinfrastruc- the telephone and, as much as possible,

123 Section 9: Cyberinfrastructure

re-create the sensation of being in the • Integration of exascale cyberinfrastruc- same room with the remote participants. ture developments with collaborative This will require advances in the human- environments. This will allow remote computer interfaces to prevent these in- collaborators to jointly access compute teractions from disrupting the real work states, from scientists iterating on sub- of the scientists. mission of science compute jobs to administrators assessing the health of the • Event distribution services that enable compute infrastructure. applications to share state interactively, so that all collaborators see the same • Integration of exascale data management representation of the application as one and movement tools with collaborative participant interacts with it. For example, environments. Such integration will allow in discussing and reviewing the DNA of remote collaborators to easily share, store, an organism, the participants’ view of the and access references to their datasets DNA on their screen would follow the independent of physical location. Support interactions of the lead speaker with the for the relevant data transfer clients will application. enable users to interact with their datasets direct from their desktops with single- • Development of network storage for ap- click and drag-and-drop access. plication states to support snapshots of collaborative applications in time, al- • Interfaces to the most prominent lowing collaborators to return to selected programming languages in use in exascale checkpoints in their interactions with ap- science applications. Such interfaces plications and data. will facilitate language adoption by collaborations. • Application sharing by leveraging native platform event models. This will allow The availability of open tools for use as un- scientists to utilize software specific to derlying infrastructure is key to the success their domain (e.g., molecule viewers, ma- of efforts to innovate at the application level, terials databases) without modification to allowing the infrastructure to be modified the software. by the domain scientists themselves accord- ing to their own needs and without regard for budgetary or legal constraints. Currently, the components described are largely not available as open source software. Cross-platform support is required to accommodate variation in the personal working environment of individual scientists.

New strategies are needed for cooperative work in the security-constrained exascale science network environment. Firewalls have long posed a difficult problem for science applications, and yet no suitable solution has been devised without significantly compromising security. Exascale science must employ strategies to securely enable application and data sharing and point-to- point interactions in constrained networks, Figure 9.3 The Compact Muon Solenoid (CMS) experiment is a particle physics while maintaining security standards com- detector being built on the Large Hadron Collider (LHC) at CERN in Switzerland. patible with the network policies of typical Significant cyberinfrastructure will be required to support the thousands of physicists from around the world who will be analyzing the massive amounts of data produced participating institutions. by the CMS.

124 Modeling and Simulation at the Exascale for Energy and the Environment

Data management and movement. Deploy- data (malicious or accidental) in data archival Exascale science applications ment of exascale applications needs to be systems, Data Grid management systems, and will need new strategies that accelerated by support for research into data file systems will be increasingly necessary as enable data sharing while management and movement requirements at exascale datasets start to stress component ensuring that individual the exascale and for development of the tools (network, CPU, storage) data error rates. Un- institutional security policies are or features that address these requirements, derlying all of the cyber security capabilities is maintained. both in leading Data Grid management sys- the need for more flexible, scalable, and veri- tems and as standalone tools. fiable authorization and authentication frameworks and tools. Cyberinfrastructure management tools. In order to ensure that computational resources 5. Expected Outcomes reach their full potential for exascale science, a substantial investment must be made in R&D Sufficient and timely investment to support on cyberinfrastructure management technol- multidisciplinary teams in a focused program ogy, including the following: tailored to encourage and support the development of cyberinfrastructure in the five areas • Tools and methods for scalable configu- outlined here will enable increased scientific ration management. Development of a output, productive collaborations between tiered configuration management strategy team members, simplified management of is essential. data, increased reliability of resources, and • Methodologies for monitoring tens of overall security of cyberinfrastructure for all thousands of nodes. Tools are needed to exascale science areas. effectively monitor tens of thousands of nodes to ensure machine health and to in- 6. Required Investment dicate when a large portion of the machine The effort will require multiple teams to ad- fails. Monitoring should include aspects vance the state of the art in each of the key of the machine that are of interest to us- areas identified here. Sizable investment also ers, such as node availability and historical will be needed for designing, developing, performance; aspects of interest to admin- testing, and deploying the cyberinfrastruc- istrators, such as node failures; and aspects ture. In addition, some effort will have to be of interest to cyber security analysts, such dedicated to maintaining the collaborations as possible machine misuse or attack. Ef- that will support crosscutting aspects of the fectively managing data from hundreds of development with the application areas and thousands of machine probes and terabyte- technology development teams. sized log files is critical.

• Investigation and determination of prop- 7. Major Risks er methods for dealing with machine and Several risks and potential downside conse- component failures. With tens of thou- quences are linked to exascale cyberinfra- sands of nodes, dealing with hardware structure R&D. failures will be a daily struggle. Strong relationships with hardware vendors and Workflow Management a clearly documented procedure will be • If we fail to invent and develop robust essential. user-friendly workflow management en- Cyber security. Continued support is essential vironments, exascale scientists will not be for the development of open source network able to process the vast amounts of data intrusion detection tools, such as Bro. Equally to get the most out their efforts and the important, the development of new approach- investment in these large-scale systems. es to the expected complex and distributed • If we fail to sufficiently automate the re- exascale cyberinfrastructure, could accelerate source management layers of workflow the capabilities to secure the exascale cyber- systems, vast hardware resources of exas- infrastructure soon after it becomes available. cale complexes will be underutilized and Tools to help detect unauthorized alteration of

125 Section 9: Cyberinfrastructure

result in disappointing levels of speedup tion of effort to build the underlying infra- and productivity. structure for each area.

Collaboration Frameworks and Techniques Cyber Security

• If we do not invest in R&D tools and in- • If mechanisms to validate the integrity of frastructure to support the remote opera- datasets and data collections are not de- tion of global instruments, we will lose veloped, the accuracy and integrity of the the active involvement of a large number research may be at risk. of our scientists in the science done on these resources. • If significant log analysis tools to detect and investigate anomalies are not avail- • If collaborative infrastructure does not sup- able, the security of the exascale-class port teams in a natural manner, produc- and support systems will be at risk. tivity of the community will suffer, and efforts will be fragmented and duplicative • If significant intrusion detection capabili- because of inability to share results and ties at line speed are not available, the se- collaborate effectively in large teams. curity of the exascale class and support systems will be at risk. Data Management and Movement • The current DOE policy is to certify sys- • If we fail to address issues related to both tems to the highest level of risk for the short- and long-term management of data, system and implement the corresponding we risk the loss of productivity of scien- security control level, forcing all who de- tists as they deal with management and sire to use the resource to conform regard- organization of data. Further, utilization less of the risk level of their individual of expensive resources will be reduced work. If the proposed sandbox technology because of data movement into and out is not investigated and developed, open of these machines without high-perfor- research may be at risk of moving to less mance capabilities and remote access to capable systems because of the perception storage devices. that security policies get in the way.

• If we do not develop strategies and infra- R&D in cyberinfrastructure is essential to structure to manage the vast amounts of the success of the exascale effort. It is a criti- data that exascale science will produce, cal component, providing the foundation on we risk losing intellectual property em- which tools and applications for doing sci- bedded in the data. ence will be built, the infrastructure by which scientists will collaborate, and the tools that Cyberinfrastructure Management Tools ensure applications have resources on which • If we do not invest in R&D on cyberin- to run, all in an environment that is secure. frastructure management tools and tech- References niques, we risk the investment in resources due to downtime and system errors as these Bill Allcock, Joe Bester, John Bresnahan, Ann L. complex resources are debugged. Chervenak, Carl Kesselman, Sam Meder, Veroni ka Nefedova, Darcy Quesnel, Steven Tuecke and • If we fail to address the usability of re- Ian Foster (2001), Secure, efficient data transport quired security mechanisms, users will and replica management for high-performance da continue to be hindered, at a cost of pro- ta-intensive computing, p. 13 in Proc. 18th IEEE ductivity or a reduction in security as us- Symposium on Mass Storage Systems. Without the automation of the ers find workarounds. resource management layers Charlie Catlett et al. (2007), TeraGrid: Analysis of workflow systems, exascale • If we inadequately connect the efforts in of organization, system architecture, and middle resources will be underused, cyberinfrastructure to development in ware enabling new types of applications, in High Performance Computing and Grids in Action, IOS and both performance and technology areas, we will see a duplica- productivity levels will be Press, Advances in Parallel Computing series, disappointing. Amsterdam.

126 Modeling and Simulation at the Exascale for Energy and the Environment Narayan Desai, Andrew Lusk, Rick Bradshaw and Rick Stevens, Michael E. Papka and Terry Disz Security policies on exascale Remy Evard (2003), BCFG: A configuration man- (2003), Prototyping the workspace of the future, systems must not get in the way agement tool for heterogeneous environments, in IEEE Internet Computing 7(4): 51–58. of science. Proc. IEEE International Conference on Cluster Computing, pp. 500, 2003. Gregor von Laszewski, Michael Hategan and Deepti Kodeboyina (2007), Java CoG Kit work Ian Foster, Jens Voeckler, Michael Wilde and Yong flow, pp. 340-356 in Workflows for Science, I. Tay Zhao (2003), The virtual Data Grid: A new model lor, E. Deelman, D. B. Gannon, and M. Shields, and architecture for data-intensive collaboration, eds., Springer. in Proc. Conference on Innovative Data Systems Research. Von Welch, Jim Barlow, James Basney, Doru Mar- cusiu and Nancy Wilkins-Diehr (2006), A AAAA Veronika Nefedova, Robert Jacob, Ian Foster, model to support science gateways with commu- Zhengyu Liu, Yun Liu, Ewa Deelman, Gaurang nity accounts, Concurrency and Computation: Mehta, Mei-Hui Su and Karan Vahi (2006), Au Practice and Experience 19(6):893-904. tomating climate science: Large ensemble simu lations on the TeraGrid with the GriPhyN virtual Yong Zhao, Michael Wilde and Ian Foster (2007), data system, p. 32 in Proc. 2nd IEEE International Virtual Data Language: A Typed Workflow Nota Conference on e-Science and Grid Computing. tion for Diversely Structured Scientific Data, pp. 258-278 in Workflows for eScience,Springer. Rick Stevens (2003), Access Grid: Enabling group-oriented collaboration on the Grid, in The Grid: Blueprint for a New Computing Infrastruc ture, Ian Foster and Carl Kesselman, eds., Morgan Kaufmann.

127 Section 9: Cyberinfrastructure Appendix A

Initiative Summary

129 130 Modeling and Simulation at the Exascale for Energy and the Environment U.S. Department of Energy Office of Science

Simulation and Modeling at the Exascale for Energy, Ecological Sustainability and Global Security An Initiative

The objective of this ten-year vision, which is in line with the Department of Energy’s Strategic Goals for Scientific Discovery and Innovation, is to focus the computational science experiences gained over the past ten years on the opportunities introduced with exascale computing to revolutionize our approaches to energy, environmental sustainability and security global challenges.

Executive Summary The past two decades of national investments in computer science and In order to realize this leadership the high-performance computing have DOE recognizes that the time-honored, placed the DOE at the forefront of many or subsystems, approach in which the areas of science and engineering. This forces and the physical environments of initiative capitalizes on the significant a phenomenon are analyzed, is gains in computational science and approaching a state of diminishing boldly positions the DOE to attack returns. The approach for the future global challenges through modeling and must be systems based and simulation simulation. The planned petascale programs are developed in the context of computer systems and the potential for encoding all known relevant physical exascale systems shortly provide an laws with engineering practices, unprecedented opportunity for science; production, utilization, distribution and one that will make it possible to use environmental factors. computation not only as an critical tool along with theory and experiment in This new approach will understanding the behavior of the • Integrate, not reduce. The full fundamental components of nature but suite of physical, chemical, also for fundamental discovery and biological, chemical and engineering exploration of the behavior of complex processes in the context of existing systems with billions of components infrastructures and human behavior including those involving humans. will be dynamically and realistically linked, rather than focusing on more Through modeling and simulation, the detailed understanding of smaller DOE is well-positioned to build on its and smaller components. demonstrated and widely-recognized leadership in understanding the • Leverage the interdisciplinary fundamental components of nature to be approach to computational a world-leader in understanding how to sciences. Current algorithms, assemble these components to address approaches and levels of the scientific, technical and societal understanding may not be adequate. issues associated with energy, ecology A key challenge in development of and security on a global scale. these models will be the creation of a

131 Appendix A: Initiative Summary

U.S. Department of Energy Office of Science framework and semantics for model be energy efficient, extremely interaction that allow the scalable, and tied to the needs of interconnection of discipline models scientific computing at all scales. with observational data. At the Correspondingly, recruit and develop outset, specialized scientific groups the next generation of computational will team with engineers, business and mathematical scientists. experts, ecologists and human behavior specialists comprehensive 2. Invest in pioneering large-scale models, that incorporate all known science, modeling and simulation phenomena and have the capability that contribute to advancing energy, to simulate systems characteristics ecology and global security. under the full range of uncertainties. 3. Develop scalable analysis • Capitalize on developments in data algorithms, data systems and storage management and validation of architectures needed to accelerate ultra-large datasets. It will develop discovery from large-scale new approaches to data management, experiments and enable verification visualization and analysis that can and validation of the results of the treat the scale and complexity of the pioneering applications. data and provide the insight needed Additionally, develop visualization for validation of the computations. and data management systems to manage the output of large-scale This new approach will enable DOE to computational science runs and in exploit recent developments in new ways to integrate data analysis commercially available computer with modeling and simulation. architectures, driven by the implementation of first generation multi- 4. Accelerate the build-out and future core processors and the introduction of development of the DOE open petascale computers within 18 months, computing facilities to realize the and prepare it to take advantage of large-scale systems-level science exascale computers in the next decade. required to advance the energy, This approach will also guarantee ecology and global security program. DOE’s leadership in applying these Develop an integrated network computers to critical problems computing environment that couples confronting the nation. these facilities to each other, to other large-scale national user facilities The initiative has four programmatic and to the emerging international themes: network of high-performance computing systems and facilities. 1. Engage top scientists and engineers, computer scientists and applied The success of this fourth effort is built mathematicians in the country to on the first three themes because develop the science of complexity as exascale systems are, by themselves, well as new science driven computer among the most complex systems ever architectures and algorithms that will engineered.

132 Modeling and Simulation at the Exascale for Energy and the Environment U.S. Department of Energy Office of Science This initiative will enable DOE to For further information on this address critical challenges in areas subject contact: such as: Dr. Michael Strayer, Associate Director Energy- Ensuring global sustainability Office of Advance Scientific Computing requires reliable and affordable Research pathways to low-carbon energy [email protected] production, e.g. bio-fuels, fusion and fission, and distribution on a massive scale. The existing mix of energy supplies places global security at great risk. Acceptable solutions require rapid and unprecedented scientific and technologic advances. Unfortunately, existing analytical, predictive, control, and design capabilities will not scale. An objective of this initiative is to provide new models and computational tools with the functionality needed to discover and develop complex processes inherent in a new energy economy.

Ecological Sustainability- The effort toward sustainability involves characterizing the conditions for balance in the climate system. In particular, sustainable futures involve understanding and managing the balance of chemicals in the atmosphere and ocean. The ability to fit energy production and industrial emissions within balanced global climate and chemical cycles is the major scientific and technical challenge for this century.

Security- The internet, as well as the instrumentation and control systems for the energy infrastructure, is central to the well-being of our society. There are several potential opportunities relating to accurately modeling these complex systems: understand operational data, identify anomalous behavior to isolate the disturbance and automatically repair any damage.

133 134 Appendix B

Agendas

135 Appendix B: Agendas

Simulation and Modeling at the Exascale for Energy, Ecological Sustainability and Global Security (E3SGS) Lawrence Berkeley National Laboratory Town Hall Meeting

April 17 – 18, 2007

Tuesday, April 17, 2007 8:00 – 8:15 am Welcome and Introduction...... Horst Simon, Associate Laboratory Director, Computing Sciences, LBNL

8:15 – 8:45 am Opening Remarks – Kick-off announcement on town hall meeting series.....Michael Strayer, Associate Director, ASCR 8:45 – 9:45 am “The Computational Frontiers of Earth System Modeling”...... Bill Collins, LBNL and NCAR

9:45 – 10:15 am Morning Break 10:15 – 10:45 am Panel Session...... Panelists: Horst Simon, Rick Stevens, Thomas Zacharia 11:00 – 12 noon Start breakout group discussions B1 in 54-130 (Perseverance Hall) B2 in 66-Auditorium B3 in 66-316 B9 in 62-203 12:00 – 1:00 pm Working lunch – pick up lunch and continue discussion 1:00 – 3:30 pm Continue breakout group discussions 3:30 – 5:30 pm Report back from breakout groups 5:30 pm Adjourn

Wednesday, April 18, 2007 8:00 – 8:15 am Welcome Back and Agenda for Day 2...... Horst Simon

8:15 – 12:00 pm Breakout group discussions B4 in 62-255 B5 in 66-Auditorium B7 in 62-203 B8 in 66-316 12:00 – 1:00 pm “Solutions to the Energy Crisis” (working lunch scheduled during this time)...... Steve Chu, Laboratory Director, LBNL

1:00 – 2:30 pm Report back from breakout groups 2:30 pm Adjourn

136 Modeling and Simulation at the Exascale for Energy and the Environment Simulation and Modeling at the Exascale for Energy, Ecological Sustainability and Global Security (E3SGS) Oak Ridge National Laboratory Town Hall Meeting May 17-18, 2007 Wednesday, May 16, 2007 Arrival and dinner on your own Thursday, May 17, 2007 Town Hall parking is in top two tiers of lot across Bethel Valley Road from ORNL Visitor Center

6:45 a.m. Buses pick up at hotels (Homewood and Springhill Suites in Turkey Creek area; Comfort Suites at Campbell Station Road; Comfort Inn, Double Tree, and Jameson Inn in Oak Ridge) One stop at each hotel 7:00- 7:15 a.m. Arrive Bldg. 8600 ...... Check in/visitor badging 7:25 a.m. Local participants to meet buses at ORNL Visitor Center for transportation to Bldg. 8600 7:30 a.m. Morning networking session and refreshments ...... Bldg. 8600, 1st floor atrium

8:00 a.m. Welcome and Introductions (Iran Thomas Auditorium) ...... Thomas Zacharia 8:15 a.m. Opening remarks ...... Michael Strayer, Assoc. Dir. ASCR

Climate Change Session...... David Erickson, Session Chair 8:45 a.m. Climate change: policy perspectives ...... Lincoln Pratson, Duke 9:30 a.m. Climate change: computational science perspectives ...... David C. Bader, LLNL 10:15 a.m. Break

10:30 a.m. Laboratory Director’s Welcome ...... Jeff Wadsworth

Energy Session ...... Doug Kothe, Session Chair 10:40 a.m. Energy: industry perspectives ...... Jack Bailey, TVA 11:10 a.m. Energy: computational science perspectives ...... Tom Downar, UC-Berkeley

12:00 p.m. Working lunch ...... 2nd Floor Lobby, CNMS Lunch will be picked up in the 2nd floor lobby of the CNMS –Center for Nanophase Materials Sciences-- just around the corner and a short walk away from the Iran Thomas Auditorium. Sitting areas will be available on 1st, 2nd, 3rd floor lobbies, in adjacent conference rooms, on the patio, in room 156, the CNMS executive conference room, in small common areas, and in the SNS atrium – however, no food and beverages are allowed in the auditorium.

1:00 p.m. Applications breakouts ...... Board buses at CNMS B1. Climate ...... SNS Room C156 (8600) B6. Numerical Climate-Economic-Energy...... Research Office Building L204 (5700) B3. Biology ...... SNS 354 (8600) B2. Energy ...... Iran Thomas Auditorium (8600) B9. Astrophysics ...... CNMS L183 (8610) B10. Industrial Processes and Manufacturing ...... CNMS L382 (8610)

3:00 p.m. Break 3:30 p.m. Continue applications breakout sessions

5:30 p.m. Adjourn

137 Appendix B: Agendas

Board buses to return to main campus ...... Flagpole lobby entrance

6:00 p.m. Dinner (Bldg. 5200, 2nd floor) ...... Session Summaries

9:00 p.m. Board buses for return to hotel ...... Visitor Center

* * * * * * * * * * * * * *

Friday, May 18, 2007, sessions will be held on main campus in Bldg. 5200 (Conference Center)

7:30 a.m. Networking session and refreshments ...... Bldg. 5200, 2nd floor lobby

8:00 a.m. Welcome back and agenda for day 2 (Rooms: Tennessee A, B, C) ...... Thomas Zacharia 8:15 a.m. Begin infrastructure breakouts B5. Hardware ...... Room tbd B7. Math ...... Room tbd B8. Software ...... Room tbd B4. Cyber security ...... Room tbd

9:45 a.m. Break

10:15 a.m. Continue breakout sessions......

12:15p.m. Working lunch (pick up lunch and continue breakouts) ...... 2nd floor lobby

1:45 p.m. Closing remarks

2:00 p.m. Adjourn

Board buses for return to hotels...... Visitor Center

138 Modeling and Simulation at the Exascale for Energy and the Environment Simulation and Modeling at the Exascale for Energy, Ecological Sustainability and Global Security (E3SGS) Argonne National Laboratory Town Hall Meeting

May 31 – June 1, 2007

Thursday, May 31, 2007

8:00 Welcome and Introduction...... Rick Stevens, Associate Laboratory Director, Computing and Life Sciences, ANL

8:15 Opening Remarks...... Michael Strayer, Associate Director, ASCR

8:45 “Energy, Environmental Sustainability and the Role of High-End Computing”...... Robert Rosner, Laboratory Director, ANL

10:00 Morning Break – Auditorium/Rotunda

10:30 “Potential Applications of Exascale Computing in Economics”...... Kenneth L. Judd, Paul H. Bauer Senior Fellow, Hoover Institution 11:15 Agenda and Instructions for Town Hall ...... Rick Stevens

11:30 – 3:30 Breakout group discussions

Noon – 1p.m. Working lunch scheduled noon-1p.m. – pick up lunches and continue discussions) B1 Climate – Hosts: Kemner/Kotamarthi B2 Energy – Hosts: Frank/Nowak B3 Biology – Hosts: Edwards/Meyer B6 Socioeconomic Modeling – Hosts: Foster/Jacob B9 Astrophysics – Hosts: Fischer/Lamb B10 Industrial Processes and Manufacturing – Hosts: Hassanein/Moré

2:00 - 2:30 Afternoon Break – Auditorium/Rotunda

3:30 Report back from breakout groups (each group 20min)

5:30 Adjourn

Friday, June 1, 2007

8:00 - 8:15 Welcome Back and Agenda for Day 2...... Rick Stevens

8:15 – 12:00 Breakout group discussions (working lunch scheduled noon-1p.m. – pick up lunches at Auditorium/Rotunda and continue discussions/report writing) B4 Cyberinfrastructure – Hosts: Catlett/Papka B5 Hardware – Hosts: Beckman/Gropp B7 Math – Hosts: Hereld/Norris B8 Software – Hosts: Lusk/Ross

139 Appendix B: Agendas

10:00 - 10:30 Morning Break – Auditorium/Rotunda

12:00 Report back from breakout groups (each group 15min)

1:00 Next steps/Comments...... Rick Stevens, Bill Kramer, Jeff Nichols

2:00 Adjourn

140 Appendix C

Attendees

141 142 Modeling and Simulation at the Exascale for Energy and the Environment

Combined Attendees List Abel, Tom �� Kavli Institute for Particle Astrophysics and Cosmology Agarwal, Deb �� Lawrence Berkeley National Laboratory Agarwal, Pratul K. �� Oak Ridge National Laboratory Ahern, Sean �� UT-Battelle / Oak Ridge National Laboratory Ahrens, James �� Los Alamos National Laboratory Alam, Sadaf R. �� Oak Ridge National Laboratory/UT-Battelle Alexiades, Vasilios �� University of Tennessee Knoxville Allan, Benjamin ��Sandia National Laboratories Altaweel, Mark �� Argonne National Laboratory Andrade, Jose �� Northwestern University Anitescu, Mihai �� Argonne National Laboratory Antypas, Katerina �� Lawrence Berkeley National Laboratory / National Energy Research Scientific Computing Center Apra, Edoardo �� UT-Battelle Aragon, Cecilia �� Lawrence Berkeley National Laboratory Archibald, Rick �� Oak Ridge National Laboratory Asanovic, Krste ��Massachusetts Institute of Technology / University of California Berkeley Ashby, Steven F. ��Lawrence Livermore National Laboratory Babnigg, Gyorgy �� Argonne National Laboratory Bader, David A. ��Georgia Institute of Technology Bader, David C. ��Lawrence Livermore National Laboratory Baertsch, Robert ��University of California Santa Cruz Bai, Zhaojun �� University of California Davis Bailey, David H. �� Lawrence Berkeley National Laboratory Bailey, Jack A. ��Tennessee Valley Authority Bair, Raymond A. �� Argonne National Laboratory Baker, Allen �� University of Tennessee Baldocchi, Dennis ��University of California Berkeley Banda, Michael �� Lawrence Berkeley National Laboratory Banks, David C. �� University of Tennessee Barker, Kevin �� Los Alamos National Laboratory Barrett, Richard �� Oak Ridge National Laboratory Barron, Tom O. �� Oak Ridge National Laboratory Bartels, Daniela �� University of Chicago Bashor, Jon �� Lawrence Berkeley National Laboratory Beavers, James �� University of Tennessee Beck, Micah �� University of Tennessee Beckman, Pete H. �� Argonne National Laboratory Bell, John �� Lawrence Berkeley National Laboratory Berger-Wolf, Tanya ��University of Illinois-Chicago Bernholc, Jerry ��North Carolina State University Bernholdt, David E. �� Oak Ridge National Laboratory Berry, Michael W. �� University of Tennessee Bethel, E. Wes �� Lawrence Berkeley National Laboratory Bland, Arthur S. �� Oak Ridge National Laboratory Blondin, John M. ��North Carolina State University Bode, Brett ��Ames Laboratory Bohn, Robert �� NCO/NITRD Borrill, Julian �� Lawrence Berkeley National Laboratory / University of California Berkeley Boudwin, Kathlyn �� UT-Battelle Bownas, Jennifer �� Oak Ridge National Laboratory Braam, Peter ��Cluster File Systems, Inc.

143 Appendix C: Attendees

Bramley, Randall ��Indiana University Branstetter, Marcia �� UT-Battelle Britt, Phillip F. �� Oak Ridge National Laboratory Bronevetsky, Greg ��Lawrence Livermore National Laboratory Brown, Nancy �� Lawrence Berkeley National Laboratory Brown , David L. ��Lawrence Livermore National Laboratory Brown , Maxine ��University of Illinois-Chicago Budiardja, Reuben �� University of Tennessee Buja, Lawrence E. ��National Center for Atmospheric Research Calafiura, Paolo �� Lawrence Berkeley National Laboratory Calder, Alan �� State University of New York at Stony Brook Canning, Andrew �� Lawrence Berkeley National Laboratory Cannon, William �� Battelle Memorial Foundation Canon, Richard �� Oak Ridge National Laboratory Cardall, Christian Y. �� Oak Ridge National Laboratory Carter, Jonathan �� Lawrence Berkeley National Laboratory Catlett, Charles E. ��Argonne National Laboratory/University of Chicago Chavarria, Daniel ��Pacific Northwest National Laboratory Chen, Jacqueline ��Sandia National Laboratories Cheng, Robert �� Lawrence Berkeley National Laboratory Chiang, John ��University of California Berkeley Childs, Henry ��Lawrence Livermore National Laboratory Chiu, George L. �� IBM Choudhary, Alok �� Northwestern University Christiansen, John H. �� Argonne National Laboratory Chu, Steve �� Lawrence Berkeley National Laboratory Cirillo, Richard �� Argonne National Laboratory Clarke, Leon E. ��Pacific Northwest National Laboratory Clarno, Kevin T. �� Oak Ridge National Laboratory Cobb, John W. �� Oak Ridge National Laboratory Coghlan, Susan �� Argonne National Laboratory Cohoon, Matthew �� The University of Chicago Colella, Phil �� Lawrence Berkeley National Laboratory Coleman-Smith, Christopher �� Lawrence Berkeley National Laboratory Collins, William �� Lawrence Berkeley National Laboratory / University of California Berkeley Collis, S. Scott ��Sandia National Laboratories Conway, Claudine ��Dell Cortis, Andrea �� Lawrence Berkeley National Laboratory Counce, Deborah �� Oak Ridge National Laboratory Crawford, Matt ��Fermi National Laboratory Culhane, Candace �� Oak Ridge National Laboratory Curfman McInnes, Lois �� Argonne National Laboratory Dale, Virginia H. �� Oak Ridge National Laboratory Davis, Kei �� Los Alamos National Laboratory Davis, Wayne �� University of Tennessee Davis, William �� Lawrence Berkeley National Laboratory D’Azevedo, Eduardo �� Oak Ridge National Laboratory de Almeida, Valmor �� UT-Battelle DeBenedictis, Erik ��Sandia National Laboratories Deiterding, Ralf �� Oak Ridge National Laboratory del-Castillo-Negrete, Diego �� Oak Ridge National Laboratory DePaoli, David �� Oak Ridge National Laboratory D’haeseleer, Patrik ��Lawrence Livermore National Laboratory

144 Modeling and Simulation at the Exascale for Energy and the Environment

Diachin, Lori A. ��Lawrence Livermore National Laboratory DiBennardi, Kathy �� Argonne National Laboratory Disz, Terrence �� Argonne National Laboratory Dobson, Patrick �� U.S. Department of Energy Office of Basic Energy Sciences Doctor, Richard �� Argonne National Laboratory Dongarra, Jack J. ��University of Tennessee / Oak Ridge National Laboratory Dosanjh, Sudip ��Sandia National Laboratories Downar, Thomas ��Purdue University Drake, John B. �� Oak Ridge National Laboratory Drugan, Cheryl �� Argonne National Laboratory Dubchak, Inna �� Lawrence Berkeley National Laboratory Duffy, Philip ��Lawrence Livermore National Laboratory Dupont, Todd �� University of Chicago Edwards, Robert �� Argonne National Laboratory Eigenmann, Rudolph ��Purdue University Ellis, David ��LSI Corporation Elnozahy, Elmootazbellah �� IBM Elwasif, Wael �� Oak Ridge National Laboratory Elwood, Jerry W. �� U.S. Department of Energy Endeve, Eirik ��University of Tennessee / Oak Ridge National Laboratory Erickson, David �� Oak Ridge National Laboratory Evans, Thomas �� Oak Ridge National Laboratory Fahey, Mark �� Oak Ridge National Laboratory Fann, George �� Oak Ridge National Laboratory Feng, Wu-chun �� Virginia Tech Feng, Zhili �� Oak Ridge National Laboratory Fernandez, Steven �� Oak Ridge National Laboratory Finsterle, Stefan �� Lawrence Berkeley National Laboratory Fischer, Paul �� Argonne National Laboratory Foley, Samantha �� Oak Ridge National Laboratory Folta, Peggy ��Lawrence Livermore National Laboratory Fossum, Barbara ��Purdue University Foster, Ian T. ��Argonne National Laboratory / University of Chicago Fowler, Robert �� RENCI / University of North Carolina Frank, Ed �� Argonne National Laboratory Fraser, Dan �� Argonne National Laboratory Friedman, Alex ��Lawrence Livermore National Laboratory / Lawrence Berkeley National Laboratory Fu, Joshua S. �� University of Tennessee Fu, Joshua Sing-Yih �� University of Tennessee Fujimoto, Richard M. �� Georgia Tech Fung, Inez �� Lawrence Berkeley National Laboratory / University of California Berkeley Galli, Giulia �� University of California Davis Ganguly, Auroop R. �� Oak Ridge National Laboratory Geffen, Charlette A. ��Pacific Northwest National Laboratory Gehin, Jess �� UT-Battelle / Oak Ridge National Laboratory Geist, George A. �� Oak Ridge National Laboratory Gerber, Richard �� Lawrence Berkeley National Laboratory Germain, Robert �� IBM Ghan, Steven ��Pacific Northwest National Laboratory Glass, Elizabeth �� Argonne National Laboratory Glendenin, Chad �� University of Chicago Goldberg, Stephen �� Argonne National Laboratory Golliver, Roger ��Intel

145 Appendix C: Attendees

Gorin, Audrey A. �� Oak Ridge National Laboratory Gracio, Debbie ��Pacific Northwest National Laboratory Graf, Peter �� National Renewable Energy Laboratory Graham, Richard �� Oak Ridge National Laboratory Graham, Robin �� UT-Battelle Gray, William �� UT-Battelle Grcar, Joseph �� Lawrence Berkeley National Laboratory Greene, Sherrell R. �� Oak Ridge National Laboratory Gropp, William �� Argonne National Laboratory Grossan, Bruce ��Eureka Scientific / INPA Grossman, Robert ��University of Illinois-Chicago Guidry, Michael ��University of Tennessee / Oak Ridge National Laboratory Gustafson, William ��Pacific Northwest National Laboratory Hanson, Donald �� Argonne National Laboratory Hanson, Paul J. �� Oak Ridge National Laboratory Hartman-Baker, Rebecca �� Oak Ridge National Laboratory Hauser, Loren �� UT-Battelle Hazlewood, Victor G. �� UT-Battelle / Oak Ridge National Laboratory Head-Gordon, Martin �� Lawrence Berkeley National Laboratory Head-Gordon, Teresa �� Lawrence Berkeley National Laboratory Helland, Barbara �� U.S. Department of Energy Office of Science Henry, Christopher �� Northwestern University Hereld, Mark �� Argonne National Laboratory Heroux, Michael ��Sandia National Laboratories Hetrick, David M. �� Oak Ridge National Laboratory / UT-Battelle Hix, William �� Oak Ridge National Laboratory Hoffman, Forrest M. �� Oak Ridge National Laboratory Hoisie, Adolfy �� Los Alamos National Laboratory Homen-de-Mello, Tito �� Northwestern University Hovland, Paul �� Argonne National Laboratory Howell, Louis ��Lawrence Livermore National Laboratory Huang, Jian �� University of Tennessee Iancu, Costin �� Lawrence Berkeley National Laboratory Iordache, Maria �� IBM Jackson, Keith �� Lawrence Berkeley National Laboratory Jacob, Robert �� Argonne National Laboratory Jacobs, Gary �� Oak Ridge National Laboratory Jain, Prashant ��University of Illinois Urbana-Champaign Janssen, Curtis ��Sandia National Laboratories Jemian, Pete �� Argonne National Laboratory Johnson, David �� UT-Battelle Johnson, Fred �� U.S. Department of Energy Office of Science Jones, Phillip �� Los Alamos National Laboratory Jones, Wesley �� National Renewable Energy Laboratory Joo, Kyungseon �� University of Connecticut Joseph, Anthony ��University of California Berkeley Jouline, Igor B. �� Oak Ridge National Laboratory Joy, Ken �� University of California Davis Jubin, Robert �� Oak Ridge National Laboratory Judd, Kenneth �� Hoover Institution Kalyanaraman, Anantharaman ��Washington State University Kamath, Chandrika ��Lawrence Livermore National Laboratory Karonis, Nick �� Northern Illinois University

146 Modeling and Simulation at the Exascale for Energy and the Environment

Karpov, Eduard �� University of Tennessee Kasdorf, James �� Pittsburgh Supercomputing Center Kaushik, Dinesh �� Argonne National Laboratory Keffer, David �� University of Tennessee Kendall, Ricky �� Oak Ridge National Laboratory Kerstein, Alan ��Sandia National Laboratories Kettimuthu, Raj �� Argonne National Laboratory Khaleel, Moe A. ��Pacific Northwest National Laboratory Khamayseh, Ahmed K. �� Oak Ridge National Laboratory Khokhlov, Alexei M. �� University of Chicago Khomami, Bamin �� University of Tennessee Kibblewhite, Ed �� University of Chicago Kiefer, David �� Cray, Inc. Klasky, Scott A. �� Oak Ridge National Laboratory Knotek, Michael L. �� Knotek Scientific Consulting Koester, David P. �� MITRE Corporation Kohl, James �� Oak Ridge National Laboratory Konerding, David �� Lawrence Berkeley National Laboratory Kotamarthi, Veerabhadra �� Argonne National Laboratory Kothe, Douglas B. �� Oak Ridge National Laboratory Kramer, William �� Lawrence Berkeley National Laboratory / National Energy Research Scientific Computing Center Krueger, Paul �� Cray, Inc. Kuehn, Jeffrey �� Oak Ridge National Laboratory Kumaran, Kalyan �� Argonne National Laboratory Kumfert, Gary ��Lawrence Livermore National Laboratory Kyrpides, Nikos �� Lawrence Berkeley National Laboratory Laguna, Pablo ��Penn State University Lamb, Donald �� University of Chicago Land, Miriam �� Oak Ridge National Laboratory Langston, Michael A. �� University of Tennessee Larzelere, Alexander �� Los Alamos National Laboratory Lattimer, James �� State University of New York at Stony Brook Lazaro, Michael �� Argonne National Laboratory Lee, Jason �� Lawrence Berkeley National Laboratory Lee, Lie-Quan ��Stanford Linear Accelerator Center Leininger, Matt ��Lawrence Livermore National Laboratory Leszczynski, Jerzy R. �� Jackson State University Levesque, John �� Cray, Inc. Levine, Michael ��Carnegie Mellon University Li, Fangxing �� University of Tennessee Li, Sherry �� Lawrence Berkeley National Laboratory Lightstone, Felice ��Lawrence Livermore National Laboratory Lindsay, Robert �� U.S. Department of Energy Office of Science Liu, Fang ��Indiana University at Bloomington Liu, Xiaohong ��Pacific Northwest National Laboratory Livny, Miron ��University of Wisconsin-Madison Locascio, Phil �� Oak Ridge National Laboratory Loebl, Andrew �� UT-Battelle Loft, Rich ��National Center for Atmospheric Research Lu, Guoping �� Lawrence Berkeley National Laboratory Lucas, Robert ��University of Southern California Ludaescher, Bertram �� University of California Davis Lusk, Ewing L. �� Argonne National Laboratory

147 Appendix C: Attendees

Lynch, Vicki E. �� Oak Ridge National Laboratory Ma, Xiaosong ��North Carolina State University / Oak Ridge National Laboratory Macal, Charles �� Argonne National Laboratory Mansfield, Betty K. �� Oak Ridge National Laboratory Marchesini, Stefano �� Lawrence Berkeley National Laboratory Markidis, Stefano ��University of Illinois Urbana-Champaign Markowitz, Victor �� Lawrence Berkeley National Laboratory Marquez, Andres ��Pacific Northwest National Laboratory Marronetti, Pedro ��Florida Atlantic University Martin, Dan �� Lawrence Berkeley National Laboratory Martinez, Manuel �� University of Louisville Matarazzo, Celeste ��Lawrence Livermore National Laboratory May, Elebeoba ��Sandia National Laboratories McCoy, Debbie D. �� Oak Ridge National Laboratory McFarlane, Joanna �� UT-Battelle McHardy, Alice ��IBM Research McMahon, James �� Lawrence Berkeley National Laboratory McParland, Charles �� Lawrence Berkeley National Laboratory Menon, Surabi �� Lawrence Berkeley National Laboratory Messer, Bronson �� Oak Ridge National Laboratory Meyer, Folker �� Argonne National Laboratory Meza, Juan �� Lawrence Berkeley National Laboratory Mezzacappa, Anthony �� Oak Ridge National Laboratory Michaels, George ��Pacific Northwest National Laboratory Mickelson, Sheri �� University of Chicago / Computation Institute Middleton, Don E. . ��National Center for Atmospheric Research Miller, Norman �� Lawrence Berkeley National Laboratory Mills, Richard �� Oak Ridge National Laboratory Monroe, Laura �� Los Alamos National Laboratory Moore, Shirley �� University of Tennessee Moore, Terry �� University of Tennessee Moré, Jorge J. �� Argonne National Laboratory Mundy, Chris �� Pacific Northwest National Laboratory / Battelle Munson, Todd �� Argonne National Laboratory Murphy, Richard C. �� Sandia National Laboratory Myra, Eric �� State University of New York at Stony Brook Myrick, Virginia L. �� Oak Ridge National Laboratory Najm, Habib ��Sandia National Laboratories Neaton, Jeffrey �� Lawrence Berkeley National Laboratory Neitzel, Bryon ��Cluster File Systems Ng, Cho ��Stanford Linear Accelerator Center Ng, Esmond �� Lawrence Berkeley National Laboratory Nichols, Jeffrey A. �� Oak Ridge National Laboratory Nikravesh, Masoud �� University of California Berkeley / Lawrence Berkeley National Laboratory Norman, Michael ��University of California San Diego Norris, Boyana �� Argonne National Laboratory North, Michael �� Argonne National Laboratory Nowak, David �� Argonne National Laboratory Nugent, Peter �� Lawrence Berkeley National Laboratory Nukala, Phani K. �� Oak Ridge National Laboratory Nutaro, James �� UT-Battelle Oefelein, Joseph ��Sandia National Laboratories Oehmen, Christopher ��Pacific Northwest National Laboratory

148 Modeling and Simulation at the Exascale for Energy and the Environment

Okojie, Felix A. �� Jackson State University Oliker, Leonid �� Lawrence Berkeley National Laboratory Olson, Douglas �� Lawrence Berkeley National Laboratory Olson, Robert �� Argonne National Laboratory O’Maonaigh, Heather C. ��Oak Ridge Associated Universities Ostrouchov, George �� Oak Ridge National Laboratory Otoo, Ekow �� Lawrence Berkeley National Laboratory Overbeek, Ross �� Fellowship for Interpretation of Genomes Pagerit, Sylvain �� Argonne National Laboratory Pan, Jerry �� Oak Ridge National Laboratory Pannala, Sreekanth �� UT-Battelle / Oak Ridge National Laboratory Papka, Michael E. ��Argonne National Laboratory / University of Chicago Pascucci, Valerio ��Lawrence Livermore National Laboratory Peery, James ��Sandia National Laboratories Pennington, Rob �� National Center for Supercomputing Applications / University of Illinois Penumadu, Dayakar �� University of Tennessee Perumalla, Kalyan �� Oak Ridge National Laboratory Peters, Mark �� Argonne National Laboratory Peterson, Cynthia B. �� University of Tennessee Peterson, Gregory D. �� University of Tennessee Petravick, Don ��Fermi National Laboratory Pieper, Gail �� Argonne National Laboratory Pinar, Ali �� Lawrence Berkeley National Laboratory Pindzola, Michael ��Auburn University Platt, Darren M. �� Joint Genome Institute Plechac, Petr �� University of Tennessee Polys, Nicholas �� Virginia Tech Poole, Stephen W. �� Oak Ridge National Laboratory Pordes, Ruth ��Fermi National Laboratory Post, Wilfred �� Oak Ridge National Laboratory Potok, Thomas �� Oak Ridge National Laboratory Pratson, Lincoln F. �� Duke University Price, Nathan ��Institute for Systems Biology Pugmire, David �� Los Alamos National Laboratory Quinn, Thomas ��University of Washington Racunas, Stephen �� Stanford University Radhakrishnan, Bala �� UT-Battelle Raghavan, Padmasani ��Pennsylvania State University Ray, Jaideep �� Sandia National Laboratory Ricker, Paul ��University of Illinois at Urbana-Champaign Rintoul, Mark ��Sandia National Laboratories Robert, Sahoff �� Oak Ridge National Laboratory Robinson, Sharon M. �� Oak Ridge National Laboratory Roche, Kenneth �� Oak Ridge National Laboratory Rogers, Jim �� Oak Ridge National Laboratory Rosen, Benjamin ��Dell, Inc. Roskies, Ralph Z. �� Pittsburgh Supercomputing Center Rosner, Robert �� Argonne National Laboratory Ross, Robert B. �� Argonne National Laboratory Rotem, Doron �� Lawrence Berkeley National Laboratory Runolfsson, Thordur ��University of Oklahoma Rushton, James E. �� Oak Ridge National Laboratory Ryne, Robert �� Lawrence Berkeley National Laboratory

149 Appendix C: Attendees

Sabau, Adrian �� Oak Ridge National Laboratory Saied, Faisal ��Purdue University Sale, Michael �� Oak Ridge National Laboratory Salve, Rohit �� Lawrence Berkeley National Laboratory Samatova, Nagiza �� Oak Ridge National Laboratory Sankaran, Ramanan �� Oak Ridge National Laboratory Sarma, Gorti �� Oak Ridge National Laboratory Scheraga, Harold ��Cornell University Schissel , David ��General Atomics Schopf, Jennifer �� Argonne National Laboratory Schryver, Jack �� Oak Ridge National Laboratory Schulthess, Thomas �� Oak Ridge National Laboratory Schulz, Karl W. ��Texas Advanced Computing Center Schwartz, Peter �� Lawrence Berkeley National Laboratory Scott, Stephen �� Oak Ridge National Laboratory Scott, Steve �� Cray Sears, Mark �� U.S. Department of Energy Sekine, Yukiko �� U.S. Department of Energy Office of Science Sethian, James �� Lawrence Berkeley National Laboratory Shalf, John �� Lawrence Berkeley National Laboratory Shaw, Henry ��Lawrence Livermore National Laboratory Sheetz, R. Michael �� University of Kentucky Sheldon, Frederick �� UT-Battelle Shelton, Robert B. �� University of Tennessee Shelton, William A. �� Oak Ridge National Laboratory Shen, Han-Wei ��Ohio State University Shoshani, Arie �� Lawrence Berkeley National Laboratory Siegel, Andrew �� Argonne National Laboratory Simon, Horst �� Lawrence Berkeley National Laboratory Simunovic, Srdjan �� Oak Ridge National Laboratory Sjolander, Kimmen ��University of California Berkeley Skinner, David �� Lawrence Berkeley National Laboratory Skow , Dane �� Argonne National Laboratory Smith, Barry �� Argonne National Laboratory Smith, Jeff W. �� UT-Battelle / Oak Ridge National Laboratory Smith, Jeremy �� University of Tennessee Smith, Melissa C. ��Clemson University Snavely, Allan ��San Diego Supercomputing Center Snyder, Seth �� Argonne National Laboratory Spada, Mary E. �� Argonne National Laboratory Spentzouris, Panagiotis ��Fermi National Laboratory Sterling, Thomas �� Louisiana State University Stevens, Rick ��Argonne National Laboratory / University of Chicago Storaasli, Olaf �� Oak Ridge National Laboratory Straatsma, Tjerk P. ��Pacific Northwest National Laboratory Strayer, Michael �� U.S. Department of Energy Office of Science Strohmaier, Erich �� Lawrence Berkeley National Laboratory Studham, R. Scott �� Oak Ridge National Laboratory Swain, William �� University of Tennessee Swesty, Douglas �� State University of New York at Stony Brook Szeto, Ernest ��Lawrence Berkeley National Laboratory / BDMTC Tang, William M. ��Princeton University Tautges, Timothy �� Argonne National Laboratory

150 Modeling and Simulation at the Exascale for Energy and the Environment

Taylor, Mark A. ��Sandia National Laboratories Tentner, Adrian �� Argonne National Laboratory Thakur, Rajeev �� Argonne National Laboratory Tieman, Brian �� Argonne National Laboratory Tierney, Brian �� Lawrence Berkeley National Laboratory Tobis, Michael �� University of Texas Towns, John ��National Center for Supercomputing Applications / University of Illinois Urbana-Champaign Trebotich, David ��Lawrence Livermore National Laboratory Uddin, Rizwan ��University of Illinois Urbana-Champaign Uhrig, Robert �� University of Tennessee Vaiana, Andrea C. �� Los Alamos National Laboratory Vaidya, Sheila ��Lawrence Livermore National Laboratory Vay, Jean-Luc �� Lawrence Berkeley National Laboratory Vazhkudai, Sudharshan �� Oak Ridge National Laboratory Verastegui, Becky �� Oak Ridge National Laboratory Vetter, Jeffrey S. �� Oak Ridge National Laboratory Vogt, David Paul �� Oak Ridge National Laboratory von Laszewski, Gregor �� Argonne National Laboratory Wadsworth, Jeffrey �� Oak Ridge National Laboratory Wang, Lin-Wang �� Lawrence Berkeley National Laboratory Wang, Paul T. ��Mississippi State University Wang, Yuson �� Argonne National Laboratory Ward, Robert C. �� University of Tennessee Warren, Michael �� Los Alamos National Laboratory Weaver, Nicholas ��International Computer Science Institute Wehner, Michael �� Lawrence Berkeley National Laboratory Wei, Yajun �� Northwestern University Weigand, Gilbert G. �� Oak Ridge National Laboratory Werner, Janet �� Argonne National Laboratory Wheat, Stephen ��Intel White III, James �� Oak Ridge National Laboratory Whitfield, David �� University of Tennessee Whitney, Paul ��Pacific Northwest National Laboratory Wilbanks, Thomas J. �� Oak Ridge National Laboratory Wilde, Michael ��University of Chicago / Argonne National Laboratory Wilke, Andreas �� University of Chicago Williams, Dean ��Lawrence Livermore National Laboratory Wing, William �� Oak Ridge National Laboratory Witek, Rich ��AMD Wolf, Matthew �� Georgia Tech Woodward, Paul ��University of Minnesota Worley, Brian �� UT-Battelle Wu, Kesheng �� Lawrence Berkeley National Laboratory Wullschleger, Stan D. �� Oak Ridge National Laboratory Xia, Fangfang �� University of Chicago Yang, Yunfeng �� Oak Ridge National Laboratory Yelick, Kathy ��University of California Berkeley Young, Glenn �� Oak Ridge National Laboratory Yu, Weikuan �� Oak Ridge National Laboratory Zacharia, Thomas �� Oak Ridge National Laboratory Zhang, Yingqi �� Lawrence Berkeley National Laboratory Zinner, Jenifer �� University of Chicago

151 Appendix C: Attendees Appendix D

Abbreviations and Terminology Appendix D: Abbreviations and Terminology

ABBREVIATIONS AND TERMINOLOGY 3D three-dimensional aa amino acid AGN active galactic nucleus AMIGA All Modular Industry Growth Assessment AMR adaptive mesh refinement ANL Argonne National Laboratory AOGCM Atmosphere-ocean general circulation model ASCR Advanced Scientific Computing Research BBH binary black hole BES Basic Energy Sciences BG Blue Gene BHNS black hole and neutron star BNS binary neutron star CAF Co-Array Fortran CGE computable general equilibrium CGRO Compton Gamma-Ray Observatory CMS Compact Muon Solenoid DAE differential algebraic equation DBA design basis accident DETF Dark Energy Task Force DFT density functional theory DVM dynamic vegetation model E3 Simulation and Modeling at the Exascale for Energy and the Environment ELM edge-localized mode EMF Energy Modeling Forum EOS equation of state ESM earth system model EVLA Enhanced Very Large Array EXIST Energetic X-ray Imaging Survey Telescope FFT fast Fourier transform flops floating point operations per second FPGA field programmable gate array FUSE Far Ultraviolet Spectroscopic Explorer Gbps gigabits per second GIS geographic information system GK gyrokinetic GLAST Gamma-ray Large Area Space Telescope GMT Giant Magellan Telescope GNEP Global Nuclear Energy Partnership GRB gamma-ray burst GTC Gyrokinetic Toroidal Code HCCI homogeneous charge compression ignition HPC high-performance computing

154 Modeling and Simulation at the Exascale for Energy and the Environment

HPCS High-Productivity Computer Systems HST Hubble Space Telescope I/O input/output IDS intrusion detection system IEA International Energy Agency INTEGRAL International Gamma-ray Astrophysics Laboratory IPCC Intergovernmental Panel on Climate Change IPS intrusion prevention system ISO Infrared Space Observatory IUE International Ultraviolet Explorer JDEM Joint Dark Energy Mission JFNK Jacobian-free Newton-Krylov JWST James Webb Space Telescope LBNL Lawrence Berkeley National Laboratory LES large eddy simulation LHC Large Hadron Collider LSST Large Synoptic Survey Telescope LTC low-temperature compression LWR light water reactor M&S modeling and simulation MHD magnetohydrodynamic MIT Massachusetts Institute of Technology MPI message-passing interface MPP massively parallel processing (or processors) MW megawatts NASA National Aeronautics and Space Administration NCSU North Carolina State University NEMS National Energy Modeling System NK Newton-Krylov NPP nuclear power plant NRC Nuclear Regulatory Commission NSF National Science Foundation OASCR Office of Advanced Scientific Computing Research ODE ordinary differential equation OLG overlapping generations ORNL Oak Ridge National Laboratory PB petabytes PDE partial differential equation PF petaflops PIC particle in cell R&D research and development RF radio frequency s-process slow neutron capture process SciDAC Scientific Discovery through Advanced Computing SitAware situational awareness

155 Appendix D: Abbreviations and Terminology

SKA Square Kilometer Array SN supernova SNF spent nuclear fuel SNP single-nucleotide polymorphism SOA secondary organic aerosol SOC system on chip SOS system of systems SVD singular value decomposition Tc critical temperature TDDFT time-dependent density functional theory TRU transuranic UCSD University of California, San Diego UPC Unified Parallel C V&V verification and validation VIRGO Variability of Solar Irradiance and Gravity Oscillations VO virtual organization WEO World Energy Outlook

156 Appendix E

References

157 Appendix E: References

Section 1: Climate Current status and future direction. Plasma Sci. Technol. 9:312–387. R. Alley, T. Berntsen, N. L. Bindoff, Z. Chen, A. Chidthaisong, P. Friedlingstein, J. Gregory, G. S. Ethier, W. M. Tang and Z. Lin (2005), Gyro Hegerl, M. Heimann, B. Hewitson et al. (2007), kinetic particle-in-cell simulations of plasma mi Climate change 2007: The physical science basis, croturbulence on advanced computing platforms. J. IPCC, Working Group 1 for the Fourth Assess Phys. Conf. Series 16, 1–15. ment, WMO. S. Ethier, W. M. Tang, R. Walkup and L. Oliker CNA Corporation (2007), National security and the (2007), Large scale gyrokinetic particle simula threat of climate change. National Research Council tion of microturbulence in magnetically confined (2001), Improving the effectiveness of U.S. climate fusion plasmas, IBM J. Res. Dev., accepted for modeling, National Academies Press. publication.

S. Doney, ed. (2004), Ocean carbon and climate W. W. Lee (1983), Gyrokinetic approach in particle change: An implementation strategy for U.S. simulation. Phys. Fluids 26(2) 556–562. ocean carbon research, U.S. Carbon Cycle Science Scientific Steering Group. W. W. Lee (1987), Gyrokinetic particle simulation model. J. Comput. Phys. 72, 243–269. Ian Foster, ed. (2007), Exascale global socioeconomic modeling enabling comprehensive climate Z. Lin, T. S. Hahm, W. W. Lee, W. M. Tang and change impact and response analysis, DOE. R. B. White (1998), Turbulent transport reduction by zonal flows: Massively parallel simulations. L. Ma, M. J. Shaffer and L. R. Ahuja (2001), Ap Science 281, 1835–1837. plication of RZWQM for soil nitrogen manage ment, pp. 265–301 in M. J. Shaffer, L. Ma and S. Z. Lin, T. S. Hahm, W. W. Lee, W. M. Tang and R. Hansen, eds., Modeling Carbon and Nitrogen Dy B. White (2000), Gyrokinetic simulations in general namics for Soil Management, Lewis Publ., Boca geometry and applications to collisional damping Raton, Florida. of zonal flows. Phys. Plasmas 7(5) 1857–1862.

S. Stich, B. Smith, C. I. Prentice, A. Arneth, A. Leonid Oliker, Andrew Canning, Jonathan Carter, Bondeau, W. Cramer, J. Kaplan, S. Levis, W. John Shalf and Stephane Ethier (2004), Scientific Lucht, M. Sykes, K. Thonicke and S. Venevsky computations on modern parallel vector systems. In (2003), Evaluation of ecosystem dynamics, plant Proc. SC2004, Pittsburgh, PA.. geography, and terrestrial carbon cycling in the LPJ L. Oliker, A. Canning, J. Carter, C. Iancu, M. dynamic global vegetation model, Glob. Change Likewski, S. Kamil, J. Shalf, H. Shan, E. Stroh Biol. 9, 161–185. maier, S. Ethier, and T. Goodale (2007), Scientific application performance on candidate petaScale Section 2A: Energy—Combustion platforms. In Proc. IPDPS’07, Long Beach, CA. DOE Office of Basic Energy Sciences 2006. Basic L. Oliker, J. Carter, M. Wehner, A. Canning, S. Research Needs for Clean and Efficient Combus Ethier, B. Govindasamy, A. Mirin, D. Parks, P. tion of 21st Century Transportation Fuels (http:// Worley, S. Kitawaki, Y. Tsuda (2005), Leading www.sc.doe.gov/bes/reports/files/CTF_rpt.pdf). computational methods on scalar and vector HEC DOE Office of Fossil Energy 2004. FutureGen platforms. In Proc. SC2005, Seattle, WA. Integrated Hydrogen, Electric Power Production and W.M. Tang and V. S. Chan (2005), Advances Carbon Sequestration Research Initiative (http:// and challenges in computational plasma science. www.fossil.energy.gov/programs/powersystems-/ Plasma Phys. Control. Fusion 47(2) R1–R34. futuregen/futuregen_report_march_04.pdf). Section 2C: Energy—Solar Section 2B: Energy—Nuclear Fusion DOE office of Basic Energy Sciences 2005. Basic D. A. Batchelor, M. Beck, A. Becoulet, R. V. Research Needs for Solar Energy uti¬lization: Budny, C. S. Chang, P. H. Diamond, J. Q. Dong, G. Report of the Basic Energy Sciences Workshop Y. Fu, A. Fukuyama, T. S. Hahm, D. E. Keyes, Y. on Solar Energy Utilization, April 18-21, 2005. Kishimoto, S. Klasky, L. L. Lao, K. Li, Z. Lin, B. (http:www.sc.doe.gov/bes/re¬ports/files/SEU_ Ludaescher, J. Manickam, N. Nakajima, T. Ozeki, rpt.pdf). N. Podhorszki, W. M. Tang, M. A. Vouk, R. E. Waltz, S. J. Wang, H. R. Wilson, X. Q. Xu, M. Yagi and F. Zonca (2007), Simulation of fusion plasmas:

158 Modeling and Simulation at the Exascale for Energy and the Environment

Section 2D: Energy—Nuclear Fission of Climate 19:5686–5699. IEA (2007), International Energy Agency. World DOE Office of Basic Energy Sciences (2006), Energy Outlook, www.worldenergyoutlook.org. Basic Research Needs for Advanced Nuclear Energy Systems: Report of the Basic Energy IMAGE 2.2 Model Flow Diagram (2007), www. Sciences Workshop on Basic Research Needs mnp.nl/image/model_details. for Advanced Nuclear Energy Systems, July IMAGE team (2001), The IMAGE 2.2 implemen 31–August 3, 2006 (http://www.sc.doe.gov/bes/ tation of the SRES scenarios: A comprehensive reports/files/ANES_rpt.pdf). analysis of emissions, climate change and impacts in the 21st century, RIVM CD-ROM publication DOE Office of Nuclear Energy, Science, and 481508018, National Institute for Public Health Technology (2006), Global Nuclear Energy and the Environment, Bilthoven, the Netherlands. Partnership Technology Development Plan, IPCC (2007), Intergovernmental Panel on Climate Global Nuclear Energy Partnership Technology Change (IPCC) 4th assessment report, www.ipcc. Development Program Integration Office, January ch, 2007. 9, 2007. K. Judd (1998), Numerical Methods in Econom ics, MIT Press. Section 3: Biology J. A. Laitner and D. A. Hanson (2006), Modeling S. S. Goens, S. Botero, A. Zemla, C. E. Zhou and detailed energy-efficiency technologies and tech M. L. Perdue (2004), J. Gen. Virol. 85, 3195. nology policies within a CGE framework, The En ergy Journal (Special Issue on Hybrid Modeling E. Klipp, W. Liebermeister, A. Helbig, A. Kowald of Energy-Environmental Policies: Reconciling and J. Schaber (2007), Nature Biotechnol. 25, 390. Bottom-up and Top-down). A. J. McMichael, D. H. Campbell-Lendrum, C. F. Section 4: Socioeconomic Modeling Corvalán, K. L. Ebi, A. K. Githeko, J. D., Scheraga and A. Woodward, eds. (2003), Climate change and H. M. Amman, D. A. Kendrick and J. Rust, eds. human health: Risks and Responses. World Health (1996), Handbook of Computational Economics, Organization, Geneva. vol. 1. Elsevier. B. C. Murray, A. J. Sommer, B. Depro, B. L. A. Dixit and R. Pindyck (1992), Investment Under Sohngen, B. A. McCarl, D. Gillig, B. D. Angelo Uncertainty. MIT Press. and K. Andrasko (2005), Greenhouse gas mitiga J. Edmonds, M. Wise, H. Pitcher, R. Richels, T. tion potential in US forestry and agriculture. EPA Wigley and C. MacCracken (1997), An integrated Report 430-R-05-006. assessment of climate change and the accelerated NAPAP Integrated Assessment Report. National introduction of advanced energy technologies: Mitigation and Acid Precipitation Assessment Program (NAPAP), An application of MiniCAM 1.0, Office of the Director, 1990. Adaptation Strategies for Global Change 1(4) 311–339. The National Energy Modeling System: An over view (2003). Energy Information Administration, EMF Study-21 (2006), Special issue on multi- U.S. Department of Energy. greenhouse gas mitigation and climate policy, EMF Study 21. The Energy Journal. A. P. Sokolov, C. A. Schlosser, S. Dutkiewicz, S. Paltsev, D. W. Kicklighter, H. D. Jacoby, R. G. B. C. English, D. G. De La Toore Ugarte, K. Prinn, C. E. Forest, J. Reilly, C. Wang, B. Felzer, Jenson, C. Hellwinckel, J. Menard, B. Wilson, M. C. Sarofim, J. Scott, P. H. Stone, J. M. Melillo R. Roberts and M. Welsh (2006), 25% renewable and J. Cohen (2005), The MIT Integrated Global energy for the United States by 2025: Agricultural System Model (IGSM) version 2: Model descrip and economic impacts. Department of Agricultural tion and baseline evaluation. MIT. Economics, University of Tennessee. M. L. Stein (1999), Statistical Interpolation of S. W. Hadley, D. J. Erickson III, J. L. Hernandez, C. Spatial Data: Some Theory for Kriging. Springer, T. Broniak and T. J. Blasing (2006), Responses of New York. energy use to climate change: A climate modeling study. Geophys. Res. Lett., 33 (L17703). Sustainable Bioenergy: A Framework for Deci sion Makers (2006), UN Energy. D. A. Hanson and J. A. Laitner (2006), Technol ogy policy and world greenhouse gas emissions in M. Q. Wang (2001), Development and use of the AMIGA modeling system, The Energy Journal GREET 1.6 fuel-cycle model for transportation (Special Issue on Multi-Greenhouse Gas Mitigation fuels and vehicle technologies (2001), Center and Climate Policy). for Transportation Research, Argonne National Laboratory. I. Held and B. Soden (2006), Robust responses of the hydrological cycle to global warming. Journal

159 Appendix E: References

Section 5: Astrophysics Section 6: Math & Algorithms Andreas Albrecht, Gary Bernstein, Robert Cahn, A. T. Adai, S. V. Date, S. Wieland and E. M. Wendy L. Freedman, Jacqueline Hewitt, Wayne Marcotte (2004), LGL: Creating a map of protein Hu, John Huth, Marc Kamionkowski, Edward function with an algorithm for visualizing W. Kolb, Lloyd Knox, John C. Mather, Suzanne very large biological networks, J. Mol. Biol. Staggs and Nicholas B. Suntzeff (2006), Report 340(1):179-190. of the Dark Energy Task Force, arXiv.org:astro- ph/0609591. D. A. Bader (2004), Computational biology and high-performance computing, special issue on John G. Baker, Joan Centrella, Dae-Il Choi, bioinformatics, Communications of the ACM Michael Koppitz and James van Meter (2006), 47(11):34–41. Gravitational wave extraction from an inspiraling configuration of merging black holes. Phys. Rev. D. A. Bader, A. Snavely and G. Jacobs (2006), NSF Letters 96:111102. workshop report on petascale computing in the biological sciences, August 29–30, Arlington, VA. J. M. Blondin and A. Mezzacappa (2007), Pulsar spins from an instability in the accretion shock of D. H. Bailey (2005), High-precision arithmetic in supernovae, Nature 445:58–60. scientific computation, Computing in Science and Engrg., May–June, 54–61. A. C. Calder, D. M. Townsley, I. R. Seitenzahl, F. Peng, O. E. B. Messer, N. Vladimirova, E. F. M. Barad and P. Colella (2005), A fourth-order Brown, J. W. Truran and D. Q. Lamb (2007), accurate local refinement method for Poisson’s Capturing the fire: flame energetics and neutroni equation, J. Comp. Physics 209(1) 1–18. zation for Type Ia supernova simulations, ApJ 656:313–332. M. J. Berger and S. H. Bokhari (1987), A parti tioning strategy for nonuniform problems on mul M. Campanelli, C. O. Lousto, P. Marronetti and tiprocessors, IEEE Trans. Computers C-36(5): Y. Zlochower (2006), Accurate evolutions of or 570–580. biting black-hole binaries without excision, Phys. Rev. Letters 96: 111101. J. Birge and F. Louveaux (1997), Introduction to Stochastic Programming, Springer. V. N. Gamezo, A. M. Khokhlov and E. S. Oran (2005), Three-dimensional delayed-detonation U. Catalyurek and C. Aykanat (1996), Decom model of Type Ia supernovae, ApJ 623:337–346. posing irregularly sparse matrices for parallel ma trix-vector multiplications, Lecture Notes in Com E. J. Hallman, B. W. O’Shea, J. O. Burns, M. puter Science 1117: 75–86. L. Norman, R. Harkness and R. Wagner (2007), The Santa Fe Light Cone Simulation Project, I: U. Catalyurek and C. Aykanat (1999), Hyper Confusion and the WHIM in upcoming Sun graph-partitioning-based decomposition for par yaev-Zel’dovich effect surveys, ArXiv e-prints allel sparse-matrix vector multiplication, IEEE 704:0704.2607. Trans. Parallel Dist. Systems 10(7): 673–693. Wayne Hu and Scott Dodelson (2002), Cosmic S. Chandra, M. Parashar and J. Ray (2007), Ana microwave background anisotropies, Ann. Rev. lyzing the impact of computational heterogeneity Astronomy and Astrophysics 40:171. on runtime performance of parallel scientific com ponents, in Proc. 15th High Performance Comput Hans-Thomas Janka, K. Langanke, A. Marek, G. ing Symposium (HPC-07), SCS Spring Simula Martinez-Pinedo and B. Mueller (2007), Theory of tion Multiconference, Norfolk, VA. core-collapse supernovae, Phys. Rept. 442:38–74. B. Christianson and M. Cox (2005), Automatic D. Lynden-Bell (1969), Galactic nuclei as col propagation of uncertainties, pp. 47–58 in M. lapsed old quasars, Nature 223:690. Buecker, G. Corliss, P. Hovland, U. Naumann, and B. Norris, eds., Automatic Differentiation: Ap Anthony Mezzacappa (2005), Ascertaining the plications, Theory, and Implementations. Lecture core-collapse supernova mechanism: The state of Notes in Computational Science and Engineering the art and the road ahead, Ann. Rev. Nucl. Part. vol. 50, Springer. Sci. 55, 467–515 (2005). K. Devine, E. Boman, R. Heaphy, R. Bisseling Carlos F. Sopuerta, Ulrich Sperhake and Pablo and U. Catalyurek (2006), Parallel hypergraph Laguna (2006), Hydro-without-hydro framework partitioning for scientific computing, in Proc. IP for simulations of black hole-neutron star bina DPS 2006. ries, Classical and Quantum Gravity 23:S579. V. Eijkhout (1998), Overview of iterative linear S. E. Woosley and J. S. Bloom (2006), The su system solver packages, NHSE Review 3(1). pernova – gamma-ray burst connection, Ann. Rev. Astronomy and Astrophysics 44:507. J. Gondzio and R. Sarkissian (2003), “Parallel Interior-Point Solver for Structured Linear Pro grams.” Mathematical Programming 96: 561-584.

160 Modeling and Simulation at the Exascale for Energy and the Environment B. Hayes (2003), A lucid interval, American Sci J. Steensland and J. Ray (2003), A heuristic re- entist 91(6) 484–488. mapping algorithm reducing inter-level com munication in SAMR applications, in Proc. 15th B. Hendrickson and R. Leland (1995), A multi IASTED International Conference on Parallel level algorithm for partitioning graphs. in Proc. and Distributed Computing and Systems 2003 Supercomputing ‘95. (PDCS03). H. Johansson and J. Steensland (2006), A per R. W. Walters and L. Huyse (2002), Uncertainty formance characterization of load balancing alog analysis for fluid mechanics with applications. rithms for parallel SAMR applications, Technical Technical report, NASA, Feb. Report 2006-047, Uppsala University, Dept. of Information Technology. M. Woolridge (2002), An Introduction to Multi- Agent Systems. Wiley & Sons. G. Karypis and V. Kumar (1998), A fast and high quality multilevel scheme for partitioning irregular graphs, SIAM J. Sci. Computing 20(1) 359–392. Section 7: Software Ray Bair, Lori Diachin, Stephen Kent, George G. Karypis and V. Kumar (1997), A coarse-grain Michaels, Anthony Mezzacappa, Richard Mount, parallel multilevel k-way partitioning algorithm, Ruth Pordes, Larry Rahn, Arie Shoshani, Rick in Proc. 8th SIAM Conf. Parallel Processing for Stevens, and Dean Williams (2003), Planning Scientific Computing. ASCR/Office of Science data-management G. J. Klir (1994), The many faces of uncertainty, strategy. http://www-conf.slac.stanford.edu/ pp. 3–19 in B. M Ayyub and M. M. Gupta, eds., dmw2004/docs/DM-strategy-final.doc. Uncertainty Modeling and Analysis: Theory and J. Duell, P. Hargrove and E. Roman (2002), The Applications, Elsevier Science. Design and Implementation of Berkeley Lab’s X. Li (2006), Direct solvers for sparse matrices, Linux Checkpoint/Restart, Berkeley Lab Techni http://crd.lbl.gov/~xiaoye/SuperLU/SparseDi cal Report LBNL-54941. rectSurvey.pdf, September. R. L. Graham, S.-E. Choi, D. J. Daniel, N. N. G. Nemhauser and L. Wolsey (1988), Integer and Desai, R. G. Minnich, C. E. Rasmussen, L. D. Combinatorial Optimization, John Wiley & Sons. Risinger and M. W. Sukalksi (2003), A network- failure-tolerant message-passing system for teras cale clusters, Int. J. Parallel Programming 31(4) J. Nocedal andn S. Wright (2006), Numerical Op timization, 2 ed., Springer. 285-303. M. J. North and C. M. Macal (2007), Managing W. Gropp and Ewing Lusk (2004), Fault tolerance Business Complexity: Discovering Strategic Solu in Message Passing Interface programs, Int. J. tions with Agent-Based Modeling and Simulation, High Perform. Comput. Appl. 18(3):363-372. Oxford. J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur, A. Patra and J. T. Oden (1995), Problem de W. Gropp, R. Latham, A. Siegel, B. Gallagher, and composition strategies for adaptive hp finite ele M. Zingale (2003), Parallel netCDF: A high-per ment methods, Computing Systems in Eng. 6(2) formance scientific I/O interface, inProc. SC2003, 97–109. Phoenix, AZ. J. R. Pilkington and S. B. Baden (1994), Parti E. Lusk and Kathy Yelick (2007), Languages for tioning with spacefilling curves, CSE Technical high-productivity computing: The DARPA HPCS Report CS94-349 Dept. Computer Science and language project, Parallel Processing Letters Engineering, University of California – San Diego. 17(1), March. J. Ray, C. A. Kennedy, S. Lefantzi and H. N. Najm J. Parker, T. Engelsiepen, R. Ross, R. Thakur, (2007), Using high-order methods on adaptively R. Latham, and W. Gropp (2006), High-perfor refined block-structured meshes derivatives, in mance file I/O for the BlueGene/L supercomputer, terpolations, and filters, SIAM J. Sci. Computing in Proc. 12th International Symposium on High- 29(1):139–181. Performance Computer Architecture (HPCA-12). P. Schwartz, M. Barad, P. Colella and T. Ligocki R. Ross, Jose Moreira, Kim Cupps, and Wayne (2006), A Cartesian grid embedded boundary Pfeiffer (2006), Parallel I/O on the IBM Blue method for the heat equation and Poisson’s equa Gene /L system, BlueGene Consortium Quarterly tion in three dimensions, J. Comput. Phys. 211(2) Newsletter, February. 531–550. Rob Ross, Evan Felix, Bill Loewe, Lee Ward, Gary H. D. Simon (1991), Partitioning of unstructured Grider and Rob Hill (2005), HPC file systems and problems for parallel processing, Computing Sys scalable I/O: Suggested research and development tems in Engrg. 2: 135–148. topics for the fiscal 2005–2009 time frame, ftp:// ftp.lanl.gov/public/ggrider/HEC-IWG-FS-IO- J. Steensland (2002), Efficient partitioning of dy Workshop-08-15-2005/FileSystems-DTS-SIO- namic structured grid hierarchies, Uppsala Uni FY05-FY09-R&D-topics-final.pdf. versity Library, Uppsala, Sweden.

161 Appendix E: References

Douglas Thain, Todd Tannenbaum and Miron Charlie Catlett et al. (2007), TeraGrid: Analysis Livny (2005), Distributed computing in practice: of organization, system architecture, and middle The Condor experience, Concurrency – Practice ware enabling new types of applications, in High and Experience 17(2–4) 323-356. Performance Computing and Grids in Action, IOS Press, Advances in Parallel Computing series, Amsterdam. Section 8: Hardware Narayan Desai, Andrew Lusk, Rick Bradshaw J. Blyler (2005), Navigating the Silicon Jun and Remy Evard (2003), BCFG: A configura gle: FPGA or ASIS?, Chip Design, June/ tion management tool for heterogeneous environ July, http://www.chipdesignmag.com/display. ments, in Proc. IEEE International Conference on php?articleId=115&issueId=11 Cluster Computing, pp. 500, 2003. C. Coarfa, Y. Dotsenko, J. Mellor-Crummey, F. Ian Foster, Jens Voeckler, Michael Wilde and Yong Cantonnet, T. El-Ghazawi, A. Mohanty and Y. Zhao (2003), The virtual Data Grid: A new model Yao (2006), An evaluation of global address space and architecture for data-intensive collaboration, languages: Co-array Fortran and Unified Parallel in Proc. Conference on Innovative Data Systems C, in Proc. Principles and Practice of Parallel Research. Programming (PPoPP), New York. Veronika Nefedova, Robert Jacob, Ian Foster, M. Elnozahy (2006), IBM has its PERCS, HP Zhengyu Liu, Yun Liu, Ewa Deelman, Gaurang CWire 15(14), April. http://www.hpcwire.com/ Mehta, Mei-Hui Su and Karan Vahi (2006), Au hpc/614724.html. tomating climate science: Large ensemble simu lations on the TeraGrid with the GriPhyN virtual F. Johnson (2006), DARPA HPCS Program, Sci Proc. 2nd IEEE International DAC Review 2 data system, p. 32 in , Fall, http://www.scidacreview. Conference on e-Science and Grid Computing. org/0602/html/news2.html Rick Stevens (2003), Access Grid: Enabling A. Krasnov, Andrew Schultz, John Wawrzynek, (2007), group-oriented collaboration on the Grid, in The Greg Gibeling and Pierre-Yves Droz Grid: Blueprint for a New Computing Infrastruc RAMP Blue: A message-passing manycore sys ture, Ian Foster and Carl Kesselman, eds., Morgan tem in FPGAs, in Proc. FPL 2007 - International Conference on Field Programmable Logic and Kaufmann. Applications, Amsterdam. Rick Stevens, Michael E. Papka and Terry Disz (2003), Prototyping the workspace of the future, C. Maxcer (2007), D-Wave claims quantum com IEEE Internet Computing 7(4): 51–58. puting breakthrough, TechNewsWorld, www.tech newsworld.com/story/55801.html. Gregor von Laszewski, Michael Hategan and Deepti Kodeboyina (2007), Java CoG Kit work M. Reilly, L. C. Stewart, J. Leonard and D. Gin flow, pp. 340-356 in Workflows for Science, I. Tay gold (2006), SiCortex technical summary, www. lor, E. Deelman, D. B. Gannon, and M. Shields, sicortex.com/whitepapers/sicortex-tech_summa eds., Springer. ry.pdf. Von Welch, Jim Barlow, James Basney, Doru D. E. Shaw et al. (2007), Anton, a special-purpose Marcusiu and Nancy Wilkins-Diehr (2006), A machine for molecular dynamics simulations, in AAAA model to support science gateways with Proc. 34th Annual International Symposium on Concurrency and Computa Computer Architecture, community accounts, San Diego, pp. 1-12. tion: Practice and Experience 19(6):893-904. M. Wehner, L. Oklier, and J. Shalf (2007), To Yong Zhao, Michael Wilde and Ian Foster (2007), wards ultra-high performance resolution models Virtual Data Language: A Typed Workflow Nota of Climate and Weather, Intl. J. High-Performance Comp. Apps. tion for Diversely Structured Scientific Data, pp. , to appear. 258-278 in Workflows for eScience,Springer. Section 9: Cyberinfrastructure Bill Allcock, Joe Bester, John Bresnahan, Ann L. Chervenak, Carl Kesselman, Sam Meder, Veroni ka Nefedova, Darcy Quesnel, Steven Tuecke and Ian Foster (2001), Secure, efficient data transport and replica management for high-performance da ta-intensive computing, p. 13 in Proc. 18th IEEE Symposium on Mass Storage Systems.

162 Modeling and Simulation at the Exascale for Energy and the Environment

DISCLAIMER

This report was prepared as an account of a workshop sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees or of- ficers, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of document authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. E3