ALCF-2 Early Science Program Technical Reports
Total Page:16
File Type:pdf, Size:1020Kb
ANL/ALCF/ESP-13/17 ALCF-2 Early Science Program Technical Reports Compendium of Individual Technical Reports for the 16 ESP Projects Argonne Leadership Computing Facility About Argonne National Laboratory Argonne is a U.S. Department of Energy laboratory managed by UChicago Argonne, LLC under contract DE-AC02-06CH11357. The Laboratory’s main facility is outside Chicago, at 9700 South Cass Avenue, Argonne, Illinois 60439. For information about Argonne and its pioneering science and technology programs, see www.anl.gov. Availability of This Report This report is available, at no cost, at http://www.osti.gov/bridge. It is also available on paper to the U.S. Department of Energy and its contractors, for a processing fee, from: U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, TN 37831-0062 phone (865) 576-8401 fax (865) 576-5728 [email protected] Disclaimer This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor UChicago Argonne, LLC, nor any of their employees or officers, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of document authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, Argonne National Laboratory, or UChicago Argonne, LLC. ANL/ALCF/ESP-13/17 ALCF-2 Early Science Program Technical Reports Compendium of Individual Technical Reports for the 16 ESP Projects Edited by Timothy J. Williams Argonne Leadership Computing Facility, Argonne National Laboratory July 18, 2013 ALCF-2 Early Science Program Technical Reports Editor: Timothy J. Williams, Argonne Leadership Computing Facility July 2013 1 Contents ALCF-2 Early Science Program Technical Reports Contents 1 The Early Science Program 3 1.1 These Technical Reports . .3 2 Acknowledgements 3 3 Climate-Weather Modeling Studies Using a Prototype Global Cloud-System Re- solving Model 5 4 Materials Design and Discovery: Catalysis and Energy Storage 11 5 Direct Numerical Simulation of Autoignition in a Jet in a Cross-Flow 23 6 High Accuracy Predictions of the Bulk Properties of Water 39 7 Cosmic Structure Probes of the Dark Universe 45 8 Accurate Numerical Simulations Of Chemical Phenomena Involved in Energy Production and Storage with MADNESS and MPQC 57 9 Petascale, Adaptive CFD 71 10 Using Multi-scale Dynamic Rupture Models to Improve Ground Motion Esti- mates 85 11 High-Speed Combustion and Detonation (HSCD) 91 12 Petascale Simulations of Turbulent Nuclear Combustion 97 13 Lattice Quantum Chromodynamics 119 14 Petascale Direct Numerical Simulations of Turbulent Channel Flow 129 15 Ab-initio Reaction Calculations for Carbon-12 141 16 NAMD - The Engine for Large-Scale Classical MD Simulations of Biomolecular Systems Based on a Polarizable Force Field 157 17 Global Simulation of Plasma Microturbulence at the Petascale & Beyond 161 18 Multiscale Molecular Simulations at the Petascale 175 Title Page Photos: Top: Mira|one of three rows of 16 IBM Blue Gene/Q racks. Bottom: Attendees at the ALCF Early Science Program Kick-Off Workshop, 18-19 October 2010, in front of Argonne National Laboratory's TCS Conference Center. 2 The ESP ALCF-2 Early Science Program Technical Reports 1 The Early Science Program The Early Science Program (ESP) is managed for the Department of Energy by the Argonne Leadership Computing Facility (ALCF). ESP was funded to prepare key scientific applications for the architecture and scale of Mira, a 48K-node IBM Blue Gene/Q system that went into production at ALCF in April 2013, and to solidify libraries and infrastructure that will pave the way for other production applications. To distinguish it from past and future Early Science Programs, we denote this one as the ALCF-2 ESP (ALCF-2 is the project name associated with procurement of Mira). The 16 Early Science projects are the result of a call for proposals in 2010, and were chosen based on computational and scientific reviews. The projects, in addition to promising delivery of exciting new science, are all based on state-of-the-art, petascale, parallel applications. Starting in October 2010, the project teams, in collaboration with ALCF staff and IBM, have undertaken intensive efforts to adapt their software to take advantage of Mira's Blue Gene/Q architecture, which, in a number of ways, is a precursor to future HPC architectures. Together, the 16 projects span a diverse range of scientific fields, numerical methods, programming models, and computational approaches. The latter include particle-mesh methods, adaptive meshes, spectral methods, Monte Carlo, molecular dynamics, and ab initio computational chemistry methods. These applications also represent a large portion of the ALCFs current and projected production computational workload. The official Early Science period lasted only about three months (between machine acceptance and commencement of production), during which the ESP had dedicated use of Mira. Projects have about 2 billion core-hours to use on Mira, as much as possible during that 3 month period, so It was essential that the projects were "ready to run" when the clock started ticking. The long lead time of the Program, and dedicated postdoctoral appointees for most projects, working with ALCF staff, helped make that possible. 1.1 These Technical Reports The purpose of the Technical Reports gathered here is to document and publicly circulate what the ESP projects have done and learned in preparing their application codes for Mira. This includes, for example, approaches for better use of in-node thread parallelism, accessing the BG/Q's QPX functionality (4-way SIMD vectorization), use of BG/Q specific libraries such as the low-level PAMI communication system, and using ESSL and Mass libraries. We hope that these examples will inform and help our production users on Mira, and those working on optimization applications en route to applying for computer time grants on ALCF systems via the INCITE and ALCC programs. 2 Acknowledgements For all projects in these reports: This research used resources of the Argonne Leadership Computing Facility at Argonne National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under contract DE-AC02-06CH11357. 3 The ESP ALCF-2 Early Science Program Technical Reports 4 GFDL esp ALCF-2 Early Science Program Technical Reports 3 Climate-Weather Modeling Studies Using a Prototype Global Cloud-System Resolving Model PI: V. Balaji (Geophysical Fluid Dynamics Laboratory) Project Summary We expect our understanding of the role of clouds in climate to undergo a qualitative change as the resolutions of global models begin to encompass clouds. At these resolutions, non-hydrostatic dynamics become significant and deep convective processes are resolved. We are poised at the threshold of being able to run global scale simulations that include direct, non-parameterized, simulations of deep convective clouds. The goal of this project is to use Mira to explore the frontier of weather prediction and climate modeling with the newly developed Geophysical Fluid Dynamics Laboratory (GFDL) global cloud-resolving model. A single, unified atmospheric modeling system with a cubed-sphere dynamical core and bulk cloud microphysics running at hydrostatic ( 10 km) and non-hydrostatic ( 5 km) resolutions will be run with the goal of capturing the climatology∼ of clouds and severe storms≤ in a warming world. The ability to reproduce historical tropical storm statistics will be used as a test of this ground-breaking model. The purpose of the experiments proposed is to validate the global cloud-resolving climate model via hurricane hindcasts. For this purpose, we will perform hurricane verification studies for the 2008 Atlantic Season. Storms in 2008 lasted a total of 100 days; performing 5-day forecasts on each of the days would give a total of 500 forecast days for the season. Report originally released as ANL/ALCF-ESP-13/1 5 GFDL esp ALCF-2 Early Science Program Technical Reports Climate-Weather Modeling Studies Using a Prototype Global Cloud-System Resolving Model Research Team Members: V. Balaji – Computational Scientist, Princeton University Shian-Jiann Lin – Physical Scientist, GFDL Christopher Kerr – Computational Scientist, GFDL Executive Summary: Clouds remain the largest source of uncertainty in our understanding of global climate change. Different aspects of the planetary cloud field can provide positive and negative feedback to the Earth’s energy balance, and clouds of course are directly implicated in changes to the planetary distribution of precipitation. A fundamental problem in our current understanding of the role of clouds in the dynamics of climate are that current resolutions do not resolve the fundamental length scales associated with clouds. We expect our understanding of the role of clouds in climate to undergo a qualitative change as the resolutions of global models begin to encompass clouds. At these resolutions (which