BLUE WATERS ANNUAL REPORT 2015

Today’s first-principles calculations need METHODS & RESULTS and long runs. By contrast, the QHA used in COMPUTATIONAL MINERAL databases of equilibrium thermal elastic conjunction with first principles vibrational PHYSICS properties of minerals and their assemblages to We used standard and extended/hybrid density density of states [3,5] is much more efficient model Earth and other planets [1]. The challenge functional theory (DFT)-based methods as for calculations corresponding to very large to computations is to perform these highly implemented in the Quantum ESPRESSO numbers of atoms. demanding calculations efficiently in a large software [e.g., 2–6]. These DFT calculations We routinely performed calculations Allocation: GLCPC/0.35 Mnh combined QHA and MD to address thermal 4 PI: Renata M. Wentzcovitch1 number of phases with variable compositions. corresponding to 10 atoms [4,7,9,15]. However, Collaborators: Tao Sun2, Dong-Bo Zhang3, Han Hsu4, Mehmet Topsakal1, Gaurav For example, the chemical composition of the elastic properties. these were high throughput calculations 1 1 1 Shukla , Kanchan Sarkar , Michel Marcondes terrestrial mantle includes at least five major Thermodynamics methods requiring typically 102 to 103 medium-size Thermodynamic states were investigated 2 1 oxide components (MgO, SiO2, Al2O3, FeO, CaO) calculations with ~10 atoms. In particular, University of Minnesota, Twin Cities using QHA [22] and MD. QHA and MD are T 2 and tens of solid phases (fig. 1). thermoelastic constants, cijkl (P,T), a central University of Chinese Academy of Sciences complementary approaches, the former from 3Computational Science Research Center, Chinese Academy of Engineering Physics, Our work is motivated and informed by central topic in our project, was one of the most CPU Beijing questions in Earth and planetary sciences. It low to intermediate temperatures (T < ~2/3 intensive calculations we performed. These 4 National Central University, Taiwan focuses on building highly realistic models that Tmelting) and the latter from intermediate to Tmelting jobs could be executed concurrently in single or and beyond. Both methods are CPU intensive EXECUTIVE SUMMARY: account for all physical and chemical variations in distributed compute nodes. Ref. [6] summarizes the system (e.g., chemical composition, pressure, because they involve statistical sampling of phase a few of these calculations [4,7,15,18]. They were Mineral physics is one of three branches of temperature, etc.). space. In MD this translates into large supercells geophysics (the others being geodynamics We perform calculations on tens of mantle and seismology). Geophysics advances by phases using primarily quasi-harmonic (QHA) close cooperation between these fields. Thus, computations. However, molecular dynamics mineral physics focuses on mineral properties (MD) or hybrid QHA/MD calculations are that are needed to interpret seismic data or are performed in the case of anharmonic phases or at essential for geodynamics simulations. To be temperatures near melting. As far as elasticity is useful, mineral properties must be investigated concerned, a rigorous treatment of solid solutions across a wide range of pressure, temperature, and is unnecessary for mineral aggregates. These chemical composition. Chemical composition is results will address questions such as: complex, encompassing at least five major oxide • What is the seismic signature in the deep components and tens of solid phases. Today, mantle produced by the sinking lithospheric first-principles computations address these plates made of mid-ocean ridge basalt (MORB), challenges. Minerals’ properties are fundamental harzburgite, and peridotite? Conversely, to what to elucidate a planet’s state, and atomistic studies extent can mantle heterogeneities be attributed of these complex materials are fundamental to to seismic contrasts between these assemblages understanding their properties. We will use Blue and “normal” pyrolite mantle [19]? Waters to perform first-principles calculations of • What are the temperatures, chemistries, and unprecedented magnitude and scope in mineral mineral assemblages of the two large, low-shear physics. velocity provinces in the deep mantle [21]? Did they result of accumulation of MORB enriched material? INTRODUCTION • What seismic signature do spin changes The goal of high pressure mineral physics is to produce in iron in lower mantle minerals [e.g., increase our knowledge of the materials that 10,16–17]? make up the Earth and other planets [1]. For • What are the phase boundary and velocity many problems, first-principles theory [e.g., 2–6] contrast produced by the post-perovskite is the only practical method of investigation transition [8,11,20] in MORB, harzburgite, available. First-principles methods have been peridotite, and pyrolite? This information must addressing materials problems important to be available before one can interpret seismic FIGURE 1: understanding the present state and evolution observations of the D” region, the ~300 km Earth’s seismic of the Earth and other planets [4,7–14]. However, above the CMB. discontinuities computation of full thermal elastic anomalies (top) [59] caused by spin crossovers at high temperatures and mantle and pressures has not been realized. mineralogy (bottom; e.g., [60]).

58 59 BLUE WATERS ANNUAL REPORT 2015

carried out by submitting bag-of-task (BoT) [14] multiple configurations (~101–102) need to be effects of much higher horizontal (10–30 km) type jobs to single or distributed systems using explored in each of up to ~104 calculations. USING PETASCALE COMPUTING and vertical resolution. There is considerable the VLab service-oriented cyberinfrastructure We have essentially investigated all major end- CAPABILITIES TO ADDRESS CLIMATE evidence that increased resolution leads to better [12–14]. member phases and simple versions of the main simulations of both transient small-scale activity On Blue Waters, these calculations were mineral solid solutions of the mantle using only CHANGE UNCERTAINTIES (e.g., eddies and waves) and large-scale features performed using BoTs or MPI directives, “image” a few few atomic configurations. While elastic of the mean climate. parallelization (1st level of parallelization). For properties of crystalline aggregates are well We expect that results from our studies will Allocation: NSF PRAC/5.02 Mnh a typical system with 20 atoms (e.g., MgSiO3 described by using few atomic configurations, be an integral part of the scientific analysis Donald J. Wuebbles1 perovskite or post-perovskite [7,15]), the best phase diagrams containing binary (or multi- PI: of climate change for the major international Co-PIs: Zachary Zobel1, Warren Washington2, Thomas Bettge2, Julio Bacmeister2, performance was achieved with 128 cores per phase) loops need accurate free energies, which Kevin Reed2, Xin-Zhong Liang3, Chao Sun3 climate assessments (e.g., the follow-on to job. Thus, a full thermodynamic calculation can only be achieved through careful sampling the Assessment Report 5, AR5, 2014) of the could be performed in less than one minute, or of atomic configurations. An example of this 1University of Illinois at Urbana–Champaign Intergovernmental Panel on Climate Change 2 a calculation of thermoelastic constants could problem can be seen in fig. 1, where OPx and National Center for Atmospheric Research (IPCC), which will likely influence U.S. and 3University of Maryland, College Park be performed in a couple of minutes. CPx phases “dissolve” into Gt (garnet) within a international policies regarding the effects of FIGURE 2: (a) broad pressure range (i.e. 5 GPa < P < 18 GPa). A EXECUTIVE SUMMARY: human activities on the Earth’s climate, and Chemical detailed description of this phenomenon requires provide supplementary data for use in various This collaborative research between the University compositions of WHY BLUE WATERS? a considerably larger number of (smarter) climate impact assessments at regional to local of Illinois, the National Center for Atmospheric mantle rocks [85- calculations. scales. The major reason for using Blue Waters is Research (NCAR), and the University of Maryland 87]. Mineralogy the number of cores available. We developed uses Blue Waters to address key uncertainties in (in mol%) of (b) wrappers around several modules of the numerically modeling the Earth’s climate system pyrolite [88], (c) PUBLICATIONS METHODS AND RESULTS Quantum ESPRESSO software that allow us to and accuracy in analyses of past and projected harzburgite, and use the system for high-throughput calculations. Lacerda, M. M., and R. M. Wentzcovitch, future changes in climate at a level that would CESM (d) basalt [89] in As explained above, phase space sampling for Hybrid ab-initio/experimental high temperature be impossible without petascale computing. Our After tuning the physics and testing model settings the lowermost thermodynamics calculations is intrinsically a equations of state: the NaCl pressure scale. J. Appl. studies used the latest, most advanced versions of (the atmospheric portion of) CESM, we began mantle. Mg- high-throughput problem. Phys., 117 (2015), 215902, doi:10.1063/1.4921904. of the Community Earth System Model (CESM) a series of simulations that will support studies Silicate: MgSiO 3 A future Track-1 system will allow us to Shukla, G., et al., Thermoelasticity of (Mg,Fe) and two versions of NCAR’s Weather Research for the next Climate Model Intercomparison perovskite expand the dimension of phase space currently SiO3 perovskite. Geophys. Res. Lett., 42:6 (2015), and Forecasting Model (WRF and CWRF) for Project 6 (CMIP6), which is important to the and/or post- sampled. Minerals are solid solutions and a full pp. 65–77, doi:10.1016/S0031-9201(96)03213-X. high-resolution regional climate analyses. These next IPCC assessment of climate change. perovskite, Fp: investigation of phase equilibrium in these solids Wu, R., et al., The Atomic and Electronic model runs put us on the pathway for major Recent results from simulations that ran 1979 ferropericlase, involves sampling of atomic configurations. We Structure of Black Phosphorus. J. Vac. Sci. Technol. international leadership in high-resolution to the present suggested that the global number Ca-Pv: CaSiO3 have just started addressing this issue. We are A, 33 (2015), 060604, doi:10.1116/1.4926753. climate modeling studies. of tropical storms and hurricanes per year will perovskite, and exploring techniques to do maximally efficient decrease in a warming climate. However, the CF: calcium- sampling of atomic configurations. Nevertheless, maximum intensity seemed to increase, meaning ferrite-type INTRODUCTION more major hurricanes (category 4 and 5). In the phase. North Atlantic basin the RCP8.5 scenario showed This collaborative research used Blue Waters a decrease in the number of tropical cyclones to address key uncertainties in numerically and hurricanes. The selection of a dynamical modeling the Earth’s climate system and accuracy core can have a significant impact on tropical in analyses of past and projected future changes cyclone intensity and frequency even in the in climate. Our studies used the latest versions presence of similar climatology and large-scale of the Community Earth System Model (CESM), environments [1]. For example, CAM5 with the the Weather Research and Forecasting model spectral element (SE) core produced stronger (WRF), and the Climate-WRF (CWRF). cyclones, and therefore more hurricanes and We explored the effect of external forcings (e.g., major hurricanes per year, than CAM5 with concentrations of greenhouse gases) on globally the finite volume (FV) core (figs. 1–2). The averaged temperature, focusing on uncertainties exact causes for these differences is an area of associated with the representation of processes continued work. and interactions between clouds, aerosols, and Compared to the out-of-box model, we radiative transfer in the models and how these improved the performance of CAM5 from 1.3 uncertainties influence the climate sensitivity. years/day to 2.5 years/day. Pat Worley (ORNL) Our second objective aimed to evaluate CESM and Ryan Mokos (NCSA) helped remove with different model dynamical cores and the bottlenecks in MPI calls, while John Truesdale

60 61