SUSTAINED PETASCALE IN ACTION: ENABLING TRANSFORMATIVE RESEARCH 2019 ANNUAL REPORT SUSTAINED PETASCALE IN ACTION: ENABLING TRANSFORMATIVE RESEARCH 2019 ANNUAL REPORT Editor Catherine Watkins Creative Director Steve Duensing Graphic Designer Megan Janeski Project Director William Kramer The research highlighted in this book is part of the Blue Waters sustained-petascale computing project, which is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana–Champaign and its National Center for Supercomputing Applications. Visit https://bluewaters.ncsa.illinois.edu/science-teams for the latest on Blue Waters- enabled science and to watch the 2019 Blue Waters Symposium presentations. CLASSIFICATION KEY To provide an overview of how science teams are using Blue Waters, researchers were asked if their work fit any of the following classifications (number responding in parentheses): Data-intensive: Uses large numbers of files, large disk space/ DI bandwidth, or automated workflows/offsite transfers, etc. (83) GPU-accelerated: Written to run faster on XK nodes than on GA XE nodes (55) LEADING BY EXAMPLE Thousand-node: Scales to at least 1,000 nodes for production TN science (74) Memory-intensive: Uses at least 50 percent of available memory MI on 1,000-node run (21) BW Only on Blue Waters: Research only possible on Blue Waters (43) The National Center for Super- Earlier in 2020 I was privileged to be part of a del- computing Applications (NCSA) egation of University of Illinois leaders that partici- Multiphysics/multiscale: Job spans multiple length/time scales or was funded in 1986 to enable dis- pated in discussions on the next frontier of AI, data MP physical/chemical processes (59) coveries not possible anywhere science, and design thinking with Illinois alumnus TABLE OF else. We have a long history of Tom Siebel and his staff at C3.ai. In his remarks, Illi- Machine learning: Employs deep learning or other techniques; ML includes “big data” (18) doing things that have not been nois’ Vice Chancellor for Academic Affairs and Pro- done before. The Blue Waters Proj- vost Andreas Cangellaris commented that “no mat- Communication-intensive: Requires high-bandwidth/low- CONTENTS ect admirably continued that tradi- ter how powerful the computers are, the real power CI latency interconnect for frequent tightly coupled messaging (33) tion and spawned new avenues of comes from the people behind them.” 3 Industrial application: Researcher has private sector NCSA DIRECTOR BILL GROPP – research and discovery for NCSA Blue Waters is not just a peak petascale system; IA collaborators or results directly applicable to industry (10) LEADING BY EXAMPLE and our partners. it is a true sustained-petascale system, specifically We are leading the way in mul- designed to be well balanced and deliver petascale Frontier science: This science has produced a first-of-its-kind FS outcome (21) 4 PROJECT DIRECTOR BILL KRAMER – timessenger astrophysics (MMA). computing power and productivity across a variety TOGETHER, WE CHANGED THE WORLD In fact, we were recently awarded of applications. Because of this balance, for almost Big idea: This research relates to the National Science over $20 million in grants in MMA areas that would seven years the Blue Waters Project has enabled re- BI Foundation’s “10 Big Ideas” initiative (83) not have been possible without the discoveries Blue markable work in biology, chemistry, physics, geo- 6 BLUE WATERS THROUGH THE YEARS Waters helped achieve. MMA combines data from sciences, cosmology and astrophysics, atmospher- cosmic messengers such as gravitational waves, neu- ic science, and many other fields such as agriculture, 10 BLUE WATERS METRIC INFORMATION trinos, and electromagnetic waves. We are bringing economics, political science, and the social sciences. together large-scale computing, data science, arti- You’ll find these remarkable discoveries in the pag- 12 EXTENDED ABSTRACTS ficial intelligence (AI), machine learning (ML), and es that follow. Discoveries that were made by the peo- software development not only to advance these do- ple behind the computer—the extraordinary NCSA mains but to develop techniques, technologies and staff and the scientists they support in their quest for 12 space science software that may transfer to other domains as well. frontier science. They are leading by example. And as 76 geoscience We’re helping to marry medicine and technology as the leader of NCSA, I would not have it any other way. a partner with the Carle Illinois College of Medicine, 118 physics & engineering the world’s first engineering-based medical school. And for many years we’ve partnered with Mayo Clinic 210 computer science & engineering to advance precision medicine—drugs or treatments designed for small groups rather than large popula- 236 biology, chemistry & health Dr. William “Bill” Gropp tions—by applying ML and AI. Director We are incorporating ML and AI into agriculture as 318 social science, economics, & humanities National Center for a partner in the University of Illinois’ new Center for Supercomputing Applications 326 mathematical & statistical sciences Digital Agriculture. With the Institute for Genomic Biology; the College of Agricultural, Consumer and KB = kilobytes 330 graduate fellows Environmental Sciences; and the Grainger College MB = megabytes of Engineering we’ll be facilitating student education GB = gigabytes 372 SCIENCE AND ENGINEERING ADVISORY in the use of technology in agriculture as well as as- TB = terabytes TEAM COMMITTEE sisting farmers and agricultural industries to advance PB = petabytes digital agriculture to keep pace with the ways tech- I/O = input/output nology is transforming how we feed and support a 373 Knh = thousand node hours OTHER BLUE WATERS PROJECTS growing global population. Mnh = million node hours In addition, we recently launched the Center for MPI = message-passing interface 374 REFERENCES Artificial Intelligence Innovation at NCSA that will focus on leveraging past and existing research proj- Allocations denoted as 406 INDEX ects to develop software to amplify current AI and type/size in extended abstracts. ML research areas and explore new ones. 3 TOGETHER, WE CHANGED THE WORLD I take great pride in the dict weather 10 to 15 days in advance, from study- We also inspire future generations to become in- achieving breakthrough research, improved tools and Blue Waters Project and in ing basic fluid dynamics to using computing for com- terested in science and technology and educate the processes, and trained the next generation of compu- the outstanding science and plete aircraft and ship design, and for moving from public on how HPC facilitates scientific discovery. By tational researchers. This publication celebrates the project teams that make Blue studying small molecules to understanding the ba- offering guided tours of the National Petascale Com- accomplishments of our partners and the research- Waters an exceptional open sic principles of life. puting Facility as well as open houses, more than ers, educators, and students they support but also resource for research. One of the most unexpected and remarkable suc- 12,000 visitors have come to see Blue Waters in ac- celebrates the dedication and innovation of the Blue Every year I write how cesses of Blue Waters in 2019 was the role it played tion, many of them K–12 and college students. Waters staff. Blue Waters enabled com- in a gerrymandering decision reached by Ohio’s Sixth By sharing the many best practices we developed, The end of the NSF funding for Blue Waters’ “broad putational, data analysis, and District Court, where University of Illinois at Urba- our staff is distributing the expertise and experiences access” phase is not the end of the story. Blue Waters now machine learning/artifi- na–Champaign’s Wendy Cho presented an analysis we have acquired from our accumulated knowledge. will continue providing groundbreaking insight as it cial intelligence investigations conducted on Blue Waters and based on more than More importantly, our Blue Waters science teams transitions to being a high-performance computation- that could not be done oth- three million simulations of possible electoral maps have published more than 1,000 articles and papers al and data resource for the National Geospatial-Intel- erwise. Once again, as you go for Ohio. Cho used a novel evolutionary Markov chain in their respective domains. ligence Agency (NGA). Blue Waters will focus ninety through this report you will Monte Carlo algorithm designed to take advantage of By being conscientious stewards of the grant mon- percent of the system resources on NGA-related proj- see the many accomplishments that carry the badge Blue Waters’ massively parallel architecture to pro- ies awarded to us, we were able to operate the Blue ects, with ten percent of Blue Waters reserved for use indicating “Only on Blue Waters” to signify that the duce an array of map scenarios that could be com- Waters supercomputer an additional 18 months be- by researchers at the University of Illinois at Urba- research would not have been possible on other pared to the current map. Further, she served as an yond original plans, providing nearly seven years of na–Champaign. As Illinois’ Vice Chancellor for Aca- open-science resources deployed at the time. expert witness and presented results from her sim- access to Blue Waters instead of the five years orig- demic Affairs and Provost Andreas Cangellaris com- Through the years, increasing numbers of proj- ulations on Blue Waters to objectively compare bi- inally planned by NSF. We were also able to add ad- mented, “no matter how powerful the computers are, ects have been identified as involving “data inten- ased and unbiased redistricting. ditional programs not in our original project scope, the real power comes from the people behind them.” sive” processing or as incorporating machine learn- Making high-performance computing (HPC) avail- such as the aforementioned Broadening Participa- NGA will be building upon the amazingly impact- ing/artificial intelligence.