SUSTAINED PETASCALE IN ACTION: ENABLING TRANSFORMATIVE RESEARCH 2018 ANNUAL REPORT SUSTAINED PETASCALE IN ACTION: ENABLING TRANSFORMATIVE RESEARCH 2018 ANNUAL REPORT Editors Barbara Jewett Catherine Watkins Creative Director Steve Duensing Graphic Designer Megan Janeski Project Director William Kramer The research highlighted in this book is part of the Blue Waters sustained-petascale computing project, which is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. Visit https://bluewaters.ncsa.illinois.edu/science-teams for the latest on Blue Waters- enabled science and to watch the 2018 Blue Waters Symposium presentations. CLASSIFICATION KEY To provide an overview of how science teams are using Blue Waters, researchers were asked if their work fit any of the following classifications (number responding in parentheses): Data-intensive: uses large numbers of files, e.g. large disk space/ DI bandwidth, or automated workflows/off-site transfers (69) TABLE OF BLUE WATERS CHANGED GPU-accelerated: written to run faster on XK nodes than on XE GA nodes (40) THE WORLD AND ILLINOIS Thousand node: scales to at least 1,000 nodes for production CONTENTS TN science (72) 3 NCSA DIRECTOR BILL GROPP – BLUE WATERS Memory intensive: uses at least 50 percent of available memory MI on 1,000-node runs (22) CHANGED THE WORLD AND ILLINOIS Blue Waters: research only possible on Blue Waters (35) For the past five years, the Blue Agricultural and Consumer Economics (ACES) at the BW 4 PROJECT DIRECTOR BILL KRAMER – ENABLING Waters Project at the National University of Illinois at Urbana-Champaign, found FRONTIER SCIENCE AND RESEARCH Multi-physics/multi-scale: job spans multiple length/time scales Center for Supercomputing that from October 2007 to June 2019 the Project has MP or physical/chemical processes (59) Applications has enabled a projected $1.08 billion direct economic impact 6 PAST NCSA DIRECTOR THOM DUNNING – Machine learning: employs deep learning or other techniques, remarkable work in biology, on Illinois' economy. Furthermore, Blue Waters is ML includes "big data” (16) CONGRATULATIONS ON FIVE YEARS OF chemistry, physics, geosciences, the catalyst for many of NCSA’s engagements with cosmology and astrophysics, Industry and other research organizations even if they Communication-intensive: requires high-bandwidth/low-latency SERVICE TO THE NATION’S SCIENCE CI interconnect for frequent, tightly coupled messaging (40) AND RESEARCH COMMUNITY atmospheric science, and many do not use Blue Waters. other fields such as agriculture, NCSA was founded in 1986 to enable discoveries Industry applicable: researcher has private sector collaborators economics, political science, and not possible anywhere else. The Blue Waters Project IA or results directly applicable to industry (10) 8 BECAUSE OF BLUE WATERS the social sciences. You’ll find has admirably continued and expanded that tradition. Frontier Science: this science has produced a first of its kind these remarkable discoveries in FS outcome (21) 10 BLUE WATERS MELDS DATA ANALYSIS WITH the pages that follow. HIGH-PERFORMANCE COMPUTING Big idea: this research relates to the NSF's "10 Big Ideas" Blue Waters has always been BI initiative (70) about much more than aiding scientific discovery. 14 BLUE WATERS SYMPOSIUM KEYNOTES Blue Waters was the first petascale system for Dr. William “Bill” Gropp HIGHLIGHT SIGNIFICANT SCIENCE ADVANCES academic use, continuing NCSA’s long history of Director doing things that have not been done before. And National Center for Supercomputing 18 EXTENDED ABSTRACTS Blue Waters is not just a peak petascale system—it is a Applications true sustained-petascale system, specifically designed to be well-balanced and deliver petascale computing 18 space science power across a variety of applications. 70 geoscience Those pioneering efforts helped us attract the best talent to not only NCSA, but also the University of 102 physics & engineering Illinois. The Project helped recruit and retain 20 professors at the UI by designating them as “Blue 178 computer science & engineering Waters Professors,” an honor that comes with 200 biology, chemistry & health substantial computing and data resources intended to boost their research efforts. 262 social science, economics, & humanities This amazing marvel of technology and engineering is one of our area’s biggest attractions, drawing visitors 268 graduate fellows from across the country and around the world. In the last five years, more than 12,000 people—ranging in 318 BLUE WATERS FULL SERVICE ASSESSMENT age from 2 weeks to 95—have toured NCSA’s National SUMMARY Petascale Computing Facility, which includes a visit to NCSA’s machine room and an opportunity to be up KB = kilobytes 322 SCIENCE AND ENGINEERING ADVISORY close to Blue Waters. Our outreach staff has standing TB = terabytes TEAM COMMITTEE annual reservations from K-12 schools, colleges, PB = petabytes and universities to bring students as part of their I/O = input/output curriculum, often traveling hundreds of miles to do so. 323 OTHER BLUE WATERS PROJECTS Knh = thousand node hours An often overlooked aspect of the Blue Waters Mnh = million node hours Project is the economic impact the Project has had 324 REFERENCES on the state of Illinois. A 2017 study conducted by Dr. Allocations denoted as Sandy Dall’erba, director of the Regional Economics type/size in extended abstracts. 348 INDEX Applications Laboratory and associate professor of 3 ENABLING FRONTIER SCIENCE AND RESEARCH Every year when I sit to best of breed and “out front” uses. This process is In the first 2014 report, the projects were all teams. Included among the principal investigators write this letter and reflect how the world moved from the first 2-D atmospheric traditional large-scale, physics-based modeling and (PIs) are 10 females and two underrepresented on the doors we have opened circulation model to being able to accurately predict simulation efforts. In this report, 53% of the projects minorities. In addition, there are four female and eight to discovery, I feel the same weather 10-15 days in advance, from studying basic identify as “data intensive” processing and 11% as underrepresented minority colleagues listed as co-PIs. wonder and delight that I did fluid dynamics to using computing for complete doing machine learning/artificial intelligence. This Among the lead institutions, five are Minority Serving when we published our first aircraft and ship design, and going from studying shift is another indication of the leadership and Institutions and nine are within EPSCoR jurisdictions. annual report five years ago. small molecules to understanding the basic principles impact Blue Waters is making. Many projects are not Our Blue Waters intern program for undergraduates As you read the reports in this of life. only data intensive or machine learning, but instead has trained over 100 of the next generation HPC book, I hope you share in this are combining these methods with multi-scale and workforce. In addition, our highly-competitive amazement at the incredible EMBEDDED FRONTIER multi-physics simulation to achieve cutting edge Blue Waters Fellows program has helped 46 young SCIENCE RESEARCH scientific discoveries that are BoB Method BoB Method results. Blue Waters is truly a catalyst for convergence, researchers gain experience on advanced research Ubiquitous possible as a result of the Blue New not just through supporting teams using different methods to take full advantage of leadership-class Problems Waters Project. methods, but also helping science teams integrate computing. As part of its mission, Blue different methods within their work to achieve At the suggestion of our NSF review panel, we’ve Waters enables computational and data analysis Improvements remarkable new results. shared some “lessons learned” over the course of investigations that could not be done otherwise. COMMUNITY OUT FRONT The numbers are impressive, but the Blue Waters our five years. By sharing the more than 35 best RESEARCH RESEARCH In fact, you can see many reports carry the badge BoB in Many Others Project has always been about much more than just practices we developed, our staff is distributing the Implementations Apply BoB indicating “Only on Blue Waters” to signify that providing physical computing, memory capacity, and expertise and experiences we have acquired from our the research would not have been possible on other storage resources. While high performance resources accumulated knowledge. More importantly, our Blue currently-deployed U.S. open-science resources. I am amazed that over just five years of service, Blue are necessary for success, they are not sufficient for Waters science teams have published more than 1,000 When I am asked why leadership computing systems Waters and the science teams using it have repeatedly effectiveness. articles and papers in their respective domains and the like Blue Waters are required, I often refer to the pushed the frontier into common practice process Through the Project’s Petascale Application project staff was also active delivering presentations cycle of discovery represented in the included diagram. multiple times, in multiple disciplines. Examples Improvement Discovery (PAID) program that and publishing papers on best practices developed Whether called Frontier Science or Grand Challenges, include the first all-atom simulation of the HIV capsid concluded last year, the Project provided millions of through our operation of Blue Waters. these “breakthroughs” are first-of-a-kind advances. (and now all-atom simulations of the flu and other dollars to science teams and computational and data I take great pride in the Blue Waters Project These make the infeasible feasible; they raise the diseases), to the first ever simulations of a billion years experts to improve the performance of applications and in the outstanding science and project teams bar in a given field through higher fidelity, longer of evolution of the universe and galaxy formation, to in a measurable manner.