NSF Extensible Terascale Facility TeraGrid Report for Final Report: October 1, 2001 through July 31, 2011 Submitted December 20, 2011 Principal Investigators (GIG and RPs) Ian Foster (GIG) University of Chicago/Argonne National Laboratory (UC/ANL) Patricia Kovatch University of Tennessee (UT-NICS) Jay Boisseau Texas Advanced Computing Center (TACC) John Cobb Oak Ridge National Laboratory (ORNL) Michael Levine Pittsburgh Supercomputing Center (PSC) Rich Loft National Center for Atmospheric Research (NCAR) Honggao Liu Louisiana Optical Network Initiative/Louisiana State University (LONI/LSU) Richard Moore San Diego Supercomputer Center (SDSC) Carol Song Purdue University (PU) Rick Stevens University of Chicago/Argonne National Laboratory (UC/ANL) Craig Stewart Indiana University (IU) John Towns National Center for Supercomputing Applications (NCSA) i TeraGrid Forum (TGF) John Towns, chair (NCSA) Matt Heinzel (UC/ANL) Patricia Kovatch (UT-NICS) Jay Boisseau (TACC) John Cobb (ORNL) Honggao Liu (LONI/LSU) Michael Levine (PSC) Rich Loft (NCAR) Richard Moore (SDSC) Mike Papka (UC-ANL) Carol Song (PU) Craig Stewart (IU) Grid Infrastructure Group Ian Foster (GIG) PI of TeraGrid GIG Matt Heinzel (UC) Director of the TeraGrid GIG Tim Cockerill (NCSA) Project Management Working Group Kelly Gaither (TACC) Visualization Christopher Jordan (TACC) Data Analysis Daniel S. Katz (UC/ANL) GIG Director of Science Jeff Koerner (UC/ANL) Operations and Networking, Operations and Security Scott Lathrop (UC/ANL) Education, Outreach and Training; External Relations Elizabeth Leake (UC) External Relations Lee Liming (UC/ANL) Software Integration and Scheduling Amit Majumdar (SDSC) Advanced User Support J.P. Navarro (UC/ANL) Software Integration and Scheduling Mike Northrop (UC) GIG Project Manager Sergiu Sanielevici (PSC) User Services and Support Nancy Wilkins-Diehr (SDSC) Science Gateways ii Working Group Leaders Dave Hart (SDSC/NCAR) Accounting Amit Majumdar (SDSC) Advanced User Support Kent Milfield (TACC) Allocations Shawn Brown (PSC) Common User Environment Jeff Koerner (UC) Core Services Christopher Jordan (TACC) Data Scott Lathrop (UC/ANL) Education, Outreach and Training Elizabeth Leake (UC) External Relations Nick Nystrom (PSC) Extreme Scalability Linda Winkler (ANL) Networking Jeff Koerner (UC) Operations Tim Cockerill (NCSA) Project Management Kate Ericson (SDSC) and Shava Smallen (SDSC) Quality Assurance Warren Smith (TACC) Scheduling Nancy Wilkins-Diehr (SDSC) Science Gateways Jim Marsteller (PSC) Security Lee Liming (UC/ANL) and J.P. Navarro (UC/ANL) Software Dave Hart (SDSC/NCAR) User Facing Projects Sergiu Sanielevici (PSC) User Services Kelly Gaither (TACC) and Mike Papka (UC/ANL) Visualization iii TERAGRID FINAL REPORT Contents 1 Executive Summary ......................................................................................................... 5 2 Introduction and Overview ............................................................................................. 6 2.1 Key lessons learned ........................................................................................................ 8 2.2 Recommendations ........................................................................................................ 10 3 Impact to Science and Engineering............................................................................... 11 3.1 Deep Usage .................................................................................................................. 12 3.2 Wide Usage .................................................................................................................. 14 3.3 Allocations .................................................................... Error! Bookmark not defined. 3.4 Science Output ............................................................................................................. 15 3.5 Lessons learned ............................................................................................................ 15 4 Impact to Researchers, Educators and the Broader Community .............................. 19 4.1 How TeraGrid Helped .................................................................................................. 19 4.2 Strategies ...................................................................................................................... 20 4.3 Shortcomings ............................................................................................................... 23 4.4 Directions for Improvement ......................................................................................... 24 5 Impacts on Education, Outreach and Training ........................................................... 25 5.1 Training ........................................................................................................................ 26 5.2 Education ..................................................................................................................... 27 5.3 Outreach ....................................................................................................................... 29 5.4 Cross-cutting Perspectives ........................................................................................... 32 6 Administration and Organization ................................................................................. 34 6.1 Leadership, Authority and Conflict in a Distributed Organization .............................. 34 6.2 Management and Organization .................................................................................... 36 6.3 Interactions with the Funding Agency ......................................................................... 37 6.4 Managing Change in Light of Uncertainty .................................................................. 38 6.5 Closing Thoughts on Administration and Organization .............................................. 38 iv 1 Executive Summary The original Distributed Terascale Facility (DTF) proposal—the genesis of TeraGrid—set the stage for a new era of computational science and engineering. The following is excerpted from the Project Summary of that proposal: As a comprehensive computational, data management, and networking environment of unprecedented scale and capability, the TeraGrid will be the enabling cyberinfrastructure for 21st century U.S. science and engineering. Operating as a single distributed resource that is co-allocated and scheduled, the TeraGrid will support both traditional scientific computing and emerging disciplinary scientific groups. A unified TeraGrid Operations Center will coordinate system management, user support, and access. … By engaging vendors, application researchers, computing experts, and government collaborators in a partnership that leverages the explosive growth of Grid technology, open source software, and high-performance commodity hardware, we can deploy a cyberinfrastructure far more powerful and flexible than any single supercomputing system, catalyzing scientific discovery across broad disciplines. This vision was the basis for the TeraGrid. Its fundamental message has not significantly changed, but our understanding of what it fully means and what it implies has changed over the life of the project. As envisioned, the TeraGrid’s impact to science and engineering has been profound. An astounding amount of work has been enabled by the TeraGrid’s systems and integrated services, at all levels. This can be seen in the annual Science Highlight booklet produced by TeraGrid. These examples have come from a variety of fields, including astrophysics, material science, climate, biomolecular science, medicine, geophysics, energy, fluid dynamics, weather, particle physics, imaging physics, literature and text studies, electronic structures, economics, machine learning, bioinformatics, epidemiology, network analysis, and ecology. A number of these were not anticipated a decade or more ago when the original TeraGrid proposal was written. These highlights are but a subset of the nearly 9,000 peer-reviewed publications supported by TeraGrid. Scientists and engineers have been successful because TeraGrid made user success one of its highest priorities. The infrastructure and supporting services catalyzed and supported an increasing number of collaborative science, computer science, and computational science teams allowing complex questions to be addressed in multiple domains. The TeraGrid encouraged not only U.S., but also global collaborations, which led to new science and new capabilities. Non- U.S. scientists have been able to make use of TeraGrid resources as part of collaborations with U.S. scientists. This has facilitated the work of the U.S. partners, and helped TeraGrid by bringing in leading-edge users who had demands that foreshadowed future user needs. The TeraGrid not only provided a single-system information and support interface for using all the National Science Foundation’s (NSF) Office of Cyberinfrastructure (OCI) funded supercomputers, it also pioneered various successful mechanisms for engaging broader communities. The Science Gateways program has had tremendous impact on the way researchers and educators alike go about their business. Simplified interfaces to domain-specific tools hid many of the complicated details of running on HPC resources, lowering the bar of entry and enabling a wider community of users. The Campus Champions program enrolled volunteer faculty and IT staff members from universities across the country