XSEDE: The Extreme Science and Engineering Discovery Environment (OAC 15-48562) Interim Project Report 12: Report Year 5, Reporting Period 1 May 1, 2020 – July 31, 2020

RY5 IPR12 Page i

XSEDE Senior Management Team (SMT) John Towns (NCSA) PI and Project Director Tim Boerner (NCSA) Deputy Project Director Kelly Gaither (TACC) Co -PI and Community Engagement and Enrichment Director Philip Blood (PSC) Co-PI and Extended Collaborative Support Service Co- Director Bob Sinkovits (SDSC) Co-PI and Extended Collaborative Support Service Co- Director Greg Peterson (UT-NICS) XSEDE Operations Director David Hart (NCAR) Resource Allocations Service Director David Lifka (Cornell) XSEDE Cyberinfrastructure Integration Director Ron Payne (NCSA) Program Office Director Emre Brookes (UT-San Antonio) User Advisory Committee Chair Ruth Marinshaw (Stanford) XD Service Providers Forum Chair Leslie Froeschl (NCSA) Senior Project Manager (ex officio) Lizanne Destefano (Georgia Tech) Lead of External Evaluation (ex officio) Bob Chadduck (NSF) Cognizant NSF Program Office (ex officio)

RY5 IPR12 Page ii

Table of Contents XSEDE: ...... i The Extreme Science and Engineering Discovery Environment ...... i XSEDE Senior Management Team (SMT) ...... ii List of Tables...... vi Reading this Report ...... vii 1. Executive Summary ...... 1 Strategic Goals ...... 2 Summary & Project Highlights ...... 4 COVID-19 Contributions ...... 7 2. Science, Engineering, & Program Highlights ...... 9 Slower and Noisier ...... 9 Supercomputer Simulations Show How DNA Prepares itself for Repair ...... 10 XSEDE allocation Facilities Expanded Solar Wind Predictions ...... 12 LED’s Bright Early Light ...... 13 Star Crash ...... 14 Every Calculation Stabs ...... 16 Georgia Tech Engineers Simulate Solar Cell Work Using XSEDE-Allocated Supercomputers ...... 18 Settling In ...... 19 XSEDE Supercomputers Aid Drug Screening For Deadly Heart Arrhythmias ...... 20 XSEDE Resources Used for High Tech Materials Science Study ...... 22 Supercomputers Simulations Help Advance Electrochemical Reaction Research ...... 23 3. Discussion of Strategic Goals and Key Performance Indicators ...... 25 Deepen and Extend Use ...... 25 3.1.1. Deepening Use to Existing Communities ...... 25 3.1.2. Extending Use to New Communities ...... 26 3.1.3. Prepare the Current and Next Generation ...... 28 3.1.4. Raising Awareness ...... 28 Advance the Ecosystem ...... 29 3.2.1. Create an Open and Evolving e-Infrastructure ...... 29 3.2.2. Enhance the Array of Technical Expertise and Support Services ...... 30 Sustain the Ecosystem...... 31 3.3.1. Provide Reliable, Efficient, and Secure Infrastructure ...... 31 3.3.2. Provide Excellent User Support ...... 31

RY5 IPR12 Page iii

3.3.3. Effective and Productive Virtual Organization...... 32 3.3.4. Innovative Virtual Organization ...... 33 4. Community Engagement & Enrichment (WBS 2.1) ...... 35 CEE Director’s Office (WBS 2.1.1) ...... 37 Workforce Development (WBS 2.1.2) ...... 37 User Engagement (WBS 2.1.3) ...... 39 Broadening Participation (WBS 2.1.4) ...... 40 User Interfaces & Online Information (WBS 2.1.5) ...... 40 Campus Engagement (WBS 2.1.6) ...... 41 5. Extended Collaborative Support Service (WBS 2.2) ...... 43 ECSS Director’s Office (WBS 2.2.1) ...... 44 Extended Support for Research Teams (WBS 2.2.2) ...... 45 Novel & Innovative Projects (WBS 2.2.3) ...... 46 Extended Support for Community Codes (WBS 2.2.4) ...... 48 Extended Support for Science Gateways (WBS 2.2.5) ...... 48 Extended Support for Training, Education, & Outreach (WBS 2.2.6) ...... 49 6. XSEDE Cyberinfrastructure Integration (WBS 2.3) ...... 51 XCI Director’s Office (WBS 2.3.1) ...... 52 Requirements Analysis & Capability Delivery (WBS 2.3.2) ...... 53 Cyberinfrastructure Resource Integration (WBS 2.3.3) ...... 54 7. XSEDE Operations (WBS 2.4) ...... 55 Operations Director’s Office (WBS 2.4.1) ...... 56 Cybersecurity (WBS 2.4.2) ...... 56 Data Transfer Services (WBS 2.4.3) ...... 56 XSEDE Operations Center (WBS 2.4.4) ...... 56 System Operations Support (WBS 2.4.5) ...... 57 8. Resource Allocation Service (WBS 2.5) ...... 58 RAS Director’s Office (WBS 2.5.1) ...... 59 XSEDE Allocations Process & Policies (WBS 2.5.2) ...... 59 Allocations, Accounting, & Account Management CI (WBS 2.5.3) ...... 60 9. Program Office (WBS 2.6) ...... 62 Project Office (WBS 2.6.1) ...... 64 External Relations (WBS 2.6.2) ...... 64 Project Management, Reporting, & Risk Management (WBS 2.6.3) ...... 65 Business Operations (WBS 2.6.4) ...... 65

RY5 IPR12 Page iv

Strategy, Planning, Policy, Evaluation & Organizational Improvement (WBS 2.6.5)...... 65 10. Financial Information ...... 67 11. Project Improvement Fund ...... 82 12. Appendices ...... 85 Glossary and List of Acronyms ...... 85 Metrics ...... 89 12.2.1. SP Resource and Service Usage Metrics ...... 89 12.2.2. Other Metrics...... 99 Scientific Impact Metrics (SIM) and Publications Listing ...... 112 12.3.1. Summary Impact Metrics ...... 112 12.3.2. Historical Trend ...... 112 12.3.3. Publications Listing ...... 113 13. Collaborations ...... 154 14. Service Provider Forum Report ...... 160 SP Forum administrative and membership activities during this reporting period 160 15. UAC ...... 162 16. XMS Summary ...... 163 Executive Summary ...... 163 17. XAB Executive Summaries ...... 164 Executive Summary of XSEDE Advisory Board Meeting, Tuesday, April 21, 2020 ... 164 18. XSEDE Project Execution Plan ...... 171

RY5 IPR12 Page v

List of Tables Table 1-1: Summary of Key Performance Indicators (KPIs) for XSEDE...... 3 Table 3-1: KPIs for the sub-goal of deepen use (existing communities)...... 25 Table 3-2: KPIs for the sub-goal of extend use (new communities)...... 27 Table 3-3: KPI for the sub-goal of preparing the current and next generation...... 28 Table 3-4: KPIs for the sub-goal of raise awareness of the value of advanced digital research. .... 28 Table 3-5: KPI for the sub-goal of create an open and evolving e-infrastructure...... 30 Table 3-6: KPI for the sub-goal of enhance the array of technical expertise and support services...... 30 Table 3-7: KPI for the sub-goal of provide reliable, efficient, and secure infrastructure...... 31 Table 3-8: KPIs for the sub-goal of provide excellent user support...... 32 Table 3-9: KPIs for the sub-goal of operate an effective and productive virtual organization...... 33 Table 3-10: KPIs for the sub-goal of operate an innovative virtual organization...... 34 Table 4-1: KPIs for Community Engagement & Enrichment...... 36 Table 5-1: KPIs for Extended Collaborative Support Service...... 43 Table 6-1: KPIs for XSEDE Cyberinfrastructure Integration (XCI)...... 51 Table 7-1: KPIs for Operations...... 55 Table 8-1: KPIs for Resource Allocation Service...... 58 Table 9-1: KPIs for Program Office...... 62 Table 10-1: Project Level Financial Summary...... 67 Table 10-2: Partner Institution Level Financial Summary...... 68 Table 12-1: Quarterly activity summary...... 90 Table 12-2: End of quarter XSEDE open user accounts by type, excluding XSEDE staff...... 92 Table 12-3: Active institutions in selected categories. Institutions may be in more than one category...... 93 Table 12-4: Project summary metrics...... 94 Table 12-24: Project activity by allocation board type...... 95 Table 16: Resource activity, by type of resource, excluding staff projects...... 96 Table 17: Globus data transfer activity to and from XSEDE endpoints, excluding XSEDE speed page user...... 98

RY5 IPR12 Page vi

Reading this Report This report is the result of an ongoing process of improving reporting on the progress in delivering on our mission and realizing our goals thus communicating the value XSEDE brings towards enhancing the productivity of a growing community of scholars, researchers, and engineers. For a large, complex, highly-distributed project such as XSEDE, this is a considerable undertaking. This process has helped XSEDE improve as an organization and as a provider and broker of services to the compute- and data-enabled science and engineering research and education community. XSEDE reports on its activities and progress by using a metrics-based approach. Based on feedback from our review panels, advisory bodies, the NSF, and other stakeholders, we have defined KPIs (Key Performance Indicators) that measure progress toward our high-level strategic goals. The key concept is not that the metrics [KPIs] themselves have a direct causal effect on eventual outcomes, but rather that the metrics are chosen so that actions and decisions which move the metrics in the desired direction also move the organization in the direction of the desired outcomes and goals. KPIs at the project and Work Breakdown Structure Level 2 areas are intended to focus the attention of external stakeholders on what we believe to be the best (key) indicators of progress toward our long-term strategic goals. The Executive Summary (§1) is intended to effectively and concisely communicate the status of the project toward delivery of the mission and realization of the vision by reaching three strategic goals. Stoplight indicators (§1.1) are used to visually provide a quick understanding of our assessment of overall project progress with respect to the strategic goals in light of our KPIs. The Science, Engineering, & Program Highlights (§2) provide a small selection in a series of scientific and engineering research and education successes XSEDE has enabled. These successes are an ongoing testament to the importance of our services to the research community. The Discussion of Strategic Goals and Key Performance Indicators (§3) provides the next level of detail in understanding project progress. It decomposes the strategic goals into subgoals and discusses progress toward each of the sub-goals using KPIs that, where possible, represent measures of impact to the communities XSEDE supports. These first three sections take a project-wide view. A more detailed analysis of progress from the view of the areas responsible for supporting each of the sub-goals—and contributing toward the KPIs associated with those sub-goals—is provided by looking at each of the areas of the project in the remaining sections (§4, §5, §6, §7, §8, §9). These sections also contain area highlights and Key Performance Indicators (KPIs) that are deemed important, along with links to corresponding sections of the XSEDE KPIs & Metrics wiki page. The metric tables on the wiki page contain definitions, descriptions, and collection methodology information about each metric in the tables. When a new metric is added, table cells from previous reporting periods will contain an asterisk (*) to designate that data for that metric was not being collected or reported at that time. Prior to the RY2 Annual Report we have reported on metrics at the Work Breakdown Structure Level 3. Based on feedback from reviewers and discussion with the Cognizant Program Officer, we have discontinued this to reduce duplication and improve readability of the document.

RY5 IPR12 Page vii

Note that XSEDE Project Years (PY) run September-August and Report Years (RY) run May-April. The table below lists the schedule for Reporting Periods (RP) within each RY, including RY1 and RY5 which are slightly altered due to a shorter RY1 and a longer RY5. Also note that, in light of the awarding of the supplemental request to extend the XSEDE 2.0 award for 12 months, a revised version of this table will be provided in subsequent reports.

RP4 (Included in RP1 RP2 RP3 Annual Report) Typical Schedule May 1- July 31 Aug. 1- Oct. 31 Nov. 1- Jan. 31 Feb. 1- Apr. 30

RY1 Period doesn’t exist Sept. 1- Oct. 31, Nov. 1, 2016- Jan. Feb. 1- Apr. 30, 2017 due to shortened 2016 (Abbreviated 31, 2017 (Included in Annual Sept. 2016-April first year. due to shortened Report) 2017 year)

RY2 May 1- July 31, 2017 Aug. 1- Oct. 31, 2017 Nov. 1, 2017- Jan. Feb. 1- Apr. 30, 2018 31, 2018 (Included in Annual May 2017-April Report) 2018

RY3 May 1- July 31, 2018 Aug. 1- Oct. 31, 2018 Nov. 1, 2018- Jan. Feb. 1- Apr. 30, 2019 31, 2019 (Included in Annual May 2018-April Report) 2019

RY4 May 1- July 31, 2019 Aug. 1- Oct. 31, 2019 Nov. 1, 2019- Jan. Feb. 1- Apr. 30, 2020 31, 2020 (Included in Annual May 2019-April Report) 2020

RY5 May 1- July 31, 2020 Aug. 1- Oct. 31, 2020 Nov. 1, 2020- Jan. Feb. 1- Aug. 31, 31, 2021 2021 (Longer May 2020-Aug. period included in 2021 Final Report)

It is anticipated that this report is read in electronic form (PDF) using Adobe Reader®. There is extensive cross-linking to facilitate referencing content across the document. In general, all text that has blue underlining (e.g., §2) is clickable. Clicking on the underlined text will take you to the referenced section. These are set up to facilitate moving back and forth between the high level discussions in §1 and §3, to more detailed discussions regarding specific project areas and financial information in §4, §5, §6, §7, §8, §9. As noted, this represents an ongoing effort at improvement, and we welcome comments on how to improve any and all aspects of our reporting process. At the beginning of the XSEDE project, a Project Execution Plan (PEP) was submitted to and approved by NSF. Content in this and all preceding Interim Project and Annual Reports supersede information submitted in the original PEP. The most recent version of the PEP can be viewed in Appendix 18 of this report and on the wiki.1 Changes in Version 2.4 of the PEP include:

1 https://confluence.xsede.org/display/XT/XSEDE+Project+Execution+Plan

RY5 IPR12 Page viii

• Addition of link to web page listing subaward partners. (p. 5) • Change to project end date to August 31, 2022 and update to project schedule per approval by NSF of XSEDE supplement year. (p. 33) (PCRPROJ-112) • Addition of Deputy Project Director role in the following sections: I.1 Project Governance (p. 34), I.2 XSEDE Senior Management Team (p. 34), K Description of the financial and business controls to be used. (p. 37) (PCRPROJ-103) • Changes in Senior Management Team members: Marinshaw, Froeschl, Chadduck. (p. 34) • Addition of link to XSEDE Metrics Dashboard (p. 39)

RY5 IPR12 Page ix

1. Executive Summary Computing across all fields of scholarship is becoming ubiquitous. Digital technologies underpin, accelerate, and enable new, even transformational, research in all domains. Researchers continue to integrate an increasingly diverse set of distributed resources and instruments directly into their research and educational pursuits. Access to an array of integrated and well-supported high-end digital services is critical for the advancement of knowledge. XSEDE (Extreme Science and Engineering Discovery Environment) is a socio-technical platform that integrates and coordinates advanced digital services within the national ecosystem to support contemporary science. This ecosystem involves a highly distributed, yet integrated and coordinated, assemblage of software, supercomputers, visualization systems, storage systems, networks, portals, gateways, collections of data, instruments, and personnel with specific expertise. XSEDE supports the need for an advanced digital services ecosystem distributed beyond the scope of a single institution and provides a long-term platform to empower modern science and engineering research and education. As a significant contributor to this ecosystem, driven by the needs of the open research community, XSEDE substantially enhances the productivity of a growing community of scholars, researchers, and engineers. XSEDE federates with other high-end facilities and campus-based resources, serving as the foundation for a national e-science infrastructure with tremendous potential for enabling new advancements in research and education. Our vision is a world of digitally-enabled scholars, researchers, and engineers participating in multidisciplinary collaborations while seamlessly accessing computing resources and sharing data to tackle society’s grand challenges. Researchers use advanced digital resources and services every day to expand their understanding of our world. More pointedly, research now requires more than just supercomputers, and XSEDE represents a step toward a more comprehensive and cohesive set of advanced digital services through our mission: to substantially enhance the productivity of a growing community of scholars, researchers, and engineers through access to advanced digital services that support open research; and to coordinate and add significant value to the leading cyberinfrastructure resources funded by the NSF and other agencies. XSEDE has developed its strategic goals in a manner consistent with NSF’s strategic plan, Building the Future: Investing in Discovery and Innovation - NSF Strategic Plan for Fiscal Years (FY) 2018 - 20222, NSF’s strategies stated broadly in the Cyberinfrastructure Framework for 21st Century Science and Engineering3 vision document, and the more specifically relevant Advanced Computing Infrastructure: Vision and Strategic Plan4 document. Though the latter two documents are now out of date for the NSF, in the absence of documents that supplant them, XSEDE continues to use them for general guidance until such time as successor documents are released. It should be noted here that three draft documents under the collective heading of Transforming Science Through Cyberinfrastructure: NSF’s Blueprint for a National Cyberinfrastructure Ecosystem for Science and Engineering in the 21st Century, distributed by NSF’s Office Director for the Office of Advanced Cyberinfrastructure, are currently available for public comment5.

2 https://www.nsf.gov/pubs/2018/nsf18045/nsf18045.pdf 3 http://www.nsf.gov/cise/aci/cif21/CIF21Vision2012current.pdf 4 http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf12051 5 https://www.nsf.gov/cise/oac/vision/blueprint-2019/

RY5 IPR12 Page 1 Strategic Goals To support our mission and to guide the project’s activities toward the realization of our vision, three strategic goals are defined: Deepen and Extend Use: XSEDE will deepen the use—make more effective use—of the advanced digital services ecosystem by existing scholars, researchers, and engineers, and extend the use to new communities. We will contribute to preparation—workforce development—of the current and next generation of scholars, researchers, and engineers in the use of advanced digital services via training, education, and outreach; and we will raise the general awareness of the value of advanced digital services. Advance the Ecosystem: Exploiting its internal efforts and drawing on those of others, XSEDE will advance the broader ecosystem of advanced digital services by creating an open and evolving e- infrastructure, and by enhancing the array of technical expertise and support services offered. Sustain the Ecosystem: XSEDE will sustain the advanced digital services ecosystem by ensuring and maintaining a reliable, efficient, and secure infrastructure, and providing excellent user support services. XSEDE will further operate an effective, productive, and innovative virtual organization. The strategic goals of XSEDE cover a considerable scope. To ensure XSEDE is delivering on its mission and to assess progress toward its vision, XSEDE has identified key metrics to measure its progress toward meeting sub-goals of each of the strategic goals. These Key Performance Indicators (KPIs) are a high-level encapsulation of XSEDE’s project metrics that measure how well each sub-goal is met. Planning is driven by XSEDE’s vision, mission, goals, and these metrics—which are in turn rooted in the needs and requirements of the communities served. The key concept is not that the KPIs themselves must have a direct causal effect on eventual outcomes, or measure eventual outcomes or long-term impacts, but rather that the KPIs are chosen so that actions and decisions which move the metrics in the desired direction also move the organization in the direction of the desired outcomes and goals. Table 1-1 below shows the project’s progress toward the three strategic goals and associated sub-goals in RY4. Status icons are used in the table as follows: A green status is defined as a strategic goal for which at least 90% of the targets for all KPIs are met. A yellow status is defined as a strategic goal within which at least 60% of the targets for all KPIs are met. A red status is a strategic goal with less than 60% of the KPI targets met.

A white status indicates there are currently no metrics tracked for this sub-goal or there is not complete data for any of the metrics tracked. Multiple indicators represent a strategic goal that has sub-goals for which there is incomplete data or that have metrics not currently tracked. In these cases, the second indicator is a qualitative assessment of the status provided in lieu of sufficient data or a formal metric being in place.

RY5 IPR12 Page 2 Table 1-1: Summary of Key Performance Indicators (KPIs) for XSEDE.

Status Sub-goals KPIs

Strategic Goal: Deepen and Extend Use (§3.1)

Number of sustained users of XSEDE resources and services via the portal Deepen use (existing ● Number of sustained underrepresented individuals using XSEDE resources and services via the portal communities) (§3.1.1) ● Percentage of sustained allocation users from non-traditional disciplines of XSEDE resources and service ●

Number of new users of XSEDE resources and services via the portal Number of new underrepresented individuals using XSEDE resources Extend use (new ● and services via the portal communities) (§3.1.2) ● Percentage of new allocation users from non-traditional disciplines of XSEDE resources and services ●

Prepare the current and Number of participant hours of live training delivered by XSEDE next generation (§3.1.3) ● Raise awareness of the Aggregate mean rating of user awareness of XSEDE resources and value of advanced digital services services (§3.1.4) ● Percent increase in social media impressions over time ● Strategic Goal: Advance the Ecosystem (§3.2)

Create an open and evolving e-infrastructure Total number of capabilities in production (§3.2.1) ●

Enhance the array of Aggregate mean rating of user satisfaction with XSEDE technical technical expertise and support services support services (§3.2.2) ●

Strategic Goal: Sustain the Ecosystem (§3.3)

Provide reliable, efficient, and secure infrastructure Mean composite availability of core services (%) (§3.3.1) ● Mean time to ticket resolution (hours) Provide excellent user Aggregate mean rating of user satisfaction with allocations process ● support (§3.3.2) and support services ● Percentage of research requests successful (not rejected)

● Mean rating of importance of XSEDE resources and services to Operate an effective and researcher productivity productive virtual ● Percentage of users who indicate the use of XSEDE-managed and/or organization (§3.3.3) XSEDE-associated resources in the creation of their work product ●

Operate an innovative Percentage of Project Improvement Fund funded projects resulting in virtual organization innovations in the XSEDE organization (§3.3.4) ● Mean rating of innovation within the organization by XSEDE staff ●

RY5 IPR12 Page 3 In grasping the scope of XSEDE’s activities, it is perhaps useful to liken XSEDE’s integrated set of efforts to that of a physical center supporting advanced digital services and research computing. In this case, research computing resources are operated and funded by separate NSF awards to Service Providers (SPs). XSEDE provides central services to complement and integrate those SP awards and resources into a common ecosystem and provides these services more efficiently and cost-effectively for NSF and the user community. XSEDE coordinates and supports education and outreach (CEE), front-line and advanced user support (CEE and ECSS), allocations review and processing (RAS), operation of shared enterprise services (Operations), and well-designed strategies for resource integration (XCI), all guided by common practices, communication, and administration (Program Office). XSEDE activities are logically organized across L2 (Work Breakdown Structure Level 2) areas to eliminate duplication of effort and allow effective management and focus; however, as with a physical center, each area depends on and supports the others in successfully completing XSEDE's mission and goals. And like a physical center, all parts of XSEDE are bound into a coherent and more effective whole by a shared set of values of the XSEDE project that bring synergy to the project: • A focus on people: people are the most valuable resource in the pursuit of knowledge. XSEDE creates a strong sense of coherent community, connecting the research community and the technical expertise they partner with from XSEDE to harness a set of coherent services and infrastructure. • Raising awareness of the value of advanced digital services: given that effective integration and coordination of advanced digital services within the national ecosystem to support contemporary science by nature makes that integration and coordination less visible, it is precisely the rich set of less visible cross-connections among the activities of XSEDE that make it much more than the sum of its parts and much more effective in the support of science and education. • Integrating and helping expand effective use of the national cyberinfrastructure across disciplines and across campuses: XSEDE is driven to bring the capabilities of the advanced digital services ecosystem to scholars, researchers, and engineers across all domains. XSEDE aids academic institutions across the US by providing resources and enabling more effective use and integration of local resources as part of the national cyberinfrastructure, better leveraging local investments in CI. • A significant, persistent, secure, and reliable environment for conduct of research: the highly integrated nature of XSEDE enables a coherent approach to creating a trusted and protected environment that provides a long-term platform to empower modern science and engineering research and education. The long-term lifespan of XSEDE means that researchers can plan activities and count on years of use of resources and support from XSEDE. • Accountability and transparency: though XSEDE has organized around its strategic goals that provide primary focus for the organizational units of the project, it is only through the unified management of the project that full accountability and transparency can be realized given the underlying cross-dependencies and synergies of the organizational units. Shared values are a social construct that XSEDE has developed over many years working with all its stakeholders. These values, in conjunction with the effective cross-organizational management techniques developed by XSEDE, enables XSEDE to not only articulate a vision of the future, but to demonstrate the trusted and dependable leadership necessary to make that vision a reality for the community. Summary & Project Highlights The XSEDE project continues to provide significant value to the national research community by enabling high-impact scientific advances across a broad range of disciplines. In XSEDE’s continuing documentation of science success, a few key examples of efforts that have been enabled by XSEDE in

RY5 IPR12 Page 4 conjunction with our Service Provider (SP) partners have been selected. These are highlighted in §2 of this report and span a range of domains including: noise generated from neutron star collisions forming black holes, how DNA repairs itself, solar wind predictions, new materials that could make LED lighting brighter and more affordable, predictions of neutron star mergers, development of antimicrobial agents to address human diseases, simulation of solar cells, predictions of protein structures, drug screening for heart arrhythmias, high tech materials study, and electrochemical reaction research. A continually updated collection of these successes is documented on the XSEDE website (see: https://www.xsede.org/science-successes). Adding value and expanding the value of NSF’s investment in advanced computing for the research community, nationally and internationally, is the focus of the portion of the XSEDE mission statement which reads, “coordinate and add significant value to the leading cyberinfrastructure resources funded by the NSF and other agencies.” This reporting period includes many examples of the project’s efforts to advance this mission. Project Advancements The most notable update on the project’s advancements is that after a successful NSF Panel Review in June, the XSEDE project has received NSF’s “authorization to spend” for award project year 5 (XSEDE Project Year 10). Funding amendments for the XSEDE subaward partners are in process for the final award year. Over the last three months, significant HPC system compromises affected major European Union (EU) HPC centers. The Cybersecurity group (SecOps), within Operations, has maintained close communications with security groups within these EU groups, centered mainly through contacts with Organisation Européenne pour la Recherche Nucléaire (CERN), Worldwide LHC Computing Grid (WLCG), and Wise Information Security for E-infrastructure (WISE). XSEDE’s Trusted Incident Response group, led by SecOps co-lead Derek Simmel, is the primary route of collaboration with these organizations and communities. Because of this, XSEDE was quickly informed of these attacks and was able to take preventative measures before any security incidents occurred. Such aspects of our collaborative outreach efforts have proven to be invaluable. During this reporting period, the RAS team completed the transition to a new Fields of Science hierarchy. This effort allowed XSEDE to replace a more than 20-year-old categorization (based on a long-past NSF directorate, division, and program structure) with a structure based on a standard from the Organisation of Economic Co-operation and Development (OECD). This change will be reflected in several XSEDE metrics starting with the next reporting period. New allocation requests aligned with the new fields of science have already started coming in. Community Involvement CEE-led virtual instances of the Advanced Computing for Social Change (ACSC) workshops for faculty and students were offered in July. The one-week faculty workshop featured a mixed schedule of training, small group working sessions, and hands-on practice using tools on the desktop and in the Jetstream Cloud environment. Twenty-two courses were updated with social justice data science exercises; nineteen courses will be offered this fall and three courses will be offered in spring 2021. A general education data science course is being developed for the Atlanta University Center Data Science Initiative and will be offered to all 8,000 undergraduate students at Clark Atlanta University, Morehouse College, and Spelman College. XSEDE partially supported the four-week data science summer immersion program for SPICE (Supporting Pacific Islanders Computing Excellence). Twenty Native Hawaiian and Pacific Islander students from the Islands of Hawaii, American Samoa, and Micronesia conducted data-driven research in topics ranging from COVID-19, environmental science, healthcare and health informatics, and social science. The immersion program was conducted online and students were trained by XSEDE staff to use

RY5 IPR12 Page 5 XSEDE resources for their research. As a result of this program, Chaminade University of Honolulu has decided to pursue becoming a Campus Champions member and will leverage the use of XSEDE resources and services for high-end computing needs rather than investing in local campus infrastructure. Attendance at monthly ECSS symposia grew significantly over the past several months and now routinely tops 100 participants. At this rate, it’s likely that ECSS will triple the annual goal of 300 participants/year. This increase in growth is due in part to successful, ongoing collaborations between ECSS and Rudi Eigenmann’s (University of Delaware) NSF-funded Xpert Network project, which has the goal of “Exchanging Best Practices and Tools for Computational and Data-Intensive Research,” CaRCC (Campus Research Computing Consortium), and Campus Champions. Continuing its efforts to engage directly with the advanced research community, the Data Transfer Services (DTS) group reached out to Campus Champions leadership to investigate how to effectively interact with the members and, by extension, the universities and institutions they represent. The DTS group plans to offer one-on-one consultation services to Campus Champions similar to services offered to the SP forum members. Also, as part of DTS outreach efforts, 29 of the 580 respondents from the data transfer section of the recent XSEDE annual user survey indicated they would like a follow-up on problems they have experienced. In response, DTS staff will begin reaching out to these users and begin one-on-one consultations this next reporting period. A summary of the overall user consultation effort will be included in a future IPR. PEARC20 Contributions PEARC20 continues to be a primary event for the XSEDE project. As in past years, the XSEDE Evaluation team provided the evaluation services for the conference. A significant number of XSEDE staff members attended PEARC20, and several staff also served on the conference committees and chaired sessions during the conference. Associated with XSEDE, an early count indicates that over 250 Campus Champions participated in PEARC20. There were many individual contributions by XSEDE staff to the conference, including: • Tutorial: An Introduction to Advanced Features in MPI (CEE) • Tutorial: Customizing OpenHPC: Integrating Additional Software and Provisioning New Services including Open OnDemand (XCI) • Workshop: Strategies for Enhancing HPC Education and Training (CEE) • Workshop: What Does it Mean to be a Campus Champion? (CEE) • Poster and Paper: Use Case Methodology in XSEDE System Integration (XCI) • Poster and Paper: Secure XSEDE Information APIs (XCI) • Poster and Paper: SciTokens SSH: Token-based Authentication for Remote Login to Scientific Computing Environments (XCI & Ops) • Paper: Best Practices in Project Management in a Large, Distributed Organization: Lessons Learned from XSEDE (Program Office) • Paper: Using Containers to Create More Interactive Online Training and Education Materials (CEE) • Paper: Implementing a Prototype System for 3D Reconstruction of Compressible Flow (CEE) • Panel: Towards a National Cyberinfrastructure Ecosystem from Campus to National Facilities (XCI)

RY5 IPR12 Page 6 • BoF: Bridging the Data Transfer Gap: An Open Discussion between Researchers, Administrators, and Network Engineers (Ops) • BoF: User Training and Engagement in Scientific Computing (CEE) COVID-19 Contributions XSEDE continues to play a key role in the nation’s (and the world’s) response to the COVID-19 pandemic. COVID-19 HPC Consortium:6 The Consortium continues to provide support to research addressing the COVID-19 pandemic. PI Towns and RAS Allocations Process & Policies (APP) L3 Manager Ken Hackworth continue in their respective roles as previously described in the XSEDE RY4 Annual Report. During this Reporting Period the number of allocations made via the Consortium has fallen off, but requests continue to be submitted. It is also suspected that as allocations made early in the process expire, new/supplemental requests will be submitted by the PIs of a number of those allocations. This has also induced a discussion of how the Consortium will continue to operate in the coming months. Though decisions have not yet been made, this is anticipated in the next month or two. Details are provided in a series of weekly reports submitted by Towns to NSF aggregating much of the information regarding not only efforts related to the Consortium, but also allocations made by other means. These should all be available in the record of the project at NSF and will not be repeated here. In total, as of August 1, 2020, the Consortium has received 158 requests through XRAS (80 new since the XSEDE Annual Report for Report Year 4 and Program Plan for Project Year 107 was delivered reporting on this through May 2, 2020). Of these, 83 (41 new since May 2, 2020) projects were provided access to resources with 29 (16 new since May 2, 2020) of these being allocated on NSF resources. A complete list of active projects provided access via the Consortium is available on the Consortium’s Active Projects page. All projects allocated via the Consortium on resources typically allocated via XSEDE are also offered ECSS support to expedite their work. CEE’s Campus Engagement team has worked to pivot their events to a virtual format due to the COVID- 19 pandemic. Campus Champions, Virtual Residents, and the CaRCC People Network shared with each other how they were changing their practices to support computational and data-intensive research remotely. Many reported an increase in demand for support as researchers moved to more digital research when they were unable to be in their labs. ECSS’s Extended Support for Science Gateways (ESSGW) team supported work to enhance the Distant Reader Gateway (https://distantreader.org/) with additional datasets focused on the COVID-19 research literature. This enhancement will allow researchers to scan and draw understanding from the vast amount of published COVID research happening quickly. More details on this effort can be found in §5.5. ECSS consultant Roberto Gomez (PSC) worked with Olexandr Isayev (CMU) on his COVID-19 HPC Consortium project "Assisting SARS-CoV-2 computational drug discovery efforts with artificial intelligence (AI) and AI-accelerated Quantum Mechanics" which has produced a database of molecules to help develop drugs to combat COVID-19, published in https://covid.molssi.org/therapeutics/#res_therapeutics. It has datasets containing low-energy conformers and tautomers for 20,306 compounds from the CAS Antiviral database, and for 6,433 drugs approved and under investigation by the FDA. After extensive sampling using Bridges-AI and Bridges- GPU, for every tautomer the researchers found low laying conformers within a ~6 kcal/mol window using the ANI-2x neural-network molecular potential, which is approaching the accuracy of high-level QM calculations.

6 https://covid19-hpc-consortium.org/ 7 https://www.ideals.illinois.edu/handle/2142/107285 RY5 IPR12 Page 7 As mentioned above, RAS has continued ongoing support of the COVID-19 HPC Consortium, which has transitioned from the original twice-daily meetings (one review and matching meeting per day) to Mon/Wed review meetings and Tue/Thu matching meetings. As demonstrated in Table 12-24 in the metrics Appendix, there have been 78 projects (representing 3% of currently open projects) approved via the Consortium “rapid review” process. The associated Active and %NUs reflect only Consortium projects on XSEDE resources; many consortium projects are on non-XSEDE systems.

RY5 IPR12 Page 8 2. Science, Engineering, & Program Highlights This section provides a select set of science and engineering highlights as well as program highlights from the community of researchers with whom we collaborate. These are drawn from the most recent reporting period (RP1 of RY5). A complete collection of highlights can be found at: https://www.xsede.org/science-successes. Slower and Noisier It seems strange to talk about "quiet" versus "noisy" collisions of neutron stars. But many such impacts form a black hole that swallows all but the gravitational evidence. A series of simulations using XSEDE- allocated supercomputers and other systems by a Penn State scientist suggested that, when the neutron stars' masses are different enough, the result is far noisier. The model predicts an electromagnetic "bang," which isn't present when the merging stars' masses are similar, that astronomers should be able to detect. When two objects roughly the mass of the sun and the size of cities slam together, it seems strange to talk about how "quiet" it is. But for many neutron-star collisions, it is quiet, at least in terms of radiation we can detect. A strong surge of gravitational waves emerges from the impact—now being sensed by gravity-wave detectors such as LIGO, in Hanford, Washington, and Livingston, Louisiana; and Virgo, in Cascina, near Pisa. But precious little else appears. That's because the incredibly dense collapsed stars combine to form a black hole, which swallows any of the radiation that could have come out of the merger. But that's not the only way it can play out. After reporting the first detection of a neutron-star merger in 2017, the LIGO team reported in 2019 the second, which they named GW190425. The first of the two collisions was what astronomers expected, with a total mass of about 2.7 times the mass of our Sun and each of the two neutron stars nearly equal in mass. But GW190425 was heavier, with a combined mass of around 3.5 Solar masses. More importantly, the ratio of the masses of the two participants was more unequal, possibly as high as 2 to 1. That may not seem like such a huge difference. But neutron stars can exist only in a narrow range of masses between about 1.2 and 3 times the mass of our Sun. Lighter stellar remnants don't collapse to form neutron stars and form white dwarfs instead. Heavier objects collapse directly to form black holes. When the difference between the merging stars gets as large as in GW190425, scientists suspected that the merger could be messier—and louder in electromagnetic radiation. Astronomers had detected no such signal from GW190425's location. But coverage of that area of the sky by conventional telescopes that day wasn't good enough to rule it out. David Radice of Penn State, working as member of CoRe, the Computational Relativity International Collaboration, which includes scientists in the U.S., Germany, Italy and Brazil, wanted to better understand the phenomenon of unequal neutron stars colliding, and to predict signatures of such collisions that astronomers could look for. He turned to simulations on a number of supercomputers, but the ones most useful to these simulations were the XSEDE-allocated systems Comet at the San Diego Supercomputer Center (SDSC) and Bridges at the Pittsburgh Supercomputing Center (PSC). To run his simulations, Radice needed an unusual combination of computing speed, large memory, and flexibility in moving data between memory and computation. That's partly because scientists know so little about these mergers for certain. To test their ideas required running about 20 simulations, each of which needed 500 compute cores for several weeks.

RY5 IPR12 Page 9 Radice employed a number of systems for this work, including XSEDE-allocated Stampede2 at the Texas Advanced Computing Center and the XSEDE SP-2 resource Blue Waters at the National Center for Supercomputing Applications. The computations did not disappoint the scientists' expectations of an electromagnetic bang. As the two simulated neutron stars spiraled in toward each other, the gravity of the larger star tore its partner apart. That meant that the smaller neutron star didn't hit its more massive companion all at once (see Figure 1). The initial dump of the smaller star's matter turned the larger into a black hole. But the rest Figure 1: A neutron star is ripped apart by tidal forces from its massive companion in an unequal-mass binary neutron star of its matter was too far away for the black hole merger. to capture immediately. Instead, the slower rain of matter into the black hole created a flash of electromagnetic radiation. The group reported their results in the journal Monthly Notices of the Royal Astronomical Society. Their hope is that the simulated signature they found can be used by astronomers using a combination of gravity-wave and conventional telescopes to detect the paired signals that would herald the breakup of a smaller neutron star merging with a larger one. You can read their report here.8 This work was supported by EU H2020 ERC Starting Grant no. BinGraSp-714626. Numerical relativity simulations were performed on the supercomputer SuperMUC at the LRZ Munich (Gauss project pn56zo), on supercomputer Marconi at CINECA (ISCRA-B project number HP10BMHFQQ); on the supercomputers Bridges, Comet, and Stampede2 (NSF XSEDE allocation TG-PHY160025); on NSF/NCSA BlueWaters (NSF AWD-1811236); on ARA cluster at Jena FSU. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Data postprocessing was performed on the Virgo "Tullio" server at Torino supported by INFN. The Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) also funded the project by providing computing time on the GCS Supercomputer SuperMUC at Leibniz Supercomputing Centre (www.lrz.de). Supercomputer Simulations Show How DNA Prepares itself for Repair It's common knowledge that the human body consists of trillions of cells. The cell nucleus, which houses DNA, is under attack every second of every day by environmental and behavioral factors. Now researchers from Harvard University and the University of Texas Medical Branch at Galveston (UTMB) have used detailed simulations that sheds light on how DNA prepares itself for repair.

8 https://doi.org/10.1093/mnras/staa1860 RY5 IPR12 Page 10 Published in the Physical Review E9 journal, the study started with two DNA strands soaked in saltwater solution. The scientists watched as the DNA strands twisted and moved toward one another, due to water molecules and nearby sodium ions. The team used allocations via XSEDE on the Comet supercomputer at the San Diego Supercomputer Center (SDSC) as well as Bridges at the Pittsburgh Supercomputing Center and Stampede2 at the Texas Advanced Computing Center. This work was unique in that the research team used these supercomputers to show specifically how the molecules interact with one another in order to line up before repair. Figure 2: Several XSEDE resources were used to generate this plot of free energy to bring two parallel DNA double helices While previous experiments primarily together as a function of distance and rotation of one around the focused on potential mechanisms to other. This illustration helps researchers better understand the demonstrate the interactions of larger DNA mechanism strands, these calculations used shorter models to get a detailed look at the actual process. "The interactions or free energies found in our study were proven to be caused by certain structures in the surrounding water molecules and sodium ions," said Monte Pettitt, who directs the Sealy Center for Structural Biology and Molecular Biophysics at UTMB. Pettitt, also the Robert A. Welch Distinguished University Chair in Chemistry, further explained that the charges and electrostatics in solution of these molecules allowed them to interact with the phosphates within the DNA. Before being placed in the solution, the DNA strands repelled one another. Once placed in the solution, however, the free energy allowed the molecules to sometimes locally attract one another in a geometrically and sequence-specific way. Having access to XSEDE-allocated resources to conduct highly detailed simulations allowed the researchers to calculate the exact amount of free energy needed for these two DNA double helices to interact with one another (see Figure 2). "Our simulations showed the free energy available relative to the specific rotation angle of the DNA strands as well as the distance between the strands," said Pettitt. "This now helps researchers to understand how DNA interacts with itself in order to condense or separate during certain cellular processes." More specifically, the study also showed how the local attraction of DNA to itself only occurs for certain geometries, which contributes to an important fundamental discovery of how DNA is aligned before it is repaired. DNA strands within a cell are not protected from environmental factors such as UV radiation and carcinogens. This makes the DNA vulnerable in the sense that it can be damaged in a spot where repair enzymes will need a roadmap to repair it—that is, another piece of DNA with the same sequence that is geometrically lined up as a template for repair. "We still need more work to understand the sequence specificity," noted Pettitt. "The calculations to illustrate these concepts would not have been possible without XSEDE and we are appreciative of the ability these supercomputers gave us to achieve our goal of furthering DNA research."

9 https://journals.aps.org/pre/abstract/10.1103/PhysRevE.101.032414 RY5 IPR12 Page 11 This research was supported in part by The National Institutes of Health (NIH) (GM066813) and the Robert A. Welch Foundation. Computational time was allocated by the National Science Foundation's Extreme Science and Engineering Discovery Environment (TG-MCA93S001). XSEDE allocation Facilities Expanded Solar Wind Predictions While space weather can produce dancing lights here on Earth, such as the beautiful Aurora borealis and Aurora australis that sometimes streak across the northernmost and southernmost skies, geomagnetic storms can cause severe damage to power grids, satellites, and many other electrical systems. Heliophysicist Bala Poduval has dedicated her space weather research to predicting the solar wind conditions that cause these storms, with a recent highlight being the validation of a machine learning model that she and her colleagues developed for solar wind prediction in which simulated data using the Comet supercomputer at the San Diego Supercomputer Center (SDSC) played a pivotal role. "To mitigate the adverse effects of space weather, it is necessary to forecast these events with sufficient lead time so that appropriate safety precautions can be taken," said Poduval, a research scientist with the University of New Hampshire (see Figure 3). "One of the important pieces of information we needed to improve our prediction model was the ambient solar wind velocity near the Earth." Upon receiving an allocation from the National Science Foundation's Extreme Science and Engineering Discovery Environment (XSEDE), Poduval and her international colleagues utilized Comet to validate a model that predicts time-lagged effects of the solar wind and the dependence on the wind's velocity. To accomplish this, they used a machine learning technique called Dynamic Time Lag Regression (DTLR) and validated their work by predicting the solar wind arrival near the Earth's orbit from physical parameters of the Sun as measured from the ground and space. "We used the changes in the solar magnetic field, in terms of the magnetic flux tube expansion factor, as well as other characteristics of the sun such as the number of sunspots, to determine the speed of the solar wind that can be detected at Earth after a lag of ‘dynamic time'," explained Poduval. "The validation of DTLR and the solar wind forecasting would not have been possible in any reasonable time frame without XSEDE high performance computing resources since the computation of magnetic flux expansion factor for a period sufficient to train a neural network would take months on a local machine as compared to just a few weeks with Comet." The code for this research was written in Interactive Data Language (IDL) and takes at least five hours to run a single model on a personal computer. Scaling up Poduval's work required over one thousand simulations, which would have taken over 5000 hours on a personal computer. "We not only saved computational time thanks to XSEDE, but were also able to optimize our code thanks to SDSC staff," said Poduval. "Further, we were able to complete two papers thanks to XSEDE and I am really thankful for this amazing service available for our academic research community." Figure 3: These comparisons of predicted and actual hourly solar wind forecasts, for a period spanning Poduval and her colleagues' work was recently approximately one month (11/16/16-12/14/16), presented at the International Conference on were generated using data computed on Comet at the San Diego Supercomputer Center. Credit: Mandar 10 Learning Representations , which was held in a Chandorkar, Cyril Furtlehner, Bala Poduval, Enrico virtual format due to the COVID-19 pandemic. Camporeale, and Michele Sebag.

10 https://iclr.cc/virtual_2020/poster_SkxybANtDB.html RY5 IPR12 Page 12 Poduval is now exploring methods of artificial intelligence for substantially improving the accuracy of prediction, and with a recent National Science Foundation award, she is developing a neural network model for predicting solar energetic particle events. This research was funded by a grant from the Centrum Wiskunde and Informatica, Amsterdam. Computational work on Comet was allocated by the National Science Foundation's XSEDE (TG- ATM170013) Initiative. LED’s Bright Early Light LED lamps are lighting up the world more and more. Global LED sales in residential lighting have risen from five percent of the market in 2013 to 40 percent in 2018, according to the International Energy Agency11, and other sectors mirror these trends. An unmatched energy efficiency and sturdiness have made LED lights popular with consumers. Scientists are currently using supercomputers to gain insight on the crystal structure of new materials that could make LED lighting even brighter and more affordable (see Figure 4). New properties have been found in a promising LED material for next-generation solid-state lighting. A January 202012 study in the chemistry journal ACS Omega revealed evidence pointing to a brighter future for cubic III-nitrides in photonic and electronic devices. "The main finding is that next generation LEDs can, should, and will do better," said study co-author Can Bayram, an assistant professor of electrical and computer engineering at the University of Illinois at Urbana-Champaign. His motivation for studying cubic III-nitrides stems from the fact that today's LED loses much of its efficiency under high injection conditions of current passing through the device, necessary for general lighting. Bayram's lab builds newly discovered crystals atom by atom in real life as well as in their simulations so that they can correlate experiments with theory. "We need new materials that are scalable to be used for next generation lighting," Bayram said. "Searching for such materials in a timely and precise manner requires immense computational power." "In this study we are exploring the fundamental properties of cubic-phase aluminium gallium indium nitride materials" Bayram said. "To date, indium gallium nitride-based green LED research has been restricted to naturally-occurring hexagonal-phase devices. Yet they are limited in power, efficiency, speed, and bandwidth, particularly when emitting the green color. This problem fueled our research. We found that cubic phase materials reduce the necessary indium content for the green color emission by ten percent because of a lower bandgap. Also, they quadruple radiative recombination dynamics by virtue of their zero polarization." study co-author and graduate student Yi-Chia Tsai added. Bayram explained that it's challenging to model compound semiconductors such as gallium nitride because they are compound, unlike elemental semiconductors such as silicon or germanium. Modeling alloys of the compound semiconductors, such as aluminum gallium nitride, are further challenging because, as the saying goes, it's all about location, location, location. Relative atomic positions matter. "In a unit cell sketch of a crystallography class, Al and Ga atoms are interchangeable but not so in our computational research," Bayram explained. That's because each atom and its relative position matter when you are simulating the unit cell, a small volume of the entire semiconductor material. "We simulate the unit cell to save computational resources and use proper boundary conditions to infer the entire material properties. Thus, we had to simulate all possible unit cell combinations and infer

11 https://www.iea.org/fuels-and-technologies/lighting 12 https://pubs.acs.org/doi/10.1021/acsomega.9b03353 RY5 IPR12 Page 13 accordingly — this approach gave the best computational matching to the experimental ones," Bayram said. Using this approach, they further explored new though not experimentally-realized materials. To overcome the computational challenges, Bayram and Tsai applied for and were awarded supercomputer allocations on the Stampede2 and Ranch systems at the Texas Advanced Computing Center. "XSEDE is a unique resource. We primarily use XSEDE hardware to enable material computations. First, I want to stress that XSEDE is an enabler. Without XSEDE, we could not perform this research. We started with Startup then Research allocation grants. XSEDE—over the last two years—provided us with Research allocations valued at nearly $20,000 as well. Once implemented, the outcome of our research will save billions of dollars annually in energy savings alone," Bayram said. The next stage in Bayram's research is to understand how impurities impact new materials and to explore how to dope the new material effectively. Through searching the most promising periodic table groups, they're looking for the best elemental dopants, which will eventually help the experimental realization of devices immensely. Said Bayram: "Supercomputers are super-multipliers. They super-multiply fundamental research into mainstream industry. One measure of success comes when the research outcome promises a unique solution. A one-time investment of $20K into our computational quest will at least lead to $6 billion in savings annually. If not, meaning that the research outcome eliminates this material for further investigation, this early investment will help the industry save millions of dollars and research-hours. Our initial findings are quite promising, and regardless of the outcome the research will ultimately benefit society." The study, "Band Alignments of Ternary Wurtzite and Zincblende III-Nitrides Investigated by Hybrid Density Functional Theory," was published in the journal ACS Omega on January 30, 2020. The study co-authors are Yi-Chia Tsai and Can Bayram, Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign. This work is supported by the National Science Foundation Faculty Early Career Development (CAREER) Program under award number NSF-ECCS-16-52871. The authors acknowledge the computational resources allocated by the Extreme Science and Engineering Discovery Environment (XSEDE) with Nos. TG-DMR180050 and TG-DMR180075. Star Crash Collisions between neutron stars involve some of the most extreme physics in the Universe. The intense effects of vast matter density and magnetic fields make them computation- hungry to simulate. A team from the National Center for Supercomputing Applications (NCSA) used artificial intelligence on the advanced graphics-processing-unit (GPU) nodes of the XSEDE-allocated supercomputers Bridges at the Pittsburgh Supercomputing Center (PSC) as well as Stampede2 at the Texas Advanced Computing Center (TACC) to obtain a correction factor that will allow much Figure 4: Determination of band alignments in ternary III-nitrides. Element-projected electronic structure of faster, less detailed simulations to produce (a) wz-AlN, (b) wz-GaN, (c) wz-InN, (d) zb-AlN, (e) zb- accurate predictions of these mergers. GaN, and (f) zb- Bizarre objects the size of a city but with more indicates an anion-like character, while the light InN. The red−light green- likecolormap behavior. mass than our Sun, neutron stars spew Credit: Tsai et. al, ACS Omega 2020, 5, 8, 3917-3923. magnetic fields a hundred thousand times green−blue colormap represents cation

RY5 IPR12 Page 14 stronger than an MRI medical scanner. A teaspoon of neutron-star matter weighs about a billion tons. It stands to reason that when these cosmic bodies smack together it will be dramatic. And nature does not disappoint on that count. Scientists have directly detected two neutron-star mergers to date. These detections depended on two gravitational-wave-detector observatories. LIGO consists of two detectors, one in Hanford, Wash., and the other in Livingston, La. The European Virgo detector is in Santo Stefano a Macerata, Italy. Scientists who analyze the data collected by LIGO and Virgo would like to see the highest-quality computer simulations of neutron star mergers. This allows them to identify what they should be looking for to better recognize and understand these events. But these simulations are slow and computationally expensive. Graduate student Shawn Rosofsky, working with advisor E. A. Huerta at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana- Champaign, set out to speed up such simulations. To accomplish this, he turned to artificial intelligence using the advanced graphics processing units (GPUs) of the National Science Foundation-funded, XSEDE-allocated Bridges supercomputing platform at PSC. Rosofsky set out to simulate the phenomenon of magnetohydrodynamic turbulence in the gasses surrounding neutron stars as they merge. This physical process is related to the turbulence in the atmosphere that produces clouds. But in neutron star mergers, it takes place under massive magnetic fields that make it difficult to simulate in a computer (see Figure 5). The scale of the interactions is small—and so detailed—that the high resolutions required to resolve these effects in a single simulation could take years. Rosofsky wondered whether deep learning, a type of artificial intelligence (AI) that uses multiple layers of representation, could recognize features in the data that allow it to extract correct predictions faster than the brute force of ultra-high resolutions. His idea was to produce a correction factor using the AI to allow lower-resolution, faster computations on conventional, massively parallel supercomputers while still producing accurate results. Deep learning starts with training, in which the AI analyzes data in which the "right answers" have been labeled by humans. This allows it to extract features from the data that humans might not have recognized but which allow it to predict the correct answers. Next the AI is tested on data without the right answers labeled, to ensure it's still getting the answers right. Rosofsky designed his deep-learning AI to progress in steps. This allowed him to verify the results at each step and understand how the AI was obtaining its predictions. This is important in deep learning computation, which otherwise could produce a result that researchers might not fully understand and so can't fully trust. Rosofsky used the XSEDE-allocated Stampede2 supercomputer at the Texas Advanced Computing Center (TACC) to produce the data that is used to train, validate and test his neural network models. For the training and testing phases of the project, Bridges' NVIDIA Tesla P100 GPUs, the most advanced available at the time, were ideally suited to the computations. Using Bridges, he was able to obtain a correction factor for the lower resolution simulations much more accurately than with the alternatives. The ability of AI to accurately compute subgrid scale effects with low resolution grids should allow the scientists to perform a large Figure 5: The intense magnetic fields accompanying simulation in months rather than years. The NCSA movement of matter from neutron-stars past each other team reported their results in the journal Physical causes increasingly complicated turbulence that is computationally expensive with standard simulation Review D in April 2020. methods. In this time series, a deep learning AI provides a simulation of this process at a fraction of the computing time.

RY5 IPR12 Page 15 The AI computations on Bridges showed that the method would work better and faster than gradient models. They also present a roadmap for other researchers to use AI to speed other massive computations. Future work by the group may include the even more advanced V100 GPU nodes of the XSEDE- allocated Bridges-AI system at PSC, or the upcoming Bridges-2 platform. Their next step will be to incorporate the AI's correction factors into large-scale simulations of neutron-star mergers and further assess the accuracy of the AI and of the quicker simulations. Their hope is that the new simulations will demonstrate details in neutron-star mergers that can be identified in gravitational wave detectors. These could allow observatories to detect more events, as well as explain more about how these massive and strange cosmic events unfold. This work was funded by National Science Foundation (NSF) Grants No. OAC-1931561, OAC-1934757, NSF-1550514, NSF-1659702, and TG-PHY160053. NVIDIA donated several Tesla P100 and V100 GPUs used for the analysis. Every Calculation Stabs Medical science is in a race to develop new and better antimicrobial agents to address infection and other human diseases. One promising example of such agents is the beta-defensins. These naturally occurring molecules stab microbes' outer membranes, dagger-like, causing their contents to spill out. A scientist at Tennessee Tech University used the XSEDE-allocated Bridges platform at PSC, as well as the D .E. Shaw Research (DESRES) Anton 2 system hosted at PSC, in a "one-two" simulation that shed light on a beta-defensin's initial binding to a microbial membrane. The work promises clues to agents that can better destroy microbes with membranes. Living in a post-pandemic world, it's hardly necessary to point out how important new antimicrobial agents can be for treating afflictions from drug-resistant bacterial infections to COVID-19. One promising avenue of research focuses on the beta-defensins, a family of small protein-like peptides. These molecules, which consist of a chain of amino acids, are naturally produced by the body to kill bacteria. Better, since intact cell membranes are so fundamental to survival, bacteria can't become resistant to this kind of attack. Beta-defensins can also suppress some viruses that have membranes, such as HIV. Engineered versions of beta- defensin may also be able to attack the SARS-CoV-2 virus Scientists would like to know more about how the beta- defensins work. Such information could help them to design both new antimicrobial agents and drugs that help the natural versions of these peptides work better. Beta-defensins work like little daggers, stabbing their Figure 6: Molecular dynamics way into the membrane of a microbe and spilling its contents simulations of human beta-defensin so it type 3 (red and blue) in wildtype. can no longer infect healthy cells. But these peptides Reprinted in part with permission work in a changing environment. The conditions in the body from The Journal of Physical Chemistry range from oxygen-rich oxidizing conditions surrounding cells, B. © 2020, American Chemical Society. for example, in the lungs to oxygen-poor reducing conditions in cells in the intestines. This causes important changes in the folding of a beta-defensin peptide such as human beta-defensin 3. The amino-acid chain in this beta-defensin's wild-type form is crosslinked to itself in three places via disulfide bonds. These links form in oxidizing conditions and break in reducing conditions. Scientists have long wondered how, and whether, beta-defensin can still destroy microbes in both forms.

RY5 IPR12 Page 16 To shed light on this question, Liqun Zhang at Tennessee Tech University found she needed to combine the complementary powers of the XSEDE-allocated Bridges platform and the DESRES Anton 2 supercomputer, both hosted at PSC. As a first step, Zhang simulated the equilibrated form of human beta-defensin type 3. This consists of starting with the peptide's chain in a disordered tangle and using the rules of chemical interaction to let that chain find the combination of twists and turns that it naturally settles into (see Figure 6). Zhang found Bridges to be a great tool for this step. The National Science Foundation-funded platform's massive computational abilities in both large memory and computational efficiency allowed her to simulate the initial 20 to 500 nanoseconds—billionths of a second—needed for the chain to find this lowest-energy form. To simulate beta-defensin sticking to the membrane, though, she needed a much longer simulation—5 to 7.5 microseconds (millionths of a second), over 10 times longer. To perform this simulation, she used Anton 2, which is made available to PSC without cost by DESRES and supported through operational funding from the National Institutes of Health. Anton 2 is a special-purpose supercomputer designed and developed by DESRES that greatly accelerates such molecular dynamics simulations. Because of its specialized hardware and software, Anton 2 can perform simulations nearly two orders of magnitude longer in a given length of real time than a general-purpose supercomputer. While not an XSEDE system, Anton 2's location at XSEDE partner PSC and the common support staff was a big help to her in making use of both supercomputers. Zhang simulated both the disulfide-crosslinked wild-type peptide and the uncrosslinked analog version of the peptide as it interacted with a virtual membrane typical of bacteria. In the initial Bridges simulations, she found that the wild-type version is much more rigid. Its crosslinks hold its shape more firmly than the analog version, which because of the lack of crosslinks is more flexible. The Anton 2 simulations showed an interesting difference that stems from this difference in flexibility. Two loops of the peptide chain initially stick to the membrane in both versions. But the analog version is flexible enough for an additional region, the "head" of the peptide, also to stick. Zhang reported her results in the Journal of Physical Chemistry in February, 2020. It isn't yet clear what the effects of this different way of initial binding to the membrane may mean for beta-defensin's ability to destroy microbes. An important next step will be for Zhang to simulate the actual insertion of the peptide into the membrane and the disruption of the membrane. Another important step will be for Zhang's colleagues to test her predictions on real peptides in the lab, verifying the results and in turn uncovering details she can use to create better simulations. Ultimately, she hopes that these simulations will offer clues for designing drugs to combat microbes that cause disease. Zhang and her colleagues also plan to design beta-defensin-based small antimicrobial peptides to combat the SARS-CoV-2 virus. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system, which is supported by NSF award number ACI-1445606, at PSC. Anton 2 computer time was provided by PSC through Grant R01GM116961 from the National Institutes of Health. The Anton 2 machine at PSC was generously made available without cost by D. E. Shaw Research. Access to Bridges and Anton 2 was provided via the COVID-19 HPC Consortium. Access to Bridges and Anton 2 was provided via the COVID-19 HPC Consortium. Some of the short-term simulations and analysis were carried out at the high performance computers at Tennessee Technological University. You can read Zhang's Journal of Physical Chemistry article here.13

13 https://doi.org/10.1021/acs.jpcb.9b10529 RY5 IPR12 Page 17 Georgia Tech Engineers Simulate Solar Cell Work Using XSEDE-Allocated Supercomputers Solar energy has become a popular renewable Figure 7: Four lead-free perovskites were simulated source of electricity around the world with silicon using Comet at the San Diego Supercomputer Center and serving as the primary source for solar cells due to Stampede2 at the Texas Advanced Computing Center. These simulations show that these materials exhibit its efficiency and stability. Because of silicon's promising features for solar energy options. They are relatively high cost, hybrid organic-inorganic now being synthesized for further investigation. Credit: perovskites (HOIPs) have emerged as a lower- H. Tran, et al, Georgia Institute of Technology; and V. cost—and highly efficient option—for solar power, Ngoc Tuoc, Hanoi University of Science and Technology. according to a recent study by Georgia Institute of Technology (Georgia Tech) researchers. The name perovskite refers not only to a specific mineral found in Russia's Ural Mountains (CaTiO3), but also to any compound that shares its structure. A search for stable, efficient, and environmentally safe perovskites has shaped an active avenue in current materials research with the new Georgia Tech findings relying on simulations done on Comet at the San Diego Supercomputer Center (SDSC) and Stampede2 at the Texas Advanced Computing Center (TACC). However, the presence of lead in the most promising perovskite candidates, methylammonium and formamidinium lead halides, has raised concerns. Moreover, these materials have shown to be unstable under certain environmental conditions. The Georgia Tech researchers worked with colleagues at the Hanoi University of Science and Technology in Vietnam to create simulations that identified four lead-free perovskites as promising candidates for solar cell materials (see Figure 7). Two of them have already been synthesized and the other two are recommended for further investigations. "This XSEDE-supported research relies on large-scale computations—a first step in our overall plan, which begins with showing simulations of this chemical space of HOIPs," said Huan Tran, a Georgia Tech materials science and engineering professor and co-author of Lead-free HOIPs for Solar Applications14, which was published earlier this year in The Journal of Chemical Physics. "Next, we will use these simulations to collaborate with experimental experts who can synthesize and test the predicted HOIPs—no personal computer can handle this level of computations hence the XSEDE supercomputers are a critically important aspect of our project." Tran and co-author Vu Ngoc Tuoc, a theoretical physics professor at the Hanoi University of Science and Technology, relied heavily upon Comet and Stampede2 for the large-scale computations that allowed them to conduct their research at a much higher level of detail. They also relied on the SDSC and TACC support staff to help when needed. "The technical support provided by both XSEDE groups was simply excellent as they helped us solve our problems very efficiently and promptly," Tran said. "XSEDE offered us access to leading computational facilities, and this is a very important factor for enabling my research topics and accelerating my projects," Tran continued. "In the coming era of materials informatics, computational materials data is the most important infrastructure and I find Comet, Stampede2, and other XSEDE facilities provide the ideal platform for boosting up the development of these areas."

14 https://aip.scitation.org/doi/full/10.1063/1.5128603 RY5 IPR12 Page 18 This research was supported by Vingroup Innovation Foundation under project VINIF.2019.DA03, and XSEDE (TG-DMR170031). The structures of the HOIPs reported in this work are available in the supplementary material15 and at http://godeepdata.org/. Settling In To understand how the tiny machinery of life works in health and disease, scientists need accurate pictures of how proteins fold and move. But laboratory methods for imaging proteins are slow, and so the structures of hundreds of thousands of proteins that have been discovered are still unknown. Scientists have used a number of methods for predicting structures via computer simulation. But sometimes even high-quality simulations aren't as accurate as drug designers may want. A Michigan State University team used the GPU nodes in XSEDE-allocated supercomputers to optimize predictions made by other scientists. In the process, they made predictions of the structures for a number of proteins with accuracy that approached the most precise, X-ray based lab measurements. One of the most important advances in how scientists think about how the contents of our bodies' cells work is that they no longer imagine proteins as stationary ball-and-stick models. Instead, they see the molecules as ever-moving structures. Understanding how proteins move has opened a window on exactly how they work—and how various disease processes interfere with those movements. The problem is that there are so many proteins. Humans alone have more than 30,000 different types of protein in their cells. And to understand human health, we also need to understand tens or hundreds of thousands more, in viruses, bacteria and parasites. Scientists have a number of tools—the most precise being X-ray crystallography—for taking snapshots of a given protein's structure. But these methods are relatively slow, and simply can't catch up with the backlog of unknown protein structures. To solve the problem of too few known protein structures, scientists have used computer modeling of what a protein might look like. But these methods aren't always as precise as some scientists need for tasks like designing new medicines. To address this issue, Lim Heo, a postdoctoral fellow at Michigan State University (MSU) and his advisor Michael Feig turned to molecular dynamics (MD) refinements using the graphics processing unit (GPU) nodes of the XSEDE- allocated supercomputers Bridges at the Pittsburgh Supercomputing Center and Comet at the San Diego Supercomputer Center (see Figure 8). Figure 8: An example of refinement in a portion of one of One problem with even some quite good protein- the proteins simulated by the MSU team. At top, the initial structure simulations, the Michigan State scientists ML predictions (red and blue) are superimposed on the real, laboratory-measured structure (yellow and pink). At realized, is that the proteins in them haven't quite bottom is the refined prediction of the MSU team's settled into their lowest-energy forms. A person molecular dynamics method (red and blue) superimposed easing into a recliner may need to wiggle a bit to on the real structure (yellow and pink). Green arrows find the most comfortable position. In a similar show regions that the molecular dynamics refinement way, a protein that isn't folded quite right needs to improved significantly. From Heo L, Feig M. High-accuracy protein structures by combining machine-learning with nudge itself to find a position that doesn't require physics-based refinement. Proteins. 2019;1–6. https://doi.org/ 10.1002/prot.25847. Reproduced with permission. 15 https://doi.org/10.1063/1.5128603#suppl RY5 IPR12 Page 19 unnecessary physical energy. Scientists understand the physics that makes the links between atoms in a protein twist and vibrate. MD simulation uses these rules to refine the thousands of such links in a given protein. Over a long enough simulated time period, the whole structure settles into its least-energy position. But such a simulation takes massive computing power. GPUs, or graphics processing units—which were originally developed to create better pictures in video games—are ideal for MD simulations. But the team simply couldn't get hold of enough GPU nodes on the computers initially available to them. Bridges and Comet—particularly their advanced NVIDIA P100 GPUs—proved ideal for giving them the power they needed to optimize the previous predictions. The molecular modeling community offered the scientists an opportunity to test their predictions. At the 13th Critical Assessment of Techniques for Protein Structure Prediction (CASP13) competition in 2018, other scientists used a simulation technique called machine learning (ML) to predict the structures of 27 whole or partial proteins that had recently been determined by lab methods. The MSU team then took these predictions the other scientists had made and refined them using their MD method. Using the same scoring method as in the CASP13 competition, they found that MD improved the ML predictions across the board—their refinements improved the earlier predictions by 3 to 30 percent. In some cases, the MD predictions achieved a precision that rivaled that of the best laboratory imaging methods. Next, the team will work on improving the efficiency of their calculations so that they can simulate longer time periods. Longer simulations allow better refinement, because proteins in an almost-correct initial model may need additional time to settle into their proper shapes. The team reported their latest results in the journal Proteins: Structure, Function and Bioinformatics in November 2019. This research was supported by National Institutes of Health Grants R01 GM084953 and R35 GM126948. Computational resources were used at the National Science Foundation’s Extreme Science and Engineering Discovery Environment (XSEDE) facilities under Grant TG-MCB090003. XSEDE Supercomputers Aid Drug Screening For Deadly Heart Arrhythmias Death from sudden cardiac arrest makes headlines when it strikes athletes. But it also causes the most deaths by natural causes in the U.S., estimated at 325,000 per year16. According to the Cleveland Clinic, the heart's bioelectrical system goes haywire during arrest. The malfunction can send heartbeats racing out of control, cutting off blood to the body and brain. This differs from a heart attack, which is caused by a blockage of the heart's arteries. The leading risk factors for sudden cardiac arrest are a past attack and the presence of disease. Another risk factor is the side effects from medications, which can potentially cause deadly arrhythmias. Using supercomputers allocated by the NSF-funded Extreme Science and Engineering Discovery Environment (XSEDE), scientists have developed for the first time a way to screen drugs through their chemical structures for induced arrhythmias (see Figure 9). Up until the early 2000s, the reason most drugs got removed from the market following FDA approval was cardiotoxicity in the form of deadly arrhythmia. In 2005, the FDA required a separate test for all drugs17. It measured the average time between the Q and T waves on an electrocardiogram, a record of the heartbeat. QT prolongation became a red flag for drug cardiotoxicity. But one problem with it is that some harmless substances, like grapefruit juice, also prolong QT interval, and using it as a proxy for heart arrhythmia could mean the loss of potentially useful and safe drugs. "What we set out to do was to try to solve that problem by building a computer-based pipeline for screening," said Colleen Clancy, a professor in the Department of Physiology and Membrane Biology

16 https://my.clevelandclinic.org/health/diseases/17522-sudden-cardiac-death-sudden-cardiac-arrest 17 https://www.fda.gov/drugs/regulatory-science-action/impact-story-finding-better-test-predicting-risk-drugs-pose-heart RY5 IPR12 Page 20 and the Department of Pharmacology at the UC Davis School of Medicine. Clancy co-authored a study18 on the computational cardiotoxicity drug screening pipeline in the journal Circulation Research in April 2020. "The major novelty of the pipeline is that we found a way to connect the atomistic scale to higher level function scales, like protein function, cell function, and in our simulated Figure 9: The authors validated drug models by tissue-level models we can calculate the performing all-atom umbrella sampling molecular spatial and temporal gradients of electrical dynamics simulations across a lipid bilayer for both activity in those simulated pieces of tissue," dofetilide and for moxifloxacin. (Credit: Yang et al., Clancy said. "That is an approximation of the Circulation Research). electrocardiogram that's measured in the clinic. We can do a direct comparison between the electrocardiogram in the simulated tissue, and electrocardiograms from patients that have taken those drugs." The two drugs chosen in the study both prolonged the QT interval. One of them, dofetilide, is a known proarrhythmic agent. The other, moxifloxacin, has a strong safety profile in healthy humans. "There's been no way to distinguish between those two classes," Clancy said. "That's what we were able to show in the computational pipeline." Starting from the chemistry of the drug interactions with a target, the scientists used that information to predict proarrhythmia vulnerability through a machine learning approach based on multi-scale computer simulation data. Clancy and colleagues chose the hERG (human Ether-à-go-go-Related Gene) potassium channel in the heart as the drug target in the first step of their computational pipeline. The hERG mediates the electrical activity of the heart, and drug companies usually screen for whether a drug blocks it. "The big challenge computationally is the system that we studied is pretty large," said study co-author Igor Vorobyov, an assistant professor in the Department of Physiology and Membrane Biology and the Department of Pharmacology at the UC Davis School of Medicine. "It's on the atomistic scale. We have around 130,000 atoms in our system. This includes the hERG protein embedded in the lipid membrane surrounded in a salt aqueous solution in water." The calculations involved billions of individual time steps to achieve an all-atom simulation of several microseconds, enough to get detailed information on how the drug binds to the target. "Here is where supercomputers come in very handy," Vorobyov said. He was awarded allocations by XSEDE on the Stampede2 system of the Texas Advanced Computing Center (TACC). XSEDE also provided supercomputing time on Comet at the San Diego Supercomputer Center (SDSC), making use of Comet's GPU and CPU nodes. The National Center for Supercomputing Applications allocated use of its NSF-funded Blue Waters system. And the scientists made use of the Anton 2 system at the Pittsburgh Supercomputing Center (PSC). Vorobyov's team received support from XSEDE staff to help fine tune the code they used, a standard code called NAMD. "We got tremendous help, and it saved us many hours to be able to use this service. The code efficiency was increased by 50 percent or more sometimes, when we worked with XSEDE staff, who helped us to fine-tune the code," he said. He added that access to XSEDE and other supercomputing resources changed their perspective.

18 https://www.ahajournals.org/doi/10.1161/CIRCRESAHA.119.316404 RY5 IPR12 Page 21 "Now, we are not limited by our own resources," Vorobyov said. "We can use the top computational resources in the world to do these calculations. It totally changes your perspective as a scientist that you can use these resources to advance science, and feel like you belong to this large community of other scientists, for the greater good of the U.S. and the worldwide community." The study, "A Computational Pipeline to Predict Cardiotoxicity: From the Atom to the Rhythm" was published February 24, 2020, in the journal Circulation Research. The study co-authors are Pei-Chi Yang, Kevin R. DeMarco, Parya Aghasafari, Mao-Tsuen Jeng, John R.D. Dawson, Vladimir Yarov-Yarovoy, Igor Vorobyov, Colleen E. Clancy of the University of California, Davis; Slava Bekker of American River College; Sergei Y. Noskov of University of Calgary, Alberta, Canada. The study was funded by the National Institutes of Health, the American Heart Association, and other agencies. XSEDE Resources Used for High Tech Materials Science Study For thousands of years, humans have produced ceramics by simply combining specific minerals with water or other solvents to create ceramic slurries that cure at room temperature and become some of the hardest known materials. In more recent times, zirconia-based ceramics have been useful for an array of applications ranging from dental implants and artificial joints to jet engine parts. The National Science Foundation's XSEDE (Extreme Science and Engineering Discovery Environment) program has long been helping to advance this and other types of materials science discoveries. In this case, during the past year, researchers from the Colorado School of Mines have been using multiple supercomputers—Comet at the San Diego Supercomputer Center (SDSC), Stampede2 at the Texas Advanced Computing Center (TACC), and Bridges at the Pittsburgh Supercomputing Center (PSC)—to study certain characteristics of zirconia. The team recently published their findings in The Journal of the European Ceramic Society.19 According to corresponding author Mohsen Asle Zaeem, a mechanical engineering professor at the Colorado School of Mines, the publication featured simulations that address zirconia-based ceramic's ability to withstand harsh conditions as well as its fracture and fatigue limitations. "By utilizing large-scale atomistic simulations, we revealed how specific type nanoscale structures, twin boundaries and pre- existing defects, control the mechanical behavior and the corresponding plastic deformation of an advanced shape memory ceramic, yttria-stabilized tetragonal zirconia (YSTZ)," explained Asle Zaeem (see Figure 10). "Some important applications, such as jet engines, require advanced materials that can perform reliably at extreme conditions; shape memory ceramics have shown superior properties at high temperatures such as high strength and excellent oxidation/corrosion resistance, and addressing their deformation, fracture and fatigue limitations will open the door for creating the next generation of high- temperature materials." Asle Zaeem, who has been using XSEDE resources since 2012, was Figure 10: The findings of this work provided significant insight into the provided with both computing allocations and technical support behavior of twin boundaries and pre- for this study. The XSEDE Extended Collaborative Support existing defects in shape memory Services (ECSS) group assisted Asle Zaeem and his team on the ceramics. Credit: N. Zhang and M. Asle installation of complex software and also helped address any Zaeem, both at the Colorado School of issues that the group's students and postdoctoral researchers Mines. experienced while collecting data, running simulations, and compiling results.

19 https://www.sciencedirect.com/science/article/pii/S0955221919306168 RY5 IPR12 Page 22 "My XSEDE-related work has resulted in more than 35 peer-reviewed journal articles over the past eight years—with eight doctorate students and five postdocs relying on the supercomputing facilities at SDSC, TACC, and PSC," said Asle Zaeem. "XSEDE has pioneered ways to provide world-class computing resources to academic researchers based solely on the scientific merit and computational readiness of the proposed research projects with no restrictions. This is a brilliant approach which has already resulted in scientific advancements that could not be achieved otherwise." This research was supported by the U.S. Department of Energy, Office of Science, Basic Energy Sciences, under award DE-SC0019279. The XSEDE portion of the project was via allocation TG-DMR140008. Supercomputers Simulations Help Advance Electrochemical Reaction Research Single-atom catalysts have recently emerged as promising innovations for solving environmental and energy issues. One such example, nickel embedded in graphene (a thin layer of graphite), has been shown to convert carbon dioxide, a molecule that causes the greenhouse effect, into carbon monoxide, an important feedstock for chemical engineering (see Figure 11). As a poisonous gas, carbon monoxide is often converted into carbon dioxide, such as in cars and trucks equipped with catalytic converters. This process is the reverse, which at first may sound a bit odd, but provides an important role in synthesizing gases to use as fuel in generating electricity in lieu of the Earth's quickly depleting fossil fuels. However, a better understanding of the atomic structure of this concoction is needed before we can use nickel-embedded graphene on a regular basis. To help with this challenge, researchers from the University of Texas at Austin (UT Austin) recently simulated the catalytic mechanism and atomic structure of nickel-doped graphene using Comet at the San Diego Supercomputer Center (SDSC) and Stampede2 at the Texas Advanced Computing Center (TACC). The simulations showed a clear picture of the catalyst's atomic structure so that researchers were able to better understand critical effects of surface change and hydrogen bonding, which were overlooked in previous models. "Our work unveils the structure and mechanism underlying the performance of the catalyst - we found that the catalysis arises from the structure where the nickel atom is bonded with one nitrogen and three carbon atoms," said Yuanyue Liu, an assistant professor of computational materials science at UT Austin. "This structure can carry more charges than other hypothesized structures, and having more charges makes the electrochemical reaction faster, which means that the carbon dioxide can be converted to carbon monoxide with a higher rate." Liu further explained that this reaction can be facilitated by hydrogen bonding between water and the polar intermediate species generated during reactions, making it more favorable over other competing reactions that do not produce polar intermediates. "This study not only explained a long-standing puzzle for an important catalyst but also highlighted the critical roles of change capacity and hydrogen bonding, which can help elucidate the mechanisms of other heterogeneous Figure 11: Electrochemical reduction of electrocatalysts in water to enable better design," said Liu. carbon dioxide catalyzed by single nickel atom embedded in graphene with nitrogen dopant. Image Credit: Xunhua Zhao and Yuanyue Liu, UT Austin.

RY5 IPR12 Page 23 Published last month in the Journal of The American Chemical Society20, Liu and co-author Xunhua Zhao discussed their future plans to advance their studies on electrochemical reactions for energy conversion. Zhao, a postdoctoral student at UT Austin, explained the importance of utilizing supercomputers for this work. "Our simulations required hundreds of CPUs running parallelly for tens of hours per calculation," said Zhao. "We relied heavily on Comet and Stampede2 to accurately model the complex processes of charge transfer and dynamically evolving hydrogen-bond networks." Specifically, Liu and Zhao used a molecular dynamics simulation technique based on quantum mechanics—an examination of molecular structure. They are now focused on collaborations with experimentalists to test their computational results. This research was funded by the National Science Foundation (1900039), the Welch Foundation (F- 1959-20180324), and the Extreme Science and Engineering Discovery Environment (TG-CHE1900065).

20 https://pubs.acs.org/doi/full/10.1021/jacs.9b13872 RY5 IPR12 Page 24 3. Discussion of Strategic Goals and Key Performance Indicators The strategic goals of XSEDE (§1.1) cover a considerable scope. Additionally, the specific activities within XSEDE’s scope are often very detailed; therefore, to ensure that this significant and detailed scope will ultimately deliver and realize the project’s mission and vision, the three strategic goals are decomposed into components or sub-goals to be considered individually. In determining the best measures of progress toward each of the sub-goals, KPIs that correlate to impact on the scientific community are used. These often pair measurements of outcome with an assessment of quality or impact to provide both a sense of scope and significance of the supporting activities. Deepen and Extend Use XSEDE will 1) deepen the use—make more effective use—of the advanced digital services ecosystem by existing scholars, researchers, and engineers and 2) extend the use to new communities. XSEDE will 3) contribute to preparation—workforce development—of scholars, researchers, and engineers in the use of advanced digital technologies via training, education, and outreach; and XSEDE will 4) raise the general awareness of the value of advanced digital research services. 3.1.1. Deepening Use to Existing Communities XSEDE engages in a range of activities that serve to deepen use including identifying new technologies and new service providers, evolving the e-infrastructure, and enhancing the research prowess of current and future researchers. However, the ongoing use of resources and services available via XSEDE is the key indicator of this deepening use. As a result, the project has chosen three KPIs (Table 3-1) that together measure the ongoing engagement with the community with an emphasis on exposing the diversity of those consuming these services: 1) number of sustained users of XSEDE resources and services via the portal, 2) number of sustained underrepresented individuals using XSEDE resources and services via the portal, and 3) percentage of sustained allocation users from non-traditional disciplines of XSEDE resources and services.

Table 3-1: KPIs for the sub-goal of deepen use (existing communities).

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year Number of 4,500/ CEE (§4) RY5 4,644 sustained users qtr of XSEDE 3,500/ resources and RY4 4,137 4,728 4,070 4,615 6,578 qtr services via the portal1 3,500/ RY3 4,196 4,089 3,099 4,864 6,851 qtr 3,000/ RY2 3,962 3,754 2,488 3,020 4,527 qtr >5,000/ RY1 * 4,755 4,446 4,924 6,186 qtr Number of 1,750/ CEE (§4) RY5 831 sustained yr under- 1,750/ represented RY4 625 809 564 705 1,014 yr individuals using XSEDE 1,500/ RY3 529 5091 343 636 1,818 resources and yr

RY5 IPR12 Page 25 Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year services via the 1,500/ 1 RY2 490 408 296 402 1,0531 portal yr >1,000/ RY1 * 322 238 535 6491 yr ECSS (§5) Percentage of RY5 33/yr 23.2 sustained allocation users from non- RY4 22/yr 21.1 22.7 23.0 23.5 30.4 traditional disciplines of RY3 20/qtr 22.1 20.7 25.8 22.7 22.8 XSEDE resources and services RY2 * 21.5 20.4 21.1 21.0 21.0

RY1 * * 18.2 19.9 18.6 18.9

1 The totals of these KPIs do not equal the sum of the data from each reporting period because one person could be counted as a sustained user/individual in more than one reporting period if they continue to log in for multiple reporting periods; however, they will only be counted once in the total. The number of sustained users of XSEDE resources and services exceeded expectations. Furthermore, it surpassed previous years during the same reporting periods. Additionally, the number of sustained users from underrepresented groups using XSEDE resources was higher than anticipated and may indicate greater availability to conduct research and participate in training events this summer due to COVID-19 travel restrictions. All of CEE education, training, and campus engagement events had much higher participation rates than in previous years. This directly contributes to increased numbers of sustained users and increased numbers of sustained users from underrepresented groups. Moreover, shifting to online platforms for what would have previously been face-to-face engagements has resulted in a positive shift in increased participation and engagement over the past quarter. The percentage of sustained allocated users from non-traditional disciplines is in line with recent quarterly reporting periods. As discussed in the RY4 annual report, these quarterly snapshots are orientative with respect to the annual target. The change in the definition of XSEDE Fields of Science, which will become effective as of the next reporting period, may result in changes in the classification of allocated projects that are considered within the scope of NIP. This, in turn, may cause a discontinuity in this KPI. We will monitor this and make changes to the KPI definition and/or the collection methodology as appropriate. 3.1.2. Extending Use to New Communities New communities are defined as fields of science, industry, and underrepresented communities that represent less than one percent of XSEDE Resource Allocation Committee (XRAC) allocations. The Novel & Innovative Projects (NIP) team and the Broadening Participation team both work to bring advanced digital services to new communities. XSEDE measures both the number of new users and the number of new users on research projects from underrepresented communities and non-traditional disciplines of XSEDE resources and services as the indicators of progress ( Table 3-2).

RY5 IPR12 Page 26 Table 3-2: KPIs for the sub-goal of extend use (new communities).

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Number of new RY5 2,500/qtr 2,157 CEE (§4) users of XSEDE resources and RY4 3,000/qtr 2,415 3,209 2,365 3,089 11,078 services via the portal RY3 3,000/qtr 1,905 2,763 2,5271 2,757 9,952

RY2 2,000/qtr 2,305 2,813 2,346 2,917 10,381

RY1 >1,000/qtr * 1,973 1,849 2,359 6,181

Number of new RY5 250/qtr 301 CEE (§4) underrepresented individuals using RY4 175/qtr 380 332 214 400 1,326 XSEDE resources and services via the RY3 200/qtr 134 155 129 238 656 portal RY2 150/qtr 251 175 222 234 882

RY1 100/qtr * 150 135 240 525

Percentage of new RY5 35/yr 30.1 ECSS (§5) allocation users

from non- RY4 35/yr 26.4 37.1 33.4 38.4 34.8 traditional disciplines of XSEDE RY3 30/yr 33.1 26.0 38.7 32.5 32.8 resources and services RY2 24.8 26.0 26.6 33.9 27.8

RY1 * 21.8 24.9 31.0 25.7 The number of new users of XSEDE resources and services did not meet expectations. This reporting period covers the summer months, which traditionally skews lower due to less academic activity. The number of new users from underrepresented groups exceeded the quarterly target and can be attributed to strong participation in XSEDE summer training events and curriculum development workshops. Although this number is lower than last year at this time, it positively reflects the work to shift all training and education activities to an online format. While deeper engagements with students and faculty have resulted, recruitment was an issue that needed to be addressed. The recruitment issue can likely be attributed to ‘Zoom fatigue’ and has been directly expressed by participants in the online activities over the summer. Accordingly, adaptations to the online offerings have been devised to compensate for this fatigue, such as building in breaks and activities that ease online video conferencing fatigue. The percentage of new allocated users from non-traditional disciplines is lower than in the three most recent quarterly reporting periods, but higher than in the first period of RY4. As discussed in the RY4 annual report, these quarterly snapshots are orientative with respect to the annual target. The change in the definition of XSEDE Fields of Science, which will become effective as of the next reporting period, may result in changes in the classification of allocated projects that are considered within the scope of NIP. This, in turn, may cause a discontinuity in this KPI. We will monitor this and make changes to the KPI definition and/or the collection methodology as appropriate.

RY5 IPR12 Page 27 3.1.3. Prepare the Current and Next Generation Part of XSEDE’s mission is to provide a broad community of existing and future researchers with access and training to use advanced digital services via the sub-goal of preparing the current and next generation of computationally-savvy researchers. While many activities support this sub-goal, such as the various Champion (§4.6), Student Engagement (§4.4), and Education (§4.2) programs, the training offered through Community Engagement & Enrichment (CEE) impacts the most people directly. Therefore, the key indicator (Table 3-3) of performance toward this goal, which is reflective of industry standards, is the number of participant hours of live training delivered by XSEDE.

Table 3-3: KPI for the sub-goal of preparing the current and next generation.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Number of RY5 40,000/yr 19,751 CEE (§4) participant hours of live RY4 40,000/yr 15,461 16,135 6,380 12,667 50,643 training delivered by RY3 40,000/yr 14,140 8,274 7,259 12,352 42,025 XSEDE1 RY2 NA 12,787 8,876 6,004 14,753 42,421

RY1 NA 5,994 3,770 5,180 9,199 24,143

1This was a new KPI in RY3. Data provided for RY1 and RY2 was reported retroactively.

The number of participant hours of live training was larger than normal—almost 20% more than the maximum reported participant-hours in any previous Reporting Period—and may be the result of the general increase in demand for training due to the COVID-19 travel restrictions and the need for online content for faculty moving from in-person to remote delivery of their courses. Training, education, broadening participation, and campus engagement all experienced higher demand and increased event registration and attendance. This will be monitored to see if this is a sustained change or a temporary effect due to the impact of COVID-19. 3.1.4. Raising Awareness While many activities led by teams throughout the XSEDE organization, such as Workforce Development (§4.2), User Engagement (§4.3), Broadening Participation (§4.4), and Campus Engagement (§4.6) contribute to the ability to raise the general awareness of the value of advanced digital research services, the project has chosen to focus on measures in two areas (Table 3-4): user input and social media. Desirable trends in these key outcomes can be correlated to success for this sub-goal.

Table 3-4: KPIs for the sub-goal of raise awareness of the value of advanced digital research.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year Aggregate mean 3.7 of 5/ PgO (§9) RY5 3.8 rating of user yr awareness of XSEDE 3.7 of 5/ resources and RY4 3.8 - - - 3.8 services yr 3.5 of 5/ RY3 3.7 - - - 3.7 yr

RY5 IPR12 Page 28 Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

RY2 * 3.6 - - - 3.6

RY1 * * - - - NA

Number of social 429,282/y PgO (§9) RY5 86,046 media impressions r over time1 426,198/y RY4 87,482 75,164 84,185 110,904 357,735 r

359,714/y RY3 112,806 63,269 93,803 85,287 355,165 r

RY2 NA 69,607 55,506 59,490 128,180 312,783

RY1 NA * 52,200 128,675 88,332 269,207

- Data reported annually 1 This was the new KPI in RY3. Data provided for RY1 and RY2 was reported retroactively. Beginning in RY5, this KPI is calculated as the number of social media impressions with the annual target calculated as a percentage of increase over the previous year. “Aggregate mean rating of user awareness of XSEDE resources and services” surpassed its annual target again this year. The number of social media impressions stayed about the same this reporting period as the same period last year, decreasing slightly by ~1.6%. External Relations believes this decrease to be due in part to the ongoing pandemic and potential audience fatigue surrounding disease research. Even given the decrease, these results are encouraging because they include a mixture of Science Stories and hands-on events, nominations, and workshop promotions that help to actively engage new communities. During RY4, ER decided to establish a voice that mixed both of these areas, and doing that without seeing a substantial drop-off in impressions indicates that XSEDE has successfully integrated into that digital space. ER is confident that the number of impressions will increase in the next reporting period due to increased social media coverage of the PEARC20 conference that included cross posting of HPCwire stories submitted by XSEDE. As a result the team has already started to see an increase in social media impressions that is expected to continue throughout August and September, and because of this, ER believes RY5 RP2 will show an increase in impressions beyond what has typically been seen in RP2 in previous years. Advance the Ecosystem Exploiting its internal efforts and drawing on those of others, XSEDE will advance the broader ecosystem of advanced digital services by 1) creating an open and evolving e-infrastructure, and by 2) enhancing the array of technical expertise and support services offered. 3.2.1. Create an Open and Evolving e-Infrastructure There are a variety of factors that affect the evolution of the e-infrastructure. These range from external factors, such as the number of XSEDE Federation members and the variety of services they provide, to internal factors, like Operations (§7) of critical infrastructure and services and the evaluation and integration of new capabilities. While XSEDE actively seeks new Federation members and Service Providers, as well as partnerships with national and international cyberinfrastructure projects, the group views their role as connectors of these elements to have the most impact. Thus, XSEDE focuses on

RY5 IPR12 Page 29 the number of new capabilities in production as an indicator of performance with respect to this sub- goal (Table 3-5).

Table 3-5: KPI for the sub-goal of create an open and evolving e-infrastructure.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Total number of RY5 110 102 XCI (§6) capabilities in production1 RY4 100 88 88 91 102 102

RY3 81 76 85 87 87 87

RY2 NA 72 74 74 75 75

RY1 NA * 63 63 69 69

1 This was a new KPI in RY3. Data provided for RY1 and RY2 was reported retroactively. Although the group did not deliver any new capabilities during the summer, RACD has enough capabilities in the pipeline to reach the goal of 110 capabilities for the program year. 3.2.2. Enhance the Array of Technical Expertise and Support Services To enhance the technical expertise of XSEDE’s staff to offer an evolving set of support services, the project will continue many activities including workshops, symposia, and training events hosted by Extended Collaborative Support Services (ECSS) and Service Providers (§5.6). The KPI for this is feedback provided from the XSEDE user-base through the annual user survey (Table 3-6).

Table 3-6: KPI for the sub-goal of enhance the array of technical expertise and support services.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year Aggregate mean 3.5 of 5/ PgO (§9) RY5 4.4 rating of user yr satisfaction with 3.5 of 5/ XSEDE technical RY4 3.9 - - - 3.9 yr support services1 3.5 of 5/ RY3 3.6 - - - 3.6 yr

RY2 NA 3.4 - - - 3.4

RY1 * * * * * *

- Data reported annually. 1 This is a new KPI in RY3. Data provided for RY2 was reported retroactively. “Aggregate mean rating of user satisfaction with XSEDE technical support services” greatly surpassed its annual target. The XSEDE evaluation team believes this is largely a result of changes that were made to the data collection instrument and methodology this year as part of efforts to refactor how its service-level KPIs are calculated. An additional screening question was added which asked respondents to self-identify as a “user,” a “service provider,” or a “gateway developer or operator.” This allowed the evaluators to more closely align the services being evaluated to the most appropriate population; that is, only those who identified as a “service provider” were presented with the opportunity to evaluate services intended for service providers. This change in the data collection methodology allowed an even greater ability to target the most appropriate population, which, in turn, can generally be expected to

RY5 IPR12 Page 30 elicit more accurate and often more positive results. With no data to support any trend given the new collection methodology, the evaluation team will monitor the performance of this KPI for another year before determining whether or not the target should be raised. Sustain the Ecosystem XSEDE will sustain the advanced digital services ecosystem by 1) ensuring and maintaining a reliable, efficient, and secure infrastructure, and 2) providing excellent user support services. Furthermore, XSEDE will operate an 3) effective, 4) productive, and 5) innovative virtual organization. 3.3.1. Provide Reliable, Efficient, and Secure Infrastructure Many activities support the provisioning and support of reliable, efficient and secure infrastructure— such as User Interfaces & Online Information (§4.5), Security (§7.2), Data Transfer Services (§7.3), Systems Operations and Support (§7.5), support for Allocations (§8.2), and Allocations, Accounting & Account Management (§8.3)—but perhaps the truest measure of an infrastructure’s reliability is its robustness as reflected by sustained availability. Thus, the KPI for this sub-goal is the mean composite availability of core services, shown as a percentage (Table 3-7), measured as a geometric mean. This is a composite measure of the availability of critical enterprise services and the XRAS allocations request management service.

Table 3-7: KPI for the sub-goal of provide reliable, efficient, and secure infrastructure.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Mean composite RY5 99.9/qtr 99.9 Ops (§7) availability of core services RY4 99.9/qtr 99.9 99.9 99.9 99.9 99.9 (%) RY3 99.9/qtr 99.9 99.9 99.9 99.9 99.9

RY2 99.9/qtr 99.8 99.9 99.9 99.9 99.9

RY1 99.0/qtr * 99.9 99.9 99.9 99.9 The core services availability target for the project was met this reporting period. 3.3.2. Provide Excellent User Support Although nearly every group in the organization has some support function, XSEDE has chosen to focus on metrics with respect to two primary support interfaces to the community: the XSEDE Operations Center (XOC) and the Resource Allocation Services (RAS) team. The XOC is the frontline centralized support group that either resolves or escalates tickets to the appropriate resolution center depending on the request. RAS is responsible for the allocations process and the allocation request system. These two support interfaces are the focus for gauging the progress towards achieving the sub-goal of providing excellent user support, specifically: 1) the mean time to resolution on support tickets that are resolved by the XOC or routed to, and resolved by, other XSEDE areas, 2) the aggregate mean rating of user satisfaction with allocations process and support services measured via a quarterly survey of users who have interacted with the allocations request system and the allocations process more generally, and 3) the percentage of research requests successful (not rejected) determined following the quarterly allocations session (Table 3-8).

RY5 IPR12 Page 31 Table 3-8: KPIs for the sub-goal of provide excellent user support.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Mean time to ticket RY5 < 16/qtr 15.3 Ops (§7) resolution (hours)

RY4 < 16/qtr 14.0 14.6 21.0 14.3 16.0

RY3 < 16/qtr 12.3 18.5 23.2 17.5 17.9

RY2 < 24/qtr 26.0 20.1 22.8 15.0 21.0

RY1 < 24/qtr * 24.0 28.2 23.1 25.1

Mean rating of user RY5 4 of 5/yr 4.4 RAS (§8) satisfaction with allocations process RY4 4 of 5/yr 4.2 4.3 4.4 4.2 4.3 and support services1 RY3 4 of 5/yr 4.1 4.2 4.1 4.4 4.2

RY2 4 of 5/yr 4.1 4.0 4.1 3.9 4.0

RY1 4 of 5/yr * 4.0 4.0 4.0 4.0

Percentage of RY5 85.0/qtr 80.0 RAS (§8) research requests successful (not RY4 85.0/qtr 81.0 80.0 78.0 87.0 82.0 rejected) RY3 85.0/qtr 65.0 70.0 72.0 75.0 70.5

RY2 85.0/qtr 70.0 69.0 72.0 68.0 69.8

RY1 85.0/qtr * 76.0 75.0 74.0 75.0

1 KPI name updated in RY4. Though slightly higher than the previous period, ticket resolution time was still below the target this reporting period. RAS continues to receive satisfaction ratings above the target of 4.0 (out of 5) for both the allocations process as whole, as well as the XRAS system, despite continuing high request levels for the quarterly Research opportunity. The target of 4.0 will remain given the uncertain and qualitative nature of this metric, as well as the continued reductions to recommended amounts for Research allocations. While the success rate, at 80%, for Research requests remained below target, efforts continue to pursue improvements to the users’ ability to prepare successful requests. While the 80% measure represents a drop from the previous period’s anomalously high 87%, it does continue the trend for this KPI at or above 80%. 3.3.3. Effective and Productive Virtual Organization During the first five years of XSEDE, in conjunction with developing a methodology for driving and assessing performance excellence, XSEDE adopted the Baldrige Criteria21 and has assessed and applied criteria from all seven criteria by that methodology. These include annual reviews of the vision, mission, strategic goals, project-wide processes and standards (KPIs); user and staff surveys (§4.3,

21 https://www.nist.gov/baldrige/

RY5 IPR12 Page 32 §9.5); stakeholder communications (§9.2); advisory boards (§9.1); community engagement (§4); workforce development (§4.2); and the analysis of organizational data that leads to organizational learning, strategic improvement, and innovation. With this foundation, it is now appropriate to look to the XSEDE users to give an indication of the project’s effectiveness by rating the importance of the resources and services provided by XSEDE (Table 3-9).

Table 3-9: KPIs for the sub-goal of operate an effective and productive virtual organization.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Mean rating of RY5 4.4 of 5/yr 4.2 PgO (§9) importance of XSEDE resources RY4 4.4 of 5/yr 4.2 - - - 4.2 and services to researcher RY3 4.2 of 5/yr 4.4 - - - 4.4 productivity RY2 NA 4.42 - - - 4.4

RY1 NA 4.32 - - - 4.3

Percentage of users RY5 80/yr 83 PgO (§9) who indicate the use of XSEDE-managed RY4 80/yr 79 - - - 79 and/or XSEDE- associated RY3 79/yr 79 - - - 79 resources in the creation of their RY2 * * * * * * work product1 RY1 * * * * * *

1 New KPI added in RY3 RP2. 2 These historical numbers are based on other survey data that was vaguely related to this KPI. We created a new survey item in RY3 to address it directly. - Data reported annually. While “Mean rating of importance of XSEDE resources and services to researcher productivity” was slightly under the target, this rating is still very high and indicates that users are finding XSEDE resources and services important to their productivity. The evaluation team believes the slight decrease is due, in part, to oversubscription of XSEDE-allocated resources, as well as the age of the project. As any project reaches its capacity or end of its life cycle, users look to incorporate newer, less-utilized resources into their workflows as a way of mitigating the risks of relying too heavily on oversubscribed or retiring resources, thus reducing the importance of any single resource. The project is working to improve on this metric, but this is working against the evaluation team’s expectation that this trend will continue. The target for “Percentage of users who indicate the use of XSEDE-managed and/or XSEDE-associated resources in the creation of their work product” was surpassed this year. 3.3.4. Innovative Virtual Organization Measuring innovation for an organization like XSEDE (or for organizations in general) is difficult and represents an area of open research. After much thought and discussion both internally and with external stakeholders and advisors, XSEDE has identified two indicators that correlate to innovation within the project: 1) percentage of Project Improvement Fund proposals resulting in innovations in the XSEDE organization and 2) mean rating of innovation within the organization by XSEDE staff (Table 3-10). The first indicator is a measurement of XSEDE’s ability to fund smaller innovative improvements within the project; the second measures how staff rate the level of innovation within the project. These RY5 IPR12 Page 33 KPIs will continue to be the subject of an open conversation within the organization and with stakeholders and advisors as XSEDE assesses these measurements and how to best quantify innovation.

Table 3-10: KPIs for the sub-goal of operate an innovative virtual organization.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Percentage of RY5 70/yr - PgO (§9) Project Improvement Fund RY4 70/yr - - - 66.7 66.7 funded projects resulting in RY3 60/yr - - - 71.4 71.4 innovations in the XSEDE organization RY2 * * * * * *

RY1 * * * * * *

Mean rating of RY5 4 of 5/yr - PgO (§9) innovation within the organization by RY4 4 of 5/yr - 4.0 - - 4.0 XSEDE staff RY3 3.5 of 5/yr - 4.0 - - 4.0

RY2 * * * * * *

RY1 * * * * * *

- Data reported annually. “Percentage of Project Improvement Fund funded projects resulting in innovations in the XSEDE organization” is reported annually in RP4, so there is no data to report during this reporting period. “Mean rating of innovation within the organization by XSEDE staff” is reported annually in RP2 based on Staff Climate Study results, so there is no data to report during this reporting period.

RY5 IPR12 Page 34 4. Community Engagement & Enrichment (WBS 2.1) Community Engagement & Enrichment (CEE) sits at the front lines of XSEDE and is tasked with balancing support for a large and diverse portfolio of existing users and the broader population of potential users and future leaders in cyberinfrastructure. While maintaining high quality support for existing users engaged in science at all levels, CEE is concerned with training and educating future generations and trying to creatively address what has been widely accepted as a leaky pipeline of potential users, leaders, practitioners, and researchers. At the core of Community Engagement & Enrichment (CEE) is the researcher, broadly defined to include anyone who uses or may potentially use the array of resources and services offered by XSEDE. The CEE team is dedicated to actively engaging a broad and diverse cross-section of the open science community, bringing together those interested in using, integrating with, enabling, and enhancing the national cyberinfrastructure. Vital to the CEE mission is the persistent relationship with existing and future users, including allocated users, training participants, XSEDE collaborators, and campus personnel. CEE will unify public offerings to provide a more consistent, clear, and concise message about XSEDE resources and services, and bring together those aspects of XSEDE that have as their mission teaching, informing, and engaging those interested in advanced cyberinfrastructure. The five components of CEE are Workforce Development (§4.2), which includes Training, Education and Student Preparation, User Engagement (§4.3), Broadening Participation (§4.4), User Interfaces & Online Information (§4.5), and Campus Engagement (§4.6). These five teams ensure routine collection and reporting of XSEDE’s actions to address user requirements. They provide a consistent suite of web- based information and documentation and engage with a broad range of campus personnel to ensure that XSEDE’s resources and services complement those offered by campuses. Additionally, CEE teams expand workforce development efforts to enable many more researchers, faculty, staff, and students to make effective use of local, regional, and national advanced digital resources. CEE expands efforts to broaden the diversity of the community utilizing advanced digital resources. The success of the CEE team depends on effective collaboration across all L2 areas of the project. Specifically, User Engagement works closely with RAS and ECSS to establish a dialogue with XSEDE’s User Community in order to better understand their needs and desires. Workforce Development and Broadening Participation partner with ECSS to develop impactful training and education opportunities for the community, especially underrepresented students, researchers, and faculty. The User Interfaces & Online Information team relies heavily on all areas of the project to ensure that the website remains accurate and informative. The Campus Engagement team likewise depends on all parts of the project to facilitate the effective participation of a diverse national community of campuses in the application of advanced digital resources and services to accelerate discovery, enhance education, and foster scholarly achievement. CEE is focused on personal interactions, ensuring that existing users, potential users, and the general public have sufficient access to materials and have a positive and effective experience with XSEDE public offerings and frontline user support. As such, the CEE Key Performance Indicators are designed to broadly assess this performance. CEE focuses on metrics that quantify how many users in aggregate are benefiting from XSEDE resources and services. Additionally, CEE focuses on how well the user base is sustained over time and how well training offerings evolve with changing user community needs. Key Performance Indicators for CEE are listed in the table below. Additional information about these KPIs can be found on the XSEDE KPIs & Metrics wiki page. For other metrics with respect to this WBS, see Appendix §12.2.2.1.

RY5 IPR12 Page 35 Table 4-1: KPIs for Community Engagement & Enrichment.

Report Sub-goal Area Metric Target RP1 RP2 RP3 RP4 Total Year Supported Number of 2,000/ Deepen/ RY5 2,277 students qtr Extend — benefiting Prepare the 1,500/ from XSEDE RY4 2,210 2,634 1,649 2,252 6,196 current and qtr resources and next services 1,250/ generation through RY3 1,613 1,666 1,145 2,104 2,323 (§3.1.3) qtr training, XSEDE RY2 950/qtr 1,722 1,478 1,170 1,522 1,802 projects, or conference RY1 50/qtr * 997 815 2,679 3,122 attendance Number of Deepen/ RY5 650/qtr 741 under- Extend — represented Prepare the students RY4 500/qtr 674 974 492 753 2,072 current and benefiting next from XSEDE generation resources and RY3 625/qtr 449 438 307 436 1,630 (§3.1.3) services through RY2 475/qtr 488 399 347 423 1,104 training, XSEDE projects, or conference RY1 50 / qtr * 34 33 19 28 attendance Aggregate 4.4 of 5 Deepen/ RY5 4.6 mean rating of /qtr Extend — training Prepare the 4.4 of 5 impact for RY4 4.5 4.3 4.3 4.6 4.4 current and /qtr attendees next registered 4.4 of 5 generation RY3 4.5 4.4 4.5 4.3 4.4 through the /qtr (§3.1.3) portal 4 of 5 RY2 4.3 4.3 4.6 4.5 4.4 /qtr 4 of 5 RY1 * 4.5 4.4 4.3 4.4 /qtr

Number of RY5 340 327 Deepen/ institutions Extend — with a RY4 300 304 305 315 325 325 Deepen use to Champion existing RY3 250 259 266 277 284 284 communities (§3.1.1) RY2 240 218 238 239 246 246

RY1 225 * 224 231 234 234

Percentage of 100 Sustain — RY5 98/qtr user (16/16) Provide requirements excellent user 100 96 100 97 98 RY4 98/qtr (34/34) (23/24) (26/26) (36/37) (119/121)

RY5 IPR12 Page 36 Report Sub-goal Area Metric Target RP1 RP2 RP3 RP4 Total Year Supported support addressed 100/ 89 90 100 100 94 RY3 (§3.3.2) within 30 days qtr (40/45) (47/52) (36/36) (27/27) (150/160) 100/ 78 102 86 93 91 RY2 qtr (36/44) (47/46) (32/37) (41/44) (156/157) 100/ 50 89 75 74 RY1 * qtr (16/32) (40/45) (40/53) (96/130)

Both the number of students and students from underrepresented groups benefiting from XSEDE resources and services continues to exceed the increased quarterly target. The number of students participating in XSEDE EMPOWER (Expert Mentoring Producing Opportunities for Work, Education, and Research), the California State University Los Angeles' (CSULA) NASA-funded Data Intensive Research and Education Center for STEM (NASA Direct STEM) partnership, Advanced Computing for Social Change at PEARC, and Computing4Change at SC is growing. This will be monitored for possible declines, as there are concerns about ‘Zoom fatigue’ as more institutions announce their decisions to deliver their fall semesters online and all conferences, including SC20, will be virtual this fall. The aggregate mean rating of training impact is consistent with prior reporting periods and exceeds its target. The number of institutions with a Campus Champion continues to meet or exceed targets. User Engagement connects with all active PIs every reporting period to ensure their projects are progressing and any issues their teams may encounter are identified and addressed. UE relies on SPs and other areas within XSEDE to engage most issues but, even so, the metric for the current reporting period remains consistent and has met the desired target for the reporting period. CEE Highlights Twenty undergraduate students from 17 institutions were accepted for summer participation in the XSEDE EMPOWER program and began working on projects in a variety of computational science research areas. Half of the students are female; four are underrepresented minorities. Alexandra Ballow, an EMPOWER student from Youngstown State University, presented on her project work at the Scientific Computing with Python Virtual Conference (SciPy 2020) and is preparing a poster for the SC20 conference. CEE Director’s Office (WBS 2.1.1) The CEE Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the CEE area. This oversight includes providing direction to the L3 management team, coordination of, and participation in, CEE planning activities and reports through the area’s Project Manager, and monitoring compliance with budgets, and retarget effort if necessary. The Director’s Office also attends and supports the preparation of project level reviews and activities. The CEE Director’s Office will continue to manage and set the direction for CEE activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls; contribute to the project level plan, schedule, and budget; contribute to XSEDE quarterly, annual, and other reports as required by the NSF; and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area. Workforce Development (WBS 2.1.2) The Workforce Development mission is to provide a continuum of learning resources and services designed to address the needs and requirements of researchers, educators, developers, integrators, and

RY5 IPR12 Page 37 students utilizing advanced digital resources. This includes providing professional development for XSEDE team members. Workforce Development fulfills its mission through an integrated suite of training, education, and student preparation activities to address formal and informal learning about advanced digital resources. Workforce Development provides business and industry with access to XSEDE’s workforce development efforts including training services and student internships that have historically proven beneficial to industry. Workforce Development is comprised of three areas: Training, Education, and Student Preparation. The Training team develops and delivers training programs to enhance the skills of the national open science community and ensure productive use of XSEDE’s cyberinfrastructure. The Education team works closely with Training and Student Preparation to support faculty in all fields of study with their incorporation of advanced digital technology capabilities within the undergraduate and graduate curriculum. The Student Preparation program actively recruits students to use the aforementioned training and education offerings to enable the use of XSEDE resources by undergraduate and graduate students to motivate and prepare them to pursue advanced studies and careers to advance discovery and scholarly studies. The Education team continued their efforts to increase access to computational and data science curriculum. The team held two virtual workshops in collaboration with Broadening Participation this reporting period: Computational Chemistry for Chemistry Educators (CCCE) and Advanced Computing for Social Change (ACSC) Faculty Curriculum Development Workshop. Both workshops had more attendees than previous workshops: 45 for CCCE and 18 for the ACSC Faculty. Additionally, the workshops were updated and altered for online delivery; CCCE used Moodle for materials sharing and ACSC used Google Drive. Twenty undergraduate students from 17 institutions were accepted for summer participation in the XSEDE EMPOWER program and began working on projects in a variety of computational science research areas. Half of the students are female; four are underrepresented minorities. The Campus Champions community, External Relations team, and Broadening Participation team remained instrumental in recruiting well-qualified students and mentors to the program as well as assisting with the application review process. Alexandra Ballow, an EMPOWER student from Youngstown State University, presented on her project work at the Scientific Computing with Python Virtual Conference (SciPy 2020) and is preparing a poster for the SC20 conference. The XSEDE training team delivered training materials on timely topics and current resources in a variety of formats to meet the community’s requirements. This reporting period two multicast training events were offered as webinars due to COVID-19, XSEDE HPC Workshop - MPI22 and XSEDE HPC Workshop - Boot Camp23, and eight training webinars were offered. • 5/14/2020: CMU Deep Learning for Physicists Workshop • 5/21/2020: XSEDE Webinar: Running Jupyter Notebooks on Comet • 5/28/2020: XSEDE Webinar: Python and Performance • 6/17/2020: Writing a Successful XSEDE Allocation Proposal • 6/18/2020: XSEDE Webinar: Introduction to Expanse • 7/1/2020: Writing a Successful XSEDE Allocation Proposal • 7/7/2020: XSEDE Webinar: Python and Performance

22 https://www.psc.edu/hpc-workshop-series/big-data-april-2020 23 https://www.psc.edu/hpc-workshop-series/openacc-march-2020 RY5 IPR12 Page 38 • 7/16/2020: XSEDE Webinar: Indispensable Security: Tips to Use SDSC's HPC Resources Securely Three new badges were developed: XSEDE Data Science with Python Beginner Badge, Lustre I/O Beginner, and Matlab for HPC Beginner Badge; and asynchronous content updated. • May 2020: CVW topic Python for High Performance24 Updated • June 2020: CVW topic Using the Jetstream APIs25 New Topic Added • July 2020: Using the Lustre File System course moved from CI-Tutor to the Moodle site (www.hpc-training.org). Content was updated and a “Lustre I/O Beginner” badge offered. Two training roadmaps were published. • July 2020: Roadmap on “How to run a containerized application” • July 2020: Roadmap on “How can I run an MPI-based parallel program?” Additionally, the XSEDE Workforce Development staff was very active at PEARC20. • Victor Eijkhout - Tutorial: An Introduction to Advanced Features in MPI, https://sched.co/cnSg (Lead presenter) • Victor Eijkhout - Paper: Implementing a Prototype System for 3D Reconstruction of Compressible Flow (Co-presenter) • Kate Cahill and Susan Mehringer - Workshop: Strategies for Enhancing HPC Education and Training, https://sched.co/cnTP • Susan Mehringer - Paper: Using Containers to Create More Interactive Online Training and Education Materials (Co-presenter) • Susan Mehringer - BoF: User Training and Engagement in Scientific Computing (Panelist) • Susan Mehringer - Workshop: What Does it Mean to be a Campus Champion? (Presenting “XSEDE Training”) For other metrics with respect to this WBS, see Appendix §12.2.2.1.1 User Engagement (WBS 2.1.3) The mission of the User Engagement (UE) team is to capture community needs, requirements, and recommendations for improvements to XSEDE’s resources and services, and to report to the national community how their feedback is being addressed. XSEDE places an emphasis on maintaining consistent user contact, traceability in tracking user issues, and closing the feedback loop. UE continues to connect with all active PIs quarterly to ensure their projects are progressing and any issues their teams may be encountering are identified and addressed. In the current reporting period, inquiries were sent to 1,217 unique PIs: 62 responses were received, 16 issues were identified, and all 16 (100%) of these issues were addressed within 30 days. The goal is to completely address all issues within the reporting period, but UE relies on SPs and other areas within XSEDE to engage most issues. This KPI metric value for the current reporting period is consistent and has met the desired target for the quarter. For other metrics with respect to this WBS, see Appendix §12.2.2.1.2.

24 https://cvw.cac.cornell.edu/python/ 25 https://cvw.cac.cornell.edu/jetstreamapi/ RY5 IPR12 Page 39 Broadening Participation (WBS 2.1.4) Broadening Participation’s mission is to engage underrepresented minority researchers from domains that are not traditional users of HPC and from Minority Serving Institutions. This target audience ranges from potential users with no computational experience to computationally savvy researchers, educators, Champions, and administrators who will promote change at their institutions for increased use of advanced digital services for research and teaching. Broadening Participation will continue the most effective recruitment activities - conference exhibiting, campus visits, and regional workshops - while increasing national impact through new partnerships and the utilization of lower cost awareness strategies to continue the growth in new users from underrepresented communities. The Diversity Forum and the Minority Research Community listservs and community calls focus on user persistence in their use of XSEDE services and their deepening engagement through participation in committees such as the User Advisory Committee (UAC) and XSEDE Resource Allocations Committee (XRAC), and participation in Champions, Campus Bridging, and other programs. Persistent institutional engagement is enabled by curriculum reform and larger numbers of researchers adopting the use of advanced digital resources as a standard research method. The virtual instances of the Advanced Computing for Social Change (ACSC) workshops for faculty and students were successfully offered in July. The faculty workshop was one week with a mixed schedule of training, small group working sessions, and hands-on practice using tools on the desktop and in the Jetstream Cloud environment. Eighteen faculty participated with more than 50% representing humanities and social sciences. Seventy percent of the attendees were from the Atlanta University Center Data Science Initiative. Twenty-two courses were updated with social justice data science exercises; nineteen courses will be offered this fall and three courses offered in spring 2021. One course, a general education data science course, is being developed for the Atlanta University Center Data Science Initiative and will be offered to all 8,000 undergraduate students at Clark Atlanta University, Morehouse College, and Spelman College. Several of the courses will use Jetstream for student exercises. The courses updated included: Africana Feminist Theory, C++, Econometrics, Intro to Asian Studies, and Urban Politics and Policy. The workshop wrapped up with presentations on their updated course plans. Both the ACSC faculty and student workshops included XSEDE New User and R training. The faculty training used Jetstream, and the student workshop used Stampede2 and the TACC Visualization portal. The student one-week experience was modeled on the SPICE Data Science Institute and the agenda was organized to accommodate participants across different time zones, including Guam and Hawaii. The 19 undergraduate participants chose social issues to explore via data analysis and wrapped up with recorded presentations that provided the problem, hypothesis, software tools and data sources, findings, and conclusions. Nine peer mentors selected from prior workshops contributed to the students’ positive experience. The XSEDE External Evaluation team conducted observations, focus groups, and surveys of both workshops. Planning and preparation is underway for participation and sponsorships at the ACM Tapia Celebration of Diversity, ACM Grace Hopper Celebration of Women in Computing, the Society for the Advancement of Chicanos and Native Americans (SACNAS) National Conference, and Computing4Change at SC20. This planning and preparation aims to ensure maximum impact at these events now being offered virtually. All of these events have been rich sources of highly qualified student applicants for ACSC, C4C, and EMPOWER. For other metrics with respect to this WBS, see Appendix §12.2.2.1.3. User Interfaces & Online Information (WBS 2.1.5) User Interfaces & Online Information (UII) is committed to enabling the discovery, understanding, and effective utilization of XSEDE’s powerful capabilities and services. Through UII’s ongoing effort to RY5 IPR12 Page 40 improve and engage a variety of audiences via the XSEDE website and User Portal, UII has an immediate impact on a variety of stakeholders including the general public, potential and current users, educators, service providers, campus affiliates, and funding agencies. These stakeholders will gain valuable information about XSEDE through an information-rich website, the XSEDE User Portal, and a uniform set of user documentation. The User Interfaces and Online Information team continued regular improvements and maintenance on the website and user portal. This includes regular user guide updates and user documentation improvements. The team is actively completing development and testing of ORCID integration for publications and improving the account creation flow. These two features are expected to be in production during the next reporting period. This reporting period the User Survey data was submitted, and the results revealed that the website was rated as 4.3 (out of 5), the user portal was rated 4.3 (out of 5), and the online technical documentation was 4.2 (out of 5). All of these are rated above the target of 4 and exceed expectations. These areas have always been rated high in terms of satisfaction expressed by users, but, while all of the ratings trended upward, the increases over last year were negligible. As a result, the targets will continue to be monitored. For other metrics with respect to this WBS, see Appendix §12.2.2.1.4. Campus Engagement (WBS 2.1.6) The Campus Engagement program promotes and facilitates the effective participation of a diverse national community of campuses in the application of advanced digital resources and services to accelerate discovery, enhance education, and foster scholarly achievement. Campus Engagement, via the Campus Champions, works directly with institutions across the U.S. both to facilitate computing and data-intensive research and education, nationally and with collaborators worldwide, and to expand the scale, scope, ambition, and impact of these endeavors. This is done by increasing scalable, sustainable institutional uptake of advanced digital services from providers at all levels (workgroup, institutional, regional, national, and international), fostering a broader, deeper, more agile, more sustainable and more diverse nationwide cyberinfrastructure ecosystem across all levels, and cultivating inter-institutional interchange of resources, expertise, and support. Campus Engagement also aims to assist with the establishment and expansion of consortia (e.g., intra-state, regional, domain-specific) that collaborate to better serve the needs of their advanced computing stakeholders. Summer has been a busy time for Campus Engagement, especially in light of the effects of COVID-19 on the community. While many of the regular CE activities have always been virtual, others have had to pivot to a virtual format. Campus Champions, Virtual Residents, and the CaRCC People Network shared with each other how they were changing their practices to support computational and data-intensive research remotely. Many reported an increase in demand for support as researchers moved to more digital research when they were unable to be in their labs. As always, the pinnacle of the Campus Engagement year is PEARC. Campus Champions, whose registrations were covered from XSEDE participant support, attended in record numbers. Early counts indicate over 250 Champions participated. Campus Champions are also well-represented on the PEARC20 committee, paper presenters, tutorials and workshops organizers, posters, BoFs, etc. Of special interest this year at PEARC, Campus Champions Leadership team member Shelley Knuth led a group of Champions that developed and held an Onboarding workshop intended for new and experienced Champions looking for more information about what it means to be an effective Champion at their respective institutions. They held talks on creating an allocation, available training, and resources and tools that would be of interest to Champions. They also held an interesting session with examples from Campus Champions discussing how they work effectively in the role at their individual institutions. With participation around 50 people throughout the day, organizers found this to be a

RY5 IPR12 Page 41 useful and interesting discussion, as evidenced by the feedback received. The Campus Champions Onboarding group plans to take this information and make it more available to the Champions via various avenues, including the web and during quarterly Community Chat sessions. Earlier in the summer (June 1-5), the Virtual Residency Program (VRP) 2020 workshop also went virtual, holding four 90-minute sessions each day for a week. The 2020 VRP workshop had 430 attendees from 225 institutions in 49 U.S. states, three U.S. territories, and seven other countries. This brought the total number of participants in all Virtual Residency activities to 924, which included 370 institutions from every U.S. state, three U.S. territories, and 11 other countries, which included: 56 Minority Serving Institutions (15% of VRP institutions); 94 non-PhD-granting institutions (25%), which included three high schools and five community colleges; 101 institutions (27%) in 27 of the 28 Established Program to Stimulate Competitive Research (EPSCoR) jurisdictions; 243 of the 370 VRP institutions are Campus Champion institutions (75% of Campus Champion institutions, 66% of VRP institutions); 119 of 141 Campus Research Computing Consortium (CaRCC) Researcher-Facing Track institutions (84% of CaRCC Researcher-Facing Track institutions, 32% of VRP institutions). In 2020, the XSEDE evaluation team (Lorna Rivera, Lizanne DeStefano) conducted the first ever evaluation of a VRP workshop. Results included: (1) Demographics: (a) Women: the VRP's 2020 workshop had 30% women, considerably lower than the U.S. population (51% women), but more than the 26% reported for all computing/IT occupations nationwide, and more than double the 13-14% reported for the SC15-17 supercomputing conferences. (b) Underrepresented Minorities (URM): the VRP's 2020 workshop had 21% URM, considerably lower than the U.S. population (34% URM), but more than double the 10% reported for all computing/IT occupations nationwide. (2) Satisfaction: Sessions at the VRP's 2020 workshop were rated 3.90 to 4.42 on a 1 to 5 scale. Underrepresented Minorities reported modestly higher satisfaction than others, including reporting their 2020 workshop experience to be 5% more successful than others (4.75 vs. 4.52), the session Google Docs to be 13% more useful (4.58 vs 4.07), and two of the 20 sessions to be 12% higher value/quality (4.71 vs. 4.19, 4.71 vs. 4.20). Women reported one session to be 10% lower value/quality than men (3.85 vs. 4.27). (No other statistically significant population differences were discovered.) In addition, the VRP team uses an indirect measure of the value of the VRP as follows: of the 330 institutions that started participating in the Virtual Residency Program before the 2020 workshop (and therefore could have participated in multiple activities), 257 institutions (78%) have participated in multiple VRP activities, and 237 institutions (72%) have participated in multiple types of VRP activities (workshops, workshop planning calls, grant proposal writing apprenticeship, paper writing apprenticeship). The continual and repeat participation indicates that they consider the VRP to be valuable. Also this summer, the Champions continued their new tradition of electing half of the leadership team and is excited to welcome one new member to their leadership team: Tom Cheatham from the University of Utah, and three re-elected members: Douglas Jennewein of Arizona State University, Julie Ma of the Massachusetts Green High Performance Computing Center, and Timothy Middelkoop of the University of Missouri. They will serve a two-year term. Joining these newly-elected members for the second year of their two-year term on the Leadership Team will be: Shelley Knuth, University of Colorado; Torey Battelle, Colorado School of Mines; and BJ Lougee of the Federal Reserve Bank of Kansas. Special thanks to outgoing Leadership Team member Hussein Al-Azzawi of the University of New Mexico for serving on the team. For other metrics with respect to this WBS, see Appendix §12.2.2.1.5.

RY5 IPR12 Page 42 5. Extended Collaborative Support Service (WBS 2.2) The Extended Collaborative Support Service (ECSS) improves the productivity of the XSEDE user community through meaningful collaborations and well-planned training activities. The objective is to optimize applications, improve work and data flows, increase effective use of the XSEDE digital infrastructure, and broadly expand the XSEDE user base by engaging members of underrepresented communities and domain areas. The ECSS program provides professionals who can be part of a collaborative team—dedicated staff who develop deep, collaborative relationships with XSEDE users— helping them make the best use of XSEDE resources to advance their work. These professionals possess combined expertise in many fields of computational science and engineering. They have a deep knowledge of underlying computer systems and of the design and implementation principles for optimally mapping scientific problems, codes, and middleware to these resources. ECSS includes experts in not just the traditional use of advanced computing systems but also in data-intensive work, workflow engineering, and the enhancement of scientific gateways. ECSS projects fall into five categories: Extended Support for Research Teams (ESRT), Novel and Innovative Projects (NIP), Extended Support for Community Codes (ESCC), Extended Support for Science Gateways (ESSGW), and Extended Support for Training, Education and Outreach (ESTEO). Project-based ECSS support is requested by researchers via the XSEDE peer-review allocation process, or, in some cases, suggested by reviewers as something that would benefit the researchers. If reviewers recommend support and if staff resources are available, projects progress through three activities. First, the project is assigned to an ECSS expert. Second, the project is quantified with the formation of a work plan through collaboration with the research group. The work plan includes concrete quarterly goals and staffing commitments from both the PI team and ECSS. Third, when the project is completed, the ECSS expert produces a final report with input from the research group. A successful project is the completion of all three phases. Each state of the progression is measured to provide an assessment of progress. Submission of work plans within 45 days of initial contact, 90% of projects with work plans completed, and 85% of completed projects with final reports within three months are additional criteria for success. The ECSS managers review work plans and also track progress via Interim Project Reports. The success of the ECSS team depends on effective collaboration across all L2 areas of the project. Specifically, ECSS works closely with XCI to expand software capabilities; External Relations within PgO to communicate the science successes enabled by ECSS assistance; RAS to review allocations requests; and CEE to develop and deliver training in HPC, data intensive computing, effective use of XSEDE resources and other topics. In addition, ECSS partners with CEE to manage the Campus Champions Fellows program, which can involve mentors from any L2 area. Key Performance Indicators for Extended Collaborative Support Service are listed in the table below. Additional information about these KPIs can be found on the XSEDE KPIs & Metrics wiki page. For other metrics with respect to this WBS, see Appendix §12.2.2.2.

Table 5-1: KPIs for Extended Collaborative Support Service.

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Number of RY5 45/yr 17 Deepen/Extend completed ECSS — Deepen use to projects RY4 45/yr 15 11 16 10 52 existing communities

RY3 45/yr 17 10 12 8 47 (§3.1.1)

RY2 50/yr 16 9 10 12 47

RY5 IPR12 Page 43 Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

RY1 50/yr * 10 13 25 48

Aggregate mean RY5 4 of 5/yr 4.1 Deepen/Extend rating of ECSS — Deepen use to impact by PIs RY4 4 of 5/yr 3.0 4.7 4.3 4.7 4.5 existing communities RY3 4 of 5/yr NA 3.9 4.4 4.3 4.2 (§3.1.1)

RY2 4 of 5/yr 4.1 4.0 3.8 4.3 4.0

4 of 5 RY1 * 4.6 4.6 3.3 4.1 /qtr

Aggregate mean 4.5 of 5/ Deepen/Extend RY5 4.8 rating of PI yr — Deepen use to satisfaction with existing 4.5 of 5/ ECSS support RY4 3.0 5.0 5.0 4.7 4.8 communities yr (§3.1.1) 4.5 of 5/ RY3 NA 4.3 4.8 4.5 4.6 yr 4.5 of 5/ RY2 4.7 4.6 4.2 4.8 4.5 yr 4.5 of 5/ RY1 * 4.9 4.7 4.6 4.5 qtr NA – Interviews were not conducted during this reporting period so no Impact and Satisfaction ratings were collected. ECSS continues to support the research community at a high level as demonstrated by quarterly metrics for the number of completed projects, impact, and satisfaction that are on track to meet or exceed annual goals. The large number of projects completed (17) is consistent with values reported in RP1 from the previous four years and may reflect a correction from the typically low numbers for RP4. ECSS Highlights Campus Champion Fellows: ECSS has selected six Fellows for 2020-21, which includes one Fellow who was accepted in the previous year but had to defer for personal reasons. While the Fellows program had expanded in scope over time to include XCI, Workforce Development, and Fellows- designed projects, this year all six Fellows chose to work with ECSS. ECSS Symposia: Attendance at monthly ECSS symposia grew significantly over the past several months and now routinely tops 100 participants. At this rate, it’s likely that ECSS will triple the annual goal of 300 participants/year. This increase in growth is due in part to successful, ongoing collaborations between ECSS and Rudi Eigenmann’s (University of Delaware) NSF-funded Xpert Network project, which has the goal of “Exchanging Best Practices and Tools for Computational and Data-Intensive Research,” CaRCC (Campus Research Computing Consortium), and Campus Champions. Further highlights can be found in the sections for each of the ECSS L3 areas. ECSS Director’s Office (WBS 2.2.1) The ECSS Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the ECSS area. This oversight includes providing direction to the L3 management team, coordination of and participation in ECSS planning activities and reports through the area’s Project Manager, monitoring compliance with budgets, and retargeting effort, if necessary. The Director’s Office also attends and supports the preparation of project-level reviews and activities.

RY5 IPR12 Page 44 The ECSS Director’s Office will continue to manage and set direction for ECSS activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls; contribute to the project level plan, schedule, and budget; contribute to XSEDE quarterly, annual, and other reports as required by the NSF; and attend XSEDE quarterly and annual meetings. The Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area. The office consists of two Level 2 Co-Directors, Philip Blood, who manages ESRT and NIP activities, and Bob Sinkovits, who manages ESCC, ESSGW, and ESTEO activities. The office also has three project managers (Marques Bland, Sonia Nayak, and Leslie Morsek). Blood and Sinkovits carry out the post-project interviews with all project PIs who have received ECSS support, both to get their assessment of how the project went, and to hear and act on any concerns they may express. Sinkovits also organizes the monthly symposium series, serves as one of the contributors to staff training, and runs the Campus Champions Fellows program (§4.6). Blood convenes User Advisory Committee meetings and supports the User Advisory Committee Chair. The project managers aid in the management of the day-to-day activities of ECSS, which includes the management of project requests (XRAC and startups), active projects, project assignments, and staffing. They continuously refine the ECSS project lifecycle, further defining processes to aid in the management of over 100 active projects. They also administer Jira for the management and tracking of projects, both for the managers and directors of ECSS and for ECSS staff. Extended Support for Research Teams (WBS 2.2.2) Extended Support for Research Teams (ESRT) accelerates scientific discovery by collaborating with researchers, engineers, and scholars to optimize their application codes, improve their work and data flows, and increase the effectiveness of their use of XSEDE digital infrastructure. ESRT projects are initiated as a result of support requests or recommendations obtained during the allocation process. Most projects focus on research codes associated with specific research teams, as community codes fall under ESCC (§5.4), but are not exclusively restricted to this classification. The primary mandate of ESRT is the support of individual research teams within the context of their research goals. ESRT is on track to meet its metric of 27 projects per year. During this reporting period, 5 projects have been completed, which is below the average of 6-7 projects needed each reporting period to meet the annual metric. However, ESRT expects to make up this difference in future reporting periods given the 28 currently active projects. ESRT has completed two PI interviews this quarter resulting in a high average satisfaction and impact scores, 5.0 in impact and 4.5 in satisfaction, as well as an average of 12 months of effort saved (1 respondent). These results meet or surpass the current goals of 4.0 and 4.5, respectively, for impact and satisfaction. Given similar results in these metrics last RP, it is not necessary to change these goals at this time. Highlights for ESRT include: A Performant Matrix of Pearson's Correlation Coefficient (MPCC) Calculations with Support for Missing Data on Emerging HPC Architectures:

RY5 IPR12 Page 45 A service for web-based genetics, GeneNetwork, routinely performs Pearson Correlation Coefficient (PCC) calculations to find relationships between and among genotypes and phenotypes (see Figure 12). With the rapidly growing data acquired from sequencing, combined with a growing body of phenotype data from medical and experimental biology, computing correlations is a recurring bottleneck, particularly in the presence of incomplete data. A team led by Figure 12: Genotype to genotype correlation between genetic markers on Dr. Pjotr Prins at the University chromosome 1 of the BxD family. Pairwise correlation between all 7,321 of Tennessee Health Science markers was ~6 times faster using MPCC. Computation was performed on a Center is developing a high- Intel(R) Xeon(R) CPU E5-2680 v4@ 2.40GHz. performance Matrix Pearson Correlation Coefficient (MPCC) algorithm to reduce these performance bottlenecks. The algorithmic redesign focused on bundling vectors into matrices and computing dot-products in bulk as general matrix-matrix multiplications (GEMMs). The incomplete data handling replaces an O(n3) conditional statement with an O(n2) conditional and a GEMM. These GEMMs are handled by importing highly efficient architecturally optimized libraries (MKL, cuBLAS) to get cross platform performance. While both algorithms have O(n3) complexity, performance comparisons between the original and improved algorithms show gains due to increased arithmetic intensity (7x increase) and efficient use of memory. ECSS ESRT consultants Dr. Chad Burdyshaw and Dr. Glenn Brook have optimized this MPCC algorithm on Intel Xeon Gold 6148 (Skylake) processors to achieve 4.3 TFlop/s in single precision (77% of theoretical peak). For large square matrices MPCC outperforms the existing PCC calculation utilizing the R cor function by 200-fold. This algorithm was originally targeted for the Stampede2 Skylake nodes, but cross compilation for GPUs and their smaller local memory has shifted focus to streaming segments of matrices too large to be held in memory. For other metrics with respect to this WBS, see Appendix §12.2.2.2.1. Novel & Innovative Projects (WBS 2.2.3) Novel and Innovative Projects (NIP) accelerates research, scholarship, and education provided by new communities that can strongly benefit from the use of XSEDE’s ecosystem of advanced digital services. Working closely with the XSEDE Outreach team, the NIP team identifies a subset of scientists, scholars, and educators from new communities, i.e., from disciplines or demographics that have not yet made significant use of advanced computing infrastructure, who are now committed to projects that appear to require XSEDE services, and are in a good position to use them efficiently. NIP staff then provide personal mentoring to these projects, helping them to obtain XSEDE allocations and use them successfully. XSEDE projects generated by, and mentored by, the personal efforts of the NIP experts should stimulate additional practitioners in their field to become interested in XSEDE. Strategies used include building

RY5 IPR12 Page 46 and promoting science gateways serving communities of end-users and the enhancement of the Domain Champions program by which successful practitioners spread the word about the benefits of XSEDE to their colleagues. NIP will continue to focus on generating and mentoring projects that have a strong data analytics component. Highlights of recent activities include: Image analysis for digital surrogates of historical motion picture film: PI Greg Wilsbacher, University of South Carolina. NIP expert Alan Craig (Shodor) worked with this multidisciplinary team to generate a startup project on Jetstream at Indiana University and is developing an ECSS work plan with them. For almost a century, celluloid-based imagery was the dominant medium for recording the history of the world, creating a global library of still and moving images. The archival digitization of motion picture film is still in a developmental phase because it is derived from a complex system and results in data intensive files. But given the rise of deep fake technology it is essential that mature systems are developed soon. Moving Image Research Collections (MIRC) at the University of South Carolina recently entered into a partnership with the United States Marine Corps History Division to preserve, digitize and make accessible the legacy 16mm and 35mm film collection housed at Marine Corps University, Quantico. The collection is very large, containing over 18,000 cans of film (a typical can contains 7 to 8 minutes of footage). It includes, for example, a curated collection of archival footage documenting the Civil Rights movement, and the long struggle for equality and social justice in South Carolina and beyond (see Figure 13). This collection has a high research value for historians and is of sufficient size to create a data set able to train image analysis algorithms. MIRC seeks to identify new methods for deploying these digital film assets as trusted historical resources (in contrast to the chaos of user- contributed online video). To accomplish this, MIRC is partnering with the university’s Computer Vision Lab and its Research Computing unit to develop and deploy three initial projects: Machine Learning (ML) algorithms for identifying and tracking textual information in historical imagery; ML algorithms for facial recognition in historical imagery; and a new method for certifying the chain of Figure 13: In February 1961, John Lewis came to Rock Hill, SC in support of the historical provenance from a “Friendship Nine,” jailed for 30 days after a sit-in. (University of South Carolina celluloid film to a master digital Center for Civil Rights History & Research) surrogate copy, and then to all subsequent copies derived from that master. Using XSEDE and ECSS, MIRC seeks to build a virtual home that not only hosts the online collections for the public, but also allows researchers and developers to collaborate with others and experiment with mechanisms for image and video analysis projects. For other metrics with respect to this WBS, see Appendix §12.2.2.2.2

RY5 IPR12 Page 47 Extended Support for Community Codes (WBS 2.2.4) Extended Support for Community Codes (ESCC) extends the use of XSEDE resources by collaborating with researchers and community code developers to deploy, harden, and optimize software systems necessary for research communities to create new knowledge. ESCC supports users via requested projects and XSEDE-initiated projects. ESCC projects may be created in two different ways. Most ESCC projects are initiated as a result of requests for assistance during the allocation process. These projects are similar in nature to ESRT projects but involve community codes rather than codes developed for and by individual research groups. ESCC projects may also be initiated by staff to support a community’s needs. ESCC is on target to meet its goal for the number of completed projects. Also, ESCC earned a satisfaction score of 5.0 which contributes positively to the overall ECSS KPI. However, the impact score was 2.0, which falls short of the goal. One project, while having a high score for satisfaction, received a low impact score. The project started out with problems during software install and subsequent profiling with Tuning and Analysis Utilities (TAU). Hence the list of accomplishments was short. Nevertheless one slow function was identified and progress was made. ESCC highlights include: ECSS consultant, B. Vanderwende, made excellent progress with PI Y. Zhang from Northwestern University. In this ongoing project a special version of the widely used WRF weather forecast software (WRF-Chem/ROMS, which includes explicit air-sea interactions) was debugged and successfully installed and used on XSEDE resources. The ECSS consultant identified the cause of run-time segmentation faults to be coming from mistakes in recent code changes. In this particular case, inconsistencies in the interplay of Fortran77 and Fortran90 calls and subtle language differences caused problems that were ‘mysterious’ to the PI and his team. The project demonstrates the value of the ECSS program in the area of classical HPC. Problems that may mystify even seasoned PIs become tractable with the help of a knowledgeable ECSS team member. For other metrics with respect to this WBS, see Appendix §12.2.2.2.3. Extended Support for Science Gateways (WBS 2.2.5) Extended Support for Science Gateways (ESSGW) broadens science impact and accelerates scientific discovery by collaborating in the development and enhancement of science-centric gateway interfaces and by fostering a science gateway community ecosystem. ESSGW continued to work at capacity during this reporting period. ESSGW completed 10 projects, which exceeds the yearly goal of 9, in the first reporting period. This heavy workload did not negatively impact customer service metrics, maintaining a 5.0 satisfaction rating for a goal of 4.5 and a 4.5 impact rating for a goal of 4.0. With 17 currently active projects, ESSGW will continue to work near if not above capacity during the next reporting period. If new projects continue to be assigned to ESSGW at the current rate, ESSGW will need to reach out to other ECSS areas for additional support, create a waitlist,

RY5 IPR12 Page 48 or increase effort for ESSGW going forward. This growing workload speaks to the success and popularity of Science Gateways within the XSEDE research community. The team will continue to monitor the number of completed projects and will consider adjusting the goal in the future if the trend continues. Recently concluded projects span many disciplines including the areas of land and food management, energy and water resources, text analysis, identifying fake news, archeology, neuroimaging, and RNA characterization. New projects focused on Alzheimer’s Disease Drug Discovery, Geoscience, Molecular Figure 14: Word balloon derived by Distant Reader from keywords in a Physics, and Energy Modeling were dataset about COVID-19. also started during this period. Highlights for ESSGW include: COVID-19 Text Analysis on the Distant Reader Gateway The Distant Reader Gateway (https://distantreader.org/) was created to help users browse and understand vast amounts of textual information. During the ongoing worldwide health crisis, ESSGW consultants Eroma Abeysinghe and Eric Coulter worked with PI Eric Morgan to enhance the gateway with additional datasets focused on the COVID-19 research literature. This enhancement will allow researchers to scan and draw understanding from the vast amount of published COVID research happening quickly. This project is an example of a project provided access via the COVID-19 HPC Consortium. Figure 14 shows a word balloon that was derived by Distant Reader from keywords in a dataset described as: “In response to the COVID-19 pandemic, the Allen Institute for AI has partnered with leading research groups to prepare and distribute the COVID-19 Open Research Dataset (CORD-19), a free resource of over 29,000 scholarly articles, including over 13,000 with full text, about COVID-19 and the coronavirus family of viruses for use by the global research community.” More details about the text analysis of this dataset are available here.26 For other metrics with respect to this WBS, see Appendix §12.2.2.2.4. Extended Support for Training, Education, & Outreach (WBS 2.2.6) Extended Support for Training, Education & Outreach (ESTEO) prepares the current and next generation of researchers, engineers, and scholars in the use of advanced digital technologies by providing the technical support for Training, Education, and Outreach planned activities. Typical events include train-the-trainers events, on-site classes requested by Campus Champions, regional workshops, conferences, and summer schools (including the International HPC Summer School). Staff also create and review online documentation and training modules. This on-demand

26 https://carrels.distantreader.org/library/covid-19/ RY5 IPR12 Page 49 training is increasingly popular with the user community when both time and travel budgets are limited. ESTEO is on track to meet its targets for all of its metrics, including the staff training with the addition of new Service Providers Expanse and Bridges2, with the need for in-depth training for ECSS staff as these resources are brought into production. Highlights for ESTEO include: ESTEO continued collaborations with CEE-Broadening Participation and CEE-Workforce Development to build on the success with virtual live training at Cal State LA to deliver virtual training for the SPICE Data Science Institute and the Advanced Computing for Social Change faculty and student workshops. Each instance has built on the lessons learned from the previous session, including providing material in advance of the session, shorter live sessions, smaller group sessions, and narrowing the scope of the training to material directly relevant to each workshop. One Campus Champion Fellow, Helen Kershaw of Brown University, who chose ECSS projects as her focus, participated in two projects during her fellowship. In the second project, in collaboration with ECSS mentor Yang Wang, she focused on developing a Jupyter notebook for running MuST on the Jetstream cloud. MuST is an ab-initio electronic structure package for the study of disordered materials that is supported by NSF/CSSI. The Jupyter notebook was central to a two-hour tutorial on the use of MuST, developed as part of the Campus Champion Fellowship. ESTEO will look to add this tutorial to its set of training offerings, as a persistent outcome from the fellow project. ESTEO also managed the review and selection process for the new round of Fellows. Finally, ESTEO team member Amit Chourasia served as Awards Chair for PEARC20, further establishing excellence in the practices for the PEARC conference series. For other metrics with respect to this WBS, see Appendix §12.2.2.2.5.

RY5 IPR12 Page 50 6. XSEDE Cyberinfrastructure Integration (WBS 2.3) The mission of XSEDE Cyberinfrastructure Integration (XCI) is to facilitate interaction, sharing, interoperability, and compatibility of all relevant software and related services across the national CI community, building and improving upon the foundational efforts of XSEDE. XCI envisions a national cyberinfrastructure that is consistent, straightforward to understand, and practical for use by researchers and students. Service to XSEDE Service Providers (SPs) is a particularly important aspect of XCI’s activities. XCI strives to make it possible for researchers and students to effortlessly use computational and data analysis resources ranging from those allocated by XSEDE to campus-based CI facilities, an individual’s workstation, and commercial cloud providers, and to interact with these resources via CI software services such as science gateways and Globus Online. XCI provides two essential integrating services: XCI provides the software glue that ties XSEDE together; particularly, it enables the interoperability of advanced computing resources supported by XSEDE with each other and with the XSEDE portal and other underlying infrastructure (e.g., accounting information), and XCI also improves the capabilities of campus cyberinfrastructure administrators anywhere in the US to manage local facilities in ways that are easily interoperable with the evolving national CI fabric while simultaneously leveraging training and educational materials created and disseminated by XSEDE. The success of the XCI team depends on effective collaboration across all L2 areas of the project. Specifically, the Requirements Analysis & Capability Delivery (RACD) team relies on Ops to integrate new capabilities as well as CEE, RAS, and the PgO’s External Relations (ER) team to help improve XSEDE services and inform the user community of their existence. The Cyberinfrastructure Resource Integration (CRI) team also collaborates with ER to communicate the tools and services which XSEDE makes available to the national CI community. Key Performance Indicators for XSEDE Cyberinfrastructure Integration are listed in the table below. Additional information about these KPIs can be found on the XSEDE KPIs & Metrics wiki page. For other metrics with respect to this WBS, see Appendix §12.2.2.3.

Table 6-1: KPIs for XSEDE Cyberinfrastructure Integration (XCI).

Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Aggregate mean rating RY5 4 of 5/yr 4.4 Advance — of user satisfaction with Create an open XCI software and and evolving e- RY4 * * * * * * technical services, infrastructure capabilities, and (§3.2.1) RY3 * * * * * * resources RY2 * * * * * *

RY1 * * * * * *

Aggregate mean rating RY5 4 of 5/yr 4.5 Advance — of Service Provider Create an open satisfaction with XCI RY4 * * * * * * and evolving e- software and technical infrastructure services, capabilities, RY3 * * * * * * (§3.2.1) and resources RY2 * * * * * *

RY1 * * * * * *

RY5 18/yr 14

RY5 IPR12 Page 51 Report KPI Target RP1 RP2 RP3 RP4 Total Owner Year

Number of non-XSEDE RY4 12/yr 9 10 11 21 21 Advance — partnerships with XCI2 Create an open RY3 8/yr 13 12 11 12 201 and evolving e- infrastructure RY2 NA - - - - 8 (§3.2.1)

RY1 NA * - - - 8

Mean time to issue RY5 10/qtr 4.2 Sustain — resolution (days) Provide excellent RY4 10/qtr 5.4 4.5 8.8 35.8 13.7 user support (§3.3.2) RY3 14/qtr 22.3 7.3 2.9 2.9 8.8

RY2 <30 days 4.0 7.0 8.0 5.0 6.1

RY1 <45 days * 7.0 4.0 16.0 9.0

- Data reported annually. 1 Engagements often continue over multiple reporting periods and get counted in each period they are active. The annual total is calculated by the number of unique engagements across all reporting periods. 2 Number of non-XSEDE partnerships with XCI was a new KPI in RY3 so there were no targets set for RY1 or RY2. Annual totals were calculated retroactively.

Beginning with the 2020 XSEDE Annual Satisfaction Survey XCI started collecting user and service provider satisfaction ratings for software, technical services, capabilities, and resources. Both KPIs exceeded their annual targets. Given that these are new KPIs with new User Survey items informing them, the evaluation team will continue to monitor their performance and determine next year if the targets should be raised. XCI made good quarterly progress on the partnership KPI. The team met its goal for issue resolution in under 10 days on average. XCI Highlights During this reporting period, XCI released a significant new version of Information Publishing Framework (IPF27 v1.5) tool to SPs enabling more fine grained publishing of batch scheduler partition information. XCI also presented the following three posters and associated papers at PEARC20: Use Case Methodology in XSEDE System Integration, Secure XSEDE Information APIs, and SciTokens SSH: Token-based Authentication for Remote Login to Scientific Computing Environments. In addition, XCI contributed to leading a PEARC20 tutorial for use of containerized applications and virtual cluster systems on the Jetstream resource and led a PEARC20 panel session discussing campus integration to the national cyberinfrastructure environment. XCI Director’s Office (WBS 2.3.1) The XCI Director’s Office has been established to provide necessary oversight to ensure the greatest efficiency and effectiveness of the XCI area. This oversight includes providing direction to the L3 management team, coordination of and participation in XCI planning activities and reports through the

27 The IPF is a tool used by resource operators to publish dynamic HPC, HTC, Visualization, Storage, or Cloud resource information to XSEDE's Information Services. RY5 IPR12 Page 52 area’s project manager, and monitoring compliance with budgets, retargeting effort, if necessary. The Director’s Office also attends and supports the preparation of project-level reviews and activities. The XCI Director’s Office will continue to manage and set direction for XCI activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls; contribute to the project level plan, schedule, and budget; contribute to XSEDE IPRs, annual reports, and other reports as required by the NSF; and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area. Requirements Analysis & Capability Delivery (WBS 2.3.2) The Requirements Analysis & Capability Delivery (RACD) team facilitates the integration, maintenance, and support of cyberinfrastructure capabilities addressing user technical requirements. The process begins by preparing Capability Delivery Plans (CDPs) that describe the technical gaps in XSEDE’s prioritized Use Cases. To fill the gaps, RACD evaluates and/or tests existing software solutions, engages with software providers, and facilitates software and service integration. To ensure software and service adoption and ROI, RACD involves users, Service Providers (SPs), and operators in an integration process that uses engineering best practices and instruments components to measure usage. Once components are integrated, RACD facilitates software maintenance and enhancements in response to evolving user needs and an evolving infrastructure environment. RACD delivered one significant new capability to production this reporting period: the ability to discover container registries available in the community (RC-06) through the Research Software Portal. At PEARC20, RACD presented the following three posters and associated poster papers: Use Case Methodology in XSEDE System Integration, Secure XSEDE Information APIs, and SciTokens SSH: Token- based Authentication for Remote Login to Scientific Computing Environments. Although the group did not deliver any new capabilities during the summer, RACD has enough capabilities in the pipeline to reach the goal of 110 capabilities for the program year. RACD’s significant fixes and enhancements to production components this IPR (12 total) included: a significant new IPF 1.5 tool to SPs enabling more fine grained publishing of batch scheduler partition information and replacing legacy Python 2.7 support with newer Python 3.x support, a new Information Services view advertising the availability of Community Software Areas (CSAs) on XSEDE resources, publishing Jetstream cloud images into the Research Software Portal software discovery catalog, several improvements for handling Python dependencies in Information Services, improved Information Services warehouse backup procedures, upgraded to Globus ID Explorer version 1.1 which is used by developers that integrate their services with XSEDE’s identity services, streamlined the forms used to grant new users access to the Research Software Portal, and secured RACD-operated web services by disabling old less secure TLS 1.0 and 1.1 encryption algorithms. Beginning with the 2020 XSEDE Annual Satisfaction Survey, RACD started collecting user and service provider satisfaction ratings for our software, technical services, capabilities, and resources. The group met the goal of 4 out of 5 for both KPIs. RACD also conducted a metadata management tools micro- survey that resulted in 18 responses that the group will analyze for opportunities to help users with metadata management, or to expand XSEDE’s capabilities in the area of metadata management. This reporting period, RACD met our defect and support request response metric by addressing 28 issues on an average of 3.8 calendar days. RACD did not instrument any new components to track usage this past reporting period, but did identify which components could be instrumented this program year. For other metrics with respect to this WBS, see Appendix §12.2.2.3.1.

RY5 IPR12 Page 53 Cyberinfrastructure Resource Integration (WBS 2.3.3) The mission of the Cyberinfrastructure Resource Integration (CRI) team is to work with SPs, CI providers, and campuses to maximize the aggregate utility of national cyberinfrastructure. CRI facilitates the incorporation of XSEDE software at SPs and encourages SPs to publish their information in the RDR. CRI’s activities are reflected in the uptake of CRI-integrated toolkits, such as the XSEDE Campus Bridging Cluster toolkit and XSEDE National Integration Toolkit, but also Globus Transfer clients and other toolkits as developed. Through XCI, XSEDE serves an aligning function within the nation by assembling a technical infrastructure that facilitates interaction and interoperability across the national CI ecosystem. In turn, this infrastructure is adopted by campus, regional, and national CI providers because it makes their task of delivering services easier and the delivered services better. The suite of interoperable and compatible software tools that XSEDE makes available to the CI community is based on those already in use, and services are added that address emerging needs including data and computational services. Because the XSEDE Cyberinfrastructure Resource Integration team (CRI) deals primarily with Level 1, 2, and 3 SPs, along with campus cyberinfrastructure administrators and support experts, the SP Forum and Campus Champions are XCI’s primary sources of direction regarding prioritization of efforts. CRI continued to work this reporting period to pivot cluster and cloud consulting offerings to support remote assistance methods. This led to fruitful engagements with the University of Central Oklahoma and Langston University, and also to a consulting engagement with George Mason University on the development of a hybrid system incorporating OpenHPC and OpenStack components. The remote model presents some challenges in terms of immediacy, but adds some flexibility and balance to the process. CRI engaged in outreach at the PEARC20 conference, contributing to a tutorial session and hosting a panel on cyberinfrastructure integration in the campus context. In addition, the team submitted a paper to the IEEE HPEC conference which is currently under review. An additional paper for the SIAM CSE conference in 2021 is under discussion. In addition to engagement work with campus cyberinfrastructure organizations, CRI made significant enhancements to existing toolkit offerings, largely thanks to the work of this year’s Campus Champions Fellow, Mike Renfro (Tennessee Tech University), who provided numerous updates to the cluster management toolkit. In addition to these, enhancements were made enabling access to GPU accelerators in virtual clusters and additional bug fixes to the cluster toolkit. Further work in application containerization has resulted in a templated container toolkit. This provides docker and singularity templates that research software engineers can use to build their own reproducible application containers with software they require. The resulting application container always uses the same versions of software dependencies and libraries. SP Forum integration continues with a number of novel Service Provider resources on ways to integrate with the XSEDE Federation. The addition of innovative cyberinfrastructure systems will add more resources to the slate of those integrating with XSEDE. Purdue, SUNY Stonybrook, University of Delaware, and Johns Hopkins University all represent service providers who are new to the XSEDE system and will require additional support in integration. For other metrics with respect to this WBS, see Appendix §12.2.2.3.2.

RY5 IPR12 Page 54 7. XSEDE Operations (WBS 2.4) The mission of XSEDE Operations is to install, connect, maintain, secure, and evolve an integrated cyberinfrastructure that incorporates a wide range of digital capabilities to support national scientific, engineering, and scholarly research efforts. In addition to the Operations Director’s Office (§7.1), Operations staff is subdivided into four teams based on the work breakdown structure: Cybersecurity (SecOps) (§7.2), Data Transfer Services (DTS) (§7.3), XSEDE Operations Center (XOC) (§7.4), and Systems Operational Support (SysOps) (§7.5). The Operations management team meets weekly and individual Operations groups meet approximately bi- weekly with all meeting minutes posted to the XSEDE wiki. The success of the Operations (Ops) team depends on effective collaboration across all L2 areas of the project. In particular, Ops relies on XCI to support new capabilities and services (e.g., security or networking technologies) and RAS to create and deploy solutions to improve help ticket response or the central database. In addition, Ops relies on all WBS teams and the Service Providers to respond to help tickets for their areas that are submitted by users. Key Performance Indicators for Operations are listed in the table below. Additional information about these KPIs can be found on the XSEDE KPIs & Metrics wiki page. For other metrics with respect to this WBS, see Appendix §12.2.2.4.

Table 7-1: KPIs for Operations.

Report Sub-goal Area Metric Target RP1 RP2 RP3 RP4 Total Year Supported Mean rating of 4.5 of 5/ Sustain — RY5 4.7 user qtr Provide satisfaction excellent user 4.5 of 5/ with tickets RY4 4.8 4.7 4.7 4.7 4.7 support (§3.3.2) qtr closed by the XOC RY3 4.5 of 5 / qtr 4.5 4.9 4.6 4.7 4.7

RY2 4.5 of 5 / qtr 4.5 4.8 4.8 4.4 4.6

4 of 5 / RY1 * 4.8 4.2 5.0 4.7 qtr

Hours of RY5 0/qtr 0 Sustain — downtime with Provide reliable, direct user RY4 0/qtr 0 0 22 0 22 efficient, and impacts from secure an XSEDE RY3 0 / qtr 0 0 0 0 0 infrastructure security (§3.3.1) incident RY2 0 / qtr 0 0 0 0 0

RY1 < 24 / qtr * 0 146 0 146

Satisfaction with user tickets was once again high this reporting period, exceeding the target. The target for no downtime from a security incident was met this reporting period. Operations Highlights As XSEDE continues to leverage AWS for infrastructure, SecOps continues to work to better audit and secure this environment. NCSA is working to hire a security engineer to work specifically on auditing and securing XSEDE's enterprise computing within AWS. Furthermore, as XSEDE's critical Hardware

RY5 IPR12 Page 55 Service Module (HSM) infrastructure ages, the SecOps group began investigating modernizing its HSMs by possibly refreshing hardware or deploying within AWS. At PEARC20, Tabitha Samuel of Data Transfer Services (DTS) chaired a Birds of a Feather (BoF) session titled “Bridging the Data Transfer Gap: An Open Discussion between Researchers, Administrators, and Network Engineers.” The BoF’s goal was to create an open platform to discuss network and data transfer issues directly affecting users, data and network engineers, and leadership at Advanced Research Computing centers. More than 80 participants attended and there was a great deal of engaged interaction between the participants and panelists. A crowd-sourced Google document was compiled by participants and panelists, comprising useful links and information with regards to data transfer services. This document will be shared with participants soon. Operations Director’s Office (WBS 2.4.1) The Operations Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the Operations area. This oversight includes providing direction to the L3 management team, coordination of and participation in Operations planning activities and reports through the area’s Project Manager, and monitoring compliance with budgets, retargeting effort if necessary. The Director’s Office also attends and supports the preparation of project-level reviews and activities. The Operations Director’s Office will continue to manage and set direction for Operations activities and responsibilities. The Office will contribute to and attend bi-weekly Senior Management Team calls; contribute to the project-level plan, schedule, and budget; contribute to XSEDE IPR, annual, and other reports as required by the NSF; and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area. Cybersecurity (WBS 2.4.2) The Cybersecurity Security (SecOps) group protects the confidentiality, integrity and availability of XSEDE resources and services. Users expect XSEDE resources to be reliable and secure, thus the security team’s goal is to minimize any interruption of services related to a security event. SecOps took advantage of the PEARC20 conference to focus on outreach. Prior to the conference, members of the group engaged with the SciTokens project to help explore capabilities-based SSH. A student worked with SecOps co-lead Alex Withers on extending XSEDE's SSH OAuth code to enable SciTokens. This work was presented as a poster and paper at PEARC20. For other metrics with respect to this WBS, see Appendix §12.2.2.4.1. Data Transfer Services (WBS 2.4.3) The Data Transfer Services (DTS) group facilitates data movement and management for the community by maintaining and continuously evolving XSEDE data services and resources. Dave Wheeler (NCSA) took over as the DTS manager, replacing Tim Boerner (NCSA) who assumed the newly created Deputy Project Director role for XSEDE. Thanks to Tabitha Samuel (UT/NICS) who served as the interim DTS manager. See the main Operations Highlights section above for additional DTS highlights. For other metrics with respect to this WBS, see Appendix §12.2.2.4.2. XSEDE Operations Center (WBS 2.4.4) The XSEDE Operations Center (XOC) staff serve as user advocates, providing timely and accurate assistance to the XSEDE community, while simultaneously monitoring and troubleshooting user-facing systems and services.

RY5 IPR12 Page 56 The XOC was once again excellent at fielding user tickets this reporting period with a 4.7 user satisfaction rating. For other metrics with respect to this WBS, see Appendix §12.2.2.4.3. System Operations Support (WBS 2.4.5) Systems Operational Support (SysOps) provides enterprise-level support and system administration for all XSEDE central services. As part of its effort to ensure high availability of critical XSEDE enterprise services (XES) and as a result of software audits, SysOps added new software offerings this reporting period and made numerous updates to its monitoring toolkit. Included in the aforementioned maintenance efforts were updates to XSEDE’s Nagios service, which is used by both SysOps and the XOC to alert of any failed services or servers, as well as run reports on service availability. These updates included checks to verify certain security standards were met on XSEDE enterprise services, part of XSEDE’s overall effort to maintain a strong security posture. Also, in this reporting period, SysOps performed another round of enterprise failover testing and began an upgrade to the XES Index. This index upgrade effort checks and updates the underlying schema, as well as updates the User Interface and usability for reporting purposes. Finally, SysOps continues to work with other XSEDE WBS areas, specifically SecOps. This tight collaboration between the two groups helps ensure any security vulnerabilities are quickly identified and mitigated. For other metrics with respect to this WBS, see Appendix §12.2.2.4.4.

RY5 IPR12 Page 57 8. Resource Allocation Service (WBS 2.5) The Resource Allocation Service (RAS) is building on XSEDE’s current allocation processes and evolving to meet the challenges presented by new types of resources to be allocated via XSEDE, new computing and data modalities to support increasingly diverse research needs, and large-scale demands from the user community for limited XSEDE-allocated resources. RAS is pursuing these objectives through three activities: managing the XSEDE allocations process in coordination with the XD Service Providers (§8.2), enhancing and maintaining the RAS infrastructure and services (§8.3), and anticipating changing community needs. The success of the RAS team depends on effective collaboration across all L2 areas of the project. Specifically, RAS collaborates closely with Ops to ensure capabilities are available, secure, and up-to- date; with External Relations (ER) within the Program Office to promote allocations periods and services; with XCI for continual product improvement and optimization; with CEE’s User Interfaces and Online Information (UII) for user optimization of services and processes; and with ECSS for efficient review of quarterly allocation requests. Key Performance Indicators for the Resource Allocation Service are listed in the table below. Additional information about these KPIs can be found on the XSEDE KPIs & Metrics wiki page. For other metrics with respect to this WBS, see Appendix §12.2.2.5.

Table 8-1: KPIs for Resource Allocation Service.

Report Sub-goal Area Metric Target RP1 RP2 RP3 RP4 Total Year Supported Mean rating of 4 of 5/ Sustain — RY5 4.4 user satisfaction qtr Provide with allocations excellent user 4 of 5/ process RY4 4.2 4.3 4.3 4.3 4.3 support qtr (§3.3.2) 4 of 5/ RY3 4.1 4.1 4.2 4.4 4.2 qtr 4 of 5/ RY2 4.1 4.0 4.1 3.9 4.0 qtr 4 of 5/ RY1 * 4.0 4.0 4.0 4.0 qtr Mean rating of 4 of 5/ Sustain — RY5 4.3 user satisfaction qtr Provide with XRAS reliable, 4 of 5/ RY4 4.3 4.3 4.4 4.2 4.3 efficient, and qtr secure 4 of 5/ infrastructure RY3 4.0 4.2 4.1 4.3 4.2 qtr (§3.3.1) 4 of 5/ RY2 4.0 4.0 4.1 3.9 4.0 qtr 4 of 5/ RY1 * 4.0 4.0 4.0 4.0 qtr

RAS continues to receive satisfaction ratings above the target of 4.0 (out of 5) for both the allocations process as whole, as well as the XRAS system, despite continuing high request levels for the quarterly Research opportunity. The target of 4.0 will remain given the uncertain and qualitative nature of this metric, as well as the continued reductions to recommended amounts for Research allocations.

RAS Highlights RY5 IPR12 Page 58 RAS has begun preparing for the arrival of new resources and Service Providers in the coming months. The APP team defined and A3M implemented a new “Early User Period” action type to allow XRAS to support Service Providers in setting up projects and allocations for users before the production start of new resources. This new action type allows Service Providers to implement simplified requirements and rapid review for users requesting pre-production access to these resources. A3M also continued work on the new AMIE (Account Management Information Exchange) implementation to simplify the Service Provider effort required to integrate with XSEDE’s accounting system. For XRAS, RAS published the first version of the XRAS Client Administrator’s Guide28. The guide documents a snapshot of the features in XRAS and provides instructions for taking advantage of the full power of XRAS capabilities. RAS will be maintaining the guide as new features are deployed and plans to publish updated versions up to twice per year. RAS Director’s Office (WBS 2.5.1) The RAS Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the RAS area. This oversight includes providing direction to the L3 management team, coordination of and participation in RAS planning activities and reports through the area’s Project Manager, monitoring compliance with budgets, and retargeting effort if necessary. The Director’s Office also attends and supports the preparation of project-level reviews and activities. The RAS Director's Office also contributes to an analytics effort to support NSF, Service Providers, and XSEDE in understanding and projecting the stewardship of, demand for, and impact of CI resources and services. The RAS Director’s Office will continue to manage and set direction for RAS activities and responsibilities. They will contribute to and attend bi-weekly Senior Management Team calls; contribute to the project-level plan, schedule, and budget; contribute to XSEDE IPR, annual, and other reports as required by the NSF; and attend XSEDE quarterly and annual meetings. Lastly, the Director’s Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area. XSEDE Allocations Process & Policies (WBS 2.5.2) Allocations enable the national open science community to easily gain access to XSEDE’s advanced digital resources, allowing them to achieve their research and education goals. For the June 2020 XRAC meeting, 197 proposals were submitted for XSEDE resources, of which 157 were recommended for award—a success rate of 80%. To continue addressing the success rate KPI, the team also continued to collaborate with ECSS/ ESTEO to produce user-oriented training for the preparation of Code Performance and Scaling documents; these documents are often the cause of Research requests being rejected by the XRAC. For the meeting, requests for compute resources were just over two times the amount available; approximately 352M SUs (198B NUs) were requested with 299M SUs (84.5B NUs) recommended and 247M SUs (77.8B NUs) awarded. Note that SU values are impacted by both Stampede2 being allocated in node-hours, GPU resources in GPU-hours, and allocations being moved among these and resources allocated in core-hours. In terms of agency support, 94 requests (48%) were either entirely or partially supported by NSF awards; 67 requests (34%) had support only from non-NSF sources, and 36 (18%) listed no supporting grants. During the June Research opportunity, 108 different PI institutions were represented, 14 from MSIs and 20 from EPSCoR states. A total of 466 reviews were contributed by 102 individuals from both the XRAC panel and the scientific specialists from the XSEDE ECSS area. Materials Research led all fields of sciences with 11.4% of the approved SUs, followed closely by Chemistry with 11.3% of approved SUs.

28 https://www.ideals.illinois.edu/handle/2142/107178 RY5 IPR12 Page 59 Along with the New and Renewal submissions for the June Research opportunity, the RAS APP team managed the usual steady stream of requests for other allocation types and management actions for active projects in the May-July reporting period. The team processed 212 Startup New and Renewal requests, 48 Educational New and Renewal requests, and 40 Campus Champions New and Renewal requests. For management actions across all allocation types, the team also processed 293 Extensions, 134 Supplements, 74 Transfers, 33 Advances, and 28 Adjustments. Also during this reporting period, the APP team worked with the A3M team to define and complete a number of activities, including replacing the old Fields of Science list, establishing an Early Access Period action to support the onboarding of new resources and Service Providers, and completing the first version of the XRAS Client Administrator’s Guide. On a day-to-day basis, the RAS APP group fields user tickets and inquiries, processes Startup and Educational Requests within 8.5 days on average (exceeding the goal of 14 days), and handles allocation management requests (e.g., advances, transfers, supplements, and extensions). The team hosted two training webcasts to assist with Research request writing. Throughout the year, the APP team supported the XSEDE-wide reporting effort and ad hoc queries related to allocation information and other data in the XSEDE Central Database (XDCDB). The APP group also worked closely with the RAS A3M team of developers to test and recommend updates to the XRAS system. For other metrics with respect to this WBS, see Appendix §12.2.2.5.1. Allocations, Accounting, & Account Management CI (WBS 2.5.3) The Allocations, Accounting and Account Management CI (A3M) group maintains and improves the interfaces, databases, and data transfer mechanisms for XSEDE-wide resource allocations, accounting of resource usage, and user account management. During this reporting period, the bulk of the team's focus was on three major areas of improvement. First, the team continued the implementation of the new REST API version of the AMIE protocol. In preparation for new Service Providers and new resources coming online in the near future, A3M is nearing completion of the API development and has been working on supporting infrastructure, such as a reference client library that Service Providers can use when developing their AMIE clients and a test harness version of the API for Service Providers to use to verify that their client implementation is working correctly. The second area of focus has been on redesigning the XSEDE Accounting Service (XACCT). To bring the service and associated database up to modern standards requires a large redesign of both the core schema and associated interfaces. During this reporting period, the team analyzed the existing interfaces and requirements, drafted a proposal for a new core database schema, and created a design plan for reworking XACCT. The development and implementation of this design will follow in subsequent months. The team also focused on improving the usability and User Experience of the XRAS Administrator's interface. They worked with XRAS Administrators, both within XSEDE and from other clients, to find and address inefficient or awkward interfaces in the administrator's workflow. These were then evaluated and addressed to improve the efficiency of the user interface. In addition to these three major efforts, a number of other improvements were made during this reporting period: • The transition to the new Fields of Sciences based on the OECD standard was completed. • A new "Early User Period" action type was created to allow users to request resources on pre- production HPC systems. • The XRAS Administrator's Guide was posted to Illinois IDEALS.

RY5 IPR12 Page 60 • A new user interface was created in XRAS Admin to manage default reviewers, allowing for the automatic assignment of reviewers based on various request criteria. For other metrics with respect to this WBS, see Appendix §12.2.2.5.2.

RY5 IPR12 Page 61 9. Program Office (WBS 2.6) The purpose of the Program Office (PgO) is to ensure critical project level functions are in place and operating effectively and efficiently. The oversight provided via the Project Office is necessary to provide consistent guidance and leadership to the L3 managers across the project. A common and consistent approach to managing projects and risks is provided by the Project Management, Reporting, and Risk Management (PM&R) team (§9.3), while Business Operations (§9.4) manages all financial functions and sub-awards. The crucial aspect of communications to all stakeholders is the focus of the External Relations team (§9.2). Finally, Strategy, Planning, Policy, Evaluation & Organizational Improvement (SP&E) (§9.5) focuses attention in precisely those areas to ensure the best possible structure continues to exist within XSEDE to allow the support of all significant project activities and enable efficient and effective performance of all project responsibilities. The success of the PgO depends on effective collaboration across all L2 areas of the project. The PgO conducts all administrative work for XSEDE, ensuring each project area is able to stay in operation and focused on the user-base. In addition, External Relations works with all L2 areas of the project to ensure that the user community is aware of the services offered by each area and to highlight project successes. Key Performance Indicators for the Program Office are listed in the table below. Additional information about these KPIs can be found on the XSEDE KPIs & Metrics wiki page. For other metrics with respect to this WBS, see Appendix §12.2.2.6.

Table 9-1: KPIs for Program Office.

Report Sub-goal Area Metric Target RP1 RP2 RP3 RP4 Total Year Supported Variance between RY5 0 0 Sustain — relevant report Operate an submission and due RY4 0 0 0 0 -1 -1 effective and date (days) productive RY3 0 0 0 0 0 0 virtual organization RY2 0 0 0 0 0 0 (§3.3.3)

RY1 0 * NA 0 0 0

Percentage of RY5 90/qtr 92.6 Sustain — subaward invoices Operate an processed within RY4 95/qtr 95.0 100.0 79.5 90.9 91.3 effective and target duration productive RY3 95/qtr 82.4 77.8 94.4 92.9 86.9 virtual organization RY2 95/qtr 100.0 90.9 87.7 67.4 86.64 (§3.3.3)

RY1 90/qtr * NA1 100.0 NA2 100.0

Percentage of RY5 90/yr 100 Sustain — recommendations Operate an addressed by RY4 90/yr 47 48 0.1 0 24 effective and relevant project productive areas within 90 days RY3 90/yr 46 74 0 62 45.5 virtual organization

RY2 90/qtr 23 15 37 49 31 (§3.3.3)

RY1 90/qtr * NA3 100 57 78.54

RY5 IPR12 Page 62 Report Sub-goal Area Metric Target RP1 RP2 RP3 RP4 Total Year Supported Aggregate mean 3.9 of 5/ Sustain — RY5 - rating of satisfaction yr Operate an with content and effective and 3.9 of 5/ accessibility of the RY4 - 3.8 - - 3.8 productive yr XSEDE Staff Wiki virtual 3.5 of 5/ organization RY3 - 3.9 - - 3.9 yr (§3.3.3)

RY2 * * * * *

RY1 * * * * *

Number of staff RY5 50/yr 2 Sustain — publications Operate an RY4 32/yr 11 44 7 18 80 innovative virtual RY3 20/yr 19 16 7 6 48 organization (§3.3.4) RY2 20/yr 2 6 0 1 9

RY1 70/yr * 5 0 13 18

Aggregate mean RY5 4.1/yr - Sustain — rating of Inclusion in Operate an XSEDE RY4 4.1/yr - 4.2 - - 4.2 innovative virtual RY3 4.1/yr - 4.1 - - 4.1 organization (§3.3.4) 5% RY2 improve- * 4.3 * * 4.3 ment/yr

RY1 * * * * *

Aggregate mean RY5 4.0/yr - Sustain — rating of Equity in Operate an XSEDE RY4 4.0/yr - 4.2 - - 4.2 innovative virtual RY3 4.0/yr - 4.1 - - 4.1 organization (§3.3.4) RY2 * * * * *

RY1 * * * * *

Number of XSEDE- RY5 325/yr 624 Deepen/ related media hits Extend — RY4 165/yr 66 92 101 237 496 Raise awareness of RY3 169/yr 23 29 54 60 166 the value of advanced RY2 169/yr 42 30 44 29 145 digital services RY1 147/yr * 32 30 18 80 (§3.1.4))

- Data reported annually. 1 Subaward institutions did not have XSEDE2 contracts in place yet, so no invoices had been issued. 2 No subaward invoices received during this reporting period. 3 No recommendations received during this reporting period. RY5 IPR12 Page 63 XSEDE continues to meet the goal of submitting all project reports on time. “Percentage of subaward invoices processed within target duration” exceeded the target of 90% of subaward invoices processed within 42 days. “Percentage of recommendations addressed by relevant project areas within 90 days” exceeded the target with 100% of recommendations addressed within 90 days. The KPIs measuring the XSEDE staff’s satisfaction with the staff wiki, inclusion, and equity are all reported annually in RP2 based on Staff Climate Study results, so there is no data to report during this reporting period. “Number of staff publications” was lower than expected this reporting period, so staff will be reminded to upload their publications to the User Portal. With many staff contributing publications to the PEARC conference, this number is expected to be significantly higher next reporting period once those publications are added to their User Portal accounts. The number of media hits greatly exceeded expectations this reporting period. A company working with a researcher who was using XSEDE resources mentioned XSEDE explicitly in multiple press releases, which resulted in a large increase in media hits for XSEDE. External Relations sees this as an unpredictable and potentially inflated event. As time goes on, the team expects these numbers to normalize to something closer to what has been experienced in previous reporting periods. PgO Highlights Contributing to the research community, the PM&R team presented a best practices paper at PEARC20 based on the lessons learned from executing the XSEDE Project. As expected, this reporting period was extremely busy for the Evaluation team as they undertook multiple evaluation efforts, both within and outside the project. §9.5 contains more information about the evaluations conducted. The Business Operations team completed the financial analysis for the NSF Panel Review and is now processing subaward amendments for the final year of the award. The project’s involvement in the response to the COVID-19 pandemic has had a very positive impact on the visibility of the project. §9.2 has further details. Project Office (WBS 2.6.1) The Project Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the Program Office area and to establish responsibility for assuring advisory activities of the project occur. This oversight includes providing direction to the L3 management team and coordination of and participation in Program Office planning activities and reports through the area’s Project Manager. The Project Office also attends and supports the preparation of project-level reviews and activities. Importantly, the Project Office is responsible for ensuring that the XSEDE Advisory Board, the User Advisory Committee, and the SP Forum are functioning. The Project Office is responsible for coordination of project-level meetings such as the bi- weekly Senior Management Team (SMT) teleconference calls and the project quarterly meetings. Lastly, the Project Office will advise the XSEDE PI on many issues, especially those relevant to this WBS area. External Relations (WBS 2.6.2) External Relations’ (ER) mission is to communicate the value and importance of XSEDE to all stakeholders (including the internal audience) through creative and strategic communications. External Relations continued to effectively raise awareness about XSEDE during this reporting period. While the social media strategy of mixing narrative science stories with hands-on training offerings and events has taken time to establish its voice, impressions are now starting to fall in line to where they were previously, indicating that XSEDE is starting to re-establish its digital voice. Though the number of social media impressions decreased slightly when compared to the same period last year, the team

RY5 IPR12 Page 64 expects an expanded amount of impressions as the project continues to assert itself as both a place for information (science stories) and a place to gain real-world skills. Due to the COVID-19 pandemic, the number of media hits greatly exceeded expectations this reporting period. Hoth Therapeutics, a company working with a researcher who was using XSEDE resources, mentioned XSEDE explicitly in multiple press releases, resulting in a huge increase in media hits for XSEDE. External Relations sees this as an unpredictable and potentially inflated event, with COVID research being one of the most talked-about topics on the internet right now, especially when it concerns a therapeutic or a vaccine. As time goes on, the team expects these numbers to normalize to something closer to what has been experienced in previous reporting periods. For other metrics with respect to this WBS, see Appendix §12.2.2.6.1. Project Management, Reporting, & Risk Management (WBS 2.6.3) The Project Management, Reporting & Risk Management (PM&R) team enables an effective virtual organization through the application of project management principles; provides visibility to project progress, successes, and challenges; brings new ideas and management practices into the project; and disseminates lessons learned in XSEDE to other virtual organizations. Communication is critical to success in this highly distributed virtual organization. During this reporting period, the PM&R team worked with project leadership to ensure the on-time submission of the Annual Report and Program Plan in May. The L2 project managers also supported their area leaders in preparing for and presenting their area’s highlights during the project’s annual NSF panel review in June. Three members of the team presented a paper on best practices in project management based on lessons learned from XSEDE project management during the PEARC20 conference. For other metrics with respect to this WBS, see Appendix §12.2.2.6.2. Business Operations (WBS 2.6.4) The Business Operations (BO) group, working closely with staff at the University of Illinois’ Grants and Contracts Office (GCO) and National Center for Supercomputing Applications’ (NCSA) Business Office, manages budgetary issues and sub-awards, and ensures timely processing of sub-award amendments and invoices. During this reporting period, Business Operations collected budget documents from subaward institutions for the proposed supplemental year of XSEDE. This effort included collecting updated spend plans for PY9 and PY10 from all subaward institutions, and once the approval to spend in PY10 was issued by NSF, working with subaward institutions to complete their documentation for PY10 funding amendments. For other metrics with respect to this WBS, see Appendix §12.2.2.6.3. Strategy, Planning, Policy, Evaluation & Organizational Improvement (WBS 2.6.5) XSEDE dedicates effort to project-wide strategic planning, policy development, evaluation and assessment, and organizational improvement in support of sustaining an effective and productive virtual organization. XSEDE has engaged an independent Evaluation Team designed to provide XSEDE with information to guide program improvement and assess the impact of XSEDE services. Evaluations are based on five primary data sources: (1) an Annual User Survey that is part of the XSEDE annual report and program plan; (2) an Enhanced Longitudinal Study, encompassing additional target groups (e.g., faculty, institutions, disciplines, etc.) and additional measures (e.g., publications, citations, research funding,

RY5 IPR12 Page 65 promotion and tenure, etc.); (3) an Annual XSEDE Staff Climate Study; (4) XSEDE KPIs, Area Metrics, and Organizational Improvement efforts, including ensuring that procedures are in place to assess these data; and (5) Specialized Studies as contracted by Level 2 directors and the Program Office. The SP&E team finalized the Annual User Survey and the Staff Climate study during this reporting period. In addition, they contributed content to the annual NSF panel review, evaluated the 5-day virtual residency workshop at University of Oklahoma, conducted an evaluation of the PEARC conference, conducted an evaluation of the SPICE program, continued efforts on the XSEDE longitudinal studies, and continued compiling data and publishing reports related to the XSEDE Return on Investment (ROI) study. For other metrics with respect to this WBS, see Appendix §12.2.2.6.4.

RY5 IPR12 Page 66 10. Financial Information The XSEDE Business Operations team (§9.4) tracks and manages the financial aspect of the XSEDE project. This section conveys the financial status at a project level. The focus is on spending against the approved budget. Note that closing out any given reporting period could take up to nine months after the reporting period ends. The actual duration is dependent on the timeliness of invoice submissions by the partner institutions, plus the University of Illinois invoice processing, typically 30-45 days. The table below shows the status of reported and paid costs during the defined reporting period. The understood delay in receipt and processing of invoices results in the need to update data associated with prior reporting periods, upon the release of each IPR. The tables below show the financial summary, at a project level and a partner institution level, as of the submission of this report.

Table 10-1: Project Level Financial Summary.

Invoices Reporting Period Budgeted Spent Projected (Paid/Expected) RY1 RP2: Sept '16 – Oct '16 38 of 38 $3,504,153 $2,478,940 N/A RY1 RP3: Nov '16 – Jan '17 57 of 57 $5,256,230 $4,238,946 N/A RY1 RP4: Feb '17 – Apr '17 57 of 57 $5,256,230 $5,131,523 N/A RY1 Total: Sept ’16 – Apr ‘17 152 of 152 $14,016,613 $11,813,504 N/A RY2 RP1: May '17 – Jul '17 57 of 57 $5,256,230 $5,455,524 N/A RY2 RP2: Aug '17 – Oct '17 57 of 57 $5,325,759 $5,012,569 N/A RY2 RP3: Nov '17 – Jan '18 57 of 57 $5,423,416 $4,909,549 N/A RY2 RP4: Feb '18 – April '18 57 of 57 $5,423,416 $4,831,465 N/A RY2 Total: Sept '17 – April '18 228 of 228 $21,428,820 $20,179,942 N/A RY3 RP1: May '18 – Jul '18 56 of 56 $5,332,257 $5,918,791 N/A RY3 RP2: Aug '18 – Oct '18* 57 of 57 $5,485,889 $5,204,824 N/A RY3 RP3: Nov '18 – Jan '19* 57 of 57 $5,560,866 $5,012,275 N/A RY3 RP4: Feb '19 – Apr '19* 58 of 58 $5,628,783 $5,343,120 N/A RY3 Total: Sept '18 – April '19* 228 of 228 $22,007,794 $21,479,009 N/A RY4 RP1: May '19 – Jul '19* 57 of 57 $5,662,741 $5,340,260 N/A RY4 RP2: Aug '19 – Oct '19 53 of 53 $5,743,027 $5,338,504 N/A RY4 RP3: Nov '19 – Jan '20 51 of 51 $5,783,170 $5,443,052 N/A RY4 RP4: Feb '20 – Apr '20 51 of 51 $5,783,170 $5,146,415 N/A RY4 Total: Sept '19 – April '20 212 of 212 $22,972,109 $21,268,232 N/A RY5 RP1: May '20 – Jul '20 26 of 51* $5,750,235 $2,628,429 $2,783,299 RY5 RP2: Aug '20 – Oct '20 RY5 RP3: Nov '20 – Jan '21 RY5 RP4: Feb '21 – Apr '21 Extended RY5: May '21 – Aug '21 * Partial information available; will be updated in future IPRs.

RY5 IPR12 Page 67 .

The following tables reflect received invoices and the breakdown distribution within the XSEDE Invoice Portal. The expenses reported may reflect back transfers, and we are reporting based on the month the expense posted to the grant/subaward account.

Table 10-2: Partner Institution Level Financial Summary.

RY1 RP2: Sept ’16 – Oct '16 RY1 RP3: Nov ’16 – Jan ‘17 Invoices Invoices Partner Paid Budgeted Spent Paid Budgeted Spent Institution (of 2) (of 3) NCSA 2 $733,278 $179,527 3 $1,099,917 $623,728 TACC 2 $550,304 $31,027 3 $825,457 $55,141 PSC/MPC 2 $532,169 $487,486 3 $789,253 $738,316 SDSC/UCSD 2 $463,680 $471,304 3 $695,520 $681,232 NICS/UTK 2 $286,575 $263,222 3 $429,862 $397,643 U Chicago/ANL 2 $226,231 $188,387 3 $339,346 $221,649 Indiana 2 $189,172 $122,602 3 $283,758 $293,470 University Shodor 2 $107,262 $109,862 3 $160,893 $137,963 Cornell 2 $105,112 $89,470 3 $157,668 $124,968 University NCAR/UCAR 2 $66,215 $0 3 $99,323 $69,224 Purdue 2 $52,897 $54,084 3 $79,345 $111,537 University Georgia Tech 2 $55,357 $4,320 3 $83,035 $6,480 SURA 2 $38,333 $31,027 3 $57,500 $55,141 OK State (OSU) 2 $33,573 $8,574 3 $50,361 $16,321 Ohio State (OSC) 2 $18,367 $0 3 $27,550 $32,197 USC-ISI 2 $13,333 $31,027 3 $20,000 $55,141 U Oklahoma 2 $11,261 $10,578 3 $16,892 $15,866 (OU) U Georgia 2 $10,567 $2,788 3 $15,850 $4,182 U Arkansas 2 $10,467 $8,749 3 $15,700 $16,184 Project Level 38 of 38 $3,504,153 $2,094,031 57/57 $5,256,230 $3,656,383

RY5 IPR12 Page 68 RY1 RP4: Feb ’17 – Apr ‘17 RY1 Total: Sept ’16 – May ‘17 Invoices Invoices Partner Paid Budgeted Spent Paid Budgeted Spent Institution (of 3) (of 8) NCSA 3 $1,099,917 $1,207,527 8 $2,933,112 $2,010,781 TACC 3 $825,457 $63,772 8 $2,201,218 $149,940 PSC/MPC 3 $798,253 $801,638 8 $2,128,675 $2,027,439 SDSC/UCSD 3 $695,520 $711,326 8 $1,854,720 $1,863,862 NICS/UTK 3 $429,862 $406,402 8 $1,146,299 $1,067,266 U Chicago/ANL 3 $339,346 $491,477 8 $904,923 $901,513 Indiana University 3 $283,758 $256,475 8 $756,688 $672,547 Shodor 3 $160,893 $117,830 8 $429,048 $365,655 Cornell University 3 $157,668 $107,949 8 $420,448 $322,387 NCAR/UCAR 3 $99,323 $90,254 8 $264,861 $159,478 Purdue University 3 $79,345 $58,688 8 $211,587 $224,308 Georgia Tech 3 $83,035 $6,480 8 $221,427 $17,281 SURA 3 $57,500 $63,772 8 $153,333 $149,940 OK State (OSU) 3 $50,361 $14,547 8 $134,295 $39,442 Ohio State (OSC) 3 $27,550 $32,471 8 $73,467 $64,667 USC-ISI 3 $20,000 $63,772 8 $53,333 $149,940 U Oklahoma (OU) 3 $16,892 $15,867 8 $45,045 $42,310 U Georgia 3 $15,850 $4,182 8 $42,267 $11,151 U Arkansas 3 $15,700 $13,123 8 $41,867 $38,056 Project Level 57 of 57 $5,256,230 $4,527,552 152 of 152 $14,016,613 $10,277,966

RY5 IPR12 Page 69

RY2 RP1: May ’17 – Jul ‘17 RY2 RP2: Aug ’17 – Oct ‘17 Invoices Invoices Partner Paid Budgeted Spent Paid Budgeted Spent Institution (of 3) (of 3) NCSA 3 $1,099,917 $1,147,967 3 $1,098,678 $974,612 TACC 3 $825,457 $916,498 3 $816,652 $847,586 PSC/MPC 3 $798,253 $830,786 3 $807,749 $828,314 SDSC/UCSD 3 $695,520 $722,633 3 $705,544 $723,763 NICS/UTK 3 $429,862 $496,015 3 $459,812 $404,335 U Chicago/ANL 3 $339,346 $336,242 3 $350,002 $147,712 Indiana University 3 $283,758 $263,689 3 $277,231 $275,840 Shodor 3 $160,893 $164,529 3 $162,986 $134,506 Cornell University 3 $157,668 $152,928 3 $178,616 $125,746 NCAR/UCAR 3 $99,323 $83,082 3 $107,430 $98,807 Purdue University 3 $79,345 $68,088 3 $72,795 $70,039 Georgia Tech 3 $83,035 $62,090 3 $75,542 $107,992 SURA 3 $57,500 $59,371 3 $58,229 $62,519 OK State (OSU) 3 $50,361 $38,016 3 $54,250 $125,817 Ohio State (OSC) 3 $27,550 $33,963 3 $27,830 $25,220 USC-ISI 3 $20,000 $33,955 3 $20,300 $13,152 U Oklahoma (OU) 3 $16,892 $15,404 3 $20,455 $14,054 U Georgia 3 $15,850 $31,924 3 $15,888 $16,871 U Arkansas 3 $15,700 $17,975 3 $15,772 $15,682 Project Level 57 of 57 $5,256,230 $5,475,155 57 of 57 $5,325,759 $5,012,569

RY5 IPR12 Page 70

RY2 RP3: Nov ’17 – Jan ‘18 RY2 RP4: Feb ’18 – Apr ‘18 Invoices Invoices Partner Paid Budgeted Spent Paid Budgeted Spent Institution (of 3) (of 3) NCSA 3 $1,154,307 $1,016,137 3 $1,154,307 $935,931 TACC 3 $812,249 $869,772 3 $812,249 $303,524 PSC/MPC 3 $812,496 $851,171 3 $812,496 $809,751 SDSC/UCSD 3 $710,556 $716,848 3 $710,556 $727,397 NICS/UTK 3 $474,787 $395,172 3 $474,787 $413,610 U Chicago/ANL 3 $355,330 $148,934 3 $355,330 $730,303 Indiana University 3 $273,968 $272,403 3 $273,968 $258,732 Shodor 3 $164,032 $127,160 3 $164,032 $128,962 Cornell University 3 $189,090 $144,466 3 $189,090 $141,914 NCAR/UCAR 3 $111,483 $101,823 3 $111,483 $136,090 Purdue University 3 $69,520 $74,499 3 $69,520 $66,940 Georgia Tech 3 $78,439 $42,557 3 $78,439 $24,414 SURA 3 $58,594 $42,633 3 $58,594 $55,266 OK State (OSU) 3 $56,195 $37,584 3 $56,195 $30,217 Ohio State (OSC) 3 $27,970 $22,023 3 $27,970 $22,727 USC-ISI 3 $20,450 $16,735 3 $20,450 $19,177 U Oklahoma (OU) 3 $22,237 $13,229 3 $22,237 $13,228 U Georgia 3 $15,907 $18,633 3 $15,907 $18,633 U Arkansas 3 $15,808 $16,404 3 $15,808 $13,283 Project Level 57 of 57 $5,423,416 $4,928,182 57 of 57 $5,423,416 $4,850,097

RY5 IPR12 Page 71 RY2 Total: May ’17 – Apr ‘18 Invoices Partner Paid Budgeted Spent Institution (of 12) NCSA 12 $4,507,209 $4,074,646 TACC 12 $3,266,607 $2,937,380 PSC/MPC 12 $3,230,994 $3,320,022 SDSC/UCSD 12 $2,822,175 $2,890,641 NICS/UTK 12 $1,839,249 $1,709,132 U Chicago/ANL 12 $1,400,007 $1,363,192 Indiana University 12 $1,108,925 $1,070,664 Shodor 12 $651,942 $555,158 Cornell University 12 $714,463 $565,054 NCAR/UCAR 12 $429,719 $419,801 Purdue University 12 $291,179 $279,566 Georgia Tech 12 $315,454 $237,053 SURA 12 $232,917 $219,789 OK State (OSU) 12 $217,001 $231,634 Ohio State (OSC) 12 $111,320 $103,934 USC-ISI 12 $81,200 $83,018 U Oklahoma (OU) 12 $81,822 $55,914 U Georgia 12 $63,553 $86,060 U Arkansas 12 $63,087 $63,345 Project Level 228 of 228 $21,428,820 $20,266,003 † Subawardee transitioned institutions during this period, thus, there was only one invoice from each of those institutions during this period.

RY5 IPR12 Page 72

RY3 RP1: May ’18 – July ‘18 RY3 RP2: Aug ’18 – Oct ‘18 Invoices Invoices Partner Paid Budgeted Spent Paid Budgeted Spent Institution (of 3) (of 3) NCSA 3 $1,079,267 $1,061,000 3 $1,196,029 $1,134,369 TACC 3 $799,010 $1,371,895 3 $793,074 $839,725 PSC/MPC 3 $813,725 $822,138 3 $841,714 $825,881 SDSC/UCSD 3 $710,556 $709,180 3 $721,408 $696,515 NICS/UTK 3 $474,787 $543,968 3 $461,707 $410,580 U Chicago/ANL 3 $355,446 $278,271 3 $353,810 $282,262 Indiana University 3 $273,968 $280,742 3 $269,780 $251,812 Shodor 3 $164,132 $187,195 3 $173,967 $128,270 Cornell University 3 $189,090 $156,765 3 $191,982 $171,593 NCAR/UCAR 3 $111,483 $107,236 3 $106,323 $77,394 Purdue University 3 $69,520 $64,669 3 $70,595 $72,138 Georgia Tech 3 $78,439 $146,226 3 $86,559 $65,189 SURA 3 $58,594 $70,674 3 $64,542 $62,935 OK State (OSU) 3 $56,195 $34,666 3 $24,427 $102,256 Ohio State (OSC) 3 $27,970 $21,567 3 $33,257 $26,865 USC-ISI 3 $20,450 $27,463 3 $20,757 $29,259 U Oklahoma (OU) 3 $22,238 $14,601 3 $22,563 $14,276 U Georgia 1† $6,363 $6,211 0 $0 $0 Notre Dame 1† $5,219 $0 3 $15,656 $0 U Arkansas 3 $15,808 $14,326 3 $15,940 $13,506 Project Level 56 of 56 $5,332,257 $5,918,791 57 of 57 $5,485,889 $5,204,824 † Subawardee transitioned institutions during this period, thus, there was only one invoice from each of those institutions during this period.

RY5 IPR12 Page 73

RY3 RP3: Nov ’18 – Jan ‘19 RY3 RP4: Feb ‘19 – Apr ‘19 Invoices Invoices Partner Paid Budgeted Spent Paid Budgeted Spent Institution (of 3) (of 3) NCSA 3 $1,254,410 $1,029,286 3 $1,254,410 $1,067,177 TACC 3 $790,106 $678,184 3 $790,106 $961,363 PSC/MPC 3 $855,709 $883,438 3 $855,709 $826,496 SDSC/UCSD 3 $726,834 $682,790 3 $726,834 $685,154 NICS/UTK 3 $455,168 $345,594 3 $455,168 $427,196 U Chicago/ANL 3 $352,992 $423,139 3 $352,992 $317,198 Indiana University 3 $300,386 $261,815 3 $300,386 $280,099 Shodor 3 $178,884 $144,815 3 $178,884 $153,524 Cornell University 3 $193,428 $198,933 3 $193,428 $266,505 NCAR/UCAR 3 $103,743 $97,268 3 $103,743 $103,775 Purdue University 3 $71,133 $72,890 3 $71,133 $47,359 Georgia Tech 3 $90,619 $49,657 3 $90,619 $46,103 SURA 3 $67,517 $46,680 3 $67,517 $86,153 OK State (OSU) 3 $8,543 $26,391 1† $2,848 $8,878 Internet2 3† $73,612 $0 Ohio State (OSC) 3 $35,901 $22,952 3 $35,901 $27,291 USC-ISI 3 $20,910 $7,402 3 $20,910 $7,837 U Oklahoma (OU) 3 $22,726 $27,441 3 $22,726 $17,411 U Georgia 0 $0 $0 0 $0 $0 Notre Dame 3 $15,656 $0 3 $15,656 $0 U Arkansas 3 $16,204 $13,601 3 $16,204 $13,601 Project Level 57 of 57 $5,560,866 $5,012,275 58 of 58 $5,628,783 $5,343,120 † Subawardee transitioned between institutions during this period resulting in net one additional invoice due to transition timing and billing periods.

RY5 IPR12 Page 74 RY3 Total: May ‘18 – Apr ‘19 Invoices Partner Paid Budgeted Spent Projected Institution (of 12) NCSA 12 $4,784,116 $4,291,832 $0 TACC 12 $3,172,295 $3,851,167 $0 PSC/MPC 12 $3,366,857 $3,357,953 $0 SDSC/UCSD 12 $2,885,632 $2,773,638 $0 NICS/UTK 12 $1,846,830 $1,727,338 $0 U Chicago/ANL 12 $1,415,239 $1,300,869 $0 Indiana University 12 $1,166,320 $1,074,468 $0 Shodor 12 $695,866 $613,804 $0 Cornell University 12 $767,927 $793,797 $0 NCAR/UCAR 12 $425,293 $385,673 $0 Purdue University 12 $282,381 $257,055 $0 Georgia Tech 12 $346,235 $307,175 $0 SURA 12 $258,170 $266,441 $0 OK State (OSU) 10 $92,012 $172,191 $0 Internet2 3 $73,612 $0 $0 Ohio State (OSC) 12 $133,028 $98,674 $0 USC-ISI 12 $83,027 $71,960 $0 U Oklahoma (OU) 12 $90,253 $73,728 $0 U Georgia 1 $6,363 $6,211 $0 Notre Dame 10 $52,185 $0 $0 U Arkansas 12 $64,154 $55,034 $0 Project Level 228 of 228 $22,007,794 $21,479,009 $0

RY5 IPR12 Page 75 RY4 RP1: May ‘19 – Jul ‘19 Invoices Partner Paid Budgeted Spent Projected Institution (of 3) NCSA 3 $1,254,410 $987,332 $0 TACC 3 $790,106 $776,066 $0 PSC/MPC 3 $855,709 $835,783 $0 SDSC/UCSD 3 $726,834 $711,843 $0 NICS/UTK 3 $455,168 $452,047 $0 U Chicago/ANL 3 $352,992 $347,184 $0 Indiana University 3 $300,386 $247,478 $0 Shodor 3 $178,884 $188,947 $0 Cornell University 3 $193,428 $213,863 $0 NCAR/UCAR 3 $103,743 $161,398 $0 Purdue University 3 $71,133 $84,844 $0 Georgia Tech 3 $90,619 $98,865 $0 SURA 3 $67,517 $48,937 $0 OK State (OSU) 0 $0 $0 $0 Internet2 3 $110,418 $53,879 $0 Ohio State (OSC) 3 $35,901 $27,115 $0 USC-ISI 3 $20,910 $7,297 $0 U Oklahoma (OU) 3 $22,726 $18,357 $0 U Georgia 0 $0 $0 $0 Notre Dame 3 $15,656 $63,645 $0 U Arkansas 3 $16,204 $15,382 $0 Project Level 57 of 57 $5,662,741 $5,340,260 $0

RY5 IPR12 Page 76 RY4 RP2: Aug ‘19 – Oct ‘19 Invoices Partner Paid Budgeted Spent Projected Institution (of 3) NCSA 3 $1,221,496 $1,040,514 $0 TACC 3 $809,638 $549,787 $0 PSC/MPC 3 $857,989 $894,619 $0 SDSC/UCSD 3 $788,548 $692,240 $0 NICS/UTK 3 $469,599 $422,264 $0 U Chicago/ANL 3 $364,621 $353,438 $0 Indiana University 3 $331,570 $390,765 $0 Shodor 3 $160,243 $213,694 $0 Cornell University 3 $195,934 $206,135 $0 NCAR/UCAR 3 $105,301 $141,144 $0 Purdue University 3 $72,163 $74,618 $0 Georgia Tech 3 $88,560 $73,146 $0 SURA 3 $87,176 $72,419 $0 OK State (OSU) 0 $0 $0 $0 Internet2 3 $77,222 $133,036 $0 Ohio State (OSC) 3 $36,174 $30,494 $0 USC-ISI 3 $21,224 $18,376 $0 U Oklahoma (OU) 3 $22,991 $18,137 $0 U Georgia 0 $0 $0 $0 Notre Dame 1 $16,101 $7,317 $23,654 U Arkansas 2 $16,476 $6,361 $5,135 Project Level 53 $5,743,027 $5,338,504 $0

RY5 IPR12 Page 77 RY4 RP3: Nov ‘19 – Jan ‘20 Invoices Partner Paid Budgeted Spent Projected Institution (of 3) NCSA 3 $1,205,039 $1,091,031 $0 TACC 3 $819,405 $966,294 $0 PSC/MPC 3 $859,130 $837,147 $0 SDSC/UCSD 3 $819,405 $730,210 $0 NICS/UTK 3 $476,815 $473,779 $0 U Chicago/ANL 3 $370,436 $357,562 $0 Indiana University 3 $347,163 $316,887 $0 Shodor 3 $150,923 $94,127 $0 Cornell University 3 $197,188 $143,999 $0 NCAR/UCAR 3 $106,080 $114,560 $0 Purdue University 3 $72,678 $78,428 $0 Georgia Tech 3 $87,531 $58,608 $0 SURA 3 $61,369 $77,099 $0 OK State (OSU) 0 $0 $0 $0 Internet2 3 $60,624 $42,958 $0 Ohio State (OSC) 3 $36,311 $31,142 $0 USC-ISI 3 $21,381 $19,135 $0 U Oklahoma (OU) 3 $23,123 $10,086 $0 U Georgia 0 $0 $0 $0 Notre Dame 0 $16,324 $0 $0 U Arkansas 0 $16,612 $0 $0 Project Level 51 $5,747,533 $5,443,052 $0

RY5 IPR12 Page 78 RY4 RP4: Feb ‘20 – Apr ‘20 Invoices Partner Paid Budgeted Spent Projected Institution (of 3) NCSA 3 $1,205,039 $1,055,366 $0 TACC 3 $819,405 $730,304 $0 PSC/MPC 3 $859,130 $857,841 $0 SDSC/UCSD 3 $819,405 $712,713 $0 NICS/UTK 3 $476,815 $500,998 $0 U Chicago/ANL 3 $370,436 $179,802 $0 Indiana University 3 $347,163 $410,438 $0 Shodor 3 $150,923 $124,724 $0 Cornell University 3 $197,188 $137,333 $0 NCAR/UCAR 3 $106,080 $127,488 $0 Purdue University 3 $72,678 $45,525 $0 Georgia Tech 3 $87,531 $48,373 $0 SURA 3 $97,006 $97,126 $0 OK State (OSU) 0 $0 $0 $0 Internet2 3 $60,624 $40,031 $0 Ohio State (OSC) 3 $36,311 $32,066 $0 USC-ISI 3 $21,381 $31,160 $0 U Oklahoma (OU) 3 $23,123 $15,129 $0 U Georgia 0 $0 $0 $0 Notre Dame 0 $16,324 $0 $0 U Arkansas 0 $16,612 $0 $0 Project Level 51 $5,783,170 $5,146,415 $0

RY5 IPR12 Page 79 RY4 Total: May ‘19 – Apr ‘20 Invoices Partner Paid Budgeted Spent Projected Institution (of 12) NCSA 12 $4,885,984 $4,174,243 $0 TACC 12 $3,238,553 $3,022,452 $0 PSC/MPC 12 $3,431,957 $3,425,390 $0 SDSC/UCSD 12 $3,154,191 $2,847,006 $0 NICS/UTK 12 $1,878,397 $1,849,088 $0 U Chicago/ANL 12 $1,458,484 $1,237,987 $0 Indiana University 12 $1,326,282 $1,365,567 $0 Shodor 12 $640,973 $621,492 $0 Cornell University 12 $783,737 $701,329 $0 NCAR/UCAR 12 $421,204 $544,589 $0 Purdue University 12 $288,653 $283,415 $0 Georgia Tech 12 $354,242 $278,992 $0 SURA 12 $348,706 $295,581 $0 OK State (OSU) 0 $0 $0 $0 Internet2 12 $308,887 $269,904 $0 Ohio State (OSC) 12 $144,697 $120,816 $0 USC-ISI 12 $84,895 $75,967 $0 U Oklahoma (OU) 12 $91,963 $61,708 $0 U Georgia 0 $0 $0 $0 Notre Dame 4 $64,404 $70,962 $0 U Arkansas 4 $65,903 $21,743 $0 Project Level 212 $22,972,109 $21,268,232 $0

RY5 IPR12 Page 80 RY5 RP1: May ‘20 – July ‘20 Invoices Partner Paid Budgeted Spent Projected Institution (of 3) NCSA 0 $1,205,039 $0 $999,175 TACC 2 $819,405 $681,298 $23,137 PSC/MPC 3 $859,130 $810,950 $0 SDSC/UCSD 1 $819,405 $265,545 $472,006 NICS/UTK 2 $476,815 $441,011 $151,609 U Chicago/ANL 3 $370,436 $534,409 $0 Indiana University 2 $347,163 $258,484 $92,045 Shodor 2 $150,923 $175,637 $62,585 Cornell University 1 $197,188 $56,237 $156,888 NCAR/UCAR 3 $106,080 $90,259 $0 Purdue University 3 $72,678 $45,531 $0 Georgia Tech 3 $87,531 $97,821 $0 SURA 2 $97,006 $78,824 $23,137 OK State (OSU) 0 0 0 0 Internet2 1 $60,624 $13,297 $54,512 Ohio State (OSC) 3 $36,311 $71,917 $0 USC-ISI 3 $21,381 $47,115 $0 U Oklahoma (OU) 2 $23,123 $18,126 $6,382 U Georgia 0 0 0 0 Notre Dame 0 0 0 0 U Arkansas 0 0 0 0 Project Level 36 $5,750,235 $3,686,461 $2,041,478

RY5 IPR12 Page 81 11. Project Improvement Fund The XSEDE Project Improvement Fund (PIF) is an extension of the XSEDE PY7-PY9 annual planning and budget review process to strategically invest approximately 2% of the annual budget ($465k annually) towards short-term project improvements specific to the XSEDE project. A lightweight Phase-Gate process is used to facilitate the review and prioritization of proposed project improvements by the XSEDE Senior Management Team (SMT). Funding is allocated based on this prioritization. The details of the process can be found on the XSEDE wiki PIF page. It is important to note that the annual budget for project improvement funds, while expected to be used during a specific project year, are not intended to be fully allocated during the project year planning period. The funds are allocated prior to and throughout the first half of the project year via a review process for the project to fund important ideas that bring value to the project and, in particular, to the community we support. There have been a wide range of PIFs approved, including: updates and improvements to tools, staff training, new offerings to the community, and analysis efforts of the project’s tools and services. To date, 25 idea submissions have been received and reviewed. The following is a summary of the current status: PY7 Project Improvement Fund Status Total PIF funds allocated: $389K of $389K

State/Phase Submissions Comments Complete 6 Projects officially closed • ECSS cloud support specialist • XDCDB improvements • Jira bootcamp • ECSS: Ease transition to Stampede2 • 2K Duo licenses • 2017 IHPCSS

Not Funded 4 Other recent efforts were similar or the submission requested annually recurring funding

Withdrawn 1 Withdrawn due to significant overlap with another submission

Notes: PY7 PIF funds have been reduced by $76K, as per PCR #14, to fund the 2018 International HPC Summer School. The NSF proposal to cover the funding of the summer school was not approved by NSF, resulting in PCR #14 being triggered to ensure sufficient funding for the summer school. PY8 Project Improvement Fund Status Total PIF funds allocated: $356K of $356K

State/Phase Submissions Comments

RY5 IPR12 Page 82 Complete 3 Activity was completed during PY8: • ER Market Analysis • ORCID Membership Fee • 2019 IHPCSS

Funded/Execute 4 No PY8 PIFs moved to Funded this reporting period. PY8 PIF proposals that were funded and continue to be executed: • Update Applications of Parallel Computing online course (PY8 & PY9) • ROI analysis of CI systems (PY8-10) • Longitudinal studies (PY8-10) • Equity & Belonging (PY8-10)

In Process/Planning 2 Proposals received in PY8 that are still in review: • OpenStack Toolkit • ECSS staff training

Withdrawn 2 Two proposals withdrawn due to lack of available funds

Notes: The PY8 PIF funds have been reduced by approximately $108K to fund the 2019 International HPC Summer School. This is due to the lack of NSF funding for the Summer School event. Two PIF submissions that were received during PY8 continue to be reviewed by the SMT. Although all budgeted PIF funds have been fully allocated, the PIF submission window will remain open to collect and prioritize valuable ideas to prepare for possible future unspent funds being reallocated for high impact ideas. PY9 Project Improvement Fund Status Total PIF funds allocated: $465K of $465K

State/Phase Submissions Comments

Complete 1 Funded activities to be completed during PY9 • Duo licenses

Funded/Execute 4 No PIFs moved to Funded this reporting period. Four PIF activities continued from PY8: • Update Applications of Parallel Computing online course (PY8 & PY9) • ROI analysis of CI systems (PY8-10) • Longitudinal studies (PY8-10)

RY5 IPR12 Page 83 • Equity & Belonging (PY8-10)

In Process/Planning 1 Proposals received in PY9 that are still in review: • Integrate online textbook exercises with CCRS toolkit

Withdrawn 2 One proposal withdrawn due to lack of available funds. One proposal was withdrawn because funds were allocated from another area of the project to cover the cost.

One PIF submission that was received during PY9 continues to be reviewed by the SMT. Although all budgeted PIF funds have been fully allocated, the PIF submission window will remain open to collect and prioritize valuable ideas to prepare for possible future unspent funds being reallocated for high impact ideas. PY10 Project Improvement Fund Status While there are no PIF funds budgeted for PY10, there are multi-year PIF funded activities that were funded in PY8 and 9 and will be continued in PY10 using unspent non-PIF project funds from PY6-8.

State/Phase Submissions Comments

Complete 0

Funded/Execute 3 Three activities continue from previous year: • ROI analysis of CI systems (PY8-10) • Longitudinal studies (PY8-10) • Equity & Belonging (PY8-10)

In Process/Planning 0

RY5 IPR12 Page 84 12. Appendices Glossary and List of Acronyms ACRONYM DESCRIPTION Notes A3M Allocations, Accounting & Account Management A&AM Accounting & Account Management A&D Architecture & Design ADR Architecture Design Review AL2S Advanced Layer 2 Service Enables Internet2 users to create point- to-point VLANs AMIE Account Management Information Exchange API Application Programming Interface Area Metric A quantifiable measure that is used to track and assess the status of a specific process. Area Metrics can measure performance or operational status. Area Metrics relating to performance can be used alone or in combinations as a key performance indicator (KPI) for the project. AWS Amazon Web Services

BoF Birds of a Feather Group of community members who informally gather to discuss best practices and/or plans C4C Computing4Change

CaRCC Campus Research Computing Consortium

CB Campus Bridging Infrastructure to make XSEDE resources appear to be proximal to the researcher’s desktop CC Campus Champion CDPs Capability Delivery Plans CEE Community Enhancement & Engagement CERN Organisation Européenne pour la Recherche Nucléaire co-Pi Co-Principal Investigator CRI Cyberinfrastructure Resource Integration CRM Customer Relationship Management CS&E Computational Science & Engineering CSR Community Software Repository CTSC Center for Trustworthy Scientific Infrastructure DNS Domain Name Service DNSKEY Domain Name Service Key DNSSEC DNS Security RY5 IPR12 Page 85 DTS Data Transfer Services E&O Education and Outreach ECSS Extended Collaborative Service e-infrastructure The integration of networks, grids, data centers and collaborative environments, and are intended to include supporting operation centers, service registries, and credential delegation services. ER External Relations ESRT Extended Support for Research Teams ESSGW Extended Collaborative Support for Science Gateways ESTEO Extended Support for Training, Education, & Outreach FTE Full Technical Equivalent GAAMP General Automated Atomic Model Parameterization GFFS Globus Federated File System GridFTP Grid File Transfer Protocol HBCUs Historically Black Colleges and Universities HPC High Performance Computing HPCU HPC University HSI Hispanic Service Institution HSM Hardware Security Models I2 Internet2 IC Industry Challenge IdM Identity Management IGTF Interoperable Global Trust Federation INCA/Nagios A service monitoring tool IPR Interim Project Report IR Incident Reports JIRA an activity tracking tool KB KB documents KPI Key Performance Indicators - A metric or combination of metrics meant to measure performance in key areas of the program so that actions and decisions which move the metrics in the desired direction also move the program in the direction of the desired outcomes and goals. L2 WBS Level 2 L3 WBS Level 3 MFC Minority Faculty Council MS Microsoft MSI Minority Serving Institution MTTR Mean Time To Resolution NCAR National Center for Atmospheric Research NCSA National Center for Supercomputing Applications

RY5 IPR12 Page 86 NICS National Institute of Computational Science NIP Novel & Innovative Projects OSG Open Science Grid OTP One Time Password PEARC Practice & Experience in Advanced Research Computing Conference Series (www.pearc.org) PEP Program Execution Plan perfSONAR PERFormance Service Oriented Network monitoring Architecture PI Principal Investigator PM Project Management/Project Manager PM&R Project Management, Reporting & Risk Management POPS PACI Online Proposal System this is no longer an acronym and POPS is just the name for the allocation submission system; being supplanted by XRAS PRACE Partnership for Advanced Computing in Europe PSC Pittsburgh Supercomputing Center PY Program Year RAS Resource Allocations Service RACD Requirements Analysis & Capability Delivery RDR Resource Description Repository RESTful Representational state transfer rocks roll An open source cluster distribution solution that simplifies the processes of deploying, managing, upgrading, and scaling high-performance parallel computing clusters. RT Request Tracker Ticketing System SACNAS Society for Advancement of Chicanos and Native Americans SCxy Supercomputing Conference (e.g. SC16) SD&I Software Development & Integration SDIACT Software Development & Integration Activity SDSC San Diego Supercomputer Center SecOps Operations - Cybersecurity SH2 SH2 Security Shodor A National Resource for Computational Science Education SP Service Provider SP&E Strategy, Planning, Policy, Evaluation & Organizational Improvement STEM Science Technology Engineering Mathematics SURA Southeastern Universities Research Association SysOps Systems Operations

RY5 IPR12 Page 87 TACC Texas Advanced Computing Center TAGPMA The Americas Grid Policy Management Authority TEOS Training, Education and Outreach Service TAS Technology Audit Service

TeraGrid An e-Science grid computing infrastructure combining resources at eleven partner sites. TTX Table Top Exercise UCCAN Canonical Use Case UCCB Campus Bridging Use Case UCDA Data Analytics Use Case UCDM Data Management Use Case UCF Federation & Interoperation Use Case UCFC First Connecting Instrumentation Use Case UCHPC High Performance Computing Use Case UCHTC High Throughput Computing Use Case UCSGW Science Gateway Use Case UCSW Scientific Workflow Use Case UCVIS Visualization Use Case UE User Engagement UII User Interfaces & Information URC Under-represented communities URCE Under-Represented Community Engagement UREP User Requirement Evaluation & Prioritization URM Under-Represented Minority US United States WBS Work Breakdown Structure Numerical code for each group within XSEDE WISE Wise Information Security for E-infrastructure

WLCG Worldwide LHC Computing Grid

XCBC XSEDE Compatible Basic Cluster Enables campus resource administrators to build a local cluster operating on open source software and compatible with XSEDE supported resources from scratch. XCI XSEDE Cyberinfrastructure Integration

RY5 IPR12 Page 88 XDCDB XSEDE Central Database The XDCDB contains 24 schemas, notably the accounting, resource repository, portal, and AMIE databases.

XDMoD XSEDE Metrics on Demand Comprehensive HPC system management tool

XES XSEDE Enterprise Services XMS XD Net Metrics Services XNIT XSEDE National Integration Toolkit A suite of software modules intended for extant clusters so they are easily interoperable with XSEDE-supported resources. XOC XSEDE Operation Center XRAC XSEDE Resource Allocation Committee XRAS XSEDE Resource Allocations System XSEDE eXtreme Science and Engineering Discovery Environment XSEDE CA XSEDE Certificate Authorities Entity responsible for certifying encryption keys for identity management XSEDE KDC XSEDE Kerberos XSEDE14 XSEDE Conference in 2014 XSEDEnet an XSEDE-only network XSO XSEDE Security Officer XSP XSEDE Scholars Program XSWoG XSEDE Working Group XTED XSEDE Technology Evaluation Database XUP XSEDE User Portal The XSEDE web pages at http://xsede.org XWFS XSEDE Wide File System

Metrics 12.2.1. SP Resource and Service Usage Metrics To demonstrate its success and help focus management attention on areas in need of improvement, XSEDE monitors a wide range of metrics in support of different aspects of “success” for the program. The metrics presented in this section provides a view into XSEDE’s user community, including XSEDE’s success at expanding that community, the projects and allocations through which XSEDE manages access to resources, and the subsequent use of the resources by the community. Table 12-1 summarizes a few key measures of the user community, the projects and allocations, and resource utilization. Expanded information and five-year historical trends are shown in three corresponding subsections. In Q2 2020, XSEDE user community metrics were mixed. For traditional users, the number of open HPC user accounts dropped just below 11,000; the number of active users dropped to 4,200, and the number

RY5 IPR12 Page 89 of institutions represented among the users running jobs declined slightly to 466. These declines are most likely related to an interruption in Jetstream usage reporting that spanned all of Q2 2020; we expect the situation to be resolved soon. However, the number of gateway climbed dramatically to nearly 22,000. The increase was tied to the I-TASSER and Galaxy gateways, both of which deal with protein structure and function, and thus may be a response to the pandemic and studies of SARS-COV-2. More details are in §12.2.1.1. Project and allocation activity held strong, with resource requests about 2.5 times what was available; and the XRAC recommended support for 1.06 times what was available. More details are in §12.2.1.2. Total XSEDE-allocated resource capacity held steady at 18.4 Pflops (peak) with no changes to the XSEDE-allocated portfolio. The central accounting system showed 8 compute resources reporting activity. Altogether, SP resources reported 50.2 billion NUs of computing delivered, a slight increase from the previous quarter. More details are in §12.2.1.3.

Table 12-1: Quarterly activity summary.

User Community Q3 2019 Q4 2019 Q1 2020 Q2 2020 Open user accounts 11,006 10,915 11,110 10,975 Active individuals 4,283 4,386 4,535 4,209 Gateway users 14,863 18,922 16,599 21,941 New user accounts 2,676 1,690 2,511 1,914 Active fields of science 40 40 40 40 Active institutions 551 472 474 466 Projects and Allocations NUs available at XRAC 53.5B 53.5B 67.6B 79.8B NUs requested at XRAC 157.3B 143.7B 144.7B 197.9B NUs recommended by XRAC 72.3BB 70.3B 80.9B 84.5B NUs awarded at XRAC 52.4B 50.6B 70.5B 77.8B Open projects 2.331 2,298 2,298 2,305 Active projects 1,450 1,422 1,432 1,343 Active gateways 19 19 19 16 New projects 222 242 201 326 Closed projects 314 295 287 265 Resources and Usage Resources open (all types) 20 19 23 23 Total peak petaflops 18.5 18.4 18.4 18.4 Resources reporting use 11 9 9 8 Jobs reported 4.33M 3.30M 3.87M 3.36M NUs delivered 49.9B 49.7B 49.9B 50.2B

12.2.1.1. User community metrics Figure 12-1 shows the five-year trend in the XSEDE user community, including open user accounts, total active XSEDE users, active individual accounts, active gateway users, the number of new HPC user

RY5 IPR12 Page 90 accounts, and the total number of new XUP accounts at the end of each quarter. The quarter had 10,975 open accounts and saw 4,209 traditional users charging jobs. The number of active gateway hit a new high of 21,941. Figure 12-2 shows the activity on XSEDE resources according to field of science across program years, including the relative fraction of PIs, open accounts, active users, allocations, and NUs used according to discipline. The figure shows the fields of science that consume ~2% or more of delivered NUs per quarter. PIs and users are counted more than once if they are associated with projects in different fields of science. The quarterly data show that the percentages of PIs and accounts associated with the “other” disciplines represent 30% of all PIs, 35% of direct-access user accounts, and 35% of active users. Collectively the “other” fields of science represented 7.5% of total quarterly usage. Note that XSEDE introduced an updated set of fields of science in Q3 2020 and future charts may show slightly different field of science labels. We have developed a mapping from the prior fields of science to the new ones so that we can report usage across the cutover date.

Figure 12-1: XSEDE user census, excluding XSEDE staff. The dramatic increases in gateway users starting in Q4 2016 are due to the I-TASSER gateway beginning to use XSEDE-allocated resources.

RY5 IPR12 Page 91

Figure 12-2: Quarterly XSEDE user, allocation, and usage summary by field of science, in order by usage, excluding staff projects. Note: PIs and users may appear under more than one field of science. Table 12-2 and Table 12-3 highlight aspects of the broader impact of XSEDE. The former shows that graduate students, postdoctoral researchers, and undergraduates consistently make up two-thirds of the XSEDE user base. The latter table shows XSEDE’s reach into targeted institutional communities, including a substantial increase in representation from MSIs and from EPSCoR state institutions. Institutions with Campus Champions represent a large portion of usage because this table shows all users at Campus Champion institutions, not just those on the champion’s project. The table also shows XSEDE’s reach into EPSCoR states, the MSI community, and countries outside the U.S.

Table 12-2: End of quarter XSEDE open user accounts by type, excluding XSEDE staff.

Category Q3 2019 Q4 2019 Q1 2020 Q2 2020 Graduate Student 4,635 4,273 4,532 4,380 Faculty 2,016 2,051 2,033 2,056 Postdoctoral 1,223 1,261 1,202 1,182 Undergraduate Student 1,460 1,577 1,677 1,745 University Research Staff (excluding postdocs) 609 613 566 580 High school 74 153 164 82 Others 989 987 936 950 TOTALS 11,006 10,915 11,110 10,975

RY5 IPR12 Page 92 Table 12-3: Active institutions in selected categories. Institutions may be in more than one category.

Category Q3 2019 Q4 2019 Q1 2020 Q2 2020 Campus Sites 128 122 123 121 Champions Users 2,260 2,342 2,192 2,086 % total NUs 51% 52% 51% 55% EPSCoR Sites 90 89 81 80 states Users 584 676 573 574 % total NUs 13% 13% 12% 11% MSIs Sites 48 46 43 41 Users 276 379 363 385 % total NUs 2.0% 2.9% 2.3% 3.0% International Sites 117 64 65 72 Users 167 87 82 104 % total NUs 2% 1% 2% 3% Total Sites 551 472 474 466 Users 4,250 4,379 4,515 4,193

12.2.1.2. Project and allocation metrics

Figure 12-3 shows the five-year trend for requests and awards at XSEDE quarterly allocation meetings. NUs requested were 2.5x greater than NUs available, and the XRAC recommendations were about 1.06x greater than the NUs available, once again requiring the reconciliation process to be invoked.

RY5 IPR12 Page 93

Figure 12-3: Five-year allocation history, showing NUs requested, awarded, available, and recommended. Figure 12-4 showing continued high levels of usage and users from these projects. Table 12-4 presents a summary of overall project activity, and Table 12-24 shows projects and activity in key project categories as reflected in allocation board type. Note that Science Gateways may appear under any board. The new Rapid Response project type identifies the projects awarded through the COVID-19 HPC Consortium. As a special class of projects, science gateway activity is detailed in Figure 12-4 showing continued high levels of usage and users from these projects.

Table 12-4: Project summary metrics.

Project metric Q3 2019 Q4 2019 Q1 2020 Q2 2020

XRAC requests 192 219 206 197

XRAC request success 80% 78% 87% 80%

XRAC new awards 48 45 46 50

Startups requested 178 195 198 198

Startups approved 161 197 185 178

Projects new 222 242 201 326

Projects closed 314 295 287 265

RY5 IPR12 Page 94

Table 12-5: Project activity by allocation board type.

Q3 2019 Q4 2019 Q1 2020 Q2 2020

Open Active % Open Active % Open Active % Open Active projects projects NUs projects projects NUs projects projects NUs projects projects % NUs

Campus 156 70 0.25% 151 64 0.3% 149 61 0.2% 154 60 0.1% Champions

Discretionary 4 5 0.11% 6 4 0.0% 6 5 0.1% 10 1 0.1%

Educational 196 106 1.71% 173 100 0.8% 196 88 0.7% 181 76 0.3%

Staff 13 13 0.12% 13 11 0.1% 13 11 0.1% 13 10 0.0%

Startup 1078 473 3.63% 1065 462 2.3% 1049 479 3.0% 1070 446 2.0%

XRAC 884 783 94.18% 890 781 96.5% 885 788 95.9% 877 750 92.8%

Rapid Response 78 4 4.7%

Totals 2,331 1,450 100% 2,298 1,422 100% 2,298 1,432 100% 2,305 1343 100%

Figure 12-4: Quarterly gateway usage (NUs), jobs submitted, users (reported by ECSS), registered gateways, and active gateways. 12.2.1.3. Resource and usage metrics SP systems delivered 50.2 billion NUs in Q2 2020, up slightly from the previous quarter. Table 16 breaks out the resource activity according to different resource types. Figure 12-5 shows the total NUs delivered by XSEDE-allocated SP computing systems, as reported to the central accounting system over the past five years.

RY5 IPR12 Page 95 Table 16: Resource activity, by type of resource, excluding staff projects.

Q3 2019 Q4 2019 Q1 2020 Q2 2020

High-performance Resources 7 7 6 6 computing Jobs 2,583,846 2,583,846 2,527,396 3,294,211

Users 3,742 3,742 4,018 4,166

Nus 46,542,293,315 46,542,293,315 47,240,372,890 49,504,998,604

Data-intensive Resources 2 2 1 1 computing Jobs 131,491 131,491 69,876 58661

Users 139 139 130 129

Nus 247,993,769 247,993,769 283,724,864 306,401,481

High-throughput Resources 1 1 1 1 computing Jobs 1,504,935 19,129 17,542 8,252

Users 532 8 8 4

Nus 2,638,221,585 860,022,066 733,073,988 402,999,559

Cloud system Resources 1 1 1 *

Jobs 19,129 1,980,268 1,231,300 *

Users 8 533 492 *

Nus 860,022,066 3,122,149,022 2,274,951,447 *

Note: A user will be counted for each type of resource used. * The missing usage for Cloud Systems—i.e., Jetstream—in Q2 2020 is due to issues with reporting to XDCDB and not low utilization of that system. We expect the interruption to be resolved soon and the usage to catch up in time for the next report.

RY5 IPR12 Page 96

Figure 12-5: Total XSEDE resource usage in NUs. 12.2.1.4. Data Services XSEDE supports monitoring for the Globus data transfer service for connecting XSEDE service providers and external sites. Table 17 shows summary metrics and increasing Globus adoption over the past five years. Figure 12-6 shows the trends in Globus data transfer activity and user adoption over five years.

RY5 IPR12 Page 97 Table 17: Globus data transfer activity to and from XSEDE endpoints, excluding XSEDE speed page user.

RY4 RP2 RY4 RP3 RY4 RP4 RY5 RP1

Files to XSEDE (millions) 94 37 33 61

TB to XSEDE 2,796 2,392 1,825 2,640 To/from XSEDE Files from XSEDE (millions) 90 159 58 31 endpoint TB from XSEDE 2,633 3,040 2,714 2,761

Users 600 623 719 658

Files to XSEDE (millions) 53 2 7 10

TB to XSEDE 47 48 64 45 To/from XSEDE Files from XSEDE (millions) 39 54 11 7 via Globus Connect TB from XSEDE 151 203 76 164

Users 404 427 519 458

TB to XSEDE 1,195 913 561 751

To/from TB from XSEDE 737 1,050 800 935 XSEDE from/to Campuses Campuses 62 55 50 57

Campus endpoints 80 81 83 89

TB to Campuses 29,323 38,516 30,163 25,565

TB from Campuses 31,847 39,420 31,339 25,235 To/from Campus Campuses 132 133 131 135

Campus endpoints 431 404 437 427

RY5 IPR12 Page 98

Figure 12-6: Top: Aggregate Globus file and data transfer activity to and from XSEDE endpoints. Bottom: Numbers of active Globus users and campuses moving data to and from XSEDE endpoints.

12.2.2. Other Metrics For previous year’s metrics, please refer to the XSEDE Project-wide KPIs & Metrics wiki page.

12.2.2.1. Community Engagement & Enrichment (WBS 2.1) (Gaither) RY5 Metrics RP1 RP2 RP3 RP4 Total Number of sustained users of XSEDE resources 4,500/qtr 4,644 and services via the portal (Project KPI) Number of sustained underrepresented individuals using XSEDE resources and 1,750/yr 831 services via the portal (Project KPI)

Number of new users of XSEDE resources and 2,500/qtr 2,157 services via the portal (Project KPI)

RY5 IPR12 Page 99 Number of new underrepresented individuals using XSEDE resources and services via the 250/qtr 301 portal (Project KPI) Number of participant hours of live training 40,000/yr 19,751 delivered by XSEDE (Project KPI) Number of students benefiting from XSEDE resources and services through training, 2,000/qtr 2,277 XSEDE projects, or conference attendance (Area KPI) Number of underrepresented students benefiting from XSEDE resources and services 650/qtr 741 through training, XSEDE projects, or conference attendance (Area KPI)1 Aggregate mean rating of training impact for attendees registered through the portal (Area 4.4 of 5/qtr 4.6 KPI) Number of institutions with a Champion (Area 340 327 KPI)

Percentage of user requirements addressed 100 98%/qtr within 30 days (Area KPI) (16/16) 1 The reporting of underrepresented students is no longer being reported as a percentage, but instead as a number as of RP4. The use of percentages does not provide the appropriate lens for understanding our progress in engaging underrepresented students.

12.2.2.1.1. Workforce Development (WBS 2.1.2) (Akli) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of unique attendees, synchronous 1,200/yr 1,010 training Number of total attendees, synchronous training 1,400/yr 1,270 (One person can take several classes)

Number of unique attendees, asynchronous 1,200/yr 144 training

Number of total attendees, asynchronous training (One person can take several 4,000/yr 403 classes) Aggregate mean rating of training impact for attendees registered through the portal 4.4 of 5 4.6 (Area KPI) Number of formal degree, minor, and 3/yr 0 certificate programs added to the curricula Number of materials contributed to public 50/yr 5 repository Number of materials downloaded from the 62,000/ 16,895 repository yr Number of computational science modules 40/yr 0 added to courses

RY5 IPR12 Page 100 RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of students benefiting from XSEDE resources and services through training, 1,500/ 2,277 XSEDE projects, or conference attendance qtr (Area KPI) Number of underrepresented students benefiting from XSEDE resources and 500/qtr 741 services through training, XSEDE projects, or conference attendance (Area KPI)1 1 The reporting of underrepresented students is no longer being reported as a percentage, but instead as a number as of RP4. The use of percentages does not provide the appropriate lens for understanding our progress in engaging underrepresented students.

12.2.2.1.2. User Engagement (WBS 2.1.3) (Snead) RY5 Metrics Target RP1 RP2 RP3 RP4 Total

Percentage of active and new PIs contacted 100 100% quarterly (1,217)

Percentage of user requirements addressed 100 98/qtr within 30 days (Area KPI) (16/16) Number of responses to PI emails each 62 quarter Number of responses to each microsurvey NA1 Number of annual user satisfaction survey NA2 respondents interviewed Number of XSEDE-wide tickets 28 Number of XSEDE-wide tickets addressed 28 1 No microsurveys this reporting period. 2 Survey report not yet available.

12.2.2.1.3. Broadening Participation (WBS 2.1.4) (Akli) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of new underrepresented individuals using XSEDE resources and 250/qtr 301 services via the portal (Project KPI) Number of sustained underrepresented individuals using XSEDE resources and 1,750/yr 831 services via the portal (Project KPI)1 1The total for this KPI does not equal the sum of the data from each reporting period because one person could be counted as a sustained individual in more than one reporting period if they continue to log in for multiple reporting periods; however, they will only be counted once in the total.

12.2.2.1.4. User Interfaces & Online Information (WBS 2.1.5) (Dahan)

RY5 Metrics Target RP1 RP2 RP3 RP4 Total

Number of new users of XSEDE resources 2,500/ 2,157 and services via the portal (Project KPI) qtr RY5 IPR12 Page 101 Number of sustained users of XSEDE 4,500/ resources and services via the portal 4,644 qtr (Project KPI)1 80,000/ Number of pageviews to the XSEDE website 42,166 qtr Number of pageviews to the XSEDE User 250,000/ 255,184 Portal qtr User satisfaction with website 4 of 5 4.3 User satisfaction with User Portal 4 of 5 4.3 User satisfaction with user documentation 4 of 5 4.2 1 The total for this KPI does not equal the sum of the data from each reporting period because one person could be counted as a sustained user in more than one reporting period if they continue to log in for multiple reporting periods; however, they will only be counted once in the total.

12.2.2.1.5. Campus Engagement (WBS 2.1.6) (Neeman, Brunson) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of Institutions with a Champion 300 327 (Area KPI) Number of unique contributors to the Champion email list 125/yr 110 ([email protected]) Number of activities that (i) expand the emerging CI workforce and/or (ii) improve 40/yr 64 the extant CI workforce, participated in by members of the Campus Engagement team

12.2.2.2. Extended Collaborative Support Services (WBS 2.2) (Blood, Sinkovits) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Percentage of sustained allocation users from non-traditional disciplines of XSEDE 33%/yr 23.2 resources and services (Project KPI) Percentage of new allocation users from non-traditional disciplines of XSEDE 35%/yr 30.1 resources and services (Project KPI) Number of completed ECSS projects 45/yr 17 (ESRT + ESCC + ESSGW) (Area KPI)

Aggregate mean rating of ECSS impact by PIs 4 of 5/yr 4.1 (Area KPI) Aggregate mean rating of PI satisfaction with 4.5 of 5/yr 4.8 ECSS support (Area KPI) Average estimated months saved due to 12 mo/ 11 ECSS support project

12.2.2.2.1. Extended Support for Research Teams (WBS 2.2.2) (Crosby) RY5 Metrics Target RP1 RP2 RP3 RP4 Total

RY5 IPR12 Page 102 Number of completed ESRT projects 27/yr 5 Average ESRT impact rating 4 of 5/yr 5.0 Average satisfaction with ESRT support 4.5 of 5/yr 4.5 Number of projects initiated 10 Number of projects discontinued 2 Number of PI interviews 2 Number of active projects 28 Average estimated months saved due to ESRT 12 mo/ 12 support project

12.2.2.2.2. Novel & Innovative Projects (WBS 2.2.3) (Sanielevici) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of new users from non-traditional 500/yr 154 disciplines of XSEDE resources and services Number of sustained users from non- traditional disciplines of XSEDE resources 500/qtr 1,833 and services Number of new XSEDE projects from target 30 31 communities generated by NIP Number of successful XSEDE projects from 25 41 target communities mentored by NIP 12.2.2.2.3. Extended Support for Community Codes (WBS 2.2.4) (Koesterke) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of completed ESCC projects 9/yr 2 Average ESCC impact rating 4 of 5/yr 2.0 Average satisfaction with ESCC support 4.5 of 5/yr 5.0 Number of projects initiated 0 Number of projects discontinued 0 Number of active projects 10 Number of PI interviews 1 Average estimated months saved due to ESCC 12 mo/ 0 support project

12.2.2.2.4. Extended Support for Science Gateways (WBS 2.2.5) (Quick) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of completed ESSGW projects 9/yr 10 Average ESSGW impact rating 4 of 5/yr 4.5 Average satisfaction with ESSGW support 4.5 of 5/ yr 5.0 Number of projects initiated 0 Number of projects discontinued 0 Number of active projects 17

RY5 IPR12 Page 103 Number of PI interviews 1 Average estimated months saved due to 12 mo/ 21 ESSGW support project

12.2.2.2.5. Extended Support for Education Outreach, & Training (WBS 2.2.6) (Alameda) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of Campus Champions fellows 4 - Average score of fellows assessment 4.5 of 5 - Number of live training events staffed 20 12

Number of staff training events 2 0 Attendees at staff training events 40 0 Attendees at ECSS Symposia 300 143

Live training event contact hours 26.5 Live training event attendees 185 Live training even attendee hours 676 Requests for service 15

Training modules reviewed 0 Training modules produced 2 Meetings and BoFs 8

Mentoring 9

Talks and presentations 10

Education proposals reviewed 38 - Data reported annually.

12.2.2.3. XSEDE Cyberinfrastructure Integration (WBS 2.3) (Lifka) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Total number of capabilities in production 110 102 (Project KPI) Aggregate mean rating of user satisfaction with XCI software and technical services, capabilities, and resources8 4/5 4.4

Aggregate mean rating of Service Provider satisfaction with XCI software and technical services, capabilities, and resources8 4/5 4.5

Number of non-XSEDE partnerships with 14/yr 14 XCI (Area KPI)

Mean time to issue resolution (Area KPI) 10 days 4.2

RY5 IPR12 Page 104 Number of new capabilities made available 7 102 for production deployment 700 by the Total number of systems that use one or end of RY2 2,196 more CRI provided toolkit

Percentage of Level 1 SPs that fully incorporate all of the recommended tools 100 1001 from the XSEDE Community Repository Percentage of Level 2 SPs that allocate resources through XSEDE that fully 100 332 incorporate all of the recommended tools from the XSEDE Community Repository Percentage of Level 2 SPs that do not allocate resources through XSEDE that fully 100 803 incorporate all of the recommended tools from the XSEDE Community Repository Percentage of Level 3 SPs that fully incorporate all of the recommended tools 100 734 from the XSEDE Community Repository 1All Level 1 SPs are up to date. 2One out of three. One Level 2 SP added (University of Delaware) and not integrated yet; Open Storage Network not integrated yet — joined late Oct.; OSG fully integrated. 34 of 5 have all the required tools installed: RDR entry and information publishing framework. 419 out of 26 have the required RDR entry.

12.2.2.3.1. Requirements Analysis & Capability Delivery (WBS 2.3.2) (Navarro) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Number of capability delivery plans (CDPs) 8/yr 0 prepared for UREP prioritization Number of CI integration assistance 61 7 engagements Average time from support request to 10 days or 3.8

solution less (n=28) Number of new components instrumented 4/yr 0 and tracked for usage and ROI analysis Number of significant fixes and 40/yr1 12 enhancements to production components Number of maintenance releases and upgrades delivered of service provider 4/yr 2 software Number of fixes and enhancements to 36/yr1 10 centrally operated services Operator rating of components delivered 4 of 5/yr NA for production deployment Software/Service Provider rating of our 4 of 5/yr NA integration assistance

RY5 IPR12 Page 105 NA No components were delivered this period. 1 Changes based on PY1-PY8 observed rates.

12.2.2.3.2. Cyberinfrastructure Resource Integration (WBS 2.3.3) (Knepper) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Total number of systems that use one or 1500 by 2,196 more CRI provided toolkits RY5 User satisfaction with CRI services1 4 of 5/yr 4.5 Number of repository subscribers to CRI 150 114 cluster and laptop toolkits

Aggregate number of TeraFLOPS of cluster 200/yr 1,020 systems using CRI toolkits Number of partnership interactions between CRI and SPs, national CI 12 7 organizations, and campus CI providers Toolkit updates 4/yr 3 New Toolkits released 2/yr 1

Average time from support request to 10.5 <14 days solution (n=2) 1This KPI was retired after RP1. Going forward satisfaction with XCI services will be measured at the L2 level only.

12.2.2.4. XSEDE Operations (WBS 2.4) (Peterson) RY4 Metrics Target RP1 RP2 RP3 RP4 Total Average composite availability of core services (geometric mean of critical 99.9%/qtr 99.9 services and XRAS) (Project KPI) Hours of downtime with direct user impacts from an XSEDE security incident 0/qtr 0 (Area KPI) Mean time to ticket resolution by XOC and <16 /qtr 15.3 WBS ticket queues (hrs) (Project KPI) Mean rating of user satisfaction with 4.5 of 5/qtr 4.7 tickets closed by the XOC (Area KPI)

12.2.2.4.1. Cybersecurity (WBS 2.4.2) (Withers, Simmel) RY5 Metrics Target RP1 RP2 RP3 RP4 Total

Hours of downtime with direct user impacts from an XSEDE security incident. 0/qtr 0 (Area KPI)

Hours of downtime WITHOUT direct user impacts from an XSEDE (affects central < 24 0 service or multiple SPs) security incident XSEDE account exposures < 10 4 Time, beyond 24 hours, to disable XSEDE 0 0 accounts RY5 IPR12 Page 106

12.2.2.4.2. Data Transfer Services (WBS 2.4.3) (Wheeler) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Performance (Gbps) of instrumented, 3.0 Gbps 3.17 intra-XSEDE transfers > 1GB New services added 0 Services retired 0 Total Globus Online users 658 Total new Globus Online users 197 Total transfers (Million) inbound 61 Total transfers (Million) outbound 31 Size of transfers (TBs) inbound 2,640 Size of transfers (TBs) outbound 2,761 Total number of days in which any 0 Network Interface error occurred XSEDEnet maximum bandwidth used 146.5 (Gbps)

12.2.2.4.3. XSEDE Operations Center (WBS 2.4.4) (Hendricks) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Mean time to resolution in XOC queue < 1 0.46 (hrs) Mean time to ticket resolution by XOC and < 16 15.3 WBS ticket queues (hrs) (Project KPI) User satisfaction with tickets closed by 4.5 of 5 4.7 the XOC (Area KPI) Mean time to resolution in WBS queue 36.6 Number of Support tickets opened for 304 WBS queues Number of Support tickets closed by WBS 265 queues Number of Support tickets opened for 381 XOC Number of Support tickets closed by XOC 381 Mean time to first response by XOC (hrs) 0.44 12.2.2.4.4. System Operations Support (WBS 2.4.5) (Rogers) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Average availability of critical enterprise services (%) [geometric mean] (Project 99.9% 99.9% KPI) Average availability of core enterprise 99.9% 99.9% services (%) Total enterprise services 47

RY5 IPR12 Page 107 RY5 Metrics Target RP1 RP2 RP3 RP4 Total Core enterprise services 8 Services added/subtracted 0

12.2.2.5. Resource Allocation Service (WBS 2.5) (Hart) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Mean rating of user satisfaction with 4 of 5/qtr 4.4 allocations process (Area KPI) Mean rating of user satisfaction with XRAS 4 of 5/qtr 4.3 (Area KPI)1 Mean rating of user satisfaction with allocations process and support services 4 of 5/yr 4.4 (Project KPI)1 Percentage of research requests successful 85%/qtr 80.0 (not rejected) (Project KPI) 12.2.2.5.1. XSEDE Allocations Process & Policies (WBS 2.5.2.) (Hackworth) RY5 Metrics Target RP1 RP2 RP3 RP4 Total User satisfaction with allocations process 4 of 5 4.4 (Area KPI) Average time to process Startup requests 14 calendar days or 8.5 less/qtr Percentage of XRAC-recommended SUs 100% 92% allocated Percentage of research requests successful 85%/qtr 80.0% (not rejected) (Project KPI) Continuous allocation requests processed 780

Research allocation requests processed 197

12.2.2.5.2. Allocations, Accounting, & Account Management CI (WBS 2.5.3) (Tolbert)

RY5 Metrics Target RP1 RP2 RP3 RP4 Total

User satisfaction with XRAS system (Area 4 of 5 4.3 KPI)

Availability of the XRAS systems 99.9% 99.9%

Number of XRAC client organizations 8

12.2.2.6. Program Office (WBS 2.6) (Payne) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Aggregate mean rating of user awareness of 3.7 of 5 / yr 3.8 XSEDE resources and services (Project KPI)

Number of social media impressions over 429,282 / 86,046 time (Project KPI) yr

RY5 IPR12 Page 108 RY5 Metrics Target RP1 RP2 RP3 RP4 Total Aggregate mean rating of user satisfaction with XSEDE technical support services 3.5 of 5/yr 4.4 (Project KPI) Percentage of users who indicate the use of XSEDE-managed and/or XSEDE-associated 80/yr 83 resources in the creation of their work product (Project KPI) Mean rating of importance of XSEDE resources and services to researcher 4.4 of 5/yr 4.2 productivity (Project KPI) Percentage of Project Improvement Fund funded projects resulting in innovations in 70%/yr - the XSEDE organization (Project KPI) Mean rating of innovation within the 4 of 5/yr - organization by XSEDE staff (Project KPI) Variance between relevant report submission 0 0 and due date (days) (Area KPI) Percentage of sub-award invoices processed 90%/qtr 92.6 within target duration (Area KPI) Percentage of recommendations addressed by relevant project areas within 90 days 90% 100 (Area KPI) Aggregate mean rating of satisfaction with content and accessibility of the XSEDE Staff 3.9 of 5/yr - Wiki (Area KPI) Number of staff publications (Area KPI) 50/yr 2 Aggregate mean rating of Inclusion in XSEDE 4.1/yr - (Area KPI) Aggregate mean rating of Equity in XSEDE 4.0/yr - (Area KPI) Number of XSEDE-related media hits (Area 325/yr 624 KPI) - Data reported annually 12.2.2.6.1. External Relations (WBS 2.6.2) (Hutson) RY4 Metrics Target RP1 RP2 RP3 RP4 Total Number of social media impressions over 429,282/ 86,046 time (Project KPI) yr Number of XSEDE-related media hits (Area 325/yr 718 KPI) Open: Open: 32% 35% Monthly open and click-through rates of Click- Click- XSEDE’s newsletter through: through: 3% 1.3%

RY5 IPR12 Page 109 12.2.2.6.2. Project Management, Reporting, & Risk Management (WBS 2.6.3) (Froeschl) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Variance, in days, between relevant report 0/report 0 submission and due date (Area KPI) Aggregate mean rating of satisfaction with content and accessibility of the XSEDE Staff 3.9 of 5/yr - Wiki (Area KPI) Percentage of risks reviewed 100% 100% Number of total risks 174 Number of active risks 140 Number of new risks 1 Number of risks triggered 4 Number of risks retired 1 Number of PCRs submitted 4 KPI/Metrics 3 Technical 0 Scope 0 Budget 0 Staff 1 Other 0 - Data reported annually. 12.2.2.6.3. Business Operations (WBS 2.6.4) (Payne) RY5 Metrics Target RP1 RP2 RP3 RP4 Total Percentage of subaward invoices processed 90%/qtr 92.6 within target duration (Area KPI) 12.2.2.6.4. Strategic Planning, Policy, Evaluation & Organizational Improvement (WBS 2.6.5) (Payne) RY5Metrics Target RP1 RP2 RP3 RP4 Total Aggregate mean rating of user awareness of 3.7 of 5 / yr 3.8 XSEDE resources and services (Project KPI) Aggregate mean rating of user satisfaction with XSEDE technical support services 3.5 of 5/yr 4.4 (Project KPI) Percentage of users who indicate the use of XSEDE-managed and/or XSEDE-associated 80%/yr 83 resources in the creation of their work product1 (Project KPI) Mean rating of importance of XSEDE resources and services to researcher 4.4 of 5/yr 4.2 productivity (Project KPI) Percentage of Project Improvement Fund proposals resulting in innovations in the 70%/yr - XSEDE organization (Project KPI) Mean rating of innovation within the 4 of 5/yr - organization by XSEDE staff (Project KPI)

RY5 IPR12 Page 110 RY5Metrics Target RP1 RP2 RP3 RP4 Total Percentage of recommendations addressed by relevant project areas within 90 days 90% 100 (Area KPI) Number of staff publications (Area KPI) 50/yr 2 Aggregate mean rating of Inclusion in XSEDE 4.1/yr - (Area KPI) Aggregate mean rating of Equity in XSEDE 4.0/yr - (Area KPI) - Data reported annually.

RY5 IPR12 Page 111 Scientific Impact Metrics (SIM) and Publications Listing This appendix presents the current Scientific Impact Metrics data as of July 21 of year 2020. This is part of the XD Metrics Service (XMS) (formerly NSF Technology Audit Service (TAS)) effort. 12.3.1. Summary Impact Metrics Table SIM-1 shows the essential scientific summary impact metrics as of July 31 of year 2020. The increasing values for each metric are listed in the table indicating the changes during the last quarter. By calculating such metrics periodically we can show the trends, as depicted in Figure SIM- 2 and Figure SIM-3. Both show steadily increasing trends.

Table SIM-1: Overall Scientific Impact Metrics Data Number of i10-index Overall externally (Number of citation h-index g-index verified unique publications cited count* publications* at least 10 times) Since 2005 (TG+XD) 18,389 10,474 640,878 263 473 Since 2011 (XD) 15,841 8,439 438,305 209 365 Change since last quarter (TG+XD) +548 +421 +34,943 +8 +13 Change since last quarter (XD) +546 +416 +31,062 +9 +16 * Data updated as of July 31st, 2020.

12.3.2. Historical Trend Figure SIM-2 and SIM-3 show the increasing quarterly trend regarding publications, citations, and other impact metrics such as H-Index and G-Index. Both suggest the increasing impact of XSEDE during the past years, based on verified unique publication count; citation count; H-index and G- index.

RY5 IPR12 Page 112

Figure SIM-2. Counts of all externally verified publications for TG/XD (since 2005) and XD (since 2011) and of those being cited at least 10 times (i10-index).

Figure SIM-3. Accumulated citation count (line, left axis) as well as h-index and g-index metrics (bar, right axis) for TG/XD (since 2005) and XD (since 2011).

12.3.3. Publications Listing shows the number of publications, conference papers, and presentations reported by XSEDE users each quarter, including the 617 reported by 190 projects in Q2 2020; these publications are listed below according to allocated project. Starting with the December 2017 XRAC meeting, all submitters must add publications to their user profiles in the XSEDE User Portal, which may have contributed to the decline beginning at the end of 2017.

RY5 IPR12 Page 113

Figure 12-7 shows the number of publications, conference papers, and presentations reported by XSEDE users each quarter, including the 617 reported by 190 projects in Q2 2020; these publications are listed below according to allocated project. Starting with the December 2017 XRAC meeting, all submitters must add publications to their user profiles in the XSEDE User Portal, which may have contributed to the decline beginning at the end of 2017.

Figure 12-7: Publications, conference papers, and presentations reported by XSEDE users.

12.3.3.1. XSEDE Staff Publications XSEDE staff reported 2 publications from May 2020 to July 2020. One publication was reported via staff members’ XSEDE User Portal user profiles, and one was published by staff in the XSEDE Digital Object Repository (XDOR). 1. Hovious, A., Navarro, J. 2019. Discovering Research IT Resources Across Campuses: Collaboration Is Key. Poster presented at the EDUCAUSE 2019 Annual Conference. https://events.educause.edu/annual- conference/2019/agenda/discovering-research-it-resources-across-campuses-collaboration-is-key. (published)

RY5 IPR12 Page 114 2. XSEDE. 2020. XRAS Client Administrator’s Guide, Version 1.0. This document was compiled to help the allocation administrators at XSEDE Resource Allocation Service (XRAS) client sites understand the features provided by the system. http://hdl.handle.net/2142/107178

12.3.3.2. Publications from XSEDE Users The following publications were submitted by users to their XSEDE User Portal profiles in Q2 2020. Most publications are associated with submissions to the June 2020 meeting, while some may be from Startup or other projects. Also note that the final set of publications are associated with “XSEDE Allocation Request” and not a specific project; we will be working to encourage users to associate such publications with the proper XSEDE project. The publications are organized by the proposal with which they were associated. This quarter, 190 projects identified 617 publications and other products that were published, in press, accepted, submitted, or in preparation. Because these publications are submitted by users and not manually verified by XSEDE staff, there is a small chance of data entry error.

1. TG-ASC050039N 1. Choi, J., D. F. Richards, L. V. Kale, and A. Bhatele (2020), End-to-end performance modeling of distributed GPU applications, Proceedings of the 34th ACM International Conference on Supercomputing, doi:10.1145/3392717.3392737. (published) [Bridges GPU, PSC]

2. TG-ASC090076 2. Haley, C. L., and X. Zhong (2020), Mode F/S Wave Packet Interference And Acoustic-like Emissions in a Mach 8 Flow Over a Cone, AIAA Scitech 2020 Forum, doi:10.2514/6.2020-1579. (published) [Comet, Stampede] 3. He, S., and X. Zhong (2020), Hypersonic Boundary Layer Receptivity over a Blunt Cone to Freestream Pulse Disturbances, AIAA Scitech 2020 Forum, doi:10.2514/6.2020-2057. (published) [Comet, SDSC, Stampede, TACC]

3. TG-ASC100004, TG-CHE140079 4. Hendinejad, N., and Q. K. Timerghazin (2020), Biological control of S-nitrosothiol reactivity: potential role of sigma- hole interactions, Physical Chemistry Chemical Physics, 22(12), 6595–6605, doi:10.1039/c9cp06377c. (published) [Comet, SDSC] 5. Hosseininasab, V., A. C. McQuilken, A. (Gus) Bakhoda, J. A. Bertke, Q. K. Timerghazin, and T. H. Warren (2020), Lewis Acid Coordination Redirects S‐Nitrosothiol Signaling Output, Angewandte Chemie International Edition, 59(27), 10854–10858, doi:10.1002/anie.202001450. (published) [Comet, SDSC] 6. Mirzaei, S., M. V. Ivanov, and Q. K. Timerghazin (2019), Improving Performance of the SMD Solvation Model: Bondi Radii Improve Predicted Aqueous Solvation Free Energies of Ions and pKa Values of Thiols, The Journal of Physical Chemistry A, 123(44), 9498–9504, doi:10.1021/acs.jpca.9b02340. (published) [Comet, SDSC] 7. Mirzaei, S., D. Wang, S. V. Lindeman, Q. K. Timerghazin, and R. Rathore (2019), Redox-Induced Molecular Actuators: The Case of Oxy-Alternate Bridged Cyclotetraveratrylene, Organic Letters, 21(19), 7987–7991, doi:10.1021/acs.orglett.9b02971. (published) [Comet, SDSC]

4. TG-ASC150027 8. Hicks, W., Wells, A., Norman, M., Wise, J., Smith, B., et al. 2020. External Enrichment of Minihalos by the First Supernovae. (submitted)

5. TG-ASC160018, TG-MCB150104 9. Spellmon, N., X. Sun, N. Sirinupong, B. Edwards, C. Li, and Z. Yang (2015), Molecular Dynamics Simulation Reveals Correlated Inter-Lobe Motion in Protein Lysine Methyltransferase SMYD2, edited by S. J. Du, PLOS ONE, 10(12), e0145758, doi:10.1371/journal.pone.0145758. (published) 10. Spellmon, N., X. Sun, N. Sirinupong, B. Edwards, C. Li, and Z. Yang (2015), Molecular Dynamics Simulation Reveals Correlated Inter-Lobe Motion in Protein Lysine Methyltransferase SMYD2, edited by S. J. Du, PLOS ONE, 10(12), e0145758, doi:10.1371/journal.pone.0145758. (published)

RY5 IPR12 Page 115 11. Spellmon, N. et al. (2016), New open conformation of SMYD3 implicates conformational selection and allostery, AIMS Biophysics, 4(1), 1–18, doi:10.3934/biophy.2017.1.1. (published)

6. TG-ASC180058 12. Kalyanam, R., Zhao, L., Jin, J., Biehl, L., Campbell, R., et al. 2019. GeoEDF: An Extensible Geospatial Data Framework for FAIR Science. Poster at AGU 2019. (published) [IU, Jetstream, Science Gateways]

7. TG-ASC190032 13. Castillo, P., Rodriguez, S., Wan, D., Yu, k., Gross, B., et al. 2019. Numerical Study of Coherent Radiation from Induced Plasma Dipole Oscillation by Detuned Laser Pulses. North American Particle Accelerator - NAPAC 2019 (Lansing, MI, USA). https://ref.ipac19.org/reference/show/94636. (published)

8. TG-ASC190071 14. Volvach, I., J. G. Alzate, Y.-J. Chen, A. J. Smith, D. L. Kencke, and V. Lomakin (2020), Thermal stability and magnetization switching in perpendicular magnetic tunnel junctions, Applied Physics Letters, 116(19), 192408, doi:10.1063/5.0005211. (published) [Bridges GPU, Comet, PSC, Pylon, Science Gateways, SDSC] 15. Volvach, I., Alzate, J., Chen, Y., Smith, A., Lomakin, V. 2020. Thermal stability and magnetization switching in Perpendicular Magnetic Tunnel Junctions. (submitted)

9. TG-AST050012N 16. Anninos, P., P. C. Fragile, S. S. Olivier, R. Hoffman, B. Mishra, and K. Camarda (2018), Relativistic Tidal Disruption and Nuclear Ignition of White Dwarf Stars by Intermediate-mass Black Holes, The Astrophysical Journal, 865(1), 3, doi:10.3847/1538-4357/aadad9. (published) [Ranch, Stampede] 17. Anninos, P., R. D. Hoffman, M. Grewal, M. J. Lavell, and P. C. Fragile (2019), Nuclear Ignition of White Dwarf Stars by Relativistic Encounters with Rotating Intermediate Mass Black Holes, The Astrophysical Journal, 885(2), 136, doi:10.3847/1538-4357/ab4ae0. (published) [Ranch, Stampede] 18. Fragile, P. C., D. R. Ballantyne, and A. Blankenship (2020), Interactions of type I X-ray bursts with thin accretion disks, Nature Astronomy, 4(5), 541–546, doi:10.1038/s41550-019-0987-5. (published) [Ranch, Stampede] 19. Fragile, P. C., D. R. Ballantyne, T. J. Maccarone, and J. W. L. Witry (2018), Simulating the Collapse of a Thick Accretion Disk due to a Type I X-Ray Burst from a Neutron Star, The Astrophysical Journal, 867(2), L28, doi:10.3847/2041- 8213/aaeb99. (published) [Ranch, Stampede] 20. Fragile, P. C., D. Nemergut, P. L. Shaw, and P. Anninos (2019), Divergence-free magnetohydrodynamics on conformally moving, adaptive meshes using a vector potential method, Journal of Computational Physics: X, 2, 100020, doi:10.1016/j.jcpx.2019.100020. (published) [Ranch, Stampede] 21. athing oscillations in a global simulation of a thin accretion disc, Monthly Notices of the Royal Astronomical Society, 483(4), 4811–4819, doi:10.1093/mnras/sty3124. (published) Mishra,[Ranch, B., Stampede W. Kluźniak,] and P. C. Fragile (2018), Bre

10. TG-AST090107 22. Campante, T. L. et al. (2019), TESS Asteroseismology of the Known Red-giant Host Stars HD 212771 and HD 203949, The Astrophysical Journal, 885(1), 31, doi:10.3847/1538-4357/ab44a8. (published) [Science Gateways, Stampede2, TACC] 23. Chaplin, W. J. et al. (2020), Age dating of an early Milky Way merger via asteroseismology of the naked- Nature Astronomy, 4(4), 382–389, doi:10.1038/s41550-019-0975-9. (published) [Science Gateways, Stampede2, TACC] eye star ν Indi, 24. Huber, D., Chaplin, W., Chontos, A., Kjeldsen, H., Christensen-Dalsgaard, J., et al. 2019. A Hot Saturn Orbiting an Oscillating Late Subgiant Discovered by TESS. (published) [Science Gateways, Stampede2, TACC] 25. Karoff, C., T. S. Metcalfe, B. T. Montet, N. E. Jannsen, A. R. G. Santos, M. B. Nielsen, and W. J. Chaplin (2019), Sounding stellar cycles with Kepler – III. Comparative analysis of chromospheric, photometric, and asteroseismic variability, Monthly Notices of the Royal Astronomical Society, 485(4), 5096–5104, doi:10.1093/mnras/stz782. (published) [Science Gateways, Stampede2, TACC] 26. Santos, A. R. G. et al. (2019), Signatures of Magnetic Activity: On the Relation between Stellar Properties and p-mode Frequency Variations, The Astrophysical Journal, 883(1), 65, doi:10.3847/1538-4357/ab397a. (published) [Science Gateways, Stampede2, TACC]

RY5 IPR12 Page 116 11. TG-AST100035 27. Xu, R., Spitkovsky, A. 2019. Electron acceleration in non-relativistic quasi-perpendicular collisionless shocks. (submitted)

12. TG-AST130002 28. Yang, C.-C., and Z. Zhu (2019), Morphological signatures induced by dust back reaction in discs with an embedded planet, Monthly Notices of the Royal Astronomical Society, doi:10.1093/mnras/stz3232. (published) [Stampede, Stampede2, TACC]

13. TG-AST130021 29. Buie, E., M. Fumagalli, and E. Scannapieco (2020), Interpreting Observations of Absorption Lines in the Circumgalactic Medium with a Turbulent Medium, The Astrophysical Journal, 890(1), 33, doi:10.3847/1538- 4357/ab65bc. (published) [Stampede2, TACC] 30. Safarzadeh, M., R. Sarmento, and E. Scannapieco (2019), On Neutron Star Mergers as the Source of r-process- enhanced Metal-poor Stars in the Milky Way, The Astrophysical Journal, 876(1), 28, doi:10.3847/1538-4357/ab1341. (published) [Stampede, TACC] 31. Sarmento, R., E. Scannapieco, and B. Côté (2019), Following the Cosmic Evolution of Pristine Gas. III. The Observational Consequences of the Unknown Properties of Population III Stars, The Astrophysical Journal, 871(2), 206, doi:10.3847/1538-4357/aafa1a. (published) [Stampede, TACC]

14. TG-AST140023 32. Benincasa, S., Loebman, S., Wetzel, A., Hopkins, P., Murray, N., et al. 2019. Live Fast, Die Young: GMC lifetimes in the FIRE cosmological simulations of Milky Way-mass galaxies. (published) [Ranch, Stampede2, TACC] 33. Besla, G., Huppenkothen, D., Lloyd-Ronning, N., Schneider, E., Behroozi, P., et al. 2019. Training the Future Generation of Computational Researchers. 51. 11. (published) [Ranch, Stampede2, TACC] 34. Bozek, B. et al. (2018), Warm FIRE: simulating galaxy formation with resonant sterile neutrino dark matter, Monthly Notices of the Royal Astronomical Society, 483(3), 4086–4099, doi:10.1093/mnras/sty3300. (published) [Ranch, Stampede2, TACC] 35. Chen, H., Johnson, S., Rudie, G., Simcoe, R., Boettcher, E., et al. 2019. Tracking the Baryon Cycle in Emission and in Absorption. (published) [Ranch, Stampede2, TACC] 36. Debattista, V. P., O. A. Gonzalez, R. E. Sanderson, K. El-Badry, S. Garrison-Kimmel, A. Wetzel, C.-A. Faucher-Giguère, and P. F. Hopkins (2019), Formation, vertex deviation, and age of the Milky Way’s bulge: input from a cosmological simulation with a late-forming bar, Monthly Notices of the Royal Astronomical Society, 485(4), 5073–5085, doi:10.1093/mnras/stz746. (published) [Ranch, Stampede2, TACC] 37. Faucher-Giguere, C. 2018. The Physics, Observational Signatures, and Consequences of AGN-driven Galactic Winds. Multiphase AGN Feeding & Feedback; Linking the Micro to Macro Scales in Galaxies, Groups, and Clusters. 17. (published) [Ranch, Stampede2, TACC] 38. Faucher-Giguère, C.-A. (2018), Recent progress in simulating galaxy formation from the largest to the smallest scales, Nature Astronomy, 2(5), 368–373, doi:10.1038/s41550-018-0427-y. (published) [Ranch, Stampede2, TACC] 39. Faucher-Giguère, C.-A. (2020), A cosmic UV/X-ray background model update, Monthly Notices of the Royal Astronomical Society, 493(2), 1614–1632, doi:10.1093/mnras/staa302. (published) [Ranch, Stampede2, TACC] 40. Fitts, A. et al. (2019), Dwarf galaxies in CDM, WDM, and SIDM: disentangling baryons and dark matter physics, Monthly Notices of the Royal Astronomical Society, 490(1), 962–977, doi:10.1093/mnras/stz2613. (published) [Ranch, Stampede2, TACC] 41. Garrison-Kimmel, S. et al. (2019), The Local Group on FIRE: dwarf galaxy populations across a suite of hydrodynamic simulations, Monthly Notices of the Royal Astronomical Society, 487(1), 1380–1399, doi:10.1093/mnras/stz1317. (published) [Ranch, Stampede2, TACC] 42. Garrison-Kimmel, S. et al. (2019), Star formation histories of dwarf galaxies in the FIRE simulations: dependence on mass and Local Group environment, Monthly Notices of the Royal Astronomical Society, 489(4), 4574–4588, doi:10.1093/mnras/stz2507. (published) [Ranch, Stampede2, TACC] 43. Graus, A. S. et al. (2019), A predicted correlation between age gradient and star formation history in FIRE dwarf galaxies, Monthly Notices of the Royal Astronomical Society, 490(1), 1186–1201, doi:10.1093/mnras/stz2649. (published) [Ranch, Stampede2, TACC]

RY5 IPR12 Page 117 44. Gronke, M., Faucher-Giguere, C., Hafen, Z., Oh, S. 2019. The Transport and Growth of Cold Gas from Galactic Winds into the Circumgalactic Medium. (published) [Ranch, Stampede2, TACC] 45. Grudic, M., Boylan-Kolchin, M., Faucher-Giguère, C., Hopkins, P. 2019. Stellar feedback sets the universal acceleration scale in galaxies. (published) [Ranch, Stampede2, TACC] 46. -A. Faucher-Giguère, and L. C. Johnson (2019), On the nature of variations in the measured star formation efficiency of molecular clouds, Monthly Notices of the Royal Astronomical Society,Grudić, M.488(2), Y., P. F.1501 Hopkins,–1518, E. doi:10.1093/mnras/stz1758. J. Lee, N. Murray, C. (published) [Ranch, Stampede2, TACC] 47. R. Offner, M. Boylan-Kolchin, C.-A. Faucher-Gigère, A. Wetzel, S. M. Benincasa, and S. Loebman (2019), Evolution of giant molecular clouds across cosmic time, Monthly Notices of the Royal Astronomical Society,Guszejnov, 492(1), D., M. 488 Y. Grudić,–502, doi:10.1093/mnras/stz3527. S. S. (published) [Ranch, Stampede2, TACC] 48. Hafen, Z., Faucher-Giguere, C., Angles-Alcazar, D., Stern, J., Keres, D., et al. 2019. The Fates of the Circumgalactic Medium in the FIRE Simulations. (published) [Ranch, Stampede2, TACC] 49. Hani, M. H., C. C. Hayward, M. E. Orr, S. L. Ellison, P. Torrey, N. Murray, A. Wetzel, and C.-A. Faucher-Giguère (2020), Variations in the slope of the resolved star-forming main sequence: a tool for constraining the mass of star-forming regions, Monthly Notices of the Royal Astronomical Society: Letters, 493(1), L87–L91, doi:10.1093/mnrasl/slaa013. (published) [Ranch, Stampede2, TACC] 50. Hopkins, P., Chan, T., Squire, J., Quataert, E., Ji, S., et al. 2020. Effects of Different Cosmic Ray Transport Models on Galaxy Formation. (published) [Ranch, Stampede2, TACC] 51. -A. Faucher-Giguère, X. Ma, N. Murray, and N. Butcher (2019), Radiative Stellar Feedback in Galaxy Formation: Methods and Physics, Monthly Notices of the Royal Astronomical Hopkins,Society, doi:10.1093/mnras/stz P. F., M. Y. Grudić, A. Wetzel,3129. D.(published Kereš, C.) [Ranch, Stampede2, TACC] 52. Johnson, S., Stern, J., Wylezalek, D., Alatalo, K., Chen, H., et al. 2019. UV diagnostics as barometers for galactic scale AGN outflows. (published) [Ranch, Stampede2, TACC] 53. Kimock, B., Narayanan, D., Smith, A., Feldmann, R., Ma, X., et al. 2020. The Origin of Lyman Alpha Blobs in Cosmological Galaxy Formation Simulations. American Astronomical Society Meeting Abstracts. 322.08. (published) [Ranch, Stampede2, TACC] 54. Lehner, N., Berek, S., Howk, J., Wakker, B., Tumlinson, J., et al. 2020. Project AMIGA: The Circumgalactic Medium of Andromeda. (published) [Ranch, Stampede2, TACC] 55. Liang, L., R. Feldmann, C.-A. Faucher- (2018), Submillimetre flux as a probe of molecular ISM mass in high-z galaxies, Monthly Notices of the Royal Astronomical Society: Letters, 478(1),Giguère, L83–L88, D. Kereš,doi:10.1093/mnrasl/sly071. P. F. Hopkins, C. C. Hayward, (published E. Quataert,) [Ranch, and Stampede2, N. Z. Scoville TACC ] 56. Ma, X., -A. Faucher-Giguère, M. Boylan-Kolchin, A. Wetzel, J. Kim, N. Murray, -consistent proto-globular cluster formation in cosmological simulations of high-redshift galaxies,M. Monthly Y. Grudić, Notices E. Quataert, of the P.Royal F. Hopkins, Astronomical C. Society, 493(3), 4315–4332, doi:10.1093/mnras/staa527. (published)and D. Kereš [Ranch, (2020), Stampede2, Self TACC] 57. Ma, X., Quataert, E., Wetzel, A., Hopkins, P., Faucher-Giguère, C., et al. 2020. No missing photons for reionization: moderate ionizing photon escape fractions from the FIRE-2 simulations. (published) [Ranch, Stampede2, TACC] 58. Orr, M., Hayward, C., Medling, A., Hopkins, P., Murray, N., et al. 2019. Swirls of FIRE: Spatially Resolved Gas Velocity Dispersions and Star Formation Rates in FIRE-2 Disk Environments. (published) [Ranch, Stampede2, TACC] 59. Peeples, M., Behroozi, P., Bordoloi, R., Brooks, A., Bullock, J., et al. 2019. Understanding the circumgalactic medium is critical for understanding galaxy evolution. (published) [Ranch, Stampede2, TACC] 60. Richings, A., Faucher-Giguere, C. 2018. Molecular Emission Lines from Simulations of AGN-Driven Molecular Outflows. Walking the Line 2018. DOI:10.5281/zenodo.1209445. (published) [Ranch, Stampede2, TACC] 61. Richings, A. J., and C.-A. Faucher-Giguère (2018), Radiative cooling of swept-up gas in AGN-driven galactic winds and its implications for molecular outflows, Monthly Notices of the Royal Astronomical Society, 478(3), 3100–3119, doi:10.1093/mnras/sty1285. (published) [Ranch, Stampede2, TACC] 62. Rudie, G., Chen, H., Newman, A., Johnson, S., Simcoe, R., et al. 2019. Observing Galaxies and Dissecting their Baryon Cycle at Cosmic Noon. (published) [Ranch, Stampede2, TACC] 63. Ruszkowski, M., Nagai, D., Zhuravleva, I., Brummel-Smith, C., Li, Y., et al. 2019. Supermassive Black Hole Feedback. (published) [Ranch, Stampede2, TACC]

RY5 IPR12 Page 118 64. Samuel, J. et al. (2019), A profile in FIRE: resolving the radial distributions of satellite galaxies in the Local Group with simulations, Monthly Notices of the Royal Astronomical Society, 491(1), 1471–1490, doi:10.1093/mnras/stz3054. (published) [Ranch, Stampede2, TACC] 65. Sanderson, R. E., A. Wetzel, S. Loebman, S. Sharma, P. F. Hopkins, S. Garrison-Kimmel, C.-A. Faucher- and E. Quataert (2020), Synthetic Gaia Surveys from the FIRE Cosmological Simulations of Milky Way-mass Galaxies, The Astrophysical Journal Supplement Series, 246(1), 6, doi:10.3847/1538-4365/ab5b9d. (publishedGiguère,) [Ranch, D. Kereš, Stampede2, TACC] 66. Santistevan, I., Wetzel, A., El-Badry, K., Bland-Hawthorn, J., Boylan-Kolchin, M., et al. 2020. Growing Pains: The Formation Times and Building Blocks of Milky Way-mass Galaxies in the FIRE Simulations. (published) [Ranch, Stampede2, TACC] 67. Shen, X., Hopkins, P., Faucher-Giguère, C., Alexander, D., Richards, G., et al. 2020. The Bolometric Quasar Luminosity Function at z = 0-7. (published) [Ranch, Stampede2, TACC] 68. Shiferaw, M., Gurvich, A., Geller, A., Faucher-Giguère, C., Richings, A. 2019. Visualizing HII Regions in FIRE Galaxy Simulations. American Astronomical Society Meeting Abstracts #233. 233. 253.05. (published) [Ranch, Stampede2, TACC] 69. Smith, A., X. Ma, V. Bromm, S. L. Finkelstein, P. F. Hopkins, C.-A. Faucher- Lyman -redshift galaxies, Monthly Notices of the Royal Astronomical Society, 484(1), 39–59, doi:10.1093/mnras/sty3483. (published) [Ranch, Stampede2, TACC] Giguère, and D. Kereš (2018), The physics of α escape from high 70. Stern, J., C.-A. Faucher-Giguère, J. F. Hennawi, Z. Hafen, S. D. Johnson, and D. Fielding (2018), Does Circumgalactic O vi Trace Low-pressure Gas Beyond the Accretion Shock? Clues from H i and Low-ion Absorption, Line Kinematics, and Dust Extinction, The Astrophysical Journal, 865(2), 91, doi:10.3847/1538-4357/aac884. (published) [Ranch, Stampede2, TACC] 71. Stern, J., D. Fielding, C.-A. Faucher-Giguère, and E. Quataert (2019), Cooling flow solutions for the circumgalactic medium, Monthly Notices of the Royal Astronomical Society, 488(2), 2549–2572, doi:10.1093/mnras/stz1859. (published) [Ranch, Stampede2, TACC] 72. Stern, J., D. Fielding, C.-A. Faucher-Giguère, and E. Quataert (2020), The maximum accretion rate of hot gas in dark matter haloes, Monthly Notices of the Royal Astronomical Society, 492(4), 6042–6058, doi:10.1093/mnras/staa198. (published) [Ranch, Stampede2, TACC] 73. Su, K.-Y., P. F. Hopkins, C. C. Hayward, C.-A. Faucher- and V. H. Robles (2019), Cosmic Rays or Turbulence can Suppress Cooling Flows (Where Thermal Heating or Momentum Injection Fail), Monthly Notices of the Royal Astronomical Society,Giguère, doi:10.1093/mnras/stz3011. D. Kereš, X. Ma, M. E. Orr, T. ( publishedK. Chan, ) [Ranch, Stampede2, TACC] 74. Su, K.-Y., P. F. Hopkins, C. C. Hayward, X. Ma, M. Boylan- -A. Faucher-Giguère, M. E. Orr, and C. Wheeler (2018), Discrete Effects in Stellar Feedback: Individual Supernovae, Hypernovae, and IMF Sampling in Dwarf Galaxies, Monthly Notices of the Royal AstronomicalKolchin, Society, D. Kasen, doi:10.1093/mnras/sty1928. D. Kereš, C. (published) [Ranch, Stampede2, TACC] 75. Su, K.-Y., P. F. Hopkins, C. C. Hayward, X. Ma, C.-A. Faucher- (2019), The failure of stellar feedback, magnetic fields, conduction, and morphological quenching in maintaining red galaxies, Monthly Notices of the Royal Astronomical Society,Giguère, 487(3), D. 4393 Kereš,–4408, M. E. doi:10.1093/mnras/stz1494.Orr, T. K. Chan, and V. H. Robles (published) [Ranch, Stampede2, TACC] 76. Taylor, S., Kelley, L., Faucher-Giguere, C. 2020. A Monte Carlo Approach to Modeling Dynamical Friction in Realistic Galactic Environments. American Astronomical Society Meeting Abstracts. 208.01. (published) [Ranch, Stampede2, TACC] 77. Tillman, M., Wellons, S., Faucher-Giguère, C., Anglés-Alcázar, D. 2019. Testing Models of Supermassive Black Hole Evolution with the Quasar Luminosity Function. American Astronomical Society Meeting Abstracts #233. 233. 242.25. (published) [Ranch, Stampede2, TACC] 78. Tillman, M., Wellons, S., Faucher-Giguère, C., Kelley, L., Anglés-Alcázar, D. 2020. Running Late: The observable implications of delayed supermassive black hole growth.. American Astronomical Society Meeting Abstracts. 344.06. (published) [Ranch, Stampede2, TACC] 79. Wellons, S., Faucher-Giguère, C., Anglés-Alcázar, D., Hayward, C., Feldmann, R., et al. 2019. Measuring dynamical masses from gas kinematics in simulated high-redshift galaxies. (published) [Ranch, Stampede2, TACC] 80. Wheeler, C., P. F. Hopkins, A. B. Pace, S. Garrison-Kimmel, M. Boylan-Kolc -A. Faucher-Giguère, and E. Quataert (2019), Be it therefore resolved: cosmological simulations of dwarf galaxies with 30 hin, A. Wetzel, J. S. Bullock, D. Kereš, C.

RY5 IPR12 Page 119 solar mass resolution, Monthly Notices of the Royal Astronomical Society, 490(3), 4447–4463, doi:10.1093/mnras/stz2887. (published) [Ranch, Stampede2, TACC] 81. Yu, S. et al. (2020), Stars made in outflows may populate the stellar halo of the Milky Way, Monthly Notices of the Royal Astronomical Society, 494(2), 1539–1559, doi:10.1093/mnras/staa522. (published) [Ranch, Stampede2, TACC]

15. TG-AST140041 82. Abruzzo, M. W., and Z. Haiman (2019), The impact of photometric redshift errors on lensing statistics in ray-tracing simulations, Monthly Notices of the Royal Astronomical Society, 486(2), 2730–2753, doi:10.1093/mnras/stz1016. (published) [Ranch, Stampede2, TACC] 83. Coulton, W. R., J. Liu, M. S. Madhavacheril, V. Böhm, and D. N. Spergel (2019), Constraining neutrino mass with the tomographic weak lensing bispectrum, Journal of Cosmology and Astroparticle Physics, 2019(05), 043–043, doi:10.1088/1475-7516/2019/05/043. (published) [Ranch, Stampede2, TACC] 84. Coulton, W., Liu, J., McCarthy, I., Osato, K. 2019. Weak Lensing Minima and Peaks: Cosmological Constraints and the Impact of Baryons. (submitted) [Ranch, Stampede2, TACC] 85. Gupta, A., J. M. Z. Matilla, D. Hsu, and Z. Haiman (2018), Non-Gaussian information from weak lensing data via deep learning, Physical Review D, 97(10), doi:10.1103/physrevd.97.103515. (published) [Ranch, Stampede2, TACC] 86. Kreisch, C. D., A. Pisani, C. Carbone, J. Liu, A. J. Hawken, E. Massara, D. N. Spergel, and B. D. Wandelt (2019), Massive neutrinos leave fingerprints on cosmic voids, Monthly Notices of the Royal Astronomical Society, 488(3), 4413–4426, doi:10.1093/mnras/stz1944. (published) [Ranch, Stampede2, TACC] 87. Liu, J., and M. S. Madhavacheril (2019), Constraining neutrino mass with the tomographic weak lensing one-point probability distribution function and power spectrum, Physical Review D, 99(8), doi:10.1103/physrevd.99.083508. (published) [Ranch, Stampede2, TACC] 88. Lu, T., and Z. Haiman (2019), The matter fluctuation amplitude inferred from the weak lensing power spectrum and correlation function in CFHTLenS data, Monthly Notices of the Royal Astronomical Society, 490(4), 5033–5042, doi:10.1093/mnras/stz2931. (published) [Ranch, Stampede2, TACC] 89. Marques, G. A., J. Liu, J. M. Z. Matilla, Z. Haiman, A. Bernui, and C. P. Novaes (2019), Constraining neutrino mass with weak lensing Minkowski Functionals, Journal of Cosmology and Astroparticle Physics, 2019(06), 019–019, doi:10.1088/1475-7516/2019/06/019. (published) [Ranch, Stampede2, TACC] 90. Ribli, D., B. Á. Pataki, J. M. Zorrilla Matilla, D. Hsu, Z. Haiman, and I. Csabai (2019), Weak lensing cosmology with convolutional neural networks on noisy data, Monthly Notices of the Royal Astronomical Society, 490(2), 1843–1860, doi:10.1093/mnras/stz2610. (published) [Ranch, Stampede2, TACC] 91. Waterval, S., Zorrilla, J., Haiman, Z. 2019. Optimizing simulation parameters for weak lensing analyses involving non- Gaussian observables. (submitted) [Ranch, Stampede2, TACC] 92. Zack, G., Li, Z., Liu, J., Spergel, D., Kreisch, C., et al. 2019. The void halo mass function: a promising probe of neutrino mass. (submitted) [Ranch, Stampede2] 93. Zorrilla, J., Haiman, Z. 2019. Probing gaseous galactic halos through the rotational kSZ effect. (accepted) [Ranch, Stampede2, TACC]

16. TG-AST150068 94. Bruch, K. 2020. Undergraduate physics students N-body simulations of galactic dynamics showcased. This piece is also about our AAS 235 iPoster + publications. XSEDE Science Success Story. https://www.xsede.org/-/xsede- allocated-comet-enables-findings-presented-at-aas-235. (published) [Comet] 95. Cummings, B. 2020. N-Body Simulation of Cosmological Structure. Honor's thesis, Physics, Reed College. Reed College (?). (in preparation) [Comet, SDSC] 96. Lydgate, C. 2020. Life of a Galaxy, In 10 Seconds. This piece for the Reed Magazine highlight our AAS 235 paper with the movie of the barred spiral. Reed Magazine. https://docs.google.com/document/d/1MSLqID7YvJ1JlDMkxKdp2izDdSr5xlmKTn1BMkqYxrI/edit?ts=5e682504. (accepted) [Comet, SDSC]

17. TG-AST160043 97. Snyder, G., Yung, L., Mantha, K., Somerville, R., Ferguson, H. 2020. Predicting Galaxy Merger Rates to be Measured by JWST. American Astronomical Society Meeting Abstracts. 283.01. (published) [Comet, Data Oasis, SDSC]

RY5 IPR12 Page 120 18. TG-AST170012 98. Stone, J., Tomida, K., White, C., Felker, K. 2020. The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers. (submitted) [Ranch, Stampede, TACC] 99. White, C., Dexter, J., Blaes, O., Quataert, E. 2020. The Effects of Tilt on the Images of Black Hole Accretion Flows. (accepted) [Ranch, Stampede, TACC] 100. White, C. J., E. Quataert, and C. F. Gammie (2020), The Structure of Radiatively Inefficient Black Hole Accretion Flows, The Astrophysical Journal, 891(1), 63, doi:10.3847/1538-4357/ab718e. (published) [Ranch, Stampede, TACC]

19. TG-AST190009 101. Mangena, T., S. Hassan, and M. G. Santos (2020), Constraining the reionization history using deep learning from 21- cm tomography with the Square Kilometre Array, Monthly Notices of the Royal Astronomical Society, 494(1), 600– 606, doi:10.1093/mnras/staa750. (published) [Bridges GPU, Bridges Regular, PSC, Pylon]

20. TG-AST190028 102. Binggeli, C., E. Zackrisson, X. Ma, A. K. Inoue, A. Vikaeus, T. Hashimoto, K. Mawatari, I. Shimizu, and D. Ceverino (2019), Balmer breaks in simulated galaxies at z>6, Monthly Notices of the Royal Astronomical Society, doi:10.1093/mnras/stz2387. (published) [Stampede2, TACC] 103. Ma, X., P. F. Hopkins, S. Garrison-Kimmel, C.-A. Faucher-Giguère, E. Quataert, M. Boylan-Kolchin, C. C. Hayward, R. -2: galaxy scaling relations, stellar mass functions, and luminosity functions, Monthly Notices of the Royal Astronomical Society, 478(2), 1694– 1715,Feldmann, doi:10.1093/mnras/sty1024. and D. Kereš (2018), Simulating (published galaxies) [Stampede, in the reionization Stampede2, era TACC with ]FIRE

21. TG-AST190038 104. Shajib, A. J. et al. (2020), STRIDES: a 3.9 per cent measurement of the Hubble constant from the strong lens system –6102, doi:10.1093/mnras/staa828. (published) [Bridges Regular, Comet, Data Oasis, PSC, Pylon, SDSC] DES J0408−5354, Monthly Notices of the Royal Astronomical Society, 494(4), 6072 22. TG-ATM090042 105. Chan, M., Anderson, J., Chen, X. 2020. An efficient bi-Gaussian ensemble Kalman filter for satellite infrared radiance data assimilation. (in preparation) 106. Chan, M., Zhang, F., Chen, X., Leung, L. 2020. Potential Impacts of Assimilating All-sky Satellite Infrared Radiances on Convection-Permitting Analysis and Prediction of Tropical Convection. (submitted) 107. Chen, X., Nystrom, R., Davis, C. 2020. Dynamical Structures of Cross-Domain Forecast Error Covariance of a Simulated Tropical Cyclone Using a Convection-Permitting Coupled Atmosphere-Ocean Ensemble. (in preparation) 108. Chen, X., O. M. Pauluis, L. R. Leung, and F. Zhang (2020), Significant Contribution of Mesoscale Overturning to Tropical Mass and Energy Transport Revealed by the ERA5 Reanalysis, Geophysical Research Letters, 47(1), doi:10.1029/2019gl085333. (published) 109. Hayatbini, N., K. Hsu, S. Sorooshian, Y. Zhang, and F. Zhang (2019), Effective Cloud Detection and Segmentation Using a Gradient-Based Algorithm for Satellite Imagery: Application to Improve PERSIANN-CCS, Journal of Hydrometeorology, 20(5), 901–913, doi:10.1175/jhm-d-18-0197.1. (published) [Stampede2, TACC] 110. Minamide, M., Zhang, F., Clothiaux, E. 2020. Nonlinear forecast error growth of rapidly intensifying hurricane Harvey (2017) examined through convection-permitting ensemble assimilation of GOES-16 all-sky radiances. (submitted) 111. Nystrom, R., Greybush, S., Chen, X., Zhang, F. 2020. Potential for new constraints on tropical cyclone surface exchange coefficients through simultaneous state and parameter estimation. (in preparation) 112. Nystrom, R., Rotunno, R., Davis, C., Zhang, F. 2020. Consistent impacts of surface enthalpy and drag coefficient uncertainty between an analytical model and simulated tropical cyclone maximum intensity and storm structure. (submitted) 113. Seibert, J., Greybush, S., Li, J., Zhang, Z., Zhang, F. 2020. Applications of the Geometry-Sensitive Ensemble Mean for Lake-Effect Snowbands and Other Weather Phenomena. (submitted) 114. Sippel, J., Zhang, F., Weng, Y., Braun, S., Cecil, D. 2015. Further Exploring the Potential for Assimilation of Unmanned Aircraft Observations to Benefit Hurricane Analyses and Forecasts. (published) 115. Yu, C., Didlake, A., Zhang, F. 2019. Secondary eyewall formation in hurricane Matthew (2016): The impact of stratiform cooling on boundary layer thermodynamics. (in preparation) RY5 IPR12 Page 121 116. Yu, C., Didlake, A., Zhang, F., Kepert, J. 2020. Investigating axisymmetric and asymmetric signals of secondary eyewall formation using observations-based modeling of the tropical cyclone boundary layer. (in preparation) 117. Yu, C., Didlake, A., Zhang, F., Nystrom, R. 2019. Asymmetric rainband processes leading to secondary eyewall formation in a model simulation of Hurricane Matthew (2016). (submitted) 118. Zhang, Y., Chen, X., Nystrom, R. 2020. Ensemble-based forecast uncertainty and error covariance of Hurricane Harvey with the dual-resolution convection-permitting fvGFS model. (in preparation) 119. Zhang, Y., Stensrud, D. 2020. Benefits of the new-generation Advanced Baseline Imager onboard GOES-16 for the ensemble-based severe thunderstorm analyses and predictions. (in preparation) 120. Zhu, L., X. Chen, and L. Bai (2020), Relative Roles of Low‐Level Wind Speed and Moisture in the Diurnal Cycle of Rainfall Over a Tropical Island Under Monsoonal Flows, Geophysical Research Letters, 47(8), doi:10.1029/2020gl087467. (published) 121. Zhu, L., Q. Wan, X. Shen, Z. Meng, F. Zhang, Y. Weng, J. Sippel, Y. Gao, Y. Zhang, and J. Yue (2015), Prediction and Predictability of High-Impact Western Pacific Landfalling Tropical Cyclone Vicente (2012) through Convection- Permitting Ensemble Assimilation of Doppler Radar Velocity, Monthly Weather Review, 144(1), 21–43, doi:10.1175/mwr-d-14-00403.1. (published) [Stampede2, TACC]

23. TG-ATM160027 122. Arms, S., Chastang, J., Grover, M., Thielen, J., Wilson, M. 2020. Python Workshop. 100th AMS Annual Meeting. https://ams.confex.com/ams/2020Annual/webprogram/meeting.html. (published) [ECSS, IU, Jetstream, Science Gateways, TACC] 123. Arms, S., Chastang, J., Grover, M., Thielen, J., Wilson, M., et al. 2020. Introducing Students to Scientific Python for Atmospheric Science. (submitted) [ECSS, IU, Jetstream, NICS, Science Gateways, TACC] 124. Chastang, J. 2019. Deploying a Unidata JupyterHub on the NSF Jetstream Cloud, Lessons Learned and Challenges Going Forward. ESIP Summer Meeting 2019. DOI:10.6084/m9.figshare.8945078.v1. (published) [ECSS, IU, Jetstream, Science Gateways, TACC] 125. Chastang, J. 2019. A JupyterHub for Atmospheric Science Research and Education on the Unidata Science Gateway. Gateways 2019. DOI:10.17605/OSF.IO/W7SV8. (published) [ECSS, IU, Jetstream, Science Gateways, TACC] 126. Chastang, J. 2020. Demonstration of Unidata Science Gateway. Demonstration of Unidata Science Gateway. Earth Science Information Partners. https://doi.org/10.6084/m9.figshare.12124065.v1 (published) [ECSS, IU, Jetstream, Science Gateways, TACC] 127. Zonca, A., Signell, R., Chastang, J., Fischer, J., Lowe, J., et al. 2020. Deploy Kubernetes and JupyterHub on XSEDE Jetstream. Proceedings of the Practice and Experience on Advanced Research Computing (Portland, Oregon, USA). (submitted) [ECSS, IU, Jetstream, Science Gateways, TACC]

24. TG-ATM170010, TG-ATM170028 128. Howland, M. F., A. S. Ghate, and S. K. Lele (2019), Influence of the geostrophic wind direction on the atmospheric boundary layer flow, Journal of Fluid Mechanics, 883, doi:10.1017/jfm.2019.889. (published) [Stampede2, TACC]

25. TG-ATM170028 129. Ghaisas, N., Ghate, A., Lele, S. 2020. Entrainment characteristics and the inner flux in multi-rotor wind turbine wakes. (in preparation) [Ranch, Stampede2, TACC] 130. Howland, M. F., A. S. Ghate, and S. K. Lele (2020), Coriolis effects within and trailing a large finite wind farm, AIAA Scitech 2020 Forum, doi:10.2514/6.2020-0994. (published) [Stampede2, TACC] 131. Howland, M. F., A. S. Ghate, S. K. Lele, and J. O. Dabiri (2020), Optimal closed-loop wake steering, Part 1: Conventionally neutral atmospheric boundary layer conditions, , doi:10.5194/wes-2020-52. (published) [Stampede2, TACC]

26. TG-ATM190011 132. Alexis, H., Martin, A., Laurel, D. 2019. Evaluating the Seasonal Cycle of Low Clouds in a New Hybrid Resolution Downscaling for Current and Future Climates. 6th Annual Climate Change and Aerosol Research REU Symposium (Portland, OR). (published) [Comet, SDSC]

RY5 IPR12 Page 122 27. TG-BCS180003 133. Ghotbi, A. 2018. Resilience-Based Seismic Evaluation and Design of Reinforced Concrete Structures. Ph.D. dissertation . University of California, Los Angeles. https://escholarship.org/uc/item/0vb568kd#article_main. (published)

28. TG-BCS180005 134. Shi, Q., S. Zhang, G. P. Korfiatis, C. Christodoulatos, and X. Meng (2020), Identifying the existence and molecular structure of the dissolved HCO3-Ca-As(V) complex in water, Science of The Total Environment, 724, 138216, doi:10.1016/j.scitotenv.2020.138216. (published) [Stampede2, TACC]

29. TG-BCS190007 135. Gong, X., Zhang, Z., Kevin, M., Pan, Y. 2020. full resolution of extreme ship response statistics.. 33rd Symposium on Naval Hydrodynamics (Osaka, Japan). (accepted)

30. TG-BCS190015 136. Feng, R., Zhou, Z., Gotway, M., Liang, J. 2020. Self-supervised Learning: From Parts to Whole. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI’20), 2020 (Lima, Peru). (submitted) [Bridges GPU, PSC, Pylon] 137. Haghighi, F., Hosseinzadeh Taher, M., Zhou, Z., Gotway, M., Liang, J. 2020. Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI’20), 2020 (Lima, Peru). (submitted) [Bridges GPU, PSC, Pylon] 138. Hosseinzadeh Taher, M., Haghighi, F., Gotway, M., Liang, J. 2020. Transferable Visual Words: An Annotation-efficient Solution to Chest X-ray Image Analysis. Abstract submitted to Research Symposium. Biomedical Informatics, Biomedical Diagnostics, and Data Science Affinity Network Research Symposium. (submitted) [Bridges GPU, PSC, Pylon] 139. Rahman Siddiquee, M., Zhou, Z., Feng, R., Tajbakhsh, N., Gotway, M., et al. 2020. Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization. (in preparation) [Bridges GPU, PSC, Pylon] 140. Zhou, Z., Liang, J. 2020. Cost-Effective Deep Learning in Medical Image Analysis. Doctoral thesis of Zongwei Zhou. Biomedical Informatics, Biomedical Diagnostics, and Data Science Affinity Network Research Symposium, 2020. (submitted) [Bridges GPU, PSC, Pylon] 141. Zhou, Z., Shin, J., Gurudu, S., Gotway, M., Liang, J. 2020. AFT*: Active Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts.. (submitted) [Bridges GPU, PSC, Pylon] 142. Zhou, Z., Sodha, V., Pang, J., Liang, J. 2020. Models Genesis. (submitted) [Bridges GPU, PSC, Pylon] 143. Zhou, Z., Sodha, V., Siddiquee, M., Feng, R., Tajbakhsh, N., et al. 2019. Models genesis: Generic autodidactic models for 3d medical image analysis. International Conference on Medical Image Computing and Computer-Assisted Intervention. 384--393. (published) [Bridges GPU, PSC, Pylon]

31. TG-BIO150042 144. Caplins, S. A. (2020), Plasticity and Artificial Selection for Developmental Mode in a Poecilogonous Sea Slug, , doi:10.1101/2020.03.06.981324. (published)

32. TG-BIO150043 145. Asnicar, F. et al. (2020), Precise phylogenetic analysis of microbial isolates and genomes from metagenomes using PhyloPhlAn 3.0, Nature Communications, 11(1), doi:10.1038/s41467-020-16366-7. (published) 146. Balaban, M., S. Sarmashghi, and S. Mirarab (2018), APPLES: Scalable Distance-based Phylogenetic Placement with or without Alignments, , doi:10.1101/475566. (published) 147. Bolyen, E. et al. (2019), Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2, Nature Biotechnology, 37(8), 852–857, doi:10.1038/s41587-019-0209-9. (published) 148. Estaki, M. et al. (2020), QIIME 2 Enables Comprehensive End‐to‐End Analysis of Diverse Microbiome Data and Comparative Studies with Publicly Available Data, Current Protocols in Bioinformatics, 70(1), doi:10.1002/cpbi.100. (published)

RY5 IPR12 Page 123 149. Fang, X. et al. (2018), Metagenomics-Based, Strain-Level Analysis of Escherichia coli From a Time-Series of Microbiome Samples From a Crohn’s Disease Patient, Frontiers in Microbiology, 9, doi:10.3389/fmicb.2018.02559. (published) 150. Hillmann, B., G. A. Al-Ghalith, R. R. Shields-Cutler, Q. Zhu, R. Knight, and D. Knights (2020), SHOGUN: a modular, accurate and scalable framework for microbiome quantification, edited by A. Valencia, Bioinformatics, 36(13), 4088– 4090, doi:10.1093/bioinformatics/btaa277. (published) 151. Karst, S. M., R. M. Ziels, R. H. Kirkegaard, E. A. Sørensen, D. McDonald, Q. Zhu, R. Knight, and M. Albertsen (2019), Enabling high-accuracy long-read amplicon sequences using unique molecular identifiers with Nanopore or PacBio sequencing, , doi:10.1101/645903. (published) 152. Marotz, C. A., J. G. Sanders, C. Zuniga, L. S. Zaramela, R. Knight, and K. Zengler (2018), Improving saliva shotgun metagenomics by chemical host DNA depletion, Microbiome, 6(1), doi:10.1186/s40168-018-0426-3. (published) 153. Mills, R. H., Y. Vázquez-Baeza, Q. Zhu, L. Jiang, J. Gaffney, G. Humphrey, L. Smarr, R. Knight, and D. J. Gonzalez (2019), Evaluating Metagenomic Prediction of the Metaproteome in a 4.5-Year Study of a Patient with Crohn’s Disease, edited by M. J. Claesson, mSystems, 4(1), doi:10.1128/msystems.00337-18. (published) 154. Morton, J. T., C. Marotz, A. Washburne, J. Silverman, L. S. Zaramela, A. Edlund, K. Zengler, and R. Knight (2019), Establishing microbial composition measurement standards with reference frames, Nature Communications, 10(1), doi:10.1038/s41467-019-10656-5. (published) 155. Poore, G. D. et al. (2020), Microbiome analyses of blood and tissues suggest cancer diagnostic approach, Nature, 579(7800), 567–574, doi:10.1038/s41586-020-2095-1. (published) 156. Salosensaari, A. et al. (2020), Taxonomic Signatures of Long-Term Mortality Risk in Human Gut Microbiota, , doi:10.1101/2019.12.30.19015842. (published) 157. Sanders, J. G. et al. (2019), Optimizing sequencing protocols for leaderboard metagenomics by combining long and short reads, Genome Biology, 20(1), doi:10.1186/s13059-019-1834-9. (published) 158. Smarr, L., E. Hyde, D. McDonald, W. Sandborn, and R. Knight (2017), Tracking Human Gut Microbiome Changes Resulting from a Colonoscopy, Methods of Information in Medicine, 56(06), 442–447, doi:10.3414/me17-01-0036. (published) 159. Zhu, Q. et al. (2019), Phylogenomics of 10,575 genomes reveals evolutionary proximity between domains Bacteria and Archaea, Nature Communications, 10(1), doi:10.1038/s41467-019-13443-4. (published)

33. TG-BIO170096 160. Easterly, C. W., R. Sajulga, S. Mehta, J. Johnson, P. Kumar, S. Hubler, B. Mesuere, J. Rudney, T. J. Griffin, and P. D. Jagtap (2019), metaQuantome: An Integrated, Quantitative Metaproteomics Approach Reveals Connections Between Taxonomy and Protein Function in Complex Microbiomes, Molecular & Cellular Proteomics, 18(8 suppl 1), S82–S91, doi:10.1074/mcp.ra118.001240. (published) [IU, Jetstream, Science Gateways] 161. Hubler, S. L., P. Kumar, S. Mehta, C. Easterly, J. E. Johnson, P. D. Jagtap, and T. J. Griffin (2019), Challenges in Peptide- Spectrum Matching: A Robust and Reproducible Statistical Framework for Removing Low-Accuracy, High-Scoring Hits, Journal of Proteome Research, 19(1), 161–173, doi:10.1021/acs.jproteome.9b00478. (published) [IU, Jetstream, Science Gateways] 162. Kumar, P., J. E. Johnson, C. Easterly, S. Mehta, R. Sajulga, B. Nunn, P. D. Jagtap, and T. J. Griffin (2019), A sectioning and database enrichment approach for improved peptide spectrum matching in large, genome-guided protein sequence databases, , doi:10.1101/843078. (published) [IU, Jetstream, Science Gateways] 163. McGowan, T., J. E. Johnson, P. Kumar, R. Sajulga, S. Mehta, P. D. Jagtap, and T. J. Griffin (2020), Multi-omics Visualization Platform: An extensible Galaxy plug-in for multi-omics data visualization and exploration, GigaScience, 9(4), doi:10.1093/gigascience/giaa025. (published) [IU, Jetstream, Science Gateways] 164. Mehta, S. et al. (2020), Precursor intensity-based label-free quantification software tools for proteomic and multiomic analysis within the Galaxy Platform, , doi:10.1101/2020.04.01.003988. (published) [IU, Jetstream, Science Gateways] 165. Sajulga, R. et al. (2020), Survey of metaproteomics software tools for functional microbiome analysis, , doi:10.1101/2020.01.07.897561. (published) [IU, Jetstream, Science Gateways] 166. Stewart, P. A., B. M. Kuenzi, S. Mehta, P. Kumar, J. E. Johnson, P. Jagtap, T. J. Griffin, and E. B. Haura (2019), The Galaxy Platform for Reproducible Affinity Proteomic Mass Spectrometry Data Analysis, Mass Spectrometry of Proteins, 249– 261, doi:10.1007/978-1-4939-9232-4_16. (published) [IU, Jetstream, Science Gateways]

RY5 IPR12 Page 124 34. TG-CCR180016, TG-CIE160022 167. Ma, A., A. McDermaid, J. Xu, Y. Chang, and Q. Ma (2020), Integrative Methods and Practical Challenges for Single-Cell Multi-omics, Trends in Biotechnology, doi:10.1016/j.tibtech.2020.02.013. (published) 168. Ma, A., C. Wang, Y. Chang, F. H. Brennan, A. McDermaid, B. Liu, C. Zhang, P. G. Popovich, and Q. Ma (2020), IRIS3: integrated cell-type-specific regulon inference server from single-cell RNA-Seq, Nucleic Acids Research, 48(W1), W275–W286, doi:10.1093/nar/gkaa394. (published) 169. Xie, J., A. Ma, Y. Zhang, B. Liu, S. Cao, C. Wang, J. Xu, C. Zhang, and Q. Ma (2019), QUBIC2: a novel and robust biclustering algorithm for analyses and interpretation of large-scale RNA-Seq data, edited by I. Birol, Bioinformatics, doi:10.1093/bioinformatics/btz692. (published) 170. Yang, J., A. Ma, A. D. Hoppe, C. Wang, Y. Li, C. Zhang, Y. Wang, B. Liu, and Q. Ma (2019), Prediction of regulatory motifs from human Chip-sequencing data using a deep learning framework, Nucleic Acids Research, 47(15), 7809–7824, doi:10.1093/nar/gkz672. (published)

35. TG-CCR180053 171. Mathias, H., Foley, S. 2020. A Parallel Two-stage Genetic Algorithm for Route Planning. Genetic and Evolutionary Computation Conference (GECCO) (Cancun, Mexico). (published) [Comet, SDSC]

36. TG-CCR180056, TG-CCR190031 172. Phillips, T., Byrd, K., Zou, X. 2019. Reexamine the identities of soldiers in an iconic image using deep-learning-based facial recognition techniques. (published) [IU, Jetstream] 173. Phillips, T., X. Yu, B. Haakenson, and X. Zou (2019), Design and Implementation of Privacy-Preserving, Flexible and Scalable Role-Based Hierarchical Access Control, 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), doi:10.1109/tps-isa48467.2019.00015. (published) 174. Phillips, T., X. Zou, F. Li, and N. Li (2019), Enhancing Biometric-Capsule-based Authentication and Facial Recognition via Deep Learning, Proceedings of the 24th ACM Symposium on Access Control Models and Technologies, doi:10.1145/3322431.3325417. (published) [Jetstream]

37. TG-CCR190027 175. Larkin, J., Jonsson, M., Justice, D., Guerreschi, G. 2020. Evaluation of Quantum Approximate Optimization Algorithm based on the approximation ratio of single samples. (published) [Bridges GPU, Bridges Large, Bridges Regular, PSC, Pylon]

38. TG-CCR190031 176. Yu, X., Haakenson, B., Phillips, T., Zou, X. 2020. User-Friendly Design of Cryptographically-Enforced Hierarchical Role- based Access Control Model. IEEE ICCCN 2020 (Honolulu, Hawaii, USA). (accepted) [IU, Jetstream]

39. TG-CCR190051 177. Bhattacharya, S., Yu, W., Fahim Tahmid Chowdhury, . 2020. O(1) Communication for Distributed SGD through Two- Level Gradient Averaging. (published) [Bridges GPU, PSC]

40. TG-CCR200022 178. Alamgir, J. 2020. Results from simulating 1115 drug compounds against SARSCOV2 protein 6w4b – a viral replicase. SARS-COV2 replicase interruption simulation using structure 6w4b and 1115 drug compounds. Submitted for publication. (submitted) [Bridges Regular, PSC]

41. TG-CDA170003 179. Mondejar, M. E., and F. Haglind (2020), The potential of halogenated olefins as working fluids for organic Rankine cycle technology, Journal of Molecular Liquids, 310, 112971, doi:10.1016/j.molliq.2020.112971. (published) [Bridges Regular, Comet, IU, Jetstream, PSC, Science Gateways, SDSC, TACC]

42. TG-CDA200003 180. Wang, R., Hozumi, Y., Yin, C., Wei, G. 2020. Mutations on COVID-19 diagnostic targets. (published)

RY5 IPR12 Page 125 43. TG-CHE060002P, TG-DMR100047, TG-DMR110037, TG-PHY140011 181. Lingerfelt, D. B., P. Ganesh, J. Jakowski, and B. G. Sumpter (2019), Electronically Nonadiabatic Structural Transformations Promoted by Electron Beams, Advanced Functional Materials, 29(52), 1901901, doi:10.1002/adfm.201901901. (published) 182. Lingerfelt, D. B., P. Ganesh, J. Jakowski, and B. G. Sumpter (2020), Understanding Beam-Induced Electronic Excitations in Materials, Journal of Chemical Theory and Computation, 16(2), 1200–1214, doi:10.1021/acs.jctc.9b00792. (published) 183. McCaskey, A. J., Z. P. Parks, J. Jakowski, S. V. Moore, T. D. Morris, T. S. Humble, and R. C. Pooser (2019), Quantum chemistry as a benchmark for near-term quantum computers, npj Quantum Information, 5(1), doi:10.1038/s41534- 019-0209-0. (published)

44. TG-CHE090043 184. Birke, R. L., and J. R. Lombardi (2020), Relative contributions of Franck–Condon to Herzberg–Teller terms in charge transfer surface-enhanced Raman scattering spectroscopy, The Journal of Chemical Physics, 152(22), 224107, doi:10.1063/5.0005012. (published) [Comet, SDSC] 185. Islam, S. K., Y. P. Cheng, R. L. Birke, M. V. Cañamares, C. Muehlethaler, and J. R. Lombardi (2020), An analysis of tetrahydrocannabinol (THC) and its analogs using surface enhanced Raman Scattering (SERS), Chemical Physics, 536, 110812, doi:10.1016/j.chemphys.2020.110812. (published) [Comet, SDSC]

45. TG-CHE110009 186. Bajaj, P., D. Zhuang, and F. Paesani (2019), Specific Ion Effects on Hydrogen-Bond Rearrangements in the Halide– Dihydrate Complexes, The Journal of Physical Chemistry Letters, 10(11), 2823–2828, doi:10.1021/acs.jpclett.9b00899. (published) 187. Egan, C. K., B. B. Bizzarro, M. Riera, and F. Paesani (2020), Nature of Alkali Ion–Water Interactions: Insights from Many-Body Representations and Density Functional Theory. II, Journal of Chemical Theory and Computation, 16(5), 3055–3072, doi:10.1021/acs.jctc.0c00082. (published) 188. Egan, C. K., and F. Paesani (2019), Assessing Many-Body Effects of Water Self-Ions. II: H3O+(H2O)n Clusters, Journal of Chemical Theory and Computation, 15(9), 4816–4833, doi:10.1021/acs.jctc.9b00418. (published) 189. Moberg, D. R., D. Becker, C. W. Dierking, F. Zurheide, B. Bandow, U. Buck, A. Hudait, V. Molinero, F. Paesani, and T. Zeuch (2019), The end of ice I, Proceedings of the National Academy of Sciences, 116(49), 24413–24419, doi:10.1073/pnas.1914254116. (published) 190. Özçelik, V. O., Y. Li, W. Xiong, and F. Paesani (2020), Modeling Spontaneous Charge Transfer at Metal/Organic Hybrid Heterostructures, The Journal of Physical Chemistry C, 124(8), 4802–4809, doi:10.1021/acs.jpcc.9b10055. (published) 191. Paesani, F., P. Bajaj, and M. Riera (2019), Chemical accuracy in modeling halide ion hydration from many-body representations, Advances in Physics: X, 4(1), 1631212, doi:10.1080/23746149.2019.1631212. (published) 192. Rallapalli, K. L., A. C. Komor, and F. Paesani (2020), Computer simulations explain mutation-induced effects on the DNA editing by adenine base editors, Science Advances, 6(10), eaaz2309, doi:10.1126/sciadv.aaz2309. (published) 193. Riera, M., E. Lambros, T. T. Nguyen, A. W. Götz, and F. Paesani (2019), Low-order many-body interactions determine the local structure of liquid water, Chemical Science, 10(35), 8211–8218, doi:10.1039/c9sc03291f. (published) 194. Riera, M., E. P. Yeh, and F. Paesani (2020), Data-Driven Many-Body Models for Molecular Fluids: CO2/H2O Mixtures as a Case Study, Journal of Chemical Theory and Computation, 16(4), 2246–2257, doi:10.1021/acs.jctc.9b01175. (published) 195. a metal-organic framework with open metal sites, Nature Communications, 10(1), doi:10.1038/s41467-019-12751-z. (Rieth,published A. J.,) K. M. Hunter, M. Dincă, and F. Paesani (2019), Hydrogen bonding structure of confined water templated by 196. Zhai, Y., A. Caruso, S. Gao, and F. Paesani (2020), Active learning of many-body configuration space: Application to the Cs+–water MB-nrg potential energy function as a case study, The Journal of Chemical Physics, 152(14), 144103, doi:10.1063/5.0002162. (published)

RY5 IPR12 Page 126 46. TG-CHE130010 197. Du, W.-G. H., A. W. Götz, and L. Noodleman (2019), DFT Fea3–O/O–O Vibrational Frequency Calculations over Catalytic Reaction Cycle States in the Dinuclear Center of Cytochrome c Oxidase, Inorganic Chemistry, 58(20), 13933– 13944, doi:10.1021/acs.inorgchem.9b01840. (published) [Bridges Regular, Comet, PSC, SDSC]

47. TG-CHE130089 198. Greer, A. 2001. On the Origin of Cytotoxicity of the Natural Product Varacin. A Novel Example of a Pentathiepin Reaction That Provides Evidence for a Triatomic Sulfur Intermediate. (published)

48. TG-CHE140070 199. Alnaas, A., Oviedo, J., Watson-Siriboe, A., Tran, S., Negussie, M., et al. 2019. Membrane binding by synaptotagmin-like protein 4: site-directed mutagenesis of the lipid interaction surface. 2019 Biophysical Society Annual Meeting (Baltimore, MD). (published) [SDSC] 200. Aydintug, B., Duster, A., Garza, C., Negussie, M., Lin, H. 2019. Proton transport in E. coli CLC transport protein by adaptive QM/MM calculations. 2019 Biophysical Society Annual Meeting (Baltimore, MD). (published) [SDSC] 201. Aydintug, B., Duster, A., Lin, H. 2020. Proton transport through E. coli CLC chloride/proton antiporter in the presence of bound fluoride. 2020 Biophysical Society Annual Meeting (San Diego, CA). (published) [SDSC] 202. Bhat, S., Talachutla, S., Lin, H. 2019. Quantitative assessment of equilibration in molecular dynamics simulations. 2019 Rocky Mountain Advanced Computing Consortium Symposium (Boulder, CO). (published) [SDSC] 203. Chon, N., Tran, S., Miller, C., Lin, H., Knight, J. 2020. Using high-throughput structure prediction and evolutionary alignment to map electrostatic protein-membrane interactions. 2020 Biophysical Society Annual Meeting (San Diego, CA). (published) [SDSC] 204. Chon, N., Zheng, H., Lin, H. 2019. A computational study of the essential transmembrane protein Nark as nitrate/nitrite exchanger. 2019 Biophysical Society Annual Meeting (Baltimore, MD). (published) [SDSC] 205. Duster, A. W., C. M. Garza, B. O. Aydintug, M. B. Negussie, and H. Lin (2019), Adaptive Partitioning QM/MM for Molecular Dynamics Simulations: 6. Proton Transport through a Biological Channel, Journal of Chemical Theory and Computation, 15(2), 892–905, doi:10.1021/acs.jctc.8b01128. (published) [Comet, SDSC] 206. Duster, A., Lin, H. 2019. Simulations of ion solvation and transfer by adaptive-partitioning QM/MM dynamics. International Society of Theoretical Chemical Physics (ISTCP 2019) (Tromsø, Norway). (published) [SDSC] 207. Duster, A., Lin, H. 2019. Adaptive QM/MM dynamics simulations of proton transfer through biological channels. 2019 Beijing Symposium of Electronic Structure and Dynamics of Complex Systems (Beijing, China). (published) [SDSC] 208. Duster, A., Lin, H. 2019. Neural network corrections to semi-empirical quantum chemistry methods for accurate descriptions of proton transfer reactions. 2019 Rocky Mountain Advanced Computing Consortium Symposium (Boulder, CO). (published) [SDSC] 209. Duster, A. W., and H. Lin (2019), Tracking Proton Transfer through Titratable Amino Acid Side Chains in Adaptive QM/MM Simulations, Journal of Chemical Theory and Computation, 15(11), 5794–5809, doi:10.1021/acs.jctc.9b00649. (published) [Comet, SDSC] 210. Duster, A., Lin, H. 2020. Tracking proton transfer through amino acids in adaptive QM/MM simulations. 2020 Biophysical Society Annual Meeting (San Diego, CA). (published) [SDSC] 211. Duster, A., Lin, H. 2020. Machine-learning-enhanced adaptive-partitioning multiscale simulations. 2020 Rocky Mountain Advanced Computing Consortium Symposium (Boulder, CO). (published) [Comet, SDSC] 212. Negussie, M., Tran, S., Chon, N., Knight, J., Lin, H. 2019. Membrane binding of synaptotagmin-like protein 4: Insight from molecular dynamics simulation of two mutants. 2019 Biophysical Society Annual Meeting (Baltimore, MD). (published) [SDSC] 213. Negussie, M., Tran, S., Chon, N., Oviedo, J., Alnaas, A., et al. 2019. Synaptotagmin-like protein 4: Membrane binding simulations of single and triple mutants. 2019 Rocky Mountain Membrane Trafficking Meeting (Denver, CO). (published) [SDSC] 214. Negussie, M., Tran, S., Chon, N., Oviedo, J., Alnaas, A., et al. 2020. Membrane interaction of synaptotagmin-like protein 4: simulations of mutant C2A domains. 2020 Biophysical Society Annual Meeting (San Diego, CA). (published) [SDSC] 215. Negussie, M., Tran, S., Chon, N., Oviedo, J., Alnaas, A., et al. 2020. Electrostatic membrane interaction of synaptotagmin-like protein 4: Simulations of mutant C2A domains. 2020 Colorado Single Molecules and Membranes Meeting (Denver, CO). (published) [SDSC]

RY5 IPR12 Page 127 49. TG-CHE140072 216. Li, J., Zhang, H., Samarakoon, W., Shan, W., Cullen, D., et al. 2019. Thermally Driven Structure and Performance Evolution of Atomically Dispersed FeN4 Sites for Oxygen Reduction. DOI:10.1002/anie.201909312. (published) 217. Pan, F., B. Li, W. Deng, Z. Du, Y. Gang, G. Wang, and Y. Li (2019), Promoting electrocatalytic CO2 reduction on nitrogen- doped carbon with sulfur addition, Applied Catalysis B: Environmental, 252, 240–249, doi:10.1016/j.apcatb.2019.04.025. (published) 218. Pan, F., H. Zhang, Z. Liu, D. Cullen, K. Liu, K. More, G. Wu, G. Wang, and Y. Li (2019), Atomic-level active sites of efficient imidazolate framework-derived nickel catalysts for CO2 reduction, Journal of Materials Chemistry A, 7(46), 26231–26237, doi:10.1039/c9ta08862h. (published) 219. Qiao, Z. et al. (2019), 3D porous graphitic nanocarbon for enhancing the performance and durability of Pt catalysts: a balance between graphitization and hierarchical porosity, Energy & Environmental Science, 12(9), 2830–2841, doi:10.1039/c9ee01899a. (published) 220. Stecker, C., K. Liu, J. Hieulle, R. Ohmann, Z. Liu, L. K. Ono, G. Wang, and Y. Qi (2019), Surface Defect Dynamics in Organic–Inorganic Hybrid Perovskites: From Mechanism to Interfacial Properties, ACS Nano, 13(10), 12127–12136, doi:10.1021/acsnano.9b06585. (published) 221. Xie, P., Y. Yao, Z. Huang, Z. Liu, J. Zhang, T. Li, G. Wang, R. Shahbazian-Yassar, L. Hu, and C. Wang (2019), Highly efficient decomposition of ammonia using high-entropy alloy catalysts, Nature Communications, 10(1), doi:10.1038/s41467-019-11848-9. (published) 222. Zou, L., J. Li, Z. Liu, G. Wang, A. Manthiram, and C. Wang (2019), Lattice doping regulated interfacial reactions in cathode for enhanced cycling stability, Nature Communications, 10(1), doi:10.1038/s41467-019-11299-2. (published)

50. TG-CHE140073 223. Bajaj, A., F. Liu, and H. J. Kulik (2019), Non-empirical, low-cost recovery of exact conditions with model-Hamiltonian inspired expressions in jmDFT, The Journal of Chemical Physics, 150(15), 154115, doi:10.1063/1.5091563. (published) [Bridges GPU, Comet, SDSC] 224. Duan, C., J. P. Janet, F. Liu, A. Nandy, and H. J. Kulik (2019), Learning from Failure: Predicting Electronic Structure Calculation Outcomes with Machine Learning Models, Journal of Chemical Theory and Computation, 15(4), 2331– 2345, doi:10.1021/acs.jctc.9b00057. (published) [Comet, SDSC] 225. Gugler, S., J. P. Janet, and H. J. Kulik (2020), Enumeration of de novo inorganic complexes for chemical discovery and machine learning, Molecular Systems Design & Engineering, 5(1), 139–152, doi:10.1039/c9me00069k. (published) [Comet, SDSC] 226. Janet, J. P., C. Duan, T. Yang, A. Nandy, and H. J. Kulik (2019), A quantitative uncertainty metric controls error in neural network-driven chemical discovery, Chemical Science, 10(34), 7913–7922, doi:10.1039/c9sc02298h. (published) [Comet, SDSC] 227. Janet, J. P., F. Liu, A. Nandy, C. Duan, T. Yang, S. Lin, and H. J. Kulik (2019), Designing in the Face of Uncertainty: Exploiting Electronic Structure and Machine Learning Models for Discovery in Inorganic Chemistry, Inorganic Chemistry, 58(16), 10592–10606, doi:10.1021/acs.inorgchem.9b00109. (published) [Comet, SDSC] 228. Janet, J. P., S. Ramesh, C. Duan, and H. J. Kulik (2020), Accurate Multiobjective Design in a Space of Millions of Transition Metal Complexes with Neural-Network-Driven Efficient Global Optimization, ACS Central Science, 6(4), 513–524, doi:10.1021/acscentsci.0c00026. (published) [Comet, SDSC] 229. Liu, F., and H. J. Kulik (2019), Impact of Approximate DFT Density Delocalization Error on Potential Energy Surfaces in Transition Metal Chemistry, Journal of Chemical Theory and Computation, 16(1), 264–277, doi:10.1021/acs.jctc.9b00842. (published) [Comet, SDSC] 230. Liu, F., T. Yang, J. Yang, E. Xu, A. Bajaj, and H. J. Kulik (2019), Bridging the Homogeneous-Heterogeneous Divide: Modeling Spin for Reactivity in Single Atom Catalysis, Frontiers in Chemistry, 7, doi:10.3389/fchem.2019.00219. (published) [Bridges GPU, Comet, SDSC] 231. Mehmood, R., and H. J. Kulik (2020), Both Configuration and QM Region Size Matter: Zinc Stability in QM/MM Models of DNA Methyltransferase, Journal of Chemical Theory and Computation, 16(5), 3121–3134, doi:10.1021/acs.jctc.0c00153. (published) [Comet, SDSC] 232. Nandy, A., C. Duan, J. P. Janet, S. Gugler, and H. J. Kulik (2018), Strategies and Software for Machine Learning Accelerated Discovery in Transition Metal Chemistry, Industrial & Engineering Chemistry Research, 57(42), 13973– 13986, doi:10.1021/acs.iecr.8b04015. (published) [Comet, SDSC]

RY5 IPR12 Page 128 233. Nandy, A., J. Zhu, J. P. Janet, C. Duan, R. B. Getman, and H. J. Kulik (2019), Machine Learning Accelerates the Discovery of Design Rules and Exceptions in Stable Metal–Oxo Intermediate Formation, ACS Catalysis, 9(9), 8243–8255, doi:10.1021/acscatal.9b02165. (published) [Comet, SDSC] 234. Park, Y.-G. et al. (2018), Protection of tissue physicochemical properties using polyfunctional crosslinkers, Nature Biotechnology, 37(1), 73–83, doi:10.1038/nbt.4281. (published) [Comet, SDSC] 235. Qi, H. W., and H. J. Kulik (2019), Evaluating Unexpectedly Short Non-covalent Distances in X-ray Crystal Structures of Proteins with Electronic Structure Analysis, Journal of Chemical Information and Modeling, 59(5), 2199–2211, doi:10.1021/acs.jcim.9b00144. (published) [Comet, SDSC] 236. Taylor, M. G., T. Yang, S. Lin, A. Nandy, J. P. Janet, C. Duan, and H. J. Kulik (2020), Seeing Is Believing: Experimental Spin States from Machine Learning Model Structure Predictions, The Journal of Physical Chemistry A, 124(16), 3286– 3299, doi:10.1021/acs.jpca.0c01458. (published) [Comet, SDSC] 237. Transue, W. J. et al. (2018), Anthracene as a Launchpad for a Phosphinidene Sulfide and for Generation of a Phosphorus–Sulfur Material Having the Composition P2S, a Vulcanized Red Phosphorus That Is Yellow, Journal of the American Chemical Society, 141(1), 431–440, doi:10.1021/jacs.8b10775. (published) [Comet, SDSC] 238. Yang, Z., F. Liu, A. H. Steeves, and H. J. Kulik (2019), Quantum Mechanical Description of Electrostatics Provides a Unified Picture of Catalytic Action Across Methyltransferases, The Journal of Physical Chemistry Letters, 10(13), 3779–3787, doi:10.1021/acs.jpclett.9b01555. (published) [Bridges GPU, Comet, SDSC] 239. Zhao, Q., and H. J. Kulik (2019), Stable Surfaces That Bind Too Tightly: Can Range-Separated Hybrids or DFT+U Improve Paradoxical Descriptions of Surface Chemistry?, The Journal of Physical Chemistry Letters, 10(17), 5090– 5098, doi:10.1021/acs.jpclett.9b01650. (published) [Comet, SDSC]

51. TG-CHE140114 240. Chakraborty, P., Y. Liu, T. Weinacht, and S. Matsika (2020), Excited state dynamics of cis,cis-1,3-cyclooctadiene: Non- adiabatic trajectory surface hopping, The Journal of Chemical Physics, 152(17), 174302, doi:10.1063/5.0005558. (published) [Comet, SDSC]

52. TG-CHE150074 241. Wang, Y., Wang, E. 2020. Computational Research in a PUI using XSEDE and ECSS Resources. PEARC20 (Portland, Oregon). (accepted) 242. Zeng, J., Tong, Z., Bao, H., Chen, N., Wang, F., et al. 2020. Controllable depolymerization of lignin using carbocatalyst graphene oxide under mild conditions. (published)

53. TG-CHE160009 243. Izzo, J. A., Y. Myshchuk, J. S. Hirschi, and M. J. Vetticatt (2019), Transition state analysis of an enantioselective Michael addition by a bifunctional thiourea organocatalyst, Organic & Biomolecular Chemistry, 17(16), 3934–3939, doi:10.1039/c9ob00072k. (published) [Comet, SDSC] 244. Lin, Y., W. J. Hirschi, A. Kunadia, A. Paul, I. Ghiviriga, K. A. Abboud, R. W. Karugu, M. J. Vetticatt, J. S. Hirschi, and D. Seidel (2020), A Selenourea-Thiourea Brønsted Acid Catalyst Facilitates Asymmetric Conjugate Additions of Amines -Unsaturated Esters, Journal of the American Chemical Society, 142(12), 5627–5635, doi:10.1021/jacs.9b12457. (published) [Comet, SDSC] to α,β 54. TG-CHE170060 245. Sun, G., and P. Sautet (2019), Toward Fast and Reliable Potential Energy Surfaces for Metallic Pt Clusters by Hierarchical Delta Neural Networks, Journal of Chemical Theory and Computation, 15(10), 5614–5627, doi:10.1021/acs.jctc.9b00465. (published) [Bridges Regular, Comet, PSC, Pylon, SDSC]

55. TG-CHE180027 246. Gozem, S., P. J. M. Johnson, A. Halpin, H. L. Luk, T. Morizumi, V. I. Prokhorenko, O. P. Ernst, M. Olivucci, and R. J. D. Miller (2020), Excited-State Vibronic Dynamics of Bacteriorhodopsin from Two-Dimensional Electronic Photon Echo Spectroscopy and Multiconfigurational Quantum Chemistry, The Journal of Physical Chemistry Letters, 11(10), 3889– 3896, doi:10.1021/acs.jpclett.0c01063. (published) [Comet, SDSC] 247. Gozem, S., R. Seidel, U. Hergenhahn, E. Lugovoy, B. Abel, B. Winter, A. I. Krylov, and S. E. Bradforth (2020), Probing the Electronic Structure of Bulk Water at the Molecular Length Scale with Angle-Resolved Photoelectron Spectroscopy, The Journal of Physical Chemistry Letters, 11(13), 5162–5170, doi:10.1021/acs.jpclett.0c00968. (published) [Comet, SDSC] RY5 IPR12 Page 129 248. Kabir, M. P., Y. Orozco-Gonzalez, and S. Gozem (2019), Electronic spectra of flavin in different redox and protonation states: a computational perspective on the effect of the electrostatic environment, Physical Chemistry Chemical Physics, 21(30), 16526–16537, doi:10.1039/c9cp02230a. (published) [Comet, SDSC]

56. TG-CHE180038 249. Rogers, J., Geissler, P. 2020. Breakage of Hydrophobic Contacts Limits the Rate of Passive Lipid Exchange Between Membranes. (submitted) [Comet, SDSC]

57. TG-CHE180056 250. Hintzen, J. C. J., J. Poater, K. Kumar, A. H. K. Al Temimi, B. J. G. E. Pieters, R. S. Paton, F. M. Bickelhaupt, and J. Mecin (2020), Comparison of Molecular Recognition of Trimethyllysine and Trimethylthialysine by Epigenetic Reader Proteins, Molecules, 25(8), 1918, doi:10.3390/molecules25081918. (published) ović 251. Karabiyikoglu, S., A. V. Brethomé, T. Palacin, R. S. Paton, and S. P. Fletcher (2020), Enantiomerically enriched tetrahydropyridine allyl chlorides, Chemical Science, 11(16), 4125–4130, doi:10.1039/d0sc00377h. (published) 252. Porey, S., X. Zhang, S. Bhowmick, V. Kumar Singh, S. Guin, R. S. Paton, and D. Maiti (2020), Alkyne Linchpin Strategy for Drug:Pharmacophore Conjugation: Experimental and Computational Realization of a Meta-Selective Inverse Sonogashira Coupling, Journal of the American Chemical Society, 142(8), 3762–3774, doi:10.1021/jacs.9b10646. (published)

58. TG-CHE180057 253. Alamudun, S. F., K. Tanovitz, A. Fajardo, K. Johnson, A. Pham, T. Jamshidi Araghi, and A. S. Petit (2020), Structure– Photochemical Function Relationships in Nitrogen-Containing Heterocyclic Aromatic Photobases Derived from Quinoline, The Journal of Physical Chemistry A, 124(13), 2537–2546, doi:10.1021/acs.jpca.9b11375. (published) [Comet, SDSC]

59. TG-CHE180074, TG-CHE180082 254. Zhao, Z., Zemerov, S., Roose, B., Dmochowski, I. 2020. Rational design of a genetically encoded Xe-129 NMR contrast agent for Zn2+ detection. This journal paper is currently in preparation as we are collecting key experiment results.. (in preparation) [Bridges Regular, PSC]

60. TG-CHE180093 255. Lemay, J.-C., Y. Dong, V. Albert, M. Inouye, M. N. Groves, J. Boukouvalas, and P. H. McBreen (2020), Relative Abundances of Surface Diastereomeric Complexes Formed by Two Chiral Modifiers That Differ by a Methyl Group, ACS Catalysis, 10(5), 3034–3041, doi:10.1021/acscatal.9b04682. (published) [Comet, SDSC]

61. TG-CHE190009 256. Bononi, F., Chen, Z., Andreussi, O., Hullar, T., Anastasio, C., et al. 2020. Bathochromic shift in the UV-Visible Absorption Spectra of Phenolic Molecules at Ice Surfaces: insights from First-Principles Calculations. (in preparation) 257. Hullar, T., Bononi, F., Chen, Z., Magadia, D., Palmer, O., et al. 2020. Photodegradation rate constants for guaiacol are faster in/on ice than in aqueous solution. (in preparation)

62. TG-CHE190014 258. Iovanac, N. C., and B. M. Savoie (2020), Simpler is Better: How Linear Prediction Tasks Improve Transfer Learning in Chemical Autoencoders, The Journal of Physical Chemistry A, 124(18), 3679–3685, doi:10.1021/acs.jpca.0c00042. (published) 259. Khot, A., S. B. Shiring, and B. M. Savoie (2019), Evidence of information limitations in coarse-grained models, The Journal of Chemical Physics, 151(24), 244105, doi:10.1063/1.5129398. (published) 260. Zhao, Q., and B. M. Savoie (2020), Self-Consistent Component Increment Theory for Predicting Enthalpy of Formation, Journal of Chemical Information and Modeling, 60(4), 2199–2207, doi:10.1021/acs.jcim.0c00092. (published)

63. TG-CHE190045 261. Rosneck, L., Rivera-Rivera, L. 2019. Molecular Dynamics Thermal Partial-Denaturation of the Amyloid Precursor Protein's C99 Transmembrane Domain. 13th Annual West Michigan Regional Undergraduate Science (WMRUGS) Research Conference (Van Andel Institute, Grand Rapids, MI 49503). (published) [Comet, Data Oasis, SDSC, Training]

RY5 IPR12 Page 130 64. TG-CHE190062 262. Singh, M., N. Garza, Z. Pearson, J. Douglas, and Z. Boskovic (2020), Broad assessment of bioactivity of a collection of spiroindane pyrrolidines through “cell painting,” Bioorganic & Medicinal Chemistry, 28(13), 115547, doi:10.1016/j.bmc.2020.115547. (published) [Bridges Regular, PSC]

65. TG-CHE190065 263. Li, G., Z. Chen, Y. Li, D. Zhang, W. Yang, Y. Liu, and L. Cao (2020), Engineering Substrate Interaction To Improve Hydrogen Evolution Catalysis of Monolayer MoS2 Films beyond Pt, ACS Nano, 14(2), 1707–1714, doi:10.1021/acsnano.9b07324. (published) [Comet, Lonestar, SDSC, Stampede2, TACC] 264. Tang, C. et al. (2020), CO2 Reduction on Copper’s Twin Boundary, ACS Catalysis, 10(3), 2026–2032, doi:10.1021/acscatal.9b03814. (published) [SDSC, TACC]

66. TG-CHE190066 265. Abularrage, N. S., B. J. Levandowski, and R. T. Raines (2020), Synthesis and Diels–Alder Reactivity of 4-Fluoro-4- Methyl-4H-Pyrazoles, International Journal of Molecular Sciences, 21(11), 3964, doi:10.3390/ijms21113964. (published) 266. Levandowski, B. J., N. S. Abularrage, and R. T. Raines (2020), Differential Effects of Nitrogen Substitution in 5‐ and 6‐ Membered Aromatic Motifs, Chemistry – A European Journal, 26(41), 8862–8866, doi:10.1002/chem.202000825. (published) [Bridges Regular, PSC, Pylon]

67. TG-CHE190111 267. Alegre-Requena, J. V., A. Valero-Tena, I. G. Sonsona, S. Uriel, and R. P. Herrera (2020), Simple iodoalkyne-based organocatalysts for the activation of carbonyl compounds, Organic & Biomolecular Chemistry, 18(8), 1594–1601, doi:10.1039/c9ob02688f. (published) [Bridges Regular, Comet, Data Oasis, PSC, Pylon, SDSC] 268. Sonsona, I. G., J. V. Alegre‐Requena, E. Marqués‐López, M. C. Gimeno, and R. P. Herrera (2020), Asymmetric Organocatalyzed Aza‐Henry Reaction of Hydrazones: Experimental and Computational Studies, Chemistry – A European Journal, 26(24), 5469–5478, doi:10.1002/chem.202000232. (published) [Bridges Regular, Comet, Data Oasis, PSC, Pylon, SDSC]

68. TG-CIE170005 269. Chen, X., R. Dathathri, G. Gill, and K. Pingali (2020), Pangolin, Proceedings of the VLDB Endowment, 13(8), 1190– 1205, doi:10.14778/3389133.3389137. (published) 270. Dathathri, R., G. Gill, L. Hoang, V. Jatala, K. Pingali, V. K. Nandivada, H.-V. Dang, and M. Snir (2019), Gluon-Async: A Bulk-Asynchronous System for Distributed and Heterogeneous Graph Analytics, 2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT), doi:10.1109/pact.2019.00010. (published) 271. Gill, G., Dathathri, R., Hoang, L., Peri, R., Pingali, K. 2020. Single Machine Graph Analytics on Massive Datasets Using Intel Optane DC Persistent Memory. 13(8). (published) 272. Hoang, L., R. Dathathri, G. Gill, and K. Pingali (2019), CuSP: A Customizable Streaming Edge Partitioner for Distributed Graph Analytics, 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), doi:10.1109/ipdps.2019.00054. (published) 273. Hoang, L., V. Jatala, X. Chen, U. Agarwal, R. Dathathri, G. Gill, and K. Pingali (2019), DistTC: High Performance Distributed Triangle Counting, 2019 IEEE High Performance Extreme Computing Conference (HPEC), doi:10.1109/hpec.2019.8916438. (published) 274. Jatala, V., Dathathri, R., Gill, G., Hoang, L., Krishna Nandivada, V., et al. 2020. A Study of Graph Analytics for Massive Datasets on Distributed GPUs. International Parallel and Distributed Processing Symposium (IPDPS). (published)

69. TG-CIE170034 275. Cheng, C., Tan, F., Hou, X., Wei, Z. 2019. A Deep Learning Approach to Competing Risks Representation in Peer-to-Peer Lending. International Joint Conference on Artificial Intelligence (IJCAI) (Macao, China). (published) 276. Cheng, C., Tan, F., Wei, Z. 2020. DeepVar: An End-to-End Deep Learning Approach for Genomic Variant Recognition in Biomedical Literature. AAAI Conference on Artificial Intelligence (New York City, USA). (published)

RY5 IPR12 Page 131 70. TG-CTS090025 277. Papavassiliou, D., Nguyen, Q. 2019. USING HELICITY TO INVESTIGATE SCALAR TRANSPORT IN WALL TURBULENCE. (submitted) 278. Papavassiliou, D., Nguyen, V. 2020. Hydrodynamic Dispersion in Porous Media and the Significance of Lagrangian Time and Space Scales . (submitted) [Stampede, TACC]

71. TG-CTS100024 279. Freund, A., and A. Ferrante (2019), Wavelet-spectral analysis of droplet-laden isotropic turbulence, Journal of Fluid Mechanics, 875, 914–928, doi:10.1017/jfm.2019.515. (published)

72. TG-CTS110007 280. Khanwale, M. A., A. D. Lofquist, H. Sundar, J. A. Rossmanith, and B. Ganapathysubramanian (2020), Simulating two- phase flows with thermodynamically consistent energy stable Cahn-Hilliard Navier-Stokes equations on parallel adaptive octree based meshes, Journal of Computational Physics, 419, 109674, doi:10.1016/j.jcp.2020.109674. (published) [Stampede2, TACC]

73. TG-CTS110029 281. Baranwal, A., D. A. Donzis, and R. D. Bowersox (2020), Vibrational turbulent Prandtl number in flows with thermal non-equilibrium, AIAA Scitech 2020 Forum, doi:10.2514/6.2020-2052. (published) [TACC] 282. Chen, C. H., D. A. Donzis, and R. D. Bowersox (2020), Characteristic Locations in Shock-Turbulence Interactions, AIAA Scitech 2020 Forum, doi:10.2514/6.2020-1812. (published) [TACC]

74. TG-CTS110039 283. - Conjugated n-Ethylene-Glycol-Terminated Quaterthiophene Oligomers: A Computational and Experimental Study, ACSMisra, Macro M., Z. Letters, Liu, B. X.9(3), Dong, 295 S.– N.300, Patel, doi:10.1021/acsmacrolett.9b00935. P. F. Nealey, C. K. Ober, and F. A. Escobedo (published (2020),) Thermal Stability of π

75. TG-CTS130006 284. Beardsell, G., Blanquart, G. 2019. Fully compressible simulations of the impact of acoustic waves on the dynamics of laminar premixed flames for engine-relevant conditions. (submitted) 285. Beardsell, G., Blanquart, G. 2020. A cost-effective semi-implicit method for the time integration of fully compressible reacting flows with stiff chemistry. (accepted) 286. Ruan, J., Blanquart, G. 2020. Direct Numerical Simulations of a Statistically Stationary Stream-wise Periodic Boundary Layer. (submitted)

76. TG-CTS160057 287. Hu, H., Weibel, J., Garimella, S. 2019. A Figure of Merit for the Design of Surface Structures for Enhanced Critical Heat Flux. CTRC Poster. (published) [Bridges Regular, Comet, PSC, Stampede, TACC]

77. TG-CTS170005 288. Iyemperumal, S. K., T. D. Pham, J. Bauer, and N. A. Deskins (2018), Quantifying Support Interactions and Reactivity Trends of Single Metal Atom Catalysts over TiO2, The Journal of Physical Chemistry C, 122(44), 25274–25289, doi:10.1021/acs.jpcc.8b05611. (published) [Stampede, Stampede2]

78. TG-CTS170045 289. Chew, A., Jiang, S., Zavala, V., Van Lehn, R. 2020. Fast Predictions of Liquid-Phase Acid-Catalyzed Reaction Rates Using Molecular Dynamics and Convolutional Neural Networks. (submitted) [Stampede2, TACC] 290. Gahan, C., Patel, S., Boursier, M., Nyffeler, K., Abbott, N., et al. 2020. Gram-Negative Bacterial Quorum Sensing Signals Self-Assemble in Aqueous Media to Form Micelles and Vesicles: An Integrated Experimental and Molecular Dynamics Study. (submitted) [Stampede2, TACC] 291. Hoover, B., Shen, Z., Gahan, C., Lynn, D., Van Lehn, R., et al. 2020. Membrane Remodeling Stimulates the Aggregation of -Synuclein. (submitted) [Stampede2, TACC]

Α

RY5 IPR12 Page 132 292. Shen, Z., Van Lehn, R. 2020. Solvent Selection for the Separation of Lignin-Derived Monomers Using the Conductor- Like Screening Model for Real Solvents. (published) [Stampede2, TACC] 293. Walker, T., Chew, A., Van Lehn, R., Dumesic, J., Huber, G. 2020. Rational Design of Mixed Solvent Systems for Acid- Catalyzed Biomass Conversion Processes Using a Combined Experimental, Molecular Dynamics and Machine Learning Approach. (published) [Stampede2, TACC] 294. Walker, T., Frelka, N., Shen, Z., Chew, A., Bannick, J., et al. 2020. Recycling of Multilayer Plastic Packaging Materials by Solvent-Targeted Recovery and Precipitation. (submitted) [Stampede2, TACC]

79. TG-CTS180002 295. Kasbaoui, M., Kulkarni, T., Bisetti, F. 2020. Characterization of the swirling von Kármán flow with a moving immersed-boundary method. (submitted) 296. Kulkarni, T., Bisetti, F. 2020. Evolution and scaling of peak flame surface density in spherical turbulent premixed flames subjected to decaying isotropic turbulence. (accepted) [Stampede2, TACC] 297. Kulkarni, T., Bisetti, F. 2020. Surface morphology and inner fractal cutoff scale of spherical turbulent premixed flames in decaying isotropic turbulence. (accepted) [Stampede2, TACC] 298. Kulkarni, T., Buttay, R., Kasbaoui, M., Attili, A., Bisetti, F. 2020. Reynolds number scaling of burning rates in spherical turbulent premixed flames. (submitted)

80. TG-CTS180037 299. Agarwal, A., Trujillo, M. 2019. A Computational Study of Nozzle Internal Flow and its Effect on Spray Atomization. ILASS-Americas 30th Annual Conference on Liquid Atomization and Spray Systems (Tempe, AZ). http://www.ilass.org/2/recent-papers-form.html. (published) 300. Kuo, C., Trujillo, M. 2019. Speedup Analysis of Adaptive Mesh Refinement in the Simulation of Spray Formation. ILASS-Americas 30th Annual Conference on Liquid Atomization and Spray Systems (Tempe, AZ). http://www.ilass.org/2/recent-papers-form.html. (published) 301. Kuo, C., Trujillo, M. 2019. A Study of Adaptive Mesh Refinement Speedup in Spray Atomization. International Multidimensional Engine Modeling User's Group Meeting at the SAE Congress (Detroit, Michigan). https://www.combustioninstitute.org/ci-event/international-multidimensional-engine-modeling-meeting/. (published) [Bridges Regular, PSC]

81. TG-CTS180057 302. Rosen, A. S., M. R. Mian, T. Islamoglu, H. Chen, O. K. Farha, J. M. Notestein, and R. Q. Snurr (2020), Tuning the Redox Activity of Metal–Organic Frameworks for Enhanced, Selective O2 Binding: Design Rules and Ambient Temperature O2 Chemisorption in a Cobalt–Triazolate Framework, Journal of the American Chemical Society, 142(9), 4317–4328, doi:10.1021/jacs.9b12401. (published) [Stampede2, TACC] 303. Rosen, A., Notestein, J., Snurr, R. 2020. Comparing GGA, GGA+U, and Meta-GGA Functionals for Redox-Dependent Bindi accepted) 304.Rosen, A. S., J. M. Notestein, and R. Q. Snurr (2020), High‐Valent Metal–Oxo Species at the Nodes of Metal–Triazolate ng at Open Metal Sites in Metal−Organic Frameworks. ( Frameworks: The Effects of Ligand Exchange and Two‐ Bond Activation, Angewandte Chemie International Edition, doi:10.1002/anie.202004458. (published) [Stampede2, TACC] State Reactivity for C−H 82. TG-CTS190001, TG-MCB190100 305. Zhang, X., C. Caruso, W. A. Lam, and M. D. Graham (2020), Flow-induced segregation and dynamics of red blood cells in sickle cell disease, Physical Review Fluids, 5(5), doi:10.1103/physrevfluids.5.053101. (published) [Comet, SDSC]

83. TG-CTS190021 306. Jeun, J., and J. W. Nichols (2017), Wavepacket modeling of turbulent jet noise generation using input-output analysis, 23rd AIAA/CEAS Aeroacoustics Conference, doi:10.2514/6.2017-3378. (published) 307. Jeun, J., and J. W. Nichols (2018), Non-compact sources of sound in high-speed turbulent jets using input-output analysis, 2018 AIAA/CEAS Aeroacoustics Conference, doi:10.2514/6.2018-3467. (published) 308. Jeun, J., J. W. Nichols, and M. R. Jovanovic (2016), Input-output analysis of heated axisymmetric turbulent jets, 22nd AIAA/CEAS Aeroacoustics Conference, doi:10.2514/6.2016-2934. (published)

RY5 IPR12 Page 133 309. 2016), Input-output analysis of high-speed axisymmetric isothermal jet noise, Physics of Fluids, 28(4), 047101, doi:10.1063/1.4946886. (published) Jeun, J., J. W. Nichols, and M. R. Jovanović ( 310. Jeun, J., G. J. Wu, and S. K. Lele (2020), Large eddy simulations of screeching twin rectangular jets, AIAA Scitech 2020 Forum, doi:10.2514/6.2020-0998. (published) 311. Wu, G., S. K. Lele, and J. Jeun (2019), Towards Large Eddy Simulations of Supersonic Rectangular Jets including Screech, 25th AIAA/CEAS Aeroacoustics Conference, doi:10.2514/6.2019-2520. (published) 312. Wu, G., Lele, S., Jeun, J. 2020. Numerical Study of Screech Closure Mechanism in a Rectangular Supersonic Jet. 2020 AIAA Aviation Forum (Reno, Nevada). (accepted) [Stampede2, TACC]

84. TG-DBS180013 313. Lacomis, J., Yin, P., Schwartz, E., Allamanis, M., Le Goues, C., et al. 2019. DIRE: A Neural Approach to Decompiled Identifier Naming. 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) (San Diego, CA, USA). (published)

85. TG-DEB090011 314. Miller, M., Pfeiffer, W., Cooper, T., Mishin, D., Chen, T., et al. 2019. Cloud bursting to AWS from the CIPRES Science Gateway. Gateways 2019 (San Diego, CA). (accepted)

86. TG-DEB110008 315. Cardoso, G. C., B. T. Klingbeil, F. A. La Sorte, C. A. Lepczyk, D. Fink, and C. H. Flather (2020), Exposure to noise pollution across North American passerines supports the noise filter hypothesis, edited by C. Sheard, Global Ecology and Biogeography, 29(8), 1430–1434, doi:10.1111/geb.13085. (published) [Lonestar, Ranch, TACC] 316. Fink, D., LaSorte, F., Farnsworth, A., Selling, S., Rohrbaugh, R. 2012. eBird Species Distribution Modeling of Migrant Landbirds for Balancing Wind and Wildlife: New Data and Tools to Improve Wind Project Siting for Biodiversity Conservation. Final report for New York State Energy Research and Development Authority (NYSERDA) Contract 1320853890. (published) [Lonestar, Ranch, TACC] 317. Fink, D., LaSorte, F., Iliff, M., Rosenberg, K. 2013. Final Technical Report: Species Distribution Modeling of Priority Bird Species on Bureau of Land Management Lands to Determine Stewardship Responsibility for Conservation Planning. Final Report for U.S. Bureau of Land Management Contract . (published) [Lonestar, Ranch, TACC] 318. Klingbeil, B. T., F. A. La Sorte, C. A. Lepczyk, D. Fink, and C. H. Flather (2019), Geographical associations with anthropogenic noise pollution for North American breeding birds, edited by C. Sheard, Global Ecology and Biogeography, 29(1), 148–158, doi:10.1111/geb.13016. (published) [Lonestar, Ranch, TACC] 319. Rosenberg, K., Fink, D. 2015. Final Report: The State of North America’s Birds and Modeling North Carolina Bird Species Distribution Dynamics. Final report to North Carolina Wildlife Resources Commission.. (accepted) [Lonestar, Ranch, TACC] 320. Wood, C., LaSorte, F., Fink, D., Rosenberg, K., Iliff, M., et al. 2013. Final Report: Species Distribution Modeling of Priority Bird Species on United States Forest Service Lands to Determine Stewardship Responsibility for Conservation Planning.. Final report for U.S. Forest Service Contract Number 12-CS-11132422-405.. (published) [Lonestar, Ranch, TACC]

87. TG-DEB160018, TG-EAR160002 321. Bassiouni, M., S. P. Good, C. J. Still, and C. W. Higgins (2020), Plant Water Uptake Thresholds Inferred From Satellite Soil Moisture, Geophysical Research Letters, 47(7), doi:10.1029/2020gl087077. (published)

88. TG-DMR050013N 322. Fyhrie, M., Q.-J. Hong, D. Kapush, S. V. Ushakov, H. Liu, A. van de Walle, and A. Navrotsky (2019), Energetics of melting of Yb2O3 and Lu2O3 from drop and catch calorimetry and first principles computations, The Journal of Chemical Thermodynamics, 132, 405–410, doi:10.1016/j.jct.2019.01.008. (published) [Stampede, TACC] 323. Hong, Q.-J., S. V. Ushakov, D. Kapush, C. J. Benmore, R. J. K. Weber, A. van de Walle, and A. Navrotsky (2018), Combined computational and experimental investigation of high temperature thermodynamics and structure of cubic ZrO2 and HfO2, Scientific Reports, 8(1), doi:10.1038/s41598-018-32848-7. (published) [Stampede, TACC] 324. Hong, Q.-J., and A. van de Walle (2019), Reentrant melting of sodium, magnesium, and aluminum: General trend, Physical Review B, 100(14), doi:10.1103/physrevb.100.140102. (published) [Stampede, TACC]

RY5 IPR12 Page 134 325. Sun, R., M. Asta, and A. van de Walle (2019), First-principles thermal compatibility between Ru-based Re-substitute alloys and Ir coatings, Computational Materials Science, 170, 109199, doi:10.1016/j.commatsci.2019.109199. (published) [Stampede, TACC] 326. Ushakov, S. V., A. Navrotsky, Q.-J. Hong, and A. van de Walle (2019), Carbides and Nitrides of Zirconium and Hafnium, Materials, 12(17), 2728, doi:10.3390/ma12172728. (published) [Stampede, TACC] 327. Van de Walle, A., and M. Asta (2019), High-throughput calculations in the context of alloy design, MRS Bulletin, 44(4), 252–256, doi:10.1557/mrs.2019.71. (published) [Stampede, TACC] 328. Van de Walle, A., and Q. Hong (2019), Assessing Phase Diagram Accuracy, Journal of Phase Equilibria and Diffusion, 40(2), 170–175, doi:10.1007/s11669-019-00711-5. (published) [Stampede, TACC] 329. Van de Walle, A., J. E. C. Sabisch, A. M. Minor, and M. Asta (2019), Identifying rhenium substitute candidate multiprincipal-element alloys from electronic structure and thermodynamic criteria, Journal of Materials Research, 34(19), 3296–3304, doi:10.1557/jmr.2019.179. (published) [Stampede, TACC]

89. TG-DMR060049N 330. Balankura, T., X. Qi, and K. A. Fichthorn (2018), Solvent Effects on Molecular Adsorption on Ag Surfaces: Polyvinylpyrrolidone Oligomers, The Journal of Physical Chemistry C, 122(26), 14566–14573, doi:10.1021/acs.jpcc.8b03156. (published) [Stampede, TACC] 331. Balankura, T., T. Yan, O. Jahanmahin, J. Narukatpichai, A. Ng, and K. A. Fichthorn (2020), Oriented attachment mechanism of triangular Ag nanoplates: a molecular dynamics study, Nanoscale Advances, 2(6), 2265–2270, doi:10.1039/d0na00124d. (published)

90. TG-DMR080058N 332. Batra, R., Pilania, G., Uberuaga, B., Ramprasad, R. 2019. Multifidelity Information Fusion with Machine Learning: A Case Study of Dopant Formation Energies in Hafnia. (published) [Comet, SDSC, Stampede2, TACC] 333. Batra, R., Tran, H., Johnson, B., Zoellner, B., Maggard, P., et al. 2020. Search for Ferroelectric Binary Oxides: Chemical and Structural Space Exploration Guided by Group Theory and Computations. (published) [Comet, SDSC, Stampede2, TACC] 334. Batra, R., H. D. Tran, C. Kim, J. Chapman, L. Chen, A. Chandrasekaran, and R. Ramprasad (2019), General Atomic Neighborhood Fingerprint for Machine Learning-Based Methods, The Journal of Physical Chemistry C, 123(25), 15859–15866, doi:10.1021/acs.jpcc.9b03925. (published) [Comet, SDSC, Stampede2, TACC] 335. Chen, L., Venkatram, S., Kim, C., Batra, R., Chandrasekaran, A., et al. 2019. Electrochemical Stability Window of Polymeric Electrolytes. (published) [Comet, SDSC, Stampede2, TACC] 336. Kamal, D., Chandrasekaran, A., Batra, R., Ramprasad, R. 2020. A charge density prediction model for hydrocarbons using deep neural networks. (published) [Comet, SDSC, Stampede2, TACC]

91. TG-DMR090023 337. Ma, T., Jacobs, R., Booske, J., Morgan, D. 2020. Understanding the interplay of surface structure and work function in oxides: a case study on SrTiO3. (submitted)

92. TG-DMR110037 338. Hachtel, J. A., J. Huang, I. Popovs, S. Jansone-Popova, J. K. Keum, J. Jakowski, T. C. Lovejoy, N. Dellby, O. L. Krivanek, and J. C. Idrobo (2019), Identification of site-specific isotopic labels by vibrational spectroscopy in the electron microscope, Science, 363(6426), 525–528, doi:10.1126/science.aav5845. (published) [GaTech, IU, LSU, NICS, OSG, PSC, SDSC, Stanford, TACC] 339. Hachtel, J. A., J. Huang, I. Popovs, S. Jansone-Popova, J. K. Keum, J. Jakowski, T. C. Lovejoy, N. Dellby, O. L. Krivanek, and J. C. Idrobo (2019), Damage-Free Nanoscale Isotopic Analysis of Biological Materials with Vibrational Electron Spectroscopy, Microscopy and Microanalysis, 25(S2), 1088–1089, doi:10.1017/s1431927619006172. (published) [GaTech, IU, LSU, NICS, OSG, PSC, SDSC, Stanford, TACC] 340. Lingerfelt, D., J. Jakowski, P. Ganesh, and B. Sumpter (2019), A TD-DFT Treatment of Electronic Excitations in the STEM Spanning Dipole and Impact Scattering Regimes, Microscopy and Microanalysis, 25(S2), 2300–2301, doi:10.1017/s1431927619012236. (published)

RY5 IPR12 Page 135 93. TG-DMR110093 341. Dutta, R., B. Kiefer, E. Greenberg, V. B. Prakapenka, and T. S. Duffy (2019), Ultrahigh-Pressure Behavior of AO2 (A = Sn, Pb, Hf) Compounds, The Journal of Physical Chemistry C, 123(45), 27735–27741, doi:10.1021/acs.jpcc.9b06856. (published) [Stampede, TACC] 342. Ghazisaeed, S., M. Minuddin, H. Nakotte, and B. Kiefer (2020), Density-functional-theory-predicted symmetry lowering from cubic to tetragonal in nickel hexacyanoferrate, Journal of Applied Crystallography, 53(1), 117–126, doi:10.1107/s1600576719016492. (published) [Stampede, TACC]

94. TG-DMR120073 343. Chen, W., L. Sun, B. Kozinsky, C. M. Friend, E. Kaxiras, P. Sautet, and R. J. Madix (2019), Effect of Frustrated Rotations on the Pre-Exponential Factor for Unimolecular Reactions on Surfaces: A Case Study of Alkoxy Dehydrogenation, The Journal of Physical Chemistry C, 124(2), 1429–1437, doi:10.1021/acs.jpcc.9b10017. (published) [Stampede, TACC] 344. Kuate Defo, R., Gadalla, M., Johnson, K., Greenspon, A., Kaxiras, E., et al. 2020. Mechanism for the Enhanced Luminescence from the Negative Silicon Monovacancy in 4H-SiC under Laser Illumination and Thermal Annealing. (in preparation) 345. Kuate Defo, R., E. Kaxiras, and S. L. Richardson (2019), How carbon vacancies can affect the properties of group IV color centers in diamond: A study of thermodynamics and kinetics, Journal of Applied Physics, 126(19), 195103, doi:10.1063/1.5123227. (published) 346. Kuate Defo, R., Zhang, X., Kaxiras, E. 2020. Mechanism for the ionization of the NV(-) defect center in diamond. (submitted) [Stampede2, TACC] 347. (submitted) Kuate Defo, R., Zhang, X., Kaxiras, E. 2020. Mechanism for the Ionization of the NV(−) Defect Center in Diamond. 348. Kucukbenli, E., Kaxiras, E. 2020. Neural network architectures for interatomic potentials of layered materials. (in preparation) [Stampede2, TACC] 349. Li, Q., Kucukbenli, E., Lam, S., Khaykovich, B., Kaxiras, E., et al. 2012. A neural-network interatomic potential for molten NaCl. (in preparation) [Stampede2, TACC] 350. Shaidu, Y., Kucukbenli, E., Lot, R., Pellegrini, F., Kaxiras, E., et al. 2020. A systematic approach to generating accurate neural network potentials: the case of carbon. (in preparation) [Stampede2, TACC] 351. Tritsaris, G., Carr, S., Zhu, Z., Xie, Y., Torrisi, S., et al. 2020. Electronic structure calculations of twisted multi-layer graphene superlattices. (published) [Stampede2, TACC] 352. Tritsaris, G., Xie, Y., Rush, A., Carr, S., Mattheakis, M., et al. 2019. LAN -- A materials notation for 2D layered assemblies. (submitted) [Stampede2, TACC]

95. TG-DMR130036 353. Dong, X., X. Chen, and E. Gull (2019), Dynamical charge susceptibility in the Hubbard model, Physical Review B, 100(23), doi:10.1103/physrevb.100.235107. (published) [Stampede2, TACC] 354. Li, S., and E. Gull (2020), Magnetic and charge susceptibilities in the half-filled triangular lattice Hubbard model, Physical Review Research, 2(1), doi:10.1103/physrevresearch.2.013295. (published) [Stampede2, TACC]

96. TG-DMR130119 355. Ke, X. et al. (2019), Ideal maximum strengths and defect-induced softening in nanocrystalline-nanotwinned metals, Nature Materials, 18(11), 1207–1214, doi:10.1038/s41563-019-0484-3. (published) [LSU, SuperMIC] 356. Wang, J., G. Cao, Z. Zhang, and F. Sansoz (2019), Size-dependent dislocation–twin interactions, Nanoscale, 11(26), 12672–12679, doi:10.1039/c9nr03637g. (published) [Comet, SDSC]

97. TG-DMR140005 357. Cooper, A. M., J. Kästner, A. Urban, and N. Artrith (2020), Efficient training of ANN potentials by including atomic forces via Taylor expansion and application to water and a transition-metal oxide, npj Computational Materials, 6(1), doi:10.1038/s41524-020-0323-8. (published) [Stampede, TACC]

RY5 IPR12 Page 136 98. TG-DMR140008 358. Thomas, S., M. S. Manju, K. M. Ajith, S. U. Lee, and M. Asle Zaeem (2020), Strain-induced work function in h-BN and BCN monolayers, Physica E: Low-dimensional Systems and Nanostructures, 123, 114180, doi:10.1016/j.physe.2020.114180. (published) [Comet, SDSC]

99. TG-DMR140129 359. VanSaders, B., and S. C. Glotzer (2020), Pinning dislocations in colloidal crystals with active particles that seek stacking faults, Soft Matter, 16(17), 4182–4191, doi:10.1039/c9sm02514f. (published) [Comet, SDSC]

100. TG-DMR150103 360. Altan, I., and P. Charbonneau (2019), Obtaining Soft Matter Models of Proteins and their Phase Behavior, Protein Self- Assembly, 209–228, doi:10.1007/978-1-4939-9678-0_15. (published) 361. Altan, I., A. R. Khan, S. James, M. K. Quinn, J. J. McManus, and P. Charbonneau (2019), Using Schematic Models to -Crystallin, The Journal of Physical Chemistry B, 123(47), 10061–10072, doi:10.1021/acs.jpcb.9b07774. (published) Understand the Microscopic Basis for Inverted Solubility in γD 362. Berthier, L., Charbonneau, P., Kundu, J. 2020. Finite-dimensional vestige of spinodal criticality above the dynamical glass transition. (submitted) 363. Khan, A. R., S. James, M. K. Quinn, I. Altan, P. Charbonneau, and J. J. McManus (2019), Temperature-Dependent -Crystallin Mutant, Biophysical Journal, 117(5), 930–937, doi:10.1016/j.bpj.2019.07.019. (published) Interactions Explain Normal and Inverted Solubility in a γD 364. Morse, P., Charbonneau, P. 2020. Memory formation in hard sphere glasses. (in preparation)

101. TG-DMR160007 365. Hemmat, Z. et al. (2020), Quasi‐Binary Transition Metal Dichalcogenide Alloys: Thermodynamic Stability Prediction, Scalable Synthesis, and Application, Advanced Materials, 32(26), 1907041, doi:10.1002/adma.201907041. (published) [Comet, SDSC, Stampede, Stampede2, TACC] 366. Ranga, P., S. B. Cho, R. Mishra, and S. Krishnamoorthy (2020), Highly tunable, polarization-engineered two- dimensional - -Ga2O3 heterostructures, Applied Physics Express, 13(6), 061009, doi:10.35848/1882-0786/ab9168. (published) [Comet, SDSC, Stampede2, TACC] electron gas in ε AlGaO3/ε 102. TG-DMR160111 367. Hinkle, K. R., and F. R. Phelan (2020), Solvation Free Energy of Self-Assembled Complexes: Using Molecular Dynamics to Understand the Separation of ssDNA-Wrapped Single-Walled Carbon Nanotubes, The Journal of Physical Chemistry C, 124(24), 13127–13140, doi:10.1021/acs.jpcc.0c00983. (published) [Stampede2, TACC]

103. TG-DMR160146 368. Christiansen, D. T., S. Ohtani, Y. Chujo, A. L. Tomlinson, and J. R. Reynolds (2019), All Donor Electrochromic Polymers Tunable across the Visible Spectrum via Random Copolymerization, Chemistry of Materials, 31(17), 6841–6849, doi:10.1021/acs.chemmater.9b01293. (published) [Comet, SDSC]

104. TG-DMR160159 369. Heine, M., O. Hellman, and D. Broido (2019), Effect of thermal lattice and magnetic disorder on phonons in bcc Fe: A first-principles study, Physical Review B, 100(10), doi:10.1103/physrevb.100.104304. (published)

105. TG-DMR160170 370. Daggag, D., T. Dorlus, and T. Dinadayalane (2019), Binding of histidine and proline with graphene: DFT study, Chemical Physics Letters, 730, 147–152, doi:10.1016/j.cplett.2019.05.043. (published) [PSC, SDSC] 371. Daggag, D., J. Lazare, and T. Dinadayalane (2019), Data related to conformation dependence of tyrosine binding on the surface of graphene: Bent prefers over parallel orientation, Data in Brief, 26, 104420, doi:10.1016/j.dib.2019.104420. (published) [PSC, SDSC] 372. Dinadayalane, T., and N. J. Bowen (2019), Computational Chemistry and Biology Courses for Undergraduates at an HBCU: Cultivating a Diverse Computational Science Community, ACS Symposium Series, 67–81, doi:10.1021/bk- 2019-1328.ch005. (published) [PSC, SDSC]

RY5 IPR12 Page 137 106. TG-DMR170008 373. Beach, K., Lucking, M., Terrones, H. 2020. Strain dependence of second harmonic generation in transition metal dichalcogenide monolayers and the fine structure of the C exciton. DOI:10.1103/physrevb.101.155431. (published) [LSU, SuperMIC] 374. Kahn, E. et al. (2020), Selective Synthesis of Bi2Te3/WS2 Heterostructures with Strong Interlayer Coupling, ACS Applied Materials & Interfaces, doi:10.1021/acsami.0c03656. (published) [Stampede2, TACC] 375. Meng, Y. et al. (2018), Excitonic Complexes and Emerging Interlayer Electron–Phonon Coupling in BN Encapsulated Monolayer Semiconductor Alloy: WS0.6Se1.4, Nano Letters, 19(1), 299–307, doi:10.1021/acs.nanolett.8b03918. (published) [Stampede2, TACC] 376. Pradhan, N. R. et al. (2019), Raman and electrical transport properties of few-layered arsenic-doped black phosphorus, Nanoscale, 11(39), 18449–18463, doi:10.1039/c9nr04598h. (published) [Stampede2, TACC] 377. Zhang, T. et al. (2020), Universal In Situ Substitutional Doping of Transition Metal Dichalcogenides by Liquid-Phase Precursor-Assisted Synthesis, ACS Nano, 14(4), 4326–4335, doi:10.1021/acsnano.9b09857. (published) [Stampede2, TACC]

107. TG-DMR170031 378. Tuoc, V., Le, T., Huan, T., Thao, N. 2020. Novel cage-like nanoporous ZnO polymorphs with cubic lattice frameworks. (published) [Comet, SDSC] 379. Tuoc, V., Le, T., Huan, T., Trung, N. 2020. Structural, Electronic and Mechanical Properties of Few-Layer GaN Nanosheet: A First-Principle Study. (published) [Comet, SDSC]

108. TG-DMR170050 380. Rao, R., Carozo, V., Wang, Y., Islam, A., Perea-Lopez, N., et al. 2019. Dynamics of cleaning, passivating and doping monolayer MoS 2 by controlled laser irradiation. DOI:10.1088/2053-1583/ab33ab. (published) [Stampede2, TACC] 381. Wang, Y., Carvalho, B., Crespi, V. 2018. Strong exciton regulation of Raman scattering in monolayer MoS2. DOI:10.1103/physrevb.98.161405. (published) [Stampede2, TACC] 382. Xuan, Y. et al. (2019), Multi-scale modeling of gas-phase reactions in metal-organic chemical vapor deposition growth of WSe2, Journal of Crystal Growth, 527, 125247, doi:10.1016/j.jcrysgro.2019.125247. (published) [Stampede2, TACC] 383. Zhang, F., Wang, Y., Erb, C., Wang, K., Moradifar, P., et al. 2019. Full orientation control of epitaxial MoS2 on hBN assisted by substrate defects. DOI:10.1103/physrevb.99.155430. (published) [Stampede2, TACC] 384. Zhang, K., Y. Wang, J. Joshi, F. Zhang, S. Subramanian, M. Terrones, P. Vora, V. Crespi, and J. A. Robinson (2019), Probing the origin of lateral heterogeneities in synthetic monolayer molybdenum disulfide, 2D Materials, 6(2), 025008, doi:10.1088/2053-1583/aafd9a. (published) [Stampede2, TACC] 385. Zhang, X. et al. (2019), Defect-Controlled Nucleation and Orientation of WSe2on hBN: A Route to Single-Crystal Epitaxial Monolayers, ACS Nano, 13(3), 3341–3352, doi:10.1021/acsnano.8b09230. (published) [Stampede2, TACC]

109. TG-DMR170067, TG-DMR180003 386. Fisch, R. 2020. Behavior of the random-field $XY$ model on simple cubic lattices at $h_r =1.5$. (published) [Bridges Regular, PSC, Pylon]

110. TG-DMR170091 387. Kocevski, V., D. A. Lopes, A. J. Claisse, and T. M. Besmann (2020), Understanding the interface interaction between U3Si2 fuel and SiC cladding, Nature Communications, 11(1), doi:10.1038/s41467-020-16435-x. (published) [Comet]

111. TG-DMR180013 388. Ghatak, K., K. N. Kang, E.-H. Yang, and D. Datta (2020), Controlled edge dependent stacking of WS2-WS2 Homo- and WS2-WSe2 Hetero-structures: A Computational Study, Scientific Reports, 10(1), doi:10.1038/s41598-020-58149-6. (published) [Comet, SDSC]

112. TG-DMR180044 389. Chorsi, H., Yue, S., Iyer, P., Goyal, M., Schumann, T., et al. 2020. Widely Tunable Optical and Thermal Properties of Dirac Semimetal C3As2. DOI:10.1002/adom.201901192. (published)

RY5 IPR12 Page 138 390. Choudhry, U., S. Yue, and B. Liao (2019), Origins of significant reduction of lattice thermal conductivity in graphene allotropes, Physical Review B, 100(16), doi:10.1103/physrevb.100.165401. (published) 391. Hou, T., S. Yue, X. Sun, A. Fan, Y. Chen, M. Wang, S. Cai, C. Zheng, B. Liao, and J. Zhao (2020), Nitrogen-Doped graphene coated FeS2 microsphere composite as high-performance anode materials for sodium-ion batteries enhanced by the chemical and structural synergistic effect, Applied Surface Science, 505, 144633, doi:10.1016/j.apsusc.2019.144633. (published) 392. Vega-Flick, A., D. Jung, S. Yue, J. E. Bowers, and B. Liao (2019), Reduced thermal conductivity of epitaxial GaAs on Si due to symmetry-breaking biaxial strain, Physical Review Materials, 3(3), doi:10.1103/physrevmaterials.3.034603. (published) 393. Yue, S., Chorsi, H., Goyal, M., Schumann, T., Yang, R., et al. 2019. Soft phonons and ultralow lattice thermal conductivity in the Dirac semimetal Cd3As2. DOI:10.1103/physrevresearch.1.033101. (published) 394. Yue, S.-Y., R. Yang, and B. Liao (2019), Controlling thermal conductivity of two-dimensional materials via externally induced phonon-electron interaction, Physical Review B, 100(11), doi:10.1103/physrevb.100.115408. (published)

113. TG-DMR180075 395. Tsai, Y.-C., and C. Bayram (2020), Band Alignments of Ternary Wurtzite and Zincblende III-Nitrides Investigated by Hybrid Density Functional Theory, ACS Omega, 5(8), 3917–3923, doi:10.1021/acsomega.9b03353. (published)

114. TG-DMR180111 396. Ke, M., H. D. Nguyen, H. Fan, M. Li, H. Wu, and Y. Hu (2020), Complementary doping of van der Waals materials through controlled intercalation for monolithically integrated electronics, Nano Research, 13(5), 1369–1375, doi:10.1007/s12274-020-2634-y. (published) [Bridges Large, Bridges Regular, PSC, Pylon]

115. TG-DMR180112 397. Quan, Y., S. S. Ghosh, and W. E. Pickett (2019), Compressed hydrides as metallic hydrogen superconductors, Physical Review B, 100(18), doi:10.1103/physrevb.100.184505. (published) [Bridges Large, Bridges Regular, PSC, Pylon]

116. TG-DMR190005 398. Sharma, S., H. Singh, and X. Ko (2019), A Quantitatively Accurate Theory To Predict Adsorbed Configurations of Linear Surfactants on Polar Surfaces, The Journal of Physical Chemistry B, 123(34), 7464–7470, doi:10.1021/acs.jpcb.9b05861. (published) [Comet, SDSC] 399. Singh, H., and S. Sharma (2020), Disintegration of Surfactant Micelles at Metal–Water Interfaces Promotes Their Strong Adsorption, The Journal of Physical Chemistry B, 124(11), 2262–2267, doi:10.1021/acs.jpcb.9b10780. (published) [Comet, SDSC]

117. TG-DMR190035 400. Hu, Y.-J. et al. (2020), Predicting densities and elastic moduli of SiO2-based glasses by machine learning, npj Computational Materials, 6(1), doi:10.1038/s41524-020-0291-z. (published) [Stampede2]

118. TG-DMR190056 401. Rowsey, R., Taylor, E., Szilagyi, R., Stadie, N. 2020. Methane Adsorption on a Heteroatom-Modified Subunit of Porous Carbon Surfaces. (in preparation) [Comet, SDSC]

119. TG-DMR190068 402. Clay, R. T., and D. Roy (2020), Superconductivity due to cooperation of electron-electron and electron-phonon interactions at quarter filling, Physical Review Research, 2(2), doi:10.1103/physrevresearch.2.023006. (published) [Bridges Regular, PSC]

120. TG-DMR190089, TG-DMR200009 403. Jayan, R., and M. M. Islam (2020), Functionalized MXenes as effective polyselenide immobilizers for lithium–selenium batteries: a density functional theory (DFT) study, Nanoscale, 12(26), 14087–14095, doi:10.1039/d0nr02296a. (published) [Comet, SDSC]

RY5 IPR12 Page 139 121. TG-DMS060014 404. Brach, S., M. Z. Hossain, B. Bourdin, and K. Bhattacharya (2019), Anisotropy of the effective toughness of layered media, Journal of the Mechanics and Physics of Solids, 131, 96–111, doi:10.1016/j.jmps.2019.06.021. (published) [Stampede, TACC] 405. Brach, S., E. Tanné, B. Bourdin, and K. Bhattacharya (2019), Phase-field study of crack nucleation and propagation in elastic–perfectly plastic bodies, Computer Methods in Applied Mechanics and Engineering, 353, 44–65, doi:10.1016/j.cma.2019.04.027. (published) [Stampede, TACC] 406. Dunkel, A., Bourdin, B., Brandt, S. 2019. vDef-Web: A Case-Study on Building a Science Gateway Around a Research Code. Gateways 2019 (San Diego, CA). https://osf.io/meetings/gateways2019/. (published) [Stampede, TACC]

122. TG-DMS180010 407. Zheng, C., and N. Wang (2019), Collaborative representation with k-nearest classes for classification, Pattern Recognition Letters, 117, 30–36, doi:10.1016/j.patrec.2018.11.005. (published) [SDSC]

123. TG-DMS180026 408. Agarwal, D., J. Wang, and N. R. Zhang (2020), Data Denoising and Post-Denoising Corrections in Single Cell RNA Sequencing, Statistical Science, 35(1), 112–128, doi:10.1214/19-sts7560. (published) [ECSS, PSC, Pylon] 409. Wang, J., D. Agarwal, M. Huang, G. Hu, Z. Zhou, C. Ye, and N. R. Zhang (2019), Data denoising with transfer learning in single-cell transcriptomics, Nature Methods, 16(9), 875–878, doi:10.1038/s41592-019-0537-1. (published) [ECSS, PSC, Pylon]

124. TG-DMS190018 410. Ramasubramanian, V., and W. D. Beavis (2020), Factors affecting Response to Recurrent Genomic Selection in Soybeans, , doi:10.1101/2020.02.14.949008. (published) [Bridges Large, PSC, Pylon]

125. TG-DMS200011 411. Molina, E., B. Y. Zhou, J. J. Alonso, M. Righi, and R. G. Silva (2019), Flow and Noise Predictions Around Tandem Cylinders using DDES approach with SU2, AIAA Scitech 2019 Forum, doi:10.2514/6.2019-0326. (published)

126. TG-EAR160028 412. Jiménez‐Martínez, J., J. D. Hyman, Y. Chen, J. W. Carey, M. L. Porter, Q. Kang, G. Guthrie, and H. S. Viswanathan (2020), Homogenization of Dissolution and Enhanced Precipitation Induced by Bubbles in Multiphase Flow Systems, Geophysical Research Letters, 47(7), doi:10.1029/2020gl087163. (published) [Stampede2, TACC]

127. TG-EAR160036 413. Klesse, S. et al. (2020), Continental‐scale tree‐ring‐based projection of Douglas‐fir growth: Testing the limits of space‐ for‐time substitution, Global Change Biology, doi:10.1111/gcb.15170. (published) [IU, Jetstream, TACC]

128. TG-EAR190021 414. Dierauer, J., Zhu, C. 2019. Future Water Indiana: Modeling climate change impacts on Indiana's water resources. 2019 Geological Society of America Meeting (Phoenix, AZ). (published) [ECSS, IU, Jetstream, Science Gateways] 415. Dierauer, J., Zhu, C. 2019. Future Water Indiana: Climate change impacts on water quantity in the Wabash River Basin.. 2019 American Geophysical Union Meeting (San Francisco, CA). (published) [ECSS, IU, Jetstream, Science Gateways] 416. Dierauer, J. R., and C. Zhu (2020), Drought in the Twenty-First Century in a Water-Rich Region: Modeling Study of the Wabash River Watershed, USA, Water, 12(1), 181, doi:10.3390/w12010181. (published) [ECSS, IU, Jetstream, Science Gateways, Stampede] 417. Dierauer, J., Zhu, C., Gong, L., Alan, W., Pamidighantam, S., et al. 2020. FutureWater Indiana: A science gateway for spatio-temporal modeling of water in Wabash basin with climate change in focus. PEARC (Virtual due to COVID-19). (accepted) [ECSS, IU, Jetstream, Science Gateways, Stampede]

RY5 IPR12 Page 140 129. TG-ECD190001 418. Chu, C. et al. (2020), Spatially separating redox centers on 2D carbon nitride with cobalt single atom for photocatalytic H2O2production, Proceedings of the National Academy of Sciences, 117(12), 6376–6382, doi:10.1073/pnas.1913403117. (published) [Bridges Regular, PSC]

130. TG-GEO150002 419. Belgin, M., T. A. Perini, F. (Cherry) Liu, N. Zhang, S. Sarajlic, A. McNeill, P. Manno, and N. C. Bright (2019), A data‐ driven support strategy for a sustainable research software repository, Concurrency and Computation: Practice and Experience, 31(20), doi:10.1002/cpe.5338. (published) 420. Belgin, M., T. A. Perini, F. (Cherry) Liu, N. Zhang, S. Sarajlic, A. McNeill, P. Manno, and N. C. Bright (2019), A data‐ driven support strategy for a sustainable research software repository, Concurrency and Computation: Practice and Experience, 31(20), doi:10.1002/cpe.5338. (published)

131. TG-GEO160006 421. Abbaspour-Tamijani, A., J. W. Bennett, D. T. Jones, N. Cartagena-Gonzalez, Z. R. Jones, E. D. Laudadio, R. J. Hamers, J. A. Santana, and S. E. Mason (2020), DFT and thermodynamics calculations of surface cation release in LiCoO2, Applied Surface Science, 515, 145865, doi:10.1016/j.apsusc.2020.145865. (published) 422. Buchman, J. T. et al. (2020), Nickel enrichment of next-generation NMC nanomaterials alters material stability, causing unexpected dissolution behavior and observed toxicity to S. oneidensis MR-1 and D. magna, Environmental Science: Nano, 7(2), 571–587, doi:10.1039/c9en01074b. (published) 423. Hailu, A., A. A. Tamijani, S. E. Mason, and S. K. Shaw (2020), Efficient Conversion of CO2 to Formate Using Inexpensive and Easily Prepared Post-Transition Metal Alloy Catalysts, Energy & Fuels, 34(3), 3467–3476, doi:10.1021/acs.energyfuels.9b03783. (published)

132. TG-IBN180019, TG-IBN190010 424. De Bézieux, H. R., K. Street, S. Fischer, K. Van den Berge, R. Chance, D. Risso, J. Gillis, J. Ngai, E. Purdom, and S. Dudoit (2020), Improving replicability in single-cell RNA-Seq cell type discovery with Dune, , doi:10.1101/2020.03.03.974220. (published) [Bridges Large, Bridges Regular, PSC]

133. TG-IBN190010 425. Brann, D. H. et al. (2020), Non-neuronal expression of SARS-CoV-2 entry genes in the olfactory system suggests mechanisms underlying COVID-19-associated anosmia, , doi:10.1101/2020.03.25.009084. (published) [Bridges Large, Bridges Regular, PSC, Pylon]

134. TG-IRI180002 426. Chen, Y., Z. Zhang, C. Wu, D. Davaasuren, J. A. Goldstein, A. D. Gernand, and J. Z. Wang (2020), AI-PLAX: AI-based placental assessment and examination using photos, Computerized Medical Imaging and Graphics, 84, 101744, doi:10.1016/j.compmedimag.2020.101744. (published) [Bridges GPU, Bridges Regular, PSC] 427. Kamani, M., Farhang, S., Mahdavi, M., Wang, J. 2020. Targeted Data-driven Regularization for Out-of-Distribution Generalization. ACM SIGKDD Conference on Knowledge Discovery and Data Mining (San Diego, CA). http://infolab.stanford.edu/~wangz/project/imsearch/climate/KDD20/. (published) [Bridges GPU, Bridges Regular, PSC] 428. Wu, C., Chen, Y., Luo, J., Su, C., Dawane, A., et al. 2020. MEBOW: Monocular Estimation of Body Orientation In the Wild. International Conference on Computer Vision and Pattern Recognition (Seattle, WA). http://infolab.stanford.edu/~wangz/project/imsearch/BODY/CVPR20/wu.pdf. (published) [Bridges GPU, PSC] 429. Wu, C., Wang, J. 2020. MEBOW: Monocular Estimation of Body Orientation In the Wild Dataset. (published) [Bridges GPU, Bridges Regular, PSC]

135. TG-IRI190005 430. Shao, J., S. Ji, and T. Yang (2019), Privacy-aware Document Ranking with Neural Signals, Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, doi:10.1145/3331184.3331189. (published) [Comet, SDSC]

RY5 IPR12 Page 141 136. TG-MCA06N060 431. Aster, A., G. Licari, F. Zinna, E. Brun, T. Kumpulainen, E. Tajkhorshid, J. Lacour, and E. Vauthey (2019), Tuning symmetry breaking charge separation in perylene bichromophores by conformational control, Chemical Science, 10(45), 10629–10639, doi:10.1039/c9sc03913a. (published) [Stampede2, TACC] 432. Berman, H. M. et al. (2019), Federating Structural Models and Data: Outcomes from A Workshop on Archiving Integrative Structures, Structure, 27(12), 1745–1759, doi:10.1016/j.str.2019.11.002. (published) [Stampede2, TACC] 433. Coleman, J. A., D. Yang, Z. Zhao, P.-C. Wen, C. Yoshioka, E. Tajkhorshid, and E. Gouaux (2019), Serotonin transporter– ibogaine complexes illuminate mechanisms of inhibition and transport, Nature, 569(7754), 141–145, doi:10.1038/s41586-019-1135-1. (published) [Stampede2, TACC] 434. Jiang, T., P.-C. Wen, N. Trebesch, Z. Zhao, S. Pant, K. Kapoor, M. Shekhar, and E. Tajkhorshid (2020), Computational Dissection of Membrane Transport at a Microscopic Level, Trends in Biochemical Sciences, 45(3), 202–216, doi:10.1016/j.tibs.2019.09.001. (published) [Stampede2, TACC] 435. Jiang, X., A. J. Halmes, G. Licari, J. W. Smith, Y. Song, E. G. Moore, Q. Chen, E. Tajkhorshid, C. M. Rienstra, and J. S. Moore (2019), Multivalent Polymer–Peptide Conjugates: A General Platform for Inhibiting Amyloid Beta Peptide Aggregation, ACS Macro Letters, 8(10), 1365–1371, doi:10.1021/acsmacrolett.9b00559. (published) [Stampede2, TACC] 436. Kumar, P., Y. Wang, Z. Zhang, Z. Zhao, G. D. Cymes, E. Tajkhorshid, and C. Grosman (2020), Cryo-EM structures of a lipid-sensitive pentameric ligand-gated ion channel embedded in a phosphatidylcholine-only bilayer, Proceedings of the National Academy of Sciences, 117(3), 1788–1798, doi:10.1073/pnas.1906823117. (published) [Stampede2, TACC] 437. Li, J., Z. Zhao, and E. Tajkhorshid (2019), Locking Two Rigid-body Bundles in an Outward-Facing Conformation: The Ion-coupling Mechanism in a LeuT-fold Transporter, Scientific Reports, 9(1), doi:10.1038/s41598-019-55722-6. (published) [Stampede2, TACC] 438. Macchione, M., A. Goujon, K. Strakova, H. V. Humeniuk, G. Licari, E. Tajkhorshid, N. Sakai, and S. Matile (2019), A Chalcogen‐Bonding Cascade Switch for Planarizable Push–Pull Probes, Angewandte Chemie International Edition, 58(44), 15752–15756, doi:10.1002/anie.201909741. (published) [Stampede2, TACC] 439. Mahdavi, M., A. Fattahi, E. Tajkhorshid, and S. Nouranian (2020), Molecular Insights into the Loading and Dynamics of Doxorubicin on PEGylated Graphene Oxide Nanocarriers, ACS Applied Bio Materials, 3(3), 1354–1363, doi:10.1021/acsabm.9b00956. (published) [Stampede2, TACC] 440. Martens, C., M. Shekhar, A. M. Lau, E. Tajkhorshid, and A. Politis (2019), Integrating hydrogen–deuterium exchange mass spectrometry with molecular dynamics simulations to probe lipid-modulated conformational changes in membrane proteins, Nature Protocols, 14(11), 3183–3204, doi:10.1038/s41596-019-0219-6. (published) [Stampede2, TACC] 441. Misra, C. et al. (2020), Aberrant Expression of a Non-muscle RBFOX2 Isoform Triggers Cardiac Conduction Defects in Myotonic Dystrophy, Developmental Cell, 52(6), 748–763.e6, doi:10.1016/j.devcel.2020.01.037. (published) [Stampede2, TACC] 442. Misra, S. K., Z. Wu, F. Ostadhossein, M. Ye, K. Boateng, K. Schulten, E. Tajkhorshid, and D. Pan (2019), Pro-Nifuroxazide Self-Assembly Leads to Triggerable Nanomedicine for Anti-cancer Therapy, ACS Applied Materials & Interfaces, 11(20), 18074–18089, doi:10.1021/acsami.9b01343. (published) [Stampede2, TACC] 443. Muller, M. P., T. Jiang, C. Sun, M. Lihan, S. Pant, P. Mahinthichaichan, A. Trifan, and E. Tajkhorshid (2019), Characterization of Lipid–Protein Interactions and Lipid-Mediated Modulation of Membrane Protein Function through Molecular Simulation, Chemical Reviews, 119(9), 6086–6161, doi:10.1021/acs.chemrev.8b00608. (published) [Stampede2, TACC] 444. Padayatti, P. S., S. C. Lee, R. L. Stanfield, P.-C. Wen, E. Tajkhorshid, I. A. Wilson, and Q. Zhang (2019), Structural Insights into the Lipid A Transport Pathway in MsbA, Structure, 27(7), 1114–1123.e3, doi:10.1016/j.str.2019.04.007. (published) [Stampede2, TACC] 445. Pant, S., and E. Tajkhorshid (2019), Microscopic Characterization of GRP1 PH Domain Interaction with Anionic Membranes, Journal of Computational Chemistry, 41(6), 489–499, doi:10.1002/jcc.26109. (published) [Stampede2, TACC] 446. Facilitated by Its Strong Partitioning in the Membrane, The Journal of Physical Chemistry B, 124(10), 1866–1880, doi:10.1021/acs.jpcb.9b10092.Shahoei, R., and E. Tajkhorshid (2020), (published Menthol) [Stampede2, Binding to TACC the Human] α4β2 Nicotinic Acetylcholine Receptor

RY5 IPR12 Page 142 447. Singharoy, A. et al. (2019), Atoms to Phenotypes: Molecular Design Principles of Cellular Energy Metabolism, Cell, 179(5), 1098–1111.e23, doi:10.1016/j.cell.2019.10.021. (published) [Stampede2, TACC] 448. Smith, J. W., X. Jiang, H. An, A. M. Barclay, G. Licari, E. Tajkhorshid, E. G. Moore, C. M. Rienstra, J. S. Moore, and Q. Chen

Association, ACS Applied Nano Materials, 3(2), 937–945, doi:10.1021/acsanm.9b01331. (published) [Stampede2, TACC(2019),] Polymer−Peptide Conjugates Convert Amyloid into Protein Nanobundles through Fragmentation and Lateral 449. Sparks, R. P. et al. (2019), A small-molecule competitive inhibitor of phosphatidic acid binding by the AAA+ protein NSF/Sec18 blocks the SNARE-priming stage of vacuole fusion, Journal of Biological Chemistry, 294(46), 17168– 17185, doi:10.1074/jbc.ra119.008865. (published) [Stampede2, TACC] 450. Terekhova, K., S. Pokutta, Y. S. Kee, J. Li, E. Tajkhorshid, G. Fuller, A. R. Dunn, and W. I. Weis (2019), Binding partner- and force- -catenin conformation probed by native cysteine labeling, Scientific Reports, 9(1), doi:10.1038/s41598-019-51816-3. (published) [Stampede2, TACC] promoted changes in αE 451. Thangapandian, S., K. Kapoor, and E. Tajkhorshid (2020), Probing cholesterol binding and translocation in P- glycoprotein, Biochimica et Biophysica Acta (BBA) - Biomembranes, 1862(1), 183090, doi:10.1016/j.bbamem.2019.183090. (published) [Stampede2, TACC] 452. Yadav, A. K., C. J. Reinhardt, A. S. Arango, H. C. Huff, L. Dong, M. G. Malkowski, A. Das, E. Tajkhorshid, and J. Chan (2020), An Activity‐Based Sensing Approach for the Detection of Cyclooxygenase‐2 in Live Cells, Angewandte Chemie International Edition, 59(8), 3307–3314, doi:10.1002/anie.201914845. (published) [Stampede2, TACC] 453. Yu, K., T. Jiang, Y. Cui, E. Tajkhorshid, and H. C. Hartzell (2019), A network of phosphatidylinositol 4,5-bisphosphate binding sites regulates gating of the Ca2+- Academy of Sciences, 116(40), 19952–19962, doi:10.1073/pnas.1904012116. (published) [Stampede2, TACC] activated Cl− channel ANO1 (TMEM16A), Proceedings of the National 137. TG-MCA07S014 454. Puggioni, L., Kritsuk, A., Musacchio, S., Boffetta, G. 2020. Conformal invariance of weakly compressible two- dimensional turbulence. (published) [Comet, Data Oasis, SDSC]

138. TG-MCA93S001 455. Bores, C., and B. M. Pettitt (2020), Structure and the role of filling rate on model dsDNA packed in a phage capsid, Physical Review E, 101(1), doi:10.1103/physreve.101.012406. (published) [Bridges Regular, Comet, PSC, SDSC] 456. Lai, C.-L., C. Chen, S.-C. Ou, M. Prentiss, and B. M. Pettitt (2020), Interactions between identical DNA double helices, Physical Review E, 101(3), doi:10.1103/physreve.101.032414. (published) [Comet, Gordon, SDSC, Stampede, TACC] 457. Nassar, O. M., K. Wong, G. C. Lynch, T. J. Smith, and B. M. Pettitt (2019), Allosteric discrimination at the NADH/ADP regulatory site of glutamate dehydrogenase, Protein Science, 28(12), 2080–2088, doi:10.1002/pro.3748. (published) [Stampede, TACC] 458. Seckfort, D., G. C. Lynch, and B. M. Pettitt (2020), The lac repressor hinge helix in context: The effect of the DNA binding domain and symmetry, Biochimica et Biophysica Acta (BBA) - General Subjects, 1864(4), 129538, doi:10.1016/j.bbagen.2020.129538. (published) [Stampede, TACC]

139. TG-MCA94P030 459. Azizi, A., M. Dogan, J. D. Cain, R. Eskandari, X. Yu, E. C. Glazer, M. L. Cohen, and A. Zettl (2020), Frustration and Atomic Ordering in a Monolayer Semiconductor Alloy, Physical Review Letters, 124(9), doi:10.1103/physrevlett.124.096101. (published) [Stampede2, TACC] 460. Chen, Y. et al. (2020), Strong correlations and orbital texture in single-layer 1T-TaSe2, Nature Physics, 16(2), 218– 224, doi:10.1038/s41567-019-0744-9. (published) [Stampede2, TACC] 461. Da Jornada, F. H., L. Xian, A. Rubio, and S. G. Louie (2020), Universal slow plasmons and giant field enhancement in atomically thin quasi-two-dimensional metals, Nature Communications, 11(1), doi:10.1038/s41467-020-14826-8. (published) [Stampede2, TACC] 462. Meyer, S., Pham, T., Oh, S., Ercius, P., Kisielowski, C., et al. 2019. Metal-insulator transition in quasi-one-dimensional HfTe3 in the few-chain limit. DOI:10.1103/physrevb.100.041403. (published) [Stampede2, TACC] 463. Rizzo, D. J. et al. (2019), Length-Dependent Evolution of Type II Heterojunctions in Bottom-Up-Synthesized Graphene Nanoribbons, Nano Letters, 19(5), 3221–3228, doi:10.1021/acs.nanolett.9b00758. (published) [Stampede2, TACC]

RY5 IPR12 Page 143 464. Wu, M., Z. Li, T. Cao, and S. G. Louie (2019), Physical origin of giant excitonic and magneto-optical responses in two- dimensional ferromagnetic insulators, Nature Communications, 10(1), doi:10.1038/s41467-019-10325-7. (published) [Stampede2, TACC] 465. Yong, C.-K. et al. (2019), Valley-dependent exciton fine structure and Autler–Townes doublets from Berry phases in monolayer MoSe2, Nature Materials, 18(10), 1065–1070, doi:10.1038/s41563-019-0447-8. (published) [Stampede2, TACC]

140. TG-MCA98N017 466. Goel, H., W. Yu, V. D. Ustach, A. H. Aytenfisu, D. Sun, and A. D. MacKerell (2020), Impact of electronic polarizability on protein-functional group interactions, Physical Chemistry Chemical Physics, 22(13), 6848–6860, doi:10.1039/d0cp00088d. (published) [Comet, SDSC, Stampede, TACC] 467. Kognole, A. A., and A. D. MacKerell (2020), Mg2+ Impacts the Twister Ribozyme through Push-Pull Stabilization of Nonsequential Phosphate Pairs, Biophysical Journal, 118(6), 1424–1437, doi:10.1016/j.bpj.2020.01.021. (published) [Comet, SDSC, Stampede, TACC] 468. Kumar, A., O. Yoluk, and A. D. MacKerell (2019), FFParam: Standalone package for CHARMM additive and Drude polarizable force field parametrization of small molecules, Journal of Computational Chemistry, 41(9), 958–970, doi:10.1002/jcc.26138. (published) [Comet, SDSC, Stampede, TACC] 469. Lin, F., and A. D. MacKerell (2019), Improved Modeling of Cation‐ ‐Ring Interactions Using the Drude Polarizable Empirical Force Field for Proteins, Journal of Computational Chemistry, 41(5), 439–448, doi:10.1002/jcc.26067. (published) [Comet, SDSC, Stampede, TACCπ and] Anion 470. MacKerell, A. D., S. Jo, S. K. Lakkaraju, C. Lind, and W. Yu (2020), Identification and characterization of fragment binding sites for allosteric ligand design using the site identification by ligand competitive saturation hotspots approach (SILCS-Hotspots), Biochimica et Biophysica Acta (BBA) - General Subjects, 1864(4), 129519, doi:10.1016/j.bbagen.2020.129519. (published) [Comet, SDSC, Stampede, TACC] 471. Turchi, M., A. A. Kognole, A. Kumar, Q. Cai, G. Lian, and A. D. MacKerell (2020), Predicting Partition Coefficients of Neutral and Charged Solutes in the Mixed SLES–Fatty Acid Micellar System, The Journal of Physical Chemistry B, doi:10.1021/acs.jpcb.9b11199. (published) [Comet, SDSC, Stampede, TACC] 472. Ustach, V. D., S. K. Lakkaraju, S. Jo, W. Yu, W. Jiang, and A. D. MacKerell (2019), Optimization and Evaluation of Site- Identification by Ligand Competitive Saturation (SILCS) as a Tool for Target-Based Ligand Optimization, Journal of Chemical Information and Modeling, 59(6), 3018–3035, doi:10.1021/acs.jcim.9b00210. (published) [Comet, SDSC, Stampede, TACC] 473. Xu, X. et al. (2020), Structure of the cell-binding component of the Clostridium difficile binary toxin reveals a di- heptamer macromolecular assembly, Proceedings of the National Academy of Sciences, 117(2), 1049–1058, doi:10.1073/pnas.1919490117. (published) [Comet, SDSC, Stampede, TACC]

141. TG-MCB070073N 474. Bhaduri, A., J. Gardner, C. F. Abrams, and L. Graham-Brady (2019), Free energy calculation using space filled design and weighted reconstruction: a modified single sweep approach, Molecular Simulation, 46(3), 193–206, doi:10.1080/08927022.2019.1688325. (published) [Stampede2, TACC] 475. Gardner, J. M., and C. F. Abrams (2019), Energetics of Flap Opening in HIV-1 Protease: String Method Calculations, The Journal of Physical Chemistry B, 123(45), 9584–9591, doi:10.1021/acs.jpcb.9b08348. (published) [Stampede2, TACC] 476. Huang, M., and C. Abrams (2019), Effects of Reactivity Ratios on Network Topology and Thermomechanical Properties in Vinyl Ester/Styrene Thermosets: Molecular Dynamics Simulations, Macromolecular Theory and Simulations, 28(6), 1900030, doi:10.1002/mats.201900030. (published) [Stampede2, TACC] 477. Lu, M. et al. (2019), Associating HIV-1 envelope glycoprotein structures with states on the virus observed by smFRET, Nature, 568(7752), 415–419, doi:10.1038/s41586-019-1101-y. (published) [Stampede2, TACC] 478. Shrivastav, G., E. Vanden-Eijnden, and C. F. Abrams (2019), Mapping saddles and minima on free energy surfaces using multiple climbing strings, The Journal of Chemical Physics, 151(12), 124112, doi:10.1063/1.5120372. (published) [Stampede2, TACC] 479. Sridhar, A. S., and C. F. Abrams (2019), Yield and Post-yield Behavior of Fatty-Acid-Functionalized Amidoamine– Epoxy Systems: A Molecular Simulation Study, Journal of Dynamic Behavior of Materials, 5(2), 143–149, doi:10.1007/s40870-019-00193-z. (published) [Stampede2, TACC]

RY5 IPR12 Page 144 480. Srikanth, A., and C. F. Abrams (2019), Effect of molecular packing and hydrogen bonding on the properties of epoxy- amido amine systems, Computational Materials Science, 169, 109082, doi:10.1016/j.commatsci.2019.109082. (published) [Stampede2, TACC]

142. TG-MCB090003 481. Feig, M., and Y. Sugita (2019), Whole-Cell Models and Simulations in Molecular Detail, Annual Review of Cell and Developmental Biology, 35(1), 191–211, doi:10.1146/annurev-cellbio-100617-062542. (published) [Bridges GPU, Comet, Maverick, PSC, SDSC, Stampede, Stanford, TACC] 482. Heo, L., C. F. Arbour, and M. Feig (2019), Driven to near‐experimental accuracy by refinement via molecular dynamics simulations, Proteins: Structure, Function, and Bioinformatics, 87(12), 1263–1275, doi:10.1002/prot.25759. (published) [Bridges GPU, Comet, PSC, SDSC] 483. Heo, L., and M. Feig (2020), High‐accuracy protein structures by combining machine‐learning with physics‐based refinement, Proteins: Structure, Function, and Bioinformatics, 88(5), 637–642, doi:10.1002/prot.25847. (published) [Bridges GPU, Comet, PSC, SDSC] 484. Heo, L., and M. Feig (2020), Modeling of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Proteins by Machine Learning and Physics-Based Refinement, , doi:10.1101/2020.03.25.008904. (published) [Bridges GPU, Comet, PSC, SDSC] 485. Nawrocki, G., W. Im, Y. Sugita, and M. Feig (2019), Clustering and dynamics of crowded proteins near membranes and their influence on membrane bending, Proceedings of the National Academy of Sciences, 116(49), 24562–24567, doi:10.1073/pnas.1910771116. (published) [Bridges GPU, Comet, Maverick, PSC, SDSC, Stampede, Stanford, TACC] 486. Sugita, Y., and M. Feig (2019), Chapter 14. All-atom Molecular Dynamics Simulation of Proteins in Crowded Environments, In-cell NMR Spectroscopy, 228–248, doi:10.1039/9781788013079-00228. (published) [Bridges GPU, Comet, PSC, SDSC]

143. TG-MCB110014 487. Chen, H., J. Pan, D. M. Gandhi, C. Dockendorff, Q. Cui, B. Chanda, and K. A. Henzler-Wildman (2019), NMR Structural Analysis of Isolated Shaker Voltage-Sensing Domain in LPPG Micelles, Biophysical Journal, 117(2), 388–398, doi:10.1016/j.bpj.2019.06.020. (published) [Comet, SDSC, Stampede2, TACC] 488. Dalchand, N., Q. Cui, and F. M. Geiger (2019), Electrostatics, Hydrogen Bonding, and Molecular Structure at Polycation and Peptide:Lipid Membrane Interfaces, ACS Applied Materials & Interfaces, 12(19), 21149–21158, doi:10.1021/acsami.9b17431. (published) [Comet, SDSC, Stampede2, TACC] 489. Das, M., U. Dahal, O. Mesele, D. Liang, and Q. Cui (2019), Molecular Dynamics Simulation of Interaction between Functionalized Nanoparticles with Lipid Membranes: Analysis of Coarse-Grained Models, The Journal of Physical Chemistry B, 123(49), 10547–10561, doi:10.1021/acs.jpcb.9b08259. (published) [Comet, SDSC, Stampede2, TACC] 490. Leander, M., Y. Yuan, A. Meger, Q. Cui, and S. Raman (2020), Functional Plasticity and Evolutionary Adaptation of Allosteric Regulation, , doi:10.1101/2020.02.10.942417. (published) [Comet, SDSC, Stampede2, TACC] 491. Lu, X., J. Duchimaza-Heredia, and Q. Cui (2019), Analysis of Density Functional Tight Binding with Natural Bonding Orbitals, The Journal of Physical Chemistry A, 123(34), 7439–7453, doi:10.1021/acs.jpca.9b05072. (published) [Comet, SDSC, Stampede2, TACC] 492. Mandal, T., W. Lough, S. E. Spagnolie, A. Audhya, and Q. Cui (2020), Molecular Simulation of Mechanical Properties and Membrane Activities of the ESCRT-III Complexes, Biophysical Journal, 118(6), 1333–1343, doi:10.1016/j.bpj.2020.01.033. (published) [Comet, SDSC, Stampede2, TACC] 493. Roston, D., D. Demapan, and Q. Cui (2019), Extensive free-energy simulations identify water as the base in nucleotide addition by DNA polymerase, Proceedings of the National Academy of Sciences, 116(50), 25048–25056, doi:10.1073/pnas.1914613116. (published) [Comet, SDSC, Stampede2, TACC] 494. Son, C. Y., J. G. McDaniel, Q. Cui, and A. Yethiraj (2019), Proper Thermal Equilibration of Simulations with Drude Polarizable Models: Temperature-Grouped Dual-Nosé–Hoover Thermostat, The Journal of Physical Chemistry Letters, 10(23), 7523–7530, doi:10.1021/acs.jpclett.9b02983. (published) [Comet, SDSC, Stampede2, TACC] 495. Watanabe, H. C., and Q. Cui (2019), Quantitative Analysis of QM/MM Boundary Artifacts and Correction in Adaptive QM/MM Simulations, Journal of Chemical Theory and Computation, 15(7), 3917–3928, doi:10.1021/acs.jctc.9b00180. (published) [Comet, SDSC, Stampede2, TACC]

RY5 IPR12 Page 145 144. TG-MCB120097 496. Sayfutyarova, E. R., and S. Hammes-Schiffer (2019), Substituent Effects on Photochemistry of Anthracene–Phenol– Pyridine Triads Revealed by Multireference Calculations, Journal of the American Chemical Society, 142(1), 487–494, doi:10.1021/jacs.9b11425. (published) [Comet, SDSC]

145. TG-MCB130036 497. woltz, r., Yarov-Yarovoy, V., Sihn, C., Wang, W., Chiamvimonvat, N. 2020. Mechanisms of Autosomal Dominant Form of Progressive Hearing Loss, DFNA2: Insights Gained using Molecular Modeling. (in preparation) [Stampede, TACC]

146. TG-MCB130109 498. Suzuki, T. A., M. Phifer‐Rixey, K. L. Mack, M. J. Sheehan, D. Lin, K. Bi, and M. W. Nachman (2019), Host genetic determinants of the gut microbiota of wild mice, Molecular Ecology, doi:10.1111/mec.15139. (published) [Comet, Data Oasis, SDSC]

147. TG-MCB130112 499. De Lio, A. M., D. Paul, R. Jain, J. H. Morrissey, and T. V. Pogorelov (2020), Proteins and Ions Compete for Membrane Interaction: the case of Lactadherin, , doi:10.1101/2020.04.03.023838. (published) [Ranch, Stampede2, TACC]

148. TG-MCB150144, TG-MCB190063, TG-MCB200007 500. Minh, D. D. L. (2019), Alchemical Grid Dock (AlGDock): Binding Free Energy Calculations between Flexible Ligands and Rigid Receptors, Journal of Computational Chemistry, 41(7), 715–730, doi:10.1002/jcc.26036. (published) [OSG]

149. TG-MCB160059 501. Casalino, L., Jinek, M., Palermo, G. 2020. Ab-Initio Simulations Reveal the Catalysis of Non-Target DNA Cleavage in CRISPR-Cas9. (submitted) [Comet] 502. East, K. W., J. C. Newton, U. N. Morzan, Y. B. Narkhede, A. Acharya, E. Skeens, G. Jogl, V. S. Batista, G. Palermo, and G. P. Lisi (2019), Allosteric Motions of the CRISPR–Cas9 HNH Nuclease Probed by NMR and Molecular Dynamics, Journal of the American Chemical Society, 142(3), 1348–1358, doi:10.1021/jacs.9b10521. (published) [Comet, SDSC] 503. Harrison, R. E. S., N. T. Zewde, Y. B. Narkhede, R. V. Hsu, D. Morikis, V. I. Vullev, and G. Palermo (2020), Factor H- Inspired Design of Peptide Biomarkers of the Complement C3d Protein, ACS Medicinal Chemistry Letters, 11(5), 1054–1059, doi:10.1021/acsmedchemlett.9b00663. (published) [Comet, SDSC] 504. Harrison, R. E. S., N. T. Zewde, Y. B. Narkhede, R. V. Hsu, D. Morikis, V. I. Vullev, and G. Palermo (2020), Factor H- Inspired Design of Peptide Biomarkers of the Complement C3d Protein, ACS Medicinal Chemistry Letters, 11(5), 1054–1059, doi:10.1021/acsmedchemlett.9b00663. (published) [Comet, SDSC] 505. Mitchell, B. P., R. V. Hsu, M. A. Medrano, N. T. Zewde, Y. B. Narkhede, and G. Palermo (2020), Spontaneous Embedding of DNA Mismatches Within the RNA:DNA Hybrid of CRISPR-Cas9, Frontiers in Molecular Biosciences, 7, doi:10.3389/fmolb.2020.00039. (published) [Comet, SDSC] 506. Saha, A., Arantes, P., Hsu, R., Narkhede, Y., Jinek, M., et al. 2020. DNA-Induced Dynamic Switch Triggers Activation of CRISPR-Cas12a. (submitted) [Comet, SDSC]

150. TG-MCB160069 507. Nissley, D., Vu, Q., Trovato, F., Ahmed, N., Jiang, Y., et al. 2020. Electrostatic Interactions Govern Extreme Nascent Protein Ejection Times from Ribosomes and Can Delay Ribosome Recycling. (published)

151. TG-MCB160101, TG-MCB160124 508. Li, Y., C. Zhang, E. W. Bell, D. Yu, and Y. Zhang (2019), Ensembling multiple raw coevolutionary features with deep residual neural networks for contact‐map prediction in CASP13, Proteins: Structure, Function, and Bioinformatics, 87(12), 1082–1091, doi:10.1002/prot.25798. (published) [Comet, Data Oasis, SDSC] 509. Wang, Y., Q. Shi, P. Yang, C. Zhang, S. M. Mortuza, Z. Xue, K. Ning, and Y. Zhang (2019), Fueling ab initio folding with marine metagenomics enables structure and function predictions of new protein families, Genome Biology, 20(1), doi:10.1186/s13059-019-1823-z. (published) [Comet, Data Oasis, SDSC] 510. Wei, X., C. Zhang, P. L. Freddolino, and Y. Zhang (2020), Detecting Gene Ontology misannotations using taxon-specific rate ratio comparisons, edited by Z. Lu, Bioinformatics, doi:10.1093/bioinformatics/btaa548. (published) [Comet, Data Oasis, SDSC]

RY5 IPR12 Page 146 511. Zhang, C., L. Lane, G. S. Omenn, and Y. Zhang (2019), Blinded Testing of Function Annotation for uPE1 Proteins by I- TASSER/COFACTOR Pipeline Using the 2018–2019 Additions to neXtProt and the CAFA3 Challenge, Journal of Proteome Research, 18(12), 4154–4166, doi:10.1021/acs.jproteome.9b00537. (published) [Comet, Data Oasis, SDSC] 512. Zhang, C., W. Zheng, S. M. Mortuza, Y. Li, and Y. Zhang (2019), DeepMSA: constructing deep multiple sequence alignment to improve contact prediction and fold-recognition for distant-homology proteins, edited by A. Valencia, Bioinformatics, 36(7), 2105–2112, doi:10.1093/bioinformatics/btz863. (published) [Comet, Data Oasis, SDSC] 513. Zheng, W., Y. Li, C. Zhang, R. Pearce, S. M. Mortuza, and Y. Zhang (2019), Deep‐learning contact‐map guided protein structure prediction in CASP13, Proteins: Structure, Function, and Bioinformatics, 87(12), 1149–1164, doi:10.1002/prot.25792. (published) [Comet, Data Oasis, SDSC] 514. Zheng, W., Q. Wuyun, Y. Li, S. M. Mortuza, C. Zhang, R. Pearce, J. Ruan, and Y. Zhang (2019), Detecting distant- homology protein structures by aligning deep neural-network based contact maps, edited by C. M. Deane, PLOS Computational Biology, 15(10), e1007411, doi:10.1371/journal.pcbi.1007411. (published) [Comet, Data Oasis, SDSC]

152. TG-MCB160127 515. Michalak, D., Loesche, M., Hoogerheide, D. 2020. A generalized interaction potential for bilayers near oxidic surfaces in buffer provides nanoscale control of membrane association. (in preparation) [Bridges Regular, Globus Online, PSC, Pylon]

153. TG-MCB160183 516. Levintov, L., and H. Vashisth (2020), Ligand Recognition in Viral RNA Necessitates Rare Conformational Transitions, The Journal of Physical Chemistry Letters, 11(14), 5426–5432, doi:10.1021/acs.jpclett.0c01390. (published) [Comet, SDSC] 517. Tannir, S., L. Levintov, M. A. Townley, B. M. Leonard, J. Kubelka, H. Vashisth, K. Varga, and M. Balaz (2020), Functional Nanoassemblies with Mirror-Image Chiroptical Properties Templated by a Single Homochiral DNA Strand, Chemistry of Materials, 32(6), 2272–2281, doi:10.1021/acs.chemmater.9b04092. (published) [Comet, SDSC]

154. TG-MCB170012 518. Chen, M., C. Li, X. Fu, W. Wei, X. Fan, A. Hattori, Z. Chen, J. Liu, and W. Zhong (2020), Let It Catch: A Short‐Branched Protein for Efficiently Capturing Polysulfides in Lithium–Sulfur Batteries, Advanced Energy Materials, 10(9), 1903642, doi:10.1002/aenm.201903642. (published) [Comet, SDSC] 519. Deng, H., P. Dutta, and J. Liu (2018), Stochastic simulations of nanoparticle internalization through transferrin receptor dependent clathrin-mediated endocytosis, Biochimica et Biophysica Acta (BBA) - General Subjects, 1862(9), 2104–2111, doi:10.1016/j.bbagen.2018.06.018. (published) [Comet, SDSC] 520. Deng, H., P. Dutta, and J. Liu (2019), Entry modes of ellipsoidal nanoparticles on a membrane during clathrin- mediated endocytosis, Soft Matter, 15(25), 5128–5137, doi:10.1039/c9sm00751b. (published) [Comet, SDSC] 521. Deng, H., P. Dutta, and J. Liu (2019), Stochastic modeling of nanoparticle internalization and expulsion through receptor-mediated transcytosis, Nanoscale, 11(23), 11227–11235, doi:10.1039/c9nr02710f. (published) [Comet, SDSC] 522. Jewel, Y., Q. Van Dinh, J. Liu, and P. Dutta (2020), Substrate‐dependent transport mechanism in AcrB of multidrug resistant bacteria, Proteins: Structure, Function, and Bioinformatics, 88(7), 853–864, doi:10.1002/prot.25877. (published) [Comet, SDSC] 523. Li, C., X. Fu, W. Zhong, and J. Liu (2019), Dissipative Particle Dynamics Simulations of a Protein-Directed Self- Assembly of Nanoparticles, ACS Omega, 4(6), 10216–10224, doi:10.1021/acsomega.9b01078. (published) [Comet, SDSC]

155. TG-MCB170016 524. Fornace, M., Cheng, L., Miller, T., Pierce, N. 2020. Experimental design and centered Gaussian processes in reduced complexity. (in preparation) [Jetstream, TACC] 525. Fornace, M., Gonzalvo, M., Cheng, L., Winfree, E., Miller, T., et al. 2020. Nearest neighbor parameter sets for RNA and DNA derived via molecular dynamics. (in preparation) [Jetstream, TACC] 526. Fornace, M., Porubsky, N., Pierce, N. 2020. A Unified Dynamic Programming Framework for the Analysis of Interacting Nucleic Acid Strands: Enhanced Models, Scalability, and Speed. (accepted) [Jetstream, TACC] 527. Fornace, M., Porubsky, N., Pierce, N. 2020. NUPACK 4.0: Analysis and Design of Nucleic Acid Structures, Devices, and Systems. (in preparation) [Jetstream, TACC]

RY5 IPR12 Page 147 156. TG-MCB170053 528. Seo, S., Bacolla, A., Yoo, D., Koo, Y., Cho, S., et al. 2020. Replication‐Based Rearrangements Are a Common Mechanism for SNCA Duplication in Parkinson's Disease. DOI:10.1002/mds.27998. (published) [PSC]

157. TG-MCB170063 529. Karandur, D. et al. (2020), Breakage of the Oligomeric CaMKII Hub by the Regulatory Segment of the Kinase, , doi:10.1101/2020.04.15.043067. (published) [Comet, SDSC, Stampede, TACC]

158. TG-MCB170097 530. Vermaas, J. V., R. Kont, G. T. Beckham, M. F. Crowley, M. Gudmundsson, M. Sandgren, J. Ståhlberg, P. Väljamäe, and B. C. Knott (2019), The dissociation mechanism of processive cellulases, Proceedings of the National Academy of Sciences, 116(46), 23061–23067, doi:10.1073/pnas.1913398116. (published) 531. Yang, J., G. Bak, T. Burgin, W. J. Barnes, H. B. Mayes, M. J. Peña, B. R. Urbanowicz, and E. Nielsen (2020), Biochemical and Genetic Analysis Identify CSLD3 as a beta-1,4-Glucan Synthase That Functions during Plant Cell Wall Synthesis, The Plant Cell, 32(5), 1749–1767, doi:10.1105/tpc.19.00637. (published) [Comet, SDSC]

159. TG-MCB170146 532. Baral, S., M. Phillips, H. Yan, J. Avenso, L. Gundlach, B. Baumeier, and E. Lyman (2020), Ultrafast Formation of the Charge Transfer State of Prodan Reveals Unique Aspects of the Chromophore Environment, The Journal of Physical Chemistry B, 124(13), 2643–2651, doi:10.1021/acs.jpcb.0c00121. (published) 533. Pinkwart, K., F. Schneider, M. Lukoseviciute, T. Sauka-Spengler, E. Lyman, C. Eggeling, and E. Sezgin (2019), Nanoscale dynamics of cholesterol in the cell membrane, Journal of Biological Chemistry, 294(34), 12599–12609, doi:10.1074/jbc.ra119.009683. (published) [Stampede, TACC] 534. Yang, L., and E. Lyman (2019), Local Enrichment of Unsaturated Chains around the A2A Adenosine Receptor, Biochemistry, 58(39), 4096–4105, doi:10.1021/acs.biochem.9b00607. (published) [Stampede, TACC] 535. Zgorski, A., R. W. Pastor, and E. Lyman (2019), Surface Shear Viscosity and Interleaflet Friction from Nonequilibrium Simulations of Lipid Bilayers, Journal of Chemical Theory and Computation, 15(11), 6471–6481, doi:10.1021/acs.jctc.9b00683. (published)

160. TG-MCB180035 536. Solares, E., Tao, Y., Long, A., Gaut, B. 2020. HapSolo: An optimization approach for removing secondary haplotigs during diploid genome assembly. . (in preparation) [Bridges Regular, Comet, PSC, Pylon, SDSC]

161. TG-MCB180037 537. Lee, S., Y. Hu, S. K. Loo, Y. Tan, R. Bhargava, M. T. Lewis, and X.-S. Wang (2020), Landscape analysis of adjacent gene rearrangements reveals BCL2L14–ETV6 gene fusions in more aggressive triple-negative breast cancer, Proceedings of the National Academy of Sciences, 117(18), 9912–9921, doi:10.1073/pnas.1921333117. (published) [Bridges Large, Bridges Regular, PSC, Pylon]

162. TG-MCB180199, TG-MCB190044 538. Bafna, K., C. Narayanan, S. C. Chennubhotla, N. Doucet, and P. K. Agarwal (2019), Nucleotide substrate binding characterization in human pancreatic-type ribonucleases, edited by F. Fraternali, PLOS ONE, 14(8), e0220037, doi:10.1371/journal.pone.0220037. (published) [Comet, SDSC, Stampede2, TACC] 539. Hester, K. P., K. Bhattarai, H. Jiang, P. K. Agarwal, and C. Pope (2019), Engineering Dynamic Surface Peptide Networks on ButyrylcholinesteraseG117H for Enhanced Organophosphosphorus Anticholinesterase Catalysis, Chemical Research in Toxicology, 32(9), 1801–1810, doi:10.1021/acs.chemrestox.9b00146. (published) [Comet, SDSC, Stampede2, TACC] 540. Kumar, P., P. K. Agarwal, M. B. Waddell, T. Mittag, E. H. Serpersu, and M. J. Cuneo (2019), Low‐Barrier and Canonical Hydrogen Bonds Modulate Activity and Specificity of a Catalytic Triad, Angewandte Chemie, 131(45), 16406–16412, doi:10.1002/ange.201908535. (published) [Comet, SDSC, Stampede2, TACC] 541. Narayanan, C., D. N. Bernard, M. Létourneau, J. Gagnon, D. Gagné, K. Bafna, C. Calmettes, J.-F. Couture, P. K. Agarwal, and N. Doucet (2020), Insights into Structural and Dynamical Changes Experienced by Human RNase 6 upon Ligand Binding, Biochemistry, 59(6), 755–765, doi:10.1021/acs.biochem.9b00888. (published) [Comet, SDSC, Stampede2, TACC]

RY5 IPR12 Page 148 163. TG-MCB190007 542. Treece, B. W., F. Heinrich, A. Ramanathan, and M. Lösche (2020), Steering Molecular Dynamics Simulations of Membrane-Associated Proteins with Neutron Reflection Results, Journal of Chemical Theory and Computation, 16(5), 3408–3419, doi:10.1021/acs.jctc.0c00136. (published) [Bridges Regular, PSC]

164. TG-MCB190025, TG-MCB190186 543. Padilla-Sanchez, V. 2020. XSEDE-allocated Bridges supercomputer powers visualization of whole viruses at atomic level. Website publication. https://www.xsede.org/-/microscopic-and-huge. (published) [Bridges Large, ECSS, PSC, Pylon] 544. Padilla-Sanchez, V. 2020. XSEDE-Allocated Bridges Supercomputer Powers Visualization of Whole Viruses at Atomic Level. Website publication. https://www.hpcwire.com/off-the-wire/xsede-allocated-bridges-supercomputer- powers-visualization-of-whole-viruses-at-atomic-level/. (published) [Bridges Large, ECSS, PSC, Pylon]

165. TG-MCB190039 545. Nguyen, P., Rubin, S., Fainman, Y. 2019. Towards Direct Sequencing of DNA facilitated by Surface-Enhanced Raman Spectroscopy. Optics & Photonics Taiwan, International Conference (Taichung, Taiwan). (published)

166. TG-MCB190089 546. Chen, M. D., I. J. Fucci, K. Sinha, and G. S. Rule (2020), dGMP Binding to Thymidylate Kinase from Plasmodium falciparum Shows Half-Site Binding and Induces Protein Dynamics at the Dimer Interface, Biochemistry, 59(5), 694– 703, doi:10.1021/acs.biochem.9b00898. (published) [Bridges Regular, PSC]

167. TG-MCB190098 547. Singh, U., Li, J., Seetharam, A., Wurtele, E. 2020. pyrpipe: a python package for RNA-Seq workflows. (published) [Bridges Regular, PSC]

168. TG-MCB190186, TG-MCB200052 548. Padilla-Sanchez, V. 2020. Bacteriophage T4 Capsid Electrostatics. Figure Share. Figure Share. https://figshare.com/articles/Bacteriophage_T4_Capsid_Electrostatics/12117588/1. (published) [Bridges Large, Globus Online, PSC, Pylon, Ranch, Stampede2, TACC] 549. Padilla-Sanchez, V. (2020), In silico analysis of SARS-CoV-2 spike glycoprotein and insights into antibody binding, Research Ideas and Outcomes, 6, doi:10.3897/rio.6.e55281. (published) [Bridges Large, PSC, Pylon, Ranch, Stampede2, TACC]

169. TG-MCB200008 550. Becket, E. et al. (2020), Draft Genome Sequences of Bacillus glennii V44-8, Bacillus saganii V47-23a, Bacillus sp. Strain V59.32b, Bacillus sp. Strain MER_TA_151, and Paenibacillus sp. Strain MER_111, Isolated from Cleanrooms Where the Viking and Mars Exploration Rover Spacecraft Were Assembled, edited by C. Putonti, Microbiology Resource Announcements, 9(26), doi:10.1128/mra.00354-20. (published) [Jetstream, TACC, Training]

170. TG-MCB200061 551. Peters, M. H., O. Bastidas, D. S. Kokron, and C. Henze (2020), Static All-Atom Energetic Mappings of the SARS-Cov-2 Spike Protein with Potential Latch Identification of the Down State Protomer, , doi:10.1101/2020.05.12.091090. (published)

171. TG-MSS160016 552. DeVries, M., G. Subhash, and A. Awasthi (2020), Shocked ceramics melt: An atomistic analysis of thermodynamic behavior of boron carbide, Physical Review B, 101(14), doi:10.1103/physrevb.101.144107. (published) [Comet, SDSC]

172. TG-MSS170004 553. Ripp, M., Démery, V., Zhang, T., Paulsen, J. 2020. Geometry underlies the mechanical stiffening and softening of an indented floating film. (published) [Comet, SDSC]

RY5 IPR12 Page 149 173. TG-MSS190010 554. Johlas, H. M., L. A. Martínez-Tossas, M. A. Lackner, D. P. Schmidt, and M. J. Churchfield (2020), Large eddy simulations of offshore wind turbine wakes for two floating platform types, Journal of Physics: Conference Series, 1452, 012034, doi:10.1088/1742-6596/1452/1/012034. (published) [Comet, SDSC]

174. TG-MSS190015 555. Qian, Y., Cereceda, D. 2019. Using DFT Calculations to Design W-Re-X Alloys for Fusion Energy Applications. ASME IMECE 2019 (Salt Lake City). (published) 556. Qian, Y., Gilbert, M., Cereceda, D. 2020. Using first-principles calculations to predict the mechanical properties of transmuting tungsten under first wall fusion power-plant conditions. (in preparation)

175. TG-OCE150006 557. Daily, J., and M. J. Hoffman (2020), Modeling the three-dimensional transport and distribution of multiple microplastic polymer types in Lake Erie, Marine Pollution Bulletin, 154, 111024, doi:10.1016/j.marpolbul.2020.111024. (published) [Comet, SDSC] 558. Mason, S. A., J. Daily, G. Aleid, R. Ricotta, M. Smith, K. Donnelly, R. Knauff, W. Edwards, and M. J. Hoffman (2020), High levels of pelagic plastic pollution within the surface waters of Lakes Erie and Ontario, Journal of Great Lakes Research, 46(2), 277–288, doi:10.1016/j.jglr.2019.12.012. (published) [Comet, SDSC]

176. TG-OCE180011 559. Herdman, L., Manley, T., Mehler, P., Kernkamp, H. 2019. Infrastructure Impacts on Circulation in Lake Champlain. 3D Numerical modeling on how circulation dynamics within a large region of Lake Champlain have been modified by the emplacement of large rock-filled causeways. IAGLR Conference, Brockport, NY, June 2019. (published) [Data Oasis, SDSC]

177. TG-PHY080014N 560. lished) [Globus Online, Ranch, Stampede,Blake, T., Meinel, TACC] S., van Dyk, D. 2020. Bayesian analysis of b→sμ+μ− Wilson coefficients using the full angular distribution of Λb→Λ(→pπ−)μ+μ− decays. DOI:10.1103/physrevd.101.035023. (pub 561. Kane, C., Lehner, C., Meinel, S., Soni, A. 2019. Radiative leptonic decays on the lattice. (published) [Globus Online, Ranch, Stampede2, TACC] 562. Leskovec, L., Meinel, S., Pflaumer, M., Wagner, M. 2019. Lattice QCD investigation of a doubly-bottom b¯b¯ud tetraquark with quantum numbers I(JP)=0(1+). DOI:10.1103/physrevd.100.014503. (published) [Globus Online, Ranch, Stampede2, TACC]

178. TG-PHY150040 563. Aartsen, M., others, . 2019. Searches for neutrinos from cosmic-ray interactions in the Sun using seven years of IceCube data. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 564. Aartsen, M. G. et al. (2020), Search for PeV Gamma-Ray Emission from the Southern Hemisphere with 5 Yr of Data from the IceCube Observatory, The Astrophysical Journal, 891(1), 9, doi:10.3847/1538-4357/ab6d67. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 565. Aartsen, M., others, . 2019. Velocity Independent Constraints on Spin-Dependent DM-Nucleon Interactions from IceCube and PICO. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 566. Aartsen, M. G. et al. (2019), Search for Sources of Astrophysical Neutrinos Using Seven Years of IceCube Cascade Events, The Astrophysical Journal, 886(1), 12, doi:10.3847/1538-4357/ab4ae2. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 567. Aartsen, M. G. et al. (2020), A Search for MeV to TeV Neutrinos from Fast Radio Bursts with IceCube, The Astrophysical Journal, 890(2), 111, doi:10.3847/1538-4357/ab564b. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 568. Aartsen, M. G. et al. (2019), Cosmic ray spectrum and composition from PeV to EeV using 3 years of data from IceTop and IceCube, Physical Review D, 100(8), doi:10.1103/physrevd.100.082002. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 569. Aartsen, M., others, . 2019. A Search for Neutrino Point-Source Populations in 7 Years of IceCube Data with Neutrino- count Statistics. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] RY5 IPR12 Page 150 570. Aartsen, M., others, . 2019. Constraints on Neutrino Emission from Nearby Galaxies Using the 2MASS Redshift Survey and IceCube. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 571. Aartsen, M. G. et al. (2019), Efficient propagation of systematic uncertainties from calibration to analysis with the SnowStorm method in IceCube, Journal of Cosmology and Astroparticle Physics, 2019(10), 048–048, doi:10.1088/1475-7516/2019/10/048. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 572. Aartsen, M. G. et al. (2020), A Search for IceCube Events in the Direction of ANITA Neutrino Candidates, The Astrophysical Journal, 892(1), 53, doi:10.3847/1538-4357/ab791d. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 573. Aartsen, M. G. et al. (2020), Neutrinos below 100 TeV from the southern sky employing refined veto techniques to IceCube data, Astroparticle Physics, 116, 102392, doi:10.1016/j.astropartphys.2019.102392. (published) [Bridges GPU, Comet, GaTech, IU, LSU, NICS, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 574. Aartsen, M., others, . 2020. In-situ calibration of the single-photoelectron charge response of the IceCube photomultiplier tubes. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 575. Aartsen, M. G. et al. (2020), Design and performance of the first IceAct demonstrator at the South Pole, Journal of Instrumentation, 15(02), T02002–T02002, doi:10.1088/1748-0221/15/02/t02002. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 576. Aartsen, M. G. et al. (2020), Combined sensitivity to the neutrino mass ordering with JUNO, the IceCube Upgrade, and PINGU, Physical Review D, 101(3), doi:10.1103/physrevd.101.032006. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 577. Aartsen, M., others, . 2020. Characteristics of the diffuse astrophysical electron and tau neutrino flux with six years of IceCube high energy cascade data. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 578. Aartsen, M., others, . 2020. IceCube Search for High-Energy Neutrino Emission from TeV Pulsar Wind Nebulae. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 579. Aartsen, M. G. et al. (2020), Development of an analysis to probe the neutrino mass ordering with atmospheric neutrinos using three years of IceCube DeepCore data, The European Physical Journal C, 80(1), doi:10.1140/epjc/s10052-019-7555-0. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 580. Aartsen, M. G. et al. (2020), Time-Integrated Neutrino Source Searches with 10 Years of IceCube Data, Physical Review Letters, 124(5), doi:10.1103/physrevlett.124.051103. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 581. Albert, A., others, . 2020. ANTARES and IceCube Combined Search for Neutrino Point-like and Extended Sources in the Southern Sky. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 582. Albert, A., others, . 2020. Combined search for neutrinos from dark matter self-annihilation in the Galactic Centre with ANTARES and IceCube. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 583. Garrappa, S. et al. (2019), Investigation of Two Fermi-LAT Gamma-Ray Blazars Coincident with High-energy Neutrinos Detected by IceCube, The Astrophysical Journal, 880(2), 103, doi:10.3847/1538-4357/ab2ada. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream] 584. Kankare, E., others, . 2019. Search for transient optical counterparts to high-energy IceCube neutrinos with Pan- STARRS1. DOI:10.1051/0004-6361/201935171. (published) [Bridges GPU, Comet, OSG, PSC, SDSC, Stampede2, Stanford, TACC, XStream]

179. TG-PHY160007 585. Bernard, T., Stoltzfus-Dueck, T., Gentle, K., Hakim, A., Hammett, G., et al. 2020. Investigating shear flow through continuum gyrokinetic simulations of limiter biasing in the Texas Helimak. (submitted) 586. Juno, J., Howes, G., TenBarge, J. 2020. A Field-Particle Correlation Analysis of a Perpendicular Magnetized Collisionless Shock: II. Vlasov Simulations. (in preparation) 587. Juno, J., Swisdak, M., TenBarge, J., Skoutnev, V., Hakim, A., et al. 2020. Noise-Induced Magnetic Field Saturation in Kinetic Simulations. (submitted) 588. TenBarge, J., Juno, J., Howes, G. 2020. Electron Energization in Zero Guide Field Magnetic Reconnection. (in preparation)

RY5 IPR12 Page 151 180. TG-PHY160053 589. Rosofsky, S. G., and E. A. Huerta (2020), Artificial neural network subgrid models of 2D compressible magnetohydrodynamic turbulence, Physical Review D, 101(8), doi:10.1103/physrevd.101.084024. (published) [Bridges GPU, Stampede2]

181. TG-PHY170023 590. - RuCl3, 2D Materials, 7(3), 035004, doi:10.1088/2053-1583/ab7e0e. (published) [Comet] Dai, Z. et al. (2020), Crystal structure reconstruction in the surface monolayer of the quantum spin liquid candidate α 591. Yu, J.-X., D.-T. Chen, J. Gu, J. Chen, J. Jiang, L. Zhang, Y. Yu, X.-G. Zhang, V. S. Zapf, and H.-P. Cheng (2020), Three Jahn- Teller States of Matter in Spin-Crossover System Mn(taa), Physical Review Letters, 124(22), doi:10.1103/physrevlett.124.227201. (published) [Comet, SDSC]

182. TG-PHY170049 592. Lu, Y., Huang, C., Kilian, P., Guo, F., Li, H., et al. 2019. An O(N) Maxwell solver with improved numerical dispersion properties. (submitted) [Stampede2, TACC] 593. Lu, Y., P. Kilian, F. Guo, H. Li, and E. Liang (2020), Time-step dependent force interpolation scheme for suppressing numerical Cherenkov instability in relativistic particle-in-cell simulations, Journal of Computational Physics, 413, 109388, doi:10.1016/j.jcp.2020.109388. (published) [Stampede2, TACC] 594. Lu, Y., H. Li, K. A. Flippo, K. Kelso, A. Liao, S. Li, and E. Liang (2019), MPRAD: A Monte Carlo and ray-tracing code for the proton radiography in high-energy-density plasma experiments, Review of Scientific Instruments, 90(12), 123503, doi:10.1063/1.5123392. (published) [Stampede2, TACC] 595. Lu, Y. et al. (2020), Modeling hydrodynamics, magnetic fields, and synthetic radiographs for high-energy-density plasma flows in shock-shear targets, Physics of Plasmas, 27(1), 012303, doi:10.1063/1.5126149. (published) [Stampede2, TACC]

183. TG-PHY180010 596. Hamed Moosavian, A., Garrison, J., Jordan, S. 2019. Site-by-site quantum state preparation algorithm for preparing vacua of fermionic lattice field theories. (published) [Bridges Regular, PSC] 597. Titum, P., J. T. Iosue, J. R. Garrison, A. V. Gorshkov, and Z.-X. Gong (2019), Probing Ground-State Phase Transitions through Quench Dynamics, Physical Review Letters, 123(11), doi:10.1103/physrevlett.123.115701. (published) [Bridges Regular, PSC]

184. TG-PHY180014 598. Dev, P. 2020. Fingerprinting quantum emitters in hexagonal boron nitride using strain. (submitted) 599. Kumar, P., Manchanda, P., Dev, P. 2020. A new class of intrinsic magnet: two-dimensional yttrium sulphur selenide. We are preparing the manuscript to be submitted to Nature Materials.. (in preparation) 600. Manchanda, P., Kumar, P., Dev, P. 2020. Platinum vacancy-induced magnetism in PtSe2: an engineered 4p magnet. This is being written and will be submitted to PRL. (in preparation) 601. Manchnda, P., Kumar, P., Dev, P. 2020. Thickness-dependence of hydrogen-induced phase transition in MoTe2. (accepted) 602. McBean, C., Manchanda, P., Dev, P. 2020. Revisiting Wettability of Graphene. We are preparing the manuscript, to be submitted in PRR. (in preparation) 603. Naumov, I., Dev, P. 2020. An Interplay of Rashba and Exchange Effects in Bismuth nanoribbons. Manuscript under preparation. (in preparation) 604. Naumov, I., Dev, P. 2020. Quantum materials interfaces: graphene/Bismuth (111) heterostructures. (submitted)

185. TG-PHY180023 605. Schneider, B., Bartschat, K., Zatsarinny, O., Bray, I., Scrinzi, A., et al. 2020. A Science Gateway for Atomic and Molecular Physics . PEARC 19 (Chicago, Ill). https://arxiv.org/abs/2001.02286. (published) 606. Schneider, B., Scrinzi, A., Zatsarinny, O., Hamilton, K., Bray, I., et al. 2021. Atomic and Molecular Scattering Applications in an ApacheAiravata Science Gateway. PEARC 20 (Portland, OR). https://pearc.acm.org/pearc20/program/. (accepted)

RY5 IPR12 Page 152 186. TG-PHY180026 607. Brown, M., K. Gelber, and M. Mebratu (2020), Taylor State Merging at SSX: Experiment and Simulation, Plasma, 3(1), 27–37, doi:10.3390/plasma3010004. (published)

187. TG-PHY190014 608. Heinonen, R., Diamond, P. 2020. Turbulence model reduction by deep learning. (submitted) [Comet, SDSC]

188. TG-SES120016 609. Bond, R. M., C. J. Fariss, J. J. Jones, A. D. I. Kramer, C. Marlow, J. E. Settle, and J. H. Fowler (2012), A 61-million-person experiment in social influence and political mobilization, Nature, 489(7415), 295–298, doi:10.1038/nature11421. (published) 610. Christakis, N. A., and J. H. Fowler (2014), Friendship and natural selection, Proceedings of the National Academy of Sciences, 111(Supplement_3), 10796–10801, doi:10.1073/pnas.1400825111. (published) [SDSC]

189. TG-SES140024 611. Contractor, N., Antone, B., Lungeanu, A., Bell, S., DeChurch, L. 2019. Validating Models of Team Composition Decisions for Long-duration Space Exploration. 2019 NASA Human Research Program Investigators’ Workshop Annual Conference (Galveston, TX). (accepted) 612. Lungeanu, A., DeChurch, L., Contractor, N. 2019. Leading Teams Over Time Through Space: Computational Experiments on Leadership Networks. . Academy of Management (AoM (Boston, MA). (accepted) 613. Lungeanu, A., DeChurch, L., Contractor, N. 2019. The Emergence and Development of Crew Shared Mental Models: An Agent-based Model.. 2019 NASA Human Research Program Investigators’ Workshop Annual Conference (Galveston, TX). (accepted) 614. Lungeanu, A., DeChurch, L., Contractor, N. 2020. Leading Teams Over Time Through Space: Computational Experiments on Leadership Networks. (submitted) 615. Lungeanu, A., Park, P., DeChurch, L., Contractor, N. 2019. Leveraging Simulations to Improve the Functioning of Multiteam Systems. European Association of Work and Organizational Psychology (Turin, Italy). (accepted) 616. Mesmer-Magnus, J., Lungeanu, A., Harris, A., Niller, A., DeChurch, L., et al. 2019. Working in Space: Managing Transitions between Tasks. NASA: Psychology and Human Performance in Space Programs: NASA. (submitted)

190. TG-SES180019 617. Lee, K., and M.-P. Kwan (2019), The Effects of GPS-Based Buffer Size on the Association between Travel Modes and Environmental Contexts, ISPRS International Journal of Geo-Information, 8(11), 514, doi:10.3390/ijgi8110514. (published) [Jetstream, TACC]

RY5 IPR12 Page 153 13. Collaborations XSEDE considers collaboration within the research community to be essential to the advancement of all fields of research. It is XSEDE’s policy and practice to encourage and engage in collaborations, where applicable. In addition to the collaborations with Service Providers via the SP Forum (§14), XSEDE collaborates with NSF awardees and other domestic and international projects. The following list represents the active collaborations with NSF awardees:

PI/Contact NSF Proposal Title Award Number David Anderson Mainstreaming Volunteer Computing 1105572 Ian Foster SI2-SSI: SciDaaS -- Scientific data management as a 1148484 service for small/medium labs

Abani Patra Collaborative Research: Integrated HPC Systems Usage 1203560 and Performance of Resources Monitoring and Modeling (SUPReMM- SUNY Buffalo)

Abani Patra Collaborative Research: Integrated HPC Systems Usage 1203604 and Performance of Resources Monitoring and Modeling (SUPReMM- UT-Austin)

Von Welch Center for Trustworthy Scientific Cyberinfrastructure 1234408 (CTSC) Kevin Franklin Latin America-US Institute 2013: Methods in 1242216 Computational Discovery for Multidimensional Problem Solving Nicholas Berente EAGER proposal: Toward a Distributed Knowledge 1348461 Environment for Research into Cyberinfrastructure: Data, Tools, Measures, and Models for Multidimensional Innovation Network Analysis

Jerzy Bernholc Multiscale Software for Quantum Simulations in 1339844 Materials Design, Nano Science and Technology

Seung-Jong Park MRI: Acquisition of SuperMIC-- A Heterogeneous 1338051 Computing Environment to Enable Transformation of Computational Research and Education in the State of Louisiana

Marlon Pierce Open Gateway Computing Environments Science 1339774 Gateways Platform as a Service (OGCE SciGaP)

Steven Tuecke Sustaining Globus Toolkit for the NSF Community 1339873 (Sustain-GT)

Kathy L. Benninger CC-NIE Integration: Developing Applications with 1341005 Networking Capabilities via End-to-End SDN (DANCES)

Renata Wentacovitch Quantum Mechanical Modeling of Major Mantle 0635990 Materials Dr. Kate Keahey A Large-Scale, Community-Driven Experimental 1419141 Environment for Cloud Research

Shaowen Wang MRI: Acquisition of a National CyberGIS Facility for 1429699 Computing- and Data-Intensive Geospatial Research and Education

RY5 IPR12 Page 154 Todd Martinez Acquisition of an Extreme GPU cluster for 1429830 Interdisciplinary Research

Donna Cox The Centrality of Advanced Digitally-ENabled Science: 1445176 CADENS

Robert Ricci CloudLab: Flexible Scientific Infrastructure to Support Fundamental Advances in Cloud Architectures and Applications

Dr. Kerk F. Kee, Almadena RUI: CAREER Organizational Capacity and Capacity 1453864 Y. Chtchelkanova Building for Cyberinfrastructure Diffusion

Nicholas Berente Fostering Successful Innovative Large-Scale, Distributed 1551609 Science and Engineering Projects through Integrated Collaboration

Allen Pope EarthCube RCN: Collaborative Research: Research 1541620 Coordination Network for High Performance Distributed Computing in the Polar Sciences

Thomas Hauser MRI Collaborative Consortium: Acquisition of a Shared 1532236 Supercomputer by the Rocky Mountain Advanced Computing Consortium

Edward Seidel BD Hubs: Midwest: “SEEDCorn: Sustainable Enabling 1550320 Environment for Data Collaboration that you are proposing in response to the NSF Big Data Regional Innovation Hubs (BD Hubs): Accelerating the Big Data Innovation Ecosystem (NSF 15-562) solicitation

Alexander Withers Secure Data Architecture: Shared Intelligence Platform 1547249 for Protecting our National Cyberinfrastructure” that you are proposing in response to the NSF Cybersecurity Innovation for Cyberinfrastructure (NSF 15-549) solicitation James Basney CILogon 2.0 project that you are proposing in response 1547268 to the NSF Cybersecurity Innovation for Cyberinfrastructure (NSF 15-549) solicitation

Bertram Ludaescher DIBBs: Merging Science and Cyberinfrastructure 1541450 Pathways: The Whole Tale

Philip J. Puxley Associated Universities, Inc. (AUI) and the National 1519126 Radio Astronomy Observatory (NRAO)

J. Bernholc SI2-SSE: Multiscale Software for Quantum Simulations of 1615114 Nanostructured Materials and Devices

David Anderson Collaborative Research: SI2-SSI: Adding Volunteer 1550601 Computing to the Research Cyberinfrastructure

Thomas Crawford Molecular Sciences Software Institute (MolSSI) that you 1547580 are proposing in response to the NSF Scientific Software Innovation Institutes (S2I2, NSF 15-553) solicitation

Nancy Wilkins-Diehr Science Gateways Software Institute for NSF Scientific 1547611 Software Innovation Institutes (S2I2, NSF 15-553) solicitation

RY5 IPR12 Page 155 Ron Hawkins CC* Compute: BioBurst in response to the Campus 1659104 Cyberinfrastructure (CC*) Program solicitation (NSF 16- 567) Farzad Mashayek CC* Networking Infrastructure: Building HPRNet (High- 1659255 Performance Research Network) for advancement of data intensive research and collaboration

Ewa Deelman SI2-SSI: Pegasus: Automating compute and data 1664162 intensive science

Doug Jennewein MRI: Acquisition of the Lawrence Supercomputer to 1626516 Advance Multidisciplinary Research in South Dakota

Dirk Colbry Cybertraining:CIP – Professional Training for 1730137 CyberAmbassadors

Eric Shook Collaborative Research: CyberTraining: CIU: Hour of 7/26/6909 Cyberinfrastructure: Developing Cyber Literacy for Geographic Information Science

Jennifer M. Schopf CC* NPEO: A Sustainable Center for Engagement and 1826994 Networks

Alex Szalay Collaborative Research: Building the Community for the 1747493 Open Storage Network

Von Welch CICI: CSRC: Research Security Operations Center 1840034 (ResearchSOC)

Larry Smarr CC* NPEO: Towards the National Research Platform 1826967

Thomas Doak Collaborative Research: ABI Sustaining: The National 1759906 Center for Genome Analysis Support

Peter Kasson SI2-SSI Collaborative Research: SCALE-MS-Scalable 1835780 Adaptive Large Ensembles of Molecular Simulations

Mao Ye A Workshop to Jumpstart High-Performance Computing 1838183 in Finance/ BIGDATA: IA: Collaborative Research: Understanding the Financial Market Ecosystem Dmitry Pekurovsky Elements: Software: Multidimensional Fast Fourier 1835885 Transforms on the Path to Exascale

Von Welch CICI: CCoE: Trusted CI: Advancing Trustworthy Science 1920430

Rudolph Eigenmann MRI: Acquisition of a Big Data and High Performance 1919839 Computing System to Catalyze Delaware Research and Education Timoty Menzies Can Empirical Software Engineering be Adapted to Computational Science 1908762 William Gropp Category I: Crossing the Divide Between Today's 2005572 Practice and Tomorrow's Science X. Carol Song Category I: Anvil - A national advanced computational 2005632 resource to meet the changing needs of the nation’s research and education communities

RY5 IPR12 Page 156 David Y. Hancock Category I – Jetstream 2: On-demand high performance 2005506 computing Honggao Liu MRI: Acquisition of FASTER - Fostering Accelerated 2019129 Sciences Transformation Education and Research

The following list represents other active formal domestic and international collaborations:

Project Collaboration Summary

Domestic Collaborations

Marshall University - Indiana University CRI staff visited Marshall University for server build Campus Bridging Site (CRI) and XSEDE software toolkit installation

Southern Illinois University - Indiana University CRI staff visited Southern Illinois University for Campus Bridging Site (CRI) server build and XSEDE software toolkit installation

Bentley University - Indiana University CRI staff visited Bentley University for server build Campus Bridging Site (CRI) and XSEDE software toolkit installation

University Texas El Paso - Indiana University CRI staff visited University Texas El Paso for server Campus Bridging Site (CRI) build and XSEDE software toolkit installation

Brandeis University - Indiana University CRI staff visited Brandeis University for server build Campus Bridging Site (CRI) and XSEDE software toolkit installation

Incorporated Research Indiana University partnering with IRIS to use the Jetstream system to Institutions for Seismology (IRIS) disseminate data to the research community

Indiana University partnered with Cyverse projects to deploy and CyVerse operate the Jetstream system as part of the XSEDE ecosystem

Indiana University partnered with the National Center for Genome National Center for Genome Analysis Analysis Support to use the Jetstream system for creation of virtual Support (NCGAS) machine images for research and analysis of genome data

South Dakota State University - Indiana University CRI staff visited SDSU for cluster build and XSEDE Campus Bridging Site software toolkit installation

Slippery Rock University - Campus Indiana University CRI staff visited Slippery Rock University for server Bridging Site build and XSEDE software toolkit installation

Doane University - Campus Bridging Indiana University CRI staff visited Doane University for cluster build Site and XSEDE software toolkit installation

UltraScan Science Gateway - Virtual Cluster CRI staff worked with Ultrascan to create a virtual cluster in Jetstream

RY5 IPR12 Page 157 3-D Quantitative Phenotyping CRI staff worked with A. Murat Maga in order to create a virtual cluster Gateway - Virtual Cluster in Jetstream for the biological sciences

RACD is helping XSEDE federated services providers (XDMoD, CI-Tutor, and Cornell Virtual Workshop) to integrate with the XSEDE Web SSO XSEDE Web SSO capability.

Use of XRAS by NCAR/CISL Agreement to use XRAS for managing the NCAR allocations

Agreement to use XRAS to manage EOL Lower Atmosphere Observing Use of XRAS by NCAR/EOL Facilities (LAOF) allocations

Use of XRAS by Blue Waters Agreement to use XRAS for managing the Blue Waters allocations

Agreement to use Information Services to enable resource discovery in Use of Information Services by UIUC the UIUC Research Portal (https://researchit.illinois.edu)

XSEDE Providing Capability Integration Assistance to make Kepler Delivering Kepler to XSEDE Users available to XSEDE users

XCI is participating in OSN software working group to facilitate Open Storage Network CI integration documenting requirements and use cases, support the engineering collaboration process, and provide other useful XSEDE services

Agreement to use network science methods and administrative data maintained by the Institute for Research on Innovation & Science to Institute for Research on Innovation develop preliminary models that examine the scientific effects of & Science (IRIS) XSEDE usage by researchers on more than 30 U.S. university campuses.

International Collaborations

International HPC Summer School Partnership with PRACE (EU), RIKEN AICS (Japan), and Compute Canada (Canada) to familiarize the best students of the respective continents or countries in computational sciences with a strong bond to supercomputing with all major state-of-the-art aspects related to HPC for a broad range of scientific disciplines, catalyze the formation of networks, provide mentoring through faculty members and supercomputing experts from renowned HPC centers, and to facilitate international exchange and open further career options.

Open Grid Forum (www.ogf.org) JP Navarro is GLUE working group chair. XSEDE leverages and influences infrastructure information management through this collaboration.

HPC Development and Summer Craig Stewart entered into a formal MOU of Collaboration with TU- Exchange Program for HPC visiting Dresden staff

Membership in the Research Data Indiana University entered into a formal membership agreement with Alliance (RDA) organization RDA to explore open data standards as part of the international scope of the organization

RY5 IPR12 Page 158 International grid computing research and education organization Indiana University participates as a contributing partner to summer teaching in the African Grid School

Distributed Organization for International grid computing research and education organization; Scientific and Academic Research Indiana University participates as a contributing partner to summer (DOSAR) teaching in the African Grid School

International collaboration of XSEDE entered into an MOU jointly with PRACE and RIKEN regional research infrastructures committing to opening lines of communications and seeking more (XSEDE, PRACE, and RIKEN) areas of collaboration. This occurred in May 2017.

Accounting Service Exploring possible collaboration on re-design of Federated Accounting Service

WISE (Wise Information Security for Support trusted global framework where security experts can share Collaborating e-Infrastructures) information on topics such as risk management, experiences about certification processes and threat intelligence

IGTF/TAGPMA To establish common policies and guidelines that help establish interoperable, global trust relations between providers of e- Infrastructures and cyber-infrastructures, identity providers, and other qualified relying parties.

International Identity Federation XCI-302: Participate in REFEDS Assurance Framework Pilot

Engagement Group for XCI-256: Participate in AARC Engagement Group for Infrastructures Infrastructures (AEGIS) (AEGIS)

Technische Universitat Darmstadt: Understanding ROI on university-owned advanced cyberinfrastructure ROI on academic advanced systems, including cloud systems with Technische Universitat cyberinfrastructure systems Darmstadt

ROI on academic advanced Understanding ROI on university-owned advanced cyberinfrastructure cyberinfrastructure systems systems, including cloud systems with RTWH Aachen University

Understanding ROI on university-owned advanced cyberinfrastructure ROI on academic advanced systems, including cloud systems with Technische Universitat Darmstadt, cyberinfrastructure systems RTWH Aachen University, & Technishe Universitat Dresden

RY5 IPR12 Page 159 14. Service Provider Forum Report Service Providers (SPs) are independently funded projects and/or organizations that provide cyberinfrastructure (CI) services to the science and engineering community. There is a rich diversity of SPs in the US academic community, spanning centers that are funded by NSF to operate large-scale resources for the national research community to universities that provide resources and services to their campus researchers. The Service Provider Forum (SPF) is intended to facilitate this ecosystem of Service Providers, thereby advancing the science and engineering researchers that rely on these cyberinfrastructure services. The SPF has two primary elements of its charter: An open forum for discussion of topics of interest to the SP community A formal communication channel between the SPF members and the XSEDE project The SPF conducts its business primarily through conference calls scheduled on a biweekly cadence on Thursdays at 4PM Eastern Time. Agendas are distributed in advance of the meetings and minutes are maintained on the XSEDE SP wiki (https://confluence.xsede.org/display/XT/XSEDE+Federation). NSF Program Officers are invited and occasionally participate. Many people from the XSEDE program routinely participate in SPF meetings to facilitate direct interaction with the XSEDE program. For example, regular updates are provided by John Towns (XSEDE PI) and Tim Boerner (XSEDE Deputy Project Director), Victor Hazlewood (XSEDE SP Coordinator), and other XSEDE management and area leads. Additional contributors from XSEDE and other organizations are frequently invited to brief the SPF on XSEDE topics or seek the Forum’s input in the development of program plans and activities. This report is the quarterly summary of the SPF’s activities covering the period of May 1, 2020 – July 31, 2020. SP Forum administrative and membership activities during this reporting period • Mostly biweekly SPF meetings – attendance typically ranges from 18 to more than 25 participants • SP Forum participation at the June NSF virtual annual review of XSEDE (Ruth Marinshaw, Dave Hancock, Jon Anderson) • SP Forum participation at the biweekly XSEDE Senior Management Team calls and June virtual Quarterly Staff Meeting (Ruth Marinshaw) • Participation in PEARC20 (all) The full membership of the SP Forum is maintained on the XSEDE website, https://www.xsede.org/ecosystem/service-providers. Technical and programmatic discussions Noteworthy SPF activities from this reporting period include: • Regular attendance and updates by John Towns and Tim Boerner on the XSEDE project. • Regular attendance and updates by Victor Hazlewood XSEDE Engineering and Operations topics. • SP input regarding XSEDE PY10 program plans • XSEDE annual review, quarterly meeting, and XAB updates • Importance of cybersecurity vigilance especially around COVID-19 related research • Transition of the University of Delaware from a Level 3 SP to Level 2 • Key presentations and discussion topics: o Barr von Oehsen (Rutgers University) – Overview of cyberinfrastructure activities and services at Rutgers o Presentations on and discussions around five newly awarded Track1 and Track2 systems: . Delta @NCSA (Tim Boerner) . Anvil @ Purdue University (Carol Song) RY5 IPR12 Page 160 . Voyager @ SDSC (Amit Majumdar) . Jetstream2 @ Indiana University (Dave Hancock) . NeoCortex @ PSC (Paola Buitrago)

RY5 IPR12 Page 161 15. UAC XSEDE User Advisory Committee Report Reporting period: May 1, 2020 through July 31, 2020 The XSEDE User Advisory Committee (UAC) represents the "user's voice" to XSEDE management, presenting recommendations regarding emerging needs and services and act as a sounding board for plans and suggested developments. It meets two or three times a year via conference call. The UAC makes suggestions to improve XSEDE operations and helps identify areas where more support expertise is required. It reviews findings of XSEDE's Performance Evaluation Plan and Self Assessment, and its User Survey. Subsets of the UAC advise XSEDE's Extended Collaborative Support Service (ECSS) management on where to dedicate support effort on community codes. One member of the UAC serves on the User Requirements Evaluation and Prioritization Working Group (UREP). The chair of the UAC (elected by its members) participates in regular XSEDE senior management meetings (currently bi- weekly). There was one regular meeting of the UAC during this reporting period which took place on 16 July 2020. Six UAC members were present, including Richard Braatz, Dhruva Chakravoty, Tom Cheatham, Mark Miller, Deidre Shoemaker and Chair Emre Brookes. XSEDE staff present included Sergiu Sanielevici and John Towns. Towns presented a general XSEDE update to the committee members. This presentation included discussion on the results of the panel review. Highlighted were the proposed longitudinal studies, XSEDE’s response to COVID-19, and their enabling support of the COVID-19 HPC Consortium. This led to a discussion of how the lessons learned from XSEDE’s COVID-19 response could be leveraged to develop an “emergency preparedness plan”. The committee was interested in the impact of COVID-19 research on the support of existing users, which to date appears to be minimal. Of particular concern to committee members was the continuity of XSEDE services after the end of the XSEDE award. Towns indicated that a one year extension was possible, which would allow the extension of XSEDE services until August 2022, but that we had no assurance of continuity past the end of XSEDE. Concern was also addressed by the committee on the future of the Campus Champions program after XSEDE. Towns and Sanielevici provided highlights on upcoming NSF funded resources. This final point initiated committee discussion on developing user training for non-traditional resources. No other actions were made by or requested of the UAC over this period.

RY5 IPR12 Page 162 16. XMS Summary Executive Summary During the current reporting period substantial progress was achieved in a number of areas. The integration of XDMoD with Open OnDemand continues to move forward with Open OnDemand users now able to directly link to XDMoD’s Job Viewer tab to obtain detailed analysis of a given jobs performance. XMS has made progress improving the cybersecurity of Open XDMoD through a collaboration with Trusted CI. Recognizing the growing importance of Gateways in scientific research, the XMS team is in the process of transforming the present limited set of custom gateway queries into a full gateway XDMoD realm. A power/energy monitoring capability has been developed for XDMoD and was recently presented at PEARC20. An Open XDMoD instance monitoring TACC Frontera is now in production, with access to both TACC and NSF staff. XDMoD and Open XDMoD usage, deliverables, key performance indicators and XMS publications have been updated.

RY5 IPR12 Page 163 17. XAB Executive Summaries

Executive Summary of XSEDE Advisory Board Meeting, Tuesday, April 21, 2020 Session I: Approval of February Meeting Minutes • Approved Session II: PI Update • Invited by NSF to submit a supplement to extend the project for 12 months. Will be submitting a supplemental request for NSF review & approval in conjunction with the Annual report & program plan. Working on the supplement proposal now. Expectation/hope for ongoing operations for an additional year. Planning in an optimistic way that it will be approved, and will operate through August 2022 once approved. • NSF still needs to issue a solicitation for a new award. Assume NSF blueprint doc is a draft of such solicitation. XSEDE submitted a response to the blueprint doc with many comments pointing out strengths & weaknesses. NSF appreciated the project’s careful review of that doc. • Q: Any substantive feedback from NSF re. the blueprint response? • JT: Mostly thanked us for the careful review. We pointed out large holes in the plan. Doc is largely phrased in a way that they anticipate making multiple awards, and nothing in it about a program/coordination office. We called this out as a significant weakness as awardee(s) will need coordination – esp if there are separate awardees. Removed workforce decelopment from award and moved into their larger WD in OAC. Feel this will present challenges. • JT to share responses to the blueprint with XAB today (this was sent on Feb 3. Leslie presented to board members today.) • Received approval from NSF to make adjustments in project leadership. John has increased responsibilities at IL that require him to pull back on XSEDE a bit. John’s time reduced from 60% to 40%. Ron has reduced time by 10% to handle non-XSEDE responsibilities. Tim Boerner appointed deputy projet director to pick up some of John and Ron’s responsibilities at 40%. Effective in early March.

o Looked at the full set of John & Ron’s responsibilities and shuffled some to Tim. John is now more external/agency facing issues. Tim handling day to day tasks, business continuity planning. • Kevin Droegemeier appointed acting NSF director until the permanent director can be appointed. He is well-versed in NSF. Summary of appointment status: https://www.sciencemag.org/news/2020/04/white-house-science-adviser-kelvin- droegemeier-will-also-lead-nsf-now • XSEDE’s involvement in COVID-19 HPC Consortium: They realized that they didn’t have a mechanism for making allocations. XSEDE’s XRAS system is exactly what was needed some minor adjustments. Give credit to Dave Hart & RAS team for their time and effort to make this work. XSEDE’s role is important to process. Has been an intense activity. A review team meets every morning, a matching committee meets in the afternoon to determine who will get allocations. Some manual hand-off required for systems not already integrated. Bringing together this scale & diversity of resources never done before.

o 65 submissions received thus far with 35 approved and running. 18 declined. A few

RY5 IPR12 Page 164 in process. Proposals limited to 3 pages.

o Have reached out to colleagues in Europe about potential collaborations. XSEDE’s MOU with PRACE is allowing this to move forward more quickly. Goal of sharing progress and moving research forward more quickly. Moving towards also opening a similar collaboration with RIST (Japan).

o Submissions have begun to slow a bit so now backing off to meeting 3 days/week. o Great role for XSEDE to provide a service much-needed by the community. XSEDE & NSF have gotten some recognition for this.

o John provides weekly updates to NSF on the NSF-funded resources that includes progress and accounting. Working to pull together staff time devoted thus far to understand full value of effort by XSEDE.

o Will we have a measurement later of scientific efficacy of resources/scientific payoffs? . Those who have been provided access to resources will be providing 1-2 page updates (internal). Consortium wants to know that progress is being made. Don’t expect results immediately, but eventually expect results. If projects seem not to make progress, the consortium will consider pulling back on resources. Still working through what such a policy looks like. . Evaluation team willing to provide additional evaluation of efficacy. Session III: CEE PY10 Plan Highlights • Not clear what will follow XSEDE 2.0, but know something will follow. If we ramp down, it may not ramp back up. Trying to be optimistic about what may transition to subsequent awardee(s). • Q: Wrap up so the project is in a well-defined state for whatever comes next. o JT: submitted draft transition plan last year. Will provide an update to that to tell subsequent awardee(s) what we think is necessary to continue the project. Will need to figure out what is needed in the final year of the program. This is deliverable, and we have satisfied already. • Q: Plans to split out BP from whatever next version of XSEDE is--impacts? Still want to take advantage of close integration with XSEDE even if separately funded.

o A: Believe this is a mistake. BP & student program funds at my SP that are not part of XSEDE. BP & student program funds are typically small, so have tried to leverage the two as the sum of the parts is greater. Advantage of keeping this as part of the project as you’re more invested, greater degree of understanding the user community. Splitting it up makes it not as much a part of the org’s mission. Lose ability to leverage. • Q: Where will effort come from to write papers? Papers on broadening participation & student programs recipe is on Kelly’s list to do--would go to a journal.

o JT: Staff have been publishing papers. o RECOMMENDATION: Frame language about staff publications to highlight that we’ve done this in the past. Need to show that this builds on our history of staff publications so it doesn’t look like something new. • Q: +-5%: Would we realistically expect such funding changes at this stage of the project? o JT: Redirecting resources to support COVID-19 response currently. As projects are awarded through the consortium process, also assigning ECSS staff to those projects (so

RY5 IPR12 Page 165 taking them away from other work). Will put in the annual report an accounting of what we’ve devoted to this work. Can that work count towards our metrics? Need to understand how this crisis is impacting the project & impact to our planned activities.

o Kelly: Participant support funds that included travel for people this spring/summer, and those people won’t be traveling. Need to figure out the correct method to handle those travel funds, and working with NSF now. One option is to defer spending of these funds until supplement year.

o Tom: Other funds unspent? . We are in the process of making sure we’re spent to zero by Aug 2021, the fact that travel won’t be happening is impacting this. Working through this with Bob Chadduck. . What else can participant support be used for: travel, lodging, registration fees, support for participation in an activity? • Q: NSF’s education programs are larger than XSEDE’s. An opportunity to move XSEDE funds into a wider range of programs is possible. • RECOMMENDATION: In terms of education programs, if you can show that you have the larger context in mind (larger than XSEDE), that would be a strong argument.

o Kelly: Includes grant: difficult to tap into structure. A lot of one-way info vs. 2-way relationship. Articulate that we have a 2-way relationship & larger structure.

o JT: Have to draw a line between education & workforce development. Have done a lot of training. If it moves to a workforce development program within OAC, it creates concerns. Session IV: RAS PY10 Plan Highlights ● How is satisfaction measured? Via quarterly survey of everyone who goes through XRAS service. ● Previously the % of successful requests was in the 60% range, which is not desirable. Have been steadily increasing this towards an 85% goal. Held a training event to help with writing successful proposals, and that was well-attended. ● Q: What’s wrong with the 13%? Resource limitation? No--that is out of our control. Invoke reconciliation process at each meeting. Here is what panel awarded, here is what resources we have… then scale back to meet what is available. If they ○get something we consider it a success. Hit 87% at the March meeting, and many members of the panel were remote. This could have impacted acceptance rates. Would like to see us sustain at the 87% level. ● How much beyond available resources are the requests? • Varies. Not uncommon to reduce by 30% to meet resource availability. ● How do you reconcile? Scientific merit? SP reps & allocation team walks through requests of over requested resources & try to balance ○ Next phase looks at how big a request is. If you shave a larger request you make room for more. Those funded by NSF are treated most favorably. ○ Documented in allocations procedures & processes. ● How is this balanced with broadening participation? Issues where people from larger institutions ○ might have more grant funds. Considerations for first time users? RY5 IPR12 Page 166 Data showing that new requests tend to do worse than those who have received allocations previously. Have added the ability for the panel to recommend provisional acceptance for 6 mos & ○address concerns. This has been popular with panels as it helps even the playing field. BP: more challenging as we don’t have a good way to track this. We can look at institutional info. Process seems to treat institutions equally. Have been asked about demographic data about users○ in the past, but we don’t have adequate information about users to answer this. ■ RECOMMENDATION: Broadening Participation and RAS could work together to consider ways to expand allocations for researchers at smaller institutions with less grant funding. Possibly in extension year could be something good to look at. ● Dave: can look at this with CEE. Have looked at R1 vs. not R1 institutions. R1 make up majority. Success rates across different types was similar. R2s aren’t suffering, but are asking for less. ● Linda: Metric that is reported on underrepresented users/data are people who have accessed & used resources. Don’t report by institution type. Promote heavily how to write a successful allocation proposal to make sure smaller institutions are getting info on how to be more successful in the process. Process appears to be equitable across different institutions/groups. ● Provide support through novel & innovative projects, which are often newer users. Trying in multiple ways to promote greater success among more groups. But still don’t have enough resources to fulfill all requests.

Session V: Ops PY10 Plan Highlights ● Q: How are cloud resources funded? Does RAS allocate? We had funding in the XSEDE2 budget for XDCDB and did an evaluation to look at moving the database to the cloud. Replaced purchasing hardware to paying for cloud services. Then we were ○able to move other services to the cloud. Approx 15 services housed in the AWS cloud currently. Have pre-funded cloud services through the end of this project year. We don’t allocate through RAS, but buy through project funds. AWS is a good solution for these enterprise services. ■ Q: Not recommending that in the next version of XSEDE the project could live with cloud only? JT: Would have to do a cost benefit ratio, but more advantageous for enterprise services. We look at this carefully before we move any services. ■ Cloud will make it easier to hand off services to a new awardee as well. ■ XDCDB & XRAS processing need a high level of redundancy & failover, so will be easier to transition. Others are on VM services. Some things need to be physically controlled so it wouldn't make sense to move to the cloud.

Session VI: XCI PY10 Plan Highlights ● Q: Long term 5 year/10 year accomplishments compared to the vision that you had for XCI. Where are we now and what does it mean for the future? Benefit across the 10 years has largely been to come up with a solid process for delivering software. Get requirements from the community & translate into what will get delivered & plan to ○deliver. Well documented plan/process for this. This will suit the community for a long period into the future. Idea of a research community software portal isn’t just to get software, but to engage in development/deployment. Place to discover and figure out how to make use of it. CRI strengths has

RY5 IPR12 Page 167 learned how to translate best practices at national centers into something that can be used on campuses. Implementation of local resources in a way that not everyone has to learn for the first time. ● How your efforts complement or work with training efforts. Some efforts of the XCI team surely require some training. Do you work with the training team? Do you benefit from what the CEE team is offering for training? Integrate closely with CEE training. Have had a useful working relationship with the training team through all of the 10 years. Used to be part of outreach & engagement group. Those contacts ○have stayed largely thanks to Susan, Linda A, Jay Alameda & ECSS folks, campus champions fellows. A lot of successful outreach as a result. A: A lot of the power of XSEDE is the integration & collaboration across teams. ● Q: Bigger vision for software ecosystem XSEDE was going to build. What are the obstacles? Hard to ○ build academic software? Heterogeneity of SPs? Observing this from the early days, there was a market change in the way XSEDE was asked to address community needs for software. XSEDE is an integrator of solutions--not a ○designer/developer. Trying to pull in from other successful projects best solutions & promote and deliver as capabilities within XSEDE. Issues of adoptions across SPs--they’ll adopt what they adopt. We make recommendations, but delicate tension between identifying what’s available and making sure people implement. Have developed a process to make it less painful. JT: Some are reluctant to adopt academic software because no long term support for it. Result is that they often develop something locally that they can’t support either. Not NSF’s role, but no one○ is picking this up. Victor: SSH software that XSEDE uses has specific modifications for high performance data transfer. Modifications to authentication to Globus. These 2 modifications don’t have developers of ○mainline products to adopt, so someone has to maintain these changes. These capabilities are important to XSEDE. No other version of these that provide these capabilities. Ken: Long term concern of open science grid. JT: Have adopted ongoing support of some tools because no alternative. XSEDE has taken over ○ support in some cases. That builds up a mortgage of things we need to support. This problem is bigger○ than XSEDE. Where does Open MPI funding come from? Standards effort, multiple implementations. Some projects decide to take on support. Globus has found sustainability through subscriptions. Acad○ emics don’t like to pay for these things. Don’t feel it is the responsibility of XSEDE. No good answer from those that fund these things in the first place.

Session VII: Program Office PY10 Plan Highlights ● Cision software both helps provide connection to more media outlets as well as provides more accurate reporting of number of hits. ● Staff publications target is set to provide as much documentation & info to the community as possible as the project winds down. Have also made it a point to have a greater focus on staff publications throughout the project. ● % subaward invoices target decreasing slightly due to delay in time for subawards to submit monthly breakdowns. ● Longitudinal studies

RY5 IPR12 Page 168 During the last review, the panel questioned the utility of conducting these studies because it wouldn’t benefit XSEDE2--shouldn’t engage in anything that doesn’t directly impact the funded ○project. Can also bring value to larger community re. Lessons learned along the way. XAB thoughts? ■ Value in understanding impacts of this project to funding a future project. Value to NSF in designing future programs. ■ Will any of this inform/impact XSEDE 3? Have they been designed to impact XSEDE 3? ● Designed to help understand XSEDE & make as effective as can be. ● More thinking about how results can benefit the community in general.

Session VIII: ECSS PY10 Plan Highlights ● Q: ECSS used to report on a metric re. time ECSS invests in projects vs. estimated time to complete that project without ECSS support? We do track this (every ECSS PI exit interview includes this question) but not as project KPI. Some PIs say they couldn’t have done it without ECSS, so difficult to measure in that case. Currently using○ 24 mos as a cap on months saved in that case. NSF could decide to put more funds into domain sciences instead of OAC, but this metric demonstrates the value of the funds they’re channeling to XSEDE ○ JT: Blueprint doc ignores ECSS. We included this in our response. No specific feedback to our response. ○

XSEDE Supplement Proposal ● Approach is to shift the year 5 transition plan into a year 6 & sustain activities that we’ve been supporting throughout the project. Awardee is limited to asking for no more than 20%of the overall award. $22M is 20%. Just salaries would exceed the 20% cap. May get saved by the ability to use unspent participant support funds--may get us where we need to be. Depending on overall impacts of COVID19, may need to ask for funds to accomplish goals of the project since effort is being devoted to COVID19. Non-trivial amount of effort already incurred for this. Once we have those numbers, we’ll discuss with NSF about how they want to address this (back off on other efforts or provide additional funds?). Q: consider going to 11 mos? ■ JT: Something to consider. Don’t know NSF’s schedule for getting out solicitation ○ David: Many salary freezes across institutions. If you show standard salary escalations could result in push-back given the current climate. ○ ■ JT: Some institutions moving to furloughs, others freezing salaries. Don’t know how it will play out. Could have a year of lesser salaries as a result Q: Have you worked up a budget? ■ JT: Working on this. Talked with Bob about some of the budget concerns, and he is looking into ○ it. ● XSEDE will submit a supplement request soon & NSF will review it during the June review. ● Q: Panel review can approve more than 20% budget. Maybe some restrictions in how proposal is requested.

RY5 IPR12 Page 169 JT: Not submitting a proposal, but a supplement to the existing award. Not sure if a supplement over $10M is required to go by the science board or not. ○ ● Q: Will there be other parts of the proposal besides the budget of concern? JT: can’t go beyond scope of original proposal. Will continue doing what we’re doing. Costs grow over time, other costs crop up (will incur some license costs) ○ ● Q: What lessons will be learned by doing everything remotely? How in the future XSEDE might save $ by continuing to do things remotely? Don’t always have to be together. Fewer in person events can help with budget issues. • JT: XSEDE has been an online distributed project from the beginning. Also moved to 2 of the quarterly staff meetings being held remotely. Value in in-person format, but have cost savings with 2 virtual meetings. This June, the project will hold its first XRAC meeting virtually. Have resisted online format for this meeting up to now. If this works well, may consider moving more of these to a virtual format. ● Emre: wouldn’t take on too much risk by trying to cut the budget excessively. Let NSF decide how they want to review it. ● Q: What happens if they still don’t have a solicitation in another year? Creating hardship by asking for supplement year after year with continued uncertainty. JT: Concerned about this, but can only point this out to our program officer. Makes it difficult for us to go outside the current scope with a supplement because we have to stay within○ the existing scope. ● Q: Envision engaging XAB in proposal once solicitation is released? JT: Yes we will engage, but can’t do that in this meeting. This meeting is supported by XSEDE grant funds. John may reach out individually to request input. Would need solicitation first.○ Encourage XAB members to look at Coordination Blueprint doc and provide feedback to Manish.

Wrap-up/Close Meeting • Overall, XAB members like the virtual format, and it leaves more time for discussion.

RY5 IPR12 Page 170 18. XSEDE Project Execution Plan At the beginning of the XSEDE project, a Project Execution Plan (PEP) was submitted to and approved by NSF. Content in this and all preceding Interim Project and Annual Reports supersede information submitted in the original PEP. The most recent version of the PEP (V. 2.4) is appended here and can also be viewed on the wiki29. Changes in Version 2.4 of the PEP include: • Addition of link to web page listing subaward partners. (p. 5) • Change to project end date to August 31, 2022 and update to project schedule per approval by NSF of XSEDE supplement year. (p. 33) (PCRPROJ-112) • Addition of Deputy Project Director role in the following sections: I.1 Project Governance (p. 34), I.2 XSEDE Senior Management Team (p. 34), K Description of the financial and business controls to be used. (p. 37) (PCRPROJ-103) • Changes in Senior Management Team members: Marinshaw, Froeschl, Chadduck. (p. 34) • Addition of link to XSEDE Metrics Dashboard (p. 39)

29 https://confluence.xsede.org/display/XT/XSEDE+Project+Execution+Plan

RY5 IPR12 Page 171

A XSEDE Project Execution Plan

Submitted to the National Science Foundation, Directorate for Computer and Information Science and Engineering, Division of Advanced Cyberinfrastructure, As a deliverable in preparation for “XSEDE 2.0: Integrating, Enabling and Enhancing National Cyberinfrastructure with Expanding Community Involvement” Proposal 1548562

August 17, 2020 Version 2.4

Table of Contents

A XSEDE Project Execution Plan A. Summary of the purpose of the award and its sub-awardees B. Description of the project deliverables, milestones and schedule C. High-level Work Breakdown Structure (WBS) D. Work Breakdown Structure dictionary defining the scope of the WBS elements 2.1. Community Engagement & Enrichment 2.1.1. D irecto r’s Office 2.1.2. Workforce Development 2.1.4. Broadening Participation 2.1.5. User Interfaces & Online Information 2.1.6. Campus Engagement 2.2. Extended Collaborative Support Service: Enabling the Research Community 2.2.1. Directors Office 2.2.2. Extended Support for Research Teams (ESRT) 2.2.3. Novel & Innovative Projects (NIP) 2.2.4. Extended Support for Community Codes (ESCC) 2.2.5. Extended Support for Science Gateways (ESSGW) 2.2.6. Extended Support for Education, Outreach & Training (ESTEO) 2.3. XSEDE Community Infrastructure 2.3.1. D irecto r’s Office 2.3.2. Requirements Analysis & Capability Delivery (RACD) 2.3.3. Capability & Resource Integration (CRI) 2.4. XSEDE Operations 2.4.1. Directors Office 2.4.2. Cybersecurity 2.4.3. Data Transfer Services 2.4.4. XSEDE Operations Center (XOC)

2

2.4.5. Systems Operational Support (SysOps) 2.5. Resource Allocations Service (RAS): Stewarding the National Investments 2.5.1. D irecto r’s Office 2.5.2. XSEDE Allocations Process & Policies 2.5.3. Allocations CI Enhancement & Maintenance 2.6. Program Office 2.6.1. Project Office 2.6.2. External Relations (ER) 2.6.3. Project Management, Reporting & Risk Management (PM) 2.6.4. Business Operations 2.6.5. Strategic, Planning, Policy, & Evaluation E. Project budget and staffing broken out by WBS element and by institution F. Description of the methodology and assumptions used for estimating the budget components G. Project risk analysis and a description of the analysis methodology H. Project schedule I. Description of the organizational structure of the project team and governance of the project including advisory groups and the processes that facilitate interaction with all external entities J. Description of the sub-contracting strategy and controls K. Description of the financial and business controls to be used L. Plan for reporting on the technical and financial status of the project M. Description of anticipated safety or health issues associated with the project, if any N. Cyber security plan for protecting the confidentiality, integrity and availability of XSEDE resources and services O. Comprehensive performance management plan which supplies reporting data P. Description of Project Policies and Standard Operating Procedures

3

A. Summary of the purpose of the award and its sub-awardees “XSEDE 2.01: Integrating, Enabling and Enhancing National Cyberinfrastructure with Expanding Community Involvement” was submitted in response to the “XSEDE Renewal RFP,” under NSF’s Extreme Digital (XD) program. NSF determined that it is in the community’s best interest, and can achieve minimal disruption of critical community services, for NSF to invoke the provisions of the cooperative agreement ACI-1053575 that allows submission of a proposal by the University of Illinois for a renewal project, under the conditions set forth in the “XSEDE Renewal RFP.” The goal of XSEDE is to accelerate open scientific discovery by enhancing the productivity and capability of researchers, engineers, and scholars, and by broadening their participation in science and engineering. It does so by making advanced computational resources easier to use, integrating existing resources into new, powerful services and building the community of users and providers. XSEDE is a virtual organization that provisions complex distributed infrastructure, support services, and technical expertise. A prominent opportunity for XSEDE is the growing, diverse collection of advanced computing, high-end visualization, data analysis, and other resources and services available to researchers, engineers, and scholars; these resources have the potential to help understand and solve the most important and challenging problems facing the nation and world. The challenge for XSEDE, as a virtual organization, is to organize these disparate resources, creating integrated services and a coordinated environment that serves the end user needs. The challenge also includes fostering awareness of, and training for, full utilization of the capabilities offered by XSEDE and its associated resources, as well as catalyzing workforce developments. All these tasks need to be accomplished in light of evolving user requirements, resources, and NSF strategies.

With this award, NSF will continue to support an advanced cyberinfrastructure (CI) that uses an increasingly virtualized approach to the provision of high-end services. These services provide a common framework for researchers in computational and data-enabled science & engineering (CDS&E) at all levels of sophistication and aim to create a seamless environment from the desktop, to local university resources to national resources. The objective is to provide the cybertools, software, know-how, assistance, and associated infrastructure required for their research to both sophisticated and novice users, the traditional high performance computing community and new communities who have not used high performance computing resources before.

XSEDE will help assure that NSF-supported compute and data-intensive cyberinfrastructure continues to provide leading-edge capabilities for the research and education community and will facilitate transformative advances in science and engineering.

The XSEDE project is a collaboration between the University of Illinois at Urbana-Champaign (National Center for Supercomputing Applications), the University of Tennessee at Knoxville

1 The term “XSEDE” refers to the vision, mission, services and support established through the execution of the initial XSEDE award (#1053575) as well as the entire project during its ten year period. "XSEDE 2.0" is used to reference specific additions and/or changes to the XSEDE project as a direct result of this proposal (#1548562).

4

(National Institute for Computational Sciences), the Carnegie Mellon University and the University of Pittsburgh (Pittsburgh Supercomputing Center), the University of Texas at Austin (Texas Advanced Computing Center), the University of California at San Diego (San Diego Supercomputing Center), the University of Chicago, Indiana University, Purdue University, the Shodor Education Foundation, the Ohio Supercomputer Center, the Southeastern Universities Research Association, Cornell University, the National Center for Atmospheric Research (NCAR), the Georgia Institute of Technology, the Oklahoma State University, the University of Georgia, Oklahoma University, the University of Southern California, the University of Arkansas, Notre Dame, and Internet2. For a current list of subaward partners, see https://www.xsede.org/about/governance/partnerships.

XSEDE will support six core service areas: Community Engagement & Enrichment (CEE), the Extended Collaborative Support Service (ECSS), XSEDE Community Infrastructure (XCI), XSEDE Operations, the Resource Allocations Service (RAS) and the Program Office.

*Note table updated April 11, 2019 to reflect the addition of Internet2 and Notre Dame as subawardees per Amendment #006.

Strategic Plan: Achieving Our Mission and Goals XSEDE’s mission is to enhance the productivity of a growing community of scholars, researchers, and engineers through access to advanced digital services that support open research by coordinating

5 and adding value to the leading cyberinfrastructure resources funded by the NSF and other agencies. Our strategic goals fully support NSF’s vision as stated in Investing in Science, Engineering and Education for the Nation’s Future and strategies stated broadly in the Cyberinfrastructure Framework for 21st Century Science and Engineering and the more specifically relevant Advanced Computing Infrastructure: Vision and Strategic Plan.

Strategic Goals Our strategic goals support our mission and guide the project’s activities toward the realization of our vision of an advanced digital services ecosystem. Three strategic goals are defined:

Deepen and Extend Use: XSEDE will deepen the use of the ecosystem by existing scholars, researchers, and engineers, and extend the use to new communities. We will contribute to the preparation—workforce development—of the current and next generation of scholars, researchers, and engineers in the use of this ecosystem; and raise the general awareness of the value of advanced digital services.

Advance the Ecosystem: XSEDE will advance the ecosystem by creating an open and evolving e- infrastructure and enhance the technical expertise and support services offered.

Sustain the Ecosystem: XSEDE will sustain the ecosystem by ensuring and maintaining a reliable, efficient, and secure infrastructure, and providing excellent user support services. XSEDE will further operate an effective, productive, and innovative virtual organization.

The XSEDE project is operational in nature and is unlike a research project in that, in XSEDE, the expectations and offerings are constantly evolving and are a direct result of the community needs, available CI resources and NSF strategies. While operating in a continued state of CI resource and community needs evolution, it is impractical to define long term goals that are more detailed than the Strategic Goals above, as doing so may unintendedly constrain the project in ways unknown at the present time. There is direct alignment between the mission statement and Key Performance Indicators for every level 2 area, and the project’s strategic goals. In addition, each annual program plan contains specific goals for each level 2 area and level 3 group.

The XSEDE project uses a wiki that is largely open to the public. The home page of the staff wiki can be used as a project map for governance, policies, processes, project documentation, event archives, etc.

6

The XSEDE Staff Wiki is located at: https://confluence.xsede.org/display/XT/XSEDE+Staff+Wiki

B. Description of the project deliverables, milestones and schedule

Project Deliverables: ● High level transition plan for the solicitation of a successor project ● Detailed transition plan once the successor(s) have been announced ● A comprehensive performance management plan detailing how the XSEDE project is performing against expectations as well as its impact on the research community ● Annual “XSEDE Highlights” booklet with key research projects supported by the XSEDE project ● XSEDE Resource Allocation Services (XRAS) ● A Software Repository of approved tools and software that is supported by the XSEDE project ● A public Wiki page containing information and documents about the XSEDE organizational structure as well as the governance and operation of the XSEDE project ● Documented Use Cases that capture the community needs ● Capability Delivery Plans that document the solutions XSEDE is integrating into the ecosystem and the process for doing so ● Evaluation and assessment reports of the External Evaluators ● Training materials and online resources delivered to the community ● Cybersecurity Plan ● Annual Program Plan will be provided to NSF prior the beginning of each project year detailing the governance, management and specific milestones to be reached by each of the service areas in the upcoming XSEDE project year. The Annual Program Plans will be stored on the wiki and can be found here: https://confluence.xsede.org/display/XT/Annual+Program+Plans

7

C. High-level Work Breakdown Structure (WBS)

D. Work Breakdown Structure dictionary defining the scope of the WBS elements

2.1. Community Engagement & Enrichment (CEE)

Mission: To actively engage a broad and diverse cross-section of the open science community, bringing together those interested in using, integrating with, enabling, and enhancing the national cyberinfrastructure including support of learning and workforce development via training and education efforts. CEE supports XSEDE’s strategic goals with the following activities: ● Deepen and Extend Use ○ Extend use to new communities ○ Deepen use to existing communities

8

○ Preparing the current and next generation ○ Raise awareness of the value of advanced digital research services ● Sustain the Ecosystem ○ Provide excellent user support ● Advance the Ecosystem ○ Enhancing the array of technical expertise and support services

The core of Community Engagement & Enrichment (CEE) is the user, broadly defined to include anyone who uses or may potentially use the array of resources and services offered by or via XSEDE. The CEE team, led by co-PI and L2 director Kelly Gaither (TACC), is dedicated to actively engaging a broad and diverse cross-section of the open science community, bringing together those interested in using, integrating with, enabling, and enhancing the national cyberinfrastructure. Vital to the CEE mission is the persistent relationship with existing and future users, including allocated users, training participants, XSEDE Conference attendees, XSEDE collaborators, and campus personnel.

The five components of CEE are User Engagement, User Interfaces & Online Information, Campus Engagement, Workforce Development (including Training, Education & Student Preparation), and Broadening Participation. These five teams will ensure routine collection and reporting of XSEDE’s actions to address user requirements. They will provide a consistent suite of web-based information and documentation and engage with a broad range of campus personnel to ensure that XSEDE’s resources and services complement those offered by campuses. Additionally, CEE teams will expand workforce development efforts to enable many more researchers, faculty, staff, and students to make effective use of local, regional, and national advanced digital resources. CEE will expand efforts to broaden the diversity of the community utilizing advanced digital resources. The CEE team will tightly coordinate with the rest of XSEDE, particularly Extended Collaborative Support Services, Resource Allocation Services, Community Infrastructure, and External Relations.

2.1.1. Director’s Office The L2 Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the L2 area. This oversight includes providing direction to the L3 management team, coordination of, and participation in, L2 planning activities and reports through the area’s Project Manager. The Director’s Office also attends and supports the preparation of project level reviews and activities.

2.1.2. Workforce Development 2.1.2.1. Training The Training team will develop and deliver training programs to enhance the skills of the national open science community and ensure productive use of XSEDE’s cyberinfrastructure. XSEDE will expand the breadth and depth of XSEDE training

9

content based upon a gap analysis of current programs and needs identified by the User Engagement team. XSEDE will expand on existing training roadmaps to include information on which training courses have been vetted and provide pointers to materials available from XSEDE as well as external training providers. Survey data will be collected to assess and improve upon respondents’ abilities to easily find the needed material.

The training team will fully implement the XSEDE training certification program for users and staff in an effort to recognize learners who demonstrate competencies attained through participation in XSEDE training offerings, enabling them to gain recognition for their accomplishments. The Moodle Learning Management System and Mozilla’s Open Badges Infrastructure (OBI) are the basis for implementation. Badges for an additional three competencies will be offered in PY6 with a goal to issue at least ten badges to XSEDE staff and fifty badges to XSEDE users, with 10% growth planned for successive years.

During PY5 of XSEDE, we will offer short duration Massive Open Online Courses (MOOCs). Based on a user assessment of these MOOCs, XSEDE will enrich the interactive and hands-on portions of these training offerings and transform them into smaller, more effective SPOCs (Small Private Online Courses) offered quarterly. Past evaluation data shows SPOC students were much more motivated and had a higher completion rate when provided with mentoring and a badge or university credit.

XSEDE will coordinate training development and offerings with campus representatives and HPC centers interested in developing, delivering, and/or using training materials. The University of Illinois’ Computational Science and Engineering group, the Software Carpentry group, and the Data Carpentry groups have committed to collaborate with XSEDE. These collaborations will gather user requirements for training, share plans for developing training materials among these groups, and foster sharing of training development and the resulting materials. The objective is to expand the breadth and depth of training so researchers, users, students, and XSEDE staff will have ready access to an ever- expanding portfolio of training opportunities delivered via live, broadcast, and online learning platforms.

2.1.2.2. Education 10

The education team will work closely with training and student preparation to create a cohesive team supporting faculty in all fields of study about advanced digital technologies, and incorporating those capabilities within the undergraduate and graduate curriculum. XSEDE will develop an online community for faculty to share experiences and get advice on curriculum materials and development. XSEDE will work with faculty to develop 50 new, re-usable learning modules and materials. This will include modules for introducing computation and data-enabled techniques within STEM classes and student oriented projects. XSEDE will disseminate educational materials to provide public access to a growing base of peer-reviewed materials that will enhance the graduate and undergraduate experience and contribute to preparing future generations.

The education team will visit campuses and attend regional workshops for faculty. This outreach has proven to be crucial in engaging faculty with integrating computational and data-enabled tools and methods into the curriculum. The campus visits and faculty support have been instrumental in motivating and assisting departments and colleges with developing certificate and degree programs. The outreach also helps raise awareness and usage of the repository of training and education materials available from the XSEDE User Portal for re-use by the community.

2.1.2.3. Student Preparation The Student Preparation program will actively recruit students to use the aforementioned training and education offerings to enable the use of XSEDE resources by undergraduate and graduate students. Evaluation data show XSEDE’s overwhelmingly positive impact preparing college students to conduct computational science and research. XSEDE will reach thousands of students annually via the vast array of training offerings. XSEDE will provide badging and certification for students on a diverse range of topics including parallel programming, visualization, data analytics, and software engineering practices. XSEDE will broaden participation by engaging with students via conference exhibitions, campus visits, regional workshops, and national conferences. XSEDE will reach out to externally funded student programs, such as NSF Graduate Research Fellows, the NSF Research Experience for Undergraduates (REU), Integrative Graduate Education and Research Traineeship (IGERT), and Broadening Participation in Computing programs. The student preparation program will also establish partnerships with national student organizations (e.g. SIAM, ACS, ACM). 11

The students in these programs will have the opportunity to access XSEDE’s resources and services, including the workforce offerings.

2.1.2.4. Broadening Participation Broadening Participation will engage underrepresented minority researchers from domains that are not traditional users of HPC, and from Minority Serving Institutions. This target audience ranges from potential users with no computational experience to computationally savvy researchers, educators, Campus Champions, and administrators that will promote change at their institutions for increased use of advanced digital services for research and teaching.

XSEDE will provide awareness activities—conference exhibitions, campus visits, and regional workshops—while increasing national impact through new partnerships such as the Southern Region Education Board Doctoral Scholars Program, the Institute for African-American Mentoring in Computing Sciences, and the Computing Alliance for Hispanic-Serving Institutions. XSEDE will aggressively promote the submission of papers at professional societies by XSEDE under-represented users and expand our dissemination partners to include new initiatives such as the IEEE Special Technical Committee on Broadening Participation.

Persistent participation is enabled by curriculum reform and larger numbers of researchers adopting the use of advanced digital resources as standard methods. Collaboration with Campus Engagement and Education will support institutional change and capacity building. XSEDE will target institutions with funded initiatives to implement curriculum changes and increase research capacity.

Using the model of the Service Provider Forum, an XSEDE Diversity Forum will be established with outreach and diversity managers at HPC centers and on campuses. The forum participants will share best practices, identify ways to leverage XSEDE activities, and review XSEDE programs to ensure they are encouraging diversity. The diversity forum will be responsible for engaging new national programs and initiatives, institutions with funding to make curriculum change and research infrastructure investments, and major research grant awards at MSIs or with a focus on broadening participation. 2.1.3. User Engagement (UE) 12

The mission of the User Engagement (UE) team is to capture community needs, requirements, and recommendations for improvements to XSEDE’s resources and services, and report to the national community how their feedback is being addressed. The UE team will process and track actionable items obtained from user feedback and monitor them throughout the UE loop, from assignment to a responsible XSEDE party through communication of subsequent actions back to the user community. To obtain user feedback, we will engage users of XSEDE’s resources and services to gauge overall satisfaction, pervasive problems, emerging needs, and requirements. Integral to this process is the derivation of requirements from diverse sources—micro-surveys, user satisfaction surveys, user interviews—and turning them into actionable Use Cases that can be tracked and handled in all areas of the XSEDE organization. The UE team will use tools provided by the XSEDE Project team including JIRA and issue tracking software to monitor requests and enhancements linked to the stakeholders who originated the requirement. The UE team will use this feedback to create a lightweight Use Case document—an encapsulation of user needs via scenarios—attach it to the JIRA issue, and assign it to the responsible XSEDE area. UE personnel will provide issue status on the user portal to keep the stakeholders and the general community apprised of progress on actionable items. This ongoing feedback loop will encourage further community input for improving XSEDE’s resources and services.

2.1.4. Broadening Participation (BP) Broadening Participation will engage underrepresented minorities, women, and Minority Serving Institution faculty and students. BP will provide awareness through conference exhibiting, campus visits, and training events —while increasing national impact through new partnerships with other organizations focused on inclusion and diversity such as National Council of Women in Technology (NC-WIT) and the Institute for Broadening Participation (IBP).

2.1.5. User Interfaces & Online Information (UII) The website is the first place XSEDE stakeholders come to find information about the project, addressing the needs of internal and external stakeholders. The website and user portal will be improved to create a more consistent and easy-to navigate look and feel. The User Interfaces & Online Information (UII) team will develop an information architecture to support a variety of stakeholders. This information-centric approach is rooted in the ability to answer fundamental questions when browsing the website: Am I in the right place? Do they have what I am looking for? What do I do now? The redesign will include a new layout, enabling a single web and mobile site regardless of device type. The UII team will expand mobile capabilities and build upon the new iOS and Android applications. Managing and publishing approved content to the site will be handled via workflows that enable multiple members of XSEDE to contribute in an organized and effective manner.

13

Prospective and current users of XSEDE quickly navigate from the website to the XSEDE User Portal for user and project related needs. For example, PIs can apply for and manage allocations, and record their research accomplishments via the publications feature in the Portal. UII will expand the initial XSEDE User Portal to integrate features such as data management, job execution, and task management. The UII team will incorporate the XSEDE software catalog and its administrative interface and continue to improve capabilities based on stakeholder feedback.

The UII team manages documentation enabling users to easily find resource and service offerings. In addition, the UII team will enable users to create a dynamic environment and tailor the user portal experience to their individual needs. For example, users with allocations on multiple XSEDE resources will be presented with content related to those specific resources, e.g. job submission.

2.1.6. Campus Engagement Ongoing communication and cooperation with campuses will help to ensure that the resources and services being offered on campuses complement those offered by XSEDE, and vice-versa. This collaboration will enhance the advanced digital resources and services provided to the community.

The Campus Champions effort has established Memoranda of Understanding (MOUs) with more than 190 campuses. On these campuses there are more than 250 Campus, Domain and Student Champions focused on assisting local users to make informed choices of resources and services that may best meet their needs. The Campus Engagement effort will extend XSEDE’s relationship with campus personnel by establishing regular communications with CIOs and VPs for research.

CIOs have indicated that they value communicating with each other and with XSEDE staff to plan the development and delivery of resources and services on their campuses. There will be monthly conferences calls, email lists, and forums for CIOs and VPRs to share challenges, solutions, and information. Other campus individuals who have service roles complementary to XSEDE (e.g. cyberinfrastructure integration and support, training, education, and broadening participation) will be engaged to enhance cooperation among campuses and XSEDE.

The Campus Engagement program will collect information from each campus quarterly to assess the level of activity in working with local users. The Campus Engagement team will provide additional training and consulting, and work with campuses to strengthen their Champion’s productivity and engagement. Campus Engagement will enhance the “Welcome Wagon/New

14

Champion Development” efforts to provide individualized attention to new Champions so they can more quickly become actively engaged.

The number of campus members has more than doubled in four years, and we project this growth rate to continue based on continuing requests from campuses. To address this rapid growth, the XSEDE Regional Champions program is actively developing models for regional support. The lessons learned from the Regional Champions program will guide further improvements for scaling support of the member campuses. XSEDE alone will not be able to sustain the support needed for the predicted growth of the program. Through collaborations with ACI-REF, Open Science Grid, and the SP Forum, the Campus Engagement program will develop strategies for long-term sustainability.

2.2. Extended Collaborative Support Service (ECSS)

Mission: Improves the productivity of the XSEDE user community through successful, meaningful collaborations to ● optimize their applications, ● improve their work and data flows, and ● increase their effective use of the XSEDE digital infrastructure and ● broadly expand the XSEDE user base by engaging members of underrepresented communities and domain areas

ECSS supports XSEDE’s strategic goals with the following activities: ● Deepen and Extend Use ○ Extend use to new communities ○ Deepen use to existing communities ○ Preparing the current and next generation

Domain scientists should not have to be experts in all areas of cyberinfrastructure to achieve their goals. The ECSS program provides dedicated staff who develop deep, collaborative relationships with XSEDE users, helping them make best use of XSEDE resources to advance their work. These professionals possess combined expertise in many fields of computational science and engineering. They have a deep knowledge of underlying computer systems and of the design and implementation principles for optimally mapping scientific problems, codes, and middleware to these resources. ECSS includes experts in not just the traditional use of petascale computing systems but also in data-intensive work, workflow engineering, and the enhancement of scientific gateways.

ECSS collaborations complement initial engagements with users through the XSEDE Operations Center helpdesk and CEE. They last for at least one month and are expected to have significant

15 deliverables within a year. Staff members typically spend 20-25% of their time on a single project, but there is flexibility in how these projects unfold.

ECSS support is usually requested by researchers via the XSEDE peer-review allocation process. If reviewers recommend support and if staff resources are available, the ECSS expert and the requesting PI develop a work plan outlining the project tasks. The work plan includes concrete quarterly goals and staffing commitments from both the PI team and ECSS. ECSS managers review work plans and also track progress via quarterly reports. But ECSS is often proactive, reaching out to groups. The Novel and Innovative Projects group within ECSS reaches out to communities which have not traditionally been users of advanced computing, while the Community Codes group works with developers to improve the performance of widely used community codes.

ECSS staff also provides the expertise for the CEE training program and will assist the Resource Allocation Service by conducting allocation reviews of smaller-scale Research requests and all Educational requests. For XCI, ECSS will provide use cases and participate in technical reviews.

2.2.1. Directors Office The L2 Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the L2 area. This oversight includes providing direction to the L3 management team, coordination of, and participation in, L2 planning activities and reports through the area’s Project Manager. The Director’s Office also attends and supports the preparation of project level reviews and activities.

2.2.2. Extended Support for Research Teams (ESRT) ESRT is a subarea of the XSEDE Collaborative Support Service (ECSS) focused on the support of research teams. Research team support includes, but is not limited to performance analysis, petascale optimization, effective use of accelerators, I/O optimization, data analytics, visualization, and domain knowledge. This support is provided by staff members organized under ECSS who provide expertise in computational sciences, domain science (many at the doctoral level), data analysis, scientific applications, and visualization.

2.2.3. Novel & Innovative Projects (NIP) NIP proactively develops projects in areas of science and scholarship that have traditionally not used advanced CI, such as bioinformatics, machine learning; image, text and social network analysis. It focuses on leveraging the science gateway, virtual environment, and data hosting and analysis capabilities of XSEDE service providers, steering nontraditional user groups to the most suitable resources, and mentoring them to ensure the success of their projects.

NIP will focus on efficiently exploiting the capabilities of new SPs and complementary components of the national advanced computing and data ecosystem. NIP experts will work 16 closely with SPs, recruit and steer appropriate user groups to the most suitable resources, and mentor them to ensure the success of their projects. In particular, the efficient use of the science gateway, virtual environments, and data hosting and analysis support offered by the new SPs should significantly boost the return on NIP effort. These environments promise to greatly reduce the barriers between end-users and the advanced computing ecosystem, especially for people in non-STEM fields and at under-resourced institutions.

NIP will expand its efforts to additional disciplines, such as computational mathematics, applications of geographical information systems, and the arts. Suggestions will be sought from advisory bodies, NSF program directors, and XSEDE internal sources. To improve its impact on underserved minorities, NIP will further strengthen its collaboration with CEE, paying special attention to the development and mentoring of projects that improve the quality and efficiency of teaching at under-resourced institutions. We will use the contract hiring and Domain Champion recruitment processes, as well as the Campus and Regional Champion programs, to ensure active participation by underrepresented groups in the work of NIP.

2.2.4. Extended Support for Community Codes (ESCC) Extended Collaborative Support for Community Codes (ESCC) extends the use of XSEDE resources by collaborating with researchers and community code developers to deploy, harden, and optimize software systems necessary for research communities to create new knowledge. ESCC projects include collaboration with the developers of widely used community applications and models.

ESCC projects can be proposed by the developers of community codes, the ESCC manager or suggested by staff, XSEDE leadership, and advisory boards. Priority will be given to helping projects funded by NSF programs (e.g., PetaApps, SDCI, STCI, SI2, MREFC) to generate robust, sustainable, and maintainable community applications. XSEDE also supports user-controlled Community Software Areas (CSAs) where any developer can get an account and install and publicize their software. The ability to request CSAs will be featured more prominently in XSEDE.

2.2.5. Extended Support for Science Gateways (ESSGW) Science Gateways are community-designed, web-based interfaces that build on XSEDE (and other) resources to provide services to their communities. Gateways play a critical role in expanding XSEDE’s user base and account for 40% of all XSEDE users. But the needs of gateway developers can be significantly different from those of researchers requesting other types of ECSS assistance. Gateways require well-defined, secure, web-accessible programming interfaces which are used for remote job submission, monitoring, and management; remote file and data management and transfer; and information services describing the state of hardware and networks, available software, queuing systems wait times, and similar information. ESSGW staff can often use lessons learned working with one user team to advise another. Best practices will continue to be captured through activities like the gateway cookbook. ESSGW staff

17

members also bring in expertise in areas such as workflows, data analytics and digital humanities and often recruit new gateways through their connections in the community.

2.2.6. Extended Support for Education, Outreach & Training (ESTEO) ESTEO coordinates bringing technical expertise of ECSS staff members to support CEE efforts. ESTEO staff deliver training in many venues—at XSEDE sites, on campuses, at conferences and offered virtually, as well as serve on committees within XSEDE’s CEE area to jointly plan and support these activities. ESTEO experts develop, review, and present technical content in all areas of ECSS expertise, review education allocation requests and also serve as mentors for Campus Champions Fellows.

The Campus Champions Fellows program pairs XSEDE Campus Champions with ECSS staff members to work together on ECSS projects for one year. Fellows commit 400 hours per year and receive a stipend and travel support in order to participate. For ECSS staff, acting as a mentor to a Fellow counts as an additional ECSS project, allowing time to participate substantially in the mentoring exercise. The goal is to enhance the effectiveness of Fellows on their campus.

2.3. XSEDE Community Infrastructure (XCI) Mission: To facilitate interaction, sharing and compatibility of all relevant software and related services across the national CI community by building and improving on the foundational efforts of XSEDE. XCI supports XSEDE’s strategic goals with the following activities: ● Sustain the Ecosystem ○ Provide reliable and secure infrastructure ● Advance the Ecosystem ○ Create an open and evolving e-infrastructure

Through XCI, XSEDE will serve as an aligning function within the nation not by rigorously defining a particular architecture, but rather by assembling a technical architecture that facilitates interaction and interoperability across the national CI community. The suite of interoperable and compatible software tools that XSEDE will make available to the community will be based on those already in use by XSEDE, such as Globus. XSEDE will also add additional services that address emerging needs, including data and computational services. The software and tools distributed by XSEDE will adhere to widely held community standards that will provide a foundation for a high degree of interoperability and compatibility among the CI community partners.

XCI is responsible for understanding the community infrastructure requirements in the form of use cases gathered by the XSEDE User Advisory Committee (UAC), XSEDE users via CEE, XD SPs, and commercial cloud service providers. XCI uses those requirements to identify existing tools and services that meet those requirements or identifies and evaluates new tools from the

18 community that do so. After testing those tools to ensure proper security and integration with existing XSEDE services and tools, they will be tested with the stakeholders that requested them to ensure they address the expressed needs. The tools and services will then be made available in the XCSR along with instructions on how to deploy them. XCI will work with CEE to promote the availability of these new capabilities and hold regular workshops and training to assist the community in their deployment. Feedback will be requested regularly on how well these capabilities are meeting or can be extended to better meet the requirements of the community.

XCI will create the XSEDE Community Software Repository (XCSR), a service and tool catalog available to the national community via the XSEDE website. This catalog will list all services and tools, which SPs have them installed, and links to the source code and/or installation packages along with documentation necessary to install and configure them. A list of all use cases, their stakeholders, and current status will also be cross-referenced with each service or tool. This information will inform discussions of priority and importance with stakeholders and the national community. All this information will be stored in the XCSR, a core deliverable and vehicle for our handoff strategy to the XSEDE successor(s) at the end of PY10.

XSEDE is moving from a direct support model to a subscription model. XCI will identify opportunities to leverage cloud providers for selected elements of service delivery in order to provide a sustainable and scalable approach for integrating critical services and tools into the ecosystem. We will also provide gateway-hosting services as part of the XSEDE organizational infrastructure—hosted within XSEDE on a server to be called XGH (XSEDE Gateway Hosting), based on a refresh of the existing Quarry Gateway Hosting. We expect that over time XSEDE will adopt more cloud-hosted services for its technical infrastructure. Rather than treating these services as part of XSEDE operations, we will take a peer-to-peer approach where XSEDE will interact, contract, and report on the value of such cloud-like infrastructure services as part of our community interaction activities.

Because return-on-investment will be a priority, we will also work with outside developers and software providers to instrument their tools so we can measure usage in a consistent way so as to ultimately feed into XDMoD—the portal by which we share resource and service usage information. Software as a Service (SaaS) providers such as Globus and Science Gateways will be required to provide usage data as well. It will not be possible or even practical to instrument all codes for usage tracking, but anything that requires a significant financial or personnel investment by XSEDE will be an important target. Working with the community, we will communicate this aspect as a critical part of developing community code. We will provide examples and workshops where possible to assist the community in this effort, or a request can be made for ECSS support.

2.3.1. Director’s Office The L2 Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the L2 area. This oversight includes providing direction

19 to the L3 management team, coordination of, and participation in, L2 planning activities and reports through the area’s Project Manager. The Director’s Office also attends and supports the preparation of project level reviews and activities.

2.3.2. Requirements Analysis & Capability Delivery (RACD) The Requirements Analysis and Capability Delivery (RACD) team prepares and supports software and services that: 1) enable user access to and use of XSEDE federated infrastructure, and 2) enable infrastructure and service providers to federate with XSEDE. Starting from XSEDE prioritized user requirements (use cases) RACD coordinates the engineering work necessary to integrate software and services into production at SPs and campuses, as XSEDE central services, as external vendor services, or on user personal systems. RACD uses engineering best practices and tools, works with external vendors and software partners to minimize integration cost to XSEDE, and aims to maximize ROI to XSEDE and the NSF. The RACD Engineering Overview can be viewed here on the XSEDE wiki: https://confluence.xsede.org/display/XT/WBS+2.3.2+RACD+Engineering+Overview

All of the software implementation, dissemination, and support will be carried out with a sense of “enabled by XSEDE,” rather than “created and branded XSEDE.” Software will be distributed to the XSEDE community through the XCSR. CI operators and the national user community will be enabled and encouraged to treat this repository much like a large menu—where people who manage a CI resource can select those tools that are relevant to the needs of their resource users and the purpose of their CI resource. XSEDE will recommend tools that are appropriate for use in a particular circumstance, or the sets of tools for particular CI provider groups (e.g., Level 1 SPs or campus resources). XCI will create and manage use case capability delivery plans. These plans may be put on hold if constraints limit XSEDE’s ability to deliver a capability, or there is insufficient ROI.

2.3.3. Capability & Resource Integration (CRI) The CRI team will manage and coordinate working with SPs and campuses to maximize the aggregate utility of national cyberinfrastructure. For SP integration, CRI will have an SP coordinator who will focus on XSEDE interactions with SPs and the SP Forum. CRI will also engage with other national CI organizations such as ACI-REF, EDUCAUSE, SURA, CASC, the Open Science Grid, and campus CI providers. These interactions will play a strong role in cost/benefit analyses and priority setting. They will inform and help all CI providers serving the U.S. research community understand each other’s needs and the needs faced by their users, promote best practices, and synergize provided services. By working with CRI, CI providers will gain a clear understanding of the costs and benefits of interoperability and interaction with XSEDE. CRI will extend and complement Campus Bridging activities in PY1-5 by establishing closer links with organizations that make use of these technologies and soliciting input and greater participation from these stakeholder organizations.

29

CRI will help CI providers of particular system types by creating “toolkits” within the XCSR that correspond to common usage modalities. The first of these toolkits will be the XSEDE National Integration Toolkit (XNIT). XNIT will include tools that can be installed on a campus cluster to promote interoperability with the national cyberinfrastructure, including XSEDE. XNIT will largely replace what is now called the XSEDE Compatible Basic Cluster (XCBC); however, we will maintain a Rocks distribution of the XCBC for those interested in new cluster installations. XNIT will include a “laptop suite” of tools that can be installed on a workstation or laptop computer at any site, with infrastructure and scientific software to enable researchers to interact effectively with the national cyberinfrastructure from their own personal system.

2.4. XSEDE Operations

Mission: XSEDE Operations installs, connects, maintains, secures, and evolves an integrated cyberinfrastructure that incorporates a wide range of digital capabilities to support national scientific, engineering, and scholarly research efforts. XSEDE Operations supports the project’s strategic goals with the following activities: ● Sustain the Ecosystem ○ Provide reliable and secure infrastructure ○ Provide excellent user support

XSEDE Operations will maintain and evolve an integrated CI capability of national scale, incorporating a wide range of digital capabilities to support the diverse national scientific and engineering research effort. XSEDE will provide first-class facilities, support, and services for users via improved technical capabilities and services, coordinated operation of distributed resources, an operations center, and highly accessible documentation. Operations will innovate by providing new insight and business intelligence in guiding decision making through expanded trend tracking and monitoring, and the analysis and dashboard visualization of operational data related to the ticket system, data transfers, and other collected operational information. XSEDE Operations will build upon the current operational successes with continued improvement based on XSEDE management guidance, advisory inputs, NSF review panel recommendations, and increased interactions with other CI providers (Blue Waters, NCAR, OSG).

2.4.1. Directors Office The L2 Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the L2 area. This oversight includes providing direction to the L3 management team, coordination of, and participation in, L2 planning activities and reports through the area’s Project Manager. The Director’s Office also attends and supports the preparation of project level reviews and activities.

2.4.2.Cybersecurity The Cybersecurity group will continue to expand and improve XSEDE security while minimizing impact on users and their productivity. To further increase awareness and rapid response to 21 threats, knowledge from individual sites will be aggregated and applied across all of XSEDE. A real-time intelligence sharing service for SPs will be deployed that will leverage the Research and Education Networking Information Sharing and Analysis Center (REN-ISAC) Collective Intelligence Framework (CIF) for exchanging attack intelligence. The CIF is an NSF-funded project to improve local protection against cyber threats by sharing security event information in near- real time. This real-time intelligence will feed into a system that will shunt traffic related to the IP addresses of bad actors such as password attackers or network scanners into a black hole network, thereby eliminating the threat. We will extend this service beyond XSEDE sites to include campus participants and operators of Science DMZs. We will further extend our intelligence system to perform cross-site analysis to look for scans, account attacks, and other suspicious activities that don’t reach thresholds at any one site but do trigger an alert when the same action is identified across multiple sites. This derived intelligence will then be shared with all participating sites. This has promise to transform our security ability to monitor and respond to a broad spectrum of attacks, with concomitant potential impact on the entire national CI ecosystem. The widespread deployment of virtual machine technologies including Docker and OpenStack highlights the critical need to understand cybersecurity best practices for these environments. Similar issues result from the adoption of public, private, and hybrid cloud services. We will develop best practices around these topics, document them, and aggressively work to disseminate this information. Outreach to campus CIOs and IT staff through the CEE Campus Engagements program, and Science DMZ operators will further include a collaborative effort working with ESnet, the Bro Center of Excellence, and the new Cybersecurity Center of Excellence with the goal of further documenting and training campus operators in security best practices. Finally, we will also develop specific training for security staff at XSEDE SPs to cover policy, process, controls, and best practices within XSEDE.

2.4.3. Data Transfer Services (DTS) Data Transfer Services (DTS) will focus on end-to-end data transfer performance, functionality, and efficiency in user workflows between instruments, resources, centers, and campuses. Where applicable, DTS will leverage emerging analytics capabilities and software defined networking (SDN) tools to improve performance, provide quality of service capabilities, and monitor network health and efficiency. The scope of these end-to-end efforts will include the Internet2 network that underpins the national XSEDEnet wide-area network, and, working in conjunction with site-local contacts, the data transfer nodes and local networks at XSEDE SP sites.

2.4.4. XSEDE Operations Center (XOC) The XSEDE Operations Center provides 24x7 helpdesk support via ticket system and call center, as well as 24x7 monitoring of critical services. The XOC resolves common problems, answers common questions, and routes other tickets and calls appropriately to WBS groups or XSEDE service providers, usually within several minutes. On the monitoring side, the XOC evaluates and documents any critical incidents, contacts the appropriate administrators, and remains a point of contact until the incident is resolved. 22

2.4.5. Systems Operational Support (SysOps) The SysOps group provides system administration and monitoring for all of the approximately 50 XSEDE centralized services. SysOps provides 24x7 monitoring and high availability for critical services for XSEDE, including geographically distributed backup and failover capabilities for enterprise services. SysOps will continue to employ server virtualization to control costs without sacrificing high availability.

2.5. Resource Allocations Service (RAS)

Mission: The Resource Allocations Service (RAS) connects users to resources that can help meet their science needs, using well-defined procedures and integrated infrastructure to ensure the most efficient and effective use of these limited resources. RAS supports XSEDE’s strategic goals with the following activities: ● Sustain the Ecosystem ○ Provide reliable, efficient, and secure infrastructure ○ Provide excellent user support

RAS will build on XSEDE’s current allo catio n pro cesses and evolve to meet the challenges presented by new types of resources to be allocated via XSEDE, new computing and data modalities to support increasingly diverse research needs, and large-scale demands from the user community for limited XSEDE-allocated resources. RAS will accomplish its objectives through three activities: 1. Carry out the NSF-approved allocation policies and manage the quarterly Research opportunities for large-scale allocation requests, including the associated meetings of the XSEDE Resource Allocations Committee (XRAC). The service also handles other allocation requests for Startups, Educational projects, transfers, extensions, and so on, coordinating all these activities with the Service Providers. 2. Maintain and improve the interfaces, databases and data transfer mechanisms for XSEDE- wide resource allocations, accounting of resource usage, and user account management. These systems include XDCDB, the XSEDE accounting system, and the XSEDE Resource Allocations Service (XRAS) 3. Analyze trends in the availability and use of resources, current technologies, computational science applications, and user requirements to inform project governance.

Supporting the XSEDE allocation and SP activities, RAS will also increase its analytics focus and mine the XSEDE Central Database (XDCDB) to document and project user demand for high-end CI resources. Such efforts will help SPs meet their award deliverables and provide NSF with data it can use to guide the direction of national CI investments. Coordinated within the RAS office of the director and working closely with the XD Metrics Service (providers of XDMoD) and XSEDE Evaluation Team, this analytics effort will investigate metrics-driven approaches to improving

23 allocations processes and policies (including NSF policies) to better meet user needs and steward national investments in CI. For example, we will conduct analyses leveraging user survey data, XDMoD usage reports, allocation requests and awards, and the XSEDE publications database to understand how users have adapted to the severe resource constraints over the past several years and to identify possible responses from XSEDE, the SPs, or NSF.

2.5.1. Director’s Office The L2 Director’s Office has been established to provide the necessary oversight to ensure the greatest efficiency and effectiveness of the L2 area. This oversight includes providing direction to the L3 management team, coordination of, and participation in, L2 planning activities and reports through the area’s Project Manager. The Director’s Office also attends and supports the preparation of project level reviews and activities.

2.5.2. XSEDE Allocations Process & Policies Allocations will oversee the central XSEDE allocation process, a major focus of RAS. The most visible aspect will be support for the quarterly review by the XSEDE Resource Allocation Committee (XRAC) of larger-scale research requests. The XRAC also serves as a key advisory board for RAS and the allocations process. The XSEDE allocation manager will oversee and direct the peer-review process for the largest research proposals (currently nearly 200 submissions per quarter are reviewed by XRAC members) along with the review by XSEDE staff experts of dozens smaller-scale requests. The RAS team recruits new XRAC members as terms expire, makes review assignments, ensures reviews are completed, manages the logistics of the quarterly meetings, and initiates the resulting awards.

The RAS team will support the review and handling of Startup, Education, and smaller-scale allocation requests. Furthermore, the RAS team will process requests, such as extensions, transfers and, to a lesser degree, supplements and appeals. It will coordinate the XSEDE resource guidance process, which leverages ECSS staff to help users identify the best resources for their projects prior to allocation request submission.

To ensure that the allocations process meets the needs of XSEDE stakeholders, the RAS team will engage regularly with the SP Forum, SP allocations representatives, the XSEDE User Advisory Committee, the NSF, and the XRAC on improvements or changes needed to the allocation policies. They will review processes to manage the growing volume of requests and to allocate effectively the evolving and diversifying resource portfolio, such as: human resources for ECSS projects, non-traditional computational resources and, we anticipate, network bandwidth or quality of service with software-defined networking. The XSEDE allocation manager will implement any agreed-upon changes to the XRAC review and meeting format

The XSEDE Allocations Policy is on the project wiki and can be found here: https://confluence.xsede.org/display/XT/XSEDE+Allocation+Policies

24

The XSEDE Allocations Procedure is also on the project wiki and can be found here: https://confluence.xsede.org/display/XT/XSEDE+Allocations+Procedure

2.5.3. Allocations CI Enhancement & Maintenance Allocations, Accounting & Account Management (A3M): The RAS team will augment efforts to support the XSEDE Resource Allocation Service (XRAS) allocation management software. Given the intense demand for XSEDE-allocated resources, the RAS team will update XSEDE infrastructure components to better support researchers and educators in identifying the appropriate and available services across the ecosystem that can support their objectives, a need identified both by existing XSEDE use cases and by an NSF workshop.

RAS efforts encompass improvement, maintenance, and operation of critical services that enable and enhance the allocations process. Led by L3 manager Amy Schuele (NCSA), the XRAS activities not only will facilitate the allocation of resources but will also enable discovery of and access to other services, such as the resources operated by Level 2 and Level 3 SPs but not allocated via XSEDE—better informing users about the range of advanced digital services available to them. XRAS efforts will follow and leverage the processes and tools, including JIRA for feature tracking, defined by XCI.

XRAS will be operated as a robust and stable service in support of XSEDE allocations processes and for client organizations. Support efforts will include maintenance updates to XRAS as XSEDE allocations policies evolve. Additional efforts will focus on enhancing the performance and reliability of the XRAS service and enhancing the reporting and metrics capabilities of the system. Because of the central and critical nature of XRAS to XSEDE, and in view of its anticipated value to the broader CI ecosystem, new XRAS features will be prioritized based on the collective inputs and feedback from users, XSEDE, the SPs, and other stakeholders. Finally, RAS will refine the sustainability and cost model for XRAS as a service. Building upon initial collaborations with NCAR and NCSA’s CADENS project, the team will work with other organizations that have expressed interest in the XRAS technology. The cost of work to support other organizations is not included in the XSEDE2 budget and will be covered by those organizations or through separate grants.

The Resource Description Repository (RDR), part of XSEDE’s Infrastructure Discovery Services, will support XRAS and serve as a cornerstone for an enhanced RAS infrastructure. The RDR will serve as the foundation for a “Resource Selector” service that will guide users to CI resources and services that are relevant and available to them for their research needs. RAS will also formally define and complete the integration of the XRAS, accounting, and XSEDE User Portal components with the enhanced RDR.

25

The Allocations, Accounting & Account Management (A3M) Services provide centralized mechanisms that integrate usage across XSEDE-allocated SPs and support allocation review and management. To accommodate emerging resources with novel usage modalities, RAS will collaborate with XCI to define accounting use cases and implement enhancements to the A3M environment. The Account Management Information Exchange (AMIE) messaging service is a critical integration mechanism for the A3M Services. In collaboration with XCI, RAS will conduct a trade-off analysis to investigate the impacts of migrating the Accounting Service from the legacy AMIE transport system to a modern, open-source messaging service. RAS, XSEDE Operations, and XCI will work together to evaluate the current infrastructure and status of the XDCDB and, if necessary, migrate the XDCDB to an even more robust, high-availability and high- performance configuration and platform, with appropriate backup and continuity plans and processes.

The XSEDE user publications database is essential to the XSEDE allocations process, to understanding XSEDE’s scientific impact, and to downstream XSEDE services such as gateways. This service will evolve to support RAS requirements in collaboration with CEE team efforts.

The RAS infrastructure will adapt and evolve as other components of the XSEDE infrastructure evolve, i.e. Identity Management and Infrastructure Discovery Services. Because the XDCDB is a primary data source for XDMoD, RAS will ensure that XDMoD remains integrated with the allocations and accounting infrastructure. Through its interactions with the SP Forum and other stakeholders, RAS will also assess interest in building a community around CI management tools that support XSEDE and RAS integration.

The RAS team will engage closely in the sustainability and transition plan from XSEDE to any successor program. The XDCDB encompasses a significant portion of the primary data products being generated and collected by XSEDE. These products will include allocation requests, review, and award information spanning more than two decades by the end PY10, more than a decade of system usage records spread over more than 50 resources, and user profile data including a database of tens of thousands of publications acknowledging XSEDE and SP support. Software maintained and enhanced by RAS, including XRAS, will be transitioned to a successor program according to procedures defined by XSEDE and the successor awardee(s).

2.6. Program Office

Mission: Ensure the critical project level functions are in place and operating effectively and efficiently and provide consistent guidance and leadership to the L2 directors and L3 managers across the project. The Program Office supports XSEDE’s strategic goals with the following activities: ● Deepen and Extend Use ○ Raise awareness of the value of advanced digital research services ● Sustain the Ecosystem

26

○ Operate an effective and productive virtual organization ○ Operate an innovative virtual organization ● Advance the Ecosystem ○ Enhancing the array of technical expertise and support services

The XSEDE Program Office will ensure the Project Office; External Relations; Project Management, Reporting, and Risk Management; Business Operations; and Strategic Planning, Policy, and Evaluation teams will effectively support XSEDE project activities and ensure efficient and effective performance of all project responsibilities. By tightly aligning organizational units (by L2 WBS) with strategic goals, the team will simplify accountability and link effort and budget to important outcomes. This approach is consistent with best practices in the management of virtual organizations, where many traditional managerial practices do not apply directly due to the distributed and knowledge-intensive nature of the work. Clearly delineated responsibilities and interfaces reduce uncertainty and enable the autonomy and discretion required for scalability and success. This structure will provide project-wide alignment and coordination while at the same time allowing each organizational unit the autonomy to adapt to the ever changing needs and provide the best service possible to the XSEDE users, service providers and the community in general.

2.6.1. Project Office The XSEDE Project Office will be led and managed by the University of Illinois’ NCSA with key partnerships instantiated via sub-awards. Illinois will ensure that an efficient and effective project governing structure is in place throughout the award period to support all significant project activities and ensure efficient and effective performance of all project responsibilities.

2.6.2. External Relations (ER) ER will promote the resources and services provided by XSEDE and examples of its successful support for science, engineering, and education to internal and external stakeholders. The ER team will communicate upcoming events, project milestones and achievements, science successes, services and resources, etc. via the XSEDE website and other channels; research and write “science success stories” and media releases; distribute a monthly external newsletter and a monthly internal newsletter; coordinate and staff an XSEDE exhibit at each year’s SC Conference; promote the annual XSEDE Conference and create supporting materials; and grow the XSEDE social media presence.

2.6.3. Project Management, Reporting & Risk Management (PM&R) The Project Management, Reporting and Risk Management (PM&R) team members have extensive experience applying project management principles to large, complex, distributed projects, including projects in the private sector, government, and XSEDE. As a focal point for XSEDE project management efforts, the PM&R team will develop and maintain an online Project Execution Plan (PEP) on the staff wiki. The PEP describes the standard operating procedures for the project and is a living document that evolves with the project.

27

Risk Management - Risk management is incorporated into the project at all WBS levels. The NCSA risk tool—originally developed for the Blue Waters project—will be used to register and monitor risks. Risk reviews will be conducted quarterly; high-risk items and mitigation strategies are included in the PEP.

Project Change Management - As part of the XSEDE annual planning process, the project will define the schedule, milestones, budget, and scope. The PM&R team will ensure that changes to these baselines will be managed through a change management process.

Project Reporting and Communications - The project will provide NSF with regular updates via teleconference and written quarterly and annual reports. The PM&R team will develop a Communication Plan that links all project groups and describes communication methods and frequencies to maximize the effectiveness and efficiency of project communications.

2.6.4. Business Operations The Business Operations group, working closely with staff at the University of Illinois’ Grants and Contracts Office (GCO) and National Center for Supercomputing Applications’ (NCSA) Business Office, will handle budgetary issues, manage sub-awards and assure timely processing of sub- award amendments and invoices.

2.6.5. Strategic, Planning, Policy, & Evaluation XSEDE will dedicate effort to project-wide strategic planning, policy development, evaluation and assessment, and organizational improvement in support of sustaining an effective and productive virtual organization. An independent Evaluation Team will be engaged to provide XSEDE with information to guide program improvement and assess the impact of XSEDE services. Evaluations will be based on five primary data sources: (1) an Annual User Survey that will be part of the XSEDE annual report and program plan; (2) an Enhanced Longitudinal Study encompassing additional target groups (e.g., faculty, institutions, disciplines, etc.) and additional measures (e.g., publications, citations, research funding, promotion and tenure, etc.); (3) an Annual XSEDE Staff Climate Study; (4) XSEDE KPIs, Area Metrics, and Organizational Improvement efforts, including ensuring that procedures are in place to assess these data; and (5) Specialized Studies as contracted by Level 2 directors and the Program Office. The Evaluation Team will create a database to support an Area Metrics/KPI Dashboard and results of any specialized studies.

This team is charged with the creation of a comprehensive performance management plan detailing the methodology, tools and data sources used to determine the performance of the XSEDE project and its impact on the research community. The initial release of this XSEDE Performance Management Plan is expected to be provided to the NSF six months after the award start date.

28

The detailed description of the project metrics and KPIs can be found at the following link: https://confluence.xsede.org/pages/viewpage.action?pageId=1671762

29 E. Project budget and staffing broken out by WBS element and by institution

Project Budget by WBS Element

Note: PY6 - PY10 terminology is used to differentiate between the initial XSEDE award (#1053575) and the follow-up award for an additional five years of operations.

30 Project Budget by Institution

*Note table updated April 11, 2019 to reflect the addition of Internet2 and Notre Dame as subawardees per Amendment #006.

F. Description of the methodology and assumptions used for estimating the budget components Cost estimates for this project include personnel, equipment, travel, and services required to perform the tasks necessary for completion of the project deliverables. These estimates reflect our knowledge of management and support costs gained from prior experience conducting projects of this complexity, scope, and magnitude. The start date of this project is assumed to be September 1, 2016

Personnel costs are based on actual salaries for current staff that are identified to work on the project. For new hires, estimates are based on the average fully loaded salary (that is, including fringe benefits and indirect costs) necessary to replace that individual’s experience and expertise at his/her institution. Estimates for goods and services are based on discussions with prospective vendors and are forward-looking.

G. Project risk analysis and a description of the analysis methodology 31 A structured, disciplined approach for risk management has been developed using the Project Management Institute’s best practices for risk management as a model. The XSEDE Project Director has overall responsibility for risk management. The XSEDE project maintains a risk register, which provides detailed information about each identified risk.

The risk management process, which must be ongoing and dynamic, ensures that: • risk identification and analysis have the appropriate rigor; • risk issues are made visible early; • thorough, credible mitigation plans are prepared/implemented; • budgets are maintained; • appropriate personnel are notified when a risk is triggered;

Project risk management consists of a six-step process: (1) identify potential vulnerabilities/risks; (2) determine the likelihood of occurrence; (3) assess the impact on the project scope, cost, and schedule baselines; (4) determine activities, alternatives, or contingencies that would reduce/mitigate/accommodate the risk; (5) execute a plan to accomplish these risk-reducing activities; and (6) report and track risk.

The project will use a risk management software application (the JIRA Risk Management Tool), which will help the project management team to record, track, and report on identified project risks.

The risk register will be updated regularly to reflect the modification to existing risks, addition of new risks, and retirement of risks as the project moves forward. The Project Director will conduct a formal risk review quarterly as part of a quarterly status meeting with the XSEDE project team in order to proactively address risks.

Identified risks can have positive as well as negative impacts on the project's technical scope, schedule, and cost. The project team will track opportunities in order to take full advantage of information for making decisions that might affect the project. In practice, if the XSEDE team detects a chance to save money by doing X instead of Y, then we record that as a "positive" risk, set triggers, and track it like other risks. The team may even have "mitigations" that increase the project’s chances that the opportunity occurs.

The project management will promptly inform NSF of any significant risk issues or opportunities that may arise during the project lifetime, and the risk register will be maintained for routine communication of potential project risks and mitigation strategies. These alerts will be contained in the conventional status reporting activities of the project where stakeholders are informed about any issues that may impact the project. Typically, these issues will be discussed during the regular teleconferences between NSF and XSEDE management. Significant risks will be documented in the required interim and annual reports. NSF can request a complete report of the risk register in advance of any of these events.

An initial risk assessment was completed during the planning period and has been documented on the wiki, which is located here: https://confluence.xsede.org/display/XT/Risk+Management. Going forward,

32 the risks will be moved to JIRA for management.

H. Project schedule The schedule is a living document and will be updated to reflect the baseline for near-term activities (work packages) as well as placeholders for long-term activities (planning packages). Overall the project begins on September 1, 2016 and ends on August 31, 2022.

I. Description of the organizational structure of the project team and governance of the project including advisory groups and the processes that facilitate interaction with all external entities

I.1. Project Governance XSEDE governance delegates decision-making authority to the greatest extent possible, allowing for timely decisions and greater agility in response to opportunities. Making use of the Work Breakdown Structure, which aligns with the organizational structure, each manager of a WBS area has decision-making authority within the scope, schedule, and budget of that WBS area. Decisions are escalated where other WBS areas or budget changes between partner institutions are involved.

The new organizational structure of the XSEDE project allows, in part, for improved communication and interaction between the XSEDE project and the NSF. A careful balance must be struck between the responsibility of the PI and co-PIs and the increased direct

33 responsibility of L2 Directors to NSF. To this end, weekly communications between the PI, Deputy Project Director, and the NSF CPO will involve the L2 Directors on a rotating basis to ensure clarity in their reporting responsibilities with respect to NSF. To maintain balance, the decision-making process will remain unchanged with the PI holding ultimate authority within the project. The Deputy Project Director will be largely responsible for offloading daily management and some project representation tasks from the PI. The XSEDE Program Manager will be the primary conduit into the project and will frequently communicate project status, risks, issues, accomplishments, and improvements. In addition, the Program Manager will seek direction and recommendations from the NSF on major changes and decision within the project.

XSEDE’s governance model emphasizes documenting activities and decisions and responding to stakeholder needs. XSEDE will continue to use its advisory boards and other input mechanisms, including outreach activities, user engagement efforts, and help requests, to assess stakeholder needs, to prioritize and define impact for these requests, and to ensure that they are implemented within the framework of existing XSEDE best practices. Governance and decision making within XSEDE are made public through the XSEDE Quarterly Reports, and to provide greater transparency in project governance, decisions and decision-making, and in addressing findings and recommendations of review panels, management bodies and advisory bodies, XSEDE will move to using a public project wiki.

As Project Director (PD), Towns with support from the Deputy Project Director (Boerner) will oversee the management of the project as a whole and will direct activities in the XSEDE Program Office. The Co-PIs and Senior Personnel will direct the activities of the Level 2 WBS areas. The XSEDE PI (Towns) and co-PIs (Gaither, Sinkovits, and Blood) hold ultimate authority and responsibility for successful program execution.

I.2. XSEDE Senior Management Team (SMT) The XSEDE Senior Management Team (SMT), the highest-level management body, will meet biweekly to assess project status, plans, and issues. It is chaired by the Project Director (PI Towns), and includes the Deputy Project Director (Boerner), the WBS Level 2 directors of Community Engagement & Enhancement (co-PI Gaither), the Extended Collaborative Support Service (Co-PI Sinkovits and Co-PI Blood), XSEDE Community Infrastructure (Lifka), XSEDE Operations (Peterson), the Resource Allocations Service (Hart) and the Program Office (Payne). In order to be responsive to both the user community and the set of collaborating SPs, the chairs of the User Advisory Committee (currently Emre Brookes, University of Texas Health Science Center at San Antonio) and the XD Service Providers Forum (currently Ruth Marinshaw, Stanford University) are members of the SMT. These eleven individuals constitute the voting members of the SMT. The Senior Project Manager (Froeschl) is an ex officio, non-voting member of the SMT. The cognizant NSF Program Officer (Chadduck) is also an ex officio, non-voting member. The XSEDE Senior Management team meets on a bi-weekly basis to assess project status, plans, and issues.

I.3. Advisory Bodies 34 Stakeholders will have input through three distinct advisory committees that have proved beneficial and will provide guidance on strategy, service, and support priorities for the community.

The XSEDE Advisory Board (XAB) meets semi-annually, either in person or by teleconference, to help ensure that XSEDE is designed to impact a broad range of disciplines, enable both research and education, have broader impacts to society, and have a user community that is diverse (gender, ethnic background, etc.) and includes representation from all types of colleges and universities. The XAB advises in the annual planning process, reviews the annual plans, and recommends strategic directions. While primarily strategic, the XAB may make tactical recommendations that help XSEDE.

The XAB consists of five scientific leaders (selected by the XSEDE management team) from different communities who use XSEDE along with the chair of the UAC and three representatives from the SPF (the SPF chair and two others, self-selected by the Forum). These members are complemented by an additional five senior members of the broader community selected by the XSEDE management team. Members serve two-year staggered terms.

The Chair of the XAB is responsible for ensuring that the XAB meets quarterly and reports back to the XSEDE Senior Management Team the results of the meeting, including any recommendations and/or action items that need attention by XSEDE and the timescale for the action. The Chair will be self- selected by the XAB membership. The selection process will be discussed with NSF. NSF will be kept apprised of candidates and the decision process. Comments from the NSF CPO will be considered throughout the selection process. The XAB chair nomination is expected to be provided to the NSF in the Fall of 2016.

The User Advisory Committee (UAC) will meet twice or three times per year by teleconference and consists of 20 active users of XSEDE-allocated resources and services representing the needs and concerns of the community. The committee presents recommendations regarding emerging needs and will review plans and suggested developments. XSEDE will seek input from NSF directorates to include researchers representing each NSF directorate or major division. Members serve two-year staggered terms. The chair, selected by UAC members, will participate in SMT meetings and is a member of the XAB. In addition, they will have the role of User Ombudsperson—the person to whom any user of any XSEDE allocated or supported resource or service can turn if they are not having issues addressed by XSEDE.

The Chair of the UAC is responsible for ensuring that the UAC meets twice or three times per year and reports back to the XSEDE Senior Management Team the results of the meeting, including any recommendations and/or action items that need attention by XSEDE and the timescale for the action. The Chair will be self-selected by the UAC members.

The XD Service Providers Forum (SPF) meets bi-weekly and provides a means by which all Service Providers can voice concerns, make recommendations, and provide feedback on proposed changes to the XSEDE environment, policies, and services. The SPF is more fully defined in the XD Service Providers Forum Charter and the Requesting Membership in the XSEDE Federation as a Service 35 Provider documents, both available online (www.xsede.org/project-documents). The Service Provider On-boarding checklist is located on the wiki at: https://confluence.xsede.org/download/attachments/1671610/XSEDE_ServiceProvider-Checklist- v2.0.docx?version=1&modificationDate=1468433493813&api=v2

The Chair of the UAC is responsible for ensuring that the UAC meets twice or three times per year and reports back to the XSEDE Senior Management Team the results of the meeting, including any recommendations and/or action items that need attention by XSEDE and the timescale for the action. The Chair will be self-selected by the UAC members and serve a one-year term.

Based on feedback from staff during the preceding award, we will also form an Internal Advisory Committee (IAC) to give advice on internal matters such as professional development, reporting, recognition, policies, etc. This committee will be defined in conjunction with the staff to establish a committee responsive to staff needs.

J. Description of the sub-contracting strategy and controls All XSEDE procurements will follow the policies of the XSEDE partner institution. For all purchases made via the University of Illinois, procurements will follow procedures and rules of the University of Illinois Purchasing Office, which are available on their website at: https://www.obfs.uillinois.edu/purchases/procedures-rules/

J.1. Capital Items

Only the project director may approve the purchase of capital equipment that is part of the XSEDE project. Changes to the capital procurement plan may only be made as allowed by the NSF, available funding, and the approval of the project director.

J.2. Sub-awards All sub-awards will contain a statement of work (SOW), budget in NSF Form 1030 format, and budget justification, all of which are submitted through the Sponsored Research Office of the sub- award institution. The sub-awards will include an executive summary, milestones, deliverables, payment schedules, and the acceptance and certification criteria for payment. Contractual terms in the NSF cooperative agreement with the University of Illinois/NCSA will flow down to sub-awardees. Sub- awardees will submit detailed invoices for payment to NCSA monthly, unless another payment schedule has been identified in their contracts.

J.3. Consultants The project director will determine the need, scope, and timing of any consultant services in support of the XSEDE project and will direct the NCSA finance office to obtain the services under the University of Illinois procurement process.

J.4. Other Purchases XSEDE staff may purchase expensed equipment (laptops, cell phones), supplies, and other goods 36 and services when submitted and approved as part of the materials and supplies portion of the annual budget submission. Purchases of alcohol, business meals, personal gifts, and other like items are prohibited, unless approved in advance by the project director and only if allowed under the University of Illinois’ policies regarding such items. See Section 8.12 and Section 8.13 of the OBFS Business and Financial Policies and Procedures Manual (https://www.obfs.uillinois.edu/purchases/procedures-rules/).

K. Description of the financial and business controls to be used NCSA will manage the project funds in accordance with Illinois rules and procedures under the day- to-day direction of the NCSA Finance division director. The University of Illinois business procedures are found in its OBFS Policies and Procedures Manual (https://www.obfs.uillinois.edu/purchases/procedures-rules/).

A budget plan will be established and updated annually. Expenditures will be planned and actual expenses reconciled monthly with the University’s enterprise accounting system, down to Level 3 in the WBS.

Budgets and actual costs will be collected in financial accounts, which correspond with the WBS structure of the project in the Illinois financial system. Elements of costs will also be maintained so that totals for effort, equipment purchases, and other cost categories can be tracked across all WBS elements. Each level 3 WBS manager will be responsible for the charges incurred for their WBS and be responsible for remaining within the budget allocated for their work. The cost incurred at each partner institution will be billed to Illinois and reviewed by the Project Director or Deputy Project Director and the Illinois finance officer. The Project Director or Deputy Project Director, with assistance from the NCSA Finance Division and the Business Operations WBS Level 2 lead will be responsible for reporting project financial information to NSF as required.

L. Plan for reporting on the technical and financial status of the project The XSEDE project will provide interim project and annual reports as designated by the NSF cognizant Program Official with content, format, and submission time line established by the NSF cognizant Program Official. The XSEDE project will submit all required reports via Research.gov using the appropriate reporting category; for any type of report not specifically mentioned in Research.gov, the XSEDE project will use the Interim Reporting function to submit reports.

The interim project report will include monthly expenditures per the NSF 1030 format and by work breakdown structure (WBS) level 2 both per institution and across the project as a whole. Planned versus actual expenditures will be indicated. It should also include detailed descriptions of the progress, achievements, and expenditures of the sub-awardees. Each report must also include relevant performance data.

The Annual Report and Program Plan will also include a detailed plan for the following year and, if necessary, an update to the Project Execution Plan.

37

Reporting schedule:

M. Description of anticipated safety or health issues associated with the project, if any No health or safety issues are expected in the XSEDE project. Nonetheless, a component of any successful project is to ensure that environment, safety, and health issues are addressed early in a project’s life cycle and fully integrated into all project activities. The project team is committed to providing a safe work environment for all workers and the public. The project team will follow all relevant and applicable safety laws and procedures required by Illinois and the other partner institutions.

N. Cyber security plan for protecting the confidentiality, integrity and availability of XSEDE resources and services XSEDE cybersecurity must support the confidentiality, availability and integrity of XSEDE and XSEDE- allocated resources by: following best practices, employing risk-based approaches, fostering 38

teamwork throughout the XSEDE team, and integration of new proven cybersecurity technologies, procedures and approaches. The following sections document the XSEDE Cybersecurity Program Plan (CSPP), a comprehensive cybersecurity program for this distributed cyberinfrastructure. Rising to the top of this list are a number of strategies that include: 1. Support for a strong authentication and authorization service that limits access to only legitimate XSEDE users, 2. Coordination of the XSEDE cybersecurity staff among contributing XD and campus Service Providers to develop policies, design secure architectures and review risks, 3. Coordinated incident response and intelligence sharing across Service Providers, trusted partners and other federations, 4. A strong XSEDE cybersecurity education program, and 5. Proactive cybersecurity through careful risk/threat analysis, design and architecture of XSEDE at every level. Cybersecurity in a highly distributed environment such as XSEDE is built upon the social networking and trust relationships honed over time among the partner cybersecurity staff. During PY6-10, XSEDE cybersecurity will build and improve upon a well-established community of security professionals and many of the successes of the XSEDE PY1-5 cybersecurity program. Successes during PY1-5 include the formation of an incident response team for the coordination of incident response across XSEDE sites and a broadened deployment of a cybersecurity architecture across the growing XSEDE user base. XSEDE will expand the cybersecurity team and draw on the expertise and experience of new individuals.

Additional information can be found in the XSEDE PEP Supplement: Cybersecurity Plan, which will be maintained on the XSEDE Staff Wiki here: https://confluence.xsede.org/download/attachments/1671245/XSEDE-PEP-CybersecurityPlan- v2.0.pdf?version=1&modificationDate=1468255825449&api=v2

O. Comprehensive performance management plan which supplies reporting data The XSEDE project currently uses a variety of metrics and KPIs to measure performance against defined strategic goals. While these measures represent the performance of the XSEDE organization and its service areas, they do not adequately represent the impact the XSEDE project has on the research community and science in general. The Strategic Planning, Analysis, and Evaluation team, is charged with the creation of a comprehensive performance management plan detailing the methodology, tools and data sources used to determine the performance of the XSEDE project as well as its impact on the research community. The initial release of this XSEDE Performance Management Plan is expected to be provided to the NSF six months after the award start date.

In addition, an XSEDE Metrics Dashboard will be created to provide access to current and historical metrics and KPIs. This dashboard can be viewed at https://www.xsede.org/web/xsede- metrics/metrics.

39

The detailed description of the metrics and KPIs can be found at the following link: https://confluence.xsede.org/pages/viewpage.action?pageId=1671762

P. Description of Project Policies and Standard Operating Procedures All XSEDE project policies and standard operating procedures are maintained on the XSEDE Staff Wiki. Any changes to these policies and standard operating procedures will be reflected on the XSEDE Staff Wiki located here: https://confluence.xsede.org/pages/viewpage.action?pageId=1671610

40