XSEDE QUARTERLY REPORT Contents 1 XSEDE – Enabling New Discoveries

XSEDE eXtreme Science and Engineering Discovery Environment

2013 Q1: January 1, 2013, through March 31, 2013

XSEDE Senior Management Team

John Towns (NCSA) PI and Project Director Jay Boisseau (TACC) Co-PI and User Services Director Ralph Roskies (PSC) Co-PI and Extended Collaborative Support Service-Projects Director Nancy Wilkins-Diehr (SDSC) Co-PI and Extended Collaborative Support Service-Communities Director Greg Peterson (UT-NICS) Co-PI and Operations Director Scott Lathrop (Shodor) Education and Outreach Director Kathlyn Boudwin (ORNL/NICS) Manager of Project Management Janet Brown (PSC) Manager of Systems and Software Engineering JP Navarro (UChicago) Manager of Software Development and Integration Tom Cheatham (Utah) Chair of the User Advisory Committee Carol Song (PU) Chair of the XD Service Providers Forum

i XSEDE QUARTERLY REPORT Contents 1 XSEDE – Enabling New Discoveries ...... 11 1.1 Project Context ...... 11 1.1.1 Communities Served 12 1.1.2 XSEDE’s Integrated, Distributed Environment 12 1.2 Project Highlights ...... 12 2 Science and Engineering Highlights ...... 15 2.1 Turbulence Modeling: Droplet-laden Isotropic Turbulence (Antonino Ferrante, William E. Boeing Department of Aeronautics and Astronautics, University of Washington, Seattle) . 15 2.2 Seismology: Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers (Yifeng Cui, San Diego Supercomputer Center, UC San Diego) ...... 16 2.3 Molecular Dynamics Simulation of Assembling Behavior of a Triplex Complex Formed by Amylose, Beta-lactoglobulin, and Alpha-linoleic acid. (Bruce Hamaker and Osvaldo Campanella, Whistler Center for Carbohydrate Research, Purdue University) ...... 17 3 Delivering and Supporting New Capabilities in an Evolving Environment ...... 19 3.1 Architecture for a National Distributed Cyberinfrastructure Ecosystem (Architecture and Design 1.1.3) ...... 19 3.1.1 Use Case Development 19 3.1.2 XWAVE and XUAS Security Component Active Design Reviews 19 3.1.3 Architecture and Design Use Case Tracking System 20 3.1.4 Accomplishments this Quarter 20 3.1.5 Planned Activities 20 3.2 Realizing the XSEDE Architecture (Software Development and Integration 1.1.6) ...... 21 3.3 XSEDE Operations 1.2 ...... 22 3.3.1 Overview 22 3.3.2 Cybersecurity 1.2.1 23 3.3.3 Data Services 1.2.2 24 3.3.4 XSEDEnet 1.2.3 25 3.3.4.1 XSEDEnet and transition to Internet2 ...... 25 3.3.4.2 PerfSONAR ...... 26 3.3.5 Software Support 1.2.4 28 3.3.6 Accounting and Account Management 1.2.5 29 3.3.7 Systems Operational Support 1.2.6 29 3.3.7.1 XSEDE Operations Center ...... 30

ii 3.3.7.2 Central Services ...... 31 3.3.7.3 INCA ...... 31 3.3.7.4 Syslog Monitoring Project ...... 32 3.3.7.5 Globus Online ...... 32 3.4 Technology Investigation Service 1.7 ...... 33 3.4.1 Overview 33 3.4.2 1.7.1 Technology Identification 33 3.4.3 1.7.2 Technology Evaluation 34 4 Supporting and Expanding the Community ...... 35 4.1 User Services 1.3 ...... 35 4.1.1 Overview 35 4.1.2 Training 1.3.1 35 4.1.3 User Information & Interfaces 1.3.2 35 4.1.4 User Engagement 1.3.3 36 4.1.4.1 Feedback ...... 36 4.1.4.2 Consulting ...... 36 4.1.5 Allocations 1.3.4 36 4.2 Extended Collaborative Support Services ...... 37 4.2.1 Extended Collaborative Support Service – Projects 1.4 37 4.2.1.1 Extended Research Teams Support 1.4.1 ...... 38 4.2.1.2 Novel and Innovative Projects 1.4.2 ...... 43 4.2.2 Extended Collaborative Support Service – Communities 1.5 45 4.2.2.1 Overview ...... 45 4.2.2.2 Extended Support for Community Codes 1.5.1 ...... 46 4.2.2.3 Extended Science Gateways Support 1.5.2 ...... 50 4.2.2.4 Extended EOT Support (WBS 1.5.3) ...... 53 4.3 Education and Outreach ...... 53 4.3.1 Education 1.6.1 54 4.3.1.1 Assisting with the Creation of Formal Programs ...... 54 4.3.1.2 Presentations at National Meetings ...... 54 4.3.1.3 New Course Development ...... 54 4.3.1.4 Participation in Outreach and Training Events ...... 54 4.3.1.5 Creation of Repository of Education Materials ...... 55 4.3.1.6 Other Activities ...... 55 4.3.2 Outreach 1.6.2 56

iii 4.3.2.1 Under-represented Engagement ...... 56 4.3.2.2 Speakers Bureau ...... 57 4.3.2.3 Student Engagement ...... 57 4.3.2.4 Campus Champions ...... 58 4.3.3 E&O Community Requirements and External Evaluation 1.6.3 60 4.3.3.1 Community Requirements Highlights ...... 60 4.3.3.2 External Evaluation Highlights ...... 60 4.3.4 E&O Infrastructure 1.6.4 63 4.3.5 Campus Bridging 1.6.5 63 4.3.5.1 Discussion of campus bridging within XSEDE and within the national cyberinfrastructure community ...... 63 4.3.5.2 Definition of Use Cases and quality attributes for XSEDE’s Campus Bridging efforts 63 4.3.5.3 GFFS Pilot Project ...... 63 4.3.5.4 Rocks Roll Software Project ...... 63 4.3.5.5 Challenges in Program Year Two ...... 64 5 Managing the Program ...... 65 5.1 Governance ...... 65 5.2 XSEDE Project Office 1.1 ...... 65 5.2.1 Project Management and Reporting 1.1.1 66 5.2.1.1 Project Schedule and Risk Register ...... 66 5.2.1.2 Project Management Software Tool ...... 66 5.2.1.3 Reporting ...... 66 5.2.1.4 Quarterly Meeting ...... 66 5.2.1.5 Other Topics ...... 67 5.2.2 Systems and Software Engineering 1.1.2 67 5.2.2.1 Gathering New User Needs and Capabilities ...... 67 5.2.2.2 Use Case Development ...... 67 5.2.2.3 XSEDE Digital Object Repository ...... 67 5.2.3 External Relations 1.1.4 67 5.2.4 Industry Relations 1.1.5 68 6 TAIS/Audit Services 1.8 ...... 69 6.1 10.8 XDMoD Usage ...... 82 7 ExTENCI ...... 85 7.1 ExTENCI – Distributed File System (Lustre-WAN) ...... 85

iv 7.2 ExTENCI – Virtual Machines ...... 85 7.3 ExTENCI – Workflow & Client Tools ...... 86 7.4 ExTENCI – Job Submission Paradigms ...... 86 8 XD Service Provider Reports ...... 87 8.1 Overview...... 88 9 XSEDE Quarterly Report: FutureGrid Service Provider (January 1, 2013 – March 31, 2013) ...... 90 9.1 Executive Summary ...... 90 9.1.1 Resource Description 90 9.2 Science Highlights ...... 91 9.3 User-facing Activities ...... 94 9.3.1 System Activities 94 9.3.1.1 Hardware ...... 94 9.3.2 Services Activities (specific services are underlined in each activity below) 94 9.3.2.1 Accounting ...... 94 9.3.2.2 Cloud Services and Support Software ...... 95 9.3.2.3 Experiment Management ...... 96 9.3.2.4 Information Services ...... 97 9.3.2.5 Performance ...... 97 9.3.2.6 FutureGrid Portal ...... 99 9.4 Security ...... 99 9.5 Education, Outreach, and Training Activities...... 99 9.6 SP Collaborations ...... 101 9.7 SP-Specific Activities ...... 101 9.8 Publications ...... 101 9.9 Metrics ...... 102 9.9.1 Standard systems metrics 102 9.9.2 Standard systems metrics (continued) 103 9.9.3 Standard systems metrics (continued) 104 9.9.4 Standard systems metrics (continued) 105 9.9.5 105 9.9.6 Standard systems metrics (continued) 106 9.9.7 Standard User Assistance Metrics 107 9.9.8 SP-specific Metrics 108

v 10 Indiana University Pervasive Technology Institute - Service Provider Quarterly Report 110 10.1 Executive Summary ...... 110 10.1.1 Indiana University Level 2 Service Provider Systems: Resource Descriptions 110 10.2 Science Highlights ...... 111 10.2.1 National Center for Genome Analysis and Support (NCGAS) attends Plant and Animal Genome Conference (PAG) XXI 111 10.3 User-facing Activities ...... 111 10.3.1 System activities 111 10.3.1.1 Level 2 – Quarry (virtual machines) ...... 111 10.3.1.2 Level 3 (pending) – Rockhopper (Penguin on Demand commercial cluster as a service) 111 10.3.2 Services activities 111 10.3.2.1 Level 2 – Quarry ...... 111 10.3.2.2 Level 3 (pending) – Rockhopper ...... 112 10.4 Security ...... 112 10.5 Education, Outreach, and Training Activities...... 112 10.6 SP Collaborations ...... 113 10.7 SP-Specific Activities ...... 113 10.8 Publications ...... 120 10.9 Metrics ...... 121 10.9.1 Standard User Assistance Metrics 121 10.10 User Information and Interfaces (WBS 1.3.2) ...... 121 10.10.1 SP-specific Metrics 123 10.10.1.1 Usage metrics – current quarter ...... 123 10.10.1.2 Usage metrics – project year 2 ...... 123 10.10.2 Standard systems metric 124 11 Georgia Tech, Keeneland - Service Provider Quarterly Report ...... 125 11.1 Executive Summary ...... 125 11.1.1 Resource Description 125 11.2 Science Highlights ...... 125 11.3 User-facing Activities ...... 126 11.3.1 System Activities 126 11.3.2 Services Activities 127 11.4 Security ...... 127 11.5 Education, Outreach, and Training Activities...... 127

vi 11.6 SP Collaborations ...... 128 11.7 SP-Specific Activities ...... 128 11.8 Publications ...... 128 11.9 Metrics ...... 129 11.9.1 Standard User Assistance Metrics 129 11.9.2 SP-specific Metrics 130 11.9.3 Standard systems metrics 130 12 NICS - Service Provider Quarterly Report ...... 131 12.1 Executive Summary ...... 131 12.1.1 Resource Description 131 12.1.1.1 Kraken ...... 131 12.2 Science Highlights ...... 132 12.2.1.1 Chemical Biology: Conformational Changes of Cancer-suppressor and HIV- promoting Proteins During Cell-binding (Hirsh Nanda, Department of Physics, Carnegie Mellon University, and the National Institute of Standards and Technology Center for Neutron Scattering) ...... 132 12.2.1.2 Aeronautics and astronautics: Turbulence Modeling: Droplet-laden Isotropic Turbulence (Antonino Ferrante, William E. Boeing Department of Aeronautics and Astronautics, University of Washington, Seattle) ...... 132 12.2.1.3 Energy Efficiency: Autotune — A Calibration Methodology for Energy Modeling (Joshua New and Jibo Sanyal, Building Technologies Research and Integration Center, Oak Ridge National Laboratory) ...... 132 12.2.1.4 Evolutionary Biology: Protein Translation and Codon Usage Bias (Michael Gilchrist, Department of Ecology and Evolutionary Biology, University of Tennessee, Knoxville) 133 12.2.1.5 Theoretical and Computational Astrophysics: Turbulence in the Outer Regions of Protoplanetary Disks (Jacob Simon, University of Colorado, Boulder) ...... 133 12.2.1.6 Astrophysics: Alignment of Magnetized Accretion Disks and Relativistic Jets with Spinning Black Holes (Jonathan McKinney, University of Maryland, College Park; Alexander Tchekhovskoy, Princeton Center for Theoretical Science; Roger Blandford, Kavli Institute for Particle Astrophysics and Cosmology, Stanford University) ...... 135 12.3 User-facing Activities ...... 136 12.3.1 System Activities 136 12.3.1.1 Kraken ...... 136 12.3.1.2 Nautilus ...... 136 12.3.2 Services Activities 137 12.3.2.1 Kraken ...... 137 12.3.2.2 Nautilus ...... 137 12.4 Security ...... 138

vii 12.5 Education, Outreach, and Training Activities...... 138 12.6 SP Collaborations ...... 139 12.7 SP-Specific Activities ...... 140 12.8 Publications ...... 141 12.8.1 User Publications 141 12.8.2 Staff Publications 144 12.9 Metrics ...... 145 12.9.1 Standard systems metrics 145 12.9.2 Kraken 146 12.9.2.1 Nautilus ...... 151 12.9.3 Standard User Assistance Metrics 156 12.9.4 SP-specific Metrics 157 13 Pittsburgh Supercomputing Center - Service Provider Quarterly Report ...... 160 13.1 Executive Summary ...... 160 13.1.1 Resource Description 161 13.2 Science Highlights ...... 163 13.2.1 Recognition and Indexing of Hand-Written Documents: Digitizing the 1940 Census (Kenton McHenry, National Center for Supercomputing Applications) 163 13.3 User-facing Activities ...... 164 13.3.1 System Activities 164 13.3.2 Services Activities 164 13.4 Security ...... 166 13.5 Education, Outreach, and Training Activities...... 166 13.6 SP Collaborations ...... 168 13.7 SP-Specific Activities ...... 168 13.8 Publications ...... 168 13.9 Metrics ...... 169 13.9.1 Standard User Assistance Metrics 169 13.9.2 SP-specific Metrics 169 13.9.3 Standard systems metrics 170 14 Purdue University - Service Provider Quarterly Report ...... 176 14.1 Executive Summary ...... 176 14.1.1 Resource Description 176 14.2 Science Highlights ...... 177 14.3 User-facing Activities ...... 180

viii 14.3.1 System Activities 180 14.3.2 Services Activities 181 14.4 Security ...... 181 14.5 Education, Outreach, and Training Activities...... 182 14.5.1 EOT Events 182 14.5.2 Education 182 14.6 SP Collaborations ...... 182 14.7 SP-Specific Activities ...... 183 14.8 Publications ...... 183 14.9 Metrics ...... 184 14.9.1 Standard User Assistance Metrics 184 14.9.2 SP-specific Metrics 184 14.9.3 Standard systems metrics 185 15 San Diego Supercomputer Center (SDSC) Service Provider Quarterly Report ...... 196 16 196 16.1 Executive Summary ...... 196 16.1.1 Resource Descriptions 196 16.2 Science Highlights ...... 197 16.3 User-facing Activities ...... 198 16.3.1 System Activities 198 16.3.2 Services Activities 199 16.4 Security ...... 200 16.5 Education, Outreach, and Training Activities...... 200 16.5.1 Training 202 16.5.2 Education 202 16.6 SP Collaborations ...... 203 16.6.1 Collaborations with SP XSEDE Users 203 16.6.2 Collaborations with External Partners 203 16.7 SP-Specific Activities ...... 204 16.8 Publications ...... 204 16.9 Metrics ...... 205 Appendix 15.9A Standard User Assistance Metrics 206 Appendix 15.9C Trestles SP-specific Metrics 227 A XSEDE Project Milestones Update ...... 230 B XSEDE Schedule with Progress Update ...... 231

ix C XSEDE Risk Register ...... 269 D XSEDE Change Control Report ...... 276 E Metrics...... 277 E.1 XSEDE Resource and Service Usage Metrics ...... 277 E.2 XSEDE Program Metrics ...... 287 F XSEDE Publications Listing ...... 299 F.1 XSEDE Staff Publications ...... 299 F.2 Publications from XSEDE Users ...... 299 G XSEDE EOT Event Details ...... 334

1 XSEDE – Enabling New Discoveries XSEDE accelerates open scientific discovery by enhancing the productivity of researchers, engineers, and scholars and making advanced digital resources easier to use. The Extreme Science and Engineering Discovery Environment (XSEDE) is a virtual organization that provides a dynamic distributed infrastructure, support services, and technical expertise that enable researchers, engineers, and scholars to address the most important and challenging problems facing the nation and world. XSEDE supports a growing collection of advanced computing, high- end visualization, data analysis, and other resources and services. XSEDE’s goals are to:  Deepen and extend the impact of eScience infrastructure on research and education—in particular, to reach communities that have not previously made use of it;  Prepare the current and next generation of researchers, engineers, and scholars in the effective use of advanced digital technologies;  Collaborate with institutions to ensure a more seamless use of the advanced technology capabilities in the national eScience infrastructure to enhance the productivity of researchers, engineers, and scholars;  Create an open and evolving environment that facilitates integration and sharing of heterogeneous digital services into a comprehensive national eScience infrastructure.  Expand the environment through the integration of new capabilities and resources, such as instruments and data repositories, based on the identified needs of the community.  Deepen and expand the array of technical expertise and support services provided to the community to maximize the effectiveness of their use of the eScience infrastructure;  Raise awareness of the value of eScience infrastructure and, in particular, the critical technical expertise and support services.

1.1 Project Context Scientists, engineers, social scientists, and humanities experts around the world—many of them at colleges and universities—use advanced digital resources and services every day. Computational technologies and resources such as supercomputers, visualization systems, storage systems, and collections of data, software, and networks are critical to the success of those researchers, who use them to address the most important and challenging problems facing the nation and world. XSEDE integrates these resources and services, makes them easier to use, and helps more people use them. Digital services, meanwhile, provide users with seamless integration to NSF's high-performance computing and data resources. XSEDE's integrated, comprehensive suite of advanced digital services is developing and implementing tools, methods, and policies to federate with other high- end facilities and with campus-based resources, serving as the foundation for a national eScience ecosystem. Common authentication and trust mechanisms, global namespace and filesystems, remote job submission and monitoring, and file transfer services are examples of XSEDE's advanced digital services. XSEDE's distributed systems architecture allows open development for future digital services and enhancements. XSEDE also provides the expertise to ensure that researchers can effectively use the supercomputers and tools. Those include:  Extended Collaborative Support that includes teaming with individual research groups or with research communities to extend their capabilities.

11  An advanced hardware and software architecture rooted in user requirements and hardened by systems engineering that allows for individualized user experiences, consistent and enduring software interfaces, improved data management, and ways for campus resources to be transparently integrated into the overall XSEDE infrastructure.  The XSEDE User Portal, a web interface that allows users to monitor and access XSEDE resources, manage jobs on those resources, report issues, and analyze and visualize results.  Coordinated allocations of NSF's high-end resources and digital services.  A powerful and extensible network, in which each XSEDE service provider is connected to a Chicago-based hub at 10 gigabits per second and has a second 10 gigabit-per-second connection to another national research and education network.  Specialized community-provided services that serve a particular function and allow for rapid innovation and experimentation.  Advanced cybersecurity to ensure that XSEDE resources and services provide confidentiality, integrity and availability of information  Training, Education, and Outreach efforts that expand the scope and scale of activities to foster greater community participation in XSEDE-based projects through curriculum development, live and web-based training offerings, outreach at professional society meetings, and engagement of under-represented faculty and students.  Advanced support for novel and innovative projects.  A fellowship program that enables Campus Champions to work closely with Extended Collaborative Support Service staff on user-identified challenges for up to a year.  The Technology Investigation Service, which allows researchers to recommend technologies for inclusion in the XSEDE infrastructure and enables the XSEDE team to evaluate those technologies and incorporate them where appropriate. 1.1.1 Communities Served The national, and global, user community that relies on XSEDE for HPC resources has grown tremendously. XSEDE continued to see increased HPC resource user numbers in Q1 2013, with an atypically high 1,339 new users added in the quarter. For the first time the number of open individual accounts exceeded 7,000, and the number of non-gateway individuals charging jobs was also a new record at 2,342. An additional 1,528 users submitted jobs via science gateways— that is, nearly 40 percent of XSEDE users submitting jobs worked through gateways. Counting current individual accounts and gateway users, the XSEDE community numbered 8,570 users. Further details can be found Appendix E. 1.1.2 XSEDE’s Integrated, Distributed Environment XSEDE is taking on the difficult but necessary task of documenting a clearly specified architectural design for its distributed systems architecture. Given the nature of the end game of the proposal competition that ultimately resulted in the XSEDE award, the project has had to substantially redesign the architecture originally proposed in order to incorporate innovative and important elements of the previously competing proposal. While this has been difficult and has led to some confusion, the project is making progress in this area and will begin to produce design documents that specify the architecture in detail during the coming months. 1.2 Project Highlights XSEDE has begun to really hit its stride across a wide range of activities; in many areas the project has already met annual goals. This is clearly evidenced by the regular reporting of science and engineering successes XSEDE has supported and enabled. In Section 2 of this report you will note a few high-impact science and engineering successes. In this quarter we began planning the

12 Annual Highlights publication, which showcases some of the most impressive examples of XSEDE’s impact. XSEDE continues its high level of support of researchers with our Extended Collaborative Support Services (ECSS) staff engaged in 118 active projects covering a variety of areas during this past quarter. In response to the annual user survey, which indicates only 28 percent of users were aware of ECSS, the External Relations team has been working to publicize ECSS more widely. We have made progress on a capability to allow users to post their own brief write-ups of achievements enabled by XSEDE, which will be available to them early next quarter. The use of science gateways continues to grow. XSEDE User Services continues to provide high-quality support for the community in a variety of ways. The XSEDE Resource Allocations Committee awarded 384M SUs (out of 796M requested) across XSEDE systems at the first allocations meeting conducted since Stampede came into production as an XSEDE-allocated system. Training classes—online and in-person— had more than 14,000 participants, and plans for training certificates are nearing completion. The user engagement activities included completing the second annual user survey (with better response rate than the first), for which results are now being analyzed, and analyzing old tickets to identify the top areas of operations and support to address for the XSEDE user community. The user engagement team is working with the operations team to complete the transition to a new user support ticket system in 2Q13 that will greatly enhance the ability to mine user support tickets. The user information and interfaces staff added new users guides for IU’s Mason system, and for the XSEDE-Wide File System (XWFS). The UII team also continued to improve the XSEDE User Portal, including the overhaul of the POPS allocations system interface (bringing it in line with other XSEDE interfaces) and implementing a new community account process that will be easier for PIs and quicker for all. This quarter the Operations team's activities included continued work on implementing storage allocations within the XSEDE allocations systems and central and SP infrastructure. Significant efforts were made on the beta deployment of the Genesis II and UNICORE software for the Campus Bridging Pilot. XSEDEnet was transitioned from NLR infrastructure to the Internet2 infrastructure. Internet2 has 100G nationwide backbone capability with software-defined networking and capabilities that should serve XSEDEnet networking needs well into the future. Good progress was made on the XSEDE-wide file system (XWFS), with hardware and software purchases and deliveries being made for a majority of the five participating sites. There were no known significant security incidents this quarter. A small number of planned and unplanned outages occurred with the RDR and XSEDE central database systems during this quarter, but uptime was still in the 99 percent ranges. The XSEDE Operations Center (XOC) fielded 1,823 tickets and closed 1,695 tickets. The Campus Bridging activity has been moving forward and the A&D team has been working hard to develop the Level 3 Decompositions of the defined Use Cases. All of the GFFS Pilot sites were given access to the GFFS Software in this period and allowed to work with XSEDE resources as well as with resources at UVA and across participating campuses. Plans for evaluation of this activity are being developed with the external evaluators. This evaluation is expected to be ready to conduct as we complete the rollout and get early experiences with the initial campus bridging capabilities. The XSEDE Advisory Board met via teleconference on Feb. 8, 2013. The project continues to benefit from the willingness of the XAB to meet with us on a more regular basis than originally planned to discuss specific topics in greater depth. The February 2013 discussion focused on finalizing comments on the XSEDE Industry Challenge program and on discussing our progress in establishing project goals and effective metrics.

13 The TEOS Advisory Committee met via conference call on Feb. 26, 2013. TEOS managers updated the AC on Campus Bridging status and plans, as well as Education and Training activities to strengthen the ties between the two services. Managers shared their vision for integrating XSEDE certifications, assessments, core competencies, and roadmaps from novice to competent user, which they hope to share more broadly to help institutions develop their own programs. These held particular interest for faculty among the Advisory Committee. For the past quarter, the Technology Investigation Service (TIS) concentrated on continued development of the XTED database, with the goal of increasing the number of entries in the database. Additional projects have been registered following outreach (via letters) to PI's of NSF- sponsored research projects. To further the awareness of TIS activities, TIS evaluation reports are being entered into the XSEDE Digital Document Repository and TIS is preparing XSEDE news items to be sent to the entire XSEDE project. The Technology Evaluation team has been focusing on completing reviews in progress and starting to plan for the next set of reviews. XSEDE External Relations has been ramping up interactions with other area of the project to assist in communications needs, including additional stories about ECSS support activities. Planning for the next annual highlights publication is also now under way. SSE, A&D and SD&I have continued to make significant progress on the development of additional Use Cases to drive the architectural design and subsequent processes in XSEDE for delivering new capabilities to the community. Building on defined Use Cases, the A&D team has made progress on the Level 3 decomposition of the security components and has conducted Active Design Reviews of these components with the SD&I team. SD&I has also been wrapping up final elements of the original 100 activities planned as part of “Increment 2” work and initiating 16 activities prioritized by the User Requirements and Prioritization Working Group for “Increment 3.” The Use Case tracking system has also progressed and we anticipate that by next quarter we will have a usable tool to provide visibility into the status of progress of Use Cases and their related components. This will help many others in the project to understand timeline expectations for new capabilities and prepare to support them. From a staffing perspective, SD&I welcomes Shava Smallen to the role of deputy director in addition to her leading the SD&I testing team. We have also made progress on identifying a replacement for Susan McKenna, who departed as our External Relations lead in December 2012. We anticipate having a new lead aboard early in the next quarter.

14 2 Science and Engineering Highlights 2.1 Turbulence Modeling: Droplet-laden Isotropic Turbulence (Antonino Ferrante, William E. Boeing Department of Aeronautics and Astronautics, University of Washington, Seattle) While most people think of turbulence as the source of unsettling bouts of chaotic airflow during a flight on an airplane, physicists have a much deeper perspective. In fact, Nobel Laureate and theoretical physicist Richard Feynman once said, “Turbulence is the most important unsolved problem of classical physics.” Probing the puzzling nature of turbulence is essential because of its impact on the flow physics of liquids and gases, and, by extension, the influence of those fundamental states of matter flowing outside and inside things— whether the setting is the atmosphere (pollution and volcanic ash) or a jet engine, a combustor, or a nuclear reactor, for Figure 2.8.1 Fully resolved droplet-laden decaying example. Antonino Ferrante of the isotropic turbulence. The plotted cube is a 512th of the University of Washington, Seattle, is entire cubic box simulated using a mesh of 10,243 grid leading a research team employing points and 7,000 droplets. turbulence simulations and modeling to investigate fluid dynamics (both gases and liquids in motion). Resolving the wide-range scales of motion requires fine computational grids with billions of points. “Without high-performance computing (HPC), turbulence simulation would just not be feasible, and our understanding of turbulent flows would not progress much,” he says. Moreover, as HPC becomes more robust, he adds, it will allow researchers to address the challenges of reducing fuel consumption and CO emissions. XSEDE’s Extended Collaborative 2 Support Services provided an avenue for Ferrante and his team to access the HPC resources and expertise offered by the National Institute for Computational Sciences and the National Center for Supercomputing Applications. Ferrante workedFigure with. Vince Fully resolved Betro, adroplet NICS-laden computational decaying isotropic scientist and a member of ECSS, to make it possible turbulence. to simulate The turbulent plotted cube flows is a 512 as th well of the as entire turbulence cubic coupled with other phenomena—including chemicallybox simulated reactive using and a mesh multiphase of 10,243 turbulent grid points flows and — while getting results in a reasonable time. The7,000 team droplets. worked closely with David Bock, a NCSA visualization programmer, and Darren Adams, an NCSA research programmer, both members of ECSS. They were able to provide petascale scaling and deployment, development of a high-level HDF5 parallel I/O layer (H5DNS), and custom visualization of simulation results. Variables such as velocity and pressure had to be advanced in time in billions of grid points; minimum resolution was 1,0243 data points. Kraken enabled the researchers to address the challenge. Ferrante says: “For the first time, we are simulating the effects of droplets on turbulence. In practice, we are running a real-life experiment on a supercomputer rather than a wind-tunnel.” The result was a turbulence model that can be coupled with flow-solver computer applications to better understand fluid flow and design all manner of objects affected by fluids (e.g., air and water).

15 2.2 Seismology: Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers (Yifeng Cui, San Diego Supercomputer Center, UC San Diego) A team of researchers at the San Diego Supercomputer Center (SDSC) and the Department of Electronic and Computer Engineering at UC San Diego has developed a highly scalable computer code that promises to dramatically cut both research times and energy costs in simulating seismic hazards. The team, led by SDSC computational scientist Yifeng Cui, performed GPU-based benchmark simulations of the 5.4 magnitude earthquake that occurred in July 2008 below Chino Hills, near Los Angeles. Compute systems used in the project included Keeneland, an XSEDE- allocated resource managed by Georgia Tech, Oak Ridge National Laboratory (ORNL), and the National Institute for Computational Sciences (NICS). The simulation, performed on Keeneland using 128 NVIDIA M2090 GPUs, indicates that small-scale heterogeneities (causing the highly irregular pattern of shaking in the image) may significantly affect ground motion in geologic basins. The work on Keeneland was instrumental to the team’s ultimately achieving five-fold speedup over the heavily optimized CPU code on Cray’s XK7 and a sustained performance of 2 petaflops per second on the tested system. A previous benchmark of the AWP-ODC code reached only 200 teraflops of sustained performance. By delivering a significantly higher level of computational power, researchers can provide more accurate earthquake predictions with increased physical reality and resolution, with the potential of saving lives and minimizing property damage. “The increased capability of GPUs, combined with the high-level GPU programming language CUDA, has provided tremendous horsepower required for acceleration of numerically intensive 3D simulation of earthquake ground motions,” said Cui, who recently presented the team’s new development at the NVIDIA 2013 GPU Technology Conference (GTC) in San Jose, Calif. The project is part of a larger computational effort coordinated by the Southern California Earthquake Center (SCEC). A technical paper based on this work will be presented June 5-7 at the 2013 International Conference on Computational Science Conference in Barcelona, Spain.

Figure 2.4.1 The image shows a snapshot of ground motion of the 2008 magnitude-5.4 Chino Hills earthquake in an east-to-west direction; the red-yellow and green-blue colors depict the amplitude of shaking. The simulation indicates that small-scale heterogeneities (causing the highly irregular pattern of shaking in the image) may significantly affect ground motion in geologic basins. Simulation by Efecan Poyraz/UC San Diego and Kim Olsen/San Diego State University. Visualization by Efecan Poyraz; map image courtesy of Google.

16 2.3 Molecular Dynamics Simulation of Assembling Behavior of a Triplex Complex Formed by Amylose, Beta-lactoglobulin, and Alpha-linoleic acid. (Bruce Hamaker and Osvaldo Campanella, Whistler Center for Carbohydrate Research, Purdue University) Food science researchers have recently become aware of the XSEDE resources. At the Whistler Center for Carbohydrate Research, Purdue University, researchers study fundamental structure-function relationships of carbohydrates and other biopolymers for their practical implications. Structure-function relationships are normally determined via elucidation of chemical and three- dimensional structures. With recent computer hardware advancements, computational methods are becoming Figure2.1.1 Structural snapshots from simulations of amylose (shown in lines) and α-linoleic acid (shown in VDW) complex. a critical part of food science research and expanding into other areas of studies. XSEDE resources assist Bruce Hamaker, professor at the Whistler Center, in studies of the detailed complexion behavior of amylose and α-linoleic acid at atomic level in order to understand the binding mechanism of small molecules to amylose. They had run simulations on a small set of lab computers, which limited them to only 2-10 ns simulation over a 10-day period, insufficient to produce conclusive data to explain experiment observations. With the help of Purdue’s XSEDE campus champion, this group began to use XSEDE resources for their MD simulations in 2013 and have run on the Stampede, Kraken and Lonestar systems. This computation now typically uses 64-128 cores. The initial result of 500 ns all-atom MD simulations starting from the initial structure including α- linoleic acid indicates a large-scale order-to-disorder conformational transition within the first 200 ns, suggesting weak coupling between amylose and α-linoleic acid. Upon closer inspection, it is clear that α-linoleic acid is necessary and sufficient for amylose to reach the thermodynamically stable configuration. Such a result demonstrates the importance of real-time dynamics in the formation of thermodynamically stable configuration of the complex, which have broad implications for understanding amylose small molecular binding behaviors. The results improve models characterizing such systems, which are important in both natural and manmade materials. The results also enhance understanding of how the local, particle-scale interactions within these materials lead to collective or emergent properties.

The Whistler Center is a university-industry research center that conducts fundamental research related to practical applications of food carbohydrates, provides analytical services, and provides training and education. Its research and service directly impact the U.S. food industry and its computational need has the potential to grow to a much larger scale.

Figure2.1.2. Geometrical rearrangement of amylose complex. (a) RMSD of the entire amylose and α-linoleic acid complex; (b) RMSD calculated for amylose alone. RMSD was calculated by superimposing the backbone atoms of amylose domains GLC, GLM and GLK, onto the crystal structure. Low RMSD values indicate similarity to the crystal structure

18 3 Delivering and Supporting New Capabilities in an Evolving Environment 3.1 Architecture for a National Distributed Cyberinfrastructure Ecosystem (Architecture and Design 1.1.3) During this quarter, the A&D team focused on three areas: 1) continued Use Case development; 2) level 3 decomposition and active design reviews of XWAVE and XUAS security components; and 3) the Architecture and Design Use Case tracking system. 3.1.1 Use Case Development The architects and area leads continued development and review of new Use Cases. Once Use Cases are reviewed and accepted as complete/ready by the architects on team calls, Altaf Hossain and Janet Brown will be adding the formatted Use Case documents to the XSEDE document repository and adding links to them on the XSEDE public-facing documents wiki page. There were extensive discussions around the development of canonical Use Cases 7, 8 and 9. UCCAN 7.0 – Resource/software/service information management UCCAN 8.0 – Resource/software/service status management UCCAN 9.0 – User Management Updated Use Case status: • Campus Bridging (7): 7 - Use Cases now public facing • Science Gateways (5): 4 - Use Cases now ready for public facing • Computing High-Performance Computing (4): 1 – Use Case now ready for public facing High-Throughput Computing (5) Scientific Workflows (3) • BIG Data Data Analytics (5) Data Movement, Storage, Backup & Archival (5): 1 – Use Case now ready for public facing Visualization (5) • Connecting Instrumentation (1) • Collaboration (1) • Canonical (10): 2 - Use Cases now ready for public facing Federation and Interoperation (1): 1 – Use Case now ready for public facing 3.1.2 XWAVE and XUAS Security Component Active Design Reviews We have successfully completed the active design review for the XWAVE architecture security components and held the first active design review for the XUAS architecture security components and common security components used by both XWAVE and XUAS. We are on schedule for new public-facing XSEDE architecture document by the end of Q4 that will contain this new documentation.

19 3.1.3 Architecture and Design Use Case Tracking System Altaf Hossain and David Lifka continue updating the Google Docs spreadsheet that is the template for the web-based Use Case tracking system. Use case data in that document is being used to drive the agenda for weekly A&D calls. Susan Mehringer from Cornell, with support from JP Navarro, is continuing development on our first version of the Use Case Registry System with a goal of having all of the A&D processes incorporated and ready for team use at the completion of PY2. This web-based system will provide views for various stakeholder groups so they can track progress on their Use Case submissions. 3.1.4 Accomplishments this Quarter  XSEDE Architecture Documentation A new version of the XSEDE Architecture level 3 decomposition is being completed that includes documentation for the XWAVE security components. Each section of this latest version of the document will clearly state the current review and acceptance status of each architectural component. This will allow us to have a public-facing document that continually evolves over time.  Number of new Use Cases per area Over 50 Use Cases are being actively developed and reviewed. Active participation in weekly conference calls by over 40 participants (XSEDE staff and outside stakeholders) including rchitects, SD&I, SSE, Security, SYS-OPS and Architecture area leads  Number of Use Cases accepted in the A&D process 16 Use Cases ready for architects to prepare level 3 decomposition.

 Active Design Reviews Completed the active design review for the XWAVE security components. Preparation for the XUAS security components started.  Use Case Registry System Development Development of system covering A&D processes continues 3.1.5 Planned Activities  XSEDE Architecture Documentation o Release updated version of XSEDE Architecture Document include level 3 decompositions for XWAVE and XUAS security components and common security components. o Includes notation identifying which sections have passed their Active Design Review o All completed Use Cases will be added to XSEDE document repository and linked to from the XSEDE wiki public facing documents page  New Use Cases developed and accepted in the A&D process o Canonical Use Case 9.0 (UCCAN 9.0): o Use case & quality attributes documented and public facing o L3 Decomposition Complete

20 o Active Design Review Complete  Active Design Reviews o Complete XUAS Security Component Active Design Review  Use Case Registry System Development o Version 1 of Web-based Use Case Tracking system will be available

3.2 Realizing the XSEDE Architecture (Software Development and Integration 1.1.6) This quarter Shava Smallen was selected as SD&I deputy director in addition to her responsibilities leading the SD&I testing team and will be assisting SD&I director JP Navarro with the management responsibilities of the group. As previously reported, Software Development and Integration (SD&I) “Increment 2” activities began in February 2012 after our first semi-annual planning effort. Of the five incomplete increment 2 activities that were carried into January 2013, two were in active testing at the time and were finished this past quarter. This included an activity to deliver GridFTP stage-in/out support in UNICORE 6, which was handed off to Operations. The other activity that included a refresh of UNICORE 6 Execution Management Services (EMS) and Genesis II Global Federated File-System (GFFS) was not handed off to Operations as decided by a special plan co-developed by SD&I, Operations, and Campus Bridging in December 2012. This was due to unanticipated delays in Increment 2 testing and a decision to instead focus on a new Increment 3 version of integrated EMS and GFFS targeted for production. Instead EMS and GFFS will be delivered to Operations in June 2013 as part of Increment 3 and will incorporate feedback from the Increment 2 reviews as well as the Campus Bridging pilot effort. The three other incomplete Increment 2 activities for information services and testing improvements were still in development and are still being worked on in Increment 3. Also note that due to a strange intermittent bug discovered by Operations while deploying MyProxy Oath inside the XSEDE User Portal container (SDIACT-50), this activity was returned to development and redesigned to deploy as a standalone service. Also previously reported, SD&I “Increment 3” priorities were established in September 2012 by a User Requirements Evaluation and Prioritization (UREP) team. This was the first time that a team with User Advisory Committee (UAC) and XSEDE senior management representatives directly set SD&I priorities. Based on UREP established priorities, SD&I identified over 30 candidate development activities last quarter and selected a subset of those for effort between October 2012 and June 2013. Specific Increment 3 accomplishments this quarter include:  Completed development, testing, and acceptance of an improved xdusage utility enabling users to view allocation usage information from a shell on SP login nodes (SDIACT- 102). Note that because this activity was small, SD&I and Operations conducted joint testing for faster production delivery.  Completed a pilot effort to manage SD&I activities in JIRA for simplified management and improved team collaboration (SDIACT-103).  Continuing design of an improved XSEDE accepted certificate authority configuration file installer and distribution mechanisms to improve grid service and software configuration (SDIACT-003).  Completed design of a command-line single sign-on hub/server that will enable users to login via ssh to an XSEDE wide login server, and then login to any other XSEDE login node without having to re-authenticate (SDIACT-070).

21  Developing a Genesis II internal improvement to replace signed Java serialized tokens (GAML) with signed SAML tokens while establishing a baseline for long-term use of SAML in XSEDE (SDIACT-110).  Designing support for replication of the Secure Token Service (or Identity Provider Service) port-type resources in Genesis II (SDIACT-112).  Designing a Globus Connect Multi-User installer that will significantly simplify campus deployment and operation of GridFTP servers for moving data to/from their resource using Globus Online (SDIACT-108).  Designing an effort to prepare a Globus Online Transfer REST API so that software developers can use Globus Online Transfer from their software (SDIACT-106).  Completed development of a System-wide Genesis II installer that will make it easier to install and use Genesis II software on SP resources (SDIACT-099).  Worked with the Science Gateways team to identify the scope of EMS and GFFS Java API for Science Gateways (SDIACT-115).  Designing an Increment 3 version of EMS and GFFS as described above (SDIACT-123).  Continued development of registration packages to register new EMS and GFFS components availability thru capability registries (SDIACT-54).  Continued progress re-designing and streamlining SD&I activity management wiki workspaces, documentation, deliverable templates (SDIACT-121).  Started an effort to design a global GFFS namespace, to identify policy decisions that need to be resolved, and to implement operational mechanisms to maintain the global GFFS namespace (SDIACT-126). SD&I also began an Increment 4 planning activity in March 2013 and identified over 30 candidate development activities that are being evaluated by the UREP and will drive SD&I’s PY3 plan.

3.3 XSEDE Operations 1.2 3.3.1 Overview The Operations group consists of ~30 FTEs and is responsible for implementing, delivering, maintaining, and evolving an integrated cyberinfrastructure of unprecedented scale that incorporates a wide range of digital capabilities to support the national scientific and engineering research effort. The Operations group follows the XSEDE project management methodologies detailed in the Project Execution Plan by allocating and coordinating staff in accordance with the XSEDE work breakdown structure (WBS), scheduling tasks in the XSEDE project schedule, and identifying and reviewing risk on an ongoing basis. Operations staff is subdivided into six teams based on the WBS: 1.2.1 Cybersecurity 1.2.2 Data Services 1.2.3 XSEDEnet (Networking) 1.2.4 Software Testing and Deployment 1.2.5 Accounting and Accounts Management 1.2.6 Systems Operational Support This quarter activities included continued work on implementing storage allocations within the XSEDE allocations systems and central and SP infrastructure. Significant efforts were made on the beta deployment of the Genesis II and UNICORE software for the campus bridging pilot. XSEDEnet was transitioned from NLR infrastructure to the Internet2 infrastructure, which has 100G nationwide backbone capability with software-defined networking and capabilities that

22 should serve XSEDEnet networking needs well into the future. Good progress was made on the XSEDE-wide file system (XWFS), with hardware and software purchases and deliveries being made for a majority of the five participating sites. No security incidents were reported this quarter and some modification to the XSEDE trusted Certificate Authorities was completed. A small number of planned and unplanned outages occurred with the RDR and XSEDE central database systems during this quarter, but uptime was still around 99 percent. The XSEDE Operations Center (XOC) fielded 1,823 tickets and closed 1,695. Among those fielded, 1,785 were submitted via email to [email protected], eight were submitted via the XSEDE User Portal, and 30 were submitted via phone to the XOC. 3.3.2 Cybersecurity 1.2.1 XSEDE operates with a goal of protecting the confidentiality, integrity, and availability of resources. For this quarter, the security team began addressing issues discovered from the risk- assessment process that concluded in late 2012 along with continued work on existing projects, including the XSEDE scanning service, One Time Password service, and XSEDE Certificate Authority (CA) service. The incident response team issued one security advisory but reported no security incidents. A number of changes were made to the XSEDE trusted Certificate Authority repository to remove expiring CAs and to add a new DOEGrids CA. Security Incidents: There were no known security events and no reported compromised accounts during this period. There were no other significant security events during Q1. Security Vulnerabilities: The security group issued one security advisory in March for a TORQUE 4.0+ security vulnerability. The advisory was assessed as a high priority by the team and mitigation options were provided. SP responses to the advisory were documented on the secure wiki for those sites that were affected. Activities during the quarter included: Policy Development: One of the findings from last year's Risk Assessments process was the need for formal XSEDE policies resulting in a new Acceptable Use Policy (AUP) that was created for all XSEDE users. The policy outlines expected user responsibilities as well as the proper use of XSEDE resources. In addition to the AUP, the XSEDE Security Working Group Charter was drafted, which establishes goals of the XSEDE Security Working Group (XSWoG). Both documents are considered to be in final form and ready for formal acceptance from XSEDE management. One Time Password (OTP): To improve the security of staff accounts at the SP sites, XSEDE is deploying an OTP service with tokens for staff accounts. The hardware needed for the XSEDE security operational OTP service was ordered and will be sent to the primary and backup locations. The primary OTP server will be hosted by NICS with the back-up OTP server at PSC for failover purposes. The software has been delivered and we're simply awaiting delivery of the hardware. Host CA Update and InCommon CA News: In order to offer an XSEDE service to issue host certificates for resources, InCommon has submitted a CP/CPS to the TAGPMA for accreditation. The InCommon CP/CPS has gone through two rounds of reviews. There are some remaining comments from the reviewers that are being worked out. Security enhancement rollouts: The following Table shows changes that were made to the XSEDE Approved Certificate Authorities this quarter: Date CA modification 3/10/2013 Removed retired NCSA GridShib CA; Replaced 10718cba.crl_url: new file

23 removed http://crl.doegrids.org/cilogon-basic.crl 1/30/2013 Added DOEGrids CA S/N 0x47 valid 2002-12-5..2018-01-25 12d0da68.* 1c3f2ca8.* 1/28/2013 Removed expiring UK EScience CA 53729190.* 367b75c3.*, DOE Grids CA 12d0da68.* 1c3f2ca8.*, and SDSC NPACI CA 9117797f.* b89793e4.* 1/11/2013 Removed Decommissioned TACC CAs 9a1da9f9 and f30e4b25

Qualys Vulnerability Scanning: The XSEDE vulnerability scanning service is in production status and available to all SPs. To assist XSEDE SPs in identifying security vulnerabilities to XSEDE services and to inform XSEDE management and NSF on the state of XSEDE security, the security team has created a list of all known core XSEDE resources. For each core resource the following information is tracked on the secure wiki: Resource Type Physical Location Resource Name IPs XSEDE - Sys Admin Point of Contact (POC) XSEDE - Service POC Backup Service POC (e.g., when primary on vacation) Responsible for Scanning (the SP or XSEDE) Scan Frequency (Daily, Weekly, etc.) 3.3.3 Data Services 1.2.2 Data Services activities for this quarter included maintaining the existing operational infrastructure, such as GridFTP servers and archive resources, improving documentation for data resources in XSEDE, and preparing for upcoming software deployments and accounting changes. The XSEDE-wide file system (XWFS) deployment process has begun, with all hardware arriving by the end of the quarter to support XWFS becoming operational with user data allocations via the XRAC beginning in July. Data Services team members from TACC, SDSC, NICS, PSC, and NCSA are engaged in this process and participating in bi-weekly planning and implementation discussions for the XWFS. Several SD&I components affecting Data Services remain in preparation for deployment, including updates to GridFTP, Globus Online, and the Global Federated File System. Operations staff participate in testing and operational readiness reviews for all such components. Data Services is actively participating in providing feedback to both the Architecture and Design and SD&I processes by providing input on Use Cases, design reviews, and operational and testing readiness reviews. Activity has been increased in the area of GridFTP logging, performance characterization, and configuration recommendations. In collaboration with staff at all sites and the SD&I team, enhanced logging has been enabled and processes have been initiated to perform detailed analysis of the resulting data. Data Services team members have also been assisting at the XSEDE-wide and individual SP levels with user services, documentation, and accounting staff, to support implementation of storage allocations and accounting throughout XSEDE.

24 3.3.4 XSEDEnet 1.2.3 The XSEDE Networking group is responsible for the operation and monitoring of the XSEDEnet wide area infrastructure as well as the network connections for each XSEDEnet service provider site. The quarter’s primary accomplishment was the successful transition of XSEDEnet wide area provider from National LambdaRail to Internet2. Tasks worth noting were weekly calls to discuss and evaluate positives and negatives of the transition to the new research and education network provided by Internet2, known as Advanced Layer 2 Services (AL2S). Implementation details included cross-connects, routing and stats collection for AL2S was planned and implemented in support of the transition. 3.3.4.1 XSEDEnet and transition to Internet2 XSEDEnet, the high-speed interconnect between the eight core XSEDE service provider sites, was recently transitioned from NLR to Internet2 infrastructure. Each site connects at 10 Gb/s to Internet2’s Advanced Layer 2 Service (AL2S). All XSEDE service provider sites have full site- to-site connectivity via the Internet2 100Gb/s AL2S backbone. In addition to XSEDEnet connectivity, all service provider sites maintain connectivity to Internet2 IP service network and/or NLR's PacketNet for general access to/from the research and education community. The decision to transition to Internet2’s AL2S was based on new capabilities and services Internet2 is providing. The AL2S backbone is configured with 100 Gb/s links to each of its point of presence sites across the United States. If a particular link becomes congested, additional 100 Gb/s links will be added. This will ensure that service provider sites will have full access to bandwidth as usage of AL2S increases. All links between XSEDEnet service provider sites are now point-to-point, and the previous 10 Gb/s bottleneck link between Denver and Chicago that existed with NLR has been removed. Also Internet2 will allow each service provider the ability to upgrade its connection to XSEDEnet from 10 Gb/s to either 40 Gb/s or 100 Gb/s if needed.

As more sites attach to Internet2 AL2S service, XSEDEnet will be able through the use of a real- time portal VLAN provisioning service to allow for Campus Bridging sites to provision VLANs to allow direct point-to-point access for campuses to service provider resources.

XSEDEnet's transition to Internet2's AL2S service was mostly completed on March 8, 2013. The only exception was Texas Advanced Computing Center's (TACC) connection. There were optics compatibility issues that forced TACC to rehome their new AL2S connection back to NLR for both layer 1 and layer 2 instead of just layer 1 transport to Houston. Once the link was reverted to NLR, TACC's AL2S connection became active and stable on March 11, 2013. Internet2 AL2S service has proven to be reliable in the case of fiber cuts affecting its backbone. On March 27, there was a fiber cut between Detroit and Chicago. Only the PSC network connections were affected by the outage. After contacting the Internet2 NOC, it was discovered that there was a bug, which was subsequently fixed, in their backup path provisioning code, and PSC's XSEDEnet service was restored over a backup path. XSEDEnet is the first production customer of the Internet2 AL2S. A server has been purchased by XSEDEnet to begin testing integration between XSEDEnet and Internet2's AL2S dynamic provisioning service. 3.3.4.2 PerfSONAR Performance-focused Service Oriented Network monitoring ARchitecture (perfSONAR) is an infrastructure for network performance monitoring of IP networks, making it easier to solve end- to-end performance problems on paths crossing several networks. perfSONAR includes a set of services delivering performance measurements in a federated environment. These services act as an intermediate layer between the performance measurement tools and the diagnostic or visualization applications. This layer is aimed at making and exchanging performance measurements between networks, using well-defined protocols. The perfSONAR architecture includes measurement points, measurement archives, distributed services (such as server registration and lookup services), and a web-based graphical user interface to the measurement archives. A dedicated 10GbE-connected perfSONAR is installed at each of the eight XSEDE SP sites (IU, NCAR, NCSA, NICS, PSC, Purdue, SDSC, and TACC). The perfSONARs run a full mesh of scheduled tests including Bandwidth Test Controller (bwctl with iperf throughput test), One-Way Active Measurement Protocol (OWAMP), pingER, and traceroute. Network engineers are the primary users of the perfSONAR infrastructure for network testing and debugging. psarch.psc.xsede.org is installed at PSC for collecting backups of the performance data from each of the perfSONARs installed at the SP sites. ps.ncar and psarch.psc both run the maddash web user interface software to present a simple graphic display of perfSONAR test results. The ps.ncar display is on a hostname basis; the psarch.psc display is IP-address-based to work around perfSONAR problems dealing with multiple IP addresses on a single host. The mesh generator software is an automated way to set up a mesh of tests between multiple sites. (The perfSONAR mesh testing was manually set up before the tool existed.) NCAR – ps.ncar.xsede.org The ps.ncar move to a new machine room and reconnection was completed in January 2013. Previous location and connectivity had a persistent inbound throughput problem to the perfSONAR. Low input appeared to be coincident with congestion on a shared link (based on observed improved throughput to ps.ncar during an NCAR supercomputer outage in 2012).

Purdue – ps.purdue.xsede.org The ps.purdue connection was upgraded from 1GbE to 10GbE in December 2012. Inbound throughput is constrained, possibly by a buffer limitation in the path with throughput decreasing as RTT increases to farther away sites. The input/output drop in early January coincided with a routing instability in the wide area network. No user problem reports were received and an investigation is ongoing.

SDSC – ps.sdsc.xsede.org The ps.sdsc connection shows limited inbound throughput. This problem appears to be limited to ps.sdsc, and the investigation is ongoing.

3.3.5 Software Support 1.2.4 The Software Deployment and Testing (ST&D) group completed acceptance testing for two software components this quarter that were carried over from the previous quarter: Basic EMS (SDIACT-97) and GridFTP Update (SDIACT-31). Both components were accepted for production deployment. The ST&D group also did a trial of a streamlined process for combined SD&I/ST&D testing for small, low-risk software components using the xdusage (SDIACT) component as the test subject, which was also accepted for production deployment. A structured process for creating deployment implementation plans was developed by the SP coordinator in cooperation with representatives from the SPs, and deployment planning is under way for all three of the components accepted this quarter. An operational readiness review (ORR) for one software component, GridFTP in UNICORE (SDIACT-75), was conducted this quarter. The review was a success, but several documentation and packaging changes were requested in the ORR, and these were not completed by the end of the quarter. Acceptance testing of this component is expected to start early in the next quarter. The items above show progress of the XSEDE Engineering process through to Operations. Table 1 shows an overview of the items that have started and completed operational reviews and items that have been accepted and started deployment. The GFFS beta deployment continued to be a major area of effort in the ST&D group this quarter. This deployment depends on the SD&I Increment 1 EMS and GFFS components, which did not pass the XSEDE Engineering process but are being deployed in a beta deployment to gain valuable experience in operating EMS/GFFS and end user feedback through campus bridging pilot users. The EMS and GFFS are continuing through the XSEDE Engineering process, and we expect new versions from SD&I Increment 2 to come to Operations for operational review. The beta deployment includes deploying a Genesis II root container at NCSA as well as Genesis II version 2.5 client software, Genesis II version 2.5 non-root containers, and UNICORE 6.4 client and servers at NICS, PSC, and TACC. Currently, the Genesis II and UNICORE client software and Genesis II non-root container have been successfully deployed at all three sites. The

28 UNICORE client and server software have been deployed and installed at NICS, PSC, and TACC, but the configuration and testing is incomplete at TACC. Table 1: ST&D operational reviews and deployments

PY 1 PY 2

Q1 Q2 Q3 Q4 Q1 Q2 Q3 readiness reviews completed 0 0 2 1 0 3 1 acceptance tests completed 0 0 2 1 0 1 3 components accepted for beta 0 0 0 1 1 0 0 components accepted for production 0 0 0 0 0 1 3 components deployed in beta 0 0 0 0 0 1 0 components deployed for production 0 0 0 0 0 1 3

3.3.6 Accounting and Account Management 1.2.5 Work continued from last quarter on formalized implementation of “Storage Allocations” in the Partnerships Online Proposal System (POPS) and the XSEDE Central Database (XDCDB). This planning was completed in February 2013, and the work was completed in the POPS and the XDCDB. This work included standalone storage allocations, enforced request limits, default allocations, and resource dependencies. Next the A&AM team will be assisting the SPs with implementing storage allocation capabilities at the SP-level. IU's Mason system was implemented in POPS and the XDCDB. Work was also completed on transferring the remaining allocations on TACC's Ranger system to the new Stampede system. A new board was created in POPS, called "Software Testbeds,” with the FutureGrid system as the sole available resource. This will be made available for submissions once the FutureGrid team has completed the Account Management Information Exchange (AMIE) implementation. This board will allow variable length awards to meet the unique requirements of FutureGrid. The A&AM team has also continued working with the FutureGrid team on completing the AMIE implementation, and this should be completed in the next quarter. Planning on the new "POPS 2.0" system, a complete rewrite of the existing POPS system using the XDCDB Postgres database, continued and intensified this quarter. The A&AM team assisted in the development of the "POPS 2.0" project plan and an architecture document. The team has also begun developing a database schema, along with a RESTful API specification. The new "POPS 2.0" system is targeted for completion in Q4 of PY3. 3.3.7 Systems Operational Support 1.2.6 Systems Operational Support (SysOps) is the group responsible for operating the XSEDE Operation Center and providing system administration for all XSEDE centralized services. Significant progress was made in the following WBS tasks in the past quarter: 1.2.6.10 and 1.2.6.12. Tasks worth noting include cross group collaboration with User Engagement to deploy the new RT ticket system, weekly operations RT team status calls, central Nagios monitoring deployment, and Application Support candidate interviews. Even though there were several planned and unplanned outages, the SysOps team maintained high overall uptime, which ensured data integrity and availability. By leveraging failover resources, where appropriate, user-facing downtime was greatly minimized. As such, no central server experienced any less than 99.35 percent uptime for the quarter and no central service experienced any less than 88.7 percent uptime for the reporting period. This specific instance was tracked down to a problem with notifications for the Karnak service owner and is out of scope for the SysOps group as that

29 service is “owned” by another individual. For completeness, we have included the downtime number in the SysOps report. Aside from this outlier, no other service experienced any less than 99.35 percent of uptime. During the reporting period, SysOps successfully deployed seven new central services, which included: MHonArc, OAuth, Nagios, RT – Primary, RT – Backup, RT – Development, and Single Sign On Hub. 3.3.7.1 XSEDE Operations Center The XSEDE Operations Center (XOC) fielded 1,823 tickets and closed 1,695 tickets. Among these 1,785 were submitted via email to [email protected], eight were submitted via the XSEDE User Portal, and 30 were submitted via phone to the XOC. There were 720 tickets or 39 percent closed within two business days. The stated metric of 80 percent in the original XSEDE proposal should be revisited to further define the metric in a more meaningful way as two business days can easily be lost just in initial communications. Nonetheless, the XOC continues to strive toward this metric. There were a total of 1,806 tickets, 99 percent, responded to within 24 hours. The following chart shows the ticket breakdown (opened/closed) for each major resolution center:

500 450 400 350 300 250 200 Opened 150 Closed 100 50 0

The tickets can also be further broken into seven problem categories. The below pie chart shows this breakdown:

30 Tickets Opened by Category

4% 8% jobs / batch queues login / access 25% 10% software / apps 5% mss / data account issues 19% 29% system issues filesystems

This pie chart represents a significant portion of the 1,823 tickets but does not represent the entire range. Tickets largely fit in the deven categories but there are other categories that are not significant enough to visually represent. 3.3.7.2 Central Services There were several outages both planned and unplanned that affected various central services during the quarter. Many of these outages were the result of individual servers or sites experiencing unexpected technical difficulties or routine maintenance. Outages varied between site-specific power events, networking interruptions, system failures, planned activities, and user- initiated interruptions. The following table describes each server that experienced an outage, the corresponding downtime/uptime, the nature of the outage (e.g., planned or unplanned), and the total number of hours down: Percentage of Uptime Planned Unplanned Total Outage(s) Outage(s) Outage(s) Service (Number of downtime hours) RDR 99.35% 14 hours 14 hours XDCDB Primary 99.91% 2 hours 2 hours

The remaining central servers did not experience an outage during the reporting period. 3.3.7.3 INCA At the time of this report, the Inca deployment was executing 874 tests for XSEDE software and services. Of these, 120 tests were running for six central XSEDE services: Inca, Information Services, Karnak, MyProxy, User Portal, and XDCDB. The table below shows the definition of an outage for each service and the uptime percentages as detected by Inca. All services fall within acceptable limits of their high availability service definition. NOTE: Uptime numbers shown below will vary from the above uptime numbers. The numbers represented below show all of the outages that INCA detected during the reporting period. Any outages or race conditions within the INCA system could prevent INCA from detecting further outages. Also, these downtime numbers are strictly from the user perspective. In situations where

31 failover resources are leveraged, the user may not see a downtime. For official system-level uptime numbers, the data in the ‘Central Services’ section should be used.

Uptime Service Definition of outage (Details of outages) Inca Inca status pages are unavailable or not able ~100% to fetch data from the database (i.e., test (Two failures for 25 details page fails to load). Tests every 5 minutes of downtime) mins. Information Information web pages are unavailable. ~100% Services Tests every 15 mins. (One brief outage for 15 minutes of downtime) Karnak Karnak front page fails to load. Tests every 88.7% 30 mins. (Three outages for a total of 244 hours of downtime) MyProxy MyProxy server does not respond to 100% credential query check. Tests every hour. (No outages detected) User Portal Portal homepage fails to load correctly. ~100% Tests every 30 mins. (One brief outage for 10 minutes of downtime) XDCDB Connection to database refused or slow ~100% (using check_postgres.pl script). Tests (Two brief outages for 25 every 5 mins. minutes of downtime)

3.3.7.4 Syslog Monitoring Project Over the past three months, staff at Cornell’s Center for Advanced Computing (CAC) have upgraded some key components of the Cornell Log Analysis and Monitoring (CLAMP) project. The SQLstream server CAC has been using was upgraded from a beta version to the newest release version, 3.0, which includes some key features for the project. Most notably, the client GUI can now be used by multiple users concurrently. Additionally, the Log Activity graphic tool has been re-implemented using Google Chart Tools rather than Adobe Flex due to the failure of a third-party, Flex-based library. Methods for accessing data within this tool were also streamlined and optimized. This makes the utility vastly more extensible and sustainable as well as yields improved responsiveness. And finally, CAC staff submitted a paper on the CLAMP project to the XSEDE13 conference. 3.3.7.5 Globus Online During the reporting period, there were 34 million files transferred to and 39 million files transferred from XSEDE endpoints using Globus Online (GO). In total, GO facilitated transfers of 642 TBs to and 650 TBs from XSEDE endpoints. There are 258 distinct GO XSEDE users. Of the total files transferred, 2 million were transferred from Globus Connect (GC) to an XSEDE endpoint and 8 million files were transferred from an XSEDE endpoint to GC. In total, 51 TBs of

32 data were transferred from GC to an XSEDE endpoint while 47 TBs were transferred from an XSEDE endpoint to GC. Of the previously mentioned distinct XSEDE GO users, 139 of them are distinct GC users. The above data does not include stats from automated performance testing. In total, there were 23 tickets opened for the GO team. The tickets can be lumped into the following categories: user education, endpoint operational issue, bug fix, feature request, and user action required notification. The table below shows the number of tickets that fit within each distinct category. Number of Category Explanation and Details Tickets User Education 4 Information provided to resolve the problem. Endpoint Operational Issue 11 Problems using a specific endpoint. A problem occurred that warrants a Bug Fix 5 change/fix to the GO software/system. GO lacking in some way and an Feature Request 3 improvement/new feature is identified. User Action Required Unsolicited email sent to user(s) for a 0 Notification problem that they should be aware of.

3.4 Technology Investigation Service 1.7 3.4.1 Overview For the past quarter, TIS has concentrated on continued development of the XTED database with the goal of increasing the number of entries in the database. An ongoing inventory of NSF- sponsored research projects is being used to send individual letters to project investigators. This has resulted in additional projects being registered in XTED. To further the awareness of TIS activities, work was started to enter TIS evaluation reports in the XSEDE Digital Document Repository. The necessary information is being gathered and put in the format needed by the repository. TIS is also preparing XSEDE news items, which will be sent to the entire XSEDE project. In addition to the XTED work, the Technology Evaluation team has been focusing on completing reviews in progress and starting to plan for the next set of reviews. TIS has continued coordinating with Operations and SD&I. Several meetings have been held to understand how the three groups can better interact. The group is working on the schedule of evaluations so that system provisioning for testing can be economized by doing one hardware and software provisioning that can serve evaluations in Operations, SD&I, and TIS. Internal to the project, we have reviewed and updated the structure and content of the TIS wiki pages. This is an ongoing process to better reflect the new status of TIS with XSEDE as well as the TIS internal reorganization. 3.4.2 1.7.1 Technology Identification XTED release 0.7.1 was pushed to QA for testing. Design and documentation of release 0.8 was begun. The next release will focus on opening up the XTED technology entry Use Case to a

33 wider audience, while also controlling ownership of those entries by the appropriate party. It will also enable XTED users to enter additional content, such as comments and expressions of interest, in a particular technology entry. This type of information will be useful in prioritizing technologies to evaluate. 3.4.3 1.7.2 Technology Evaluation Tech Evaluation has started three high-priority evaluations of commercial OTP systems. Additionally, the evaluation of UNICORE for WMS continues. The annual review of the evaluation process is under way. The training package has been completed and our new team member was trained with it. Progress on the three OTP evaluations was delayed due to unforeseen issues getting software, licenses, and support from the commercial vendors themselves. The evaluations will be finished in the upcoming quarter, with priority given to the initial internal report for consumption by other XSEDE subgroups. The Unicore evaluation was preempted to some extent by the emphasis on the OTP evaluations, but progress was made toward the production of the internal report, which is under review now. Three evaluations were started in this quarter, one was in progress and zero were completed.

34 4 Supporting and Expanding the Community 4.1 User Services 1.3 4.1.1 Overview The XSEDE User Services activities continue to mature in quality and scope, supporting a growing, and increasingly diverse, XSEDE user community. PIs requested 800M SUs (57.4B NUs) and the allocations process granted 322M SUs (26B NUs) across XSEDE-allocated systems at the first allocations meeting conducted since Stampede came into production as an XSEDE- allocated system. Training classes—online and in-person—had more than 14,000 participants, and plans for training certificates are nearing completion. The user engagement activities included completing the second annual user survey (with better response rate than the first), for which results are now being analyzed, and analyzing old tickets to identify the top areas of operations and support to address for the XSEDE user community. The user engagement team is working with the operations team to complete the transition to a new user support ticket system that will greatly enhance the ability to mine user support tickets. The user information and interfaces staff added new user guides for IU’s Mason system and for the XSEDE-wide file system (XWFS). The UII team also continued to improve the XSEDE User Portal, including the overhaul of the POPS allocations system interface (bringing it in line with other XSEDE interfaces) and implementing a new community account process that will be easier for PIs and quicker for all. All User Services activities continue on pace to achieve PY2 goals 4.1.2 Training 1.3.1 The XSEDE training efforts continued this quarter at a pace that will exceed our annual goals. Training reached more than 14,000 people via online, webcast, and in-person methods. Thirteen scheduled events took place, including the online course Applications of Parallel Computers, with 371 registrants. The events had more than 1,200 registrants. Demand for both beginner and advanced training continues to grow. In online training, 835 users enrolled for courses offered through CI-Tutor. The Virtual Workshop (VW) site had more than 12,000 unique visits. Return visits numbered more than 3,000. More workshops are in the works including VWs on Allocations , Data Transfers, R , MATLAB PCT/MDSC/MEX, Optimization and Scalability (parts 2 & 3), and Advanced Batch/SLURM. Planned outreach to minority-serving institutions continued with a heavily subscribed visit to University of Texas El Paso for a multi-track session that attracted 106 registrants. Ten XSEDE staff facilitated the sessions. The Florida International University multi-track regional workshop has reached the cap of 100 registrants, and the University of Maryland Baltimore County regional workshop is on track to reach its cap of 50 registrants. Conversations continue on coordination with the Virtual School for Computation Science and Engineering (VSCSE), and VSCSE event registration will be managed by the XSEDE User portal this summer. The plan for training certificates is nearing completion, with Education and Training working together to align competencies with courses. Implementation is proceeding on schedule. In both delivery and in course development, XSEDE remains on track to deliver on the milestones. 4.1.3 User Information & Interfaces 1.3.2 The XSEDE website and XSEDE User Portal (XUP) development team released new features and met deliverables for this quarter. The XSEDE website continued to expand with the online content being improved by adding and updating new user guides including the IU Mason system and XSEDE-wide file system. In addition, supporting documentation for the allocation of storage services was integrated. Efforts this quarter also included collaborating with Education and

35 Outreach to release a Summer School 2013 application form and administrative interface. This quarter the XUP team also participated in updating the POPS user interface. An overhaul of the front-end submission interface was done to improve usability and conform with the XSEDE look and feel. Furthermore, a new community accounts process was put in place to make a request less cumbersome for the PI and more streamlined on the backend. This allows community accounts to be processed more effectively with less human intervention and delay. Finally, during the reporting period there were over 24,000 file transfers accounting for over 9.5 TB of data transferred via the portal file transfer service. The Knowledgebase (KB) team added 10 new KB items and updated existing articles to insure accuracy. There are now 586 documents in the KB and there have been over 171,000 document retrievals. Use of both the website and the XUP continues to increase, with over 3.4 million hits on the website and over 2.1 million on the XUP within the reporting period. Many features within the XUP saw an increase in usage. Furthermore, 5,722 users accessed the XUP and 76 percent of the 2,382 users running jobs on XSEDE logged into the XUP. 4.1.4 User Engagement 1.3.3 XSEDE User Engagement is organized as two working teams: Feedback and Consulting. The Feedback team focuses on proactive support, while the Consulting team focuses on reactive support. 4.1.4.1 Feedback The Feedback team completed all required feedback activities and required reporting for the quarter, except for conducting a focus group. Two focus groups will be conducted next quarter, to catch up on the requirements. The Feedback team completed data mining on the tickets submitted to [email protected] during the previous quarter. As in previous quarters, the majority of tickets were related to routine operational issues. Aside from routine operational issues, the most common issues were related to data transfer and storage facilities. A report documenting the results of the quarterly data mining activities is posted on the XSEDE Staff Wiki for reference. The Feedback team completed the annual user survey, and the response rate was significantly better than last year. The team is analyzing the survey data and preparing the associated report. The survey team is also working with XSEDE Allocations to establish a targeted survey for use following each allocations review period. The team is also preparing to conduct interviews with some of the teams identified by the NIP group. 4.1.4.2 Consulting The Consulting team conducted all required consulting activities for the quarter and made significant progress toward the milestones associated with the deployment and release of the new XSEDE ticket system. A detailed project plan for the deployment of and transition to the new ticket system is available in SciForma. Finally, a cost-risk-benefits analysis for the proposed XSEDE CRM is under way. 4.1.5 Allocations 1.3.4 This objective encompasses the allocations process, both for Startup, Education and Campus Champion allocations as well as the merit-review XRAC Research request process, the POPS system for request handling and management, mechanisms by which PIs manage allocations through transfers, extensions and so on, and interfaces by which PIs manage the users who are authorized to use their allocations. Operationally, this objective includes the XRAC review

36 process, the Startup allocations review and decision process, and the maintenance and operations of the POPS system. The table below shows the overall allocations management activity handled by POPS and the allocations staff for the reporting period. Note that for transfers, the table shows only the positive side of the transaction to show the total transfer level; there is a corresponding negative amount, adjusted for resource exchange rates. POPS Requests and Awards

The March XRAC quarterly allocations meeting was planned and held in Las Vegas. The next two XRAC meetings have been scheduled for Indianapolis in June 2013 and Arlington, Virginia, in September 2013. At the March 2013 XRAC meeting, 800M SUs were requested. Recommendations totaled 384M SUs (57.4B NUs) with 452M SUs (33B NUs) available and 322M SUs (26B NUs) awarded. The review board had discussed a trend of poor or insufficient efficiency/scaling numbers in proposals and wanted to make sure they were not “giving benefit of the doubt” to PIs, therefore more than the normal cuts were made in recommended awards. The XSEDE Allocations staff received 400 tickets within the reporting period. Most, if not all, were addressed and a high rate of user satisfaction achieved. Lastly, XSEDE Allocations staff along with the XSEDE Operations/accounting group and UII group worked to include the ability to have storage requests in the Spring 2013 Research submission period. 4.2 Extended Collaborative Support Services 4.2.1 Extended Collaborative Support Service – Projects 1.4 The Extended Collaborative Support Service (ECSS) pairs members of the XSEDE user community with expert ECSS staff members for an extended period to work together to solve challenging science and engineering problems through the application of cyberinfrastructure. In- depth staff support, lasting weeks to up to a year, can be requested at any time through the XSEDE allocations process. Expertise is available in a wide range of areas, from performance analysis and petascale optimization to the development of community gateways and work and data flow systems. ECSS staff members also participate in reviewing adaptive proposals associated with XRAC meetings. ECSS efforts are divided into two parts: Projects, headed by Ralph Roskies, and Communities, headed by Nancy Wilkins-Diehr. These groups have very close interactions, with common project management support. ECSS consists of 37 FTEs, spread over ~80 people at about a dozen sites. ECSS worked with External Relations and User Services staff to develop the ability to allow users to contribute brief descriptions of achievements enabled by XSEDE; this feature is ready for deployment.

37 ECSS-Projects consists of ESRT (Extended Support for Research Teams) and NIP (Novel and Innovative Projects). Roskies has begun interviewing PIs who receive ECSS support to assess ECSS staff performance, to hear firsthand what the scientific impact of the ECSS has been, and to accept suggestions for improving ECSS operations. To date, he has interviewed eight PIs, and the results are very gratifying. PIs are almost unanimous in their appreciation of the work done by the consultants and again, almost unanimously, feel that their productivity has increased as a result. One problem that emerged is that the calendars of the proposed consultant and that of the proposing teams may not align. Even after a workplan has been developed, staffing issues in the proposing team may preclude further work with the ECSS consultant, and a project may have to be abandoned mid- stream. Roskies has suggested to the PI that they can re-apply for the ECSS support when their staffing issues are resolved. We also learned that it would be good to inform PIs about whom to contact if they feel that the interaction with the consultant is not going as well as it might. This information will now be in the award letter that PIs get when they get ECSS support. Roskies has encouraged PIs to mention the impact of ECSS support when they give talks on their work. ECSS-Projects has continued to contribute to developing Use Cases for the A&D effort. They have drafted Use Cases for discussion and review by the architecture team. Together with the Blue Waters team, ECSS has planned and promoted the annual Extreme Scaling Workshop to be held Aug. 19-20 in Boulder, Colorado, with a focus on heterogeneous computing. Detailed metrics are contained in the individual reports below. ECSS project managers Karla Gendler and Natalie Henriques continue their tasks of managing ECSS activities, namely project requests, active projects, project assignments, and staffing. This quarter marked the transition from using and maintaining spreadsheets to using Sciforma entirely. They continue to review and track workplans, entering staff allocations and quarterly objectives on each. They manage and attend ECSS meetings and XSEDE project management (PM) meetings, posting notes and action items to the ECSS wiki once the meeting has concluded. They provide ECSS information to the XSEDE PM office and relay information from the PM team to ECSS. They also maintain the ECSS wiki and mailing lists. They coordinated the gathering of information for the Q7 report and have published all of the information to the wiki. At the quarterly meeting held in Las Vegas, Gendler held a session for all of ECSS management to demonstrate how to use Sciforma and she also met with Mike Northrop and Justin Whitt, spending an extra afternoon testing and customizing Sciforma. User guides and new reports for Sciforma are being generated based on the needs of the managers. The goal is to train ECSS staff at XSEDE13 and have all reports, workplans, and data entry handled through Sciforma. 4.2.1.1 Extended Research Teams Support 1.4.1 An ESRT project is a collaborative effort between an XSEDE user group and one or more ECSS staff members, whose goal is to enhance the research group’s capability to transform knowledge using XSEDE resources and related technologies. Typical ESRT projects have a duration of several months up to one year and include the optimization and scaling of application codes to use 100,000 nodes or more per job; aggregating petabyte databases from distributed heterogeneous sources and mining them interactively; or helping to discover and adapt the best work and dataflow solution for simulation projects that generate ~100 TB of persistent data per 24-hour run. A request for ESRT support is made by the PI of a research team via the XSEDE resource allocation process. If the request is recommended by the reviewers and suitable to be an ESRT

38 project, and if staff resources are available, a statement of work for up to one year will be developed in collaboration by the PI, the ESRT team leader, and the ESRT manager and project manager. The work plan will include staff assignments from the pool of available advanced support experts who have the necessary skills. The ESRT team leader, working with the ECSS project manager, will be responsible for project tracking and reporting and for requesting additional resources or assistance from XSEDE management as needed. Metrics that quantify ESRT requests this quarter and total active projects are provided in Table 1 and Table 2. Table 1 ESRT project metrics for this quarter Metric XRAC Startups/Edu Number of requests 9 12 Number of projects initiated 9 7 Number of work plans completed 4 2 Number of new work plans completed for previous quarters 4 0

Table 2 ESRT project breakdown Metric XRAC Startup/Edu Number of active projects 37 30

As of January 2013, there are 15.69 FTEs assigned to ESRT from NCSA, NICS, PSC, SDSC, and TACC. Table 3 summarizes the number of requests, unjustified/rejected requests, workplans completed (and as such projects in progress), and work plans still in process. All projects are contacted within the first week or two of being notified of getting recommended for ECSS support. Table 3 ESRT project metrics since XSEDE began. Requests Not justified Workplans In completed process Jun-Aug startups (2011) 12 8 4 0 Aug XRAC 10 2 8 0 Sep-Dec startups 12 9 3 0 Dec XRAC 11 6 5 0 Jan-Mar startups (2012) 11 4 7 0 Mar XRAC 13 6 7 0 Apr-June startups 12 9 3 0 June XRAC 11 5 5 1 Year 1 Totals 92 49 42 1 July-Sep startups 9 2 (+1) 5 2 (-1)

39 Sep XRAC 7 4 (+1)* 3 (-1) 0 Oct-Dec startups 7 (+1) 2 (+2) 2 3 (-1) Dec XRAC 9 (+1) 1 (+1) 4 (+4) 4 (-4) Jan-Mar startups (2013) 12 5 2 5 Mar XRAC 12 3 0 9 Year 2 to-date Totals 56 17 16 23  project with a workplan was closed due to unforeseen circumstances.  (+/-) numbers show the change from last quarter report

For the first 18 months, there has been one main challenge with managing the ESRT program: the management of projects and people. ESRT has a large distributed staff and many different projects with different start/end dates. In this quarter, we switched to using the project management software to manage the projects, which has made some parts of management easier. Other parts are not quite as easy yet, but that will be addressed as we better learn how to use all the functionality of the software. The following sections highlight a few projects that provide examples of the kind of work that is being done in the ESRT program.

4.2.1.1.1 Computational Studies of the Interaction of Time-Dependent Electromagnetic Fields and Charged Particles (Klaus Bartschat, Drake University) ECSS Project Team: Lars Koesterke (TACC) The goal of the project was to use XSEDE resources efficiently, and in particular special emphasis was given to Stampede, which contains Xeon Sandy-Bridge and Xeon Phi processors. They were very successful in optimizing the homegrown code for the Xeon processors. A speedup of about 1.6x on Stampede was achieved. They had limited success in using the Intel Phi for this project. The original code is MPI-only. This project was all about serial optimization. The developers of the code did not write the code so that the hot loops would have stride-one data access. This was addressed partially by using temporary arrays, but the problem is by no means solved. A complete rewrite is required, which that is not feasible as an ECSS project. Nevertheless the performance in the hot routine is now about 20 percent of peak, which is excellent. It was a very good collaboration with some good results ultimately, although the Xeon Phi’s were not utilized. They are preparing a paper for Physical Review Letters, which will present results from the much improved code. These results could not have been produced without the large performance increase. They are now tackling even larger calculations. “Overall, the experience with Stampede has been very positive -- Xiaoxu got going really quickly and managed to get a lot done. Chances are that he would never have been able to get such beautiful results if it hadn't been for Lars' help with the optimization (even 50% is worth a lot when it comes to using millions of SU's) and our selection as early users.” Klaus Bartschat

4.2.1.1.2 Large-Scale High Performance Computation of Aerodynamics, Free-Surface Flow and Fluid Structure Interaction (Bazilevs, UCSD) ECSS Project Team: Amit Majumdar (UCSD)

40 The overall objective of the collaboration was to profile the fluid structure interaction parallel code and suggest performance improvement possibilities and implement those in the code. The fluid part of the code is implemented using MPI and the structural part was a serial code. Although at the beginning there was thought about looking at the MPI scaling of the fluid code, once the project started it was decided that looking into shared memory parallelization of the serial structural part of the code was of interest to the PI. The PI was interested in using the fluid structure interaction code in the bigger perspective of Dynamic Data Driven Application Systems (DDDAS) framework and hence speeding up the serial structural part of the code was of interest. The structural code was profiled and routines were identified as candidate routines for parallelization. Both OpenMP and OpenACC, for GPUs, were explored for the structural code. Modest speedup was obtained. Help was also provided to the PI regarding writing his XSEDE allocation proposal. XSEDE staff worked with one of PI’s graduate student for OpenMP parallelization. The serial structural code was profiled, using gprof, and the results were analyzed to see where most of the time was spent. Two of the routines that were identified as candidates for shared memory parallelization were shell_SparseProd.f90 and shell_SparseCG.f90. They were parallelized using OpenMP. Lot of OpenMP implementations were tested for multiple loops within these two routines that perform sparse matrix product and conjugate gradient solution respectively. Various OpenMP tuning options (e.g. scheduling chunk size variation, binding threads to cores) were tested using different compilers (Intel, PGI) on various XSEDE machines (Trestles, Gordon, Kraken). Problems were tested for 1,800 and 18,000 element cases that the PI’s team provided. A graduate student of the PI also worked on this OpenMP parallelization effort. Modest speedup was obtained with the OpenMP implementation and the plot below shows the result on SDSC’s Gordon machine.

Structural OpenMP Code Speedup on SDSC Gordon (Intel Sandy Bridge Procs)

4.5 4 3.5 3 2.5 2 1.5 1 1 2 4 8 16 NormalizedSpeedup # of Cores

In the last quarter of the collaboration, OpenACC implementation of the serial structural code was explored on the GPUs of the Lonestar machine. OpenACC was first implemented in the SparseProd.f90 routine. Tests were done with the 1,800 element case. But timing analysis (done with PGI flags set during compilation) showed that too much time was spent in transferring data between the host and GPU than the time spent in computation by the GPU kernel. The OpenMP implementation and the OpenACC implementation were the two types of shared memory parallel codes provided as possible options for shared memory parallelization of the serial structural code. It is possible that these two may be combined if the PGI compiler can

41 effectively handle OpenMP and OpenACC. This is a potential future option. Further tests can be done on larger problems to see if the speedup continues to scale. It is possible that further optimization, using OpenACC, may be explored but that may require restructuring the code.

4.2.1.1.3 Modeling Ship Propellers (Chryssostomidis, MIT) ECSS Project Team: Vince Betro (NICS) To model the propeller "crashback" scenario, wherein a propeller quickly reverses direction while still moving ahead, CFD simulations are required to determine the loads on the propeller and propulsion system. The resulting flow about the propeller is both unsteady and very different from the design point. Thus, the propeller blades experience both the dynamic-stall and fully separated flow regimes, which cannot be modeled by existing propeller panel method codes. Here, the modeling is based on high-order spectral/hp element solver NEKTAR. NEKTAR was developed at Brown University and has been successfully employed to simulate unsteady 3D flow in complex geometry and at high Reynolds numbers. The project team has developed and implemented additional capabilities for the CFD solver required to model fast moving objects, such as propellers, without the need for re-meshing. In order to properly model propeller “crashback” physics within a reasonable time constraint, a highly optimized hybrid OpenMP/MPI code is sought. The conjugate gradient loop and many of its sub-functions have already been parallelized, and many loops have been replaced by calls to functions by each thread. These functions work on different chunks of data, wherein 16-byte memory alignment is preserved for each chunk and SSE code is generated. This has resulted in cutting the CPU time per time step roughly in half; by applying these parallelization techniques to more candidate sections of the code, the intention is to cut the CPU time in half again.

The ECSS project was able to get results for a paper for SC12, and it has been suggested to the PI that they continue to add OpenMP parallelism throughout the code.

Figure 1 Interaction between wind farms and the atmospheric boundary layer flow

4.2.1.2 Novel and Innovative Projects 1.4.2 The mission of the Novel and Innovative Projects (NIP) team is to provide proactive efforts to develop and sustain XSEDE projects by non-traditional (to HPC/CI) users. Activities range from initial contact to the conception and execution of successful projects, including those that receive extended collaborative support. The scope of NIP includes disciplines whose practitioners have rarely availed themselves of HPC/eScience resources in the past. It also includes demographic diversity, such as researchers and educators based at MSIs and EPSCoR institutions and SBIR recipients. Bringing these communities to XSEDE leads to the consideration of applications and programming modes that have not been the focus of HPC in the past, such as those necessary for data analytics and informatics, and of innovative technologies such as streaming from instruments, mobile clients, and the integration and mining of distributed, heterogeneous databases. The implementation of campus bridging processes and technologies will be particularly important for these communities. Biweekly teleconferences and the use of the project wiki and email list have been successful in catalyzing communications among team members, who benefit from each other’s contacts and expertise and share best practices. We have structured the team into task forces: Minorities; Genomics; Humanities, Arts and Social Sciences (HASS); Economics; Databases & Data Analytics; Geographic Information Systems (GIS) & Visualization; Matlab/Python/R/Java; Campus Bridging & Cloud Bridging, and Accelerators. Thus, some of our effort is focused on understanding and meeting the requirements of specific communities and disciplines, and some is focused on developing and sharing expertise in specific technologies that are likely to be required across communities.

43 Most of the members of the NIP team (5.1 FTEs) are also active in the ESRT, ESCC, ESSGW, and ESTEO areas. This allows them to participate in, or lead, the technical execution of ECS projects they have mentored in the initial stages of their XSEDE development. We have built a close connection between NIP and the XSEDE systems engineering, user engagement, and technology investigation services. Systems engineering staff now attend each of our biweekly NIP teleconferences and take note of the feedback our team members pick up from the potential and current users they contact and mentor. Since the beginning of the program (July 2011), the NIP team has initiated and executed 31 outreach events; engaged 59 groups of potential XSEDE users; and mentored 82 XSEDE user groups. Team members are involved in the technical execution of 13 active ECS projects and worked on four ECS projects that were completed by March 31, 2013. They are leading the planning of two possible future ECS projects recommended by reviewers. Below are some examples of NIP assisted projects that have made progress in this quarter. Michel Regenwetter and his team at U. Illinois study how to correct potential flaws and biases in decision making. They look specifically at decisions under risk and decisions with trade-offs over time, both of which are at the core of much human activity. The team has developed a very general mathematical modeling and statistical testing framework implemented in MATLAB for PC that they ported to Blacklight during their startup and ECS allocation. The initial contact, leading up to the formulation of the startup and ECS proposal was made by NIP HASS specialist Alan Craig. The ECS project they formulated is led by NIP and RT staff member Roberto Gomez. The MATLAB framework the team developed with ECS uses specialized statistical software based on “order-constrained statistical inference” algorithms still unavailable in commercial packages. During their startup allocation, Regenwetter’s team computed more analyses in two months on Blacklight than all PC-based analyses carried out in their lab over the past three to four years. They are now preparing a full production proposal for XRAC, aiming to organize different theories, models, and methods in systematic ways and to increase the scope and rate of analysis of behavioral research by orders of magnitude. Kenton McHenry and his team at U. Illinois received NIP assistance from Alan Craig and Dora Cai to obtain an XRAC allocation for their project From Raw 1940s Census Images to Searchable Information on Blacklight with ECS by Roberto Gomez. This work uses the 1940s Census as a test case for developing efficient methods to interpret and index cursive text, an important challenge in the meaningful digitization of hand-written records and other historical documents. The 1940 Census data consist of 4,646 reels containing about 5 million forms, each of which includes 50 rows and 44 columns, for a total of 11 billion cells. The processing of the forms can be split into three stages: (1) aligning the form and cutting out the cells, (2) extracting descriptive features for each cell, and (3) creating indices for the entire collection of cells. For each cell, a feature vector is calculated to serve as a description of the cell’s content. The group extracts a 30-dimensional feature vector for each cell and builds two types of indices to later search the collection’s data. The first, a simple binary tree, requires during its computation around 50 GB of memory to hold all feature vectors and the corresponding distance matrix for a single column and single reel. The second index, a modified version of a Lucene textual index, is built through an I/O-intensive process, requiring a total of 8 GB of memory equally divided between RAM disk and computational needs. While the processing of the data is highly parallel, each of the stages required at least 4 GB of memory per process. Because many HPC systems have more limited amounts of memory available per core, PSC Blacklight’s large shared memory proved to be helpful for this calculation, avoiding the additional cost of an idle core that running stage 1 and 2 on many systems would entail. The final stage requires 50 GB of memory per process when building the binary tree, which is only provided by a shared-memory system such as Backlight;

44 the machine also speeds the computation of the Lucene index by virtue of a very fast I/O, enabled by using memory as a RAM disk. The figure shows digitized hand-written census form with selected cells highlighted. The problem is to (1) align forms and cut out cells, (2) extract descriptive features for each cell, and (3) create indices for search queries.

4.2.2 Extended Collaborative Support Service – Communities 1.5 4.2.2.1 Overview ECSS Communities focuses on collaborations that impact large numbers of users–those using community codes (ESCC), those using science gateways (ESSGW), and those benefiting from XSEDE’s education, outreach and training activities (ESTEO). Thirty-one ESCC projects (5.35 FTEs) and 20 ESSGW projects (4.97 FTEs) are in progress. 12 of the ESCC projects are part of the Trinity and Allpaths-LG work being undertaken for multiple user groups. In addition to the user-requested projects, ESCC and ESSGW staff continue with internally initiated projects, such as optimization and development of commonly used community codes, work on gateway and workflow Use Cases for the XSEDE architecture team, and debugging of grid software on new architectures. This quarter was also active for ESTEO (3.78 FTEs), featuring a 300-person online parallel programming course, an Introduction to OpenACC tutorial presented simultaneously at one primary and eight satellite labs, regional workshops, and conference activities. ECSS staff are significantly involved in the XSEDE13 conference, filling key roles in the organizing committee including general chair, chair of the technical program, and chairs of several of the technical tracks and visualization showcase. Many staff also submit contributions to the conference–papers co-authored with the user community, tutorial submissions, and poster, visualization, and BOF submissions. The load on both staff and the conference committee increased significantly this quarter as submission deadlines approached.

45 The ECSS Symposium continues each month and is open to the public to highlight work going on in ECSS projects and allow ECSS staff to learn from one another. The topics this reporting period were:  Graph Analytics: An XSEDE Introduction. Presenter: Nick Nystrom (PSC)  I/O Analysis for the Community Multiscale Air Quality (CMAQ) Simulation. Presenter: Kwai Wong (NICS), PI: Joshua Fu (University of Tennessee)  Visualization of Volcanic Eruption Simulations (CFDlib). Presenter: Amit Chourasia (SDSC), PI Darcy Ogden (SIO, UCSD)  Using Hybrid MPI/OpenMP Approach to Improve the Scalability of a Phase-Field- Crystal Code. Presenter: Reuben Budiardja (NICS), PI Katsuyo Thornton (U Michigan) The symposium audience includes the user community, Campus Champions and staff. 4.2.2.2 Extended Support for Community Codes 1.5.1 Extended Support for Community Codes (ESCC) efforts are aimed at deploying, hardening, and optimizing software systems necessary for extensive research communities to create new knowledge using XD resources and related technologies. ESCC projects are focused on helping users with community codes and tools on XSEDE systems. There were eight requests for ESCC support over the past quarter. However, only four have progressed to the workplan stage. The other four projects were either closed because the PI wasn’t ready, or the PI wasn’t responsive, or the issue turned out to be simple enough to be handled in the short term via the ticketing system. Of the four projects that went forward, two were from XRAC requests, one was from a renewal request, and one was from a startup request. This was slightly unusual in that the XRAC requests outnumbered the startup requests. Usually, the startup allocations have more need of assistance than the XRAC users. The four projects initiated covered a range of community codes. These projects range from porting a watershed resource model from a Windows platform to Gordon to optimizing the WRF adjoint model for the XEON Phi on Stampede. There are also projects to improve the large-scale parallel I/O performance of EPICC V2, a plasma fluid model, and a new project to provide guidance to the user community on building CESM applications on XSEDE resources. The two internal projects, Collaboration with the Broad Institute: Genomics Community Capabilities and the Amber Sustainability Project, are approaching one year. The genomics project is primarily focused on adapting the community codes Trinity and Allpaths-LG for use on HPC systems. The group at PSC, led by Phil Blood, has assisted over 15 research groups with the use of these codes and is working with the developers of Trinity to incorporate options into the code to improve HPC performance. In the next stage of the project, we will expand Trinity to other XSEDE resources, such as Stampede and Gordon, and add documentation for XSEDE users. The Amber project is focused on ensuring that each XSEDE system has an up-to-date version of AMBER. Over the past quarter, there was a problem with AMBER 12 on Blacklight. The lead consultant, Ross Walker, worked with PSC staff to resolve the issue and get AMBER to build correctly for the SGI. The next stage for this project will be to add documentation and test cases for XSEDE. Below is a team summary from an active ESCC project to provide an example of ESCC support. This project involves the use of analytics on Gordon to create a tool that allows researchers to index and search videos based on the content. This project also involves Campus Champion Fellow Dirk Colbry from Michigan State.

46 Project Title: Interactive Large Scale Media Analytics PI Name: Virginia Kuhn, Michael Simeone Project Team: Luigi Marini, Liana Diesendruck, Ritu Arora Summary: We have extended Medici (version 2) to support content-based retrieval of videos. Users can play videos from within Medici (Figure 1), but the system has now been extended to recognize and store sections of videos, in this case called “shots.” A user can select a shot and search the entire database based on a list of pre-computed feature descriptors (Figure 3 and 4). For each feature descriptor we list the top 10 results. When a user uploads a video, three preprocessing steps are executed: shot identification, extraction of feature descriptors, and encoding of original video in HTML5-compatible format for web viewing. The extraction steps are offloaded asynchronously to the compute node over the RabbitMQ event bus using the new Medici 2 extraction mechanism. We have also implemented web-based viewers for the sections of a video. They are shown both as a gallery and as a list of individual shots on the page where the user accesses the video (Figure 2 and 3). When a user plays a video the gallery advances to the appropriate key frame for the shot being shown. Based on an extensive content-based video retrieval literature review, we have implemented the following content-based image retrieval feature descriptors and appropriate distance metrics: HSV color histogram, HSL color histogram, YCbCr color layout, Gabor texture, edge histogram, discrete cosine transform, rough color layout, color and edge directivity descriptor. Shots are identified using a modified version of the cinemetrics library. The first frame of a shot becomes the key frame of that shot. Key frames are indexed using the feature descriptors listed above. The system has been deployed on Gordon dedicated resources (one I/O node and four compute nodes). The I/O node runs the web front end, the MongoDB database, and the RabbitMQ event bus. Each compute node runs one instance of each extraction and indexing processes. We also have explored use of the storage resources (Ranch and Gordon file-system) and mentored the Campus Champion Fellow’s integration into the project. Also updated was a parallel application—genetic algorithm for content-based image retrieval—which could be used for clustering image segments and hence images in the future.

Figure 2. User can play video from the browser.

Figure 3. Shots identified using computer vision algorithms are shown in a gallery view below the video player. The highlighted shot advances as the user plays the video or jumps to a particular time. Users can tag video on the right.

Figure 4. Identified shots and their time coding listed sequentially. Clicking on the "Similar" link will executing a content-based search against the extracted feature descriptors.

Figure 5. Query results. Original query key frame on the left and results on the right. Results are ordered by relevancy and by featured representation.

4.2.2.3 Extended Science Gateways Support 1.5.2 The Extended Support for Science Gateways (ESSGW) team provides assistance to researchers wishing to access XSEDE resources through web portals and science gateways. The group assists both new and advanced groups and has experience in the use of web technologies, grid software, fault tolerance, complex workflows, and security and accounting aspects of the program. Key activities this quarter:  Over 5.5 million SU’s have been charged through gateway community accounts from 38,001 jobs. These jobs were executed on behalf of 1,528 unique end users.  Continued organization of interleaving biweekly gateway community and developer meetings. o The biweekly gateway symposia on gateways and gateway-related efforts included the following presentations: . Experiences with XSEDE and OSG Job Submissions by Yan Liu (NCSA) . Using the Eclipse Parallel Tools Platform in Support of High Performance Computing on XSEDE Resources by Jay Alameda (NCSA) . The Nimbus Platform: IaaS cloud computing capabilities of acquisition and management of customized on-demand resources, suitability for scientific applications, and impact on Science Gateways by Kate Keahey (ANL) o The gateways staff calls occur on alternating weeks and focused this quarter on Use Cases, community code gateways, gateway user attributes, and educating staff on XSEDE Engineering process.  The ESSGW effort during this period devoted significant effort to the Science Gateways and Scientific Workflow Use Case discussions with the XSEDE Architecture and Design team.  Continued discussions with XSEDE Architects in strategizing gateway transition to emerging XSEDE Architectural components.  PY2 unspent funds effort of half an FTE was awarded to ESSGW. This experimental effort will proactively work with science PI’s in setting up new gateways. PI’s have to be willing to share research scripts and eventually “take-over” the gateway. Concrete outcomes will be to improvise on the foundational blocks like community gateway allocations and integration with XSEDE security. Initial target is to align with PI Benoit Roux (U Chicago).  Participated in the SD&I Reviews related to Science Gateways and working with Operations in review, test and SP coordination activities.  Migrated to using Sciforma project management software.  Chairing XSEDE13 software and software environments technical program track.

4.2.2.3.1 Highlights from ECSS SGW projects

4.2.2.3.1.1 Dark Energy Survey Simulation Working Group This ESSGW project focused on transition from Ranger to Stampede. In collaboration with TACC system administrators and the Globus team, GRAM5 was installed on Stampede. A joint effort for DES and UltraScan gateway was made to test GRAM5 for different types of jobs. In addition to the middleware testing and migration, ECSS staff worked with the DES team to run benchmark comparisons of the performance of their LGADGET code on Ranger and Stampede.

50 During this quarter we also extended the DES workflow to include post-processing with the applications RNN and Rockstar. Rockstar is a non-MPI parallel code: It needs to be started on a master node with single CPU, and then a second job needs to be submitted with the required processors to start processing. Communication between master and workers use TCP. Setting the walltime for such jobs is very tricky. We created a wrapper script to do the job but there are some manual steps to do this processing. In the upcoming quarter, we will work with TACC administrators to find a better solution for running Rockstar in the DES workflow. ESSGW staff co-authored with the DES collaborators a paper that was submitted for XSEDE13.

4.2.2.3.1.2 VLAB The VLAB project provides petascale computations in mineral physics with the Quantum ESPRESSO codes. VLAB support activities were minimal during this quarter because of the unavailability of the primary VLAB developer. The ESSGW team was able to reach the developer near the end of the reporting period and did make progress on some technical issues. Detailed activities during this reporting period include:  Continued work with primary VLAB developer on integrating their portal with current version of Apache Airavata Gateway middleware. This will allow VLAB to outsource migration to different XSEDE resources to the ESSGW team.  Improved the gateway specifying a local file as an input staging URL. The gateway previously only supported GridFTP urls for input file transfers. The effort involved testing local file protocols in the gateway job submission clients.  Investigating ways to support VLAB on Stampede.

4.2.2.3.1.3 Ultrascan ESSGW assisted the UltraScan team in transiting to Stampede from Ranger. We initially explored the integration of GSI-SSH into UltraScan’s job management infrastructure. This work was performed and contributed also to the open-source Apache Airavata project. However, job monitoring in this approach is a challenge. Fortunately, we instead were able to make successful transition to GRAM5 on Stampede in collaboration with TACC systems administrators and the Globus team. This involved the conversion of SGE modules to SLURM modules and significant testing. with several issues reported and resolved by TACC administrators and Globus developers. Testing included both serial and MPI jobs. We also identified an issue where job threads on GRAM5 were not correctly disposed of after the jobs completed. We developed an internal workaround solution for UltraScan, which was superseded by suggested fixes from the Globus team. These fixes are still being tested. The ESSGW team has co-authored a paper in collaboration with the PI for submission to XSEDE13.

4.2.2.3.1.4 CyberGIS During this quarter, the ESSGW team continued to engage political science domain users in developing a spatial optimization algorithm for redistricting. A scalable parallel genetic algorithm (PGA) library was further enhanced. The PGA library work and experiment results were reported and submitted to the Journal of Parallel Computing. Using this library, we are developing another application in agricultural science for land use optimization. Three abstracts were prepared and submitted to XSEDE13 to report our progress in cropland use optimization, CyberGIS integration with a NSF geospatial data facility, and spatial pattern analysis of disease mapping, respectively. The following research publications have been produced:

51 Wang, S., Anselin, L., Bhaduri, B., Crosby, C., Goodchild, M. F., Liu, Y., and Nyerges, T. L. “CyberGIS Software: A Synthetic Review and Integration Roadmap.” International Journal of Geographical Information Science, DOI:10.1080/13658816.2013.776049 Liu Y.Y., Wang S. "A Scalable Parallel Genetic Algorithm for the Generalized Assignment Problem," Parallel Computing, Under review. Liu Y., Guo M., Wang S. "Large-scale Land Use Optimization by Enhancing a Scalable Parallel Genetic Algorithm Library." XSEDE13, July 2013, San Diego, CA. Under review. Padmanabhan A., Youn C., Hwang M., Liu Y., Wang S., Wilkins-Diehr N., Crosby C. "Integration of Science Gateways: A Case Study with CyberGIS and OpenTopography." XSEDE13, July 2013, San Diego, CA. Under review. Padmanabhan A., Wang S., Cao G., Hwang M., Zhao Y., Zhang Z., Gao Y. "An Interactive CyberGIS Environment for Massive Location-based Social Media Data Analysis." XSEDE13, July 2013, San Diego, CA. Under review. Next quarter effort will continue to work on the two spatial optimization problems using the PGA library, i.e., the redistricting problem in political science and the cropland-use optimization problem in agricultural science. We will continue to enhance our middleware services to manage our use of XSEDE resources, in particular Stampede at TACC and Gordon at SDSC.

4.2.2.3.1.5 Open Science Grid Support In XSEDE-OSG collaboration, the sampleapp package was extended to include the SLURM job scheduler. Parameter-sweeping examples are included in Condor support to illustrate how to use XSEDE-OSG to do parameter-sweeping study. Yan gave a gateway community talk in the bi-weekly meeting on Jan 18 to share the experience of using XSEDE-OSG. Yan served on the review committee, together with IU's Suresh Marru, for the option selection review of SDIACT-105 API for Science Gateways, as part of the soft engineering process in XSEDE software development and integration (SDI). We continue to contribute our experience in developing gateway-wide job and data tools as part of gateway team's effort to assist the design of XSEDE architecture.

4.2.2.3.1.6 Galaxy Installation on Mason Louise Laurent and Rathi Thiagarajan expressed interest in using Galaxy on XSEDE resources to perform RNA sequencing. Although they were satisfied with the capabilities of the public instance of Galaxy, they found the file transfer and storage limits inadequate for their needs. Because of these limitations, they were forced to use command line tools on Trestles instead of using the integrated, graphical, web-based interface of Galaxy that they preferred. ESSGW staff has installed an instance of Galaxy on a Quarry gateway hosting VM configured to run sequencing jobs on the Mason cluster. We used GlobusOnline to transfer data from Laurent's TSRI lab to the DataCapacitor file system at IU, imported the data into Galaxy, and showed the researchers how to perform subsequent transfers and imports themselves. Transferring data this way proved to be much faster and more reliable than the web upload mechanism provided by the public instance of Galaxy. It allows significantly larger datasets to be transferred and stored. ESSGW collaborated with NCGAS project to install and configured all of the backend application required by Galaxy.

4.2.2.3.1.7 CIPRES

52 This quarter, effort focused on an initial implementation of the CIPRES REST API. The result will be restructuring job submission to be more responsive, placing less load on the web application by offloading file staging and job submission to a different process. The REST API requires a more clearly defined job state transition than the CIPRES Portal does, so the restructuring will add that clarity as well. 4.2.2.4 Extended EOT Support (WBS 1.5.3) Over this quarter, ESTEO continues to contribute to many tutorials, mentoring opportunities, meetings, as well as presented numerous talks and presentations at scientific and high performance computing conferences. In many cases, ESTEO staff initiated the contributions, developing submissions for XSEDE13, especially with Amit Majumdar serving as the Technical Program Chair for XSEDE13. Additionally, ESTEO staff served as reviewers for not only XSEDE13, but also for other conferences and workshops. Some of the training events for this quarter include an introduction to using Hadoop on SDSC’s Gordon resource, which included contributions from Mahidhar Tatineni; a course on preparing data for use with VisIt, at SDSC, which was organized by Amit Chourasia (SDSC); and an introduction to the Neuroscience Gateway organized by Amit Majumdar at SDSC. Of particular note is the encore presentation of the very successful Introduction to OpenACC tutorial that was presented from PSC, in January 2012, with satellite labs at eight sites across XSEDE. XSEDE ESTEO staff provided the on-site support at the various sites for the tutorial, and as before, John Urbanic prepared and presented a significant amount of the course material. Additionally, several ESTEO staff, including Marcela Madrid from PSC and Mahidhar Tatineni, supported online version of the Berkeley course on parallel programming that is being offered this spring. The course attracted over 300 people to register, which is prompting formulation of new policy regarding effective mechanisms to support large online courses with XSEDE resources. Several ESTEO and XSEDE staff conducted a successful XSEDE regional workshop at University of Texas at El Paso, which featured content tailored to community interests and needs. ESTEO and XSEDE staff also were involved in planning for two upcoming XSEDE regional workshops, one at Florida International University and the other at the University of Maryland, Baltimore County. Staff provided outreach and training activities at a wide variety of conferences, including the American Institute of Chemical Engineers annual meeting, the Analytics 2012 Conference, and the eScience conference. Additionally, a number of staff, including Ross Walker, mentored students this quarter. We also continue to review web tutorials and provide new content. Requests for ESTEO time are discussed at a monthly call involving XSEDE TEOS staff and XSEDE Training staff. The call is led by Jay Alameda, lead for ESTEO. The full list of training courses for the period is available online at https://www.xsede.org/web/xup/course-calendar. 4.3 Education and Outreach The Education and Outreach (E&O) team is making very good progress achieving its goals and objectives engaging and serving the community. The team has developed a set of services documents that are being used to improve information about the resources and services available on the XSEDE web site.

53 4.3.1 Education 1.6.1 During the first quarter of 2013, we continued to promote the adoption of formal programs in computational science. We also participated in a number of campus visits, outreach events, and education/training workshops. Work also continued on the repository of computational science education materials and on the competencies for computational science education. The team submitted a supplemental proposal for the 4th annual International HPC Summer School, which will be held June 23-28 in New York City in collaboration with PRACE and RIKEN. A call for papers for the 6th annual Extreme Scaling Workshop, which will be held August 15-16 in Boulder, Colorado, was issued in collaboration with Blue Waters. 4.3.1.1 Assisting with the Creation of Formal Programs Our work continued with institutions interested in starting or expanding computational science education programs. A draft proposal for a Ph.D. program in computational science at Clark Atlanta University was reviewed as a step toward submission for formal review. We assisted Montgomery College, Maryland with the creation of an introductory seminar on computational biology for students being funded through an NSF S-STEM grant. A visit was made to the college to meet with administrators and faculty concerning the steps to create a formal computational science program. Other discussions with faculty and administrators about computational science programs were initiated at University of Texas El Paso, with a consortium of Colorado community colleges through their STEM advancement program, and with Philander Smith College (Little Rock, Arkansas). 4.3.1.2 Presentations at National Meetings We made a poster presentation about computational science education at the SIGCSE conference in Denver, Colorado, in March. The poster fostered discussions with four potential XSEDE clients for education program support. Shodor Education Foundation also sponsored a Little FE build-out at that conference. 4.3.1.3 New Course Development Working with our colleagues at University of California Berkeley, Susan Mehringer from Cornell, and Jay Alameda and other ESTEO staff, we launched the parallel computing course for online use. The course was originally restricted to 300 registrants. That level was reached within 24 hours. We added an additional 75 to a waiting list. Because of restrictions on accounts for people from T-6 countries, this was pared to 345 students that were admitted. Of those, 252 were from the United States with the remainder from Europe, Asia, and Africa. The drop-out rate was high with only 172 students actually taking quizzes and actively participating in the course. At this stage, only 36 are completing the computing exercises. We have begun discussions on how to connect the course in a formal way with educational institutions in the United States to reduce the dropout rate and make the effort at live interaction more productive. We expect to try a new version of this course and a second course next academic year. 4.3.1.4 Participation in Outreach and Training Events We participated in a number of outreach and training events over the past quarter. Aside from the national conferences discussed above, we conducted a number of education oriented workshops both in person and via webinar. These are summarized in table 8.2.1. The events included keynote presentations at workshops, workshop sessions on computational thinking, and discussions with faculty on building formal computational science education programs.

54 Table 8.2.1 Education Related Events in First Quarter of 2013 Event Description Date Lead Instructor Florida International Consultation with faculty on joint 1/7 – Robert Panoff University education activities 1/8 Barton College Computational thinking workshop 1/16 Robert Panoff Oneonta College Colloquium and discussions with 2/6-2/8 Robert Panoff faculty Clemson University Keynote talk on computational 2/11 - Steven Gordon CI-Days science education; workshops on 2/13 Robert Panoff computational thinking UTEP Regional Computational thinking and 2/19 Steven Gordon Workshop curriculum workshops U Texas San Antonio Computational thinking workshop 2/23 Robert Panoff Montgomery College Overview of computational biology 2/28 Steven Gordon Computational and discussion of a formal Biology curriculum UNLV XSEDE Presentation on computational 3/14 Steven Gordon Outreach science curriculum

4.3.1.5 Creation of Repository of Education Materials We have completed the design of the repository of education and training materials and have been working on implementation steps over the past quarter. This includes preparation of instructions for potential reviewers, changes to the site to make searching for materials more straightforward, and updates to the other information on the HPC University site relating to news, career, and internship opportunities. 4.3.1.6 Other Activities We have begun to work on the definition of competencies for data-intensive computing. A draft set of competencies has been completed and made available for comment to a cross-section of experts from the field via an online survey. We will be following up with several webinar discussions and expect to have a publically available set of competencies in place by the end of the next quarter. We have also been working on updating the materials being used in computational thinking workshops. In cooperation with Shodor, we will be adding example exercises associated with the datasets and models that have been used in computational thinking workshops. Those will be reviewed as part of the effort for the educational repository so that the materials are widely available. We have also been coordinating our education and training efforts with the Virtual School for Computational Science Education. That has included planning for summer workshops, use of the competencies to tag the summer workshop activities, and possible future offerings of certificates for the participating students. Finally, an article summarizing the XSEDE education program authored by Steven Gordon was published in the January-February issue of Computing in Science and Engineering.

55 4.3.2 Outreach 1.6.2 4.3.2.1 Under-represented Engagement SURA coordinated a number of campus visits, conferences and presentations to engage the under-represented community in XSEDE.  A two-day XSEDE regional workshop held at UTEP on February 19-20 attracted 100 registrants. UTEP serves 22,700 students, 77 percent of whom are Hispanic. TACC provided training staff for the technical sessions, which included Research Data Management, Introduction to Scientific Visualization, Introduction to FORTRAN programming, Introduction to C Programming, Introduction to Parallel Programming on Stampede, and Introduction to Parallel Programming with MPI and OpenMP. Steve Gordon presented Computational Thinking and Adding Computational Science to Curriculum. Lorna Rivera is preparing the workshop evaluation report.  February 22, Dwayne John and Amy Szczepanski of NICS presented a mini-workshop at Clark Atlanta University as the inaugural event for the new CAU Visualization lab.  February 28–March 2, exhibited at the Emerging Researchers National Conference (ERN) in Washington, D.C. Over 900 students and their advisors attended the conference, and XSEDE staff interacted with 70 participants. Dwayne John (NICS), Samuel Moore (TACC), and Linda Akli (SURA) staffed the table and presented a min- workshop on high-performance computing with Mike Smith, Academic Program Director, Intel Software Partners and Emerging Markets. John presented “Introduction to High-performance Computing: It’s Not Only for Computer Scientists.” Student contact information was collected for followup.  March 4, participated with Marcela Madrid (PSC) of the NIPS team in a campus visit at Howard University. Marcela provided an overview of XSEDE and NIPS. The visit was organized by Marcus Alfred, the campus champion and attended by one representative from the mathematics department.  Five XSEDE MSI representatives were sponsored to the NSF Research Data Management Workshop held March 12-13 in Washington, D.C. This workshop provided the opportunity for the participants to submit position papers, hear from agency program officers about federal initiatives in managing research data, and participate in breakout groups to identify issues and develop recommendations. A lunch session was held with Bob Chadduck, the NSF program officer, and the MSI representatives, to share thoughts and concerns. The MSI reps that were sponsored to the meeting presented to the Minority Research Community monthly call at the end of March..  With Steve Gordon, held a second call on March 19 with Philander Smith College in Arkansas to provide an overview of XSEDE education for the senior administration.  Several MSI representatives submitted papers to the XSEDE13 EOT track including Mark Jack (FAMU), Pat Teller (UTEP), Peter Molnar (CAU), Rachel Vincent-Finley (Southern), and Hongmei Chi (FAMU).

4.3.2.1.1 Allocations A key measure of SURA’s impact is the number of research allocations that result from the efforts. Mark Jack (FAMU) is successfully using his XRAC allocation, which has been transitioned from Ranger to Stampede. New allocations have been made at Central State University (1 PI/Startup); Florida A&M University (3 PIs/2 XRACs and 1 Startup); Hampton (1

56 PI/Startup); Tennessee State University (3 PIs/2 Startups and 1 Education); and UTEP (3 PI/2 XRACs and 1 Startup). 4.3.2.2 Speakers Bureau XSEDE Outreach is moving toward a more community-driven agenda. Events in the first quarter reflected this in the balance between campus outreach and national and professional conferences. Four events were staffed by XSEDE, and feedback from these sessions strongly shows that smaller, more focused interactions with the community are generating stronger, more tangible impact.

4.3.2.2.1 Local Events XSEDE was invited to present workshop material at the 2013 Harvard University Computefest. Hands-on sessions introduced XSEDE, including a detailed walk through of services available through the portal without an allocation. Twelve of the 17 participants created portal accounts (the other five already have allocations). The second workshop was an encore presentation of the visualization tutorial delivered at XSEDE12, which introduces Paraview and VisiT. Both sessions were well-attended and received. The second local event was an Introduction to XSEDE seminar at the University of Nevada–Las Vegas following the XSEDE XRAC/Quarterly meeting in March 2013. The timing and location were chosen to leverage the travel commitments of XSEDE staff, enabling the program to be presented by XSEDE leadership who might not otherwise be available. Three presentations were given: an introduction, STEM highlights and examples from ECSS, and an overview of the XSEDE Education program. Contact information was collected from the 22 attendees and the appropriate XSEDE teams are following up with them.

4.3.2.2.2 National Events XSEDE also exhibited at two national events. The Richard Tapia Celebration of Diversity in Computing attracted over 500 students, academics, and STEM professionals who shared experiences, insights, and opportunities for building and sustaining careers in computation and STEM. Seventy-six students provided contact information for follow-up and have been invited to apply for the XSEDE Scholars program. Students were also provided with information about applying to the Student Engagement opportunities for the summer. XSEDE exhibited at the spring meeting of AAAS in Boston in March 2013. XSEDE staffed a booth during the exhibits and Family Science Days events but did not generate significant interest. This result has been seen at most of the national events at which XSEDE has exhibited. Without a major speaker, a workshop or tutorial to generate interest in XSEDE, national conferences are not a cost-effective way to raise awareness of XSEDE services and generate new projects and users. 4.3.2.3 Student Engagement Ruth Kravetz joined the Rice team to replace Alice Fisher to coordinate Rice’s activities. The Rice team revised and posted the XSEDE Scholars application to include information about specific programming background and some selected additional demographics. Rice used new and existing contacts as well as the faculty council, Henry Neeman, and others to increase publicity for the program. Rice made connections with potential and existing XSEDE Scholars at the Tapia Conference. Rice initiated a Facebook relationship with Native communities to increase the number of Native Americans with knowledge about XSEDE. Rice is establishing an XSEDE Scholars “Elder Statesmen” Mentoring program for two to three XSEDE Scholars. In return for the travel grant and a second or third year in the program, “elder

57 statesmen” will support new XSEDE scholars in June-July with weekly assistance on C homework assignments, etc. and will support the younger scholars during the XSEDE13 student programming contest. Tentatively, Paul Delgado, Manuel Zubieta, and possibly Grace Silva will be the XSEDE Elder Statesmen for Cohort 3. Rice started plans for the three XSEDE Scholars/Minority Faculty Council events at XSEDE13. Ruth Kravetz co-wrote an outreach paper for XSEDE13 with Linda Akli about the positive effect of collaboration among MSI Outreach, campus champions, and the XSEDE Scholars program The Minority Faculty Council members are recruiting XSEDE Scholars and some, including UTEP, are assembling student groups at their sites for group webinar viewings. MFC members have also spent time during this quarter working on MSI recruitment with Linda Akli. Rice met with outgoing and incoming campus champions to establish connections between XSEDE Scholars and Campus Champions. Few students participated in the January 16 webinar offered for Scholars, probably because it was immediately after the winter break. We worked hard to promote the March 7 webinar as an XSEDE13 preview and had 60 sign up; 35 watched the live webinar, and at least six watched the recorded webinar.

4.3.2.3.1 Timelines/plans for next steps The Empowering Leadership Alliance (ELA) and XSEDE webinars will continue to be offered throughout the spring. Planning for the meetings for XSEDE Scholars Program at XSEDE13 began in February and will continue through May. The team will develop differentiated online (pre-recorded or live webinar) training for new XSEDE Scholars to include an intro track (C programming and intro to Linus) and for intermediate and advanced users. Rice will work with the Minority Faculty Council members and the outreach team to develop a structured agenda for XSEDE13 that will culminate in: tangible activities in year 3 for the Minority Faculty Council and XSEDE leadership collaboration; and in opportunities for student internships for XSEDE scholars and other underrepresented students.

4.3.2.3.2 Summer Engagement Opportunities XSEDE staff and researchers have stepped up to offer 23 projects for consideration for the summer immersion opportunity for students. The projects have been posted on the XSEDE website and students were notified to apply. Applications were received from 113 students. The Student Engagement team is now working with the project supervisors to review and interview candidates and make selections as quickly as possible. 4.3.2.4 Campus Champions Four new campuses have joined the program: University of Nevada Reno, University of Connecticut, Florida International University, and M D Anderson at the University of Texas. There are new Champion representatives at the following five institutions: University of Texas El Paso, University of North Carolina, Rutgers University, Purdue University, and Temple University. There are now 138 member institutions and 186 Champions at those institutions. The types of institutions include: o Regular institutions: 74 o EPSCoR state institutions: 45 o Minority-Serving Institutions: 11 o Both MSI and EPSCoR: 8

58 4.3.2.4.1 Campus Champions Working Groups The Champions Working Groups continue to conduct a significant amount of planning work to benefit the overall program. The following are brief updates from each workshop group.  User Assistance: The group will be meeting biweekly during the next quarter to plan for XSEDE13. They have posed the idea of providing forums to aid in the communication between and among champions.  Outreach: The group is meeting every two weeks and is editing and renewing the information that was initially prepared for the XSEDE12 conference. New materials are being prepared for distribution at XSEDE13.  Student Champion and Regional Champion: These two groups have worked very hard to develop a prototype program in each of these areas. Documents which define these two programs and outline the goals for the two groups are in the final stages of development. The plans will be sent to the XSEDE Leadership in the fourth quarter. We plan to be able to announce these programs at XSEDE13.

4.3.2.4.2 Campus Champion Fellows The fellows program is beginning to wind to a close for the inaugural group of four. The fellows have been meeting monthly to discuss the program and assist with tweaking the program for the next cohort. Two of the Champions have presented their work at the monthly Champion Meetings and the other two will present in the fourth quarter. All will be a part of a panel to present to the larger community at XSEDE13. A meeting has been held with the Champions and the ECSS staffers who are acting as mentors for this group. This was very helpful to see how the interactions have gone and what progress has been made. Plans were made for the next application period, which will open the first week of April with the announcement of the new Fellows to be made no later than XSEDE13.

4.3.2.4.3 Campus Champion Metrics An initial meeting was held to determine what data might be available and how it might be used to assess the impact of the Champions. The question is “How can we measure the impact of a champion at a given campus?” We are initially looking at various pieces of data at a given institution prior to a Champion joining the program and that same data over the period after joining.

4.3.2.4.4 Champion Welcome Wagon: Mentoring There has been a concentrated effort to connect with new champions in a variety of ways. Initial email is sent to each new Champion, which welcomes them to the program, and then a follow-up email outlines the initial steps to get started. Additionally, a call is set up for the Champion to have a general discussion and to determine how things are going and where help is needed. This quarter nine such calls have been made. Some assessment is also made about whether the Champion might need or want a mentor to be assigned. We are testing the waters with a mentor program. A handful of champions have volunteered. Matching of mentors/mentees will take place as determined through the phone calls.

4.3.2.4.5 Outreach and Training A Champion 101 (New Champion Orientation) was held in January. “Champions—connecting your university” webinar was held in February for seven attendees in Nevada. Attempts to continue the development of a training plan for Champions has continued without a lot of success. We will continue that effort over the next quarter.

59 4.3.2.4.6 Open Science Grid OSG All Hands Presentation “Meeting the HTC Demand with Diagrid and XSEDE” Updated the Champions with new information from the OSG All Hands meeting.

4.3.2.4.7 Champion Staff Support This quarter a half-time support person (Dirk Colbry, Michigan State) was added to the champion support team. Initially this position will be looking at new champion support, mentoring, training, and web updates. 4.3.3 E&O Community Requirements and External Evaluation 1.6.3 4.3.3.1 Community Requirements Highlights Outcomes of the TEOS Strategic Planning Retreat—both recommendations and action items— have helped TEOS managers articulate and prioritize their plans and requests for additional support as we focus on Year 3. The outcomes have also given the TEOS Advisory Committee useful challenges where their guidance can be most valuable. The TEOS Advisory Committee met via conference call on February 26. During the 90-minute call, TEOS managers updated the Advisory Committee on Campus Bridging status and plans, as well as Education and Training activities to strengthen the ties between the two services. Managers shared their vision for integrating XSEDE certifications, assessments, core competencies, and roadmaps from novice to competent user, which they hope to share more broadly to help institutions develop their own programs. These held particular interest for faculty among the Advisory Committee. Managers also shared Use Cases they developed to demonstrate workflows among services. The advisory committee was particularly interested in values for the Use Cases for documenting impact and process variables. XSEDE evaluators and advisors held follow-up discussion addressing how the Use Cases might be strengthened into case studies for more rigorous evaluation. XSEDE evaluators also reviewed the next iteration of the user needs survey that is due to be distributed this spring. Its focus is on gathering more information on XSEDE TEOS early users’ computing modalities, research questions, and data challenges, to understand where new education and training materials and activities would have the greatest impact. An in-person Advisory Committee meeting was announced for October 2013. XSEDE13 is a major focus of the next quarter’s activities. We will have a meeting of members of the Advisory Committee who attend XSEDE13 to discuss Year 3 plans as well as priorities for audience surveys and interviews to support Year 3 initiatives. 4.3.3.2 External Evaluation Highlights

4.3.3.2.1 Training The evaluation team has been engaged in discussions regarding the new training certificate program in order to ensure a comprehensive and robust evaluation design is implemented in a timely manner. Longitudinal Tracking System: The external evaluation team successfully tested the longitudinal tracking system through the XSEDE User Portal (XUP). We’ve hired a graduate student, Michael Culbertson, to work on this task. The first two tests were done with the first cohort of the Student Engagement Program and the first two cohorts of the XSEDE Scholars Program. Findings from these pilot tests can be found in the Outreach section of this report. The evaluation team thanks XSEDE staff for their help in developing this system, especially Stephen McNally, Maytal Dahan, and Tom Maiden.

60 4.3.3.2.2 Education Steve Gordon and Berkeley’s James Demmel have engaged the evaluation team in the assessment of the CS267 Spring 2013 course on Applications of Parallel Computers. An online post-course evaluation form was developed for those students who completed the course’s requirements. Questions center on course delivery, content, difficulty, and student satisfaction. Additional items address student intentions to adopt XSEDE resources. The evaluation team also plans to administer an online evaluation survey to students who did not complete the requirements. This form will be significantly shorter and will include items related to reasons for not completing the course. The final course modules will be available May 2, 2013. An interim evaluation report will be submitted to Steve Gordon, James Demmel, and Scott Lathrop by August 2013 and will also be posted on the E&O Evaluation wiki.

4.3.3.2.3 Outreach Student Engagement: The first pilot test of the longitudinal tracking system through the XSEDE User Portal was conducted with cohort one of the Student Engagement program in February 2013. The analysis tracked interaction only in terms of training registration, allocations, and XSEDE resource use. Other forms of HPC/XSEDE engagement are not available. The most notable findings are listed below:  Most participants did not continue to interact with XSEDE after the program, unless they had been involved with XSEDE before the program.  Relatively few of the participants were introduced to XSEDE resources hands-on during engagement. A detailed report was submitted to Laura McGinnis and Scott Lathrop in February 2013 and can also be found on the E&O Evaluation wiki under TEOS Evaluation Reports. XSEDE Scholars Program: The second pilot test of the longitudinal tracking system through the XSEDE User Portal was conducted with cohorts one and two of the XSEDE Scholars Program in April 2013. The analysis tracked interaction only in terms of training registration, allocations, help desk tickets, and XSEDE resource usage. Other forms of HPC/XSEDE engagement are not available. The most notable findings are listed below:  Engagement with XSEDE was low among scholars: 25 percent (17/68) showed minimal activity beyond having an XUP account.  Four scholars (all graduate students) made use of XSEDE resources.  One scholar made use of the XSEDE Scholars allocation, a faculty member’s startup allocation, and the same faculty member’s subsequent full allocation. This scholar’s XSEDE engagement began before enrollment in the Scholars program.

Campus Champions: The external evaluation team has met with Kay Hunt and Laura McGinnis to finalize the Campus Champions Case Study plans. The first two cases were chosen due to their successful use of the XSEDE ecosystem of resources (TEOS, User Services, ECSS, allocations etc.) to increase awareness and use of XSEDE at their institution. Case 1 was chosen to highlight broad XSEDE success as a Champion. Case 2 was chosen to provide a detailed example of model participation in the Campus Champions program. The evaluation team plans on presenting case study data with practical and detailed examples that can be used by champions and XSEDE staff as reference materials to inform best participation practices and program planning. These studies

61 may also document and shed light on the initial new XSEDE user experience with Campus Champions and thus inform other areas of the XSEDE project on how to streamline this process.

Minority Outreach: Evaluators were engaged in the planning of all three regional events (UTEP, FIU, & UMBC). Data from the Nashville Regional Event (May 2012) was used to guide program planning. The evaluation team worked with program management to develop a training session survey, post workshop survey, and interview protocol. Evaluators attended a regional events at the University of Texas, El Paso (UTEP) on February 19-20, 2013. Two more events are planned for early in the next quarter. An interim evaluation report on each workshop will be drafted and submitted to Linda Akli, Laura McGinnis, and Scott Lathrop and will also be posted on the E&O Evaluation wiki.

4.3.3.2.4 Campus Bridging Evaluators continue to work with the Campus Bridging team to formalize the evaluation plans. Lorna Rivera and Lizanne DeStefano have attended regular Campus Bridging phone calls to inform evaluation plans. Evaluators also met with Craig Stewart, Rich Knepper, and Andrew Grimshaw in March to discuss evaluation needs and plans. The group determined the evaluation team should conduct interviews with four key groups at each GFFS/EMS pilot site to determine: (1) what worked well; (2) recommendations for improvement; (3) what can be learned; and (4) perceptions of the pilot project. The key groups to be interviewed include: (1) key contacts/leads; (2) technical/systems administrators; (3) end users; and (4) XSEDE staff. Interviews will begin in April 2013 in order to provide data to program management by the June 2013 NSF review.

4.3.3.2.5 Key Trends The evaluation has identified the following key trends from the last quarter within TEOS:  Relatively few of the student program (XSEDE Scholars Program and Student Engagement) participants were introduced to XSEDE resources hands-on during engagement.  Most student participants do not continue to interact with XSEDE after their program, unless they had been involved with XSEDE before the program.

4.3.3.2.6 Plans for Next Quarter In addition to the aforementioned plans, evaluators will continue to develop and implement the longitudinal tracking system. The evaluation team will also disseminate the formalized TEOS Case Studies that have been developed and shared with TEOS leads and program coordinators. This dissemination will begin with the TEOS Advisory Committee followed by the larger XSEDE community through the E&O Evaluation wiki page.

62 4.3.4 E&O Infrastructure 1.6.4 Sixty-seven events were added to the Education & Outreach Blog on the XSEDE website. Ange Mason (SDSC), with the assistance of a student, collects and vets events for their suitability for various components of our perceived audience, and rewrites the introductions as appropriate. The new Facebook presence initiated in December 2012 has begun to be populated with events and other items from the E&O XSEDE Blog. Most items added to the blog also appeared on the Facebook page, which is called Computational Science Education News. The new presence will be pushed out to collaborators and thence through their various outlets into the communities that care about computational science and research in our schools at all levels. Mason and Jim Ferguson (NICS) have been working with the Blue Waters project and the HPC University effort to re-design the site and feature a feed from the E&O Blog directly to the HPC University front page. This effort began in summer 2012 with an XSEDE-sponsored student intern at Shodor and has moved to regular web staff at Shodor. All items that get put on the E&O Blog at XSEDE are now duplicated in the news feed on the HPC University front page. There are also new web forms in place for those who wish to submit events and resources to share on the site. No missed milestones or risks activated for this quarter. 4.3.5 Campus Bridging 1.6.5 4.3.5.1 Discussion of campus bridging within XSEDE and within the national cyberinfrastructure community The XSEDE Campus Bridging team continues to have discussions with units in XSEDE and interested parties in the national community (OSG, EDUCAUSE ACTI) in order to disseminate XSEDE’s Campus Bridging initiatives and gather comments and suggestions for community needs. Barbara Hallock introduced the XSEDE Campus Bridging strategy at the OSG All Hands Meeting in March 2013. The two major thrust areas of XSEDE Campus Bridging, the GFFS Pilot Project and the Campus Bridging Cluster Software, made major progress in this quarter. 4.3.5.2 Definition of Use Cases and quality attributes for XSEDE’s Campus Bridging efforts The CB Team has continued to work with the Architecture and Design team to complete Use Case definition and the Level 3 Decomposition of Campus Bridging Use Cases and assist in the refinement of general Use Cases for all of XSEDE. 4.3.5.3 GFFS Pilot Project All of the GFFS pilot sites were given access to the GFFS software in this period and allowed to work with XSEDE resources as well as with resources at UVA and across participating campuses. Campus sites benefited greatly from UVA “office hours” in which UVA staff worked with pilot sites directly via phone or Skype in order to address installation and configuration issues. All sites have set up the software locally and tested file sharing within the GFFS, testing of job submission to Unicore on Kraken is still in progress, and the CUNY site has started the process of setting up Unicore locally in order to accept jobs submitted from XSEDE. TEOS Evaluation, Operations, and SD&I have all created means for capturing pilot site responses to the software and implementation processes. It is the responsibility for the Campus Bridging team to ensure that responses to the software are adequately captured in order to inform XSEDE and Genesis II Teams. 4.3.5.4 Rocks Roll Software Project The additional staff at Cornell dedicated to the Campus Bridging Software Project have created Rocks Rolls and RPM packages that are ready for distribution and testing. The Cornell staff will

63 develop a test plan that is approved by SD&I and carry testing, presenting verification of the test data to SD&I and Operations. Campus Bridging and SD&I have agreed on a plan for distributing the software packages, and the Campus Bridging team has a list of friendly users for testing the software packages from Linda Akli’s MSI Campus call. 4.3.5.5 Challenges in Program Year Two The remaining challenges in Program Year Two will be to continue to gather responses from the GFFS pilot sites and to appropriately span the gap between the pilot implementations of the software and the production implementation of the software scheduled for PY3. The Software Package Program will require adequate documentation and information for potential users to facilitate the installation and maintenance of clusters created with the software packages, by explicitly providing pointers to self-help resources upstream of XSEDE and forums and documentation available via XSEDE in order to ensure that users of the software will be able to find the correct resources to facilitate implementation and usage of the software packages locally.

64 5 Managing the Program 5.1 Governance The XSEDE project has established an organizational structure and governance that promotes efficient and effective project performance. As this is a distributed project involving 17 partner institutions and with many other stakeholders, including NSF and thousands of users, it was necessary to establish a governance model that balances efficiency and inclusiveness. The XSEDE governance model has strong central management to provide rapid response to issues and opportunities, delegation and decentralization of decision-making authority, openness to genuine stakeholder participation, and improved professional project management practices including formal risk management and change control. The XSEDE governance model is geared toward inclusion of, and responsiveness to, users, service providers, and the NSF scientific community. The various stakeholders have input through three advisory bodies, which have direct access to the XSEDE project director and the XSEDE senior management team through regularly scheduled meetings. In order to remain well informed of the requirements of the user community, XSEDE leadership receives advice and counsel from the User Advisory Committee, the XD Service Providers Forum, the XSEDE Advisory Board, and the TEOS Advisory Committee. These advisory committees are intimately involved with XSEDE management in guiding the project toward optimal operations, service, and support for users. The XSEDE project is managed by a senior management team consisting of the PI/project director as chair, the co-PIs and key leaders of major areas of the XSEDE project, the chair of the User Advisory Committee, and the chair of the XD Service Providers Forum. This team is constituted from those responsible for the day-to-day operation of the project and is the highest- level management body in the organization. In order to be responsive to both the user community and the set of service providers with whom we will collaborate, the chairs of the User Advisory Committee and the XD Service Providers Forum are members of this team. 5.2 XSEDE Project Office 1.1 The first quarter of the calendar year has seen great progress made on multiple fronts in the project office and its units. Of particular note, SD&I welcomes Shava Smallen to the role of deputy director in addition to her leading the SD&I testing team. This is a role much needed to assist in the many tasks in managing the SD&I area. We have also made progress on identifying a replacement for Susan McKenna, who departed as our External Relations lead in December 2012. We anticipate having a new lead aboard in the next quarter. XSEDE External Relations has been ramping up interactions with other area of the project to assist in communications needs, including additional stories about ECSS support activities and success. Planning for the next annual XSEDE highlights publication is also now under way; production of this project and many others is being greatly improved by XSEDE graphic designer Steve Duensing, who joined the project this quarter. Significant effort by SSE, A&D and SD&I along with appropriate subject matter experts (SMEs) has continued in the development of additional Use Cases to drive the architectural design and subsequent processes in XSEDE for delivering new capabilities to the community. Building on defined Use Cases, the A&D team has made progress on the Level 3 decomposition of the security components and has conducted Active Design Reviews of these components with the SD&I team. SD&I has also been wrapping up final elements of the original 100 activities planned as part of “Increment 2” work and initiating 16 activities prioritized by the User Requirements and

65 Prioritization Working Group for “Increment 3.” This was the first instance of fully using our prioritization process to guide the activities of SD&I. “Increment 4” planning also began this quarter, with over 30 activities identified and being presented to the UREP for prioritization. The Use Case tracking system has also progressed with the expectation that during the next quarter we will have a usable tool to provide visibility into the status of progress of Use Cases and their related components through the various stages from the A&D teams design on through to deployment for production operations. This will help many others in the project to understand timeline expectations for new capabilities and prepare to support them. Following direction from our first annual review, XSEDE has been developing a concise set of statements that present the projects mission, vision, and goals. Drafts of these were discussed with the XSEDE Advisory Board. In addition, we have also been making progress on defining an appropriate adaptation of the Baldrige Criteria. This also was discussed with the XAB.

5.2.1 Project Management and Reporting 1.1.1 5.2.1.1 Project Schedule and Risk Register The project schedule and risk register were updated to reflect the most current information about XSEDE. 5.2.1.2 Project Management Software Tool The team tested and installed version 5.0 of the Sciforma software and worked with Sciforma to resolve some JAVA related issues. ECSS is using the tool to manage ECSS resources across at least 100 projects. The Sciforma project management tool also provides the overall project schedule for XSEDE. One of the ECSS project managers demonstrated the Sciforma software at the March quarterly for ECSS WBS managers. 5.2.1.3 Reporting  The ECSS quarterly reports were produced.  ECSS metrics were reported to Dave Hart for inclusion in XSEDE metrics  The Q1 2013 Report was completed.  The project management team has continued working with XSEDE management on Project Year 3 planning. 5.2.1.4 Quarterly Meeting The seventh XSEDE quarterly meeting was planned and held March 11-13 in Las Vegas. In addition to other topics, the tailored use of the Baldrige Criteria for the XSEDE project was presented. Some of the key actions as a result of the discussion were:  State mission and vision for the future in simple and clear terms  Include in the vision both short-term and longer-term goals  Communicate the mission and vision to stakeholders  Assess the XSEDE organizational profile—what are the key organizational characteristics?  Identify future activities for every part of XSEDE and tie these to the vision, mission, and goals.  Determine metrics to measure progress toward goals  Identify how metrics influence future direction of XSEDE

66 5.2.1.5 Other Topics  Proposals for unspent funds were developed and submitted for review by the XSEDE PI and co-PIs.  A Project Change Request was developed and approved to incorporate the authorized unspent funds distribution and work scope.  Drafted Industry Challenge call for proposals for XSEDE leadership team review. 5.2.2 Systems and Software Engineering 1.1.2 5.2.2.1 Gathering New User Needs and Capabilities SSE continued its ongoing efforts to identify and elicit new user needs and capabilities in order to improve and develop XSEDE. SSE staff continue to participate in the user-facing project areas of XSEDE (ECSS, NIP, User Engagement, Campus Champions, etc.), joining in their regular conferences calls and email discussions to look for new user needs and capabilities. We also review reports of meetings, conferences, surveys, and other user outreach events for similar information. The information collected is of a widely disparate nature, from simple requests and fixes to things that are much more complex and have architectural significance. We also participate in discussions about how best to get these capabilities and needs into the XSEDE “pipe” for processing by the appropriate group. 5.2.2.2 Use Case Development Several areas of strategic importance for the XSEDE architecture have been identified including: science gateways, data analytics, high-performance computing, high-throughput computing, scientific workflows, and visualization. Appropriate subject matter experts (SMEs) have been identified for each of these areas; SSE works with these SMEs to help them generate effective Use Cases and quality attributes for each of these strategic activity areas. This quarter, additional Use Cases have been identified, including a series of foundational, or “Canonical,” Use Cases. SSE staff continue to work with SMEs to develop these new Use Cases as they are identified. After Use Cases have been properly developed they are put through a series of reviews by both the A&D group and then by SD&I; SSE organizes and facilitates the A&D Use Case reviews and serves as reviewers on the related SD&I reviews. 5.2.2.3 XSEDE Digital Object Repository The XSEDE Digital Object Repository will preserve all the key documents and digital objects of the XSEDE project within the University of Illinois IDEALS archival system. During this quarter we began to populate the repository. Documents in the repository include: key project foundation documents, project governance and policy documents, survey and evaluation documents, plans and reports, various operational baselines and checklists, and XSEDE Federation Letters of Intent and acceptance. These documents have permanent Digital Object Identifiers and are available by normal search methods. 5.2.3 External Relations 1.1.4 The XSEDE External Relations team provides extensive support for XSEDE events. This includes, but isn’t limited to, design of materials, web support, and publicity. Events this quarter have included the Extreme Scaling Workshop, International HPC Summer School, XSEDE13, student involvement in a range of activities, and TEOS events. ER is increasing collaboration with XSEDE WBS teams. This includes more stories focused on particular teams or aspects of XSEDE, i.e. ECSS, to enhance awareness. This is being accomplished through direct pitches to HPC media and inclusion in the What’s New in XSEDE

67 external newsletter. Increasing awareness will also be directed toward internal communications, such as announcing quarterly allocations to XSEDE staff and users. Planning for the next annual highlights book is under way, with new layout and content ideas being implemented. This quarter new XSEDE staffer Steve Duensing took over responsibility for the design of this publication in addition to other XSEDE design needs. The ER team’s primary challenge is the need for a new coordinator to take over for Susan McKenna, who departed the project in the previous quarter. The search closed in late February but the new coordinator will not be on board until May and will then need some time to get up to speed. Trish Barker will continue to fill the role on an interim basis and will delegate/share responsibilities with other members of the ER team. 5.2.4 Industry Relations 1.1.5 The industry call of interest was drafted and reviewed. Based on the review comments, significant edits were made to what is now called the XSEDE Industry Challenge Program. The draft has been reviewed with key XSEDE personnel and the Private Sector Program Office at NCSA. Their comments are being incorporated and the current draft will be shared at the April XAB meeting. The search continued during this quarter to find an appropriate Level 3 Manager for the Industry Relations position. A plan will be developed once a Level 3 manager is identified including tasks, metrics, risks and resource usage.

68 6 TAIS/Audit Services 1.8 10.1. Open-Source Version of XDMoD Preliminary work has begun to prepare for the release of the open-source version of XDMoD. Initially, as a proof of concept, HPC utilization data from the University at Buffalo’s Center for Computational Research (CCR) has been ingested into the XDMoD data warehouse structure. The various XSEDE roles were then mapped onto academic equivalents such as dean, department chair, and PI. The net effect is to adapt the XDMoD framework, which was written to serve the XSEDE cyberinfrastructure, to serve cyberinfrastructure typical of a university setting. Ultimately, we will provide an open-source XDMoD template that will be suitable to support academic and commercial HPC systems. The response to XDMoD and its predecessor UBMoD has been so strong that there are a number of academic and commercial HPC centers that have requested to act as beta test sites for the open-source version of XDMoD. We anticipate beta testing to start later in 2013. Figure 10.1-1 shows a summary page with output data from CCR’s clusters obtained during this development effort. The upper left chart is CPU hours generated on various CCR clusters. The upper right chart analyzes CPU hours by job size (number of processors). The lower left chart breaks the usage down by academic divisions. Finally, the lower right chart shows the average daily core count. This is only a small sample of the many metrics that will be available through the open-source version of XDMoD.

Figure 10.1-1. Open-source version of XDMoD showing a summary page of usage from SUNY at Buffalo computers.

69 10. 2 Incremental Improvements to the XDMoD User Interface We continue to make incremental improvements to several areas of the XDMoD user interface in order to improve the user experience. Quick-Start instructions (instructions displayed when no actions have been selected) have been added to the Custom Report Generator, Usage Explorer, and the Allocations tab. A complete redesign of the Allocations tab is under way to provide a more organized and detailed look at the status of a PI’s allocations. This will allow a user, PI, or program manager to easily view the current state of an allocation including top consumers, resources allocated, burn rate, and remaining SUs. Preliminary design work has begun on “Use Case Wizards,” a tool to provide users with a guided, hands-on tutorial for using XDMoD to answer specific questions. These wizards will allow a user to select from a list of common tasks such as “Show me which users are consuming the most CPU hours on Kraken in the fourth quarter of 2012” and interactively guide them through the process of answering the question using XDMoD. 10. 3 Application Kernel Progress Running application kernels continuously on the XSEDE (or other) resources generates a tremendous amount of performance data that makes manual oversight to identify underperforming hardware or software impractical. Accordingly, we have been exploring the development of automated processes to monitor application kernel performance. A preliminary model is being developed to use the application kernel data to assess quality of service. Figure 10.3-1 shows a representative plot of this effort for MPI-Tile-IO and IOR application kernel data. A region is automatically selected as the control region. The control region is by definition considered to be the normal operating envelope of the given application kernel. The process, in this case the performance of a given application kernel, is assumed to be nominally in control in this region. If the five-point running average at a given point beyond the control region exceeds a specified tolerance (based upon the data range in the control region) the process is flagged as out of control. In Figure 10.3-1 the region where the write aggregate throughput bandwidth of the MPI-Tile-IO and IOR application kernels drops substantially is automatically evaluated to be out of control. The details of when to automatically readjust the control region to account for system upgrades or normal changes, when to flag a process as out of control, etc., are still being finalized. However, note that the automatic process control system successfully flagged the deterioration of the write throughput performance of Lonestar4. Using application kernel process control, site administrators can easily monitor application kernel run failures for troubleshooting performance issues at their site.

Figure 10.3-1. Application Kernel process control. Time histories for two I/O-based application kernels selected to demonstrate the application kernel control process for two underperforming application kernels. The solid purple line is the measured data, the dashed black line is a five- point average, the blue shading indicates the control zone range. The red zones indicate that the process is out of control in an unfavorable sense while the green zones indicate superior performance compared to the in-control performance.

Figure 10.3-2, shows a sudden decrease in file system performance on Lonestar4 as measured by three application kernels (IOR, MPI-Tile-IO, and IMB). The IOR and MPI-Tile-IO both show a sudden decrease in the aggregate write throughput bandwidth, while IMB, which measures latency, shows an equally sudden increase in latency. Without application kernels periodically surveying this space, the loss in performance would have gone unnoticed. We are working with TACC to understand this file system performance deterioration.

Figure 10.3-2. Application kernel data for IMB (blue), IOR (red) and MPI-Tile-IO (black) on Lonestar4. The IOR and MPI-Tile-IO data show a sudden drop of aggregate write throughput bandwidth and the IMB data shows a sudden increase in latency starting on 7/24-25/2012.

10.4 Addition of initial SUPReMM data to XDMoD SUPReMM is a separately funded NSF project that is formally not part of the TAS project, but it is closely related in that one of its main goals is to display detailed job-level performance data for XSEDE resources in XDMoD. SUPReMM (Integrated HPC Systems Usage and Performance of Resources Monitoring and Modeling) leverages the open-source software package TACC_Stats or other metrics collection systems to periodically sample node and system state values and a wide spectrum of hardware counters to provide a rich (and extensive) dataset of performance data for every application run on each node in a given HPC cluster. Initially the focus has been on the TACC systems Lonestar4, Ranger, and Stampede. Ultimately, however, the goal is to extend this data collection to all XSEDE resources. When fully integrated with XDMoD, SUPReMM will provide users, and more importantly system personnel, with the ability to identify underperforming applications and, based on the collected SUPReMM data, suggest methods that may be employed to improve end-user application performance and in so doing make more efficient use of these already oversubscribed platforms. TACC_Stats enhances the UNIX sysstat/sar utilities, a useful and often underutilized collection of performance monitoring utilities for the UNIX/Linux-based HPC environments. Currently TACC_Stats can gather core-level CPU usage (user time, system time, idle, etc), socket-level memory usage (free, used, cached, etc), swapping/paging activities, system load and process statistics, network and block device counters, inter-process communications (SysV IPC), software/hardware interrupt request (IRQ) counts, filesystems usage (NFS, Lustre, Panasas), interconnect fabric traffic (InfiniBand and Myrinet), and CPU hardware performance counters. Currently, TACC_Stats uses CPU performance counters as follows (note that performance counters are processor-specific and have many restrictions on simultaneous sampling): on AMD Opteron, the events are FLOPS, memory accesses, data cache fills, SMP/NUMA traffic; on Intel Nehalem/Westmere, the events are FLOPS, SMP/NUMA traffic, and L1 data cache hits. To not interfere with a user's own profiling and instrumentation activities, at periodic invocations (currently every 10 minutes), TACC_Stats only reads values from performance registers.

72 The node-level TACC_Stats data is then processed to produce job-level data. To date, the TAS team has ingested job-level Ranger TACC_Stats data into XDMoD to provide users with the ability to easily generate plots showing the performance of Ranger on a fine-grained (job) level, as indicated in Figures 10.4-1 to 10.4-4 (all for the last quarter of 2012, Ranger’s last full quarter in production). In particular, we consider CPU usage, FLOPS, memory usage and network usage. Figure 10.4-1 shows the CPU usage compared with the CPU idle time for all XSEDE jobs run on Ranger during the final quarter of 2012. Note that the CPU usage is very efficient with most jobs showing very low CPU idle time. Figure 10.4-2 shows the daily output of FLOPS by Ranger, which shows a somewhat surprisingly small value considering that the peak value for Ranger has been benchmarked as 579 TFLOPS. This is dependent on the job mix running on any specific day. Figure 10.4-3 shows the memory usage per core for Ranger during the same period. Note that each core is equipped with 2 GB of memory so even on peak usage days less than 40 percent of the memory is used on average. Finally, Figure 10.4-4 shows the I/O scratch write for the Ranger file system. Note the very large daily variation. In general, file system usage shows considerable variation over any time period considered.

Figure 10.4-1. Total user CPU hours (red), total idle CPU hours (blue), and system time (black) for all XSEDE jobs run on Ranger aggregated daily during the last quarter of 2012.

Figure 10.4-2. Total FLOPS for Ranger aggregated daily during the last quarter of 2012.

Figure 10.4-3. Memory usage per core for Ranger aggregated daily during the last quarter of 2012.

Figure 10.4-4. IO scratch write for Ranger file system aggregated daily during the last quarter of 2012.

10.5 Indiana University Sub-contract: Progress report on Publications and Scientific Impact We continued to make progress on the scientific metric data extraction, transformation as planned. We have correlated the user, organization, project, and field of science information from the XDcDB database with the publication data from the NSF award search database and carried out some summary statistics based on number of publications at user, project and field of science level. This data is used in our initial metric to measure the scientific impact of XSEDE users and projects. We also identified some significant issues with the data from the NSF award search database. Author names are highly incomplete, i.e. no full name associated with a publication but only last name and first initial, often preventing unique identification of the author. We plan to mitigate this issue while integrating data from other data sources, as well as applying some machine- learning approaches to identify more clearly which publications belong to which user. We are in the process of defining a general database to host the publication data, which is a mashup from various data sources including the NSF award search data, publications from the XSEDE portal, as well as other sources such as Microsoft Academic Search, Mendeley, Google Scholar Profile, etc. This design enables us to visualize the preliminary scientific impact metric data already obtained while supporting gradual improvement with the integration of more data and increased accuracy. Planed work for the next quarter: delivery of a database on scientific impact and publication with emphasis on XSEDE, the transitioning of our database technology to MongoDB, the ability to export data from MongoDB as an SQL database, the use of direct queries into the MongoDB from XDMOD. Another activity was done in correlation with the FutureGrid project while improving the FG cloud metric project. Most recently we have contributed the creation of a framework that allows us easily to generate a PDF report from cloud metric data. The metrics are very different from HPC metrics and are therefore stored in our own database at IU.

75 10.6 NICS Sub-contract: PEAK progress 10.6-1. Objectives In this quarter, we have been working to extend our research from Kraken to other XSEDE platforms. An XSEDE allocation proposal was submitted and approved. We were awarded 200,000 core hours in total on Blacklight, Gordon, Keeneland, Longhorn, Stampede, and Trestles. We selected Gordon as our first target since Kraken and Gordon have representative features, such as different processor vendors (AMD vs. Intel) and different interconnects (proprietary vs. commodity). Therefore, a direct comparison between these two distinct platforms would be interesting. In addition, we intended to compute an application-based service unit (SU) conversion rate between Kraken and Gordon. 10.6-2. Results Benchmark runs of five applications, NAMD, LAMMPS, Gromacs, CPMD, and Amber, were performed on Gordon. For each application, we compiled a number of versions with a combination of available flavors (i.e., compiler, numerical library, and FFTW version). For each version, we ran three trials on 16, 32, 64, 128, 256, 512, and 1,024 cores, respectively. The fastest run time of each version on each core count was recorded for further analysis. 10.6-2.1 NAMD NAMD is built merely for each compiler flavor because the sole numerical library NAMD uses is single-precision FFTW2 (version 2.1.5). The input dataset is STMV. Figure 10.6-1 shows the result. On Kraken, the PGI build does not run, so there are no results for this configuration. The GNU version exhibits 2 percent to 8 percent performance advantage over Intel. On Gordon, the PGI version fails to produce expected result when core counts are 256 and 512. Overall, Intel edges out GNU and exceeds PGI by 15 percent.

Kraken Gordon

Figure 10.6-1. Performance data of NAMD on Kraken and Gordon.

10.6-2.2 LAMMPS We compiled six versions of LAMMPS, one for each compiler and FFTW combination. LAMMPS does not need BLAS/LAPACK. The benchmark problem is Rhodo.

76 Figure 10.6-2 shows the result. Generally, the effect due to FFTW libraries is almost negligible on Gordon, while on Kraken, they have certain impact when 48 cores are used. The GNU and Intel versions have very similar performance, while the Intel version is only slightly faster than its GNU counterpart on Kraken. The PGI version, on the other hand, is slower by a large margin of 8 percent to 16 percent.

Kraken Gordon

Figure 10.6-2. Performance data of LAMMPS on Kraken and Gordon.

10.6-2.3 Gromacs Gromacs' main simulation program “mdrun” supports a variety of FFT libraries and we built for FFTW2 and FFTW3. Regarding BLAS/LAPACK, Gromacs only makes use of them in its utility programs but not in “mdrun,” so we omit them. The benchmark input is d.dppc. The results are shown in Figure 10.6-3. Overall, PGI builds are the worst on both systems, lagging behind Intel and GNU by 6 percent to 11 percent on Kraken and 10 percent to 18 percent on Gordon. Regarding FFTW2 and FFTW3, intel-fftw3 is the fastest build combination on Kraken, but there is no clear winner on Gordon. Kraken Gordon

Figure 10.6-3. Performance data of Gromacs on Kraken and Gordon.

10.6-2.4 CPMD

77 CPMD is a typical application that can be optimized by many build options: compiler, BLAS/LAPACK library, and FFTW version. Theoretically we are able to produce 18 versions of CPMD on Kraken. However, because the latest source code version of CPMD is not compatible with the version of GNU compiler on Kraken, we only produced 12 builds of CPMD on Kraken. We drove CPMD simulation with input problem Si512. Due to memory requirements, the minimal core count to run Si512 benchmark on Kraken is 48. From the benchmark results in Figure 10.6-4, it is obvious that on Kraken, the choice of compiler has substantial influence on the performance. Overall, the PGI compiler produces much faster CPMD executables. In most cases, the Intel versions run three to six times slower than their PGI counterparts. The only two exceptions are intel-libsci-fftw2 and intel-libsci-fftw3, which have very similar performance to PGI versions. The use of FFTW library does not seem to have significant impact. On Gordon, the influence of compiler becomes smaller but the choice of numerical library does matter; the MKL versions outperform ACML versions to a great extent. The difference caused by numerical libraries diminishes when more cores are used. As we can see from the figure, intel- mkl-fftw3 is the best combination that we can use to build CPMD on Gordon.

Kraken Gordon

Figure 10.6-4. Performance data of CPMD on Kraken and Gordon.

10.6-2.5 Amber We built Amber's PMEMD using all three compiler flavors. PMEMD uses FFTW3 but not BLAS & LAPACK. We tested the Joint AMBER/CHARMM (JAC) benchmark input. Figure 10.6-5 shows the results. On Kraken the Intel version does not run, so there is no result for it. On Gordon, the PGI version falls behind GNU and Intel versions by up to 30 percent, and the Intel version has a slight performance advantage over the GNU version. Amber's scalability on the JAC benchmark starts to worsen as the core count grows beyond 64 (4 nodes) so we do not include that part in the plot. The case for Kraken is more interesting. On Kraken the PGI version runs faster when core count is 12 and 24. The performance deteriorates significantly when we use 48 cores (four nodes) or more.

78 Kraken Gordon

Figure 10.6-5. Performance data of Amber on Kraken and Gordon.

10.6-2.6 Application-based SU Conversion Rate In order to help guide HPC users to select the most appropriate platform to run their applications, we have proposed a concept of “application-based SU conversion factors” and tried to use these factors derived by application benchmark to validate the official conversion rate value (i.e., 2.42). Specifically, we use the following formula: Gordon/Kraken Ratio = (Kraken SUs consumed) / (Gordon SUs consumed) Based on our performance benchmark data, we derived the application-based SU conversion rate as well as its 95 percent confidence interval for each application (see Table 1 and Figure 10.6-6). It appears that running certain applications, such as Gromacs and NAMD, on Kraken is much more cost-effective than running them on Gordon, while Gordon is a better choice when running Amber, CPMD, and LAMMPS. Our goal is to provide an application-based SU conversion rate for HPC user to help them select the most “cost effective” platform to run their applications. Table 1. The application-based SU conversion rates between Kraken and Gordon for five applications. Applications Application-Based SU Conversion 95% Confidence Rate Interval Amber 3.03 (2.00, 4.05) CPMD 5.10 (4.47, 5.73) Gromacs 1.69 (1.61, 1.77) LAMMPS 2.55 (2.34, 2.76) NAMD 1.07 (1.04, 1.11)

Figure 10.6-6. SU Conversion confidence intervals.

10.6-3. Conclusions In this quarter we extended our research from Kraken to Gordon and successfully built and benchmarked different versions of five popular scientific applications. The performance of these applications on two platforms are compared and analyzed. Moreover, we proposed “application- based SU conversion rate” to validate official SU conversion rate value and help improve the efficiency of resource usage. The application-based SU conversion rates of five applications between Kraken and Gordon demonstrate that it is more cost-effective to run Gromacs and NAMD on Kraken and run Amber, CPMD, and LAMMPS on Gordon. We suggest using this application-based SU conversion rate to guide XSEDE users to choose the most appropriate platform. 10.7 University of Michigan Sub-contract: Usability assessment report: As a first measure of the usability of the XDMoD interface, the team of researchers at the University of Michigan (UM) conducted an in-person interview study of local Michigan XSEDE users (n=10) and potential XSEDE users (n=13). Even though the sample size was small, analysis of the interview data showed a number of areas of the XDMoD interface that could be improved, including: (1) developing a less complex default view for some of XDMoD’s tabs to prevent information overload, (2) improving the ability to locate and adjust the time period for graphs in the Report Generator, (3) making it easier to add charts to the Report Generator, (4) improving navigation in the Usage tab, (5) making the location of help resources easier to find, and 6) implementing “right click” capability for some functions. This analysis provided useful feedback to the early XDMoD development efforts and many of the areas identified as needing improvement were addressed in subsequent releases. In order to explore XDMoD usability in a broader context, the UM team chose to administer a web-based questionnaire to a random sample of XSEDE users. Survey respondents were sampled from the population of individual XSEDE users who ran a minimum of 25 jobs between March 1, 2012, and Sept. 1, 2012. Information on jobs came from XSEDE logs for this period. Jobs run

80 from group, staff, or other non-individual accounts were excluded. A minimum of 25 jobs run on XSEDE was selected as evidence of significant use, as opposed to occasional or test use. Using these criteria produced a sampling frame of 1,914 account holders, from which 20 percent (N=398) were randomly selected. Respondents were invited via email to complete a web-based questionnaire. In this invitation, respondents were offered a pre-incentive in the form of a $5 Amazon.com gift code included in their personalized email invitation. Respondents were free to use the gift code independent of whether they opted to complete the survey. Of those invited to participate, a usable response rate of 28.7 percent (n=110) was obtained. Of the 110 respondents only five had logged into XDMoD prior to the survey, meaning 105 of the respondents were new XDMoD users and therefore had not been previously exposed to the interface. In addition to the standard multiple choice questions and free format answers found in most survey tools, the survey also included a novel technology that can track where survey participants click in a given window. This capability is particularly useful for interface development since it allows the investigator to determine where in a given interface screen shot users go to find the answer to the posed question. For example, Figure 10.7-1 show all locations on a given window of the XDMoD interface where users clicked in response to a particular survey question. The resulting image can be thought of as a heat map of the response space, with areas of high click frequency represented by continually “warmer” colors (red indicating the highest frequency of clicks). In Figure 10.7-1 users predominantly clicked in one of two areas, namely on a tab along the top of the window or on a summary plot in the bottom center. This provides useful information regarding how intuitive and easy to use the interface is with respect to a given query.

Figure 10.7-1 is the aggregate heat map of the clicks by respondents for question 20 in the web- based survey. The question “Usability Study - Summary Page The XSEDE XDMoD (Metrics on Demand) Usage Summary Tab is shown below. Click on the image below to indicate where you would find awarded grants by field of science.”

Results of the survey showed that some users are overwhelmed by the amount of data presented to them. Users are confused why information is presented that is not in a relevant format, and tutorials are useful in orienting users but other help resources would be beneficial. These findings suggest a set of priorities for future evolution of the XDMoD interface, including: simplifying the

81 interface for the Usage tab to present less text in the tree navigation and switching to a drop down navigation more similar to the Report Generator, concentrating on making sure the data presented to the user is in a format most useful for their need (role), and making sure the help manual is indexed to the web. Specific findings and recommendations based on the survey results are as follows: 1. Some default graphs make little sense for their representation of data. For example, pie charts for a single value as displayed in the Summary page when in the User role. Accordingly, the information presented should be tailored to be relevant to a particular user (role). 2. Some users feel overloaded when approaching XDMoD with the amount of default information presented through the interface. In some respects this is not a particularly surprising result given the powerful features built into the interface along with the rapid injection of new features during the first three years of the award. However, we expect this situation to improve substantially over the next two years as the XDMoD user interface matures and as more attention can be devoted to making the interface intuitive. In addition, we will continue to investigate alternative designs that reduce the presented information until requested by a user’s action. For example, when landing on the XDMoD Summary page for the first time, we may find it beneficial to direct the user to select the metrics they are interested in seeing when they first land on the page. This would limit the amount of information first presented as well as provide new users with insight that the summary page can be customized. 3. Users turn to search engines for help and accordingly the documentation needs to be indexed by public search engines. This is the same sentiment shared by users in the previous XDMoD survey. 4. Tutorials and other help tips for new and returning users are desired. The effectiveness of the report generator instructions when presented with no charts appears to be effective in helping users understand how to begin creating a custom chart. This can be expanded to other areas. 6.1 10.8 XDMoD Usage XDMoD usage during the present reporting period as reported by Google analytics is shown in Figure 10.8-1. Figure 10.8-2 shows the growth in XDMoD usage over the past two years. The usage has continued to grow over the past quarter and we anticipate further growth as capabilities are added and it becomes better known to XSEDE users.

Figure 10.8-1. Google analytics overview of XDMoD usage for January 1, 2013 to March 31, 2013.

Figure 10.8-2. Growth in XDMoD usage for the past two years.

10.9 Meetings, Events Publications and Presentations TAS has been very active during this period in preparation for XSEDE13. We have submitted two papers and plan to also present a BOF on XDMoD: 1. T. R. Furlani, B. L. Schneider, M. D. Jones, J. Towns, D. L. Hart, S. M. Gallo, R. L. DeLeon, C. Lu, A. Ghadersohi, R. J. Gentner, A. Patra, G. von Lazewski, F. Wang, J. T. Palmer, and N. Simakov, “Using XDMoD to Facilitate XSEDE Operations, Planning and Analysis

83 2. C. Lu, J. Browne, R. L. DeLeon, J. Hammond, B. Barth, T. R. Furlani, S. M. Gallo, M. D. Jones, and A. K. Patra, “Comprehensive Job Level Resource Measurement and Analysis for XSEDE HPC Systems 3. XSEDE Metrics on Demand (XDMoD) Technology Auditing Framework: A Guide for the XSEDE Community: Presenters: Thomas R. Furlani, Matthew D. Jones, Amin Ghadersohi, Steven M. Gallo and Robert L. DeLeon. Center for Computational Research, State University of New York at Buffalo

84 7 ExTENCI The goal of The Extending Science Through Enhanced National Cyberinfrastructure (ExTENCI) project is to develop and provide production-quality enhancements to the national cyberinfrastructure that will enable specific science applications to more easily use both OSG and XSEDE or broaden access to a capability to both XSEDE and OSG users. The ExTENCI project is a joint Open Science Grid (OSG) and XSEDE project, funded by the National Science Foundation. The PIs are Paul Avery (U. Florida) and Ralph Roskies (PSC). The planned two-year project began in August 2010 and has received a no-cost extension through July 2013. This quarter, some project funding was redistributed to allow the addition of nine new deliverables for work that will start in April. Also, four new deliverables were documented for work done over the last six months at the UW. Overall, the project has completed 110 of the 126 deliverables, not including the nine new ones that begin next quarter. ExTENCI has four primary areas of work, each of which is discussed sections below. 7.1 ExTENCI – Distributed File System (Lustre-WAN) This quarter, all remaining deliverables from PSC and UF were completed. Some work continued at UF on experiments with OpenNebula and running VMs with the ExTENCI Lustre client. Beginning in the next quarter, work will begin on six new deliverables.  UF replaced hardware for their Object Storage Servers (OSSs) and Object Storage Targets (OSTs) and completed performance benchmarks verifying the new configuration.  Tested an additional configuration of Xrootd-fs with the Lustre filesystem and discovered an issue that was reported to the Xrootd development group.  Installed OpenNebula at the UF and ran a test cloud containing images of the ExTENCI Lustre client and the official CERNVM for CMS. Verified the clients can be instantiated and managed through a web interface.  Set up a Kerberized Lustre server at Fermilab to verify Fermilab can support a CMS Tier 3 site (FIU) using the Lustre filesystem.  Worked with Xyratex developer to resolve compilation problem in getting Lustre 2 working on Centos6. Repeated benchmark testing on Centos6 and verified performance (300 MB/s) was similar to Centos5. 7.2 ExTENCI – Virtual Machines Four additional deliverables are now completed. They are all related to improving the HTCondor support of Clouds:  Enabled HTCondor to work with OpenStack and Eucalyptus.  Added support for Amazon's "spot instances," which use a bidding mechanism to determine the price charged for surplus resources.  Dramatically increased the maximum number of simultaneous EC2 jobs HTCondor can efficiently manage to enable large cloud submissions.  Improved the cloud information available from HTCondor. Improved the utility of HTCondor's responses to EC2 error conditions like invalid credentials or typos in the instance specification. Increased the amount of information conveyed by HTCondor to the user from the cloud service.

85 7.3 ExTENCI – Workflow & Client Tools This quarter, the team focused on improvement of Swift documentation and setting up access to OSG via the new UChicago Campus Grid.  The User Guide was reviewed and improvements were made for the next release.  A satellite sample application was enhanced to make it easier for any user to run it.  A new software release (Swift 0.94) was frozen, passed validation tests, and released.  A Swift demonstration was made at the OSG All-Hands meeting.  Tested Swift can submit jobs to OSG using the new UChicago Campus Grid (UC3) flocking interface. 7.4 ExTENCI – Job Submission Paradigms The JSP team continues to enhance and add plug-in adaptors on SAGA to enable science jobs and data to be distributed to both XSEDE and OSG.  Wrote a paper for HPDC that extensively uses OSG along with other DCI to do data- intensive bioinformatics. See Pilot-Data: An Abstraction for Distributed Data, http://arxiv.org/abs/1301.6228.  Did extensive testing and characterization of iRODS on OSG.  Wrote a newsletter article on SAGA for OSG.  Worked on design and interface issues with the BigJob Gateway that is now implemented and ready for user test.  The BigJob Gateway ( http://gw68.quarry.iu.teragrid.org ) is now available for users.  Completed an independent review and test of SAGA-Python documentation and tutorials and documented the results for SAGA developers.  Incorporated improvements to SAGA-Python documentation and tutorials based on the review.  Submitted three papers to XSEDE13 with the collaborating scientists.

86 8 XD Service Provider Reports

2013 Q1: January 1, 2013, through March 31, 2013

XD Service Provider Forum Leadership Carol Song, Chair Purdue University David Y. Hancock, Vice Chair Indiana University

XD Service Provider Principal Investigators Sean Ahern U Tennessee – National Institute for Computational Science Jay Boisseau U Texas - Texas Advanced Computing Center Geoffrey Fox Indiana University Kelly Gaither U Texas - Texas Advanced Computing Center William T.C. Kramer U Illinois - National Center for Supercomputing Applications Michael Levine Pittsburgh Supercomputing Center Miron Livny U Wisconsin-Madison Richard Loft National Center for Atmospheric Research Richard Moore U California San Diego - San Diego Supercomputer Center Michael Norman U California San Diego - San Diego Supercomputer Center Manish Parashar Rutgers University Gregory D. Peterson U Tennessee – National Institute for Computational Science Carol Song Purdue University Craig Stewart Indiana University John Towns U Illinois - National Center for Supercomputing Applications Jeffrey Vetter Georgia Institute of Technology

87 8.1 Overview The XD Service Provider (SP) Forum is the representative body of federated providers of leading edge computational, storage and visualization resources, software, and associated services to the open science community. There are 16 members. In partnership with the XSEDE project, the SPs have delivered more than 1.1 billion service units (SUs) collectively to the XSEDE user community in this quarter, supporting 1,024 projects (including 2,384 individual researchers) from 368 institutions. Below is a summary of the SP Forum’s main activities in this reporting quarter. New membership and resources: As the new XSEDE-allocated resource Stampede went into production in January, the SP Forum has accepted the Stampede SP as a new member of the forum as Level 1 Service Provider. The application and the acceptance letters are available at the SPF wiki (under “other documents”). Indiana University SP recently added Mason as a Level 2 resource on XSEDE. Mason is a large- memory campus computer cluster (512 cores, 8 TB RAM) for data-intensive, high-performance computing tasks. Mason is connected to the XSEDEnet via a 10 Gbps link. Mason is configured to support researchers running genomics applications that require large amounts of memory. It is also intended for educators to use for instruction on genome analysis software and development. Mason is available for XSEDE allocation in the April 2013 cycle. Storage allocations implementation: The SP Forum worked with XSEDE to create a storage resource allocation policy and guidelines for implementation. The forum has approved the final version of the XSEDE storage allocations policy. The XSEDE team provided timeline, implementation specifics, and updates to the forum. The forum provided feedback, including the integration of a test period for usage reporting in the plan to ensure readiness for operation on July 1, 2013. XSEDE network change: XSEDEnet has transitioned from NLR to Internet2. This transition aims at improving the backhaul network to support 100G traffic and reduce bottleneck issues with the NLR backbone for XSEDE. All SPs that are affected by this transition approved the change in late January. Technical staff at each of the SPs worked with XSEDE through the transition. Discussion regarding support for classes: The issue of class accounts and allocations was brought into focus recently by a large class with 300 students (from UC Berkeley). While the SPs have long supported classes, they are typically small ones. Issues related to allocation and support came up due to the scale of the class, even though XSEDE and SPs knew about the class in advance. The SP Forum is working with XSEDE on developing a policy and process to support classes more effectively and sustainably. This will include defining a threshold for class size above which additional information will need to be collected from the instructors. This work has just begun and will continue into the next quarter. Miscellaneous: SPs provide feedback on various issues on an ongoing basis. For example, a recent discussion concluded that some SPs do not want to receive user citizenship information from XSEDE account creation and that the XSEDE account mechanism needs to provide an opt-in option to SP sites. The SP Forum conducts its business and coordination through regular conference calls. Victor Hazlewood, XSEDE SP coordinator, has been updating the SP Forum on a monthly basis. The

88 forum chair and vice chair participated in the XSEDE quarterly meeting in March, as well as the regular senior management team calls.

89 9 XSEDE Quarterly Report: FutureGrid Service Provider (January 1, 2013 – March 31, 2013)

9.1 Executive Summary ScaleMP cluster installed and in “early user” testing. This is a 16-node cluster dedicated to ScaleMP software use. Began use of Cluster Compatibility Mode (CCM) on the Cray XT5m (xray) Precip 0.2 released with two new features: an automatic instance retry feature in case of booting/infrastructure problems, and an improvement on how SSH keys are handled for multiple machines/accounts. The Inca monitoring test configuration for FutureGrid’s cloud infrastructure was updated to adapt to changes in the infrastructure New network tunneling mechanisms have been integrated into ViNe overlays for better performance and improved usability.

9.1.1 Resource Description FG Hardware Systems # # # Total RAM Storage Name Site System type Nodes CPUs Cores TFLOPS (GB) (TB) india IU IBM iDataPlex 128 256 1024 11 3072 512 hotel UC IBM iDataPlex 84 168 672 7 2016 120 sierra SDSC IBM iDataPlex 84 168 672 7 2688 96 foxtrot UF IBM iDataPlex 32 64 256 2 768 24 alamo TACC Dell PowerEdge 96 192 768 8 1152 30 xray IU Cray XT5m 1 168 672 6 1344 180 bravo IU Large Disk / Large Memory 16 32 128 1.5 3072 192 delta IU Large Disk / Large Memory 16 32 CPU 192 9 3072 192 With Tesla GPUs 32 GPU 14336 lima SDSC SSD 8 16 128 1.3 512 3.8 (SSD) 8 (SATA) echo IU Large Memory (ScaleMP) 16 32 192 2 6144 192 [in Beta testing] TOTAL 481 1128 4704 54.8 23840 1550 +32 GPU +14336 GPU

FG Storage Systems System Type Capacity (TB) File System Site Xanadu 360 180 NFS IU DDN 6620 120 GPFS UC SunFire x4170 96 ZFS SDSC Dell MD3000 30 NFS TACC IBM dx360 M3 24 NFS UF

9.2 Science Highlights Mining Interactions between Network Community Structure and Information Diffusion Yong-Yeol Ahn School of Informatics and Computing Indiana University

Abstract Recent technological advances are opening up unprecedented opportunities to understand the dynamics of society in incredible resolution and scale. Each day mobile phones and online social network services record when, how, what, where, and with whom we communicate and the data reveals fascinating details of human behavior and our society. The understanding of the structure, dynamics, and roles of communities will have huge impacts on both industry and public sectors. The research objective of this research project is to analyze and understand interactions between community structures and information diffusion, and ultimately develop predictive models of information diffusion based on community structure. Intellectual Merit The proposed research will advance our understanding of the interplay between network structure and information diffusion by combining two distinct topics — communities and information diffusion — that rarely have been studied together. This research will lead us both to more accurate discovery of highly overlapping communities and to deeper understanding of how network community structure affects information spreading. Such understanding will also allow us to develop predictive models of information diffusion, which leverages the information of network community structure. Furthermore, the intellectual merits are not confined to social network research, as the methods can potentially be broadly applicable to other types of networks, as the PI’s research on biological networks exemplify. Broader Impacts This project will have strong broader impacts. Because most students actively engage social networking services, it appeals to broader populations. There is a strong demand for network analysis curricula and research opportunities. The PI will leverage this opportunity to initiate a curriculum on network analysis and mining and to provide research opportunities to broad populations. The PI will provide research opportunities and mentoring to students from HBCUs (Historically Black Colleges and Universities) through A4RC (Alliance for Advancement of African- American Researchers in Computing) and DREU program. The PI will develop online courses on social networks and reach out to underrepresented populations. Since many social phenomena are related to community structure (e.g. racial and cultural segregation, conflict between groups, emergence of political polarization, rumors, misinformation), understanding the structure, dynamics, and roles played by communities will help us to understand these phenomena. Furthering our understanding of these phenomena will, in turn, help us to understand and mitigate some of their harmful effects, and will facilitate improved public policy design and campaigns for social good. Use of FutureGrid FutureGrid will be used to process and analyze large-scale network datasets. The algorithms will be developed based on map reduce framework and FutureGrid will be used as a testbed for the algorithms. Results Lilian Weng, Filippo Menczer, Yong-Yeol Ahn, "Community Structure and Spreading of Social Contagions" (Preprint, to be submitted to WWW'13)

HyFlowTM Roberto Palmieri Department of Electrical and Computer Engineering Virginia Polytechnic Institute and State University Abstract The HyFlowTM project is developing distributed transactional memory (or DTM) as an alternative to lock-based distributed concurrency control. HyFlow is a framework for DTM, with pluggable support for policies for directory lookup, transactional synchronization and recovery, contention management, and cache coherence. HyFlow exports a distributed programming model that excludes locks: using (Java 5’s) annotations, a programmer can define atomic sections as transactions, in which reads and writes to shared, local and remote objects appear to take effect instantaneously. No changes are needed to the underlying virtual machine or compiler. Intellectual Merit We propose fault-tolerant distributed transactional memory. Distributed transactional memory (TM) promises to alleviate the problems of lock-based distributed concurrency control-- e.g., distributed deadlocks, livelocks, and lock convoying. Distributed TM exports a simple programming interface, which avoids locks. Fault-tolerance is essential to distributed TM to cope with node/network failures, but that must be achieved with scalability. Object replication is central to fault-tolerant D-STM. However, replication protocols must allow read concurrency and ensure transactional consistency and progress properties, while being message-efficient on metric-space networks (i.e., one in which communication cost between nodes form a metric). This is an open problem. Broadcasting -- classical replication primitive in database systems -- is inherently non-scalable in metric-space networks, as messages transmitted grow quadratically with the number of nodes. The project proposes a novel replicated distributed TM framework, whose key idea is to split a transaction into two independent phases with orthogonal responsibilities: 1) regular read/write phases, in which latest copies of required objects are quickly collected, in the presence of node/link failures, without any consistency guarantees, and 2) a request-commit phase, in which conflicts are detected and consistency is ensured by consulting the results collected from conflict resolution modules of other nodes. The project proposes quorum-based replication protocols, in which a transaction reads an object by reading object copies from a read quorum, and writes an object by writing copies to a write quorum. Additionally, the project proposes correctness criterion including weaker consistency models, and distributed conflict resolution protocols and cache coherence protocols. The proposed framework and protocols will be implemented in the HyFlow distributed TM Java package, which is open-source and freely distributed. Broader Impacts Project's broader impacts include: 1) integration of the research results into a graduate course taught at Blacksburg, VA and VT-MENA/Egypt; 2) dissemination of the project's results through an open- source implementation of the project's distributed TM infrastructure and publication of results at relevant conferences; and 3) increasing the cultural interaction between students and faculty in the US and those in the Middle East and North Africa region, by advisement of VT-MENA students and teaching an advanced course at VT-MENA/Egypt. Use of FutureGrid One of the main purposes of the HyFlowTM project is to build scalable protocols for Distributed Transactional Memories. Scalability in terms of the size of the system so we would use FutureGrid resources for running experiments using several machines (>8). Results

92 This is the link of the project (http://www.hyflow.org/hyflow) and here there are all the papers and technical reports in the context of the project: (http://www.hyflow.org/hyflow/wiki/Publications).

Parallel Watershed and Hydrodynamic Models Meghna Babbar-Sebens Civil and Construction Engineering Oregon State University

Abstract This project will develop a decision support tool for enhanced understanding and managing flow and water quality in streams and reservoir in the Eagle Creek watershed. The project uses computationally intensive simulation models, and requires the use of distributed computing for solving complex environmental problems in a tractable manner. Additionally, these models are also being used to visualize and design solutions via a web-based interface, which requires on-demand use of computational resources to implement these simulation models. Hence, the use of cloud computing will be explored for this purpose. Intellectual Merit-- The proposed research develops transformative concepts for search and design algorithms that provide more robust, transparent, and comprehensive means for supporting environmental planning and management processes, at the same time integrating knowledge from hydrologic sciences, environmental engineering, computer science, economics, and decision science. The investigators and collaborators have diverse and complementing expertise in the areas of hydrology, environmental engineering, computer science, economics, and ecosystem science. Broader Impacts This research embodies a profound new approach in watershed management: community interaction in the design process. Past management approaches have included public participation at the beginning or end of a design process, whereas here the watershed community will be included in all aspects of the design process. Community participation is particularly important for the adoption of distributed upland storage as an alternative to past structural methods for controlling floods: non- structural methods are often on private lands and require the acceptance and participation of watershed inhabitants. As well as its research contributions, the project will provide training of graduate students and undergraduate students in Earth and Environmental Sciences and Computer Science, jointly by the investigators and collaborators, in a multi-disciplinary project environment. Further, the methodologies can be extended to other domains involving the intersection of manmade, environmental, and societal systems. Presenting our results widely to audiences involved in environmental and ecosystem management, we will help promote more participatory management of our nation’s environmental resources. Use of FutureGrid The project uses computationally intensive simulation models, and requires the use of distributed computing for solving complex environmental problems in a tractable manner. Additionally, these models are also being used to visualize and design solutions via a web-based interface, which requires on-demand use of computational resources to implement these simulation models. Hence, the use of cloud computing will be explored for this purpose. Results FutureGrid resources were used to support the work in the following publications: Babbar-Sebens, M., Barr, R.C., Tedesco, L.P., Anderson, M., 2013. Spatial identification and optimization of upland wetlands in agricultural watersheds. Ecological Engineering, 52, pp. 130– 142.

9.3 User-facing Activities 9.3.1 System Activities 9.3.1.1 Hardware  IU iDataPlex (“india”). No major activities reported this quarter.  IU Cray (“xray”). An experiment was initiated to compare performance of Cray’s “Cluster Compatibility Mode” (CCM) to more traditional Linux and Windows clusters. This has resulted in opening a case with Cray for investigation of intermittent abnormal termination of CCM jobs. A new user guide is under development.  IU HP (“bravo”). No major activities reported this quarter.  IU GPU (“delta”). Upgraded the CUDA code base from version 4.2 to 5.0.  IU ScaleMP (“echo”). The hardware was received and deployed into the data center. Software configuration is ongoing. System available for beta testers.  SDSC iDataPlex (“sierra”). Storage capacity was upgraded from 160GB to 3 TB in three Sierra compute nodes used for cloud services.  SDSC Aeon (“lima”). Initial HDFS experiments show good I/O performance using SSDs compared to HDDs.  UC iDataPlex (“hotel”). No major activities reported this quarter.  UF iDataPlex (“foxtrot”). System operational for production Nimbus users.  TACC Dell (“alamo”). Java updated to version 1.7.0_15 on all login, compute nodes and images. Network map showing network topography to XSEDE and FutureGrid resources updated. 9.3.2 Services Activities (specific services are underlined in each activity below) 9.3.2.1 Accounting XSEDE AMIE Gold integration We continued to work with XSEDE on AMIE integration. Using the “AmieGold” bridge software developed by LONI/LSU/CCT, we configured and successfully tested a local installation of AMIE and Gold integrating with XSEDE via the AmieGold software. We are currently working to develop integration between AmieGold and FutureGrid’s internal account management systems. FG Metrics FutureGrid Cloud Metrics has been developed as a practical approach to assist users in understanding cloud usage data and to help system administrators access cloud utilization statistics. This quarter we have started the followiing activities (a) the creation of documentation to allow the installation by administrators and others rather than the developers only (b) the refactoring of the code (c) better usage examples (d) improving of the web framework to display the results. In particular we have achieved the following:  Automated documentation: We have completely reworked the way the information is presented and use now the bootstrap theme. IN addition we have completely reorganized the manual and are now adding more documentation to address installation and usage.  Code refactoring:

94  We have agreed that all code must follow pep8 standard and pylint is to be used. We also improved the quality of source code, simplified and reorganized with better descriptions for the functions and generalized coding styles. Various python packages (Docopt, flask, etc.) allow us to enhance code readability. As a part of code refactoring, we also provide FG Metrics as a package, including an installation and configuration script. yaml is going to be our new standard to configure settings. A new dynamic Cmd3 module was developed that allows integration of cloud metrics with our cloud mesh code more easily via plugins.  Moving to mongodb: We have decided to transition the database from SQL to Mongodb. We have installed mongodb system in production (mongo.futuregrid.org) and started to add our data into the mongodb server. We will have the following advantages: (a) integration with our web framework will be easier (b) integration with our cloud mesh will be easier (c) deployment of mongodb is easier on other machines than mySQL (d) integration with our new cmd3 framework can follow the same pattern as in (a) and (b) and vice versa. 9.3.2.2 Cloud Services and Support Software Eucalyptus Determined that only administrators can register kernel images. Determined that large data volumes cannot be mounted to Eucalyptus instances due to permission limitations in NFS. Nimbus The highlights of our work this quarter was the release of an alpha version of the Nimbus Phantom service on FutureGrid. Phantom provides a gateway to all FutureGrid clouds, plus the Amazon cloud, allowing users to deploy virtual machines on multiple private, community, and commercial clouds. In addition, Phantom also provides high availability support (restarting virtual machines for the user) as well as scalability support (allowing users to scale up and down, cloud-bursting between multiple clouds). Phantom provides an intuitive web interface lowering the entry barrier for FutureGrid access as well as a scripting interface providing access to more complex functionality. The University of Chicago is operating Phantom as a FutureGrid service. OpenStack OpenStack Folsom was installed on sierra and released to friendly users. The LDAP module of Keystone doesn’t have some necessary functionality to integrate account management with FutureGrid LDAP, so users, projects and keys are synchronized with FutureGrid LDAP by a custom script. The OpenStack Grizzly version will be released soon and we will plan to upgrade our Essex version, running on india, to Grizzly in the next quarter. OpenStack Folsom is available to friendly users on alamo. Developed a custom script to synchronize users, passwords, projects, and ssh keys between this installation and the FutureGrid LDAP infrastructure. Provided example virtual machine images for CentOS 6.4, Fedora 17, and Ubuntu 12.10 to users. No problems have been raised during the friendly user period so we expect to make OpenStack Folsom on alamo available to all users once the user guide is completed. ViNe This quarter, the UF team developed extensions to ViNe software focusing on two tunneling mechanisms: (1) code that support Generic Routing Encapsulation (GRE) tunneling has been tested and validated; and (2) ssh-tunneling-like mechanism that will enable FutureGrid users to run ViNe software without the need for root privilege has been designed and implemented. More specifically:

95  Enabled GRE tunneling between ViNe routers on sierra and foxtrot, and confirmed the stability of the developed code (over 1 week without problems).  A new ViNe tunneling mechanism that enables standalone machines (e.g., FG users’ client laptops) to access resources on ViNe overlays without the need to instantiate a ViNe router is being implemented and tested. The new ViNe tunnels are very much like ssh-tunneling, but users do not need access to a login machine to establish the forwarding tunnels. Unlike ssh, ViNe supports both TCP and UDP tunnels. The implementation uses regular sockets, and does not require administrative/root privileges. The basic functionality of the tunnels has been tested and verified. Control features are being designed to enable easy management of tunnels. 9.3.2.3 Experiment Management Interactive Experiment Management The Host List Manager and TakTuk continue to be available to users to support interactive experiment management on our distributed infrastructure. The development of Message-Based Execution and Management System (MEMS) is still focused on providing the underlying infrastructure that it requires. Specifically, a messaging infrastructure (see the Information Services section below). Experiment Management with Pegasus To support higher level experiments, such as running workflow and other workloads, testing different filesystem setups and time-based repeatable experiments, a Python API was implemented. The name of the API is Pegasus Repeatable Experiments for the Cloud in Python, or Precip. The API provides flexible experiment management for running experiments on infrastructure clouds. Precip was developed for use on FutureGrid cloud platforms such as Eucalyptus (>=3.2), Nimbus, OpenStack, and commercial clouds such as Amazon EC2. The API enables users to easily provision virtual machines and run scripts and transfer data to/from sets of VMs identified by tags. The goal of the API is to provide a simple interface for writing repeatable experiments in Python. During the report period two new features were implemented and a version 0.2 of Precip was released. The first feature is the retry feature. Previously, some of the cloud instances fail to boot correctly and they will reach timeout in the middle of the experiments. So in the provisioning phase, if an instance has booting problems, Precip will retry the booting phase the number of times specified by the user. The second feature is a change to the way the SSH keypairs are stored. This enabled Precip to be used from multiple machines or accounts at the same time. Now at the beginning of every connection between the submit host and the cloud a unique id is generated for each account and the keys are stored in "precip_{id}". In this case different machines with different ids can use it at the same time. Image Management. FG RAIN We have completed a code QA review and refactoring in which we were able to reduce significantly the number of lines of the code as well as improve the maintainability of the code. We worked closely with the users to support their experiments using the FG Rain software. Bug Fixes included improvements to bare metal provisioning. We are currently working on a bug in regards to the image generated for certain linux flavor and kernel version could not be properly provisioned. FG Teefaa

96 FG Teefaa is installed on india and released to friendly users. Users can make a snapshot of their OS image locally on their machine or their VM, and provision it on india nodes, with scheduled by Torque Resource Manager. This service will allow users to build a customized kernel at home and provision it on our HPC infrastructure and do performance testing. In the next quarter, with combining Teefaa with RAIN or deployment tools(such as Chef, Puppet, SaltStack or Fabric) or simply customized scripts, we will plan to provide build-to-order custom HPC and Cloud infrastructure. 9.3.2.4 Information Services The FutureGrid messaging system provides a single location where users and tools can obtain a variety of information about FutureGrid. During this quarter, a new virtual machine was configured to host the messaging service. We also examined the Qpid message broker as an alternative to the RabbitMQ broker that we have been using. The examination of Qpid was prompted because the Erlang SSL library used by RabbitMQ more strictly follows the X.509 standards and doesn’t accept a few CA certificates that we would like it to. We like the features of RabbitMQ and have decided to continue to use it and will work around this difficulty with a few CA certificates not being accepted by it. One of the types of information that is sent to the messaging system is GLUE 2 resource descriptions. During this quarter, we spent some time improving our GLUE2 software and expect to deploy a new version of this software next quarter. This new version will be more compatible with the ongoing discussions in the Open Grid Forum about a JSON rendering of GLUE2 and will include support for OpenStack. Work is also being done to transition the Inca reporter that collects system partition data from Netlogger over to GLUE2. We continued to examine document-oriented databases to store information received from the messaging system. We expect to deploy CouchDB into production in the next quarter (it appears to have acceptable performance for our needs and comes with a REST API which we would like), but have not completed our evaluations. HPC Services Globus The Globus GRAM 5 and GridFTP services continue to be available on alamo and hotel. Users are added to the grid-mapfiles on these systems upon request. 9.3.2.5 Performance Inca Deployed Inca cloud tests on alamo for the new Folsom Openstack Fixed a bug in the Inca cloud test with using curl to verify external connectivity of a VM and enhanced the test to ssh into a VM instance as non-root user (needed for Ubuntu VMs). A script was written to download Inca test histories as a CSV file for easier analysis of the historical data. In addition to Inca data, we are looking at outage data for understanding the reliability of the overall testbed, cloud infrastructure, and HPC environment. We are investigating how different FutureGrid sites compare in terms of reliability (mean time between failures), number of failures, and failure distributions using both the outage data and the existing Inca data for (a) SSH access, (b) cloud infrastructures including Nimbus, Eucalyptus, and Openstack, and (c) HPC jobs. We are working to submit the results of this study as a paper by the summer. These results will also enable us to better understand the type and pattern of failures and potentially serve as a foundation for predicting failures, which can aid users and improve system reliability.

97 A UCSD undergraduate student funded from the NSF-funded Pacific Rim Undergraduate Experiences (PRIME) program will also investigate how to use Inca for monitoring services on a resource-constrained field-deployed cell phone running Android OS that acts as a data acquisition platform and will be leveraging FutureGrid resources for the software development. PAPI For PAPI-V, we have confirmed that counters are accessible system provided by IU. We have also confirmed that counters are available with linux kernel 3.3+and an up-to-date qemu (the ubuntu repo version was actually out of date). We have now tested on qemu-kvm and it is working on the most recent kvm version available (at http://git.kernel.org/cgit/virt/kvm/qemu- kvm.git/snapshot/qemu-kvm-1.2.0.tar.gz). The tests performed are four minibenchmarks provided by the Mantevo Project: http://www.mantevo.org/. On all benchmarks run, 30 of the 50 preset events available through PAPI exhibit under 2% difference in means. It appears that there are just as many events that have a smaller standard deviation on kvm than events with a larger STDEV on kvm. Out of the 50 events tested, 11 consistently have 30% or greater counts on kvm than on bare metal and are all related to TLB or Instruction Cache. This TLB/Instruction cache issue is consistent with identical tests we have done on the VMWare platform. Data Cache Misses also show a over 20% difference in half the benchmarks. There are 9 events that with over 10% differences which are not related to TLB or Instruction Cache. The 9 events are: PAPI_BR_UCN, PAPI_BR_MSP, PAPI_TOT_CYC, PAPI_L3_DCW, PAPI_L3_TCW, PAPI_SP_OPS, PAPI_VEC_SP, PAPI_REF_CYC. These results should be indicative of what FutureGrid should expect if they upgrade their kernels and qemu-kvm. Therefore, most events should be reliable, although there are a few events that warrant more inspection to determine if the values can be trusted inside a kvm guest. We may need to investigate more into these particular events to understand why they are significantly different from bare metal to kvm. perfSONAR

SDSC’s networking team fixed a major performance problem from the STAR router in Chicago to Sierra (from <.5 Gbps to 8 Gbps) in early January. The perfSONAR machines were instrumental in the debugging effort.

SSD Performance Benchmarking

Solid-state drives (SSDs) are becoming cheaper and more common in Data Centers and we believe that this trend will continue to grow. Current 6 GB/s SATA III NAND-based SSDs are delivering improved random I/O performance compared to traditional hard-disk drives. The goal of this project is to understand how big data technologies can benefit from SSDs. As the first effort, we are benchmarking Apache Hadoop. A key component of Apache Hadoop is the Hadoop Distributed File System (HDFS), a distributed file system that provides high-throughput access to application data. In this study, we used the TestDFSIO benchmark to generate random write patterns for HDFS and measured I/O performance using Lima. Lima is a FutureGrid cluster at SDSC that consists of 8 nodes equipped with 480 GB SSD SATA drives. As shown in Figure 1, our preliminary results indicate that the use of SSDs can significantly increase the performance compared to HDD. We are also collaborating with Prof. Madhusudan Govindraju and his graduate students at SUNY Binghamton on this effort. Next, we will investigate the impact of SSDs on virtualization.

Figure 1: HDFS I/O throughput (Mbps) comparison for SSD and HDD using the TestDFSIO benchmark. For each file size, ten files were written to the disk.

9.3.2.6 FutureGrid Portal

The following activities have been conducted as part of the FG Portal development:

 Publications. A simplified publication form was implemented to encourage users to submit their publications.  Email Notifications. Various email notifications were implemented to alert project members and owners when members have been added or removed, and when the status of the project has changed.  Front Page. The front page was redesigned with a smaller font and slightly different color scheme.  Social Media Integration. Links to the FutureGrid Google+ and Blogger pages were added to the header.  User Approval. A confirmation screen was added to the user approval process for increased security.

9.4 Security No security issues occurred during this period. 9.5 Education, Outreach, and Training Activities Integration with XSEDE Warren Smith coordinated the integration of FutureGrid information into the XSEDE website. A new Testbed category was added to the XSEDE Resource listing to share FutureGrid information: https://www.xsede.org/testbeds. A User Guide was developed with support from TACC technical writing staff and contributions from FutureGrid staff: https://www.xsede.org/futuregrid. Social Media

99 FutureGrid expanded social media engagement during the Jan - March 2013 quarter with the creation of the FutureGrid Testbed blog and related Google+ page dedicated to providing opportunities for FutureGrid team members to share information about systems and software innovations, research and educational projects, publications, presentations, and other activities related to FutureGrid, an experimental testbed that allows researchers and educators to collaborate on the development and testing of innovative approaches to parallel, grid, and cloud computing. FutureGrid continued to share information online through Twitter and Facebook. Other A new article was contributed to the FutureGrid blog explaining how FutureGrid resources can be used with Phantom: http://futuregridtestbed.blogspot.com/2013/02/announcing-nimbus-phantom-alpha-on.html.

Classes this quarter: Type Title Location Date(s) Number of Number of Participants Under- represented people FG-301 Advanced University of Spring 12-15 Unknown Networking Colorado, 2013 Class Boulder, CO FG-308 Computational Michigan State Spring 44 4 Techniques for University, East 2013 Large Scale Lansing, MI Data Analysis FG-316 Cloud University of Spring 15 Unknown Computing Piemonte 2013 Class (3rd Orientale, Italy Edition) FG-318 High University of Spring ~50 ~50 Performance Puerto Rico at 2013 Computing Mayaguez Class FG-291 Clouds, Grids, University of Dec 10-22 10 10 and Greenwich, 2012 Virtualization: London, UK Distributed Computing

Events this quarter (in order of most recent):

Type Title Location Date(s) Hours Number of Number of Method Participants Under- represented people

100 Indiana University

Presentation FutureGrid PTI major 03/07/2013 1.0 ~25 Live RAIN: More Project Review, than Dynamic Indiana Provisioning University Across Virtual and Bare-metal Resources Presentation Big Data and MAGIC 02/06/2013 1.0 Telecon Clouds: Telecon on Challenges and Clouds and Opportunities Testbeds Presentation Big Data and NIST Cloud and 01/15/2013 1.0 Live Clouds: Big Data Challenges and Workshop, Opportunities Gaitherburg, MD Presentation salsaDPI: A Indiana 01/11/2013 1.0 Live Dynamic University Provisioning Interface for IaaS University of Chicago Presentation FutureGrid XSEDE 02/17/2013 1.0 Telecon clouds Gateway Community meeting University of Florida

Keynote Self-organizing DAS Workshop, 02/02/2013 0.75 ~40 Live Virtual Private Delft University Networks and of Technology, Applications Delft, NL Webinar Self-organizing Organized by 03/22/2013 1.0 14 Live Virtual Private Milibo Networks and Applications

9.6 SP Collaborations No new collaborations during this quarter.

9.7 SP-Specific Activities See 1.3.2 Services Activities. 9.8 Publications Bosin, A., "A SOA-based model for the integrated provisioning of cloud and grid resources", Advances in Software Engineering, vol. 2013, 2013.

101 Geoffrey Fox and Dennis Gannon, “Using Clouds for Technical Computing,” Technical Report January 2 2013. Judy Qiu and Bingjing Zhang, “Mammoth Data in the Cloud: Clustering Social Images,” Technical Report January 2 2013. 9.9 Metrics 9.9.1 Standard systems metrics Top 35 HPC FutureGrid Users #Jobs – Largest to Smallest (Jan-Mar 2013) Avg. Job Avg. Wait Wall Avg. User Area # Jobs Size (cpus) Time (h) Time (d) Mem (MB) xcguser University of Virginia - Genesis 42067 1 1 1055.6 321.8 pela3247 University of Virginia 9261 39.7 1 22237.4 35.7 unicore University of Virginia - UNICORE 5064 1 0 1 4.5 inca SDSC - INCA 4678 6.2 1.7 14.9 12.7 charngda University of Buffalo 3114 40 2.2 607.7 98.3 ssmallen UC San Diego 1583 1 0 0 2 oweidner LSU - SAGA 637 3.3 1.6 7.5 104 vshah505 Rutgers University 432 7.2 0.4 36.4 32.4 ktanaka Indiana University 154 9.7 0 12.2 88.5 jdiaz Rutgers University 145 60.7 11.6 11166.4 5.9 jasonkwan University of Utah 91 7.4 0 92.3 22578.3 emheien UC Davis 74 48.9 0.1 104.1 8266.4 feiteng Indiana University 68 20.5 0.1 181.7 110.5 lihui Indiana University 55 20.7 1.4 327.5 891.8 marksant University of Amsterdam - AMC 46 7.7 0 0 2.5 merzky LSU - SAGA 43 2.1 0 0 2.3 isnardoan University of Puerto Rico at Mayahguez 42 35 0.6 0.6 344.8 nnmahaja Indiana University 37 7.8 0 15 8065.9 jwalters USC - ISI 33 8.7 0.3 13 900 malik University of Innsbruck 29 1.8 0 0.2 6.9 jianwu UC San Diego 26 26.3 0 108 296.9 dino24 Rutgers University 24 8 1.4 0.7 15.7 azadeh Indiana University 22 118.2 25 4092.7 20.4 jonrgold Indiana University 22 1 0 2.7 161 jychoi Oak Ridge National Laboratory 22 7.1 0 1.5 72.7 donny University of Florida 21 42.4 0 4.1 325 skamburu Indiana University 21 15.3 0 4.1 16.3 dgignac TACC 18 7.2 20.2 3.8 58.9 lizix Indiana University 17 1 0 0.8 242.2 weng Indiana University 17 116.2 35.1 2153.8 15.7 yangruan Indiana University 17 54.8 77.4 3302.5 4.8 artiavis Rutgers University 15 10 0 0 3.8 blagodurov Simon Fraser University 13 2.6 0 0.1 11.3 sjeganat Indiana University 13 31.1 0 2.2 202.3

102 9.9.2 Standard systems metrics (continued) Top 35 HPC (Average) Memory Users - Largest to Smallest (Jan-Mar 2013) Avg. Job Avg. Wait Wall Avg. User Area # Jobs Size (cpus) Time (h) Time (d) Mem (MB) jasonkwan University of Utah 91 7.4 0 92.3 22578.3 upitamba Indiana University 3 12 0 6 11166 emheien UC Davis 74 48.9 0.1 104.1 8266.4 nnmahaja Indiana University 37 7.8 0 15 8065.9 adnanozsoy Indiana University 5 1.6 0 0.4 1784.8 eholk Indiana University 12 12 0 5.4 1459.9 jwalters USC - ISI 33 8.7 0.3 13 900 lihui Indiana University 55 20.7 1.4 327.5 891.8 zola Rutgers University 11 162.9 0 1.9 601.5 leggett University of Chicago 3 43 0 0 467 nykim LSU 6 20.5 0 22.6 373.4 isnardoan University of Puerto Rico at Mayahguez 42 35 0.6 0.6 344.8 donny University of Florida 21 42.4 0 4.1 325 xcguser University of Virginia - Genesis 42067 1 1 1055.6 321.8 jianwu UC San Diego 26 26.3 0 108 296.9 ptan Michigan State University 1 32 56.8 0 271.3 lizix Indiana University 17 1 0 0.8 242.2 sjeganat Indiana University 13 31.1 0 2.2 202.3 jzeng Indiana University 1 32 0 0.1 181.6 jonrgold Indiana University 22 1 0 2.7 161 eamsden Indiana University 10 10.9 0 1.2 152.3 dshrestha University of Massachusetts 5 160 0 6.8 138.9 shubhada Indiana University 3 16 0.1 2.1 136.9 sekanaya Indiana University 2 8 0 2.2 121 feiteng Indiana University 68 20.5 0.1 181.7 110.5 rrnewton Indiana University 12 3.8 0 4.6 107.2 oweidner LSU - SAGA 637 3.3 1.6 7.5 104 charngda University of Buffalo 3114 40 2.2 607.7 98.3 ktanaka Indiana University 154 9.7 0 12.2 88.5 jychoi Oak Ridge National Laboratory 22 7.1 0 1.5 72.7 gaoxm Indiana University 11 263.3 7.5 7582.3 63.9 dgignac TACC 18 7.2 20.2 3.8 58.9 pela3247 University of Virginia 9261 39.7 1 22237.4 35.7 vshah505 Rutgers University 432 7.2 0.4 36.4 32.4

103 9.9.3 Standard systems metrics (continued)

Top 38 HPC (Average) Job Size Users - Largest to Smallest (Jan-Mar 2013) Avg. Job Avg. Wait Wall Avg. User Area # Jobs Size (cpus) Time (h) Time (d) Mem (MB) gaoxm Indiana University 11 263.3 7.5 7582.3 63.9 zola Rutgers University 11 162.9 0 1.9 601.5 dshrestha University of Massachusetts 5 160 0 6.8 138.9 azadeh Indiana University 22 118.2 25 4092.7 20.4 weng Indiana University 17 116.2 35.1 2153.8 15.7 jdiaz Rutgers University 145 60.7 11.6 11166.4 5.9 yangruan Indiana University 17 54.8 77.4 3302.5 4.8 emheien UC Davis 74 48.9 0.1 104.1 8266.4 leggett University of Chicago 3 43 0 0 467 donny University of Florida 21 42.4 0 4.1 325 charngda University of Buffalo 3114 40 2.2 607.7 98.3 pela3247 University of Virginia 9261 39.7 1 22237.4 35.7 isnardoan University of Puerto Rico at Mayahguez 42 35 0.6 0.6 344.8 zincum University of Florida 8 35 0 0 32.2 ayl Washington University at St Louis 1 32 0 0 0.9 jzeng Indiana University 1 32 0 0.1 181.6 ptan Michigan State University 1 32 56.8 0 271.3 sameer UC San Diego 1 32 0 1.4 23.8 sjeganat Indiana University 13 31.1 0 2.2 202.3 jianwu UC San Diego 26 26.3 0 108 296.9 lihui Indiana University 55 20.7 1.4 327.5 891.8 feiteng Indiana University 68 20.5 0.1 181.7 110.5 nykim LSU 6 20.5 0 22.6 373.4 yye00 TACC 2 20 0 0.2 11.1 shubhada Indiana University 3 16 0.1 2.1 136.9 skamburu Indiana University 21 15.3 0 4.1 16.3 zhangbj Indiana University 8 13 0 20.7 3.7 eholk Indiana University 12 12 0 5.4 1459.9 melrom Rutgers University 2 12 0 0 23.8 upitamba Indiana University 3 12 0 6 11166 eamsden Indiana University 10 10.9 0 1.2 152.3 artiavis Rutgers University 15 10 0 0 3.8 sanketw Rutgers University 9 9.9 4.9 0 8 ktanaka Indiana University 154 9.7 0 12.2 88.5 jwalters USC - ISI 33 8.7 0.3 13 900 azebro1 Rutgers University 10 8 29.9 0 0 dino24 Rutgers University 24 8 1.4 0.7 15.7

104 9.9.4 Standard systems metrics (continued) Cloud Environment: Wall Hours By Cluster india Eucalyptus, Openstack sierra Eucalyptus, Nimbus hotel Nimbus alamo Nimbus foxtrot Nimbus

This stacked column chart represents average monthly usage of Wall Hours:

9.9.5 Cloud Environment: VM Count By Cluster india Eucalyptus, Openstack sierra Eucalyptus, Nimbus hotel Nimbus alamo Nimbus foxtrot Nimbus

105 This stacked column chart represents average counts of launched VM instances per month:

9.9.6 Standard systems metrics (continued) Cloud Environment: User Count By Cluster india Eucalyptus, Openstack sierra Eucalyptus, Nimbus hotel Nimbus alamo Nimbus foxtrot Nimbus

106 This stacked column chart represents average count of active users per month:

9.9.7 Standard User Assistance Metrics

RT Ticket System

New/Open tickets (49) in period, grouped by queue (category):

a) 001 FutureGrid account requests b) 001 Portal account requests c) 002 Eucalyptus issues d) 001 Nimbus issues e) 001 OpenStack issues f) 032 General issues g) 005 hotel issues h) 002 alamo issues i) 002 xray issues j) 000 User Support issues k) 002 Systems issues

107 Resolved tickets (173) in period, grouped by queue (category):

a) 010 FutureGrid account requests b) 009 Portal account requests c) 001 Eucalyptus issues d) 011 Nimbus issues e) 008 OpenStack issues f) 102 General issues g) 007 hotel issues h) 009 alamo issues i) 005 xray issues j) 011 User Support issues

9.9.8 SP-specific Metrics Knowledge Base:

Month All FG docs FG docs in XSEDE domain Percentage of all FG docs Jan 2013 3455 819 23.70% Feb 2013 3136 897 28.60% Mar 2013 4692 924 19.69%

Total 11283 2640 23.40% Table 1: Hits on FG KB Docs by Domain (1st quarter 2013)

Month FG KB docs modified Docs created Oct 2012 34 12 Nov 2012 10 10 Dec 2012 10 2 Jan 2013 9 0 Feb 2013 27 0 Mar 2013 11 0 Table 2: FG KB Document Editing (4th quarter 2012 – 1st quarter 2013) Users: Top 10 countries with most registered users:

UNITED STATES: 1373(79%) PUERTO RICO: 47(2.7%) ITALY: 37(2.1%) INDIA: 31(1.8%) CHINA: 26(1.5%) UNITED KINGDOM: 23(1.3%) FRANCE: 22(1.3%) SWITZERLAND: 21(1.2%) GERMANY: 20(1.2%) CANADA: 15(0.9%) The rest 40 countries: 122(7%) Projects:

108  Nineteen (19) new projects added this quarter (292 total projects)  Categorization of projects to date:

a) Project Status: Active Projects: 256(83.9%) Completed Projects: 33(10.8%) Pending Projects: 1(0.3%) Cancelled Projects: 3(1.0%) Incomplete Projects: 0(0%) Denied Projects: 12(3.9%)

b) Project Orientation: Research Projects: 239(81.8%) Education Projects: 50(17.1%) Industry Projects: 2(0.7%) Government Projects: 1(0.3%)

c) Project Primary Discipline : Computer Science: 143(46.9%) Technology Evaluation: 25(8.2%) Life Science: 29(10%) Education: 41(13.4%) Interoperability: 10(3.3%) Domain Science excluding Life Science: 27(8.9%) Not Assigned: 17(5.6%)

d) Project Service Request/Wishlist:

HPC: 152(49.8%) Eucalyptus: 152(49.8%) Nimbus: 155(50.8%) OpenNebula: 61(20%) OpenStack: 92(30.2%) Hadoop: 112(36.7%) Twister: 41(13.4%) MapReduce: 105(34.4%) Genesis II: 31(10.2%) XSEDE: 53(17.4%) gLite: 22(7.2%) Unicore 6: 18(5.9%) Vampir: 16(5.2%) CUDA: 15(4.9%) SAGA: 5(1.6%) Globus: 21(6.9%) Pegasus: 19(6.2%) MPI: 21(6.9%) PAPI: 17(5.6%) ScaleMP: 4(1.3%)

109 10 Indiana University Pervasive Technology Institute - Service Provider Quarterly Report 10.1 Executive Summary During the current quarter, IU continued hosting its Level 2 and Level 3 (Pending) systems – the Quarry virtual machine (VM) hosting and Rockhopper. We continue to work through the testing and approval process to have Rockhopper certified as an Internet2 NET+ service. Becoming registered as a NET+ service identifies this service to an important constituency that – as a group – is likely to find it easier to discover a NET+ Service than an Internet2 service. (More information is available at www.internet2.edu/netplus/.) Normal operational activities proceeded routinely and without incident. There were many significant educational and outreach activities carried out during this reporting period. Perhaps the most notable was a booth at the Plant and Animal Genome conference that became a joint National Center for Genome Analysis Support/XSEDE booth. The booth was staffed by personnel from NCGAS, XSEDE, Indiana University Research Technologies, Texas Advanced Computing Center, Pittsburgh Supercomputing Center, and San Diego Supercomputing Center. This booth allowed NCGAS and its partners to interact directly with members of the agricultural genomics community, and to showcase how we can assist them in overcoming their computational and “big data” challenges. XSEDE’s participation in the Plant and Animal Genome conference was suggested to XSEDE PI John Towns by a NSF BIO program officer, as a good venue to meet people who were not traditional XSEDE users. We made dozens of contacts with potential users of XSEDE at this conference and plan to go back next year.

Figure 1. NCGAS scientists interact with researchers at PAGXXI. 10.1.1 Indiana University Level 2 Service Provider Systems: Resource Descriptions Level 2 - Quarry Virtual Machines - The Quarry Gateway Web Services Hosting resource at Indiana University consists of multiple Intel-based HP systems geographically distributed for failover in Indianapolis and Bloomington, IN. Currently there are four HP DL160 front-end systems at each site. Each is configured with dual quad-core Intel E5603 processors, 24 GB of RAM, and a 10-gigabit Ethernet adapter. The front-end systems host the KVM-based virtual machines. VM block storage is provided by two HP DL180 servers at each site configured with a quad-core Intel X5606 processor, 12 GB of RAM, a 10-gigabit Ethernet adapter, and a RAID controller attached to an HP storage array. Quarry is used solely for hosting science gateway and

110 web service allocations, or services to support central XSEDE infrastructure. Requests are restricted to members of approved projects that have a web service component. Level 3 (Pending) -– Rockhopper: Rockhopper is a collaborative effort between Penguin Computing, IU, the University of Virginia, the University of California Berkeley, and the University of Michigan to provide supercomputing “cluster on demand” services in a secure US facility. Researchers at US institutions of higher education and federally funded research centers can purchase computing time from Penguin Computing and receive access via high-speed national research networks operated by IU. It takes just minutes to go from submitting credit card information via a web form to computing on Rockhopper (the system itself is owned by Penguin; cycles on Rockhopper are purchased from Penguin). Rockhopper is a 4.4 TFLOPS system based on AMD processors.

10.2 Science Highlights 10.2.1 National Center for Genome Analysis and Support (NCGAS) attends Plant and Animal Genome Conference (PAG) XXI The National Center for Genome Analysis Support sponsored a booth at Plant and Animal Genomics (PAG) XXI. The booth was staffed by personnel from NCGAS, XSEDE, Indiana University Research Technologies, Texas Advanced Computing Center, Pittsburgh Supercomputing Center, and San Diego Supercomputing Center. This booth allowed NCGAS and its partners to interact directly with members of the agricultural genomics community, and to showcase how we can assist them in overcoming their computational and “big data” challenges. By NCGAS and its XSEDE partners cooperating to create a unified presence to a scientific domain, we created a “one stop shop” where researchers with little or no exposure to the national cyberinfrastructure could map a computational strategy from start to finish. As potential users met with NCGAS bioinformaticians and genomic scientists, we were able to “pull in” XSEDE experts to clarify solutions to computational challenges, often right as the initial conversation was taking place. 10.3 User-facing Activities 10.3.1 System activities 10.3.1.1 Level 2 – Quarry (virtual machines) During Q1, administrators retired the OpenVZ system (moving all virtual machines to the KVM environment), and brought online 18TB of synchronous, cross-campus mirrored storage. 10.3.1.2 Level 3 (pending) – Rockhopper (Penguin on Demand commercial cluster as a service) There were no issues with Rockhopper during this quarter.

10.3.2 Services activities 10.3.2.1 Level 2 – Quarry The Quarry Gateway Hosting service was not modified during this period. Quarry continues to be used solely for hosting science gateway and web service allocations, or services to support central XSEDE infrastructure. Requests are restricted to members of approved projects that have a web service component. An external request form can be found at: http://pti.iu.edu/hps/vm-account-request.

111 10.3.2.2 Level 3 (pending) – Rockhopper The Rockhopper service usage statistic forQ1 of 2013. There were 1,112 user logins recorded, and 16,803 jobs were run. In addition, 4 new users were added to the system, increasing the total to 89.

10.4 Security There were no security issues during the reporting period. 10.5 Education, Outreach, and Training Activities Type Title Location Date(s) Hours Number Number Method of of Participa *Under- nts represent ed people Tutorial High IU 9 Jan- 2 hours 7 0 S Throughput Indianapolis, 13 Computing IN Campus 2013 Conference IU-UITS South Bend, 13-15 6 hours 70 5 S talk/presentation Regional Gary and Feb- /panel Northern Kokomo IN 2013 Campus Tour Campuses Conference Open Science IU 11-14 32 131 15 S talk/presentation Grid All Hands Indianapolis, Mar- hours /panel Meeting IN Campus 2013 Workshop Intro to Parallel IU 2 Feb- 8 hours 2 0 S Programming Indianapolis, 2013 IN Campus Workshop Intro to Parallel IU 2 Feb- 8 hours 7 0 S Programming Bloomingto 2013 n, IN Campus Conference Apache ApacheCon 26 -28 1 hour 25 5 S talk/presentation Airavata: North Feb /panel Building America 2013 Gateways to Portland, innovation OR Conference Apache Student ApacheCon 26-28 1 hour 25 5 S talk/presentation Induction: North Feb /panel Catalyzing America 2013 Collaborative Portland, Student OR Research Project

112 Type Title Location Date(s) Hours Number Number Method of of Participa *Under- nts represent ed people Conference Science Open March 20 150 25 S talk/presentation Gateways Science Grid 11- min. /panel All Hands 14th

Meeting, 2013 Indianapolis IN

Conference National Center Plant and Jan 32 200 30 S talk/presentation for Genome Animal 12- hours /panel Analysis and Genome 16th Support and Conference, 2013 XSEDE San Diego, CA Table 1. EOT activities for Q1 of Calendar 2013. *Traditionally underrepresented groups.

10.6 SP Collaborations Indiana University secured a booth at the annual National Plant and Genome Conference in January, which was shared and hosted with Pittsburgh Supercomputer Center and Texas Advanced Computation Center. The three partner organizations collaborated to promote XSEDE and the services their respective sites could provide to this community of researchers. 10.7 SP-Specific Activities Data Services (WBS 1.2.2) File system support Justin Miller continues to work with XSEDE users who are using Data Capacitor resources, most recently working with Purdue to insure that users of Steele continue to reap the benefits from using DC-WAN. Systems Operational Support (WBS 1.2.6) Virtual machines IU staff provided ongoing operational support for virtual machines (VMs) hosted on the IU Quarry system. IU staff have completed the installation of a central XSEDE Nagios deployment and are currently adding it to systems to be monitored. XD Operations Center Fail-over In the event of an emergency and/or an extended outage, Indiana University’s GlobalNOC (Global Research Network Operations Center) will serve in the role of a backup XNOC. GlobalNOC is located in Indianapolis on the IUPUI campus. GlobalNOC will be prepared to receive/send emails that are directed to [email protected]. In addition, GlobalNOC set up a dedicated phone line at (317) 274-7782 and is prepared to take phone calls directed to XNOC should the primary XOC at NCSA be unreachable due to an emergency and/or an extended outage.

113 The list below includes items on which we have made significant progress, and which should be completed in Q1 of 2013: • Phones: An XSEDE NOC greeting was added on the IU phone system, and we are waiting on the pending toll-free provider redirecting the number and process. Details are still being worked out as far as who should be authorized to request a pull of the phone to be answered by GlobalNOC. XNOC will initiate a push for all the incoming calls if need be during an emergency and/or an outage. • Email: Setup of email acceptance for failover on the IU GlobalNOC side is awaiting XSEDE/NCSA setup, policy and procedures, and documentation before we continue. An inbox is already set up, but a process/procedure is still in the works to determine how email can be pushed to GlobalNOC and by whom. The list below includes items that are on hold pending decisions by XSEDE management: • Ticketing: GlobalNOC awaits announcement of a final decision on a new trouble ticket system for XNOC. A shared account has been created for GlobalNOC to have access to the ticketing system once a decision is made as far as what ticketing system to use. The current timeline for the new ticketing system implementation is Q2 of 2013. New RT servers were deployed both at TACC and NICS. • Monitoring: Still pending; we are waiting on a setup to access current XNOC monitoring. • Process and Procedure Documentation: GlobalNOC is still waiting on centralizing all XSEDE documentation to the XSEDE Staff Wiki. • Fail-over Documentation: If process/procedure documentation is centralized into the staff wiki, then the fail-over process for documentation will become part of the overall failover process. • Training: GlobalNOC staff needs to receive training from XNOC on trouble ticket system usage and/or other tools that will be needed during an emergency and/or an extended outage. • XNOC Staff relocation: This item is still in the works. If need be, XNOC staff can relocate to GlobalNOC to work at Indianapolis during an extended XNOC outage. User Information and Interfaces (WBS 1.3.2) Documentation During Q1, the Mason User Guide was edited, reviewed by the system administrators, and updated. It was submitted to Susan Lindsay for inclusion on the XSEDE web site. Multiple docs were added to IU internal XSEDE wiki for preservation. Additional work and consulting were provided to Dr. Samy Meroueh at IUPUI for an OSG/XSEDE gateway service he provides. Working with his staff, we got the gateway home to visual standards expected for an XSEDE resource and those required by Indiana University. This quarter also began the transition of KB services away from the role of writer/originator of XSEDE KB content and towards the role of service provider. KB content will be provided by service area experts and maintained by XSEDE KB staff. Editorial staff also participated in final revisions of Campus Bridging documentation, adding clarity and readability to documents intended for wide distribution.

User Engagement (WBS 1.3.3)

114 Annual User Satisfaction Survey The annual XSEDE USER Satisfaction Survey was deployed on February 11, 2013, and closed on March 18. Data in this report reflects responses though March 12; final data is being analyzed presently and will be complete my late-April. A full report will be available in early-May. 5000 XSEDE users were randomly selected from those designated as “active” in the XSEDE user database; of these, approximately 18% (943) no longer had valid email addresses, reducing the survey sample to 4057. (The vast majority of invalid email addresses belonged to students, who likely have changed their institutional affiliation in the last 12 months.) The survey sample reflected the overall XSEDE population. That is, if the XSEDE population was comprised of 20% faculty, the survey sample included this same percentage, and so on. Over the course of the survey’s administration, there were six contacts with the respondent pool, with each reminder resulting in a marked increase in participation:

Figure 2. Graph shows participation response at each point of contact as of March 12, 2013.

In a preliminary review of the raw data, the response rate, excluding those who affirmatively refused to participate, is just over 20%, representing a 100%+ increase over last year’s response rate for XSEDE users. The factors contributing to the increased response rate may include:  A shorter, more focused survey (average duration-to-date is 9.5 minutes)  Invitation and reminders coming under PI Towns signature  Notices placed on the XSEDE website and portal  XSEDE monthly newsletter and XSEDE News articles  Increased number of contacts (a total of six contacts, including the initial invitation)

While the annual survey is a tool aimed at gauging broad, overall satisfaction with XSEDE activities and services -- a basic “report card,” if you will – there remains a need to focus on specifics for certain resources and services. With additional funding for staff and resources recently approved, several smaller, targeted surveys will be conducted in the coming project year. Planning for the first of these targeted surveys in now under way and will study the allocations process.

Preliminary Results and Highlights

115 Populations (as defined by the XSEDE central database) comprising the “completed” and “partially completed” categories are as follows, with bracketed percentages representing the category as a percentage of the survey sample:

 Graduate Students: 31.1%[40%]  Faculty: 30.8 [21%}  Post-Docs: 18.5% [14%]  University research Staff: 10.2% [8%]  Undergraduates: 2.4% [10%]  Center, Research Staff: 2.2% [2%]  Center, Non-research staff: 1.3% [less than 1%]  Other: 1.1% [1%]  Government (non Center): 1% [1%]  University, Non-research staff: 1%[1%]  Industry: .2% [less than 1%]  Non-Profit: .3%[less than 1%]

Nearly 38% of those responding report having used XSEDE resources for more than three years, with over 24% reporting one to two years of experience. Some 30% indicated less than one year of experience using XSEDE resources, and just over eight (8) percent of respondents report that they have yet to use XSEDE resources. Some 38% of respondents describe their level of experience as “novice” or “low,” with just over 35% describing their experience level as “moderately experienced” and nearly 26% of respondents describing themselves and “experienced” or “highly experienced” users. Nearly 83% of respondents report that it would be difficult or impossible to conduct their work without the use of XSEDE resources. Fewer than 14% of respondents indicate XSEDE resources are useful, but not critical in conducting their work. A small number of respondents, or 3.5%, report that using XSEDE resources is either sometimes or always unhelpful, with additional processes and requirements outweighing the benefits. Notably among these, 33% are faculty, 29% are graduate students, and 19% are post-docs. Respondents are largely unaware of resource personnel at their institutions able to assist with their use of XSEDE. Only 14% report being aware of an XSEDE staff member, and just 12% are aware of an XSEDE Campus Champion. Fifteen (15) percent indicate they are aware of local IT support personnel who are able to assist their use of XSEDE. Some 32% indicate they are aware of a colleague at their institution who is able to assist in using XSEDE resources. Forty-three (43) percent of respondents indicate they are unaware of personnel at their institution who are able to assist with their use of XSEDE. Respondents report that they are at least “somewhat aware” of most XSEDE resources and services, with Computational Resources, the XSEDE User Portal, the XSEDE website, and Support/Consulting Desk Services having the highest levels of awareness among users. As might be expected, newer services (e.g., Technology Insertion Services, XSEDE Mobile) have lower level levels of awareness. Mean satisfaction outpaces awareness in all key service areas, indicating those who use a particular service are in most cases “very” to “completely” satisfied with their experience. Respondents were largely neutral to positive about about the training methods they were asked to rate, but showed a clear preference for the ability to self serve through the use of online reources.

116 Novel & Innovative Projects (WBS 1.4.2) Pierce presented on XSEDE and Science Gateways at the Open Science Grid All Hands Meeting held in Indianapolis from March 11th through 14th. Pierce and Marru have discussed XSEDE as part of their Apache Airavata: Building Gateways to innovation talk at the annual ApacheCon held in Portland, OR from February 26th-28th 2013.

Extended Support for Science Gateways (WBS 1.5.2) Gateway Management ● Organized biweekly XSEDE Science Gateway staff calls. These feature presentations of potential interest to the entire XSEDE gateway community as well as XSEDE staff. This quarter’s presentations included: o Experiences with XSEDE & OSG Job Submissions by Yan Liu (NCSA). o Using the Eclipse Parallel Tools Platform in Support of High Performance Computing on XSEDE Resources - Jay Alameda. o Nimbus Platform - Kate Keahey - Argonne National Laboratory - The talk discussed, IaaS cloud computing capabilities of acquisition and management of customized on-demand resources. Its suitability for scientific applications, and its impact on Science Gateways. ● Led discussions and client testing of GRAM5 with Slurm provider on Stampede. ● Planned out PY3 key Activities. ● Participated in and represented Science Gateways at Architecture & Design team meetings. ● Participated in the SD&I Reviews related to Science Gateways. ● Work with Operations in review, test and SP co-ordination activities. ● Migrated to using SciForma project management software starting February. ● ● Track Chairs for the the XSEDE 13 software and software environments technical program. Scientific Workflow Use Cases IU XSEDE team members Marlon Pierce and Suresh Marru, along with Ravi Madduri (UC/Argonne National Labs) are leading the Science Workflow Use Case activity for the XSEDE Architecture and Design Team. The first Use Case is under final preparation for review. This case builds significantly on previous work by IU’s Rich Knepper and Marru on Campus Bridging and Science Gateway Use Cases, respectively. SD&I Gateway End User Activity Science gateways commonly use community credentials as part of the community allocation procedure. This allows gateways to support a large community of users without requiring individual users obtain XSEDE allocations. The PY2 additional funds activity is focused to overcome the current challenges of:

117 ● Contacting each gateway by XSEDE management to find out how many individual gateway users were supported during a given reporting period. This makes reporting inherently inconsistent and inaccurate. ● XSEDE has no direct way of tracking or auditing who is using its resources; it must trust the gateways to vet the users. ● There is no mechanism for XSEDE or Service Providers to disable individual users of a gateway at the Service Provider level.Gateways are individually responsible for tracking allocation usage of their users. The activity has just started with discussions with SD&I. Based on discussions, we are drafting a workplan that we will also discuss with the XSEDE ECSS Science Gateways staff. The goal is to make a two phase progress over the next two quarters. Community Code Gateways Experimental effort to proactively work with science PI’s in setting up new gateways. PI’s have to be willing to share research scripts and eventually “take-over” the gateway. Lay out foundational blocks like community gateway allocations, integration with XSEDE Security. Initial target is to align with Benoit Roux, U Chicago

ESSGW Projects Currently Assigned to IU: UltraScan During Q7, our primary focus was on supporting UltraScan’s execution on TACC’s Stampede and completing the transition away from Ranger. We initially explored the integration of GSI- SSH into UltraScan’s job management infrastructure. This work was performed and contributed also to the open source Apache Airavata project. However, job monitoring over in this approach is a challenge. Fortunately, we instead were able to make successful transition to GRAM5 on Stampede in collaboration with TACC systems administrators and the Globus team. This involved the conversion of SGE modules to SLURM modules and significant testing phase with several issues reported and resolved by TACC administrators and Globus developers. Testing included both serial and MPI jobs. We also identified an issue where job threads on GRAM5 were not correctly disposed of after the jobs completed. We developed an internal workaround solution for UltraScan, which was superseded by suggested fixes from the Globus team. These fixes are still being tested. DES Our support for the DES Simulations Working Group’s workflow effort also concentrated on a transition away from Ranger to Stampede. In addition to the middleware testing and migration described above for UltraScan, we worked with the DES team to run benchmark comparisons of the performance of their LGADGET code on Ranger and Stampede. During this quarter we also extended the DES workflow to include post-processing with the applications RNN and Rockstar. Rockstar is a non-MPI parallel code: it needs to be started on a master node with single CPU, and then a second job needs to be submitted with the required processors to start processing. Communication between master and workers use TCP. Setting the walltime to such jobs is very tricky. We created a wrapper script to do the job but there are some manual steps to do this processing. In the upcoming quarter, we will work with TACC administrators to find a better solution for running Rockstar in the DES workflow. Finally, we co-authored an XSEDE 13 paper with the DES collaborators.

118 Biodrugscreen ○ ECSS work plan

■ Team meetings to understand the requirements.

■ Have a draft with list of objectives. PI need to fill the effort and resources.

Extended Support for Training, Education, and Outreach (WBS 1.5.3)

Type Title Location Dates Hours Number Number Method of of Particip *Under- ants represe nted persons

Conference Apache ApacheCon February 1 hour 25 5 S talk/presenta Airavata: North 26-28th, tion/panel Building America 2013 Gateways Portland, OR to innovation

Conference Apache ApacheCon February 1 hour 25 5 S talk/presenta Student North 26-28th, tion/panel Induction: America 2013 Catalyzing Portland, OR Collaborati ve Student Research Project

Conference Science Open March 11- 20 150 25 S talk/presenta Gateways Science Grid 14th 2013 min. tion/panel All Hands

Meeting, Indianapolis IN Table 2. ECSS EOT Activities for Q1 of Calendar 2013. *Traditionally underrepresented groups.

Campus Bridging (WBS 1.6.5) Discussion of campus bridging within XSEDE and within the national cyberinfrastructure community The XSEDE Campus Bridging team continues to have discussions with units in XSEDE and interested parties in the national community (OSG, Educause ACTI) in order to disseminate XSEDE’s Campus Bridging initiatives and gather comments and suggestions for community needs. Barbara Hallock introduced the XSEDE Campus Bridging strategy at the OSG All Hands

119 Meeting in March of 2013. The two major thrusts areas of XSEDE Campus Bridging, the GFFS Pilot Project and the Campus Bridging Cluster Software, made major progress in this quarter. Definition of Use Cases and quality attributes for XSEDE’s Campus Bridging efforts The CB Team has continued to work with the Architecture and Design team to complete Use Case definition and the Level 3 Decomposition of Campus Bridging Use Cases, and assist in the refinement of general Use Cases for all of XSEDE. GFFS Pilot Project All of the GFFS Pilot sites were given access to the GFFS Software in this period and allowed to work with XSEDE resources as well as with resources at UVA, and across participating campuses. Campus sites benefited greatly from UVA “office hours” in which UVA staff worked with pilot sites directly via phone or skype in order to address installation and configuration issues. At the time of this report, all sites have set up the software locally and tested file sharing within the GFFS, testing of job submission to Unicore on Kraken is still in progress, and the CUNY site has started the process of setting up Unicore locally in order to accept jobs submitted from XSEDE. TEOS Evaluation, Operations, and SD&I have all created means for capturing pilot site responses to the software and implementation processes. It is the responsibility for the Campus Bridging Team to ensure that responses to the software are adequately captured in order to inform XSEDE and Genesis II Teams. Rocks Roll Software Project The additional staff at Cornell dedicated to the Campus Bridging Software Project have created Rocks Rolls and rpm packages that are ready for distribution and testing. The Cornell staff will develop a test plan that is approved by SD&I and carry testing, presenting verification of the test data to SD&I and Operations. Campus Bridging and SD&I have agreed on a plan for distributing the software packages, and the Campus Bridging team has a list of friendly users for testing the software packages from Linda Akli’s MSI Campus call. Challenges in Program Year Two The remaining challenges in Program Year Two will be to continue to gather responses from the GFFS Pilot sites and to appropriately span the gap between the pilot implementations of the software and the production implementation of the software scheduled for PY3. The Software Package Program will require adequate documentation and information for potential users to facilitate the installation and maintenance of clusters created with the software packages, by explicitly providing pointers to self-help resources upstream of XSEDE and forums and documentation available via XSEDE in order to ensure that users of the software will be able to find the correct resources to facilitate implementation and usage of the software packages locally.

10.8 Publications Conference Proceedings Zhang, H., and M. J. Boyles, "Visual exploration and analysis of human-robot interaction rules" SPIE, Visualization and Data Analysis, vol. 8654, Burlingame, California, Feb 2013. Journal Article Kluge, M., S. Simms, T. William, R. Henschel, A. Georgi, C. Meyer, M. S. Mueller, C. A. Stewart, W. Wünsch, and W. Nagel, "Performance and quality of service of data and video

120 movement over a 100 Gbps testbed" Future Generation Computer Systems, vol. 29, no. 1, Jan 2013. Presentation Hallock, B., Campus Bridging: What is it and why is it important?, OSG All Hands Meeting, Mar 2013. Pierce, M., Science Gateways, OSG All Hands Meeting , Indiana University - Purdue University 420 University Blvd. Indianapolis, IN 46202, Mar 2013. Pierce, M., S. Marru, S. Wijeratne, R. Singh, and H. Suriyaarachchi, Apache Airavata: Building Gateways to Innovation, ApacheCon NA, Portland Or., Feb 2013. Zhang, H., and M. J. Boyles, Visual exploration and analysis of human-robot interaction rules, SPIE, Visualization and Data Analysis 2013, Burlingame, California, Feb 2013. Hallock, B., Rockhopper: Penguin on Demand at Indiana University, Presented in-booth at the Plant and Animal Genome XXI conference, San Diego, CA., Jan 2013. LeDuc, R., Statistical Consideration for Identification and Quantification in Top-Down Proteomics, American Society for Mass Spectrometry - Sanibel Conference 2013, St Pete Beach, FL, Jan 2013. 10.9 Metrics 10.9.1 Standard User Assistance Metrics 10.10 User Information and Interfaces (WBS 1.3.2) Knowledge Base Summary statistics for the XSEDE Knowledge Base: Metric For current Number of KB documents available at end of quarter quarter 586 Number of new KB documents added 10 Total number of retrievals 285,162 Total number of retrievals minus bots 171,097 Table 3. High-level XSEDE Knowledge Base metrics for current quarter. Metric PY to date (when Number of KB documents available at end of quarter applicable) 586 Number of new KB documents added 108 Total number of retrievals 678,737 Total number of retrievals minus bots 375,325 Table 4. High-level XSEDE Knowledge Base metrics for current period and project year to date.

121 Knowledge Base Summary statistics for the XSEDE Knowledge Base:

Number of Documents Viewed 150000 Total XSEDE 100000 Accesses By Month 50000 Accesses by Users (minus 0 Bots) Jan-13 Feb-13 Mar-13

Figure 3. Total Knowledge Base accesses by month, current reporting period (January-March 2013). KB Document Editing 60 50 Number of 40 Documents 30 Modified 20 Number of 10 Documents 0 Added 1-Jan-13 1-Feb-13 1-Mar-13

Figure 4. Knowledge Base editing activity (new entries added; existing entries modified or removed) by month, current reporting period (January-March 2013).

Number of Documents 590 585 580 Number of 575 Documents 570

Figure 5. Total number of documents in XSEDE Knowledge Base by month, current reporting period (January-March 2013).

122 10.10.1 SP-specific Metrics 10.10.1.1 Usage metrics – current quarter System # # # Syste Storage TB TB read Allocati VMs TB m high written ons alloca allocated water ted mark Quarry VM 9 39 1950.8G Rockhopper B Data Capacitor WAN 40 6.5 2B 319 85 250 Mason (starts Q2 2013) Table 5. Service Provider system key usage metrics for the current quarter (Q1 of 2013–January- March 2013). System Overall # Planned # Unplann Total % planned downti unplann ed minutes in uptime downti me ed downti reporting mes duration downti me period total mes duration (minute total s) (minute s) Quarry VM 100% 0 0 0 1,400 132,480 Rockhopper 100% 0 0 0 0 129,600 Data Capacitor WAN 99.8% 0 0 1 254 129,600 Mason (starts Q2 2013) Table 6. Service Provider system key usage metrics for the current quarter (Q1of 2013 –January- March 2013) 10.10.1.2 Usage metrics – project year 2 System # # # # Storage TB TB read Allocati VMs TB acces high writte ons allocat allocat ses water n ed ed mark Quarry VM 9 39 1950.8 Rockhopper 1,112 GB Data Capacitor WAN 47 6.5 43.1B 322 345 555 Mason (starts Q2 2013) Table 7. Service Provider system key usage metrics for PY2 (July 2012–March 2013).

System Overall # Planned # Unplann Total % planned downti unplann ed minutes uptime downti me ed downti in mes duration downti me reporting total mes duration period (minutes total ) (minutes ) Quarry VM 100% 2 225 3 1,418 263,520 Rockhopper 100% 0 0 0 0 388,800 Data Capacitor WAN 99.72% 2 780 2 306 394,560

123 Mason (starts Q2 2013) Table 8. Service Provider system key usage metrics for PY2 (July 2012–March 2013). 10.10.2 Standard systems metric N/A.

124 11 Georgia Tech, Keeneland - Service Provider Quarterly Report 11.1 Executive Summary The Keeneland Project completed its first full quarter of production on the Keeneland Full Scale System (KFS) at the end of March 2013. Usage has grown to ~200 users with quarterly allocations of 1.5M GPU-hrs (SUs). The Keeneland Initial Delivery System (KIDS) has been shifted from production to development, discretionary/start-up accounts, and supporting education, outreach, and training. The software on KIDS was upgraded and reconfigured to match KFS. 11.1.1 Resource Description KFS – 264-node cluster composed of HP SL250 nodes, each with two Sandy Bridge CPUs, three NVIDIA M2090 GPUs, 32 GB host memory, and a Mellanox FDR IB interconnect through a Mellanox FDR IB enterprise switch. KIDS – 120-node cluster composed of HP SL390 nodes, each with two Westmere CPUs, three NVIDIA M2090 GPUs, 24 GB host memory, and a Mellanox QDR IB interconnect through a QLogic QDR IB enterprise switch. 11.2 Science Highlights The following table lists the top 10 users on Keeneland with their research topic. Research Area PI Research Title Molecular Biosciences Justin MacCallum MeLD: Modeling with Limited Data--a physics-based integrative modeling approach to protein structure Materials Research David Morse Testing universality in diblock copolymers using Graphics Processing Units Chemistry Bernard Montgomery Pettitt Salt and Co-Solvent Effects for Proteins and Nucleic Acids Molecular Biosciences Ross Walker Benchmarking and Profiling of the GPU

Chemistry Andreas Goetz Understanding ion solvation at the air/water interface from adaptive QM/MM molecular dynamics simulations Molecular Biosciences Michael Shirts Modeling Biological Mcromolecules in Chromatography and Drug Design Molecular Biosciences Isaiah Charles Sumner Custom Force Field Development for the Calculation of Protein Conformational Change Astronomical Sciences James Lombardi The Hydrodynamics of Binary Mergers

Chemistry Andreas Goetz GPU accelerated adaptive QMMM MD with

Molecular Biosciences David A Case Determination of solution structures by NMR and dependence on pH and salt of biomolecular interactions

125 11.3 User-facing Activities 11.3.1 System Activities The Keeneland Full Scale System (KFS) became an XSEDE production resource at the end of October 2012. Although KFS frequently is at very high utilization levels during prime time, queues have sometimes emptied over long holiday weekends, and during March a problem with the NICS parallel file system took the system down for about a week. We anticipate the during the next quarter utilization will stabilize as initial production issues are resolved and more users are added to the system. KFS has a wide range of job size distributions. Although the majority of the number of jobs are small, the majority of the workload is in the mid- to large-size jobs. The bar chart shows the distribution of the KFS workload by node count (3 GPUs/node). The peak in large jobs corresponds to the peak utilization on KFS in December 2012. In January 2013 all XSEDE workload had migrated to KFS from KIDS, so running large jobs became more difficult (longer wait times), and the main workload shifted to the mid-size jobs. Since KFS became an XSEDE production resource at the end of October 2012, the usage of the system has been dominated by the top 5- 10 accounts, even as the number of accounts has grown. In particular the top 10 accounts has accounted for between 95-100% of the system utilization each month. Although it shows greater fluctuation, the top 5 accounts account for between 82-90%. As more information become available with new users being added, we will watch how the additional accounts change the usage patterns on the system or whether former top 5 or top 10 accounts are just replaced with new large accounts. Over half of the Keeneland XSEDE production workload is being used for Molecular Biosciences, followed by Material Research and Chemistry. Much smaller amounts go for Astronomical Sciences, Computer and Computational Research, Cross-Disciplinary Activities, Mathematical Sciences, and Physics. About 1/3% of Keeneland use is for Earth Sciences, Chemical, Thermal Systems, Networking and Communications Research, Advanced Scientific Computing, and Information, Robotics, and Intelligent Systems.

126

11.3.2 Services Activities The Keeneland Advanced Application Support completed their activities working with the BEAST/BEAGLE phylogenetics code and turned the code over to the community repository. The modified code allows users to solve problems in hours that previously took months on a GPU-accelerated workstation or small server. The Keeneland Advanced Application Support worked with the UCSD and SDSC on SCEC AWP-ODC, a scalable GPU-accelerated code to simulate earthquake ground motions. Keeneland profiled and helped debug the code, which now runs on Keeneland, the DoE Titan system, and elsewhere. The Keeneland Advanced Application Support has begun work on a scalable GPU code for Brownian dynamics (BD) simulations with an algorithm specifically for GPU-accelerated systems. The algorithm was developed by a Georgia Tech professor for use 11.4 Security Nothing. 11.5 Education, Outreach, and Training Activities Keeneland collaborated with the Pittsburgh Supercomputing Center to provide the platform for the OpenACC Workshop and hosted a remote site at Georgia Tech.

127 Type Title Location Date(s) Hours Number of Number Method Participants of Under- represen ted people Work OpenACC Workshop Georgia 15-16 10 11 S shop Tech, Jan Atlanta, 2013 GA

11.6 SP Collaborations Nothing. 11.7 SP-Specific Activities Nothing. 11.8 Publications CONFERENCES/PRESENTATIONS 1. Azzam Haidar and Stanimire Tomov (organizers), “Computational Challenges for Large Scale Heterogeneous Applications”, Mini-symposiums MS 44 and MS63 at SIAM CSE’13, Boston, MA, February 25-March 1, 2012.

2. Azzam Haidar (presenter), R. Solca, S. Tomov, J. Dongarra, and T. Schulthess, “A Hybrid CPU-GPU Generalized Eigensolver for Electronic Structure Calculations”, SIAM CSE’13, Boston, MA, February 25-March 1, 2012.

3. M. Gates, “Achieving high performance with multiple-GPU non-symmetric eigenvalue solver”, SIAM CSE’13, Boston, MA, February 25-March 1, 2012.

4. E. Agullo, J. Dongarra, H. Ltaief (presenter), and S. Tomov, “Developing Numerical Algorithms on Heterogeneous Architectures with High Productivity in Mind”, SIAM CSE’13, Boston, MA, February 25-March 1, 2012.

5. T. Dong, J. Dongarra, S. Tomov, and I. Yamazaki (presenter), “Multi-GPU Tridiagonalization on Shared-and-distributed-memory systems”, SIAM CSE’13, Boston, MA, February 25-March 1, 2012.

6. S. Tomov (presenter), J. Dongarra, I. Yamazaki, A. Haidar, M. Gates, P. Luszczek, J. Kurzak, S. Donfack, T. Dong, K. Kabir, C. Cao, Y. Jia, “Numerical Linear Algebra Libraries for Emerging Architectures: Challenges and Approaches”, Numerical Methods for PDEs: in Occasion of Raytcho Lazarov’s 70th Birthday College Station, TX, January 25-26, 2013.

PUBLICATIONS

128 1. C. Cao, J. Dongarra, P. Du, M. Gates, P. Luszczek, and S. Tomov, “clMAGMA: High Performance Dense Linear Algebra with OpenCL”, The International Workshop on OpenCL (IWOCL’13, submitted 02/15), Atlanta, GA, May 13-14, 2013.

2. T. Dong, V. Dobrev, T. Kolev, R. Rieben, S. Tomov, and J. Dongarra, “Hydrodynamic Computation with Hybrid Programming on CPU-GPU Clusters”, ICS’2013 (submitted 01/18), Eugene, Oregon, USA.

3. A. Haidar, M. Gates, S. Tomov, and J. Dongarra, “Toward a scalable multi-GPU eigensolver via compute-intensive kernels and efficient communication”, ICS’2013 (submitted 01/18), Eugene, Oregon, USA.

4. A. Haidar, R. Solca, M. Gates, S. Tomov, T. Schulthess, and J. Dongarra, “Leading edge hybrid multi-GPU algorithms for generalized eigenproblems in electronic structure calculations”, ISC’13 (submitted 02/10), Leipzig, Germany.

5. T. Dong, J. Dongarra, S. Tomov, and I. Yamazaki, “Tridiagonalization of a symmetric dense matrix on a GPU cluster”, AsHES’13 IPDPS workshop (submitted 01/11), Boston, USA.

6. S. Donfack, S. Tomov, and J. Dongarra, “Dynamically balanced synchronization-avoiding LU factorization with multicore and GPUs”, in progress (available upon request), 2013.

7. Lee, Seyong, and Vetter Jeffrey S., Early Evaluation of Directive-Based GPU Programming Models for Productive Exascale Computing, SC12: ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis, 11/2012, Salt Lake City, Utah, USA, (2012)

8. “Improving Utilization and Application Performance in High-Performance GPGPU Clusters with GPGPU Assemblies,” Alexander Merritt, Naila Farooqui, Vishakha Gupta, Magda Slawinska, Ada Gavrilovska, Karsten Schwan - under submission to a first tier HPC conference

9. “Multi-tenancy on GPGPU-based Servers,” Dipanjan Sengupta, Raghavendra Belapure, Karsten Schwan – under submission.

11.9 Metrics 11.9.1 Standard User Assistance Metrics

Keeneland January-March 2013: tickets opened: 146 tickets closed: 161 distribution of response time:

0-1 hrs:16 1-24 hrs: 55 1-7 days: 50 1-2 weeks: 19 > 2 weeks: 6

129 11.9.2 SP-specific Metrics 11.9.3 Standard systems metrics XDMod team will provide tables of metrics. The originally proposed list: 1. Job summaries by core count or memory footprint 2. Machine usage by scientific discipline 3. Machine usage by project 4. Top 20 users, sorted my machine use 5. Other standard metrics that the SP-Forum and NSF agree to.

130 12 NICS - Service Provider Quarterly Report 12.1 Executive Summary In 2009, the National Institute for Computational Sciences (NICS) delivered the first academic petaflop computer to the NSF community—a Cray XT5 called Kraken. By the end of 2010, systems at NICS were delivering more than 70% of all NSF compute cycles. This quarter, Kraken sustained a utilization of 96% and a 98% uptime while providing roughly 56% of the total CPU hours delivered by XSEDE resources (Table 4, Figure 29).

The addition of the SGI Altix, called Nautilus, and the Remote Data and Visualization (RDAV) center serves to broaden the services provided by NICS to the NSF community and increases the potential for breakthrough science (Section 12.2). RDAV's purpose is to aid in the significant challenge of transforming large-scale data into knowledge and insight by providing scientists with well-engineered and well-supported remote visualization, analysis, and scientific workflow technologies. Nautilus provided an 89% uptime for the quarter and 45% utilization (Table 2).

Support staff at NICS responded to 823 new XSEDE tickets and closed 839 tickets (Figure 21) during the quarter.

12.1.1 Resource Description NICS currently has two NSF funded computational resources: Kraken and Nautilus. These systems share a Network File System (NFS) that contains user directories, project directories and software directories. One-time password tokens provide secure access to both the computational and storage resources at NICS.

12.1.1.1 Kraken Kraken is a Cray XT5 consisting of 9,408 compute nodes, each containing two 6-core AMD Istanbul Opteron processors and 16 GB of on-node memory. The result is 112,896 compute cores that deliver 1.17 PF at peak performance with 147 TB total memory. Communications take place over the Cray SeaStar2+ interconnect. A parallel Lustre file system provides 3.3 PB (raw) of short-term data storage.

12.1.1.2 Nautilus Nautilus, an SGI Altix UV 1000 system, is the centerpiece of NICS Remote Data and Visualization (RDAV) Center that is also located at ORNL. It has 1024 cores (Intel Nehalem EX processors), 4 TB of global shared memory, and 8 GPUs in a single system image yielding 8.2 TF at peak performance. A parallel Lustre file system provides 427 TB (raw) of short-term data storage.

12.1.1.3 HPSS Archival Storage The High Performance Storage System (HPSS), developed and operated by ORNL, is capable of archiving hundreds of petabytes of data and can be accessed by all major leadership computing platforms. Incoming data is written to disk and later migrated to tape for long term archiving. This hierarchical infrastructure provides high-performance data transfers while leveraging cost-effective tape technologies. Robotic tape libraries provide tape storage. The center has four SL8500 tape libraries holding up to 10,000 cartridges each. The libraries house a total of 24 T10K-A tape drives (500 GB cartridges, uncompressed), 60 T-10K-B tape drives (1 terabyte cartridges, uncompressed), and 20 T10K-C tape drives (5 terabyte cartridges, uncompressed). Each T10K-A and T10K-B

131 drive has a bandwidth of 120 MB/s. Each T10K-C tape drive has a bandwidth of 240 MB/s. Disk storage is provided by DDN storage arrays with nearly a petabyte of capacity and over 12 GB/s of bandwidth. This infrastructure has allowed the archival system to scale to meet increasingly demanding capacity and bandwidth requirements with more than 12.7 PB of NICS data stored as of April 2013. 12.2 Science Highlights 12.2.1.1 Chemical Biology: Conformational Changes of Cancer-suppressor and HIV-promoting Proteins During Cell-binding (Hirsh Nanda, Department of Physics, Carnegie Mellon University, and the National Institute of Standards and Technology Center for Neutron Scattering) See main body of XSEDE quarterly report

12.2.1.2 Aeronautics and astronautics: Turbulence Modeling: Droplet-laden Isotropic Turbulence (Antonino Ferrante, William E. Boeing Department of Aeronautics and Astronautics, University of Washington, Seattle) See main body of XSEDE quarterly report, 2.3

12.2.1.3 Energy Efficiency: Autotune — A Calibration Methodology for Energy Modeling (Joshua New and Jibo Sanyal, Building Technologies Research and Integration Center, Oak Ridge National Laboratory)

Worldwide, buildings are major energy consumers. In the U.S., for example, they use 41 percent of the nation’s primary energy; and in other countries, the building energy consumption is similar, ranging from 30 percent to 40 percent. One way to improve the energy efficiency of buildings is computer modeling and simulation. Energy models are representations of buildings applied in simulations, which consist of design and operating parameters — window-performance values, installed electricity, ventilation and more — associated with energy consumption. While energy models have been commonly employed to effect energy savings, and a number of energy-modeling computer applications exist, getting models right is very difficult because of the margin for error created by the numerous input parameters. The missing element has been a methodology for calibrating energy models, says Joshua New of the Building Technologies Research and Integration Center at Oak Ridge National Laboratory (ORNL). As an answer to the need, New and ORNL colleague Jibo Sanyal are leading a research project focused on developing Autotune, an advanced analytical and optimization methodology that leverages terabytes of high-performance-computing-generated simulation data and data mining with multiple-machine learning algorithms for quickly calibrating a building energy model to measured (utility or sensor) data. The project has been made a reality by XSEDE allocations on the Nautilus supercomputer — approximately 300,000 service units so far. Housed at ORNL, Nautilus is managed by the University of Tennessee’s National Institute for Computational Sciences (NICS). The Center for Remote Data Analysis and Visualization of NICS is providing consultative services to New and Sanyal’s team. The Autotune project has employed Nautilus to run about 75 percent of the simulations for warehouse and 100 percent for retail buildings, two of the most popular types, as categorized by square-footage use in the U.S. The researchers plan to deploy Autotune as a desktop application, website and web service. In addition, as part of an “open science” approach, the massive amounts

132 of data emanating from the project are being made available to citizen scientists through the Autotune project website (http://autotune.roofcalc.com). 12.2.1.4 Evolutionary Biology: Protein Translation and Codon Usage Bias (Michael Gilchrist, Department of Ecology and Evolutionary Biology, University of Tennessee, Knoxville)

Codon usage bias — the non-uniform use of codons, or triplets of different nucleic acids, within a gene sequence — is a ubiquitous biological phenomenon. A research team led by Michael Gilchrist of the University of Tennessee, Knoxville, is investigating how ribosomes choose certain codons over others in manufacturing proteins. This allows them to visualize the relationship between the genetic makeup of a cell and reproductive success, as well as extract information on gene expression from the codon sequences. “This research is helping to advance our quantitative understanding of the costs and errors associated with protein translation,” Gilchrist explains. “It also demonstrates how we can extract more and more meaningful information from Figure. How codon usage changes with genomic datasets using an integrated set of biologically gene expression, the process in which based models rather than generic algorithms or simple linear information from a gene is used in the models.” XSEDE funding allowed Gilchrist to receive formation of a gene product (in this consultative support from the Center for Remote Data research, a protein). Analysis and Visualization (RDAV) of the National Institute for Computational Sciences (NICS). RDAV optimized the R programming code Gilchrist’s team was using so that the code could be run on the Nautilus supercomputer. The consultants also helped Gilchrist’s team understand the trade-offs between ease of programming and quality of information output. Because of the high-performance-computing support, Gilchrist says, calculations that took 24 hours on the desktop computer could be executed in less than an hour on Nautilus, greatly increasing the researchers’ ability to develop their methods and test alternative hypotheses about the forces driving the evolution of codon usage bias. He also says that expediting the calculations facilitates his team’s ability to develop their intuitive understanding because they get the results while the ideas are fresh in their minds. In addition to developing models to investigate codon usage bias, the researchers plan to develop computational tools that other researchers can use in easily fitting their models to data.

12.2.1.5 Theoretical and Computational Astrophysics: Turbulence in the Outer Regions of Protoplanetary Disks (Jacob Simon, University of Colorado, Boulder)

Many newly formed stars are surrounded by swirling masses of warm dust and gas called protoplanetary disks. These disks may become celestial bodies such as planets and asteroids.

But how they make that transformation will remain a mystery until researchers can get a grasp on the disordered movement, or turbulence, that characterizes the constituent gases of the disks. “By understanding the nature of the gases, we can learn something about how small particles interact with each other, coagulate to become larger particles and then ultimately form planets,” says Jake Simon of the University of Colorado, principal investigator of a

133 research project currently facing two primary challenges in the quest to understand protoplanetary disk turbulence: developing correct models for the simulations and designing algorithms that capture the nature of the Hall effect, which refers to a voltage difference that occurs across an electrical conductor.

While Simon and his colleagues address those encumbrances, they are achieving successes. For example, they now understand ambipolar diffusion, in which electrons and ions in protoplanetary disks are dragged along by a magnetic field. “If the ions and electrons don’t collide with the neutrals frequently enough, ambipolar diffusion acts to damp out the turbulence,” Simon explains. “The degree to which this happens has been explored with our high-resolution numerical simulations that we have run on the Kraken supercomputer. We believe we now have a much better understanding of how disks behave in their outer regions, far from the central star.”

Simon and his team have used more than 4 million service units (compute hours) on Kraken so far, including an average of approximately 585 cores per run and a single-run high of 18,432 cores. He says: “With HPC resources like Kraken, we can run several of these high-resolution computer simulations simultaneously. This speeds up our research considerably.”

Figure. Diagram of a protoplanetary disk around a young star. Angular momentum is transported outward and mass is transported inward and onto the star via disk turbulence. This turbulence is studied using local simulations of a small patch of disk (the 3D box on the lower right; this shows the gas density in the turbulent state). The presence of an external magnetic field similar to the one shown enhances this turbulence in the outer disk regions.

134

12.2.1.6 Astrophysics: Alignment of Magnetized

Accretion Disks and Relativistic Jets with Spinning Black Holes (Jonathan McKinney, University of Maryland, College Park; Alexander Tchekhovskoy, Princeton Center for Theoretical Science; Roger Blandford, Kavli Institute for Particle Astrophysics and Cosmology, Stanford University)

Dense and compact collections of matter with gravitational fields so strong that not even light can escape, black holes are compelling scientific curiosities chock full of information. Indeed, the black Figure. 3D snapshot of an evolved black hole model. The disk and jet near the black hole — which NASA likens to having 10 times the hole are aligned with the black-hole spin mass of the Sun packed into a sphere about the size of axis and point mostly in and out of the New York City — could unveil revelations about the figure plane, whereas at larger distances early universe and either confirm or disprove the jet points roughly halfway between the black-hole spin axis and disk’s rotational Einstein’s general relativity theory. axis (pointing along the orange cylinder). Investigations require calculations and simulations, The predictions from this research challenge an overly simplified and coupled with observations of the entities that uniformly symmetrical view of the jet and accompany black holes — namely accretion disks and disk relative to the black hole. relativistic jets. Researchers Jonathan McKinney,

Alexander Tchekhovskoy and Roger Blandford in a paper published in Science in January 2013 explain how, contrary to assumptions, black holes, accretion disks and relativistic jets are not in a simply aligned configuration but are actually in one that twists and bends. To make the discovery, the researchers reduced the symmetry in the numerical code by using spherical polar coordinates that employ radius and two different angles to describe the coordinates. The team’s simulations also dispelled the belief that the accretion disk controlled the relativistic jet — the interaction was shown to be just the opposite. XSEDE resources provided the highly parallel computing capability and the compute time necessary to create several models and run 12 simulations on the Kraken supercomputer. The Texas Advanced Computing Center (Lonestar and Ranch supercomputers) as well as NASA Advanced Supercomputing (Pleiades supercomputer) also provided support. The researchers performed most of the analyses and visualizations on the Nautilus supercomputer. McKinney says the visualizations were crucial to understanding what was happening. Recently published images (in Science, by Sheperd Doeleman of MIT) of the jet-launching structure near the supermassive black hole in the neighboring M87 galaxy will provide McKinney et al. with the observational element necessary for comparison with simulations. They anticipate that within two to three years they will be able to make statements as to whether they believe the general relativity theory is correct.

135 12.3 User-facing Activities 12.3.1 System Activities 12.3.1.1 Kraken Availability Kraken had an overall system availability of 98% for this quarter with 47 total hours of downtime. Downtime for the quarter consisted of 43 hours of scheduled downtime and 4 hours of unscheduled downtime (Table 1).

Table 4: Summary of maintenance statistics for Kraken in Q1 2013.

Maintenance Stats - Kraken Cray XT5

Number of planned reboots 7

Number of unplanned reboots 2

Total reboots 9

Number of job failures due to system faults 505

Total time in period 2,159 hours (100%)

Scheduled Downtime 43 hours (2%)

Unscheduled Downtime 4 hours (0%)

Total Downtime 47 hours (2%)

Total time available to users (total - downtime) 2,112 hours (98%)

% System Utilization 96%

12.3.1.2 Nautilus Availability Nautilus had an overall system availability of 89% for this quarter. Downtime for the quarter consisted of 25 hours of scheduled downtime and 219 hours of unscheduled downtime (Table 2).

136 Table 5: Summary of maintenance statistics for Nautilus in Q1 2013.

Maintenance Stats – Nautilus SGI UV

Number of planned reboots 2

Number of unplanned reboots 7

Total reboots 9

Total time in period 2,160

Scheduled Downtime 25 (1%)

Unscheduled Downtime 219 (10%)

Total Downtime 244 (11%)

Total time available to users (total - downtime) 1,916 (89%)

System Utilization 45%

12.3.2 Services Activities 12.3.2.1 Kraken Software Packages NICS currently supports 409 unique application builds on Kraken that include pre-compiled binaries and builds with PGI, GNU, Cray, and Intel compilers. These builds include 217 unique versions and 129 unique applications and libraries.

Environment In the last quarter, the NICS shared Lustre file system, “Medusa”, was mounted on the Kraken system. This change allows users of the NICS Nautilus system to easily migrate date between the two systems to facilitate analysis and/or visualization with the results of computations on Kraken.

12.3.2.2 Nautilus Software Packages NICS’ staff currently supports 311 unique application builds on Nautilus that include pre- compiled binaries and builds with PGI, GNU, and Intel compilers. These builds include 218 unique versions and 120 unique applications and libraries.

137

Environment In the first quarter of 2013, no notable environment changes occurred on the Nautilus system.

12.4 Security There were no security incidents in the first quarter of 2013.

12.5 Education, Outreach, and Training Activities

Table 6: Summary of Education, Outreach, and Training activities through Q1 2013. Type Title Location Date(s) Hours Number of Method Funding Participants Sources

Seminar “Modeling Materials at 53rd Sanibel 2/17/13 In Person NICS the Nanoscale- Quantum Symposium, St. 1 80 Dynamical Perspective” Simons Island, GA Seminar “Portal for Petascale Regional 2/11/13 In Person NICS Lifescience Applications CyberInfrastructur and Research” e Symposium and 1 50 Workshops, Clemson, SC Seminar “Impact of computing on U.Tennessee, 3/13 In Person XSEDE, next generation Environmental 1 25 NICS knowledge discovery” Biology Workshop "Beacon Project Training Oak Ridge Nat’l 3/21- In Person NICS Workshop on the Intel Lab 22/13 12 20 Xeon Phi” Seminar “Impact of computing on 2/13 In Person XSEDE, U.Tennessee, next generation 1 35 NICS Psychology Dept. knowledge discovery” Workshop Crash Course in 3/4/13 In Person XSEDE, University of 6 23 Supercomputing Tennessee NICS Workshop Crash Course in 3/11/13 In Person XSEDE, University of 6 19 Supercomputing Tennessee NICS Workshop Crash Course in 3/18/13 In Person XSEDE, University of 6 18 Supercomputing Tennessee NICS Workshop Crash Course in 3/25/13 In Person XSEDE, University of 6 22 Supercomputing Tennessee NICS Workshop “ParaView & Parallel R” 3/26/13 In Person XSEDE, University of 8 20 Tennessee NICS Workshop “Programming the new 3/27/13 In Person XSEDE, University of 7 16 Intel Xeon” Tennessee NICS

138 12.6 SP Collaborations EPSCOR The Experimental Program to Stimulate Competitive Research, or EPSCoR, establishes partnerships with government, higher education and industry that are designed to effect lasting improvements in a state's or region's research infrastructure, R&D capacity and hence, its national R&D competitiveness. NICS participates along with researchers from twenty-seven other states and the Commonwealth of Puerto Rico. The partnership is based on existing and planned collaborations in the advanced materials and systems biology domains where computational science is driving new approaches and insights. The collaborative team has proposed to build cyberinfrastructure (CI) linked, community specific knowledge environments that embody the desktop to XSEDE ecosystem by using campus-based CI at a regional research institution as an essential bridge for connecting faculty investigators to national resources such as the XSEDE.

Recently, NICS EPSCoR staff generated new modules for HMMER for the latest upgrade on Kraken and began working on Systems Biology Science Gateway with the web interface known as PoPLAR: the Portal for Petascale Lifescience Applications and Research, currently can run BLAST and HMMER through portal and is still under testing for transferring large files for analysis. Staff also engaged with researchers from Medical University of South Carolina to use AutoDock on Kraken. In addition, Bhanu Rekepalli collaborated with Nick Panasik from Claflin University to port the LINUS python package to Kraken and with William Mondy on biocad modeling on Nautilus.

Keeneland Keeneland is a five-year, $12 million NSF Track 2D award made to the Georgia Institute of Technology. The Keeneland goal is to make an experimental, high-performance computing system consisting of an HP system with NVIDIA Fermi accelerators available for use by the XSEDE user community. System support and user support are provided by NICS. Staff at Georgia Tech and Oak Ridge National Lab (ORNL) provides advanced application-support. Software development activities are funded at Georgia Tech, University of Tennessee, and ORNL. The Keeneland Project completed its first full quarter of production on the Keeneland Full Scale System (KFS) at the end of March 2013. Usage has grown to ~200 users with quarterly allocations of 1.5M GPU-hrs (SUs). The Keeneland Initial Delivery System (KIDS) has been shifted from production to development, discretionary/start-up accounts, and supporting education, outreach, and training. The software on KIDS was upgraded and reconfigured to match KFS. Since KFS became an XSEDE production resource at the end of October 2012, the usage of the system has been dominated by the top 5-10 accounts, even as the number of accounts has grown. In particular the top 10 accounts has accounted for between 95-100% of the system utilization each month. Although it shows greater fluctuation, the top 5 accounts account for between 82- 90%. As more information becomes available with new users being added, we will watch how the additional accounts change the usage patterns on the system or whether former top 5 or top 10 accounts are just replaced with new large accounts. Over half of the Keeneland XSEDE production workload is being used for Molecular Biosciences, followed by Material Research and Chemistry. Much smaller amounts go for Astronomical Sciences, Computer and Computational Research, Cross-Disciplinary Activities, Mathematical Sciences, and Physics. About 33% of Keeneland use is for Earth Sciences, Chemical, Thermal Systems, Networking and Communications Research, Advanced Scientific Computing, and Information, Robotics, and Intelligent Systems.

139

NCSA Blue Waters Project The National Institute for Computational Sciences (NICS) staff has collaborated with the National Center for Supercomputing Applications (NCSA) Blue Waters project starting in January 2012 to facilitate and assist with the configuration and deployment of the Cray XE/XK system. This effort is focused on application support and system management.

NICS staff has been assigned to two PRAC support projects with NCSA staff members. Additionally, NICS has provided assistance with the installation, maintenance, and utilization of a software/library tracking solution. Following a CLE upgrade on the system, NICS assisted with application testing by verifying that two codes continued to function, perform, and produce correct results. Two project milestones were completed during the quarter composed of one from each of the application support and system management components.

Joint Institute for Computational Sciences The University of Tennessee (UT) and Oak Ridge National Laboratory (ORNL) established the Joint Institute for Computational Sciences (JICS) in 1991 to encourage and facilitate the use of high performance computing in the state of Tennessee. JICS advances scientific discovery and state-of-the-art engineering by taking full advantage of the petascale computers supported by DOE and NSF and housed at ORNL facilities by enhancing knowledge of computational modeling and simulation through educating a new generation of scientists and engineers well versed in the application of computational modeling and simulation to solving the world’s most challenging scientific and engineering problems. JICS is staffed by joint faculty who hold dual appointments as faculty members in departments at UT and as staff members in ORNL research groups. The institute also employs professional research staff, postdoctoral fellows and students, and administrative staff. The JICS facility represents an investment by the state of Tennessee and features a state-of-the art interactive distance learning center auditorium with seating for 66 people, conference rooms, informal and open meeting space, executive offices for distinguished scientists and directors, and incubator suites for students and visiting staff. The JICS facility is a hub of computational and engineering interactions. Joint faculty, postdocs, students, and research staff share the building, which is designed specifically to provide intellectual stimulation and interaction. The auditorium serves as the venue for invited lectures and seminars by representatives from academia, industry, and other laboratories, and the open lobby doubles as casual meeting space and functions as a site for informal presentations and poster sessions, including an annual 100+ student poster session.

12.7 SP-Specific Activities AACE

In 2011 the Joint Institute for Computational Sciences (JICS) established the Application Acceleration Center of Excellence (AACE) in partnership with NICS, industry leading vendors, and academic institutions. The center’s objectives are:  Accelerate NSF projects toward exascale with state-of-the-art heterogeneous architectures  Spur development of new algorithms and codes optimized for accelerator-based architectures  Disseminate fundamental knowledge

140  Facilitate effective exchange of expertise and cross-disciplinary collaboration

NICS is assisting NSF users with their transition to the Intel MIC architecture by researching parallelization techniques on the Intel MIC platform and by porting key NSF applications (already millions of lines of code) to the Intel MIC architecture in advance of its commercial release. NICS will also offer training on the Intel MIC architecture following its commercial debut.

"NICS has provided insight to Intel regarding the technology requirements of the scientific computing community," added Joe Curley, director of marketing for Intel's Technical Computing Group. "The impact of our partnership can be seen in the focus of our Intel MIC ‘Knights Ferry’ software development platform on extending well understood, high-level, standard programming languages and models."

NICS continues to engage a variety of technology providers to determine the role their technologies will play in the quest for exascale, while at the same time providing valued input for product development to meet the needs of the NSF research community.

Industrial Partnerships

Through the Cooperative Service Agreement with NSF, NICS is authorized to provide up to 5% of Kraken compute resources to industrial partners. This time on Kraken is made available at a cost such that commercial entities offering the same resources are not undercut.

NICS has established four of these industrial partnerships each with different requirements, support needs, and goals. A common consideration is that all of the industrial partners use the NICS resources because they do not have sufficient capability at their home site. We anticipate that each of the current partners will return to NICS and Kraken when they encounter other large computing projects. We are having active conversations with other large and small manufacturing companies, engineering service providers, and non-profit organizations who might be future partners.

12.8 Publications 12.8.1 User Publications Submitted 1. Araya G., Jansen K. and Castillo L., “DNS of turbulent thermal boundary layers subjected to adverse pressure gradients,” Physics of Fluids. 2. Lucci F., L’Vov V., Ferrante A., Rosso M. & Elghobashi S.,"Eulerian-Lagrangian bridge for the energy and dissipation spectra in isotropic turbulence" Theoretical and Computational Fluid Dynamics.

3. P. Ramaprabhu et al., The Rayleigh-Taylor instability driven by an accel-decel-accel profile, Phys. Rev E.

4. Sadowski, A., Narayan, R., Sironi, L., & Ozel, F. 2013, "Location of the bow shock ahead of cloud G2 at the Galactic Center," MNRAS.

141 5. Liu, Yi-Hsin, W. Daughton, H. Li, H. Karimabadi, V. Roytershteyn. “Bifurcated structure of the electron diffusion region in three-dimensional magnetic reconnection,” Phys. Rev. Lett.

6. Egedal, J., A. Le and W. Daughton. “A review of pressure anisotropy caused by electron trapping in collisionless plasma, and its implications for magnetic reconnection,” Phys. Plasmas.

7. Scudder, J. D., W. Daughton, H. Karimabadi, V. Roytershteyn. “Theoretical corollaries of electron demagnetization: a defensible framework for detection of the electron diffusion region of magnetic reconnection,” Phys. Plasmas.

Accepted 1. Reynolds, S. P., & Blondin, J. M. “Azimuthal Density Variations Around the Rim of Tycho's Supernova Remnant,” ApJ. 2. Maxime Viallet, Casey Meakin, David Arnett, Miroslav Mocak. "Turbulent convection in stellar interiors. III. Mean-field analysis and stratification effects," Astrophysical Journal.

3. Sadowski, A., Sironi, L., Abarca, D., Guo, X., Ozel, F., & Narayan, R. 2013, "Radio light curves during the passage of cloud G2 near Sgr A," MNRAS. 4. H. Karimabadi, V. Roytershteyn, and W. Daughton. “Evolution in Understanding Magnetic Reconnection and Its Connection With Turbulence,” Space Science Review. 5. Roytershteyn, V., S. Dorfman, W. Daughton, H. Ji, M. Yamada, and H. Karimabadi. “Electromagnetic instability of thin reconnection layers: Comparison of 3D simulations with MRX observations,” Phys. Plasmas.

6. Leonardis, E., S. C. Chapman, W. Daughton, V. Royterhsteyn, H. Karimabadi. “Identification of intermittent multi-fractal turbulence in fully kinetic simulations of magnetic reconnection,” Phys. Rev. Lett.

Published 1. Skory, S., Hallman, E., Burns, J. O., Skillman, S. W., O’Shea, B. W., Smith, B. D. “On the Road to More Realistic Galaxy Cluster Simulations: The Effects of Radiative Cooling and Thermal Feedback Prescriptions on the Observational Properties of Simulated Galaxy Clusters,” The Astrophysical Journal, Volume 763, Isssue 1, article id 38, 18 pp. DOI: 10.1088/0004-637X/763/1/38. 2. Skillman, Samuel W.; Xu, Hao; Hallman, Eric J.; O'Shea, Brian W.; Burns, Jack O.; Li, Hui; Collins, David C.; Norman, Michael L. “Cosmological Magnetohydrodynamic Simulations of Galaxy Cluster Radio Relics: Insights and Warnings for Observations,” The Astrophysical Journal, Volume 765, Issue 1, article id 21,16 pp. DOI: 10.1088/0004- 637X/765/1/21 3. P.S. Krstic, J.P. Allain, C. Taylor, J. Dadras, S. Maeda, K. Morokuma, J. Jakowski and A. Allouche, C. H. Skinner,“Deuterium uptake in magnetic-fusion devices with lithium- conditioned carbon walls", Phys. Rev. Lett. 110, 105001

142

4. Kasper Kristensen, Thomas Kjaergaard, Ida-Marie Hoyvik, Patrick Ettenhuber, Poul Jorgensen, Branislav Jansik, Simen Reine, Jacek Jakowski, “The Divide-Expand- Consolidate MP2 scheme goes massively parallel,” Molecular Physics [doi:10.1080/00268976.2013.783941] 5. Araya G., Castillo L. and Jansen K., “DNS of stable spatially-developing turbulent thermal boundary layers under weak stratification, in Turbulence V, Springer Proceedings in Physics. 6. M. Choi, A. Janotti, and C. G. Van de Walle. “Native point defects and dangling bonds in α-Al2O3,” J. Appl. Phys. 113, 044501 7. Warren, D. C. & Blondin, J. M. “Three-dimensional numerical investigations of the morphology of Type Ia SNRs,” Monthly Notices of the Royal Astronomical Society, 429, 3099. 8. M. Choi, J.L. Lyons, A. Janotti, and C.G. Van de Walle. “Impact of native defects in high-k dielectric oxides on GaN/oxide metal-oxide-semiconductor devices,” phys. stat. sol. (b), 250, 787–791. 9. Janotti, C. Franchini, J. B. Varley, G. Kresse, and C. G. Van de Walle. “Dual behavior of excess electrons in rutile TiO2A,” phys. stat. sol. (RRL) 7, 199. 10. Myers CG, Pettitt BM. “Communication: Origin of the contributions to DNA structure in phages,” J Chem Phys. 2013;138(7):071103. PMCID: 3592880. 11. Blondin, J. M. “Accretion Disks in Two-Dimensional Hoyle-Lyttleton Flow,” Astrophysical Journal, 767, 135. 12. Burkey, M. T., Reynolds, S. P., Borkowski, K. J., & Blondin, J. M. “X-Ray Emission from Strongly Asymmetric Circumstellar Material in the Remnant of Kepler's Supernova,” ApJ, 764, 63. 13. Ka-Ming Tam, H. Fotso, S.-X. Yang, Tae-Woo Lee, J.Moreno, J. Ramanujam, M. Jarrell. “Solving the Parquet Equations for the Hubbard Model beyond Weak Coupling,” Phys. Rev. E 87 013311. 14. Ka-Ming Tam, Mark Jarrell, and Juana Moreno. “Bose-Hubbard Model on a Kagome Lattice with Sextic Ring-Exchange Terms,” Phys. Rev. B 87, 075146. 15. Peng Zhang, Peter Reis, Ka-Ming Tam, Mark Jarrell, Juana Moreno, Fakher Assaad, Andy McMahan. “Periodic Anderson model with electron-phonon correlated conduction band,” Phys. Rev. B 87, 121102(R). 16. Vázquez, F.X., Unger, V.M., Voth, G.A. “Autoinhibition of Endophilin in Solution via Interdomain Interactions,” Biophys. J. 104, 396-403. 17. Cui, H, Mim, C, Lyman, E, Unger, V.M., Voth, G.A. “Understanding the Role of Amphipathic Helices in N-BAR Domain Driven Membrane Remodeling,” Biophys. J. 104, 404-411. 18. Dimonte, G. et al. “Ejecta source model based on the nonlinear Richtmyer-Meshkov instability,” J. Appl. Phys., 113, 024905. 19. Zhi Liang, William Evans, Tapan Desai and Pawel Keblinski. “Improvement of heat transfer efficiency at solid-gas interfaces by self-assembled monolayers,” Applied Physics Letters 102, 061907.

143 20. Zhi Liang, William Evans, and Pawel Keblinski, “Equilibrium and nonequilibrium molecular dynamics simulations of thermal conductance at solid-gas interfaces,” Physical Review E 87, 022119. 21. Yong Zhang, TingTing Zuo, YongQiang Cheng & Peter K. Liaw. “High-entropy Alloys with High Saturation Magnetization, Electrical Resistivity, and Malleability,” Scientific Reports 3, number 1455. 22. Perilla, Juan R., Leahy, Daniel J., Woolf, Thomas B. “Molecular dynamics simulation of transitions for ECD epidermal growth factor receptors show key differences between human and drosophila forms,” Proteins: Structure, Function, and Bioinformatics. doi: 10.1002/prot.24257. 23. Karimabadi, H., V. Roytershteyn, M. Wan, W. H. Matthaues, W. Daughton, P. Wu, M. Shay, B. Loring, J. Borovsky, E. Leonardis, S. Chapman, T.K.M. Nakamura. “Coherent structures, intermittent turbulence and dissipation in high-temperature plasmas,” Physics of Plasmas, 20, 012303; doi: 10.1063/1.4773205. 24. Le, A., J. Egedal, O. Ohia, W. Daughton, H. Karimabadi, V.S. Lukin, Regimes of the electron diffusion region in magnetic reconnection, Phys. Rev. Lett., 110, 135004. Nakamura, T.K.M., W. Daughton, H. Karimabadi, S. Eriksson, Three-dimensional dynamics of vortex-induced reconnection and comparison with THEMIS observations, submitted, J. Geophys. Res. 12.8.2 Staff Publications Submitted 1. Liu, Q., Logan, J., et al., “Hello ADIOS: The Challenges and Lessons of Developing Leadership Class I/O Frameworks,” Submitted to Concurrency and Computation: Practice and Experience, September 28, 2012. 2. Betro, V., Godo, M., Wyman, N. “Meshing, Visualization, and Computational Environments Technical Committee Year In Review,” AIAA Aerospace America, December 2012. 3. Bhanu Rekapalli, Kristin Wuichet, Gregory Peterson and Igor Zhulin. Dynamics of domain coverage of the protein sequence universe, BMC Genomics 2012, 13:642. 4. Bhanu Rekapalli, Paul Giblock and Christopher Reardon. PoPLAR: The Portal for Petascale Life Sciences Applications and Research. Under Review in BMC Bioinformatics.

Accepted

1. Fahey, M. R.; Crosby, L. D.; Rogers, G. L.; Hazlewood, V. G. “Kraken: The First Academic Petaflop Computer” In Contemporary High Performance Computing: From Petascale toward Exascale, Jeffrey S. Vetter Ed.; CRC Computational Science; Chapman & Hall: New York. 2. Kasper Kristensen, Thomas Kjaergaard, Ida-Marie Hoyvik, Patrick Ettenhuber, Poul Jorgensen, Branislav Jansik, Simen Reine, Jacek Jakowski, The Divide-Expand- Consolidate MP2 scheme goes massively parallel, Molecular Physics.

144 Published

1. P.S. Krstic, J.P. Allain, C. Taylor, J. Dadras, S. Maeda, K. Morokuma, J. Jakowski and A. Allouche, C. H. Skinner, “Deuterium uptake in magnetic-fusion devices with lithium- conditioned carbon walls", Phys. Rev. Lett. 110, 105001.

2. Kinter, J. L. et al., 2013: Revolutionizing Climate Modeling with Project Athena: A Multi-Institutional, International Collaboration. Bull. Amer. Meteor. Soc., 94, 231–245.

3. Skory, S., Hallman, E., Burns, J. O., Skillman, S. W., O’Shea, B. W., Smith, B. D. “On the Road to More Realistic Galaxy Cluster Simulations: The Effects of Radiative Cooling and Thermal Feedback Prescriptions on the Observational Properties of Simulated Galaxy Clusters,” The Astrophysical Journal, Volume 763, issue 1, 18pp.

4. Skillman, S. W., Xu, H., Hallman, E. J., O’Shea, B. W., Burns, J. O., Li, Hui., Collins, D. C., Norman, M. L. “Cosmological Magnetohydrodynamic Simulations of Galaxy Cluster Radio Relics: Insights and Warnings for Observations,” The Astrophysical Journal, Volume 765, issue 1, 16pp.

12.9 Metrics 12.9.1 Standard systems metrics The following subsections contain system metrics for NICS’ resources that are allocated through XSEDE: Kraken and Nautilus.

Note that job wait times and job expansion factors as reported by XDMoD are skewed by user specified job dependencies. NICS has implemented an “effective queue time” metric to eliminate the influence of job dependencies on these statistics. The effective queue time is a measure of the wait time incurred only once a job is eligible to run and is not a factor of individual workflows. In the future job wait times and expansion factors will be reported based on effective queue times.

Another issue with wait time and expansion factor by job size, as currently reported, is that the job size bins overlap multiple scheduling queues at NICS, and thereby, overlap multiple scheduling policies. This too will be corrected in future reporting. Also note that the error bars associated with the mean values in Figure 3, Figure 4, Figure 13 and Figure 14 represent the standard deviation of the sampled mean which is the standard deviation divided by the square root of N, where N is the sample size.

145 NICS-KRAKEN Quarterly Report

Total NUs Charged by Resource

Service Provider = U Tennessee

2013-01-01 to 2013-03-31

12.9.2 Kraken

Total NUs Charged by Job Size

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Figure 1: Daily resource consumption in mega-normalized units (1e3) charged on Kraken in Q1 2013.

Figure 2: Total resource consumption in giga-normalized units (1e9) by job size for Kraken in Q1 2013.

146 Avg Wall Hours Per Job by Job Size

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Avg Wait Hours Per Job by Job Size

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Figure 3: Average wall hours by job size on Kraken in Q1 2013.

Figure 4: Average wait hours by job size on Kraken in Q1 2013.

147 User Expansion Factor by Job Size

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Total NUs Charged by Job Wall Time

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Figure 5: Expansion factor by job size for Kraken in Q1 2013.

Figure 6: Total resource consumption in giga-normalized units (1e9) by wall time for Kraken in Q1 2013.

148 User Expansion Factor by Job Wall Time

Resource = NICS-KRAKEN -- Service Provider = U Tennessee

2013-01-01 to 2013-03-31

Total NUs Charged by Field of Science

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Figure 7: Expansion factor by wall time for Kraken in Q1 2013.

Figure 8: Resource consumption in mega-normalized units (1e6) by field of science for Kraken in Q1 2013.

149 Total NUs Charged by User Institution

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Total NUs Charged by PI

Resource = NICS-KRAKEN

2013-01-01 to 2013-03-31

Figure 9: Resource consumption in mega-normalized units (1e6) by institution in mega-normalized units for Kraken in Q1 2013.

Figure 10: Resource consumption by PI in mega-normalized units for Kraken in Q1 2013.

150 NICS-NAUTILUS Quarterly Report

Total NUs Charged by Resource

Service Provider = U Tennessee

2013-01-01 to 2013-03-31

12.9.2.1 Nautilus

Total NUs Charged by Job Size

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Figure 11: Daily resource consumption in kilo-normalized units (1e3) charged on Nautilus in Q1 2013.

Figure 12: Total resource consumption in mega-normalized units (1e6) by job size for Nautilus in Q1 2013.

151 Avg Wall Hours Per Job by Job Size

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Avg Wait Hours Per Job by Job Size

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Figure 13: Average wall time in hours by job size on Nautilus in Q1 2013.

Figure 14: Average wait time in hours by job size for Nautilus in Q1 2013.

152 User Expansion Factor by Job Size

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Total NUs Charged by Job Wall Time

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Figure 15: Expansion factor by job size for Nautilus in Q1 2013.

Figure 16: Total resource consumption in mega-normalized units (1e6) by wall time for Nautilus in Q1 2013.

153 User Expansion Factor by Job Wall Time

Resource = NICS-NAUTILUS -- Service Provider = U Tennessee

2013-01-01 to 2013-03-31

Total NUs Charged by Field of Science

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Figure 17: Expansion factor by wall time for Nautilus in Q1 2013.

Figure 18. Total NUs charged by field of science Q1 2013.

154 Total NUs Charged by User Institution

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Total NUs Charged by PI

Resource = NICS-NAUTILUS

2013-01-01 to 2013-03-31

Figure 19: Resource consumption by institute in mega-normalized units for Nautilus in Q1 2013.

Figure 20: Resource consumption by PI in kilo-normalized units for Nautilus in Q1 2013.

155 12.9.3 Standard User Assistance Metrics NICS’ front line user support responded to 823 new XSEDE tickets in the first quarter (Figure 22). Open tickets experienced an avg. median resolution time of 20.9 hours for the quarter (Figure 22). These tickets corresponded to a variety of issues (Figure 23) with the majority falling into two groups: login/access issues and jobs/batch queues.

Figure 21: XSEDE tickets opened/closed by month.

Figure 22. Ticket volume and resolution time.

156

Figure 23: Number of tickets issued by category.

12.9.4 SP-specific Metrics NICS’ resources provided roughly 56% of computational cycles that were delivered to the NSF community in this quarter (Figure 29), and NSF charges account for most of the total charges on these resources (Figure 24 and Figure 25).

1% 0% 1% 0% Kraken NUs

XSEDE - 9.5 Billion TENNESSEE - 57.8 Million NATIONAL - 75.8 Million EDUCATION - 2.5 Million INDUSTRIAL - 54.7 Million

98%

Figure 24: XSEDE charges as a percentage of total Figure 25: XSEDE charges as a percentage of total charges on Nautilus in Q1 2013. charges on Kraken in Q1 2013.

157

Figure 26: Monthly utilization for Kraken through Q1 2013.

14000 13,164 12,716 12,969 12,254 12,058 12,976 12000 11,800 11,021 11,530 10,672 10,222 9,806 10000 8,708 8,737 8,412 8,557 8,605 8,769 8,804 7,769 8000 7,339 7,614 8,355 6,493 # of Users 6000 # of Files

4000 Total TB stored

2000 536 574 559 490 497 510 521 546 555 559 567 582

Oct-12 Jan-13

Dec-12

Feb-13

Nov-12

Aug-12

July-12

Mar-13

Sept-12

May-12 June-12 April-12 Figure 27: Archival storage usage on HPSS through Q1 2013.

158 4500 4025 4113 3969 4140 4074 3871 3913 3870 4000 3691 3745 3469 3600 3500 3000 # of Users 2500 2000 # of Projects 1500 727 767 716 1000 714 718 671 732 668 745 702 741 715 500

Oct-12

Jan-13

Dec-12 Feb-13

Apr-12 Nov-12

Aug-12

July-12

Sept-12 Mar-13

May-12 June-12

Figure 28: Number of active users and projects on NICS resources through Q1 2013.

8% NICS TACC SDSC 56% 33% PURDUE PSC OSG

Figure 29: CPU hours delivered by NICS resources as a percentage of total CPU hours delivered by XSEDE resources in Q1 2013.

159 13 Pittsburgh Supercomputing Center - Service Provider Quarterly Report 13.1 Executive Summary The Pittsburgh Supercomputing Center operates Blacklight, a powerful and unique resource for the national research community. Blacklight, an SGI Altix UV 1000 acquired with the assistance of an NSF grant and operated as an XSEDE resource, is the world’s largest shared-memory system, providing two partitions of 16TB each. For large-scale, rapid graph analytics as well as support for heterogeneous applications, PSC operates Sherlock, a YarcData uRiKA™ (Universal RDF Integration Knowledge Appliance) data appliance with PSC enhancements. Sherlock, which was funded via NSF’s STCI program, contains 32 YarcData Graph Analytics Platform nodes, each containing 2 Threadstorm 4.0 processors, a SeaStar 2 interconnect ASIC, and 32 GB of RAM plus additional Cray XT5 nodes having standard x86 processors. With operational funding from NIH, PSC also operates Anton, a special-purpose computer for molecular dynamics which is used by many NSF-supported researchers. PSC systems are supported by a central file system Brashear (except for Sherlock), and extensive LAN, MAN and WAN infrastructure. For persistent storage such as archiving files, hosting data collections, etc., PSC operates Data Supercell, a scalable, disk-only file repository that provides fast access to files. Its initial deployment has four petabytes. PSC resources enabled significant progress in genomics and in recognition and indexing of hand- written documents. Users in those application areas found Blacklight’s operating characteristics, particularly the RAM disk, to be very valuable. In the genomics application, PSC staff devised a workflow which allowed the application to take advantage of both the local filesystem and RAM disk and made other optimizations that led to a four-fold speedup. In the hand-written documents application, Blacklight’s large shared memory proved to be helpful for the three-stage calculation in several ways. First, Blacklight’s large amount of memory per core relative to other systems avoids the additional cost of an idle core which running the first- and second-stage calculations on the other systems would entail. Second, the third stage requires 50 GB of memory per process, which is only provided by a shared-memory system such as Backlight. Third, Blacklight also speeds one aspect of the third-stage computation by virtue of very fast I/O, enabled by utilizing memory as a RAM disk. MATLAB on Blacklight via MDCS is bringing new user communities into high performance computing. For instance, PSC staff members assisted in porting a PC-based MATLAB framework for mathematical psychology research on human decision making to Blacklight, and the research team reports that they computed more analyses in two months on Blacklight than all PC-based analyses carried out in their lab over the past 3-4 years combined. PSC people make significant intellectual contributions and continue to earn high praise for their efforts. For example, PSC’s Phil Blood helped economist Kenneth Judd to become an XSEDE user, which led to Judd becoming a Blue Waters user. Judd wrote to Blood, “I got a 20,000,000- hours allocation on Blue Waters. Our experience on XSEDE was a big help in getting up to speed on big machines and was surely a positive factor in getting that award. Thank-you XSEDE!!!!” Ralph Roskies, a PI for XSEDE, said, “That’s the way it’s supposed to work!” PSC’s David O’Neal helped to improve the performance of FRED (Framework for Reconstructing Epidemic Dynamics), an open source agent-based modeling system. For the most demanding runs, service unit charges fell by a factor of 60. These changes have significantly enhanced the work that can be done with FRED because they enable larger, more relevant simulations.

160 In collaborations with non-XSEDE partners, PSC’s Jim Marsteller is a leader in the Center for Trustworthy Scientific Cyberinfrastructure, and Josephine Palencia has made significant contributions to the ExTENCI project. PSC engaged in a range of Training, Education and Outreach activities, which included middle school, high school and undergraduate STEM programs and HPC training workshops. Specifically, PSC has continued its leadership in training on programming and support for accelerators. 13.1.1 Resource Description Computing and Storage: PSC provides a range of computing and storage platforms for the national science community. For applications requiring very large shared memory, high-productivity programming models, and/or moderate parallelism with a high-performance system-wide interconnect, PSC operates Blacklight, an SGI UV 1000 cc-NUMA shared-memory system comprising 256 blades. Each blade shares 128GB of local memory, and holds two Intel Xeon X7560 (Nehalem) eight-core processors, for a total of 4,096 cores and 32 TB across the whole system. Each core has a clock rate of 2.27 GHz, supports two hardware threads and can perform 9 Gflop/s for a total system floating point capability of 37 Tflop/s. Up to 16 TB of this memory is accessible as a single memory space to a shared-memory program. Message-passing and PGAS programs can access all 32 TB on the system. Blacklight is part of the National Science Foundation XSEDE integrated national system of cyberinfrastructure. Additionally, PSC has an SGI Altix 4700 system called Salk which is also targeted at applications requiring large shared memory, high-productivity programming models, or moderate parallelism with a high-performance, system-wide interconnect. Salk is administered for the NIH-funded National Resource for Biomedical Supercomputing (NRBSC) and offers 144 Montvale processors providing a peak aggregate speed of 0.96 Tflop/s with 288 GB shared memory. This system supports advanced programming languages and models including UPC. PSC operates Sherlock, a YarcData uRiKA™ (Universal RDF Integration Knowledge Appliance) data appliance with PSC enhancements. An experimental system, Sherlock enables large-scale, rapid graph analytics through massive multithreading, a shared address space, sophisticated memory optimizations, a productive user environment, and support for heterogeneous applications. Sherlock contains 32 YarcData Graph Analytics Platform nodes, each containing 2 Threadstorm 4.0 (TS4) processors, a SeaStar 2 (SS2) interconnect ASIC, and 32 GB of RAM. Aggregate shared memory is 1 TB, which can accommodate a graph of approximately 10 billion edges. The TS4 processors and SS2 interconnect contain complementary hardware advances specifically for working with graph data. PSC has customized Sherlock with additional Cray XT5 nodes having standard x86 processors to add valuable support for heterogeneous applications that use the Threadstorm nodes as graph accelerators. Other x86 nodes serve login, filesystem, database, and system management functions. PSC operates an Anton special-purpose supercomputer for molecular dynamics (MD) simulation that performs up to 100 times faster than conventional supercomputers. Designed by D. E. Shaw Research (DESRES) and provided to PSC without cost by DESRES, it is available for non- commercial research use by universities and other non-profit institutions. This machine, the only Anton computer operated outside DESRES, is hosted by PSC and is available to the national biomedical community with funding from NIH’s National Institute of General Medical Sciences. Computing time on Anton is allocated by a peer-review committee convened by the National Research Council. A large number of Anton users are NSF-supported investigators. The Anton computer is supplemented by a high performance file storage system for simulation trajectories

161 and an analysis cluster (Kollman). Each of the four nodes in the analysis cluster consists of two Intel Westmere six-core processors and 96 GB of memory. The high-performance file storage system consists of a 500-TB Lustre file system. The file system and the analysis cluster nodes are interconnected over Quad Data Rate (QDR) InfiniBand. Availability of the Anton system has been extended until September 2014. PSC operates several Linux clusters for scientific research as well as several high-end servers and powerful workstations for development, analysis, and visualization tasks. The production workload on all of the PSC computing platforms is managed by PBS/Torque. Several scheduler policy modules used include a locally-developed module, Simon, and the Maui scheduler. All of the PSC computing platforms except Sherlock have access to Brashear, PSC’s shared, central file system using the Lustre file system architecture. It comprises eight storage nodes and 350 TB of direct-attached disks, forming a large I/O cluster globally accessible within the PSC site. Access to the file system is provided by InfiniBand, 10-Gigabit Ethernet and 1-Gigabit Ethernet. Each node in the I/O cluster is a Lustre Object Storage Server (OSS) hosting multiple Object Storage Targets (OSTs). PSC’s Data Supercell for persistent storage of information is a disk-only file repository that is less costly than a disk-tape archive system and provides much faster file access. Each building block in the repository has one petabyte of useable disk storage, which is managed by the ZFS file system and the PSC-developed SLASH2 replicating distributed file system. ZFS and SLASH2 provide multiple layers of robust data integrity checking to protect user data against data corruption. This building-block architecture will enable the repository to scale well beyond its initial deployment of four petabytes. Users can access the repository from within PSC using the familiar PSC file archiving utility, far. From outside PSC, users can employ a variety of well-known file transfer methods such as SCP and GridFTP. These transfers are handled by a series of dedicated data transfer servers. Networking: PSC network facilities consist of production and research Local Area Network (LAN), Metropolitan Area Network (MAN), and Wide Area Network (WAN) infrastructures. Local Area Network Infrastructure - The LAN infrastructure consists of switched Ethernet with speeds up to 10 Gb/s. The LAN architecture was constructed to overcome issues of buffer contention in data center Ethernet switches on the Science DMZ1. This allows for higher bandwidth data transfers to the data transfer nodes. 3 Rivers Optical Exchange: PSC operates and manages the 3 Rivers Optical Exchange (3ROX), a regional network aggregation point that provides high-speed commodity and research network access, primarily to sites in Western and Central Pennsylvania and West Virginia. While the primary focus of 3ROX is to provide cost-effective, high-capacity, state-of-the-art network connectivity to the university community, this infrastructure also provides well-defined network services to both community (K-12, government) and commercial entities in Western Pennsylvania. University member sites currently include Carnegie Mellon University, the Pennsylvania State University, the Pittsburgh Supercomputing Center, the University of Pittsburgh, WVNET (West Virginia’s state-wide research and education network), and West Virginia University.

1 From http://fasterdata.es.net/science-dmz/: The Science DMZ is a portion of the network, built at or near the campus or laboratory’s local network perimeter that is designed such that the equipment, configuration, and security policies are optimized for high-performance scientific applications rather than for general-purpose business systems or “enterprise” computing.

162 3ROX Metropolitan Area Network Infrastructure - 3ROX MAN infrastructure is DWDM- based and supports multiple 10-Gigabit Ethernet waves. It is capable of supporting 40- and 100-Gigabit waves as the need arises. This DWDM network connects four different locations around Pittsburgh that include long haul service providers, a co-location hotel, a campus based co-location facility, and PSC’s Northern Pike machine room. 3ROX Wide Area Network Infrastructure - 3ROX WAN infrastructure has both Commodity Internet and Research and Education components. Explicit routing is used to maintain the acceptable use policies associated with the various production and research network infrastructures. The 3ROX Commodity Internet component consists of multiple high-performance WAN connections to major Internet service providers, including a Gigabit Ethernet connection to Cogent and a 10-Gigabit Ethernet connection to Level 3. In addition, 3ROX provides connectivity to both regional and national Internet2 and content peering infrastructures, in particular access to the Internet2 based TR/CPS content peering services; regional peering with Southern Cross Roads (SOX), OARnet and Comcast; along with a recent direct peering connection with Google. The 3ROX Research and Education component includes a 10-Gigabit Ethernet connection, with 5 Gb/s of bandwidth, to the Internet2 network. In addition to the Internet2 connection, 3ROX also has a 10-Gigabit Ethernet connection to National LambdaRail; a 10-Gigabit Ethernet connection to the XSEDE backbone network; a 10-Gigabit Ethernet connection between PSC’s offices and its remote supercomputing machine room at 4350 Northern Pike; and a 10-Gigabit Ethernet connection to Penn State University (PSU) to provide XSEDE connectivity to PSU. 13.2 Science Highlights In addition to major science accomplishments that are highlighted in the XSEDE report, we present selected others specific to PSC. 13.2.1 Recognition and Indexing of Hand-Written Documents: Digitizing the 1940 Census (Kenton McHenry, National Center for Supercomputing Applications) Meaningful digitization of hand-written records and other historical documents is a computational challenge. Interpreting and indexing cursive text is nontrivial; searching such collections using simple, linear indices involves time lags that the average Web user would not find acceptable. Kenton McHenry and colleagues at NCSA have used the 1940s Census as a test case for better methods. The 1940 Census data consist of 4,646 reels containing about 5 million forms, each of which Figure Digitized hand-written census form with selected cells highlighted. The includes 50 rows and 44 problem is to (1) align forms and cut out cells, (2) extract descriptive features for each cell, and (3) create indices for search queries. columns, for a total of 11

163 billion cells. The processing of the forms can be split into three distinct stages: (1) aligning the form and cutting out the cells, (2) extracting descriptive features for each cell, and (3) creating indices for the entire collection of cells. For each cell, a feature vector is calculated to serve as a description of the cell’s content. The group extracts a 30-dimensional feature vector for each cell, and builds two types of indices to later search the collection’s data. The first, a simple binary tree, requires during its computation around 50 GB of memory to hold all feature vectors and the corresponding distance matrix for a single column and single reel. The second index, a modified version of a Lucene textual index, is built through an I/O intensive process, requiring a total of 8 GB of memory equally divided between RAM disk and computational needs. While the processing of the data is highly parallel, each of the stages required at least 4 GB of memory per process. Because many HPC systems have more limited amounts of memory available per core, PSC Blacklight’s large shared memory proved to be helpful for this calculation, avoiding the additional cost of an idle core that running stage 1 and 2 on many systems would entail. The final stage requires 50 GB of memory per process when building the binary tree, which is only provided by a shared-memory system such as Backlight; the machine also speeds the computation of the Lucene index by virtue of very fast I/O, enabled by utilizing memory as a RAM disk. 13.3 User-facing Activities 13.3.1 System Activities Brocade 100-GbE OpenFlow-capable Switch/Router: A Brocade 100-GbE OpenFlow-capable switch/router has been delivered and is in the process of being deployed (with a target of mid-to- late May). Final specifications are being completed for the purchase of 100-GbE line cards for 3ROX DWDM infrastructure to support an upgrade of PSC’s Internet2 connection to 100-GbE (planned for the third quarter of 2013). This upgraded connection will enable OpenFlow technologies, a form of software defined networking, allowing 3ROX and its members to begin experimenting with new networking protocols and services. 10-GbE connection to NOAA: A second 10-GbE connection to the NOAA supercomputing facility in Fairmont, West Virginia was placed in production in late February 2013. This circuit is provisioned through collaboration between 3ROX, WVnet, OARnet and Lumos, and connects the Fairmont, WV facility to the Internet2 PoP in Cleveland OH and provides failover for the primary circuit between Fairmont and Pittsburgh. 3ROX/Drexel/MAPGI Meeting with Internet2: Representatives from 3ROX, MAGPI, Internet2 and the CIO’s or their designees from the research universities in Pennsylvania, Delaware, New Jersey and West Virginia met in Philadelphia on January 30, 2013. Internet2 provided updates on, and solicited input about, the proposed revisions to their connection and participation fee model that would shift some fees from the connector networks to the universities. 13.3.2 Services Activities Genomics: PSC’s Phil Blood is a co-author with a group led by PI Christopher Mason (Weill Medical College of Cornell University) of a paper2 describing one of the largest transcriptome assembly projects ever attempted, which has laid the foundation for a new reference resource for comparative functional genomics. In order to accomplish these assemblies on Blacklight using

2 Pipes L, Li S, Bozinoski M, Palermo R, Peng X, Blood P, Kelly S, Weiss JM, Thierry-Mieg J, Thierry- Mieg D, Zumbo P, Chen R, Schroth GP, Mason CE, Katze MG, “The non-human primate reference transcriptome resource (NHPRTR) for comparative functional genomics”, Nucleic Acids Res. 2013 Jan 1; 41(D1):D906-D914, see http://www.ncbi.nlm.nih.gov/pubmed/23203872 and http://nhprtr.org/

164 the Trinity code, Blood debugged a problem with failing jobs, devising a workflow which allowed the team to take advantage of both the local filesystem and RAM disk. Blood also benchmarked the final stages of Trinity, determining the optimal number of threads to use. These changes resulted in the jobs running successfully and sped up the final stages of Trinity by ~4x. Blood continues to optimize these Trinity jobs on Blacklight. Bringing New Communities of Users to HPC-Economics: Phil Blood was also instrumental in getting economist Kenneth Judd, the Paul H. Bauer Senior Fellow at the Hoover Institution at Stanford University, up and running in XSEDE. Among Blood’s contributions is speaking about XSEDE at several sessions of Judd’s annual Initiative for Computational Economics. On March 13, 2013 Judd wrote to Blood, “Yesterday I got a 20,000,000-hours allocation on Blue Waters. Our experience on XSEDE was a big help in getting up to speed on big machines and was surely a positive factor in getting that award. Thank-you XSEDE!!!!” In response, Ralph Roskies, PSC Scientific Co-Director and a PI on the XSEDE project, said, “That’s the way it’s supposed to work!” Bringing New Communities of Users to HPC-Mathematical Psychology: Michel Regenwetter and his team at have developed a very general mathematical modeling and statistical testing framework implemented in MATLAB for PC for their mathematical psychology research on human decision making. The MATLAB framework they developed uses specialized statistical software based on “order-constrained statistical inference” algorithms currently still unavailable in commercial packages. With guidance from PSC’s Roberto Gomez and Anirban Jana, they ported their framework to Blacklight via MATLAB Distributed Computing Server under a startup allocation. Having an adequate understanding of decision making and how to correct potential flaws and biases in decision making bears immeasurable value to society. The Regenwetter group looks specifically at decisions under risk and decisions with trade-offs over time, both of which are at the core of much human activity. The medium term goal of the Regenwetter laboratory is to increase their scientific throughput by at least two orders of magnitude through massive parallelization in modeling and data analysis. In the past, contemporary research on risky or intertemporal choice was limited to considering only few theories, few mathematical formulations, few probability models and few statistical methods due to limitations in mathematical and statistical computing resources. Even testing a few variants of one theory on a few dozen laboratory data sets can take weeks of computation on a PC. High performance computing, however, enables full-fledged quantitative analyses of many more theories with fewer simplifying convenience assumptions on many more laboratory data sets from more human decision makers. During their start-up allocation, Regenwetter’s team computed more analyses in two months on Blacklight than all PC-based analyses carried out in their lab over the past 3-4 years combined. They say, “The likely impact of this endeavor is an unprecedented increase in the scale and rigor of decision making research, both in the development of new theories and in the empirical testing of these theories.” The Regenwetter team has carried out partial analyses of two recent NSF funded human decision making experiments, one in risky choice, and one in intertemporal choice. The analysis of the risky choice experiment has already yielded on the order of ten times as many results as the group typically publishes in one publication. For the intertemporal study, the startup allocation generated on the order of 100 times as many results as a typical journal article in that field. Code Performance: Working on an ECSS project, PSC’s David O’Neal and Jay DePasse of the University of Pittsburgh improved the performance of FRED (Framework for Reconstructing Epidemic Dynamics), which is an open source agent-based modeling system that uses detailed synthetic populations derived from census data to capture geographic and demographic

165 distributions. The original serial implementation required approximately 96 hours and 540 GB of memory to complete a 100-day US simulation. The current version completes the same analysis in 3-4 hours using less than 200 GB of memory. For the most demanding runs, service unit charges fell by a factor of 60. These changes have significantly enhanced the work that can be done with FRED, as it not only reduced the time to solution, but also enabled larger, more relevant simulations, such as influenza spread throughout the entire US. Anton: On March 18, 2013 PSC’s National Resource for Biomedical Supercomputing issued a new Request for Proposals for simulation time on Anton. See http://www.nrbsc.org/anton_rfp/. Submission deadline for proposals is May 28, 2013. Proposal review will again be conducted by the National Research Council at the National Academies of Science. Allocations are expected to commence in October 2013. 13.4 Security PSC had no security incidents. 13.5 Education, Outreach, and Training Activities Type Title Location Date(s) Hours Number of Number Method Participants of Under- represen ted people Worksh OpenACC Workshop PSC 15-16 10 104 NA Live and op Pittsburgh JAN Webcast PA 2013 Worksh XSEDE New User Training PSC 8 FEB 1.5 60 NA Webcast op Pittsburgh 2013 PA BoF Web10G TIP 2013 16 JAN 1.5 9 NA Live Conf. 2013 Honolulu, HI Present SLASH2 and Data MIT 20 FEB .2 ~75 NA Live ation Supercell Enterprise 2013 Forum Pittsburgh PA Worksh Shawn Brown, Workshop International 11-28 21 62 NA Live op on Computer Programming Centre for MAR and Advanced Tools for Theoretical 2013 Scientific Research Work & Physics Quantum ESPRESSO Trieste, Italy Developer Training User Pittsburgh Hadoop User PSC 20 MAR 2 19 NA Live Group Group Meeting Pittsburgh 2013 PA Confer Yang Wang, “A APS 21 MAR .2 NA NA Live ence computational study of high Meeting 2013 Talk entropy alloys” Baltimore MD

166 Type Title Location Date(s) Hours Number of Number Method Participants of Under- represen ted people Poster Yang Wang, “An APS 18-22 NA NA NA Poster investigation of the internal Meeting MAR sum convergence in the full Baltimore 2013 potential multiple scattering MD theory”

Regarding the webcast OpenACC workshop on January 15-16, there were 104 people who attended the workshop across 8 sites. PSC’s John Urbanic delivered his usual excellent lectures and Mark Ebersole from NVIDIA was extremely helpful as well. The workshop ran from 11AM- 5PM each day, with a 1-hour lunch break, providing 10 hours of lecture with hands-on exercises. PSC’s Steve Cunningham provided video conferencing support for the workshop which was held in PSC’s Deerfield Training Center. The videoconferencing for this workshop was provided in collaboration with MCNC, replacing Internet2 as the videoconferencing provider. MCNC immediately addressed any issues that arose during the workshop. We are currently investigating the cost of acquiring additional HD323 equipment for PSC so that we can continue to provide production support for videoconferencing-based distributed workshops. Here are a few quotes from the most recent set of attendees’ evaluations:  Very professionally done from registration throughout.  Excellent speaker, Thanks John!  The workshop was very well organized. In particular, I appreciated the hands-on examples since they made clear how easy it is to start using the material covered in the workshop.  John Urbanic has done an outstanding job!  It was a really good overview. It looks like OpenACC will be really easy to set up/demo to convince my adviser that we should use it. Course on BigData and Sustainability: PSC’s Bryon Gill, Nick Nystrom, and J. Ray Scott are assisting with “BigData and Sustainability,” a course in the CMU Information Systems Program. Gill and Scott are helping design a hands-on experience for the students using a Hadoop cluster that Gill has built. Nystrom will give a lecture late in the course on BigData analytics. We will work together to bring more hands-on data analytics into the class. Pittsburgh Middle School & High School Computer Fairs: On March 19-20, 2013, PSC’s Bryon Gill and Derek Simmel served as judges at the Pittsburgh Middle School & High School Computer Fairs. The fairs are hosted at the Carnegie Science Center by the Allegheny Intermediate Unit. Gill and Simmel were assigned to judge projects in the Programming, Web Design, and (for High School) Animation categories. For 2013, programming projects were required to develop an app for a mobile device (IOS or Android). Examples included math practice exercise apps, an online social networking service, and an academic “daytimer” and grades-tracking app. Website development projects included a videogame review site, a Civil War retrospective site, an informational/events announcement site about a church youth group, and a tablet-browser-optimized site exploring the intersection between cyberspace and innovation. The high-school animation entry was a well-crafted, animated movie trailer for a post-apocalyptic ninja/samurai action movie. Gill said, “I was surprised how well middle school students were able to do with mobile app development.”

167 13.6 SP Collaborations Center for Trustworthy Scientific Cyberinfrastructure (CTSC): CTSC is a joint project of people from Indiana University, the National Center for Supercomputing Applications, the University of Wisconsin-Madison, the University of Wisconsin-Milwaukee, and the Pittsburgh Supercomputing Center. CTSC has been quickly ramping up. Jim Marsteller of PSC is leading a cybersecurity planning activity for the IceCube and CyberGIS projects. An engagement plan was written by Marsteller and sent to representatives of the two partnering projects to review. The goal is to clearly outline the project plan and establish time commitments by CTSC staff as well as people from the respective partners. CTSC requires an equal amount of participation from each partner in order to enhance their cybersecurity skills. Jim Marsteller and other CTSC team members visited the Long Term Ecological Research (LTER) network office to conduct a risk assessment and assist with developing a cybersecurity plan. LTER provides scientific expertise, research platforms, and long-term datasets that are necessary to document and analyze environmental change. Over the course of four days, the team conducted a series of activities to identify assets and assign value, and to document threats and their impact on the LTER network office. Draft documents including risk assessment findings and initial cybersecurity plan are nearing completion. CTSC is submitting a proposal to NSF to restart NSF’s annual Cybersecurity Summit. Jim Marsteller, Von Welch (IU) and Craig Jackson (IU) co-authored the proposal. The summit is scheduled for September 30 - October 2, 2013. Marsteller and Welch have started selecting potential individuals for the program committee. Extending Science Through Enhanced National CyberInfrastructure (ExTENCI): PSC’s Josephine Palencia has completed work on speeding up the parallel processing of Higgs reconstruction on the Florida International University (FIU) compute cluster. Under Palencia’s ExTENCI production setup, the local Condor job scheduler at FIU launched and completed 192 Higgs reconstructions in parallel (processing ~100GB of Monte Carlo data stored separately in 32 input files, each repeatedly used by 6 processes) in ~2 hours. In contrast, before the deployment of Palencia’s ExTENCI production setup, it took ~15 minutes to complete each individual run of Higgs reconstruction for a total of ~48 hours. Thus, the time to completion of the total task decreased from ~48 hours to ~2 hours. 13.7 SP-Specific Activities The following tutorials with participation by PSC staff members have been accepted for presentation at XSEDE13:  Sergiu Sanielevici, Raymond Norris (MathWorks) and Anirban Jana, Parallel Computing with MATLAB and Scaling to XSEDE with Blacklight  Marcela Madrid, Ken Hackworth and Jim Marsteller, XSEDE New User Tutorial  John Urbanic, Accelerator Programming Using OpenACC  Kathy Benninger and Jason Zurawski (Internet2), Network Performance Tutorial featuring perfSONAR 13.8 Publications Datta R, Brown S, Lee BY, Huang SS, “Dissemination of multidrug-resistant organisms (MDROs) across statewide health facilities”, IDWeek (1st Annual Joint Meeting of IDSA, SHEA, HIVMA, and PIDS), October 17-21, 2012 (San Diego, CA).

168 Datta R, Brown S, Lee BY, Huang SS, “Quantifying the spread of methicillin-resistant Staphylococcus aureus (MRSA) across California healthcare facilities”, IDWeek (1st Annual Joint Meeting of IDSA, SHEA, HIVMA, and PIDS), October 17-21, 2012 (San Diego, CA). Pipes L, Li S, Bozinoski M, Palermo R, Peng X, Blood P, Kelly S, Weiss JM, Thierry-Mieg J, Thierry-Mieg D, Zumbo P, Chen R, Schroth GP, Mason CE, Katze MG, “The non-human primate reference transcriptome resource (NHPRTR) for comparative functional genomics”, Nucleic Acids Res. 2013 Jan 1; 41(D1):D906-D914; see also http://www.ncbi.nlm.nih.gov/pubmed/23203872 and http://nhprtr.org/. 13.9 Metrics 13.9.1 Standard User Assistance Metrics Numbers in Table 1 refer to tickets handled by the PSC help desk in PSC’s local ticket system.

Table 1. Distribution of times to resolution for the 387 tickets that were created as well as resolved between 1/1/2013 and 3/31/2013 Ticket Account File Jobs/batch Login/access Software System Type issues systems queue issues issues /apps issues Other 0-1 hr 1 9 5 0 2 0 7 1-24 hrs 15 8 56 0 15 8 42 1-7 days 36 19 31 0 20 30 31 1-2 wks 2 4 2 0 14 1 6 >2 wks 3 3 0 0 17 0 0

Numbers in Table 2 refer to tickets relating to PSC that were handled in the central XSEDE Ticket System.

Table 2: Distribution of times to resolution Time to account file grid jobs/batch login/access mss/data network software system other Resolution issues systems software queues issues issues issues /apps issues 0-1 hr 1-24 hr 1 1-7 d 3 1 2 1 1-2 wk 3 3 4 2 4 3 > 2 wk 2 1 3 5 3 4 8 Still 1 Open

13.9.2 SP-specific Metrics Key system statistics for Blacklight for 1/1/2013 to 3/31/2013 are shown in Table 3. Table 3: Operational Statistics - Blacklight Number of unplanned outages 6 Number of planned outages 9 Total outages 15 Number of job failures due to system faults 87

169 Table 3: Operational Statistics - Blacklight Total time* in period (hours) 4320.00 100.00% Scheduled Downtime (hours) 30.50 0.71% Unscheduled Downtime (hours) 71.00 1.64% Total Downtime (hours) 101.50 2.35% Total time available to users (total-downtime) 4218.50 97.65% % System Utilization 74.16% * On Blacklight a node is half the machine. Time values listed are expressed in node hours. 13.9.3 Standard systems metrics The charts of standard system metrics for Blacklight on the following five pages were provided by the XDMod team:

170 Total NUs Charged by Resource

Service Provider = PSC

2013-01-01 to 2013-03-31

Total NUs Charged by Job Size

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

171

Avg Wall Hours Per Job by Job Size

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

Avg Wait Hours Per Job by Job Size

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

172

User Expansion Factor by Job Size

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

Total NUs Charged by Job Wall Time

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

173

User Expansion Factor by Job Wall Time

Resource = PSC-BLACKLIGHT -- Service Provider = PSC

2013-01-01 to 2013-03-31

Total NUs Charged by Field of Science

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

174

Total NUs Charged by User Institution

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

Total NUs Charged by PI

Resource = PSC-BLACKLIGHT

2013-01-01 to 2013-03-31

175 14 Purdue University - Service Provider Quarterly Report 14.1 Executive Summary The Purdue service provider (SP) provides an HPC cluster (Steele), a high-throughput computing resource (the Purdue Condor pool), and a cloud resource (Wispy) to XSEDE community XSEDE’s resource allocation process. The SP operates the systems and provides helpdesk, user support and scientific consulting services, as well as participates in XSEDE-wide operations, security, software, training and outreach activities. Purdue contributes its expertise in HPC, high- throughput computing, virtualization and science gateway development to assist XSEDE users through training events, tutorials and demonstrations, as well as to the XSEDE ECSS staff on its conference calls and at conferences. These activities are funded by the NSF awards #0503992, #0932251. Purdue XSEDE resources have supported 97 users (58 projects) from 46 institutions during the reporting quarter. More than 786K jobs, including 425K science gateway jobs were completed, consuming approximately 12 million service units (SUs) on Purdue SP resources during this quarter. We expect the usage to decline –users migrating to other XSEDE resources -- as we approach the retirement date of the Purdue resources. After July 31, 2013, Purdue’s Steele will be decommissioned from the XSEDE as it will be physically retired from Purdue’s community cluster facility. Purdue’s Condor pool will no longer be allocated via the XRAC but will still be accessible on XSEDE via the Open Science Grid. Purdue’s roles in XSEDE also include the following (funded by the XSEDE award):  ECSS staff to provide expert support to user requests for advanced technical assistance.  Lead for the XSEDE Campus Champion Program to further expand the XSEDE user base by reaching out to campuses, providing training and outreach activities, and getting feedback from campus users to improve XSEDE services.  An XSEDE – OSG liaison to bridge the efforts of the two cyberinfrastructure projects by providing assistance and guidance to users in utilizing the OSG resources and identifying and organizing training and outreach activities to broaden user base. Purdue continues to contribute to the XSEDE project by leading the Campus Champions program, providing OSG liaison to help users utilize OSG resources and leverage OSG user support and training activities. A total of 9 new campus champions from 9 institutions have joined XSEDE in this quarter. Purdue SP staff actively participated in Campus Champion related conferences and trainings, such as the online Campus Champion 101 training program (New Champion Orientation), which was held in this January, and the OSG All Hands meeting in Indianapolis. As a service provider, Purdue continues to support a number of science gateways that utilize XSEDE data and computational resources, bridge OSG computation high-throughput HPC (HTHPC) jobs to XSEDE resource, and develop, deploy virtualization, bioinformatics, biostatics and Molecular Dynamics (MD) simulation tools to support research work across the country. These activities and impacts are reported in the overall XSEDE report sections. 14.1.1 Resource Description Steele The Steele cluster consists of 893 dual quad-core Dell 1950 compute nodes, running Red Hat Enterprise Linux, version 4. Each node thus has 8 64-bit Intel 2.33 GHz E5410 CPUs and either 16 GB or 32 GB of RAM. They are interconnected with either Gigabit Ethernet or InfiniBand.

176 The machine offers access to the 120 TB scratch space. Steele’s peak performance is rated at 66.59 TFLOPS. Steele cluster is well suited for a wide range of both serial and parallel jobs. Steele replaced the Purdue Lear cluster and was made available to TG users in May 2008. Its projected useful lifetime is through July 2013. In October 2009, Purdue RP has increased the TG dedicated portion of Steele from 22 nodes to nearly 200 nodes (1600 cores). Steele has no effective runtime limit on XSEDE jobs. Additionally, XSEDE users may leverage the larger Steele cluster by utilizing the standby queues with no node limit but subject to 4 or 8 hour runtime limits. Condor Pool The Purdue Condor pool is a shared resource among the resource owners (academic users at Purdue) and XSEDE/OSG users. Consisting of approximately 50,000 processor cores, the Condor pool is an opportunistic resource which allows Condor jobs access to machines that are not being used by their owners. The Purdue Condor pool is designed for high-throughput computing, and is excellent for parameter sweeps, Monte Carlo simulation, or most any serial application. In addition, some classes of parallel jobs (master-worker) may be run effectively in Condor. 30% of all Condor-usable cycles are available to XSEDE users at a minimum level of service. On average the Purdue Condor pool is able to provide up to 10 million CPU hours to XSEDE users per year. The Purdue Condor resource, recently named DiaGrid, has expanded tremendously from a total of 7700 CPUs at the end of 2007 to its current size of about 50,000 cores, the largest Condor pool in the U.S. Majority of the nodes runs Linux OS, while a small portion runs Windows. It is a federated pool from 10 institutions, including Purdue’s West Lafayette campus, Purdue Calumet campus, University of Wisconsin-Madison, University of Nebraska-Lincoln, Indiana University, and University of Notre Dame. Memory on most of the compute nodes ranges from 2 GB, 4 GB to 10 GB per core. With a total of approximately 390 TFLOPS (peak) available, the Purdue Condor pool can provide large numbers of cycles in a short amount of time. All shared areas and software packages available on Steele are available on Condor. Available to TeraGrid/XSEDE users since 2006, the Condor pool is self-renewing as old machines in the pool are retired and new ones, e.g., from Purdue’s community clusters, added over time. Wispy Purdue’s Wispy is a special XSEDE resource, a cloud computing platform for research and educational use. Wispy consists of eight 64bit, 16-core HP SL230 connected via 1 Gigabit Ethernet network with the capacity of supporting 128 VMs. Wispy runs KVM and the Nimbus cloud software. It provides users with the capability of packaging their applications and operating systems completely inside the Virtual Machine (VM) images, submitting these VMs to run in Wispy with up to 14 CPUs and 32GB of memory each, and have full control over the execution environment. Current usage includes small, instant, on-demand clusters for various tasks and running complicated or prepackaged applications on additional hardware resources. 14.2 Science Highlights A Molecular Dynamics Simulation of assembling behavior of a triplex complex formed by amylose, beta-lactoglobulin, and alpha-linoleic acid.

PI: Dr. Bruce Hamaker and Dr. Osvaldo Campanella, Whistler Center for Carbohydrate Research, Purdue University

177 Food science researchers have recently become aware of the XSEDE resources. At the Whistler Center for Carbohydrate Research, researchers study fundamental structure- function relationships of carbohydrates and other biopolymers as related to practical uses. Structure-function relationships are normally determined via elucidation of chemical and three-dimensional structures. With recent computer hardware advancements, computational methods are becoming a critical part of food science research and expanding into other areas of studies. XSEDE resources Figure1. Image of simulated amylose and GLK assisted Dr. Bruce Hamaker, a professor of complex. Whistler Center for Carbohydrate Research at Purdue University, in studies of the detailed complexion behavior of amylose and α-linoleic acid at atomic level in order to understand the binding mechanism of small molecules to amylose. Traditionally, they ran simulations on a small set of lab computers, which limited them for only 2-10 ns simulation over a 10-day period, which is insufficient to produce conclusive data to explain experiment observations. This group started their computation on the Purdue campus resource DiaGrid.org. With the help of Purdue’s XSEDE campus champion, this group began to utilize XSEDE resources for their MD simulations in 2013 and have run on Stampede, Kraken and Lonestar systems. This computation typically uses 64-128 cores. The initial result of 500 ns all-atom MD simulations starting from the initial structure including α-linoleic acid indicates a large-scale order-to-disorder conformational transition within the first 200ns, suggesting weak coupling between amylose and α-linoleic acid. Upon closer inspection, it is clear that α-linoleic acid is necessary and sufficient for amylose to reach the thermodynamically stable configuration. Such result demonstrates the importance of real-time dynamics in the formation of thermodynamically stable configuration of the complex, which have broad implications for understanding amylose small molecular binding behaviors. The results improve models characterizing such systems, which are important in both natural and manmade materials. The results also enhance understanding of how the local, particle-scale interactions within these materials lead to collective or emergent properties.

The Whistler Center is a university-industry research Figure2. Protein Contact Map for a 500ns MD center that conducts fundamental research related to simulation. practical applications of food carbohydrates, provide analytical services, and provide training and education. Its research and service directly impact U.S. food industry and its computational need has the potential to grow to a much larger scale.

A hidden Markov model-based algorithm for identifying tumour subtype using array CGH data

PI: Ke Zhang, Director of ND INBRE Bioinformatics Core, University of North Dakota

178

Cancer is a heterogeneous disease caused by genomic aberrations and characterized by significant variability in clinical outcomes and response to therapies. In the meantime, cancer is a complex disease driven by the interaction of multiple genes. It has been reported that the copy number of individual gene is not sufficient to define cancer subtypes and predict responses to treatments. Therefore, a classification based on genome-wide copy number patterns would be better suited for this purpose. The recent advancement in array comparative genomic hybridization (aCGH) research has significantly improved tumor identification using DNA copy number data. A number of unsupervised learning methods have been proposed for clustering aCGH samples by the cancer genome pattern community. Two of the major challenges for developing aCGH sample clustering are the high spatial correlation

between aCGH markers and the low computing efficiency. Professor Ke Zhang at the University of North Dakota uses Purdue XSEDE Steele resources for developing a Markov model-based algorithm for identifying tumor subtype using array CGH data. Dr. Zhang and colleagues are interested in the spatial correlation between aCGH markers. They have developed an efficient clustering algorithm to identify tumor Figure 3 Workflow of HMMC clustering procedure. subtypes based on DNA copy number aberrations. The performance of the proposed hidden Markov model based clustering (HMMC) method has been evaluated using both simulated and real patient aCGH data. Professor Zhang’s work properly models the high spatial correlation between aCGH markers. Clusters of tumor samples are modeled with a mixture of hidden Markov model (HMM) models where each HMM fits a cluster of samples. XSEDE recourses helped Zhang’s group develop and refine their HMM-based clustering algorithm utilizing 256 threads simultaneously. Analysis on 95 glioma patient CGH data using this algorithm has shown a rapid convergence to the optimal cluster and a shortened computation time in comparison to other clustering Figure2. A coarse-grained model for biomembrane methods, such as hierarchical simulations clustering (HC), support vector machine, probability model-based clustering, and nonnegative matrix factorization (NMF). Specifically, this HMMC method has a 50% lower error rate than that of the NMF method. The HMMC method was able to locate the optimal number of groups automatically when applied to the glioma aCGH data. The resulting clustering of glioma samples has Figure 4 Heatmap of glioma clustering. 179

shown strong correlation to clinical data on survival rate. The newly developed algorithm would potentially have wide applications in tumor subtype identification, genomic signature discovery, and diagnostic and prognostic biomarker search. 14.3 User-facing Activities 14.3.1 System Activities The Steele cluster continues to be busy and highly utilized by the XSEDE users during the reporting period. XSEDE users access Steele through the XSEDE queues, and in addition to the NSF funded portion of the cluster, XSEDE users have access to the entire cluster through its standby queues with a wall clock limit of 4 hours for each job. In this manner, XSEDE users typically consumes as high as three times of the cycles allocated for XSEDE on a monthly basis. Figure 5 shows the monthly usage on Steele by XSEDE users to date since Mar. 2012. The SP strives to provide the highest level of system availability to its cluster users (see table2). There was an unscheduled 92 minutes downtime on Steele Monthly CPU Hours Delivered to XSEDE Users Steele in January. The SP Actual usage NSF funded staff discovered a network 4,000,000 interruption for 10 nodes (lasting 47 hours) on 2/9- 3,500,000 2/11 and was restored promptly. This interruption 3,000,000 caused a slight degradation 2,500,000 in capacity but did not impact access by XSEDE 2,000,000 users. The SP had a 1,500,000 scheduled maintenance all Purdue systems, including 1,000,000 Steele/Wisp/Condor and campus systems during the 500,000 winter break. All of our - systems were unavailable for a period of 90 hours (01/01- 01/04) continuing the maintenance service from Figure 5. CPU hours of Steele used by XSEDE researchers, Jan. 2012 – Mar. December 2012. 2013 The scheduled maintenance to upgrade home directory has been completed in early January. During this downtime, systems also received OS updates, network updates, upgrades to the PBS batch system software, and updates to the software stack. After 5 years of service, the NAS system currently providing research home directories is being retired. Beginning January 3, 2013, Purdue research computing home %Uptime (monthly) directories will leverage the new Extensible Storage Facility hardware 2013 Condor Steele Wispy deployed for the new BoilerBackpack Jan 87% 87% 100% service. This upgrade has direct impact on XSEDE users, including a larger Feb 100% 100% 100% default space quota and the removal of Mar 100% 100% 100% file limit quota. The extensibility of the storage offers project space for large Table 2: Uptime Percentages for Steele/Condor/ data storage in a storage “condo” Wispy (01/01 – 03/31/2013).

180

Figure 5. Usage of Steele delivered to XSEDE users, Oct. 2011 – Dec. 2012 model, allowing projects, including the SP project, to purchase more storage as user needs increase. Purdue aims to provide a software stack that allows for optimal use (performance and stability) of the clusters. This necessitates periodic updates to the stack as compilers, libraries and software are improved over time. During this scheduled service period, software upgrades were performed, maintenance to the research computing network was completed. Lastly, the OpenMPI on Steele was upgraded to the latest 1.6.3 version. This new version offers many performance and stability improvements for RDMA and InfiniBand technologies. We also introduced a new set of compliers:  Intel 13.0.1.117  PGI 12.10-0  Popular MPIs and libraries built with these compilers These new compilers include bug fixes and enhanced performance and stability. The default GCC compiler version was upgraded from GCC 4.4.0 to 4.7.2. Other changes include the upgrade of the default Matlab module to version R2012a and removal of old versions of Matlab. 14.3.2 Services Activities Purdue SP provides both helpdesk support and consulting support to XSEDE users. The SP user support staff worked with many XSEDE users during the quarter. Most of the support requests were related to troubleshoot issues. Categories of user issues and requests are summarized in the table in Section 1.8.2. Purdue’s SP staffs have identified 4 potential contacts to discuss possible access to XSEDE resources. Starting in January, Dr. Feng (Kevin) Chen is the new XSEDE Campus Champion to help Purdue researchers. Dr. Chen and Kim Dillman (both Purdue Campus Champions) work with local users, and their findings in this process often translate into training materials for the larger XSEDE community. They actively participate in the face-to-face working meetings and phone meetings for both the Outreach and XSEDE allocation proposal Preparation, and recently worked on various documents for XSEDE13. They added two users who are now ready to use XSEDE to their Campus Champion allocations. Dillman attended the OSG All Hands meeting in Indianapolis Mar. 12-14, 2013 and presented “Meeting the HTC Demand with DiaGrid and XSEDE”. Purdue staff also is working with OSG on how to handle larger allocation requests for OSG in the future. Currently OSG makes available 2 million SUs per quarter to XRAC. As new and potential users of OSG are identified, this will become severely inadequate. Purdue used to give out at least 6-8 million SUs from the Condor Pool for high-throughput requests and was very flexible if more was needed. With the impending decommission of the Condor pool from XSEDE allocation process and the current OSG availability, we needed to find new ways to help new high- throughput users and for existing Condor Pool users to transfer to OSG. At the last allocation meeting in March, it was decided that there will be two “levels” of OSG allocations going forward, i.e. the “guaranteed” and “not guaranteed”. This will hopefully allow OSG to offer more SUs each allocation period. OSG will work out the details for how to handle this new method from the “technical” implementation side on their submit host. 14.4 Security No security incident was reported during this quarter.

181 14.5 Education, Outreach, and Training Activities 14.5.1 EOT Events

Number Date(s Number of of Under- Type Title Location Hours Method ) Participants represent ed people

Tutorial Campus Network 01/23/ 1 Approx. 11 n/a Remotely Champions Meeting 2013 presented 101 Training to conference Meeting the Indian- Presented HTC Demand 03/12/ Presentation apolis, 1 Approx. 55 n/a to with Diagrid 2013 conference and XSEDE IN West SubmitR V2.0 02/10/ Classroom, Tutorial Lafayette 2 Approx. 35 n/a Event 2013 hands-on ,IN

14.5.2 Education Purdue SP Staff is preparing for the 2013 OSG Summer School. During the school, June 26-30, students will learn to use high-throughput computing (HTC) systems — at their own campuses or using the national Open Science Grid (OSG) — to run large-scale computing applications that are at the heart of today’s cutting-edge science. Through lectures, discussions, and lots of hands-on activities with experienced OSG staff, students will learn how HTC systems work, how to run and manage lots of jobs and huge datasets to implement a scientific computing workflow, and where to turn for more information and help. The school is ideal for graduate students in computer science or other sciences where large-scale computing is a vital part of the research process, but any qualified and interested applicant will be considered.

14.6 SP Collaborations The SP staff continued to work with an USDA funded project (USDA-NIFA no. 2011-68002- 30220), an integrated research and extension project working to improve farm resilience and profitability in the North Central Region by transforming existing climate information into usable knowledge for the agricultural community. The overall purpose of the project is to develop decision support tools for use in understanding the potential impact of climate change on the production of maize (corn) in the region. The researchers in this project are conducting modeling and data synthesis which often require long runs on resources such as those available on XSEDE and high performance data storage. The SP staff is assisting the research group to investigate how to make the large number of simulation runs manageable by designing workflows, identifying appropriate resources, as well as assisting with data processing tasks. 23 of the desired 31 years of gridded simulation data have been created. These gridded simulation data are being used in a corn crop development model. In the last three months, the SP staff has also been involved in developing web enabled decision support tools based on a minimum of 30 years of climate data for the region including min and max temperature, rainfall, growing degree days and county yield data.

182 Purdue is also participating in a collaborative SI2-SSI project to develop “An Interactive Software Infrastructure for Sustaining Collaborative Community Innovation in the Hydrologic Sciences”. Purdue PI is co-PI to this project and provides computational (including XSEDE and high-throughput) and online modeling and data management expertise to the team (NSF award # 1148090).

14.7 SP-Specific Activities In the past quarter Purdue SP staff worked with Professor Larry Band in the Geography department of University of North Carolina to use WaterHUB (funded by NSF CI-TEAM program) in his watershed hydrology course in spring semester. The students will run different versions of the SWAT models using the SWATShare tool to study the input/output for different watersheds and seasons, as well as exploring dynamics of soil water, evapotranspiration, and groundwater. The Purdue staff has focused on improving the reliability and usability of SWATShare based on the feedbacks from the class. In the meantime, we also made significant progress on supporting model auto calibration and sensitivity analysis in SWATShare. Purdue SP staff developed a plant isotopic model in the IsoMAP gateway (funded by NSF ABI program). The staff has completed the software modules for raster data preprocessing and is currently working on testing the C++ model with different model inputs. The staff also improved system robustness by migrating its Solr installation to a production server and the Solar data store to a more reliable file system. The gateway will be used to support two stable isotope short courses at University of Utah in June. A new OSG related ECSS project from a PI at University of Michigan was assigned to Purdue ECSS Staff. Our ECSS staff worked with the PI and his Campus Champion, Brock Palen to determine if they could request OSG instead of the Purdue Condor Pool for their XRAC request which they did.

14.8 Publications Staff publications: (1) Dillman, Kimberley A . Meeting the HTC Demand with Diagrid and XSEDE Open Science grid All Hands meeting, Mar. 12, 2013 . (2) M. McLennan, S. Clark, E. Deelman, M. Rynge, F. McKenna, D. Kearney, and C. Song. “Bringing Scientific Workflow to the Masses via Pegasus and HUBzero”, 5th International Workshop on Science Gateways, IWSG 2013, Zurich, Switzerland, June 3- 5, 2013. Submitted (3) B. Cotton, C. Thompson, B. Raub. BLASTer: “BLASTer: A hub-based tool for bioinformatics”, 2013 XSEDE annual conference, San Diego, CA, July 22-25, 2013. Submitted (4) Rajesh Kalyanam, Lan Zhao, Carol Song, Yuet Ling Wong, Jaewoo Lee and Nelson Villoria. “iData: A Community Geospatial Data Sharing Environment to Support Data- driven Science”, 2013 XSEDE annual conference, San Diego, CA, July 22-25, 2013. Submitted (5) Kim Dillman. Selecting and Using XSEDE Resources for Maximum Productivity. Submitted to XSEDE13.

183 User publications Park, S.; Im, W. Two Dimensional Window Exchange Umbrella Sampling for Transmembrane Helix Assembly Journal of chemical theory and computation 2013, 9, 13. Jo, S.; Lee, H. S.; Skolnick, J.; Im, W. Restricted N-glycan Conformational Space in the PDB and Its Implication in Glycan Structure Modeling PLoS computational biology 2013, 9, e1002946. Jo, S.; Im, W. Glycan fragment database: a database of PDB-based glycan 3D structures Nucleic acids research 2013, 41, D470. Jo, S.; Jiang, W.; Lee, H. S.; Roux, B.; Im, W. CHARMM-GUI Ligand Binder for absolute binding free energy calculations and its application Journal of chemical information and modeling 2013, 53, 267. De Stefano, G.; Vasilyev, O. V. Wavelet-based adaptive large-eddy simulation with explicit filtering J Comput Phys 2013, 238, 240. Liu, J. K.; Wen, S. H.; Zou, X. X.; Zuo, F.; Beran, G. J. O.; Feng, P. Y. Visible-light-responsive copper(II) borate photocatalysts with intrinsic midgap states for water splitting J Mater Chem A 2013, 1, 1553. Liu, J.; Wen, S.; Hou, Y.; Zuo, F.; Beran, G. J.; Feng, P. Boron carbides as efficient, metal-free, visible-light-responsive photocatalysts Angew Chem Int Ed Engl 2013, 52, 3241.

14.9 Metrics 14.9.1 Standard User Assistance Metrics Purdue SP ticket resolution times by category from XSEDE ticket system: Time to account file grid jobs/batch login/access mss/data network software system other Resolution issues systems software queues issues issues issues /apps issues 0-1 hr 1-24 hr 1 1 1 1 1-7 d 6 5 8 3 1-2 wk 1 6 8 > 2 wk 2 1 3 9 Still 1 Open

14.9.2 SP-specific Metrics Purdue SP ticket summary: Tickets Tickets Category Activities Received Closed account issues 0 0 csa requests 0 0 file systems 0 0

Gateways 0 0

184 gpfs-wan 0 0 grid software 0 0 inca test MDS stale providers and failures, host certificates 20 20 reports expired, INCA Java errors Project number, jobs rejected by server, jobs jobs / batch deleted,NAMD, qstat errors, g_debug queue 17 15 queues questions, virtual memory errors, segmentation fault in code, compiled MATLAB errors unable to login, gsi-ssh applet error, login shell login / access 10 10 change, file transfer with SCP,environment variables, issues system access methods mss / data file transfer from Steele, Condor Pool shared 3 2 issues filesystem, data loss network issues 0 0

Other 3 2 space on DC-WAN, change login shell, SFTP error refund request 0 0 reservation 0 0 request Security 0 0

Running DESMOND, using Python on Condor, usgin software / apps 10 9 HSI, Gromacs question, R on Condor, LAMMPS error, VASP question system issues 18 18 Inca errors, PBS server unresponsive, qstat errors Workshops 0 0

TOTAL 81 76

14.9.3 Standard systems metrics See charts below.

185 PURDUE-STEELE Quarterly Report

Total NUs Charged by Resource

Service Provider = PURDUE

Total NUs Charged by Job Size

Resource = PURDUE-STEELE

186 Avg Wall Hours Per Job by Job Size

Resource = PURDUE-STEELE

Avg Wait Hours Per Job by Job Size

Resource = PURDUE-STEELE

187 User Expansion Factor by Job Size

Resource = PURDUE-STEELE

Total NUs Charged by Job Wall Time

Resource = PURDUE-STEELE

188 User Expansion Factor by Job Wall Time

Resource = PURDUE-STEELE -- Service Provider = PURDUE

Total NUs Charged by Field of Science

Resource = PURDUE-STEELE

189 Total NUs Charged by User Institution

Resource = PURDUE-STEELE

Total NUs Charged by PI

Resource = PURDUE-STEELE

190 PURDUE-CONDOR Quarterly Report

Total NUs Charged by Resource

Service Provider = PURDUE

Total NUs Charged by Job Size

Resource = PURDUE-CONDOR

191 Avg Wall Hours Per Job by Job Size

Resource = PURDUE-CONDOR

Avg Wait Hours Per Job by Job Size

Resource = PURDUE-CONDOR

192 User Expansion Factor by Job Size

Resource = PURDUE-CONDOR

Total NUs Charged by Job Wall Time

Resource = PURDUE-CONDOR

193 User Expansion Factor by Job Wall Time

Resource = PURDUE-CONDOR -- Service Provider = PURDUE

Total NUs Charged by Field of Science

Resource = PURDUE-CONDOR

194 Total NUs Charged by User Institution

Resource = PURDUE-CONDOR

Total NUs Charged by PI

Resource = PURDUE-CONDOR

195 15 San Diego Supercomputer Center (SDSC) Service Provider Quarterly Report 16 16.1 Executive Summary Gordon completed its fifth quarter of operations during the reporting period. Gordon was conceived and deployed as the first XSEDE system to address challenges of data intensive computing. The use of massive amounts of flash memory, large memory nodes, and a high performance parallel file system are proving useful to researchers in a wide range of domains. The latest XRAC meeting in March 2013 continued the trend of increasing demand/awards on Gordon, with an over-request factor of 1.65X and all available cycles awarded. Gordon also now hosts jobs for the thriving CIPRES and GRIDCHEM gateways, along with Trestles. Trestles enters its third year of production and continues to be highly successful in its objectives to support the modest-scale/gateway user community, with a focus on user productivity and fast turnaround. The system now hosts five gateways. The system utilization remains reasonably high while still maintaining our primary objective of short wait times and low expansion factors. We have committed to extend the operational life of Trestles at least through June 2014. As a planned element of the Gordon project, a major upgrade of the Project Storage file system was procured and deployed this quarter, with production availability April 5, 2013. This upgrade, implemented by Cray/Appro and their supplier Aeon Computing, expands Project Storage from ~400TB to ~2PB. This medium-term disk storage resource, which is co-mounted across both Gordon and Trestles, represents a new paradigm for XSEDE, allowing users extended access to their data for analysis, but not the long-term expectations of tape archives. The initial resource has been allocated via an internal proposal review process, and that process is now transitioned to the XRAC storage allocation request process. A highlight this reporting period was a rapid-response effort working with a Fermilab/UCSD team to use Gordon to rapidly process large volumes of LHC/CMS data in their search for dark matter. After hearing of the team’s surge computing requirements, we used an allocation of director’s discretionary time and volunteered to host their OSG-based workflow on Gordon and accelerate their time-to-science by a factor of about three compared to their current OSG resources. In less than two months from start to finish, we not only facilitated significant scientific impact, but also essentially demonstrated the feasibility of making an XSEDE resource part of the Open Science Grid. SDSC continues to be active in presenting XSEDE-wide training in Hadoop (now available on Gordon), data analytics, visualization and the new NeuroScience Gateway. These are part of SDSC’s thrust to provide training in areas relevant to data-intensive area. 16.1.1 Resource Descriptions Gordon Gordon is a dedicated XSEDE cluster designed by Appro and SDSC consisting of 1,024 compute nodes and 64 I/O nodes. Each compute node contains two 8-core 2.6 GHz Intel EM64T Xeon E5 (Sandy Bridge) processors and 64 GB of DDR3-1333 memory. The I/O nodes each contain two 6-core 2.67 GHz Intel X5650 (Westmere) processors, 48 GB of DDR3-1333 memory, and sixteen 300 GB Intel 710 solid state drives (4.7TB total SSD). The network topology is a 4x4x4 3D torus with adjacent switches connected by three 4x QDR InfiniBand links (120 Gbit/s). Compute nodes (16 per switch) and I/O nodes (1 per switch) are connected to the switches by 4x QDR (40 Gbit/s). The theoretical peak performance of Gordon is 341 TFlop/s.

196 Trestles Trestles is a dedicated XSEDE cluster designed by Appro and SDSC consisting of 324 compute nodes. Each compute node contains four sockets, each with an 8-core 2.4 GHz AMD Magny- Cours processor, for a total of 32 cores per node and 10,368 total cores for the system. Each node has 64 GB of DDR3 RAM, with a theoretical memory bandwidth of 171 GB/s. The compute nodes are connected via QDR InfiniBand interconnect, fat tree topology, with each link capable of 8 GB/s (bidirectional). Trestles has a theoretical peak performance of 100 TFlop/s. Project Storage Project Storage is an XSEDE storage resource, which now provides ~2 PB of medium-term persistent storage to XSEDE users, and is available from both Gordon and Trestles. Project Storage has been allocated since early 2012 via a pilot internal proposal process, and is not being transitioned to the formal XRAC storage allocation process. Project Storage is part of Data Oasis, a multi-component Lustre-based parallel file system designed by SDSC and supplied by Aeon Computing and Cray/Appro, 16.2 Science Highlights SDSC’s Gordon Assists in Crunching Large Hadron Collider Data SDSC’s Gordon recently completed its most data-intensive task so far: rapidly processing raw data from almost one billion particle collisions as part of a project to help define the future research agenda for the Large Hadron Collider (LHC).

Gordon provided auxiliary computing capacity to the Open Science Grid by processing massive data sets generated by the Compact Muon Solenoid, one of two large general-purpose particle detectors at the LHC used by researchers to find the elusive Higgs particle. The around-the-clock data processing run was completed in about four weeks’ time, making the data available for analysis several months ahead of the team’s original schedule.

About 1.7 million core hours – or about 15% of Gordon’s compute capacity during the month- long processing– were dedicated to this task, with more than 125 terabytes of data streaming through Gordon’s nodes and into SDSC’s Data Oasis storage system for further analysis. For more details please see the full press release at http://www.sdsc.edu/News%20Items/PR040413_lhc.html.

UC San Diego Team Achieves Petaflop-Level Earthquake Simulations on GPU-Powered Supercomputers - Accelerated Code Cuts Time and Cost in Seismic Modeling A team of researchers at SDSC and the Department of Electronic and Computer Engineering at UC San Diego has developed a highly scalable computer code that promises to dramatically cut both research times and energy costs in simulating seismic hazards throughout California and elsewhere.

Led by SDSC computational scientist Yifeng Cui, the team developed a scalable GPU accelerated code for use in earthquake engineering and disaster management through regional earthquake simulations at the petascale level, as part of a larger computational effort coordinated by the Southern California Earthquake Center (SCEC).

The team performed GPU-based benchmark simulations of the 5.4 magnitude earthquake that occurred in July 2008 below Chino Hills, near Los Angeles. Compute systems used included Keeneland, an XSEDE resource managed by Georgia Tech, ORNL, and NICS. “The increased capability of GPUs, combined with the high-level GPU programming language CUDA, has

197 provided the tremendous horsepower required for acceleration of numerically intensive 3D simulation of earthquake ground motions,” said Cui, who recently presented the team’s new development at the NVIDIA 2013 GPU Technology Conference (GTC) in San Jose, Calif. A technical paper based on this work will be presented June 5-7 at the 2013 International Conference on Computational Science Conference in Barcelona, Spain. For the full press release please see http://www.sdsc.edu/News%20Items/PR040213_earthquake.html UC San Diego Awarded NIH Grant to Expand Diabetes and Obesity Research HubSDSC to Provide Computational Resources and Host Research Database UC San Diego researchers have been awarded a National Institutes of Health (NIH) grant to expand and enhance a cyberinfrastructure designed to provide scientists with easily accessible, Web-based resources to help fight diabetes and metabolic diseases.

The grant will focus on establishing, coordinating, and making available large pools of datasets to researchers as part of a project to further develop the National Institute of Diabetes and Digestive and Kidney Diseases’ (NIDDK) Interconnectivity Network community and infrastructure.

Efforts will also focus on enhancing the analytical capabilities of the portal through the creation of workflows, such as SDSC’s Kepler project, using SDSC’s data-intensive Gordon supercomputer and the SDSC Cloud to make datasets accessible via the portal to ensure that the network is sustainable beyond the term of the award. For the full press release please see http://www.sdsc.edu/News%20Items/PR021313_diabetes.html. 16.3 User-facing Activities 16.3.1 System Activities In addition to routine system monitoring and administration, the systems effort last quarter focused on:  Expanding the Data Oasis Project Storage;  Working with the Open Science Grid to extend their data analysis capabilities.

The NSF project space accessible from Trestles and Gordon expansion was performed by Aeon Computing on behalf of Cray, and consisted of adding 8 new JBODs (disk arrays) with 45 3TB drives each, and replacing the existing 36 2TB drives per server with 3TB ones. In preparation for the expansion, all of the JBODs were installed ahead of time, and ready to be cabled to the servers. The drive replacements necessitated temporarily relocating user data during the upgrade, and while the project file system was in production, user data was copied to Gordon's scratch Lustre file system in a restricted directory. After user access was removed, a final sync of the data copy was made. After verifying the replica, the drives were replaced and the JBODs attached, and we validated the performance of the individual storage targets before creating the Lustre file system. With the file system in place, Lustre performance was verified from Trestles and Gordon, with all servers sustaining greater than 2200 MB/s write speeds, and greater than 2100 MB/s read speeds. While the upgrade went smoothly overall, it was delayed temporarily while we worked with SDSC's facilities staff to improve the airflow to the Data Oasis systems and reduce the drive temperatures, which were nearing the upper limit of the operating range. The portion of the SDSC machine room containing Data Oasis uses hot aisle containment to improve cooling efficiency. However, the high-density storage servers and JBODs in Data Oasis have drive trays in the back, facing the hot aisle, which makes maintaining the intake temperatures and pressure differential critical. Complicating this are the side intakes on the servers and JBODs which provide additional air to the rear drives. Normally, the fronts of the racks are sealed, and the inside of the rack is

198 considered part of the hot aisle. Our facilities staff was able to shift this barrier to the rear of the racks, which created a cold plenum chamber for the side intakes. This inexpensive modification brought the hottest drives temperatures down by about 20°F, providing a reasonable margin. A challenging – and very rewarding - project last quarter was enabling the Open Science Grid (OSG) analysis on Gordon for the LHC/CMS experiment. Accommodating the CMS workflow required few changes to the Gordon production environment; most of the requirements fit within the standard XSEDE models of user access and computing. Some work was required to add distinguished names for CMS users, but this was not out of the ordinary. There were a few features of Gordon that were well suited to hosting this project.  SDSC's high-performance GridFTP servers attached to Data Oasis, which were critical to getting data to Gordon at a fast enough pace that the analysis could be completed in the desired schedule;  The SSD drives on the compute nodes providing fast local scratch space, and reducing small-block IO on the Lustre file systems.  An I/O node dedicated to the project. The IO node's capabilities in terms of public network access, fast storage, and sharing the dual InfiniBand rails with the compute nodes made it an ideal host for the needed HTTP proxy and application server. In order to maintain security controls, SDSC staff had responsibility for installing and configuring the I/O node.

A paper discussing the technology integration between the Open Science Grid and SDSC as an XSEDE Service Provider has been submitted for presentation at the XSEDE’13 conference. Other work included transitioning several Gordon nodes from vSMP to standard compute nodes that mount I/O node SSDs, to improve overall system utilization. There were no significant equipment failures during the quarter. With the project storage expansion complete, our attention for Q2 will be on completing the upgrade of Trestles and Gordon to CentOS 6. 16.3.2 Services Activities Between the two tickets systems used to support Trestles and Gordon (the XSEDE ticket system and SDSC’s local ticket system), a total of 544 tickets were created between January 1 and March 31, 2012. These tickets included account questions, file system issues, software requests, Globus support, code support, password resets, code optimizations and debugging, allocation refunds/problems, project space requests, software support, licensing queries, and resource availability. 508 of those tickets were closed, leaving 36 tickets that we are still working to resolve. The average time to close the tickets across the two systems was 6 days, with a median time of 2.5 days. In support of Gordon users, SDSC staff fielded questions and provided information on Gordon’s architecture, software install requests, utilizing SSD scratch space, and configuration of Hadoop on Gordon. New software installs/support included SPRNG, mono (v. 2.10.8), AMBER, Weka, Abaqus (6-11.2), and LAMMPS. SDSC user services staff supported a dedicated I/O node project, which was set up to handle a Hadoop cluster for students in a class. Assistance was provided in installing software in conjunction with Hadoop, including hbase, Revolution-R, RHadoop, and thrift. In support of Trestles users, SDSC user services staff fielded questions on gridftp usage, allocations/accounts, new software installs, system performance, and filesystem use/troubleshooting. New software installs/support included Octave (v.3.6.2), PCRE (v8.32),

199 LAMMPS, python (v2.7.4), raxml (v 7.4.2), and Abaqus (6-11.2). SDSC staff also worked on an Abaqus issue for a large sized problem. Tests were conducted on both Trestles and Gordon, and issue was resolved by moving simulations to Gordon where there is more SSD scratch space available. Benchmark results were provided to the user to assist in the process.

SDSC user services staff supported users through the Data Oasis Project Storage expansion process on both Gordon and Trestles. Users were assisted with moving job-critical data (to Data Oasis scratch locations) in advance of the expansion process. Users were provided with alternate jobs options for running utilizing local scratch or Data Oasis scratch as applicable during the expansion process. As a result, the expansion process was completed with minimal impact on user jobs.

16.4 Security SDSC had one minor incident this quarter. One user account was preemptively locked on both Gordon and Trestles.

16.5 Education, Outreach, and Training Activities San Diego Supercomputer Center’s Education, Outreach and Training activities for the first quarter of 2013 are summarized in the table below.

Type Audie Hr # # Meth Title Location Date(s) nce s Partic. U-r od TRAINING Tuto F. D. Visualization for Harvard January 2. 32 na Sync, rial G. U. Big Data University 17 5 web I. Exploratory Research CI Days by Campus Champions (Chourasia) Tuto F. D. Introduction to UCSD/SDSC January 2 86 42 Sync, rial G. U. using Hadoop on 31 web I. Gordon (Tatineni) Tuto F. D. Visualization UCSD/SDSC February 1. 84 54 Sync, rial G. U. Training: 11 5 web I. Preparing/Transl ating Data for use with VisIt application (Chourasia) Tuto F. D. Introduction to UCSD/SDSC March 14 2 50 24 Sync, rial G. U. the Neuroscience web I. Gateway (NSG) - A Portal for Computational

200 Neuroscience (Majumdar) Wor F. D. OpenACC GPU UCSD/SDSC January 12 11 n.a. Web ksho G. U. Programing 15-16 p I. Workshop (satellite location) Wor F. D. PACE: Data UCSD/SDSC January 16 23 8 Sync, ksho G. I. Mining Boot Camp 23-24 web p 1 (Balac) Wor F. D. PACE: Data UCSD/SDSC February 16 19 8 Sync ksho G. I. Mining Boot Camp 7-8 p 2 (Balac)

EDUCATION W F. S. Smart Team UCSD/SDSC January 6 23 9 S “From Genes to 12 Drugs – a Lab in a Laptop” W S San Diego Jan 10 5 5 S StudentTECH – 15,17,22, Alice 24, 29 W S, F SMART Team UCSD/SDSC Feb 9 3 22 9 S Modeling Clinic W S, F SMART Team UCSD/SDSC March 9 27 2 24 S Middle School Protein Exploration Day W S SMART Team TSRI, La March 14 2 34 11 S Poster Session Jolla OUTREACH Conf F.G. Tapia Celebration Washington February 24 550 500 S U.D. of Diversity in D.C. 20 Computing W S S.D. Mayor’s Cup UCSD/SDSC March 2 6 86 5 S Cybersecurity Challenge Wrk F. D. UCSD’s Research San Diego, March 12, 1 100 20 S shp I. Cyberinfrastructure CA 2013 Program: Enabling Research Thru Shared Services- at CENIC 2013: Building Blocks for Next Gen Networks M F San Diego CSTA UCSD/SDSC March 12 2 15 8 S Chapter Meeting Wrk F. D. Data Services for Arlington, March 13, 1 150 30 S shp I. Campus Researchers. VA 2013 Research Data Management Implementations Workshop

201 Conf Publ San Diego Science San Diego March 23 8 500 300 S ic Festival * S=synchronous, A=asynchronous 16.5.1 Training Training workshops in the first quarter of 2013 focused on Big Data topics, recognizing the growing interest in data-intensive computing among XSEDE and new XSEDE-ready communities. SDSC hosted tutorials on data visualization, Hadoop, predictive data analytics, and the new Computational Neuroscience Gateway (CNG). All but one were offered both as an on- site event and as a webcast, to maximize the users reached across the country. This approach has proven effective at increasing SDSC’s training audiences by four or five fold over on-site offerings. Particularly when introducing cutting-edge technologies and tools, the wider audiences help to make them available more quickly across the country and encourage potential users to contact SDSC and XSEDE for assistance incorporating these tools into their research. 16.5.2 Education SDSC’s Research Experience for High School Students collected 172 applications by the March 31 application deadline. This program welcomes students for part-time internships of all kinds within the supercomputer center. Most are within research teams, but some will be helping with the planning and coordination of the XSEDE’13 conference. These valuable experiences have proven valuable introductions to computational science and other XSEDE-related fields for many students. Alumni have garnered prizes at local, regional, and national research competitions as well as won scholarships to attend prestigious colleges and universities – many with declared majors related to their internship experience. SDSC’s NSF-funded CE21-ComPASS project recruited 23 new teachers seeking professional development to teach Computer Science Principles. This represents more than double the number projected in the original project proposal. SDSC’s 2013 StudentTECH Summer Programs were announced on February 15. By March 31, ten of the fourteen classes had already filled, with requests for additional sections. The high demand for computing and computational science learning opportunities at all levels may reflect increased public awareness of national shortages in computing-savvy workers. XSEDE blog compiled 288 posts from January- March 2013. Posts currently include single news items and the XSEDE weekly HPC Research and Education weekly newsletter posts. Some of this content has also enhanced the XSEDE TEOS. The HPCU site was updated regularly with news, events, career postings, internships and fellowship opportunities. Two students have started to help with this effort: Rachel Wilson at Shodor Foundation and Alex Kissinger at SDSC. Rachel will be working on site improvements to enhance both the site design and the submission process options. Alex will focus on posting emerging news and events to HPCU and CSEN (Computational Science Education News Facebook page). He will also be helping with improvements to TEOS web pages. With help from these dedicated students, our goal is to keep the content dynamic and compelling, with additional images, where available and appropriate. SDSC outreach to colleagues through professional conference presentations, workshops, and tutorials related to XSEDE services are included in the table above. In support of campus bridging, SDSC’s Richard Moore presented in two workshops this quarter: UCSD’s Research Cyberinfrastructure Program: Enabling Research Thru Shared Services, at CENIC 2013: Building Blocks for Next Gen Networks, March 12, 2013 in San Diego; and Data Services for Campus Researchers, at the CASC & NSF-sponsored Workshop: Research Data Management Implementations, March 13, 2013 in Arlington, VA.

202 16.6 SP Collaborations 16.6.1 Collaborations with SP XSEDE Users

 Dongju Choi continued work with David Haussler’s team to run the whole genome alignment pipeline on Gordon. A “global” queue was created for this project to enable users to run batch jobs that span Gordon’s vSMP and regular compute nodes.  Dongju Choi and Robert Sinkovits continued working with Mao Ye (U. Illinois, Computational Finance) and achieved a further 2x speedup of the limit order book (LOB) construction code, bringing the total speedup to more than 100x for heavily traded stocks. The newly improved code was applied to three full days of NASDAQ activity, including the May 6, 2010 “flash crash”.  Robert Sinkovits continued collaboration with Doug White (mathematical anthropology) to port pairwise cohesion application to vSMP. With modifications, the code was able to achieve near linear scaling on 256 cores of a vSMP node. Code changes are underway to reduce the memory footprint and rapidly identify the largest strongly connected groups in very large graphs.  Mahidhar Tatineni, Pietro Cicotti and Rick Wagner deployed alternative flash configurations (e.g. 16 SSDs exported to compute node and mounted as separate devices) to help George Porter (UCSD, CSE) achieve optimal I/O performance for large scale sorting benchmarks.  Amit Majumdar, Subhashini Sivagnanam & Kenneth Yoshimoto, collaborating with Ted Carnevale & Michael Hines from Yale School of Medicine and MaryAnn Martone, Anita Bandrowski & Vadmin Astakhov from NIF, continued development of the Neuroscience Gateway (NSG). NEURON version 7.3 and parallel GENESIS were added as new sumulation tools within the NSG and are available on Trestles. Efforts are underway to install other neuronal simulation tools such as MOOSE, NEST, PyNN on Trestles. An XSEDE-wide NSG tutorial was held in March and was attended by about 10 attendees in- person at SDSC and by about 35 attendees via the web. The number of registered gateway users has grown to about 80 in the past few months.  Amit Chourasia provided visualization support for the following projects o Work with the ENZO team on interleaved visualization; o Homa Karimabadi: provided new version of Paraview for visualization of Magneto Hydrodynamics simulation; o Mark Vonmoer: provided documentation and vSMP build of VisIt software for visualizing GridChem data; o Completed visualization work with Darcy Ogden on visualization of volcano eruption simulation data.

16.6.2 Collaborations with External Partners See description above regarding the collaboration with Fermilab/UCSD physicists on processing LHC/CMS data. Rick Wagner completed a UCSD-funded project with the UCSD Library to curate astrophysics simulation data as part of UCSD's Research Cyberinfrastructure Initiative. The curated data included the initial condition, several datasets, and some derived data and images from an adaptive mesh cosmology simulation completed using Enzo on TeraGrid resources. The collected files and metadata are available through the Online Archive of California.

203 16.7 SP-Specific Activities Scheduling and Resource Optimization A shared queue for CIPRES-specific, half-node jobs was implemented on Gordon. Also on Gordon, a reservation for jobs in the 8-16 node range was put in place to help prevent starvation of resources for large, single-switch topology jobs. Optimization and bug fixing for Catalina on Trestles and Gordon was performed. XSEDE-related Activities SDSC participated in a review of status of XSEDE software with XSEDE’s Software Development and Integration team. On Trestles, the configuration of the GRAM Audit capability was updated. When the local audit database is integrated with AMIE packet generation, Trestles will be able to send gateway user attributes to XSEDE. SDSC security staff recently received an invitation to assist XSEDE's Software Development and Integration group by providing additional security expertise in the review process. SDSC has accepted the invitation and looks forward to collaborating with XSEDE in this capacity.

SDSC once again surpassed its availability goal of 95% for the XCDB (XSEDE central database) and the AMIE central instance in Q1 2013. There were only 6 minutes of XCDB downtime and that was during a planned maintenance window. The AMIE central instance did not experience any service interruptions in Q1. In late 2013 Postgresql will be upgraded to take advantage of new features as support for the current version expires in mid-2014. In addition, SDSC continues to be very active in XSEDE through various working groups and the Service Provider Forum. These included:  Development and deployment of storage allocations  XWFS pilot project  XRAC reviews  Allocations-related policy and process issues

16.8 Publications Isborn, C.M., Goetz, A.W., Clark, M.A., Walker, R.C., Martinez, T.J., " Electronic Absorption Spectra from MM and ab Initio QM/MM Molecular Dynamics: Environmental Effects on the Absorption Spectrum of Photoactive Yellow Protein", J. Chem. Theory Comput., 2012, 8 (12), pp. 5092-5106, DOI: 10.1021/ct3006826

Skjevik, A.A., Madej, B., Teigen, K., Walker, R.C., “LIPID11: A Modular Framework for Lipid Simulations using Amber”, Journal of Physical Chemistry B, 2012, 116 (36), pp 11124-11136, DOI: 10.1021/jp3059992.

Talks "Identifying novel drug targets with advanced molecular dynamics simulations: Application to adenovirus proteases", University of Bergen Seminar Series, Bergen Norway, Ross C. Walker, Invited Speaker. January 24, 2013.

“Data Services for Campus Researchers,” Research Data Management Implementations Workshop, Arlington VA, Richard Moore, Invited Speaker, March 13, 2013.

204 Posters Ben Madej, Ross C. Walker - "Molecular Dynamics with a GPU Turbocharger: Delving Deeper into Drug Development Science", NVIDIA Graphics Technology Conference, San Jose, CA, Mar 2013

16.9 Metrics Appendices 1.9A-C includes the following metrics:  1.9-A XSEDE-generated user ticket statistics  1.9-B Trestles and Gordon Quarterly stats from XDMoD (January – March 2013)  1.9-C Local Trestles stats related to achieving user productivity objectives

205

Appendix 15.9A Standard User Assistance Metrics

Time to account file grid jobs/batch login/access mss/data network software/apps system other Resolution issues systems software queues issues issues issues issues 0-1 hr 3 2 1 1 1-24 hr 1 9 19 2 1 10 3 1 1-7 d 8 10 1 42 5 2 13 2 2 1-2 wk 11 2 20 2 1 7 1 2 > 2 wk 3 2 39 1 2 19 1 2 Still 2 4 4 21 2 2 12 1 4 Open

206

Appendix 15.9B Trestles and Gordon Quarterly Usage Statistics

207 SDSC-TRESTLES Quarterly Report

Total NUs Charged by Resource

Service Provider = SDSC

2013-01-01 to 2013-03-31

208 Total NUs Charged by Job Size

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

209 Avg Wall Hours Per Job by Job Size

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

210 Avg Wait Hours Per Job by Job Size

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

211 User Expansion Factor by Job Size

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

212 Total NUs Charged by Job Wall Time

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

213 User Expansion Factor by Job Wall Time

Resource = SDSC-TRESTLES -- Service Provider = SDSC

2013-01-01 to 2013-03-31

214 Total NUs Charged by Field of Science

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

215 Total NUs Charged by User Institution

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

216 Total NUs Charged by PI

Resource = SDSC-TRESTLES

2013-01-01 to 2013-03-31

SDSC-GORDON Quarterly Report

Total NUs Charged by Resource

Service Provider = SDSC

2013-01-01 to 2013-03-31

Total NUs Charged by Job Size

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

Avg Wall Hours Per Job by Job Size

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

Avg Wait Hours Per Job by Job Size

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

User Expansion Factor by Job Size

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

Total NUs Charged by Job Wall Time

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

User Expansion Factor by Job Wall Time

Resource = SDSC-GORDON -- Service Provider = SDSC

2013-01-01 to 2013-03-31

Total NUs Charged by Field of Science

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

Total NUs Charged by User Institution

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

Total NUs Charged by PI

Resource = SDSC-GORDON

2013-01-01 to 2013-03-31

Appendix 15.9C Trestles SP-specific Metrics 2013-01-01 to 2013-03-31 Number of Jobs run by Job Size (Cores)

Average System Wait Time (Hours) by Job Size (Cores)

Average Requested Wall Time (Hours) by Job Size (Cores)

Average “Scheduler” Expansion Factor by Job Size (Cores) (Requested Wall Time + System Wait Time)/Requested Wall Time

228

Average Scheduler Expansion Factor by Requested Wall Time (Hours)

229 A XSEDE Project Milestones Update Starting with the next quarterly report this appendix will be renamed to “Progress Towards Goals” and will track the metric’s measurements.

230 B XSEDE Schedule with Progress Update

Page: 1

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Prodution: XSEDE Schedule 10/6/10 7/9/21 35.93%

XSEDE 10/6/10 7/9/21 35.93%

Project Office 10/6/10 11/3/16 36.38%

Project Management and Reporting 10/6/10 8/31/16 38.06%

Management 7/1/11 6/29/16 62.50%

Ongoing: Senior Management Team - 7/1/11 6/29/16 50.00% Oversight and management of XSEDE Ongoing: Business and finance office 7/1/11 6/29/16 50.00% support - subaward management Ongoing: Annual Planning, Budgeting, Change Control resulting from SEMP 7/1/11 6/29/16 50.00% Spiral Design Process

Update Project Execution Plan 7/2/12 10/31/12 100.00%

Project Reporting 10/6/10 8/31/16 33.33%

Prepare 2011 Q3 Quarterly Report 10/3/11 10/28/11 100.00%

Milestone: 2011 Q3 Quarterly Report 10/28/11 10/28/11 100.00% completed Prepare 2011 Q4 Quarterly Report 1/2/12 1/27/12 100.00%

Milestone: 2011 Q4 Quarterly Report 1/27/12 1/27/12 100.00% completed Prepare 2012 Q1 Quarterly Report 4/2/12 4/27/12 100.00%

Milestone: 2012 Q1 Quarterly Report 4/27/12 4/27/12 100.00% completed Prepare Y1 Annual Report 7/23/12 8/31/12 100.00%

Milestone: Y1 Annual Report completed 10/6/10 11/2/10 100.00%

Prepare 2012 Q3 Quarterly Report 10/1/12 10/26/12 100.00%

Milestone: 2012 Q3 Quarterly Report 7/28/11 7/28/11 100.00% completed Prepare 2012 Q4 Quarterly Report 1/1/13 1/28/13 100.00%

Milestone: 2012 Q4 Quarterly Report 10/28/11 10/28/11 100.00% completed

Prepare 2013 Q1 Quarterly Report 4/1/13 4/26/13 100.00%

Milestone: 2013 Q1 Quarterly Report 1/27/12 1/27/12 100.00% completed

Prepare Y2 Annual Report 7/22/13 8/30/13 0.00%

Milestone: Y2 Annual Report completed 11/1/13 11/1/13 0.00%

Prepare 2013 Q3 Quarterly Report 10/1/13 10/29/13 0.00%

Milestone: 2013 Q3 Quarterly Report 10/1/13 10/1/13 0.00% completed Prepare 2013 Q4 Quarterly Report 1/1/14 1/28/14 0.00%

Printed on Monday, April 15, 2013

231 Page: 2

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Milestone: 2013 Q4 Quarterly Report 10/28/11 10/28/11 0.00% completed Prepare 2014 Q1 Quarterly Report 4/1/14 4/28/14 0.00%

Milestone: 2014 Q1 Quarterly Report 1/27/12 1/27/12 0.00% completed

Prepare Y3 Annual Report 7/21/14 8/29/14 0.00%

Milestone: Y3 Annual Report completed 5/25/12 5/25/12 0.00%

Prepare 2014 Q3 Quarterly Report 10/1/14 10/28/14 0.00%

Milestone: 2014 Q3 Quarterly Report 7/28/11 7/28/11 0.00% completed

Prepare 2014 Q4 Quarterly Report 1/1/15 1/28/15 0.00%

Milestone: 2014 Q4 Quarterly Report 10/28/11 10/28/11 0.00% completed

Prepare 2015 Q1 Quarterly Report 4/1/15 4/28/15 0.00%

Milestone: 2015 Q1 Quarterly Report 1/27/12 1/27/12 0.00% completed Prepare Y4 Annual Report 7/20/15 8/31/15 0.00%

Milestone: Y4 Annual Report completed 5/25/12 5/25/12 0.00%

Prepare 2015 Q3 Quarterly Report 10/1/15 10/28/15 0.00%

Milestone: 2015 Q3 Quarterly Report 7/28/11 7/28/11 0.00% completed Prepare 2015 Q4 Quarterly Report 1/1/16 1/28/16 0.00%

Milestone: 2015 Q4 Quarterly Report 10/28/11 10/28/11 0.00% completed Prepare 2016 Q1 Quarterly Report 4/1/16 4/28/16 0.00%

Milestone: 2016 Q1 Quarterly Report 1/27/12 1/27/12 0.00% completed

Prepare Y5 Annual Report 7/25/16 8/31/16 0.00%

Milestone: Y5 Annual Report completed 5/25/12 5/25/12 0.00%

Prepare Final Report 7/25/16 8/31/16 0.00%

Milestone: Final Report completed 1/3/11 1/3/11 0.00%

Project Management - Risk Management 7/1/11 1/5/16 42.86%

Identify risks with Level 3 WBS 7/1/11 8/25/11 100.00% Managers Review risk register with Level 2 WBS 7/29/11 8/25/11 100.00% Managers Quarterly review of risk register - 2011 10/3/11 10/4/11 100.00% Q3 Quarterly review of risk register - 2011 1/2/12 1/3/12 100.00% Q4 Quarterly review of risk register - 2012 4/2/12 4/3/12 100.00% Q1

Printed on Monday, April 15, 2013

232 Page: 3

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Quarterly review of risk register - 2012 4/2/12 4/2/12 100.00% Q2 Quarterly review of risk register - 2012 7/2/12 7/2/12 100.00% Q3 Quarterly review of risk register - 2012 10/1/12 10/1/12 100.00% Q4 Quarterly review of risk register - 2013 1/7/13 1/7/13 100.00% Q1 Quarterly review of risk register - 2013 4/1/13 4/1/13 0.00% Q2 Quarterly review of risk register - 2013 7/1/13 7/1/13 0.00% Q3 Quarterly review of risk register - 2013 10/1/13 10/1/13 0.00% Q4 Quarterly review of risk register - 2014 1/6/14 1/6/14 0.00% Q1 Quarterly review of risk register - 2014 4/1/14 4/1/14 0.00% Q2 Quarterly review of risk register - 2014 7/1/14 7/1/14 0.00% Q3 Quarterly review of risk register - 2014 10/1/14 10/1/14 0.00% Q4 Quarterly review of risk register - 2015 1/5/15 1/5/15 0.00% Q1 Quarterly review of risk register - 2015 4/1/15 4/1/15 0.00% Q2 Quarterly review of risk register - 2015 7/1/15 7/1/15 0.00% Q3 Quarterly review of risk register - 2015 10/1/15 10/1/15 0.00% Q4 Quarterly review of risk register - 2016 1/5/16 1/5/16 0.00% Q1 Sciforma Administration 7/1/13 6/30/16 0.00%

Systems Engineering 7/2/12 6/28/13 55.00%

User needs collection 7/2/12 6/28/13 75.00%

Requirements analysis 7/2/12 6/28/13 75.00%

Managing the UREP 7/2/12 6/28/13 75.00%

Updating the Requirements Management 7/2/12 9/28/12 35.00% Plan Updating the SEMP 7/2/12 12/14/12 0.00%

XDOR/IDEALS: establishing 7/2/12 9/28/12 75.00% procedures/developing documentation

Management of XDOR process 6/28/13 6/28/13 50.00%

Systems Architecture 10/6/10 9/30/14 10.81%

Deploy Grid Middleware Infrastructure 3/1/11 3/28/11 0.00%

Milestone: Grid Middleware Infrastructure 10/6/10 10/6/10 0.00% deployed Deploy Data Management software 4/1/11 3/29/12 0.00%

Milestone: Data Management software 10/6/10 10/6/10 0.00% deployed Printed on Monday, April 15, 2013

233 Page: 4

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Deploy Account Management software 3/1/11 3/28/11 0.00%

Milestone: Account Management software 10/6/10 10/6/10 0.00% deployed

Deploy Information Services Infrastructure 3/1/11 3/28/11 0.00%

Milestone: Information Services 10/6/10 10/6/10 0.00% Infrastructure deployed Deploy Common User Environment 3/1/11 3/28/11 0.00%

Milestone: Common User Environment 10/6/10 10/6/10 0.00% deployed

Deploy System of Systems Test Environment 3/1/11 3/28/11 0.00%

Milestone: System of Systems Test 10/6/10 10/6/10 0.00% Environment deployed Spiral 1.0 1/3/11 12/21/11 0.00%

Inc 1.0 Start 1/3/11 1/3/11 0.00%

Refine Engineering Plan 1/3/11 1/21/11 0.00%

Prod. Baseline 1.0 Approved 1/3/11 1/3/11 0.00%

UNICORE Stack (1.0) 1/3/11 3/4/11 0.00%

Genesis II Stack (1.0) 1/3/11 2/25/11 0.00%

Execution Mgmt. (1.0) 1/3/11 2/25/11 0.00%

Execution Mon. Sys. (1.0) 1/3/11 1/28/11 0.00%

Information Serv. (1.0) 1/3/11 1/28/11 0.00%

GFFS Initial 1/3/11 1/11/11 0.00%

XWFS Initial 1/3/11 1/21/11 0.00%

Data Management (1.0) 1/3/11 3/25/11 0.00%

G&VO Management (1.0) 1/3/11 3/25/11 0.00%

API&CLT (1.0) 1/3/11 3/25/11 0.00%

GUI Tools (1.0) 1/3/11 1/28/11 0.00%

Replica Management Sys (1.0) 1/3/11 3/25/11 0.00%

AAM System (1.0) 1/3/11 1/28/11 0.00%

Ticket System (1.0) 1/3/11 1/28/11 0.00%

Ops Ctr Spt System (1.0) 1/3/11 1/28/11 0.00%

XSEDE Gateway Framework (1.0) 1/3/11 3/25/11 0.00%

Campus Interoperability Framework (1.0) 1/3/11 3/25/11 0.00%

Printed on Monday, April 15, 2013

234 Page: 5

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Investigate Identity Mgmt and Auth w 1/3/11 4/22/11 0.00% InCommon Security Mgmt and Services(1.0) 1/3/11 6/17/11 0.00%

Create Security PSG 9/1/11 10/12/11 0.00%

Approval of Security PSG 10/12/11 10/12/11 0.00%

XSEDE Certificate Authority 9/1/11 12/21/11 0.00%

Unified System Log (1.0) 1/3/11 1/28/11 0.00%

CM & Deployment Mgmt System (1.0) 1/3/11 1/21/11 0.00%

OSG Bridge (1.0) 1/3/11 2/25/11 0.00%

XSEDE Portal 1/3/11 3/25/11 0.00%

System Test Planning (1.0) 5/23/11 6/10/11 0.00%

Usability Panel Planning (1.0) 5/23/11 6/3/11 0.00%

TRR for Inc 1.0 7/4/11 7/5/11 0.00%

Inc 1.0 Integration 6/20/11 7/1/11 0.00%

Inc 1.0 System Test 7/6/11 8/2/11 0.00%

Inc 1.0 Spiral I&T Complete 8/2/11 8/2/11 0.00%

Inc 1.0 Deployment Planning 7/4/11 7/15/11 0.00%

Training Material Dev. (1.0) 3/28/11 5/20/11 0.00%

Inc 1.0 Deployment 8/3/11 8/16/11 0.00%

Inc 1.0 Deployment Complete 8/16/11 8/16/11 0.00%

XSEDE IOC 8/16/11 8/16/11 0.00%

Spiral 2.0 5/3/11 10/18/11 0.00%

Prod. Baseline 2.0 Approved 5/3/11 5/3/11 0.00%

Syslog Analysis Sys (2.0) 5/3/11 5/3/11 0.00%

AAM System (2.0) 5/3/11 6/27/11 0.00%

Execution Monitoring Sys (2.0) 5/3/11 6/27/11 0.00%

RQOSA Mgmt Sys (2.0) 5/3/11 7/25/11 0.00%

Security Mgmt and Services (2.0) 5/3/11 8/22/11 0.00%

API&CLT (2.0) 5/3/11 6/27/11 0.00%

GUI Tools (2.0) 5/3/11 5/30/11 0.00%

CM & Deploy Mgmt (2.0) 5/3/11 5/30/11 0.00%

Printed on Monday, April 15, 2013

235 Page: 6

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Campus Interop Fwrk (2.0) 5/3/11 7/25/11 0.00%

Ticket System (2.0) 5/3/11 6/27/11 0.00%

XSEDE Gateway Framework (2.0) 5/3/11 7/25/11 0.00%

Replica Mgmt (2.0) 5/3/11 7/25/11 0.00%

File System (2.0) 5/3/11 8/22/11 0.00%

Ops Ctr Spt Sys (2.0) 5/3/11 5/3/11 0.00%

Execution Mgmt (2.0) 5/3/11 6/27/11 0.00%

Info Services (2.0) 5/3/11 8/22/11 0.00%

Foreign Grid Policy Adapters (2.0) 5/3/11 6/13/11 0.00%

Bandwidth Provisioning (2.0) 5/3/11 7/25/11 0.00%

XSEDE Portal (2.0) 5/3/11 6/27/11 0.00%

G&VO Mgmt (2.0) 5/3/11 7/25/11 0.00%

Collab. Tools (2.0) 5/3/11 5/30/11 0.00%

Data Management (2.0) 5/3/11 7/25/11 0.00%

Inc 2.0 Integration 8/23/11 9/5/11 0.00%

Inc 2.0 System Test 9/7/11 9/20/11 0.00%

Training Material Dev. (2.0) 5/3/11 6/27/11 0.00%

System Test Planning (2.0) 8/15/11 9/2/11 0.00%

Usability Panel Planning (2.0) 8/22/11 8/26/11 0.00%

TRR for Inc 2.0 9/6/11 9/6/11 0.00%

Inc 2.0 Deployment Planning 9/21/11 10/4/11 0.00%

Inc 2.0 Deployment 10/5/11 10/18/11 0.00%

Inc 2.0 Deployment Complete 10/18/11 10/18/11 0.00%

Inc 2.0 Spiral I&T Complete 9/20/11 9/20/11 0.00%

Spiral 3.0 10/18/11 4/4/12 0.00%

Prod. Baseline 3.0 Approved 10/18/11 10/18/11 0.00%

Execution Mgmt (3.0) 10/18/11 12/12/11 0.00%

G&VO Mgmt (3.0) 10/18/11 1/9/12 0.00%

Data Managementt (3.0) 10/18/11 12/12/11 0.00%

Info Services (3.0) 10/18/11 2/6/12 0.00%

Printed on Monday, April 15, 2013

236 Page: 7

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Replica Mgmt Sys (3.0) 10/18/11 10/18/11 0.00%

Bandwidth Provisioning (3.0) 10/18/11 10/18/11 0.00%

RQOSA Mgmt (3.0) 10/18/11 10/18/11 0.00%

AAM Sys (3.0) 10/18/11 12/12/11 0.00%

Execution Mon Sys (3.0) 10/18/11 11/14/11 0.00%

Collab Tools (3.0) 10/18/11 11/14/11 0.00%

API&CLT (3.0) 10/18/11 11/14/11 0.00%

GUI Tools (3.0) 10/18/11 10/31/11 0.00%

File System (3.0) 10/18/11 1/9/12 0.00%

Ops Ctr Support Sys (3.0) 10/18/11 10/18/11 0.00%

Syslog Analysis Sys (3.0) 10/18/11 10/31/11 0.00%

XSEDE Portal (3.0) 10/18/11 11/14/11 0.00%

System Test Planning (3.0) 2/1/12 2/21/12 0.00%

Usability Panel Planning (3.0) 2/7/12 2/13/12 0.00%

TRR for Inc 3.0 2/22/12 2/22/12 0.00%

Security Mgmt and Services (3.0) 10/18/11 1/9/12 0.00%

Inc 3.0 Integration 2/7/12 2/20/12 0.00%

Inc 3.0 System Test 2/23/12 3/7/12 0.00%

Inc 3.0 Spiral I&T Complete 3/7/12 3/7/12 0.00%

Training Material Dev. (3.0) 11/7/11 12/30/11 0.00%

Inc 3.0 Deployment Planning 3/8/12 3/21/12 0.00%

Inc 3.0 Deployment 3/22/12 4/4/12 0.00%

Inc 3.0 Deployment Complete 4/4/12 4/4/12 0.00%

XSEDE FOC 4/4/12 4/4/12 0.00%

Ongoing: Incremental improvements 10/6/10 9/30/14 0.00% continue via SEMP Spiral Design Process

Public Facing XSEDE Architecture 12/5/11 3/5/12 100.00% Document Agreement of contents and level of 12/5/11 12/8/11 100.00% detail Establish time frame to produce public 12/9/11 1/5/12 100.00% facing architecture document Outline for initial level 1 & level 1/6/12 1/12/12 100.00% 2-decomposition documentation First draft of public facing document 1/13/12 2/1/12 100.00%

Printed on Monday, April 15, 2013

237 Page: 8

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Revise, comment add content to 2/2/12 2/9/12 100.00% document as necessary Architects review first draft with 2/10/12 2/10/12 100.00% Bachman Identify remaining steps to complete 2/13/12 2/16/12 100.00% first draft A&D team including liaisons from SD&I, Security, Campus Bridging and 2/17/12 2/27/12 100.00% management review first draft Architects address comments/revisions and request endorsement from A&D 2/28/12 3/1/12 100.00% team and liaisons First version of XSEDE Architecture Document (Level 1 & 2 Decomp) 3/2/12 3/5/12 100.00% available at Qrtrly Mtg Campus Bridging - Architectural Response 1/19/12 5/31/12 65.83% to Stakeholder Requirements

Preliminary background work 1/19/12 2/16/12 100.00%

Documentation and review of use cases 2/17/12 2/23/12 100.00% and requirements matrix completed Architectural response at a Level 3 2/24/12 3/22/12 100.00% Decomposition prepared by the architectsStakeholder review of Architectural 3/23/12 4/19/12 95.00% response Active Design Review 4/20/12 5/3/12 0.00%

Incorporation into public facing 5/4/12 5/31/12 0.00% XSEDE Architecture Document Science Gateways - Architectural Response 5/4/12 9/6/12 20.00% to Stakeholder Requirements Documentation and review of use cases 5/4/12 5/31/12 100.00% and requirements matrix completed Architectural response at a Level 3 6/1/12 6/28/12 0.00% Decomposition prepared by the Stakeholderarchitects review of Architectural 6/29/12 7/26/12 0.00% response Active Design Review 7/27/12 8/9/12 0.00%

Incorporation into public facing 8/10/12 9/6/12 0.00% XSEDE Architecture Document Computing - Architectural Response to 8/10/12 12/13/12 16.00% Stakeholder Requirements Documentation and review of use cases 8/10/12 9/6/12 80.00% and requirements matrix completed Architectural response at a Level 3 9/7/12 10/4/12 0.00% Decomposition prepared by the Stakeholderarchitects review of Architectural 10/5/12 11/1/12 0.00% response

Active Design Review 11/2/12 11/15/12 0.00%

Incorporation into public facing 11/16/12 12/13/12 0.00% XSEDE Architecture Document BIG Data - Architectural Response to 11/16/12 4/4/13 15.00% Stakeholder Requirements Documentation and review of use cases 11/16/12 12/13/12 75.00% and requirements matrix completed

Printed on Monday, April 15, 2013

238 Page: 9

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Architectural response at a Level 3 12/14/12 1/25/13 0.00% Decomposition prepared by the Stakeholderarchitects review of Architectural 1/28/13 2/21/13 0.00% response Active Design Review 2/22/13 3/7/13 0.00%

Incorporation into public facing 3/8/13 4/4/13 0.00% XSEDE Architecture Document Connecting Instrumentation - Architectural 3/8/13 7/11/13 1.00% Response to Stakeholder Requirements Documentation and review of use cases 3/8/13 4/4/13 5.00% and requirements matrix completed Architectural response at a Level 3 4/5/13 5/2/13 0.00% Decomposition prepared by the Stakeholderarchitects review of Architectural 5/3/13 5/30/13 0.00% response Active Design Review 5/31/13 6/13/13 0.00%

Incorporation into public facing 6/14/13 7/11/13 0.00% XSEDE Architecture Document Collaboration - Architectural Response to 6/14/13 10/17/13 1.00% Stakeholder Requirements Documentation and review of use cases 6/14/13 7/11/13 5.00% and requirements matrix completed Architectural response at a Level 3 7/12/13 8/8/13 0.00% Decomposition prepared by the Stakeholderarchitects review of Architectural 8/9/13 9/5/13 0.00% response

Active Design Review 9/6/13 9/19/13 0.00%

Incorporation into public facing 9/20/13 10/17/13 0.00% XSEDE Architecture Document XSEDE Architectural Canonical Use Cases 6/14/13 10/17/13 16.00%

Documentation and review of use cases 6/14/13 7/11/13 30.00% and requirements matrix completed Architectural response at a Level 3 7/12/13 8/8/13 30.00% Decomposition prepared by the Stakeholderarchitects review of Architectural 8/9/13 9/5/13 20.00% response Active Design Review 9/6/13 9/19/13 0.00%

Incorporation into public facing 9/20/13 10/17/13 0.00% XSEDE Architecture Document External Relations 10/6/10 11/3/16 64.52%

Generate Publications: Highlights (Science, 4/2/12 11/3/16 62.73% EOT, Digital Resources) SciHi subcommittee from XSEDE ER 4/9/12 4/9/12 100.00% established Collect story ideas 5/25/12 5/25/12 100.00%

Story choices approved by XSEDE 6/29/12 6/29/12 100.00% leadership Graphic designer selected 7/2/12 7/2/12 100.00%

Printed on Monday, April 15, 2013

239 Page: 10

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE About 15 science highlights stories selected, edited (incl tech review) and 7/27/12 7/27/12 100.00% shared w sr leadership

Cover-to-cover edit complete 8/10/12 8/10/12 100.00%

Overall design and test story mockup 9/5/12 9/5/12 90.00% complete and reviewed

Final design complete 9/20/12 9/20/12 0.00%

Completed book delivered to printer 9/21/12 9/21/12 0.00%

Milestone: Science Highlights 11/21/12 11/21/12 0.00% published Ongoing: Repeat previous 10 tasks 4/2/12 11/3/16 0.00% annually Create XSEDE website and translate 1/3/11 4/18/11 100.00% relevant TG website content XSEDE WebsiXSEDE website 1/3/11 1/3/11 100.00% committee established Website requirements document 1/17/11 1/17/11 100.00% complete Content requirements document 1/17/11 1/17/11 100.00% complete First rev of design reviewed by website 2/15/11 2/15/11 100.00% committee Rev of website reviewed by XSEDE 3/1/11 3/1/11 100.00% leadership

Short form usability test completed 3/15/11 3/15/11 100.00%

Final version of website reviewed by 3/22/11 3/22/11 100.00% website committee Content approved 4/1/11 4/1/11 100.00%

Final build complete 4/7/11 4/7/11 100.00%

Content ported and built 4/15/11 4/15/11 100.00%

Initial version of XSEDE website 4/18/11 4/18/11 100.00% launched Milestone: XSEDE website completed 4/18/11 4/18/11 100.00%

Generate Annual Conference Proceedings 10/6/10 10/7/14 48.00%

Genereate Annual Conference 2012 3/1/12 7/12/12 100.00%

Proceedings chair selected and 3/1/12 3/1/12 100.00% incorporated into event planning Submission site publicized 3/1/12 3/1/12 100.00%

Approval from ACM or whatever 6/1/12 6/1/12 100.00% publisher received Templates and copyright permission 6/5/12 6/5/12 100.00% forms out to authors All papers and copyright permission forms received from authors and 6/22/12 6/22/12 100.00% OKed by track chairs All papers sent to production house 6/25/12 6/25/12 100.00% for reproduction to USB drive

Printed on Monday, April 15, 2013

240 Page: 11

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Milestone: Annual Conference 7/12/12 7/12/12 100.00% Proceedings Complete Genereate Annual Conference 2013 3/1/13 7/12/13 0.00%

Proceedings chair selected and 3/1/13 3/1/13 0.00% incorporated into event planning

Submission site publicized 3/1/13 3/1/13 0.00%

Approval from ACM or whatever 6/1/13 6/3/13 0.00% publisher received Templates and copyright permission 6/5/13 6/5/13 0.00% forms out to authors All papers and copyright permission forms received from authors and 6/22/13 6/24/13 0.00% OKed by track chairs All papers sent to production house 6/25/13 6/25/13 0.00% for reproduction to USB drive Milestone: Annual Conference 7/12/13 7/12/13 0.00% Proceedings Complete Ongoing: Repeat previous 7 tasks 10/6/10 10/7/14 20.00% annually Milestone: Ongoing - Generate press 4/4/11 3/24/16 25.00% releases & website content Milestone: Ongoing - Generate publicity via 1/3/12 3/24/16 25.00% social media Monthly: Gather, edit and format content for internal (plain text) e-newsletter; distribute to 1/10/12 6/9/16 25.00% XSED Milestone: Ongoing - Generate monthly 1/10/12 6/9/16 25.00% internal e-newsletter Monthly: Gather, edit and format content for external (HTML/designed) e-newsletter; 1/25/12 6/23/16 25.00% distribute to m Milestone: Ongoing - Generate monthly 1/25/12 6/23/16 25.00% external e-newsletter Industry Relations 10/6/10 6/30/16 0.25%

Workforce Development 7/1/11 6/30/16 0.00%

Increase industry partners' awareness of 7/1/11 6/30/16 0.00% all XSEDE SP's training opportunites Elicit industry partners' input to enhance training programs for workforce 7/1/11 6/30/16 0.00% development Software Development 10/6/10 10/2/12 0.50%

Hold conference call with XAB to flesh 7/12/12 7/12/12 1.00% out software development activity Select and execute the software 10/6/10 10/2/12 0.00% development project SD&I 9/1/11 1/1/14 82.74%

PDR 9/1/11 9/1/11 100.00%

Increment Planning 9/1/11 9/1/11 100.00%

develop incrment plan 9/1/11 9/1/11 100.00%

Printed on Monday, April 15, 2013

241 Page: 12

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

IRR 9/7/11 9/7/11 100.00%

conduct IRR 9/7/11 9/7/11 100.00%

IRR complete and passed 9/7/11 9/7/11 100.00%

CI Detailed Design 9/8/11 9/8/11 100.00%

develop detail design 9/8/11 9/8/11 100.00%

GFFS 9/8/11 9/8/11 100.00%

Execution Management 9/8/11 9/8/11 100.00%

XUAS Data 9/8/11 9/8/11 100.00%

CDR 9/9/11 9/15/11 100.00%

conduct CDR 9/9/11 9/9/11 100.00%

CDR complete and passed 9/15/11 9/15/11 100.00%

CI Development 9/16/11 9/16/11 100.00%

develop CI 9/16/11 9/16/11 100.00%

GFFS 9/16/11 9/16/11 100.00%

Execution Management 9/16/11 9/16/11 100.00%

XUAS Data 9/16/11 9/16/11 100.00%

CI TRR 9/19/11 10/7/11 100.00%

conduct CI TRR 9/19/11 9/19/11 100.00%

GFFS 9/19/11 9/19/11 100.00%

Execution Management 9/19/11 9/19/11 100.00%

XUAS Data 9/19/11 9/19/11 100.00%

CI TRR complete and passed 10/7/11 10/7/11 100.00%

CI Tests 10/10/11 12/15/11 100.00%

conduct Ci tests 10/10/11 10/10/11 100.00%

GFFS 10/10/11 10/10/11 100.00%

Execution Management 10/10/11 10/10/11 100.00%

XUAS Data 10/10/11 10/10/11 100.00%

CI Tests complete and passed 12/15/11 12/15/11 100.00%

STRR 12/16/11 12/16/11 100.00%

conduct STRR 12/16/11 12/16/11 100.00%

Printed on Monday, April 15, 2013

242 Page: 13

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

STRR complete and passed 12/16/11 12/16/11 100.00%

System Integration Test 12/19/11 12/28/11 100.00%

conduct system test 12/19/11 12/19/11 100.00%

System test complete and passed 12/28/11 12/28/11 100.00%

ORR 12/29/11 12/29/11 100.00%

conduct ORR 12/29/11 12/29/11 100.00%

ORR complete and passed 12/29/11 12/29/11 100.00%

Increment De-brief 12/30/11 1/6/12 100.00%

increment reflection workshop 12/30/11 12/30/11 100.00%

reflection and practice report 1/2/12 1/2/12 100.00%

de-brief complete 1/6/12 1/6/12 100.00%

Implement Open, Continuous Planning 7/2/12 6/28/13 100.00%

Implement Continuous Development and 7/2/12 6/28/13 100.00% Integration

Implement Engineering Improvements 7/2/12 6/28/13 100.00%

SDIACT-010 - Deliver Operational Tests with 4/19/12 6/29/12 50.00% Cis SDIACT-015 - Genesis II/UNICORE 6 GAML 4/3/12 7/2/12 100.00% SAML SDIACT-018 - Replicated/Synchronized 4/18/12 6/18/12 100.00% stateful resource SDIACT-028 - (CANCELED): GO Transfer 4/25/12 7/2/12 100.00% REST API as XSEDE Production service SDIACT-031 - Improve GridFTP for SPs 4/25/12 7/2/12 100.00%

SDIACT-043 - Genesis II Documentation 4/30/12 7/2/12 100.00%

SDIACT-044 - (CANCELED, MERGED with 4/30/12 7/2/12 100.00% SDIACT-100): Campus bridging beta support SDIACT-049 - Link Globus Online into 4/25/12 7/2/12 100.00% XSEDE User Portal SDIACT-050 - MyProxy OAuth Limited Proxy 4/20/12 7/2/12 70.00% Support SDIACT-054 - Register new increment 1 4/19/12 6/5/12 75.00% components SDIACT-073 - System information 4/24/12 6/29/12 20.00% publishing pilot SDIACT-075 - GridFTP in UNICORE 6 5/1/12 7/2/12 100.00%

SDIACT-096 - Identify XSEDE not TeraGrid in resource names for new OSG resource 3/22/12 5/9/12 100.00% integration SDIACT-097 - Basic Execution Service 5/2/12 6/25/12 100.00%

Printed on Monday, April 15, 2013

243 Page: 14

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE SDIACT-100 - Globus Online Increment 2 4/25/12 7/2/12 100.00% addressing security concerns SDIACT-101 - EMS and GFFS Increment 2 5/8/12 7/2/12 100.00% updates SCIDACT-071 - Improve cmd line single 3/27/12 6/29/12 100.00% sign-on access SDIACT-070 - SSO Hub (CLI) 11/9/12 2/28/13 20.00%

SDIACT-102 - Replace tgusage w/ xdusage 10/4/12 12/14/12 100.00%

SDIACT-103 - Prototype Acct Mgmt w/ JIRA 11/26/12 1/14/13 65.00%

SDIACT-121 - Activity workspace & process 11/21/12 2/20/13 40.00% imp.

SDIACT-106 Globus Transfer REST APIv1.0 12/5/12 2/14/13 10.00%

SDIACT-108 Globus Connect Multi-user 1.0 12/5/12 2/15/13 10.00%

SDIACT-003 CA Certificate Installer 12/11/12 2/28/13 10.00%

SDIACT-110 GenesisII GAML to SAML 1/22/13 4/19/13 20.00% Migration SDIACT-099 System-wide Genesis II 1/15/13 2/11/13 100.00% installer SDIACT-115 (EMS, Genesis II) API for 1/24/13 3/27/13 10.00% Science Gateways SDIACT-112 Replicated Kerberos-aware 2/11/13 4/15/13 10.00% STS SDIACT-123 Integrated EMS and GFFS 1/15/13 5/31/13 10.00% Increment 3 Update SDIACT-107 GLUE2 Update 3/1/13 1/1/14 10.00%

Operations 10/6/10 7/1/16 49.54%

User Services 1/3/11 7/9/21 25.21%

Training 1/3/11 6/30/16 16.80%

Milestone: Develop guidelines for online and in-person training materials across 3/1/11 3/28/11 0.00% XSEDE Sites Milestone: Develop 10 Training Modules 4/1/11 6/30/16 15.00% Annually Milestone: Develop 10 Training 4/1/11 3/30/12 75.00% Modules Year 1 Develop and Post 2 new online 4/1/11 7/1/11 0.00% training tutorials Develop and Post 2 new online 7/5/11 9/29/11 100.00% training tutorials Develop and Post 3 new online 10/3/11 12/30/11 100.00% training tutorials Develop and Post 3 new online 1/4/12 3/30/12 100.00% training tutorials Milestone: Develop 10 Training 7/2/12 6/28/13 0.00% Modules Year 2 Develop and Post 2 new online 7/2/12 9/28/12 0.00% training tutorials

Printed on Monday, April 15, 2013

244 Page: 15

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Develop and Post 2 new online 10/1/12 12/31/12 0.00% training tutorials Develop and Post 3 new online 1/1/13 3/29/13 0.00% training tutorials Develop and Post 3 new online 4/1/13 6/28/13 0.00% training tutorials Milestone: Develop 10 Training 7/1/13 6/30/14 0.00% Modules Year 3 Develop and Post 2 new online 7/1/13 9/30/13 0.00% training tutorials Develop and Post 2 new online 10/1/13 12/31/13 0.00% training tutorials Develop and Post 3 new online 1/1/14 3/31/14 0.00% training tutorials Develop and Post 3 new online 4/1/14 6/30/14 0.00% training tutorials Milestone: Develop 10 Training 7/1/14 6/30/16 0.00% Modules Year 4 Develop and Post 2 new online 7/1/14 9/30/14 0.00% training tutorials Develop and Post 2 new online 10/1/14 12/31/14 0.00% training tutorials Develop and Post 3 new online 1/1/15 3/31/15 0.00% training tutorials Develop and Post 3 new online 4/1/16 6/30/16 0.00% training tutorials Milestone: Develop 10 Training 7/1/15 6/30/16 0.00% Modules Year 5 Develop and Post 2 new online 7/1/15 9/30/15 0.00% training tutorials Develop and Post 2 new online 10/1/15 12/31/15 0.00% training tutorials Develop and Post 3 new online 1/1/16 3/31/16 0.00% training tutorials Develop and Post 3 new online 4/1/16 6/30/16 0.00% training tutorials

Conduct 50 Training Sessions Annually 4/1/11 6/30/16 18.75%

Milestone: Conduct 50 Training 4/1/11 3/30/12 93.76% sessions Year 1 Conduct first 10 in-person or 4/1/11 7/1/11 100.00% webcast training sessions Conduct first 15 in-person or 7/5/11 12/30/11 100.00% webcast training sessions Conduct first 10 in-person or 10/3/11 12/30/11 75.05% webcast training sessions Conduct first 15 in-person or 1/4/12 3/30/12 100.00% webcast training sessions Milestone: Conduct 50 Training 7/2/12 6/28/13 0.00% sessions Year 2 Conduct first 10 in-person or 7/2/12 9/28/12 0.00% webcast training sessions Conduct first 15 in-person or 10/1/12 12/31/12 0.00% webcast training sessions Conduct first 10 in-person or 1/1/13 3/29/13 0.00% webcast training sessions

Printed on Monday, April 15, 2013

245 Page: 16

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Conduct first 15 in-person or 4/1/13 6/28/13 0.00% webcast training sessions Milestone: Conduct 50 Training 7/1/13 6/30/14 0.00% sessions Year 3 Conduct first 10 in-person or 7/1/13 9/30/13 0.00% webcast training sessions Conduct first 15 in-person or 10/1/13 12/31/13 0.00% webcast training sessions Conduct first 10 in-person or 1/1/14 3/28/14 0.00% webcast training sessions Conduct first 15 in-person or 4/1/14 6/30/14 0.00% webcast training sessions Milestone: Conduct 50 Training 7/1/14 6/30/15 0.00% sessions Year 4 Conduct first 10 in-person or 7/1/14 9/30/14 0.00% webcast training sessions Conduct first 15 in-person or 10/1/14 12/31/14 0.00% webcast training sessions Conduct first 10 in-person or 1/1/15 3/31/15 0.00% webcast training sessions Conduct first 15 in-person or 4/1/15 6/30/15 0.00% webcast training sessions Milestone: Conduct 50 Training 7/1/15 6/30/16 0.00% sessions Year 5 Conduct first 10 in-person or 7/1/15 9/30/15 0.00% webcast training sessions Conduct first 15 in-person or 10/1/15 12/31/15 0.00% webcast training sessions Conduct first 10 in-person or 1/1/16 3/31/16 0.00% webcast training sessions Conduct first 15 in-person or 4/1/16 6/30/16 0.00% webcast training sessions Milestone: Complete Federation of existing online training materials with XSEDE 1/3/11 12/30/11 0.00% Repositories Milestone: Complete 2 targeted community 7/1/11 6/30/16 20.00% workshops annually Conduct first targeted community 7/1/11 12/30/11 100.00% workshop Year 1 Conduct second targeted community 1/2/12 6/29/12 100.00% workshop Year 1 Conduct first targeted community 7/2/12 12/31/12 0.00% workshop Year 2 Conduct second targeted community 1/1/13 6/28/13 0.00% workshop Year 2 Conduct first targeted community 7/1/13 12/31/13 0.00% workshop Year 3 Conduct second targeted community 1/1/14 6/2/14 0.00% workshop Year 3 Conduct first targeted community 7/1/14 12/31/14 0.00% workshop Year 4 Conduct second targeted community 1/1/15 6/30/15 0.00% workshop Year 4 Conduct first targeted community 7/1/15 12/31/15 0.00% workshop Year 5

Printed on Monday, April 15, 2013

246 Page: 17

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Conduct second targeted community 1/1/16 6/30/16 0.00% workshop Year 5 Milestone: Conduct 4 technical training, content-based and mentoring webinars in 7/1/11 6/30/16 20.00% support of the XSE Conduct 4 webinars in support of 7/1/11 6/29/12 100.00% XSEDE Scholars Program Year 1 Conduct 4 webinars in support of 7/2/12 6/28/13 0.00% XSEDE Scholars Program Year 2 Conduct 4 webinars in support of 7/2/13 6/30/14 0.00% XSEDE Scholars Program Year 3 Conduct 4 webinars in support of 7/1/14 6/30/15 0.00% XSEDE Scholars Program Year 4 Conduct 4 webinars in support of 7/1/15 6/30/16 0.00% XSEDE Scholars Program Year 5

Portfolio Review 1/3/11 1/3/11 100.00%

Focus areas 1/3/11 12/30/11 0.00%

Support of new systems (Stampede, 1/3/11 7/4/11 0.00% Gordon, Keeneland, Blue Waters)

XSEDE architecture and tools 1/3/11 12/30/11 0.00%

Security 1/3/11 7/4/11 0.00%

Train the trainers 1/3/11 7/4/11 0.00%

Trainiing for non-traditional areas 1/3/11 12/30/11 0.00%

Portal to API for gateways 7/2/12 6/28/13 0.00%

User Information Resources 1/3/11 6/30/16 47.14%

Milestone: Release Production User Portal 4/1/11 7/1/11 100.00% & Web Site Transition existing production portal 4/1/11 7/1/11 100.00% capabilities and web site

Define user information architecture 4/1/11 6/23/11 100.00%

Milestone: Maintain Production User News 4/1/11 7/1/11 100.00% System Transition existing user news system to 4/1/11 7/1/11 100.00% production Milestone: Release Allocation & User 4/1/11 10/11/11 100.00% Guide for New and Transitioning Users Document instructions for new users 4/1/11 7/1/11 100.00% coming to XSEDE

Document instructions for existing users 7/4/11 9/5/11 100.00%

Document allocation policies for 9/6/11 10/11/11 100.00% resources Milestone: Release new user guide with 4/1/11 7/1/11 100.00% user comment capabilities Create user guide template for HPC, 4/1/11 6/23/11 100.00% Viz, Storage, etc. Create user guide examples for each 4/1/11 6/23/11 100.00% resource type

Printed on Monday, April 15, 2013

247 Page: 18

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Ensure all user guides have been 4/1/11 7/1/11 100.00% transitioned to template Publish all user guides across web 4/1/11 7/1/11 100.00% presence Milestone: Production mobile user portal 4/1/11 6/30/11 56.61%

Transition existing mobile framework 4/1/11 4/28/11 100.00%

Evaluate requirements for mobile 4/29/11 6/22/11 49.84% features Create schedule for releasing future 4/29/11 6/30/11 20.00% mobile features

Milestone: Release updated user news 7/1/11 9/22/11 100.00%

Define requirements for user news 7/1/11 7/28/11 100.00% system Evaluate existing and alternative 7/29/11 8/11/11 100.00% technologies Release new user news system (if 8/12/11 9/22/11 100.00% appropriate) Milestone: Create new social media 10/3/11 9/26/13 12.53% presence for XD

Define requirements for social media 10/3/11 9/3/12 50.10%

Evaluate requirements that come out of 10/31/11 9/26/13 0.00% User Engagement Create twitter and facebook presence 7/2/12 12/31/12 0.00% for users of XD and specifically XUP users Display twitter feeds on user portal 7/2/12 12/31/12 0.00%

Milestone: Release collaborative 7/2/12 12/31/12 0.00% capabilities to user portal Define requirements based on User 7/2/12 12/31/12 0.00% Engagement feedback Enable users to be able to share calendars, chat, files, etc. with 7/2/12 12/31/12 0.00% collaborators via user portal Milestone: Release integrated training 1/2/12 3/23/12 100.00% system Define requirements for integrated training system based on TEOS and 1/2/12 1/27/12 100.00% User Requiremenst Enable sites to post training courses on 1/30/12 3/23/12 100.00% user portal Enable sites to add online training 1/30/12 3/23/12 100.00% resources to user portal Enable users to register for training 1/30/12 3/23/12 100.00% online via user portal Create one stop shop for user training on user portal with calendar, SMS 1/30/12 3/23/12 100.00% notification, online regi Resource Selector 1/3/11 9/2/11 19.81%

Ongoing: Develop and implement improvements based on SEMP Spiral 1/3/11 6/30/16 25.97% Design Process XSEDE User Portal 1/3/11 6/30/16 26.15%

Printed on Monday, April 15, 2013

248 Page: 19

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Redesign dock at the top and apply 1/2/12 3/2/12 100.00% theme Update profile portlet - expand with 1/2/12 6/29/12 100.00% picture, publications,etc. Link checker for XUP & XSEDE web 1/2/12 3/30/12 0.00% site Add forget username feature 1/2/12 4/30/12 100.00%

Integrate future grid status in XUP 1/2/12 6/29/12 0.00% system monitor Expand system status beyond 1/2/12 6/29/12 0.00% up/down/etc. Implement new News categories 1/2/12 3/30/12 100.00%

GridShib/InCommon integration 1/2/12 9/28/12 0.00%

Enable dynamic feedback on each 1/2/12 3/30/12 100.00% page Migrate TGU staff queries to XUP 1/2/12 3/30/12 75.00% staff area

Compete guest homepage redesign 1/2/12 6/29/12 0.00%

Disable/gray out login link when 4/2/12 6/29/12 0.00% resources are down Merge add/remove user page and 7/2/12 6/30/16 0.00% allocation page Look at giving gateways a different 7/2/12 6/30/16 0.00% 'view' for their community allocations Merge DN listing with user profile 7/2/12 6/30/16 0.00%

Integrate new ticketing system 7/2/12 12/31/12 0.00%

Chat for help 7/2/12 6/30/16 0.00%

Make hot links/bookmarking feature 7/2/12 6/30/16 0.00%

Screen sharing with support staff 7/2/12 6/30/16 0.00%

Expand allocations/usage/job 7/2/12 6/30/16 100.00% history with graphs

Integrate XDMoD services 7/2/12 6/30/16 19.99%

Custom views for communities: campus champions, gateways, 1/3/11 1/1/15 7.99% fields of science

Network connectivity monitor 7/2/12 6/30/16 0.00%

Dynamic visualization information 7/2/12 6/30/16 0.00% in system monitor Remote visualization services 7/2/12 6/30/16 0.00%

RSS feeds and SMS notifications 7/2/12 6/30/16 50.00% for user portal functions OSG documentation integration 7/2/12 6/30/16 100.00%

Videos & interactive guide on 7/2/12 6/30/16 0.00% features of XUP

Printed on Monday, April 15, 2013

249 Page: 20

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Online training for XUP 7/2/12 6/30/16 0.00%

Online survey for users on features 7/2/12 6/30/16 0.00%

Workflow management interface 7/2/12 6/30/16 0.00%

Metascheduling, job reservations, 7/2/12 6/30/16 0.00% and ensemble job submission Change community account form to enter information in XCDB and 7/2/12 6/30/16 10.00% approve via portal Ongoing: Prioritize and integrate SD&I 1/3/11 12/31/15 20.00% configuration items in US UII User Engagement 4/1/11 7/9/21 21.39%

Annual User Surveys 1/2/12 6/30/16 10.00%

Develop and Implement Y1 Annual 1/2/12 3/30/12 100.00% User Survey Milestone: Y1 Annual User Survey 7/2/12 7/2/12 0.00% report Develop and Implement Y2 Annual 1/1/13 4/1/13 0.00% User Survey Milestone: Y2 Annual User Survey 7/1/13 7/1/13 0.00% report Develop and Implement Y3 Annual 1/1/14 3/31/14 0.00% User Survey Milestone: Y3 Annual User Survey 6/30/14 6/30/14 0.00% report Develop and Implement Y4 Annual 1/1/15 3/31/15 0.00% User Survey Milestone: Y4 Annual User Survey 6/30/15 6/30/15 0.00% report Develop and Implement Y5 Annual 1/1/16 3/31/16 0.00% User Survey Milestone: Y5 Annual User Survey 6/30/16 6/30/16 0.00% report XSEDE CRM 1/2/12 7/2/12 0.00%

Document current XSEDE activities 1/2/12 1/2/12 0.00% athat act as CRM Design and Implemnt initial CRM 1/2/12 6/29/12 0.00% system Milestone: Initial CRM system deployed 7/2/12 7/2/12 0.00%

Data Mining 10/3/11 7/1/16 13.16%

Q1Y1 Data mining operations 10/3/11 12/30/11 100.00%

Milestone: Q1Y1 Data mining report 12/30/11 12/30/11 100.00%

Q2Y1 Data mining operations 1/2/12 3/30/12 100.00%

Milestone: Q2Y1 Data mining report 4/2/12 4/2/12 100.00%

Q3Y1 Data mining operations 4/3/12 7/2/12 100.00%

Milestone: Q3Y1 Data mining report 7/2/12 7/2/12 0.00%

Printed on Monday, April 15, 2013

250 Page: 21

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Q4Y1 Data mining operations 7/3/12 10/1/12 0.00%

Milestone: Q4Y1 Data mining report 10/2/12 10/2/12 0.00%

Q1Y2 Data mining operations 10/2/12 12/31/12 0.00%

Milestone: Q1Y2 Data mining report 1/1/13 1/1/13 0.00%

Q2Y2 Data mining operations 1/2/13 4/2/13 0.00%

Milestone: Q2Y2 Data mining report 4/2/13 4/2/13 0.00%

Q3Y2 Data mining operations 4/1/13 6/28/13 0.00%

Milestone: Q3Y2 Data mining report 7/1/13 7/1/13 0.00%

Q4Y2 Data mining operations 7/1/13 9/30/13 0.00%

Milestone: Q4Y2 Data mining report 10/1/13 10/1/13 0.00%

Q1Y3 Data mining operations 10/1/13 12/31/13 0.00%

Milestone: Q1Y3 Data mining report 1/1/14 1/1/14 0.00%

Q2Y3 Data mining operations 1/1/14 3/31/14 0.00%

Milestone: Q2Y3 Data mining report 4/1/14 4/1/14 0.00%

Q3Y3 Data mining operations 4/1/14 6/30/14 0.00%

Milestone: Q3Y3 Data mining report 7/1/14 7/1/14 0.00%

Q4Y3 Data mining operations 7/1/14 9/30/14 0.00%

Milestone: Q4Y3 Data mining report 10/1/14 10/1/14 0.00%

Q1Y4 Data mining operations 10/1/14 12/31/14 0.00%

Milestone: Q1Y4 Data mining report 1/1/15 1/1/15 0.00%

Q2Y4 Data mining operations 1/1/15 3/31/15 0.00%

Milestone: Q2Y4 Data mining report 4/1/15 4/1/15 0.00%

Q3Y4 Data mining operations 4/1/15 6/30/15 0.00%

Milestone: Q3Y4 Data mining report 7/1/15 7/1/15 0.00%

Q4Y4 Data mining operations 7/1/15 9/30/15 0.00%

Milestone: Q4Y4 Data mining report 10/1/15 10/1/15 0.00%

Q1Y5 Data mining operations 10/1/15 12/31/15 0.00%

Milestone: Q1Y5 Data mining report 1/1/16 1/1/16 0.00%

Q2Y5 Data mining operations 1/1/16 3/31/16 0.00%

Milestone: Q2Y5 Data mining report 4/1/16 4/1/16 0.00%

Printed on Monday, April 15, 2013

251 Page: 22

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Q3Y5 Data mining operations 4/1/16 6/30/16 0.00%

Milestone: Q3Y5 Data mining report 7/1/16 7/1/16 0.00%

Develop Focus Group Topics and conduct 9/30/11 6/30/16 15.00% focus groups Milestone: Q1Y1 Focus Group Report 9/30/11 9/30/11 100.00%

Milestone: Q2Y1 Focus Group Report 1/2/12 1/2/12 100.00%

Milestone: Q3Y1 Focus Group Report 4/2/12 4/2/12 100.00%

Milestone: Q4Y1 Focus Group Report 7/2/12 7/2/12 0.00%

Milestone: Q1Y2 Focus Group Report 10/1/12 10/1/12 0.00%

Milestone: Q2Y2 Focus Group Report 12/31/12 12/31/12 0.00%

Milestone: Q3Y2 Focus Group Report 4/1/13 4/1/13 0.00%

Milestone: Q4Y2 Focus Group Report 7/1/13 7/1/13 0.00%

Milestone: Q1Y3 Focus Group Report 9/30/13 9/30/13 0.00%

Milestone: Q2Y3 Focus Group Report 12/31/13 12/31/13 0.00%

Milestone: Q3Y3 Focus Group Report 3/31/14 3/31/14 0.00%

Milestone: Q4Y3 Focus Group Report 6/30/14 6/30/14 0.00%

Milestone: Q1Y4 Focus Group Report 9/30/14 9/30/14 0.00%

Milestone: Q2Y4 Focus Group Report 12/31/14 12/31/14 0.00%

Milestone: Q3Y4 Focus Group Report 3/31/15 3/31/15 0.00%

Milestone: Q4Y4 Focus Group Report 6/30/15 6/30/15 0.00%

Milestone: Q1Y5 Focus Group Report 9/30/15 9/30/15 0.00%

Milestone: Q2Y5 Focus Group Report 12/31/15 12/31/15 0.00%

Milestone: Q3Y5 Focus Group Report 3/31/16 3/31/16 0.00%

Milestone: Q4Y5 Focus Group Report 6/30/16 6/30/16 0.00%

Develop and Conduct BoF Sessions 9/30/11 9/2/16 20.00%

Milestone: Conduct Y1 XSEDE BoF 9/30/11 9/30/11 100.00% and Report Milestone: Conduct Y1 SC BoF and 1/2/12 8/25/16 100.00% Report Milestone: Conduct Y2 XSEDE BoF 9/30/12 10/1/12 0.00% and Report Milestone: Conduct Y2 SC BoF and 12/31/12 12/31/12 0.00% Report Milestone: Conduct Y3 XSEDE BoF 9/2/16 9/2/16 0.00% and Report

Printed on Monday, April 15, 2013

252 Page: 23

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Milestone: Conduct Y3 SC BoF and 9/2/16 9/2/16 0.00% Report Milestone: Conduct Y4 XSEDE BoF 9/2/16 9/2/16 0.00% and Report Milestone: Conduct Y4 SC BoF and 9/2/16 9/2/16 0.00% Report Milestone: Conduct Y5 XSEDE BoF 9/2/16 9/2/16 0.00% and Report Milestone: Conduct Y5 SC BoF and 9/2/16 9/2/16 0.00% Report Conduct Usability Panels and Testing (as 4/1/11 7/9/21 20.00% needed)

User Engagement General Operations 7/1/11 9/3/20 20.00%

Monitor tickets 7/1/11 9/3/20 20.00%

Resolve XSEDE wide tickets 7/1/11 9/3/20 20.00%

Bi-weekly User Engagement status 7/1/11 9/3/20 20.00% meetings

Quarterly reporting 7/1/11 9/3/20 20.00%

XSEDE Ticket System 4/1/11 4/29/13 45.45%

Transition existing TG ticket system to 4/1/11 4/27/12 100.00% production in XSEDE Milestone: Release Production 7/7/11 4/29/13 100.00% Ticketing System Deploy new/improved XSEDE Ticket 7/1/11 10/29/12 33.33% System Define requirements for ticket 7/1/11 8/31/12 100.00% system Evaluate candidate ticket systems 8/15/11 10/29/12 100.00%

Milestone: Select new/improved 10/3/11 8/31/12 100.00% ticket system Develop and deploy new/improved 10/3/11 3/30/12 0.00% ticket system Milestone: Release new/improved 3/31/12 4/2/12 0.00% ticket system Integrate new/improved ticket 4/2/12 6/29/12 0.00% system with CRM Milestone: Release Integrated CRM 6/29/12 6/29/12 0.00% interface Integrate new/improved ticket 4/2/12 6/29/12 0.00% system with XSEDE User Portal Milestone: Release Integrated XUP 6/29/12 6/29/12 0.00% interface

Consulting Policies 10/3/11 5/31/16 25.00%

Create consulting policies, procedures, 10/3/11 5/31/16 50.00% and support guide Milestone: Deploy consulting policies, 1/30/14 1/30/14 0.00% procedures, and support guide Contact PI at allocation start 7/15/11 11/17/17 28.57%

Printed on Monday, April 15, 2013

253 Page: 24

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Contact "new"/"renewal" PIs at the 7/15/11 11/17/17 100.00% beginning of each XRAC allocation periodContact "new"/"renewal" PIs at the beginning of each XRAC allocation 7/15/11 1/30/14 100.00% period 1 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 10/17/11 1/30/14 100.00% period 2 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 1/16/12 1/30/14 100.00% period 3 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 4/16/12 4/28/15 100.00% period 4 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 4/29/15 4/29/15 100.00% period 5 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 9/1/15 9/1/15 0.00% period 6 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 1/30/14 1/30/14 0.00% period 7 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 10/29/15 10/29/15 0.00% period 8 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 1/30/14 1/30/14 0.00% period 9 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 1/29/16 1/29/16 0.00% period 10 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 1/30/14 1/30/14 0.00% period 11 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 4/29/16 4/29/16 0.00% period 12 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 7/15/14 7/15/14 0.00% period 13 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 10/15/14 10/15/14 0.00% period 14 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 1/15/15 1/15/15 0.00% period 15 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 4/15/15 4/15/15 0.00% period 16 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 7/15/15 7/15/15 0.00% period 17 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 10/15/15 10/15/15 0.00% period 18

Printed on Monday, April 15, 2013

254 Page: 25

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 1/15/16 1/15/16 0.00% period 19 Contact "new"/"renewal" PIs at the beginning of each XRAC allocation 4/15/16 4/15/16 0.00% period 20

Contact "startup" PIs each month 7/15/11 9/2/16 25.00%

1 7/15/11 9/1/16 100.00%

2 8/15/11 9/1/16 100.00%

3 9/15/11 9/16/11 100.00%

4 10/17/11 9/2/16 100.00%

5 11/15/11 9/2/16 100.00%

6 12/15/11 9/2/16 100.00%

7 1/16/12 9/2/16 100.00%

8 2/15/12 9/2/16 100.00%

9 3/15/12 9/2/16 100.00%

10 4/16/12 9/2/16 100.00%

11 5/15/12 9/2/16 100.00%

12 9/2/16 9/2/16 100.00%

13 9/2/16 9/2/16 100.00%

14 8/15/12 8/15/12 100.00%

15 9/17/12 9/17/12 100.00%

16 10/15/12 10/15/12 0.00%

17 11/15/12 11/15/12 0.00%

18 4/29/13 4/29/13 0.00%

19 1/15/13 1/15/13 0.00%

20 2/15/13 2/15/13 0.00%

21 3/15/13 3/15/13 0.00%

22 4/15/13 4/15/13 0.00%

23 5/15/13 5/15/13 0.00%

24 6/17/13 6/17/13 0.00%

25 8/25/16 8/25/16 0.00%

26 8/31/16 8/31/16 0.00%

Printed on Monday, April 15, 2013

255 Page: 26

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

27 9/16/13 9/16/13 0.00%

28 9/2/16 9/2/16 0.00%

29 9/2/16 9/2/16 0.00%

30 9/2/16 9/2/16 0.00%

31 9/2/16 9/2/16 0.00%

32 9/2/16 9/2/16 0.00%

33 9/2/16 9/2/16 0.00%

34 9/2/16 9/2/16 0.00%

35 9/2/16 9/2/16 0.00%

36 9/2/16 9/2/16 0.00%

37 9/2/16 9/2/16 0.00%

38 8/15/14 8/15/14 0.00%

39 9/15/14 9/15/14 0.00%

40 10/15/14 10/15/14 0.00%

41 11/17/14 11/17/14 0.00%

42 12/15/14 12/15/14 0.00%

43 1/15/15 1/15/15 0.00%

44 2/16/15 2/16/15 0.00%

45 3/16/15 3/16/15 0.00%

46 4/15/15 4/15/15 0.00%

47 5/15/15 5/15/15 0.00%

48 6/15/15 6/15/15 0.00%

49 8/25/16 8/25/16 0.00%

50 8/31/16 8/31/16 0.00%

51 9/15/15 9/15/15 0.00%

52 9/2/16 9/2/16 0.00%

53 9/2/16 9/2/16 0.00%

54 9/2/16 9/2/16 0.00%

55 9/2/16 9/2/16 0.00%

56 9/2/16 9/2/16 0.00%

Printed on Monday, April 15, 2013

256 Page: 27

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

57 9/2/16 9/2/16 0.00%

58 9/2/16 9/2/16 0.00%

59 9/2/16 9/2/16 0.00%

60 9/2/16 9/2/16 0.00%

Allocations 1/3/11 9/2/16 22.45%

Allocations policy in place 4/1/11 9/2/16 100.00%

Host Quarterly Allocations Meetings 9/1/11 9/2/16 31.25% Annually Host Year 1 Quarterly Allocations 9/1/11 9/2/16 100.00% Meetings Host Quarterly Allocations Meeting 9/1/11 9/2/16 100.00%

Host Quarterly Allocations Meeting 12/1/11 9/2/16 100.00%

Host Quarterly Allocations Meeting 3/1/12 9/2/16 100.00%

Host Quarterly Allocations Meeting 9/2/16 9/2/16 100.00%

Host Year 2 Quarterly Allocations 9/2/16 9/2/16 25.00% Meetings Host Quarterly Allocations Meeting 9/2/16 9/2/16 100.00%

Host Quarterly Allocations Meeting 9/2/16 9/2/16 0.00%

Host Year 3 Quarterly Allocations 9/2/16 9/2/16 0.00% Meetings Host Quarterly Allocations Meeting 9/2/16 9/2/16 0.00%

Host Quarterly Allocations Meeting 9/2/16 9/2/16 0.00%

Host Year 4 Quarterly Allocations 9/2/16 9/2/16 0.00% Meetings Host Quarterly Allocations Meeting 9/2/16 9/2/16 0.00%

Host Quarterly Allocations Meeting 9/2/16 9/2/16 0.00%

Conduct How to Write a Successful Proposal 1/3/11 9/2/16 25.00% Webcasts Annually Conduct Year 1 How to Write a 8/1/11 9/2/16 100.00% Successful Proposal Webcast Conduct How to Write a Successful 8/1/11 9/2/16 100.00% Proposal Webcast Printed on Monday, April 15, 2013

257 Page: 28

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Conduct How to Write a Successful 11/1/11 9/2/16 100.00% Proposal Webcast Conduct How to Write a Successful 2/1/12 9/2/16 100.00% Proposal Webcast Conduct How to Write a Successful 5/1/12 9/2/16 100.00% Proposal Webcast Conduct Year 2 How to Write a 9/2/16 9/2/16 25.00% Successful Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 100.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct Year 3 How to Write a 1/3/11 9/2/16 0.00% Successful Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 1/3/11 1/3/11 0.00% Proposal Webcast Conduct Year 4 How to Write a 9/2/16 9/2/16 0.00% Successful Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct Year 5 How to Write a 9/2/16 9/2/16 0.00% Successful Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast Conduct How to Write a Successful 9/2/16 9/2/16 0.00% Proposal Webcast New POPS Interface 2/13/12 9/2/16 0.00%

Design new interface 2/13/12 4/2/12 0.00%

Implement and test 4/2/12 6/29/12 0.00%

Deploy 9/2/16 9/2/16 0.00%

Add Tier 2 resources to allocation process 1/3/11 12/30/11 0.00%

Printed on Monday, April 15, 2013

258 Page: 29

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Implement Allocation Levels and Types 1/3/11 6/28/13 0.00%

Levels 7/2/12 6/28/13 0.00%

Small 7/2/12 6/28/13 0.00%

Standard 7/2/12 6/28/13 0.00%

XRAC 7/2/12 6/28/13 0.00%

Types 1/3/11 6/28/13 0.00%

Storage 1/3/11 12/30/11 0.00%

Visualization 7/2/12 6/28/13 0.00%

Throughput 7/2/12 6/28/13 0.00%

Advanced Support for Research 7/2/12 6/28/13 0.00% Teams (ECSS) GPU, MIC, other 7/2/12 6/28/13 0.00% non-heterogeneous x86 compute Extended Collaborativesystems Support Service 7/1/11 7/22/16 40.96% (ECSS)-Projects Create/ test proj. mgmt. framework for ECSS 7/1/11 10/28/11 100.00% work plans/reporting Add at least 1 external FTE to fill an identified 7/1/11 6/30/16 100.00% skills gap Fill 1.5 Discretionary Hires (as needed) 7/2/12 6/28/13 66.66% Host monthly symposium open to ECSS, XSEDE 10/3/11 6/30/16 30.00% staff, and Users Conduct continuing training of ECSS Staff as 7/2/12 6/30/16 30.00% needed Extended Support for Research Team (ESRT) 7/1/11 7/22/16 39.29%

Establish ESRT group 7/1/11 7/21/11 100.00%

Set up ESRT staff and management 7/1/11 7/21/11 100.00% teams and communications Milestone: Support 20 ESRT Projects 7/1/11 7/22/16 40.59% Annually Work w/TG AUS to transition ASTA 7/1/11 10/28/11 100.00% Projs. To ESRT mgmt. Milestone: All TG ASTA proj. managed 10/28/11 10/28/11 100.00% as XD ESRT Projs. Work with XD CMS to Generate 20 new 7/1/11 6/30/16 40.00% XD ESRT projs. Annually Year 1 Generate 20 new XD ESRT 7/1/11 6/29/12 100.00% projects Year 2 Generate 20 new XD ESRT 7/2/12 6/28/13 100.00% projects Year 3 Generate 20 new XD ESRT 7/1/13 6/30/14 0.00% projects Year 4 Generate 20 new XD ESRT 7/1/14 6/30/15 0.00% projects Year 5 Generate 20 new XD ESRT 7/1/15 6/30/16 0.00% projects Printed on Monday, April 15, 2013

259 Page: 30

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE 20 ESRT work plans documented and 10/31/11 7/22/16 29.00% actively managed annually Year 1 - 20 work plans documented 10/31/11 6/29/12 100.00%

milestone: Y1 prepare and 7/2/12 7/27/12 90.00% complete Final Reports

Year 2 - 20 work plans documented 7/2/12 6/28/13 100.00%

milestone: Y2 prepare and 7/1/13 7/26/13 0.00% complete Final Reports Year 3 - 20 work plans documented 7/1/13 6/30/14 0.00%

milestone: Y3 prepare and 7/1/14 7/25/14 0.00% complete Final Reports

Year 4 - 20 work plans documented 7/1/14 6/30/15 0.00%

milestone: Y4 prepare and 7/1/15 7/24/15 0.00% complete Final Reports Year 5 - 20 work plans documented 7/1/15 6/30/16 0.00%

milestone: Y5 prepare and 7/1/16 7/22/16 0.00% complete Final Reports Organize and execute HPC workshop at 7/1/11 7/7/11 0.00% IEEE E-Science conference Host Annual Workshop on Petascale 7/1/11 7/1/15 0.00% Computing

Conduct XRAC Meetings Adaptive Reviews 7/2/12 6/30/16 35.00%

Novel & Innovative Projects 7/1/11 6/29/16 28.57%

Establish NIP group 7/1/11 9/30/11 100.00%

Set up NIP staff and management 7/1/11 9/30/11 100.00% teams and communications Milestone: Generate 20 new XSEDE+ ECS 7/1/11 6/29/16 20.00% NIPs Annually Year 1-work w/XD CMS,TEOS,TIS to 7/1/11 6/29/12 100.00% generate 20 XD NIP projects Year 2-work w/XD CMS,TEOS,TIS to 7/2/12 6/28/13 0.00% generate 20 XD NIP projects Year 3-work w/XD CMS,TEOS,TIS to 7/1/13 6/30/14 0.00% generate 20 XD NIP projects Year 4-work w/XD CMS,TEOS,TIS to 7/1/14 6/30/15 0.00% generate 20 XD NIP projects Year 5-work w/XD CMS,TEOS,TIS to 7/1/15 6/29/16 0.00% generate 20 XD NIP projects

Milestone: Create ECSS Project work plans 7/1/11 12/30/11 0.00% Extended Collaborative Support Service - 7/1/11 7/22/16 34.17% Communities Create/ test proj. mgmt. framework for ESCC 7/1/11 10/28/11 100.00% work plans/reporting Add at least 1 external FTE to fill an identified 7/1/11 6/30/16 25.00% skills gap Host monthly symposium for ECSS and XSEDE 10/3/11 6/30/16 30.00% staff

Printed on Monday, April 15, 2013

260 Page: 31

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Conduct Continuing Training of ECSS Staff as 7/2/12 6/30/16 30.00% needed Extended Support for Community Codes 7/1/11 7/22/16 35.00% (ESCC) Establish ESCC group 7/1/11 8/25/11 100.00%

Set up ESCC staff and management 7/1/11 8/25/11 100.00% teams and communications Transition TG ASP proj. to ESCC 7/1/11 10/28/11 100.00% management Milestone:Active TG ASP proj. managed as 10/28/11 10/28/11 100.00% XD ESCC proj. Milestone: Support 10 ESCC Projects 7/1/11 7/22/16 26.67% Annually Year 1-Work w/TG CMS,TEOS,TIS to 7/1/11 6/29/12 100.00% generate 10 XD ESCC proj. Year 2-Work w/TG CMS,TEOS,TIS to 7/2/12 6/28/13 100.00% generate 10 XD ESCC proj. Year 3-Work w/TG CMS,TEOS,TIS to 7/1/13 6/30/14 0.00% generate 10 XD ESCC proj. Year 4-Work w/TG CMS,TEOS,TIS to 7/1/14 6/30/15 0.00% generate 10 XD ESCC proj. Year 5-Work w/TG CMS,TEOS,TIS to 7/1/15 6/30/16 0.00% generate 10 XD ESCC proj. Milestone: Create 10 ESCC work plans 10/31/11 7/22/16 20.00% Annually Y1 - 10 work plans documented 10/31/11 6/29/12 100.00%

milestone: Y1 prepare and 7/2/12 7/27/12 50.00% complete Final Reports

Y2 - 10 work plans documented 7/2/12 6/28/13 50.00%

milestone: Y2 prepare and 7/1/13 7/26/13 0.00% complete Final Reports

Y3 - 10 work plans documented 7/1/13 6/30/14 0.00%

milestone: Y3 prepare and 7/1/14 7/25/14 0.00% complete Final Reports

Y4 - 10 work plans documented 7/1/14 6/30/15 0.00%

milestone: Y4 prepare and 7/1/15 7/24/15 0.00% complete Final Reports Y5 - 10 work plans documented 7/1/15 6/30/16 0.00%

milestone: Y5 prepare and 7/1/16 7/22/16 0.00% complete Final Reports Work with the TIS group to evaluate and recommend SI2 software projects for 7/2/12 6/30/16 0.00% inclusion in the XSEDE Develop documentation, sample scripts, optimized builds to cover the top 7/2/12 6/30/16 0.00% community codes in use on Extended Collaborative Support Service - 7/1/11 7/22/16 25.50% Science Gateways Establish ESSGW group 7/1/11 6/30/16 100.00%

Printed on Monday, April 15, 2013

261 Page: 32

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Set up ESSGW staff and management 7/1/11 8/25/11 100.00% teams and communications Organize bi-weekly gateway community 9/1/11 6/30/16 100.00% calls open to all XSEDE users Milestone: Support 10 Science Gateways 7/1/11 7/22/16 16.67% Annually Year 1-Work w/TG CMS,TEOS,TIS to 7/1/11 6/29/12 100.00% generate 10 XD ESSGW proj. Year 2-Work w/TG CMS,TEOS,TIS to 7/2/12 6/28/13 70.00% generate 10 XD ESSGW proj. Year 3-Work w/TG CMS,TEOS,TIS to 7/1/13 6/30/14 0.00% generate 10 XD ESSGW proj. Year 4-Work w/TG CMS,TEOS,TIS to 7/1/14 6/30/15 0.00% generate 10 XD ESSGW proj. Year 5-Work w/TG CMS,TEOS,TIS to 7/1/15 6/30/16 0.00% generate 10 XD ESSGW proj.

Milestone: Create ESSGW work plans 10/31/11 7/22/16 8.00%

At least 10 ESSGW work plans documented&actively managed 10/31/11 7/22/16 8.00% annually

Y1 - 10 work plans documented 10/31/11 6/29/12 50.00%

milestone: Y1 prepare and 7/2/12 7/27/12 10.00% complete Final Reports Y2 - 10 work plans documented 7/2/12 6/28/13 20.00%

milestone: Y2 prepare and 7/1/13 7/26/13 0.00% complete Final Reports

Y3 - 10 work plans documented 7/1/13 6/30/14 0.00%

milestone: Y3 prepare and 7/1/14 7/25/14 0.00% complete Final Reports Y4 - 10 work plans documented 7/1/14 6/30/15 0.00%

milestone: Y4 prepare and 7/1/15 7/24/15 0.00% complete Final Reports Y5 - 10 work plans documented 7/1/15 6/30/16 0.00%

milestone: Y5 prepare and 7/1/16 7/22/16 0.00% complete Final Reports Gateway Outreach: Constantly reach out to new potential gateways independently and 7/1/11 6/30/16 20.00% in collaboratio XSEDE Requirements: Work with Gateway community in analyzing the XSEDE 7/1/11 6/30/16 20.00% architecture requirements XSEDE Architecture Test Cases: Provide Test Cases to SD&I teams in nature of tests 7/1/11 6/30/16 20.00% to evaluate sca Extended Support for Training Education and 7/1/11 6/30/16 45.00% Outreach (ESTEO) Establish ESTEO group 7/1/11 8/25/11 100.00%

Set up ESTEO staff and management 7/1/11 8/25/11 100.00% teams and communications

Printed on Monday, April 15, 2013

262 Page: 33

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Milestone: Contribute content for TEO 7/1/11 10/28/11 100.00% modules Work w/TG AUS to transition ASEOT 7/1/11 10/28/11 100.00% projs. to XD ESTEO mgmt. Milestone: All TG ASEOT projs. 10/28/11 10/28/11 100.00% Managed as XD ESTEO projs. Milestone: 50 ESTEO projects/activities 7/1/11 6/30/16 30.00% supported Annually Year 1-work w/XD CMS, TEOS, TIS to 7/1/11 6/29/12 100.00% generate 50 XD ESTEO projects/activitiesYear 2- work w/XD CMS, TEOS, TIS to 7/2/12 6/28/13 50.00% generate 50 XD ESTEO projects/activitiesYear 3 - work w/XD CMS, TEOS, TIS to 7/1/13 6/30/14 0.00% generate 50 XD ESTEO projects/activitiesYear 4 - work w/XD CMS, TEOS, TIS to 7/1/14 6/30/15 0.00% generate 50 XD ESTEO projects/activitiesYear 5 - work w/XD CMS, TEOS, TIS to 7/1/15 6/30/16 0.00% generate 50 XD ESTEO Testprojects/activities and Document initial work assignments 7/2/12 6/28/13 0.00% for UCB CS class using XSEDE resouces Arrange ECSS Internal Training Seminars 7/2/12 6/30/16 0.00% Annually Education and Outreach 4/4/11 7/1/13 58.19%

Education 7/1/11 7/1/13 66.25%

Milestone: 2 HPC Graduate level summer 7/1/11 6/28/13 75.00% schools annually Milestone: 2 HPC Graduate level 7/1/11 10/3/11 100.00% summer schools annually Milestone: 2 HPC Graduate level 7/2/12 6/28/13 50.00% summer schools annually Milestone: 5 summer workshops annually 7/1/11 7/1/13 55.00%

Milestone: 5 summer workshops 7/1/11 10/3/11 100.00% annually Milestone: 5 summer workshops 7/2/12 7/1/13 10.00% annually Milestone: Add certificate programs at 7/1/11 7/1/13 60.00% specific universities Milestone: ID univ to work with to dev 7/1/11 12/30/11 100.00% vert pgm Milestone:Add cert and/or deg pgm @ 7/2/12 7/1/13 20.00% univ and cont to ID univs for cert pgm Milestone: Provide online educational 7/1/11 6/28/13 75.00% services Milestone: Provide online educational 7/1/11 10/3/11 100.00% services Milestone: Provide online educational 7/2/12 6/28/13 50.00% services Outreach 6/1/11 7/1/13 72.86%

Underrepresented Engagement 7/1/11 7/1/13 75.00%

Milestone: 10 campus visits (SURA) 7/1/11 6/28/13 75.00%

Printed on Monday, April 15, 2013

263 Page: 34

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Milestone: 10 campus visits (SURA) 7/1/11 10/3/11 100.00%

Milestone: 10 campus visits (SURA) 7/2/12 6/28/13 50.00%

Milestone: Engage 40 underrepresented 7/1/11 6/28/13 75.00% individuals (Rice) Milestone: Engage 40 7/1/11 10/3/11 100.00% underrepresented individuals (Rice) Milestone: Engage 40 7/2/12 6/28/13 50.00% underrepresented individuals (Rice) Milestone: Create Faculty Council with 7/1/11 7/1/13 75.00% 20 minority faculty (Rice) Milestone: Create Faculty Council 7/1/11 12/30/11 100.00% with 20 minority faculty (Rice) Milestone: Create Faculty Council 7/2/12 7/1/13 50.00% with 20 minority faculty (Rice)

Speakers' Bureau 7/1/11 6/28/13 75.00%

Milestone: 10 National/Regional 7/1/11 6/28/13 75.00% presentations Milestone: 10 National/Regional 7/1/11 10/3/11 100.00% presentations Milestone: 10 National/Regional 7/2/12 6/28/13 50.00% presentations Student Engagement 7/1/11 6/28/13 60.00%

Milestone: Recruit 20 students for 7/1/11 6/28/13 60.00% training/mentoring/internship Milestone: Recruit 20 students for 7/1/11 10/3/11 100.00% training/mentoring/internship Milestone: Recruit 20 students for 7/2/12 6/28/13 20.00% training/mentoring/internship

Campus Champions 7/1/11 6/28/13 75.00%

Milestone: Increase impact in Campus 7/1/11 6/28/13 75.00% Champions program Milestone: Increase membership in 7/1/11 10/3/11 100.00% Campus Champions program Milestone: Increase impact in 7/2/12 6/28/13 50.00% Campus Champions program

XSEDE Annual Conference 6/1/11 6/28/13 75.00%

XSEDE 12 6/1/11 10/3/11 100.00%

XSEDE 13 7/2/12 6/28/13 50.00%

Community Requirements 4/4/11 6/28/13 75.00%

TEOS Advisory Committee 4/4/11 6/28/13 75.00%

Semi-Annual consultation with TEOS 4/4/11 11/16/11 100.00% Advisory Group TEOS Advisory Committee 7/2/12 6/28/13 50.00%

Collect Community Requirements 4/4/11 6/28/13 75.00%

Annual collection and analysis of 4/4/11 11/16/11 100.00% community needs and requirements Printed on Monday, April 15, 2013

264 Page: 35

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Collect Community Requirements 7/2/12 6/28/13 50.00%

Infratructure 4/4/11 6/28/13 66.67%

Curation of TEOS information on public 4/4/11 10/3/11 100.00% web and XSEDE portal E&O Curation 7/2/12 6/28/13 50.00%

E&O Infrastructure Lead 7/2/12 6/28/13 50.00%

Campus Bridging 4/4/11 6/28/13 39.33%

Lead Campus Bridging Effort 4/4/11 6/28/13 53.33%

Ongoing: Share information with 4/4/11 10/3/11 100.00% campuses interested in campus bridging Lead Campus Bridging Effort 7/2/12 6/28/13 50.00%

Campus Bridging Travel to Pilot 7/2/12 6/28/13 100.00% Program Sites(Task replaced 1/23/13) Pilot program, software packaging, documentation and suppport (Task 7/2/12 6/28/13 50.00% replaced 1/23/13) Rocks Roll test cluster 10GbE at 7/2/12 6/28/13 10.00% Cornell(Task replaced 1/23/13) Rocks Roll test cluster Infiniband at 7/2/12 6/28/13 10.00% IU(Task Replaced 1/23/13)

Pilot program 2/1/12 4/30/13 55.64%

Pilot site preparatory meetings 2/1/12 12/31/12 100.00%

Operations deployment plan for beta 7/2/12 12/28/12 100.00% grid XSEDE beta grid in place 8/1/12 3/29/13 80.06%

XSEDE GFFS Container Software 1/1/13 3/29/13 19.76% installed at SP's XSEDE GFFS Software installers ready 12/3/12 2/1/13 75.07% for pilot sites

Pilot sites using GFFS Software 1/1/13 3/28/13 19.96%

Pilot metrics and case studies created 2/1/13 4/30/13 50.30%

Evaluation discussions with pilots 2/1/13 4/30/13 0.00%

Software packaging program 12/3/12 6/28/13 8.68%

Software package list 12/3/12 2/28/13 50.29%

Rocks roll for XSEDE-like cluster 1/1/13 2/28/13 10.47%

Rocks testing by Operations 3/1/13 5/31/13 0.00%

Cobbler/Puppet configuration 2/1/13 4/30/13 0.00% scripts/recipes RPMs for XSEDE software packages 3/1/13 3/29/13 0.00%

RPM testing by Operations 4/1/13 6/28/13 0.00%

Printed on Monday, April 15, 2013

265 Page: 36

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE

Software distribution site & channels 3/1/13 3/29/13 0.00%

External Evaluation 4/4/11 6/28/13 75.00%

External Evaluator Quarterly Reports 4/4/11 10/3/11 100.00%

External Evaluation 7/2/12 6/28/13 50.00%

Technology Investigation Service 6/1/12 12/30/15 59.69%

Technology Identification 7/2/12 8/5/13 56.82%

Plan and Execute TIS Merge 8/20/12 9/28/12 100.00%

Create QA/Testing/backup plan 7/2/12 7/3/12 100.00%

Identify ER content coordinator for new TIS 7/2/12 7/3/12 100.00% site Market TIS to Technologies 7/2/12 6/28/13 11.35%

Market TIS to user 7/2/12 6/28/13 10.00%

Plan for increased access to TIS - ex. In 4/1/13 8/5/13 0.00% common Authentication Enable users to add comments to 8/28/12 12/31/12 15.00% technologies - such as likes and feedbacks Enable evaluators to enter evaluations for 7/2/12 9/21/12 100.00% multiple pieces of a single technology Plan for future release iterations of XTED 7/2/12 6/28/13 75.00% based on requirements Technology Evaluation 6/1/12 12/30/15 60.43%

Receive the requirements from other Level 10/1/12 9/27/13 100.00% 3s. Providing evaluation results to the XSEDE 4/1/13 7/18/14 75.00% Level 3 implementers. Transfer Evaluation results to the Level 4/1/13 9/27/13 75.00% 3 implementers. Transfer TIS knowledge and 7/22/13 7/18/14 75.00% deployment objects to Level 3 Makeimplementers data available to other XSEDE Level 3s about the evaluation process, data 10/1/13 9/29/14 75.00% collection and report Maintain/update system access accounts to 1/1/14 12/30/14 75.00% XSEDE RP systems for TEP members

Maintain Test hardware 7/21/14 9/29/15 75.00%

Maintain/administer TEP acquired 7/21/14 7/17/15 75.00% hardware Maintenance/administration of the TEL 10/1/14 9/29/15 75.00% dedicated hardware

Update XTED as appropriate 1/1/15 12/30/15 75.00%

Accomplish multiple evaluations annually 6/1/12 11/11/15 52.50%

Evaluation 5: Pegasus WMS- finish 6/1/12 11/11/15 100.00%

6/1/12 11/15/12 90.00%

Printed on Monday, April 15, 2013

266 Page: 37

% Name Start Finish Completed 2011 2012 2013 2014 2015 2016 2017 2018 2019 FTE Evaluation 7: GFFS Reliable File 7/2/12 8/2/12 100.00%

Evaluation 7: GFFS Reliable File 7/2/12 8/2/12 100.00%

Evaluation 8: Safenet - start 10/15/12 11/15/12 100.00%

7/2/12 8/2/12 50.00%

7/2/12 8/2/12 100.00%

7/2/12 8/2/12 50.00%

7/2/12 8/2/12 100.00%

7/2/12 8/2/12 50.00%

7/2/12 8/2/12 0.00% start

7/2/12 8/2/12 0.00% finish

7/2/12 8/2/12 0.00% start

7/2/12 8/2/12 0.00% finish

7/2/12 8/2/12 0.00% start

7/2/12 8/2/12 0.00% finish Verify the evaluation process annually 7/2/12 4/8/13 50.00%

review the evaluation process 4/1/13 4/8/13 100.00%

update the evaluation process post 7/2/12 7/9/12 0.00% review as necessary Assist in updating the process to identify 7/23/12 7/19/13 75.00% User Requirements and the next item for Maintain/updateevaluation the TEP portion of the 7/23/12 7/19/13 75.00% XSEDE wiki Documentation 7/23/12 4/12/13 66.67%

Document TEL Dedicated Hardware 7/23/12 8/3/12 100.00%

Document FutureGrid systems 4/1/13 4/12/13 0.00%

Provide short document for specific TIS 10/1/12 10/4/12 100.00% activities for others

Create an evaluation process for hardware 4/1/13 5/24/13 0.00%

Create a training package for new TEP 10/15/12 11/15/12 100.00% members to orient them on the TEP process Train new members with the training 7/2/12 7/11/12 100.00% package Virtual Machines 4/1/13 4/19/13 0.00%

Create images that closely mirrors 4/1/13 4/19/13 0.00% XSEDE SP systems

Printed on Monday, April 15, 2013

267 Page: 38

Gantt Legend

Project (Gantt Bar Styles)

Default Completed Scheduled Baseline 1

Task (Gantt Bar Styles)

Default Completed Scheduled Free Float Total Float Negative Float Start Delay Baseline 1

Critical Filter Scheduled

Parent Filter Scheduled

Buffers Scheduled

Buffer Incursion 0% - 25% Buffer Incursion

Buffer Incursion 25% - 75% Buffer Incursion

Buffer Incursion 75% - 100% Buffer Incursion

Task (Gantt Symbols)

Default Objective Start Objective Finish Gantt Baseline2 Finish Gantt Baseline3 Finish Gantt Baseline4 Finish Gantt Baseline5 Finish

Printed on Monday, April 15, 2013

268 C XSEDE Risk Register

Home > Risks Register > XSEDE Risks Summary

Risks Summary

Risk Summary Stats All Risks Non-retired Retired Risks Risks Total Number of Risks: 154 1 8 7 1 1 1 7 6 Number of non-retired 122 Risks: 17 36 22 5 13 17 31 9 High Impact 32 Risks: 28 13 22 6 1 5 22 12 17 High Probability 14 Risks: Impact Impact High Risk-Level: 6 Impact

Key Risks: 2

Current Risks: 117

(stats count non-retired risks)

Non-retired Risks

122 risks Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

273 Grid Software Scaling High High High Monitor 1.1.3 Architecture Apr 01, Mar 31, & Design 2011 2016

Grid Software System 1.1.3 Architecture Apr 01, Mar 31, 272 Integration High High High Monitor & Design 2011 2016

Lack of ability to support a large 1.5.3 Extended Mar 15, Jul 01, 380 number of online High High High Monitor Collaborative EOT 2013 2014 students. Support

Lack of ability to support a large Mar 15, Jul 01, 379 number of students in High High High Monitor 1.3.1 Training 2013 2014 full online courses

Lack of ability to support a large Mar 15, Jul 01, 378 number of students in High High High Monitor 1.6.1 Education 2013 2014 full online courses

Parent Risk for large Mar 12, Jul 01, 377 number of online High High High Monitor 2013 2014 students

Admins are not required to use strong 350 authentication for High High Medium Monitor 1.2.1 Security Oct 22, Oct 22, management of XSEDE 2012 2016 services.

Campus Bridging 1.6.5 Campus Oct 01, Dec 31, 328 expectation High High Medium Monitor management Bridging 2012 2013

Capacity of XSEDE Jun 30, 296 reources less than High Medium High Monitor 1.1 Project Office Jul 01, 2011 demand 2016

269 1.2.4 Software 278 Implementation delays High High Medium Monitor Testing & Jul 01, 2011 Jul 01, and inconsistencies Deployment 2016

Lack of Process to insure that the 369 application and user High Medium High Monitor 1.7 TIS Oct 30, Jun 30, cases/requirements 2012 2015 are defined up front.

Loss of Project PI or Jan 01, Jun 30, 287 co-PI High Medium High Monitor 1 XSEDE 2011 2016

Mismatch between research teams’ needs, 1.4 Extended Apr 01, Mar 31, 258 XD resources, and High Medium High Monitor Collaborative 2011 2016 AUSS staff availability. Support - Projects

Network Disruption Jul 01, 312 Disables Critical High Medium High Monitor 1.2.3 XSEDEnet Jul 01, 2011 Services 2015

Opportunity: Leverage 301 new/novel resources High Medium High Monitor 1.1 Project Office Jul 01, 2011

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

1.1.1 Project Feb 16, Jul 01, 331 Program Plan High Medium High Monitor Management and Reporting 2012 2015

Security policies and procedures from Oct 23, Oct 23, 362 TeraGrid are High High Medium Monitor 1.2.1 Security 2012 2016 out-of-date

Suitable project 1.4.1 Extended management Collaborative Mar 31, 250 framework and process High Medium High Monitor Research Teams Jul 01, 2011 2016 is not available. Support

There is a zero-day root escalation exploit in the wild for Linux or Oct 23, Oct 23, 360 some common piece of High High Medium Monitor 1.2.1 Security 2012 2016 the XSEDE software stack

There is no regular security training for Oct 23, Oct 23, 364 users and service High High Medium Monitor 1.2.1 Security providers, and security 2012 2016 policies lack visibility.

Usage of deployed 1.2.4 Software Jul 01, 274 software and services High High Medium Monitor Testing & Jul 01, 2011 2016 Deployment

XSEDE archival Jun 30, 298 storage gap High Medium High Monitor 1.2.2 Data Services Jul 01, 2011 2016

Active Design Review 1.1.3 Architecture Mar 12, Mar 31, 376 Effort Medium Medium Medium Monitor & Design 2013 2016

Automated services may use unencrypted Oct 22, Oct 22, 343 private keys to access Medium Medium Medium Monitor 1.2.1 Security 2012 2016 XSEDE

252 Campus Infrastructure Medium Medium Medium Monitor 1.6.5 Campus Jul 01, 2011 Mar 31, Bridging 2016

Communication 1.2.6 Systems Jul 01, 318 Breakdown Medium Medium Medium Monitor Operational Jul 01, 2011 2016 Support

Delay in program Sep 19, Sep 30, 325 implementation Medium Medium Medium Monitor 1.6.1 Education 2011 2012

Dependencies on 1.7.1 Technology Apr 04, Jun 30, 392 access to development Medium Medium Medium Monitor Identification 2013 2015 resources

270 Deployed software could be out of date, especially without 356 coordinated patch Medium Medium Medium Monitor Oct 23, Oct 23, manage-ment and 2012 2016 stale CTSS registrations info

Federated identity Jun 30, 305 management does not Medium Medium Medium Monitor 1.1 Project Office Jul 01, 2011 2014 catch on

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

GFFS Pilot - lack of 1.6.5 Campus Mar 18, Jul 01, 388 responses/engagement Medium Medium Medium Monitor Bridging 2013 2014

Grand-fathered in Oct 23, Oct 23, 363 services or software Medium Medium Medium Monitor 1.2.1 Security 2012 2016 are exploited

Incident response resources at different SPs vary, and the 365 XSEDE incident Medium Medium Medium Monitor 1.2.1 Security Oct 23, Oct 23, response (IR) team is 2012 2016 geographically distributed.

Inconsistent or Oct 22, Oct 22, 347 non-existent backup Medium Medium Medium Monitor processes 2012 2016

Insufficient Service Jan 01, Mar 31, 271 Provider Integration Medium Medium Medium Monitor 1.1 Project Office into XSEDE activities 2011 2016

Mar 31, 251 Mentoring Program Medium Medium Medium Monitor 1.6.2 Outreach Jul 01, 2011 2016

No consistent patch management process Oct 23, Oct 23, 352 for all XSEDE services Medium Medium Medium Monitor 1.2.1 Security 2012 2016 and systems

Non-optimum arrival 373 of tool evaluation Medium Medium Medium Execute 1.7.2 Technology Oct 30, Jun 30, requests. Contingency Evaluation 2012 2015

277 Noncompliant service Medium Medium Medium Monitor 1.1 Project Office Apr 01, Mar 31, provider 2011 2016

1.6.5 Campus Mar 31, 253 Remote User Support Medium Medium Medium Monitor Bridging Jul 01, 2011 2016

1.1.3 Architecture Mar 12, Mar 31, 374 Scheduling participants Medium Medium Medium Monitor & Design 2013 2016

Suitable project 1.5.2 Extended management Collaborative Mar 31, 322 framework and process Medium Medium Medium Monitor Science Gateways Jul 01, 2011 2016 is not available Support

Suitable project management 1.5.1 Extended Mar 31, 255 framework and process Medium Medium Medium Monitor Support for Jul 01, 2011 2016 is not available. Community Codes

Suitable project 1.5.3 Extended 257 management Medium Medium Medium Monitor Collaborative EOT Jul 01, 2011 Mar 31, framework and process 2016 is not available. Support

There are no security baselines for most 366 services, and there is Medium Medium Medium Monitor 1.2.1 Security Oct 23, Oct 23, no regular auditing 2012 2016 with respect to security

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

271 There is a common 361 XSEDE service with a Medium Medium Medium Monitor 1.2.1 Security Oct 23, Oct 23, remote exploit 2012 2016

Training - Sync Mar 31, 246 delivery Medium Medium Medium Monitor 1.3.1 Training Jul 01, 2011 2016

375 Working through Medium Medium Medium Monitor 1.1.3 Architecture Mar 12, Mar 31, slumps & Design 2013 2016

XSEDE depends upon software that is no Oct 23, Oct 23, 359 longer actively Medium Medium Medium Monitor 2012 2016 supported

XSEDE hardening Oct 22, Oct 22, 346 guidelines for SPs are Medium Medium Medium Monitor 1.2.1 Security 2012 2016 optional and unaudited

XSEDE has no Oct 22, Oct 22, 351 centralized security Medium Medium Medium Monitor 1.2.1 Security 2012 2016 monitoring

XSEDE has very complex organizational Oct 23, Oct 23, 367 and procedural Medium Medium Medium Monitor 1.2.1 Security 2012 2016 structures.

XSEDE relies heavily 357 on in-house software Medium Medium Medium Monitor 1.2.1 Security Oct 23, Oct 23, that has not had code 2012 2016 audits

Architecture 1.1.2 Systems and 266 Obsolescence & Medium Low High Monitor Software Apr 01, Mar 31, Software Risks Engineering 2011 2016

Dependencies on 370 operational support for Medium Low High Monitor 1.7 TIS Oct 30, Jun 30, services from outside 2012 2015 sources.

Differing architectural views hinder 1.1.2 Systems and Mar 01, Jun 30, 288 deployment and Medium Low High Monitor Software 2011 2016 operation of XSEDE Engineering

1.2.6 Systems 332 Failure of XDCDB Medium Low High Monitor Operational Jun 18, Jul 01, failover process Support 2012 2016

Failure of XSEDE 1.2.6 Systems Jul 01, 260 Operational Medium Low High Monitor Operational Jul 01, 2011 Infrastructure Support 2016

Funds are not provided 295 for initial network Medium Low High Monitor 1.2.3 XSEDEnet Jul 01, 2011 Jun 30, "gap" costs 2016

Inadequate Jan 01, Mar 31, 276 communication with Medium Low High Monitor 1.1 Project Office 2011 2016 SPs

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

LifeRay license expires 1.7.1 Technology Oct 30, Jun 30, 372 without renewal. Medium Low High Monitor Identification 2012 2015

Mismatch between 1.5.1 Extended 254 research teams’ needs, Medium Low High Monitor Support for Jul 01, 2011 Mar 31, XD resources, and 2016 ECSS staff availability. Community Codes

Mismatch between research teams’ needs, 1.5.3 Extended Mar 31, 256 XD resources, and Medium Low High Monitor Collaborative EOT Jul 01, 2011 2016 ECSS staff availability. Support

Mismatch maintained 1.4.1 Extended between research Collaborative Mar 31, 248 teams’ needs, XD Medium Low High Monitor Research Teams Jul 01, 2011 2016 resources, and ECSS staff availability. Support

272 NLR Services Are Jul 01, 311 Inadequate Medium Low High Monitor 1.2.3 XSEDEnet Jul 01, 2011 2016

1.2.4 Software 316 Prohibitive Cost for Medium Low High Monitor Testing & Jul 01, 2011 Jul 01, Required Software Deployment 2016

265 Technical Obsolesence Medium Low High Monitor 1.1.3 Architecture Apr 01, Mar 31, & Design 2011 2016

TeraGrid Domain 1.2.6 Systems 317 Name Hardcoded into Medium High Low Monitor Operational Jul 01, 2011 Jul 01, SP Software Support 2016

242 XRAC Meeting Costs Medium Low High Monitor 1.3.4 Allocations Jul 01, 2011 Mar 31, 2016

XSEDE DNS Service Jul 01, 314 Availability Medium Low High Monitor 1.2.3 XSEDEnet Jul 01, 2011 2016

XSEDE RS Service Jul 01, 315 Availability Medium Low High Monitor 1.2.3 XSEDEnet Jul 01, 2011 2015

334 Abuse of shared Low Medium Low Monitor 1.2.1 Security Oct 22, Oct 22, accounts 2012 2016

Deviation from project 1.1.1 Project 275 management Low Medium Low Monitor Management and Jan 01, Mar 31, procedures Reporting 2011 2016

1.7.2 Technology Apr 04, Jul 01, 394 Evaluation timeliness Low Medium Low Monitor Evaluation 2013 2015

Genesis II FUSE/RNS Oct 31, 309 scalability and Low Low Medium Monitor 1.2.2 Data Services Jul 31, 2011 2012 performance

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

354 Helpdesk tickets are Low Medium Low Monitor 1.2.1 Security Oct 23, Oct 23, emailed in plaintext 2012 2016

Identity vetting Oct 22, Oct 22, 338 procedures are abused Low Medium Low Monitor 1.2.1 Security 2012 2016 or circumvented

Inconsistent authentication Oct 22, Oct 22, 335 standards exploited to Low Medium Low Monitor 1.2.1 Security 2012 2016 spread attacks

Integrating services 307 fail to meet standards Low Medium Low Monitor 1.1 Project Office Jul 01, 2011 Jun 30, in service level 2016 agreements

1.2.4 Software 330 Lack of Testing Low Medium Low Monitor Testing & Jan 10, Jul 01, Resources Deployment 2012 2016

Lack of applications to 1.7.2 Technology Apr 04, Jul 01, 393 evaluate Low Low Medium Monitor Evaluation 2013 2015

270 Loss of Senior Low Low Medium Monitor 1 XSEDE Apr 01, Mar 31, Technical Personnel 2011 2016

Oct 22, Oct 22, 339 Mail list abuse Low Medium Low Monitor 1.2.1 Security 2012 2016

Mismatch between non-traditional users’ 1.4.2 Novel & Jun 30, 321 needs, XSEDE Low Medium Low Monitor Jul 01, 2011 services, and NIP staff Innovative Projects 2016 allocation.

NLR Business Focus Sep 01, Jul 01, 326 Changes Low Low Medium Monitor 1.2.3 XSEDEnet 2011 2016

Namespace collisions 1.2.4 Software 327 between XSEDE Low Medium Low Monitor Testing & Oct 03, Jul 01, software and local 2011 2016 software Deployment

273 Outdated Critical 1.2 XSEDE Aug 16, Jul 01, 333 Documents Low Low Medium Monitor Operations 2012 2016

Overloading XSEDE 1.2.6 Systems 319 Operations Center Low Medium Low Monitor Operational Jul 01, 2011 Jul 01, Staff Support 2016

Plan for long term service scaling may Jun 30, 306 not meet short term Low Medium Low Monitor 1.1 Project Office Jul 01, 2011 2016 needs

Potential equipment 1.2.5 239 failure can disrupt Low Low Medium Monitor Accounting/Account Jul 01, 2011 Jul 01, database services. Mgmt 2016

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

Prohibitive Operating 1.2.6 Systems Jul 01, 241 Costs for Hardware or Low Low Medium Monitor Operational Jul 01, 2011 Software Support 2016

Resources for security 368 in XSEDE could be Low Medium Low Monitor 1.2.1 Security Oct 23, Oct 23, inadequate 2012 2016

Self-service password Oct 22, Oct 22, 341 resets are abused Low Medium Low Monitor 1.2.1 Security 2012 2016

Software Partner 1.1.3 Architecture Apr 01, Mar 31, 268 Failure Low Low Medium Monitor & Design 2011 2016

Mar 31, 249 Student Internships Low Medium Low Monitor 1.6.2 Outreach Jul 01, 2011 2016

Training - AUSS Mar 31, 245 support Low Low Medium Monitor 1.3.1 Training Jul 01, 2011 2016

UNICORE project is 1.1.3 Architecture Jan 01, Mar 31, 261 cancelled Low Low Medium Monitor & Design 2011 2015

Uses may not Oct 22, Oct 22, 344 adequately protect Low Medium Low Monitor 1.2.1 Security their private keys 2012 2016

XD Architecture not Apr 01, Mar 31, 244 fully implemented at Low Low Medium Monitor 1.1 Project Office an XD Service Provider 2011 2016

XD Service Provider has insufficient 1.2.4 Software 262 resources to Low Low Medium Monitor Testing & Jul 01, 2011 Jul 01, implement XSEDE Deployment 2016 Architecture

Acceleration of web 1.6.4 E&O Mar 18, Jul 01, 387 hits is less than Low Low Low Monitor Infrastructure 2013 2016 desired

Advisory Committee Mar 18, Jul 01, 385 Member retention Low Low Low Monitor 1.6.3 Evaluation 2013 2016

Conference call system Oct 22, Oct 22, 340 uses weak or loosely Low Low Low Monitor 2012 2016 managed PINs

Delay in "In-Common 371 Authentication" Low Low Low Monitor 1.7.1 Technology Oct 30, Jun 30, availability Identification 2012 2013

Delay in ticketing 1.3.2 User 329 system interface Low Low Low Monitor Information & Jan 01, Jan 01, release Interfaces 2012 2013

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

May 01, Mar 31, 247 Education Workshops Low Low Low Monitor 1.6.1 Education 2011 2016

Forensic trail is lost because of no Oct 22, Oct 22, 345 centralized logging in Low Low Low Monitor 1.2.1 Security 2012 2016 XSEDE

274 IR IM chats could be 355 exposed without one's Low Low Low Monitor 1.2.1 Security Oct 23, Oct 23, knowledge 2012 2016

Lack of Program Mar 18, Jul 01, 381 Knowledge Low Low Low Monitor 1.6.2 Outreach 2013 2014

Lack of Qualified 1.1.2 Systems and Mar 01, Mar 31, 269 System Administration Low Low Low Monitor Software Personnel Engineering 2011 2016

Lack of available 382 XSEDE Staff to support Low Low Low Monitor 1.6.2 Outreach Mar 18, Jul 01, growing communities 2013 2014

Lack of fit between 1.6.5 Campus Mar 18, Jul 01, 390 offereings and needs Low Low Low Monitor Bridging 2013 2016

One-off authentication Oct 22, Oct 22, 337 systems make account Low Low Low Monitor 1.2.1 Security revocation more fragile 2012 2012

Portal registration Mar 18, Jul 01, 386 inconsistencies Low Low Low Monitor 1.6.3 Evaluation 2013 2016

1.2.4 Software Jul 01, 267 Software Complexity Low Low Low Monitor Testing & Jul 01, 2011 Deployment 2016

Software packaging 1.6.5 Campus Mar 08, Jul 01, 389 plan Low Low Low Monitor Bridging 2013 2014

Some XSEDE resources depend upon Oct 23, Oct 23, 358 proprietary, unvetted Low Low Low Monitor 2012 2016 protocols.

Support and 391 installation issues that Low Low Low Monitor 1.6.5 Campus Mar 18, Jul 01, create a large demand Bridging 2013 2016 on [email protected]

Survey users may not 384 continue to participate Low Low Low Monitor 1.6.3 Evaluation Mar 18, Jul 01, is subsequent years 2013 2016

Unencrypted Oct 22, Oct 22, 342 credentials are Low Low Low Monitor 1.2.1 Security 2012 2016 harvested for attacks

Risk Id Risk Risk Level Probability Impact Status Subproject Monitor Date Retire Dat

User data has weak 353 isolation guarantees on Low Low Low Monitor 1.2.1 Security Oct 23, Oct 23, most resources 2012 2016

support for staff and Mar 18, Jul 01, 383 participants to build Low Low Low Monitor 1.6.2 Outreach 2013 2014 meaningful relationships

Retired risks

Retired Risks

275 D XSEDE Change Control Report PCR # 201301 to incorporate unspent funds into a new cost baseline.

276 E Metrics To demonstrate its success and help focus management attention on areas in need of improvement, XSEDE monitors a wide range of metrics in support of different aspects of “success” for the program. The metrics presented in the quarterly reports provide a view into XSEDE’s user community, including its success at expanding that community, the projects and allocations through which XSEDE manages access to resources, and the use of the resources by those projects (§E.1). In addition, XSEDE has identified metrics describing the program’s success at delivering centralized services to this community, including operations, user support, advanced user support, and education and outreach activities (§E.2). Together, these metrics provide perspectives on how XSEDE works to ensure that the XSEDE-associated services and resources deliver science impact for the science and engineering research community. E.1 XSEDE Resource and Service Usage Metrics Table 7 highlights key XSEDE measures that summarize the user community, the projects and allocations, and resource utilization for the quarter. Notably, XSEDE saw a rebound in open user accounts, exceeding 7,000 for the first time and due in large part the record 1,339 new user accounts in the quarter. Along with the record number of new users, the XSEDE website and portal saw a 30 percent increases in activity, and XSEDE’s online training activity saw twice the visitors from the previous quarter (see §E.2.3). The number of active individuals also climbed, with 2,342 recording Table 7. Quarterly activity summary usage on SP User Community Q2 2012 Q3 2012 Q4 2012 Q1 2013 systems. Gateways Open user accounts 6,636 6,964 6,464 7,042 continue to represent a major part of the Active individuals 2,245 2,148 2,229 2,342 XSEDE community, Gateway users 1,580 1,624 1,629 1,528 with 1,528 users New user accounts 760 863 644 1,339 submitting jobs via Active fields of science 29 29 31 32 gateways. More user Active institutions 375 354 331 364 community details Projects and Allocations are in §E.1.1. NUs available at XRAC 17.939B 22.429B 31.142B 32.781B Project and alloca- NUs requested at XRAC 37.539B 57.611B 71.429B 57.454B tion activity showed NUs recommended by XRAC 23.194B 19.738B 37.963B 26.305B continued high de- NUs awarded at XRAC 17.502B 18.149B 29.820B 26.305B mand, with XSEDE Open projects 1,600 1,612 1,606 1,240 resources requested Active projects 1,027 1,028 1,014 1,096 at 175 percent of Active gateways 16 16 15 15 what was available New projects 247 212 201 224 by 143 XRAC requests. The XRAC Closed projects 207 218 229 280 recommendations, Resources and Usage however, fit within Resources open (all types) 24 24 24 18 the resources Total peak petaflops 2.92 3.54 3.39 6.77 available. During the Resources reporting use 13 14 13 13 quarter, a record Jobs reported 1.70M 1.32M 1.62M 1.63M 1,096 projects made NUs delivered 16.67B 17.9B 17.4B 23.0B use of the resources. Avg wtd run time (hrs) 20.9 22.4 23.4 19.4 More details are in Avg wtd wait time (hrs) 32.6 36.5 30.1 23.8 §E.1.2. Avg wtd slow down 4.7 4.1 3.6 2.9

277 XSEDE computing resources represented 6.77 Pflops (peak) at the end of the quarter, due to the introduction of the TACC Stampede system. The central accounting system showed 13 resources reporting activity, and together they delivered 23.04 billion NUs of computing. This represents an increase of approximately 33 percent over the previous quarter, again due to Stampede. Perhaps also due to the increased capacity from Stampede, XSEDE users experienced shorter wait times, response times, and slow downs, on average. More details are in §E.1.3. E.1.1 User community metrics Figure 6 shows the five-year trend in the XSEDE user community, including open user accounts, total active XSEDE users, active individual accounts, active gateway users, the number of new accounts, and the total number of unvetted user accounts (i.e., portal-only accounts) at the end of each quarter. Unvetted user accounts can be used for training course registration and other functions. The quarter saw a record level of 7,042 open accounts—surpassing 7,000 for the first time—and 1,339 new users—far exceeding the previous record of 1,055 new users one year ago.

Figure 6. XSEDE user census, excluding XSEDE staff.

Figure 7 shows the activity on XSEDE resources according to field of science, including the relative fraction of PIs, open accounts, active users, allocations, and NUs used according to discipline. For consistency across quarters, we show the nine fields of science that typically consume ~2 percent or more of delivered NUs per quarter. PIs and users are counted more than once if they are associated with projects in different fields of science. The quarterly data show that the percentages of PIs and accounts associated with the “other” disciplines represent more than 30 percent of all PIs, 40 percent of user accounts, and more than 30 percent of active users. Collectively the “other” fields of science represent 10 percent of total quarterly usage, led by

278 activity in earth sciences (3.9%) and environmental biology (1.0%).

Figure 8 shows the number of publications, conference papers, and presentations reported by XSEDE users each quarter, including the 1,112 reported by 122 projects in Q4; Appendix F lists these publications according to allocated project. The large increase in Q4 is likely explained, in part, by the number of requests at the XRAC meeting, which received 30 more requests than the prior quarter. Table 8 and Table 9 highlight aspects of the broader impact of XSEDE. The former shows that graduate students, post-doctoral researchers, and undergraduates make up 65 percent of the XSEDE user base. The latter table shows XSEDE’s reach into targeted institutional communities. Institutions with Campus Champions represent a large portion of XSEDE’s usage because this table shows all users at Campus Champion institutions, not just those on the champion’s project. The table also shows XSEDE’s reach into EPSCoR states, the MSI community, and internationally.

279

Figure 7. Quarterly XSEDE user, allocation, and usage summary by field of science, in order by usage, excluding staff projects. Note: PIs and users may appear under more than one field of science.

Figure 8. Publications, conference papers, and presentations reported by XSEDE users

Table 8. End of quarter XSEDE open user accounts by type, excluding XSEDE staff. Category Q2 2012 Q3 2012 Q4 2012 Q1 2013 Graduate Student 2,574 2,678 2,555 2,848 Faculty 1,344 1,386 1,322 1,384 Postdoctorate 1,075 1,109 1,002 1,028

280 Undergraduate Student 639 733 627 721 University Research Staff (excluding postdocs) 535 559 492 528 High school 4 10 13 23 Others 465 489 453 510 TOTALS 6,636 6,964 6,464 7,042

Table 9. Active institutions in selected categories. Institutions may be in more than one category. Category Q2 2012 Q3 2012 Q4 2012 Q1 2013 Campus Sites 62 69 65 71 Champions Users 795 812 876 888 % total NUs 45% 39% 34% 39% EPSCoR Sites 67 66 61 62 states Users 318 324 321 303 % total NUs 20% 15% 17% 10% MSIs Sites 18 17 15 15 Users 40 37 38 48 % total NUs 1% 1% 0.4% 0.7% International Sites 63 44 36 53 Users 86 70 61 90 % total NUs 3% 2% 4% 2% Total Sites 375 354 331 364 Users 2,245 2,148 2,229 2,342 E.1.2 Project and allocation metrics Figure 9 shows the five-year trend for requests and awards at XSEDE quarterly allocation meetings. The figure shows the continued strong growth in demand even with the 40 percent increase in available resources due to the first allocations on the TACC Stampede system. NUs requested were 175 percent of NUs available, and the XRAC recommendations were 80 percent of the NUs available. XSEDE awarded slightly fewer NUs than recommended because requests could not be moved to alternate resources in all cases, due to architectural differences. Table 10 presents a summary of overall project activity. Table 11 shows projects and activity in key project categories as reflected in allocation board type, including the number of open and active user accounts with each type of project. (Science Gateways may appear under any board.). The quarter saw a large drop in open projects, likely due to the retirement of the Ranger and Spur systems; however, the number of active projects—those making use of resources—showed an increase. Table 12 shows detailed information about allocations activity for the various request types available for the different classes of projects. XSEDE had 146 Research (XRAC) requests, of which 123 (84%) received awards, including 47 new projects. There were also 211 Startup requests, of which 163 (77%) received awards; 21 Education requests with 16 awards; and 31 Campus Champion requests with 31 awards.

281 As a special class of projects, science gateway activity is detailed in

Figure 10, showing continued high levels of usage and users from these projects. Table 13 shows gateway activity supported by specific XSEDE resources.

282

Figure 9. Allocation meeting history, showing NUs requested, awarded, available, and recommended. June 2008 was the last MRAC meeting with only “medium” requests.

Table 10. Project summary metrics Project metric Q2 2012 Q3 2012 Q4 2012 Q1 2013 XRAC requests 147 158 188 146 XRAC request success 90% 87% 89% 84% XRAC new awards 39% 36% 38% 38% Startup requests 206 176 142 211 Startup request success 82% 81% 78% 77% Projects open 1,600 1,612 1,606 1,240 Projects new 247 212 201 224 Projects active 1,027 1,028 1,014 1,096 Projects closed 207 218 229 280 Resource diversity (wtd) 1.5 (2.0) 1.5 (2.1) 1.4 (2.0) 1.6 (2.2) SP diversity (wtd) 1.3 (1.7) 1.3 (1.7) 1.3 (1.6) 1.3 (1.6)

Table 11. Project activity by allocation board type. Board Open projects Open users Active projects Active users NUs XRAC 532 4,112 582 1,586 22,078,203,340 Startup 542 1,396 401 519 671,203,288 Campus Champions 80 611 48 99 122,133,160 Educational 59 1,426 47 389 95,998,851 Staff 27 535 16 72 73,432,083 Discretionary 0 0 2 5 1,804,247 Totals 1,240 8,080 1,096 2,670 23,042,774,969

Table 12. Allocations activity in POPS, excluding staff and discretionary projects. Research Startup # Req SUs Req # Awd SUs Awd # Req SUs Req # Awd SUs Awd

283 New 58 205,236,082 47 58,724,836 189 22,294,550 150 14,004,112 Prog. Report 2 7,110,444 2 5,500,000 n/a Renewal 81 541,852,708 74 282,099,808 22 3,163,041 13 1,008,018 Advance 39 40,068,404 36 20,439,151 n/a Justification 9 77,657,226 1 30,000,000 0 0 0 0 Supplemental 33 66,296,757 26 14,281,009 30 2,972,032 25 2,003,027 Transfer 77 25,267,905 72 20,414,053 44 2,594,564 36 1,819,847 Extension 57 n/a 55 n/a 50 n/a 48 n/a Education Campus Champions # Req SUs Req # Awd SUs Awd # Req SUs Req # Awd SUs Awd New 15 2,370,350 11 955,000 9 4,247,000 9 4,096,000 Prog. Report n/a n/a Renewal 6 650,512 5 440,000 22 11,926,070 22 9,232,008 Advance n/a n/a Justification 0 0 0 0 0 0 0 0 Supplemental 8 333,012 4 247,000 5 1,000,000 3 500,000 Transfer 4 222,588 4 237,428 0 0 0 0 Extension 1 n/a 1 n/a 0 n/a 0 n/a

Figure 10. Quarterly gateway usage (NUs), jobs submitted, users (reported by ECSS), registered gateways, and active gateways.

Table 13. Gateway activity by resource. Resource Gateways Jobs NUs SDSC Gordon 2 5,873 121,576,554 SDSC Trestles 6 18,746 116,606,420 NICS Kraken 4 8,761 67,931,390 TACC Stampede 9 3,248 29,996,628 TACC Lonestar4 3 624 4,418,858 TACC Ranger 8 242 2,891,263 PSC Blacklight 1 77 1,063,112

284 Purdue Steele 2 425 2,537 OSG Grid1 1 5 0 E.1.3 Resource and usage metrics This quarter, SP systems delivered 23.04 billion NUs, an increase of about 33 percent from the previous quarter, and 56 percent more NUs than the year-ago quarter. Table 14 breaks out the resource activity according to different resource types. Figure 11 shows the total NUs delivered by XSEDE computing systems, as reported to the central accounting system over the past five years. Figure 12 presents a perspective of the capacity and capability use of XSEDE resources by project. The figure shows the cumulative percentage of projects and resource usage according to each project’s largest reported job size (in cores). The point at which the proverbial 80/20 rule holds precisely is at 72/28; that is the 72 percent of projects whose largest jobs were between 512 and 1,024 cores consumed only 28 percent of the delivered NUs, while the remaining 28 percent of projects, whose largest jobs were all of larger sizes, consumed the remaining 72 percent of delivered NUs. Finally, Table 15 presents some summary metrics to reflect aggregate “usage satisfaction,” including the average run time, wait time, response time (run + wait), and slow down (or expansion factor). These values are presented as unweighted averages, which show the impact of small jobs, and as averages weighted by each job’s portion of the workload (in core-hours), which show responsiveness to the jobs responsible for most of the delivered NUs. Notably for the quarter, while the “average” job is only two hours long, the average weighted job is just more than 23 hours long, and all the weighted usage satisfaction metrics showed decreases, an indicator of faster responsiveness and thus user satisfaction. The weighted average for slow down (3.6) eliminates the skew in the job slow down attributed to small jobs and shows a much more realistic average perceived slowdown for the work delivered. XSEDE provides central monitoring of GRAM5 job submission activity at XSEDE SP sites (Figure 13). GRAM has been deprecated in favor of GRAM5, and thus we are no longer reporting pre-GRAM5 jobs separately.

Table 14. Resource activity, by type of resource, excluding staff projects. Note: A user will be counted for each type of resource used. Type Resources Jobs Users NUs High-performance computing 7 1,337,132 1,897 20,802,651,031 Data-intensive computing 2 69,907 533 2,031,250,926 High-throughput computing 2 221,257 16 92,987,799 Visualization system 2 6,586 105 43,011,312 Totals 13 1,634,882 2,551 22,969,901,068

285

Figure 11. Total XSEDE resource usage in NUs.

Figure 12. Cumulative distribution of projects, jobs, and usage according to project’s maximum job size in cores (excludes staff projects). Vertical line (black) shows “joint ratio” of 72/28 between 512 and 1,024

286 cores. I.e., 72% of projects use fewer than (a bit less than) 1,024 cores and consume 28% of XSEDE NUs; the other 28% of projects have larger jobs and consume the other 72% of XSEDE NUs.

Table 15. Usage satisfaction metrics, for HPC and data-intensive computing resources only, excluding staff projects. Job attribute Q2 2012 Q3 2012 Q4 2012 Q1 2013 Run time (hrs) 2.0 2.0 2.0 2.0 Unweighted Wait time (hrs) 4.4 6.2 4.4 3.1 average Response time (hrs) 7.3 9.7 7.6 5.9 Slow down 324.7 512.2 334.0 562.2 Wtd run time (hrs) 20.9 22.4 23.4 19.4 Weighted Wtd wait time (hrs) 32.6 36.5 30.1 23.8 average Wtd response time (hrs) 53.5 58.9 53.5 43.1 Wtd slow down 4.7 4.1 3.6 2.9

Figure 13. GRAM5 jobs by site 01-01-2013 to 04-01-2013

E.2 XSEDE Program Metrics E.2.1 Project Office 1.1 1.1.4 External Relations The XSEDE External Relations team reported the following media hits for the quarter.

Table 16. XSEDE media hits. Date Source Headline Category Notes 3/28/13 HPCWire NSF Official On New Supers, Data-Intensive Blue Launch, what Blue

Future Waters, Waters and Stampede XSEDE will mean for the future of research and

287 Date Source Headline Category Notes computing 3/27/13 Scientific At Nearly 10 Petaflops, Stampede Provides XSEDE Stampede launch

Computing World Advanced Computing for US Scientists

3/27/13 HPCWire TACC Unveils Dell HPC System Stampede XSEDE Stampede launch

3/27/13 HPCWire Cray and Intel Commit to XSEDE13 Conference XSEDE XSEDE13 3/26/13 NSF NSF-funded Superhero Supercomputer Helps XSEDE Gordon at SDSC Battle Autism - US National Science Foundation

(NSF)

3/26/13 Science Daily Superhero supercomputer helps battle autism XSEDE Gordon at SDSC

3/26/13 Florida FIU and XSEDE to host engineering workshops XSEDE International University Student Media

3/21/13 HPCWire Students Invited to Participate in XSEDE13 XSEDE also on HPC in the Cloud 3/15/13 HPC in the Cloud International Summer School on HPC Challenges XSEDE

in Computational Sciences 3/14/13 Scientific International Summer School on HPC Challenges XSEDE

Computing World in Computational Sciences 3/3/13 InsideHPC Life Sciences on High Performance Computing XSEDE video

Workshop 3/3/13 InsideHPC Video: Life Sciences on High Performance XSEDE John Fonner at the

Computing Workshop TACC

3/1/13 HPCWire SDSC Creating BigData Top100 List XSEDE

2/28/13 HPCWire EGI and XSEDE Launch Data Sharing Initiative XSEDE

2/28/13 HPC in the Cloud EGI and XSEDE Launch Data Sharing Initiative XSEDE 2/28/13 HPC in the Cloud Blue Waters, XSEDE Host Extreme Scaling XSEDE,

Workshop 2013 Blue Waters 2/28/13 HPCWire Blue Waters, XSEDE Host Extreme Scaling XSEDE,

Workshop 2013 Blue Waters 2/26/13 HPCWire Applications for Free HPC Summer School Due XSEDE

March 18 2/26/13 HPCWire At XSEDE13 Conference, Jul 24 Panel On XSEDE

Bioscience and Computation 2/22/13 redOrbit Researchers Use Black Holes To Probe The Limits XSEDE

Of Spacetime - Space News

2/20/13 Datanami SQLstream Announces University Partner Program XSEDE 2/20/13 HPCWire 2013 Tapia Conference Celebrates Most Successful XSEDE Sponsoring

Meeting 2/13/13 HPCWire SDSC’s Chaitan Baru Named Associate Director, XSEDE

Data Initiatives

2/6/13 TribLIVE Supercomputing Center, Ansys in simulation effort XSEDE 2/4/13 HPCWire PRACE Announces International Summer School XSEDE

for HPC Challenges 2/2/13 InsideHPC International Summer School on HPC Challenges XSEDE

in Computational Sciences

2/1/13 Newswise.com SDSC Mourns the Loss of Dr. Robert P. Harkness Blue A research using Blue Waters, Waters passed away XSEDE

2/1/13 HPCWire SDSC Mourns the Loss of Dr. Robert P. Harkness Blue A research using Blue Waters, Waters passed away XSEDE

1/31/13 HPCWire XSEDE Role in Science Education XSEDE Page 5 of "The Week in HPC Research"

1/30/13 InsideHPC Using Supercomputers to Model Aortic Aneurysms XSEDE Got it from the XSEDE external newsletter

1/30/13 Datanami eScience and a Tale of Two Cities XSEDE Written by Elizabeth

288 Date Source Headline Category Notes Alum Leake, XSEDE mentioned in bio

1/24/13 HPCWire Globus Online Passes XSEDE Acceptance Test XSEDE *Same release on HPC in the Cloud* 1/14/13 HPCWire Genome Analysis Support Center Announces New XSEDE

Initiatives 1/14/13 Indiana University Genome analysis support center announces new XSEDE,

initiatives to foster discovery NCSA 1/11/13 Quartz Two supercomputers crunched the data and XSEDE concluded high frequency trading has “little impact

on our lives” 1/10/13 Inside Illinois High-frequency stock trading of little value to XSEDE

investors, public 1/10/13 e! Science News High-frequency stock trading of little value to XSEDE Doesn't clarify

investors, general public XSEDE-allocated resources were used, just says "The research was sponsored by the National Science Foundation's Extreme Science and Engineering Discovery Environment program." 1/3/13 EurekAlert Unconventional visualization method wins jury XSEDE,

prize at media festival NCSA

E.2.2 Operations 1.2 1.2.1 Security The XSEDE security team has identified the following metrics for tracking security incidents and response. The table summarizes the metrics, and details on any incidents are provided in the main body of the report. Table 17. XSEDE security metrics and incident response Q2 2012 Q3 2012 Q4 2012 Q1 2013 XSEDE-wide notice of vulnerability 0 0 0 1 Compromised user accounts 0 2 0 0 Other incident response 0 0 0 0 Critical rollout of vulnerability patches 0 0 0 0 Security enhancement rollouts 1 0 1 4 1.2.2 Data Services XSEDE supports monitoring for two central data movement services: the gridFTP service connecting the XSEDE service providers and the Globus Online service for connecting XSEDE service providers as well as external sites. Table 18 shows quarterly summary metrics and increasing Globus Online adoption over the past four quarters, while Figure 14 and Figure 15 show Globus Online and GridFTP activity, respectively, by SP site. (We are examining why Globus Online activity at some sites, such as NICS, is larger than their more general GridFTP activity.)

Table 18. Globus Online activity to and from XSEDE endpoints, excluding GO XSEDE speed page user. Q2 2012 Q3 2012 Q4 2012 Q1 2013 To/from Files to XSEDE 34.8 million 57.1 million 14.1 million 33.9 million XSEDE TB to XSEDE 824 311 559 642

289 endpoint Files from XSEDE 23.9 million 44.9 million 17.7 million 39.1 million TB from XSEDE 185 325 453 650 Faults detected 971,000 985,000 1,561,000 1,296,000 Users 187 218 217 258 Files to XSEDE 2.5 million 24.9 million 2.1 million 2.3 million To/from TB to XSEDE 18 37 37 51 XSEDE Files from XSEDE 2.2 million 9.4 million 4.9 million 7.6 million via TB from XSEDE 11 34 23 47 Globus Connect Faults detected 381,000 575,000 770,000 570,000 Users 97 138 124 139

Figure 14. Globus Online activity into and out of XSEDE SP end points

290

Figure 15. GridFTP volume and file transfers, per SP site.

1.2.3 XSEDEnet Traffic utilization of the Chicago-Denver XSEDEnet NLR link is shown in two figures below. Figure 16 shows the peak bandwidth across the link for the period. Figure 17 shows link utilization as a percentage. The network team monitors backbone utilization for capacity planning; 50 percent utilization for three weeks was the threshold that would trigger a link upgrade process. Traffic across all XSEDEnet NLR links is shown in Figure 18. This quarter, XSEDE transitioned from NLR to the Internet2 AL2S backbone, which is why the network figures end in early March. The transition also entailed migrating to new reporting tools, and new charts will be ready for the next quarterly report. The new tools will allow XSEDE to use site metrics instead of backbone-only metrics. When usage reaches 50 percent over three weeks at any site, the site will be contacted to determine the possibility of upgrading their bandwidth.

291

Figure 16. XSEDEnet Chicago-Denver peak bandwidth

Figure 17. XSEDEnet Chicago-Denver utilization (as a percentage)

Figure 18. XSEDEnet Q3 2012 aggregate bandwidth across all links

292 1.2.5 Accounting and Account Management The Accounting and Account Management group administers and operates the software for the XSEDE allocations system (POPS), the accounting system, and user account management. Table 19 shows the processing time for ongoing allocation requests outside of the quarterly XRAC requests. XSEDE reduced the account creation time to a matter of minutes with the deployment of POPS and User Portal components that allow users to create their own portal logins; because of this improvement, this metric will now be constant, so we will no longer be reporting it.

Table 19. Average time to process allocation requests and account creation requests, in days. (Excludes quarterly XRAC requests; “n/a” indicates none submitted.) ALLOCATION REQUESTS Q2 2012 Q3 2012 Q4 2012 Q1 2013 Research Advance 7 5 12 4 Transfer 3 5 3 3 Supplement 14 7 17 12 Justification n/a 44 5 68 Startup, Education, New 9 10 12 9 Campus Champions, Renewal 6 3 7 5 Discretionary Transfer 3 4 3 2 Supplement 4 8 5 8 Account creation requests 0.03 0.03 0.03 0.03* 1.2.6 Systems Operational Support The Systems Operational Support group encompasses the XSEDE Operations Center (XOC), which includes front-line user support and the ticket system, and the system administration of all XSEDE centralized services. In the ticket system, XSEDE tracks total ticket volume and responsiveness (Table 20), which groups (“resolution centers”) field the tickets (Table 21), and the numbers of tickets in seven common categories (Table 22 and Figure 19). Starting this quarter, the tables and figure consider only the tickets that fall into the standard resolution centers and categories, which represent the bulk of XSEDE ticket activity. For the central services, XSEDE tracks the uptime reported by system administrators (Table 23) as well as the Inca-reported uptime for seven key user-visible services (Table 24). The Inca- reported uptime better reflects “user-visible outages,” that is, what the average user would experience, and typically exceeds the actual system uptime, reflecting the effectiveness of XSEDE’s backup systems, failover capabilities, and operational responsiveness.

Table 20. XSEDE Operations Center ticket system metrics. Q2 2012 Q3 2012 Q4 2012 Q1 2013 Total tickets opened 2,744 2,421 2,098 1,823 Tickets opened – email 2,448 2,175 1,872 1,785 Tickets opened – portal 26 18 18 8 Tickets opened – phone 270 228 208 30 Total tickets closed 2,394 2,028 2,021 1,695 Tickets, response in 24 hrs 2,326 (85%) 2,021 (83%) 1,727 (82%) 1,806 (99%) Tickets closed within 2 bus. days 1,050 (38%) 880 (36%) 742 (35%) 720 (39%)

Table 21. Ticket breakdown (opened/closed) for each major resolution center. Q2 2012 Q3 2012 Q4 2012 Q1 2013 Open Closed Open Closed Open Closed Open Closed NICS 657 640 481 458 514 503 449 435 XOC 415 298 349 231 290 171 131 131 TACC 276 251 292 257 253 218 461 416 Proposal issues 408 376 403 358 381 300 355 324 SDSC 379 327 313 251 298 242 235 211 PSC 89 87 75 64 50 48 42 41

293 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Open Closed Open Closed Open Closed Open Closed Purdue 118 115 82 82 60 60 57 56 NCSA 89 72 83 81 21 18 12 7 User facing services 73 65 25 22 67 65 36 35 UST 11 11 11 11 5 5 42 38 IU 10 7 3 3 2 2 1 1 OSG 3 3 1 1 2 1 2 1 Others 215 135 303 209 153 85 n/a

Figure 19. Tickets in the seven primary problem categories.

Table 22. Ticket counts for the seven primary problem categories shown in Figure 19. Q2 2012 Q3 2012 Q4 2012 Q1 2013 Login / access issues 445 345 326 350 Jobs / batch queues 510 407 342 311 Software / apps 295 224 210 239 Account issues 152 150 148 126 MSS / data issues 79 76 63 57 File systems 99 75 80 93 System issues 105 84 70 50

294 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Service % Up Hrs (P|U) % Up Hrs (P|U) % Up Hrs (P|U) % Up Hrs (P|U) Authority Data Movement Service Globus Listener 99.82% 4|0 IIS Metrics 99.82% 4|0 Inca 99.31% 0|15 98.71% 26|2.5 Inca backup 99.50% 11|0 99.91% 0|2 Information 99.82% 4|0 Services Karnak 99.82% 4|0 99.98% 0|0.5 91.17% 0|195 88.70% 0|244 Kerberos backup Knowledgebase MyProxy Openfire Jabber POPS RDR 99.64% 0|8 99.35% 0|14 Sciforma 99.84% 3.5|0 Secure Wiki SELS Sharepoint Software 99.82% 4|0 Distribution Source Repository 99.82% 4|0 Speedpage 99.95% 1|0 TG Wiki 99.82% 4|0 Ticket System Usage Reporting Tools User Portal User Portal backup 99.95% 1|0 99.91% 0|2 User Profile Service XDCDB 99.91% 0|2 99.97% 0|0.75 99.90% 0.06|2.25 99.91% 2|0 XDCDB backup 99.91% 2|0 98.91% 0|24 99.64% 0|8

Table 24. Inca-monitored XSEDE central services, Inca-detected uptimes, and outages. Service Outage definition. Test frequency. Q2 2012 Q3 2012 Q4 2012 Q1 2013 Inca status pages unavailable or test details page Inca 99.3% 98.71% 99.71% 99.98% fails to load. Every 5 min. Information Information web pages unavailable. Every 15 min. 99.9% 100% 99.76% 99.99% Services Karnak Karnak front page fails to load. Every 30 min. 99.9% 99.98% 91.17% 88.70% MyProxy server does not respond to credential MyProxy 100% 100% 100% 100.00% check. Every hour. Portal home page fails to load correctly. Every 30 User Portal 100% 99.98% 100% 99.99% min. Connection to database refused or slow (using XDCDB 99.99% 99.96% 100% 99.98% check_postgres.pl script). Every 5 min. E.2.3 User Services 1.3 1.3.1 Training We are now including aggregate training metrics, including number of events held and attendees as well as online modules and visitors, in Table 25.

295 Table 25. Training events and attendees Q4 2012 Q1 2013 Events held 23 20 Event attendees 858 1,268 Online modules available 41 44 Online module unique visitors 7,774 13,163 Online module repeat visitors 1,444 3,173 1.3.2 User Information & Interfaces The User Information and Interfaces group provides XSEDE users with central information and services via the XSEDE User Portal (XUP), website, XUP mobile, and knowledgebase. Table 26 shows increasing activity on most user information interfaces, as well as increases in the numbers of logged-in users accessing these interfaces. Table 27 shows the most popular XUP applications, by visits.

Table 26. XSEDE website, user portal and XUP Mobile activity. (Note: “Users” indicates logged-in users.) UII Activity Q2 2012 Q3 2012 Q4 2012 Q1 2013 Web hits 2,676,532 2,362,105 2,269,506 3,434,497 Web visitors 33,362 33,053 33,145 51,018 XUP hits 1,465,537 1,540,402 1,558,209 2,135,182 XUP visitors 13,787 16,118 16,998 23,582 XUP accounts 7,563 9,015 10,000 12,563 XUP users 3,976 4,399 4,346 5,722 XUP users running jobs 1,552 1,552 1,640 1,818 XUP Mobile hits 1,693 1,931 3,569 6,668 XUP Mobile users 25 20 17 20 KB docs retrieved 57,451 87,987 187,966 171,097 Total KB docs 497 538 573 586 New KB docs 19 62 36 10

Table 27. XUP and website application visits. “Users” indicates logged-in users. Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Application Visits Users Visits Users Visits Users Visits Users Visits Users Allocations/Usage 49,261 3,803 50,998 3,724 45,899 3,773 44,572 3,649 55,103 4,863 User News 62,784 526 47,194 2,013 25,707 263 26,992 338 42,830 509 Training Regis’n 19,535 828 11,642 631 10,556 650 18,726 1,011 28,503 1,703 File Manager 31,952 79 21,375 78 42,219 97 37,852 69 24,307 85 (xfers) (2.5 TB) (xfers) (20 TB) (xfrs) (3.6 TB) (10 TB) (9.5 TB) GSI-SSH 24,714 1,343 28,317 1,354 22,411 1,286 21,352 1,248 22,543 1,509 Resource Listing 36,244 1,104 22,483 1,265 16,934 1,391 16,869 1,436 19,901 1,696 Publications 2,908 595 12,528 1,372 18,214 1,694 RSS news feed 11,959 92 14,171 109 Help Desk/Consult 4,659 748 4,815 733 4,704 652 10,031 721 13,955 975 Knowledge Base 11,436 377 10,964 338 8,269 156 8,699 217 12,178 301 Software Search 8,204 265 8,879 316 9,903 318 9,707 344 10,911 434 System Accounts 12,897 1,841 12,235 1,813 10,919 1,731 9,037 1,632 10,828 1,937 User Profile 1,954 833 5,476 1,262 5,930 1,345 6,314 1,312 8,083 1,626 System Monitor 8,213 1,257 9,401 1,034 6,819 985 6,996 888 7,933 1,085 Online Training 2,093 347 1,668 437 1,414 220 1,452 304 7,923 753 Listing POPS Submit 5,639 1,204 5,982 1,117 5,735 1,118 5,974 1,294 Add User Form 5,751 626 4,628 617 4,446 547 3,710 523 5,367 664 Gateways List 16,338 229 8,914 234 5,470 158 5,043 152 4,929 161 Ticketing System 2,366 682 2,600 654 2,229 609 1,996 532 2,483 622 My Jobs 3,500 960 3,195 900 2,634 788 2,484 777 2,463 836 SU Calculator 2,048 243 1,466 239 1,240 109 1,336 188 1,519 244 XSEDE Tech DB 330 15 1,388 41

296 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Application Visits Users Visits Users Visits Users Visits Users Visits Users Karnak Q Predict 1,108 222 871 264 643 182 601 153 628 149 Community Accts 436 311 318 259 279 206 387 231 Gateway Regis’n 243 16 238 8 275 12 234 11

E.2.4 Extended Collaborative Support Service 1.4, 1.5 The Extended Collaborative Support Service pairs members of the XSEDE user community with expert staff in projects lasting up to a year to solve challenging science and engineering problems. Table 28 shows project and staffing metrics. Table 29 shows metrics for Extended Support for Training, Education, and Outreach. Table 28. Extended Collaborative Support project and staffing activity Q2 2012 Q3 2012 Q4 2012 Q1 2013 Project requests XRAC 20 14 14 25 Supplemental/Startups 28 30 18 30 ECSS In-house project 0 0 0 0 Projects initiated Research Team 14 15 13 15 Community Codes 7 3 7 8 Science Gateways 6 2 1 6 Unassigned 0 2 1 0 Projects cancelled/no- XRAC 5 5 0 11 go Supplemental/Startups 16 17 10 15 Projects active Research Team (XRAC) 29 26 30 37 Research Team (S/S) 20 19 24 30 Research (TG) 1 0 0 0 Subtotal 50 45 54 67 Community Codes (XRAC) 10 9 9 9 Community Codes (S/S) 17 18 7 20* Community Codes (TG) 2 1 0 0 In-house 1 1 0 2 Subtotal 30 29 16 31 Science Gateways (XRAC) 9 9 7 7 Science Gateways (S/S) 10 8 8 13 Science Gateways (TG) 1 0 0 0 Subtotal 20 17 15 20 Total Projects Active 100 91 85 118 Work plans submitted 10 14 9 6 Projects completed (allocation ended in quarter) 15 11 11 39 Novel, Innovative User groups engaged 47 65 72 82 Projects (NIP) NIP-led ECS planning efforts 9 3 5 2 NIP ECS requests 38 42 54 59 (prospective user groups) NIP ECS projects active 9 10 10 13 NIP outreach events 16 21 30 31 ECSS staffing (FTE) Research Team 12.8 13.19 15.54 15.69 Community Codes 8.24 8.14 5.60 5.35 Science Gateways 4.73 4.63 4.89 4.97 NIP 5.64 5.64 5.05 5.10 TEO 3.53 3.43 3.87 3.18 Total 34.94 35.03 34.95 34.29 * Note: Includes 12 Trinity and Allpaths-LG projects

Table 29. Extended Support for Training, Education and Outreach 1.5.3 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Description # Staff # Staff # Staff # Staff Requests for service 2 2 0 0 4 4 0 0

297 User meetings and BOFs 18 20 11 12 13 13 6 7 Mentoring 12 12 9 9 2 2 2 2 Talks and presentations 20 20 15 22 9 12 3 3 Tutorials 31 32 12 18 5 10 9 11 Online tutorials and webinars 0 0 3 3 0 0 2 4 Online tutorial reviews 17 17 2 2 4 5 3 3

298 F XSEDE Publications Listing F.1 XSEDE Staff Publications F.1.1 Project Office 1.1 F.1.2 Operations 1.2 1. Kluge, M., S. Simms, T. William, R. Henschel, A. Georgi, C. Meyer, M. S. Mueller, C. A. Stewart, W. Wünsch, and W. Nagel, "Performance and quality of service of data and video movement over a 100 Gbps testbed" Future Generation Computer Systems, vol. 29, no. 1, Jan 2013. 2. Hallock, B., Rockhopper: Penguin on Demand at Indiana University, Presented in-booth at the Plant and Animal Genome XXI conference, San Diego, CA., Jan 2013. F.1.3 User Services 1.3 F.1.4 Extended Collaborative Support Service – Projects 1.4 3. Zhang, H., and M. J. Boyles, "Visual exploration and analysis of human-robot interaction rules" SPIE, Visualization and Data Analysis, vol. 8654, Burlingame, California, Feb 2013. 4. Zhang, H., and M. J. Boyles, Visual exploration and analysis of human-robot interaction rules, SPIE, Visualization and Data Analysis 2013, Burlingame, California, Feb 2013. 5. LeDuc, R., Statistical Consideration for Identification and Quantification in Top-Down Proteomics, American Society for Mass Spectrometry - Sanibel Conference 2013, St Pete Beach, FL, Jan 2013. 6. Walker, R., and Clark, M. “An extensible interface for QM/MM molecular dynamics simulations with AMBER,” submitted to J Comp Chem, 2013. F.1.5 Extended Collaborative Support Service – Communities 1.5 7. Pierce, M., Science Gateways, OSG All Hands Meeting , Indiana University - Purdue University 420 University Blvd. Indianapolis, IN 46202, Mar 2013. 8. Pierce, M., S. Marru, S. Wijeratne, R. Singh, and H. Suriyaarachchi, Apache Airavata: Building Gateways to Innovation, ApacheCon NA, Portland Or., Feb 2013. F.1.6 Education and Outreach 1.6 9. Gordon, S.I., “Advancing Computational Science Education through Xsede,” Computing in Science & Engineering , 15(1):90-92, Jan.-Feb. 2013, doi: 10.1109/MCSE.2013.2. 10. Hallock, B., Campus Bridging: What is it and why is it important?, OSG All Hands Meeting, Mar 2013. F.2 Publications from XSEDE Users The following publications were gathered from research submissions to the March 2013 XSEDE Resource Allocations Committee (XRAC) meeting. Renewal submissions are required to provide a file specifically to identify publications resulting from the work conducted in the prior year. The publications are organized by the proposal with which they were associated. This quarter, 83 requests (out of 143) identified 735 publications and conference papers that were published, in press, accepted, submitted, or in preparation. 1. AST020001 1. Aubert, O., Le Bars, M., Le Gal, P., Marcus, P., 2012. The universal aspect ratio of vortices in rotating stratified flows: experiments and observations. Journal of Fluid Mechanics 706, 34. 2. Hassanzadeh, P., Marcus, P., Le Gal, P., 2012. The universal aspect ratio of vortices in rotating stratified flows: theory and simulation. Journal of Fluid Mechanics 706, 46. 3. Marcus, P., Asay-Davis, X.,Wong, M., de Pater, I., 2013. Jupiter’s Red Oval BA: Dynamics, Color, and Relationship to Jovian Climate Change. The Journal of Heat Transfer accepted for publication September 2012 and to appear Januray 2013. 4. Marcus, P. S., Pei, S., Jiang, C.-H., Hassanzadeh, P., 2012. Three-dimensional vortex lattices created by the self-replication of stable stratified layers of rotating flows. Physical Review Letters Under review. 5. Martin, S., de Pater, I., Marcus, P., 2012. Neptune’s zonal winds from near-IR Keck adaptive optics imaging in August 2001. Astrophysics and Space Science 337 (1), 65–78.

299 2. AST040008 6. Three-Dimensional Relativistic Magnetohydrodynamic Simulations of Current-Driven Instability. III. Rotating Relativistic Jets, Mizuno, Y., Y. Lyubarsky, K.-I. Nishikawa, & P. E Hardee, ApJ, 757, 16 (14pp), 2012 7. Current Driven Instability of a SUB-ALFV´ENIC Relativistic Jet, Hardee, P. E., Y. Mizuno, K.-I. Nishikawa, Int. J. of Mod. Phys. Conf. Ser., 8, 340, 2012 8. Relaxation of Pulsar Wind Nebula via Current-Driven Kink Instability, Mizuno, Y., Y. Lyubarsky, K.-I. Nishikawa, & P. E Hardee, Int. J. of Mod. Phys. Conf. Ser., 8, 368, 2012 9. Magnetic Field Amplification by Relativistic Shocks in an Inhomogeneous Medium, Mizuno, Y., M. Pohl, J. Niemiec, B. Zhang, K.-I. Nishikawa, & P. E Hardee, Int. J. of Mod. Phys. Conf. Ser., 8, 364, 2012 10. Simulation of relativistic shocks and associated radiation, Nishikawa, K.-I., J. Niemiec, B. Zhang, M. Medvedev, P. Hardee, Y. Mizuno, Å. Nordlund, J. Frederiksen, H. Sol, M. Pohl, D. H. Hartmann, G. J. Fishman, Int. J. Mod. Phys. Conf. Ser., 8, 259, 2012 11. Nishikawa, K.-I., B. Zhang, I. Dutan, M. Medvedev, P. Hardee, E.-J. Choi, K. Min, J. Niemiec, Y. Mizuno, Å. Nordlund, J. Frederiksen, H. Sol, M. Pohl, D. H. Hartmann, J. F. Fishman, Radiation from accelerated particles in relativistic jets with shock, shear-flow, and reconnection, Proceedings for Fall 2012 Gamma Ray Burst Symposium, EAS Pub. Ser., submitted, 2012 12. Mizuno, Y. M. Pohl, J. Niemiec, B. Zhang, K.-I. Nishikawa, P. E. Hardee, Magnetic Field Amplification and Saturation by Turbulence in A Relativistic Shock Propagating through An Inhomogeneous Medium, Proceedings for Fall 2012 Gamma Ray Burst Symposium, EAS Pub. Ser., submitted, 2012 13. Nishikawa, K.-I., E.-J. Choi, K. Min, P. Hardee, Y. Mizuno, B. Zhang, J. Niemiec, M. Medvedev, A. Nordlund, J. Frederiksen, H. Sol, M. Pohl, D. H. Hartmann, J. F. Fishman, Radiation from shock-accelerated particles, Proceeding of Science, PoS(GRB 2012)028, 2012 (http://pos.sissa.it/cgi- bin/reader/conf.cgi?confid=152) 14. Nishikawa, K.-I., B. Zhang, E.-J. Choi, K. Min, P. Hardee, Y. Mizuno, J. Niemiec, M. Medvedev, A. Nordlund, J. Frederiksen, H. Sol, M. Pohl, D. H. Hartmann, J. F. Fishman, Radiation from accelerated particles in shocks and reconnection regions, IAU Symposium 279: Death of Massive Stars: Supernovae & Gamma-Ray Bursts, 371, 2012 15. Mizuno, Y., Y. Lyubarsky, K.-I. Nishikawa & P. E. Hardee, The Current-Driven Kink Instability in Magnetically Dominated Relativistic Jets, Proceedings for Waves and Instabilities in Space and Astrophysical Plasmas, AIP Conf. Proc. 1439, pp. 226-236, 2012 16. Nishikawa, K.-I., E.-J. Choi, K. Min, P. Hardee, Y. Mizuno, B. Zhang, J. Niemiec, M. Medvedev, A. Nordlund, J. Frederiksen, H. Sol, M. Pohl, D. H. Hartmann, J. F. Fishman, Simulation of Relativistic Jets and Associated Self-consistent Radiation, the proceeding of 6th International Conference on Numerical Modeling of Space Plasma Flows, June 13 - 17, 2011, Valencia, Spain, ASP Conference Series, v. 459, 143, 2012 3. AST040024 17. Pan, K.-C., Ricker, P. M., and Taam, R. E. “Evolution of Post-Impact Companion Stars in SNIa Remnants within the Single-Degenerate Scenario.” 2012, ApJ, 760, 21 18. Pan, K.-C., Ricker, P. M., and Taam, R. E. “Impact of Type Ia Supernova Ejecta on Binary Companions in the Single-Degenerate Scenario.” 2012, ApJ, 750, 151 19. Ricker, P. M. and Taam, R. E. “An AMR Study of the Common Envelope Phase of Binary Evolution.” 2012, ApJ, 746, 74 20. Sutter, P. M., Yang, H.-Y., Ricker, P. M., Foreman, G., and Pugmire, D. “An Examination of Magnetized Outflows from Active Galactic Nuclei in Galaxy Clusters.” 2012, MNRAS, 419, 2293 4. AST100001 21. W. Zhang, L. Howell, A. Almgren, A. Burrows, J.C. Dolence, & J. Bell, ”CASTRO: A New Compressible Astrophysical Solver. III. Multigroup Radiation Hydrodynamics,” accepted to Astrophys. J. (arXiv:1207.3845). 22. J. Nordhaus, T. Brandt, A. Burrows, & A. Almgren, ”The Hydrodynamic Origin of Neutron Star Kicks,” M.N.R.A.S., 423, 1805, 2012 (arXiv:1112.3342). 23. E. Abdikamalov, A. Burrows, C.D. Ott, F. Loeffler, E. O’Connor, J.C. Dolence, & E. Schnetter, ”A New Monte Carlo Method for Time-Dependent Neutrino Radiation Transport,” Astrophys. J., 755, 111, 2012 (arXiv:1203.2915). 24. Adam Burrows, Josh Dolence, & Jeremiah Murphy, ”An Investigation into the Character of Pre-Explosion Core-Collapse Supernova Shock Motion,” 2012, accepted to Astrophys. J., arXiv:1204.3088. 25. J.W. Murphy, J.C. Dolence, & A. Burrows, ”The Dominance of Neutrino-Driven Convection in Core-Collapse Supernovae,” submitted to Astrophys. J., 2012 (arXiv:1205.3491). 26. Adam Burrows, ”Thoughts on Core-Collapse Supernova Theory,” submitted to Rev. Mod. Phys., 2012.

300 27. J.C. Dolence, A. Burrows, & J.W. Murphy, ”Dimensional Dependence of the Hydrodynamics in Core- Collapse Supernovae,” submitted to the Astrophysical Journal, 2012. 18.2 28. Adam Burrows, ”Multi-Dimensional Core-Collapse Supernova Explosions,” to appear in the proceedings of the 11th International Symposium on the Origin of Matter and Evolution of Galaxies (OMEG11)), held at RIKEN, Tokyo, Japan on Nov. 14 - Nov. 17, 2011 (AIP Conference series), eds. K. Nakamura, T. Kajino, and S. Kubono. 5. AST100014 29. “Direct MD simulation of liquid-solid phase equilibria for two-component plasmas,” A. S. Schneider, J. Hughto, C. J. Horowitz, D. K. Berry, Physical Review E 85, 066405 (2012). 30. “Durability of neutron star crust", Andrey Chugunov and C. J. Horowitz, Contributions to Plasma Physics 52, 122 (2012). 31. “Direct MD simulation of liquid-solid phase equilibria for three-component plasma", J. Hughto, C. J. Horowitz, A. S. Schneider, Zach Medin, Andrew Cumming, Physical Review E 86, 066413 (2012). 32. “Multi-messenger probes of neutron rich matter", C. J. Horowitz, proceedings of Yukawa Institute Symposium Frontier issues in physics of exotic nuclei, Progress Theoretical Physics Suppl. 196 (2012) 451. 33. “The carbon-oxygen phase diagram of plasma mixtures in white dwarf stars", A. S. Schneider, C. J. Horowitz, J. Hughto and D. K. Berry, J. Phys. Conf. Ser. 402, 012026 (2012). 34. “Structure and Shear Modulus of the Neutron Star Crust", J. Hughto, J. Phys. Conf. Ser. 342, 012005 (2012). 35. “Multi-messenger observations of neutron rich matter", Asia Pacific Center for Theoretical Physics (APCTP), C. J. Horowitz, Pohang, Korea, April 2012. 36. “Equation of state in astrophysics and the symmetry energy", Asia Pacific Center for Theoretical Physics (APCTP), C. J. Horowitz, Pohang, Korea, April 2012. 37. “Multi-messenger observations of neutron rich matter", Conference Intersections of Particle and Nuclear Physics, C. J. Horowitz, St. Petersburg, FL, June 2012. 38. “Multi-messenger observations of neutron rich matter", Conference COMPSTAR: the physics and astrophysics of compact stars, C. J. Horowitz, Tahiti, French Polynesia, June 2012. 39. “Neutron stars, neutrinos, and gravitational waves", Low Energy Town Hall Meeting, Argonne, Ill, C. J. Horowitz, Aug 2012. 40. “Nuclear structure, neutron stars, and gravitational waves", Conference on Nuclear Physics: Extremes of Nuclear Landscape, Zakopane, Poland, C. J. Horowitz, Aug 2012. 41. “Review of Multi-messenger observations of neutron rich matter", Plenary talk at conference Quark Confinement and Hadron Spectrum X, Munich, Germany, C. J. Horowitz, Oct 2012. 42. “Neutron rich nuclei and neutron stars", International Conference Fusion and Neutron Rich Nuclei 5, Sanibel Island FL, Nov 2012. 6. AST110050 43. Zalucha, A. M., and T. I. Michaels (2012), “A 3D general circulation model for Pluto and Triton with fixed volatile abundance and simplified surface forcing,” Icarus, submitted. 7. ATM050012 44. Dorier, M., G. Antoniu, F. Cappello, M. Snir, and L. Orf, 2012: Damaris: How to efficiently leverage multicore parallelism to achieve scalable, jitter-free I/O. 2012 45. Chiu, S. L., B. F. Jewett and R. B. Wilhelmson, 2012: Mesoscale and stormscale ingredients of tornadic supercells producing long-track tornadoes in the 2011 Alabama super outbreak. 26th Conf. on Severe Local Storms, Amer. Meteor. Soc., Nashville, 5.4. 46. ___, ___, and ___, 2012: The 2010 Mississippi Valley tornado outbreak - a mesoscale and stormscale perspective. 14th Conference on Mesoscale Processes, American Meteorological Society, Los Angeles, CA. IEEE International Conference. 47. Syrowski, A., B. F. Jewett, and R. Wilhelmson, 2012: An assessment of internal and external forcings in supercell interactions and their impact on storm morphology. 26th Conf. on Severe Local Storms, Amer. Meteor. Soc., Nashville, 11B.5. 48. Van Leer, K. W., B. F. Jewett and R. B. Wilhelmson, 2012: Rapid intensification mechanisms including the role of storm mergers in the 22 May 2011 Joplin, MO tornadic storm. 26th Conf. on Severe Local Storms, Amer.Meteor.Soc., Nashville, 6.2. 49. ___, ___, ___, 2013: Conference notebook summary. Bull. Amer. Meteor. Soc., in press.

301 8. ATM070007 50. P.L. Pritchett, F.S. Mozer, and M. Wilber, “Intense Perpendicular Electric Fields Associated with Three- Dimensional Magnetic Reconnection at the Subsolar Magnetopause,” J. Geophys. Res., 117, A06212, doi:10.1029/2012JA017533, 2012. 51. P.L. Pritchett and F.V. Coroniti, “Structure and Consequences of the Kinetic Ballooning/Interchange Instability in the Magnetotail,” J. Geophys. Res., in press, 2012. 52. P.L. Pritchett, “The Influence of Intense Electric Fields on Three-Dimensional Asymmetric Reconnection,” Phys. Plasmas, in press, 2012. 9. ATM070010 53. P. Zhu, J. Raeder, C. C. Hegna, and C. R. Sovinec, Nature of axial tail instability and bubbleblob formation in near-Earth plasma sheet, J. Geophys. Res. (in press) (2012). 10. ATM100026 54. "Kinetic Simulations of 3D Farley-Buneman Turbulence and Anomalous Electron Heating" by M. Oppenheim and Y. Dimant (submitted on December 4, 2012, manuscript #: 2012JA018556). This will come out in early 2013. This includes acknowledgment of NICS and the Teragrid grant by number. 11. ATM110006 55. DePalma, J.W.; Bzdek, B.R.; Doren, D.J.; Johnston, M.V. Structure and Reactivity of Nanometer Size Clusters of Sulfuric Acid with Ammonia and Dimethylamine. J. Phys. Chem. A, 2012, 116, 1030-1040. 56. DePalma, J.W.; Horan, A.J.; Hall IV, W.A.; Johnston, M.V. Thermodynamics of Oligomer Formation: Implications for Secondary Organic Aerosol Formation and Reactivity. Submitted to Physical Chemistry Chemical Physics. 57. Bzdek, B.R.; DePalma, J.W.; Ridge, D.P.; Laskin, J.; Johnston, M.V. Fragmentation Energetics of Clusters Relevant to Atmospheric New Particle Formation. Submitted to the Journal of the American Chemical Society. 12. ATM120010 58. 1. Li, H., & Misra, V. (submitted). Global Seasonal Climate Predictability in a Two Tiered Forecast System. Part II: Boreal winter and spring seasons. Climate Dynamics. Manuscript submitted for review, 47 pages. Available from http://floridaclimateinstitute.org/images/document_library/publications/fish50-paperpartII- final.pdf 59. Li, H., Kanamitsu, M., Hong, S. -Y., Yoshimura, K., Cayan, D. R., & Misra, V. (submitted). A high-resolution ocean-atmosphere coupled downscaling of present climate over California. Climate Dynamics. Manuscript submitted for review, 34 pages. 60. Misra, V., Li, H., Wu, Z., & DiNapoli, S. (submitted). Global seasonal climate predictability in a two tiered forecast system. Part I: Boreal summer and fall seasons. Climate Dynamics. Manuscript submitted for publication, 47 pages. Available from http://floridaclimateinstitute.org/images/document_library/publications/fish50-paperpartI-final.pdf 61. Misra, V. and H. Li, 2012: Seasonal predictability of the Asian summer monsoon in a two-tiered forecast system. Climate Dynamics. Manuscript submitted for review, 37 pages 62. Bastola, S. and V. Misra, 2013: Seasonal predictability of streamflow in southeastern US: Part I: Boreal summer and fall seasons 63. Li, H. and V. Misra, 2013: Impact of air-sea coupling on southeastern US climate 3. Li, H. and V. Misra, 2013: The fidelity of two global reanalysis examined by their forcing of a coupled downscaled integration 64. Misra, V. and H. Li, 2013: Seasonal predictability of the West African monsoon 65. Bastola, S. and V. Misra, 2013: Seasonal predictability of streamflow in southeastern US: Part II: Boreal winter and spring seasons 66. Kozar, M., H. Li and V. Misra, 2013: Seasonal predictability of vertical wind shear in the Maximum development region of the Atlantic 67. Misra, V. and H. Li, 2013: Seasonal predictability of freeze events in the southeastern United States 68. Misra, V. and H. Li, 2013: Seasonal predictability of the west African monsoon. 69. Misra, V., 2012: The efficacy of two tiered and bias corrected SST forced seasonal hindcasts. NOAA MAP PI Webinar, Tallahassee, December 13 70. Misra, V., 2012: FISH50: Florida Climate Institute Seasonal Hindcasts at 50km grid resolution, Southeast Climate Consortium Fall Planning meeting, Tifton, Georgia, November 14

302 71. Li, H., 2012: FISH50: Florida Climate Institute Seasonal Hindcasts at 50km grid resolution, RSM VI International conference, San Diego, California, November 6-10 72. Misra, V., 2012: FISH50: Florida Climate Institute Seasonal Hindcasts at 50km grid resolution, COAPS seminar series, Tallahassee, November 5 73. Misra, V., 2012: FISH50: Florida Climate Institute Seasonal Hindcasts at 50km grid resolution, Florida Alliance for Climate and Water, Orlando, Florida, September 27 13. CCR110026 74. C.D. Sudheer, S. Krishnan, A. Srinivasan, and P. R. C. Kent. Dynamic Load Balancing for Petascale Quantum Monte Carlo Applications: The Alias Method, Computer Physics Communications, Vol. 184, pg. 284-292, 2013. 75. Zheng Gu, Matthew Small, Xin Yuan, Aniruddha Marathe and David K. Lowenthal. Trace-driven Protocol Customization for Optimizing MPI Performance on RDMA-enabled Clusters, accepted in International Journal of Parallel Programming. 76. Optimization of the Hop-Byte Metric for Effective Topology Aware Mapping, C.D. Sudheer, Ashok Srinivasan, Proceedings of the 19th IEEE International Conference on High Performance Computing (HiPC), 2012. 77. Matthew Small and Xin Yuan. A New Design of RDMA-based Small Message Channels for InfiniBand Clusters, submitted to the International Conference on Supercomputing, 2013. 14. CCR130015 78. P. Cicotti and S. B. Baden. “Latency Hiding and Performance Tuning with Graph-Based Execution.” Proc. Seventh IEEE eScience Conference, Data-Flow Execution Models for Extreme Scale Computing (DFM 2011), Galveston Island, Texas, October 2011, IEEE. 15. CDA100010 79. S. Y. Mashayak and N. R. Aluru, “Thermodynamic state-dependent structure-based coarse-graining of confined water”, Journal of Chemical Physics, Vol. 137, No. 21, Art. No. 214707, 2012. 80. V. V. R. Nandigana and N. R. Aluru, “Understanding anomalous current-voltage characteristics in microchannel-nanochannel interconnect devices”, Journal of Colloid and Interface Science, Vol. 384, No. 1, pp. 162-171, 2012. 81. K. Park, N. Kim, D. T. Morisette, N. R. Aluru and R. Bashir, “Resonant MEMS mass sensors for measurement of microdroplet evaporation”, Journal of Microelectromechanical Systems, Vol. 21, No. 3, pp. 702-711, 2012. 82. S. Y. Mashayak and N. R. Aluru, “Coarse-grained potential model for structural prediction of confined water”, Journal of Chemical Theory and Computation, Vol. 8, No. 5, pp. 1828-1840, 2012. 83. B. M. Venkatesan, D. Estrada, S. Banerjee, X. Jin, V. E. Dorgan, M. Bae, N. R. Aluru, E. Pop, R. Bashir, “Stacked graphene-Al2O3 nanopore sensors for sensitive detection of DNA and DNA-protein complexes”, ACS Nano, Vol. 6, No. 1, pp. 441-450, 2012. 84. T. Sanghi and N. R. Aluru, ``Coarse-grained potential models for structural prediction of carbon dioxide (CO2) in confined environments'', Journal of Chemical Physics, Vol. 136, No. 2, Art. No. 024102, 2012. 85. P. Sumant, H. Wu, A. Cangellaris, and N. R. Aluru, ``Reduced-order models of finite element approximations of electromagnetic devices exhibiting statistical variability'', IEEE Transactions on Antennas and Propagation, Vol. 60, No. 1, pp. 301-309, 2012. 16. CHE030089 86. Hong, Y. J.; Tantillo, D. J. Chem. Commun. 2012, 48, 1571-1573: "Theoretical Calculations on Carbocations Involved in the Biosynthesis of Bergamotenes and Related Terpenes - The Same and Not the Same." Part of the Emerging Investigators issue. 87. Lodewyk, M. W.; Siebert, M. R.; Tantillo, D. J. Chem. Rev. 2012, 112, 1839-1862: "Computational Prediction of 1H and 13C Chemical Shifts: A Useful Tool for Natural Product, Mechanistic and Synthetic Organic Chemistry" 88. Gutierrez, O.; Aube, J.; Tantillo, D. J. J. Org. Chem. 2012, 77, 640-647: "Mechanism of the Acid Promoted Intramolecular Schmidt Reaction. Theoretical Assessment of the Importance of Lone Pair-Cation, Cation-pi and Steric Effects in Controlling Regioselectivity" 89. Knapp, J. M.; Wood, A. B.; Phuan, P.-W.; Lodewyk, M. W.; Tantillo, D. J.; Verkman, A. S.; Kurth, M. J. J. Med. Chem. 2012, 55, 1242-1251: "Structure-Activity Relationships of Cyanoquinolines with Corrector- Potentiator Activity in (delta)F508-Cystic Fibrosis Transmembrane Conductance Regulator Protein" 90. Felix, R. J.; Weber, D.; Gutierrez, O.; Tantillo, D. J.; Gagne, M. R. Nature Chem. 2012, 4, 405-409: "A Gold- Catalysed Enantioselective Cope Rearrangement of Achiral 1,5-Dienes," highlighted in SYNFACTS.

303 91. Laulhe, S; Bogdanov, B.; Johannes, L. M.; Gutierrez, O.; Harrison, J. G.; Tantillo, D. J.; Zhang, X.; Nantz, M. H. J. Mass Spectrom. 2012, 47, 676-686: "Fragmentation of Oxime and Silyl Oxime Ether Odd-electron Positive Ions by the McLafferty Rearrangement: New Insights on Structural Factors that Promote alpha,beta- Fragmentation" 92. Liu, R.; Gutierrez, O.; Tantillo, D. J.; Aube, J. J. Am. Chem. Soc. 2012, 134, 6528-6531: "Stereocontrol in a Combined Allylic Azide Rearrangement and Intramolecular Schmidt Reaction: Application to Asymmetric Synthesis of Pinnaic Acid" 93. Gutierrez, O.; Harrison, J. G.; Pemberton, R. P; Tantillo, D. J. Chem. Eur. J. 2012, 18, 11029-11035: "Reexamining the Mechanisms of Competing Pericyclic Reactions of 1,3,7-Octatriene" 94. Leverett, C. A.; Purohit, V. C.; Johnson, A. G.; Davis, R. A.; Tantillo, D. J.; Romo, D. J. Am. Chem. Soc. 2012, 134, 13348-13356: "Dyotropic Rearrangements of Fused Tricyclic-beta-Lactones: Application to the Synthesis of (-)-Curcumanolide A and (-)-Curcumalactone" 95. Harrison, J. G.; Tantillo, D. J. J. Molec. Model. 2013, in press: "Role of Gold in a Complex Cascade Reaction Involving Two Electrocyclization Steps," invited for special issue on Quitel 2011. 96. Nouri, D. H.; Tantillo, D. J. Org. Biomol. Chem. 2012, 10, 5514-5517: "Attack of Radicals and Protons on Ladderane Lipids: Quantum Chemical Calculations and Biological Implications" 97. Harrison, J. G.; Tantillo, D. J. Phys. Chem. Chem. Phys. 2012, 14, 14756-14759: "Fusing Cubanes to 1,5- Hexadiene," invited for themed issue on "Predicting New Molecules by Quantum Chemical Methods". 98. Zu, L.; Xu, M.; Lodewyk, M. W.; Cane, D. E.; Peters, R. J.; Tantillo, D. J. J. Am. Chem. Soc. 2012, 134, 11369-11371: "Effect of Isotopically Sensitive Branching on Product Distribution for Pentalenene Synthase - Support for a Mechanism Predicted by Quantum Chemistry" 99. Hong, Y. J.; Ponec, R.; Tantillo, D. J. J. Phys. Chem. A 2012, 116, 8902-8909: "Changes in Charge Distribution, Molecular Volume, Accessible Surface Area and Electronic Structure Along the Reaction Coordinate for a Carbocationic Triple Shift Rearrangement of Relevance to Diterpene Biosynthesis" 100. Knapp, J. M.; Zhu, J. S.; Tantillo, D. J.; Kurth, M. J. Angew. Chem. Int. Ed. 2012, 51, 10588-10591: "Multicomponent Assembly of Highly Substituted Indoles by Dual Palladium-Catalyzed Coupling Reactions," highlighted in SYNFACTS. 101. Avila, B.; El-Dakdouki, M. H.; Nazer, M. Z.; Harrison, J. G.; Tantillo, D. J.; Haddadin, M. J.; Kurth, M. J. Tetrahedron Lett. 2012, 53, 6475–6478: "Acid and Base Catalyzed Davis-Beirut Reaction: Experimental and Theoretical Mechanistic Studies and Synthesis of Novel 3-Amino-2H-indazoles" 102. Gutierrez, O.; Tantillo, D. J. J. Org. Chem. 2012, 77, 8845-8850: "Analogies Between Synthetic and Biosynthetic Reactions in which [1,2]-Alkyl Shifts are Combined with Other Events - Dyotropic, Schmidt and Carbocation Rearrangements," invited JOCSynopsis. 103. Sio, V.; Harrison, J. G.; Tantillo, D. J. Tetrahedron Lett. 2012, 53, 6919-6922: "Theoretical Assessment of the Viability of Thermal [2+2] Processes for Formation of Plumisclerin A" 104. Lebold, T. P.; Wood, J. L.; Deitch, J.; Lodewyk, M. W.; Tantillo, D. J.; Sarpong, R. Nature Chem., in press, DOI: 10.1038/nchem.1528: "A Divergent Approach to the Synthesis of the Yohimbinoid Alkaloids Venenatine and Alstovenine" 105. Ogawa, Y.; Painter, P. P.; Tantillo, D. J.; Wender, P. A. J. Org. Chem. 2013, 78, 104-115: "Mechanistic and Computational Studies of Exocyclic Stereocontrol in the Synthesis of Bryostatin-like cis-2,6-Disubstituted-4- Alkylidene Tetrahydropyrans by Prins Cyclization," part of Robert Ireland Memorial Issue. 106. Hong, Y. J.; Giner, J.-L.; Tantillo, D. J. J. Org. Chem. 2013, in press, DOI: 10.1021/jo3024208: "Triple-Shifts and Thioether Assistance in Rearrangements Associated with an Unusual Biomethylation of the Sterol Side Chain" 17. CHE040014 107. Elizabeth L. Noey, Yingdong Luo, Liming Zhang, and K. N. Houk: "Mechanism of Gold(I)-Catalyzed Rearrangements of Acetylenic Amine-N-Oxides: Computational Investigations Lead to a New Mechanism Confirmed by Experiment," J. Am. Chem. Soc., 134, 1078-1084 (2012). 108. Yu Lan, Rick L. Danheiser, and K. N. Houk: "Why Nature Eschews the Concerted [2 + 2 + 2] Cycloaddition of a Nonconjugated Cyanodiyne. Computational Study of a Pyridine Synthesis Involving an Ene-Diels-Alder- Bimolecular Hydrogen Transfer Mechanism," J. Org. Chem., 77, 1533-1538 (2012). 109. Peng Liu, Xiufang Xu, Xiaofei Dong, Benjamin K. Keitz, Myles B. Herbert, Robert H. Grubbs, and K. N. Houk: "Z-Selectivity in Olefin Metathesis with Chelated Ru Catalysts: Computational Studies of Mechanism and Selectivity," J. Am. Chem. Soc., 134, 1464-1467 (2012). 110. Yu-Hong Lam, K. N. Houk, Ulf Scheffler, and Rainer Mahrwald: "Stereoselectivities of Histidine-Catalyzed Asymmetric Aldol Additions and Contrasts with Proline Catalysis: A Quantum Mechanical Analysis," J. Am. Chem. Soc., 134, 6286-6295 (2012).

304 111. Myles B. Herbert, Yu Lan, Benjamin Keith Keitz, Peng Liu, Koji Endo, Michael W. Day, K. N. Houk, and Robert Howard Grubbs: "Decomposition Pathways of Z-Selective Ruthenium Metathesis Catalysts," J. Am. Chem. Soc., 134, 7861-7866 (2012). 112. Yu Lan, Peng Liu, Stephen G. Newman, Mark Lautens, and K. N. Houk: "Theoretical Study of Pd(0)- Catalyzed Carbohalogenation of Alkenes: Mechanism and Origins of Reactivities and Selectivities in Alkyl Halide Reductive Elimination from Pd(II) Species," Chem. Sci., 3, 1987-1995 (2012). 113. Xiufang Xu, Peng Liu, Adam Leser, Lauren E. Sirois, Paul A. Wender, and K. N. Houk: "Ligand Effects on Rates and Regioselectivities of Rh(I) Catalyzed (5+2) Cycloadditions: A Computational Study of Cyclooctadiene and Dinaphthocyclooctatetraene as Ligands," J. Am. Chem. Soc., 134, 11012-11025 (2012). 114. Peng Liu, Xing Yang, Vladimir B. Birman, and K. N. Houk: "Origin of Enantioselectivity in Benzotetramisole-Catalyzed Dynamic Kinetic Resolution of Azlactones," Org. Lett., 14, 3288-3291 (2012). 115. Qingquan Zhao, Yu-hong Lam, Mahboubeh Kheirabadi, Chongsong Xu, K. N. Houk, and Christian E. Schafmeister: “Hydrophobic Substituent Effects on Proline Catalysis of Aldol Reactions in Water” J. Org. Chem., 77, 4784-4792 (2012). 116. Kersey Black, Peng Liu, Lai Xu, Charles Doubleday, and K. N. Houk: “Dynamics, Transition States, and Timing of Bond Formation in Diels-Alder Reactions” Proc. Natl. Acad. Sci. U. S. A., 109, 12860-12865 (2012). 117. Ramesh Giri, Yu Lan, Peng Liu, K. N. Houk, and Jin-Quan Yu: “Understanding Reactivity and Stereoselectivity in Palladium-Catalyzed Diastereoselective sp3 C–H Bond Activation: Intermediate Characterization and Computational Studies” J. Am. Chem. Soc., 134, 14118-14126 (2012). 118. Sarah M. Bronner, Joel L. Mackey, K. N. Houk, and Neil K. Garg: “Steric Effects Compete with Aryne Distortion to Control Regioselectivities of Nucleophilic Additions to 3-Silylarynes” J. Am. Chem. Soc. 134, 13966-13969 (2012). 119. Xing Yang, Peng Liu, K. N. Houk, and Vladimir B. Birman: “Manifestation of Felkin-Anh Control in Enantioselective Acyl Transfer Catalysis: Kinetic Resolution of Carboxylic Acids” Angew. Chem., Int. Ed. 51, 9638–9642 (2012). 120. Martin J. Schnermann, Nicholas L. Untiedt, Gonzalo Jiménez-Osés, K. N. Houk, Larry Overman: “Forming Tertiary Organolithiums and Organocuprates from Nitrile Precursors and their Bimolecular Reactions with Carbon Electrophiles to form Quaternary Carbon Stereocenters”. Angew. Chem. Int. Ed., 51, 9581-9586 (2012). 121. Hao Wang, Philipp Kohler, Larry E. Overman, and K. N. Houk: “Origins of Stereoselectivities of Dihydroxylations of cis-Bicyclo[3.3.0]octenes” J. Am. Chem. Soc., 134, 16054-16058 (2012). 122. Arne Dieckmann, K. N. Houk: “Elucidation of Strong Cooperative Effects Caused by Dispersion in a Recognition-Mediated Diels-Alder Reaction”. J. Chem. Theory Comput. In press, DOI: 10.1021/ct300655b. 123. Xing Yang, Valentina D. Bumbu, Peng Liu, Ximin Li, Hui Jiang, Eric W. Uffman, Lei Guo, Wei Zhang, Xuntian Jiang, K. N. Houk, and Vladimir B. Birman: “Catalytic, Enantioselective N-Acylation of Lactams and Thiolactams Using Amidine-Based Catalysts”, J. Am. Chem. Soc. 134, 17605-17612 (2012). 124. Marinus A. Bigi, Peng Liu, Lufeng Zou, K. N. Houk, and M. Christina White: “Cafestol to tricalysiolide B and 6β-hydroxy-tricalysiolide B: Biosynthetic and derivatization studies using non-heme iron catalyst Fe(PDP)”, Synlett, 23, 2768-2772 (2012). 125. Xiufang Xu, Peng Liu, Adam Leser, Lauren E. Sirois, Paul A. Wender, and K. N. Houk: "Ligand Effects on Rates and Regioselectivities of Rh(I) Catalyzed (5+2) Cycloadditions: A Computational Study of Cyclooctadiene and Dinaphthocyclooctatetraene as Ligands," J. Am. Chem. Soc., 134, 11012-11025 (2012). 126. Peng Liu, Xing Yang, Vladimir B. Birman, and K. N. Houk: "Origin of Enantioselectivity in Benzotetramisole-Catalyzed Dynamic Kinetic Resolution of Azlactones," Org. Lett., 14, 3288-3291 (2012). 127. Ramesh Giri, Peng Liu, Yu Lan, K. N. Houk, and Jin-Quan Yu: "Understanding Reactivity and Stereoselectivity in Palladium-Catalyzed Diastereoselective sp3 C-H Bond Activation: Intermediate Characterization and Computational Studies," J. Am. Chem. Soc., 134, 14118-14126 (2012). 128. Kersey Black, Peng Liu, Lai Xu, Charles Doubleday, and Kendall N. Houk: "Inaugural Article: Dynamics, Transition States, and Timing of Bond Formation in Diels-Alder Reactions," Proc. Natl. Acad. Sci. USA, 109, 12860-12865 (2012). 129. Yong Liang, Joel L. Mackey, Steven A. Lopez, Fang Liu, K. N. Houk: “Control and Design of Mutual Orthogonality in Bioorthogonal Cycloadditions,” J. Am. Chem. Soc., 134, 17904-17907 (2012). 130. Yu-hong Lam, K. N. Houk, U. Scheffler, R. Mahrwald: “Fluorine as a Regiocontrol Element in the Ring Openings of Bicyclic Aziridiniums,” Helv. Chim. Acta., 95, 2265-2277 (2012). 131. Xin Hong, Peng Liu, and K. N. Houk: "Mechanism and Origins of Ligand-Controlled Selectivities in [Ni(NHC)]-Catalyzed Intramolecular (5+2) Cycloadditions and Homo-Ene Reactions: A Theoretical Study". J. Am. Chem. Soc. ASAP 132. Steven A. Lopez, Munk, M. E.; K. N. Houk: “Mechanisms and Transition States of 1,3-Dipolar Cycloadditions of Phenyl Azide with Enamines: A Computational Analysis” In preparation

305 133. Arne Dieckmann, Martin Breugst, K. N. Houk: “Kinetic Control and Zwitterionic Intermediates in Organocatalytic Diels-Alder Reactions of Cross-Conjugated Trienamines”. Submitted 134. Gonzalo Jiménez-Osés, Carlos F. Barbas. III, K. N. Houk: “Perfect Selectivity from a Thiourea-Catalyzed Diels-Alder Reaction: Strategies for Predictions of Organic Selectivities”. In Preparation. 135. Gonzalo. Jiménez-Osés, K. C. Nicolaou, K. N. Houk: “Computational Assessment of Stereoselectivity in Biyouyanagins Synthesis”. In Preparation. 136. Martin Breugst, Albert Eschenmoser, K. N. Houk: “Theoretical Exploration of the Mechanism of Riboflavin Formation” In Preparation. 137. Lei Fang, Peng Liu, Benjamin R. Sveinbjornsson, Sule Atahan-Evrenk, Koen Vandewal, Sílvia Osuna, Gonzalo Jiménez-Osés, Supriya Shrestha, Gaurav Giri, Peng Wei, Alberto Salleo, Alán Aspuru-Guzik, Robert H. Grubbs, K. N. Houk, and Zhenan Bao: “Confined organization of fullerene units along high polymer chains”, submitted. 18. CHE070035 138. Photoisomerization of cis-1-(3-Methyl-2-naphthyl)-2-phenylethene in Glassy Methylcyclohexane at 77 K Jack Saltiel *†, Stuart R. Hutchinson †, Katelyn Chitwood †, and Olga Dmitrenko J. Phys. Chem. A, 2012, 116 (22), pp 5293–5298. 139. The Role of Acid Catalysis in the Baeyer–Villiger Reaction. A Theoretical Study Robert D. Bach J. Org. Chem., 2012, 77 (16), pp 6801–6815 140. Journal of Computational Chemistry, Volume 33, Issue 6, pages 708–714, 5 March 2012, An extension of the grid empowered molecular simulator to quantum reactive scattering†, Sergio Rampino1,*, Noelia Faginas Lago1, Antonio Laganà1, Fermin Huarte-Larrañaga2 141. Earth Science Informatics ,Volume 5, Number 1 (2012), 33-41, DOI: 10.1007/s12145-012-0094-y Talkoot: software tool to create collaboratories for earth science, Rahul Ramachandran, Manil Maskey, Ajinkya Kulkarni, Helen Conover, U. S. Nair and Sunil Movva 142. Photoisomerization of cis-1-(3-Methyl-2-naphthyl)-2-phenylethene in Glassy Methylcyclohexane at 77 K Jack Saltiel, Stuart R. Hutchinson, Katelyn Chitwood, and Olga Dmitrenko, The Journal of Physical Chemistry A 2012 116 (22), 5293-5298 19. CHE090047 143. “Can ORMAS be used for nonadiabatic coupling calculations? SiCH4 and Butadiene Contours.”, A.C. West and T.L. Windus, Theor. Chem. Acc., 2012, 131, 1251. 144. “O + C2H4 Potential Energy Surface: Lowest-lying Singlet at the Multireference Level”, A.C. West, J.D. Lynch, B. Sellner, H. Lischka, W.L. Hase, and T.L. Windus, Theor. Chem. Acc., 2012, 131, 1279. DOI: 10.1007/s00214-012-1279-7 145. “O + C2H4 Potential Energy Surface: Excited States and Biradicals at the Multireference Level”, A.C. West, J.D. Lynch, B. Sellner, H. Lischka, W.L. Hase, and T.L. Windus, Theor. Chem. Acc., 2012, 131, 1123-1131. 20. CHE090098 146. Wang, J.; Cieplak, P.; Cai, Q.; Hsieh, M. J.; Wang, JM.; Duan, Y. Luo, R.; Development of polarizable models for molecular mechanical calculations III: Polarizable water models conforming to Thole polarization screening schemes. J. Phys. Chem. B, 2012, 116, 7999-8008. 147. Wang, J.; Cieplak, P.; Li, J.; Cai, Q.; Hsieh, M. J.; Luo, R.; Duan, Y. Development of polarizable models for molecular mechanical calculations IV: van der Waals parameterization. J. Phys. Chem. B, 116, 7088-7101, 2012. 148. Wang, JM.; Hou, T. Develop and test a solvent accessible surface area-based model in conformational entropy calculations. J. Chem. Info. Model., 2012, 52, 1199-1212. 21. CHE110011 149. Manna, K.; Ellern, A.; Sadow, A. D. " Highly Enantioselective Zirconium-Catalyzed Cyclization of Aminoalkenes " submitted for publication. 150. Yan, K.; Schoendorff, G.; Upton, B. M.; Ellern, A.; Windus, T. L.; Sadow, A. D. " Intermolecular β -hydrogen abstraction in ytterbium, calcium and potassium tris(dimethylsilyl)methyl compounds" Submitted to Organometallics, Revision requested. 151. "Main group compounds supported by oxazolinylborate ligands in catalysis" Columbia University, 27 September 2012. 152. "Main group compounds supported by oxazolinylborate ligands in catalysis" University of Wyoming, 1 October 2012.

306 153. "Main group compounds supported by oxazolinylborate ligands in catalysis" Colorado State University, 2 October 2012. 154. "Main group compounds supported by oxazolinylborate ligands in catalysis" Wayne State University, 11 October 2012. 155. "Main group compounds supported by oxazolinylborate ligands in catalysis" University of Windsor, Ontario CA, 12 October 2012. 156. "Main group catalysts and sustainability" Macalester College, St. Paul MN, 31 October 2012. 157. "Main group catalysts and sustainability" 4th Dalton Transactions International Symposium on Organometallic Chemistry and Catalysis. Institute of Materials Research and Engineering, Singapore, 5 November 2012. 158. "Main group catalysts and sustainability" 4th Dalton Transactions International Symposium on Organometallic Chemistry and Catalysis. Fudan University, Shanghai, China, 7 November 2012. 159. "Oxazolinylborates compounds for Catalytic C-N bond formation" Shanghai Institute of Organic Chemistry, Shanghai, China, 9 November 2012. 22. CHE120025 160. “Structures of Cage, Prism, and Book Isomers of Water Hexamer from Broadband Rotational Spectroscopy” Cristóbal Pérez, Matt T. Muckle, Daniel P. Zaleski, Nathan A. Seifert, Berhane Temelso, George C. Shields, Zbigniew Kisiel, and Brooks H. Pate*. Science. 336, 897-901 (2012) 161. “A Computational Study of the Hydration of Sulfuric Acid Dimers: Implications for Acid Dissociation and Aerosol Formation” Berhane Temelso, Judy Phan* and George C. Shields. J. Phys. Chem. A. 116 (39), 9745- 9758. (2012) 162. “Hydration of the Bisulfate Ion: Atmospheric Implications” Devon E. Husar,* Berhane Temelso, Alexa L. Ashworth* and George C. Shields. J. Phys. Chem. A. 116, 5151-5163. (2012) 163. “Quantum Mechanical Study of Sulfuric Acid Hydration: Atmospheric Implications” Berhane Temelso, Tom Morrell,* Robert Shields,* Marco Allodi,* Elena Wood,* Karl N. Kirschner, Thomas Castonguay, Kaye A. Archer* and George C. Shields. J. Phys. Chem. A. 116, 2209-2224. (2012) 164. “Isotopic Ratios in Titan’s Methane: Measurements and Modeling.” C. A. Nixon, B. Temelso, S. Vinatier, N. A. Teanby, B. Bézard, R. K. Achterberg, K. E. Mandt, C. D. Sherrill, P. G. J. Irwin, D. E. Jennings, P. N. Romani, A. Coustenis, F. M. Flasar. Astrophys. J. (2012) 749, 159 23. CTS030025 165. Yazdani, A.Z.K., & Bagchi, P., 2012. Influence of membrane viscosity on capsule dynamics in shear flow. Journal of Fluid Mechanics. Accepted, in press. 166. Vahidkhah, K. Diamond, S.L, and Bagchi, P. 2013. Hydrodynamic interaction between a platelet and erythrocyte: Effect of erythrocyte deformation, dynamics, and wall proximity. Journal of Biomechanical Engineering, Accepted. 167. Bagchi, P. & Yazdani, A. 2012. Analysis of membrane tank-tread of nonspherical capsules and red blood cells. European Physical Journal E, 35 (10), 9780-12. 168. Yazdani, A., & Bagchi, P. 2012. Three-dimensional numerical simulation of vesicle dynamics using a front- tracking method. Physical Review E, 85, 056308-18. 24. CTS070066 169. Bonilla R. & Leonardi S. (2012) DNS, LES and RANS of the turbulent flow in a channel with V-shaped and inclined rib turbulators, submitted to J. of Turbomachinery 170. Leonardi S., Orlandi P. & Antonia R.A. (2012) Heat transfer in a turbulent channel flow with transverse square bars and circular rods: DNS. submitted to J. Fluid Mech. 171. Martinez-Tossas L., Leonardi S., M. J. Churchfield & P. J. Moriarty (2012) A Comparison of Actuator Disk and Actuator Line Wind turbine Models and Best Practices for Their Use. submitted to J. Comp. Physics 172. Leonardi S. (2012) Direct Numerical Simulation of a turbulent channel flow with 3D random roughness on a wall. European Fluid Mechanics Conference, Rome 9/10-14/12 25. CTS100027 173. T. A. Manz and D. S. Sholl, “Improved Atoms-in-Molecule Charge Partitioning Functional for Simultaneously Reproducing the Electrostatic Potential and Chemical States in Periodic and Nonperiodic Materials,” Journal of Chemical Theory and Computation, 8 (2012) 2844-2867. 174. T. A. Manz, J. M. Caruthers, S. Sharma, K. Phomphrai, K. T. Thomson, W. N. Delgass, and M. M. Abu- Omar, “Structure–Activity Correlation for Relative Chain Initiation to Propagation Rates in Single-Site Olefin Polymerization Catalysis,” Organometallics, 31 (2012) 602-618.

307 175. T. A. Manz, “Estimating Unimolecular Rate Constants in Solution: Deactivation of Half-Metallocene Complexes Activated with B(C6F5)3,” submitted. 176. T. A. Manz and D. S. Sholl, “Computing Accurate Net Atomic Charges, Atomic Spin Moments, and Effective Bond Orders in Complex Materials,” A. Asthagiri and M. Janik, (Eds.), Computational Catalysis, RSC, Cambridge, UK, in press. 177. T. A. Manz (speaker) and D. S. Sholl, “Improved Methods for Assigning Point Charges, Bond Orders, and Magnetic Moments in Complex Materials,” AICHE Annual Meeting, Minneapolis, Minnesota, October 19, 2011. 178. T. A. Manz (speaker), S. Sharma, K. Phomphrai, G. Medvedev, K. T. Thomson, M. M. Abu-Omar, W. N. Delgass, and J. M. Caruthers, “Structure-Activity Correlation for Relative Chain Initiation to Propagation Rates in Single-Site Olefin Polymerization Catalysis,” AICHE Annual Meeting, Minneapolis, Minnesota, October 20, 2011. 26. CTS110026 179. Rao, A., Reddy, R. K., Nandakumar, K., Franz E., Valsaraj, K.T. Dynamics of organic droplets in water in presence of mass transfer: A model system for the deep water oil spill. November 2012. South-West Regional ACS meeting (ACS-SWRM) in Baton Rouge. 180. Reddy, R.K., Rao, A., Yu, Z, Wu, C., Nandakumar, K., Thibodeaux, L., Valsaraj, K.T. “Challenges in and approaches to modeling the complexities of deep water oil and gas release". Book chapter in “Oil Spill Remediation: Colloid Chemistry-Based Principles and Solutions, edited by Somasundaran, P”. John Wiley and Sons Ltd.(In press) 181. Reddy, R. K., Rao, A., Nandakumar, K., 2012. Towards a comprehensive model of the transport processes in deep water oil and gas spill subject to surfactant exposure. 243rd ACS annual meeting in San Diego. 27. CTS110039 182. U. Agarwal and F.A. Escobedo, “Yielding and shear induced melting of 2D mixed crystals of spheres and dimers”, Soft Matter 8, 5916 (2012). 28. CTS110051 183. Yaniv D. Scherson, Shela J. Aboud, Brian J. Cantwell, Ashley Micks, Density Functional Theory Studies of N2O Decomposition on Clean and Defective (0001) Surfaces of Rh2O3 and Fe2O3 (in preparation) and the Stanford University 184. Yaniv D. Scherson titled: Energy Recovery from Waste Nitrogen through N2O Decomposition: The Coupled Aerobic-Anoxic Nitrous Decomposition Operation (CANDO), Ph.D Dissertation, Stanford University (2012) 29. DBS130003 185. Judith Gelernter, Gang Wu, High performance mining of social media data, XSEDE’12, July 16-20, 2012, Chicago, Illinois, U.S.A. Copyright 2010 ACM 1-58113-000-0/00/0010…$10.00. 30. DMR000003 186. Wang Yechun & Dimitrakopoulos P. “Low-Reynolds-number droplet motion in a square microfluidic channel”, Theor. Comput. Fluid Dyn., 26, 361–379 (2012). 187. Dodson W. R. III & Dimitrakopoulos P. “Tank-treading of swollen erythrocytes in shear flows”, Phys. Rev. E., 85 021922 (9 pages) (2012). 188. Dissanayake I. D. & Dimitrakopoulos P. “Coil-helix-rod transition of an initially flexible bead-rod polymer after rapid stiffening: configuration and stress evolution”, J. Chem. Phys., under review (2011). 31. DMR010001 189. L. Miljacic and A. van de Walle. Ab initio calculation of the equation of state of solid, liquid and gaseous Tantalum. Submitted, 2012. 190. L. Miljacic and A. van de Walle. Ab initio calculation of the critical point of Tantalum. Submitted, 2012. 191. A. van deWalle, C. Balaji Gopal, S. Demers, Q. Hong, A. Kowalski, L. Miljacic, G. Pomrehn, and P. Tiwary. Symmetry-adapted bases for parametrizing anisotropic properties. Submitted, 2012. 192. P. Tiwary and A. van de Walle. Accelerated molecular dynamics simulations through SISYPHUS. Submitted, 2012. 193. M. de Jong, D. L. Olmsted, A. van de Walle, and M. Asta. First-principles study of the structural and elastic properties of rhenium-based transition-metal alloys. Phys. Rev. B, 86:224101, 2012. 194. S. Demers and A. van de Walle. Intrinsic defects and dopability of zinc phosphide. Phys. Rev. B, 85:195208, 2012.

308 195. B. G. Chirranjeevi and A. van de Walle. Ab initio thermodynamics of intrinsic oxygen vacancies in ceria. Phys. Rev. B, 86:134117, 2012. 196. Q. Hong and A. van de Walle. Direct first-principles chemical potential calculations of liquids. J. Chem. Phys., 137:094114, 2012. 197. P. Dalach, D. E. Ellis, and A. van de Walle. First-principles thermodynamic modeling of lanthanum chromate perovskites. Phys. Rev. B, 85:014108, 2012. 198. C. Ravi, B. K. Panigrahi, M. C. Valsakumar, and A. van de Walle. First-principles calculation of phase equilibrium of V-Nb, V-Ta, and Nb-Ta alloys. Phys. Rev. B, 85:054202, 2012. 199. B. P. Burton and A. van de Walle. First principles phase diagram calculations for the octahedral-interstitial system HfOx, 0 _ x _ 1/2. Calphad Journal, 37:151–157, 2012. 200. L. G. Wang and A. van de Walle. Ab initio calculations of the melting temperatures of refractory bcc metals. Phys. Chem. Chem. Phys., 14:1529, 2012. 201. B. P. Burton, A. van de Walle, and H. T. Stokes. First principles phase diagram calculations for the octahedral- interstitial system ZrOX, 0 _ X _ 1/2. J. Phys. Soc. Jpn., 81:014004, 2012. 32. DMR090071 202. Charge transfer excitation energies by perturbative delta self consistent method Tunna Baruah, Marco Olguin, and Rajendra R. Zope, Journal of Chemical Physics 137, 084316 (2012). 203. Charge transfer excitation energies in co-facial porphyrin fullerene complexes Rajendra R. Zope, Marco Olguin, and Tunna Baruah, Journal of Chemical Physics 137, 084317 (2012). 204. Low-lying planar isomers of neutral and charged B22 clusters B. C. Hikmat, T. Baruah and Rajendra R. Zope, Journal of Physics: At. Mol. Opt. Phys. 45, 225101 (2012). 205. Geometry and electronic structure of neutral and charged B21 clusters Ruben Casillas, T. Baruah and Rajendra R. Zope, Chemical Physics Letters ( 2012) (http://dx.doi.org/10.1016/j.cplett.2012.10.076). 206. Effect of Geometrical Orientation on the Charge-Transfer Energetics of Supramolecular (tetraphenyl)- porphyrin/C60 Dyads, Marco Olguin, Rajendra R. Zope, and Tunna Baruah Journal of Chemical Physics (under review). 33. DMR100025 207. “Is dislocation flow turbulent in deformed crystals?”, Woosong Choi, Yong S. Chen, Stefanos Papanikolaou, and James P. Sethna, Computing in Science and Engineering 14 13, 2012 208. “Scaling theory of continuum dislocation dynamics: Self-organized critical pattern formation”, Yong S. Chen, Woosong Choi, Stefanos Papanikolaou, and James P. Sethna. 34. DMR100033 209. Y.M. Jin, “Phase Field Microelastic Modeling of Microstructure Evolutions in Free-Standing Thin Films”, Acta Mater., 60, 6547-6554, 2012. 210. L.D. Geng, Y.M. Jin, “Generation and Storage of 360° Domain Walls in Planar Magnetic Nanowires”, J. Appl. Phys., 112, 083903-1-4, 2012. 211. B. Wang, Y.M. Jin, “Magnetization and Magnetostriction of Terfenol-D near Spin Reorientation Boundary,” J. Appl. Phys., 111, 103908-1-5, 2012. 212. J.E. Zhou, T.L. Cheng, Y.U. Wang, “Correlated Nucleation and Self-Accommodating Kinetic Pathway of Ferroelectric Phase Transformation,” J. Appl. Phys., 111, 024105-1-13, 2012. 213. T.L. Cheng, Y.U. Wang, “Spontaneous Formation of Stable Capillary Bridges for Firming Compact Colloidal Microstructures in Phase Separating Liquids: A Computational Study,” Langmuir, 28, 2696-2703, 2012. 214. T.L. Cheng, Y.U. Wang, “Diffuse Interface Field Approach to Modeling Sterically-Shielded Shape- Anisotropic Particles at Curved Fluid Interfaces: The Role of Laplace Pressure,” J. Colloid Interface Sci. (submitted) 215. J.E. Zhou, Y. Yan, S. Priya, Y.U. Wang, “Phase Field Modeling and Simulation of Templated Grain Growth and Crystallographic Texture Development in Ferroelectric Polycrystalline Ceramics,” J. Appl. Phys. (submitted) 216. W.F. Rao, J.E. Zhou, Y.U. Wang, “Poling-Induced Single-Crystal-Like Piezoelectric Anisotropy in Untextured Ferroelectric Polycrystalline Ceramics: Phase Field Modeling” (in preparation) 217. W.F. Rao, Y.U. Wang, “Stress Effects on Piezoelectric Properties of Ferroelectric Polycrystalline Ceramics around Morphotropic Phase Boundary: Modeling and Simulation” (in preparation) 218. I.A. Kuyanov, T.L. Cheng, Y.M. Jin, Y.U. Wang, “Charge Compensation and Aging Mechanisms in Iron- Doped Ferroelectric Titanate Perovskites” (in preparation) 219. (Invited Talk) Y.U. Wang, J.E. Zhou, “Computational Study of Microstructure and Property Relations in Ferroelectric Polycrystals,” Electronic Materials and Applications, Orlando, Florida, January 23-25, 2013.

309 220. Y.M. Jin, L.D. Geng, “Generation and Storage of 360° Domain Walls for Magnetic Devices,” Electronic Materials and Applications, Orlando, Florida, January 23-25, 2013. 221. Y.M. Jin, F.D. Ma, Y.U. Wang, S.L. Kampe, S. Dong, “Phase Field Modeling of Multicomponent Multiferroic Composites,” Electronic Materials and Applications, Orlando, Florida, January 23-25, 2013. 222. Y.U. Wang, W.F. Rao, T.L. Cheng, J.E. Zhou, “Nanodomain Structures, Mechanisms and Diffraction Effects in Morphotropic Phase Boundary Ferroelectrics,” Electronic Materials and Applications, Orlando, Florida, January 23-25, 2013. 223. (Invited Talk) Y.M. Jin, B.L. Wang, “Magnetization and Magnetostriction of Terfenol-D near Spin Reorientation Boundary,” TMS Annual Meeting, San Antonio, Texas, March 3-7, 2013. 224. (Invited Talk) Y.M. Jin, S.A. Hackney, “Phase Field Modeling of Microstructure Engineering in Spinodal- Type Magnetic Materials,” TMS Annual Meeting, San Antonio, Texas, March 3-7, 2013. 225. (Invited Talk) Y.U. Wang, J.E. Zhou, “Computational Study of Microstructure and Property Relations in Ferroelectric Polycrystals,” TMS Annual Meeting, San Antonio, Texas, March 3-7, 2013. 226. Y.M. Jin, S.L. Kampe, “Ferroelectric Ceramic-Reinforced Metal Matrix Composites,” TMS Annual Meeting, San Antonio, Texas, March 3-7, 2013. 227. T.L. Cheng, Y.U. Wang, “Diffuse Interface Field Approach (DIFA) to Modeling and Simulation of Capillarity-Related Self-Organization Phenomena in Colloidal Systems,” TMS Annual Meeting, San Antonio, Texas, March 3-7, 2013. 228. (Invited Instructor) Y.M. Jin, “Critical Roles of Long-Range Interactions in Microstructure Evolutions: Phase Field Modeling,” IIMEC 2013 Second Winter School on Computational Materials Science, College Station, Texas, January 6-15, 2013. 35. DMR110004 229. D. Marrocchelli et al., Advanced Functional Materials 22, 1958 (2012). 230. D. Marrocchelli et al., Physical Chemistry Chemical Physics 14, 3089 (2012). 231. Y. Kuru et al., Journal of the Electrochemical Society 159 (2012). 232. M. Youssef, and B. Yildiz, Phys Rev B 86, 14 (2012). 233. K. Aravind et al., Journal of Physics: Condensed Matter 25, 045004 (2013). 234. J. W. Han and B. Yildiz, the Journal of the American Chemical Society (submitted) 36. DMR110092 235. Wang M, Alexander-Katz A, Olsen BD. “Diffusion of entangled rod-coil block copolymers.” ACS Macro Letters. 2012:1:676-680. 236. Wang M, Timachova K, Olsen BD. “Diffusion mechanisms and experimental measurements of rod-coil diblock copolymers.” In preparation. 237. Wang M, Timachova K, Alexander-Katz A, Olsen BD. “Diffusion of entangled rod-coil block copolymers.” Poster presented at Gordon Research Conference on Polymer Physics, South Hadley, MA. July, 2012. 238. Olsen BD. “Physics of protein-polymer hybrid block copolymers and complexes.” Invited presentation at Peking University Department of Polymer Chemistry, Beijing, China. June, 2012. 239. Olsen BD. “Physics of protein-polymer hybrid block copolymers and complexes.” Invited presentation at Tsinghua University, Beijing, China. June, 2012. 240. Olsen BD, Wang M, Timachova K. “Measurement of diffusion in entangled coil-rod-coil triblock copolymers.” Presented at American Physical Society March Meeting, Boston, MA. March, 2012. 241. Wang M, Alexander-Katz A, Olsen BD. “Theoretical and computational studies of entangled rod-coil block copolymer diffusion.” Presented at American Physical Society March Meeting, Boston, MA. March, 2012. 37. DMR120023 242. Iris W. Fu, Cade B. Markegard, Brian K. Chu, and Hung D. Nguyen, “The role of electrostatics and temperature on morphological transitions of hydrogel nanostructures self-assembled by peptide amphiphiles via molecular dynamics simulations,” Advanced Functional Materials, in review 243. Iris W. Fu and Hung D. Nguyen, “Solvent effects on kinetic mechanisms responsible for hydrogel self- assembly by peptide amphiphiles through molecular dynamics simulations,” planned to submit to JACS 38. DMS120006 244. M. Safari, F.P.M. Hadi and M.R.H. Sheikhi “Entropy Generation Analysis of a Turbulent Nonpremixed Jet Flame,” Combustion and Flame, 2013, under preparation. 245. M. Safari, F.P.M. Hadi and M.R.H. Sheikhi “Entropy Filtered Density Function for Large Eddy Simulation of Turbulent Reacting Flows,” Physics of Fluids, 2013, under preparation.

310 246. M. Safari, F.P.M. Hadi and M.R.H. Sheikhi “Prediction of Local Entropy Generation Using Large Eddy Simulation of Turbulent Reacting Flows,” AIAA Journal, 2013, under preparation. 247. J.E. Kopriva, G.M. Laskowski and M. R. H. Sheikhi, “Assessment of High Pressure Cooled and Uncooled Turbine Blade Wakes via RANS and URANS at Engine Scale Conditions” Proceedings of the 2013 ASME Turbo Expo, San Antonio, TX, June 3-7, 2013, 1-10, 2013. 248. M. Safari, F.P.M. Hadi and M.R.H. Sheikhi “Analysis of Local Entropy Generation Using Large Eddy Simulation of Turbulent Reacting Flows,” AIAA Paper AIAA-2013-0428, 51st American Institute of Aeronautics and Astronautics Aerospace Sciences Meeting, Grapevine, TX, January 7-10, 2013. 249. M. R. H. Sheikhi, M. Safari and H. Metghalchi, “Large Eddy Simulation for Local Entropy Generation Analysis of Turbulent Flows” ASME Journal of Energy Resources Technology, 134(4), 041603 1-6, 2012. 250. M. Safari and M.R.H. Sheikhi, “Filtered Density Function for Large Eddy Simulation of Local Entropy Generation in Turbulent Reacting Flows,” Bulletin of the American Physical Society, 57(17), p. 123, 65th Annual Meeting of the Division of Fluid Dynamics of the American Physical Society, San Diego, CA, November 18-20, 2012. 251. F.P.M. Hadi, M. Safari, J. Kopriva, M. Janbozorgi and M.R.H. Sheikhi, “Clean Sustainable Energy through Supercomputer Simulation of Turbulent Combustion of Alternative Fuels,” Research, Innovation and Scholarship Expo, Northeastern University, Boston, MA, March 29, 2012. 252. M. Safari and M.R.H. Sheikhi, “Development of Filtered Density Function for Large Eddy Simulation of Entropy Transport in Turbulent Reacting Flows,” Bulletin of the American Physical Society, 56(18), p. 161, 64th Annual Meeting of the Division of Fluid Dynamics of the American Physical Society, Baltimore, MD, November 20-22, 2011. 253. M.R.H. Sheikhi, M. Safari and H. Metghalchi, “Large Eddy Simulation of Local Entropy Generation in a Turbulent Mixing Layer, Proceedings of the 2011 Fall Technical Meeting of the Combustion Institute, Eastern States Section, in Chemical and Physical Processes in Combustion, University of Connecticut, Storrs, CT, October 9-12, 2011. 254. M. R. H. Sheikhi, “Filtered Density Function for Prediction of Entropy Generation in Turbulent Flames,” SIAM International Conference on Numerical Combustion, San Antonio, TX, April 8-10, 2013. 255. M. R. H. Sheikhi, “Analysis of Local Entropy Generation Using Large Eddy Simulation of Turbulent Reacting Flows, 51st American Institute of Aeronautics and Astronautics Aerospace Sciences Meeting, Grapevine, TX, January 7-10, 2013. 256. M. Safari, “Filtered Density Function for Large Eddy Simulation of Local Entropy Generation in Turbulent Reacting Flows,” Bulletin of the American Physical Society, 57(17), p. 123, 65th Annual Meeting of the Division of Fluid Dynamics of the American Physical Society, San Diego, CA, November 18-20, 2012. 257. F.P.M. Hadi, M. Safari, J. Kopriva, “Clean Sustainable Energy through Supercomputer Simulation of Turbulent Combustion of Alternative Fuels,” Research, Innovation and Scholarship Expo, Northeastern University, Boston, MA, March 29, 2012. 258. M. Safari, “Development of Filtered Density Function for Large Eddy Simulation of Entropy Transport in Turbulent Reacting Flows,” Bulletin of the American Physical Society, 56(18), p. 161, 64th Annual Meeting of the Division of Fluid Dynamics of the American Physical Society, Baltimore, MD, November 20-22, 2011. 39. EAR080015 259. Kenji Ohta, R. E. Cohen, Kei Hirose, Kristjan Haule, Katsuya Shimizu & Yasuo Ohishi, Experimental and theoretical evidence for pressure-induced metallization in FeO with the rocksalt type structure, submitted. 260. R. Caracas, R.E. Cohen, and F. Gygi, Carbon-bearing silicate melts the deep Earth, AGU, San Francisco, (2011). 261. R. Caracas, Multimegabar compression of glasses of planetary materials, AGU, San Francisco (2012). 262. Carbon-bearing melts in the deep Earth R. Caracas, R. E. Cohen, and F. Gygi To be submitted to Earth Planet. Sci. Lett. 263. Carbon-bearing silicate glasses at high pressure R. Caracas In preparation for Phys. Rev. B 40. EAR100002 264. S. Huang, H. Wang, K. -H. Ding, and L. Tsang, " Subwavelength Imaging Enhancement through 3- Dimensional Plasmon Superlens with Rough Surface, " Opt. Lett., vol. 37, pp1295-1297, April 2012 265. X. Xu., L. Tsang, and S. Yueh, " Electromagnetic Models of Co/Cross Polarization of Bicontinuous/DMRT in Radar Remote Sensing of Terrestrial Snow at X- and Ku-band for CoReH2O and SCLP Applications, " IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 5, pp. 1024-1032, 2012 266. S. Huang, and L. Tsang, " Electromagnetic scattering of randomly rough soil surfaces based on numerical solutions of Maxwell equations in 3 dimensional simulations using a hybrid UV/PBTG/SMCG method, " IEEE Trans. Geosci. Remote Sens., vol. 50, pp. 4025-4035 , 2012

311 267. P. Xu, L. Tsang and K.S. Chen, " Fourth Stokes Parameter in Polarimetric Passive Remote Sensing from Two- layer Rough Surfaces, " Progress in Electromagnetic Research, vol. 129, pp. 125-141 , 2012 268. H G Wang, L. Tsang and S. Huang, " The Loss and back-Coupling Effects on the Images of the Three Dimensional Superlens, "Optics Letters, vol. 37, pp. 2262-2264 , 2012 269. R. Ding, L. Tsang, H. Braunisch, and W Chang, " Wave propagation in parallel plate metallic waveguide with finite conductivity and three dimensional roughness, " IEEE Trans. Antenna and Propag, vol.60, pp. 5867- 5880 , 2012 270. L. Tsang, K.H. Ding, S. Huang, and X. Xu, " Electromagnetic Computation in Scattering of Electromagnetic Waves by Random Rough Surface and Dense Media in Microwave Remote Sensing of Land Surfaces, " Proceedings of IEEE , in press 2012 271. W. Chang and L. Tsang, " A partial coherent physical model of third and fourth stokes parameters of sastrugi snow surfaces over layered media with rough surface boundary conditions of conical scattering combined with vector radiative transfer theory, " Progress In Electromagnetics Research B, Vol. 45, 57-82 , in press 2012 41. EAR130009 272. Fan, Y., Li, HB, Miguez-Macho, G (2012): Global Patterns of Groundwater Table Depth, Science, Accepted. 42. ECS120003 273. Liu XS, W.J., Zhou B, Stein M, Adams M, Shane E, Guo XE, Fast Trabecular Bone Strength Predictions of HR-pQCT and Individual Trabeculae Segmentation (ITS)-Based Plate-Rod Finite Element Model Discriminate Postmenopausal Vertebral Fractures. Submitted to JBMR. 43. IRI120005 274. Castello, Charles C. and New, Joshua R. (2012). "Autonomous Correction of Sensor Data Applied to Building Technologies Utilizing Statistical Processing Methods." In Proceedings of the 2nd Energy Informatics Conference, Atlanta, GA, Oct. 6, 2012. [PDF] 275. New, Joshua R., Sanyal, Jibonananda, Bhandari, Mahabir S., Shrestha, Som S. (2012). "Autotune E+ Building Energy Models." In Proceedings of the 5th National SimBuild of IBPSA-USA, International Building Performance Simulation Association (IBPSA), Aug. 1-3, 2012. [PDF] 276. Sanyal, Jibonananda, Al-Wadei, Yusof H., Bhandari, Mahabir S., Shrestha, Som S., Karpay, Buzz, Garret, Aaron L., Edwards, Richard E., Parker, Lynne E., and New, Joshua R. (2012). "Poster: Building Energy Model Calibration using EnergyPlus, Machine Learning, and Supercomputing." In Proceedings of the 5th National SimBuild of IBPSA-USA, International Building Performance Simulation Association (IBPSA), Aug. 1-3, 2012. [PDF] 277. Edwards, Richard E., New, Joshua R., and Parker, Lynne E. (2012). "Predicting Future Hourly Residential Electrical Consumption: A Machine Learning Case Study." In Journal of Energy and Buildings, pp.591-603, volume 49, issue 0, June 2012. [PDF] 278. Garrett, Aaron, New, Joshua R., and Chandler, Theodore. (2013). "Evolutionary Tuning of Building Models to Monthly Electrical Consumption." To appear in Proceedings of the ASHRAE Annual Conference, Denver, CO, June 22-26, 2013. 279. Edwards, Richard E., New, Joshua R. (2013). "Simulation Speedup of EnergyPlus via Machine Learning Approximation and Inverse Modeling Accuracy" May appear in Journal of Energy and Buildings. 280. Edwards, Richard E., Zhang, Hao, Parker, Lynne, and New, Joshua R. (2012). "Approximate l-fold cross- validation with Least Squares SVM and Kernel Ridge Regression." In ORNL Publication Tracking System; rejected by Proceedings of the Neural Information Processing Systems (NIPS). 281. Garrett, Aaron and New, Joshua R. (2012). "Evolutionary Tuning of Building Model Parameters" In ORNL Publication Tracking System. 44. MCA01S007 282. C. Pelissier and A. Alexandru, “Resonance parameters of the rho-meson from asymmetrical lattices,” arXiv:1211.0092 [hep-lat]. 283. A. Alexandru and I. Horvath, “Spontaneous Chiral Symmetry Breaking as Condensation of Dynamical Chirality,” arXiv:1210.7849 [hep-lat]. 284. M. Lujan, A. Alexandru, Y. Chen, T. Draper, W. Freeman, M. Gong, F. X. Lee and A. Li et al., “The ∆_mix parameter in the overlap on domain-wall mixed action,” Phys. Rev. D 86, 014501 (2012) [arXiv:1204.6256 [hep-lat]]. 285. Y. -B. Yang, Y. Chen, G. Li and K. -F. Liu, “Is 1−+ Meson a Hybrid?,” Phys. Rev. D 86, 094511 (2012) [arXiv:1202.2205 [hep-ph]].

312 286. H. -W. Lin and K. -F. Liu, “Comment on ‘Controversy concerning the definition of quark and gluon angular momentum’ by Elliot Leader (PRD 83, 096012 (2011)),” Phys. Rev. D 85, 058901 (2012) [arXiv:1111.0678 [hep-ph]]. 287. A. Alexandru and I. Horvath, “How Self-Dual is QCD?,” Phys. Lett. B 706, 436 (2012) [arXiv:1110.2762 [hep-lat]]. 288. K. F. Liu, “Charge-dependent Azimuthal Correlations in Relativistic Heavy-ion Collisions and Electromagnetic Effects,” Phys. Rev. C 85, 014909 (2012) [arXiv:1109.4883 [nucl-th]]. 289. A. Alexandru, C. Pelissier, B. Gamari and F. Lee, “Multi-mass solvers for lattice QCD on GPUs,” J. Comput. Phys. 231, 1866 (2012) [arXiv:1103.5103 [hep-lat]]. 290. W. Freeman, A. Alexandru, F. Lee and M. Lujan, “Sea Contributions to Hadron Electric Polarizabilities through Reweighting,” PoS LATTICE 2012, 015 (2012) [arXiv:1211.-5570 [hep-lat]]. 291. A. Alexandru and I. Horvath, “Chiral Polarization Scale at Finite Temperature,” PoS LATTICE 2012, 210 (2012) [arXiv:1211.2601 [hep-lat]]. 45. MCA01S018 292. Gumbart, J., Khalili-Araghi, F., Sotomayor, M. & Roux, B. Constant electric field simulations of the membrane potential illustrated with simple systems. Biochim Biophys Acta 1818, 294-302 (2012). 293. Gumbart, J. & Roux, B. Determination of membrane-insertion free energies by molecular dynamics simulations. Biophys J 102, 795-801 (2012). 294. Herrou, J., Rotskoff, G., Luo, Y., Roux, B. & Crosson, S. Structural basis of a protein partner switch that regulates the general stress response of alpha-proteobacteria. Proc Natl Acad Sci USA 109, E1415-1423 (2012). 295. Hyde, H. C., Sandtner, W., Vargas, E., Dagcan, A. T., Robertson, J. L., Roux, B., Correa, A. M. & Bezanilla, F. Nano-positioning system for structural analysis of functional homomeric proteins in multiple conformations. Structure 20, 1629-1640 (2012). 296. Jo, S., Jiang, W., Lee, H. S., Roux, B. & Im, W. CHARMM-GUI Ligand Binder for Absolute Binding Free Energy Calculations and Its Application. J Chem Inf Model (2012). 297. Khalili-Araghi, F., Tajkhorshid, E., Roux, B. & Schulten, K. Molecular dynamics investigation of the omegacurrent in the Kv1.2 voltage sensor domains. Biophys J 102, 258-267 (2012). 298. Kwon, T., Roux, B., Jo, S., Klauda, J. B., Harris, A. L. & Bargiello, T. A. Molecular dynamics simulations of the Cx26 hemichannel: insights into voltage-dependent loop-gating. Biophys J 102, 1341-1351 (2012). 299. Lacroix, J. J., Pless, S. A., Maragliano, L., Campos, F. V., Galpin, J. D., Ahern, C. A., Roux, B. & Bezanilla, F. Intermediate state trapping of a voltage sensor. The Journal of general physiology 140, 635-652 (2012). 300. Lee, K. I., Jo, S., Rui, H., Egwolf, B., Roux, B., Pastor, R. W. & Im, W. Web interface for Brownian dynamics simulation of ion transport and its applications to beta-barrel pores. J Comput Chem 33, 331-339 (2012). 301. McNeish, A., Roux, B., Aylett, S. B., Van Den Brink, A. & Cottrell, G. Endosomal proteolysis regulates calcitonin gene-related peptide responses in mesenteric arteries. Br J Pharmacol 167, 1679-1690 (2012). 302. Raghuraman, H., Cordero-Morales, J. F., Jogini, V., Pan, A. C., Kollewe, A., Roux, B. & Perozo, E. Mechanism of Cd(2+) Coordination during Slow Inactivation in Potassium Channels. Structure 20, 1332-1342 (2012). 303. Robertson, J. L., Palmer, L. G. & Roux, B. Multi-ion distributions in the cytoplasmic domain of inward rectifier potassium channels. Biophysical Journal 103, 434-443 (2012). 304. Roux, B. Ion binding sites and their representations by reduced models. The journal of physical chemistry 116, 6966-6979 (2012). 305. Roux, B. Comment on ”Probing the Thermodynamics of Competitive Ion Binding Using Minimum Energy Structures”. The journal of physical chemistry (2012). 306. Rowley, C. N. & Roux, B. The Solvation Structure of Na+ and K+ in Liquid Water Determined from High Level ab Initio Molecular Dynamics Simulations. J. Chem. Theo. Comp. 8, 3526-3535 (2012). 307. Vargas, E., Yarov-Yarovoy, V., Khalili-Araghi, F., Catterall, W. A., Klein, M. L., Tarek, M., Lindahl, E., Schulten, K., Perozo, E., Bezanilla, F. & Roux, B. An emerging consensus on voltage-dependent gating from computational modeling and molecular dynamics simulations. The Journal of general physiology 140, 587-594 (2012). 308. Ziervogel, B. K. & Roux, B. The Binding of Antibiotics in OmpF Porin. Structure (2012). 309. Lin, Y.L., Meng, Y., Jiang, W. & Roux, B. Explaining why Gleevec is a specific and potent inhibitor of Abl kinase. Proc Natl Acad Sci USA, (2013). 310. Gumbart, J.C., and B. Roux. Determination of membrane-insertion free energies by molecular dynamics simulations. Biophys. J. 102(4), 795–801 (2012).

313 311. Jiang, W., Y. Luo, L. Maragliano, and B. Roux. Calculation of free energy landscape in multi-dimensions with hamiltonian-exchange umbrella sampling on petascale supercomputer. J. Chem. Theo. Comp. 8, 4672-4680 (2012). 312. Luo, Y., W. Jiang, H. Yu, A. D. MacKerell, and B. Roux. Simulation study of ion pairing in concentrated aqueous salt solutions with a polarizable force field. Farad. Discuss. (2013). 313. Gumbart, J. C and Roux, B. & Chipot, C., Standard Binding Free Energies from Computer Simulations: What Is the Best Strategy? J. Chem. Theo. Comp. 9, 794-802, (2013). 314. Riahi, S., B. Roux, and C. N. Rowley. QM/MM Molecular Dynamics Simulations of the Hydration of Mg(II) and Zn(II) Ions. Can. J. Chem. (2013). 315. Gumbart, J.C., I. Teo, B. Roux, and K. Schulten. Reconciling the roles of kinetic and thermodynamic factors in membrane-protein insertion. J. Am. Chem. Soc. (2013). 316. Wojtas-Niziurski, W., Y. Meng, B. Roux, and S. Berneche. Automatic Self-Learning Adaptive Umbrella Sampling Method for the Determination of Free Energy Landscapes in Multiple Dimensions. J. Chem. Theo. Comp. (2013). 46. MCA02N014 317. D. Alic, C. Bona-Casas, C. Bona, L. Rezzolla, and C. Palenzuela, Conformal and covariant formulation of the Z4 system with constraint-violation damping, Phys. Rev. D85, 064040 (2012), 1106.2254. 318. D. Alic, P. Mosta, L. Rezzolla, O. Zanotti, and J. L. Jaramillo, Accurate Simulations of Binary Black-Hole Mergers in Force-Free Electrodynamics, Astrophys.J. 754, 36 (2012), 1204.2226. 319. C. Reisswig, N. T. Bishop, and D. Pollney, General relativistic null-cone evolutions with a high-order scheme (2012), 1208.3891. 320. T. Damour, A. Nagar, D. Pollney, and C. Reisswig, Energy Versus Angular Momentum in Black Hole Binaries, Phys. Rev. Lett. 108, 131101 (2012). 321. J. D. Brown, P. Diener, S. E. Field, J. S. Hesthaven, F. Herrmann, et al., Numerical simulations with a first order BSSN formulation of Einstein’s field equations, Phys.Rev. D85, 084004 (2012), 1202.1038. 322. P. Diener, I. Vega, B. Wardell, and S. Detweiler, Self-consistent orbital evolution of a particle around a Schwarzschild black hole, Phys.Rev.Lett. 108, 191102 (2012), 1112.4821. 323. B. Wardell, I. Vega, J. Thornburg, and P. Diener, A Generic effective source for scalar self-force calculations, Phys.Rev. D85, 104044 (2012), 1112.6355. 324. F. L¨offler, J. Faber, E. Bentivegna, T. Bode, P. Diener, et al., The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics, Class.Quant.Grav. 29, 115001 (2012), 1111.3344. 47. MCA03S007 325. H. Wen, M. Terrel, S. Fan, and M. J. F. Digonnet, “Sensing with slow light in fiber bragg gratings”, IEEE Sensors Journal, vol. 12, pp. 156-163 (2012). 326. S. Fan, R. Baets, A. Petrov, Z. Yu, J. D. Joannopuolos, W. Freude, A. Melloni, M. Popovic, M. Vanwolleghem, D. Jalas, M. Eich, M. Krausse, H. Renner, E. Brinkmeyer, C. R. Doerr, “Comment on ‘Nonreciprocal light propagation in a 327. silicon photonic circuit’, by Feng et al Science 333, 729, 2011”, Science, vol. 335, p. 38 (2012). 328. L. Zhu, C. R. Otey and S. Fan, “Negative differential thermal conductance through vacuum”, Applied Physics Letters, vol. 100, art. No. 044104 (2012). 329. S. E. Kocabas, E. Rephaeli, and S. Fan, “Resonance fluorescence in a waveguide geometry”, Physical Review A, vol. 85, art. No. 023817 (2012). 330. S. Sandhu and S. Fan, “Lossless intensity modulation”, Optics Express, vol. 20, no. 4, pp. 4280-4290 (2012). 331. Y. Yao, J. Yao, V. K. Narasimhan, Z. Ruan, C. Xie, S. Fan and Y. Cui, “Broadband light management using low-Q whispering gallery modes in spherical nanoshells”, Nature Communications, vol. 3, art. No. 664 (2012). 332. K. X. Wang, Z. Yu, V. Liu, Y. Cui and S. Fan, “Absorption enhancement in ultrathin crystalline silicon solar cells with anti-reflection and light-trapping nanocone gratings”, Nano Letters, vol. 12, pp. 1616-1619 (2012). 333. L. Verslegers, Z. Yu, Z. Ruan, P. B. Catrysse, and S. Fan, “From electromagnetically induced transparency to superscattering with a single structure: a coupled-mode theory for doubly resonant structures”, Physical Review Letters, vol. 108, art. No. 083902 (2012). 334. W. Shin and S. Fan, “Choice of the perfectly matched layer boundary condition for frequency domain Maxwell’s equations solvers”, Journal of Computational Physics, vol. 231, pp. 3406-3431 (2012). 335. M. A. Terrel, M. Digonnet and S. Fan, “Resonant fiber optic gyroscope using an air-core fiber”, Journal of Lightwave Technology, vol. 30, no. 7, pp. 931-937 (2012). 336. E. Rephaeli and S. Fan, “Stimulate emission from a single excited atom in a waveguide”, Physical Review Letters, vol. 108, art. No. 143602 (2012).

314 337. K. Fang, Z. Yu and S. Fan, “Photonic Aharonov-Bohm effect based on dynamic modulation”, Physical Review Letters, vol. 108, art. No. 103901 (2012). 338. Z. Ruan and S. Fan, “Temporal coupled-mode theory for light scattering by an arbitrary-shaped object supporting a single resonance”, Physical Review A, vol. 85, art. No. 043828 (2012). 339. W. Shin, A. Raman and S. Fan, “Instantaneous electric energy and electric power dissipation in dispersive media”, Journal of the Optical Society of America B, vol. 29, pp.1048-1054 (2012). 340. C. –M. Hsu, C. Battaglia, C. Pahud, Z. Ruan, F. –J. Haug, S. Fan, C. Ballif, and Y. Cui, “High-efficiency amorphous silicon solar cell on periodic nanocone back reflectors”, Advanced Energy Materials, vol. 2, pp. 628-633 (2012). 341. V. Liu, and S. Fan, “S4: a free electromagnetic solver for layered periodic structures”, Computer Physics Communications, vol. 183, pp. 2233-2244 (2012). 342. S. Jeong, E. Garnett, S. Wang, Z. Yu, S. Fan, M. L. Brongersma, M. Mcgehee, and Y. Cui, “Hybrid Si nanocone-polymer solar cells”, Nano Letters, vol. 12, pp. 2971-2976 (2012). 343. H. Lira, Z. Yu, S. Fan and M. Lipson, “Electrically driven nonreciprocity induced by interband photonic transition on a silicon chip”, Physical Review Letters, vol. 109, art. No. 033901 (2012). 344. H. Iizuka and S. Fan, “Rectification of heat transfer between dielectric-coated and un-coated silicon carbide plates”, Journal of Applied Physics, vol. 112, art. No. 024304 (2012). 345. A. Bynes, R. Pant, E. Li, D. –Y. Choi, C. G. Poulton, S. Fan, S. Madden, B. Luther-Davis and B. J. Eggleton, “On-chip tunable and reconfigurable narrowband microwave photonic filter using stimulated Brillouin scattering”, Optics Express, vol. 20, pp. 18845-18854 (2012). 346. C. G. Poulton, R. Pant, A. Byrnes, S. Fan, M. J. Steel and B. J. Eggleton, “Design for broad-band on-chip isolator using Stimulated Brillouin Scattering in dispersion engineered chalcogenide waveguides”, Optics Express, vol. 20, pp. 21235-21246 (2012). 347. B. Gupa, C. Otey, C. Poitras, S. Fan, and M. Lipson, “Near-field radiative cooling of nanostructures”, Nano Letters, vol. 12, pp. 4546-4550 (2012). 348. E. Rephaeli and S. Fan, “Few photon single atom cavity QED with input-output formalism in Fock space”, IEEE Journal of Selected Topics on Quantum Electronics, vol. 18, pp. 1754-1762 (2012). 349. Z. Yu, A. Raman and S. Fan, “Thermodynamic upper bound on broadband light coupling with photonic structures”, Physical Review Letters, vol. 109, art. No. 173901 (2012). 350. A. Kinkhabwala, Z. Yu, S. Fan, and W. E. Moerner, “Fluoresence correlation spectroscopy at high concentration using gold bowtie nanoantennas”, Chemical Physics, vol. 406, pp. 3-8 (2012). 351. K. Fang, Z. Yu and S. Fan, “Realizing effective magnetic field for photons by controlling the phase of dynamic modulation”, Nature Photonics, vol. 6, pp. 782-787 (2012). 352. V. Liu, D. A. B. Miller and S. Fan, “Ultra-compact photonic crystal waveguide spatial mode converter and its connection to the optical diode effect”, Optics Express, vol. 20, pp. 28388-28397 (2012). 353. D. Liang, Y. Huo, Y. Kang, K. X. Wang, A. Gu, M. Tan, Z. Yu, S. Li, J. Jia, X. Bao, S. Wang, Y. Yao, H. –S. P. Wong, S. Fan, Y. Cui, and J. S. Harris, “Optical absorption enhancement in freestanding GaAs thin film nanopyramid arrays”, 354. Advanced Energy Materials, vol. 2, pp. 1254-1260 (2012). 355. V. Intarprasonk and S. Fan, “Enhancing waveguide-resonator optical force with an all-optical on-chip analogue of electromagnetically induced transparency”, Physical Review A 86, 063833 (2012). 356. V. Liu, D. A. B. Miller and S. Fan, “Application of computational electromagnetics for nanophotonics design and discovery”, Proceedings of the IEEE (Invited paper, accepted). 357. T. Tanemura, P. Wahl, S. Fan, and D. A. B. Miller, “Modal source radiator model for arbitrary two- dimensional arrays of sub-wavelength apertures on metal films”, IEEE Journal of Selected Topics on Quantum Electronics (invited paper, accepted). 358. L. Hu, G. Zheng, J. Yao, N. Liu, B. Weil, Y. Cui, M. Eskilsson, E. Karabulut, L. Wagberg, Z. Ruan, S. Fan, J. T. Bloking, M. D. MiGehee, “Transparent and conductive paper from nanocellulose fibers”, Energy and Environmental Science (accepted). 359. P. B. Catrysse and S. Fan, “Routing of deep-subwavelength optical beams and images without reflection and diffraction using infinitely anisotropic metamaterials”, Advanced Materials (accepted). 360. K. X. Wang, Z. Yu, S. Sandhu and S. Fan, “Fundamental Bounds on Decay Rates in Asymmetric Single-Mode Optical Resonators”, Optics Letters (accepted). 361. Keynote Talk, “Nanophotonics for the control of thermal electromagnetic fields”, in International Workshop on Nano-Micro Thermal Radiation, May 22, 2012, Hotel Matsushima Taikanso, Sendai, Japan. 362. Distinguished Speaker, “Nanophotonics for energy and information applications”, Department of Electrical and Computational Engineering, Purdue University, December 13, 2012. 363. S. Fan, “Dynamic and aperiodic nanophotonic structures”, in Photonic and Phononic Properties of Engineering Nanostructures II, SPIE Photonics West, San Francisco, California, January 24, 2012.

315 364. S. Fan, “Dark resonances in all-optical analogue to EIT: lossless intensity modulation, force enhancement, and optical antennas”, in Advances in Slow and Fast Light V, SPIE Photonics West, San Francisco, California, January 24, 2012. 365. S. Fan, “The influence of the 4n^2 factor on solar cells”, Photons, Electrons, Bands, A Symposium on the Diversity of Opto-Electronics in Celebration of Prof. Eli Yablonovitch’s 65th birthday, University of California, Berkeley, January 27, 2012. 366. S. Fan, “Non reciprocity without magneto-optics”, Plenary Session, the 11th Annual CUDOS Workshop, Shoal Bay, New South Whales, Australia, January 31, 2012. 367. S. Fan, “Nanophotonics for solar and thermal applications”, Seminar at the School of Photovoltaic and Renewable Energy Engineering, University of New South Whales, Sydney, Australia, February 21, 2012. 368. S. Fan, “The physics of nanophotonic structures”, Physics Colloquium, Department of Physics, University of Sydney, Australia, April 2, 2012. 369. S. Fan, “Direct incorporation of the electronic degree of freedom in plasmonic modeling”, Symposium KK: Plasmonic Materials and Meta-Materials, Materials Research Society Spring Meeting, San Francisco, California, April 11, 2012. 370. “Nanophotonics for solar and thermal applications”, Swinburne University of Technology, Melbourne, Australia, May 7, 2012. 371. “Aspects of nanophotonics, photonic transitions and near-field thermal radiation”, the 10th International Congress on Photonic and Electromagnetic Crystal Structures, Santa Fe, New Mexico, June 5, 2012. 372. “Control near-field and far-field thermal electromagnetic properties with nanophotonic concepts”, Gordon Research Conference on Plamonics, Colby College, Waterville, Maine, June 13, 2012. 373. “Aspects of on-chip photonics: dynamically included non-reciprocity, and few photon quantum transport in waveguide-atom system”, Department Seminar, Department of Physics, Macquarie University, Sydney, Australia, June 22, 2012. 374. “Nanophotonics and its application for information technology”, Department Seminar, Department of Electrical Engineering, University of Sydney, Sydney, Australia, June 26, 2012. 375. “Nanophotonics theory, and information and thermal applications”, Seminar, Research School of Physical Sciences and Engineering, the Australian National University, Canberra, Australia, July 1, 2012. 376. “Nanophotonics: from fundamentals to applications”, Institute of Semiconductor, Chinese Academy of Sciences, Beijing, China, July 16, 2012. 377. “Nanophotonics: from fundamentals to applications”, Institute of Physics, Chinese Academy of Sciences, Beijing, China, July 17, 2012. 378. “Non-reciprocity and Effective Real-space Magnetic Field for Photons from Dynamic Modulation”, International Nano-Optoelectronic Workshop (INOW), Berkeley and Stanford, California, August 13, 2012. 379. “Sub-wavelength electromagnetic structures”, Frontiers of THz Science, SLAC National Accelerator Laboratory, Stanford, California, September 6, 2012. 380. Z. Yu and S. Fan, “Light management for sunlight and thermal emission”, IEEE Photonics Conference (IPC12), Burlingame, California, September 24, 2012. 381. “Interference Between Nanophotonic Resonances”, IEEE Photonics Conference (IPC12), Burlingame, California, September 24, 2012. 382. “Dynamically induced non-reciprocity and gauge fields for photons”, Workshop on New Materials and New Concepts for Controlling Light and Waves, Croucher Advanced Study Institute, the Hong Kong University of Science and Technology, Hong Kong, China, October 3, 2012. 383. “Photonic transition can induce non-reciprocity and effective gauge field for photons”, the 5th International Workshop on Theoretical and Computational Nano-Photonics (TaCoNa-Photonics 2012), Physikzentrum, Bad Honnef, Germany, October 26, 2012. 384. “Non-reciprocity in silicon from dynamic modulation”, Nanophotonics and Micro/Nano Optics, SPIE Photonics Asia, Beijing, China, November 6, 2012. 385. “Non-reciprocity and time-reversal symmetry breaking gauge field for photons”, Ginzton-Pulse Seminar Series, SLAC National Laboratory, Stanford, CA, December 5, 2012. 386. “Non-reciprocity and non-reciprocal effective gauge field in dynamically modulated photonic structures”, Joint Quantum Institute (JQI) Seminar, University of Maryland, College Park, MD, December 11, 2012. 48. MCA06N038 387. Reddy, H. and Abraham, J. (2013) Influence of Turbulence-Kernel Interactions on Flame Development in Lean Methane/Air Mixtures Under Natural Gas-Fueled Engine Conditions, Fuel, 103:1090-1105. 388. Reddy, H. and Abraham, J. (2012) Two-Dimensional Direct Numerical Simulation Evaluation of the Flame Surface Density Model for Flames Developing from an Ignition Kernel in Lean Methane/Air Mixtures Under Engine Conditions, Physics of Fluids, 105108.

316 389. Mukhopadhyay, S. and Abraham, J. (2012) Influence of Heat Release and Turbulence on Scalar Dissipation Rate in Autoigniting n-Heptane/Air Mixtures, Combustion and Flame, 159(9):2883-2895. 390. Reddy, H. and Abraham, J. (2012) Transient Evolution of Strain and Curvature in Flames Developing from Kernels, International Journal of Spray and Combustion Dynamics, 4 (4), 323-340. 391. Mukhopadhyay, S. and Abraham, J. (2012) Influence of Turbulence on Autoignition in Stratified Mixtures Under Compression Ignition Engine Conditions, Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, published online October 17, 2012. DOI:http://pid.sagepub.com/content/early/2012/10/17/0954407012459624 392. Mukhopadhyay, S. and Abraham, J. (2012) Evaluation of an Unsteady Flamelet Progress Variable Model for Autoignition and Flame Development in Compositionally Stratified Mixtures, Physics of Fluids, 075115. 49. MCA06N043 393. A. Uzun and M. Y. Hussaini, “Some Issues in Large-Eddy Simulations for Chevron Nozzle Jet Flows,” Journal of Propulsion and Power, Vol. 28, No. 2, March-April 2012, pp. 246 – 258. 394. A. Uzun and M. Y. Hussaini, “An Application of Delayed Detached Eddy Simulation to Tan- dem Cylinder Flow Field Prediction,” Computers & Fluids, Vol. 60, May 2012, pp. 71 –85. 395. A. Uzun, R. Kumar, M. Y. Hussaini and F. S. Alvi, “Simulation of Tonal Noise Generation by Supersonic Impinging Jets,” under consideration for publication in AIAA Journal. 396. A. Uzun, J. T. Solomon, C. H. Foster, W. S. Oates, M. Y. Hussaini and F. S. Alvi, “Flow Physics of a Pulsed Microjet Actuator for High-Speed Flow Control,” under consideration for publication in AIAA Journal. 397. H. Lee, A. Uzun and M. Y. Hussaini, “Identification of Jet Noise Source Based on High-Fidelity Numerical Simulation of a Round Jet Flow,” in preparation. 398. J. Bin, H. Zhao, A. Uzun and M. Y. Hussaini, “Computations of Turbulent Pipe and Confined Jet Flows Using a Continuous Model,” in preparation. 50. MCA07S017 399. Extended hadron and two-hadron operators of definite momentum for spectrum calculations in lattice QCD, C. Morningstar, J. Bulava, B. Fahy, J. Foley, Y.C. Jhang, K.J. Juge, D. Lenkner, C.H. Wong, submitted to Phys. Rev. D. 400. Progress with applications of the stochastic LapH method to excited states, J. Bulava, B. Fahy, J. Foley, Y.C. Jhang, K.J. Juge, D. Lenkner, C. Morningstar, C.H. Wong, PoS(Lattice 2012), proceedings of the 30th International Symposium on Lattice Field Theory, June 24-29, 2012, Cairns, Australia. 401. Excited States in Lattice QCD using the Stochastic LapH Method, C. Morningstar, J. Bulava, J. Foley, K.J. Juge, C.H. Wong, B. Fahy, Y.C. Jhang, and D. Lenkner, proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the campus and beyond (XSEDE12), July 16-20, 2012, Chicago, IL, USA, [doi:10.1145/2335755.2335824]. 402. Progress with Excited States in Lattice QCD, C. Morningstar, J. Bulava, B. Fahy, J. Foley, Y.C. Jhang, K.J. Juge, D. Lenkner, C.H. Wong, proceedings of the 11th Conference on the Intersections of Particle and Nuclear Physics (CIPANP 2012), May 29-June 3, 2012, St. Petersburg, FL, USA. 51. MCA07S033 403. Pogorelov, N.V., Borovikov, S.N., Zank, G.P., Burlaga, L.F., Decker, R., Stone, E.C., Radial Velocity Along the Voyager 1 Trajectory: The Effect of Solar Cycle, Astrophys. J. Letters, 750, L4 (2012). 404. Borovikov, S.N., Pogorelov, N.V., Ebert, R.W., Solar rotation effects on the heliosheath flow near solar minima, Astrophys. J., 750, 42 (2012). 405. McComas, D. J., Alexashov, D., Bzowski, M., Fahr, H., Heerikhuisen, J., Izmodenov, V., Lee, M. A., Moebius, E., Pogorelov, N., Schwadron, N. A., & Zank, G. P., The heliosphere’s interstellar interaction: No bow shock, Science, 336, 1291 (2012) 406. Gamayunov, K. V., Zhang, M., Pogorelov, N V., Heerikhuisen, J, Rassoul, H. K., Self-consistent Model of the Interstellar Pickup Protons, Alfvénic Turbulence, and Core Solar Wind in the Outer Heliosphere, Astrophys. J., 757, 74 (2012) 407. Kim, T.K., Pogorelov, N.V., Borovikov, S.N., Hayashi, K., Numerical Modeling of Solar Wind Flow Using Interplanetary Scintillation Data as Boundary Conditions, in Numerical Modeling of Space Plasma Flows: ASTRONUM-2011, ed. N.V. Pogorelov, J.A. Font, E. Audit, & G.P. Zank, pp. 209 – 215, Astronomical Society of the Pacific Conf. Ser. 459, San Francisco (2012). 408. Kryukov, I.A., Pogorelov, N.V., Zank, G.P., Borovikov, S.N., Numerical modeling of the solar wind turbulence, in Physics of the Heliosphere: A 10 Year Retrospective, Proc. 10th Annual Astrophysics Conference, American Institute of Physics Conf. Ser. 1436, 48 – 54, New York (2012).

317 409. Heerikhuisen, J., Pogorelov, N., Zank, G., Modeling energetic neutral atoms and the IBEX spectrum, in Physics of the Heliosphere: A 10 Year Retrospective, Proc. 10th Annual Astrophysics Conference, American Institute of Physics Conf. Ser. 1436, 221 – 226, New York (2012). 410. Pogorelov, N.V., Borovikov, S.N., Burlaga, L.F., Ebert, R.W., Heerikhuisen, J., Hu, Q., Kryukov, I.A., Suess, S.T., Zank, G.P., Numerical modeling of transient phenomena in the distant solar wind and in the heliosheath, in Physics of the Heliosphere: A 10 Year Retrospective, Proc. 10th Annual Astrophysics Conference, American Institute of Physics Conf. Ser. 1436, 321 – 330, New York (2012). 411. Pogorelov et al. Numerical Modeling of the Solar Wind Flow with Observational Boundary Conditions, 11th Annual Astrophysics Conference, Palm Springs, CA, March 19-23, 2012. (Invited). 412. Pogorelov et al. Unsteady Processes in the Vicinity of the Heliopause, 13th International Solar Wind Conference, 18-22 June, 2012, Big Island, HI. 413. Pogorelov et al. Modeling Solar Wind Flow with the Multi-Scale Fluid-Kinetic Simulation Suite, 7th International Conference on Numerical Modeling of Space Plasma Flows, 24-29 June, 2012, Big Island, HI. 414. Pogorelov et al.. Modeling Heliosheath Flow with Observational Boundary Conditions, 39th COSPAR Scientific Assembly, 14-22 July, 2012, Mysore, India. 415. Pogorelov et al.. Solar Wind Behavior at Voyager Spacecraft from the Simulation Perspective, Voyager Science Steering Group Meeting, 4-5 September, 2012, Pasadena, CA. 416. Pogorelov et al.. Modeling CMEs from the Sun to the Earth Orbit, 2nd In-situ Heliospheric Science Conference, September 18-20, 2012, Laurel, MD. 417. Pogorelov et al.. Solar Cycle Model based on Ulysses Measurements, 2nd In-situ Heliospheric Science Conference, September 18-20, 2012, Laurel, MD. 418. Pogorelov et al.. Magnetic Fields in the Vicinity of the Heliopause, 2nd In-situ Heliospheric Science Conference, September 18-20, 2012, Laurel, MD. 419. Pogorelov et al.. Nonstationary Phenomena in the Heliosheath, 2012 Fall AGU Meeting, December 3-7, 2012, San Francisco, CA. 420. Heerikhuisen et al. 11th AIAC Palm Springs, March 19-23, 2012: "Modeling of Energetic Neutral Atoms Observed in the Inner Heliosphere coming from the Heliosheath" 421. Heerikhuisen et al. Solar Wind 13, Big Island, June 18-22, 2012: "Connection of the Solar Wind with the Interstellar Medium through Numerical Modeling", an invited talk. 422. Heerikhuisen et al. Astronum 2012, Big Island, June 25-29, 2012: "Simulating the Heliosphere with MHD and Kinetic Hydrogen". 423. Heerikhuisen et al. Global Modeling of the Heliosphere and the IBEX Ribbon, 2012 Fall AGU Meeting, December 3-7, 2012, San Francisco, CA. 424. Borovikov et al. Solar Rotation Effects on the Heliosheath Flow near Solar Minima, Solar Wind 13, 18-22 June, 2012, Big Island, HI 425. Borovikov et al. Numerical Simulations of the Solar Wind Flow: From Sun Surface to the Heliotail, ASTRONUM-2013, 7th International Conference on Numerical Modeling of Space Plasma Flows, 24-29 June, 2012, Big Island, HI 426. Borovikov et al. The Role of the Interstellar and Interplanetary Magnetic Fields in the Heliopause Stability, 2012 Fall AGU Meeting, December 3-7, 2012, San Francisco, CA. 427. Borovikov et al. Numerical Simulation of the January 2012 CME Event and its Propagation to the Earth Orbit, 2012 Fall AGU Meeting, December 3-7, 2012, San Francisco, CA. [Invited] 52. MCA08X002 428. Lingle Wang, Bruce J. Berne, and Richard A. Friesner, On achieving high accuracy and reliability in the calculation of relative protein-ligand binding affinities, Proc. Natl. Acad. Sci., 109, 6, 1937-1942 (2012) 429. Jing Zhang, Thomas F. Hughes, Michael Steigerwald, Louis Brus, and Richard A. Friesner, Realistic Cluster Modeling of Electron Transport and Trapping in Solvated TiO2 Nanoparticles, J. Am. Chem. Soc., 134, 29, 12028 – 12042 (2012). 430. J. Mondal and B. J. Berne. How water contributes to free energy barriers in ligand binding : Insights from kinase-ligand binding. To be submitted to Proc. Natl. Acad. Sci. (2013). 431. J. Mondal, J. A. Morrone, and B. J. Berne. How the drying transition affects ligand binding in model systems. To be submitted (2013). 432. J. Mondal, G. Stirnemann, and B. J. Berne. Insights on reverse role of protecting and denaturing osmolytes on hydrophobic interactions: A computer simulation study. To be submitted (2013). 433. R. A. Nistor, T. E. Markland, and B. J. Berne. Probing the activation energy land-scape of ice growth in the supercooled regime. To be submitted to JCP (2013). 434. R. A. Nistor, T. E. Markland, and B. J. Berne. Solute induced kinetic barriers inhibiting ice growth. To be submitted to JPCL (2013).

318 53. MCA08X003 435. Olsen BN, Schlesinger PH, Ory DS, Baker NA. Side-Chain Oxysterols: From Cells to Membranes to Molecules. BBA Biomembranes, 1818, 330-336, 2012. PMC3197895. 436. Lee S-J, Schlesinger PH, Wickline SA, Lanza GM, Baker NA. Interaction of Melittin Peptides with Perfluorocarbon Nanoemulsion Particles. J Phys Chem B, 115, 15271-15279, 2012. PMC3245340. 437. Lee S-J, Schlesinger PH, Wickline SA, Lanza GM, Baker NA. Simulation of fusion-mediated nanoemulsion interactions with model lipid bilayers. Soft Matter, 8, 7024-7025, 2012. PMC3375911. 54. MCA08X004 438. M. Berzins, Q. Meng, J. Schmidt, J.C. Sutherland. DAG-Based Software Frameworks for PDEs, In Proceedings of Euro-Par 2011 Workshops, Part I, Lecture Notes in Computer Science (LNCS) 7155, Springer- Verlag Berlin Heidelberg, pp. 324–333. August, 2012. 439. J. van Rij, T. Harman, and T. Ameel., Slip Flow Fluid-Structure Interaction, International Journal of Thermal Sciences, Volume 58, August 2012, Pages 9–19, ISSN 1290-0729. 440. J. Peterson and C.Wight., An Eulerian-Lagrangian Computational Model for Deflagration to Detonation Transition, Combustion and Flame, Volume 159, Issue 7, July 2012, Pages 2491–2499. 441. J. Peterson , J. Beckvermit, T. Harman, M. Berzins and C. Wight., Multiscale Modeling of High Explosives for Transportation Accidents, in Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment (XSEDE 2012). ACM, New York, NY, USA, Article 32, DOI 10.1145/2335755.2335828. 442. M. Berzins, J. Schmidt, Q. Meng, A. Humphrey, Past, Present, and Future Scalability of the Uintah Software, Proceedings of the Blue Waters Workshop 2012, 2013 (to appear). 443. Berzins, M. Status of Release of the Uintah Computational Framework, Technical Report, No. UUSCI-2012- 001, SCI Institute, University of Utah, 2012. 444. J. Schmidt, M. Berzins, J. Thornock, T. Saad, J. Sutherland, Large Scale Parallel Solution of Incompressible Flow Problems using Uintah and Hypre, SCI Technical Report, No. UUSCI-2012-002, SCI Institute, University of Utah, 2012,(submitted to CCGRID13). 445. A. Humphrey, Q. Meng, M. Berzins, T. Harman, Radiation Modeling Using the Uintah Heterogeneous CPU/GPU Runtime System, in Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment (XSEDE 2012). ACM, New York, NY, USA, Article 4, DOI 10.1145/2335755.2335791. 446. Q. Meng, M. Berzins, Scalable Large-scale Fluid-structure Interaction Solvers in the Uintah Framework via Hybrid Task-based Parallelism Algorithms, SCI Technical Report No. UUSCI-2012-004, SCI Institute, University of Utah / Concurrency and Computation (Submitted), 2012. 447. Q. Meng, A. Humphrey, M. Berzins, The Uintah Framework: A Unified Heterogeneous Task Scheduling and Runtime System, to appear in Digital Proceedings of Supercomputing 2012 - 2nd International Workshop on Domain-Specific Languages and High-Level Frameworks for for High Performance Computing (WOLFHPC 2012), Supercomputing 2012 (SC12). 448. A. Humphrey, Q. Meng, M. Berzins, T. Harman, Radiation Modeling Using the Uintah Heterogeneous CPU/GPU Runtime System, SCI Technical Report, No. UUSCI-2012-003, SCI Institute, University of Utah, 2012. 449. J. Peterson, C. Wight, and M. Berzins, Applying High-performance Computing to Petascale Explosive Simulations. International Conference on Computational Sciences, Procedia Computer Science, 2013 (submitted) 450. Q. Meng, A. Humphrey, M. Berzins, The Uintah Framework: A Unified Heterogeneous Task Scheduling and Runtime System, Supercomputing 2012 (SC’12) - 2nd International Workshop on Domain-Specific Languages and High-Level Frameworks for for High Performance Computing (WOLFHPC 2012). 451. M. Berzins, Q. Meng, J Schmidt, A. Humphrey, J. Luitjens Programming Beyond Petascale? Scalability of Uintah Complex Engineering Applications Using Asynchronous Parallelism, November 9, 2012, Invited Presentation, HP Consortium for Advanced Scientific and Technical Computing (HP-CAST 19), Salt Lake City, UT 2012. 452. A. Humphrey, Q. Meng, M. Berzins, The Uintah Framework: A Unified Heterogeneous Task Scheduling and Runtime System”, November 10, 2012, Invited Presentation, HP Consortium for Advanced Scientific and Technical Computing (HP-CAST 19), Salt Lake City, UT 2012. 453. M. Berzins, J. Schmidt, Q. Meng, A. Humphrey, Past, Present, and Future Scalability of the Uintah Software, Invited Presentation, Blue Waters Workshop, Chicago, IL 2012. 454. Q. Meng, M. Berzins, Scaling Uintah’s AMR to 200K Cores and Beyond, Feb 15, 2012, 15th SIAM Conference on Parallel Processing for Scientific Computing, Savannah, GA.

319 455. A. Humphrey, Q. Meng, M. Berzins, ”Migrating the Uintah Computational Framework to CPU/GPU Architectures, Feb 15, 2012, 15th SIAM Conference on Parallel Processing for Scientific Computing, Savannah, GA. 456. A. Humphrey, Q. Meng, M. Berzins, T. Harman, Radiation Modeling Using the Uintah Heterogeneous CPU/GPU Runtime System, 1st Conference of the Extreme Science and Engineering Discovery Environment (XSEDE 2012). 457. J. Beckvermit, J. Peterson, T. Harman, M. Berzins and C. Wight., Multiscale Modeling of High Explosives for Transportation Accidents, 1st Conference of the Extreme Science and Engineering Discovery Environment (XSEDE 2012). 458. J. Peterson and C. Wight., Mesoscale Reaction Modeling of Granular Beds of Explosives Subject to Impact, 1st Conference of the Extreme Science and Engineering Discovery Environment (XSEDE 2012) Best Undergraduate Research Poster Award. 55. MCA08X008 459. T. Appelquist, et al. [LSD Collaboration], WW Scattering Parameters via Pseudoscalar Phase Shifts, Phys. Rev. D 85, 074505 (2012). http://dx.doi.org/10.1103/PhysRevD.85.074505 460. T. Appelquist, et al. [LSD Collaboration], Approaching Conformality with Ten Flavors, arXiv:1204.6000 [hep-ph]. http://arxiv.org/abs/arXiv:1204.6000 461. T. Appelquist, et al. [LSD Collaboration], Lattice calculation of composite dark matter form factors, arXiv:1301.1693 [hep-ph]. http://arxiv.org/abs/arXiv:1301.1693 56. MCA08X032 462. Y. MIAO, Z. YI. D. GLASS, L. HONG, M. TYAGI, J. BAUDRY, N. JAIN, J.C. SMITH. Temperature- Dependent Dynamical Transitions of Different Classes of Amino Acid Residue in a Globular Protein. Journal of the American Chemical Society. 34 (48), pp 19576–19579 (2012). 463. Y. MIAO, Z. YI, C. CANTRELL D. GLASS, J. BAUDRY, N. JAIN, J.C. SMITH. Coupled Flexibility Change in Cytochrome P450cam Substrate Binding Determined by Neutron Scattering, NMR, and Molecular Dynamics Simulation. Biophysical Journal 103: 2167-2176 (2012). 464. B. MINTZ, J.PARKS Benchmark interaction energies for biologically relevant non-covalent complexes containing divalent sulfur Journal Physical Chemistry A 116, 1086-1092 (2012). 465. A. SANGHA, J. PARKS, R. STANDAERT, A. ZIEBELL, M. DAVIS & J.C. SMITH. Radical Coupling Reactions in Lignin Synthesis: A DFT study. Journal of Physical Chemistry B 116 ,16 ,4760-4768 (2012). 466. Y. ZHENG, Y. MIAO, J. BAUDRY, N. JAIN and J.C. SMITH Derivation of Mean-Square Displacements for Protein Dynamics from Elastic Incoherent Neutron Scattering. Journal of Physical Chemistry B 116,16,5028- 5036 (2012). 57. MCA08X040 467. M. M. Budzevich, V. V. Zhakhovsky, C. T. White, and I. I. Oleynik, “Evolution of Shock-Induced Orientation-Dependent Metastable States in Crystalline Aluminum”, Phys. Rev. Lett. 109, 125505 (2012). 468. S. I. Ashitkov, N. A. Inogamov, V. V. Zhakhovskii, Yu. N. Emirov, M. B. Agranat, I. I. Oleinik, S. I. Anisimov, and V. E. Fortov, “Formation of Nanocavities in the Surface Layer of Aluminum Target Irradiated by Femtosecond Laser Pulse”, JETP Lett. 95, 176–181 (2012). 469. L. Adamska, Y. Lin, A.J. Ross, M. Batzill, and I.I. Oleynik, “Atomic and electronic structure of simple metal/graphene and complex metal/graphene/metal interfaces”, Phys. Rev. B 85, 195443 (2012). 470. L. Adamska, R. Addou, M. Batzill, and I.I. Oleynik, “Atomic and electronic structure of graphene/Sn-Ni(111) and graphene/Sn-Cu(111) surface alloy interfaces”, Appl. Phys. Lett. 101, 051602 (2012). 471. V. V. Zhakhovsky, B. J. Demaske, N. A. Inogamov, V. A. Khokhlov, S. I. Ashitkov et al., “Super-elastic response of metals to laser-induced shock waves”, AIP Conference Proceedings, 1464, 102-112 (2012). 472. S. I. Ashitkov, N. A. Inogamov, P. S. Komarov, V. V. Zhakhovsky, I. I. Oleynik et al., “Strength of metals in liquid and solid states at extremely high tension produced by femtosecond laser heating”, AIP Conference Proceedings, 1464, 120-125 (2012). 473. N. A. Inogamov, Yu. V. Petrov, V. V. Zhakhovsky, V. A. Khokhlov, B. J. Demaske et al.., “Two-temperature thermodynamic and kinetic properties of transition metals irradiated by femtosecond lasers”, AIP Conference Proceedings, 1464, 593-608 (2012). 474. M. Ishino, A. Faenov, M. Tanaka, N. Hasegawa, M. Nishikino, S. Tamotsu, T. Pikuz, N. Inogamov, V. Zhakhovsky, et al., “Interaction of soft x-ray laser pulse radiation with aluminum surface: Nano-meter size surface modification”, 3rd International Symposium on laser-driven relativistic plasmas applied to science, energy, industry, and medicine, AIP Conference Proceedings, 1465, 236-240 (2012).

320 475. V. V. Zhakhovsky, M. M. Budzevich, N. A. Inogamov, C. T. White, and I. I. Oleynik, “Single two-zone elastic-plastic shock waves in solids”, AIP Conference Proceedings, 1426, 1227-1232 (2012). 476. N. A. Inogamov, V. V. Zhakhovsky, C. T. White, I. I. Oleynik, and V. E. Fortov, “Ultrashort elastic and plastic shock waves in Aluminum”, AIP Conference Proceedings, 1426, 909-912 (2012). 477. M. M. Budzevich, V. V. Zhakhovsky, C. T. White, and I. I. Oleynik, “Shock-induced phase transitions in metals: recrystallization of supercooled melt and melting of an overheated solid”, AIP Conference Proceedings, 1426, 1267-1270 (2012) 478. M. M. Budzevich, V. V. Zhakhovsky, C. T. White, and I. I. Oleynik, “Effect of reactive chemistry on mechanisms of condensed phase detonation”, AIP Conference Proceedings, 1426, 1279-1282 (2012) 479. You Lin, R. Perriot, V. V. Zhakhovsky, Xiang Gu, C. T. White, and I. I. Oleynik, “Shockinduced phase transition in diamond”, AIP Conference Proceedings, 1426, 1171-1174 (2012) 480. R. Perriot, Y. Lin, V. V. Zhakhovsky, N. Pineau, J. H. Los, J. B. Maillet , L. Soulard, C. T. White and I. I. Oleynik, “Shock compression of diamond: Molecular dynamics simulations using different interatomic potentials”, AIP Conference Proceedings, 1426, 1175-1178 (2012) 481. B. J. Demaske, V. V. Zhakhovsky, N. A. Inogamov, C. T. White, and I. I. Oleynik, “MD simulations of laser- induced ultrashort shock waves in nickel”, AIP Conference Proceedings, 1426, 1163-1166 (2012) 482. B. J. Demaske, V. V. Zhakhovsky, C. T. White, and I. I. Oleynik, “A new nickel EAM potential for atomistic simulations of ablation, spallation and shock wave phenomena”, AIP Conference Proceedings, 1426, 1211- 1214 (2012) 483. B. J. Demaske, V. V. Zhakhovsky, C. T. White, and I. I. Oleynik, “Evolution of metastable elastic shockwaves in nickel”, AIP Conference Proceedings, 1426, 1303-1306 (2012) 484. V. V. Zhakhovsky, S.A. Pikuz, T. A. Shelkovenko, P.F. Knapp, C. C. Saylor, D.A. Hammer, “Cavitation and formation of foam-like structures inside exploding wires”, AIP Conference Proceedings, 1426, 1207-1210 (2012) 485. V. V. Zhakhovsky, B. J. Demaske, N. A. Inogamov, V. A. Khokhlov, S. I. Ashitkov et al., “Super-elastic response of metals to laser-induced shock waves”, International High-Power Laser Ablation Conference (HPLA-2012), Santa Fe, 30 April–3 May 2012 486. S. I. Ashitkov, N. A. Inogamov, P. S. Komarov, V. V. Zhakhovsky, I. I. Oleynik et al., “Strength of metals in liquid and solid states at extremely high tension produced by femtosecond laser heating”, International High- Power Laser Ablation Conference(HPLA-2012), Santa Fe, 30 April–3 May 2012 487. N. A. Inogamov, Yu. V. Petrov, V. V. Zhakhovsky, V. A. Khokhlov, B. J. Demaske et al.., “Two-temperature thermodynamic and kinetic properties of transition metals irradiated by femtosecond lasers”, International High-Power Laser Ablation Conference (HPLA-2012), Santa Fe, 30 April–3 May 2012 488. N. A. Inogamov, S. I. Ashitkov, V. V. Zhakhovsky, Yu. N. Emirov, et al., “Freezing of nanocavities beneath the surface of metal film irradiated by femtosecond laser pulse”, 8th International Conference on Photo- Excited Processes and Applications (ICPEPA-8), Rochester USA, August 12-17, 2012 489. N. A. Inogamov, V. V. Zhakhovsky, M. Ishino, S. I. Ashitkov, A. Faenov, et al., “Surface nanostructuring produced by ultrashort infrared and X-ray laser pulses”, 8th International Conference on Photo-Excited Processes and Applications (ICPEPA-8), Rochester USA, August 12-17, 2012 490. B. J. Demaske, V. V. Zhakhovsky, I. I. Oleynik, N. A. Inogamov, S. I. Ashitkov et al., “Cold ablation of metals driven by hot electrons”, 8th International Conference on Photo-Excited Processes and Applications (ICPEPA-8), Rochester USA, August 12-17, 2012 491. N. A. Inogamov, V. V. Zhakhovsky, S. I. Ashitkov, Yu. N. Emirov, A. Ya. Faenov, et al., “Surface nano- deformations caused by ultrashort laser pulse”, 19th European Conference on Fracture (ECF19), Kazan, Russia, August 26-31, 2012 492. N. A. Inogamov, V. V. Zhakhovsky, S. I. Ashitkov, V. A. Khokhlov, B. J. Demaske, et al., “Ablation and spallation of metals by femtosecond laser pulse”, 19th European Conference on Fracture (ECF19), Kazan, Russia, August 26-31, 2012 493. N. A. Inogamov, V. V. Zhakhovsky, S. I. Ashitkov, et al., “Super-elastic and plastic shock waves generated by lasers”, 19th European Conference on Fracture (ECF19), Kazan, Russia, August 26-31, 2012 494. V. A. Khokhlov, N. A. Inogamov, V. V. Zhakhovsky, S. I. Anisimov, “The structure of the superelastic and plastic shock wave”, XXVII International Conference on Equations of State for Matter, Elbrus, Russia, March 1-6, 2012 495. S. I. Tkachenko, V. V. Zhakhovsky, T. A. Shelkovenko, S. A Pikuz, “Different scenarios of the core-corona formation during wire explosion”, XXVII International Conference on Equations of State for Matter, Elbrus, Russia, March 1-6, 2012

321 58. MCA93S013 496. Sinko, W., C. de Oliveira, L. Pierce, J.A. McCammon. Protecting High Energy Barriers: A New Equation to Regulate Boost Energy in Accelerated Molecular Dynamics Simulations. J. Chem. Theory Comput. 8, 17–23 (2012). 497. Ortiz-Sanchez, J.M., D. Bucher, L.C.T. Pierce, P.R.L. Markwick, J.A.McCammon. Exploring the Photophysical Properties of Molecular Systems using Excited State Accelerated Ab Initio Molecular Dynamics. J. Chem. Theory Comput 8, 7252-7261 (2012). 498. Pierce, L.C.T., R. Salomon-Ferrer, C.A.F. de Oliveira, J.A. McCammon, R.C. Walker. Routine access to millisecond timescale events with accelerated molecular dynamics. J. Chem. Theory Comp. 8, 2997−3002 (2012). 499. Lindert, S., P. Kekenes-Huskey, G. Huber, L. Pierce, J.A. McCammon. Dynamics and calcium association to the N-terminal regulatory domain of human cardiac Troponin C: A multi-scale computational study. J. Phys. Chem. B 116, 8449-8459 (2012). 500. Kekenes-Huskey, P.M., V.T. Metzger, B.J. Grant, McCammon, J.A. Calcium binding and allosteric signaling mechanisms for sarcoplasmic reticulum Ca2+ ATPase. Protein Science 21, 1429-1444 (2012). 501. Kekenes-Huskey, P.M., A. Gillette, J. Hake, J. A. McCammon. Finite Element Estimation of Protein-Ligand Association Rates with Post-Encounter Effects: Applications to Calcium binding in Troponin C and SERCA. Comp. Sci. Disc. 5, article 014015 (2012). 7. Wang, Y., D.E. Schlamadinger, J.E Kim, J.A. McCammon. Comparative Molecular Dynamics Simulations of the Antimicrobial Peptide CM15 in Model Lipid Bilayers. BBA – Biomembranes 1818, 1402-9 (2012). 502. Schlamadinger, D., Y. Wang, J.A. McCammon, J. Kim. Spectroscopic and Computational Study of Melittin, Cecropin A, and the Hybrid Peptide CM15. J. Phys. Chem. 116, 10600-11608 (2012). 59. MCA94P014 503. H. Gao, B. Zhang, S. G. Johnson, and G. Barbastathis, “Design of thin–film photonic metamaterial Lüneburg lens using analytical approach,” Optics Express, vol. 20, pp. 1617–1628, January 2012. 504. P. Bermel, M. Ghebrebrhan, M. Harradon, Y. X. Yeng, I. Celanovic, J. D. Joannopoulos, and M. Soljačić, “Tailoring photonic metamaterial resonances for thermal radiation,” Nanoscale Research Letters, vol. 6, p. 549, October 2011. 60. MCA94P018 505. Charlotte Christensen, Fabio Governato, Thomas Quinn, Alyson M. Brooks, David B. Fisher, Sijing Shen, Jacqueline McCleary, and James Wadsley. The effect of models of the interstellar media on the central mass distribution of galaxies. ArXiv e-prints, 2012. 506. Charlotte Christensen, Thomas Quinn, Fabio Governato, Adrienne Stilp, Sijing Shen, and James Wadsley. Implementing Molecular Hydrogen in Hydrodynamic Simulations of Galaxy Formation. MNRAS, 425(4):3058–3076, 2012. 507. V. P. Debattista, R. Roˇskar, T. R. Quinn, B. Moore, and J. Wadsley. The Origin of Disc Orientations in Forming Galaxies. MNRAS, 2013 in preparation. 508. V. P. Debattista, R. Roˇskar, M. Valluri, T. R. Quinn, B. Moore, and J. Wadsley. What’s Up in the Milky Way? The Orientation of the Disc Relative to the Triaxial Halo. MNRAS, 2013 submitted. 509. F. Governato, A. Zolotov, A. Pontzen, C. Christensen, S. H. Oh, A. M. Brooks, T. Quinn, S. Shen, and J. Wadsley. Cuspy no more: how outflows affect the central dark matter and baryon distribution in cold dark matter galaxies. MNRAS, 422:1231–1240, May 2012. 510. E. K. Hicks, R. I. Davies, M. A. Malkan, and A. Rivers. Identifying Seyfert AGN Fueling Mechanisms on Scales from 1 kpc to 10 pc. In American Astronomical Society Meeting Abstracts #219, volume 219 of American Astronomical Society Meeting Abstracts, page #243.35, January 2012. 511. F. Munshi, F. Governato, A. M. Brooks, C. Christensen, S. Shen, S. Loebman, B. Moster, T. Quinn, and J. Wadsley. Reproducing the Stellar Mass/Halo Mass Relation in Simulated LCDM Galaxies: Theory vs Observational Estimates. ArXiv e-prints, September 2012. 512. Thomas R. Quinn, Pritish Jetley, Laxmikant V. Kale, and Filippo Gioachin. N –body Simulations with ChaNGa. In Laxmikant V. Kale and Bhatele Abhinav, editors, Parallel Science and Engineering Applications: The Charm++ Approach. Taylor & Francis, 2013 in press. 513. R. Roˇskar, V. P. Debattista, T. R. Quinn, and J. Wadsley. Radial migration in disc galaxies - I. Transient spiral structure and dynamics. MNRAS, 426:2089–2106, November 2012. 514. M. Valluri, V. P. Debattista, T. R. Quinn, R. Roˇskar, and J. Wadsley. Probing the shape and history of the Milky Way halo with orbital spectral analysis. MNRAS, 419:1951–1969, January 2012.

322 61. MCA94P030 515. C.-H. Park, F. Giustino, C.D. Spataru, M.L. Cohen, and S.G. Louie, \Inelastic Carrier Lifetime in Bilayer Graphene," Appl. Phys. Lett. 100, 032106 (2012). 516. B.D. Malone and M.L. Cohen, \Prediction of a metastable phase of silicon in the Ibam structure," Phys. Rev. B 85, 024116 (2012). 517. O.V. Yazyev, E. Kioupakis, J.E. Moore, and S.G. Louie, \Quasiparticle Effects in the Bulk and Surface-state Bands of Bi2Se3 and Bi2Te3 Topological Insulators," Phys. Rev. B 85, 161101 (2012). 518. J. Noffsinger, E. Kioupakis, C.G. Van deWalle, S.G. Louie, and M.L. Cohen, \Phonon-assisted Optical Absorption in Silicon from First Principles," Phys. Rev. Lett. 108, 167402 (2012). 519. K. Liu, J. Deslippe, F. Xiao, R.B. Capaz, X. Hong, S. Aloni, A. Zettl, W. Wang, X. Bai, S.G. Louie, E. Wang, and F. Wang, \An Atlas of Carbon Nanotube Optical Transitions," Nature Nanotech. 7, 325 (2012). 520. J. Deslippe, G. Samsonidze, D.A. Strubbe, M. Jain, M.L. Cohen, and S.G. Louie, \BerkeleyGW: A Massively Parallel Computer Package for the Calculation of the Quasiparticle and Optical Properties of Materials and Nanostructures," Comp. Phys. Comm. 183, 1269 (2012). 521. K. Kim, S. Coh, L.Z. Tan, W. Regan, J.M. Yuk, E. Chatterjee, M.F. Crommie, M.L. Cohen, S.G. Louie, and A. Zettl, \Raman Spectroscopy Study of Rotated Double-layer Graphene: Misorientation-angle Dependence of Electronic Structure," Phys. Rev. Lett. 108, 246103 (2012). 522. M. Orlita, L.Z. Tan, M. Potemski, M. Sprinkle, C. Berger, W.A. de Heer, S.G. Louie, and G. Martinez, \Resonant Excitation of Graphene K-Phonon and Intra-Laudau-Level Excitons in Magneto-Optical Spectroscopy," Phys. Rev. Lett. 108, 247401 (2012). 523. S.K. Choi, M. Jain, and S.G. Louie, \Mechanism for Optical Initialization of Spin in Center in Diamond," Phys. Rev. B 86, 041202 (2012). 524. N. Alem, Q.M. Ramasse, C.R. Seabourne, O.V. Yazyev, K. Erickson, M.C. Sarahan, C. Kisielowski, A.J. Scott, S.G. Louie, and A. Zettl, \Subangstrom Edge Relaxations Probed by Electron Microscopy in Hexagonal Boron Nitride," Phys. Rev. Lett 109, 205502 (2012). 525. S. Choi, J. Deslippe, R.B. Capaz, S.G. Louie, \An Explicit Formula for Optical Oscillator Strength of Excitons in Semiconducting Single-Walled Carbon Nanotubes: Family Behavior," Nano Lett. 13 (1), pp. 54-58 (2012). 62. MCA98N017 526. Zhu, X., P.E. Lopes, and A.D. Mackerell, Jr., Recent Developments and Applications of the CHARMM force fields. Wiley interdisciplinary reviews. Computational molecular science, 2012. 2: p. 167-185. 527. Best, R.B., X. Zhu, J. Shim, P.E.M. Lopes, J. Mittal, M. Feig, and A.D. MacKerell Jr., Optimization of the additive CHARMM all-atom protein force field targeting improved sampling of the backbone φ, ψ and sidechain χ1 and χ2 dihedral angles. J. Chem. Theory and Comput., 2012. 8: p. 3257–3273. 528. Best, R.B., J. Mittal, M. Feig, and A.D. MacKerell, Jr., Inclusion of many-body effects in the additive CHARMM protein CMAP potential results in enhanced cooperativity of alpha-helix and beta-hairpin formation. Biophysical Journal, 2012. 103: p. 1045-51. 529. Hart, K., N. Foloppe, C.M. Baker, E.J. Denning, L. Nilsson, and A.D. Mackerell, Jr., Optimization of the CHARMM additive force field for DNA: Improved treatment of the BI/BII conformational equilibrium. Journal of Chemical Theory and Computation, 2012. 8: p. 348-362. 530. Mallajosyula, S.S., O. Guvench, E. Hatcher, and A.D. Mackerell, Jr., CHARMM Additive All-Atom Force Field for Phosphate and Sulfate Linked to Carbohydrates. Journal of Chemical Theory and Computation, 2012. 8: p. 759-776. 531. Yu, W., X. He, K. Vanommeslaeghe, and A.D. MacKerell, Jr., Extension of the CHARMM General Force Field to sulfonyl-containing compounds and its utility in biomolecular simulations. Journal of Computational Chemistry, 2012. 33: p. 2451-68. 532. Vanommeslaeghe, K. and A.D. Mackerell, Jr., Automation of the CHARMM General Force Field (CGenFF) I: Bond Perception and Atom Typing. Journal of Chemical Information and Modeling, 2012. 52: p. 3144-54. 533. Vanommeslaeghe, K., E.P. Raman, and A.D. Mackerell, Jr., Automation of the CHARMM General Force Field (CGenFF) II: Assignment of Bonded Parameters and Partial Atomic Charges. Journal of Chemical Information and Modeling, 2012. 52: p. 3155-68. 534. Yu, W., P.E.M. Lopes, B. Roux, and J. MacKerell, A.D., Six-site Polarizable Model of Water Based on the Classical Drude Oscillator. Journal of Chemical Physics, 2013. In Press. 535. Luo, Y., W. Jiang, H. Yu, J. MacKerell, A.D., and B. Roux, Simulation study of ion pairing in concentrated aqueous salt solutions with a polarizable force field. Faraday Discussions, 2013. In Press. 536. Zhu, X., P.E. Lopes, J. Shim, and A.D. MacKerell, Jr., Intrinsic energy landscapes of amino acid side-chains. Journal of Chemical Information and Modeling, 2012. 52: p. 1559-72.

323 537. Shim, J., X. Zhu, R.B. Best, and A.D. Mackerell, Jr., (Ala)(4) -X-(Ala)(4) as a model system for the optimization of the chi(1) and chi(2) amino acid side-chain dihedral empirical force field parameters. Journal of Computational Chemistry, 2012. 538. Denning, E.J. and A.D. MacKerell, Jr., Intrinsic contribution of the 2'-hydroxyl to RNA conformational heterogeneity. Journal of the American Chemical Society, 2012. 134: p. 2800-6. 539. Jain, V., B. Hilton, B. Lin, S. Patnaik, F. Liang, E. Darian, Y. Zou, A.D. Mackerell, Jr., and B.P. Cho, Unusual sequence effects on nucleotide excision repair of arylamine lesions: DNA bending/distortion as a primary recognition factor. Nucleic Acids Research, 2012. 540. Maiti, A., M.S. Noon, A.D. MacKerell, Jr., E. Pozharski, and A.C. Drohat, Lesion processing by a repair enzyme is severely curtailed by residues needed to prevent aberrant activity on undamaged DNA. Proceedings of the National Academy of Sciences of the United States of America, 2012. 109: p. 8091-6. 541. Underwood, K.F., D.R. D'Souza, M. Mochin-Peters, A.D. Pierce, S. Kommineni, M. Choe, J. Bennett, A. Gnatt, B. Habtemariam, A.D. MacKerell, Jr., and A. Passaniti, Regulation of RUNX2 transcription factor- DNA interactions and cell proliferation by vitamin D3 (cholecalciferol) prohormone activity. Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research, 2012. 27: p. 913-25. 542. Petty, A., E. Myshkin, H. Qin, H. Guo, H. Miao, G.P. Tochtrop, J.T. Hsieh, P. Page, L. Liu, D.J. Lindner, C. Acharya, A.D. MacKerell, Jr., E. Ficker, J. Song, and B. Wang, A small molecule agonist of EphA2 receptor tyrosine kinase inhibits tumor cell migration in vitro and prostate cancer metastasis in vivo. PLoS ONE, 2012. 7: p. e42120. 543. McKnight, L.E., E.P. Raman, P. Bezawada, S. Kudrimoti, P.T. Wilder, K.G. Hartman, R. Godoy-Ruiz, E.A. Toth, A. Coop, A.D. Mackerell, Jr., and D.J. Weber, Structure-Based Discovery of a Novel Pentamidine- Related Inhibitor of the Calcium-Binding Protein S100B. ACS medicinal chemistry letters, 2012. 3: p. 975- 979. 544. Yap, J.L., X. Cao, K. Vanommeslaeghe, K.Y. Jung, C. Peddaboina, P.T. Wilder, A. Nan, A.D. MacKerell, Jr., W.R. Smythe, and S. Fletcher, Relaxation of the rigid backbone of an oligoamide-foldamer-based alpha-helix mimetic: identification of potent Bcl-xL inhibitors. Organic & biomolecular chemistry, 2012. 10: p. 2928-33. 545. Velvadapu, V., I. Glassford, M. Lee, T. Paul, C. Debrosse, D. Klepacki, M.C. Small, A.D. Mackerell, Jr., and R.B. Andrade, Desmethyl Macrolides: Synthesis and Evaluation of 4,10-Didesmethyl Telithromycin. ACS medicinal chemistry letters, 2012. 3: p. 211-215. 546. Stavitskaya, L., J. Shim, J.R. Healy, R.R. Matsumoto, A.D. MacKerell, Jr., and A. Coop, Deconstructing 14- phenylpropyloxymetopon: minimal requirements for binding to mu opioid receptors. Bioorganic & medicinal chemistry, 2012. 20: p. 4556-63. 547. Gonzalez, P.M., N. Hussainzada, P.W. Swaan, A.D. Mackerell, Jr., and J.E. Polli, Putative irreversible inhibitors of the human sodium-dependent bile acid transporter (hASBT; SLC10A2) support the role of transmembrane domain 7 in substrate binding/translocation. Pharmaceutical Research, 2012. 29: p. 1821-31. 548. Raman, E.P., K. Vanommeslaeghe, and A.D. Mackerell, Jr., Site-Specific Fragment Identification Guided by Single-Step Free Energy Perturbation Calculations. Journal of Chemical Theory and Computation, 2012. 8: p. 3513-3525. 549. Foster, T.J., A.D. MacKerell, Jr., and O. Guvench, Balancing target flexibility and target denaturation in computational fragment-based inhibitor discovery. Journal of Computational Chemistry, 2012. 33: p. 1880-91. 63. MCA99S022 550. Donzis, D.A., Sreenivasan, K.R. and Yeung, P.K. (2012) Some results on the Reynolds number scaling of pressure statistics in isotropic turbulence. Physica D, 241, 164-168. 551. Yeung, P.K., Donzis, D.A. and Sreenivasan, K.R. (2012) Dissipation, enstrophy and pressure statistics in turbulence simulations at high Reynolds numbers. J. Fluid Mech., 700, 5-15 [Focus on Fluids coverage, June 2012] 552. Gaither, K.P., Childs, H., Schulz, K.W., Harrison, C., Barth, W., Donzis, D.A. and Yeung, P.K. (2012) Visual analytics for finding critical structures in massive time-varying turbulent-flow simulations. IEEE Computer Graphics & Applications, 32(4), 34-45. 553. Verma, S., Yeung, P.K. and Blanquart, G. (2012) Scalar Filtered – Direct Numerical Simulation (SF-DNS) technique for turbulent transport of high Schmidt number passive scalars. Submitted to Phys. Fluids (in revision). 554. Sawford, B.L, Pope, S.B. and Yeung, P.K. (2012) Gaussian Lagrangian stochastic models for multi-particle dispersion. Submitted to Phys. Fluids (in review). 555. Sawford, B.L. and Yeung, P.K. (2012) Turbulent Lagrangian velocity statisyics conditioned on extreme values of dissipation. Submitted to Physica D (in review).

324 556. Gotoh, T. and Yeung, P.K (2013) “Passive scalar transport: a computational perspective”. in Ten Chapters in Turbulence review volume, Eds. P.A. Davidson, Y. Kaneda & K.R. Sreenivasan, Cambridge University Press, UK. 557. Czechowski, K., McClanahan, C., Battaglino, C., Iyer, K.P., Yeung, P.K. and Vuduc, R. (2012) On the communication complexity of 3D FFTs and its implications for exascale. Proceedings of the ACM International Conference on Supercomputing (ICS), San Servolo Island, Venice, Italy, June 2012. doi:10.1145/2312005.2312039. 558. Yeung, P.K. (2011) Petascale computing for studies of turbulent mixing and dispersion: current status and future prospects. Discussion Meeting on “Defining Guidelines for Future Extreme Simulations of Three- dimensional Fluid and Magnetohydrdynamic Turbulence”, Indian Institute of Science, Bangalore, India (December 2011). 559. Sawford, B.L., Yeung, P.K. & Hackl, J.F. (2012) Turbulent Lagrangian velocity statistics conditioned on extreme values of dissipation. IUTAM Symposium on Understanding Common Aspects of Extreme Events in Fluids, University College Dublin, Ireland (July 2012). 560. Yeung, P.K. (2012) Petacale computing and similarity scaling in turbulence. CFD Futures Conference, National Institute of Aerospace, Hampton, VA (August 2012). 561. American Physical Society, 62nd Fluid Dynamics Meeting (Nov 2011): (See Bull. Am. Phys. Soc. 56(18).) 562. American Physical Society, 63rd Fluid Dynamics Meeting (Nov 2012): (See Bull. Am. Phys. Soc. 57(17).) 563. Buaria, D., Yeung, P.K. and Domaradzki, J.A. (2012) Spectral transfer and scale locality characteristics in turbulent mixing over a wide range of Schmidt numbers. 564. Sreenivasan, K.R. and Yeung, P.K. (2012) Turbulent mixing of substances which are highly diffusive. 565. Yeung, P.K., Iyer, K.P. and Sreenivasan, K.R. (2012) Turbulent mixing of highly diffusive substances in the presence of a uniform mean gradient. 566. At Department of Mechanical and Civil Engineering, California Institute of Technology (Pasadena, CA), March 2012: on “Cyber-enabled investigations of turbulence: flow physics and Petascale algorithms”. 567. At Department of Physics, Chinese University of Hong Kong (Hong Kong), May 2012: on “Reaching Higher in Turbulence Simulations: Large-scale Computing and Future Strategies”. 568. At Department of Physics, Chinese University of Hong Kong (Hong Kong), May 2012: on “The Physics of Mixing and Dispersion in Turbulence: A Computational Perspective”. 569. At Center for Computational Science, Nagoya University (Nagoya, Japan), June 2012: on “Petascale computing and studies of turbulence and turbulent mixing”. 570. At Department of Mechanical Engineering, University of Hong Kong (Hong Kong), June 2012: on “Cyber- enabled Studies of Turbulence and Turbulent Dispersion”. 64. MCB040002 571. Pan, JJ, Chen, Q, Willenbring, D, Yoshida, K, Tillman, T, Kashlan, OB, Cohen, A, Kong, XP, Xu, Y, and Tang, P, Structure of ELIC Co-crystallized with Its Competitive Antagonist Acetylcholine, Nature Communications, DOI: 10.1038/ncomms1703, 714, (2012). PMCID: PMC3316889 572. Pan, JJ, Chen, Q, Willenbring, D, Mowrey, Kong, XP, Cohen, A, Divito, CB, Xu, Y, and Tang, P, Structure of the Pentameric Ligand-Gated Ion Channel GLIC Bound With Anesthetic Ketamine, Structure, 20, 1463-69, (2012) NIHMSID # 403148; Publ.ID: STFODE2512 573. Cui, T, Mowrey, D, Bondarenko, V, Tillman, T, Ma, D, Landrum, E, Perez-Aguilar, JM, He, J, Wang, W, Saven, JG, Eckenhoff, R, Tang, P, Xu, Y, NMR structure and dynamics of water-solubilized transmembrane domains of nicotinic acetylcholine receptor, BBA- Biomembranes, 1818, 617-26, (2012) PMID: 22155685; PMCID: PMC3273582 574. Bondarenko, V, Mowrey, D, Tillman, T, Cui, T, Liu, LT, Xu, Y, Tang, P, NMR Structures of the Transmembrane Domains of the α4β2 nAChR, BBA- Biomembranes, 1818, 1261-1268, (2012). PMID: 22361591; PMCID: PMC3319252 575. Bondarenko, V, Mowrey, D, Liu, LT, Xu, Y, Tang, P, NMR resolved multiple anesthetic binding sites in the TM domains of the α4β2 nAChR, BBA- Biomembranes, 1828(2), 398-404, (2013). 576. Tillman, T, Cheng, MH, Chen, Q, Tang, P, Xu, Y, Reversal of Ion Charge Selectivity Renders the Pentameric Ligand-Gated Ion Channel GLIC Insensitive to Anesthetics, Biochem, J, Dec 07, 449 (part 1), 61-68 (2012). 577. Jianjun Pan, Qiang Chen, Dan Willenbring, Ken Yoshida, Tommy Tillman, Ossama B. Kashlan, Aina Cohen, Xiang-Peng Kong, Yan Xu, and Pei Tang, Structure and function of ELIC bound with the antagonist acetylcholine, Biophysical Society 56th annual meeting, February 25-29, San Diego, CA, (2012)

325 65. MCB060006 578. Dynamic factors controlling targeting nanocarriers to vascular endothelium, D. M. Eckmann, R. Radhakrishnan, V. Muzykantov, R. Radhakrishnan, D. M. Eckmann, Current Drug Metabolism, 2012, 113, 70. Pubmed ID: 22292809. 579. Mesoscale Modeling and Simulations of Spatial Partitioning of Curvature Inducing Proteins under the Influence of Mean Curvature Fields in Bilayer Membranes, J. Liu, R. Tourdot, V. Ramanan, N. J. Agrawal, R. Radhakrishnan, Molecular Physics, 2012, 110(11–12), 1127–1137. (DOI:10.1080/00268976.2012.664661) 580. A hybrid formalism combining fluctuating hydrodynamics and generalized Langevin dynamics for the simulation of nanoparticle thermal motion in an incompressible fluid medium, B. Uma, D. M. Eckmann, P. S. Ayyaswamy, R. Radhakrishnan, Molecular Physics, 2012, 110(11–12), 1057–1067. (DOI: 10.1080/00268976.2012.663510) 581. Investigating Molecular Mechanisms of Specificity in Regulation of the HER2 Receptor Tyrosine Kinase through Molecular Modeling and Simulation, S. E. Telesco, A. Shih, Y. Liu, R. Radhakrishnan, International Journal of Cancer Research and Prevention (2012), ISSN: 1554-1134, Volume 4, Number 3, 2012. 582. Structural Systems Biology and Multiscale Signaling Models, S. E. Telesco, R. Radhakrishnan, Annals of Biomedical Engineering, 2012, 40(11), 2295-2306. Pubmed ID: 22539148. 583. Roles for Thermodynamics and Catalysis in Systems Biology: Case and Point, R. Radhakrishnan, Editorial in Journal of Thermodynamics and Catalysis, OMICS Publishing Group, 2012, 3:e102. doi:10.4172/2157- 7544.1000e102. 584. Temporal multiscale approach for nanocarrier motion with simultaneous adhesion and hydrodynamic interactions in targeted drug delivery, R. Radhakrishnan, B. Uma, J. Liu, P. S. Ayyaswamy, D. M. Eckmann, J. Computational Physics, 2012, in press. http://dx.doi.org/10.1016/j.jcp.2012.10.026. 585. Erlotinib binds both inactive and active conformations of the EGFR tyrosine kinase domain, J. H. Park1, Y. Liu1, M. A. Lemmon*, R. Radhakrishnan*, Biochem J., 2012, 448(3), 417-423; 1. Equal contribution, * co- corresponding authors. 586. Hybrid scheme combining fluctuating hydrodynamics and generalized Langevin dynamics for nearly neutrally buoyant particle in stationary medium, B. Uma, D. M. Eckmann, R. Radhakrishnan, P. S. Ayyaswamy, ASME Journal of Heat Transfer, 2013, 135, 011011. 587. A molecular docking study of the binding of ruthenium complex compounds to PIM1, GSK3, and CDK2 protein kinases, Y. Liu, N. J. Agrawal, R. Radhakrishnan, J. Molecular Modeling, 2013, 19, 371-382. DOI 10.1007/s00894-012-1555-4. 588. Nanocarrier-cell surface adhesive and hydrodynamic interactions: ligand-receptor bond sensitivity study, B. Uma, R. Radhakrishnan, D. M. Eckmann, P. S. Ayyaswamy, ASME Journal of Nanotechnology for Medicine and Engineering, 2012, in press. 589. Nanocarrier Hydrodynamics and Binding in Targeted Drug Delivery: Challenges in Numerical Modeling and Experimental Validation, P. S. Ayyaswamy, V. Muzykantov, D. M. Eckmann, R. Radhakrishnan, J. Nanotechnology for Engineering and Medicine, 2013, in press. 590. Modeling of a nanoparticle motion in a Newtonian fluid: A comparison between fluctuating hydrodynamics and generalized Langevin methods, B.Uma, R.Radhakrishnan, D.M.Eckmann and P.S.Ayyaswamy, Proceedings of the ASME 3rd Micro/Nanoscale Heat and Mass Transfer International Conference, Paper No. MNHMT2012-75019, 2012, in press. 591. Fluctuating hydrodynamics approach for the simulation of nanoparticle Brownian motion in a Newtonian fluid: buoyancy effects, B.Uma, R.Radhakrishnan, D.M.Eckmann and P.S.Ayyaswamy, Proceedings of the 21st National and 10th ISHMT-ASME Heat and Mass Transfer conference, 2012, in press. 592. 3. Computational Methodology for Mechanistic Profiling of Kinase Domain Mutations in Cancers, P. J. Huwe, and R. Radhakrishnan, Proceedings of the IEEE, 5th International Advanced Research Workshop on In Silico Oncology and Cancer Investigation, 2013, in press. An open access version of the article is also available at http://www.5th-iarwisoci.iccs.ntua.gr/ 593. Top-down Mesoscale Models and Free Energy Calculations of Multivalent Protein-Protein and Protein- Membrane Interactions in Nanocarrier Adhesion and Receptor Trafficking, J. Liu, N. J. Agrawal, D. M. Eckmann, P. S. Ayyaswamy, R. Radhakrishnan, Innovations in Biomolecular Modeling, Royal Society of Chemistry Publishing, 2012, chapter 11, pp272-287. ISBN-10: 1849734100; ISBN-13: 978-1849734103 66. MCB070038 594. Catanese DJ Jr, Matthews KS. Disconnected Interacting Protein 1 binds with high affinity to pre-tRNA and ADAT. (2011-10-28 Epub: 2011-09-24) Biochem Biophys Res Commun 414:506-11 595. Winkler DD, Muthurajan UM, Hieb AR, Luger K. Histone chaperone FACT coordinates nucleosome interaction through multiple synergistic binding events. (2011-12-02 Epub: 2011-10-03) J Biol Chem 286:41883-92

326 596. Brookes EH, Singh R, Pierce M, Marru S, Demeler B, Rocco M. Ultrascan solution modeler: integrated hydrodynamic parameter and small angle scattering computation and fitting tools. (2012) ACM XSEDE 12 1:1 597. Rocco, M, Brookes, E.H., Byron, O. US-SOMO: Methods for the Construction and Hydration of Macromolecular Hydrodynamic Models. G.C.K. Roberts (ed.), Encyclopedia of Biophysics, DOI 10.1007/978-3-642-16712-6, Springer-Verlag Berlin Heidelberg 2012 598. Gebendorfer KM, Drazic A, Le Y, Gundlach J, Bepperling A, Kastenmüller A, Ganzinger KA, Braun N, Franzmann TM, Winter J. Identification of a hypochlorite-specific transcription factor from Escherichia coli. (2012-02-24 Epub: 2012-01-04) J Biol Chem 287:6892-903 599. Robinson AK, Leal BZ, Chadwell LV, Wang R, Ilangovan U, Kaur Y, Junco SE, Schirf V, Osmulski PA, Gaczynska M, Hinck AP, Demeler B, McEwen DG, Kim CA. The growth-suppressive function of the polycomb group protein polyhomeotic is mediated by polymerization of its sterile alpha motif (SAM) domain. (2012-03-16 Epub: 2012-01-24) J Biol Chem 287:8702-13 600. Gabrielsen M, Beckham KS, Cogdell RJ, Byron O, Roe AJ. FolX from Pseudomonas aeruginosa is octameric in both crystal and solution. FEBS Lett: 2012-4-24; 586(8); 1160-5 601. Zheng J, Gay DC, Demeler B, White MA, Keatinge-Clay AT. Divergence of multimodular polyketide synthases revealed by a didomain structure. (2012-07-01 Epub: 2012-05-27) Nat Chem Biol 8:615-21 602. Robinson AK, Leal BZ, Nanyes DR, Kaur Y, Ilangovan U, Schirf V, Hinck AP, Demeler B, Kim CA. Human polyhomeotic homolog 3 (PHC3) sterile alpha motif (SAM) linker allows open-ended polymerization of PHC3 SAM. (2012-07-10 Epub: 2012-06-28) Biochemistry 51:5379-86 603. Harkness KM, Tang Y, Dass A, Pan J, Kothalawala N, Reddy VJ, Cliffel DE, Demeler B, Stellacci F, Bakr OM, McLean JA. Ag44(SR)30(4-): a silver-thiolate superatom complex. (2012-07-21 Epub: 2012-06-15) Nanoscale 4:4269-74 604. Le HT, Buscaglia R, Dean WL, Chaires JB, Trent JO. Calculation of Hydrodynamic Properties for G- Quadruplex Nucleic Acid Structures from in silico Bead Models. (2012-08-11 Epub: 2012-08-11) Top Curr Chem 605. Xue J, Gao Y, Hoorelbeke B, Kagiampakis I, Zhao B, Demeler B, Balzarini J, Liwang PJ. The role of individual carbohydrate-binding sites in the function of the potent anti-HIV lectin griffithsin. (2012-09-04 Epub: 2012-08-21) Mol Pharm 9:2613-25 606. Clark NJ, Kramer M, Muthurajan UM, Luger K. Alternative modes of binding of poly(ADP-ribose) polymerase 1 to free DNA and nucleosomes. (2012-09-21 Epub: 2012-07-31) J Biol Chem 287:32430-9 607. Krayukhina E, Uchiyama S, Fukui K. Effects of rotational speed on the hydrodynamic properties of pharmaceutical antibodies measured by analytical ultracentrifugation sedimentation velocity. (2012-09-29 Epub: 2012-06-21) Eur J Pharm Sci 47:367-74 608. Retzlaff M, Rohrberg J, Küpper NJ, Lagleder S, Bepperling A, Manzenrieder F, Peschek J, Kessler H, Buchner J. The Regulatory Domain Stabilizes the p53 Tetramer by Intersubunit Contacts with the DNA Binding Domain. (2013-01-09 Epub: 2012-10-24) J Mol Biol 425:144-55 609. Andrews BT, Catalano CE. The enzymology of a viral genome packaging motor is influenced by the assembly state of the motor subunits. (2012-11-20 Epub: 2012-11-07) Biochemistry 51:9342-53 610. Bepperling A, Alte F, Kriehuber T, Braun N, Weinkauf S, Groll M, Haslbeck M, Buchner J. Alternative bacterial two-component small heat shock protein systems. (2012-12-11 Epub: 2012-11-26) Proc Natl Acad Sci U S A 109:20407-12 611. Le HT, Miller MC, Buscaglia R, Dean WL, Holt PA, Chaires JB, Trent JO. Not all G-quadruplexes are created equally: an investigation of the structural polymorphism of the c-Myc G-quadruplex-forming sequence and its interaction with the porphyrin TMPyP4. (2012-12-21 Epub: 2012-10-29) Org Biomol Chem 10:9393-404 612. Fisher OS, Zhang R, Li X, Murphy JW, Demeler B, Boggon TJ. Structural studies of cerebral cavernous malformations 2 (CCM2) reveal a folded helical domain at its C-terminus. (2012-12-22 Epub: 2012-12-22) FEBS Letters 613. Brookes, E., Singh, R., Pierce, M., Marru, S., Demeler, B, Rocco, M. UltraScan Solution Modeler: Integrated Hydrodynamic Parameter and Small Angle Scattering Computation and Fitting Tools, XSEDE 12 Conference, Chicago, 7/2012 614. Brookes, E. H. and Rocco, M. UltraScan-SOMO, A2B: Bead modeling, SAXS/SANS and dynamics. AUC 2012. 3/2012. San Antonio, Texas. 615. Brookes, E. , Demeler, B., Vachette, P., Perez, J., Ding, F., Dokholyan, N.V., Rocco, M. UltraScan SOlution MOdeler (US-SOMO): Multiple model multiple experiment simulation and fitting, AUC 12 Symposium, San Antonio, 3/2012. 616. Demeler B. UltraScan software data analysis of sedimentation velocity experiments using the XSEDE USLIMS-3 science gateway. An introductory workshop. AUC 2012. 3/2012. San Antonio, Texas. 617. Demeler B. UltraScan software data analysis of sedimentation velocity experiments using the XSEDE USLIMS-3 science gateway. Advanced concepts. Workshop. AUC 2012. 3/2012. San Antonio, Texas.

327 618. Demeler B. Macromolecular Characterization in the Solution Environment with Analytical Ultracentrifugation. Departmental Seminar. The University of Düsseldorf, Germany. 9/2012 619. Brookes, E. H. UltraScan-SOMO Hydrodynamic and SAXS modeling and fitting, Harwell Research Complex, 10/2012. Oxfordshire, UK. 620. Brookes, E. H. UltraScan-SOMO Hydrodynamic and SAXS modeling and fitting, University College London, 10/2012. London, UK. 621. Brookes, E. , Singh, R., Pierce, M., Marru, S., Demeler, B, Rocco, M., UltraScan SOlution MOdeler (US- SOMO): Multiple model multiple experiment simulation and fitting, AusBioSAXS 12 Workshop, Australian Synchrotron, Melbourne, 11/2012. 622. Brookes, E. H. and Rocco, M. UltraScan-SOMO - Hydrodynamic Modeling, La Trobe University, 11/2012. Melbourne, Australia. 623. Brookes, E. Parsimonious Spatial Models from Small Angle Scattering of Biological Macromolecules, Small Angle Scattering Conference, Sydney, 11/2012 624. Demeler, B. Characterization of Kinetic Rates and Equilibrium Constants for Reversible Associations and Ligand binding using Sedimentation Velocity. EMBO Course. 10/2011, University of Oulu, Finland 625. Demeler, B. Solution Studies of Polymers and Macromolecules: Biophysical Methods with Applications in High-Performance Computing. UTHSCSA Departmental Seminar. 10/2011 626. Demeler, B. Study of CdTe nanoparticles with a multiwavelength detector for the analytical ultracentrifuge. Aesku, Germany 10/2011 67. MCB080013 627. Shim JY, Bertalovitz AC, and Kendall DA. Probing the Interaction of SR141716A with the CB1 Receptor. J Biol Chem. 2012, 287, 38741-38754. 628. Chen X, Zhang X, Lu Y, Shim JY, Sang S, Sun Z, and Chen X. Chemoprevention of 7,12- dimethylbenz[a]anthracene (DMBA)-induced hamster cheek pouch carcinogenesis by a 5-lipoxygenase inhibitor, garcinol. Nutr Cancer. 2012, 64, 1211-1218. 629. Shim JY, and Padgett P. Functional Residues Essential for the Activation of the CB1 Cannabinoid Receptor. Methods Enzymol. 2013, 520, 335-353. 630. Shim JY. A Molecular Dynamics (MD) Simulation Study on the Molecular Mechanism of Cannabinoid Receptor Activation. Keystone Symposium on G Protein-Coupled Receptors: Molecular Mechanisms and Novel Functional Insights, Banff, Alberta, Canada, 2012. 631. Shim JY, Bertalovitz AC, and Kendall DA. Probing the Interaction of HU210 and SR141716A with the CB1 Receptor. Carolina Cannabinoid Collaborative, Greenville, NC, USA, 2012. 68. MCB080071 632. D. V. Matyushov, Dipolar response of hydrated proteins, J. Chem. Phys. 136, 085102 (2012). 633. D. R. Martin and D. V. Matyushov, Solvent-renormalized dissipative electro-elastic network model of hydrated proteins, J. Chem. Phys. 137, 165101 (2012). 634. D. R. Martin and D. V. Matyushov, Non-Gaussian Statistics and Nanosecond Dynamics of Electrostatic Fluctuations Affecting Optical transitions in a Green Fluorescent Protein, J. Phys. Chem. B 116, 10294–10300 (2012). 635. D. V. Matyushov, On the theory of dielectric response of protein solutions, J. Phys.: Cond. Matter 24, 325105 (2012). 636. M. Heyden, D. J. Tobias, and D. V. Matyushov, Terahertz absorption by dilute aqueous solutions, J. Chem. Phys. 137, 235103 (2012). 637. D. R. Martin, S. B. Ozkan, and D. V. Matyushov, Dissipative electro-elastic network model of protein electrostatics, Phys. Biol. 9, 036004 (2012). 638. A. D. Friesen and D. V. Matyushov, Surface Polarity and Nanoscale Solvation, J. Phys. Chem. Lett. 3, 3685– 3689 (2012). 639. D. N. LeBard, D. R. Martin, S. Lin, and W. Woodbury, Natural selection of protein dynamics to optimize and control bacterial photosynthesis, Nat. Chem. , to be submitted (2013). 69. MCB090003 640. L. Fang, C. Chan, M. Feig: Palmitate interactions with PKR kinase. to be submitted. 641. L. Fang, C. Chan, M. Feig: Palmitate interactions with IRE1 kinase. to be submitted. 642. B. Wang, A. Predeus, Z. Burton, M. Feig: Trigger loop closing in RNA polymerase depends on correctly paired active site NTP. to be submitted

328 643. M. Sayadi, T. Mori, Y. Sugita, M. Feig: Extension of the HDGB implicit membrane model with an implicit van der Waals formalism. to be submitted 644. S. M. Gopal, P. Kar, Y. Cheng, A. Predeus, M. Feig: PRIMO: A physics-based transferable coarse-grained model for proteins. to be submitted 645. M. Sharma, A Predeus, S. Mukherjee, M. Feig: DNA Bending Propensity in the Presence of Base Mismatches: Implications for DNA Repair. to be submitted 646. A. Panahi, M. Feig: Dynamics Heterogeneous Dielectric Generalized Born (DHDGB): An implicit membrane model with a dynamically varying bilayer thickness. Under review at JCTC 647. V. Mirjalili, M. Feig: Protein structure refinement through structure selection and averaging from molecular dynamics ensembles. JCTC (2013), in press 648. R. B. Best, J. Mittal, M. Feig, A. D. MacKerell Jr.: Inclusion of many-body effects in the additive CHARMM - -hairpin formation. Biophysical Journal (2012), 103, 1045-1051 649. R. B. Best, X. Zhu, J. Shim, P. Lopes, J. Mittal, M. Feig, A. D. MacKerell Jr: Optimization of the additive CHARMM all-atom protein force field targeting improved sampling of the backbone ,  and sidechain 1 and 2 dihedral angles. Journal of Chemical Theory and Computation (2012), 8, 3257-3273 650. M. Sayadi, M. Feig: Computational study of Ser16 and Thr17 phosphorylated Phospholamban in implicit membrane. BBA Biomembranes (2013), 1828, 577-585 651. A. V. Predeus, S. Gul, S. M. Gopal, M. Feig: Conformational Sampling of Peptides in the Presence of Protein Crowders from AA/CG-Multiscale Simulations. Journal of Physical Chemistry B (2012), 116, 8610-8620 652. Y.-M. Cheng, S. M. Gopal, S. M. Law, M. Feig: Molecular dynamics trajectory compression with a coarse-grained model. IEEE/ACM Transactions on Computational Biology and Bioinformatics (2012) 116, 599-605 653. S. M. Law, M. Feig: Base-flipping mechanism in DNA mismatch recognition by MutS. Biophysical Journal (2011) 101, 2223-2231

70. MCB090174 654. Longitudinal analysis of integrase N155H variants in heavily treated patients failing raltegravir-based regimens. (H. l. Nguyen, C. Charpentier, N. Nguyen, P. de Truchis, J-M. Molina, K. Ruxrungtham, C. Delaugerre), In HIV Med, 2012. 655. Quantized Water Access to the HIV-1 Protease Active Site as a Proposed Mechanism for Cooperative Mutations in Drug Affnity. (Benjamin A Hall, David W Wright, Shantenu Jha, Peter V Coveney), In Biochemistry, volume 51, pp. 6487-6489, 2012. 656. From base pair to bedside: molecular simulation and the translation of genomics to personalized medicine. (David W Wright, Shunzhou Wan, Nour Shublaq, Stefan J Zasada, Peter V Coveney), In Wiley Interdiscip Rev Syst Biol Med, 2012. 657. Thumbs Down for HIV: Domain Level Rearrangements Do Occur in the NNRTI-Bound HIV-1 Reverse Transcriptase. (David W Wright, S. Kashif Sadiq, Gianni De Fabritiis, Peter V Coveney), In J Am Chem Soc, 2012. 658. HIV-1 reverse transcriptase complex with DNA and nevirapine reveals non-nucleoside inhibition mechanism. (Kalyan Das, Sergio E Martinez, Joseph D Bauman, Eddy Arnold), In Nat Struct Mol Biol, volume 19, pp. 253-259, 2012. 659. Homology model-guided 3D-QSAR studies of HIV-1 inte-grase inhibitors. (H. Sharma, X. Cheng, J. K. Buolamwini), In J Chem Inf Model, volume 52, pp. 515-544, 2012. 660. Response of the Atlantic Ocean circulation to Greenland Ice Sheet melting in a strongly-eddying ocean model (W. M. E. Maltrud M. W. Hecht H. A. Dijkstra Weijer, M. A. Kliphuis), In Geophysical Research Letters, 2012. 661. Stability of Free and Mineral-Protected Nucleic Acids: Implications for the RNA World (J. B. Swadling, P. V. Coveney, H.C. Greenwell), In Geochimica et Cosmochimica Acta, Elsevier, volume 1, pp. 360-378, 2012. 662. The inhibitory effects of vinylphosphonate-linked thymidine dimers on the unidirectional translocation of PcrA helicase along DNA: a molecular modelling study (H Laughton CAWang), In Phys. Chem. Chem. Phys., 2012. 663. Drug-perturbed Essential Dynamics/Molecular Dynamics (EDMD) to Virtual Screening and Rational Drug Design. (R. Carrillo O Laughton C. Orozco M. Chaudhuri), In J. Chem. Theory Comput, 2012. 664. Fast Atomistic Molecular Dynamics Simulations from Essential Dynamics Samplings (O Laughton CA Orozco M Carrillo), In J. Chem. Theory Comput, 2012. 665. Mapping application performance to HPC architecture (A Bethune I Kenway R Smith L Guest M Kitchen C Calleja P Korzynski A Rankin S Ashworth M Porterd A Todorov I Plummer M Jones E Steenman-Clark L Ralton B Laughton C. Gray), In Computer Physics Communication, 2012.

329 666. The effects of a telomere destabilizing agent on cancer cell-cycle dynamicsIntegrated modelling and experiments (BVWattis JAD Preston SP Laughton CA Hirt), In J. Theor. Biol., 2012. 667. Dynamic Application Runtime Environment (DARE): A Standards-based Middleware Framework For Building Science Gateways (Joohyun Kim Yaakoub el-Khamra Sharath Maddineni, Shantenu Jha), In , 2012. (Journal of Grid Computing (in press)) 668. Conformational Heterogeneity of the SAM-I Riboswitch Transcriptional ON State: A Chaperone-Like Role for S-Adenosyl Methionine (Wei Huang, Joohyun Kim, Shantenu Jha, Fareed Aboulela), In Journal of Molecular Biology, volume 418, pp. 331 - 349, 2012. (http://www.sciencedirect.com/science/article/pii/S0022283612001763) 669. Mechanism of Drug Efficacy within the Epidermal Growth Factor Receptor Revealed by Microsecond Molecular Dy-namics Simulation (S. Wan, D. W. Wright, P. V. Coveney), In Molecular Cancer Therapeutics, 2012. 670. Distributed Computing Practice for Large-Scale Science and Engineering Applications (Shantenu Jha, Murray Cole, Daniel S. Katz, Manish Parashar, Omer Rana, JonWeissman), In , 2012. (Computing and Concurrency: Practise and Experience (in press)) 671. Pilot-Abstractions for Data-Intensive Cloud Applications (Andre Luckow, Pradeep Mantha, Melissa Romanus, Shantenu Jha), In , 2012. (Computing and Concurrency: Practise and Experience (in press)) 672. Theory, modelling and simulation in origins of life studies (P.V. Coveney, J.B. Swadling, J.A.D. Wattis, H.C. Greenwell), In Chemical Society Reviews, Royal Society of Chemistry, 2012. 673. Advancing Next-Generation Sequencing Data Analytics with Scalable Distributed Infrastructure (Joohyun Kim Yaakoub el-Khamra Sharath Maddineni, Shantenu Jha), In , 2012. (Concurrency and Computation: Practise and Experience (in press)) 674. Quantized Water Access to the HIV-1 Protease Active Site as a Proposed Mechanism for Cooperative Mutations in Drug Affnity (Benjamin A. Hall, David W. Wright, Shantenu Jha, Peter V. Coveney), In Biochemistry, volume 51, pp. 6487-6489, 2012. 675. Global Conformational Dynamics of HIV-1 Reverse Transcriptase Bound to Non-Nucleoside Inhibitors (David W.Wright, Benjamin A. Hall, Paul Kellam, Peter V. Coveney), In Biology, volume 1, pp. 222-244, 2012. 71. MCB110056 676. J. Hsin, A. Gopinathan, and K. C. Huang. Nucleotide-dependent conformations of FtsZ dimmers and force generation observed through molecular dynamics simulations Proc. Natl. Acad. Sci. USA, 109:9432{9437, 2012. 677. J. G. Cueva, J. Hsin, K. C. Huang, and M. B. Goodman. Posttranslational acetylation of α-tubulin constrains protofilament number in native microtubules. Curr. Biol., 22:1066{1074, 2012. 678. Y. Li and J. Hsin and L. Zhao and Y. Cheng and K. C. Huang and H.-W. Wang and S. Ye. A curved FtsZ- GDP protofilament structure reveals a hinge-opening mechanism for constrictive force generation. Submitted, 2012. 72. MCB120012 679. Shenoy S., Nanda H, Loesche, M. Membrane Association of the PTEN Tumor Suppressor: Electrostatic Interactions with Phosphatidylserine-Containing Bilayers and Regulatory Role of the C-Terminal Tail. Journal of Structural Biology. v180 (2012) 394-408 Recently highlighted in: Nature Chemical Biology v9 (2013) 9 680. Shenoy S., Lösche, M., Nanda H. Conformational sampling of the PTEN disordered C-terminal tail and energetics of binding to the C2 core domain. in preparation 681. “An Issue of Life and Death: Membrane Interaction of the PTEN Tumor Suppressor” Arizona State Univ., Center for Biological Physics. Oct 2012 682. “Orientation and Structure of Membrane Proteins Studied by Neutron Reflectivity and Molecular Dynamics” National Institute of Health, Laboratory of Computational, Rockville, MD, June 2012 683. “A New Window into Protein-Membrane Interactions: What Can We Hope to Learn About Gag Assembly on Membranes from Scattering and MD Simulation Studies?” National Institute of Health, Bethesda, MD. June 2012 684. “An Issue of Life and Death: Membrane Interaction of the PTEN Tumor Suppressor” American Society of Neutron Scattering, Georgetown, DC. June 2012 685. “High-resolution Structures of Disordered Systems: Membrane Association of the PTEN Tumor Suppressor by Neutron Reflection and MD Simulation” American Chemical Society Colloids and Surfaces Symposium, Baltimore, MD. June 2012

330 73. MCB120014 686. A. Bhattacharya, R. B. Best, and J. Mittal, “Smoothing of the GB1 hairpin folding landscape by interfacial confinement”, Biophys. J. 103, 596 (2012). 687. R. B. Best, J. Mittal, M. Feig, and A.D. MacKerell, “Inclusion of many-body effects in the additive CHARMM protein CMAP potential results in enhanced cooperativity of alpha-helix and β-hairpin formation” Biophys. J. 103, 1045 (2012). 688. A. Bhattacharya, Y. C. Kim, and J. Mittal, “Protein-protein interactions in a crowded environment”, Biophysical Reviews (submitted). 689. A. Bhattacharya, Y. C. Kim, and J. Mittal, “Effects of macromolecular crowding on binding-induced folding and binding affinity” (in preparation). 690. Y. C. Kim and J. Mittal, “Quantitative theory of crowding induced entropy-enthalpy compensation in protein association” Phys. Rev. Lett. (under review). 74. MCB120050 691. Fleming PJ, Freites JA, Moon CP, Tobias DJ and KG Fleming (2012) “Outer membrane phospholipase A in phospholipid bilayers: A model system for concerted computational and experimental investigations of amino acid side chain partitioning into lipid bilayers” Biochim Biophys Acta. 1818(2):126-34. 692. Moon CP, Zaccai NR, Fleming PJ, Gessmann D, and KG Fleming (2013) “Membrane protein thermodynamic stability may serve as the energy sink for sorting in the periplasm” In review. 75. MCB120051 693. Echeverria, I., Papoian, G., Exploring the homeodomain sliding along DNA in molecular detail. In preparation 76. MCB120052 694. AMAP: introducing phi/psi coupling for implicit simulations of proteins. A. Perez, JL MacCallum, Ken A. Dill. In prep 695. Extracting physically realistic average structures from protein conformational ensembles using nonlinear averaging. JL MacCallum, A. Wagaman, A. Perez, Ken A. Dill. In prep 696. Improving REMD efficiency in biomolecular systems. JL. MacCallum, S. Presse, Ken A. Dill. In prep 697. MeLD: Modeling with Limited Data—a physics-based integrative modeling approach to protein. JL. MacCallum, A. Perez, Ken A. Dill. In prep 77. NCR120002 698. R. Lychev, S. Goldberg, and M. Schapira. Brief announcement: network-destabilizing attacks. In Proceedings of the 2012 ACM symposium on Principles of distributed computing, pages 331{332. ACM, 2012. 78. OCE100015 699. Yu, X., T.-J. Hsu and S. Balachandar 2012. A hybrid spectral-compact scheme for turbulence resolving simulation of fine sediment transport in the bottom boundary layer, research report CACR-12-07,Center for Applied Coastal Research, University of Delaware, Newark, DE . http://www.coastal.udel.edu/~thsu/CACR_report/CACR-TP-1207.pdf 700. Ozdemir, C. E., Hsu, T.-J., Balachandar, S., A Direct numerical simulation study on bottom boundary layer turbulence under a single solitary wave: instability mechanisms, flow transience and characteristics of flow turbulence, J. Fluid Mech., submitted. 701. Yu, X., C. E. Ozdemir, Hsu, T.-J., Balachandar, S., Turbulence Modulations due to Sediment-induced Density Stratification and Rheology during Fine Sediment Transport in an Oscillatory Channel - a Numerical Investigation, Journal of Waterway, Port, Coastal, and Ocean Engineering, submitted. 79. OCE130007 702. Cerovecki, I., L. D. Talley, M. R. Mazloff, and G. Maze, 2013: Subantarctic Mode Water formation, destruction and export in the eddy-permitting Southern Ocean State Estimate. Journal of Physical Oceanography, Accepted for publication. 703. Griesel, A., M. R. Mazloff, and S. T. Gille, 2012: Mean dynamic topography in the Southern Ocean: Evaluating Antarctic Circumpolar Current transport. J. Geophys. Res., 117 (C1), doi:10.1029/2011JC007573. 704. Klocker, A., R. Ferrari, and J. H. LaCasce, 2012: Estimating suppression of eddy mixing by mean ows. Journal of Physical Oceanography, 42 (9), 1566{1576, doi:10.1175/JPO-D-11-0205.1. 705. Mazloff, M. and R. Ferrari, 2013: The force balance of the Southern Ocean meridional over-turning circulation. Journal of Physical Oceanography, In revision.

331 80. PHY110015 706. “SUGRA Grand Unification, LHC and Dark Matter,” P. Nath, arXiv:1207.5501 [hep-ph]. 707. “GUT and Supersymmetry at the LHC and in dark matter,” P. Nath, AIP Conf. Proc. 1467, 29 (2012); arXiv:1205.1153 [hep-ph]. 708. “Cosmic Coincidence and Asymmetric Dark Matter in a Stueckelberg Extension,” W. -Z. Feng, P. Nath and G. Peim, Phys. Rev. D 85, 115016 (2012); arXiv:1204.5752 [hep-ph]. 709. “Higgs Boson Mass Predictions in SUGRA Unification and Recent LHC-7 Results,” S. Akula, B. Altunkaynak, D. Feldman, P. Nath and G. Peim, Phys. Rev. D 85, 075001 (2012); arXiv:1112.3645 [hep-ph]. 710. “Naturalness, Supersymmetry and Implications for LHC and Dark Matter,” S. Akula, M. Liu, P. Nath and G. Peim Phys. Lett. B 709, 192 (2012); arXiv:1111.4589 [hep-ph]. 81. PHY110016 711. Mason, J., Perez, J.C., Boldyrev, S. & Cattaneo, F. Numerical simulations of strong incompressible magnetohydrodynamic turbulence. Phys. of Plasmas, 19 (2012) 055902 712. Perez, J.C., Mason, J., Boldyrev, S. & Cattaneo, F. On the energy spectrum of strong Magnetohydrodynamic turbulence. Phys. Rev. X, 2 (2012) 041005 713. Zhdankin, V., Boldyrev, S., Mason, J. & Perez, J.C. Magnetic discontinuities in Magnetohydrodynamic turbulence and the solar wind. Phys. Rev. Lett., 108 (2012) 175004 714. Zhdankin, V., Boldyrev, S., Mason, J. Distribution of magnetic discontinuities in the solar wind and in MHD turbulence. Astrophys. J. Lett., 760 (2012) L22 82. PHY120002 715. A. Cheng, A. Hasenfratz, G. Petropoulos and D. Schaich, “Searching for the chiral limit of SU(3) gauge theory with 8 flavors”, in preparation (2013) 716. A. Cheng, A. Hasenfratz, G. Petropoulos and D. Schaich, “Exploring infrared conformality with 12 fundamental flavors”, in preparation (2013) 717. A. Cheng, A. Hasenfratz, G. Petropoulos and D. Schaich, “Improved step scaling from Monte Carlo Renormalization Group two-lattice matching”, in preparation (2013) 718. A. Hasenfratz, A. Cheng, G. Petropoulos and D. Schaich, “Reaching the chiral limit in many flavor systems”, to appear in the proceedings of Strong Coupling Gauge Theories in the LHC Perspective (SCGT12) 719. A. Cheng, A. Hasenfratz, G. Petropoulos and D. Schaich, “Scale-dependent mass anomalous dimension from Dirac eigenmodes”, submitted to Physical Review D (2013) [arXiv:1301.1355] 720. G. Petropoulos, A. Cheng, A. Hasenfratz and D. Schaich, “MCRG study of 8 and 12 fundamental flavors”, Proceedings of Science Lattice 2012:051 (2012) [arXiv:1212.0053] 721. D. Schaich, A. Cheng, A. Hasenfratz and G. Petropoulos, “Bulk and finite-temperature transitions in SU(3) gauge theories with many light fermions”, Proceedings of Science Lattice 2012:028 (2012) [arXiv:1207.7164] 722. A. Hasenfratz, A. Cheng, G. Petropoulos and D. Schaich, “Mass anomalous dimension from Dirac eigenmode scaling in conformal and confining systems”, Proceedings of Science Lattice 2012:034 (2012) [arXiv:1207.7162] 723. A. Cheng, A. Hasenfratz and D. Schaich, “Novel phase in SU(3) lattice gauge theory with 12 light fermions”, Physical Review D85:094509 (2012) [arXiv:1111.2317] 83. PHY120016 724. Bode, T. Bogdanovic, R. Haas, J. Healy, P. Laguna and D. Shoemaker, “Mergers of Supermassive Black Holes in Astrophysical Environments,” Astrophysics Journal vol. 45 744 (2012) 725. Haas, R., Shcherbakov, R., Bode, T. and Laguna, P., “Tidal Disruptions of White Dwarfs from Ultra-Close Encounters with Intermediate Mass Spinning Black Holes,” Astrophys. J. 749, 117 (2012) 726. O’Shaughnessy, R., Healy, J., London, L., Meeks, Z. and Shoemaker, D., “Is J enough? Comparison of Gravitational waves emitted along the total angular momentum direction with other preferred orientations,” Physical Review D vol. 85 084003 (2012) 727. Ajith, P. and the NINJA collaboration, ”The NINJA-2 catalog of hybrid post-Newtonian/numerical relativity waveforms for non-precessing black-hole binaries,” Classical Quantum Gravity vol. 29 124001 (2012) 728. O’Shaughnessy, R., London, L., Healy, J. and Shoemaker, D., “Precession during merger 1: Strong polarization changes are observationally accessible features of strong-field gravity during binary black hole merger,” submitted to Physics Review D, arXiv:1209.3712 (2012) 729. Shcherbakov, R., Pe’er, A., Reynolds, C., Haas, R., Bode, T. and Laguna ,P., “GRB060218 as a Tidal Disruption of a White Dwarf by an Intermediate Mass Black Hole,” submitted to Astrophys. J. arXiv:1212.4837

332 730. Shcherbakov, R., Pe’er, A., Reynolds, C., Haas, R., Bode, T. and Laguna, P., “Prompt Emission from Tidal Disruptions of White Dwarfs by Intermediate Mass Black Holes,” arXiv:1212.5267 (2012) 731. Pekowsky, L., Healy, J., Shoemaker, D. and Laguna, P., “Impact of Higher-order Modes on the Detection of Binary Black Hole Coalescences,” accepted by Physical Review D, arXiv:1210.1891 (2012) 732. Healy, J., Laguna, P., Pekowsky, L., Shoemaker, D., “Template Mode Hierarchies for Binary Black Hole Mergers,” in preparation 733. Pekowsky, L., Healy, J., London, L., Shoemaker, D., O’Shaughnessy, R., Comparing gravitational waves from nonprecessing and precessing black hole binaries in the corotating frame,” in preparation 734. Mundim, B., Faber, J., Noble, S., Bode, T., Haas, R., Löffler, F., Mösta, P., Ott, C. and Reisswig, C., “GRHydro-MHD: A public general relativistic magneto-hydrodynamics code, ” in preparation 735. Loffler, F., Faber, J., Bentivegna, E., Bode, T., Diener, P., Haas, R., Hinder, I., Mundim, B. and et al., “The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics,” Class. Quant. Grav. 29, 115001 (2012)

333 G XSEDE EOT Event Details

Type Title Location Date(s) Hours Number of Number Method Funding Participants of Sources Under- represen ted people Online Applications of Parallel Online – 2/14/201 50 345 35 A XSEDE course Computers Cornell 3 – registered Virtual 5/9/2013 Workshop 170 active and XSEDE 40 may Moodle complete Semina Discussion with faculty Florida 1/7/ - 12 6 S XSEDE r Atlantic 1/8/2013 University Worksh Introduction to Barton 1/16/201 7 28 S o Computational Thinking College, 3 Wilson, NC Semina Big Data, Parallel SUNY 2/6 - 4 35 S XSEDE r Computing and Oneonta, 2/8/2013 Computational Thinking NY Worksh SC Cyber infrastructure Clemson, SC 2/11 - 4 40 S XSEDE op Symposium – 2/13/201 Computational Thinking 3 Worksh SC Cyber infrastructure Clemson, SC 2/12/201 1 40 S XSEDE op Symposium – 3 Computational Science Curricula Worksh UTEP Regional Workshop El Paso, TX 2/19 - 16 100 70 S XSEDE op 2/20/201 3 Semina Math in the real world: an Pamlico 3/6/2013 2 28 S Other NSF r introduction to modeling County High funds School Worksh Inquiry Based Science Dublin City 3/12 - 16 65 S Other NSF op Education Enhanced by University, 3/13/201 funds computational thinking Ireland 3 Worksh Computational Thinking for University 3/23/201 4 23 S XSEDE op Algebra of Texas at 3 San Antonio Semina Computational biology and Montgomery 3/28/201 4 35 S XSEDE r formal computational College, MD 3 science curricula Worksh Introduction to XSEDE UNLV, Las 3/14/201 3 22 15 S XSEDE op Vegas, NV 3 Tutoria F. D. G. U. I. Visualizatio Harvard Januar 2.5 32 na Sync, web l n for Big Universit y 17 Data y Exploratory

334 Research CI Days by Campus Champions (Chourasia) Tutoria F. D. G. U. I. Introduction UCSD/S Januar 2 86 42 Sync, web l to using DSC y 31 Hadoop on Gordon (Tatineni) Tutoria F. D. G. U. I. Visualizatio UCSD/S Februa 1.5 84 54 Sync, web l n Training: DSC ry 11 Preparing/Tr anslating Data for use with VisIt application (Chourasia) Tutoria F. D. G. U. I. Introduction UCSD/S March 2 50 24 Sync, web l to the DSC 14 Neuroscienc e Gateway (NSG) - A Portal for Computation al Neuroscienc e (Majumdar) Worksh F. D. G. U. I. OpenACC UCSD/S Januar 12 11 n.a. Web op GPU DSC y 15- Programing 16 Workshop (satellite location) Worksh F. D. G. I. PACE: Data UCSD/S Januar 16 23 8 Sync, web op Mining Boot DSC y 23- Camp 1 24 (Balac) Worksh F. D. G. I. PACE: Data UCSD/S Februa 16 19 8 Sync op Mining Boot DSC ry 7-8 Camp 2 (Balac) W F. S. Smart Team UCSD/S Januar 6 23 9 S “From DSC y 12 Genes to Drugs – a Lab in a Laptop” W S San Jan 10 5 5 S StudentTEC Diego 15,17, H – Alice 22, 24, 29 W S, F SMART UCSD/S Feb 9 3 22 9 S Team DSC Modeling

335 Clinic W S, F SMART UCSD/S March 27 2 24 S Team DSC 9 Middle School Protein Exploration Day W S SMART TSRI, La March 2 34 11 S Team Poster Jolla 14 Session OUTREAC H Conf F.G.U.D. Tapia Washing Februa 24 550 500 S Celebration ton D.C. ry 20 of Diversity in Computing W S S.D. UCSD/S March 6 86 5 S Mayor’s DSC 2 Cup Cybersecurit y Challenge Wrkshp F. D. I. UCSD’s San March 1 100 20 S Research Diego, 12, Cyberinfrast CA 2013 ructure Program: Enabling Research Thru Shared Services- at CENIC 2013: Building Blocks for Next Gen Networks M F San Diego UCSD/S March 2 15 8 S CSTA DSC 12 Chapter Meeting Wrkshp F. D. I. Data Arlingto March 1 150 30 S Services for n, VA 13, Campus 2013 Researchers. Research Data Management Implementat ions Workshop Conf Public San Diego San March 8 500 300 S Science Diego 23 Festival

336 Tutoria High Throughput IU 9 Jan-13 2 7 0 S l Computing Indianapolis, 2013 hours IN Campus Confer IU-UITS Regional South Bend, 13-15 6 70 5 S ence Northern Campus Tour Gary and Feb- hours talk/pre Kokomo IN 2013 sentatio Campuses n/panel Confer Open Science Grid All IU 11-14 32 131 15 S ence Hands Meeting Indianapolis, Mar- hours talk/pre IN Campus 2013 sentatio n/panel Worksh Intro to Parallel IU 2 Feb- 8 2 0 S op Programming Indianapolis, 2013 hours IN Campus Confer National Center for Plant and Jan 12- 32 200 30 S ence Genome Analysis and Animal 16th 2013 talk/pre Support and XSEDE Genome hours sentatio Conference, n/panel San Diego, CA

337