GPU Clusters for HPC Bill Kramer Director of Blue Waters National Center for Supercomputing Applications University of Illinois at Urbana- Champaign National Center for Supercomputing Applications University of Illinois at Urbana-Champaign National Center for Supercomputing Applications: 30 years of leadership • NCSA • R&D unit of the University of Illinois at Urbana-Champaign • One of original five NSF-funded supercomputing centers • Mission: Provide state-of-the-art computing capabilities (hardware, software, hpc expertise) to nation’s scientists and engineers • The Numbers • Approximately 200 staff (160+ technical/professional staff) • Approximately 15 graduate students (+ new SPIN program), 15 undergrad students • Two major facilities (NCSA Building, NPCF) • Operating NSF’s most powerful computing system: Blue Waters • Managing NSF’s national cyberinfrastructure: XSEDE Source: Thom Dunning Petascale Computing Facility: Home to Blue Waters • Blue Waters • 13PF, 1500TB, 300PB • >1PF On real apps • NAMD, MILC, WRF, PPM, NWChem, etc • Modern Data Center • Energy Efficiency • 90,000+ ft2 total • LEED certified Gold • 30,000 ft2 raised floor • Power Utilization Efficiency 2 = 1.1–1.2 20,000 ft machine room gallery Source: Thom Dunning Data Intensive Computing Personalized Medicine w/ Mayo LSST, DES Source: Thom Dunning NCSA’s Industrial Partners Source: Thom Dunning NCSA, NVIDIA and GPUs • NCSA and NVIDIA have been partners for over a decade, building the expertise, experience and technology. • The efforts were at first exploratory and small scale, but have now blossomed into providing the largest GPU production resource in the US academic cyber- infrastructure • Today, we are focusing on helping world class science and engineering teams decrease their time to insight for some of the world’s most important and challenging computational and data analytical problems Imaginations unbound Original Blue Waters Goals • Deploy a computing system capable of sustaining more than one petaflops or more for a broad range of applications • Cray system achieves this goal using a well defined metrics • Enable the Science Teams to take full advantage of the sustained petascale computing system • Blue Waters Team has established strong partnership with Science Teams, helping them to improve the performance and scalability of their applications • Enhance the operation and use of the sustained petascale system • Blue Waters Team is developing tools, libraries and other system software to aid in operation of the system and to help scientists and engineers make effective use of the system • Provide a world-class computing environment for the petascale computing system • The NPCF is a modern, energy-efficient data center with a rich WAN environment (100-400 Gbps) and data archive (>300 PB) • Exploit advances in innovative computing technology • Proposal anticipated the rise of heterogeneous computing and planned to help the computational community transition to new modes for computational and data-driven science and engineering Imaginations unbound Blue Waters Computing System Aggregate Memory – 1.6 PB IB Switch >1 TB/sec 10/40/100 Gb External Servers Ethernet Switch 100 GB/sec 120+ Gb/sec Spectra Logic: 300 usable PB Sonexion: 26 usable PB 100-300 Gbps WAN Imaginations unbound Details of Blue Waters Imaginations unbound Computation by Discipline on Blue Waters Actual Usage by Discipline STEM Education Physics Social Sciences 0.01% 2.5% 0.3% Astronomy and Astrophysics Particle Physics 17.8% 25.9% Atmospheric and Climate Sciences 10.4% Nuclear Physics 0.7% Mechanical and Dynamic Systems 0.03% Biology and Biophysics Materials Humanities Fluid 23.6% Science 0.0002% Systems 3.3% 5.1% Geophysics 1.3% Engineering Chemistry 0.05% 6.5% Earth Sciences 2.0% Computer Science 0.5% Imaginations unbound XK7 Usage by NSF PRAC teams – A Behavior Experiment – First year • An observed experiment – teams self select what type of node is most useful • First year of usage Amber - NAMD/VMD - QCD – MILC and MD MD Chroma Gromacs - MD Increasing allocation size Imaginations unbound Production Computational Science with XK nodes • The Computational Microscope • PI – Klaus Schulten • Simulated flexibility of ribosome trigger factor complex at full length and obtained better starting configuration of trigger factor model (simulated to 80ns) • 100ns simulation of cylindrical HIV 'capsule’ of CA proteins revealed it is stabilized by hydrophobic interactions between CA hexamers; maturation involves detailed remodeling rather than disassembly/re-assembly of CA lattice, as had been proposed. • 200ns simulation of CA pentamer surrounded by CA hexamers suggested interfaces in hexamer-hexamer and hexamer-pentamer pairings involve different patterns of interactions • Simulated photosynthetic membrane of a chromatophore in bacterium Rps. photometricum for 20 ns -- simulation of a few hundred nanoseconds will be needed Images from Klaus Schulten and John Stone, University of Illinois at Urbana-Champaign Imaginations unbound Production Computational Science with XK nodes • Lattice QCD on Blue Waters • PI - Robert Sugar, University of California, Santa Barbara • The USQCD Collaboration, which consists of nearly all of the high-energy and nuclear physicists in the United States working on the numerical study of quantum chromodynamics (QCD), will use Blue Waters to study the theory of the strong interactions of sub-atomic physics, including simulations at the physical masses of the up and down quarks, the two lightest of the six quarks that are the fundamental constituents of strongly interacting matter Imaginations unbound Production Computational Science with XK nodes • Hierarchical molecular dynamics sampling for assessing pathways and free energies of RNA catalysis, ligand binding, and conformational change • PI - Thomas Cheatham, University of Utah • Attempting to decipher the full landscape of RNA structure and function. • Challenging because • RNA require modeling the flexibility and subtle balance between charge, stacking and other molecular interactions • structure of RNA is highly sensitive to its surroundings, and RNA can adopt multiple functionally relevant conformations. • Goal - Fully map out the conformational, energetic and chemical landscape of RNA. • "Essentially we are able to push enhanced sampling methodologies for molecular dynamics simulation, specifically replica-exchange, to complete convergence for conformational ensembles (which hasn't really been investigated previously) and perform work that normally would take 6 months to years in weeks. This is critically important for validating and assessing the force fields for nucleic acids,” - Cheatham. Images courtesy – T Cheatham Imaginations unbound Imaginations unbound Imaginations nodes Computational of XK Most RecentUse Node*Hours 1,000,000.0 2,000,000.0 3,000,000.0 4,000,000.0 5,000,000.0 6,000,000.0 7,000,000.0 8,000,000.0 9,000,000.0 - Teams and withbothXK usage XE Teams Karimabadi-3D Kinetic Sims. of… Sugar-Lattice QCD Yeung-Complex Turbulent Flows… Schulten-The Computational… Cheatham-MD Pathways and… Aksimentiev-Pioneering… Shapiro-Signatures of Compact… Mori-Plasma Physics Sims. using… Ott -CCSNe, Hypermassive… Voth -Multiscale Sims. of… Tajkhorshid -Complex Biology in… Glotzer-Many-GPU Sims. of Soft… Woosley-Type Ia Supernovae Jordan -Earthquake System… 2014 Aluru-QMC of H2O-Graphene,… Tomko-Redesigning Comm. and… Bernholc -Quantum Sims.… Pande -Simulating Vesicle Fusion - Kasson-Influenza Fusion… July 1, 2014 30, to Sept July 1, 2014 Chemla-Chemla Lusk-Sys. Software for Scalable… Thomas -QC during Steel… Fields-Benchmark Human Variant… Makri-QCPI Proton & Electron… Hirata -Predictive Comp. of… Elghobashi-DNS of Vaporizing… Jongeneel-Accurate Gene… Lazebnik -Large-Scale… Beltran -Spot Scanning Proton… Woodward -Turbulent Stellar… XE NodeXE Hrs XK Node Hrs Total Node*hrs Most Resent Computational Use of XK nodes Teams with both XE and XK usage - July 1, 2014 to Sept 30, 2014 9,000,000.0 8,000,000.0 7,000,000.0 6,000,000.0 5,000,000.0 4,000,000.0 Node*Hours 3,000,000.0 2,000,000.0 1,000,000.0 Total Node*hrs - XK Node Hrs XE Node Hrs Imaginations unbound Evolving XK7 Use on BW - Major Advance in Understanding of Collisionless Plasmas Enabled through Petascale Kinetic Simulations • PI: Homayoun Karimabadi, University of California, San Diego • Major results to date: • Global fully kinetic simulations of magnetic reconnection • First large-scale 3D simulations of decaying collisionless plasma turbulence • 3D global hybrid simulations addressing coupling between Fully kinetic simulation (all species kinetic; shock physics & Large scale hybrid kinetic simulation: code: VPIC) (kinetic ions + fluid electrons; magnetosheath turbulence ~up to 1010 cells codes: H3D, HYPERES) ~up to 4x1012 particles ~up to 1.7x1010 cells ~120 TB of memory ~up to 2x1012 particles ~107 CPU-HRS Slide courtesy of H Karimardi ~130 TB of memory ~up to 500,000 cores Imaginations unbound Evolving XK7 Use on BW - Petascale Particle in Cell Simulations of of Kinetic Effects in Plasmas • PI – Warren Mori – Presenter – Frank Tsung • Use six parallel particle-in-cell (PIC) codes to investigate four key science areas: • Can fast ignition be used to develop inertial fusion energy? • What is the source of the most energetic particles in the cosmos? • Can plasma-based acceleration be the basis of new compact accelerators for use at the energy frontier, in medicine, in probing materials, and in novel light sources? • What processes trigger substorms