DOCSLIB.ORG

Cray CS-Storm Nodes to Handle Billions of Entries 70 – K80 Will Be an Advantage Generic 2-GPU 60 50 40 “CS-Storm’S Dense, Tightly Integrated Dense GPU 30

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Cray CS-Storm Nodes to Handle Billions of Entries 70 – K80 Will Be an Advantage Generic 2-GPU 60 50 40 “CS-Storm’S Dense, Tightly Integrated Dense GPU 30 Realizing GPU Computation at Scale CS-Storm: Building a Productive & Reliable GPU Platform John K. Lee, Vice President, Cluster Products Maria Iordache PhD, Product Management Director C O M P U T E | S T O R E | A N A L Y Z E Legal Disclaimer Information in this document is provided in connection with Cray Inc. products. No license, express or implied, to any intellectual property rights is granted by this document. Cray Inc. may make changes to specifications and product descriptions at any time, without notice. All products, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. Cray hardware and software products may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Cray uses codenames internally to identify products that are in development and not yet publically announced for release. Customers and other third parties are not authorized by Cray Inc. to use codenames in advertising, promotion or marketing and any use of Cray Inc. internal codenames is at the sole risk of the user. Performance tests and ratings are measured using specific systems and/or components and reflect the approximate performance of Cray Inc. products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. The following are trademarks of Cray Inc. and are registered in the United States and other countries: CRAY and design, SONEXION, URIKA and YARCDATA. The following are trademarks of Cray Inc.: ACE, APPRENTICE2, CHAPEL, CLUSTER CONNECT, CRAYPAT, CRAYPORT, ECOPHLEX, LIBSCI, NODEKARE, THREADSTORM. The following system family marks, and trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT and XT. The registered trademark LINUX is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other names and brands may be claimed as the property of others. Other product and service names mentioned herein are the trademarks of their respective owners. Copyright 2015 Cray Inc. C O M P U T E | S T O R E | A N A L Y Z E 2 Cray Inc. - GTC, March 2014 Agenda ● About Cray ● Building a Productive & Reliable dense GPU Platform – CS-Storm ● Realizing GPU computation at scale K40 or K80 inside C O M P U T E | S T O R E | A N A L Y Z E 3 Cray Inc. - GTC, March 2014 About Cray Seymour Cray founded Cray Research in 1972 • 1972-1996, Cray Research grew to leadership in Supercomputing • 1996-2000, Cray was subsidiary of SGI • 2000- present, Cray Inc. growing to $561M in revenue in 2014 Cray Inc. • NASDAQ: CRAY • Over 1,000 employees across 30 countries • Headquartered in Seattle, WA Three Focus Areas • Computation • Storage • Analytics Seven Major • Austin, TX • San Jose, CA Development Sites: • Chippewa Falls, WI • Seattle, WA • Pleasanton, CA • Bristol, UK • St. Paul, MN C O M P U T E | S T O R E | A N A L Y Z E Cray Inc. Cray’s Vision: The Fusion of Supercomputing and Big & Fast Data Modeling The World Cray Supercomputers solving “grand challenges” in science, engineering and analytics Data Models Math Models Data-Intensive Processing Modeling and Integration of datasets and math models for simulation augmented High throughput search, analysis, with data to provide event processing & the highest fidelity data capture from predictive modeling virtual reality results sensors, data feeds and knowledge and instruments discovery Compute Store Analyze C O M P U T E | S T O R E | A N A L Y Z E Cray Inc. Supercomputing Leadership Top 500 Supercomputers in the World November 2014 Top 50 Top 100 Top 500 Cray Systems 16 28 62 Vendor Rank #1 #1 (tied) #3 C O M P U T E | S T O R E | A N A L Y Z E 6 Cray Inc. - GTC, March 2014 Cray has vast experience building very large scale, GPU-based HPC systems ● Cray has more accelerated systems in Top500 than anyone ● 75 of Top500 are systems with accelerators ● 14 of these are made by Cray ● Cray systems have more GPU performance (Rmax) on Top500 than all others combined! ● GPUs supported on both, Cray’s Cluster Systems and Supercomputer line – multiple of each in Top500 ● CS-Storm dense-GPU Computing Platform, with 8 GPUs / server – launched in Aug 2014: ● #10 on Top500, Nov 2014 ● #4 on Green500, Nov 2014 C O M P U T E | S T O R E | A N A L Y Z E 7 Cray Inc. - GTC, March 2014 Background on CS-Storm ● Started as a custom engineering project in late 2013 ● Customer funded project to build the highest performing PCIe-attached Accelerator platform ● Able to run the host processors at peak performance ● Able to run Accelerators at peak performance for cards up to 300W ● Successfully delivered a production system with over 4,480x K40 GPUs in a single system ● #10 on last year’s Top500 list ● Another system is #4 on the Green500 list ● Designed to support any full-height, double width PCIe accelerators ● Currently supports K40 and K80 GPUs C O M P U T E | S T O R E | A N A L Y Z E 8 Cray Inc. - GTC, March 2014 CS-Storm: Design What makes it special? Efficient air-cooled 2U design, with enough power and cooling capacity to operate eight current and future GPUs and CPUs at peak level without capping their performance • Pull/push fan architecture efficiently cools the chassis, providing a consistent temperature across the system and its accelerators Most efficient use of PCIe bandwidth to operate the eight GPUs at full performance • Cray R&D performed a comprehensive signal integrity study to determine the electrical signaling capabilities of the PCIe bus Innovative, easy-to-maintain server design, providing easy access to all components • 3 x 1,630W power supplies capable of 2+1 redundancy • Native 480 V and 208 V server power supplies – increases power efficiency Room-neutral cooling via optional rear-door heat exchangers C O M P U T E | S T O R E | A N A L Y Z E 9 Cray Inc. - GTC, March 2014 CS-Storm: Innovative Design Six local SSD drives Host Intel processors IVB / HSW 512 / 1024 GB max 16 DIMMS 2x4 NVIDIA K40 or K80 11.4 or 15 TF/node 2U form factor • 22 nodes / 48U rack • 176 GPUs / 48U rack C O M P U T E | S T O R E | A N A L Y Z E 10 Cray Inc. - GTC, March 2014 CS-Storm: Server PCI-e Layout Haswell motherboard High Speed High Speed Network2 Network1 QPI performance Optimized GPU Optimized C O M P U T E | S T O R E | A N A L Y Z E 11 Cray Inc. - GTC, March 2014 GPU Cage Serviceability 24”-wide chassis provides more room than standard 19” chassis to effectively service and cool the GPUs C O M P U T E | S T O R E | A N A L Y Z E 1 Cray Inc. - GTC, March 2014 2 Individual GPU Access ● Each cage holds 4x GPUs. ● Each GPU assembly can be removed easily through a blind mate quick disconnect. 3 1 4 2 GPU assembly C O M P U T E | S T O R E | A N A L Y Z E 1 Cray Inc. - GTC, March 2014 3 Future-Proof Design PCIe Gen3 standard size K40 / K80 Maximum dimension PCIe card supported: 39.06 mm x 132.08 mm x 313.04 mm C O M P U T E | S T O R E | A N A L Y Z E 14 Cray Inc. - GTC, March 2014 CS-Storm System: Cooling Performance GP GPU Power CS-Storm's pull/push fan architecture U # Temperature Consumption efficiently cools the chassis, providing 1 63 213W a consistent temperature across the 2 64 222W system and its coprocessors. 3 63 214W CS-Storm is designed to provide enough 4 61 218W air flow to keep all GPUs cool under the most challenging compute workloads 5 53 218W 6 53 215W . Measured at Chippewa Falls manufacturing facility; with 27C ambient temperature 7 57 218W . DGEMM was running on all GPUs at the time of 8 51 213W record capture . The table shows the “hottest” of the 22 nodes in the rack C O M P U T E | S T O R E | A N A L Y Z E 1 Cray Inc. - GTC, March 2014 5 CS-Storm System: Efficient, Room Neutral Cooling Customized 48U RDHx with 64.2 kW max cooling Rear-door heat exchangers are available in both 42U and 48U heights C O M P U T E | S T O R E | A N A L Y Z E 16 Cray Inc. - GTC, March 2014 Software at Scale: Partnerships deliver a complete software ecosystem Essential software and management tools needed to build a powerful, flexible and highly available supercomputer Development & Intel® Parallel PGI Cluster NVIDIA® Performance ® Studio XE Cluster GNU toolchain Cray PE on CS * Development Kit® CUDA® Tools Edition HPC Application Cray LibSci, Programming Intel® MPI, MKL Platform MPI MVAPICH2 OpenMPI Libraries LibSci_ACC Tools Rogue Wave Debuggers Allinea DDT, MAP Intel® IDB PGI PGDBG® GNU GDB TotalView® Resource Adaptive Computing: IBM Platform™ Management / SLURM MOAB® / Maui / Altair PBSPro Grid Engine LSF® Job Scheduling Torque Schedulers, File Systems and File Systems Lustre® NFS GPFS PanFS® Local (ext3, ext4, XFS) Management Cluster Cray® Advanced Cluster Engine (ACE™) Management Software** Management Drivers and Operating Accelerator software stack & drivers OFEDTM Systems and Network Mgmt.
Load more

Recommended publications
  • An Operational Perspective on a Hybrid and Heterogeneous Cray XC50 System
    An Operational Perspective on a Hybrid and Heterogeneous Cray XC50 System Sadaf Alam, Nicola Bianchi, Nicholas Cardo, Matteo Chesi, Miguel Gila, Stefano Gorini, Mark Klein, Colin McMurtrie, Marco Passerini, Carmelo Ponti, Fabio Verzelloni CSCS – Swiss National Supercomputing Centre Lugano, Switzerland Email: {sadaf.alam, nicola.bianchi, nicholas.cardo, matteo.chesi, miguel.gila, stefano.gorini, mark.klein, colin.mcmurtrie, marco.passerini, carmelo.ponti, fabio.verzelloni}@cscs.ch Abstract—The Swiss National Supercomputing Centre added depth provides the necessary space for full-sized PCI- (CSCS) upgraded its flagship system called Piz Daint in Q4 e daughter cards to be used in the compute nodes. The use 2016 in order to support a wider range of services. The of a standard PCI-e interface was done to provide additional upgraded system is a heterogeneous Cray XC50 and XC40 system with Nvidia GPU accelerated (Pascal) devices as well as choice and allow the systems to evolve over time[1]. multi-core nodes with diverse memory configurations. Despite Figure 1 clearly shows the visible increase in length of the state-of-the-art hardware and the design complexity, the 37cm between an XC40 compute module (front) and an system was built in a matter of weeks and was returned to XC50 compute module (rear). fully operational service for CSCS user communities in less than two months, while at the same time providing significant improvements in energy efficiency. This paper focuses on the innovative features of the Piz Daint system that not only resulted in an adaptive, scalable and stable platform but also offers a very high level of operational robustness for a complex ecosystem.
    [Show full text]
  • A Scheduling Policy to Improve 10% of Communication Time in Parallel FFT
    A scheduling policy to improve 10% of communication time in parallel FFT Samar A. Aseeri Anando Gopal Chatterjee Mahendra K. Verma David E. Keyes ECRC, KAUST CC, IIT Kanpur Dept. of Physics, IIT Kanpur ECRC, KAUST Thuwal, Saudi Arabia Kanpur, India Kanpur, India Thuwal, Saudi Arabia [email protected] [email protected] [email protected] [email protected] Abstract—The fast Fourier transform (FFT) has applications A. FFT Algorithm in almost every frequency related studies, e.g. in image and signal Forward Fourier transform is given by the following equa- processing, and radio astronomy. It is also used to solve partial differential equations used in fluid flows, density functional tion. 3 theory, many-body theory, and others. Three-dimensional N X (−ikxx) (−iky y) (−ikz z) 3 f^(k) = f(x; y; z)e e e (1) FFT has large time complexity O(N log2 N). Hence, parallel algorithms are made to compute such FFTs. Popular libraries kx;ky ;kz 3 perform slab division or pencil decomposition of N data. None The beauty of this sum is that k ; k ; and k can be summed of the existing libraries have achieved perfect inverse scaling x y z of time with (T −1 ≈ n) cores because FFT requires all-to-all independent of each other. These sums are done at different communication and clusters hitherto do not have physical all- to-all connections. With advances in switches and topologies, we now have Dragonfly topology, which physically connects various units in an all-to-all fashion.
    [Show full text]
  • View Annual Report
    Fellow Shareholders, Headlined by strong growth in revenue and profitability, we had one of our best years ever in 2015, executing across each of our major focus areas and positioning our company for continued growth into the future. We achieved another year of record revenue, growing by nearly 30 percent compared to 2014. In fact, our revenue in 2015 was more than three times higher than just four years prior — driven by growth in both our addressable market and market share. Over the last five years, we have transformed from a company solely focused on the high-end of the supercomputing market — where we are now the clear market leader — to a company with multiple product lines serving multiple markets. We provide our customers with powerful computing, storage and analytics solutions that give them the tools to advance their businesses in ways never before possible. During 2015, we installed supercomputing and storage solutions at a number of customers around the world. In the U.S., we completed the first phase of the massive new “Trinity” system at Los Alamos National Laboratory. This Cray XC40 supercomputer with Sonexion storage serves as the National Nuclear Security Administration’s flagship supercomputer, supporting all three of the NNSA’s national laboratories. We installed the first petaflop supercomputer in India at the Indian Institute of Science. In Europe, we installed numerous XC and storage solutions, including a significant expansion of the existing XC40 supercomputer at the University of Stuttgart in Germany. This new system nearly doubles their computing capacity and is currently the fastest supercomputer in Germany.
    [Show full text]
  • TECHNICAL GUIDELINES for APPLICANTS to PRACE 17Th CALL
    TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 17th CALL (T ier-0) The contributing sites and the corresponding computer systems for this call are: System Architecture Site (Country) Core Hours Minimum (node hours) request Joliot Curie - Bull Sequana X1000 GENCI@CEA 134 million 15 million core SKL (FR) (2.8 million) hours Joliot Curie - BULL Sequana GENCI@CEA 72 million 15 million core KNL X1000 (FR) (1,1 million) hours Hazel Hen Cray XC40 System GCS@HLRS 70 million 35 million core (DE) (2.9 million) hours JUWELS Multicore cluster GCS@JSC (DE) 70 million 35 million core (1.5 million) hours Marconi- Lenovo System CINECA (IT) 36 million 15 million core Broadwell (1 million) hours Marconi-KNL Lenovo System CINECA (IT) 612 million 30 million core (9 million) hours MareNostrum Lenovo System BSC (ES) 240 million 15 million core (5 million) hours Piz Daint Cray XC50 System CSCS (CH) 510 million 68 million core (7.5 million) hours Use of GPUs SuperMUC Lenovo NextScale/ GCS@LRZ (DE) 105 million 35 million core SuperMUC-NG Lenovo ThinkSystem (3.8 million) hours The site selection is done together with the specification of the requested computing time by the two sections at the beginning of the online form. The applicant can choose one or several machines as execution system, as long as proper benchmarks and resource request justification are provided on each of the requested systems. The parameters are listed in tables. The first column describes the field in the web online form to be filled in by the applicant. The remaining columns specify the range limits for each system.
    [Show full text]
  • Cray XC40 Power Monitoring and Control for Knights Landing
    Cray XC40 Power Monitoring and Control for Knights Landing Steven J. Martin, David Rush Matthew Kappel, Michael Sandstedt, Joshua Williams Cray Inc. Cray Inc. Chippewa Falls, WI USA St. Paul, MN USA {stevem,rushd}@cray.com {mkappel,msandste,jw}@cray.com Abstract—This paper details the Cray XC40 power monitor- This paper is organized as follows: In section II, we ing and control capabilities for Intel Knights Landing (KNL) detail blade-level Hardware Supervisory System (HSS) ar- based systems. The Cray XC40 hardware blade design for Intel chitecture changes introduced to enable improved telemetry KNL processors is the first in the XC family to incorporate enhancements directly related to power monitoring feedback gathering capabilities on Cray XC40 blades with Intel KNL driven by customers and the HPC community. This paper fo- processors. Section III provides information on updates to cuses on power monitoring and control directly related to Cray interfaces available for power monitoring and control that blades with Intel KNL processors and the interfaces available are new for Cray XC40 systems and the software released to users, system administrators, and workload managers to to support blades with Intel KNL processors. Section IV access power management features. shows monitoring and control examples. Keywords-Power monitoring; power capping; RAPL; energy efficiency; power measurement; Cray XC40; Intel Knights II. ENHANCED HSS BLADE-LEVEL MONITORING Landing The XC40 KNL blade incorporates a number of en- hancements over previous XC blade types that enable the I. INTRODUCTION collection of power telemetry in greater detail and with Cray has provided advanced power monitoring and control improved accuracy.
    [Show full text]
  • Hpc in Europe
    HPC IN EUROPE Organisation of public HPC resources Context • Focus on publicly-funded HPC resources provided primarily to enable scientific research and development at European universities and other publicly-funded research institutes • These resources are also intended to benefit industrial / commercial users by: • facilitating access to HPC • providing HPC training • sponsoring academic-industrial collaborative projects to exchange expertise and accelerate efficient commercial exploitation of HPC • Do not consider private sector HPC resources owned and managed internally by companies, e.g. in aerospace design & manufacturing, oil & gas exploration, fintech (financial technology), etc. European HPC Infrastructure • Structured provision of European HPC facilities: • Tier-0: European Centres (> petaflop machines) • Tier-1: National Centres • Tier-2: Regional/University Centres • Tiers planned as part of an EU Research Infrastructure Roadmap • This is coordinated through “PRACE” – http://prace-ri.eu PRACE Partnership foR Advanced Computing in Europe • International non-profit association (HQ office in Brussels) • Established in 2010 following ESFRI* roadmap to create a persistent pan-European Research Infrastructure (RI) of world-class supercomputers • Mission: enable high-impact scientific discovery and engineering research and development across all disciplines to enhance European competitiveness for the benefit of society. *European Strategy Forum on Reseach Infrastructures PRACE Partnership foR Advanced Computing in Europe Aims: • Provide access to leading-edge computing and data management resources and services for large-scale scientific and engineering applications at the highest performance level • Provide pan-European HPC education and training • Strengthen the European users of HPC in industry A Brief History of PRACE PRACE Phases & Objectives • Preparation and implementation of the PRACE RI was supported by a series of projects funded by the EU’s FP7 and Horizon 2020 funding programmes • 530 M€ of funding for the period 2010-2015.
    [Show full text]
  • Piz Daint”, One of the Most Powerful Supercomputers
    The Swiss High-Performance Computing and Networking Thomas C. Schulthess | T. Schulthess !1 What is considered success in HPC? | T. Schulthess !2 High-Performance Computing Initiative (HPCN) in Switzerland High-risk & high-impact projects (www.hp2c.ch) Application driven co-design Phase III of pre-exascale supercomputing ecosystem Three pronged approach of the HPCN Initiative 2017 1. New, flexible, and efficient building 2. Efficient supercomputers 2016 Monte Rosa 3. Efficient applications Pascal based hybrid Cray XT5 2015 14’762 cores Upgrade to Phase II Cray XE6 K20X based hybrid 2014 Upgrade 47,200 cores Hex-core upgrade Phase I 22’128 cores 2013 Development & Aries network & multi-core procurement of 2012 petaflop/s scale supercomputer(s) 2011 2010 New 2009 building Begin construction complete of new building | T. Schulthess !3 FACT SHEET “Piz Daint”, one of the most powerful supercomputers A hardware upgrade in the final quarter of 2016 Thanks to the new hardware, researchers can run their simula- saw “Piz Daint”, Europe’s most powerful super- tions more realistically and more efficiently. In the future, big computer, more than triple its computing perfor- science experiments such as the Large Hadron Collider at CERN mance. ETH Zurich invested around CHF 40 million will also see their data analysis support provided by “Piz Daint”. in the upgrade, so that simulations, data analysis and visualisation can be performed more effi- ETH Zurich has invested CHF 40 million in the upgrade of “Piz ciently than ever before. Daint” – from a Cray XC30 to a Cray XC40/XC50. The upgrade in- volved replacing two types of compute nodes as well as the in- With a peak performance of seven petaflops, “Piz Daint” has tegration of a novel technology from Cray known as DataWarp.
    [Show full text]
  • Accelerating Science with the NERSC Burst Buffer Early User Program
    Accelerating Science with the NERSC Burst Buffer Early User Program Wahid Bhimji∗, Debbie Bard∗, Melissa Romanus∗y, David Paul∗, Andrey Ovsyannikov∗, Brian Friesen∗, Matt Bryson∗, Joaquin Correa∗, Glenn K. Lockwood∗, Vakho Tsulaia∗, Suren Byna∗, Steve Farrell∗, Doga Gursoyz, Chris Daley∗, Vince Beckner∗, Brian Van Straalen∗, David Trebotich∗, Craig Tull∗, Gunther Weber∗, Nicholas J. Wright∗, Katie Antypas∗, Prabhat∗ ∗ Lawrence Berkeley National Laboratory, Berkeley, CA 94720 USA, Email: [email protected] y Rutgers Discovery Informatics Institute, Rutgers University, Piscataway, NJ, USA zAdvanced Photon Source, Argonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USA Abstract—NVRAM-based Burst Buffers are an important part their workflow, initial results and performance measurements. of the emerging HPC storage landscape. The National Energy We conclude with several important lessons learned from this Research Scientific Computing Center (NERSC) at Lawrence first application of Burst Buffers at scale for HPC. Berkeley National Laboratory recently installed one of the first Burst Buffer systems as part of its new Cori supercomputer, collaborating with Cray on the development of the DataWarp A. The I/O Hierarchy software. NERSC has a diverse user base comprised of over 6500 users in 700 different projects spanning a wide variety Recent hardware advancements in HPC systems have en- of scientific computing applications. The use-cases of the Burst abled scientific simulations and experimental workflows to Buffer at NERSC are therefore also considerable and diverse. We tackle larger problems than ever before. The increase in scale describe here performance measurements and lessons learned and complexity of the applications and scientific instruments from the Burst Buffer Early User Program at NERSC, which selected a number of research projects to gain early access to the has led to corresponding increase in data exchange, interaction, Burst Buffer and exercise its capability to enable new scientific and communication.
    [Show full text]
  • Cray System Monitoring: Successes, Requirements, and Priorities
    Cray System Monitoring: Successes, Requirements, and Priorities x Ville Ahlgreny, Stefan Anderssonx, Jim Brandt , Nicholas P. Cardoz, Sudheer Chunduri∗, x xi Jeremy Enos∗∗, Parks Fieldsk, Ann Gentile , Richard Gerberyy, Joe Greenseid , Annette Greineryy, Bilel Hadri{, Yun (Helen) Heyy, Dennis Hoppex, Urpo Kailay, Kaki Kellyk, Mark Kleinz, x x Alex Kristiansen∗, Steve Leakyy, Mike Masonk, Kevin Pedretti , Jean-Guillaume Piccinaliz, Jason Repik , Jim Rogerszz, Susanna Salmineny, Mike Showerman∗∗, Cary Whitneyyy, and Jim Williamsk ∗Argonne Leadership Computing Facility (ALCF), Argonne, IL USA 60439 Email: (sudheerjalexk)@anl.gov yCenter for Scientific Computing (CSC - IT Center for Science), (02101 Espoo j 87100 Kajaani), Finland Email: (ville.ahlgrenjurpo.kailajsusanna.salminen)@csc.fi zSwiss National Supercomputing Centre (CSCS), 6900 Lugano, Switzerland Email: (cardojkleinjjgp)@cscs.ch xHigh Performance Computing Center Stuttgart (HLRS), 70569 Stuttgart, Germany Email: ([email protected],[email protected]) {King Abdullah University of Science and Technology (KAUST), Thuwal 23955, Saudia Arabia Email: [email protected] kLos Alamos National Laboratory (LANL), Los Alamos, NM USA 87545 Email: (parksjkakjmmasonjjimwilliams)@lanl.gov ∗∗National Center for Supercomputing Applications (NCSA), Urbana, IL USA 61801 Email: (jenosjmshow)@ncsa.illinois.edu yyNational Energy Research Scientific Computing Center (NERSC), Berkeley, CA USA 94720 Email: (ragerberjamgreinerjsleakjclwhitney)@lbl.gov zzOak Ridge National Laboratory (ORNL), Oak Ridge, TN USA 37830 Email: [email protected] x Sandia National Laboratories (SNL), Albuquerque, NM USA 87123 Email: (brandtjgentilejktpedrejjjrepik)@sandia.gov xi Cray Inc., Columbia, MD, USA 21045 Email: [email protected] Abstract—Effective HPC system operations and utilization in the procurement of a platform tailored to support the require unprecedented insight into system state, applications’ performance demands of a site’s workload.
    [Show full text]
  • Using NERSC for Research in High Energy Physics Theory
    Using NERSC for Research in High Energy Physics Theory Richard Gerber Senior Science Advisor HPC Department Head (Acting) NERSC: the Mission HPC Facility for DOE Office of Science Research Largest funder of physical science research in U.S. Bio Energy, Environment Compung Materials, Chemistry, Geophysics Par2cle Physics, Astrophysics Nuclear Physics Fusion Energy, Plasma Physics 6,000 users, 700 projects, 700 codes, 48 states, 40 countries, universities & national labs Current Production Systems Edison 5,560 Ivy Bridge Nodes / 24 cores/node 133 K cores, 64 GB memory/node Cray XC30 / Aries Dragonfly interconnect 6 PB Lustre Cray Sonexion scratch FS Cori Haswell Nodes 1,900 Haswell Nodes / 32 cores/node 52 K cores, 128 GB memory/node Cray XC40 / Aries Dragonfly interconnect 24 PB Lustre Cray Sonexion scratch FS 1.5 PB Burst Buffer 3 Cori Xeon Phi KNL Nodes Cray XC40 system with 9,300 Intel Data Intensive Science Support Knights Landing compute nodes 10 Haswell processor cabinets (Phase 1) 68 cores / 96 GB DRAM / 16 GB HBM NVRAM Burst Buffer 1.5 PB, 1.5 TB/sec Support the entire Office of Science 30 PB of disk, >700 GB/sec I/O bandwidth research community Integrated with Cori Ha swell nodeson Begin to transition workload to energy Aries network for data / simulation / analysis on one system efficient architectures 4 NERSC Allocation of Computing Time NERSC hours in 300 millions DOE Mission Science 80% 300 Distributed by DOE SC program managers ALCC 10% Competitive awards run by DOE ASCR 2,400 Directors Discretionary 10% Strategic awards from NERSC 5 NERSC has ~100% utilization Important to get support PI Allocation (Hrs) Program and allocation from DOE program manager (L.
    [Show full text]
  • Cray XC40 Power Monitoring and Control for Intel Knights Landing Processors Steven J
    Cray XC40 Power Monitoring and Control for Intel Knights Landing Processors Steven J. Martin ([email protected]) Executive Summary ● Cray supports Advanced Power Management (APM) ● First APM features released for XC system in 2013 ● Cray works with customers, partners, and the broader HPC community to design and deliver new APM features ● APM updates in R/R and / or blades featuring Intel KNL processors ● Highly parallel blade telemetry gathering architecture ● Node-level power sampling at 1kHz in hardware ● Factory calibrated node-level power sensors ● Foundation for higher scan-rates for pm_counters ● Aggregate sensors for cpu and memory ● P-State and C-State limiting 2 CUG 2016 Copyright 2016 Cray Inc. Agenda ● Introduction to XC power monitoring and control ● Short introduction to XC power monitoring and control ● Cray XC enhanced HSS blade-level monitoring ● Motivation for enhanced blade-level monitoring ● Knights Landing Processor Daughter Card (KPDC) ● Enhances component level monitoring capabilities ● Updated Cray XC power monitoring and control interfaces ● New sensor data available in PMDB, /sys/cray/pm_counters, and RUR ● CAPMC updates 3 CUG 2016 Copyright 2016 Cray Inc. Introduction to XC power monitoring and control ● Cray PM on XC system ● First released in June of 2013 ● System Management Workstation (SMW) 7.0.UP03 ● Cray Linux Environment (CLE) 5.0.UP03 ● Power Management Database (PMDB) ● System Power Capping ● PM Counters /sys/cray/pm_counters ● Resource Utilization Reporting (RUR) (Sept 2013) ● Online documentation: ● http://docs.cray.com/books/S-0043-7204/S-0043-7204.pdf 4 CUG 2016 Copyright 2016 Cray Inc. Introduction to XC power monitoring and control ● Cray Advanced Platform Monitoring Control (CAPMC) ● Released in the fall of 2014 ● SMW 7.2.UP02 and CLE 5.2.UP02 ● Enabling Workload Managers (WLM) ● Secure, authenticated, off-smw, monitoring and control interfaces ● Supported by major WLM partners on XC systems ● CAPMC online documentation: ● http://docs.cray.com/books/S-2553-11/S-2553-11.pdf 5 CUG 2016 Copyright 2016 Cray Inc.
    [Show full text]
  • XC40 Programming Environment Form OS up to Modules
    Introduction to SahasraT RAVITEJA K Applications Analyst, Cray inc E Mail : [email protected] 1 1. Introduction to SahasraT 2. Cray Software stack 3. Compile applications on XC 4. Run applications on XC 2 What is Supercomputer? ● Broad term for one of the fastest computer currently available. ● Designed and built to solve difficult computational problems on extremely large jobs that could not be handled by no other types of computing systems. Characteristics : ● The ability to process instructions in parallel (Parallel processing) ● The ability to automatically recover from failures (Fault tolerance ) 3 4 What is SahasraT? ● SahasraT is Country’s first petaflops supercomputer. ● SahasraT : Sahasra means “Thousand” and T means “Teraflop” ● Built and designed by Cray ( XC40 Series ) 5 Cray XC System Building Blocks System Group Rank 3 Rank 2 Network Network Passive Electrical Active Chassis Network Optical Network Rank 1 Network 2 Cabinets Hundreds of 16 Compute Blades 6 Chassis Cabinets Compute Blade No Cables 384 Compute Up to 10s of Nodes 4 Compute Nodes 64 Compute Nodes thousands of nodes 6 Connecting nodes together: Aries Obviously, to function as a Cray XC40 Cabinets single supercomputer, the Compute Compute individual nodes must have node node method to communicate with Service Service each other. node node Compute Compute All nodes in the node node interconnected by the high Compute Compute speed, low latency Cray Aries node node Network. 7 XC Compute Blade 8 Cray XC Rank1 Backplane Network o Chassis with 16 compute blades o 128 Sockets o Inter-Aries communication over backplane o Per-Packet adaptive Routing 9 Types of nodes: Service nodes: • Its purpose is managing running jobs, but you can access using an interactive session.
    [Show full text]