DOCSLIB.ORG

Towards Exascale Computing with the Atmospheric Model NUMA

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Towards Exascale Computing with the Atmospheric Model NUMA Towards Exascale Computing with the Atmospheric Model NUMA Andreas Müller, Daniel S. Abdi, Michal Kopera, Lucas Wilcox, Francis X. Giraldo Department of Applied Mathematics Naval Postgraduate School, Monterey (California), USA Tim Warburton Virginia Tech Blacksburg (Virginia), USA • NUMA = Non-hydrostatic Unified Model of the Atmosphere • dynamical core inside the Navy’s next generation weather prediction system NEPTUNE (Navy’s Environment Prediction sysTem Using the Numa Engine) • developed by Prof. Francis X. Giraldo and generations of postdocs Goal • NOAA: HIWPP project plan: Goal for 2020: ~ 3km – 3.5km global resolution within operational requirements Goal • NOAA: HIWPP project plan: Goal for 2020: ~ 3km – 3.5km global resolution within operational requirements • Achieved with NUMA: baroclinic wave test case at 3.0km within 4.15 minutes per one day forecast on supercomputer Mira double precision, no shallow atmosphere approx., arbitrary terrain, IMEX in the vertical Goal • NOAA: HIWPP project plan: Goal for 2020: ~ 3km – 3.5km global resolution within operational requirements • Achieved with NUMA: baroclinic wave test case at 3.0km within 4.15 minutes per one day forecast on supercomputer Mira double precision, no shallow atmosphere approx., arbitrary terrain, IMEX in the vertical • Expect: 2km by doing more optimizations Communication between processors 4th order Finite Difference Communication between processors 4th order Finite Difference CPU 1 CPU 2 Communication between processors compact stencil methods 4th order Finite Difference (like in NUMA) CPU 1 CPU 2 Communication between processors compact stencil methods 4th order Finite Difference (like in NUMA) CPU 1 CPU 2 CPU 1 CPU 2 Communication between processors compact stencil methods 4th order Finite Difference (like in NUMA) NUMA should scale extremely well CPU 1 CPU 2 CPU 1 CPU 2 Fastest Supercomputers of the World according to top500.org # NAME COUNTRY TYPE PEAK (PFLOPS) 1 TaihuLight China Sunway 125.4 2 Tianhe-2 China Intel Xeon Phi 54.9 3 Titan USA NVIDIA GPU 27.1 4 Sequoia USA IBM BlueGene 20.1 5 K computer Japan Fujitsu CPU 11.2 6 Mira USA IBM BlueGene 10.0 1 PFlops = 1015 floating point ops. per sec. Fastest Supercomputers of the World according to top500.org # NAME COUNTRY TYPE PEAK (PFLOPS) 1 TaihuLight China Sunway 125.4 2 Tianhe-2 China Intel Xeon Phi 54.9 3 Titan USA NVIDIA GPU 27.1 4 Sequoia USA IBM BlueGene 20.1 5 K computer Japan Fujitsu CPU 11.2 6 Mira USA IBM BlueGene 10.0 1 PFlops = 1015 floating point ops. per sec. overview • Mira: Blue Gene strategy: optimize one setup for one specific computer • Titan: GPUs strategy: keep all options, portability on CPUs and GPUs • Intel Knights Landing Mira: Blue Gene optimize one setup for this specific computer Mira Titan KNL Optimization of main computations runtime of computational kernel on 30720 CPUs 1 MPI process per core, p=6 11.8 sec. p=3 9.4 sec. 4 MPI processes per core 4 sec. Lapack 3.9 sec. rewritten for optimized compiler vectorization 2.1 sec. BG/Q vector intrinsics 1 sec. OpenMP 1.1 sec. current version 0.7 sec. optimized intensity 0.5 sec. measured theory 0 3 6 9 12 more details: see Mueller et al. https://arxiv.org/abs/1511.01561 Mira Titan KNL Measurements: roofline plot rising bubble test case on Mira 103 attainable timeloop createrhs blue: peak GFlops entire timeloop 102 red: main computational peak bandwidth kernel 5.5⨉ GFlops per node 9.8⨉ 101 GFlops/s per node per GFlops/s data points: different optimization stages 100 10-1 100 101 102 arithmeticoperational intensity intensity (Flops/Bytes) (Flops/Byte) Mira Titan KNL Measurements: roofline plot rising bubble test case on Mira 103 attainable timeloop createrhs blue: peak GFlops entire timeloop 102 red: main theoretical computational peak bandwidth optimum kernel 5.5⨉ GFlops per node 9.8⨉ 101 GFlops/s per node per GFlops/s data points: different optimization stages 100 10-1 100 101 102 arithmeticoperational intensity intensity (Flops/Bytes) (Flops/Byte) Mira Titan KNL Strong scaling with NUMA 1.8 billion grid points (3.0km horizontal, 31 levels vertical) baroclinic instability:baroclinic p=3, instability, 3.2km resolution,p=3, 3.0km horizontal 31 dof resolutionin the vertical 350 450 total runtime 400computation time 300 350 250 300 200 250 99.1% 150 200 strong scaling 150 100 model days per wallclock day efficiency model days / wall clock day 100 50 50 3.14M threads 0 0 0 0 0.50.5 1 1 1.5 1.5 2 2 2.5 32.5 3.5 3 4 numbernumber of threads of threads #106 #106 Mira Titan KNL 12.1% of theoretical peak (flops) baroclinic instability, p=3, 3.0km horizontal resolution total 12 filter createrhs IMEX 10 1.21 PFlops on 8 entire Mira 6 % of peak flops 4 2 0 104 105 106 107 number of threads Mira Titan KNL Where are we heading? dynamics within 4.5 minutes runtime per one day forecast baroclinic instability, p=3, 31 vertical layers, 128 columns per node 107 106 current version 105 number of threads 104 103 1 2 3 5 10 20 30 50 100 resolution in km Mira Titan KNL Where are we heading? dynamics within 4.5 minutes runtime per one day forecast baroclinic instability, p=3, 31 vertical layers, 128 columns per node 107 106 current version 105 Mira number of threads fully 104 optimized 103 1 2 3 5 10 20 30 50 100 resolution in km Mira Titan KNL Where are we heading? dynamics within 4.5 minutes runtime per one day forecast baroclinic instability, p=3, 31 vertical layers, 128 columns per node 107 106 Aurora 2018 ? 2018 Aurora current version 105 Mira number of threads fully 104 optimized 103 1 2 3 5 10 20 30 50 100 resolution in km Mira Titan KNL Titan: GPUs keep all options, portability on CPUs and GPUs Mira Titan KNL Titan 18,688 CPU nodes (16 cores each) network 18,688 NVIDIA GPUs (2,688 CUDA cores each) Mira Titan KNL OCCA2: unified threading model (slide:OCCA2: courtesy of Lucas unified Wilcox and threading Tim Warburton) model PortabilityOCCA2: & extensibility: unified device independent threading kernel language model (or OpenCLPortability / CUDA) & extensibility and: nativedevice independent host APIs. kernel language and native host APIs. Host Language pThreads # ++ CPU Processors Xeon Phi Intel COI AMD GPUs Kernel Language FPGAs OCCA OKL CUDA-x86 NVIDIA GPUs Floopy Available at: https://github.com/tcew/OCCA2Back ends Devices 23 Mira Titan KNL Floopy / OCCA2 Code Floopy/OCCA2 code (slide: courtesy of Lucas Wilcox and Tim Warburton) Mira Titan KNL Floopy / OCCA2 Code Floopy / OCCA2 Code Floopy/OCCA2 code (slide: courtesy of Lucas Wilcox and Tim Warburton) Mira Titan KNL Introduction PortingNUMAtomany-core Scalability Test cases Conclusions NUMA - Scalability on GPU cluster AWeak 90% scaling weak scaling up to 16,384 efficiency GPUs on (88% Titan of Supercomputer Titan) using 16384using 900 elements K20x per GPUs GPU, polynomial order 8 (up to 10.7 billion grid points) 100 90 80 70 60 50 90% weak scaling 40 Efficiency(%) efficiency using DG with 30 communication overlap 20 CG nooverlap 10 DG nooverlap DG overlap 0 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 No of GPUs Abdi et al.: Acceleration of the Implicit-Explicit Non-hydrostatic Unified Model of the Atmosphere (NUMA) on Manycore Processors, IJHPCA (submitted, 2016) Figure:pre-printGPU and scalability more information: see study http://frankgiraldo.wixsite.com/mysite/numa on Titan supercomputer 24/32 Mira Titan KNL Abdi, Wilcox, Warburton and Giraldo roofline plot: single precision 17 on the NVIDIA K20X GPU on Titan 4 250 10 Horizontal volume kernel Horizontal volume kernel Vertical volume kernel Vertical volume kernel 3520 GFLOPS/s Update kernel Update kernel 200 Project kernel Project kernel 3 10 Roofline 150 12 2 10 208 GB/s GB/s 100 GFLOPS/s 4 convergence order 1 10 50 2 0 0 10 1 2 3 4 5 6 7 8 9 10 11 −1 0 1 2 N 10 10 10 10 p GFLOPS/GB (a) SP-CG kernels performance Mira Titan KNL (b) DP-CG kernels performance (c) SP-DG kernels performance (d) DP-DG kernels performance Figure 4. Performance of individual kernels: The efficiency of our kernels are tested on a mini-app developed for this purpose. The FLOPS and byte for this test are counted manually. The volume kernel, that is split into two (horizontal + vertical), has the highest rate of FLOPS/s. The time-step update kernel has the highest bandwidth usage at 208GB/s. The Single Precision (SP) and Double Precision (DP) performance of the main kernels in CG and DG are shown in-terms of GFLOPS/s, GB/s and roofline plots to illustrate their efficiency. The GPU is a Tesla K20c. time to make a day’s weather forecast. For this reason, we a given simulation time limit for two resolutions: a coarse also conducted strong scaling tests, shown in Fig. 7, on a grid of 13km resolution and a fine grid of 3km resolution. global scale simulation problem described in Sec. 8.5. Our goal here is to determine the number of GPUs required for Prepared using sagej.cls Abdi, Wilcox, Warburton and Giraldo 17 roofline plot: double precision on the NVIDIA K20X GPU on Titan (a) SP-CG kernels performance 4 250 10 Horizontal volume kernel Horizontal volume kernel Vertical volume kernel Vertical volume kernel Update kernel Update kernel 200 Project kernel Project kernel 1170 GFLOPS/s 3 10 Roofline 150 2 10 208 GB/s 12 GB/s 100 GFLOPS/s 4 1 convergence 10 50 order 2 0 0 10 1 2 3 4 5 6 7 8 9 10 11 −1 0 1 2 N 10 10 10 10 p GFLOPS/GB (b) DP-CG kernels performance Mira Titan KNL (c) SP-DG kernels performance (d) DP-DG kernels performance Figure 4.
Load more

Recommended publications
  • Ushering in a New Era: Argonne National Laboratory & Aurora
    Ushering in a New Era Argonne National Laboratory’s Aurora System April 2015 ANL Selects Intel for World’s Biggest Supercomputer 2-system CORAL award extends IA leadership in extreme scale HPC Aurora Argonne National Laboratory >180PF Trinity NNSA† April ‘15 Cori >40PF NERSC‡ >30PF July ’14 + Theta Argonne National Laboratory April ’14 >8.5PF >$200M ‡ Cray* XC* Series at National Energy Research Scientific Computing Center (NERSC). † Cray XC Series at National Nuclear Security Administration (NNSA). 2 The Most Advanced Supercomputer Ever Built An Intel-led collaboration with ANL and Cray to accelerate discovery & innovation >180 PFLOPS (option to increase up to 450 PF) 18X higher performance† >50,000 nodes Prime Contractor 13MW >6X more energy efficient† 2018 delivery Subcontractor Source: Argonne National Laboratory and Intel. †Comparison of theoretical peak double precision FLOPS and power consumption to ANL’s largest current system, MIRA (10PFs and 4.8MW) 3 Aurora | Science From Day One! Extreme performance for a broad range of compute and data-centric workloads Transportation Biological Science Renewable Energy Training Argonne Training Program on Extreme- Scale Computing Aerodynamics Biofuels / Disease Control Wind Turbine Design / Placement Materials Science Computer Science Public Access Focus Areas Focus US Industry and International Co-array Fortran Batteries / Solar Panels New Programming Models 4 Aurora | Built on a Powerful Foundation Breakthrough technologies that deliver massive benefits Compute Interconnect File System 3rd Generation 2nd Generation Intel® Xeon Phi™ Intel® Omni-Path Intel® Lustre* Architecture Software >17X performance† >20X faster† >3X faster† FLOPS per node >500 TB/s bi-section bandwidth >1 TB/s file system throughput >12X memory bandwidth† >2.5 PB/s aggregate node link >5X capacity† bandwidth >30PB/s aggregate >150TB file system capacity in-package memory bandwidth Integrated Intel® Omni-Path Architecture Processor code name: Knights Hill Source: Argonne National Laboratory and Intel.
    [Show full text]
  • Roofline Model Toolkit: a Practical Tool for Architectural and Program
    Roofline Model Toolkit: A Practical Tool for Architectural and Program Analysis Yu Jung Lo, Samuel Williams, Brian Van Straalen, Terry J. Ligocki, Matthew J. Cordery, Nicholas J. Wright, Mary W. Hall, and Leonid Oliker University of Utah, Lawerence Berkeley National Laboratory {yujunglo,mhall}@cs.utah.edu {swwilliams,bvstraalen,tjligocki,mjcordery,njwright,loliker}@lbl.gov Abstract. We present preliminary results of the Roofline Toolkit for multicore, manycore, and accelerated architectures. This paper focuses on the processor architecture characterization engine, a collection of portable instrumented micro benchmarks implemented with Message Passing Interface (MPI), and OpenMP used to express thread-level paral- lelism. These benchmarks are specialized to quantify the behavior of dif- ferent architectural features. Compared to previous work on performance characterization, these microbenchmarks focus on capturing the perfor- mance of each level of the memory hierarchy, along with thread-level parallelism, instruction-level parallelism and explicit SIMD parallelism, measured in the context of the compilers and run-time environments. We also measure sustained PCIe throughput with four GPU memory managed mechanisms. By combining results from the architecture char- acterization with the Roofline model based solely on architectural spec- ifications, this work offers insights for performance prediction of current and future architectures and their software systems. To that end, we in- strument three applications and plot their resultant performance on the corresponding Roofline model when run on a Blue Gene/Q architecture. Keywords: Roofline, Memory Bandwidth, CUDA Unified Memory 1 Introduction The growing complexity of high-performance computing architectures makes it difficult for users to achieve sustained application performance across different architectures.
    [Show full text]
  • Characterizing and Understanding HPC Job Failures Over the 2K-Day Life of IBM Bluegene/Q System
    Characterizing and Understanding HPC Job Failures over The 2K-day Life of IBM BlueGene/Q System Sheng Di,∗ Hanqi Guo,∗ Eric Pershey,∗ Marc Snir,y Franck Cappello∗y ∗Argonne National Laboratory, IL, USA [email protected], [email protected], [email protected], [email protected] yUniversity of Illinois at Urbana-Champaign, IL, USA [email protected] Abstract—An in-depth understanding of the failure features of Blue Joule (UK), also adopt the same Blue Gene/Q system HPC jobs in a supercomputer is critical to the large-scale system architecture and they are still in operation. maintenance and improvement of the service quality for users. In Studying the job failure features in a large-scale system is this paper, we investigate the features of hundreds of thousands of jobs in one of the most powerful supercomputers, the IBM nontrivial in that it involves numerous messages logged from Blue Gene/Q Mira, based on 2001 days of observations with a across multiple data sources and many messages are heavily total of over 32.44 billion core-hours. We study the impact of the duplicated [2]. In our work, we performed a joint analysis system’s events on the jobs’ execution in order to understand the by leveraging four different data sources: the reliability, avail- system’s reliability from the perspective of jobs and users. The ability, and serviceability (RAS) log; task execution log; job characterization involves a joint analysis based on multiple data sources, including the reliability, availability, and serviceability scheduling log; and I/O behavior log. The RAS log is the most (RAS) log; job scheduling log; the log regarding each job’s important system log related to system reliability such as node physical execution tasks; and the I/O behavior log.
    [Show full text]
  • June 2012 | TOP500 Supercomputing Sites
    PROJECT LISTS STATISTICS RESOURCES NEWS CONTACT SUBMISSIONS LINKS HOME Home Lists June 2012 MANNHEIM, Germany; BERKELEY, Calif.; and KNOXVILLE, Tenn.—For the first time since November 2009, a United Contents States supercomputer sits atop the TOP500 list of the world’s top supercomputers. Named Sequoia, the IBM BlueGene/Q system installed at the Department of Energy’s Lawrence Livermore National Laboratory achieved an impressive 16.32 Release petaflop/s on the Linpack benchmark using 1,572,864 cores. Top500 List Sequoia is also one of the most energy efficient systems on Press Release (PDF) the list, which will be released Monday, June 18, at the 2012 Press Release International Supercomputing Conference in Hamburg, Germany. This will mark the 39th edition of the list, which is List highlights compiled twice each year. Performance Development On the latest list, Fujitsu’s “K Computer” installed at the RIKEN Related Files Advanced Institute for Computational Science (AICS) in Kobe, Japan, is now the No. 2 system with 10.51 Pflop/s on the TOP500 List (XML) Linpack benchmark using 705,024 SPARC64 processing TOP500 List (Excel) A 1.044 persone piace cores. The K Computer held the No. 1 spot on the previous TOP500 Poster Mi piace two lists. questo elemento. Di' che Poster in PDF piace anche a te, prima di The new Mira supercomputer, an IBM BlueGene/Q system at tutti i tuoi amici. Argonne National Laboratory in Illinois, debuted at No. 3, with Drilldown 8.15 petaflop/s on the Linpack benchmark using 786,432 Performance Development cores. The other U.S.
    [Show full text]
  • Supercomputers – Prestige Objects Or Crucial Tools for Science and Industry?
    Supercomputers – Prestige Objects or Crucial Tools for Science and Industry? Hans W. Meuer a 1, Horst Gietl b 2 a University of Mannheim & Prometeus GmbH, 68131 Mannheim, Germany; b Prometeus GmbH, 81245 Munich, Germany; This paper is the revised and extended version of the Lorraine King Memorial Lecture Hans Werner Meuer was invited by Lord Laird of Artigarvan to give at the House of Lords, London, on April 18, 2012. Keywords: TOP500, High Performance Computing, HPC, Supercomputing, HPC Technology, Supercomputer Market, Supercomputer Architecture, Supercomputer Applications, Supercomputer Technology, Supercomputer Performance, Supercomputer Future. 1 e-mail: [email protected] 2 e-mail: [email protected] 1 Content 1 Introduction ..................................................................................................................................... 3 2 The TOP500 Supercomputer Project ............................................................................................... 3 2.1 The LINPACK Benchmark ......................................................................................................... 4 2.2 TOP500 Authors ...................................................................................................................... 4 2.3 The 39th TOP500 List since 1993 .............................................................................................. 5 2.4 The 39th TOP10 List since 1993 ...............................................................................................
    [Show full text]
  • An Analysis of System Balance and Architectural Trends Based on Top500 Supercomputers
    ORNL/TM-2020/1561 An Analysis of System Balance and Architectural Trends Based on Top500 Supercomputers Hyogi Sim Awais Khan Sudharshan S. Vazhkudai Approved for public release. Distribution is unlimited. August 11, 2020 DOCUMENT AVAILABILITY Reports produced after January 1, 1996, are generally available free via US Department of Energy (DOE) SciTech Connect. Website: www.osti.gov/ Reports produced before January 1, 1996, may be purchased by members of the public from the following source: National Technical Information Service 5285 Port Royal Road Springfield, VA 22161 Telephone: 703-605-6000 (1-800-553-6847) TDD: 703-487-4639 Fax: 703-605-6900 E-mail: [email protected] Website: http://classic.ntis.gov/ Reports are available to DOE employees, DOE contractors, Energy Technology Data Ex- change representatives, and International Nuclear Information System representatives from the following source: Office of Scientific and Technical Information PO Box 62 Oak Ridge, TN 37831 Telephone: 865-576-8401 Fax: 865-576-5728 E-mail: [email protected] Website: http://www.osti.gov/contact.html This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal lia- bility or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or rep- resents that its use would not infringe privately owned rights. Refer- ence herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not nec- essarily constitute or imply its endorsement, recommendation, or fa- voring by the United States Government or any agency thereof.
    [Show full text]
  • ALCF Newsbytes Argonne Leadership Computing Facility Argonne National Laboratory
    ALCF Newsbytes Argonne Leadership Computing Facility Argonne National Laboratory Volume 2, Issue 3 | Summer 2012 Table of Contents Program Awards 247 Million Hours Program Awards 247 Million Hours .......... 1 of ALCF Supercomputer Time Mira Ranks Third on TOP500 .................... 2 Rapid Procedure Unites Quantum Chemistry with Artificial Intelligence ........ 4 Nine research projects have been awarded 247 million Lithium-Air: The “Mount Everest” processor hours of computing time at the Argonne of Batteries for Electric Vehicles ............... 5 Spotlight: Steve Crusan ............................. 6 Leadership Computing Facility (ALCF) through the ALCF Systems Shine on Green500 ............. 7 U.S. Department of Energy’s (DOE) initiative—the ASCR Mira Shares Graph 500 Top Spot ............... 8 Leadership Computing Challenge (ALCC). Chosen through a Leap to Petascale Workshop in Review ..... 8 peer-review process, the projects selected reflect areas of special interest to the DOE: energy, national emergencies, and broadening the community of researchers capable of Events of Interest Argonne’s Energy Showcase using leadership-class computing resources. Saturday, September 15, 2012 9 a.m. – 4 p.m. Argonne National Laboratory will open its gates to the community for a day of discovery ALCC awards of compute time on Intrepid (the ALCF’s Blue Gene/P) become available and fun for the whole family. The event is free in July for the following recipients: and open to the public. Advance registration is required as attendance will be limited.
    [Show full text]
  • Optimizing the Performance of Reactive Molecular Dynamics Simulations for Multi-Core Architectures
    Optimizing the Performance of Reactive Molecular Dynamics Simulations for Multi-Core Architectures Hasan Metin Aktulga1, Christopher Knight2, Paul Coffman2, Kurt A. O’Hearn1, Tzu-Ray Shan3, and Wei Jiang2 1Michigan State University 2Argonne Leadership Computing Facility 3Sandia National Laboratories (currently, Materials Design Inc.) June 23, 2017 Abstract Reactive molecular dynamics simulations are computationally demanding. Reaching spatial and temporal scales where interesting scientific phenomena can be observed requires efficient and scalable implementations on modern hardware. In this paper, we focus on optimizing the performance of the widely used LAMMPS/ReaxC package for multi-core architectures. As hy- brid parallelism allows better leverage of the increasing on-node parallelism, we adopt thread parallelism in the construction of bonded and nonbonded lists, and in the computation of com- plex ReaxFF interactions. To mitigate the I/O overheads due to large volumes of trajectory data produced and to save users the burden of post-processing, we also develop a novel in-situ tool for molecular species analysis. We analyze the performance of the resulting ReaxC-OMP package on Mira, an IBM Blue Gene/Q supercomputer. For PETN systems of sizes ranging from 32 thousand to 16.6 million particles, we observe speedups in the range of 1.5-4.5 . We observe sustained performance improvements for up to 262,144 cores (1,048,576 processes)× of Mira and a weak scaling efficiency of 91.5% in large simulations containing 16.6 million particles. The in-situ molecular species analysis tool incurs only insignificant overheads across various system sizes and run configurations. 1 Introduction arXiv:1706.07772v1 [cs.DC] 23 Jun 2017 Molecular Dynamics (MD) simulations have become an increasingly important computational tool for a range of scientific disciplines including, but not limited to, chemistry, biology, and materials science.
    [Show full text]
  • Early Evaluation of the Cray XC40 Xeon Phi System 'Theta' at Argonne
    Early Evaluation of the Cray XC40 Xeon Phi System ‘Theta’ at Argonne Scott Parker, Vitali Morozov, Sudheer Chunduri, Kevin Harms, Chris Knight, and Kalyan Kumaran Argonne National Laboratory, Argonne, IL, USA fsparker, morozov, chunduri, harms, knightc, [email protected] Abstract—The Argonne Leadership Computing Facility The remainder of the paper is organized as follows, Sec- (ALCF) has recently deployed a nearly 10 PF Cray XC40 system tion II describes the Theta system architecture. Section III named Theta. Theta is nearly identical in performance capacity discusses the performance of KNL’s microarchitecture, mem- to the ALCF’s current IBM Blue Gene/Q system Mira and represents the first step in a path from Mira to a much larger ory, computation and communication subsystems using a set next generation Intel-Cray system named Aurora to be deployed of micro-benchmarks and Section IV covers the performance in 2018. Theta serves as both a production resource for scientific of several key scientific applications. computation and a platform for transitioning applications to run efficiently on the Intel Xeon Phi architecture and the Dragonfly II. THETA/XC40 ARCHITECTURE topology based network. This paper presents an initial evaluation of Theta. The Theta The Cray XC40 is the latest in the Cray line of supercom- system architecture will be described along with the results of puters. The ALCF Theta XC40 system has a nearly 10 PF peak benchmarks characterizing the performance at the processor core, with key hardware specifications shown in Table I. memory, and network component levels. In addition, benchmarks are used to evaluate the performance of important runtime TABLE I components such as MPI and OpenMP.
    [Show full text]
  • Improving Batch Scheduling on Blue Gene/Q by Relaxing 5D Torus Network Allocation Constraints
    Improving Batch Scheduling on Blue Gene/Q by Relaxing 5D Torus Network Allocation Constraints Zhou Zhou, Xu Yang, Zhiling Lan Paul Rich,∗ Wei Tang,y Vitali Morozov,∗ Narayan Desai,yz Department of Computer Science ∗Argonne Leadership Computing Facility Illinois Institute of Technology yMathematics and Computer Science Division Chicago, IL 60616, USA Argonne National Laboratory, Argonne, IL 60439, USA fzzhou1, [email protected], [email protected] zEricsson Inc. 200 Holger Way, San Jose, CA 95134, USA ∗[email protected], [email protected] ∗[email protected], [email protected] Abstract—As systems scale toward exascale, many resources resources is increasing. In the near future, management will become increasingly constrained. While some of these of shared resources such as network and bandwidth will resources have historically been explicitly allocated, many— become increasingly critical. such as network bandwidth, I/O bandwidth, or power—have not. As systems continue to evolve, we expect many such Torus-based networks are commonly used in high-end resources to become explicitly managed. This change will pose supercomputers because of their linear scaling on per-node critical challenges to resource management and job scheduling. cost as well as their competitive communication perfor- In this paper, we explore the potentiality of relaxing network mance. The IBM Blue Gene/L, Blue Gene/P [2] [3], and allocation constraints for Blue Gene systems. Our objective Cray XT systems [4] use a 3D torus network for node is to improve the batch scheduling performance, where the partition-based interconnect architecture provides a unique communication. Blue Gene/Q (BG/Q) has its nodes elec- opportunity to explicitly allocate network resources to jobs.
    [Show full text]
  • Application Power Profiling on IBM Blue Gene/Q
    Application Power Profiling on IBM Blue Gene/Q Sean Wallace,∗y Venkatram Vishwanath,y Susan Coghlan,y John Tramm,y Zhiling Lan,∗ and Michael E. Papkayz ∗Illinois Institute of Technology, Chicago, IL, USA yArgonne National Laboratory, Argonne, IL, USA zNorthern Illinois University, DeKalb, IL, USA [email protected], [email protected], [email protected], [email protected], [email protected], and [email protected] Abstract—The power consumption of state of the art supercom- in different rankings[4]. Therefore, the analysis of power puters, because of their complexity and unpredictable workloads, consumption on state-of-the-art supercomputers is imperative is extremely difficult to estimate. Accurate and precise results, as to better understand how these new systems differ from their are now possible with the latest generation of supercomputers, are therefore a welcome addition to the landscape. Only recently predecessors and in which direction key characteristics are have end users been afforded the ability to access the power heading. consumption of their applications. However, just because it’s Fortunately, hardware manufacturers have begun deploying possible for end users to obtain this data does not mean it’s various sensors on HPC systems to collect power-related data a trivial task. This emergence of new data is therefore not only and provide relatively easy access to the data store. In this understudied, but also not fully understood. In this paper, we provide detailed power consumption analysis work, we will describe two power monitoring capabilities of microbenchmarks running on Argonne’s latest generation of deployed on the Blue Gene/Q (BG/Q): one is an environ- IBM Blue Gene supercomputers, Mira, a Blue Gene/Q system.
    [Show full text]
  • No Slide Title
    20 Jahre TOP500 mit einem Ausblick auf neuere Entwicklungen Hans Meuer Prometeus GmbH & Universität Mannheim [email protected] ZKI Herbsttagung in Leipzig 11. - 12. September 2012 page 1 Outline Mannheim Supercomputer Statistics 1986 - 1992 & TOP500 Project Start in 1993 39 TOP500 Lists / Highlights of the 39th List and Developments Competition between Countries, Manufacturers and Sites Linpack Pros and Cons & Alternative Supercomputer Rankings HPC & Cloud Computing Summary ZKI Herbsttagung in Leipzig 2012 page 2 Mannheim Supercomputer Statistics & TOP500 Project Start in 1993 Mannheim Supercomputer Statistics 1986 – 1992 Presented at ISC’86 – ISC’92/Mannheim Supercomputer Seminars Counting of Supercomputers in the World 530 systems in the year 1992 ZKI Herbsttagung in Leipzig 2012 page 3 Mannheim Supercomputer Statistics & TOP500 Project Start in 1993 Manufacturer Share 100% NEC Hitachi 80% Fujitsu 60% CDC Cray 40% 20% 0% 1986 1987 1988 1989 1990 1991 1992 ZKI Herbsttagung in Leipzig 2012 page 4 Mannheim Supercomputer Statistics & TOP500 Project Start in 1993 Countries Share 100% Others 90% Others Europe 80% 70% Germany 60% France 50% UK 40% Japan 30% USA 20% 10% 0% 1986 1987 1988 1989 1990 1991 1992 ZKI Herbsttagung in Leipzig 2012 page 5 Mannheim Supercomputer Statistics & TOP500 Project Start in 1993 Deficits of the Mannheim Supercomputer Statistics Not very reliable Vector Supercomputers only – „numerical“/ mainly „scientific and engineering“ applications Definition of „Supercomputer“ necessary ZKI Herbsttagung in Leipzig 2012 page 6 Mannheim Supercomputer Statistics & TOP500 Project Start in 1993 Top500 Procedure Listing the 500 most powerful computers in the world Yardstick: Rmax of Linpack Solve Ax=b, dense problem, matrix is random Update twice a year: • ISC’xy in June in Germany • SC’xy in November in the U.S.
    [Show full text]