DOCSLIB.ORG

Exampi17.Pdf

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Optimizing Point-to-Point Communication between AMPI Endpoints in Shared Memory Sam White & Laxmikant Kale UIUC ExaMPI 2017 Motivation • Exascale trends: • HW: increased node parallelism, decreased memory per thread • SW: applications themselves becoming more dynamic • How should applications and runtimes respond? • MPI+X (X=OpenMP, Kokkos, MPI, etc)? • New language? Legion, Charm++, HPX, etc? ExaMPI 2017 2 Motivation • MPI+X performance: • Either serialize around communication … • Or incur synchronization costs inside MPI • Semantic restrictions can help • What if we hoist threading into MPI? • Performant MPI+X often requires similar hoisting • Threaded MPIs: MPC-MPI, FG-MPI, TMPI, AMPI • Similar to the MPI Endpoints proposal ExaMPI 2017 3 Motivation • Questions: • Why is AMPI’s existing implementation not as fast as we might expect (vs process-based MPIs)? • What can be done to improve it? • More generally, what are the costs of process boundaries & kernel-assisted IPC? • What can MPI Endpoints offer in terms of pt2pt latency & bandwidth in shared memory? ExaMPI 2017 4 Overview • Introduction to AMPI • Execution model • Existing shared memory performance • Shared memory optimizations • Locality: intra-core vs intra-process • Size: small vs large messages • Conclusions & future work ExaMPI 2017 5 Adaptive MPI • AMPI is an MPI implementation on top of Charm++ • AMPI offers Charm++’s application-independent features to MPI programmers: • Overdecomposition • Communication/computation overlap • Dynamic load balancing • Online fault tolerance ExaMPI 2017 6 Execution Model • AMPI ranks are User-Level Threads (ULTs) • Can have multiple per core • Fast to context switch • Scheduled based on message delivery • Migratable across cores and nodes at runtime • For load balancing & online checkpoint/restart ExaMPI 2017 7 Execution Model Rank 0 Rank 1 MPI_Send() Rank 3 Rank 4 Rank 2 Scheduler Scheduler Core 0 Core 1 Node 0 ExaMPI 2017 8 Shared Memory • AMPI can be built in two different modes • Non-SMP: 1 process per core/hyperthread • SMP: 1 process per node/socket/NUMA domain • N worker threads, 1 dedicated communication thread per process ExaMPI 2017 9 AMPI Shared Memory • Many AMPI ranks can share the same OS process ExaMPI 2017 10 Performance Analysis • OSU Microbenchmarks v5.3: osu_latency, osu_bibw • Quartz (LLNL): Intel Xeon (Ivybridge) • MVAPICH2 2.2, Intel MPI 2018, OpenMPI 2.0.0 • Cori (NERSC): Intel Xeon (Haswell) • Cray MPI 6.7.0 • AMPI (6.8.0) vs AMPI-shm (6.9.0-beta) • P1: two ranks co-located on the same core • P2: two ranks on different cores, in the same OS process ExaMPI 2017 11 Existing Performance • Small message latency on Quartz OSU MPI Latency Benchmark on Quartz (LLNL) MVAPICH P2 AMPI P2 32 IMPI P2 AMPI P1 16 (us) OpenMPI P2 8 4 Latency 2 1-way 1 0.5 1 4 16 64 256 1024 4096 16384 65536 Message Size (Bytes) ExaMPI 2017 12 Existing Performance • Large message latency on Quartz OSU MPI Latency Benchmark on Quartz (LLNL) OSU MPI Latency Benchmark on Quartz (LLNL) 2048 131072 1024 65536 512 32768 256 16384 (us) 128 (us) 8192 64 4096 Latency 32 Latency 2048 16 1024 8 512 4 256 64 128 256 512 1024 2048 4096 4 8 16 32 64 Message Size (KB) Message Size (MB) ExaMPI 2017 13 Existing Performance • Bidirectional Bandwidth on Quartz ExaMPI 2017 14 Existing Performance • Small message latency on Cori-Haswell ExaMPI 2017 15 Existing Performance • Bidirectional Bandwidth on Cori-Haswell ExaMPI 2017 16 Performance Analysis • Breakdown of P1 time (us) per message on Quartz • Scheduling: Charm++ scheduler & ULT ctx • Memory copy: message payload movement • Other: AMPI message creation & matching ExaMPI 2017 17 Scheduling Overhead 1. Even for P1, all AMPI messages traveled thru Charm++’s scheduler • Use Charm++ inline tasks 2. ULT context switching overhead • Faster ULT ctx: Boost or QuickThreads ULTs 3. Avoid resuming threads without real progress • MPI_Waitall: keep track of “blocked on” reqs P1 0-B latency: 1.27 us -> 0.66 us ExaMPI 2017 18 Memory Copy Overhead • Q: Even with inline tasks, AMPI P1 performs poorly for large messages. Why? • A: Charm++ messaging semantics do not match MPI’s • In Charm++, messages are first class objects • Users pass ownership of messages to the runtime when sending and assume it when receiving • Only app’s that can reuse message objects in their data structures can perform “zero copy” transfers ExaMPI 2017 19 Memory Copy Overhead • To overcome Charm++ messaging semantics in shared memory, use a rendezvous protocol: • Recv’er performs direct (userspace) memcpy from sendbuf to recvbuf • Benefit: avoid intermediate copy • Cost: sender must suspend & be resumed upon copy completion P1 1-MB latency: 165 us -> 82 us ExaMPI 2017 20 Other Overheads • Sender-side: • Create a Charm++ message object & a request • Receiver-side: • Create a request, create matching queue entry, enqueue in unexpected_msgs or posted_requests • Solution: use memory pools for fixed-size, frequently-used objects P1 0-B latency: 0.66 us -> 0.54 us ExaMPI 2017 21 AMPI-shm Performance • Small message latency on Quartz • AMPI-shm P2 faster than other impl’s for 2+ KB ExaMPI 2017 22 AMPI-shm Performance • Large message latency on Quartz • AMPI-shm P2 fastest for all large messages, up to 2.33x faster than process-based MPIs for 32+ MB ExaMPI 2017 23 AMPI-shm Performance • Bidirectional bandwidth on Quartz • AMPI-shm can utilize full memory bandwidth • 26% higher peak, 2x bandwidth for 32+ MB than others OSU MPI Bidirectional Bandwidth Benchmark on Cori-Haswell (NERSC) STREAM copy 40000 Cray MPI 35000 AMPI AMPI-shm (MB/s) 30000 25000 Bandwidth 20000 15000 10000 Bidirectional 5000 ExaMPI 2017 24 4 16 64 256 1024 4096 16384 65536 Message Size (KB) AMPI-shm Performance • Small message latency on Cori-Haswell OSU MPI Latency Benchmark on Cori-Haswell (NERSC) 8 Cray MPI P2 AMPI-shm P2 4 AMPI-shm P1 (us) 2 1 Latency 0.5 0.25 1 4 16 64 256 1024 4096 16384 65536 Message Size (Bytes) ExaMPI 2017 25 AMPI-shm Performance • Large message latency on Cori-Haswell • AMPI-shm P2 is 47% faster than Cray MPI at 32+ MB ExaMPI 2017 26 AMPI-shm Performance • Bidirectional bandwidth on Cori-Haswell • Cray MPI on XPMEM performs similarly to AMPI-shm up to 16 MB OSU MPI Bidirectional Bandwidth Benchmark on Cori-Haswell (NERSC) STREAM copy 40000 Cray MPI 35000 AMPI AMPI-shm (MB/s) 30000 25000 Bandwidth 20000 15000 10000 Bidirectional 5000 4 16 64 256 1024 4096 16384 65536 ExaMPI 2017 27 Message Size (KB) Future Work • User-space shared memory optimizations for: Collectives, Derived Datatypes, RMA, and SHM • Testing with applications • Interprocess “zero copy” communication • Requires new Charm++ messaging semantics • Sender & recver both need completion callbacks • New messaging API under development: implementations for OFI, Verbs, uGNI, PAMI, MPI ExaMPI 2017 28 Summary • User-space communication offers portable intranode messaging performance • Lower latency: 1.5x-2.3x for large msgs • Higher bandwidth: 1.3x-2x for large msgs • Intermediate buffering unnecessary for medium/ large msgs • Shared-memory aware endpoints can provide lower latency & higher bandwidth for messaging within node ExaMPI 2017 29 This material is based in part upon work supported by the Department of Energy, National Nuclear Security Administration, under Award Number DE- NA0002374. ExaMPI 2017 30 Questions? Thank you ExaMPI 2017 31.
Recommended publications
  • Integration of CUDA Processing Within the C++ Library for Parallelism and Concurrency (HPX)

    Integration of CUDA Processing Within the C++ Library for Parallelism and Concurrency (HPX)

    1 Integration of CUDA Processing within the C++ library for parallelism and concurrency (HPX) Patrick Diehl, Madhavan Seshadri, Thomas Heller, Hartmut Kaiser Abstract—Experience shows that on today’s high performance systems the utilization of different acceleration cards in conjunction with a high utilization of all other parts of the system is difficult. Future architectures, like exascale clusters, are expected to aggravate this issue as the number of cores are expected to increase and memory hierarchies are expected to become deeper. One big aspect for distributed applications is to guarantee high utilization of all available resources, including local or remote acceleration cards on a cluster while fully using all the available CPU resources and the integration of the GPU work into the overall programming model. For the integration of CUDA code we extended HPX, a general purpose C++ run time system for parallel and distributed applications of any scale, and enabled asynchronous data transfers from and to the GPU device and the asynchronous invocation of CUDA kernels on this data. Both operations are well integrated into the general programming model of HPX which allows to seamlessly overlap any GPU operation with work on the main cores. Any user defined CUDA kernel can be launched on any (local or remote) GPU device available to the distributed application. We present asynchronous implementations for the data transfers and kernel launches for CUDA code as part of a HPX asynchronous execution graph. Using this approach we can combine all remotely and locally available acceleration cards on a cluster to utilize its full performance capabilities.
  • Bench - Benchmarking the State-Of- The-Art Task Execution Frameworks of Many- Task Computing

    Bench - Benchmarking the State-Of- The-Art Task Execution Frameworks of Many- Task Computing

    MATRIX: Bench - Benchmarking the state-of- the-art Task Execution Frameworks of Many- Task Computing Thomas Dubucq, Tony Forlini, Virgile Landeiro Dos Reis, and Isabelle Santos Illinois Institute of Technology, Chicago, IL, USA {tdubucq, tforlini, vlandeir, isantos1}@hawk.iit.edu Stanford University. Finally HPX is a general purpose C++ Abstract — Technology trends indicate that exascale systems will runtime system for parallel and distributed applications of any have billion-way parallelism, and each node will have about three scale developed by Louisiana State University and Staple is a orders of magnitude more intra-node parallelism than today’s framework for developing parallel programs from Texas A&M. peta-scale systems. The majority of current runtime systems focus a great deal of effort on optimizing the inter-node parallelism by MATRIX is a many-task computing job scheduling system maximizing the bandwidth and minimizing the latency of the use [3]. There are many resource managing systems aimed towards of interconnection networks and storage, but suffer from the lack data-intensive applications. Furthermore, distributed task of scalable solutions to expose the intra-node parallelism. Many- scheduling in many-task computing is a problem that has been task computing (MTC) is a distributed fine-grained paradigm that considered by many research teams. In particular, Charm++ [4], aims to address the challenges of managing parallelism and Legion [5], Swift [6], [10], Spark [1][2], HPX [12], STAPL [13] locality of exascale systems. MTC applications are typically structured as direct acyclic graphs of loosely coupled short tasks and MATRIX [11] offer solutions to this problem and have with explicit input/output data dependencies.
  • MVAPICH2 2.2 User Guide

    MVAPICH2 2.2 User Guide

    MVAPICH2 2.2 User Guide MVAPICH TEAM NETWORK-BASED COMPUTING LABORATORY DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING THE OHIO STATE UNIVERSITY http://mvapich.cse.ohio-state.edu Copyright (c) 2001-2016 Network-Based Computing Laboratory, headed by Dr. D. K. Panda. All rights reserved. Last revised: October 19, 2017 Contents 1 Overview of the MVAPICH Project1 2 How to use this User Guide?1 3 MVAPICH2 2.2 Features2 4 Installation Instructions 13 4.1 Building from a tarball .................................... 13 4.2 Obtaining and Building the Source from SVN repository .................. 13 4.3 Selecting a Process Manager................................. 14 4.3.1 Customizing Commands Used by mpirun rsh..................... 15 4.3.2 Using SLURM..................................... 15 4.3.3 Using SLURM with support for PMI Extensions ................... 15 4.4 Configuring a build for OFA-IB-CH3/OFA-iWARP-CH3/OFA-RoCE-CH3......... 16 4.5 Configuring a build for NVIDIA GPU with OFA-IB-CH3.................. 19 4.6 Configuring a build for Shared-Memory-CH3........................ 20 4.7 Configuring a build for OFA-IB-Nemesis .......................... 20 4.8 Configuring a build for Intel TrueScale (PSM-CH3)..................... 21 4.9 Configuring a build for Intel Omni-Path (PSM2-CH3).................... 22 4.10 Configuring a build for TCP/IP-Nemesis........................... 23 4.11 Configuring a build for TCP/IP-CH3............................. 24 4.12 Configuring a build for OFA-IB-Nemesis and TCP/IP Nemesis (unified binary) . 24 4.13 Configuring a build for Shared-Memory-Nemesis...................... 25 5 Basic Usage Instructions 26 5.1 Compile Applications..................................... 26 5.2 Run Applications....................................... 26 5.2.1 Run using mpirun rsh ...............................
  • HPX – a Task Based Programming Model in a Global Address Space

    HPX – a Task Based Programming Model in a Global Address Space

    HPX – A Task Based Programming Model in a Global Address Space Hartmut Kaiser1 Thomas Heller2 Bryce Adelstein-Lelbach1 [email protected] [email protected] [email protected] Adrian Serio1 Dietmar Fey2 [email protected] [email protected] 1Center for Computation and 2Computer Science 3, Technology, Computer Architectures, Louisiana State University, Friedrich-Alexander-University, Louisiana, U.S.A. Erlangen, Germany ABSTRACT 1. INTRODUCTION The significant increase in complexity of Exascale platforms due to Todays programming models, languages, and related technologies energy-constrained, billion-way parallelism, with major changes to that have sustained High Performance Computing (HPC) appli- processor and memory architecture, requires new energy-efficient cation software development for the past decade are facing ma- and resilient programming techniques that are portable across mul- jor problems when it comes to programmability and performance tiple future generations of machines. We believe that guarantee- portability of future systems. The significant increase in complex- ing adequate scalability, programmability, performance portability, ity of new platforms due to energy constrains, increasing paral- resilience, and energy efficiency requires a fundamentally new ap- lelism and major changes to processor and memory architecture, proach, combined with a transition path for existing scientific ap- requires advanced programming techniques that are portable across plications, to fully explore the rewards of todays and tomorrows multiple future generations of machines [1]. systems. We present HPX – a parallel runtime system which ex- tends the C++11/14 standard to facilitate distributed operations, A fundamentally new approach is required to address these chal- enable fine-grained constraint based parallelism, and support run- lenges.
  • Scalable and High Performance MPI Design for Very Large

    Scalable and High Performance MPI Design for Very Large

    SCALABLE AND HIGH-PERFORMANCE MPI DESIGN FOR VERY LARGE INFINIBAND CLUSTERS DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Sayantan Sur, B. Tech ***** The Ohio State University 2007 Dissertation Committee: Approved by Prof. D. K. Panda, Adviser Prof. P. Sadayappan Adviser Prof. S. Parthasarathy Graduate Program in Computer Science and Engineering c Copyright by Sayantan Sur 2007 ABSTRACT In the past decade, rapid advances have taken place in the field of computer and network design enabling us to connect thousands of computers together to form high-performance clusters. These clusters are used to solve computationally challenging scientific problems. The Message Passing Interface (MPI) is a popular model to write applications for these clusters. There are a vast array of scientific applications which use MPI on clusters. As the applications operate on larger and more complex data, the size of the compute clusters is scaling higher and higher. Thus, in order to enable the best performance to these scientific applications, it is very critical for the design of the MPI libraries be extremely scalable and high-performance. InfiniBand is a cluster interconnect which is based on open-standards and gaining rapid acceptance. This dissertation presents novel designs based on the new features offered by InfiniBand, in order to design scalable and high-performance MPI libraries for large-scale clusters with tens-of-thousands of nodes. Methods developed in this dissertation have been applied towards reduction in overall resource consumption, increased overlap of computa- tion and communication, improved performance of collective operations and finally designing application-level benchmarks to make efficient use of modern networking technology.
  • MVAPICH2 2.3 User Guide

    MVAPICH2 2.3 User Guide

    MVAPICH2 2.3 User Guide MVAPICH Team Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University http://mvapich.cse.ohio-state.edu Copyright (c) 2001-2018 Network-Based Computing Laboratory, headed by Dr. D. K. Panda. All rights reserved. Last revised: February 19, 2018 Contents 1 Overview of the MVAPICH Project1 2 How to use this User Guide?1 3 MVAPICH2 2.3 Features2 4 Installation Instructions 14 4.1 Building from a tarball................................... 14 4.2 Obtaining and Building the Source from SVN repository................ 14 4.3 Selecting a Process Manager................................ 15 4.3.1 Customizing Commands Used by mpirun rsh.................. 16 4.3.2 Using SLURM................................... 16 4.3.3 Using SLURM with support for PMI Extensions................ 16 4.4 Configuring a build for OFA-IB-CH3/OFA-iWARP-CH3/OFA-RoCE-CH3...... 17 4.5 Configuring a build for NVIDIA GPU with OFA-IB-CH3............... 20 4.6 Configuring a build to support running jobs across multiple InfiniBand subnets... 21 4.7 Configuring a build for Shared-Memory-CH3...................... 21 4.8 Configuring a build for OFA-IB-Nemesis......................... 21 4.9 Configuring a build for Intel TrueScale (PSM-CH3)................... 22 4.10 Configuring a build for Intel Omni-Path (PSM2-CH3)................. 23 4.11 Configuring a build for TCP/IP-Nemesis......................... 24 4.12 Configuring a build for TCP/IP-CH3........................... 25 4.13 Configuring a build for OFA-IB-Nemesis and TCP/IP Nemesis (unified binary)... 26 4.14 Configuring a build for Shared-Memory-Nemesis.................... 26 4.15 Configuration and Installation with Singularity....................
  • Automatic Handling of Global Variables for Multi-Threaded MPI Programs

    Automatic Handling of Global Variables for Multi-Threaded MPI Programs

    Automatic Handling of Global Variables for Multi-threaded MPI Programs Gengbin Zheng, Stas Negara, Celso L. Mendes, Laxmikant V. Kale´ Eduardo R. Rodrigues Department of Computer Science Institute of Informatics University of Illinois at Urbana-Champaign Federal University of Rio Grande do Sul Urbana, IL 61801, USA Porto Alegre, Brazil fgzheng,snegara2,cmendes,[email protected] [email protected] Abstract— necessary to privatize global and static variables to ensure Hybrid programming models, such as MPI combined with the thread-safety of an application. threads, are one of the most efficient ways to write parallel ap- In high-performance computing, one way to exploit multi- plications for current machines comprising multi-socket/multi- core nodes and an interconnection network. Global variables in core platforms is to adopt hybrid programming models that legacy MPI applications, however, present a challenge because combine MPI with threads, where different programming they may be accessed by multiple MPI threads simultaneously. models are used to address issues such as multiple threads Thus, transforming legacy MPI applications to be thread-safe per core, decreasing amount of memory per core, load in order to exploit multi-core architectures requires proper imbalance, etc. When porting legacy MPI applications to handling of global variables. In this paper, we present three approaches to eliminate these hybrid models that involve multiple threads, thread- global variables to ensure thread-safety for an MPI program. safety of the applications needs to be addressed again. These approaches include: (a) a compiler-based refactoring Global variables cause no problem with traditional MPI technique, using a Photran-based tool as an example, which implementations, since each process image contains a sep- automates the source-to-source transformation for programs arate instance of the variable.
  • Exascale Computing Project -- Software

    Exascale Computing Project -- Software

    Exascale Computing Project -- Software Paul Messina, ECP Director Stephen Lee, ECP Deputy Director ASCAC Meeting, Arlington, VA Crystal City Marriott April 19, 2017 www.ExascaleProject.org ECP scope and goals Develop applications Partner with vendors to tackle a broad to develop computer spectrum of mission Support architectures that critical problems national security support exascale of unprecedented applications complexity Develop a software Train a next-generation Contribute to the stack that is both workforce of economic exascale-capable and computational competitiveness usable on industrial & scientists, engineers, of the nation academic scale and computer systems, in collaboration scientists with vendors 2 Exascale Computing Project, www.exascaleproject.org ECP has formulated a holistic approach that uses co- design and integration to achieve capable exascale Application Software Hardware Exascale Development Technology Technology Systems Science and Scalable and Hardware Integrated mission productive technology exascale applications software elements supercomputers Correctness Visualization Data Analysis Applicationsstack Co-Design Programming models, Math libraries and development environment, Tools Frameworks and runtimes System Software, resource Workflows Resilience management threading, Data Memory scheduling, monitoring, and management and Burst control I/O and file buffer system Node OS, runtimes Hardware interface ECP’s work encompasses applications, system software, hardware technologies and architectures, and workforce
  • HPXMP, an Openmp Runtime Implemented Using

    HPXMP, an Openmp Runtime Implemented Using

    hpxMP, An Implementation of OpenMP Using HPX By Tianyi Zhang Project Report Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Science in the School of Engineering at Louisiana State University, 2019 Baton Rouge, LA, USA Acknowledgments I would like to express my appreciation to my advisor Dr. Hartmut Kaiser, who has been a patient, inspirable and knowledgeable mentor for me. I joined STEjjAR group 2 years ago, with limited knowledge of C++ and HPC. I would like to thank Dr. Kaiser to bring me into the world of high-performance computing. During the past 2 years, he helped me improve my skills in C++, encouraged my research project in hpxMP, and allowed me to explore the world. When I have challenges in my research, he always points things to the right spot and I can always learn a lot by solving those issues. His advice on my research and career are always valuable. I would also like to convey my appreciation to committee members for serving as my committees. Thank you for letting my defense be to an enjoyable moment, and I appreciate your comments and suggestions. I would thank my colleagues working together, who are passionate about what they are doing and always willing to help other people. I can always be inspired by their intellectual conversations and get hands-on help face to face which is especially important to my study. A special thanks to my family, thanks for their support. They always encourage me and stand at my side when I need to make choices or facing difficulties.
  • MVAPICH USER GROUP MEETING August 26, 2020

    MVAPICH USER GROUP MEETING August 26, 2020

    APPLICATION AND MICROBENCHMARK STUDY USING MVAPICH2 and MVAPICH2-GDR ON SDSC COMET AND EXPANSE MVAPICH USER GROUP MEETING August 26, 2020 Mahidhar Tatineni SDSC San Diego Supercomputer Center NSF Award 1928224 Overview • History of InfiniBand based clusters at SDSC • Hardware summaries for Comet and Expanse • Application and Software Library stack on SDSC systems • Containerized approach – Singularity on SDSC systems • Benchmark results on Comet, Expanse development system - OSU Benchmarks, NEURON, RAxML, TensorFlow, QE, and BEAST are presented. InfiniBand and MVAPICH2 on SDSC Systems Gordon (NSF) Trestles (NSF) 2012-2017 COMET (NSF) Expanse (NSF) 2011-2014 GordonS 2015-Current Production Fall 2020 (Simons Foundation) 2017- 2020 • 324 nodes, 10,368 cores • 1024 nodes, 16,384 cores • 1944 compute, 72 GPU, and • 728 compute, 52 GPU, and • 4-socket AMD Magny-Cours • 2-socket Intel Sandy Bridge 4 large memory nodes. 4 large memory nodes. • QDR InfiniBand • Dual Rail QDR InfiniBand • 2-socket Intel Haswell • 2-socket AMD EPYC 7742, • Fat Tree topology • 3-D Torus topology • FDR InfiniBand HDR100 InfiniBand • MVAPICH2 • 300TB of SSD storage - via • Fat Tree topology • GPU nodes with 4 V100 iSCSI over RDMA (iSER) • MVAPICH2, MVAPICH2-X, GPUs + NVLINK • MVAPICH2 (1.9, 2.1) with 3-D MVAPICH2-GDR • HDR200 Switches, Fat torus support • Leverage SRIOV for Virtual Tree topology with 3:1 Clusters oversubscription • MVAPICH2, MVAPICH2-X, MVAPICH2-GDR Comet: System Characteristics • Total peak flops ~2.76 PF • Hybrid fat-tree topology • Dell primary
  • Infiniband and 10-Gigabit Ethernet for Dummies

    Infiniband and 10-Gigabit Ethernet for Dummies

    The MVAPICH2 Project: Latest Developments and Plans Towards Exascale Computing Presentation at Mellanox Theatre (SC ‘18) by Dhabaleswar K. (DK) Panda The Ohio State University E-mail: [email protected] http://www.cse.ohio-state.edu/~panda Drivers of Modern HPC Cluster Architectures High Performance Interconnects - Accelerators / Coprocessors InfiniBand high compute density, high Multi-core Processors <1usec latency, 100Gbps Bandwidth> performance/watt SSD, NVMe-SSD, NVRAM >1 TFlop DP on a chip • Multi-core/many-core technologies • Remote Direct Memory Access (RDMA)-enabled networking (InfiniBand and RoCE) • Solid State Drives (SSDs), Non-Volatile Random-Access Memory (NVRAM), NVMe-SSD • Accelerators (NVIDIA GPGPUs and Intel Xeon Phi) • Available on HPC Clouds, e.g., Amazon EC2, NSF Chameleon, Microsoft Azure, etc. Summit Sunway TaihuLight Sierra K - Computer Network Based Computing Laboratory Mellanox Theatre SC’18 2 Parallel Programming Models Overview P1 P2 P3 P1 P2 P3 P1 P2 P3 Logical shared memory Memory Memory Memory Shared Memory Memory Memory Memory Shared Memory Model Distributed Memory Model Partitioned Global Address Space (PGAS) SHMEM, DSM MPI (Message Passing Interface) Global Arrays, UPC, Chapel, X10, CAF, … • Programming models provide abstract machine models • Models can be mapped on different types of systems – e.g. Distributed Shared Memory (DSM), MPI within a node, etc. • PGAS models and Hybrid MPI+PGAS models are gradually receiving importance Network Based Computing Laboratory Mellanox Theatre SC’18 3 Designing Communication Libraries for Multi-Petaflop and Exaflop Systems: Challenges Application Kernels/Applications Co-Design Middleware Opportunities and Programming Models Challenges MPI, PGAS (UPC, Global Arrays, OpenSHMEM), CUDA, OpenMP, across Various OpenACC, Cilk, Hadoop (MapReduce), Spark (RDD, DAG), etc.
  • The Future of PGAS Programming from a Chapel Perspective

    The Future of PGAS Programming from a Chapel Perspective

    The Future of PGAS Programming from a Chapel Perspective Brad Chamberlain PGAS 2015, Washington DC September 17, 2015 COMPUTE | STORE | ANALYZE (or:) Five Things You Should Do to Create a Future-Proof Exascale Language Brad Chamberlain PGAS 2015, Washington DC September 17, 2015 COMPUTE | STORE | ANALYZE Safe Harbor Statement This presentation may contain forward-looking statements that are based on our current expectations. Forward looking statements may include statements about our financial guidance and expected operating results, our opportunities and future potential, our product development and new product introduction plans, our ability to expand and penetrate our addressable markets and other statements that are not historical facts. These statements are only predictions and actual results may materially vary from those projected. Please refer to Cray's documents filed with the SEC from time to time concerning factors that could affect the Company and these forward-looking statements. COMPUTE | STORE | ANALYZE 3 Copyright 2015 Cray Inc. Apologies in advance ● This is a (mostly) brand-new talk ● My new talks tend to be overly text-heavy (sorry) COMPUTE | STORE | ANALYZE 4 Copyright 2015 Cray Inc. PGAS Programming in a Nutshell Global Address Space: ● permit parallel tasks to access variables by naming them ● regardless of whether they are local or remote ● compiler / library / runtime will take care of communication OK to access i, j, and k wherever they live k = i + j; i k j k k k k 0 1 2 3 4 Images / Threads / Locales / Places