DOCSLIB.ORG

PVM Parallel Virtual Machine

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

PVM Parallel Virtual Machine Scientic and Engineering Computation Janusz Kowalik Editor DataParal lel Programming on MIMD Computers by Philip J Hatcher and Michael J Quinn Unstructured Scientic Computation on Scalable Multiprocessors Mehrotra Jo el Saltz and Rob ert Voigt edited by Piyush Dynamics Implementations and Results Paral lel Computational Fluid edited by Horst D Simon Enterprise Integration Modeling Proceedings of the First International Conference J Petrie Jr edited by Charles High Performance Fortran Handbook The by Charles H Ko elb el David B Loveman Rob ert S Schreib er Guy L Steele Jr and Zosel Mary E Using MPI Portable Paral lel Programming with the MessagePassing Interface Gropp Ewing Lusk and Anthony Skjellum by William Paral lel Virtual Machine A Users Guide and Tutorial for Networked Paral lel PVM Computing by Al Geist Adam Beguelin Jack Dongarra Weicheng Jiang Rob ert Manchek and Vaidy Sunderam PVM Parallel Virtual Machine A Users Guide and Tutorial for Networked Parallel Computing Al Geist Adam Beguelin Jack Dongarra Weicheng Jiang Rob ert Manchek aidy Sunderam V The MIT Press Cambridge Massachusetts England London c Massachusetts Institute of Technology All rights reserved No part of this b o ok may b e repro duced in any form by any electronic or photo copying recording or information storage and retrieval without mechanical means including p ermission in writing from the publisher a T X by the authors and was printed and b ound in the United States of America This b o ok was set in L E Library of Congress CataloginginPublication Data This b o ok is also available in p ostscript and html forms over the Internet you can use one of the following metho ds To retrieve the p ostscript le anonymous ftp netlibcsutkedu ftp cd pvmbook get pvmbookps quit from any machine on the Internet typ e anonnetlibcsutkedupvmbookpvmbookps pvmbookps rcp sending email to netlibornlgov and in the message typ e send pvmbookps from pvmbook click pvm click b o ok click pvmpvmbo okps click use Xnetlib and click library download click Get Files Now Xnetlib is an Xwindow interface to the netlib software based on from xnetlib a clientserver mo del The software can b e found in netlib send index To view the html le use the URL httpwwwnetliborgpvmbookpvmbookhtml Contents Series Foreword xi Preface xiii Intro duction Heterogeneous Network Computing rends in Distributed Computing T PVM Overview Other Packages The p System Express MPI The Linda System The PVM System Using PVM How to Obtain the PVM Software Setup to Use PVM Setup Summary Starting PVM Common Startup Problems Running PVM Programs PVM Console Details Host File Options Basic Programming Techniques Common Parallel Programming Paradigms Crowd Computations Tree Computations Workload Allo cation Data Decomp osition Function Decomp osition vi Contents Porting Existing Applications to PVM PVM User Interface Pro cess Control Information Dynamic Conguration Signaling Setting and Getting Options Message Passing Message Buers Packing Data Sending and Receiving Data Unpacking Data Dynamic Pro cess Groups Program Examples ForkJoin Dot Pro duct Failure Matrix Multiply OneDimensional Heat Equation Dierent Styles of Communication How PVM Works Comp onents Task Identiers Architecture Classes Message Mo del Asynchronous Notication PVM Daemon and Programming Library Messages Fragments and Databufs Contents vii Messages in Libpvm Messages in the Pvmd Pvmd Entry Points Control Messages PVM Daemon Startup Shutdown Host Table and Machine Conguration Tasks Wait Contexts Fault Detection and Recovery Pvmd Starting Slave Pvmds Resource Manager Libpvm Library Language Supp ort Connecting to the Pvmd Proto cols Messages PvmdPvmd PvmdTask and TaskTask Message Routing Pvmd Pvmd and Foreign Tasks Libpvm Multicasting Task Environment Environment Variables Standard Input and Output Tracing Debugging Console Program Resource Limitations viii Contents In the PVM Daemon In the Task Multipro cessor Systems MessagePassing Architectures SharedMemory Architectures Optimized Send and Receive on MPP Advanced Topics XPVM Network View SpaceTime View Other Views Porting PVM to New Architectures Unix Workstations Multipro cessors Troublesho oting Getting PVM Installed Set PVM ROOT OnLine Manual Pages Building the Release Errors During Build Compatible Versions Getting PVM Running Pvmd Log File Pvmd So cket Address File Starting PVM from the Console Starting the Pvmd by Hand Adding Hosts to the Virtual Machine PVM Host File Shutting Down Compiling Applications Header Files Contents ix Linking Running Applications Spawn Cant Find Executables Group Functions Memory Use Input and Output Scheduling Priority Resource Limitations Debugging and Tracing Debugging the System Runtime Debug Masks Tickle the Pvmd Starting Pvmd under a Debugger Sane Heap Statistics Glossary A History of PVM Versions B PVM Routines Bibliography Index Series Foreword The world of mo dern computing p otentially oers many helpful metho ds and to ols to scientists and engineers but the fast pace of change in computer hardware software and The algorithms often makes practical use of the newest computing technology dicult Scientic and Engineering Computation series fo cuses on rapid advances in computing technologies and attempts to facilitate transferring these technologies to applications in science and engineering It will include b o oks on theories metho ds and original applications in such areas as parallelism largescale simulations timecritical computing computeraided design and engineering use of computers in manufacturing visualization of scientic data and humanmachine interface technology The series will help scientists and engineers to understand the current world of ad vanced computation and to anticipate future developments that will impact their com puting environments and op en up new capabilities and mo des of computation This volume presents a software package for developing parallel programs executable allows on networked Unix computers The to ol called Parallel Virtual Machine PVM a heterogeneous collection of workstations and sup ercomputers to function as a single highp erformance parallel machine PVM is p ortable and runs on a wide variety of mo dern platforms It has b een well accepted by the global computing community and used successfully for solving largescale problems in science industry and business Janusz S Kowalik Preface In this b o ok we describ e the Parallel Virtual Machine PVM system and how to develop programs using PVM PVM is a software system that p ermits a heterogeneous collection of Unix computers networked together to b e viewed by a users program as a single parallel computer PVM is the mainstay of the Heterogeneous Network Computing research pro ject a collab orative venture b etween Oak Ridge National Lab oratory the University of Tennessee Emory University and Carnegie Mellon University The PVM system has evolved in the past several years into a viable technology for distributed and parallel pro cessing in a variety of disciplines PVM supp orts a straight forward but functionally complete messagepassing mo del PVM is designed to link computing resources and provide users with a parallel platform for running their computer applications irresp ective of the numb er of dierent computers they use and where the computers are lo cated When PVM is correctly installed it is capable of harnessing the combined resources of typically heterogeneous networked computing platforms to deliver high levels of p erformance and functionality In this b o ok we describ e the architecture of the PVM system and discuss its computing the programming interface it supp orts auxiliary facilities for pro cess groups mo del the use of PVM on highly parallel systems such as the Intel Paragon Cray TD and Thinking Machines CM and some of the internal implementation techniques employed Performance issues dealing primarily with communication overheads are analyzed and recent ndings as well as enhancements are presented To demonstrate the viability of PVM for largescale scientic sup ercomputing we also provide some example programs This b o ok is not a textb o ok rather it is meant to provide a fast entrance to the world of heterogeneous network computing We intend this b o ok to b e used by two groups students and researchers working with networks of computers As such we of readers hop e this b o ok can serve b oth as a reference and as a supplement to a teaching text on asp ects of network computing This guide will familiarize readers with the basics of PVM and the concepts used in programming on a network The information provided here will help with the following PVM tasks Writing a program in PVM Building PVM on a system Starting PVM on a set of machines Debugging a PVM application xiv Preface A Bit of History The PVM pro ject b egan in the summer of at Oak Ridge National Lab oratory The prototyp e system PVM was constructed by Vaidy Sunderam and Al Geist this version of the system was used internally at the Lab and was not released to the outside Version of PVM was written at the University of Tennessee and released in March During the following year PVM b egan to b e used in many scientic applications After user feedback and a numb er of changes
Recommended publications
  • A Case for High Performance Computing with Virtual Machines

    A Case for High Performance Computing with Virtual Machines

    A Case for High Performance Computing with Virtual Machines Wei Huangy Jiuxing Liuz Bulent Abaliz Dhabaleswar K. Panday y Computer Science and Engineering z IBM T. J. Watson Research Center The Ohio State University 19 Skyline Drive Columbus, OH 43210 Hawthorne, NY 10532 fhuanwei, [email protected] fjl, [email protected] ABSTRACT in the 1960s [9], but are experiencing a resurgence in both Virtual machine (VM) technologies are experiencing a resur- industry and research communities. A VM environment pro- gence in both industry and research communities. VMs of- vides virtualized hardware interfaces to VMs through a Vir- fer many desirable features such as security, ease of man- tual Machine Monitor (VMM) (also called hypervisor). VM agement, OS customization, performance isolation, check- technologies allow running different guest VMs in a phys- pointing, and migration, which can be very beneficial to ical box, with each guest VM possibly running a different the performance and the manageability of high performance guest operating system. They can also provide secure and computing (HPC) applications. However, very few HPC ap- portable environments to meet the demanding requirements plications are currently running in a virtualized environment of computing resources in modern computing systems. due to the performance overhead of virtualization. Further, Recently, network interconnects such as InfiniBand [16], using VMs for HPC also introduces additional challenges Myrinet [24] and Quadrics [31] are emerging, which provide such as management and distribution of OS images. very low latency (less than 5 µs) and very high bandwidth In this paper we present a case for HPC with virtual ma- (multiple Gbps).
  • 2.5 Classification of Parallel Computers

    2.5 Classification of Parallel Computers

    52 // Architectures 2.5 Classification of Parallel Computers 2.5 Classification of Parallel Computers 2.5.1 Granularity In parallel computing, granularity means the amount of computation in relation to communication or synchronisation Periods of computation are typically separated from periods of communication by synchronization events. • fine level (same operations with different data) ◦ vector processors ◦ instruction level parallelism ◦ fine-grain parallelism: – Relatively small amounts of computational work are done between communication events – Low computation to communication ratio – Facilitates load balancing 53 // Architectures 2.5 Classification of Parallel Computers – Implies high communication overhead and less opportunity for per- formance enhancement – If granularity is too fine it is possible that the overhead required for communications and synchronization between tasks takes longer than the computation. • operation level (different operations simultaneously) • problem level (independent subtasks) ◦ coarse-grain parallelism: – Relatively large amounts of computational work are done between communication/synchronization events – High computation to communication ratio – Implies more opportunity for performance increase – Harder to load balance efficiently 54 // Architectures 2.5 Classification of Parallel Computers 2.5.2 Hardware: Pipelining (was used in supercomputers, e.g. Cray-1) In N elements in pipeline and for 8 element L clock cycles =) for calculation it would take L + N cycles; without pipeline L ∗ N cycles Example of good code for pipelineing: §doi =1 ,k ¤ z ( i ) =x ( i ) +y ( i ) end do ¦ 55 // Architectures 2.5 Classification of Parallel Computers Vector processors, fast vector operations (operations on arrays). Previous example good also for vector processor (vector addition) , but, e.g. recursion – hard to optimise for vector processors Example: IntelMMX – simple vector processor.
  • Scalable and Distributed Deep Learning (DL): Co-Design MPI Runtimes and DL Frameworks

    Scalable and Distributed Deep Learning (DL): Co-Design MPI Runtimes and DL Frameworks

    Scalable and Distributed Deep Learning (DL): Co-Design MPI Runtimes and DL Frameworks OSU Booth Talk (SC ’19) Ammar Ahmad Awan [email protected] Network Based Computing Laboratory Dept. of Computer Science and Engineering The Ohio State University Agenda • Introduction – Deep Learning Trends – CPUs and GPUs for Deep Learning – Message Passing Interface (MPI) • Research Challenges: Exploiting HPC for Deep Learning • Proposed Solutions • Conclusion Network Based Computing Laboratory OSU Booth - SC ‘18 High-Performance Deep Learning 2 Understanding the Deep Learning Resurgence • Deep Learning (DL) is a sub-set of Machine Learning (ML) – Perhaps, the most revolutionary subset! Deep Machine – Feature extraction vs. hand-crafted Learning Learning features AI Examples: • Deep Learning Examples: Logistic Regression – A renewed interest and a lot of hype! MLPs, DNNs, – Key success: Deep Neural Networks (DNNs) – Everything was there since the late 80s except the “computability of DNNs” Adopted from: http://www.deeplearningbook.org/contents/intro.html Network Based Computing Laboratory OSU Booth - SC ‘18 High-Performance Deep Learning 3 AlexNet Deep Learning in the Many-core Era 10000 8000 • Modern and efficient hardware enabled 6000 – Computability of DNNs – impossible in the 4000 ~500X in 5 years 2000 past! Minutesto Train – GPUs – at the core of DNN training 0 2 GTX 580 DGX-2 – CPUs – catching up fast • Availability of Datasets – MNIST, CIFAR10, ImageNet, and more… • Excellent Accuracy for many application areas – Vision, Machine Translation, and
  • Parallel Programming

    Parallel Programming

    Parallel Programming Parallel Programming Parallel Computing Hardware Shared memory: multiple cpus are attached to the BUS all processors share the same primary memory the same memory address on different CPU’s refer to the same memory location CPU-to-memory connection becomes a bottleneck: shared memory computers cannot scale very well Parallel Programming Parallel Computing Hardware Distributed memory: each processor has its own private memory computational tasks can only operate on local data infinite available memory through adding nodes requires more difficult programming Parallel Programming OpenMP versus MPI OpenMP (Open Multi-Processing): easy to use; loop-level parallelism non-loop-level parallelism is more difficult limited to shared memory computers cannot handle very large problems MPI(Message Passing Interface): require low-level programming; more difficult programming scalable cost/size can handle very large problems Parallel Programming MPI Distributed memory: Each processor can access only the instructions/data stored in its own memory. The machine has an interconnection network that supports passing messages between processors. A user specifies a number of concurrent processes when program begins. Every process executes the same program, though theflow of execution may depend on the processors unique ID number (e.g. “if (my id == 0) then ”). ··· Each process performs computations on its local variables, then communicates with other processes (repeat), to eventually achieve the computed result. In this model, processors pass messages both to send/receive information, and to synchronize with one another. Parallel Programming Introduction to MPI Communicators and Groups: MPI uses objects called communicators and groups to define which collection of processes may communicate with each other.
  • CS650 Computer Architecture Lecture 10 Introduction to Multiprocessors

    CS650 Computer Architecture Lecture 10 Introduction to Multiprocessors

    NJIT Computer Science Dept CS650 Computer Architecture CS650 Computer Architecture Lecture 10 Introduction to Multiprocessors and PC Clustering Andrew Sohn Computer Science Department New Jersey Institute of Technology Lecture 10: Intro to Multiprocessors/Clustering 1/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Key Issues Run PhotoShop on 1 PC or N PCs Programmability • How to program a bunch of PCs viewed as a single logical machine. Performance Scalability - Speedup • Run PhotoShop on 1 PC (forget the specs of this PC) • Run PhotoShop on N PCs • Will it run faster on N PCs? Speedup = ? Lecture 10: Intro to Multiprocessors/Clustering 2/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Types of Multiprocessors Key: Data and Instruction Single Instruction Single Data (SISD) • Intel processors, AMD processors Single Instruction Multiple Data (SIMD) • Array processor • Pentium MMX feature Multiple Instruction Single Data (MISD) • Systolic array • Special purpose machines Multiple Instruction Multiple Data (MIMD) • Typical multiprocessors (Sun, SGI, Cray,...) Single Program Multiple Data (SPMD) • Programming model Lecture 10: Intro to Multiprocessors/Clustering 3/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Shared-Memory Multiprocessor Processor Prcessor Prcessor Interconnection network Main Memory Storage I/O Lecture 10: Intro to Multiprocessors/Clustering 4/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Distributed-Memory Multiprocessor Processor Processor Processor IO/S MM IO/S MM IO/S MM Interconnection network IO/S MM IO/S MM IO/S MM Processor Processor Processor Lecture 10: Intro to Multiprocessors/Clustering 5/15 12/7/2003 A.
  • Parallel Computer Architecture

    Parallel Computer Architecture

    Parallel Computer Architecture Introduction to Parallel Computing CIS 410/510 Department of Computer and Information Science Lecture 2 – Parallel Architecture Outline q Parallel architecture types q Instruction-level parallelism q Vector processing q SIMD q Shared memory ❍ Memory organization: UMA, NUMA ❍ Coherency: CC-UMA, CC-NUMA q Interconnection networks q Distributed memory q Clusters q Clusters of SMPs q Heterogeneous clusters of SMPs Introduction to Parallel Computing, University of Oregon, IPCC Lecture 2 – Parallel Architecture 2 Parallel Architecture Types • Uniprocessor • Shared Memory – Scalar processor Multiprocessor (SMP) processor – Shared memory address space – Bus-based memory system memory processor … processor – Vector processor bus processor vector memory memory – Interconnection network – Single Instruction Multiple processor … processor Data (SIMD) network processor … … memory memory Introduction to Parallel Computing, University of Oregon, IPCC Lecture 2 – Parallel Architecture 3 Parallel Architecture Types (2) • Distributed Memory • Cluster of SMPs Multiprocessor – Shared memory addressing – Message passing within SMP node between nodes – Message passing between SMP memory memory nodes … M M processor processor … … P … P P P interconnec2on network network interface interconnec2on network processor processor … P … P P … P memory memory … M M – Massively Parallel Processor (MPP) – Can also be regarded as MPP if • Many, many processors processor number is large Introduction to Parallel Computing, University of Oregon,
  • What Is SPMD? Messages

    What Is SPMD? Messages

    1 2 Outline Motivation for MPI Overview of PVM and MPI The pro cess that pro duced MPI What is di erent ab out MPI? { the \usual" send/receive Jack Dongarra { the MPI send/receive { simple collective op erations Computer Science Department New in MPI: Not in MPI UniversityofTennessee Some simple complete examples, in Fortran and C and Communication mo des, more on collective op erations Implementation status Mathematical Sciences Section Oak Ridge National Lab oratory MPICH - a free, p ortable implementation MPI resources on the Net MPI-2 http://www.netlib.org/utk/p eople/JackDongarra.html 3 4 Messages What is SPMD? 2 Messages are packets of data moving b etween sub-programs. 2 Single Program, Multiple Data 2 The message passing system has to b e told the 2 Same program runs everywhere. following information: 2 Restriction on the general message-passing mo del. { Sending pro cessor 2 Some vendors only supp ort SPMD parallel programs. { Source lo cation { Data typ e 2 General message-passing mo del can b e emulated. { Data length { Receiving pro cessors { Destination lo cation { Destination size 5 6 Access Point-to-Point Communication 2 A sub-program needs to b e connected to a message passing 2 Simplest form of message passing. system. 2 One pro cess sends a message to another 2 A message passing system is similar to: 2 Di erenttyp es of p oint-to p oint communication { Mail b ox { Phone line { fax machine { etc. 7 8 Synchronous Sends Asynchronous Sends Provide information about the completion of the Only know when the message has left.
  • Exploiting Automatic Vectorization to Employ SPMD on SIMD Registers

    Exploiting Automatic Vectorization to Employ SPMD on SIMD Registers

    Exploiting automatic vectorization to employ SPMD on SIMD registers Stefan Sprenger Steffen Zeuch Ulf Leser Department of Computer Science Intelligent Analytics for Massive Data Department of Computer Science Humboldt-Universitat¨ zu Berlin German Research Center for Artificial Intelligence Humboldt-Universitat¨ zu Berlin Berlin, Germany Berlin, Germany Berlin, Germany [email protected] [email protected] [email protected] Abstract—Over the last years, vectorized instructions have multi-threading with SIMD instructions1. For these reasons, been successfully applied to accelerate database algorithms. How- vectorization is essential for the performance of database ever, these instructions are typically only available as intrinsics systems on modern CPU architectures. and specialized for a particular hardware architecture or CPU model. As a result, today’s database systems require a manual tai- Although modern compilers, like GCC [2], provide auto loring of database algorithms to the underlying CPU architecture vectorization [1], typically the generated code is not as to fully utilize all vectorization capabilities. In practice, this leads efficient as manually-written intrinsics code. Due to the strict to hard-to-maintain code, which cannot be deployed on arbitrary dependencies of SIMD instructions on the underlying hardware, hardware platforms. In this paper, we utilize ispc as a novel automatically transforming general scalar code into high- compiler that employs the Single Program Multiple Data (SPMD) execution model, which is usually found on GPUs, on the SIMD performing SIMD programs remains a (yet) unsolved challenge. lanes of modern CPUs. ispc enables database developers to exploit To this end, all techniques for auto vectorization have focused vectorization without requiring low-level details or hardware- on enhancing conventional C/C++ programs with SIMD instruc- specific knowledge.
  • Polynomial-Time Algorithms for Enforcing Sequential Consistency in SPMD Programs with Arrays

    Polynomial-Time Algorithms for Enforcing Sequential Consistency in SPMD Programs with Arrays

    Polynomial-time Algorithms for Enforcing Sequential Consistency in SPMD Programs with Arrays ¡ Wei-Yu Chen , Arvind Krishnamurthy , and Katherine Yelick ¢ Computer Science Division, University of California, Berkeley £ wychen, yelick ¤ @cs.berkeley.edu ¥ Department of Computer Science, Yale University [email protected] Abstract. The simplest semantics for parallel shared memory programs is se- quential consistency in which memory operations appear to take place in the or- der specified by the program. But many compiler optimizations and hardware fea- tures explicitly reorder memory operations or make use of overlapping memory operations which may violate this constraint. To ensure sequential consistency while allowing for these optimizations, traditional data dependence analysis is augmented with a parallel analysis called cycle detection. In this paper, we present new algorithms to enforce sequential consistency for the special case of the Single Program Multiple Data (SPMD) model of parallelism. First, we present an algo- rithm for the basic cycle detection problem, which lowers the running time from ¥ ¦¨§ © ¦¨§ © to . Next, we present three polynomial-time methods that more ac- curately support programs with array accesses. These results are a step toward making sequentially consistent shared memory programming a practical model across a wide range of languages and hardware platforms. 1 Introduction In a uniprocessor environment, compiler and hardware transformations must adhere to a simple data dependency constraint: the orders of all pairs of conflicting accesses (accesses to the same memory location, with at least one a write) must be preserved. The execution model for parallel programs is considerably more complicated, since each thread executes its own portion of the program asynchronously, and there is no predetermined ordering among accesses issued by different threads to shared memory locations.
  • Parallel Programming Using Openmp Feb 2014

    Parallel Programming Using Openmp Feb 2014

    OpenMP tutorial by Brent Swartz February 18, 2014 Room 575 Walter 1-4 P.M. © 2013 Regents of the University of Minnesota. All rights reserved. Outline • OpenMP Definition • MSI hardware Overview • Hyper-threading • Programming Models • OpenMP tutorial • Affinity • OpenMP 4.0 Support / Features • OpenMP Optimization Tips © 2013 Regents of the University of Minnesota. All rights reserved. OpenMP Definition The OpenMP Application Program Interface (API) is a multi-platform shared-memory parallel programming model for the C, C++ and Fortran programming languages. Further information can be found at http://www.openmp.org/ © 2013 Regents of the University of Minnesota. All rights reserved. OpenMP Definition Jointly defined by a group of major computer hardware and software vendors and the user community, OpenMP is a portable, scalable model that gives shared-memory parallel programmers a simple and flexible interface for developing parallel applications for platforms ranging from multicore systems and SMPs, to embedded systems. © 2013 Regents of the University of Minnesota. All rights reserved. MSI Hardware • MSI hardware is described here: https://www.msi.umn.edu/hpc © 2013 Regents of the University of Minnesota. All rights reserved. MSI Hardware The number of cores per node varies with the type of Xeon on that node. We define: MAXCORES=number of cores/node, e.g. Nehalem=8, Westmere=12, SandyBridge=16, IvyBridge=20. © 2013 Regents of the University of Minnesota. All rights reserved. MSI Hardware Since MAXCORES will not vary within the PBS job (the itasca queues are split by Xeon type), you can determine this at the start of the job (in bash): MAXCORES=`grep "core id" /proc/cpuinfo | wc -l` © 2013 Regents of the University of Minnesota.
  • Exploring Massive Parallel Computation with GPU

    Exploring Massive Parallel Computation with GPU

    Need for parallelism Graphical Processor Units Gravitational Microlensing Modelling Exploring Massive Parallel Computation with GPU Ian Bond Massey University, Auckland, New Zealand 2011 Sagan Exoplanet Workshop Pasadena, July 25-29 2011 Ian Bond | Microlensing parallelism 1/40 Need for parallelism Graphical Processor Units Gravitational Microlensing Modelling Assumptions/Purpose You are all involved in microlensing modelling and you have (or are working on) your own code this lecture shows how to get started on getting code to run on a GPU then its over to you . Ian Bond | Microlensing parallelism 2/40 Need for parallelism Graphical Processor Units Gravitational Microlensing Modelling Outline 1 Need for parallelism 2 Graphical Processor Units 3 Gravitational Microlensing Modelling Ian Bond | Microlensing parallelism 3/40 Need for parallelism Graphical Processor Units Gravitational Microlensing Modelling Paralel Computing Parallel Computing is use of multiple computers, or computers with multiple internal processors, to solve a problem at a greater computational speed than using a single computer (Wilkinson 2002). How does one achieve parallelism? Ian Bond | Microlensing parallelism 4/40 Need for parallelism Graphical Processor Units Gravitational Microlensing Modelling Grand Challenge Problems A grand challenge problem is one that cannot be solved in a reasonable amount of time with todays computers’ Examples: – Modelling large DNA structures – Global weather forecasting – N body problem (N very large) – brain simulation Has microlensing
  • Dcuda: Hardware Supported Overlap of Computation and Communication

    Dcuda: Hardware Supported Overlap of Computation and Communication

    dCUDA: Hardware Supported Overlap of Computation and Communication Tobias Gysi Jeremia Bar¨ Torsten Hoefler Department of Computer Science Department of Computer Science Department of Computer Science ETH Zurich ETH Zurich ETH Zurich [email protected] [email protected] [email protected] Abstract—Over the last decade, CUDA and the underlying utilization of the costly compute and network hardware. To GPU hardware architecture have continuously gained popularity mitigate this problem, application developers can implement in various high-performance computing application domains manual overlap of computation and communication [23], [27]. such as climate modeling, computational chemistry, or machine learning. Despite this popularity, we lack a single coherent In particular, there exist various approaches [13], [22] to programming model for GPU clusters. We therefore introduce overlap the communication with the computation on an inner the dCUDA programming model, which implements device- domain that has no inter-node data dependencies. However, side remote memory access with target notification. To hide these code transformations significantly increase code com- instruction pipeline latencies, CUDA programs over-decompose plexity which results in reduced real-world applicability. the problem and over-subscribe the device by running many more threads than there are hardware execution units. Whenever a High-performance system design often involves trading thread stalls, the hardware scheduler immediately proceeds with off sequential performance against parallel throughput. The the execution of another thread ready for execution. This latency architectural difference between host and device processors hiding technique is key to make best use of the available hardware perfectly showcases the two extremes of this design space.