INTRODUCTION TO PARALLEL PROGRAMMING WITHMPIANDOPENMP 1–5 March 2021 | Benedikt Steinbusch | Jülich Supercomputing Centre
Member of the Helmholtz Association
IV Nonblocking Point-to-Point Communication 14 18 Introduction 14 19 Start 14 20 Completion 15 INTRODUCTION TO 21 Remarks 16 22 Exercises 17
PARALLELPROGRAMMINGWITH V Collective Communication 18 23 Introduction 18 MPIANDOPENMP 24 Reductions 18 25 Reduction Variants 19 Benedikt Steinbusch | Jülich Supercomputing Centre | 1–5 March 2021 26 Exercises 19 27 Data Movement 20 28 Data Movement Variants 22 CONTENTS 29 Exercises 22 30 In Place Mode 23 I Fundamentals of Parallel Computing 3 31 Synchronization 24 1 Motivation 3 32 Nonblocking Collective Communication 24 2 Hardware 3 3 Software 4 VI Derived Datatypes 25 33 Introduction 25 II First Steps with MPI 5 34 Constructors 26 4 What is MPI? 5 35 Exercises 28 5 Terminology 6 36 Address Calculation 28 6 Infrastructure 6 37 Padding 29 7 Basic Program Structure 7 38 Exercises 30 8 Exercises 9 VII Input/Output 31 III Blocking Point-to-Point Communication 10 39 Introduction 31 9 Introduction 10 40 File Manipulation 31 10 Sending 10 41 File Views 33 11 Exercises 11 42 Data Access 34 12 Receiving 11 43 Consistency 40 13 Exercises 12 44 Exercises 40 14 Communication Modes 12 15 Semantics 12 VIII Tools 41 16 Pitfalls 13 45 MUST 41 17 Exercises 13 46 Exercises 42 IX Communicators 42
1 47 Introduction 42 77 The task Construct 60 48 Constructors 42 78 task Clauses 61 49 Accessors 43 79 Task Scheduling 61 50 Destructors 45 80 Task Synchronization 62 51 Exercises 45 81 Exercises 62
X Thread Compliance 46 XV Wrap-up 63 52 Introduction 46 XVI Tutorial 63 53 Enabling Thread Support 46 54 Matching Probe and Receive 46 TIMETABLE 55 Remarks 47 Day 1 Day 2 Day 3 Day 4 (Day 5) XI First Steps with OpenMP 47 56 What is OpenMP? 47 09:00 Fundamentals Blocking I/O First Steps with Tutorial 57 Terminology 49 10:30 of Parallel Collective OpenMP Computing Communication 58 Infrastructure 49 59 Basic Program Structure 50 COFFEE 60 Exercises 52 11:00 First Steps with Nonblocking I/O Low-Level Tutorial XII Low-Level OpenMP Concepts 52 12:30 MPI Collective Constructs Comm. 61 Introduction 52 62 Exercises 53 UTENSILS 63 Data Environment 54 13:30 Blocking P2P Derived Tools & Loop Tutorial 64 Exercises 55 14:30 Communication Datatypes Communicators Worksharing 65 Thread Synchronization 56 COFFEE 66 Exercises 57 15:00 Nonblocking Derived Thread Task Tutorial XIII Worksharing 57 16:30 P2P Datatypes Compliance Worksharing 67 Introduction 57 Communication 68 The single construct 57 69 single Clauses 57 70 The loop construct 58 71 loop Clauses 58 72 Exercises 59 73 workshare Construct 59 74 Exercises 59 75 Combined Constructs 59
XIV Task Worksharing 60 76 Introduction 60
2 PART I 2 HARDWARE FUNDAMENTALS OF PARALLEL COMPUTING A MODERN SUPERCOMPUTER Interconnect Interconnect Interconnect
1 MOTIVATION … CPU RAM CPU RAM CPU RAM PARALLEL COMPUTING Parallel computing is a type of computation in which many calculations or the MICROCHIP MEMORY MICROCHIP MEMORY MICROCHIP MEMORY execution of processes are carried out simultaneously. (Wikipedia1) Internal Bus Internal Bus Internal Bus WHYAMIHERE? TH MEMORY TH MEMORY TH MEMORY The Way Forward Accel. RAM Accel. RAM Accel. RAM • Frequency scaling has stopped • Performance increase through more parallel hardware PARALLEL COMPUTATIONAL UNITS • Treating scientific problems Implicit Parallelism – of larger scale • Parallel execution of different (parts of) processor instructions – in higher accuracy • Happens automatically – of a completely new kind • Can only be influenced indirectly by the programmer PARALLELISM IN THE TOP 500 LIST Multi-core / Multi-CPU Average Number of Cores of the Top 10 Systems • Found in commodity hardware today • Computational units share the same memory 106 Cluster • Found in computing centers • Independent systems linked via a (fast) interconnect 104 • Each system has its own memory
Number of cores Accelerators • Strive to perform certain tasks faster than is possible on a general purpose CPU 1993 1999 2004 2009 2015 2020 • Make different trade-offs 1Wikipedia. Parallel computing — Wikipedia, The Free Encyclopedia. 2017. URL: https://en.wikipedia.org/w/index.php?title=Parallel_computing&oldid=787466585 • Often have their own memory (visited on 06/28/2017). • Often not autonomous
3 Vector Processors / Vector Units Message Passing • Perform same operation on multiple pieces of data simultaneously • Parts of program state are transferred from one process to another for coordination • Making a come-back as SIMD units in commodity CPUs (AVX-512) and GPGPU • Primitive operations are active send and active receive MPI MEMORY DOMAINS • Implements a form of Distributed State and Message Passing Shared Memory • (But also Shared State and Synchronization) • All memory is directly accessible by the parallel computational units SHARED STATE & SYNCHRONIZATION • Single address space • Programmer might have to synchronize access Shared State Distributed Memory The whole program state is directly accessible by the parallel threads. Synchronization • Memory is partitioned into parts which are private to the different computational units • Threads can manipulate shared state using common loads and stores • “Remote” parts of memory are accessed via an interconnect • Establish agreement about progress of execution using synchronization primitives, e.g. • Access is usually nonuniform barriers, critical sections, … OpenMP 3 SOFTWARE • Implements Shared State and Synchronization PROCESSES & THREADS & TASKS • (But also higher level constructs)
Abstractions for the independent execution of (part of) a program. Process Usually, multiple processes, each with their own associated set of resources (memory, file descriptors, etc.), can coexist Thread • Typically “smaller” than processes • Often, multiple threads per one process • Threads of the same process can share resources Task • Typically “smaller” than threads • Often, multiple tasks per one thread • Here: user-level construct
DISTRIBUTED STATE & MESSAGE PASSING
Distributed State Program state is partitioned into parts which are private to the different processes.
4 PART II 4. Datatypes ✓ 5. Collective Communication ✓ FIRST STEPS WITH MPI 6. Groups, Contexts, Communicators and Caching (✓) 7. Process Topologies (✓) 4 WHATISMPI? 8. MPI Environmental Management (✓) MPI (Message-Passing Interface) is a message-passing library interface specification. 9. The Info Object [...] MPI addresses primarily the message-passing parallel programming model, in 10. Process Creation and Management which data is moved from the address space of one process to that of another process through cooperative operations on each process. (MPI Forum2) 11. One-Sided Communications • Industry standard for a message-passing programming model 12. External interfaces (✓) • Provides specifications (no implementations) 13. I/O ✓ • Implemented as a library with language bindings for Fortran and C 14. Tool Support • Portable across different computer architectures 15. … Current version of the standard: 3.1 (June 2015) LITERATURE & ACKNOWLEDGEMENTS
BRIEF HISTORY Literature <1992 several message-passing libraries were developed, PVM, P4, ... • Message Passing Interface Forum. MPI: A Message-Passing 1992 At SC92, several developers for message-passing libraries agreed to develop a standard for Interface Standard. Version 3.1. June 4, 2015. URL: message-passing https://mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf 1994 MPI-1.0 standard published • William Gropp, Ewing Lusk, and Anthony Skjellum. Using MPI. Portable Parallel Programming with the Message-Passing Interface. 3rd ed. The MIT Press, Nov. 2014. 336 pp. 1997 MPI-2.0 standard adds process creation and management, one-sided communication, ISBN: 9780262527392 extended collective communication, external interfaces and parallel I/O • William Gropp et al. Using Advanced MPI. Modern Features of the Message-Passing Interface. 2008 MPI-2.1 combines MPI-1.3 and MPI-2.0 1st ed. Nov. 2014. 392 pp. ISBN: 9780262527637 2009 MPI-2.2 corrections and clarifications with minor extensions • https://www.mpi-forum.org 2012 MPI-3.0 nonblocking collectives, new one-sided operations, Fortran 2008 bindings Acknowledgements
2015 MPI-3.1 nonblocking collective I/O, current version of the standard • Rolf Rabenseifner for his comprehensive course on MPI and OpenMP • Marc-André Hermanns, Florian Janetzko and Alexander Trautmann for their course material COVERAGE on MPI and OpenMP 1. Introduction to MPI ✓ 2. MPI Terms and Conventions ✓ 3. Point-to-Point Communication ✓ 2Message Passing Interface Forum. MPI: A Message-Passing Interface Standard. Version 3.1. June 4, 2015. URL: https://mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf.
5 5 TERMINOLOGY C Compiler Wrappers
PROCESS ORGANIZATION [MPI-3.1, 6.2] $ # Generic compiler wrapper shipped with e.g. OpenMPI $ mpicc foo.c -o foo Terminology: Process $ # Vendor specific wrapper for IBM's XL C compiler on BG/Q $ bgxlc foo.c -o foo An MPI program consists of autonomous processes, executing their own code, in an MIMD style. Fortran Compiler Wrappers Terminology: Rank $ # Generic compiler wrapper shipped with e.g. OpenMPI A unique number assigned to each process within a group (start at 0) $ mpifort foo.f90 -o foo Terminology: Group $ # Vendor specific wrapper for IBM's XL Fortran compiler on BG/Q $ bgxlf90 foo.f90 -o foo An ordered set of process identifiers Terminology: Context However, neither the existence nor the interface of these wrappers is mandated by the standard. A property that allows the partitioning of the communication space Terminology: Communicator PROCESS STARTUP [MPI-3.1, 8.8] Scope for communication operations within or between groups, combines the The MPI standard does not mandate a mechanism for process startup. It suggests that a command concepts of group and context mpiexec with the following interface should exist:
Process Startup OBJECTS [MPI-3.1, 2.5.1] $ # startup mechanism suggested by the standard Terminology: Opaque Objects $ mpiexec -n
and copied, however copying a handle does not copy the object it refers to. C Destroying an object that has operations pending will not disrupt those operations. Fortran Language Bindings Terminology: Predefined Handles Consistent with F08 standard; good type-checking; highly recommended MPI defines several constant handles to certain objects, e.g. MPI_COMM_WORLD a use mpi_f08 communicator containing all processes initially partaking in a parallel execution of a F08 program. Not consistent with standard; so-so type-checking; not recommended
use mpi 6 INFRASTRUCTURE F90
COMPILING & LINKING [MPI-3.1, 17.1.7] Not consistent with standard; no type-checking; strongly discouraged include 'mpif.h' MPI libraries or system vendors usually ship compiler wrappers that set search paths and required F77 libraries, e.g.:
6 FORTRAN HINTS [MPI-3.1, 17.1.2 -- 17.1.4] R Rmpi, pdbMPI julia MPI.jl This course uses the Fortran 2008 MPI interface (use mpi_f08) which is not available in all MPI implementations. The Fortran 90 bindings differ from the Fortran 2008 bindings in the following .NET MPI.NET points: Java mpiJava, MPJ, MPJExpress • All derived type arguments are instead integer (some are arrays of integer or have a And many others, ask your favorite search engine. non-default kind) • Argument intent is not mandated by the Fortran 90 bindings 7 BASICPROGRAMSTRUCTURE • The ierror argument is mandatory instead of optional • Further details can be found in [MPI-3.1, 17.1] WORLDORDERINMPI
푁 MPI4PY HINTS • Program starts as distinct processes. 푝0 푝1 푝2 … • Stream of instructions might be different for each All exercises in the MPI part can be solved using Python with the mpi4py package. The slides do process. not show Python syntax, so here is a translation guide from the standard bindings to mpi4py. • Each process has access to its own private memory. • Information is exchanged between processes via • Everything lives in the MPI module (from mpi4py import MPI). messages. • Constants translate to attributes of that module: MPI_COMM_WORLD is • Processes may consist of multiple threads (see MPI.COMM_WORLD. OpenMP part on day 4). • Central types translate to Python classes: MPI_Comm is MPI.Comm. • Functions related to point-to-point and collective communication translate to methods on MPI.Comm: MPI_Send becomes MPI.Comm.Send. • Functions related to I/O translate to methods on MPI.File: MPI_File_write becomes MPI.File.Write. • Communication functions come in two flavors: INITIALIZATION [MPI-3.1, 8.7] – high level, uses pickle to (de)serialize python objects, method names start with MPI.Comm.send lower case letters, e.g. , Initialize MPI library, needs to happen before most other MPI functions can be used – low level, uses MPI Datatypes and Python buffers, method names start with upper int MPI_Init(int *argc, char ***argv) case letters, e.g. MPI.Comm.Scatter. C https://mpi4py.readthedocs.io help() See also and the built-in Python . MPI_Init(ierror) integer, optional, intent(out) :: ierror OTHER LANGUAGE BINDINGS F08 Exception (can be used before initialization) Besides the official bindings for C and Fortran mandated by the standard, unofficial bindings for int MPI_Initialized(int* flag)
other programming languages exist: C C++ Boost.MPI MATLAB Parallel Computing Toolbox Python pyMPI, mpi4py, pypar, MYMPI, …
7 After xeto cnb sdatrfinalization) after used be (can Exception functions its using done are you when library MPI Finalize FINALIZATION eemn h oa ubro rcse nacommunicator a in processes of number total the Determine SIZE COMMUNICATOR itself. handle the Additionally, communication. MPI_COMM_SELF for available processes all includes that communicator COMMUNICATORS PREDEFINED C F08 C F08 C F08 C F08 integer logical ierror) MPI_Finalized(flag, int integer MPI_Finalize(ierror) int integer logical ierror) MPI_Initialized(flag, int type type MPI_COMM_SELF; MPI_Comm MPI_COMM_WORLD; MPI_Comm MPI_Init MPI_Finalized( MPI_Finalize( P_omsz(P_omcomm, MPI_Comm_size(MPI_Comm (MPI_Comm) (MPI_Comm) , , , , , MI31 8.7] [MPI-3.1, a encalled, been has optional intent optional optional intent sacmuiao hti ai nec rcs n otisol h process the only contains and process each on valid is that communicator a is MI31 6.4.1] [MPI-3.1, (out) (out) :: :: , , void , int MPI_COMM_SELF MPI_COMM_WORLD intent intent intent MPI_COMM_WORLD :: ) :: *flag) flag flag (out) (out) (out) :: :: :: int ierror ierror ierror savldhnl oapredefined a to handle valid a is *size) 8 Examples RO HANDLING ERROR Examples communicator a within process calling the of rank the Determine RANK PROCESS F08 C F08 C F08 C F08 • • integer integer type ierror) size, MPI_Comm_size(comm, call :: integer int int integer integer ierror) type rank, MPI_Comm_rank(comm, int call :: integer int int MPI_ERRORS_RETURN MPI_ERRORS_ARE_FATAL be can handlers Error to attached be can which handlers error through handling error Flexible – – – (MPI_Comm), err=MICm_akMICM_OL,&rank); MPI_Comm_rank(MPI_COMM_WORLD, = ierror rank; (MPI_Comm), comm, MPI_Comm_rank(MPI_Comm &size); MPI_Comm_size(MPI_COMM_WORLD, = ierror size; P_omrn(P_OMWRD rank) MPI_Comm_rank(MPI_COMM_WORLD, size) MPI_Comm_size(MPI_COMM_WORLD, idw ntpr fti course) this of part (not Windows Files Communicators , , , , MI31 6.4.1] [MPI-3.1, optional intent optional intent MI31 .,84 8.5] 8.4, 8.3, [MPI-3.1, rank size (out) (out) intent intent , , nerrcd srtre rmteroutine the from returned is code error An intent intent :: :: rosecutrdi P otnsaotexecution abort routines MPI in encountered Errors (in) (in) rank size (out) (out) :: :: comm comm :: :: int ierror ierror *rank) Custom error handler A user supplied function is called on encountering an error Write a program print_rank_conditional.{c|cxx|f90|py} in such a way that process 0 writes out the total number of processes • By default – Communicators use MPI_ERRORS_ARE_FATAL I am process 0 of 3 I am process 1 – Files use MPI_ERRORS_RETURN I am process 2 – Windows use MPI_ERRORS_ARE_FATAL Use: MPI_Comm_size 8 EXERCISES
EXERCISE STRATEGIES
Solving • Do not have to solve all exercises, one per section would be good • Exercise description tells you what MPI functions/OpenMP directives to use • Work in pairs on harder exercises • If you get stuck – ask us – peek at solution • CMakeLists.txt is included Solutions exercises/{C|C++|Fortran|Python}/: . Most of the algorithm is there, you add MPI/OpenMP hard Almost empty files you add algorithm and MPI/OpenMP solutions Fully solved exercises, if you are completely stuck or for comparison
EXERCISES
Exercise 1 – First Steps 1.1 Output of Ranks Write a program print_rank.{c|cxx|f90|py} that has each process printing its rank.
I am process 0 I am process 1 I am process 2
Use: MPI_Init, MPI_Finalize, MPI_Comm_rank 1.2 Output of ranks and total number of processes
9 PART III • A source process sends a message to a destination process using an MPI send routine • A destination process needs to post a receive using an MPI receive routine BLOCKING POINT-TO-POINT • The source process and the destination process are specified by their ranks in the COMMUNICATION communicator • Every message sent with a point-to-point operation needs to be matched by a receive operation 9 INTRODUCTION
MESSAGE PASSING 10 SENDING SENDING MESSAGES [MPI-3.1, 3.2.1]
휋 MPI_Send(
int MPI_Send(const void* buf, int count, MPI_Datatype datatype, ↪ int dest, int tag, MPI_Comm comm) C
MPI_Send(buf, count, datatype, dest, tag, comm, ierror) 휋 휋 type(*), dimension(..), intent(in) :: buf after integer, intent(in) :: count, dest, tag type(MPI_Datatype), intent(in) :: datatype type(MPI_Comm), intent(in) :: comm integer, optional, intent(out) :: ierror BLOCKING & NONBLOCKING PROCEDURES F08
Terminology: Blocking MESSAGES [MPI-3.1, 3.2.2, 3.2.3] A procedure is blocking if return from the procedure indicates that the user is allowed to reuse resources specified in the call to the procedure. A message consists of two parts: Terminology: Nonblocking Terminology: Envelope source If a procedure is nonblocking it will return as soon as possible. However, the user is • Source process not allowed to reuse resources specified in the call to the procedure before the • Destination process dest communication has been completed using an appropriate completion procedure. • Tag tag Examples: • Communicator comm Phone • Blocking: Telephone call Terminology: Data @ • Nonblocking: Email Message data is read from/written to buffers specified by: • Address in memory buf PROPERTIES • Number of elements found in the buffer count • Communication between two processes within the same communicator • Structure of the data datatype Caution: A process can send messages to itself.
10 DATA TYPES [MPI-3.1, 3.2.2, 3.3, 4.1] 11 EXERCISES
Terminology: Data Type Exercise 2 – Point-to-Point Communication Describes the structure of a piece of data 2.1 Send Terminology: Basic Data Types In the file send_receive.{c|cxx|f90|py} implement the function/subroutine send(msg, dest). It should use MPI_Send to send the integer msg to the process with rank number dest Named by the standard, most correspond to basic data types of C or Fortran in MPI_COMM_WORLD. For the tag value, use the answer to the ultimate question of life, the 3 C type MPI basic data type universe, and everything (42) . MPI_Send signed int MPI_INT Use: float MPI_FLOAT char MPI_CHAR 12 RECEIVING … RECEIVING MESSAGES [MPI-3.1, 3.2.4] Fortran type MPI basic data type MPI_Recv(
Terminology: Typed Communication THE MPI_STATUS TYPE [MPI-3.1, 3.2.5] • Type of buffer contents must match MPI data type (e.g. inC int and MPI_INT). Contains information about received messages • Data type on send must match data type on receive operation. MPI_Status status; type(MPI_status) :: status status.MPI_SOURCE status%MPI_SOURCE • Allows data conversion when used on heterogeneous systems. status.MPI_TAG status%MPI_TAG Terminology: Packed data status.MPI_ERROR status%MPI_ERROR F08 C See [MPI-3.1, 4.2] 3Douglas Adams. The Hitchhiker’s Guide to the Galaxy. Pan Books, Oct. 12, 1979. ISBN: 0-330-25864-8.
11 ESG ASSEMBLY MESSAGE Pass eun fe acigmsaei ed to be received. is ready message a matching after Returns PROBE F08 C * F08 C • • • MPI_STATUS_IGNORE integer integer type ierror) type count, datatype, MPI_Get_count(status, int integer type type ierror) status, integer comm, tag, MPI_Probe(source, int
Process 0 1 2 3 4 16 PITFALLS root 1 1 1 1 1 x 3 1 7 4 9 DEADLOCK + Structure of program prevents blocking routines from ever completing, e.g.: tmp 8 + + Process 0 + 12 call MPI_Ssend(..., 1, ...) call MPI_Recv(..., 1, ...) 21
Process 1 24 call MPI_Ssend(..., 0, ...) call MPI_Recv(..., 0, ...) y - 24 - - - Mitigation Strategies Bonus • Changing communication structure (e.g. checkerboard) In the file global_prefix_sum.{c|cxx|f90|py} implement the function/subroutine • Using MPI_Sendrecv global_prefix_sum_ring(x, y, comm). It will be called by all processes on the • Using nonblocking routines communicator comm and on each one accepts an integer x. It should compute the global prefix sum of all values of x across all processes and return the results in y, i.e. the y returned on a process is the sum of all the x contributed by processes with lower rank number and its own. 17 EXERCISES Use the following strategy: 2.3 Global Summation – Ring 1. Every process except for the one with rank 0 receives a partial result from the process with In the file global_sum.{c|cxx|f90|py} implement the function/subroutine the next lower rank number global_sum_ring(x, y, root, comm). It will be called by all processes on the 2. Add the local x to the partial result communicator comm and on each one accepts an integer x. It should compute the global sum of all values of x across all processes and return the result in y only on the process with rank root. 3. Send the partial result to the process with the next higher rank number (except on the process with the highest rank number) Use the following strategy: 4. Return the partial result in y
13 The file contains a small main() function / program that can be used to test whether your PART IV implementation works. Use: MPI_Send, MPI_Recv NONBLOCKING POINT-TO-POINT Process 0 1 2 3 4 COMMUNICATION x 3 1 7 4 9 + tmp 4 + 18 INTRODUCTION + RATIONALE [MPI-3.1, 3.7] 11 + Premise 15 Communication operations are split into start and completion. The start routine produces a request handle that represents the in-flight operation and is used in the completion routine. The user promises to refrain from accessing the contents of message buffers while the operation isin 24 flight. Benefit y 3 4 11 15 24 A single process can have multiple nonblocking operations in flight at the same time. This enables communication patterns that would lead to deadlock if programmed using blocking variants of the same operations. Also, the additional leeway given to the MPI library may be utilized to, e.g.: • overlap computation and communication • overlap communication • pipeline communication • elide usage of buffers
19 START
INITIATION ROUTINES [MPI-3.1, 3.7.2]
Send Synchronous MPI_Issend Standard MPI_Isend Buffered MPI_Ibsend Ready MPI_Irsend Receive MPI_Irecv
14 Probe NONBLOCKING PROBE [MPI-3.1, 3.8.1] MPI_Iprobe MPI_Iprobe(
NONBLOCKING SEND [MPI-3.1, 3.7.2] MPI_Iprobe(source, tag, comm, flag, status, ierror) integer, intent(in) :: source, tag MPI_Isend(
MPI_Isend(buf, count, datatype, dest, tag, comm, request, • Uses true/false flag to indicate availability of a message. ↪ ierror) type(*), dimension(..), intent(in), asynchronous :: buf integer, intent(in) :: count, dest, tag 20 COMPLETION type(MPI_Datatype), intent(in) :: datatype type(MPI_Comm), intent(in) :: comm WAIT [MPI-3.1, 3.7.3] type(MPI_Request), intent(out) :: request MPI_Wait(
int MPI_Wait(MPI_Request *request, MPI_Status *status) NONBLOCKING RECEIVE [MPI-3.1, 3.7.2] C MPI_Wait(request, status, ierror) MPI_Irecv(
MPI_Test(
15 FREE CANCEL C * F08 C * F08 C • • • • • • • integer type ierror) MPI_Request_free(request, int )
y - - 24 - -
17 PART V 24 REDUCTIONS COLLECTIVE COMMUNICATION GLOBAL REDUCTION OPERATIONS [MPI-3.1, 5.9] Associative operations over distributed data
23 INTRODUCTION 푑0 ⊕ 푑1 ⊕ 푑2 ⊕ … ⊕ 푑푛−1, where COLLECTIVE [MPI-3.1, 2.4, 5.1] 푑푖, data of process with rank 푖 ⊕, associative operation Terminology: Collective Examples for ⊕: A procedure is collective if all processes in a group need to invoke the procedure. • Sum + and product × • Collective communication implements certain communication patterns that involve all processes in a group • Maximum max and minimum min • Synchronization may or may not occur (except for MPI_Barrier) • User-defined operations • No tags are used Caution: Order of application is not defined, watch out for floating point rounding. • No MPI_Status values are returned REDUCE [MPI-3.1, 5.9.1] • Receive buffer size must match the total amount of data sent (c.f. point-to-point communication where receive buffer capacity is allowed to exceed the message size) + + • Point-to-point and collective communication do not interfere + 7 8 9 1 2 3 4 5 6 10 11 12
CLASSIFICATION [MPI-3.1, 5.2.2] before
One-to-all MPI_Bcast, MPI_Scatter, MPI_Scatterv All-to-one 7 8 9 MPI_Gather, MPI_Gatherv, MPI_Reduce 1 2 3 4 5 6 10 11 12 All-to-all 22 26 30 after MPI_Allgather, MPI_Allgatherv, MPI_Alltoall, MPI_Alltoallv, MPI_Alltoallw, MPI_Allreduce, MPI_Reduce_scatter, MPI_Barrier Other MPI_Reduce(
int MPI_Reduce(const void* sendbuf, void* recvbuf, int count, ↪ MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) C
18 XLSV SCAN EXCLUSIVE F08 C * F08 integer type type type integer type comm, type op, datatype, count, recvbuf, MPI_Exscan(sendbuf, int
after before count, 19 . lblSmain– Summation Global 4.1 RDFNDOPERATIONS PREDEFINED Bonus Use: operation. reduction and datatype MPI_Reduce Since comm) 2.3. exercise to solution root, y, global_sum_reduce(x, file the In Communication Collective – 4 Exercise EXERCISES 26 functionality: combined or extended with Routines VARIANTS REDUCTION VARIANTS REDUCTION 25 MPI_SUM Meaning MPI_MIN MPI_MAX Name MPI_MINLOC MPI_MAXLOC MPI_BXOR MPI_LXOR MPI_BOR MPI_LOR MPI_BAND MPI_LAND MPI_PROD • • • MPI_Reduce global_sum_... MPI_Scan MPI_Reduce_scatter MPI_Allreduce global_sum.{c|cxx|f90|py} asn ntefnto ruet ntecretwyadslcigtecretMPI correct the selecting and way correct the in arguments function the on passing , efr lblpei euto,icueondt nresult in data own include reduction, prefix global a perform : Sum Minimum Maximum iiu n h is akta od it holds that rank first the and Minimum it holds that rank first the and Maximum or exclusive Bitwise or exclusive Logical or Bitwise or Logical and Bitwise and Logical Product MPI_Reduce MI31 . -5.11] -- 5.9 [MPI-3.1, efr lblrdcinadrpiaetersl noalranks all onto result the replicate and reduction global a perform : saseilzto of specialization a is MI31 5.9.2] [MPI-3.1, efr lblrdcinte cte h eutot l ranks all onto result the scatter then reduction global a perform : mlmn h function/subroutine the implement MPI_Reduce htprom h aeoeaina the as operation same the performs that MI31 5.9.4] [MPI-3.1, MI31 5.9.4] [MPI-3.1, tcnb mlmne ycalling by implemented be can it , In the file global_prefix_sum.{c|cxx|f90|py} implement the function/subroutine SCATTER [MPI-3.1, 5.6] global_prefix_sum_scan(x, y, comm) that performs the same operation as global_prefix_sum_ring. A B C D Since global_prefix_sum_... is a specialization of MPI_Scan, it can be implemented by
calling MPI_Scan, passing on the function arguments in the correct way and selecting the correct before MPI datatype and reduction operation. Use: MPI_Scan
A B C D 27 DATAMOVEMENT A C D B after BROADCAST [MPI-3.1, 5.4]
int MPI_Scatter(const void* sendbuf, int sendcount, 휋 ↪ MPI_Datatype sendtype, void* recvbuf, int recvcount, before ↪ MPI_Datatype recvtype, int root, MPI_Comm comm) C
MPI_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, ↪ recvtype, root, comm, ierror) type(*), dimension(..), intent(in) :: sendbuf 휋 휋 휋 휋 type(*), dimension(..) :: recvbuf after integer, intent(in) :: sendcount, recvcount, root type(MPI_Datatype), intent(in) :: sendtype, recvtype type(MPI_Comm), intent(in) :: comm integer, optional, intent(out) :: ierror int MPI_Bcast(void* buffer, int count, MPI_Datatype datatype, F08 ↪ int root, MPI_Comm comm) C GATHER [MPI-3.1, 5.5] MPI_Bcast(buffer, count, datatype, root, comm, ierror) type(*), dimension(..) :: buffer integer, intent(in) :: count, root B type(MPI_Datatype), intent(in) :: datatype A C D type(MPI_Comm), intent(in) :: comm before integer, optional, intent(out) :: ierror F08
B A C D A B C D after
20 GATHER-TO-ALL F08 C F08 C integer type type integer type recvcount, type recvbuf, sendtype, sendcount, MPI_Allgather(sendbuf, int integer type type integer type recvcount, type recvbuf, sendtype, sendcount, MPI_Gather(sendbuf, int ↪ ↪ ↪ ↪ ↪ ↪ (MPI_Comm), (MPI_Datatype), (*), (*), ierror) comm, recvtype, comm) MPI_Comm recvtype, MPI_Datatype sendtype, MPI_Datatype MPI_Allgather( (MPI_Comm), (MPI_Datatype), (*), ierror) comm, (*), root, recvtype, recvtype, MPI_Datatype sendtype, MPI_Datatype MPI_Gather( A A A , , , , MI31 5.7] [MPI-3.1, B optional intent dimension dimension optional intent dimension dimension C D (in) void const (in) intent intent , , (..) (..), void const (..) (..), A B B intent intent intent :: intent :: B (in) (in) C edon,rccut root recvcount, sendcount, :: edon,recvcount sendcount, :: intent intent (out) (out) sendbuf, * D (in) recvbuf (in) recvbuf :: int void :: void sendbuf, * A C C comm (in) comm (in) :: :: recvbuf, * :: :: comm) MPI_Comm root, recvbuf, * B ierror recvtype sendtype, ierror recvtype sendtype, C :: :: int D sendbuf sendbuf int sendcount, A D D int int sendcount, B recvcount, recvcount, C D
after before 21 L-OALSCATTER/GATHER ALL-TO-ALL AAMVMN SIGNATURES MOVEMENT DATA * F08 C • • • • P_olcie
DATA MOVEMENT VARIANTS [MPI-3.1, 5.5 -- 5.8] 29 EXERCISES Routines with variable counts (and datatypes): • MPI_Scatterv: scatter into parts of variable length 4.2 Redistribution of Points with Collectives • MPI_Gatherv: gather parts of variable length In the file redistribute.{c|cxx|f90|py} implement the function redistribute which should work as follows: • MPI_Allgatherv: gather parts of variable length onto all processes 1. All processes call the function collectively and pass in an array of random numbers – the MPI_Alltoallv • : exchange parts of variable length between all processes points – from a uniform random distribution on [0, 1). MPI_Alltoallw • : exchange parts of variable length and datatype between all processes 2. Partition [0, 1) among the nranks processes: process 푖 gets partition [푖/푛푟푎푛푘푠, (푖 + 1)/푛푟푎푛푘푠). DATA MOVEMENT SIGNATURES 3. Redistribute the points, so that every process is left with only those points that lie inside its partition and return them from the function. MPI_Collectivev(
22 The file contains tests that will check your implementation. If MPI_IN_PLACE is used for recvbuf on the root process, recvcount and recvtype are ignored and the root process does not send data to itself Use: MPI_Alltoall, MPI_Alltoallv
IN PLACE GATHER ALL-TO-ALL WITH VARYING COUNTS
MPI_Alltoallv(const void* sendbuf, const int sendcounts[], ↪ const int sdispls[], MPI_Datatype sendtype, void* recvbuf, A B C D
↪ const int recvcounts[], const int rdispls[], MPI_Datatype before ↪ recvtype, MPI_Comm comm) C
MPI_Alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, ↪ recvcounts, rdispls, recvtype, comm, ierror) type(*), dimension(..), intent(in) :: sendbuf A A B C D C D
type(*), dimension(..) :: recvbuf after integer, intent(in) :: sendcounts(*), sdispls(*), ↪ recvcounts(*), rdispls(*) type(MPI_Datatype), intent(in) :: sendtype, recvtype MPI_IN_PLACE sendbuf sendcount sendtype type(MPI_Comm), intent(in) :: comm If is used for on the root process, and are integer, optional, intent(out) :: ierror ignored on the root process and the root process will not send data to itself. F08
IN PLACE GATHER-TO-ALL 30 INPLACEMODE
INPLACEMODE A B C D
• Collectives can be used in in place mode with only one buffer to conserve memory before • The special value MPI_IN_PLACE is used in place of either the send or receive buffer address • count and datatype of that buffer are ignored A B C D A B C D A B C D A B C D IN PLACE SCATTER after
A B C D If MPI_IN_PLACE is used for sendbuf on all processes, sendcount and sendtype are
before ignored and the input data is assumed to already be in the correct position in recvbuf.
A A B C D C D after
23 IN PLACE ALL-TO-ALL SCATTER/GATHER IN PLACE EXCLUSIVE SCAN
+ + +
A B C D E F G H I J K L M N O P 1 2 3 4 5 6 7 8 9 10 11 12 before before
A E I M B F J N C G K O D H L P 1 2 3 1 2 3 5 7 9 12 15 18 after after
If MPI_IN_PLACE is used for sendbuf on all processes, sendcount and sendtype are If MPI_IN_PLACE is used for sendbuf on all the processes, the input data is taken from ignored and the input data is assumed to already be in the correct position in recvbuf. recvbuf and replaced by the results.
INPLACEREDUCE 31 SYNCHRONIZATION + + + BARRIER [MPI-3.1, 5.3] 1 2 3 4 5 6 7 8 9 10 11 12
before int MPI_Barrier(MPI_Comm comm) C
MPI_Barrier(comm, ierror) type(MPI_Comm), intent(in) :: comm integer, optional, intent(out) :: ierror F08 1 2 3 4 5 6 22 26 30 10 11 12
after Explicitly synchronizes all processes in the group of a communicator by blocking until all processes have entered the procedure.
If MPI_IN_PLACE is used for sendbuf on the root process, the input data for the root process is 32 NONBLOCKING COLLECTIVE COMMUNICATION taken from recvbuf. PROPERTIES
Properties similar to nonblocking point-to-point communication 1. Initiate communication • Routine names: MPI_I... (I for immediate) • Nonblocking routines return before the operation has completed. • Nonblocking routines have the same arguments as their blocking counterparts plus an extra request argument. 2. User-application proceeds with something else
24 3. Complete operation PART VI • Same completion routines (MPI_Test, MPI_Wait, …) Caution: Nonblocking collective operations cannot be matched with blocking collective DERIVED DATATYPES operations. Nonblocking Barrier 33 INTRODUCTION Barrier is entered through MPI_Ibarrier (which returns immediately). Completion (e.g. MPI_Wait) blocks until all processes have entered. MOTIVATION [MPI-3.1, 4.1]
NONBLOCKING BROADCAST [MPI-3.1, 5.12.2] Reminder: Buffer • Message buffers are defined by a tripleaddress ( , count, datatype). Blocking operation • Basic data types restrict buffers to homogeneous, contiguous sequences of values in int MPI_Bcast(void* buffer, int count, MPI_Datatype datatype, memory. ↪ int root, MPI_Comm comm) C Scenario A Problem: Want to communicate data describing particles that consists of a position (3 double) Nonblocking operation and a particle species (encoded as an int). int MPI_Ibcast(void* buffer, int count, MPI_Datatype datatype, Solution(?): Communicate positions and species in two separate operations. ↪ int root, MPI_Comm comm, MPI_Request* request) C Scenario B Problem: Have an array real :: a(:), want to communicate only every second entry MPI_Bcast(buffer, count, datatype, root, comm, ierror) a(1:n:2). type(*), dimension(..) :: buffer integer, intent(in) :: count, root Solution(?): Copy data to a temporary array. type(MPI_Datatype), intent(in) :: datatype Derived datatypes are a mechanism for describing arrangements of data in buffers. Gives the MPI type(MPI_Comm), intent(in) :: comm library the opportunity to employ the optimal solution. integer, optional, intent(out) :: ierror F08 TYPE MAP & TYPE SIGNATURE [MPI-3.1, 4.1] MPI_Ibcast(buffer, count, datatype, root, comm, request, ↪ ierror) Terminology: Type map type(*), dimension(..), asynchronous :: buffer integer, intent(in) :: count, root A general datatype is described by its type map, a sequence of pairs of basic datatype type(MPI_Datatype), intent(in) :: datatype and displacement: type(MPI_Comm), intent(in) :: comm Typemap = {(type , disp ), … , (type , disp )} type(MPI_Request), intent(out) :: request 0 0 푛−1 푛−1 integer, optional, intent(out) :: ierror F08 Terminology: Type signature A type signature describes the contents of a message read from a buffer with a general datatype:
Typesig = {type0,…, type푛−1}
Type matching is done based on type signatures alone.
25 EXAMPLE 5. MPI_Type_create_struct 푛 blocks of 푚푖 instances of oldtype푖 with displacement 푑푖 for each 푖 = 1, … , 푛 struct heterogeneous { 6. MPI_Type_create_subarray 푛 dimensional subarray out of an array with elements of int i[4]; type oldtype double d[5]; } 7. MPI_Type_create_darray distributed array with elements of type oldtype C
type, bind(C) :: heterogeneous CONTIGUOUS DATA [MPI-3.1, 4.1.2] integer :: i(4) real(real64) :: d(5) int MPI_Type_contiguous(int count, MPI_Datatype oldtype,
end type ↪ F08 MPI_Datatype* newtype) C
Basic Datatype MPI_Type_contiguous(count, oldtype, newtype, ierror) integer, intent(in) :: count 0 MPI_INT MPI_INTEGER type(MPI_Datatype), intent(in) :: oldtype 4 MPI_INT MPI_INTEGER type(MPI_Datatype), intent(out) :: newtype 8 MPI_INT MPI_INTEGER integer, optional, intent(out) :: ierror F08 12 MPI_INT MPI_INTEGER • Simple concatenation of oldtype 16 MPI_DOUBLE MPI_REAL8 24 MPI_DOUBLE MPI_REAL8 • Results in the same access pattern as using oldtype and specifying a buffer with count greater than one. 32 MPI_DOUBLE MPI_REAL8 40 MPI_DOUBLE MPI_REAL8 oldtype 48 MPI_DOUBLE MPI_REAL8
0 4 8 12 16 24 32 40 48 count = 9
STRUCT DATA [MPI-3.1, 4.1.2]
34 CONSTRUCTORS int MPI_Type_create_struct(int count, const int ↪ array_of_blocklengths[], const MPI_Aint TYPE CONSTRUCTORS [MPI-3.1, 4.1] ↪ array_of_displacements[], const MPI_Datatype ↪ array_of_types[], MPI_Datatype* newtype) C A new derived type is constructed from an existing type oldtype (basic or derived) using type constructors. In order of increasing generality/complexity: MPI_Type_create_struct(count, array_of_blocklengths, ↪ array_of_displacements, array_of_types, newtype, ierror) 1. MPI_Type_contiguous 푛 consecutive instances of oldtype integer, intent(in) :: count, array_of_blocklengths(count) 2. MPI_Type_vector 푛 blocks of 푚 instances of oldtype with stride 푠 integer(kind=MPI_ADDRESS_KIND), intent(in) :: ↪ array_of_displacements(count) 3. MPI_Type_indexed_block 푛 blocks of 푚 instances of oldtype with displacement 푑푖 type(MPI_Datatype), intent(in) :: array_of_types(count) for each 푖 = 1, … , 푛 type(MPI_Datatype), intent(out) :: newtype 4. MPI_Type_indexed 푛 blocks of 푚푖 instances of oldtype with displacement 푑푖 for each integer, optional, intent(out) :: ierror 푖 = 1, … , 푛 F08
26 Caution: Fortran derived data types must be declared sequence or bind(C), see [MPI-3.1, MPI_Type_create_subarray(ndims, array_of_sizes, 17.1.15]. ↪ array_of_subsizes, array_of_starts, order, oldtype, ↪ newtype, ierror) EXAMPLE integer, intent(in) :: ndims, array_of_sizes(ndims), ↪ array_of_subsizes(ndims), array_of_starts(ndims), order struct heterogeneous { type(MPI_Datatype), intent(in) :: oldtype int i[4]; type(MPI_Datatype), intent(out) :: newtype integer, optional, intent(out) :: ierror double d[5]; F08 } EXAMPLE count = 2; array_of_blocklengths[0] = 4; ndims = 2; array_of_displacements[0] = 0; array_of_sizes[] = { 4, 9 }; array_of_types[0] = MPI_INT; array_of_subsizes[] = { 2, 3 }; array_of_blocklengths[1] = 5; array_of_starts[] = { 0, 3 }; array_of_displacements[1] = 16; order = MPI_ORDER_C; array_of_types[1] = MPI_DOUBLE; C oldtype = MPI_INT; C type, bind(C) :: heterogeneous ndims = 2 integer :: i(4) array_of_sizes(:) = (/ 4, 9 /) real(real64) :: d(5) array_of_subsizes(:) = (/ 2, 3 /) end type array_of_starts(:) = (/ 0, 3 /) order = MPI_ORDER_FORTRAN count = 2; oldtype = MPI_INTEGER array_of_blocklengths(1) = 4 F08 array_of_displacements(1) = 0 An array with global size 4 × 9 containing a subarray of size 2 × 3 at offsets 0, 3: array_of_types(1) = MPI_INTEGER array_of_blocklengths(2) = 5 array_of_displacements(2) = 16 array_of_types(2) = MPI_REAL8 F08
0 4 8 12 16 24 32 40 48
COMMIT & FREE [MPI-3.1, 4.1.9] SUBARRAY DATA [MPI-3.1, 4.1.3] Before using a derived datatype in communication it needs to be committed int MPI_Type_create_subarray(int ndims, const int ↪ array_of_sizes[], const int array_of_subsizes[], const int int MPI_Type_commit(MPI_Datatype* datatype) C ↪ array_of_starts[], int order, MPI_Datatype oldtype, ↪ MPI_Datatype* newtype)
C MPI_Type_commit(datatype, ierror) type(MPI_Datatype), intent(inout) :: datatype integer, optional, intent(out) :: ierror F08
27 Marking derived datatypes for deallocation 36 ADDRESS CALCULATION int MPI_Type_free(MPI_Datatype *datatype)
C ALIGNMENT & PADDING
MPI_Type_free(datatype, ierror) struct heterogeneous { type(MPI_Datatype), intent(inout) :: datatype int i[3]; integer, optional, intent(out) :: ierror double d[5]; F08 }
count = 2; 35 EXERCISES array_of_blocklengths[0] = 3; Exercise 5 – Derived Datatypes array_of_displacements[0] = 0; array_of_types[0] = MPI_INT; 5.1 Matrix Access – Diagonal array_of_blocklengths[1] = 5; In the file matrix_access.{c|cxx|f90|py} implement the function/subroutine array_of_displacements[1] = 16; get_diagonal that extracts the elements on the diagonal of an 푁 × 푁 matrix into a vector: array_of_types[1] = MPI_DOUBLE; C vector = matrix , 푖 = 1 … 푁. 푖 푖,푖 type, bind(C) :: heterogeneous Do not access the elements of either the matrix or the vector directly. Rather, use MPI datatypes for integer :: i(3) accessing your data. Assume that the matrix elements are stored in row-major order in C (all real(real64) :: d(5) elements of the first row, followed by all elements of the second row, etc.), column-major order in end type Fortran. count = 2; Hint: MPI_Sendrecv on the MPI_COMM_SELF communicator can be used for copying the array_of_blocklengths(1) = 3 data. array_of_displacements(1) = 0 Use: MPI_Type_vector array_of_types(1) = MPI_INTEGER array_of_blocklengths(2) = 5 5.2 Matrix Access – Upper Triangle array_of_displacements(2) = 16 array_of_types(2) = MPI_REAL8 In the file matrix_access.{c|cxx|f90|py} implement the function/subroutine F08 get_upper that copies all elements on or above the diagonal of an 푁 × 푁 matrix to a second matrix and leaves all other elements untouched. 0 4 8 12 16 24 32 40 48
upper푖,푗 = matrix푖,푗, 푖=1…푁,푗=푖…푁
As in the previous exercise, do not access the matrix elements directly and assume row-major ADDRESS CALCULATION [MPI-3.1, 4.1.5] layout of the matrices in C, column-major order in Fortran. Make sure to un-comment the call to test_get_upper() to have your solution tested. Displacements are calculated as the difference between the addresses at the start of a buffer and Hint: MPI_Sendrecv on the MPI_COMM_SELF communicator can be used for copying the at a particular piece of data in the buffer. The address of a location in memory is found using: data. int MPI_Get_address(const void* location, MPI_Aint* address) Use: MPI_Type_indexed C
28 DRS ARITHMETIC ADDRESS Addition EXAMPLE Subtraction address. an as same the necessarily not operator C the Using C F08 C F08 C F08 P_yecet_tut2 lcln ip,tps &newtype); types, displ, MPI_Type_commit(&newtype); blocklen, base); MPI_Type_create_struct(2, MPI_Aint_diff(displ[1], = displ[1] &displ[1]); base); MPI_Get_address(&h.d, MPI_Aint_diff(displ[0], = displ[0] &displ[0]); MPI_Get_address(&h.i, &base); MPI_Get_address(&h, }; MPI_DOUBLE MPI_INT, { int = types[2] MPI_Datatype newtype; displ[2]; MPI_Datatype base, MPI_Aint heterogeneous struct integer integer b) MPI_Aint a, MPI_Aint_diff(MPI_Aint MPI_Aint integer integer b) MPI_Aint a, MPI_Aint_add(MPI_Aint MPI_Aint integer integer type ierror) address, MPI_Get_address(location, lcln2 ,5}; 5 3, { = blocklen[2] (*), (kind=MPI_ADDRESS_KIND), b) MPI_Aint_add(a, (kind=MPI_ADDRESS_KIND) , (kind=MPI_ADDRESS_KIND), (kind=MPI_ADDRESS_KIND), b) MPI_Aint_diff(a, (kind=MPI_ADDRESS_KIND) optional dimension & odtrieadessi icuae,snei eun one hc is which pointer a returns it since discouraged, is addresses determine to MI31 4.1.5] [MPI-3.1, , (..), intent h; snhoos:: asynchronous (out) :: intent intent intent ierror (in) (in) (out) location :: :: :: ,b a, b a, address 29 YEEXTENT TYPE RESIZE by added be can padding Explicit 'c' 'c' Let spacing and Extent Extent Terminology: PADDING 37
C t F08 , , lcln=( ,5/) 5 3, /) (/ MPI_REAL8 = MPI_INTEGER, blocklen (/ = types :: integer type integer type int call base) MPI_Aint_diff(displ(2), call = displ(2) base) call MPI_Aint_diff(displ(1), = displ(1) call call eatp ihtp map type with type a be The ↪ ↪ 'd' 'd' MI31 4.1.7] [MPI-3.1, extent (MPI_Datatype) (heterogeneous) P_itetn,MIDttp*newtype) MPI_Datatype* extent, lb, MPI_Aint MPI_Aint oldtype, MPI_Type_create_resized(MPI_Datatype newtype) } , MPI_Type_commit(newtype) types, displ, blocklen, MPI_Type_create_struct(2, displ(2)) MPI_Get_address(h%d, displ(1)) MPI_Get_address(h%i, base) MPI_Get_address(h, n not and 'e' MI31 4.1] [MPI-3.1, (kind=MPI_ADDRESS_KIND) fatp sdtrie rmits from determined is type a of , blocklen(2) 'f' { Typemap 'b' extent ub lb Typemap Typemap } , then , (P_HR 1)} {(MPI_CHAR, Typemap 'd' :: {( = resizing :: , P_edb ,t ...) t, 3, MPI_Send(b, = = ye() newtype types(2), type 'f' min h max 푗 = 푗 h type. the 0 } disp ub , ( . disp disp Typemap 푗 0 푗 ,…,( , … ), oe bounds lower + :: sizeof and ae displ(2) base, − type b lb type naryof array an 푛−1 Typemap 푗 휖 + ) and , disp ilrsl namessage a in result will pe bounds upper 푛−1 char )} , { = b : 'a' , { 'b' 'b' , , rt function a write ESG ASSEMBLY MESSAGE MPI_Type_size using queried be can MPI_Type_get_true_extent type a of extent true and Extent bound type derived new a Creates rdc orc results? correct produce Modification: type. point the the using of by instance tested an be copying will for function construct Your you layout. datatype data the describes that Datatype MPI committed additional with space three-dimensional in point a represents properties: that datatype a of definition a Given Structs 5.3 EXERCISES 38 F08 • • • integer type type ierror) newtype, integer extent, lb, MPI_Type_create_resized(oldtype, a 1character) (1 tag 1 precision) double z, y, (x, coordinates 3 b, g, (r, values color 3 lb n xlctuprbound upper explicit and P_evbfe,4 (b ) MIIT ),...) 4)}, (MPI_INT, 0), {(lb, 4, MPI_Recv(buffer, ...) 8)}, (ub, 0), {(MPI_INT, 4, MPI_Send(buffer, (MPI_Datatype), (MPI_Datatype), Message Buffer Buffer hneteodro h opnnso h on tutr.De orpormstill program your Does structure. point the of components the of order the Change , (kind=MPI_ADDRESS_KIND), point_datatype optional . newtype 0 255] [0, ∈ , ? 0 0 intent intent intent h ieo eutn esgscnb ure with queried be can messages resulting of size The . 0 2 1 b+extent + lb in ) ihtesm yempas map type same the with struct.{c|cxx|f90} ? 4 2 (out) (out) (in) 2 6 3 :: :: :: intent ? 4 . MPI_Type_get_extent ierror oldtype newtype 4 5 (in) ? 6 or oldtype :: 6 7 struct_.py b extent lb, ? 8 u xlctlower explicit but ? 9 and htrtrsa returns that ...... 30 MPI_Type_commit Use: MPI_Get_address , MPI_Aint_diff , MPI_Type_create_struct , PART VII Task-Local Files + Simple to implement INPUT/OUTPUT + No explicit coordination between processes needed + No false sharing of file system blocks 39 INTRODUCTION - Number of files quickly becomes unmanageable MOTIVATION - Files often need to be merged to create a canonical dataset (post-processing) - File system might introduce implicit coordination (metadata modification) I/O on HPC Systems
• “This is not your parents’ I/O subsystem” SEQUENTIAL ACCESS TO METADATA • File system is a shared resource Juqueen, IBM Blue Gene/Q, GPFS, filesystem /work using fopen() – Modification of metadata might happen sequentially parallel creation of task-local files – File system blocks might be shared among processes 103 777.8 • File system access might not be uniform across all processes 410.2
• Interoperability of data originating on different platforms ) 240.6 −1
MPI I/O (푠 102 73.8 • MPI already defines a language that describes data layout and movement Time • Extend this language by I/O capabilities 34.7 24.8 26.9 • More expressive/precise API than POSIX I/O affords better chances for optimization 101 COMMON I/O STRATEGIES 215 216 217 218 219 220 221 Number of files Funnelled I/O + Simple to implement - I/O bandwidth is limited to the rate of this single process 40 FILEMANIPULATION - Additional communication might be necessary FILE,FILEPOINTER&HANDLE [MPI-3.1, 13.1]
- Other processes may idle and waste resources during I/O operations Terminology: File All or several processes use one file An MPI file is an ordered collection of typed data items. + Number of files is independent of number of processes Terminology: File Pointer + File is in canonical representation (no post-processing) A file pointer is an implicit offset into a file maintained by MPI. - Uncoordinated client requests might induce time penalties Terminology: File Handle - File layout may induce false sharing of file system blocks An opaque MPI object. All operations on an open file reference the file through the file handle.
31 OPENINGAFILE [MPI-3.1, 13.2.1] CLOSING A FILE [MPI-3.1, 13.2.2]
int MPI_File_open(MPI_Comm comm, const char* filename, int int MPI_File_close(MPI_File* fh) C ↪ amode, MPI_Info info, MPI_File* fh) C MPI_File_close(fh, ierror) MPI_File_open(comm, filename, amode, info, fh, ierror) type(MPI_File), intent(out) :: fh type(MPI_Comm), intent(in) :: comm integer, optional, intent(out) :: ierror F08 character(len=*), intent(in) :: filename integer, intent(in) :: amode • Collective operation type(MPI_Info), intent(in) :: info • User must ensure that all outstanding nonblocking and split collective operations type(MPI_File), intent(out) :: fh associated with the file have completed integer, optional, intent(out) :: ierror F08
• Collective operation on communicator comm DELETINGAFILE [MPI-3.1, 13.2.3] • Filename must reference the same file on all processes int MPI_File_delete(const char* filename, MPI_Info info) C • Process-local files can be opened using MPI_COMM_SELF • info object specifies additional information (MPI_INFO_NULL for empty) MPI_File_delete(filename, info, ierror) character(len=*), intent(in) :: filename type(MPI_Info), intent(in) :: info ACCESS MODE [MPI-3.1, 13.2.1] integer, optional, intent(out) :: ierror F08 amode denotes the access mode of the file and must be the same on all processes. It must contain • Deletes the file identified by filename exactly one of the following: • File deletion is a local operation and should be performed by a single process MPI_MODE_RDONLY read only access • If the file does not exist an error is raised MPI_MODE_RDWR read and write access • If the file is opened by any process MPI_MODE_WRONLY write only access – all further and outstanding access to the file is implementation dependent and may contain some of the following: – it is implementation dependent whether the file is deleted; if it is not, an error is MPI_MODE_CREATE create the file if it does not exist raised MPI_MODE_EXCL error if creating file that already exists MPI_MODE_DELETE_ON_CLOSE delete file on close FILE PARAMETERS MPI_MODE_UNIQUE_OPEN file is not opened elsewhere Setting File Parameters MPI_MODE_SEQUENTIAL access to the file is sequential MPI_File_set_size Set the size of a file [MPI-3.1, 13.2.4] MPI_MODE_APPEND file pointers are set to the end of the file MPI_File_preallocate Preallocate disk space [MPI-3.1, 13.2.5] Combine using bit-wise or (| operator in C, ior intrinsic in Fortran). MPI_File_set_info Supply additional information [MPI-3.1, 13.2.8]
32 Inspecting File Parameters – MPI_Type_create_struct, MPI_File_get_size Size of a file [MPI-3.1, 13.2.6] – MPI_Type_create_subarray MPI_File_get_amode Acess mode [MPI-3.1, 13.2.7] • Displacements in their typemap must be non-negative and monotonically nondecreasing MPI_File_get_group Group of processes that opened the file [MPI-3.1, 13.2.7] • Have to be committed before use MPI_File_get_info Additional information associated with the file [MPI-3.1, 13.2.8] DEFAULT FILE VIEW [MPI-3.1, 13.3] I/OERRORHANDLING [MPI-3.1, 8.3, 13.7] When newly opened, files are assigned a default view that is the same on all processes: Caution: Communication, by default, aborts the program when an error is encountered. I/O • Zero displacement operations, by default, return an error code. • File contains a contiguous sequence of bytes int MPI_File_set_errhandler(MPI_File file, MPI_Errhandler • All processes have access to the entire file ↪ errhandler) C File 0: byte 1: byte 2: byte 3: byte ... MPI_File_set_errhandler(file, errhandler, ierror) Process 0 0: byte 1: byte 2: byte 3: byte ... type(MPI_File), intent(in) :: file type(MPI_Errhandler), intent(in) :: errhandler Process 1 0: byte 1: byte 2: byte 3: byte ... integer, optional, intent(out) :: ierror F08 ... 0: byte 1: byte 2: byte 3: byte ... • The default error handler for files is MPI_ERRORS_RETURN MPI_SUCCESS • Success is indicated by a return value of ELEMENTARY TYPE [MPI-3.1, 13.3] • MPI_ERRORS_ARE_FATAL aborts the program Terminology: Elementary Type • Can be set for each file individually or for all files by using MPI_File_set_errhandler on a special file handle, MPI_FILE_NULL An elementary type (or etype) is the unit of data contained in a file. Offsets are expressed in multiples of etypes, file pointers point to the beginning of etypes. Etypes can be basic or derived. 41 FILEVIEWS Changing the Elementary Type E.g. etype = MPI_INT: FILEVIEW [MPI-3.1, 13.3] File 0: int 1: int 2: int 3: int ... Terminology: File View Process 0 0: int 1: int 2: int 3: int ... A file view determines what part of the contents of a file is visible to a process. It is defined by a displacement (given in bytes) from the beginning of the file, an Process 1 0: int 1: int 2: int 3: int ... elementary datatype and a file type. The view into a file can be changed multiple ...... times between opening and closing. 0: int 1: int 2: int 3: int File Types and Elementary Types are Data Types • Can be predefined or derived • The usual constructors can be used to create derived file types and elementary types, e.g. – MPI_Type_indexed,
33 FILE TYPE [MPI-3.1, 13.3] "native" Data is stored in the file exactly as it is in memory Terminology: File Type + No loss of precision A file type describes an access pattern. It can contain either instances of the etype or holes with an extent that is divisible by the extent of the etype. + No overhead Changing the File Type - On heterogeneous systems loss of transparent interoperability E.g. Filetype0 = {(int, 0), (hole, 4), (hole, 8)}, Filetype1 = {(hole, 0), (int, 4), (hole, 8)}, …: "internal" Data is stored in implementation-specific format File 0: int 1: int 2: int 3: int ... + Can be used in a homogeneous and heterogeneous environment Process 0 0: int 1: int ... + Implementation will perform conversions if necessary Process 1 0: int ... - Can incur overhead ...... 0: int - Not necessarily compatible between different implementations "external32" CHANGINGTHEFILEVIEW [MPI-3.1, 13.3] Data is stored in standardized data representation (big-endian IEEE) + Can be read/written also by non-MPI programs int MPI_File_set_view(MPI_File fh, MPI_Offset disp, - Precision and I/O performance may be lost due to type conversions between native and ↪ MPI_Datatype etype, MPI_Datatype filetype, const char* external32 representations ↪ datarep, MPI_Info info) C - Not available in all implementations MPI_File_set_view(fh, disp, etype, filetype, datarep, info, ↪ ierror) type(MPI_File), intent(in) :: fh 42 DATAACCESS integer(kind=MPI_OFFSET_KIND), intent(in) :: disp Three orthogonal aspects type(MPI_Datatype), intent(in) :: etype, filetype character(len=*), intent(in) :: datarep 1. Synchronism type(MPI_Info), intent(in) :: info (a) Blocking integer, optional, intent(out) :: ierror F08 (b) Nonblocking • Collective operation (c) Split collective • datarep and extent of etype must match 2. Coordination • disp, filetype and info can be distinct (a) Noncollective • File pointers are reset to zero (b) Collective • May not overlap with nonblocking or split collective operations 3. Positioning
DATA REPRESENTATION [MPI-3.1, 13.5] (a) Explicit offsets • Determines the conversion of data in memory to data on disk (b) Individual file pointers • Influences the interoperability of I/O between heterogeneous parts of a system or different (c) Shared file pointers systems
34 POSIX read() and write() Shared File Pointers These are blocking, noncollective operations with individual file pointers. • All processes share a single file pointer • All processes must use the same file view SYNCHRONISM • Individual accesses appear as if serialized (with an unspecified order) Blocking I/O Blocking I/O routines do not return before the operation is completed. • Collective accesses are performed in order of ascending rank Nonblocking I/O • Nonblocking I/O routines do not wait for the operation to finish • A separate completion routine is necessary [MPI-3.1, 3.7.3, 3.7.5] Combine the prefix MPI_File_ with any of the following suffixes: • The associated buffers must not be used while the operation is in flight coordination Split Collective positioning synchronism noncollective collective • “Restricted” form of nonblocking collective blocking read_at, write_at read_at_all, explicit offsets write_at_all • Buffers must not be used while in flight nonblocking iread_at, iwrite_at iread_at_all, • Does not allow other collective accesses to the file while in flight iwrite_at_all
• begin and end must be used from the same thread split collective N/A read_at_all_begin, read_at_all_end, write_at_all_begin, COORDINATION write_at_all_end
Noncollective blocking read, write read_all, write_all individual file The completion depends only on the activity of the calling process. pointers nonblocking iread, iwrite iread_all, iwrite_all Collective split collective N/A read_all_begin, • Completion may depend on activity of other processes read_all_end, write_all_begin, • Opens opportunities for optimization write_all_end
blocking read_shared, read_ordered, POSITIONING [MPI-3.1, 13.4.1 -- 13.4.4] shared file write_shared write_ordered pointers Explicit Offset nonblocking iread_shared, N/A iwrite_shared • No file pointer is used split collective N/A read_ordered_begin, • File position for access is given directly as function argument read_ordered_end, write_ordered_begin, Individual File Pointers write_ordered_end • Each process has its own file pointer • After access, pointer is moved to first etype after the last one accessed
35 WRITING WRITING blocking, noncollective, explicit offset [MPI-3.1, 13.4.2] blocking, noncollective, individual [MPI-3.1, 13.4.3]
int MPI_File_write_at(MPI_File fh, MPI_Offset offset, const int MPI_File_write(MPI_File fh, const void* buf, int count, ↪ void* buf, int count, MPI_Datatype datatype, MPI_Status ↪ MPI_Datatype datatype, MPI_Status* status) C ↪ *status) C MPI_File_write(fh, buf, count, datatype, status, ierror) MPI_File_write_at(fh, offset, buf, count, datatype, status, type(MPI_File), intent(in) :: fh ↪ ierror) type(*), dimension(..), intent(in) :: buf type(MPI_File), intent(in) :: fh integer, intent(in) :: count integer(kind=MPI_OFFSET_KIND), intent(in) :: offset type(MPI_Datatype), intent(in) :: datatype type(*), dimension(..), intent(in) :: buf type(MPI_Status) :: status integer, intent(in) :: count integer, optional, intent(out) :: ierror F08 type(MPI_Datatype), intent(in) :: datatype integer, optional, intent(out) :: ierror • Starts writing at the current position of the individual file pointer F08 • Moves the individual file pointer by the count of etypes written • Starting offset for access is explicitly given
• No file pointer is updated EXAMPLE count datatype buf • Writes elements of from memory starting at blocking, noncollective, individual [MPI-3.1, 13.4.3] = … • Typesig datatype Typesig etype Typesig etype With its file pointer at element 1, process 1 calls MPI_File_write(count = 2): • Writing past end of file increases the file size File 0 1 2 3 4 5 6 7 8 9 ... EXAMPLE blocking, noncollective, explicit offset [MPI-3.1, 13.4.2] Process 0 0 1 2 3 ... Process 0 calls MPI_File_write_at(offset = 1, count = 2): Process 1 0 1 2 ... File 0 1 2 3 4 5 6 7 8 9 ...
Process 2 0 1 2 ... Process 0 0 1 2 3 ...
Process 1 0 1 2 ... WRITING nonblocking, noncollective, individual [MPI-3.1, 13.4.3]
Process 2 0 1 2 ... int MPI_File_iwrite(MPI_File fh, const void* buf, int count, ↪ MPI_Datatype datatype, MPI_Request* request) C
36 WRITING lcig olcie individual collective, blocking, EXAMPLE individual collective, blocking, F08 C F08 • • • • • • • • integer type type integer ierror) status, type datatype, type count, buf, MPI_File_write_all(fh, int integer type type integer ierror) type request, datatype, type count, buf, MPI_File_iwrite(fh, ihisfl one teeet2 rcs calls 2 process 2, element at pointer file its With calls 1 process 0, element at pointer file its With calls 0 process 1, element at pointer file its With I/O funnel to processes between communication use can MPI pointer file individual its uses process Each as signature Same a Returns as operation same the Starts ↪ (MPI_Status) (MPI_Datatype), (*), (MPI_File), status) MPI_Status* datatype, MPI_Datatype fh, MPI_File_write_all(MPI_File (MPI_Request), (MPI_Datatype), (*), (MPI_File), , , , , request optional intent dimension optional intent dimension MPI_File_write (in) (in) betta sue ocmlt h operation the complete to used is that object intent intent , :: , MI31 13.4.3] [MPI-3.1, MI31 13.4.3] [MPI-3.1, (..), (..), intent intent intent status intent :: intent :: MPI_File_write (in) (in) count count intent intent (out) (out) (out) (in) (in) :: :: u olciecoordination collective but , (in) fh (in) fh :: :: :: :: :: ierror datatype ierror request datatype P_iewiealcut=0) = MPI_File_write_all(count 2) = MPI_File_write_all(count 1) = MPI_File_write_all(count :: :: u osntwi o completion for wait not does but os void const buf buf buf, * int count, : , , 37 WRITING WRITING pi-olcie individual split-collective, individual split-collective, F08 C F08 C • • • integer type type ierror) status, type buf, MPI_File_write_all_end(fh, int integer type integer ierror) type datatype, count, type buf, MPI_File_write_all_begin(fh, int buf status as operation Same ↪ ↪ rcs 0 Process rcs 1 Process rcs 2 Process (MPI_Status) (*), (MPI_File), status) MPI_Status* fh, MPI_File_write_all_end(MPI_File (MPI_Datatype), (*), (MPI_File), datatype) MPI_Datatype count, fh, MPI_File_write_all_begin(MPI_File ruetms ac corresponding match must argument File , , , srtre ytecorresponding the by returned is optional dimension optional intent dimension MI31 13.4.5] [MPI-3.1, 13.4.5] [MPI-3.1, MPI_File_write_all (in) intent intent , :: , 0 0 (..), (..), intent intent status intent :: 0 1 (in) (in) count intent intent 2 0 (out) (out) (in) :: :: 1 3 (in) fh (in) fh end begin :: :: :: 1 4 u split-collective but , routine ierror ierror datatype :: :: routine 5 1 buf buf 2 6 os void const os void const 7 2 8 2 buf, * 3 9 buf, * ...... int EXAMPLE ... blocking, noncollective, shared [MPI-3.1, 13.4.4] File 0 1 2 3 4 5 6 7 8 9 With the shared pointer at element 2, ... • process 0 calls MPI_File_write_shared(count = 3), Process 0 0 1 2 3 4 5 6 7 8 9 • process 2 calls MPI_File_write_shared(count = 2): ... Scenario 1: Process 1 0 1 2 3 4 5 6 7 8 9
File 0 1 2 3 4 5 6 7 8 9 ... Process 2 0 1 2 3 4 5 6 7 8 9 ...
Process 0 0 1 2 3 4 5 6 7 8 9 ... READING blocking, noncollective, individual [MPI-3.1, 13.4.3] Process 1 0 1 2 3 4 5 6 7 8 9 ... int MPI_File_read(MPI_File fh, void* buf, int count, ↪ MPI_Datatype datatype, MPI_Status* status) C Process 2 0 1 2 3 4 5 6 7 8 9 ... MPI_File_read(fh, buf, count, datatype, status, ierror) type(MPI_File), intent(in) :: fh Scenario 2: type(*), dimension(..) :: buf integer, intent(in) :: count type(MPI_Datatype), intent(in) :: datatype File 0 1 2 3 4 5 6 7 8 9 ... type(MPI_Status) :: status integer, optional, intent(out) :: ierror F08 Process 0 0 1 2 3 4 5 6 7 8 9 ... • Starts reading at the current position of the individual file pointer • Reads up to count elements of datatype into the memory starting at buf Process 1 0 1 2 3 4 5 6 7 8 9 ... • status indicates how many elements have been read • If status indicates less than count elements read, the end of file has been reached Process 2 0 1 2 3 4 5 6 7 8 9 ... FILEPOINTERPOSITION [MPI-3.1, 13.4.3]
EXAMPLE int MPI_File_get_position(MPI_File fh, MPI_Offset* offset) C blocking, collective, shared [MPI-3.1, 13.4.4] MPI_File_get_position(fh, offset, ierror) With the shared pointer at element 2, type(MPI_File), intent(in) :: fh • process 0 calls MPI_File_write_ordered(count = 1), integer(kind=MPI_OFFSET_KIND), intent(out) :: offset integer, optional, intent(out) :: ierror • process 1 calls MPI_File_write_ordered(count = 2), F08 • process 2 calls MPI_File_write_ordered(count = 3): • Returns the current position of the individual file pointer in units of etype
38 • Value can be used for e.g. File 0 1 2 3 4 5 6 7 8 9 – return to this position (via seek)
– calculate a displacement Process 0 0 1 2 3 • MPI_File_get_position_shared queries the position of the shared file pointer
Process 1 0 1 2 SEEKINGTOAFILEPOSITION [MPI-3.1, 13.4.3]
int MPI_File_seek(MPI_File fh, MPI_Offset offset, int whence) C Process 2 0 1 2
MPI_File_seek(fh, offset, whence, ierror) Process 2 calls MPI_File_seek(offset = -1, whence = MPI_SEEK_END): type(MPI_File), intent(in) :: fh integer(kind=MPI_OFFSET_KIND), intent(in) :: offset File 0 1 2 3 4 5 6 7 8 9 integer, intent(in) :: whence integer, optional, intent(out) :: ierror F08 Process 0 0 1 2 3 • whence controls how the file pointer is moved: MPI_SEEK_SET sets the file pointer to offset Process 1 0 1 2 MPI_SEEK_CUR offset is relative to the current value of the pointer MPI_SEEK_END offset is relative to the end of the file Process 2 0 1 2 • offset can be negative but the resulting position may not lie before the beginning of the file • MPI_File_seek_shared manipulates the shared file pointer CONVERTING OFFSETS [MPI-3.1, 13.4.3]
EXAMPLE int MPI_File_get_byte_offset(MPI_File fh, MPI_Offset offset, ↪ MPI_Offset* disp) C Process 0 calls MPI_File_seek(offset = 2, whence = MPI_SEEK_SET): MPI_File_get_byte_offset(fh, offset, disp, ierror) File 0 1 2 3 4 5 6 7 8 9 type(MPI_File), intent(in) :: fh integer(kind=MPI_OFFSET_KIND), intent(in) :: offset integer(kind=MPI_OFFSET_KIND), intent(out) :: disp Process 0 0 1 2 3 integer, optional, intent(out) :: ierror F08
• Converts a view relative offset (in units of etype) into a displacement in bytes from the Process 1 0 1 2 beginning of the file
Process 2 0 1 2
Process 1 calls MPI_File_seek(offset = -1, whence = MPI_SEEK_CUR):
39 43 CONSISTENCY int MPI_File_sync(MPI_File fh) C
CONSISTENCY [MPI-3.1, 13.6.1] MPI_File_sync(fh, ierror) type(MPI_File), intent(in) :: fh Terminology: Sequential Consistency integer, optional, intent(out) :: ierror If a set of operations is sequentially consistent, they behave as if executed in some F08 serial order. The exact order is unspecified. The Sync-Barrier-Sync construct
• To guarantee sequential consistency, certain requirements must be met // writing access sequence through one file handle • Requirements depend on access path and file atomicity MPI_File_sync(fh0); MPI_Barrier(MPI_COMM_WORLD); Caution: Result of operations that are not sequentially consistent is implementation dependent. MPI_File_sync(fh0); // ... C ATOMIC MODE [MPI-3.1, 13.6.1] // ... Requirements for sequential consistency MPI_File_sync(fh1); Same file handle: always sequentially consistent MPI_Barrier(MPI_COMM_WORLD); File handles from same open: always sequentially consistent MPI_File_sync(fh1); // access sequence to the same file through a different file File handles from different open: not influenced by atomicity, see nonatomic mode ↪ handle C • Atomic mode is not the default setting • MPI_File_sync is used to delimit sequences of accesses through different file handles • Can lead to overhead, because MPI library has to uphold guarantees in general case • Sequences that contain a write access may not be concurrent with any other access int MPI_File_set_atomicity(MPI_File fh, int flag) sequence C
MPI_File_set_atomicity(fh, flag, ierror) type(MPI_File), intent(in) :: fh 44 EXERCISES logical, intent(in) :: flag integer, optional, intent(out) :: ierror Exercise 6 – Data Access F08 6.1 Writing Data In the file rank_io.{c|cxx|f90|py} write a function write_rank that takes a NONATOMIC MODE [MPI-3.1, 13.6.1] communicator as its only argument and does the following: Requirements for sequential consistency • Each process writes its own rank in the communicator to a common file rank.dat using Same file handle: operations must be either nonconcurrent, nonconflicting, or both "native" data representation. File handles from same open: nonconflicting accesses are sequentially consistent, conflicting • The ranks should be in order in the file: 0 … 푛 − 1. accesses have to be protected using MPI_File_sync Use: MPI_File_open, MPI_File_set_errhandler, MPI_File_set_view, File handles from different open: all accesses must be protected using MPI_File_sync MPI_File_write_ordered, MPI_File_close Terminology: Conflicting Accesses 6.2 Reading Data Two accesses are conflicting if they touch overlapping parts of a file and at least one In the file rank_io.{c|cxx|f90|py} write a function read_rank that takes a is writing. communicator as its only argument and does the following:
40 • The processes read the integers in the file in reverse order, i.e. process 0 reads the last entry, PART VIII process 1 reads the one before, etc. • Each process returns the rank number it has read from the function. TOOLS Careful: This function might be run on a communicator with a different number of processes. If there are more processes than entries in the file, processes with ranks larger than or equal to the number of file entries should return MPI_PROC_NULL. 45 MUST Use: MPI_File_seek, MPI_File_get_position, MPI_File_read MUST 6.3 Phone Book Marmot Umpire Scalable Tool The file phonebook.dat contains several records of the following form:
struct dbentry { int key; int room_number; https://doc.itc.rwth-aachen.de/display/CCP/Project+MUST int phone_number; MUST checks for correct usage of MPI. It includes checks for the following classes of mistakes: char name[200]; } • Constants and integer values C • Communicator usage type :: dbentry integer :: key • Datatype usage integer :: room_number • Group usage integer :: phone_number character(len=200) :: name • Operation usage end type F08 • Request usage
In the file phonebook.{c|cxx|f90|py} write a function look_up_by_room_number that • Leak checks (MPI resources not freed before calling MPI_Finalize) uses MPI I/O to find an entry by room number. Assume the file was written using "native" data • Type mismatches representation. Use MPI_COMM_SELF to open the file. Return a bool or logical to indicate whether an entry has been found and fill an entry via pointer/intent out argument. • Overlapping buffers passed to MPI • Deadlocks resulting from MPI calls • Basic checks for thread level usage (MPI_Init_thread)
MUST USAGE
Build your application:
$ mpicc -o application.x application.c $ # or $ mpif90 -o application.x application.f90
Replace the MPI starter (e.g. srun) with MUST’s own mustrun:
41 $ mustrun -n 4 --must:mpiexec srun --must:np -n ./application.x PART IX
Different modes of operation (for improved scalability or graceful handling of application crashes) are available via command line switches. COMMUNICATORS Caution: MUST is not compatible with MPI’s Fortran 2008 interface. 47 INTRODUCTION
46 EXERCISES MOTIVATION
Exercise 7 – MPI Tools Communicators are a scope for communication within or between groups of processes. New 7.1 Must communicators with different scope or topological properties can be used to accommodate certain Have a look at the file must.{c|c++|f90}. It contains a variation of the solution to exercise 2.3 needs. – it should calculate the sum of all ranks and make the result available on all processes. • Separation of communication spaces: A software library that uses MPI underneath isused 1. Compile the program and try to run it. in an application that directly uses MPI itself. Communication due to the library should not conflict with communication due to the application. 2. Use MUST to discover what is wrong with the program. • Partitioning of process groups: Parts of your software exhibit a collective communication 3. If any mistakes were found, fix them and go back to 1. pattern, but only across a subset of processes. Note: must.f90 uses the MPI Fortran 90 interface. • Exploiting inherent topology: Your application uses a regular cartesian grid to discretize the problem and this translates into certain nearest neighbor communication patterns.
48 CONSTRUCTORS
DUPLICATE [MPI-3.1, 6.4.2]
int MPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm) C
MPI_Comm_dup(comm, newcomm, ierror) type(MPI_Comm), intent(in) :: comm type(MPI_Comm), intent(out) :: newcomm integer, optional, intent(out) :: ierror F08
• Duplicates an existing communicator comm • New communicator has the same properties but a new context
SPLIT [MPI-3.1, 6.4.2]
int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm ↪ *newcomm) C
42 ATSA TOPOLOGY CARTESIAN F08 C F08 • • • • • • integer type logical integer type reorder, periods, dims, ndims, MPI_Cart_create(comm_old, int integer type integer ierror) newcomm, type key, color, MPI_Comm_split(comm, :0 :0 k: 0, c: c: ( rcse r ree yacnigvleof value ascending by ordered are Processes value color Special by grouped are Processes subgroups disjoint into communicator a in processes the Splits h rdhas grid The Cartesian periodic) (possibly a grid on arranged processes with communicator new a Creates ↪ ↪ ↪ MPI_COMM_NULL 0 :0 k: , 0 0 *comm_cart) (MPI_Comm), (MPI_Comm), ierror) comm_cart, dims[], comm_old, MPI_Cart_create(MPI_Comm (MPI_Comm), (MPI_Comm), , , , , , optional intent optional intent intent ndims :0 :1 k: 0, c: c: os int const 1 :1 k: , 1 1 MI31 7.5.1] [MPI-3.1, MPI_UNDEFINED srtre in returned is (in) (in) (in) iesosand dimensions intent intent intent intent , , intent intent color :: :: :: :0 :1 k: 0, c: c: (out) (in) periods[], (out) (in) oo,key color, eid(dm) reorder periods(ndims), dims(ndims) ndims, 0 n e omnctrprdsic value distinct per communicator new one , :1 k: , 2 2 newcomm (out) (out) dims[i] :: :: osntcet e communicator new a create not does :: :: comm comm_old :: comm_cart :: newcomm ) key :1 :0 k: 1, c: c: int 1 onsi dimension in points ierror ierror :0 k: , 0 3 nnwcommunicator new in ere,MPI_Comm reorder, int :1 :1 k: 1, c: c: ndims, 0 :1 k: , 1 4 i os int const :– :0 k: –, c: c: – :0 k: , 5 – 43 9ACCESSORS 49 rnltstern fapoesit t oriaeo h atsa grid. Cartesian the on coordinate its into process a of rank the Translates rcs –1 e omnctrwt oooya shown as topology with communicator new 0–11: process Output: .false. 2 = ndims comm_old Input: AKT COORDINATE TO RANK 12–: process F08 C • integer integer ierror) integer coords, maxdims, type rank, MPI_Cart_coords(comm, int If ↪ reorder (MPI_Comm), coords[]) comm, MPI_Cart_coords(MPI_Comm otis1 rcse o more) (or processes 12 contains MPI_COMM_NULL , is=[4 ] 3 4, [ = dims , , , optional intent intent stu,MIi ret sinnwrnst processes to ranks new assign to free is MPI true, is 0 1 2 MI31 7.5.5] [MPI-3.1, 0 0) (0, 1) (0, 2) (0, (out) (in) 0 1 2 0 intent , intent :: :: , eid fle,.as.]rodr= reorder ] .false. .false., [ = periods (in) ak maxdims rank, coords(maxdims) 1 0) (1, 1) (1, 2) (1, (out) 3 4 5 1 :: comm :: ierror int 2 0) (2, 1) (2, 2) (2, 6 7 8 2 rank, int 3 0) (3, 1) (3, 2) (3, 10 11 9 3 maxdims, int COORDINATE TO RANK [MPI-3.1, 7.5.5] 2 5 8 11 int MPI_Cart_rank(MPI_Comm comm, const int coords[], int ↪ *rank) C
MPI_Cart_rank(comm, coords, rank, ierror) 1 4 7 10 type(MPI_Comm), intent(in) :: comm integer, intent(in) :: coords(*) integer, intent(out) :: rank integer, optional, intent(out) :: ierror F08
Translates the coordinate on the Cartesian grid of a process into its rank. 0 3 6 9
CARTESIAN SHIFT [MPI-3.1, 7.5.6] Input: direction = 0, disp = 1, periodic Output: int MPI_Cart_shift(MPI_Comm comm, int direction, int disp, int process 0: rank_source = 9, rank_dest = 3 ↪ *rank_source, int *rank_dest) C … process 3: rank_source = 0, rank_dest = 6 MPI_Cart_shift(comm, direction, disp, rank_source, rank_dest, … ↪ ierror) process 9: rank_source = 6, rank_dest = 0 type(MPI_Comm), intent(in) :: comm … integer, intent(in) :: direction, disp integer, intent(out) :: rank_source, rank_dest 2 5 8 11 integer, optional, intent(out) :: ierror F08
• Calculates the ranks of source and destination processes in a shift operation on a Cartesian grid • direction gives the number of the axis (starting at 0) 1 4 7 10 • disp gives the displacement Input: direction = 0, disp = 1, not periodic Output: process 0: rank_source = MPI_PROC_NULL, rank_dest = 3 0 3 6 9 … process 3: rank_source = 0, rank_dest = 6 … process 9: rank_source = 6, rank_dest = MPI_PROC_NULL Input: direction = 1, disp = 2, not periodic … Output: process 0: rank_source = MPI_PROC_NULL, rank_dest = 2 process 1: rank_source = MPI_PROC_NULL, rank_dest = MPI_PROC_NULL process 2: rank_source = 0, rank_dest = MPI_PROC_NULL …
44 MPI_UNEQUAL Otherwise. 2 5 8 11
50 DESTRUCTORS
FREE [MPI-3.1, 6.4.3] 1 4 7 10 int MPI_Comm_free(MPI_Comm *comm) C
MPI_Comm_free(comm, ierror) type(MPI_Comm), intent(inout) :: comm 0 3 6 9 integer, optional, intent(out) :: ierror F08
Marks a communicator for deallocation. NULL PROCESSES [MPI-3.1, 3.11]
int MPI_PROC_NULL = /* implementation defined */ 51 EXERCISES C Exercise 8 – Communicators integer, parameter :: MPI_PROC_NULL = ! implementation defined F08 8.1 Cartesian Topology
• Can be used as source or destination for point-to-point communication In global_sum_with_communicators.{c|cxx|f90|py}, redo exercise 2.3 using a Cartesian communicator. • Communication with MPI_PROC_NULL has no effect Use: MPI_Cart_create, MPI_Cart_shift, MPI_Comm_free • May simplify code structure (communication with special source/destination instead of branch) 8.2 Split • MPI_Cart_shift returns MPI_PROC_NULL for out of range shifts In global_sum_with_communicators.{c|cxx|f90|py}, redo exercise 3.1 using a new split communicator per communication round. COMPARISON [MPI-3.1, 6.4.1] Use: MPI_Comm_split
int MPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int ↪ *result) C
MPI_Comm_compare(comm1, comm2, result, ierror) type(MPI_Comm), intent(in) :: comm1, comm2 integer, intent(out) :: result integer, optional, intent(out) :: ierror F08
Compares two communicators. The result is one of: MPI_IDENT The two communicators are the same. MPI_CONGRUENT The two communicators consist of the same processes in the same order but communicate in different contexts. MPI_SIMILAR The two communicators consist of the same processes in a different order.
45 PART X MPI_Init_thread(required, provided, ierror) integer, intent(in) :: required integer, intent(out) :: provided THREAD COMPLIANCE integer, optional, intent(out) :: ierror F08
• required and provided specify thread support levels 52 INTRODUCTION • If possible, provided = required THREAD COMPLIANCE [MPI-3.1, 12.4] • Otherwise, if possible, provided > required • An MPI library is thread compliant if • Otherwise, provided < required 1. Concurrent threads can make use of MPI routines and the result will be as if they were • MPI_Init is equivalent to required = MPI_THREAD_SINGLE executed in some order.
2. Blocking routines will only block the executing thread, allowing other threads to make INQUIRY FUNCTIONS [MPI-3.1, 12.4.3] progress. • MPI libraries are not required to be thread compliant Query level of thread support: • Alternative initialization routines to request certain levels of thread compliance int MPI_Query_thread(int *provided) C • These functions are always safe to use in a multithreaded setting: MPI_Initialized, MPI_Finalized, MPI_Query_thread, MPI_Query_thread(provided, ierror) MPI_Is_thread_main, MPI_Get_version, integer, intent(out) :: provided integer, optional, intent(out) :: ierror
MPI_Get_library_version F08
Check whether the calling thread is the main thread: 53 ENABLINGTHREADSUPPORT int MPI_Is_thread_main(int* flag) C THREAD SUPPORT LEVELS [MPI-3.1, 12.4.3] MPI_Is_thread_main(flag, ierror) The following predefined values are used to express all possible levels of thread support: logical, intent(out) :: flag integer, optional, intent(out) :: ierror MPI_THREAD_SINGLE program is single threaded F08 MPI_THREAD_FUNNELED MPI routines are only used by the main thread MPI_THREAD_SERIALIZED MPI routines are used by multiple threads, but not concurrently 54 MATCHINGPROBEANDRECEIVE
MPI_THREAD_MULTIPLE MPI is thread compliant, no restrictions MATCHING PROBE [MPI-3.1, 3.8.2] MPI_THREAD_SINGLE < MPI_THREAD_FUNNELED < MPI_THREAD_SERIALIZED < MPI_THREAD_MULTIPLE int MPI_Mprobe(int source, int tag, MPI_Comm comm, ↪ MPI_Message* message, MPI_Status* status) C INITIALIZATION [MPI-3.1, 12.4.3]
int MPI_Init_thread(int* argc, char*** argv, int required, ↪ int* provided) C
46 MPI_Recv settings, multithreaded In (e.g. request Probe same the complete to try not must threads Multiple the thread, Completion same Request the on occur should MPI of finalization and Initialization Finalization and Initialization CLARIFICATIONS REMARKS 55 RECEIVE MATCHED F08 C F08 • • • • • integer type type type ierror) status, integer message, type datatype, count, MPI_Mrecv(buf, int integer type type ierror) type status, message, integer comm, tag, MPI_Mprobe(source, obokn variant Nonblocking to handle message the Sets message probed previously the Receives variant Nonblocking message probed the exactly receive to like Works ↪ (MPI_Status) (MPI_Message), (MPI_Datatype), (*), status) MPI_Status* message, MPI_Message* MPI_Mrecv( (MPI_Status) (MPI_Message), (MPI_Comm), . , , , , MPI_Probe optional intent optional intent dimension MI31 12.4.2] [MPI-3.1, MI31 3.8.3] [MPI-3.1, void (in) (in) MPI_Probe MPI_Imrecv MPI_Improbe intent , :: , :: xetfrtereturned the for except , (..) buf, * intent intent intent intent status intent :: status :: MPI_MESSAGE_NULL (in) count :: tag source, ih ac ifrn esg sasubsequent as message different a match might (out) (inout) int (out) (out) exists (in) buf exists :: on,MIDttp datatype, MPI_Datatype count, message comm :: :: :: :: :: ierror datatype ierror message MPI_Message message MPI_Wait au hc a eused be may which value anthread main ). . 47 6WA SOPENMP? IS WHAT 56 OPENMP WITH STEPS FIRST XI PART https://www.openmp.org/about/openmp-faq/ 2020 2018 2015 2013 2011 2008 2005 2002 2000 1999 1998 1997 HISTORY BRIEF 2020) (November 5.1 specification: the of version Current 4 atisvnWvrne al. et Waveren van Matthijs • • • eso .,spotfrnwrbs agae,lo rnfrain,compare-and-swap, transformations, loop languages, base newer for support 5.1, Version and task extended allocators, compatibility, language base model, memory 5.0, Version priorities task facilities, devices and SIMD extended 4.5, Version and cancellation), groups, (dependencies, tasks devices, SIMD, affinity, thread 4.0, Version facilities task extended 3.1, Version tasks 3.0, Version clarifications variables, control internal model, memory 2.5, version combined First 2.0 version C/C++ 2.0 version FORTRAN 1.1 version FORTRAN 1.0 version C/C++ 1.0 version FORTRAN xedddvcsfacilities devices extended facilities devices compatibility 2003 Fortran improved platforms different across Portable Provides etc. accelerators, DSPs, also now systems, SMP targeted Initially Fortran in ( parallelism programs. high-level C/C++ specify and to and used routines, be library can directives, that compiler variables of environment set a for specification a is OpenMP specifications pnPFAQ OpenMP pnPFAQ OpenMP ntimplementations) (not eso ..Jn ,2018. 6, June 3.0. version . 4 ) vstdo 01/30/2019). on (visited URL : COVERAGE – Device Information Routines • Directives – Device Memory Routines – Directive Format ✓ – Lock Routines ✓ – Conditional Compilation ✓ – Timing Routines – Variant Directives – Event Routine – Internal Control Variables ✓ – Interoperability Routines – Informational and Utility Directives – Memory Management Routines – parallel Construct ✓ – Tool Control Routine – teams Construct – Environment Display Routine – masked Construct – … – scope Construct • OMPT Interface – Worksharing Constructs ✓ • OMPD Interface – Loop-Related Directives (✓) • Environment Variables (✓) – Tasking Constructs ✓ • … – Memory Management Directives LITERATURE – Device Directives – Interoperability Official Resources – Combined Constructs & Clauses on Combined and Composite Constructs (✓) • OpenMP Architecture Review Board. OpenMP Application Programming Interface. Version 5.1. Nov. 2020. URL: https://www.openmp.org/wp- – if Clause ✓ content/uploads/OpenMP-API-Specification-5.1.pdf – Synchronization Constructs and Clauses ✓ • OpenMP Architecture Review Board. OpenMP Application – Cancellation Constructs Programming Interface. Examples. Version 5.0.0. Nov. 2019. URL: https://www.openmp.org/mp-documents/openmp-examples-5.0.0.pdf – Data Environment ✓ • https://www.openmp.org – Nesting of Regions Recommended by https://www.openmp.org/resources/openmp-books/ • Runtime Library Routines • Timothy G. Mattson, Yun He, and Alice E. Koniges. The OpenMP Common Core. Making – Runtime Library Definitions (✓) OpenMP Simple Again. 1st ed. The MIT Press, Nov. 19, 2019. 320 pp. ISBN: 9780262538862 – Thread Team Routines (✓) • Ruud van der Pas, Eric Stotzer, and Christian Terboven. Using OpenMP—The Next Step. – Thread Affinity Routines Affinity, Accelerators, Tasking, and SIMD. 1st ed. The MIT Press, Oct. 13, 2017. 392 pp. ISBN: 9780262534789 – Teams Region Routines Additional Literature – Tasking Routines • Michael McCool, James Reinders, and Arch Robison. Structured Parallel Programming. – Resource Relinquishing Routines Patterns for Efficient Computation. 1st ed. Morgan Kaufmann, July 31, 2012. 432 pp. ISBN: 9780124159938
48 Older Works (https://www.openmp.org/resources/openmp-books/) Terminology: Directive • Barbara Chapman, Gabriele Jost, and Ruud van der Pas. Using OpenMP. Portable Shared In C/C++, a #pragma, and in Fortran, a comment, that specifies OpenMP program Memory Parallel Programming. 1st ed. Scientific and Engineering Computation. The MIT behavior. Press, Oct. 12, 2007. 384 pp. ISBN: 9780262533027 • Rohit Chandra et al. Parallel Programming in OpenMP. 1st ed. Morgan Kaufmann, Oct. 11, 58 INFRASTRUCTURE 2000. 231 pp. ISBN: 9781558606715 • Michael Quinn. Parallel Programming in C with MPI and OpenMP. 1st ed. McGraw-Hill, June 5, COMPILING & LINKING 2003. 544 pp. ISBN: 9780072822564 Compilers that conform to the OpenMP specification usually accept a command line argument that • Timothy G. Mattson, Beverly A. Sanders, and Berna L. Massingill. Patterns for Parallel turns on OpenMP support, e.g.: Programming. 1st ed. Software Patterns. Sept. 15, 2004. 384 pp. ISBN: 9780321228116 Intel C Compiler OpenMP Command Line Switch 57 TERMINOLOGY $ icc -qopenmp ...
THREADS & TASKS GNU Fortran Compiler OpenMP Command Line Switch
Terminology: Thread $ gfortran -fopenmp ...
An execution entity with a stack and associated static memory, called threadprivate The name of this command line argument is not mandated by the specification and differs from memory. one compiler to another. Terminology: OpenMP Thread Naturally, these arguments are then also accepted by the MPI compiler wrappers: A thread that is managed by the OpenMP runtime system. Compiling Programs with Hybrid Parallelization Terminology: Team $ mpicc -qopenmp ... A set of one or more threads participating in the execution of a parallel region. Terminology: Task RUNTIMELIBRARYDEFINITIONS [OpenMP-5.1, 3.1] A specific instance of executable code and its data environment that the OpenMP imlementation can schedule for execution by threads. C/C++ Runtime Library Definitions Runtime library routines and associated types are defined in the omp.h header file. LANGUAGE #include
49 59 BASICPROGRAMSTRUCTURE • Column 6 is blank for first line of directive, non-blank and non-zero for continuation Free Form Sentinel WORLDORDERINOPENMP sentinel = !$omp F08 • Program starts as one single-threaded process. 푡0 푡1 푡2 … • Forks into teams of multiple threads when • The usual line length, white space and continuation rules apply appropriate. • Stream of instructions might be different for each CONDITIONAL COMPILATION [OpenMP-5.1, 2.2] thread. • Information is exchanged via shared parts of C Preprocessor Macro memory. #define _OPENMP yyyymm
• OpenMP threads may be nested inside MPI C processes. yyyy and mm are the year and month the OpenMP specification supported by the compiler was published. Fortran Fixed Form Sentinels
!$ | *$ | c$ F08
• Must start in column 1 C AND C++ DIRECTIVE FORMAT [OpenMP-5.1, 2.1] • Only numbers or white space in columns 3–5 In C and C++, OpenMP directives are written using the #pragma method: • Column 6 marks continuation lines #pragma omp directive-name [clause[[,] clause]...] Fortran Free Form Sentinel C !$ • Directives are case-sensitive F08
• Applies to the next statement which must be a structured block • Must only be preceded by white space Terminology: Structured Block • Can be continued with ampersand An executable statement, possibly compound, with a single entry at the top and a single exit at the bottom, or an OpenMP construct. THE PARALLEL CONSTRUCT [OpenMP-5.1, 2.6]
FORTRAN DIRECTIVE FORMAT [OpenMP-5.1, 2.1.1, 2.1.2] #pragma omp parallel [clause[[,] clause]...] structured-block C sentinel directive-name [clause[[,] clause]...] F08 !$omp parallel [clause[[,] clause]...] • Directives are case-insensitive structured-block !$omp end parallel Fixed Form Sentinels F08
sentinel = !$omp | c$omp | *$omp • Creates a team of threads to execute the parallel region F08 • Each thread executes the code contained in the structured block • Must start in column 1 • Inside the region threads are identified by consecutive numbers starting at zero • The usual line length, white space, continuation and column rules apply
50 • Optional clauses (explained later) can be used to modify behavior and data environment of A FIRST OPENMP PROGRAM the parallel region #include
Team size int main(void){ printf("Hello from your main thread.\n"); int omp_get_num_threads(void); C #pragma omp parallel integer function omp_get_num_threads() printf("Hello from thread %d of %d.\n", F08 ↪ omp_get_thread_num(), omp_get_num_threads()); Returns the number of threads in the current team printf("Hello again from your main thread.\n"); Thread number } C int omp_get_thread_num(void); C Program Output
integer function omp_get_thread_num() $ gcc -fopenmp -o hello_openmp.x hello_openmp.c F08 $ ./hello_openmp.x Returns the number that identifies the calling thread within the current team (between zero and Hello from your main thread. omp_get_num_threads()) Hello from thread 1 of 8. Hello from thread 0 of 8. Hello from thread 3 of 8. Hello from thread 4 of 8. Hello from thread 6 of 8. Hello from thread 7 of 8. Hello from thread 2 of 8. Hello from thread 5 of 8. Hello again from your main thread.
program hello_openmp use omp_lib implicit none
print *, "Hello from your main thread."
!$omp parallel print *, "Hello from thread ", omp_get_thread_num(), " of ", ↪ omp_get_num_threads(), "." !$omp end parallel
print *, "Hello again from your main thread." end program F08
51 60 EXERCISES PART XII
Exercise 9 – Warm Up 9.1 Generalized Vector Addition (axpy) LOW-LEVEL OPENMP CONCEPTS In the file axpy.{c|c++|f90}, fill in the missing body of the function/subroutine axpy_serial(a, x, y, z[, n]) so that it implements the generalized vector addition (in 61 INTRODUCTION serial, without making use of OpenMP): MAGIC 퐳 = 푎퐱 + 퐲. Any sufficiently advanced technology is indistinguishable from magic. Compile the file into a program and run it to test your implementation. (Arthur C. Clarke5) 9.2 Dot Product INTERNAL CONTROL VARIABLES [OpenMP-5.1, 2.4] In the file dot.{c|c++|f90}, fill in the missing body of the function/subroutine dot_serial(x, y[, n]) so that it implements the dot product (in serial, without making Terminology: Internal Control Variable (ICV) use of OpenMP): A conceptual variable that specifies runtime behavior of a set of threads or tasks in an dot(퐱, 퐲) = ∑ 푥푖푦푖. 푖 OpenMP program. Compile the file into a program and run it to test your implementation. • Set to an initial value by the OpenMP implementation • Some can be modified through either environment variables (e.g. OMP_NUM_THREADS) or API routines (e.g. omp_set_num_threads()) • Some can be read through API routines (e.g. omp_get_max_threads()) • Some are inaccessible to the user • Might have different values in different scopes (e.g. data environment, device, global) • Some can be overridden by clauses (e.g. the num_threads() clause) • Export OMP_DISPLAY_ENV=TRUE or call omp_display_env(1) to inspect the value of ICVs that correspond to environment variables [OpenMP-5.1, 3.15, 6.12]
PARALLELISM CLAUSES [OpenMP-5.1, 2.6, 2.18]
if Clause
if([parallel :] scalar-expression) C
if([parallel :] scalar-logical-expression) F08
If false, the region is executed only by the encountering thread(s) and no additional threads are forked. num_threads Clause 5Arthur C. Clarke. Profiles of the future : an inquiry into the limits of the possible. London: Pan Books, 1973. ISBN: 9780330236195.
52 A A ure h C htcnrl h ubro hed ofork. to threads of number the controls that ICV the Queries omp_get_max_threads for fork to threads of num_threads number the controls that ICV the Sets omp_set_num_threads THE CONTROLLING EXAMPLE the (overrides expression the of value the to equal size team a Requests F08 C F08 C F08 parallel C parallel F08 C nee function integer int integer subroutine void parallel end !$omp statement3 statement2 ) 64 statement1 num_threads( parallel !$omp } ) threshold > length if( { parallel omp #pragma num_threads(scalar-integer-expression) num_threads(integer-expression) statement2; statement1; statement0; omp_get_max_threads( omp_set_num_threads( ietv iha with directive an with directive lue nonee subsequently. encountered clause) num_threads omp_set_num_threads(num_threads) nthreads-var P Routine API Routine API num_threads omp_get_max_threads() if lueadascae tutrdboki C: in block structured associated and clause ICV void OeM-.,3.2.3] [OpenMP-5.1, 3.2.1] [OpenMP-5.1, int ); lueadascae tutrdboki Fortran: in block structured associated and clause num_threads); parallel nthreads-var ein (without regions ICV) 53 HEDLMT&DNMCADJUSTMENT DYNAMIC & LIMIT THREAD 01CnrligteNme fThreads of Number the Controlling 10.1 omp_get_thread_limit hed okdfra for forked threads Use Controlling – 10 Exercise EXERCISES 62 a of part as executed being code this Is omp_in_parallel A OF INSIDE threads. of number the of adjustment dynamic disable or Enable program. omp_get_dynamic a in used threads of number the on bound Upper F08 C F08 C F08 C hello_openmp.{c|c++|f90} • • • • nee function integer int oia function logical int logical subroutine function logical void int The The The The if num_threads omp_set_num_threads OMP_NUM_THREADS omp_get_thread_limit( omp_in_parallel( omp_get_dynamic( omp_set_dynamic( PARALLEL clause dynamic parallel omp_set_dynamic(dynamic) P Routine API and parallel REGION? omp_set_dynamic clause P Routine API omp_in_parallel() omp_get_dynamic() omp_get_thread_limit() region: niomn variable environment void void OeM-.,3.2.5] [OpenMP-5.1, int P routine API parallel opa rudwt h aiu ast e h ubrof number the set to ways various the with around play to ); ); dynamic); void OeM-.,3.2.13] [OpenMP-5.1, ); P Routines API region? OeM-.,326 3.2.7] 3.2.6, [OpenMP-5.1, Inspect the number of threads that are actually forked using omp_get_num_threads. DATA-SHARING ATTRIBUTE RULES I [OpenMP-5.1, 2.21.1.1] 10.2 Limits of the OpenMP Implementation The rules that determine the data-sharing attributes of variables referenced from the inside of a Determine the maximum number of threads allowed by the OpenMP implementation you are construct fall into one of the following categories: using and check whether it supports dynamic adjustment of the number of threads. Pre-determined • Variables with automatic storage duration declared inside the construct are private (C and 63 DATAENVIRONMENT C++)
DATA-SHARING ATTRIBUTES [OpenMP-5.1, 2.21.1] • Objects with dynamic storage duration are shared (C and C++) • Variables with static storage duration declared in the construct are shared (C and C++) Terminology: Variable • Static data members are shared (C++) A named data storage block, for which the value can be defined and redefined during the execution of a program. • Loop iteration variables are private (Fortran) Terminology: Private Variable • Implied-do indices and forall indices are private (Fortran) With respect to a given set of task regions that bind to the same parallel region, a • Assumed-size arrays are shared (Fortran) variable for which the name provides access to a different block of storage for each task region. Explicit Terminology: Shared Variable Data-sharing attributes are determined by explicit clauses on the respective constructs. With respect to a given set of task regions that bind to the same parallel region, a Implicit variable for which the name provides access to the same block of storage for each If the data-sharing attributes are neither pre-determined nor explicitly determined, they fall back task region. to the attribute determined by the default clause, or shared if no default clause is present.
CONSTRUCTS & REGIONS DATA-SHARING ATTRIBUTE RULES II [OpenMP-5.1, 2.21.1.2] Terminology: Construct The data-sharing attributes of variables inside regions, not constructs, are governed by simpler An OpenMP executable directive (and for Fortran, the paired end directive, if any) and rules: the associated statement, loop or structured block, if any, not including the code in • Static variables (C and C++) and variables with the save attribute (Fortran) are shared any called routines. That is, the lexical extent of an executable directive. Terminology: Region • File-scope (C and C++) or namespace-scope (C++) variables and common blocks or variables accessed through use or host association (Fortran) are shared All code encountered during a specific instance of the execution of a given construct or of an OpenMP library routine. • Objects with dynamic storage duration are shared (C and C++) Terminology: Executable Directive • Static data members are shared (C++) An OpenMP directive that is not declarative. That is, it may be placed in an executable • Arguments passed by reference have the same data-sharing attributes as the variable they context. are referencing (C++ and Fortran) • Implied-do indices, forall indices are private (Fortran) • Local variables are private
54 THE SHARED CLAUSE [OpenMP-5.1, 2.21.4.2] DEFAULT CLAUSE [OpenMP-5.1, 2.21.4.1]
shared(list) C and C++ * default(shared | none) • Declares the listed variables to be shared. C • The programmer must ensure that shared variables are alive while they are shared. Fortran
• Shared variables must not be part of another variable (i.e. array or structure elements). default(private | firstprivate | shared | none) F08
THE PRIVATE CLAUSE [OpenMP-5.1, 2.21.4.3] Determines the data-sharing attributes for all variables referenced from inside of a region that have neither pre-determined nor explicit data-sharing attributes. private(list) * Caution: default(none) forces the programmer to make data-sharing attributes explicit if they are not pre-determined. This can help clarify the programmer’s intentions to someone who • Declares the listed variables to be private. does not have the implicit data-sharing rules in mind. • All threads have their own new versions of these variables. • Private variables must not be part of another variable. REDUCTION CLAUSE [OpenMP-5.1, 2.21.5.4]
• If private variables are of class type, a default constructor must be accessible. (C++) reduction(reduction-identifier : list) * • The type of a private variable must not be const-qualified, incomplete or reference to incomplete. (C and C++) • Listed variables are declared private. • Private variables must either be definable or allocatable. (Fortran) • At the end of the construct, the original variable is updated by combining the private copies using the operation given by reduction-identifier. • Private variables must not appear in namelist statements, variable format expressions or expressions for statement function definitions. (Fortran) • reduction-identifier may be +, -, *, &, |, ^, &&, ||, min or max (C and C++) or an identifier (C) or an id-expression (C++) • Private variables must not be pointers with intent(in). (Fortran) • reduction-identifier may be a base language identifier, a user-defined operator, or + - * .and. .or. .eqv. .neqv. max min iand ior ieor FIRSTPRIVATE CLAUSE [OpenMP-5.1, 2.21.4.4] one of , , , , , , , , , , or (Fortran) • Private versions of the variable are initialized with appropriate values firstprivate(list) *
Like private, but initialize the new versions of the variables to have the same value as the variable 64 EXERCISES that exists before the construct. Exercise 11 – Data-sharing Attributes • Non-array variables are initialized by copy assignment (C and C++) 11.1 Generalized Vector Addition (axpy) • Arrays are initialize by element-wise assignment (C and C++) In the file axpy.{c|c++|f90} add a new function/subroutine axpy_parallel(a, x, y, • Copy constructors are invoked if present (C++) z[, n]) that uses multiple threads to perform a generalized vector addition. Modify the main part of the program to have your function/subroutine tested. • Non-pointer variables are initialized by assignment or not associated if the original variable is not associated (Fortran) Hints: • pointer variables are initialized by pointer assignment (Fortran) • Use the parallel construct and the necessary clauses to define an appropriate data environment.
55 • Use omp_get_thread_num() and omp_get_num_threads() to decompose the • name is a compile time constant. work. • In C, names live in their own name space. • In Fortran, names of critical regions can collide with other identifiers. 65 THREAD SYNCHRONIZATION LOCK ROUTINES [OpenMP-5.1, 3.9] • In MPI, exchange of data between processes implies synchronization through the message metaphor. void omp_init_lock(omp_lock_t* lock); • In OpenMP, threads exchange data through shared parts of memory. void omp_destroy_lock(omp_lock_t* lock); • Explicit synchronization is needed to coordinate access to shared memory. void omp_set_lock(omp_lock_t* lock); void omp_unset_lock(omp_lock_t* lock); Terminology: Data Race C A data race occurs when subroutine omp_init_lock(svar) • multiple threads write to the same memory unit without synchronization or subroutine omp_destroy_lock(svar) subroutine omp_set_lock(svar) • at least one thread writes to and at least one thread reads from the same subroutine omp_unset_lock(svar) memory unit without synchronization. integer(kind = omp_lock_kind) :: svar F08 • Data races result in unspecified program behavior. • Like critical sections, but identified by runtime value rather than global name • OpenMP offers several synchronization mechanism which range from high-level/general to shared low-level/specialized. • Locks must be between threads • Initialize a lock before first use THE BARRIER CONSTRUCT [OpenMP-5.1, 2.19.2] • Destroy a lock when it is no longer needed set unset #pragma omp barrier • Lock and unlock using the and routines C • set blocks if lock is already set !$omp barrier F08 THE ATOMIC AND FLUSH CONSTRUCTS [OpenMP-5.1, 2.19.7, 2.19.8] • Threads are only allowed to continue execution of code after the barrier once all threads in the current team have reached the barrier. • barrier, critical, and locks implement synchronization between general blocks of code • A barrier region must be executed by all threads in the current team or none. • If blocks become very small, synchronization overhead could become an issue THE CRITICAL CONSTRUCT [OpenMP-5.1, 2.19.1] • The atomic and flush constructs implement low-level, fine grained synchronization for certain limited operations on scalar variables: #pragma omp critical [(name)] – read structured-block C – write !$omp critical [(name)] – update, writing a new value based on the old value structured-block – capture, like update and the old or new value is available in the subsequent code !$omp end critical [(name)] F08 • Correct use requires knowledge of the OpenMP Memory Model [OpenMP-5.1, 1.4] • Execution of critical regions with the same name are restricted to one thread at a time. • See also: C11 and C++11 Memory Models
56 66 EXERCISES PART XIII
Exercise 12 – Thread Synchronization 12.1 Dot Product WORKSHARING In the file dot.{c|c++|f90} add a new function/subroutine dot_parallel(x, y[, n]) that uses multiple threads to perform the dot product. Do not use the reduction clause. Modify 67 INTRODUCTION the main part of the program to have your function/subroutine tested. Hint: WORKSHARING CONSTRUCTS • Decomposition of the work load should be similar to the last exercise • Decompose work for concurrent execution by multiple threads parallel • Partial results of different threads should be combined in a shared variable • Used inside regions • Use a suitable synchronization mechanism to coordinate access • Available worksharing constructs: single sections Bonus – and construct Use the reduction clause to simplify your program. – loop construct – workshare construct – task worksharing
68 THE SINGLE CONSTRUCT
THE SINGLE CONSTRUCT [OpenMP-5.1, 2.10.2]
#pragma omp single [clause[[,] clause]...] structured-block C
!$omp single [clause[[,] clause]...] structured-block !$omp end single [end_clause[[,] end_clause]...] F08
• The structured block is executed by a single thread in the encountering team. • Permissible clauses are firstprivate, private, copyprivate and nowait. • nowait and copyprivate are end_clauses in Fortran.
69 SINGLECLAUSES
IMPLICITBARRIERS&THE NOWAIT CLAUSE [OpenMP-5.1, 2.10] • Worksharing constructs (and the parallel construct) contain an implied barrier at their exit. • The nowait clause can be used on worksharing constructs to disable this implicit barrier.
57 THE COPYPRIVATE CLAUSE [OpenMP-5.1, 2.21.6.2] for (range-decl: range-expr) structured-block C++
copyprivate(list) • var can be an integer, a pointer, or a random access iterator * • incr-expr increments (or decrements) var, e.g. var = var + incr • list contains variables that are private in the enclosing parallel region. • The increment incr must not change during execution of the loop • At the end of the single construct, the values of all list items on the single thread are copied to all other threads. • For nested loops, the bounds of an inner loop (b and lb) may depend at most linearly on the iteration variable of an outer loop, i.e. a0 + a1 * var-outer • E.g. serial initialization • var must not be modified by the loop body • copyprivate cannot be combined with nowait. • The beginning of the range has to be a random access iterator 70 THE LOOP CONSTRUCT • The number of iterations of the loop must be known beforehand In Fortran the do-loops must have the following form: WORKSHARING-LOOP CONSTRUCT [OpenMP-5.1, 2.11.4] do [label] var = lb, b[, incr] F08 #pragma omp for [clause[[,] clause]...] for-loops • var must be of integer type C • incr must be invariant with respect to the outermost loop !$omp do [clause[[,] clause]...] • The loop bounds b and lb of an inner loop may depend at most linearly on the iteration do-loops variable of an outer loop, i.e. a0 + a1 * var-outer [!$omp end do [nowait]] F08 • The number of iterations of the loop must be known beforehand Declares the iterations of a loop to be suitable for concurrent execution on multiple threads. Data-environment clauses 71 LOOPCLAUSES • private THE COLLAPSE CLAUSE [OpenMP-5.1, 2.11.4] • firstprivate • lastprivate collapse(n) * • reduction • The loop directive applies to the outermost loop of a set of nested loops, by default Worksharing-Loop-specific clauses • collapse(n) extends the scope of the loop directive to the n outer loops • schedule • All associated loops must be perfectly nested, i.e.: • collapse for (int i = 0; i < N; ++i) { CANONICAL LOOP FORM [OpenMP-5.1, 2.11.1] for (int j = 0; j < M; ++j) { // ... In C and C++ the for-loops must have the following form: } } C for ([type] var = lb; var relational-op b; incr-expr) ↪ structured-block C
58 THE SCHEDULE CLAUSE [OpenMP-5.1, 2.11.4] • scalar assignments • forall constructs schedule(kind[, chunk_size]) * • where statements and constructs Determines how the iteration space is divided into chunks and how these chunks are distributed • atomic, critical and parallel constructs among threads. Where possible, these are decomposed into independent units of work and executed in parallel. static Divide iteration space into chunks of chunk_size iterations and distribute them in a round-robin fashion among threads. If chunk_size is not specified, chunk size is chosen such that each thread gets at most one chunk. 74 EXERCISES dynamic Divide into chunks of size chunk_size (defaults to 1). When a thread is done workshare processing a chunk it acquires a new one. Exercise 14 – Construct 14.1 Generalized Vector Addition (axpy) guided Like dynamic but chunk size is adjusted, starting with large sizes for the first chunks and decreasing to chunk_size (default 1). In the file axpy.f90 add a new subroutine axpy_parallel_workshare(a, x, y, z) that uses the workshare construct to perform the generalized vector addition. auto Let the compiler and runtime decide. 14.2 Dot Product runtime Schedule is chosen based on ICV run-sched-var. In the file dot.f90 add a new function dot_parallel_workshare(x, y) that uses the If no schedule clause is present, the default schedule is implementation defined. workshare construct to perform the dot product. Caveat: Make sure to correctly synchronize access to the accumulator variable. 72 EXERCISES
Exercise 13 – Loop Worksharing 75 COMBINED CONSTRUCTS 13.1 Generalized Vector Addition (axpy) COMBINED CONSTRUCTS [OpenMP-5.1, 2.16] In the file axpy.{c|c++|f90} add a new function/subroutine axpy_parallel_for(a, x, y, z[, n]) that uses loop worksharing to perform the generalised vector addition. Some constructs that often appear as nested pairs can be combined into one construct, e.g. 13.2 Dot Product #pragma omp parallel In the file dot.{c|c++|f90} add a new function/subroutine dot_parallel_for(x, y[, #pragma omp for n]) that uses loop worksharing to perform the dot product. for (...; ...; ...) { ... Caveat: Make sure to correctly synchronize access to the accumulator variable. } C
73 WORKSHARE CONSTRUCT can be turned into #pragma omp parallel for WORKSHARE (FORTRAN ONLY) [OpenMP-5.1, 2.10.3] for (...; ...; ...) { ... !$omp workshare } structured-block C !$omp end workshare [nowait] F08 Similarly, parallel and workshare can be combined.
The structured block may contain: Combined constructs usually accept the clauses of either of the base constructs. • array assignments
59 PART XIV 77 THE TASK CONSTRUCT TASK WORKSHARING THE TASK CONSTRUCT [OpenMP-5.1, 2.12.1] #pragma omp task [clause[[,] clause]...] structured-block 76 INTRODUCTION C
TASK TERMINOLOGY !$omp task [clause[[,] clause]...] structured-block !$omp end task
Terminology: Task F08
A specific instance of executable code and its data environment, generated when a Creates a task. Execution of the task may commence immediately or be deferred. thread encounters a task, taskloop, parallel, target or teams construct. Data-environment clauses Terminology: Child Task • private A task is a child task of its generating task region.A child task region is not part of its generating task region. • firstprivate Terminology: Descendent Task • shared A task that is the child task of a task region or of one of its descendent task regions. Task-specific clauses Terminology: Sibling Task • if Tasks that are child tasks of the same task region. • final • untied TASK LIFE-CYCLE • mergeable • Execution of tasks can be deferred and suspended • depend • Scheduling is done by the OpenMP runtime system at scheduling points • priority • Scheduling decisions can be influenced by e.g. task dependencies and task priorities TASK DATA-ENVIRONMENT [OpenMP-5.1, 2.21.1.1]
created running completed The rules for implicitly determined data-sharing attributes of variables referenced in task generating constructs are slightly different from other constructs: If no default clause is present and • the variable is shared by all implicit tasks in the enclosing context, it is also shared by deferred the generated task, suspended • otherwise, the variable is firstprivate.
60 78 TASKCLAUSES Terminology: Untied Task A task that, when its task region is suspended, can be resumed by any thread in the THE IF CLAUSE [OpenMP-5.1, 2.12.1] team. That is, the task is not tied to any thread. if([task: ] scalar-expression) Terminology: Tied Task * A task that, when its task region is suspended, can be resumed only by the same If the scalar expression evaluates to false: thread that suspended it. That is, the task is tied to that thread. • Execution of the current task THE PRIORITY CLAUSE [OpenMP-5.1, 2.12.1] – is suspended and
– may only be resumed once the generated task is complete priority(priority-value) * • Execution of the generated task may commence immediately • priority-value is a scalar non-negative numerical value Terminology: Undeferred Task • Priority influences the order of task execution A task for which execution is not deferred with respect to its generating task region. That is, its generating task region is suspended until execution of the undeferred task • Among tasks that are ready for execution, those with a higher priority are more likely to be is completed. executed next
THE FINAL CLAUSE [OpenMP-5.1, 2.12.1] THE DEPEND CLAUSE [OpenMP-5.1, 2.19.11]
final(scalar-expression) depend(in: list) * depend(out: list) If the scalar expression evaluates to true all descendent tasks of the generated task are depend(inout: list) *
• undeferred and • list contains storage locations • executed immediately. • A task with a dependence on x, depend(in: x), has to wait for completion of previously Terminology: Final Task generated sibling tasks with depend(out: x) or depend(inout: x) A task that forces all of its child tasks to become final and included tasks. • A task with a dependence depend(out: x) or depend(inout: x) has to wait for Terminology: Included Task completion of previously generated sibling tasks with any kind of dependence on x A task for which execution is sequentially included in the generating task region. That • in, out and inout correspond to intended read and/or write operations to the listed is, an included task is undeferred and executed immediately by the encountering variables. thread. Terminology: Dependent Task A task that because of a task dependence cannot be executed until its predecessor THE UNTIED CLAUSE [OpenMP-5.1, 2.12.1] tasks have completed.
untied * 79 TASKSCHEDULING • The generated task is untied meaning it can be suspended by one thread and resume execution on another. TASK SCHEDULING POLICY [OpenMP-5.1, 2.12.6] • By default, tasks are generated as tied tasks. The task scheduler of the OpenMP runtime environment becomes active at task scheduling points. It may then
61 • begin execution of a task or 81 EXERCISES • resume execution of untied tasks or tasks tied to the current thread. Exercise 15 – Task worksharing Task scheduling points 15.1 Generalized Vector Addition (axpy) • generation of an explicit task In the file axpy.{c|c++|f90} add a new function/subroutine axpy_parallel_task(a, • task completion x, y, z[, n]) that uses task worksharing to perform the generalized vector addition. • taskyield regions 15.2 Dot Product • taskwait regions In the file dot.{c|c++|f90} add a new function/subroutine dot_parallel_task(x, y[, n]) that uses task worksharing to perform the dot product. • the end of taskgroup regions Caveat: Make sure to correctly synchronize access to the accumulator variable. • implicit and explicit barrier regions 15.3 Bitonic Sort THE TASKYIELD CONSTRUCT [OpenMP-5.1, 2.12.4] The file bsort.{c|c++|f90} contains a serial implementation of the bitonic sort algorithm. Use OpenMP task worksharing to parallelize it. #pragma omp taskyield C
!$omp taskyield F08
• Notifies the scheduler that execution of the current task may be suspended at this point in favor of another task • Inserts an explicit scheduling point
80 TASK SYNCHRONIZATION
THE TASKWAIT & TASKGROUP CONSTRUCTS [OpenMP-5.1, 2.19.5, 2.19.6]
#pragma omp taskwait C
!$omp taskwait F08
Suspends the current task until all child tasks are completed.
#pragma omp taskgroup structured-block C
!$omp taskgroup structured-block !$omp end taskgroup F08
The current task is suspended at the end of the taskgroup region until all descendent tasks generated within the region are completed.
62 PART XV PART XVI WRAP-UP TUTORIAL
ALTERNATIVES N-BODY SIMULATIONS
Horizontal Alternatives Dynamics of the N-body problem:
Parallel languages Fortran Coarrays, UPC; Chapel, Fortress, X10 푞푖푞푗 퐚 = (퐱 − 퐱 ) 푖,푗 3 푖 푗 Parallel frameworks Charm++, HPX, StarPU √(퐱푖 − 퐱푗) ⋅ (퐱푖 − 퐱푗) Shared memory tasking Cilk, TBB 푖̈퐱 = 퐚푖 = ∑ 퐚푖,푗 Accelerators CUDA, OpenCL, OpenACC, OmpSs 푗≠푖 Platform solutions PLINQ, GCD, java.util.concurrent Velocity Verlet integration: Vertical Alternatives Δ푡 Δ푡 Applications Gromacs, CP2K, ANSYS, OpenFOAM 퐯∗ (푡 + ) = 퐯 (푡) + 퐚 (푡) 2 2 Numerics libraries PETSc, Trilinos, DUNE, FEniCS Δ푡 퐱 (푡 + Δ푡) = 퐱(푡) + 퐯∗ (푡 + ) Δ푡 Big data frameworks Hadoop, Spark 2 Δ푡 Δ푡 퐯 (푡 + Δ푡) = 퐯∗ (푡 + ) + 퐚 (푡 + Δ푡) JSC COURSE PROGRAMME 2 2
• Introduction to the usage and programming of supercomputer resources in Jülich, 18 – 19 Program structure: May • Advanced Parallel Programming with MPI and OpenMP, 30 November – 02 December read initial state from file calculate accelerations • GPU Programming with CUDA, 4 – 6 May for number of time steps: • Directive-based GPU programming with OpenACC, 26 – 27 October write state to file calculate helper velocities v* • And more, see https://www.fz-juelich.de/ias/jsc/courses calculate new positions calculate new accelerations calculate new velocities write final state to file *
A SERIAL N-BODY SIMULATION PROGRAM
Compiling nbody $ cmake -B build $ cmake --build build ...
63 Invoking nbody Write a version of nbody that is parallelized using OpenMP. Look for suitable parts of the program to annotate with OpenMP directives. $ ./build/nbody Usage: nbody 16.2 MPI parallel version $ ./build/nbody ../input/kaplan_10000.bin Write a version of nbody that is parallelized using MPI. The distribution of work might be similar to Working on step 1... the previous exercise. Ideally, the entire system state is not stored on every process, thus particle data has to be communicated. Communication could be point-to-point or collective. Input and Visualizing the results output functions might have to be adapted as well. $ paraview --state=kaplan.pvsm 16.3 Hybrid parallel version
Initial conditions based on: A. E. Kaplan, B. Y. Dubetsky, and P. L. Shkolnikov. “Shock Shells in Write a version of nbody that is parallelized using both MPI and OpenMP. This might just be a Coulomb Explosions of Nanoclusters.” In: Physical Review Letters 91 (14 Oct. 3, 2003), p. 143401. combination of the previous two versions. DOI: 10.1103/PhysRevLett.91.143401 Bonus A clever solution is described in: M. Driscoll et al. “A Communication-Optimal N-Body Algorithm for SOME SUGGESTIONS Direct Interactions.” In: 2013 IEEE 27th International Symposium on Parallel and Distributed 10.1109/IPDPS.2013.108 Distribution of work Processing. 2013, pp. 1075–1084. DOI: . Implement Algorithm 1 from the paper. Look for loops with a number of iterations that scales with the problem size N. If the individual loop iterations are independent, they can be run in parallel. Try to distribute iterations evenly among the threads / processes. Distribution of data What data needs to be available to which process at what time? Having the entire problem in the memory of every process will not scale. Think of particles as having two roles: targets (푖 index) that experience acceleration due to sources (푗 index). Make every process responsible for a group of either target particles or source particles and communicate the same particles in the other role to other processes. What particle properties are important for targets and sources? Input / Output You have heard about different I/O strategies during the MPI I/O part of the course. Possible solutions include: • Funneled I/O: one process reads then scatters or gathers then writes • MPI I/O: every process reads or writes the particles it is responsible for Scalability Keep an eye on resource consumption. Ideally, the time it takes for your program to finish should be inversely proportional to the number of threads or processes running it 풪 (푁2/푝). Similarly, the amount of memory consumed by your program should be independent of the number of processes 풪 (푁).
EXERCISES
Exercise 16 – N-body simulation program 16.1 OpenMP parallel version
64 COLOPHON
This document was typeset using
• LuaLATEX and a host of macro packages, • Adobe Source Sans Pro for body text and headings, • Adobe Source Code Pro for listings, • TeX Gyre Pagella Math for mathematical formulae, • icons from Font Awesome Font-Awesome.
65