Recap of Lecture 6
• Shared memory multiprocessors Distributed Memory • Caches may be either shared or distributed. Machines and Programming • Multicore chips are likely to have shared caches • Cache hit performance is better if they are distributed (each cache is smaller/closer) but they must be kept Lecture 7 coherent -- multiple cached copies of same location must be kept equal. • Requires clever hardware (see CS258, CS252). James Demmel • Distant memory much more expensive to access. www.cs.berkeley.edu/~demmel/cs267_Spr15 • Machines scale to 10s or 100s of processors. • Shared memory programming • Starting, stopping threads. Slides from Kathy Yelick • Communication by reading/writing shared variables. • Synchronization with locks, barriers. CS267 Lecture 7 1 02/10/2015 CS267 Lecture 7 2
Outline Architectures in Top 500, Nov 2014 • Distributed Memory Architectures 500 • Properties of communication networks 450 • Topologies 400 • Performance models 350 Cluster • Programming Distributed Memory Machines 300 MPP 250 using Message Passing Constellations • Overview of MPI 200 SMP • Basic send/receive use 150 SIMD 100 • Non-blocking communication Single Processor 50 • Collectives 0
1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013
02/10/2015 CS267 Lecture 7 3 02/10/2015
CS267 Lecture 2 1 Historical Perspective Network Analogy • To have a large number of different transfers occurring at once, • Early distributed memory machines were: you need a large number of distinct wires • Collection of microprocessors. • Not just a bus, as in shared memory • Communication was performed using bi-directional queues between nearest neighbors. • Networks are like streets: • Messages were forwarded by processors on path. • Link = street. • Switch = intersection. • “Store and forward” networking • Distances (hops) = number of blocks traveled. • There was a strong emphasis on topology in algorithms, • Routing algorithm = travel plan. in order to minimize the number of hops = minimize time • Properties: • Latency: how long to get between nodes in the network. • Street: time for one car = dist (miles) / speed (miles/hr) • Bandwidth: how much data can be moved per unit time. • Street: cars/hour = density (cars/mile) * speed (miles/hr) * #lanes • Network bandwidth is limited by the bit rate per wire and #wires
02/10/2015 CS267 Lecture 7 5 02/10/2015 CS267 Lecture 7 6
Design Characteristics of a Network Performance Properties of a Network: Latency • Topology (how things are connected) • Diameter: the maximum (over all pairs of nodes) of the • Crossbar; ring; 2-D, 3-D, higher-D mesh or torus; shortest path between a given pair of nodes. hypercube; tree; butterfly; perfect shuffle, dragon fly, … • Latency: delay between send and receive times • Routing algorithm: • Latency tends to vary widely across architectures • Example in 2D torus: all east-west then all north-south • Vendors often report hardware latencies (wire time) (avoids deadlock). • Application programmers care about software • Switching strategy: latencies (user program to user program) • Circuit switching: full path reserved for entire message, like the telephone. • Observations: • Packet switching: message broken into separately- • Latencies differ by 1-2 orders across network designs routed packets, like the post office, or internet • Software/hardware overhead at source/destination • Flow control (what if there is congestion): dominate cost (1s-10s usecs) • Stall, store data temporarily in buffers, re-route data to • Hardware latency varies with distance (10s-100s nsec other nodes, tell source node to temporarily halt, per hop) but is small compared to overheads discard, etc. • Latency is key for programs with many small messages 02/10/2015 CS267 Lecture 7 7 02/10/2015 CS267 Lecture 7 8
CS267 Lecture 2 2 Latency on Some Machines/Networks End to End Latency (1/2 roundtrip) Over Time
100 8-byte Roundtrip Latency nCube/2CS2 24.2 SP2 25 CM5 22.1 Paragon MPI ping-pong CM5 SP1CS2 36.34 20 18.5 nCube/2 T3D T3D T3E18.916 14.6 15 Myrinet KSR 12.080511.027 10 SP-Power39.25 9.6 usec 7.2755 10 Cenju3 6.9745 6.905 6.6 4.81 3.3 5 SPP Roundtrip Latency (usec) Latency Roundtrip Quadrics 2.6 SPP Quadrics 0 T3E Elan3/Alpha Elan4/IA64 Myrinet/x86 IB/G5 IB/Opteron SP/Fed 1 1990 1995 2000 2005 2010 Year (approximate) • Latencies shown are from a ping-pong test using MPI • These are roundtrip numbers: many people use ½ of roundtrip time • Latency has not improved significantly, unlike Moore’s Law to approximate 1-way latency (which can’t easily be measured) • T3E (shmem) was lowest point – in 1997 Data from Kathy Yelick, UCB and NERSC 02/10/2015 CS267 Lecture 7 9 02/10/2015 CS267 Lecture 7 10
Performance Properties of a Network: Bandwidth Bandwidth on Existing Networks
• The bandwidth of a link = # wires / time-per-bit Flood Bandwidth for 2MB messages 100% 857 • Bandwidth typically in Gigabytes/sec (GB/s), 225 1504 20 90% MPI i.e., 8* 2 bits per second 244 80% • Effective bandwidth is usually lower than physical link 630 610 bandwidth due to packet overhead. 70% Routing and control 60% header 50%
• Bandwidth is important for applications 40% Data with mostly large messages payload 30%
20%
Percent HW peak (BW in MB) in (BW peak HW Percent 10% Error code Trailer 0% Elan3/A lpha Elan4/IA 64 My r inet/x 86 IB/G5 IB/Opteron SP/Fed
• Flood bandwidth (throughput of back-to-back 2MB messages)
02/10/2015 CS267 Lecture 7 11 02/10/2015 CS267 Lecture 7 12
CS267 Lecture 2 3 Bandwidth Chart Performance Properties of a Network: Bisection Bandwidth 400 Note: bandwidth depends on SW, not just HW • Bisection bandwidth: bandwidth across smallest cut that 350 divides network into two equal halves
300 • Bandwidth across “narrowest” part of the network
T3E/MPI T3E/Shmem 250 IBM/MPI not a IBM/LAPI Compaq/Put bisection 200 Compaq/Get bisection M2K/MPI cut M2K/GM cut
Bandwidth (MB/sec) Bandwidth Dolphin/MPI 150 Giganet/VIPL SysKonnect
100
50 bisection bw= link bw bisection bw = sqrt(p) * link bw
0 2048 4096 8192 16384 32768 65536 131072 • Bisection bandwidth is important for algorithms in which Message Size (Bytes) all processors need to communicate with all others Data from Mike Welcome, NERSC 02/10/2015 CS267 Lecture 7 13 02/10/2015 CS267 Lecture 7 14
Network Topology Linear and Ring Topologies • In the past, there was considerable research in network • Linear array topology and in mapping algorithms to topology. • Key cost to be minimized: number of “hops” between • Diameter = n-1; average distance ~n/3. nodes (e.g. “store and forward”) • Bisection bandwidth = 1 (in units of link bandwidth). • Modern networks hide hop cost (i.e., “wormhole • Torus or Ring routing”), so topology less of a factor in performance of many algorithms • Example: On IBM SP system, hardware latency varies from 0.5 usec to 1.5 usec, but user-level message passing latency is roughly 36 usec. • Need some background in network topology • Diameter = n/2; average distance ~ n/4. • Algorithms may have a communication topology • Bisection bandwidth = 2. • Example later of big performance impact • Natural for algorithms that work with 1D arrays.
02/10/2015 CS267 Lecture 7 15 02/10/2015 CS267 Lecture 7 16
CS267 Lecture 2 4 Meshes and Tori – used in Hopper Hypercubes
d Two dimensional mesh Two dimensional torus • Number of nodes n = 2 for dimension d. • Diameter = 2 * (sqrt( n ) – 1) • Diameter = sqrt( n ) • Diameter = d. • Bisection bandwidth = sqrt(n) • Bisection bandwidth = 2* sqrt(n) • Bisection bandwidth = n/2.
• 0d 1d 2d 3d 4d
• Popular in early machines (Intel iPSC, NCUBE). • Lots of clever algorithms. 110 111 • See 1996 online CS267 notes. 010 011 • Generalizes to higher dimensions • Greycode addressing: 100 101 • Cray XT (eg Hopper@NERSC) uses 3D Torus • Each node connected to 000 001 • Natural for algorithms that work with 2D and/or 3D arrays (matmul) d others with 1 bit different. 02/10/2015 CS267 Lecture 7 17 02/10/2015 CS267 Lecture 7 18
Trees Butterflies • Diameter = log n. • Diameter = log n. • Bisection bandwidth = 1. • Bisection bandwidth = n. Ex: to get from proc 101 to 110, • Easy layout as planar graph. • Cost: lots of wires. Compare bit-by-bit and • Many tree algorithms (e.g., summation). • Used in BBN Butterfly. Switch if they disagree, else not • Fat trees avoid bisection bandwidth problem: • Natural for FFT. • More (or wider) links near top. • Example: Thinking Machines CM-5. O 1 O 1
O 1 O 1
butterfly switch multistage butterfly network
02/10/2015 CS267 Lecture 7 19 02/10/2015 CS267 Lecture 7 20
CS267 Lecture 2 5 Does Topology Matter? Dragonflies – used in Edison • Motivation: Exploit gap in cost and performance between optical 1 MB multicast on BG/P, Cray XT5, and Cray XE6 interconnects (which go between cabinets in a machine room) and electrical 8192 networks (inside cabinet) BG/P • Optical more expensive but higher bandwidth when long XE6 4096 XT5 • Electrical networks cheaper, faster when short • Combine in hierarchy 2048 • One-to-many via electrical networks inside cabinet • Just a few long optical interconnects between cabinets 1024 • Clever routing algorithm to avoid bottlenecks: 512 • Route from source to randomly chosen intermediate cabinet • Route from intermediate cabinet to destination Bandwidth (MB/sec) 256 • Outcome: programmer can (usually) ignore topology, get good performance • Important in virtualized, dynamic environment 128 • Programmer can still create serial bottlenecks 8 64 512 4096 • Drawback: variable performance #nodes See EECS Tech Report UCB/EECS-2011-92, August 2011 • Details in “Technology-Drive, Highly-Scalable Dragonfly Topology,” J. Kim. W. Dally, S. Scott, D. Abts, ISCA 2008 02/10/2015 CS267 Lecture 7 21 02/10/2015 22 CS267 Lecture 7
Evolution of Distributed Memory Machines • Special queue connections are being replaced by direct memory access (DMA): • Network Interface (NI) processor packs or copies messages. • CPU initiates transfer, goes on computing. • Wormhole routing in hardware: • NIs do not interrupt CPUs along path. Performance • Long message sends are pipelined. • NIs don’t wait for complete message before forwarding Models • Message passing libraries provide store-and-forward abstraction: • Can send/receive between any pair of nodes, not just along one wire. • Time depends on distance since each NI along path must participate.
02/10/2015 CS267 Lecture 7 23 CS267 Lecture 7 24
CS267 Lecture 2 6 Shared Memory Performance Models Latency and Bandwidth Model • Time to send message of length n is roughly • Parallel Random Access Memory (PRAM) Time = latency + n*cost_per_word • All memory access operations complete in one clock = latency + n/bandwidth period -- no concept of memory hierarchy (“too good to be true”). • Topology is assumed irrelevant. • OK for understanding whether an algorithm has enough • Often called “α-β model” and written parallelism at all (see CS273). Time = α + n*β • Parallel algorithm design strategy: first do a PRAM algorithm, then worry about memory/communication time (sometimes • Usually α >> β >> time per flop. works) • One long message is cheaper than many short ones. • Slightly more realistic versions exist α + n*β << n*(α + 1*β) • E.g., Concurrent Read Exclusive Write (CREW) PRAM. • Can do hundreds or thousands of flops for cost of one message. • Still missing the memory hierarchy • Lesson: Need large computation-to-communication ratio to be efficient. • LogP – more detailed model (Latency/overhead/gap/Proc.)
02/10/2015 CS267 Lecture 7 25 02/10/2015 CS267 Lecture 7 26
Alpha-Beta Parameters on Current Machines Model Time VaryingDrop Page Message Fields Here Size & Machines
Sum of model • These numbers were obtained empirically 10000
machine α β α is latency in usecs T3E/Shm 1.2 0.003 β is BW in usecs per Byte 1000 machine T3E/MPI 6.7 0.003 T3E/Shm IBM/LAPI 9.4 0.003 T3E/MPI IBM/LAPI IBM/MPI 7.6 0.004 IBM/MPI Quadrics/Get 3.267 0.00498 100 Quadrics/Shm Quadrics/MPI Quadrics/Shm 1.3 0.005 How well does the model Myrinet/GM Quadrics/MPI 7.3 0.005 Myrinet/MPI Time = α + n*β GigE/VIPL Myrinet/GM 7.7 0.005 predict actual performance? 10 GigE/MPI Myrinet/MPI 7.2 0.006 Dolphin/MPI 7.767 0.00529
Giganet/VIPL 3.0 0.010 1 GigE/VIPL 4.6 0.008 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 131072
GigE/MPI 5.854 0.00872 size
02/10/2015 CS267 Lecture 7 27 02/10/2015 CS267 Lecture 7 28
CS267 Lecture 2 7 Measured MessageDrop Page Time Fields Here
Sum of gap 10000 Programming
1000 machine Distributed Memory Machines T3E/Shm T3E/MPI with IBM/LAPI IBM/MPI 100 Quadrics/Shm Message Passing Quadrics/MPI Myrinet/GM Myrinet/MPI GigE/VIPL Slides from 10 GigE/MPI Jonathan Carter ([email protected]), Katherine Yelick ([email protected]),
1 Bill Gropp ([email protected]) 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 131072
size
02/10/2015 CS267 Lecture 7 29 02/10/2015 CS267 Lecture 7 30
Message Passing Libraries (1) Message Passing Libraries (2) • All communication, synchronization require subroutine calls • Many “message passing libraries” were once available • No shared variables • Chameleon, from ANL. • Program run on a single processor just like any uniprocessor • CMMD, from Thinking Machines. program, except for calls to message passing library • Express, commercial. • Subroutines for • MPL, native library on IBM SP-2. • Communication • NX, native library on Intel Paragon. • Pairwise or point-to-point: Send and Receive • Zipcode, from LLL. • Collectives all processor get together to • PVM, Parallel Virtual Machine, public, from ORNL/UTK. – Move data: Broadcast, Scatter/gather • Others... – Compute and move: sum, product, max, prefix sum, … • MPI, Message Passing Interface, now the industry standard. of data on many processors • Need standards to write portable code. • Synchronization • Barrier • No locks because there are no shared variables to protect • Enquiries • How many processes? Which one am I? Any messages waiting?
02/10/2015 CS267 Lecture 7 31 02/10/2015 CS267 Lecture 7 32
CS267 Lecture 2 8 Novel Features of MPI MPI References • Communicators encapsulate communication spaces for • The Standard itself: library safety • at http://www.mpi-forum.org • Datatypes reduce copying costs and permit • All MPI official releases, in both postscript and HTML heterogeneity • Latest version MPI 3.0, released Sept 2012 • Multiple communication modes allow precise buffer management • Other information on Web: • Extensive collective operations for scalable global • at http://www.mcs.anl.gov/mpi communication • pointers to lots of stuff, including other talks and • Process topologies permit efficient process placement, tutorials, a FAQ, other MPI pages user views of process layout • Profiling interface encourages portable tools
Slide source: Bill Gropp, ANL 33 Slide source: Bill Gropp, ANL 34 02/10/2015 CS267 Lecture 7 02/10/2015 CS267 Lecture 7
Books on MPI Finding Out About the Environment
• Using MPI: Portable Parallel Programming with the Message-Passing Interface (2nd edition), • Two important questions that arise early in a by Gropp, Lusk, and Skjellum, MIT Press, 1999. parallel program are: • Using MPI-2: Portable Parallel Programming • How many processes are participating in this with the Message-Passing Interface, by Gropp, Lusk, and Thakur, MIT Press, 1999. computation? • MPI: The Complete Reference - Vol 1 The MPI Core, by • Which one am I? Snir, Otto, Huss-Lederman, Walker, and Dongarra, MIT Press, 1998. • MPI provides functions to answer these • MPI: The Complete Reference - Vol 2 The MPI Extensions, questions: by Gropp, Huss-Lederman, Lumsdaine, Lusk, Nitzberg, Saphir, and Snir, MIT Press, 1998. • MPI_Comm_size reports the number of processes. • Designing and Building Parallel Programs, by Ian Foster, • MPI_Comm_rank reports the rank, a number between Addison-Wesley, 1995. 0 and size-1, identifying the calling process • Parallel Programming with MPI, by Peter Pacheco, Morgan- Kaufmann, 1997. Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 Slide source: Bill Gropp, ANL 35 02/10/2015 CS267 Lecture 7 36
CS267 Lecture 2 9 Hello (C) Hello (Fortran)
#include "mpi.h" #include
Note: hidden slides show Fortran and C++ versions of each example Slide source: Bill Gropp, ANL Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 37 02/10/2015 CS267 Lecture 7 38
Hello (C++) Notes on Hello World
#include "mpi.h" • All MPI programs begin with MPI_Init and end with #include
Slide source: Bill Gropp, ANL Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 39 02/10/2015 CS267 Lecture 7 40
CS267 Lecture 2 10 MPI Basic Send/Receive Some Basic Concepts • We need to fill in the details in • Processes can be collected into groups
Process 0 Process 1 • Each message is sent in a context, and must be
Send(data) received in the same context Receive(data) • Provides necessary support for libraries • A group and context together form a • Things that need specifying: communicator • How will “data” be described? • A process is identified by its rank in the group • How will processes be identified? associated with a communicator • How will the receiver recognize/screen messages? • What will it mean for these operations to complete? • There is a default communicator whose group contains all initial processes, called MPI_COMM_WORLD
Slide source: Bill Gropp, ANL Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 41 02/10/2015 CS267 Lecture 7 42
MPI Datatypes MPI Tags
• The data in a message to send or receive is described • Messages are sent with an accompanying user- by a triple (address, count, datatype), where defined integer tag, to assist the receiving • An MPI datatype is recursively defined as: process in identifying the message • predefined, corresponding to a data type from the language • Messages can be screened at the receiving end (e.g., MPI_INT, MPI_DOUBLE) • a contiguous array of MPI datatypes by specifying a specific tag, or not screened by • a strided block of datatypes specifying MPI_ANY_TAG as the tag in a • an indexed array of blocks of datatypes receive • an arbitrary structure of datatypes • Some non-MPI message-passing systems have • There are MPI functions to construct custom datatypes, “ ” in particular ones for subarrays called tags message types . MPI calls them • May hurt performance if datatypes are complex tags to avoid confusion with datatypes
Slide source: Bill Gropp, ANL Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 43 02/10/2015 CS267 Lecture 7 44
CS267 Lecture 2 11 MPI Basic (Blocking) Send MPI Basic (Blocking) Receive
A(10) A(10) B(20) B(20)
MPI_Send( A, 10, MPI_DOUBLE, 1, …) MPI_Recv( B, 20, MPI_DOUBLE, 0, … ) MPI_Send( A, 10, MPI_DOUBLE, 1, …) MPI_Recv( B, 20, MPI_DOUBLE, 0, … ) MPI_RECV(start, count, datatype, source, tag, MPI_SEND(start, count, datatype, dest, tag, comm, status) comm) • Waits until a matching (both and ) message is • The message buffer is described by (start, count, source tag datatype). received from the system, and the buffer can be used • The target process is specified by dest, which is the rank of • source is rank in communicator specified by comm, or the target process in the communicator specified by comm. MPI_ANY_SOURCE • When this function returns, the data has been delivered to • tag is a tag to be matched or MPI_ANY_TAG the system and the buffer can be reused. The message • receiving fewer than count occurrences of datatype is may not have been received by the target process. OK, but receiving more is an error
Slide source: Bill Gropp, ANL • status contains further information (e.g. size of message) 02/10/2015 CS267 Lecture 7 45 02/10/2015 CS267 Lecture 7 Slide source: Bill Gropp, ANL 46
A Simple MPI Program A Simple MPI Program (Fortran)
#include “mpi.h” #include
CS267 Lecture 2 12 A Simple MPI Program (C++) Retrieving Further Information
• Status is a data structure allocated in the user’s program. #include “mpi.h” #include
MPI::Finalize(); return 0; }
CS267 Lecture 7 Slide source: Bill Gropp, ANL 49 CS267 Lecture 7 02/10/2015 02/10/2015 Slide source: Bill Gropp, ANL 50
MPI is Simple Another Approach to Parallelism • Many parallel programs can be written using just these • Collective routines provide a higher-level way to six functions, only two of which are non-trivial: organize a parallel program • MPI_INIT • Each process executes the same communication • MPI_FINALIZE operations • MPI_COMM_SIZE • MPI provides a rich set of collective operations… • MPI_COMM_RANK • MPI_SEND • MPI_RECV
02/10/2015 CS267 Lecture 7 Slide source: Bill Gropp, ANL 51 02/10/2015 CS267 Lecture 7 Slide source: Bill Gropp, ANL 52
CS267 Lecture 2 13 Collective Operations in MPI Alternative Set of 6 Functions
• Claim: most MPI applications can be written with only 6 functions (although which 6 may differ) • Collective operations are called by all processes in a communicator • Using point-to-point: • Using collectives: • MPI_BCAST distributes data from one process (the • MPI_INIT • MPI_INIT root) to all others in a communicator • MPI_FINALIZE • MPI_FINALIZE • MPI_REDUCE combines data from all processes in • MPI_COMM_SIZE • MPI_COMM_SIZE communicator and returns it to one process • MPI_COMM_RANK • MPI_COMM_RANK • In many numerical algorithms, SEND/RECEIVE can be • MPI_SEND • MPI_BCAST replaced by BCAST/REDUCE, improving both simplicity • MPI_RECEIVE • MPI_REDUCE and efficiency • You may use more for convenience or performance
Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 53 02/10/2015 CS267 Lecture 7 54
Example: Calculating Pi Example: PI in C – 1/2
#include "mpi.h" #include
CS267 Lecture 2 14 Example: PI in C – 2/2 Example: PI in Fortran – 1/2
h = 1.0 / (double) n; program main sum = 0.0; include ‘mpif.h’ integer done, n, myid, numprocs, i, rc for (i = myid + 1; i <= n; i += numprocs) { double pi25dt, mypi, pi, h, sum, x, z x = h * ((double)i - 0.5); data done/.false./ sum += 4.0 * sqrt(1.0 - x*x); data PI25DT/3.141592653589793238462643/ call MPI_Init(ierr) } call MPI_Comm_size(MPI_COMM_WORLD,numprocs, ierr ) mypi = h * sum; call MPI_Comm_rank(MPI_COMM_WORLD,myid, ierr) do while (.not. done) MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, if (myid .eq. 0) then MPI_COMM_WORLD); print *,”Enter the number of intervals: (0 quits)“ read *, n if (myid == 0) endif printf("pi is approximately %.16f, Error is .16f\n", call MPI_Bcast(n, 1, MPI_INTEGER, 0, * MPI_COMM_WORLD, ierr ) pi, fabs(pi - PI25DT)); if (n .eq. 0) goto 10 } MPI_Finalize(); return 0; } Slide source: Bill Gropp, ANL Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 57 02/10/2015 CS267 Lecture 7 58
Example: PI in Fortran – 2/2 Example: PI in C++ - 1/2
h = 1.0 / n #include "mpi.h" sum = 0.0 #include
CS267 Lecture 2 15 Example: PI in C++ - 2/2 Synchronization • MPI_Barrier( comm ) h = 1.0 / (double) n; • Blocks until all processes in the group of the sum = 0.0; communicator comm call it. for (i = myid + 1; i <= n; i += numprocs) { x = h * ((double)i - 0.5); • Almost never required in a parallel program sum += 4.0 / (1.0 + x*x); • Occasionally useful in measuring performance and load } balancing mypi = h * sum; MPI::COMM_WORLD.Reduce(&mypi, &pi, 1, MPI::DOUBLE, MPI::SUM, 0); if (myid == 0) std::cout << "pi is approximately “ << pi << “, Error is “ << fabs(pi - PI25DT) << “\n”; } MPI::Finalize(); return 0; } Slide source: Bill Gropp, ANL 02/10/2015 CS267 Lecture 7 61 02/10/2015 CS267 Lecture 7 62
Synchronization (Fortran) Synchronization (C++) • MPI_Barrier( comm, ierr ) • comm.Barrier(); • Blocks until all processes in the group of the • Blocks until all processes in the group of the communicator comm call it. communicator comm call it.
02/10/2015 CS267 Lecture 7 63 02/10/2015 CS267 Lecture 7 64
CS267 Lecture 2 16 Collective Data Movement Comments on Broadcast, other Collectives • All collective operations must be called by all processes in the communicator P0 A A • MPI_Bcast is called by both the sender (called the root P1 Broadcast A process) and the processes that are to receive the P2 A broadcast P3 A • “root” argument is the rank of the sender; this tells MPI which process originates the broadcast and which receive
P0 A B C D Scatter A P1 B P2 Gather C P3 D
02/10/2015 CS267 Lecture 7 65 02/10/2015 CS267 Lecture 7 66
More Collective Data Movement Collective Computation
P0 A A B C D P0 A ABCD Reduce P1 B Allgather A B C D P1 B P2 C A B C D P2 C P3 D A B C D P3 D
P0 A0 A1 A2 A3 A0 B0 C0 D0 P0 A A P1 P1 AB B0 B1 B2 B3 Alltoall A1 B1 C1 D1 B Scan P2 C0 C1 C2 C3 A2 B2 C2 D2 P2 C ABC P3 D0 D1 D2 D3 A3 B3 C3 D3 P3 D ABCD
02/10/2015 CS267 Lecture 7 67 02/10/2015 CS267 Lecture 7 68
CS267 Lecture 2 17 MPI Collective Routines MPI Built-in Collective Computation Operations
• Many Routines: Allgather, Allgatherv, • MPI_MAX Maximum Allreduce, Alltoall, Alltoallv, Bcast, • MPI_MIN Minimum • MPI_PROD Product Gather, Gatherv, Reduce, Reduce_scatter, • MPI_SUM Sum Scan, Scatter, Scatterv • MPI_LAND Logical and • All versions deliver results to all participating • MPI_LOR Logical or processes, not just root. • MPI_LXOR Logical exclusive or • V versions allow the chunks to have variable sizes. • MPI_BAND Binary and • MPI_BOR Binary or • Allreduce, Reduce, Reduce_scatter, and Scan • MPI_BXOR Binary exclusive or take both built-in and user-defined combiner functions. • MPI_MAXLOC Maximum and location • MPI-2 adds Alltoallw, Exscan, intercommunicator • MPI_MINLOC Minimum and location versions of most routines
02/10/2015 CS267 Lecture 7 69 02/10/2015 CS267 Lecture 7 70
EXTRA SLIDES
02/10/2015 CS267 Lecture 7 71
CS267 Lecture 2 18