Advanced Features of Message Passing Interface (MPI) PCOPP-2002

A June 03- 06, 2002 C-DAC, Pune

r e Message Passing Interface (MPI)

Message Passing Interface (MPI)

s a

Day – 3 Classroom Lecture 3 – Day

g e Advanced Features of

Advanced Features of

I n

PCOPP-2002

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 1

A June 03- 06, 2002 C-DAC, Pune

a n

e a

Positives and Negatives Features MPI –2 Features Types of Synchronization PassingCost of Message MPI Datatypes MPI Collective Communication and Computations MPI Communication modes MPI advanced point-to-point communication Basics of MPI

n g

Lecture Outline

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 2

A June 03- 06, 2002 C-DAC, Pune

a n

.Understanding the way in which the different MPI operations are 4. Know how to use different MPI functions to solve performance 3. Understand how MPI implementation work 2. Understand the effect of different MPI Functions that accomplish 1. MPI Goals

F e

implemented is critical in tuning for performance problems communicationthe same

a c

MPI Goals

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 3

A June 03- 06, 2002 C-DAC, Pune

c e

Message-Passing Programming Paradigm connected using a message passing interconnection network.

Message Passing Architecture Model Message

s s

M i

I n

COMMUNICATION

r f

M a

c e

NETWORK

(

P I

M ) P • M P : Processors are are : Processors Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 4

A June 03- 06, 2002 C-DAC, Pune

a n

Is MPI Large or Small?

F e

MPI is large (125 Functions) MPI is just small (6 Functions) MPI is a

r e

One need not master all parts of MPI to use it Onewhen it is can access flexibility required Many parallel programs can be written with just 6 basic functions Number of functions not necessarily a measure of complexity many functions requires functionality extensive MPI’s

s a

right

P a

Is MPI Large or Small? s

candidate for message passing

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 5

A June 03- 06, 2002 C-DAC, Pune

n c

One can utilize any of 125 functions in MPI. MPI can be large MPI_FINALIZE MPI_RECV MPI_SEND MPI_COMM_RANK MPI_COMM_SIZE MPI_INIT One can begin programming with 6 MPI function calls MPI can be small. The MPI Message Passing Interface Small or Large

Is MPI Large or Small?

e r Determines number of processors Determines the label of the calling process Initializes MPI Terminates MPI

Receives a message messageSends a

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 6

A June 03- 06, 2002 C-DAC, Pune

a n

Sending and Receiving messages Fundamental questions answered

e a

How does the receiver identify it? What is sent? data To whom is sent?

M e

Process 0 Process 1

s a

edRecv Send

s s

MPI Send and Receive

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 7

A June 03- 06, 2002 C-DAC, Pune

a n

c e

Information on MPI Send and Recv

F e

Source Communication between two processes Destination process is identified by its Communication takes place within a

process sends message to to message sends process e

MPI Point-to-Point Communication

(

I ) destination communicator rank in the communicator Copyright process PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 8

A June 03- 06, 2002 C-DAC, Pune

a n

Communication Completion

F e it is locally complete for all processes A communication operation is globally complete

processes involved have completed their part in the operation communicationA operation is globally complete if all the process has completed its part in the operation A communication operationthe is locally complete on processif a

M e

MPI Point-to-Point Communication

a c

if and only if

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 9

A June 03- 06, 2002 C-DAC, Pune

a n

Where Blocking Send

c e

A typical blocking send looks like

e a type message containing the message to be containing the message sent dest ( messages screen selectively

address

o f

is an integer identifier representing the processthe toreceive

is nonnegativeis integer that the destination can use to

s s send (

length ,

a g

MPI Blocking Send and Receive

s s

dest, type, address, length

t e

)

f a

describes a contiguous area in memory

(

I ) ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 10

A June 03- 06, 2002 C-DAC, Pune

a n

The sending and receiving of messages between pairs of processors. Point-to-Point Communications

F e

operation has been issued and the message has been transferred. BLOCKING SEND: has been issued and the message has been received. BLOCKING RECEIVE:

a g

MPI Blocking Send and Receive

I n MPI_Send MPI_Recv

returns only after the corresponding RECEIVE

a c

returns only after the corresponding SEND

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 11

A June 03- 06, 2002 C-DAC, Pune

n c S = Sender S = R = Receiver procedure.and receivesendblocking use the following If we are sending a large message, most implementations of Blocking Sends and Receives

has been posted MPI_RECV corresponding word has arrived that Transferdoesn’t begin until R S

r e

size > threshold s

MPI_SEND

a g

MPI Blocking Send and Receive

s i

task waits n

(blocking standard send)

f a

MPI_RECV

(

P I

source complete source from transfer data ) wait complete is buffer user’s to transfer taskcontinues when data Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 12

A June 03- 06, 2002 C-DAC, Pune

a n

Non-Blocking Send and Receive Fortran C

F e

processor. complete,immediatebut returns control back to the calling Non-blocking Receive:

r e

MPI_Irecv (buf, count, dtype, source, tag, comm, request, ierror) comm, request, ierror) MPI_Isend (buf, count, dtype, tag, MPI_Irecv (buf, count, dtype, dest, tag, request);comm, MPI_Isend (buf, count, dtype, dest, tag, comm, request);

MPI Non-Blocking Send and Receive

MPI_IRecv

c e

does not wait for the message transfer to

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 13

A June 03- 06, 2002 C-DAC, Pune

c e

Non-Blocking Communications

t u

Separate communicationthree phases: into r

Wait for non-blocking communication to complete. Wait for non-blocking Do some work (perhaps involving other communications ?) Initiate non-blocking communication. M

MPI Non-Blocking Send and Receive

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 14

A June 03- 06, 2002 C-DAC, Pune

o f

MPI Non-Blocking Send and Receive

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 15

A June 03- 06, 2002 C-DAC, Pune

a n

If we are sending a If weare sending been received. been already has the message believes side sending the on MPI_Wait case, the In this completed. has operation a non-blocking it to see checks An MPI-Wait R = Receiver S = Sender side. receiving on the buffer in a stored and immediately sent be can The message procedure. the following use receive and sends c

MPI_IRECV posted early enoughMPI_IRECV posted early node can be avoided if Transfer to buffer on receiving

F e

size

t u

MPI_ISEND (non-blocking standard send) send) standard (non-blocking MPI_ISEND

r e

≤

threshold

MPI Non-Blocking Send and Receive

small

n t

message, most implementations of non-blocking of non-blocking implementations most message,

e r

MPI_WAIT

(

I ) MPI_IRECV Copyright is late enough if MPI_WAIT no delay MPI_WAIT PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 16

A June 03- 06, 2002 C-DAC, Pune

a n

been sent yet. sent been not has message the that side sees the sending on MPI_Wait the this case, In completed. has operation it a non-blocking to see An MPI_Wait checks R = Receiver S = Sender by an later halted MPI_Wait. but after the send, is resumed Computation sent. immediately is not data the but is issued, send The procedure. following the use receive and sends If we are sending a sending If we are c

MPI_IRECV has been posted has arrived that corresponding transfer doesn’t begin until word

R S

size > threshold o

MPI Non-Blocking Send and Receive f

MPI_ISEND

P a

large

(non-blocking standard send) standard (non-blocking

t e

message, most implementations of non-blocking of non-blocking implementations most message, r

MPI_WAIT

a c

task waits e

MPI_IRECV

(

I ) source complete source from transfer data MPI_WAIT Copyright PCOPP PCOPP enough if wait is late interruption no  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 17

A June 03- 06, 2002 C-DAC, Pune

a n

Communication Modes

e a

Buffered mode Synchronous mode t

buffered to this ensure independentmatching of receive, and message may be to standardSimilar mode, but completion is always complete until message delivery is guaranteed the send will not except standard mode, same as The

g e

MPI Communication Modes

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 18

A June 03- 06, 2002 C-DAC, Pune

a n S = Sender S = R = Receiver non-blocking send. temporary storage on the sending processor, we can perform a type of for (buffer space) memory some the programmer allocate we If Buffered Sends and Receives

R S

t u

to buffer to copy data

MPI_BSEND

g e buffer complete buffer user-supplied to transfer data

MPI Buffered Send and Receive

I n

(buffered send)

(

I ) MPI_RECV task waits Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 19

A June 03- 06, 2002 C-DAC, Pune

v a

when a send operation is initiated, or when it completes The mode of a point to point communication operation governs Communication Modes n

F e

Standard mode Ready mode

a t

e s

been initiated initiated A send may be not been initiated A send may initiated be

MPI Communication Modes

(

I ) even if only if a matching receive a matching receive Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, has has (Contd…) 2002 2002 20

A June 03- 06, 2002 C-DAC, Pune

a n

Communication Modes

F e

Two basic ways of checking on non-blocking send and receives communication to be overlapped Use non-blocking and completion routines allow computation and a

r e

mpi_test mpi_wait Call a test routine that a returns flag to indicate if complete Call a wait routine that blocks until completion

g e MPI Communication Modes

(request_id,flag,return_status,ierr)

(request_id,return_status,ierr)

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 21

A June 03- 06, 2002 C-DAC, Pune

v a

Communication Modes

Non-blocking send Blocking send

e a

r e

Must check for local completion for local Must check bufferMessage not should be read Returns when send is locally complete completion for local Must check Message buffer should not be written Returns “immediately

g e

MPI Communication Modes

(

I ) from after return to after return Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 22

A June 03- 06, 2002 C-DAC, Pune

v a

Communication Modes n

F e

Blocking receive communication is complete mpi_wait mpi_status a

e s

Message bufferMessage from afterbe read can return Returns when receive is locally complete

s s

blocks until communication is complete

a g

returns “immediately”, and sets e

MPI Communication Modes

(

I ) flag Copyright to to true PCOPP PCOPP  C-DAC, 2002 C-DAC, if the (Contd…) 2002 2002 23

A June 03- 06, 2002 C-DAC, Pune

M Synchronous send Standard send Receive Buffered send

Ready send

s a

Sender mode

MPI Communication Modes

( M buffered. or synchronous Either

completed. has the receive whether error occurs), irrespective of an (unless completes Always P receiver. error occurs), irrespective of an (unless completes Always has arrived. has arrived. a when message Completes

receive has completed has receive completed the when Only completes

I ) Notes Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 24

A June 03- 06, 2002 C-DAC, Pune

e s MPI Sender Modes eev MPI_RECV MPI_RSEND MPI_SSEND Receive Ready send Synchronous send Standard send

ufrdsn MPI_BSEND Buffered send

s s

OPERATION

MPI Communication Modes

(

I ) MPI_SEND MPI CALL Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 25

A June 03- 06, 2002 C-DAC, Pune

c e

MPI : Persistent Operations MPI : Nonblocking operations, overlap effective

t u “ startall(recvs),sendrecv/barrier, startall(rsends),waitall waitall “ recvinit, startall, sendinit, Variation of example of MPI_Request Allocation “ Potential saving Isend, Irecv, Waitall

Vendor implementations are buggy

s s

MPI Persistent Communication

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 26

A June 03- 06, 2002 C-DAC, Pune

a n

will stall indefinitely. will stall indefinitely. the receives can never be executed, and both sets of communications executed after the sends, so if the sends do not completereturn, and corresponding receives in order to complete, but those receives are The diagram demonstrates that the two onsendstheir waiting each are for events that haven’t been initiated yet. Causes of Deadlock : Deadlock MPI Message Passing -

c e PE 0

PE 1

MPI_Send issue by both PEs

s s

MPI Cause of Deadlock

Deadlock occurs when all tasks are waiting

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 27

A June 03- 06, 2002 C-DAC, Pune

n c

MPI Message Passing : Avoiding Deadlock

e a

Change Ordering Non-blocking calls

u r

the sendsposted. are task is working on when the message arrives or in what order This allows each messagethematter what to be received, no other communication. Have each task post a non-blocking receive before it doesany its send first. to post Arrange for one task to post its receive first and for the other between tasks Different ordering of calls

a s

P-Avoiding Deadlock MPI-

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 28

A June 03- 06, 2002 C-DAC, Pune

n c

MPI Message Passing Avoiding Deadlock

e a

MPI_Sendrecv Buffered mode

t u

receive to be executed.receive the send This will allow to complete and the subsequent copying the message to the user-supplied buffer. Use buffered sends so that computation can proceed after message to a destination and the receiving from a source The send-receive combines in one call the sending of a (S). MPI_Sendrecv_replaceMPI_Sendrecv use

s s

MPI-Avoiding Deadlock

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 29

A June 03- 06, 2002 C-DAC, Pune

n c

need participate in the communication. processors. A collective communicationprocessorsall that implies The sending and/or receiving of messages to/from groups of Collective Communications Collective Communications e

e a

Two broad classes : All collective routines block until they are locally complete No message tags used Involves coordinated communication within a group of processes

t u

Global computation routines Data movement routines

a g

MPI Collective Communications

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 30

A June 03- 06, 2002 C-DAC, Pune

v a

Collective Communication n

e a

Examples: Called by all processes in a communicator. Communications involving a group of processes. t

Global sum,maximum,etc. global gather. scatter, Broadcast, Barrier synchronization.

s a

MPI Collective Communications

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 31

A June 03- 06, 2002 C-DAC, Pune

n c

Receive buffers must be exactly the right size No tags. All collective operations are blocking. Synchronization may or may not occur All processes must communicate Collective action over a communicator Characteristics of Collective Communication

MPI Collective Communications

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 32

A June 03- 06, 2002 C-DAC, Pune

a n

Communication is coordinated among a group of processes Collective Communications

F e

Group can be constructed “ be constructed can Group No non-blocking collective operations Different communicators are used instead routines or by using MPI topology-definition routines

s s

MPI Collective Communications

( M

by hand

I ) ” with MPI group-manipulation Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 33

A June 03- 06, 2002 C-DAC, Pune

n c

Collective Computation Operations

F MPI_BOR MPI_BAND MPI_LXOR MPI_LOR MPI_BXOR MPI_LAND

MPI_Name

a g

MPI Collective Computations

I n Bitwise AND Bitwise

Bitwise OR Bitwise Bitwise exclusive OR t

Logical exclusive or (xor) Logical or Logical and Operation

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 34

A June 03- 06, 2002 C-DAC, Pune

n c

Collective Computation Operation

F MPI_MAXLOC MPI_SUM MPI_PROD MPI_MIN MPI_MAX

MPI_MAXLOC e

MPI Name

s s

MPI Collective Computations

(

I ) Maximum and location Maximum and Maximum and location Maximum and Operation Maximum Minimum Product Sum Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 35

A June 03- 06, 2002 C-DAC, Pune

MPI Collective Communications and Computations

u r

in and user-defined combination functions sizes (Scatterv,Allgatherv, Gatherv) V versions deliver resultsAll to all participating processes e

l eue eue,ReduceScatter, and Scan take both built- All reduce, Reduce ,

-version allow the chunks to have different non-uniform data

M e Scatterv Reduce Scatter Gather Alltoall

Allgather

i n

Other CollectiveCells Library

(

I ) Alltoallv Gatherv Allgatherv Scan Copyright Allreduce Scatter Bcast Reduce PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 36

A June 03- 06, 2002 C-DAC, Pune

P3 P2 P1 P0 P3 P2 P1 P0

r e 0D 2D3 D2 D1 D0 C3 C2 C1 C0 B3 B2 B1 B0 A0

D C B A

Representation of collective data movement in MPI M

1A A3 A2 A1

s a

MPI Collective Communications

a c

All gather e

All to

(

I ) P3 P2 P1 P0 P3 P2 P1 P0 A3 A2 A1 A0 A A A A Copyright B3 B2 B1 B0 BCD BCD BCD BCD PCOPP PCOPP  C3 C2 C1 C0 C-DAC, 2002 C-DAC, (Contd…) D3 D2 D1 D0 2002 2002 37

A June 03- 06, 2002 C-DAC, Pune

a n

four processors. every processor accumulates the final values final the accumulates processor every processors. four Performs a scatter a scatter Performs All-to-All

All-to-All operation for an integer array of size 8 on 4 processors

s s

MPI Collective Communications

P a

and gather

(

from all four processors to all other

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 38

A June 03- 06, 2002 C-DAC, Pune

n c

MPI : Support for Regular Decompositions

e a “MPI_Cart_Create “ “Remember,contention still matters; a good mapping can “Allow MPI implementation to provide low expected contention “Simple to use (why not?)

User candefine Using topology routines reduce contention effects layout of processes (contention can matter) topology routinesyou use the Why

s s

ita topology virtual i

P Using topology MPI -

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 39

A June 03- 06, 2002 C-DAC, Pune

c e

Message type

t u

MPI datatypes: datatype A message containselements a numberof particularof some C types are different from Fortran types Derived types can be built up from basic types r

eie aatps(etr; Structs; Others) Data types (Vectors; Derived Basic types

n t

MPI Datatypes

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, 2002 2002 40

A June 03- 06, 2002 C-DAC, Pune

P ai Datatypes MPI Basic

MPI_PACKED MPI_BYTE MPI_LONG_DOUBLE MPI_DOUBLE MPI_FLOAT MPI_UNSIGNED_LONG MPI_UNSIGNED MPI_UNSIGNED_SHORT MPI_UNSIGNED_CHAR MPI_LONG MPI_INT MPI_SHORT MPI_CHAR

g e

P Datatype MPI

s i

P ai Datatypes MPI Basic

I n

-C

(

I ) Double Unsigned long int Unsignedint short Unsigned char Signed long int Signed int Signed short int Long double Float Unsigned int Signed char datatype C Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 41

A June 03- 06, 2002 C-DAC, Pune

a n

Contiguous Data

F e

The simplest derived C : contiguous items of the same Fortran :

e s nee on,odye newtype oldtype, integer count,

ltp,P_aaye*newtype); oldtype,MPI_Datatype (int count, MPI_Datatype MPI_Type_contiguous int P_yecniuu cut oldtype, newtype) MPI_Type_contiguous (count,

P a

MPI Derived Data types MPI Derived

a c

datatype

(

I ) datatype consists of a number ofof a numberconsists Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 42

A June 03- 06, 2002 C-DAC, Pune

d v

Stride = = 5; Stride a Vector Datatype Array_of_types[1] = MPI_DOUBLE Array_of_blocklengths[1] =3 Array_of_types[0] = MPI_INT; Array_of_blocklengths[0] =1 2; Count = Struct Datatype

block length = 3; Count = 2;

s s

MPI Derived Data types MPI Derived

(

I ) ; Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 43

A June 03- 06, 2002 C-DAC, Pune

a n

Datatype Extent of a Constructing a Vector Datatype

F e

Fortran C Fortran C

e s

nee datatype, extent, ierror integer ierror) MPI_Type_extent(datatype, extent, int MPI_Type_extent (MPI_Datatype datatype, int *extent); ierror) newtype, blocklength,oldtype, stride, MPI_Type_vector (count, *newtype); MPI_Datatype oldtype, MPI_Datatype blocklength, int stride, MPI_Type_vectorint (int count, int

s i

MPI Derived Data types MPI Derived

(

I ) Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 44

A June 03- 06, 2002 C-DAC, Pune

a n

Struct Datatype Constructing a

F e C : Fortran : MPI_Type_Struct (count, array_of_blocklengths, (count, MPI_Type_Struct

n P_yesrc (int count, int array_of_blocklengths, int MPI_Type_struct

r e

P_aaye*newtype); MPI_Datatype *array_of_types, MPI_Datatype *array_of_displacements, MPI_Aint

M e

ra_fdslcmns aryo_ye,nwye ierror) array_of_displacements, array_of_types, newtype,

s s

MPI Derived Data types MPI Derived

(

A June 03- 06, 2002 C-DAC, Pune

a n

datatype a Committing

F e

Fortran C This is done using before it is used. has been constructed, it needs to be committed datatype Once a

e s

nee aaye ierror datatype, integer ierror) (datatype, MPI_Type_Commit *datatype); int MPI_Type_Commit (MPI_Datatype

s s

MPI Derived Data types MPI Derived

n g

MPI_TYPE_COMMIT

(

A June 03- 06, 2002 C-DAC, Pune

a n

MPI : Performance of datatypes

e a

Use of collective when many copies bcast /gather Collect many small messages into a single large message; with time Performance very dependent on implementation; should improve Test of 1000 element vector of doubles with stride of 24 doubles. Handing non-contiguousdata; t

“User packs and unpacks by hand “MPI_Type_vector” and “MPI_Type_struct(.*,*)”;

s s

MPI Derived Data types MPI Derived

(

A June 03- 06, 2002 C-DAC, Pune

a n

e a

Common message-passingoperations component Startup time has both a to transfer messages Message passing programs that exploit oftendata parallelismuse t

Gather and Scatter operations Broadcast, Reduction, Prefix, Gather, Scalter, All to Barrier Blocking and Non-Blocking type Point to point / Collective communication

P a

Cost of Message Passing Cost of Message

t e

required data

( M

hardware

I ) among processors among as well Copyright software PCOPP PCOPP  C-DAC, 2002 C-DAC, relate 2002 2002 48

A June 03- 06, 2002 C-DAC, Pune

n c

Startup-time and transfer-time

e a

travel, and the status of the network at the time of transmission The cost depends on thehasto size of themessage,it how far receiving the message processorsinvolved must spend some time sending and Every time a message is passed between processors, the parts: startup-time The time required to send a message can be divided into two

P a

Cost of Message Passing Cost of Message

and transfer-time

(

A June 03- 06, 2002 C-DAC, Pune

c e

Hardware

t u

being sent Startup time is fixed and does not depend on the size of message and router The time to establish an interfaceprocessor betweenthelocal The time to execute the writing algorithm (such as adding header,trailer, and error correction information) Time required to prepare the message for under-lying network

M e

P a

Cost of Message Passing Cost of Message

(

A June 03- 06, 2002 C-DAC, Pune

a n

Software

e a

receiving processors receiving Negotiating thetransmissiondata of between the sending and Copying the message to internal buffers library for creating various internal data structures In general, this may include the time spent by the message-passing passing library Depends on the protocol followed by the underlying message

o f

P a

Cost of Message Passing Cost of Message

(

A June 03- 06, 2002 C-DAC, Pune

n c

The transfer time of a message

t u

to another will have traverse many links In most cases, a message traveling from one processor Per word transfer rate ( The underlying interconnection network Depends on the size of the message and bandwidth of Transfer time of a message of size

P a

Cost of Message Passing Cost of Message

c e

( M

)= P

1/r

I ) n words ( nt Copyright w ) PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 52

A June 03- 06, 2002 C-DAC, Pune

n c

Remarks

e a

there is a greater likelihood that link contention will arise. As the message sizes and distances being traveled increase, forcing the remaining messages to wait. In this case, only the one message at a time will traverse the link, same link of the interconnection network at the same time. concurrently, more than one message may want to traverse the Different messages traverse the interconnection network

P a

Cost of Message Passing Cost of Message

(

A June 03- 06, 2002 C-DAC, Pune

a n

Remarks

F e

network the bandwidthandmessage of the underlying interconnection transfera messageThe dependsof time on the size of the network statusprocessor,the receivingThe of and other traffic in the May depend on the size of the message being sent Depending on the type of MPI communication operation performed that of the hardware end The software related startup time is usuallyhigherthanof much

P a

Cost of Message Passing Cost of Message

(

A June 03- 06, 2002 C-DAC, Pune

a n

varies greatly and it depends on type of communication call used. time requiredThe by the MPI collective communication operation Cost of Collective Communication Operations

c e

a t

the interconnection network Time required by each operation depends on characteristic of collective communication operations Algorithm is more important for efficient implementation of BROADCAST of a message of size Element wise Reduce Operation of operation Barrier

P a

Cost of Message Passing Cost of Message

(

I ) n n words words words Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 55

A June 03- 06, 2002 C-DAC, Pune

Cttruh routing (Cut-through- REDUCE BROADCAST SCATTER

P n n

n a processor SCATTER PREFIX REDUCE Cost of Message Passing Cost of Message

BROADCAST BARRIER s

Operation s is size of the message received from each is size of the message being broadcasted

is size of message stored in each processor

n t

)

(

I ) 0(np) 0(np) 0(n log p) 0(n log p) p) 0(log Complexity Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 56

A June 03- 06, 2002 C-DAC, Pune

v a

Synchronization methods Synchronization Speed

F e

Assigning a unique task to each processor from a list of tasks particular answer Waiting until the first of any of the contributing processes finds a Waiting all processes finish a loop Speed of system Cost of message passingalso depends on Synchronization together to the next step have finished one step of a problem and are ready to go on Refers to the time needed for totheyall processorsagree that

Types of Synchronization

(

A June 03- 06, 2002 C-DAC, Pune

a n

Memory copies MPI-Synchronization Delays

e a

and single processor problem performance of source primary the are copies Memory requires devoting resources to checking. it There is a performance tradeoff caused by reacting quickly - react quickly, a delay results thepartner if doesn’t Message passing is a cooperative method- Measured memcpy performance non-contiguousCost of datatypes hardware

s s

P-Synchronization MPI-

f a

memcpy

(

I ) is often much slower than the Copyright PCOPP PCOPP  C-DAC, 2002 C-DAC, (Contd…) 2002 2002 58

A June 03- 06, 2002 C-DAC, Pune

a n

e a

Application-oriented process topologies Collective Environmental Profiling t

Built-in support for grids and graphs (uses groups) Subgroups defined directly or by topology Large number of data movements routines Both built-in and user-defined collective operations Inquiry and Error control Hooks allow users to intercept MPI calls

Features of MPI

(

A June 03- 06, 2002 C-DAC, Pune

a n

F e

Point-to-Point communication Non-message-passing concepts not included: General Non-message-passing concepts included a

t u

r e

Thread safety Communicators combine context & group for message security s Modes: normal (blocking and non-blocking), synchronous, datatypes, heterogeneity Structured buffers and derived shared memory shared ready (to allow access to fast protocols), buffered

Process management; Remote memorytransfers; Virtual Active messages and Threads

I n

Features of MPI

(

A June 03- 06, 2002 C-DAC, Pune

n c

e a

Positives

s s

No language binding issues Do not require any daemon to start application Rich set of collective functions Simplified sending message Performance was a high-priority in the design MPI is De-facto standard for message-passing in a box

I n

Features of MPI

(

A June 03- 06, 2002 C-DAC, Pune

c e

t u

Positives r

Not limited to “shared -data” Can send messages using any kind of data Simple to understand conceptually Simple memory model Bestpractice scaling seenin

I n

Features of MPI

(

A June 03- 06, 2002 C-DAC, Pune

c e

Cons

a t

expensive are errorwhich proneuse lots or of messages, which is Non-contiguous data handling either use derived data types in nature,communication using asynchronous indeterministic Codes may be consuming Development of production codes is much difficult and time Debugging is not easy

I n

Features of MPI

(

A June 03- 06, 2002 C-DAC, Pune

a n

P- ehius–Positives MPI-2 Techniques – c

e a

Thread Safety is ensured ProcessDynamic spawning/deletion operations Generalized requests (interrupt receive) “ put/get/barrier to reduce synchronization points One-sided communication communicationsNon-blocking collective t

Additional language bindings for Fortran90 /95 and C++ MPI-I/O

n g

Features of MPI-2

(

A June 03- 06, 2002 C-DAC, Pune

n c

Tuning Performance Tuning Performance (General techniques)

e a

Pitfalls MPI -Specific tuning Techniques Performance Changing the Algorithm Load Balancing Decomposition Aggregation

n g

P Performance MPI -

(

A June 03- 06, 2002 C-DAC, Pune

c e

e a

Features of MPI and MPI-2 PassingCost of Message Operations Summary of MPI Collective Communication and Computations Data Types, Toplogies Summary of MPI special features – Summary of MPI advanced point-to-point communication

n g

Summary of MPI

(

A June 03- 06, 2002 C-DAC, Pune

(