PCOPP2002 PCOPP-2002 Day – 3 Classroom Lecture AdvancedAdvanced FeaturesFeatures ofof MessageMessage PassingPassing InterfaceInterface (MPI)(MPI) June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 1 PCOPP2002 Lecture Outline Basics of MPI MPI advanced point-to-point communication MPI Communication modes MPI Collective Communication and Computations MPI Datatypes Cost of Message Passing Types of Synchronization MPI –2 Features Positives and Negatives Features June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 2 PCOPP2002 MPI Goals MPI Goals 1. Understand the effect of different MPI Functions that accomplish the same communication 2. Understand how MPI implementation work 3. Know how to use different MPI functions to solve performance problems 4. Understanding the way in which the different MPI operations are implemented is critical in tuning for performance June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 3 PCOPP2002 Message Passing Architecture Model Message-Passing Programming Paradigm : Processors are connected using a message passing interconnection network. COMMUNICATION NETWORK P P P • • • • P M M M M June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 4 PCOPP2002 Is MPI Large or Small? Is MPI Large or Small? MPI is large (125 Functions) MPI’s extensive functionality requires many functions Number of functions not necessarily a measure of complexity MPI is small (6 Functions) Many parallel programs can be written with just 6 basic functions MPI is just right candidate for message passing One can access flexibility when it is required One need not master all parts of MPI to use it June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 5 PCOPP2002 Is MPI Large or Small? (Contd…) The MPI Message Passing Interface Small or Large MPI can be small. One can begin programming with 6 MPI function calls MPI_INIT Initializes MPI MPI_COMM_SIZE Determines number of processors MPI_COMM_RANK Determines the label of the calling process MPI_SEND Sends a message MPI_RECV Receives a message MPI_FINALIZE Terminates MPI MPI can be large One can utilize any of 125 functions in MPI. June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 6 PCOPP2002 MPI Send and Receive Sending and Receiving messages Process 0 Process 1 Send Recv Fundamental questions answered To whom is data sent? What is sent? How does the receiver identify it? June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 7 PCOPP2002 MPI Point-to-Point Communication (Contd…) Information on MPI Send and Recv Communication between two processes Source process sends message to destination process Communication takes place within a communicator Destination process is identified by its rank in the communicator June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 8 PCOPP2002 MPI Point-to-Point Communication Communication Completion A communication operation is locally complete on a process if the process has completed its part in the operation A communication operation is globally complete if all the processes involved have completed their part in the operation A communication operation is globally complete if and only if it is locally complete for all processes June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 9 PCOPP2002 MPI Blocking Send and Receive Blocking Send A typical blocking send looks like send (dest, type, address, length) Where dest is an integer identifier representing the process to receive the message type is nonnegative integer that the destination can use to selectively screen messages (address, length) describes a contiguous area in memory June03- 06,2002 C-DAC, Pune containing the message to be sent Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 10 PCOPP2002 MPI Blocking Send and Receive (Contd…) Point-to-Point Communications The sending and receiving of messages between pairs of processors. BLOCKING SEND: returns only after the corresponding RECEIVE operation has been issued and the message has been transferred. MPI_Send BLOCKING RECEIVE: returns only after the corresponding SEND has been issued and the message has been received. MPI_Recv June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 11 PCOPP2002 MPI Blocking Send and Receive (Contd…) Blocking Sends and Receives If we are sending a large message, most implementations of blocking send and receive use the following procedure. S = Sender R = Receiver MPI_SEND (blocking standard send) data transfer from size > threshold source complete task waits S R wait Transfer doesn’t begin until word has arrived that MPI_RECV task continues when data corresponding MPI_RECV transfer to user’s buffer is June03- 06,2002 C-DAC, Pune has been posted complete Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 12 PCOPP2002 MPI Non-Blocking Send and Receive Non-Blocking Send and Receive Non-blocking Receive: does not wait for the message transfer to complete, but immediate returns control back to the calling processor. MPI_IRecv C MPI_Isend (buf, count, dtype, dest, tag, comm, request); MPI_Irecv (buf, count, dtype, dest, tag, comm, request); Fortran MPI_Isend (buf, count, dtype, tag, comm, request, ierror) June03- 06,2002 C-DAC, Pune MPI_Irecv (buf, count, dtype, source, tag, comm, request, ierror) Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 13 PCOPP2002 MPI Non-Blocking Send and Receive (Contd…) Non-Blocking Communications Separate communication into three phases: Initiate non-blocking communication. Do some work (perhaps involving other communications ?) Wait for non-blocking communication to complete. June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 14 PCOPP2002 MPI Non-Blocking Send and Receive (Contd…) June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 15 PCOPP2002 MPI Non-Blocking Send and Receive If we are sending a small message, most implementations of non-blocking sends and receive use the following procedure. The message can be sent immediately and stored in a buffer on the receiving side. S = Sender R = Receiver An MPI-Wait checks to see it a non-blocking operation has completed. In this case, the MPI_Wait on the sending side believes the message has already been received. MPI_ISEND (non-blocking standard send) size ≤ threshold MPI_WAIT Transfer to buffer on receiving MPI_WAIT MPI_IRECV node can be avoided if no delay if MPI_WAIT June03- 06,2002 C-DAC, Pune MPI_IRECV posted early enough is late enough Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 16 PCOPP2002 MPI Non-Blocking Send and Receive (Contd…) If we are sending a large message, most implementations of non-blocking sends and receive use the following procedure. The send is issued, but the data is not immediately sent. Computation is resumed after the send, but later halted by an MPI_Wait. S = Sender R = Receiver An MPI_Wait checks to see it a non-blocking operation has completed. In this case, the MPI_Wait on the sending side sees that the message has not been sent yet. MPI_ISEND (non-blocking standard send) data transfer from size > threshold MPI_WAIT source complete task waits S R transfer doesn’t begin until word no interruption MPI_IRECV MPI_WAIT has arrived that corresponding if wait is late June03- 06,2002 C-DAC, Pune MPI_IRECV has been posted enough Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 17 PCOPP2002 MPI Communication Modes Communication Modes Synchronous mode The same as standard mode, except the send will not complete until message delivery is guaranteed Buffered mode Similar to standard mode, but completion is always independent of matching receive, and message may be buffered to ensure this June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 18 PCOPP2002 MPI Buffered Send and Receive Buffered Sends and Receives If we the programmer allocate some memory (buffer space) for temporary storage on the sending processor, we can perform a type of non-blocking send. S = Sender R = Receiver MPI_BSEND (buffered send) data transfer to copy data user-supplied to buffer buffer complete S R task waits June03- 06,2002 C-DAC, Pune MPI_RECV Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 19 PCOPP2002 MPI Communication Modes (Contd…) Communication Modes The mode of a point to point communication operation governs when a send operation is initiated, or when it completes Standard mode A send may be initiated even if a matching receive has not been initiated Ready mode A send may be initiated only if a matching receive has been initiated June03- 06,2002 C-DAC, Pune Advance Features of Message Passing Interface (MPI) Copyright C-DAC, 2002 20 PCOPP2002 MPI Communication Modes (Contd…) Communication Modes Two basic ways of checking on non-blocking send and receives Call a wait routine that blocks until completion Call a test routine that returns a flag to indicate if complete Use non-blocking and completion