CMPSCI 377 Operating Systems November 13 - Lecture 13 Distributed Parallel Programming Lecturer: Emery Berger Scribes: John Ho, Ernald Nicolas
Previously:
• Programming with threads - Shared memory and single machines.
Today:
• Distributed parallel programming - Used in systems involving large computations.
• Message Passing - Message Passing Interface (MPI execution model)
13.1 Distributed Memory
13.1.1 Overview
• Networks of workstations
• Clusters
• Non-shared memory systems
13.1.2 Why Distribute? - Symmetric Multiprocessor (SMP)
• The Symmetric property is that each processor has the same access to memory.
13-1 13-2
Example #1 - Mesh of Processes
• A process partitioner, partitions the list (or “mesh”) of processes into smaller chunks. Each chunk is assigned to an individual processor. In the Quad-Tree (Tree w/ four children per node) of Figure 13.1, the first level of depth has four processors which are assigned four chunks in the mesh respectively.
Figure 13.1: A Mesh Diagram with its coresponding Quad Tree.
(Source: http://www.cs.sandia.gov/∼kddevin/LB/quadtree.html)
Q: What if a quadrant has almost all the processes? A: Recursively split quadrants with many processes into smaller meshes and assign a processor to each one. This is called Adaptive Mesh Refinement. In Figure 13.1, Adaptive Mesh Refinement is applied in Quadrant 1, 4, and 12. Each of these quadrant is split into smaller meshes coressponding to a leaf (i.e. a processor) in the Quad-Tree.
Problems wtih SMP
• If there isn’t enough RAM, additional machines are needed
• Idea of shared memory doesn’t work because there is a dramatic increase of contention on shared bus. Bus essentially becomes a bottleneck.
– Does placing interconnects between processors resolve this? No! Would require custom boards and distance between processors would become a factor for higher frequencies.
Moral of the story. . . SMP sucks! 13-3
13.1.3 Distributed Architectures & Memory Systems
• Each machine has its own local memory • There is no shared global memory • Processors communicate with each other by sending messages through network (i.e. Message Passing)
13.2 Message Passing
• Messages are usually sent over sockets • Messages transmitted between clusters or other types of DMS • Cluster - message sent over sockets • Means of communication - send / receive events • You send only what you need (that’s good, very efficient) • However, difficult to program (that’s bad) – ”Debugging hell” – Processors have to wair for data to be transmitted – Manual partitioning needed – Different programming paradigm – Deadlocks, deadlocks, deadlocks • Message Passing Interface (MPI) abstracts all transmissions to send/receive signals
13.2.1 MPI execution model
• Similar to fork - spawns multiple copies of the same program • Different behaviors are determined by who ”self” is • Master/Slave MPI Example - ”If I’m 0 (Master) I send messages to others, else (Slave) I wait for messages to receive”
13.2.2 Broadcasting
• Broadcast data - source sends to everyone, as opposed to point-to-point communi- cation, where communication is from me to you