Recap: Consensus
• On a synchronous system – There’s an algorithm that works. • On an asynchronous system CSE 486/586 Distributed Systems – It’s been shown (FLP) that it’s impossible to guarantee. Mutual Exclusion • Getting around the result – Masking faults – Using failure detectors – Still not perfect Steve Ko • Impossibility Result Computer Sciences and Engineering – Lemma 1: schedules are commutative University at Buffalo – Lemma 2: some initial configuration is bivalent – Lemma 3: from a bivalent configuration, there is always another bivalent configuration that is reachable.
CSE 486/586, Spring 2013 CSE 486/586, Spring 2013 2
Why Mutual Exclusion? Why Mutual Exclusion?
• Bank’s Servers in the Cloud: Think of two • Bank’s Servers in the Cloud: Think of two simultaneous deposits of $10,000 into your bank simultaneous deposits of $10,000 into your bank account, each from one ATM. account, each from one ATM. – Both ATMs read initial amount of $1000 concurrently from – Both ATMs read initial amount of $1000 concurrently from the bank’s cloud server the bank’s cloud server – Both ATMs add $10,000 to this amount (locally at the ATM) – Both ATMs add $10,000 to this amount (locally at the ATM) – Both write the final amount to the server – Both write the final amount to the server – What’s wrong? – What’s wrong?
• The ATMs need mutually exclusive access to your account entry at the server (or, to executing the code that modifies the account entry)
CSE 486/586, Spring 2013 3 CSE 486/586, Spring 2013 4
Mutual Exclusion Mutexes
• Critical section problem • To synchronize access of multiple threads to – Piece of code (at all clients) for which we need to ensure common data structures there is at most one client executing it at any point of time. Allows two operations: • Solutions: lock() – Semaphores, mutexes, etc. in single-node OS while true: // each iteration atomic – Message-passing-based protocols in distributed systems: if lock not in use: » enter() the critical section label lock in use » AccessResource() in the critical section break » exit() the critical section unlock() • Distributed mutual exclusion requirements: label lock not in use – Safety – At most one process may execute in CS at any time – Liveness – Every request for a CS is eventually granted – Ordering (desirable) – Requests are granted in the order they were made
CSE 486/586, Spring 2013 5 CSE 486/586, Spring 2013 6
C 1 Semaphores How Are Mutexes Used?
• To synchronize access of multiple threads to mutex L= UNLOCKED; extern mutex L; common data structures • Semaphore S=1; ATM1: ATM2 – Allows two operations lock(L); // enter lock(L); // enter – wait(S) (or P(S)): while(1){ // each execution of the while loop is atomic // critical section // critical section if (S > 0) obtain bank amount; obtain bank amount; S--; add in deposit; add in deposit; break; update bank amount; update bank amount; } unlock(L); // exit unlock(L); // exit – signal(S) (or V(S)): S++; – Each while loop execution and S++ are each atomic operations
CSE 486/586, Spring 2013 7 CSE 486/586, Spring 2013 8
Distributed Mutual Exclusion Assumptions/System Model Performance Criteria • Bandwidth: the total number of messages sent in • For all the algorithms studied, we make the following each entry and exit operation. assumptions: • Client delay: the delay incurred by a process at each – Each pair of processes is connected by reliable channels entry and exit operation (when no other process is in, (such as TCP). or waiting) – Messages are eventually delivered to recipients’ input buffer in FIFO order. – (We will prefer mostly the entry operation.) – Processes do not fail (why?) • Synchronization delay: the time interval between one process exiting the critical section and the next • Four algorithms process entering it (when there is only one process – Centralized control waiting) – Token ring • These translate into throughput — the rate at which – Ricart and Agrawala the processes can access the critical section, i.e., x – Maekawa processes per second. • (these definitions more correct than the ones in the textbook) CSE 486/586, Spring 2013 9 CSE 486/586, Spring 2013 10
1. Centralized Control 1. Centralized Control
• A central coordinator (master or leader) • Safety, liveness, ordering? – Is elected (next lecture) • Bandwidth? – Grants permission to enter CS & keeps a queue of requests to – Requires 3 messages per entry + exit operation. enter the CS. • Client delay: – Ensures only one process at a time can access the CS – one round trip time (request + grant) – Has a special token per CS • Operations (token gives access to CS) • Synchronization delay – one round trip time (release + grant) – To enter a CS Send a request to the coord & wait for token. – On exiting the CS Send a message to the coord to release the • The coordinator becomes performance bottleneck token. and single point of failure. – Upon receipt of a request, if no other process has the token, the coord replies with the token; otherwise, the coord queues the request. – Upon receipt of a release message, the coord removes the oldest entry in the queue (if any) and replies with a token.
CSE 486/586, Spring 2013 11 CSE 486/586, Spring 2013 12
C 2 2. Token Ring Approach CSE 486/586 Administrivia • Processes are organized in a logical ring: pi has a communication channel to p(i+1)mod (n). • PA2 due this Friday. • Operations: – More help by TAs this week – Only the process holding the token can enter the CS. – To enter the critical section, wait passively for the token. When in CS, hold • PA3 will be out this weekend. on to the token. – To exit the CS, the process sends the token onto its neighbor. • Practice problem set 1 & midterm example posted on – If a process does not want to enter the CS when it receives the token, it the course website. forwards the token to the next neighbor. – Will post solutions today • Features: • Safety & liveness, ordering? P0! Previous holder of token • Midterm on Wednesday (3/6) @ 3pm – Not Friday (3/8) • Bandwidth: 1 message per exit current holder PN-1! P1! • Client delay: 0 to N message of token • Come talk to me! transmissions. • Synchronization delay between one P2! process’s exit from the CS and the next holder of next process’s entry is between 1 P3! token and N-1 message transmissions.
CSE 486/586, Spring 2013 13 CSE 486/586, Spring 2013 14
3. Ricart & Agrawala’s Algorithm 3. Ricart & Agrawala’s Algorithm
• Processes requiring entry to critical section multicast • To enter the CS a request, and can enter it only when all other – set state to wanted processes have replied positively. – multicast “request” to all processes (including timestamp) • Messages requesting entry are of the form
CSE 486/586, Spring 2013 15 CSE 486/586, Spring 2013 16
3. Ricart & Agrawala’s Algorithm 3. Ricart & Agrawala’s Algorithm
On initialization state := RELEASED; To enter the section 41 state := WANTED; 41 p Multicast request to all processes; 3 T := request’s timestamp; p Reply Wait until (number of replies received = (N – 1)); 1 state := HELD;
On receipt of a request
CSE 486/586, Spring 2013 17 CSE 486/586, Spring 2013 18
C 3 Analysis: Ricart & Agrawala 4. Maekawa’s Algorithm
• Safety, liveness, and ordering? • Observation: no need to have all peers reply • Bandwidth: • Only need to have a subset of peers as long as all – 2(N-1) messages per entry operation subsets overlap. – N-1 unicasts for the multicast request + N-1 replies • Voting set: a subset of processes that grant – N-1 unicast messages per exit operation permission to enter a CS • Client delay • Voting sets are chosen so that for any two – One round-trip time processes, pi and pj, their corresponding voting sets have at least one common process. • Synchronization delay – Each process p is associated with a voting set v (of – One message transmission time i i processes) – Each process belongs to its own voting set – The intersection of any two voting sets is non-empty – Each voting set is of size K – Each process belongs to M other voting sets
CSE 486/586, Spring 2013 19 CSE 486/586, Spring 2013 20
4. Maekawa’s Algorithm 4. Maekawa’s Algorithm
• Simple example • Multicasts messages to a (voting) subset of processes
– To access a critical section, pi requests permission from all other processes in its own voting set vi – Voting set member gives permission to only one requestor P0 P3 at a time, and queues all other requests – Guarantees safety – Maekawa showed that K=M=√N works best – One way of doing this is to put N processes in a √N by √N matrix and take union of row & column containing pi as its P1 P2 voting set.
CSE 486/586, Spring 2013 21 CSE 486/586, Spring 2013 22
Maekawa’s Algorithm – Part 1 Maekawa’s Algorithm – Part 2
On initialization state := RELEASED; voted := FALSE; For pi to exit the critical section state := RELEASED; For p to enter the critical section i Multicast release to all processes in V ; state := WANTED; i On receipt of a release from pi at pj Multicast request to all processes in Vi; if (queue of requests is non-empty) Wait until (number of replies received = K); then state := HELD; remove head of queue – from pk, say; On receipt of a request from pi at pj send reply to pk; if (state = HELD or voted = TRUE) voted := TRUE; then else queue request from p without replying; i voted := FALSE; else end if send reply to pi; voted := TRUE; Continues on end if next slide!
CSE 486/586, Spring 2013 23 CSE 486/586, Spring 2013 24
C 4 Maekawa’s Algorithm – Analysis Summary
• Bandwidth: 2√N messages per entry, √N messages • Mutual exclusion per exit – Coordinator-based token – Better than Ricart and Agrawala’s (2(N-1) and N-1 – Token ring messages) – Ricart and Agrawala’s timestamp algorithm • Client delay: One round trip time – Maekawa’s algorithm – Same as Ricart and Agrawala • Synchronization delay: One round-trip time – Worse than Ricart and Agrawala • May not guarantee liveness (may deadlock) • How? P0
P1 P2
CSE 486/586, Spring 2013 25 CSE 486/586, Spring 2013 26
Acknowledgements
• These slides contain material developed and copyrighted by Indranil Gupta (UIUC).
CSE 486/586, Spring 2013 27
C 5