Distributed Algorithms

Master 2 IFI, CSSR Distributed Algorithms

Francesco Bongiovanni INRIA Sophia Antipolis Research Center OASIS Team [email protected] Course web site : deptinfo.unice.fr/~baude/AlgoDist Nov. 2009

Chapter 7 : Failure Detectors, Consensus,

Self-Stabilization 1 Acknowledgement

. The slides for this lecture are based on ideas and materials from the following sources:

. Introduction to Reliable Distributed Programming Guerraoui, Rachid, Rodrigues, Luís, 2006, 300 p., ISBN: 3-540-28845-7 (+ teaching material) . ID2203 Distributed Systems Advanced Course by Prof. Seif Haridi from KTH – Royal Institute of Technology (Sweden) . CS5410/514: Fault-tolerant Distributed Computer Systems Course by Prof. Ken Birman from Cornell University . Distributed Systems : An Algorithmic Approach by Sukumar, Ghosh, 2006, 424 p.,ISBN:1-584-88564-5 (+teaching material) . Various research papers

2 Outline 1. Failure Detectors . Definition . Properties – completeness and accuracy . Classes of FDs . Two algorithms : PFD and EPFD . Leader Election vs Failure Detector

2. Consensus . Definition . Properties . Types of Consensus : regular and uniform . Algorithm: hierarchical consensus

3. Self-stabilization . Principle

. Example: Dijkstra's Token ring 3 Failure detectors

4 System models

. synchronous distributed system . each message is received within bounded time . each step in a process takes lb < time < ub . each local clock’s drift has a known bound

. asynchronous distributed system . no bounds on process execution . no bounds on message transmission delays . arbitrary clock drifts the Internet is an asynchronous distributed system

5 Failure model

. First we must decide what do we mean by failure? . Different types of failures

. Crash-stop (fail-stop) . A process halts and Crashes does not execute any further operations Omissions . Crash-recovery Crashes and recoveries . A process halts, but then recovers (reboots) after Arbitrary (Byzantine) a while

. Crash-stop failures can be detected in synchronous systems . Next: detecting crash-stop failures in asynchronous systems

6 What's a Failure Detector ?

Pi Pj

7 What's a Failure Detector ?

Crash failure

Pi Pj

8 What's a Failure Detector ?

Needs to know about PJ's failure

Crash failure

Pi Pj

9 1. Ping-ack protocol

If pj fails, within T time units, pi will send it a ping message, and will time out within another T time units. Detection time = 2T Needs to know about PJ's failure

ping

Pi Pj ack

- Pj replies

- Pi queries Pj once every T time units

- if Pj does not respond within T time units,

Pi marks pj as failed 10 2. Heart-beating protocol

Needs to know about PJ's failure

heartbeat Pi Pj

- Pj maintains a sequence number - if P has not received a new heartbeat for the past i - P send P a heartbeat with T time units, P declares P as failed j i i j incremented seq. number after T' (=T) time units

If pj has sent x heartbeats until the time it fails, then pi will timeout within (x+1)*T time units in the worst case, and will detect p as failed. j 11 Failure Detectors

. Abstracting time

. FD provide information (not necessary fully accurate) about which processes have crashed

. Use failure detectors to encapsulate timing assumptions . Black box giving suspicions regarding node failures . Accuracy of suspicions depends on model strength

12 Failure Detectors

. Basic properties

. Completeness

. Every crashed process is suspected

. Accuracy

. No correct process is suspected

Both properties comes in two flavours . Strong and Weak

13 Failure Detectors

. Strong Completeness . Every crashed process is eventually suspected by every correct process

. Weak Completeness . Every crashed process is eventually suspected by at least one correct process

. Strong Accuracy . No correct process is ever suspected

. Weak Accuracy . There is at least one correct process that is never suspected

14 Failure Detectors

. Classes of FDs

Accuracy Weak Strong Eventually Eventually Weak Strong

Weak W Q Eventual ◊Q Completeness Weak ◊W Strong Perfect Eventual Eventually S P Strong Perfect Strong ◊S ◊P

Synchronous Systems Asynchronous Systems

15 Perfect Failure Detector (P)

16 Perfect Failure Detector (P)

17 Correctness of P

. PFD1 (strong completeness) . A crashed node doesn’t send . Eventually every node will notice the absence of

. PFD2 (strong accuracy) . Assuming local computation is negligible . Maximum time between 2 heartbeats . γ + δ time units . If alive, all nodes will recv hb in time . No inaccuracy

18 Eventually Perfect Failure Detector

19 Eventually Perfect Failure Detector

20 Correctness of EPFD

. PFD1 (strong completeness) . Same as before

. PFD2 (eventual strong accuracy) . Each time p is inaccurately suspected by a correct q . Timeout T is increased at q . Eventually system becomes synchronous, and T becomes larger than the unknown bound δ (T>γ +δ) . q will receive HB on time, and never suspect p again

21 Leader Election

22 Leader Election vs Failure Detection

. Failure detection captures failure behavior . Detect failed nodes

. Leader election (LE) also captures failure behavior . Detect correct nodes (a single & same for all)

. Formally, leader election is an FD . Always suspects all nodes except one (leader) . Ensures some properties regarding that node

23 Leader Election vs Failure Detection

. We’ll define two leader election algorithm

. Leader election (LE) which “matches” P . Eventual leader election (Ω) which “matches” eventual P

24 Matching LE and P . P’s properties . P always eventually detects failures (strong completeness) . P never suspects correct nodes (strong accuracy)

. Completeness of LE . Informally: eventually ditch crashed leaders . Formally: eventually every correct node trusts some correct node

. Accuracy of LE . Informally: never ditch a correct leader . Formally: No two correct nodes trust different correct nodes . Is this really accuracy? . Yes! Assume two nodes trust different correct nodes . One of them must eventually switch, i.e. leaving a correct node

25 LE desirable properties . LE always eventually detects failures . Eventually every correct node trusts some correct node

. LE is always accurate . No two correct nodes trust different correct nodes

. But the above two permit the following

. But P1 is “inaccurately” leaving a correct leader

26 LE desirable properties . To avoid “inaccuracy” we add . Local Accuracy:

. If a node is elected leader by pi, all previously elected leaders

by pi have crashed

Not allowed, as P is 1 correct

27 Leader election - interface

28 Leader election - algorithm

29 Matching Ω and EPFD . Eventual P weakens P by only providing eventual accuracy

. Weaken LE to Ω by only guaranteeing eventual agreement

LE Properties:

 LE1 (eventual completeness). Eventually every eventual correct node trusts some correct node

 LE2 (agreement). No two correct nodes trust different correct nodes

 LE3 (local accuracy).If a node is elected leader by pi,

all previously elected leaders by pi have crashed

30 Eventual Leader election - interface

31 Eventual Leader election - algorithm

. See in the book...

32 Consensus (agreement)

33 Consensus

. In the consensus problem, the processes propose values and have to agree on one among these values B A

. Solving consensus is key to solving many problems in distributed computing (e.g., total order broadcast, atomic commit, terminating reliable broadcast) 34 Consensus – cannonical application

. a set of servers implement a distributed database

. a subset of servers participate in a particular transaction . some of the servers may fail . the remaining servers must agree on whether to install the results of the transaction to the database or discard them

35 Consensus – cannonical application

36 Consensus – cannonical application

37 Consensus – basic properties

. Termination . Every correct node eventually decides

. Agreement . No two correct processes decide differently

. Validity . Any value decided is a value proposed

. Integrity: . A node decides at most once

38 FLP impossibility result

. Consensus in Asynchronous System

. Impossibility of consensus in the fail-silent model

. FPL (Fischer, Lynch and Peterson 1985) : consensus is impossible in the fail-silent model with deterministic processes, even if only one process crashes

. No way to satisfy agreement (safety) and termination (liveness) together

39 How to solve consensus in asynchronous systems with crashes ?

. How to solve consensus in the presence of crashes ?

. Either we relaxed our system model, that is, we assume partial synchrony

. Either we modify the specifications . Constraining the set of inputs . Change the termination property: terminates with some probability

. Or... 40 How to solve consensus in asynchronous systems with crashes ?

. Intuitively consensus is impossible to solve because : . 1) the decision depends on one process . 2) we have no idea if this process is alive (we have to wait for its message) or dead.

. Thus we add to the asynchronous system what it needs in order to solve the consensus: . Failure detectors

41 (regular) Consensus

42 (regular) Consensus

. Sample execution

Question : does it satisfy consensus ?

43 Uniform consensus

44 Uniform consensus

Question: Does it satisfy uniform consensus ? 45 Hierarchical consensus

. Use perfect fd (P) and best-effort bcast (BEB)

. Each node stores its proposal in proposal . Possible to adopt another proposal by changing proposal . Store identity of last adopted proposer in lastprop

. Loop through rounds 1 to N . In round i . node i is leader and . broadcasts proposal v, and decides proposal v . other nodes . adopt i’s proposal v and remember lastprop i or . detect crash of i 46 Hierarchical consensus idea

. Basic idea of hierarchical consensus . There must be a first correct leader p, . P decides its value v and bcasts v . BEB ensures all correct nodes get v . Every correct node adopts v . Future rounds will only propose v

47 Problem with orphan messages...

Only adopt from node i if i>lastProp? 48 Invariant to avoid orphans

. Leader in round r might crash, . but much later affect some node in round>r

. Invariant . adopt if proposer p is ranked lower than lastprop . otherwise p has crashed and should be ignored

49 Execution without failure...

50 Execution with failure...

Is it uniform ? 51 Hierarchical consensus Impl. (1)

Last adopted proposal and Last adopted proposer id

52 Hierarchical consensus Impl. (2)

set node’s initial proposal, unless it has already adopted another node’s

If I am leader Trigger once per round

Trigger if I have proposal

Permanently decide

Next round if deliver or crash

Invariant: only adopt “newer” than what you have

53 Correctness

. Validity . Always decide own proposal or adopted value

. Integrity . Rounds increase monotonically . A node only decide once in the round it is leader

. Termination . Every correct node makes it to the round it is leader in . If some leader fails, completeness of P ensures progress . If leader correct, validity of BEB ensures delivery

54 Correctness (2) . Agreement . No two correct nodes decide differently

. Take correct leader with minimum id i . By termination it will decide v . It will BEB v . Every correct node gets v and adopts it . No older proposals can override the adoption . All future proposals and decisions will be v

. How many failures can it tolerate? . N-1

55 Self-stabilization

56 Recall

. Main challenges in distributed systems: . Failures . Concurrency

. In presence of (permanent) failures, a robust algorithm guarantees . Liveness properties are eventually achieved . Safety properties are never violated

57 Self-Stabilization

. Self-stabilization is a different approach to fault tolerance

. it considers transient (temporary) failures . it is more optimistic . If bad thing happen (safety is violated), the system will recover within a finite time, and will behave nicely afterwards.

58 Definition

. “A system is self-stabilizing when, regardless of its initial state, it is guaranteed to arrive at a legitimate state in a finite number of steps.” 1 Edsger W. Dijkstra

[1] Edsger W. Dijkstra, Self-stabilizing systems in spite of distributed control, Communications of the ACM, v.17 n.11, p.643644,Nov. 1974

59 Self-Stabilization

. System S is self-stabilizing with respect to predicate P that identifies the legitimate states, if:

. Convergence . Starting from any arbitrary configuration, S is guaranteed to reach a configuration satisfying P, within a finite number of state transitions. . Closure . P is closed under the execution of S. That is, once in a legitimate state, it will stay in a legitimate state.

60 Some advantages of Self-Stabilizing systems

. No need for consistent initialization. . Starting in any arbitrary state, the system will converge to a legitimate state.

. Possibility of sequential composition without the need for termination detection.

61 A self-stabilizing algorithm: Dijkstra's Token ring

62 Dijkstra's Token ring

 A single token circulates over the ring and grants privilege to the process holding it.

 N+1 processes: P0, P2,….,Pn

 Connected in a ring

 Predecessor of Pi

pred(Pi ) = P(i-1) mod N+1

. Successor of Pi

succ(Pi ) = P(i+1) mod N+1

63 Token Ring stabilization

 Pi has a local variable Xi

 Xi can take values from 0 to K-1 (K >= N)  Each process, can read the value of its predecessor (Shared Memory Model)

 There is a scheduler, which selects a process at each step, in a random but fair manner.

64 Token Ring stabilization