Distributed Algorithms

DISTRIBUTED ALGORITHMS

Lecture Notes for

Fall

Nancy A Lynch

Boaz PattShamir

January

Preface

This rep ort contains the lecture notes used by Nancy Lynchs graduate course in Distributed

Algorithms during fall semester The notes were prepared by Nancy Lynch and

Teaching Assistant Boaz PattShamir

The main part of this rep ort is detailed lectures notes for the regular class meetings

We think the reader will nd these to b e reasonably well p olished and wewould very much

appreciate comments ab out any errors or suggestions on the presentation Following these

notes are the homework assignments Finallywe included an app endix that contains notes

for three additional lectures that were given after the regular term was over The extra

lectures cover material that did not t into the scheduled numb er of class hours These

notes are in a very rough shap e

Many thanks go to the students who to ok the course this semester they contributed

many useful comments and suggestions on the technical content and also had a very strong

inuence on the pace and level of the presentation If these notes are understandable to other

audiences it is largely b ecause of the feedbackprovided by this rst audience Thanks also

go to George Varghese for contributing an excellent lecture on selfstabilizing algorithms

Wewould also liketoacknowledge the work of Ken Goldman and Isaac Saias in preparing

earlier versions and resp ectively of the notes for this course Wewere able to

reuse much of their work The students who to ok the course during those twoyears worked

hard to prepare those earlier lecture notes and also deserve a lot of thanks Jennifer Welch

and Mark Tuttle also contributed to earlier versions of the course Finallywewould liketo

thank Joanne Talb ot for her invaluable help

Nancy Lynch lynchtheorylcsmitedu

Boaz PattShamir boaztheorylcsmitedu

January

Contents

LECTURES

Lecture Septemb er

Intro duction to the Course

The Sub ject Matter

Style

Overview of the Course

Synchronous Network Algorithms

Problem Example Leader Election in a Ring

Algorithm LeLann ChangRob erts

Lecture Septemb er

Leader Election on a Ring cont

Algorithm Hirshb ergSinclair

Counterexample Algorithms

Lower Bound on ComparisonBased Proto cols

Leader Election in a General Network

BreadthFirst Search

Extensions

Lecture Septemb er

Shortest Paths Unweighted and Weighted

Minimum Spanning Tree

Maximal Indep endentSet

Lecture Septemb er

Link Failures

The Co ordinated Attack Problem

Randomized Consensus

Faulty Pro cessors

Stop Failures

Lecture Septemb er

Byzantine Failures

Reducing the Communication Cost

Stopping Faults

The Byzantine Case

The TurpinCoan Algorithm

Lecture Septemb er

Numb er of Pro cesses for Byzantine Agreement

Byzantine Agreement in General Graphs

Weak Byzantine Agreement

Numb er of Rounds with Stopping Faults

Lecture Octob er

Numb er of Rounds With Stopping Failures cont

The Commit Problem

Two Phase Commit PC

Threephase Commit PC

Lower Bound on the Numb er of Messages

Lecture Octob er

Asynchronous Shared Memory

Informal Mo del

Mutual Exclusion Problem

Dijkstras Mutual Exclusion Algorithm

A Correctness Argument

Lecture Octob er

Dijkstras Mutual Exclusion Algorithm cont

An Assertional Pro of

Running Time

Improved Mutual Exclusion Algorithms

NoStarvation Requirements

Petersons TwoPro cess Algorithm

Tournament Algorithm

Iterative Algorithm

Lecture Octob er

Atomic Ob jects

Example ReadWrite Ob ject

Denition

Atomic Snapshots

Problem Statement

Unb ounded Algorithm

Bounded Register Algorithm

Lecture Octob er

Burns Mutual Exclusion Algorithm

Lamp orts Bakery Algorithm

Analysis

The Numb er of Registers for Mutual Exclusion

Two Pro cesses and One Variable

Three pro cesses and TwoVariables

The General Case

Lecture Octob er

Consensus Using ReadWrite Shared Memory

Imp ossibility for Arbitrary Stopping Faults

Imp ossibility for a Single Stopping Fault

Mo deling and Mo dularity for Shared Memory Systems

The Basic InputOutput Automaton Mo del

Lecture Octob er

IO Automata cont

Problem Sp ecication

Pro of Techniques

Shared Memory Systems as IO Automata

Instantaneous Memory Access

Ob ject Mo del

Mo dularity Results

Transforming from Instantaneous to Atomic Ob jects

Comp osition in the Ob ject Mo del

Lecture Octob er

Mo dularity cont

Waitfreedom

Concurrent Timestamp Systems

Denition of CTS Ob jects

Bounded Lab el Implementation

Correctness Pro of

Application to Lamp orts Bakery Algorithm

Lecture Novemb er

MultiWriter Register Implementation Using CTS

The Algorithm

A Lemma for Showing Atomicity

Pro of of the Multiwriter Algorithm

Algorithms in the ReadMo difyWrite Mo del

Consensus

Mutual Exclusion

Lecture Novemb er

ReadMo difyWrite Algorithms cont

Mutual Exclusion cont

Other Resource Allo cation Problems

Mutual exclusion is only one kind of resource allo cation problem mo deling access to a single

shared resource In this section we generalize the problem There are twoways to describ e

the generalizations The rst is exclusion problems and the second is resource problems

We start with the new denitions

Denitions

In the rst variant we describ e exclusion problemsFormallywe describ e these problems

by a collection E of conicting sets of pro cesses that are not allowed to co exist in C E

could b e any collection of sets of pro cesses that is closed under containment ie if S E

and S S then S EFor example the mutual exclusion problem is characterized by

E fS jS j g

A common generalization is the k exclusion problem mo deling k shared resources where

at most k pro cesses may b e in the critical region at anygiven time In our language the

problem is dened by

E fS jS j k g

Clearlywe can b e more arbitrary eg say E is dened to contain f g f g f g f g

and other containing subsets But note that in this case do esnt exclude and do esnt

exclude

This leads us to the second variant of the denition in terms of resources In this case

wehave a set of explicit resources in the problem statement and for each p wegive a list

of requirements in the form of a monotone Bo olean formula eg R R R R

where the R s are resources Intuitively the meaning of the example is that p needs

i i

either R or R and either R or R in order to go critical More formally a set of resources

is acceptable for a pro cess to go critical i the setting all the corresp onding variables to true

yields a satisfying assignment for the formula

Lets consider another simple example to illustrate this style of formulation Supp ose p

wants resources R and R p wants R and R p wants R and R andp wants R and

R In this example the corresp onding formulae have only a single clause ie there are no

options here

Note that any resource problem gives rise to an exclusion problem where the combina

tions excluded by the formulas cannot b e simultaneously satised Eg for the example

ab ove we get an equivalentformulation by

E fS S f g f g f g f g f gg

It is also true that every exclusion condition can b e expressed as a resource problem and

thus these formulations are equivalent

For stating a general problem to b e solved in concurrent systems we again assume the

cyclic b ehavior of the requests as b efore The mutual exclusion condition is now replaced by

a general exclusion requirement The denitions of progress and lo ckoutfreedom remain the

same as b efore We actually want to require something more since there are some trivial

solutions that satisfy the requirements listed so far any solution to mutual exclusion is a

fortiori a solution to more general exclusion But theres something wrong with this we

wanttoinsistthat moreconcurrency is somehow p ossible We dont knowwhatisthebest

way to express this intuitive notion We give here a slightly stronger condition in terms of

the resource formulation as follows

Consider anyreachable state in which pro cess i is in T andevery pro cess that

has any of the samenamedresources in its resource requirementformula is in

R Then if all the conicting pro cesses stayinR and i continues to take steps

eventually i pro ceeds to C Note We allow for other pro cesses to halt

Dijkstra had originally dened a sp ecial case called the Dining Philosophers problem

describ ed b elow as a resource problem p0 F4 F0

p4 p1

F3 F1

p3 F2 p2

Figure The dining philosophers problem for n

In the traditional scenario see Figure wehaveve say philosophers at a table

usually thinking ie in R From time to time each philosopher wants to eat ie to enter

C which requires two forks representing the resources we assume that each philosopher

can only take the forks adjacent to him or her Thus the formula for p is F F

i i i

counting mo d We can represent this as an exclusion problem namely

E fS S fF F g for some ig

i i

We assume that there is a shared variable asso ciated with each fork and the access mo del

for the variables is readmo difywrite If the pro cesses are all identical and they dont use

their indices then wehave the same kind of symmetry problem wesaw in electing a leader

in a ring which cannot b e solved To b e concrete supp ose they all try to pick up their left

fork rst and then their right This satises the exclusion condition but it can deadlo ck if

all wake up at the same time and grab their left forks then no one can pro ceed this is worse

than a deadlo cking execution the system actually wedges Thus wemust use some kind

of symmetry breaking technique and here wemake the assumption that the pro cess indices

ie lo cations are known to the pro cesses

Dijkstras original solution requires simultaneous lo cking of the two adjacentforksa

centralized shared readmo difywrite variable and has no fairness Well see a b etter solution

b elow

The LeftRight Algorithm

This algorithm app eared in Burns pap er but it is generally considered as folklore This

algorithm ensures exclusion deadlo ckfreedom in the stronger version stated ab ove and

lo ckoutfreedom Also it has a go o d time b ound indep endent of the size of the ring

This is a simple style of algorithm based on running around and collecting the forks one

at a time Wehave to b e careful ab out the order in which this is done For instance if the

order is arbitrary then we risk deadlo ckasabove Even if there is no danger of deadlo ck we

might b e doing very p o orly in terms of time p erformance Eg if everyone gets their forks

in numerical order of forks then although they dont deadlo ck this might result in a long

waiting chainTo see this consider the following scenario In Figure ab ove supp ose

that p gets b oth forks then p gets its right fork then waits for its left p gets its right

fork then waits for its left p gets its right fork then waits for its left and p waits for its

right fork See Figure for nal conguration

p0 F4 F0

p4 p1

F3 F1

p3 F2 p2

Figure A conguration with long waiting chain Arrows indicate p ossession of forks

In the last state wehavea chain where p waits for p whichwaits for p whichwaits

for p whichwaits for p The pro cesses in the waiting chain can go into the critical region

only sequentially In general this means that time can b e at least linear in the size of the

ring

In contrast the Burns algorithm has a constant b ound indep endent of the size of the

ring The algorithm accomplishes this byavoiding long waiting chains We describ e the

algorithm here for n even The variant for o dd n is similar and is left as an exercise The

algorithm is based on the following simple rule even numb ered pro cesses pick up their left

forks rst and o dd numb ered pro cesses get their right forks rst Thus the asymmetry

here is the knowledge of the parityVariables corresp ond to the forks as b efore Now each

variable contains a queue of pro cesses waiting for the fork of length at most The rst

pro cess on the queue is assumed to own the fork The co de is given in Figure

Prop erties Mutual exclusion is straightforward Deadlo ckfreedom follows from lo ckout

freedom which in turn follows from a time b ound whichwe provenow Assume as usual

that the step time at most s and that the critical region time at most c Dene T nto

b e the worst time from entry to trying region until entry to critical region in an npro cess

ring Our goal is to b ound T n As an auxiliary quantitywedeneS n to b e the worst

time from when a pro cess has its rst fork until entry to critical region

try

Eect pc lookleft

lookleft

Precondition pc lookleft

Eect if i is not on x queue then add i to end

else if i is rst on x queue then pc lookright

lookright

Precondition pc lookright

Eect if i is not on x queue then add i to end

else if i is rst on x queue then pc beforeC

crit

Precondition pc beforeC

Eect pc C

exit

Eect return set fL Rg

pc return

returnleft

Precondition pc return

L returnset

Eect returnset returnset fLg

remove i from x queue

returnright

Precondition pc return

R returnset

Eect returnset returnset fRg

remove i from x queue

rem

Precondition pc return

returnset

Eect pc rem

Figure Leftright dining philosophers algorithm co de for pro cess p i even

Let us start by b ounding T n in terms of S n p tests its rst fork within time s of

when it enters T When p rst tries either it gets its rst fork immediatelyornotIfso

then S n time later it go es critical If not then the other pro cess has the rst fork At

worst that other pro cess just got the fork so it takes at most S nc s until it gets to

C nishes C and returns b oth forks Then since p recorded its index in the queue for its

rst fork within additional time s p will retest the rst fork and succeed in getting it and

then in additional S n time p reaches C This analysis says that

T n s max fS nSnc s s S ng s c S n

Wenow b ound S n p tests its second fork within time s of when it gets the rst fork

Again we need to consider two cases If it gets the second fork immediately then within an

additional time s p go es critical If not then the other pro cess has it and b ecause of the

arrangement since its p s second fork it must b e its neighb ors second fork also So within

time at most s c s the neighb or gets to C leaves and returns b oth forks Then within

additional time s p will retest the second fork and succeed in getting it Then additional

time s will take p to C This analysis says that

S n s maxfs s c s s sg s c

Putting Eq and together we conclude that

T n s c s cs c

Note that the b ound is indep endentofn This is a very nice prop erty for a distributed

algorithm to have It expresses a strong kind of lo calityornetwork transparency

Extension to More General Resource Problems

Due to the nice features of the LeftRight algorithm it is natural to ask whether can we

extend this alternating leftright idea to more general resource allo cation problems In this

section weshow an extension that is not completely general but rather applies only to

conjunctive problems where the resource requirement of a pro cess is exactly one particular

set This case do es not cover k exclusion for example whichisadisjunctive problem with

more than one alternativeallowed

Again for each resource wehave an asso ciated variable shared by all pro cesses that use

that resource The variable contains a FIFO queue to record whos waiting as b efore As

b efore the pro cesses will wait for the resources one at a time Toavoid deadlo ck we arrange

resources in some global total ordering and let each pro cess seek resources in order according

to this ordering smallest to largest It is easy to see that this prevents deadlo ck if pro cess

i waits for pro cess j then j holds a resource which is strictly larger than the one for which i

is waiting and hence the pro cess holding the largestnumb ered resource can always advance

The FIFO nature of the queues also prevents lo ckout

However the time p erformance of this strategy is not very go o d in general There is no

limit on the length of waiting chains except for the numb er of pro cesses and so the time

b ounds can b e linearly dep endentonthenumb er of no des in the network Wesaw suchan

example ab ove for rings So wemust rene the ordering scheme by giving a waytocho ose a

go o d total ordering ie one that do esnt p ermit long waiting chains We use the following

strategy for selecting a go o d total order Dene the resourceproblem graph where the no des

represent the resources and wehave an edge from one no de to another if there is a user that

needs b oth corresp onding resources See Figure for an example of Dining Philosophers with no des

F0 F1 p1

p0 p2

F5 F2

p5 p3

F4 F3

Figure Resource graph for six dining philosophers

Now color the no des of the graph so that no two adjacent no des have the same color

We can try to minimize the numb er of colors nding minimal coloring is NPhard but a

greedy algorithm is guranteed to color the graph in no more than m colors

Now totally order the colors This only partially orders the resources but it do es totally

order the resources needed byany single pro cess See Figure for an example

Now each pro cess accesses the forks in increasing order according to the color ordering

Equivalentlytakeany top ological ordering of the given partial ordering and let all pro cesses

follow that total ordering in the earlier strategy Carrying on with our example this b oils

down to the alternating leftright strategy

It is easy to see that the length of waiting chains is b ounded by the numb er of colors

This is true since a pro cess can b e waiting for a resource of minimum color which another

pro cess has That pro cess could only b e waiting for a resource of larger color etc

The algorithm has the nice prop erty that the worstcase time is not directly dep endent F0 F1 p1

p0 p2

F5 F2

p5 p3

F4 F3

Figure coloring

on the total numb er of pro cesses and resources Rather let s b e an upp er b ound on step

time and c an upp er b ound on critical section time as b efore Let k be the numb er of colors

and let m b e the maximal numb er of users for a single resource We showthattheworst

k k

case time b ound is O m c km s That is time dep ends only on lo cal parameters

We can make a reasonable case that in a realistic distributed network these parameters are

actually lo cal not dep endent on the size of the network Note that this a big b ound it

is exp onential in k so complexity is not prop ortional to the length of the waiting chain but

is actually more complicated There are some complex interleavings that get close to this

b ound

Toprove the upp er b ound we use the same strategy is as b efore Dene T for i k

indicating colors and j m indicating p ositions on a queue to b e the worstcase time

from when a pro cess reaches p osition j on any queue of a resource with color iuntil it

reaches C Then set up inequalities as for the LeftRight algorithm The base case is when

the pro cess already has the last resource

T s

Also when a pro cess is rst on any queue within time s it will b e at worst at p osition m

on the next higher queue

T s T for all i k

i im

And lastly when a pro cess in in p osition j on queue i it only has to wait for its

predecessor on the queue to get the resource and give it up and then it gets the resource

T T c ks T for all i and for all j

ij ij i

The claimed b ound follows from solving for T after adding an extra s

Cutting Down the Time Bound This result shows that a very general class of resource

allo cation problems can have time indep endentofglobalnetwork parameters But it would

b e nice to cut down the time b ound from exp onential to linear in the numb er of colors There

are some preliminary results byAwerbuchSaks and ChoySingh But there is probably still

more work to b e done

Safe and Regular Registers

In this section we switch from the very p owerful readmo difywrite shared memory mo del to

some muchweaker variants We start with some denitions

Denitions

Wenow fo cus on readwrite ob jects registers again and consider generalizations of the

notion of an atomic readwrite register safe and regular registers These variants are weaker

than atomic registers but can still b e used to solveinteresting problems They t our general

ob ject denition in particular inputs are read and write v with outputs as b efore The

ix ix registers are mo deled as IO automata as in Figure

write(v)

write−respond

X read

read−respond(v)

Figure Concurrent ReadWrite Register X Subscripts mentioned in the text are

omitted from the diagram

As b efore the index i on the read and write op erations corresp onds to a particular line

on which a pro cess can access the register Also as b efore we assume that op erations on each

line are invoked sequentially ie no new op erations are invoked on a line until all previous

op erations invoked on that line have returned But otherwise op erations can overlap

Recall the denition of atomic registers An atomic register has three prop erties it

preserves wellformedness it satises the atomicity condition and fairness implies that op

erations complete Also dep ending on the architecture of the implementation we can have

awaitfree condition Nowwe generalize the second atomicity condition without changing

the other conditions This still ts the general ob ject sp ec mo del so the mo dularity results

hold this allows us to build ob jects hierarchicallyeven preserving the waitfree prop erty

For these weaker conditions we only consider singlewriter registers This means that

writes never overlap one another We require in all cases that any read that do esnt overlap

a write gets the value of the closest preceding write or the initial value if none This is

what would happ en in an atomic register The denitions dier from atomic registers in

allowing for more p ossibilities when a read do es overlap a write

The weakest p ossibilityisa safe registerinwhichareadthatoverlaps a write can return

an arbitrary value The other p ossibilityisa regular registerwhich falls somewhere in

between safe and atomic registers As ab ove a read op eration on a regular register returns

the correct value if it do es not overlap any write op erations However if a read overlaps one

or more writes it has to return the value of the register either b efore or after anyofthe

writes it overlaps

W2 R1 R2

Figure Read Overlapping Writes

For example consider the scenario in Figure The set of feasible writes for R

is fW W W W W g b ecause it is allowed to return a value written byany of these

write op erations Similarly the set of feasible writes for R is fW W W g The imp ortant

dierence b etween regular and atomic registers is that for regular registers consecutive reads

can return values in inverted order

Implementation Relationships for Registers

Supp ose we only have safe registers b ecause thats all our hardware provides and wewould

liketohave atomic registers We shall showthatinfactwe can start with safe singlereader

singlewriter binary registers and build all the way up to atomic multireader multiwriter

k ary registers for arbitrary k Lamp ort carries out a series of waitfree constructions starting

with the simplest registers and ending up with singlereader singlewriter k ary registers

Others have completed the remaining constructions Wehave already seen howtogetmulti

writer multireader registers from singlewriter multireader registers although there is ro om

for improved eciency Also Singh Anderson and Gouda havegiven an implementation

of singlewriter multireader registers from singlewriter singlereader registers

In this general development some of the algorithms are logically complex andor time

consuming so there is some question ab out their practicality

The general developmentwe consider here can b e formalized in the ob ject sp ec mo del

So we get transitivity as discussed earlier Recall the architecture

reader line RP

x1 writer 1 line WP1

x2 writer 2 line

WP2

Figure Example Implementing a writer reader register with two writer reader

registers

In Figure we see a particular implementation of a writer reader register in terms

of two writer reader registers There is a pro cess for each line of the register b eing

implemented ie for each of the highlevel writers and the highlevel reader These pro cesses

are connected to the lowlevel registers in suchaway that eachlineofalowlevel register is

connected to only one pro cess

In general we call the highlevel registers logical registers and the lowlevel registers

physical registers In all of the constructions we will consider a logical register is constructed

from one or more physical registers Each input line to the logical register is connected to a

pro cess These pro cesses in turn are connected to one or more of the physical registers using

internal lines Exactly one pro cess is connected to anyinternal line of a physical register

This p ermits the pro cesses to guarantee sequentiality on each line Pro cesses connected to

external write lines are called write pro cesses and pro cesses connected to external read lines

are called read pro cesses Note that nothing prevents a read pro cess from b eing connected

to an internal write line of a physical register and vice versa

We follow Lamp orts development Wehave safe vs regular vs atomic singlereader

vs multireader and binary vs k ary registers Wethus consider the twelve dierent kinds

of registers this classication gives rise to and see which register typ es can b e used to

implement which other typ es The implementation relationships in Figure should b e

obvious b ecause they indicate typ es of registers that are sp ecial cases of other typ es

Safe Regular Atomic

nreader nreader nreader

k ary k ary k ary

s s s

J J J

nreader reader nreader reader nreader reader

s s s s s s

binary k ary binary k ary binary k ary

I I J J J

J J J

s s s

reader reader reader

binary binary binary

Figure Obvious implementation relationships b etween register typ es An arrow from

typ e A to typ e B means that B can b e implemented using A

Lamp ort presents ve constructions to show other implementation relationships In the

following constructions actions on external lines are always sp ecied in upp ercase whereas

actions on internal lines are sp ecied in lowercase For example WRITE and READ denote

external op erations whereas write and read denote internal op erations

N Reader Registers from SingleReader Registers

The following construction Construction in the pap er implements an nreader safe register

from n singlereader safe registers The same algorithm also implements an nreader regular

lines of all n internal registers as in Figure

WP W

r w

RP R

r w

RP R

r w

RP n R

n n

Figure N Reader Registers from n Reader Registers

Read pro cess i is connected to the read line of the ith physical register Thus the write

pro cess writes all the physical registers while each read pro cess only reads one physical

WRITE v For all i in fnginvoke writev on x Wait for acks from all x and then

i i

do ACK The writes can b e done in any order

READ Invoke read on x Wait for return v and then do RETURN v

i i

Claim If x x are safe registers then so is the logical register

Pro of Within each WRITE exactly one write is p erformed on each particular register x

Therefore since WRITE op erations o ccur sequentially write op erations for each particular

x are also sequential Likewise read op erations for a particular x are also sequential

i i

Therefore eachphysical register has the required sequentiality of accesses This means that

the physical registers return values according to their second conditions b ehavior sp ecs

Nowifa READsayby RP do es not overlap any WRITE then its contained read do es

not overlap any write to x Therefore the correctness of x assures that the read op eration

i i

gets the value written by the last completed write to x This is the same value as written by

the last completed WRITE and since RP returns this value this READ returns the value

of the last completed WRITE

Claim If x x areregular registers then so is the logical register

Pro of We can reuse the preceding pro of to get the required sequentiality of accesses to

eachphysical register and to provethatany READ that do es not overlap a WRITE returns

the correct value Therefore we only need to showthatifaREAD pro cess R overlaps some

WRITE op erations then it returns the value written by a feasible WRITE Since the read r

for R falls somewhere inside the interval of R the set of feasible writes for r corresp onds to

a subset of the feasible WRITES for R

Therefore regularity of the physical register x implies that r gets one of the values

written by the set of feasible writes and hence R gets one of the values written by the set

of feasible WRITES

Consider two READ s done sequentially at dierent pro cesses bracketed within a single

WRITE They can obtain outoforder results if the rst sees the result of the WRITE

but the later one do es not since WRITE isnt done atomically This implies that the

construction do es not make the logical register atomic even if the x are atomic

With this construction Figure reduces to Figure

Waitfreedom The previous construction guarantees that all logical op erations terminate

in a b ounded numb er of steps of the given pro cess regardless of what the other pro cesses

do Therefore the implementation is waitfree In fact the prop erty satised is stronger

since it mentions a b ounded numb er of steps Sometimes in the literature waitfreedom is

formulated in terms of suchabound

K ary Safe Registers from Binary Safe Registers

j k

If k then we can implementa k ary safe register using l binary safe registers Con

struction in the pap er Wedothisby storing the ith bit of the value in binary register x

The logical register will allow the same numb er of readers as the physical registers do see

Figure

The co de for this construction is as follows

WRITE v For i in flg in any order write bit i of the value to register x

Safe Regular Atomic

nreader k ary k ary

k ary

s s s

nreader reader

X s s

binary k ary

s s s

binary binary reader

binary

Figure Collapsed Implementation Relationships

x x x

WP W

RP R

R RP

n n

Figure K ary Safe Registers from Binary Safe Registers

READ For i in flg in any order read bit i of value v from register x Then

i i

RETURNv

Note that unlike the Construction this construction works for safe registers onlyiea

k ary regular register cannot b e constructed from a binary regular register using this metho d

With this construction Figure reduces to Figure

Safe Regular Atomic

nreader k ary

k ary

s s s

X Q

nreader reader

X s s

binary k ary

J Q

s s Q

reader binary

binary

Figure Collapsed Implementation Relationships

JJ Distributed Algorithms Novemb er

Lecturer Nancy Lynch

Lecture

We continue investigating the implementation relationships among the dierent register mo d

els In the last lecture wesaw Constructions and and wehave consequently reduced

the relationship to the one describ ed in Figure

Safe Regular Atomic

nreader k ary

k ary

s s s

Q X

nreader reader

s s X

binary k ary

Q J

Q s s

reader binary

binary

Figure Collapsed Implementation Relationships

Construction Binary Regular Register from Binary Safe

A binary regular register can easily b e implemented using just one binary safe register see

Figure

Recall that a read from a binary safe register may return anyvalue from its domain the

read overlaps a write For binary registers the domain consists of only and Notice that

since a binary regular register is only required to return a feasible value it suces to ensure

W WP

RP RP

R R

Figure Binary Regular Register from Binary Safe Register

that all writes done on the lowlevel register actually change the value b ecause in this

case either value is feasible

The basic idea is that the write pro cess WPkeeps track of the contents of register x

This is easy b ecause WP is the only writer of x WP do es a lowlevel write only when it

is p erforming a WRITE that would actually change the value of the register If the WRITE

is just rewriting the old value then WP nishes the op eration rightaway without touching

x Therefore all low level write s toggle the value of the register Sp ecically the co de is

as follows

write v if v is the last recorded value then do WRITERESPONDelsedo write v and

up on receiving writerespond do WRITERESPOND

read do read up on receiving readrespondx do READRESPONDx

Now consider what happ ens when a READ is overlapp ed byaWRITE If the corresp ond

ing write is not p erformed ie the value is unchanged then register x will just return the

old value and this READ will b e correct If the corresp onding write is p erformed x may

return either or However b oth and are in the feasible value set of this READ b ecause

the overlapping WRITE is toggling the value of the register Therefore the READ will b e

correct

The implementation relations of Figure now reduce to Figure

Regular Atomic

k ary nreader

k ary

s s

nreader reader

s s X

binary k ary

s s

binary and reader

all safe binary

Figure Collapsed Implementation Relationships

Construction K ary Regular Register from Binary Reg

ular Register

We can implementa k ary regular register using k binary regular registers arranged as in

Figure Here we assume that the k values of the register are enco ded as k

We will use a unary enco ding of these values and the idea which has app eared in other

pap ers by Lamp ort and byPeterson of writing a sequence of bits in one order and reading

them in the opp osite order We remark that Lamp ort also gives an improved implementation

for binary representation thus reducing the space requirement logarithmically

If the initial value of the logical register is v then initially x is and the other physical

registers are all using unary representation The co de is as follows

WRITE v First write inx Then in order write inx x

v v

READ Do read x x x in order until x is found for some v Then RETURN

k v

Note that RP is guaranteed to nd a nonzero x b ecause whenever a physical register

i v

is zero ed out there is already a written in a higher index register Alternativelywe could

initialize the system so that x and then wewould have x always equal to We

k k

nowprove the correctness of Construction formally

Claim If a READ R sees in x then v must have been written by a WRITE that is

feasible for R

x x

k k

W WP

R RP

n n

Figure K ary Regular Registers from Binary Regular Registers

Pro of By contradiction Supp ose R sees x and neither an overlapping nor an

immediately preceding WRITE wrote v to the logical register Then v was written either

sometime in the past sayby WRITE W or v is the initial value We analyze b elowthe

former case the initial value case can b e treated similarly

So let W b e the last WRITE completely preceding R that wrote v Since v is not a

feasible value for R there must b e another WRITE W after W which completely precedes

RAny such W ie a WRITE that follows W and completely precedes R can write only

values strictly smaller than v b ecause if it wrote a value more than v itwould set x

before R could see x and it cannot write v bythechoice of W Note that if sucha W

in R writes value v completely precedes the read of x then the contained write to x

v v

such that v v and suchthatthe So consider the latest write to a register x

write follows W and completely precedes the read of x in R By the ab ove arguments

there is at least one such writeWenow pro ceed with case analysis

Since R reads the registers in order x x and it returns value vv R must see

is set to Therefore there exists But wehave just identied a place where x x

v v

some write to x which follows the write to x and either precedes or overlaps the

v v

read of v in R This can only happ en if there is an intervening write to some x such

vAsabove in this case wemust have v v If not then b efore doing the that v

write to x the enclosing WRITE would overwrite the in x and so R could not get

v v

it But then this is a contradiction to the choice of the write of v as the latest Note the

write to x completely precedes the read of x in R b ecause the write to x either

v v v

in R and v v precedes or overlaps the read of x

The implementation relationships now collapse as shown in Figure

nreader

k ary atomic

nreader reader

s s

binary atomic k ary atomic

reader

binary atomic

regular

and safe

Figure Collapsed Implementation Relationships

Construction Reader K ary Atomic Register from Reg

ular Register

Wenow showhow to construct a writer reader k ary atomic register from two writer

reader k ary regular registers arranged as in Figure The co de is given in Figure

The cases in the co de are designed to decide whether to return the new or the old value

read Intuitivelyif x num it means that there has b een much progress in the write or

it may b e nished so it is reasonable to return the new value At the same time it is useful

to rememb er that the new value has already b een returned in the newreturned variable

This is b ecause a later read of x that overlaps a write could get an earlier version of the

variable with num b ecause x is a regular register and not an atomic register Notice

that it couldnt get num The second case covers the situation in which the previous

read could have already returned the value of a write that overlaps b oth The conditions lo ok

a little ad hoc Unfortunatelywe cant give a correctness pro of in class For the correctness

pro of we refer the reader to Lamp ort This is not the sort of pro of that can b e done in

anyobvious way using invariants b ecause the underlying registers are not atomic Instead

W WP

R RP

Figure Reader Atomic Register from Regular Registers

Lamp ort develop ed an alternative theory of concurrency based on actions with duration

overlapping or totally preceding each other Another approach to prove the correctness is

to use the Lemma presented in Lecture

The implementation relations now collapse as in Figure This concludes Lamp orts

constructions

MultiReader Atomic Registers

Notice that the constructions thus far still leave a gap b etween the nreader writer atomic

registers and the reader writer atomic registers This gap has b een closed in a couple of

other pap ers SinghAG NewmanW We remark that these algorithms are somewhat

complicated It is also not clear if they are practical since their time complexity is high

SinghAndersonGouda have a reasonable construction related to Lamp orts Construction

Lamp orts Bakery Revisited

Wenow show that the original Lamp ort bakery algorithm presented in class still works if

the underlying registers are only safe The explanation is as follows

A read of the choosing variable while it is b eing written will b e guaranteed to get

either or since these are the only p ossible values But those corresp ond to either

the p oint b efore or after the write step

Ifanumb er is read bya cho oser while it is b eing written the cho oser can get an

arbitrary value When the cho oser takes the maximum of this arbitrary value with

Shared Variables regular registers

x stores tuples of old new num color where old new are from the value do

main of the atomic register and num f g Initially x v v red

y stores values from fred blue g Initially y blue

Co de for write v

newcolor y

oldvalue xnew record value of x lo cally no need to read it

for i f g do

x oldvalue vinewcolor

Co de for read

remember last two reads in x and x

x x

y x color

case

x num

newreturned true

return x new

x num and x num x num and newreturned and x color x color

return x new

otherwise

newreturned false

return x old

end case

Figure Co de for constructing singleread k ary atomic register from regular registers

nreader

k ary atomic

nreader

binary atomic

regular and safe

reader atomic

Figure Collapsed Implementation Relationships

the others the result is that the cho oser is guaranteed to cho ose something bigger

than all the values it sees without overlap but there is no guarantee it will cho ose

something bigger than a concurrent writer However the concurrent writer is cho osing

concurrently and the algorithm didnt need any particular relation b etween values

chosen concurrently

In L supp ose someone reads a numb er while its b eing written If the test succeeds

the writer is back at the p ointofcho osing its value and everything is ne That is

it must b e that the reader at Lsaw the writer b efore it b egin cho osing its number

so the writer is guaranteed to cho ose a larger numb er The problem is if the writers

value comes backasvery small which can delay the reader However it cannot delay

the reader forever

Asynchronous Network Algorithms

Nowwe make another ma jor switchinthemodelwe consider Namelywe switch from asyn

chronous shared memory to asynchronous networksAsinsynchronous networks we assume

a graph or a digraph G with pro cesses at the no des and communication over directed

edges But now instead of synchronous rounds of communication wehaveasynchronyin

the communication as well as the pro cess steps

We mo del each pro cess as an IOA generally with some inputs and outputs connecting it

with the outside world This is the b oundary where the problems will usually b e stated In

addition pro cess i has outputs of the form send m where j is an outgoing neighb or and

m is a message element of some message alphab et M and inputs of the form receive m

for j an incoming neighb or Except for these interface restrictions the pro cess can b e an

arbitrary IOA We sometimes restrict the numb er of classes or the numb er of states etc

to obtain complexity results

To mo del communication wegivea behavior sp ecication for the allowable sequences of

send i j and receive i j actions for eachedgei j This can b e done in twoways bya

list of properties or axiomsorby giving an IOA whose fair b ehaviors are exactly the desired

sequences The advantage of giving an explicit IOA is that in this case the entire system

is describ ed simply as a comp osition and wehave information ab out the state of the entire

system to use in invariants and simulation pro ofs But sometimes its necessary to do some

unnatural programming to sp ecify the desired b ehavior as an IOA esp ecially when wewant

to describ e liveness constraints Let us now make a short digression to dene the imp ortant

notions of safety and liveness

Let S and L b e prop erties ie sets of sequences over a given action alphab et S is a

safety property if the following conditions hold

S is nonempty ie there is some sequence that satises S

S is prexclosed ie if a sequence satises S then all its prexes satisfy S and

S is limitclosed ie if every nite prex of an innite sequence satises S then the

innite sequence satises S

L is a liveness property if for every nite sequence there is some extension that satises

L Informally safety prop erties are appropriate for expressing the idea that nothing bad

ever happ ens if nothing bad has happ ened in a nite sequence then nothing bad has

happ ened in any prex either moreover if something bad happ ens it happ ens at a nite

p oint in time Liveness on the other hand is most often used to express the idea that

something go o d eventually happ ens

Wenow return to the asynchronous message passing mo del The most common commu

nication channel studied in the research literature is a reliable FIFO channelFormally this

is an IOA with the appropriate interface whose state is a queue of messages The send i j

action puts its argument at the end of the queue The receive m action is enabled if m is

at the head of the queue and its eect is to remove the message from the queue Of course

it also delivers the message to the recipient pro cess which should handle it somehow But

that is a part of the pro cess job not of the channels The right division of resp onsibilities

is obtained bytheIOA comp osition denition The fairness partition here can put all the

lo cally controlled actions in a single class

To gain some exp erience with asynchronous network algorithms within this mo del we

go back to the b eginning of the course and reconsider some of the examples we previously

considered in synchronous networks

Leader Election

Recall the algorithms we considered earlier First the graph is a ring with UIDs built into

the initial states of the pro cessors as b efore In the action signature there are no inputs from

the outside world and the only output actions are leader one for each pro cess i

LeLannChang Algorithm

In this algorithm each pro cess sends its ID around the unidirectional ring When a no de

receives an incoming identier it compares that identier to its own if the incoming identier

is greater than its own it keeps passing the identier if it is less than its own then the

identier is discarded and if it is equal it outputs the leader action This idea still works

in an asynchronous system Figure gives the co de for the algorithm

Essentially the b ehavior of the algorithm is the same as in the synchronous but skewed

in time We mightprove it correct by relating it to the synchronous algorithm but the

relationship would not b e a forward simulation since things happ en here in dierent orders

We need a more complex corresp ondence Instead of pursuing that idea well discuss a more

common style of verifying such algorithms namely direct use of invariants and induction

Invariant pro ofs for asynchronous systems work ne The induction is now on individual

pro cesses steps whereas the induction in the synchronous mo del was on global system steps

Here as in the synchronous case wemust showtwo things

that no one except the maximum ever outputs leader and

that a leader output eventually o ccurs

Note that is a safety prop erty asserting the fact that no violation ever o ccurs and

is a liveness prop erty

Safety prop erties can b e proved using invariants Here we use a similar invarianttothe

one we stated for the synchronous case

Invariant If i i and j i i then own i send j

max max

Wewant to prove this invariantby induction again similar to the synchronous case

Recall a homework problem ab out this But now due to the intro duction of the message

channels with their own states we need to add another prop ertytomake the induction go

through

Invariant If i i and j i i then own i queue j j

max max

With these two statements together the induction works much as b efore More sp ecif

icallywe pro ceed by case analysis based on one pro cess at a time p erforming an action

The key step as b efore is where i gets pruned out by pro cess i Thisproves the safety

max

prop erty

Wenowturntotheliveness prop erty namely that a leader action eventually o ccurs

For this prop erty things are quite dierent from the synchronous case Recall that in the

synchronous case weusedavery strong invariant whichcharacterized exactly where the

maximum value had reached after k rounds Now there is no notion of round so the pro of

must b e dierent We cant give such a precise characterization of what happ ens in the

computation since there is nowsomuch more uncertaintySowe need to use a dierent

metho d based on making progress toward a goal Here weshow inductively on k for

k n that eventual l y i winds up in send We then apply this to n

max i k

max

and show that i eventually gets put into the i i channel and then that i

max max max max

message eventually is received at i and therefore eventually leader action gets done

max i

max

All these eventually arguments dep end on using the IOA fairness denition

For example supp ose that there is a state s app earing in some fair execution where

a message m app ears at the head of a send queue in sWewanttoshow that eventually

send mmust o ccur Supp ose not By examination of the allowable transitions we conclude

that m remains at the head of the send queue forever This means that the send class stays

enabled forever By the IOA fairness some send must eventually o ccur But m is the only

one at the head of the queue so the only one for which an action can b e enabled so send m

must eventually o ccur

Likewise if m app ears in the k th p osition in the send queue wecanprove that eventually

send m o ccurs This is proven by induction on k with the base case ab ove The inductive

step says that something in p osition k eventually gets to p osition k and this is based on

the head eventually getting removed

We remark that this typ e of argument can b e formalized within temp oral logic using

statements of the form P Q see Manna and Pnueli

State of pro cess i

own typ e UID initially is UID

send typ e queue of UIDs initially containing only is UID

status takes values from funknown chosen reported g initially unknown

Actions of pro cess i

send m

Precondition m is rst on send

Eect Remove rst elementof send

receive m

Eect case

m own addm to end of send

m own status chosen

otherwise do nothing

endcase

leader

Precondition status chosen

Eect status reported

Fairness partition classes For any pro cess all send actions are in one class and the

leader action in another singleton class

Figure LeLannChang algorithm for leader election in asynchronous rings

JJ Distributed Algorithms Novemb er

Lecturer Nancy Lynch

Lecture

Leader Election in Asynchronous Rings cont

The LeLannChang Algorithm cont

We conclude our discussion of this algorithm with time analysis In the synchronous case

we had an easy way to measure time bycounting the number of synchronous rounds for

the synchronous version of this algorithm we got ab out n In the asynchronous case we

havetousea real time measure similar to what we used for asynchronous shared memory

algorithms

At this p oint we pause and generalize this time measure to arbitrary IOAs This turns

out to b e straightforward thinking of the fairness classes as representing separate tasks

we can just asso ciate a p ositive real upp er b ound with each class This generalizes what

we did for shared memory in a natural way there we had upp er b ounds for each pro cess

and for each user Recall that we could mo del the user as a separate IO automaton In

sp ecializing version to the reliable network mo del we put an upp er b ound of l on eachclass

of each pro cess and an upp er b ound of d on time for the message queue to do a step that

is the time to deliver the rst message in the queue

Using these denitions we pro ceed with the analysis of the LeLannChang algorithm A

naive approach as in the CWI pap er gives an O n b ound This is obtained by using the

progress argument from the last lecture adding the time b ounds in along the way in the

worst case it takesnl nd time for the maximum UID to get from one no de to the next nl

time to send it and then nd to receive it Note that in this analysis we are allowing for the

maximum numb er of messages n to delay the message of interest in all the queues The

overall time complexityisthus O n l d

But it is p ossible to carry out a more rened analysis thus obtaining a b etter upp er

b ound The key idea is that although the queues can grow tosizeatworst n and incur nd

time to deliver everything this cannot happ en everywhere More sp ecically in order for

a long queue to form some messages must have b een traveling faster than the worstcase

upp er b ound Using this intuition we can carry out an analysis that yields an O nl d

upp er b ound on time Informally the idea is to asso ciate a progress measure with each state

giving an upp er b ound on the time from anypoint when the algorithm is in that state until

the leader is rep orted Formallywe prove the following claim

Claim Given a state in an execution of the algorithm consider the distancearound the

ring of a token ie message i from its current position to its home node iLet k be the

maximum of these distances Then within k l dl time a leader event occurs

Applying this claim to the initial state yields an upp er b ound for the algorithm of nl

dl as desired

Pro of By induction on k

Base case k ie the i token has already arrived at no de i in this case within

max max

l time units the leader will o ccur

Inductive step Supp ose the claim holds true for k and prove it holds for k Supp ose

that k is the maximum of the indicated distances Consider any token that is at distance

greater than k By denition of k no token is in distance strictly greater than k This

means that the token is either in the send queue at distance k orinthechannels queue

from distance k to k Wenow argue that this token is not blo cked byany other

token ie that it is the only token in the combination of this send buer or this queue

To see this notice that if another token were there also then that other token would b e at

distance strictly greater than k from its goal a contradiction to the assumption that k is the

maximum suchvalue So within time at most l d the token on whichwe fo cus gets to the

next no de ie to within distance k of its goal and then by the inductivehyp othesis

in additional time k l d l a leader event o ccurs This proves the inductive step

The Hirshb ergSinclai r Algorithm

Recall the Hirshb ergSinclair algorithm which used the idea of twodirectional exploration

as successively doubled distances It is straightforward to see that this algorithm still works

ne in the asynchronous setting with the same time and communication upp er b ounds

O n log n

The Peterson Algorithm

Hirshb erg and Sinclair in their original pap er conjectured that in order to get O n log n

worst case message p erformance a leader election algorithm would have to allow bidirectional

message passing Peterson however constructed an algorithm disproving this conjecture

More sp ecically the assumptions used bythePeterson algorithm are the following

The numb er of no des in the ring is unknown

Unidirectional channels

Asynchronous mo del

Unb ounded identier set

Anynodemay b e elected as the leader not necessarily the maximum

The Peterson algorithm not only has O n log nworst case p erformance but in fact the

constant factor is low it is easy to show anupperboundof n log nandPeterson used

a trickier optimized construction to get a worst case p erformance of n log n The

constant has b een broughteven further down by other researchers

The reason we didnt intro duce this algorithm earlier in the course is that synchrony

do esnt help in this case it seems like a naturally asynchronous algorithm

In Petersons algorithm pro cesses are designated as b eing either in an active mo de or

relay mo de all pro cesses are initially active We can consider the active pro cesses as the

ones doing the real work of the algorithm the pro cesses still participating in the leader

election pro cess The relay pro cesses in contrast just pass messages along

The Peterson algorithm is divided into asynchronously determined phases In each

phase the numb er of active pro cesses is divided at least in half so there are at most log n

phases

In the rst phase of the algorithm each pro cess sends its UID two steps clo ckwise Thus

everyone can compare its own UID to those of its twocounterclo ckwise neighb ors When it

receives the UIDs of its twocounterclo ckwise neighb ors each pro cess checks to see whether

it is in a conguration such that the immediate counterclo ckwise neighb or has the highest

UID of the three ie pro cess i checks if id id and id id If pro cess i nds

i i i i

that this condition is satised it remains active adopting as a temp orary UID the UID

of its immediate counterclo ckwise neighb or ie pro cess iAny pro cess for which the ab ove

condition do es not hold b ecomes a relay for the remainder of the execution The job of a

relay is only to forward messages to active pro cesses

Subsequent phases pro ceed in much the same way among active pro cessors only those

whose immediate active counterclo ckwise neighb or has the highest temp orary UID of the

three will remain active for the next phase A pro cess that remains active after a given phase

will adopt a new temp orary UID for the subsequent phase this new UID will b e that of its

immediate activecounterclo ckwise neighb or from the justcompleted phase The formal co de

for Petersons algorithm is given in Figures and

It is clear that in any given phase there will b e at least one pro cess that nds itself in a

conguration allowing it to remain active unless only one pro cess participates in the phase

State of pro cess i

mode taking values from factive relay g initially active

status taking values from funknown elected reported g initially unknown

id j where j f gtyp e UID or nil initially id is UID and id id

nil

send queue of UIDs initially containing is UID

receive queue of UIDs

Actions of pro cess i

getseconduid

Precondition mode active

receive

id nil

Eect id head receive

remove head of receive

add id to end of send

if id id then status elected

getthirduid

Precondition mode active

receive

id nil

Eect id head receive

remove head of receive

Figure Petersons algorithm for leader election part

Co de for pro cess i cont

advancephase

Precondition mode active

id nil

id max fid id g

Eect id id

id nil

add id to send

becomerelay

Precondition mode active

id nil

id max fid id g

Eect mode relay

relay

Precondition mode relay

receive

Eect move head of receive to tail of send

send message m

Precondition head send m

Eect delete head of send

message m receive

Eect add m to the tail of receive

Figure Petersons algorithm for leader election part

in which case the lone remaining pro cess is declared the winner Moreover at most half the

previously active pro cesses can survivea given phase since for every pro cess that remains

active there is an immediate counterclo ckwise activeneighbor that must go into its relay

state Thus as stated ab ove the numb er of active pro cesses is at least halved in each phase

until only one active pro cess remains

Communication and Time Analysis The total numberofphasesinPetersons algo

rithm is at most blog nc and during each phase each pro cess in the ring sends and receives

exactly two messages This applies to b oth active and relay pro cesses Thus there are

at most n b log nc messages sent in the entire algorithm Note that this a much b etter

constant factor than in the Hirshb ergSinclair algorithm

As for time p erformance one might rst estimate that the algorithm should take O n log n

time since there are log n phases and each phase could involvea chain of message deliver

ies passing through relays of total length O n As it turns out however the algorithm

terminates in O n time Wegive a brief sketch of the time analysis Again wemay ne

glect pileups which seem not to matter for the same reason as in the analysis of LeLann

This allows us to simplify the analysis by assuming that every no de gets a chance to send

a message in b etween every two receives For simplicitywe also assume that lo cal pro cess

ing time is negligible compared to the messagetransmission time dsowe just consider the

messagetransmission time Now our plan is to trace backwards the longest sequential chain

of messagesends that had to b e sent in order to pro duce a leader

Let us denote the eventual winner by P In the nal phase of the algorithm P had to

hear from two activecounterclo ckwise neighbors P and P In the worst case the chain

of messages sent from P toP is actually n in length and P P as depicted in Figure

Now consider the previous phase Wewishtocontinue pursuing the dep endency chain

backwards from P which might b e the same no de as P The key p oint is that for

anytwo consecutive phases it must b e the case that b etween anytwo active pro cesses in

the later phase there is at least one active pro cess in the previous phase Thus at the

nexttolast phase there must have b een an additional pro cess in the interval starting from

P counterclo ckwise to P and another additional pro cess in the interval starting from P

counterclo ckwise to P

Thus the chain of messages pursued backward from P in the nexttolast phase from

P and P can at worst only extend as far as P ie in the worst case P P as

depicted in Figure And there is an additional pro cess Qbetween P andP

At the phase preceding the nexttolast phase P waits to hear from P and P where

P isatworst equal to Q also there is an additional pro cess Qbetween Q and P see P0=P2

Figure The last phase of the Peterson algorithm P is the winner

P3 Q1

P1=P4

Figure The nexttolast phase of the Peterson algorithm

Figure At the phase b efore this P waits to hear from P andP etc This analysis

can go on but wenever get back around to P and so the total length of the dep endency

chain is at worst nWe conclude therefore that the time is at most around nd

Burns Lower Bound Result

All the algorithms in the previous section are designed to minimizethenumb er of messages

and the b est achieve O n log n communication Recall that in the synchronous case we had

a matching n log nlower b ound under the assumption that the algorithms are comparison

based That carries over to the asynchronous mo del since the synchronous mo del can b e

formulated as a sp ecial case of the asynchronous mo del Note that the algorithms to

which this lower b ound applies are not allowed to use the UIDs in arbitrary ways eg for

counting As wesaw if we lift this restriction then we can get O n message algorithms in P0

P3 P6=Q1

Figure The next preceding phase of the Peterson algorithm

the synchronous mo del

It turns out however that the asynchronous network mo del has enough extra uncertainty

so that the n log nlower b ound applies regardless of how the identiers are used The

pro of of this fact is completely dierent pro of from the synchronous case now the asynchrony

is used heavilyWe consider leader election algorithms with the following prop erties

The numb er of no des in the ring is unknown

Bidirectional channels

Asynchronous mo del

Unb ounded identier set

Anynodemay b e elected as leader

For this setting Burns proved the following result

Theorem Any leader election algorithm with the properties listedabove sends at least

n log n messages in the worst case where n is the number of processes in the ring

For simplicitywe assume that n is a p ower of The pro of can b e extended to arbitrary

n cf homework We mo del each pro cess as an IO automaton and stipulate that each

automaton is distinct in essence that each pro cess has a unique identier The automaton

can b e represented as in Figure Each pro cess has two output actions sendright and

sendleftandtwo input actions receiveright and receiveleft

Our job will ultimately b e to see how a collection of automata of this typ e b ehavewhen

arranged into a ring however in the course of this exploration wewould also liketosee

how the automata b ehave when arranged not in a ring but simply in a straight line as in

receiveleft sendright

sendleft receiveright

Figure A pro cess participating in a leader election algorithm

Figure A line of leaderelecting automata

Figure Formallywecansay that a line is a linear comp osition using IO automaton

comp osition of distinct automata chosen from the universal set of automata

The executions of such a line of automata can b e examined in isolation where the

two terminal automata receive no input messages in this case the line simply op erates on

its own Alternativelywe mightcho ose to examine the executions of the line when certain

input messages are provided to the two terminal automata

As an added bit of notation wewillsaythattwo lines of automata are compatible when

they contain no common automaton b etween them Wewillalsodenea join op eration on

two compatible lines which simply concatenates the lines this op eration interp oses a new

message queue to connect the rightmost receiveright message of the rst line with the

leftmost sendleft message of the second and likewise to connect the leftmost receiveleft

message of the second line with the rightmost sendright message of the rst and then uses

ordinary IOA comp osition Finallythe ring op eration on a single line interp oses new queues

to connect the rightmost sendright and leftmost receiveleft actions of the line and the

rightmost receiveright and leftmost sendleft actions The ring and join op erations are depicted graphically in Figure

join(L,M)

ring(L)

Figure The join and ring op erations

We pro ceed with a pro of that n log n messages are required to elect a leader in a bidi

rectional asynchronous ring where n is unknown to the pro cesses and pro cess identiers are

unb ounded For a system S line or ring and an execution of S we dene the following

notions

COMM S is the numb er of messages sent in execution of system S

COMM S sup COMM S Here we consider the numb er of messages sent

during any execution of S For lines we only consider executions in which no messages

come in from the ends

A state q of a ring is quiescent if there is no execution fragment starting from q in

whichany new message is sent

A state q of a line is quiescent if there is no execution fragment starting from q in which

no messages arrive on the incoming links at the ends and in whichany new message

is sent

Wenow state and proveourkey lemma

Lemma For every i there is an innite set of disjoint lines L such that for al l L

i i

L j L j and COMM L i

Pro of By induction on i

Base case For i we need an innite set of dierent pro cesses such that eachcan

send at least message without rst receiving one Supp ose for contradiction that there

are pro cesses p and q such that neither can send a message without rst receiving one

Consider rings R R and R as shown in Figure

R1 R2 R3

p q q

Figure Basis for pro of of Lemma

In all three rings no messages are ever sent so each pro cess pro ceeds indep endently

Since R solves election p must elect itself and similarly for Randq Then R elects

two leaders a contradiction It follows that there is at most one pro cess that cant send a

message b efore receiving one If there is an innite numb er of pro cesses removing one leaves

an innite set of pro cesses that will send a message without rst receiving one Let L be

this set whichproves the basis

Inductive step Assume the lemma is true for i Let n LetL M N be any

lines from L Consider all p ossible combinations of two of these lines into a line of double

size LM LN ML NL MN and NM Since innitely many disjoint sets of three lines

can b e chosen from L the following claim implies the lemma

log n messages Claim Atleast one of the lines can be made to send at least

Pro of Assume that the claim is false By the inductivehyp othesis there exists a nite

n n

execution of L for which COMM L log and in which no messages arrive

L L

from the ends

We can assume without loss of generality that the nal conguration of is quiescent

since otherwise can b e extended to generate more messages until the number log n of

messages is reached We can assume the same condition for and by similar reasoning

M N

L M

can know ab out join

Figure join L M

Now consider anytwo of the lines say L and M Consider join L M Consider an execution

that starts by running on L and on M but delays messages over the b oundary

L M

n n

log messages Nowdeliver the delayed In this execution there are at least

messages over the b oundary By the assumption that the claim is false the entire line must

additional messages messages for otherwise the total quiesce without sending more than

n n n n n

exceeds log log n This means that at most pro cesses in join L M

know ab out the join and these are contiguous and adjacent to the b oundary as shown in

Figure These pro cesses extend at most halfwayinto either segment Let us call this

execution Similarly for etc

LM LN

R1 p1

L N

Figure ring join L M N case

In ring R of Figure consider an execution in which and o ccur rst

L M N

quiescing in three pieces separately Then quiesce around b oundaries as in and

LM LN

Since the processes that know about each join extend at most half way into either

segment these messages wil l be noninterfering Similarly for R

EachofRandR elects a leader say p and p We can assume without loss of generality

that p is b etween the midp ointof L and the midp ointof M as in Figure We consider

cases based on the p osition of p in R Figure to get a contradiction

If p is b etween the midp ointof L and the midp ointof N as in Figure then let

R b e assembled by joining just M and N Consider an execution of Rinwhich the two N

p2 R2

L M

Figure ring join L N M

segments are rst run to quiescence using and and then quiescence o ccurs around

M N

the b oundaries as in and

MN NM

p3 N

Figure ring join M N leader elected in the lower half

p2 R2 p3

L M

Figure ring join L N M

By the problem denition a leader must b e elected in R say p First supp ose it is

in lower half as in Figure Then it also o ccurs in R and gets elected there to o as in

Figure There are two leaders in this case which is a contradiction If it is in the upp er

half of R then we arrive at a similar contradiction in R

If p is b etween the midp ointof L and the midp ointof M thenwearrive at a similar

contradiction based on R again

L N

Figure ring join L N

If p is b etween the midp ointof M and the midp ointof N thenwe arrive at a similar

contradiction based on R a ring containing LN as in Figure

The lemma follows from the claim

Lemma essentially proves Theorem let n be a power of Pick a line L of length

n with COMM L log n and paste it into a circle using the ring op erator Let the

pro cesses in L b ehave exactly as they would if they were not connected in the execution

that sends the large numb er of messages and delay all messages across the pasted b oundary

until all the large numberofmessageshave b een sent This gives us the desired b ound

Note the crucial part played by the asynchronyinthisargument

Problems in General Asynchronous Networks

So far wehave revisited several synchronous ring leader election algorithms from the b egin

ning of the course and have seen that they extend directly to the asynchronous setting Now

we will reexamine the algorithms from more general synchronous networks In the sequel we

shall assume that the underlying communication network is mo deled by a general undirected

graph

Network Leader Election

The synchronous algorithm wesaw earlier for leader election in a general graph do es not

extend directly to the asynchronous mo del In the synchronous algorithm every no de main

tains a record of the maximum UID it has seen so far initially its own Ateach round the

no de propagates this maximum on all its incident edges The algorithm terminates after a

numb er of rounds equal to the diameter if the pro cess has its own ID as its known maximum

then it announces itself as the leader

In the asynchronous setting we dont have rounds and we dont haveany information

ab out time that would allowustowait for enough time to hear from everyone and so the

ab ove idea do esnt extend We could still follow the basic strategy of propagating maximum

UIDs asynchronously whenever a pro cess gets a new maximum it propagates it sometime

later The problem is nowthatwe dont knowwhentostopToovercome this problem we

shall use other techniques

Breadthrst Search and Shortest Paths

Reconsider the shortest paths algorithms Recall that breadthrst search is a sp ecial case

of shortest paths when all edges haveweights In these algorithms we assume that we

are given an initiator no de i and wewant that eachnodeeventually outputs a p ointer to

a parent in a shortestpaths or breadthrst tree We can also output the distance on the

shortest path

The algorithms wehave presented earlier used the synchronyvery heavily Eg for

breadthrst search we marked no des in layers one layer for eachsynchronous round This

was very ecient O E total messages and time prop ortional to the diameter

We cannot run that algorithm directly in an asynchronous network since dierent branches

of the tree can grow more quickly than others More sp ecically this means that a no de can

get some information ab out one path rst then later nd out ab out a shorter path Thus

we need to b e able to adjust estimates downward when new information arrives Recall that

the synchronous shortestpaths algorithm wesaw earlier already did this kind of adjustment

We called it a relaxation step

This adjustment idea is used in a famous shortestpaths and BFS algorithm that imi

tates the BellmanFord algorithm from sequential complexity theory In fact this algorithm

was the routing algorithm used in the ARPANET b etween and Note that this

will not b e a terminating algorithm the no des will not know when they are done We give

the sp ecication of this algorithm in preconditioneects style in Figure

State variables

bestdist for i this is initially for others

parent initially undened

newsend for i initially neighbors i for others initially

says whom they should send a newlydiscovered distance to

Signature We assume that the weights are known a priori so no inputs are needed from

the outside world We need send and receive actions as usual the only message is a

distance value nonnegative real There are no outputs since not known when this

terminates

Actions

send m

Precondition j newsend

m bestdist

Eect newsend newsend fj g

receive m

Eect if m weight j i bestdist then

bestdist m weight j i

newsend neighbors i fj g

parent j

Figure BellmanFord algorithm for shortest paths

Analysis If the algorithm were executed in synchronous rounds then the time would b e

O n where n is the total number of nodes and the numb er of messages would b e O njE j

This do es not seem to o bad But in asynchronous execution the time b ound can b e much

worse in fact we show b elow that the time and message complexity are exponential in n

Claim The algorithm sends messages in the worst case

Pro of Consider the example illustrated in Figure

2^k 2^(k−1) 2 1

i000 i1 i2 00 ik i(k+1) i(k+2)

Figure Bad scenario for the BellmanFord Algorithm

The p ossible estimates that i can have for its bestdist are More

over we claim that it is p ossible for i to acquire all of these estimates in order from

the largest to the smallest To see how consider the following scenario Supp ose that the

messages on the upp er paths all propagate rst giving i the estimate Next the

alternative message from i arrives giving i the adjusted estimate of Next

k k

the alternative message from i to i arrives at i cause i to reduce its estimate by

k k k k

It then propagates this revision on b oth paths Again supp ose the message on the upp er

path arrives rst etc

Since i gets distinct estimates it is p ossible if i takes steps fast enoughfor

k k

i to actually put all these values in the outgoing channel to i This yields exp onentially

k k

many messages actually Moreover the time complexity is similarly bad b ecause

these messages all get delivered sequentially to i

Wenow consider upp er b ounds on the BellmanFord algorithm First we claim that

the numb er of messages is O n e at most This is b ecause each no de only sends

new messages out when it gets a new estimate Now each estimate is the total weightof

a simple path to that no de from i and the b ound follows from the fact that there are at

most n such paths This may b e a lo ose estimate Now consider any particular edge

It only gets messages when one of its endp oints gets a new estimate leading to a total of

n messages on that edge

JJ Distributed Algorithms Novemb er

Lecturer Nancy Lynch

Lecture

Asynchronous BroadcastConvergecast

In an earlier lecture we describ ed a synchronous discipline for broadcasting and converge

casting sometimes called broadcastwithecho using a breadthrst spanning tree In the

asynchronous setting the breadthrst spanning tree is not as usable since we dont know

when the construction is complete We can still do broadcastconvergecast reasonably e

ciently on an arbitrary not necessarily breadthrst spanning tree

First we consider how to construct an arbitrary spanning tree The distinguished source

no de i b egins by sending out report messages to all its neighb ors Any other no de up on

receipt of its rst report message marks itself as doneidenties the sender j as its parent

sends a parent j announcement to the outside world and sends out report messages on

its outgoing links The no de ignores additional report messages This idea works ne asyn

chronously no des know when they are done O E messages are used and it takes O diam

time until termination There are no pileups on links since only two messages ever get sent

on each edge one in each direction

Nowwe pro ceed to the task of broadcasting a message to all no des We can construct an

arbitrary spanning tree as ab ove and then i uses it to send messages over the tree links

To do this no des need to b e informed as to which other no des are their children and this

requires lo cal communication from children to parents up on discovery Note the interesting

timing anomaly here tree paths could takentimetotraverse the second time Toavoid

that we can send the message in the course of building the tree by piggybacking the

message on all report messages

We can also use an asynchronouslyconstructed spanning tree to fan in results of a

computation backto i as in the synchronous case eachnodewaits to hear from all its

children in the spanning tree and then forwards a message to its parent

This brings us to the task of broadcast and convergecast Now supp ose wewantto

broadcast a message and collect acknowledgments backat i thateveryone has received it

To solve this problem we do a combination of the asynchronous spanning tree construction

ab ove followed asynchronously by fanning in results from the leaves

In fact if the tree is b eing constructed solely for the purp ose of sending and getting

acks for a particular message it is p ossible to delete the tree information after fanning in

State

msg forthevalue b eing broadcast at i initially the message it is sending elsewhere

nil

send buer for outgoing messages at i initially contains bcast m to all neighb ors of

i elsewhere

parent p ointer for the parent edge initially nil

acked a set to keep trackofchildren that haveacked everywhere initially

done Bo olean ag initially false

Actions

send action as usual just sends anything in send buer

receive bcast m

Eect if msg nil then

msg m

parent j

for all k neighbors fj g put message bcast min send

acked

else put message ack to j in send

receive ack

Eect acked acked fj g

nish for i i

Precondition acked neighbors fparent g

done false

Eect put ack message to parent in send

done true

nish for i i

Precondition acked neighbors

done false

Eect output done message to outside world

done true

Figure Co de for the broadcastconvergecast task without deleting the tree

the results back to the parent However wehave to b e careful that the no de do esnt forget

everything since if it later gets a report message from another part of the tree whose

pro cessing is delayed it should know enough to ignore it In other words we need to keep

some kind of ag at eachnodesaying that its done The co de is given in Figure

Analysis Communication is simple it takes O E messages to set up the tree and after

the tree is set up its only O n messages to do the broadcastconvergecast work The time

analysis is trickier One might think that in O diam time the algorithm terminates since

thats howlongittakes to do the broadcast The convergecast however can take longer

Sp ecicallywehave the following timing anomaly a message can travel fast along a long

network path causing a no de to b ecome attached to the spanning tree on a long branch

even though there is a shorter path to it When the resp onse travels back along the long

path it maynow take time prop ortional to the length of the path whichisn for general

graphs To x this problem we shall need new ideas suchasa synchronizerwhichwe shall

discuss next lecture

Example application to leader election The broadcastconvergecast algorithm pro

vides an easy solution to the leader election problem Recall that the simple leader election

algorithm we mentioned ab ove didnt terminate Now we could do an alternative leader

election algorithm as follows Starting from every no de run broadcastconvergecast in par

allel while collecting the maximum ID in the convergecast on each tree The no de that nds

that the maximum is equal to its own UID gets elected

Minimum Spanning Tree

Problem Statement

Nowwe return to the problem of constructing a minimumweight spanning tree MST this

time in an asynchronous network Let us recall the problem denition We are given an

undirected graph G V E with weighted edges suchthateachvertex is asso ciated with

its own pro cess and pro cesses are able to communicate with each other via messages sent

over edges We wish to have the pro cesses vertices co op erate to construct a minimum

weight spanning tree for the graph G That is wewant to construct a tree covering the

vertices in Gwhosetotaledgeweight is less than or equal to that of every other spanning

tree for G

We assume that pro cesses have unique identiers and that each edge of the graph is

asso ciated with a unique weight known to the vertices on each side of that edge The

assumption of unique weights on edges is not a strong one given that pro cesses have unique

identiers if edges had nondistinct weights we could derive virtual weights for all edges

by app ending the identier numb ers of the end p oints onto the edge weights and use them

as tiebreakers b etween the original weights We will also assume that a pro cess do es not

know the overall top ology of the graph only the weights of its incident edges and that

it can learn nonlo cal information only by sending messages to other pro cesses over those

edges The output of the algorithm will b e a marked set of tree edges every pro cess will

mark those edges adjacent to it that are in the nal MST

There is one signicant piece of input to this algorithm namelythatanynumber of nodes

will b e awakened from the outside to b egin computing the spanning tree A pro cess can

be awakened either by the outside world asking that the pro cess b egin the spanning tree

computation or by another alreadyawakened pro cess during the course of the algorithm

The motivation for the MST problem comes mainly from the area of communications

The weights of edges might b e regarded as messagesending costs over the links b etween

pro cessess In this case if wewant to broadcast a message to every pro cess wewould use

the MST to get the message to every pro cess in the graph at minimum cost

Connections Between MST and Other Problems

The MST problem has strong connections to two other problems that of nding any undi

rected unro oted spanning tree at all for a graph and that of electing a leader in a graph

If we are given an undirected unro oted spanning tree it is pretty easy to elect a leader

this pro ceeds via a convergecast of messages from the leaves of the tree until the incoming

messages converge at some particular no de which can then b e designated as the leader It

is p ossible that the messages converge at some edge rather than some no de in which case

one of the endp oints of this edge say the one with the larger ID can b e chosen

Converselyifwe are given a leader it is easy to nd an arbitrary spanning tree as

discussed ab ove under broadcastconvergecast the leader just broadcasts messages along

each of its neighb oring edges and no des designate as their parent in the tree that no de

from which they rst receive an incoming message after which the no des then broadcast

their own messages along their remaining neighb oring edges So mo dulo the costs of these

basic algorithms the problems of leader election and nding an arbitrary spanning tree are

equivalent

A minimum spanning tree is of course a spanning tree The converse problem going from

an arbitrary spanning tree or a leader to a minimum spanning tree is much harder

One p ossible idea would b e to haveevery no de send information regarding its surrounding

edges to the leader which then computes the MST centrally and distributes the information

backtoevery other no de in the graph This centralized strategy requires a considerable

amount of lo cal computation and a fairly large amountofcommunication

The Synchronous Algorithm Review

Let us recall how the synchronous algorithm worked It was based on two basic prop erties

of MSTs

Prop erty Let G beanundirectedgraph with vertices V and weightededges E Let

V E i k be any spanning forest for G with k Fix any i i k Let e

i i

beanedge of lowest cost in in the set fe e E E and exactly one endpoint of e is

E and this treeisofas in V g Then thereisaspanning treeforG that includes feg

j i

low a cost as any spanning tree for G that includes E

Prop erty If al l edges of a connectedgraph have distinct weights then the MST is unique

These prop erties justify a basic strategy in which the MST is built up in fragments as

follows Atanypoint in the algorithm each of a collection of fragments may indep endently

and concurrently nd its own minimumweight outgoing edge MWOE knowing that all

such edges found must b e included in the unique MST Note that if the edge weights were

not distinct the fragments couldnt carry out this choice indep endently since it would b e

p ossible for them to form a cycle

This was the basis of the synchronous algorithm It worked in synchronous levels where

in each level al l fragments found their MWOEs and combined using these to form at most

half as many fragments

If we try to run the given synchronous algorithm in an asynchronous network wesee

some problems

Diculty In the synchronous case when a no de queries a neighb or to see if it is in the

same fragment it knows that the neighb or no de is uptodate at the same level Therefore

if the neighb or has a distinct fragment id then this fact implies that the neighbor is not in

the same fragment But in the asynchronous setting it could b e the case that the neighbor

has not yet heard that it is in the same fragment

Diculty The synchronous algorithm achieves a message cost of O n log n E based

on a balanced construction of the fragments Asynchronously there is danger of construct

ing the fragments in an unbalanced way leading to many more messages The number of

messages sentby a fragment to nd its MWOE will b e prop ortional to the number of nodes

in the fragment Under certain circumstances one might imagine the algorithm pro ceeding

byhaving one large fragment that picks up a single no de at a time each time requiring f

messages where f is the numb er of no des in the fragment see Figure In sucha

situation the algorithm would require n messages to b e sentoverall

Figure Howdoweavoid a big fragmentgrowing by one no de at a time

The GallagerHumbletSpi ra Algorithm Outline

These diculties lead us to rethink the algorithm to mo dify it for use in an asynchronous

network Luckily the basic ideas are the same The algorithm we shall see b elowisby

GallagerHumbletSpira They fo cus on keeping the numb er of messages as small as p ossible

and manage to achieve the same O n log nE message b ound as in the synchronous

algorithm

Before we continue let us make a few preliminary remarks First consider the question

whether this is the minimum b ound p ossible Note that at least for some graphs eg rings

the n log n term is necessary it comes from the lower b ound on the numb er of messages

for leader election that wesaw in Burns theorem What ab out the E term This seems

essential In a work byAwerbuchGoldreichPelegVainish they show that the number of

bits of communication must b e at least E log n If we assume that each message contains

only a constantnumb er of ids then this means that the numb er of messages is E If

the messages are allowed to b e of arbitrary size however then it can b e shown that O n

messages suce

Another thing wewould liketosay ab out the GallagerHumbletSpira algorithm is that

not only is it interesting but it is also extremely clever as presented in their pap er it is

ab out two pages of tight mo dular co de and there is a go o d reason for just ab out every line

in the algorithm In fact only one or twotiny optimizations have b een advanced over the

original algorithm The algorithm has b een proven correct via some rather dicult formal

pro ofs see WelchLL and it has b een referenced and elab orated up on quite often in

subsequent research It has b ecome a sort of test case for algorithm pro of techniques

As in the synchronous algorithm wesaw earlier the central idea of the GallagerHumblet

Spira algorithm is that no des form themselves into collections ie fragments of increas

ing size Initially all no des are considered to b e in singleton fragments Eachfragment

is itself connected by edges that form a MST for the no des in the fragment Within any

fragment no des co op erate in a distributed algorithm to nd the MWOE for the entire frag

ment that is the minimum weight edge that leads to a no de outside the fragment The

strategy for accomplishing this involves broadcasting over the edges of the fragment asking

each no de separately for its own MWOE leading outside the fragment Once all these edges

have b een found the minimal edge among them will b e selected as an edge to include in the

eventual MST

Once an MWOE for a fragment is found a message maybesent out over that edge to

the fragment on the other side The two fragments may then combine into a new larger

fragment The new fragment then nds its own MWOE and the entire pro cess is rep eated

until all the no des in the graph havecombined themselvesinto one giant fragment whose

edges are the nal MST

This is not the whole story of course there are still some problems to overcome First

how does a node know which of its edges lead outside its current fragment Anodein

fragment F can communicate over an outgoing edge but the no de at the other end needs

some way of telling whether it to o is in F We will therefore need some way of naming

fragments so that two no des can determine whether they are in the same fragment But the

issue is still more complicated it may b e for example that the other no de at the end of the

apparently outgoing edge is in F but hasnt learned this fact yet b ecause of communication

delays Thus some sort of overall synchronization pro cess is neededsome sort of strategy

that ensures that no des wont search for outgoing edges until all no des in the fragmenthave

b een informed of their current fragment

And there is also the second problem discussed ab ove of the unbalanced merging b e

havior causing excessive message cost This second problem should suggest a balancedtree

algorithm solution that is the diculty derives from the merging of data structures that

are very unequal in size The strategy that we will use therefore is to merge fragments of

roughly equal size Intuitivelyifwe can keep merging fragments of nearly equal size wecan

keep the numb er of total messages to O n log n

The trickwe will use to keep the fragments at similar sizes is to asso ciate a level number

with each fragment We will ensure that if level F l for a given fragment F then the

numb er of no des in F is greater than or equal to Initially all fragments are just singleton

no des at level When two fragments at level l are merged together the result is a new

fragment at level l This preserves the condition for level numb ers if two fragments of

l l

size at least are merged the result is a new fragment of size at least

So far this may lo ok similar to the way the level numb ers were used in the synchronous

algorithm but it will actually b e somewhat dierent Eg in the synchronous case we could

merge some arbitrary numb er of level l fragments to get a new level l fragment

The level numb ers as it turns out will not only b e useful in keeping things balanced

but they will also provide some identierlike information helping to tell no des whether they

are in the same fragment

In More Detail

There are twoways of combining fragments

Merging This combining rule is applied when wehavetwo fragments F and F with

the same level and the same minimumweight outgoing edge

level F level F l

MWOEF MWOEF

The result of a merge is a new fragmentata level of l

Absorbing There is another case to consider It might b e that some no des are forming

into huge fragments via merging but isolated no des or small fragments are lagging

b ehind at a low level In this case the small fragments may b e absorb ed into the

larger ones without determining the MWOE of the large fragment Sp ecically the

rule for absorbing is that if two fragments F and F satisfy level F level F and

the MWOEF leads to F then F can b e absorb ed into F by combining them along

MWOEF The larger fragment formed is still at the level of F In a sense wedont

want to think of this as a new fragment but rather as an augmented version of F

These two combining strategies are illustrated in a rough way by Figure It is

worth stressing the fact that levelF levelF do es not imply that fragment F is smaller

than F in fact it could b e larger Thus the illustration of F as a small fragmentin

Figure is meant only to suggest the typical case

Level numb ers also serve as mentioned ab ove as identifying information for fragments

For fragments of level or greater however the sp ecic fragmentidentier is the coreedge

of the fragment The core edge is the edge along which the merge op eration resulting in the

current fragmentlevel to ok place Since level numb ers for fragments are only incremented

by merge op erations any fragmentoflevel or greater must have had its level number

incremented by some previous merge along an edge The core edge also serves as the site

where the pro cessing for the fragment originates and where information from the no des of

the fragment is collected

To summarize the wayinwhichcoreedgesareidentied for fragments

For a merge op eration cor e is the common MWOE of the twocombining fragments

For an absorb op eration cor e is the core edge of the fragment with the larger level

numb er

Figure Two fragments combine by merging a fragment absorbs itself into another

Note that identifying a fragmentby its core edge dep ends on the assumption that all

edges have unique identiers Since we are assuming that the edges have unique weights

the weights could b e the identiers

Wenowwanttoshow that this strategyofhaving fragments merge together and absorb

themselves into larger fragments will in fact suce to combine all fragments into a MST for

the entire graph

Claim If we start from an initial situation in which each fragment consists of a single

node and we apply any possible sequence of merge and absorb steps then thereisalways

some applicable step to take until the result is a single fragment containing al l the nodes

Pro of Wewanttoshow that no matter what conguration we arrive at in the course of

the algorithm there is always some merge or absorb step that can b e taken

One way to see that this is true is to lo ok at all the current fragments at some stage

in the running algorithm Each of these fragments will identify its MWOE leading to some

other fragment If we view the fragments as vertices in a fragmentgraph and draw the

MWOE for eachfragment we get a directed graph with an equal number of vertices and

edges see Figure Since wehave k no des and k edges for some k such a directed

graph must have a cycle and b ecause the edges have distinct weights only cycles of size

ie cycles involving two fragments may exist Such a cycle represents two fragments

that share a single MWOE

Now it must b e the case that the two fragments in any cycle can b e combined If the

two fragments in the cycle have the same level numb er a merge op eration can take place

otherwise the fragment with the smaller level numb er can absorb itself into the fragment

Figure A collection of fragments and their minimumweight outgoing edges

with the larger one

Lets return to the question of how the MWOE is found for a given fragment The basic

strategy is as follows Each no de in the fragment is nds its own MWOE leading outside

the fragment then the information from each no de is collected at a selected pro cess which

takes the minimum of all the edges suggested by the individual no des

This seems straightforward but it reop ens the question of howanodeknows that a

given edge is outgoingthat is that the no de at the other end of the edge lies outside the

current fragment Supp ose wehavea node p that lo oks across an edge e to a no de q at

the other end How can p knowifq is in a dierent fragment or not

A fragment name or identier may b e thoughtofasapaircore level If q s fragment

name is the same as ps then p certainly knows that q is in the same fragment as itself

However if q s fragment name is dierentfromthatof p then it is still p ossible that q and

p are indeed in the same fragment but that q has not yet b een informed of that fact That

is to say q s information regarding its own current fragmentmay b e out of date

However there is an imp ortant fact to note if q s fragment name has a core unequal to

that of p and it has a level value at least as high as pthenq cant b e in the fragmentthat

p is in currently and never will b e This is so b ecause in the course of the algorithm a no de

will only b e in one fragmentatany particular level Thus wehave a general rule that q

can use in telling p whether b oth are in the same fragment if the value of core level for

q is the same as that of p then they are in the same fragment and if the value for cor e is

dierent for q and the value of level is at least as large as that of p then they are in dierent

fragments

The upshot of this is that MWOEp can b e determined only if level q level p If q

has a lower level than p it simply delays answering p until its own level is at least as great

as ps

q q

p q

Figure Fragment F absorbs itself into F while F is still searching for its own MWOE

However notice that wenowhave to reconsider the progress argument since this extra

precondition may cause progress to b e blo cked Since a fragment can b e delayed in nding

its MWOE since some individual no des within the fragment are b eing delayed we might

ask whether it is p ossible for the algorithm to reach a state in which neither a merge or

absorb op eration is p ossible To see that this is not the case we use essentially the same

argument as b efore but this time we only consider those MWOEs found by fragments

with the lowest level in the graph call this level l These succeed in all their individual

MWOEp calculations so succeed in determining their MWOE for the entire fragment

Then the argumentisasbeforeIfany fragmentatlevel l nds a MWOE to a higherlevel

fragment then an absorb op eration is p ossible and if every fragment at the level l has a

MWOE to some other fragment at level l thenwemust have a cycle b etween two fragments

at level l and a merge op eration is p ossible So again even with the new preconditions we

conclude that the algorithm must make progress until the complete MST is found

Getting back to the algorithm each fragment F will nd its overall MWOE by taking a

minimum of the MWOE for each no de in the fragment This will b e done by a broadcast

convergecast algorithm starting from the core emanating outward and then collecting all

information back at the core

This leads to yet another question what happ ens if a small fragment F gets absorb ed

into a larger one F while F is still in the course of lo oking for its own MWOE

There are two cases to consider consult Figure for the lab eling of no des Supp ose

rst that the minimum edge leading outside the fragment F has not yet b een determined

In this case we searchfora MWOE for the newlyaugmented fragment F in F as well there

is no reason it cannot b e there

has already b een found at the time that F On the other hand supp ose MWOEF

absorbs itself into F Inthatevent the MWOE for q cannot p ossibly b e e since the only

way that the MWOE for q could b e known is for that edge to lead to a fragment with a level

at least as great as F and we know that the level of F is smaller than that of F Moreover

the fact that the MWOE for q is not e implies that the MWOE for the entire fragment

F cannot p ossibly b e in F This is true b ecause e is the MWOE for fragment F and

thus there can b e no edges leading out of F with a smaller cost than the alreadydiscovered

MWOE for no de q Thus we conclude that if MWOEF is already known at the time the

absorb op eration takes place then fragment F neednt lo ok for its overall MWOE among

the newlyabsorb ed no des This is fortunate since if F did in fact have to lo ok for its

MWOE among the new no des it could b e to o late by the time the absorb op eration takes

place q mighthave already rep orted its own MWOE and fragment F might already b e

deciding on an overall MWOE without knowing ab out the newlyabsorb ed no des

A Summary of the Co de in the GHS Algorithm

Wehave describ ed intuitively the ma jor ideas of the GallagerHumbletSpira algorithm this

explanation ab ove should b e sucient to guide the reader through the co de presented in

their original pap er

Although the actual co de in the pap er is dense and complicated the p ossibilityofan

understandable highlevel description turns out to b e fairly typical for communications al

gorithms In fact the highlevel description that wehave seen can serve as a basis for a

correctness pro of for the algorithm using simulations

The following message typ es are employed in the actual co de

INITIATE messages are broadcast outward on the edges of a fragment to tell no des to

start nding their MWOE

REPORT messages are the messages that carry the MWOE information back in

These are the convergecast resp onse to the INITIATE broadcast messages

TEST messages are sentoutby no des when they search for their own MWOE

ACCEPT and REJECT messages are sent in resp onse to TEST messages from no des

they inform the testing no de whether the resp onding no de is in a dierentfragment

ACCEPT or is in the same fragment REJECT

CHANGEROOT is a message senttoward a fragments MWOE once that edge is

found The purp ose of this message is to change the ro ot of the merging or currently

b eingabsorb ed fragment to the appropriate new ro ot

CONNECT messages are sent across an edge when a fragment combines with another

In the case of a merge op eration CONNECT messages are sentbothways along the

edge b etween the merging fragments in the case of an absorb op eration a CONNECT

message is sentby the smaller fragment along its MWOE toward the larger frag

ment

In a bit more detail INITIATE messages emanate outward from the designated core

edge to all the no des of the fragment these INITIATE messages not only signal the no des

to lo ok for their own MWOEifthatedgehasnotyet b een found but they also carry

information ab out the fragment identity the core edge and level numb er of the fragment

As for the TESTACCEPTREJECT proto col theres a little b o okkeeping that no des have

to do Every no de in order to avoid sending out redundant messages testing and retesting

edges keeps a list of its incident edges in the order of weights The no des classify these

incident edges in one of three categories

Branch edges are those edges designated as part of the building spanning tree

Basic edges are those edges that the no de do esnt knowanything ab out yet they

mayyet end up in the spanning tree Initially of course all the no des edges are

classied as basic

Rejected edges are edges that cannot b e in the spanning tree ie they lead to another

no de within the same fragment

A fragmentnodesearching for its MWOE need only send messages along basic edges

The no de tries each basic edge in order lowest weight to highest The proto col that the

no de follows is to send a TEST message with the fragmentlevelnumb er and coreedge

represented by the unique weight of the core edge The recipient of the TEST message

then checks if its own identity is the same as the TESTer if so it sends back a REJECT

message If the recipients identity core edge is dierent and its level is greater than or

equal to that of the TESTer it sends backanACCEPT message Finally if the recipient

has a dierent identity from the TESTerbuthasalower level numb er it delays resp onding

until such time as it can send back a denite REJECT or ACCEPT

So each no de nds the MWOE if it exists All of this information is sentback to the

no des incident on the core edge via REPORT messages who determine the MWOE for the

entire fragment A CHANGEROOT message is then sentbacktowards the MWOE and the

endp oint no de sends a CONNECT message out over the MWOE

When two CONNECT messages cross this is the signal that a merge op eration is taking

place In this event a new INITIATE broadcast emanates from the new core edge and the

newlyformed fragment b egins once more to lo ok for its overall MWOE If an absorbing

CONNECT o ccurs from a lowerlevel to a higherlevel fragment then the no de in the high

level fragment knows whether it has found its own MWOE and thus whether to send back

an INITIATE message to b e broadcast in the lowerlevel fragment

Complexity Analysis

Message complexity The analysis is similar to the synchronous case giving the same b ound

of O n log n E More sp ecically in order to analyze the message complexity of the GHS

algorithm we app ortion the messages into two dierent sets resulting separately as we will

see in the O n log n term and the O E term

The O E term arises from the fact that eachedgeinthegraphmust b e tested at least

once in particular weknow that TEST messages and asso ciated REJECT messages can

o ccur at most once for each edge this follows from the observation that once a REJECT

message has b een sentover an edge that edge will never b e tested again

All other messages sent in the course of the algorithmthe TESTACCEPT pairs that go

with the acceptances of edges the INITIATEREPORT broadcastconvergecast messages

and the CHANGEROOTCONNECT messages that o ccur when fragments combinecan

b e considered as part of the overall pro cess of nding the MWOE for a given fragment In

p erforming this task for a fragment there will b e at most one of these messages asso ciated

with each no de each no de receives at most one INITIATE and one ACCEPT each sends

at most one successful TEST one REPORT and one of either CHANGEROOT or CON

NECT Thus the numb er of messages sent within a fragment in nding the MWOE is O m

where m is the numb er of no des in the fragment

The total numb er of messages sentintheMWOEnding pro cess therefore is prop or

tional to

number of nodes in F

all f ragments F

whichis

X X

number of nodes in F

all lev el number s L all f r ag ments F of lev el L

Now the total numb er of no des in the inner sum at eachlevel is at most n since each

no de app ears in at most one fragmentata given levelnumber L And since the biggest

p ossible value of L is log nthesumaboveisboundedby

log n

n O n log n

Thus the overall message complexity of the algorithm is O E n log n

Time Complexity It can b e shown by induction on l that the time for all the no des to

reach level at least l is at most O ln Hence the time complexity of the GHS algorithm is

O n log n

Proving Correctness for the GHS Algorithm

A go o d deal of interesting work remains to b e done in the eld of proving correctness for

communication algorithms like the GallagerHumbletSpira algorithm The level of complex

ity of this co de makes direct invariant assertion pro ofs quite dicult to do at least at the

detailed level of the co de Note that most of the discussion in this lecture has b een at the

higher level of graphs fragments levels MWOEs etc So it seems that a pro of oughtto

take advantage of this highlevel structure

One promising approachistoapplyinvariantassertion and other techniques to prove

correctness for a highlevel description of the algorithm and then prove indep endently that

the co de in fact correctly simulates the highlevel description see WelchLL

A pro of can b e formalized by describing the highlevel algorithm as an IO automaton

in which the state consists of fragments and actions include merge and absorb op erations

describing the lowlevel co de as another IO automaton and then pro ducing that there is a

simulation mapping b etween the two automata

JJ Distributed Algorithms Novemb er

Lecturer Nancy Lynch

Lecture

Minimum Spanning Tree cont

Simpler Synchronous Strategy

Note that in the asynchronous algorithm wehave complications we did not encounter in

the synchronous algorithm eg absorbing vs merging waiting for fragments to catch up

Another idea to cop e with these diculties is to try to simulate the synchronous algorithm

more closely somehowsynchronizing the levels

More sp ecicallywe can design an algorithm based on combining fragments where each

fragment has an asso ciated level numb er as follows Each individual no de starts as a single

no de fragment at level This time fragments of level l can only b e combined into fragments

of level l as in the synchronous algorithm Eachnodekeeps a locallevel variable which

is the latest level it knows of the fragment it b elongs to so initially every no des locallevel is

and when it nds out ab out its memb ership in a new fragmentoflevel l it raises its lo cal

level to l The basic idea is that a no de at level l tries not to participate in the algorithm

for nding its level l fragments MWOE until it knows that all the no des in the system have

reached level at least l Implementing this test requires global synchronization This is

barrier synchronization But in fact some kind of a weaker lo cal synchronization suces

Namely each no de only tests that all of its neighbors have locallevel at least l This requires

each no de to send a message on all of its incident edges every time its level is incremented

This algorithm has the same time p erformance as b efore ie O n log n The message

complexity gets worse howeveritisnow O E log n

Synchronizers

The idea discussed ab ove suggests a general strategy for running a synchronous algorithm in

an asynchronous network This will only apply to a faultfree asynchronous algorithm ie

no failed pro cesses or messages b eing lost duplicated etc since as we shall so on see the

faulttolerance capabilityisvery dierentinsynchronous and asynchronous systems We are

working from Awerbuch with some formalization and pro ofs done recently by Devara jan a

student

The strategy works for general synchronous algorithms without faults the idea is to

synchronize at each round rather than every so often as in the MST algorithm sketched

ab ove Doing this every so often dep ends on sp ecial prop erties of the algorithm eg that

it works correctly if it is allowed to exhibit arbitrary interleavingsofnodebehavior b etween

synchronization p oints We shall only describ e howtosynchronize at every round

Synchronous Mo del

Recall that in the synchronous mo del each no de had message generation and transition

functions It is convenient to reformulate eachnodeasanIOA with output actions synch

send and input actions synchreceive The no de must alternate these actions starting with

synchsendAt rounds at which the synchronous algorithm do esnt send anything we assume

that a synchsend action o ccurs Wecallsuch a no de automaton a client automaton The

client automaton may also have other external actions interacting with the outside world

The rest of the system can b e describ ed as a synchronous message system called synch

sys for short in the sequel whichateach round collects all the messages that are sent

at that round in synchsend actions and then delivers them to all the client automata in

synchreceive actions In particular this system synchronizes globally after all the synch

send events and b efore all the synchreceive events of each round

Note that this comp osition of IOAs is equivalent to the more tightly coupled synchronous

mo del describ ed earlier where the message system was not mo deled explicitly

We will describ e how to simulate the synchsys using an asynchronous network so that

the client automata running at the network no des cannot tell the dierence b etween running

in the simulation system and running in a real synchronous system

HighLevel Implementation Using a Synchronizer

The basic idea discussed byAwerbuch is to separate out the pro cessing of the actual messages

from the synchronization He splits up the implementation into several pieces a frontend

for each no de able to communicate with the frontends of neighb ors over sp ecial channels

and a pure synchronizer S The job of each front end is to pro cess the messages received

from the lo cal clientin synchsend events At a particular round the front end collects all

the messages that are to b e sent to neighb ors note that a no de might send to some neighbors

and not to some others It actually sends those messages to the neighb ors along the sp ecial

channels For eachsuch message it sends it waits to obtain an acknowledgment along the

sp ecial channel When it has received acks for all its messages it is safe using Awerbuchs

terminology p of Aw erbuch pap er For a no de to b e safe means that it knows that

all its neighb ors have received its messages Meanwhile while waiting for its own messages

to b e acknowledged it is collecting and acknowledging the messages of its neighb ors

When is it OK for the no de frontend to deliver all the messages it has collected from its

neighb ors to the client Only when it knows that it wont receiveany others It is therefore

sucient to determine that al l the nodes neighbors are safe for that round ie that those

neighb ors knowthat their messages for that round have all b een delivered

Thus the job of the synchronizer is to tell frontends when all their neighb ors are safe

To do this the synchronizer has OK input actions outputs from the frontends by which the

front ends tell the synchronizer that they are safe The synchronizer S sends a GO message

to a front end when it has received an OK message from each of its neighb ors

For now we are just writing an abstract sp ec for the S comp onent of course this will

have to b e implemented in the network We claim that this combination simulates the

synchronous system combination of clients and synchsys Notice we did not say that it

implements the system in the formal sense of b ehavior inclusion In fact it do esnt this

simulation is lo cal in the sense that the rounds can b e skewed at distances in the network

Rounds are only kept close for neighb ors But in this architecture where the client programs

do not haveany means of communicating directly with each other outside of the synchsys

this is enough to preserve the view of each client

Theorem If is any execution of the implementation clients front ends channels and

synchronizer then thereisanexecution of the specication clients and synchsys such

that for al l clients i jclient jclient

i i

That is as far as each client can tell its running in a synchronous system So the

b ehavior is the same the same correctness conditions hold at the outside b oundary of the

clients sub ject to reordering at dierent no des

Pro of Sketch We cannot do an implementation pro of as we did for example for the

CTSS algorithm since we are reordering the actions at dierent no des Note that certain

events dep end on other events to enable them eg an OK event dep ends on prior ack

input events at the same frontend and all events at one client automaton may dep end

on each other This is a conservative assumption We dene this depends on relation D

formally as follows For every schedule of the implementation system there is a partial

order relation D describing events o ccurring in that dep end on each other The key

prop erty of this relation D is that for anyschedule of the implementation system and

of that preserves the partial order of events given by D wehave any p ermutation

that is also a schedule of the implementation system This says that the D relations are

capturing enough ab out the dep endencies in the schedule to ensure that any reordering that

preserves these dep endencies is still a valid schedule of the implementation system

Now we start with anyschedule of the implementation and reorder the events to make

the successive rounds line up globally The goal is to make the reordered schedule lo ok as

much as p ossible likea schedule of the synchronous system We do this by explicitly putting

all the synchreceiveievents after all the synchsend ievents for the same iWenow claim

this new requirement is consistent with the dep endency requirements in D since those

never require the reverse order even when applied transitivelyWeget in this wayBy

the key claim ab ove is also a schedule of the implementation system

Were not done yet although is fairly synchronous it is still a schedule of the

implementation system wewanta schedule of the sp ecication system In the next step

This still we suppress the internal actions of the synchsys getting a reduced sequence

lo oks the same to the clients as and nowwe claim that it is a schedule of the sp ecication

system This claim can b e proved by induction

Note that the theorem started with an execution not a schedule To complete the pro of

we extract the schedule get as ab ove and then ll in the states to get an execution of

the sp ecication system lling in the client states as in

Note that if we care ab out order of events at dierentclients then this simulation strategy

do es not work

Synchronizer Implementations

Nowwe are left with the job of implementing the synchronizer part of this implementation

Its job is quite simple it gets OK inputs from eachnodeateach round and for each

no de it can output GO when it knows that all its neighb ors in the graph strictly sp eaking

including the no de itself have done input OK for that round Wewant to implement this

by a distributed algorithm one piece p er no de using a message system as usual There are

several ways to do this

Synchronizer The synchronizer works as follows When a no de gets an OK it sends

this information to all its neighb ors When a no de hears that all its neighb ors are OK and

it is OK it outputs GO The correctness of this algorithm is fairly obvious The complexity

is as follows For every round we generate O E messages since all no des send messages on

all edges at every round Also each round takes constant time measuring time from when

all the OKs at a round have happ ened until all the GOs for that round have happ ened

We conclude that this algorithm is fast but maybetoocommunicationinecientfor

some purp oses This brings us to the other extreme b elow

Synchronizer For this synchronizer we assume that there is a ro oted spanning tree in

G The rule is now as follows Convergecast all the OK info to the ro ot and then broadcast

p ermission to do the GO outputs The cost of this algorithm is O n messages for every

round but the time to complete a round is prop ortional to the height of the spanning tree

whichisnintheworst case

Hybrid Implementation

By combining synchronizer and it is p ossible to get a hybrid algorithm that dep ending

on the graph structure mightgivesimultaneously b etter time eciency than and b etter

message eciency than The basic idea is to divide the graph up into clusters of no des

each with its own spanning tree ie a spanning forest We assume that this spanning

forest is constructed using some prepro cessing The rule nowwillbetousesynchronizer

within each clustertree and use to synchronize b etween clusters Toseehow this works

we found it useful to describ e a highlevel decomp osition see Figure

front ends

Cluster− Cluster− Cluster− synch synch synch

Forest−synch

Figure Decomp osition of the synchronization task to clustersynch for intracluster

synchronization and forestsynch for intercluster synchronization

The b ehavior of clustersynch is to take OKs from all no des in one particular cluster

which is a connected subset of no des in the graph and output a single clusterOK to

forestsynch Then when a single clusterGO is input from forestsynch it pro duces a GO

for each no de in the cluster A p ossible way to implement this on the no des of the given

cluster of course is just to use the idea of synchronizer doing convergecast to handle the

OKs and broadcast to handle the GOs b oth on a spanning tree of the cluster

The other comp onent is essentially up to renaming a synchronizer for the cluster graph

G of G where the no des of G corresp ond to the clusters of G and there is an edge from

C to D in G if in G there is an edge from some no de in C to some no de in D Ignoring

for the moment the problem of howtoimplement this forestsynchronizer in a distributed

system based on G let us rst argue why this decomp osition is correct Weneedtoshow

that any GO that is output at anynodei implies that OKs for i and all its neighb ors in

G have o ccurred First consider no de i The GOi action means that clusterGO for is

cluster C has o ccurred which means that clusterOK for C has o ccurred which means that

OKi has o ccurred Now consider j neighbors i C Same argument holds clusterOK

for C means that OKj has o ccurred Finallylet j neighbors i C The GOi action

means as b efore that clusterGO for C has o ccurred which implies that clusterOK for D

has o ccurred where D is the cluster containing j The clusters are neighbors in G b ecause

the no des i and j are neighb ors in G This implies as b efore that OKj has o ccurred

Implementation of forestsynch Wecanuseanysynchronizer to synchronize b etween the

clusters Supp ose wewant to use synchronizer Note that we cant run directly b ecause it

is supp osed to run on no des that corresp ond to the clusters with edges directly connecting

these no des clusters we arent provided with such no des and channels in a distributed

system However it is not hard to simulate them We do this as follows Assume wehavea

leader no de in each cluster and let it run the no de proto col of for the entire cluster This

canbut do esnt have tob e the same no de as is used as the ro ot in the implementation

of for the cluster if the cluster is synchronized using The next problem to solveisthe

waytwo leaders in dierent clusters can communicate Wesimulate direct communication

using a path b etween them Note that there must exist such a path b ecause the leaders

that need to communicate are in adjacent clusters and each cluster is connected We need

some prepro cessing to determine these paths we ignore this issue for now

We need also to sp ecify for the nal implementation where the clusterOK action o ccurs

This is done as an output from some no de in the clustersynch proto col if synchronizer

is used it is an output of the ro ot of the spanning tree of the cluster It is also an input to

the leader no de in the forestsynch for that same cluster If these two are the same no de

then this action just b ecomes an internal action of the actual no de in the distributed system

If these no de are dierent then we need to have them communicate along some path also

determined by some prepro cessing this requires an extra piece of the implementation

The formal structure of this hybrid algorithm is quite nice each no de in the distributed

network is formally an IOAwhich is the comp osition of two other IOAs one for each of the

two proto cols intracluster and intercluster synchronizers We can consider two orthogo

nal decomp ositions of the entire implementation vertical ie to no des and channels or

horizontal ie bythetwo algorithms each distributed one piece p er no de

Analysis We shall consider the sp ecic implementation where the clustersynch is done

using and the forestsynch is done using andwe assume that the leader for within a

cluster is same no de that serves as the ro ot for

Consider one round The numb er of messages is O n for the work within the clusters

plus O E where E is the numb er of edges on all the paths needed for communication among

the leaders We remark that dep ending on the decomp osition E could b e signicantly

smaller than E The time complexity is prop ortional to the maximum heightofany cluster

spanning tree

Thus the optimal complexityofthehybrid algorithm b oils down to the combinatorial

problem of nding a graph decomp osition such that b oth the sum of the path lengths and the

maximum height are small Also we need to establish this decomp osition with a distributed

algorithm We will not address these problems here

Applications

We can use the synchronizers presented ab ovetosimulate anysynchronous algorithm in

the original mo del for synchronous systems presented earlier in the course on an asyn

chronous system Note that the synchronizer do esnt work for faulttolerant algorithms such

as Byzantine agreement b ecause it is not p ossible in this case to wait for all pro cesses

Ring leader election algorithms such as LeLannChang Hirshb ergSinclair and Peterson

can thus b e run in an asynchronous system But note that the message complexitygoesway

up if we do this Since they already worked in an asynchronous system without any such

extra synchronization this isnt interesting The following applications are more interesting

Network leader election Using the synchronizer the algorithm that propagates the max

imal ID seen so far can work as in the synchronous setting This means that it can count

rounds as b efore waiting only diam rounds b efore terminating It also means that the num

b er of messages do esnt have to b e excessive its not necessary for no des to send constantly

since eachnodenowknows exactly when it needs to send once to each neighb or at each

asynchronous round Using synchronizer the complexity of the resulting algorithm is

O diam time and O E diam messages which is b etter than the aforementioned naive

strategy of multiple spanning trees

Breadthrst search For this problem the synchronizer proves to b e a big win Recall

the horrendous p erformance of the BellmanFord algorithm in the asynchronous mo del and

how simple the algorithm was in the synchronous setting Nowwe can run the synchronous

algorithm using say synchronizer which leads to an O E diam message O diam time

algorithm Compare with the synchronous algorithm which needed only O E messages

Note O E diam might b e considered excessive communication We can give a direct

asynchronous algorithm based on building the BF tree in levels doing broadcastconvergecast

at each level adding in the new leaves for that level and terminating when no new no des are

discovered For this algorithm the time is then O diam whichisworse than the algorithm

based on the synchronizer But the message complexity is only O E n diam since each

edge is explored once and tree edges are traversed at most a constantnumb er of times for

each level It is also p ossible to obtain a timecommunication tradeo bycombining the two

strategies More sp ecicallywe can explore k levels in each phase using obtaining time

ndiam diam

and communication complexityof O E k for k diam complexityof O

k k

Weighted Singlesource Shortest Paths Using the synchronous algorithm with synchro

nizer yields an algorithm with O n time complexityand O En messages We remark

that there is an algorithm with fewer messages and more time cf Gab ow as develop ed by

Awerbuch in his notes

Maximal Independent Set We can apply the synchronizer to a randomized synchronous

algorithm like MIS to o We omit details

Lower Bound for Synchronizers

Awerbuchs synchronizer result suggests that a synchronous algorithm can b e converted into

a corresp onding asynchronous algorithm without to o great an increase in costs In particular

by using synchronizer it is p ossible not to increase the time cost at all The following

result by ArjomandiFischerLynch shows that this approach cannot b e adopted universally

In particular it establishes a lower b ound on time for an asynchronous algorithm to solve

a particular problem Since there is a very fast synchronous algorithm for the problem

this means that not every fast synchronous algorithm can b e converted to an asynchronous

algorithm with the same time complexity Note that the dierence is not caused by requiring

any faulttolerance so the result may seem almost contradictory to the synchronizer results

We start by dening the problem Let G V Ebeagraph Recall that diam G

denotes the maximum distance b etween twonodesinG The systems external interface

consists of ash output actions for all i V where ash is an output of the IO automaton

i i

at no de i As an illustration imagine that the ash is a signal that no de i has completed

some task Dene a session as a sequence of ashes in which at least one ash o ccurs for

every i We can now dene the k session problem we require simply that the algorithm

should p erform at least k disjoint sessions

The original motivation for this setting was that the no des were p erforming some kind

of co ordinated calculation on external data eg a Bo olean matrix transitive closure in the

PRAM style Eachnodei j k was resp onsible for writingainlocation i j in case

it ever saw s in b oth lo cations i k andk j Notice that for this problem it do esnt

matter whether the no des do excessivechecking and writing Correctness is guaranteed if

wehave at least log n sessions ie if there is enough interleaving

The reason the problem is stated this way is that it is a more general problem than the

simpler problem in which all pro cesses do exactly one step in each session so it strengthens

the imp ossibility result Note also that this is a weak problem statement it do esnt even

require the no des to knowwhenk sessions have completed

Before weturntoprove the lower b ound result let us make the problem statement

more precise As usual we mo del the pro cesses as IOAs connected by FIFOchannels For

simplicitywe let the no de automata partition consist of a singleclass intuitively the no des

are executing sequential programs

Our goal is to prove an inherent time complexity result Note that this is the rst lower

b ound on time that we are proving in this course We need to augment the mo del by

asso ciating times as usual with the events in a monotone nondecreasing way approaching

innity in an innite fair execution Let l b e a b ound for lo cal step time and d be the bound

for delivery of rst message in eachchannel This is a sp ecial case of the general notion of

time b ounds for IOAs We assume that d l An execution with times asso ciated with

events satisfying the given requirements is called a timed execution

Next we dene the time measure T A for algorithm A as follows For each execution

of the algorithm dene T to b e the supremum of the times at whicha ash event o ccurs

in There could b e innitely manysuchevents hence we use a supremum here Finally

we dene

T A supT

We can now state and prove our main result for this section

Theorem Suppose A is an algorithm that solves the k session problem on graph G Then

T A k diam G d

Before we turn to prove the theorem consider the k session problem in the synchronous case

We can get k sessions without the no des ever communicating each no de just do es a ash

at every round for k rounds The time would b e only kd for a time measure normalized so

that each round counts for d time This discrepancy proves that the inherentmultiplicative

overhead due to asynchrony for some problems is prop ortional to diam G

Pro of Sketch of Theorem

By contradiction Supp ose that there exists an algorithm A with T A k diam G d

Call a timed execution slow if all the actions take their maximum times ie if the deliveries

of the rst messages in the channels always takes time d and the step times always take l

Let be any slow fair timed execution of the system By the assumption that A is correct

contains k sessions By the contradiction assumption the time of the last ash in is

strictly less than k diam G dSowe can write where the time of the

last eventin is less than k diam G d and where there are no ash events in

Furthermore b ecause of the time b ound we can decomp ose where each

of the has the dierence b etween the times of its rst and last events strictly less than

diam G d

Wenow construct another fair execution of the algorithm This

will b e an ordinary untimed fair execution that is we do not assign times to this one The

execution is constructed by reordering the actions in each and removing the times to

obtain and by removing the times from to get Of course when we reorder events

we will have to make some adjustments to the intervening states We will show that the

mo died execution contains fewer than k sessions whichcontradicts the correctness of A

The reordering must preserve certain dep endencies in order to pro duce a valid execution of

AFormallywe shall prove the following claim

Claim Let i i i iand i i i j wheredisti j diam G

For al l r k there exists a reordering of ie the actions are

r r r r

reordered with the initial and nal states unchanged such that the fol lowing properties

hold

The reordering preserves the fol lowing dep endency partial order

a The order of al l actions of the same node

b The order of each send m event and its corresponding receive m event

ij ij

contains no event of i

r r

contains no event of i

r r

Let us rst showhow to complete the pro of of Theorem using Claim Since the

reordering preserves the dep endencies this means that is a fair execution of

A But we can show that this execution contains at most k sessions as follows No session

can b e entirely contained within since contains no eventof i Likewise no session can

be entirely contained within since this sequence contains no event of pro cess i

r r r

This implies that each session must contain events on b oth sides of some b oundary

r r

Since there are only k such b oundaries the theorem follows

It remains to prove Claim Todothiswe pro duce the required reorderings The

construction is done indep endently for each r Soxsomer r k We consider

two cases

Case contains no eventof i In this case wedene and is empty

r r r r r

Case contains an eventof i Let b e the rst eventof i in Nowwe claim

r r r r

that there is no step of i in that dep ends on according to the dep endency relation

r r

dened in Claim This is true since the time for a message to propagate from i to i in

r r

aslow execution is at least diam d whereas wehave constructed so that the time b etween

its rst and last events is strictly less than diam dThus we can reorder so that all the

steps of i precede wekeep the initial and nal states the same and mimic the lo cal state

changes as in This still yields allowable transitions of the algorithm by a dep endency

theorem similar to the one we used for the synchronizer construction but simpler Let

b e the part of the reordered sequence b efore and the part of the sequence starting with

It is straightforward to verify that has the prop erties required to prove Claim

r r r

Note that the reordered sequence is not a timed execution if wewere trying to preserve

the times somehow during the reordering wewould b e stretching and shrinking some time

intervals Wewould have to b e careful to observe the upp er b ound requirements We could

avoid these problems by making all the time intervals very small but we dont need to worry

ab out this at all since all we need to get the contradiction is an untimed fair execution

Theorem almost lo oks likea contradiction to the synchronizer result recall that the

synchronizer gives a simulation with constant time overhead This is not a contradiction

however b ecause the synchronizer simulation do esnt preserve the total external b ehavior

it only preserves the b ehavior at each no de while it may reorder the events at dierent

no des We remark that for most purp oses we might not care ab out global reordering but

sometimes when there is some outofband communication the order of events at dierent

no des might b e imp ortant

JJ Distributed Algorithms Novemb er

Lecturer Nancy Lynch

Lecture

Time Clo cks etc

The idea that it is OK to reorder events at dierent no des motivates Lamp orts imp ortant

notion of logical time dened in his famous pap er Time Clocks and the Ordering of

Events in a Distributed System

Logical Time

The mo del assumed is the one wehave b een using In this mo del there is no builtin notion

of realtime in particular there are no clo cks but it is p ossible to imp ose a logical notion of

time namely Lamp orts logical time Sp ecicallyevery event of the system send or receive

of messages external interface event internal event of no de gets assigned a logical time

an element of some xed totally ordered set Typically this set is either the nonnegative

integers or the nonnegative reals p erhaps with tiebreakers These logical times dont have

to haveany particular relationship to anynotionofrealtimeHowever they do need to

satisfy the following prop erties

No twoevents get assigned the same logical time

Events at each no de obtain logical times that are strictly increasing in the same order

as the events o ccur

For any message its send event gets a strictly smaller logical time than its receive

event

For anyevent there are only nitely many other events that get assigned logical times

that are smaller

The imp ortant result ab out such an assignmentisasfollows Supp ose that we are given

an execution of a network system of IO automata and a logical time assignment ltime

that satises the conditions ab ove Then there is another execution of the same system

which lo oks the same to each no de ie ji ji for all i and in which the times from

the assignment ltime o ccur in increasing order in real time In other words we can havea

reordered execution such that the logical time order of the original execution is the same as

the real time order in the reordered execution So we can say that programming in terms of

logical time lo oks to all the users as if they are programming in terms of real time

We can view this as another way to reduce the uncertaintyofanasynchronous network

system Roughly sp eaking we can write asynchronous programs where we assume that each

no de has access to real time Real time mo deling do esnt t the IOA mo del whichis

why this is rough It will b e done more carefully later in the course when we do timing

based mo dels In the basic mo del the no des dont have this access Instead we can have

them implement logical time somehow and let them use the logical time in place of the real

time The reordering result says that the resulting executions will lo ok the same to all no des

separately

One generalization is useful supp ose wewant to augment the mo del to allow broadcast

actions which serve to send the same message to all the other no des We can implement

this of course with a sequence of individual send actions There is nothing to o interesting

here thus far But the imp ortant thing here is that we can allow a broadcast to havea

single logical time and require that all the asso ciated receives have larger logical times The

reordering result still holds with this new action

Algorithms

Lamp orts implementation Lamp ort presents a simple algorithm for pro ducing such

logical times It involves each no de pro cess maintaining a lo cal variable clock that it uses as

a lo cal clo ck The lo cal clo ck gets incremented at eachevent input output or internal that

o ccurs at that no de The logical time of the event is dened to b e the value of the variable

clock immediately after the event paired with the pro cess index as a tiebreaker

This algorithm is easily seen to ensure Prop erties and ab ove In order to ensure

Prop erty the following rule is observed Whenever a node does a send or broadcast

event it rst increments its clock variable to get the logical time for the send event and

it attaches that clo ckvalue to the message when it sends it out When the receiver of the

message p erforms a receive event it makes sure that it increments its clock variable to b e

not only larger than its previous value but also strictly larger than the clo ckvalue in the

message As b efore it is this new clo ckvalue that gets assigned to the receive event To

ensure Prop erty it suces to allow each increase of a clock variable to increase the value

by some minimum amount eg

Welchs Implementation In Welchs algorithm as in Lamp orts the logical time at

each no de is maintained by a state variable named clock which can take on nonnegative

integer or real values But this time we assume that there is an input action tick tthat

is resp onsible for setting this variable The no de sets its clock variable to t when tick tis

received We can imagine this input as arriving from a separate clo ck IOA The no de

implements b oth this clo ck and the pro cess IOA

Now each nontick action gets its logical time equal to the value of clock when it is

p erformed with tie breakers rst by the pro cess index and second by execution order at

the same pro cess Note that the clock value do es not change during the execution of this

action Some additional careful handling is needed to ensure Prop erty This time wehave

an extra FIFO receivebuer at each no de which holds messages whose clo ckvalues are at

least the clo ckvalue at the lo cal no de Assuming that the lo cal clock value keeps increasing

without b ound eventually it will b e big enough so that it dominates the incoming messages

clo ckvalue The rule is to wait for the time where the rst message in the receivebuer has

an asso ciated clo ck less than the lo cal clock Then this message can b e pro cessed as usual

we dene the logical time of the receiveevent in terms of the lo cal clock variable when this

simulated receive o ccurs

The correctness of this algorithm is guaranteed by the following facts Prop ertyis

ensured by the the tiebreakers Prop erty relies on the monotonicity of the lo cal clo cks plus

the tiebreakers Prop erty is guaranteed by the buer discipline and Prop erty requires

a sp ecial prop erty of the clo ck automaton namely that it must growunb oundedly This

could b e achieved eg byhaving a p ositivelower b ound on the tick increment and having

the clo ck automaton execute fairly

Note that this algorithm leads to bad p erformance when the lo cal clo cks get out of synch

For this to work in practice we need some synchronization among the clo cks whichisnor

really p ossible in a purely asynchronous system We could use a synchronizer for the lo cal

clo cks but it is to o exp ensive to do this at every step Instead we could use a synchronizer

every so often sayevery steps Still in a pure asynchronous system the worstcase

b ehavior of this strategy will b e p o or This algorithm makes more sense in a setting with

some notion of realtime

Applications

It is not clear that the ab ove algorithms can always b e used to implementsome ltime

abstraction which then can b e used as a blackbox The problem is that the logical time

In other words we dont allow Zeno behaviors in which innite executions take a nite amount of time

abstraction do es not have a clear interface description Rather well consider uses of the

general idea of logical time Before wegointo more sophisticated applications we remark

that b oth algorithms can b e used to supp ort logical time for a system containing broadcast

actions as describ ed ab ove

Banking system

Supp ose wewant to count the total amount of money in a banking system in whichnonew

money is added to or removed from the system but it may b e sent around and received ie

the system satises conservation of money The system is mo deled as some collection of

IOAs with no external actions having the lo cal amount of money enco ded in the lo cal state

with send actions and receive actions that have money parameters The architecture is xed

in our mo del the only variation is in the decision of when to send moneyandhowmuch

Supp ose this system somehow manages to asso ciate a logical time with eacheventas

ab ove Imagine that eachnodenow consists of an augmented automaton that contains

the original automaton This augmented automaton is supp osed to know the logical time

asso ciated with eachevent of the basic automaton In this case in order to count the total

amount of money in the bank it suces to do the following

Fix some particular time tknown to all the no des

For each no de determine the amount of money in the state of the basic automaton

after pro cessing all events with logical time less than or equal to t and no later events

For eachchannel determine the amount of money in all the messages sent at a logical

time at most t but received at a logical time strictly greater than t

Adding up all these amounts gives a correct total This can b e argued as follows Consider

any execution of the basic system together with its logical time assignment By the

reordering result theres another execution of the same basic system as ab ove and same

logical time assignment that lo oks the same to all no des and all events o ccur in logical time

order What this strategy do es is cut execution immediately after time t and record

the money in all the no des and the money in all the channels This simple strategy is thus

giving an instantaneous snapshot of the system state whichgives the correct total amount

of money in the banking system

Let us consider how to do this with a distributed algorithm Each augmented automaton

i is resp onsible for overseeing the work of basic automaton i Augmented automaton i is

assumed able to lo ok inside basic automaton i to see the amount of money Note that this

is not ordinary IOA comp osition the interface abstractions are violated by the coupling It

is also assumed to know the logical time asso ciated with eachevent of the basic automaton

The augmented automaton i is resp onsible for determining the state of the basic automaton

at that no de plus the states of all the incoming channels

The augmented automaton i attaches the logical time of each send event to the message

b eing sent It records the basic automaton state as follows Automaton i records the state

after each step until it sees some logical time greater than t Then it backs up one

step and returns the previous state For recording the channel state we need to know the

messages that are sent at a logical time at the sender at most t and received at a logical

time at the receiver strictly greater than t So as so on as an event o ccurs with logical

time exceeding t that is when it records the no de state it starts recording messages

coming in on the channel It continues recording them as long as the attached timestamp

is no more than t Note to ensure termination we need to assume that each basic no de

continues sending messages every so often so that its neighb ors eventually get something

with timestamp strictly larger than t Alternatively the counting proto col itself could add

some extra messages to determine when all the senders messages with attached time at most

t have arrived

We remark that the augmented algorithm has the nice prop erty that it do esnt interfere

with the underlying op eration of the basic system

General Snapshot

This strategy ab ove can of course b e generalized b eyond banks to arbitrary asynchronous

systems Supp ose wewanttotakeanysuch system that is running and determine a global

state at some p ointintimeWe cant actually do this without stopping everything in the

system at one time or making duplicate copies of everything It is not practical in a real

distributed system eg one with thousands of no des to really stop everything it could

even b e imp ossible But for some applications it may b e sucient to get a state that just

lo oks like a correct global state as far as the no des can tell We shall see some such

applications later In this case we can use the strategy describ ed ab ove

Simulating a Single State Machine

Another use mentioned but not emphasized in Lamp orts pap er but very much emphasized

by him in the ensuing years is the use of logical time to allow a distributed system to

simulate a centralized state machine ie a single ob ject The notion of an ob ject that

works here is a very simple one essen tially it is an IO automaton with input actions only

we require that there b e only one initial value and that the new state b e determined by the

old state and the input action This may not seem very useful at rst glance but it can b e

used to solvesomeinteresting synchronization problems eg we will see how to use it to

solvemutual exclusion Its also a mo del for a database with up date op erations

Supp ose now that there are n users supplying inputs up dates at the n no de pro cesses of

the network Wewould like them all to apply their up dates to the same centralized ob ject

We can maintain one copyofthisobjectinonecentralized lo cation and send all the up dates

there But supp ose that wewould like all the no des to have access to ie to b e able to read

the latest available state of the ob ject Eg supp ose that reads are more frequent than

writes and wewant to minimize the amountofcommunication This suggests a replicated

implementation of the ob ject where eachnodekeeps a private copy we can broadcast all

the up dates to all the no des and let them all apply the up dates to their copies But they

need to up date their copies in the same way at least eventuallyIfwe just broadcast the

up dates nothing can stop dierent no des from applying them in dierent orders We require

a total order of all the up dates pro duced anywhere in the system known to all the no des

so that they can all apply them in the same order Moreover there should only b e nitely

many up dates ordered b efore any particular one so that all can eventually b e p erformed

Also a no de needs to know when its safe to apply an up date this means that no further

up dates that should have b een applied b efore ie b efore the one ab out to b e applied will

ever arrive Wewould like the prop erty of monotonicity of successive up dates submitted by

a particular no de in order to facilitate this prop erty

The answer to our problems is logical time When a user submits an up date the asso ci

ated pro cess broadcasts it and assigns it a timestamp which is the logical time assigned to

the broadcast event for the up date If a no de stops getting lo cal up dates as inputs it should

still continue to send some dummy messages out just to keep propagating information ab out

its logical time Each no de puts all the up dates it receives together with their timestamps

in a request queue not yet applying them to the ob ject Note this queue must include

all the up dates the no de receives in broadcasts by other no des plus those the no de sends in

its own broadcasts The no de applies up date u if the following conditions are satised

Up date u has the smallest timestamp of any request on the request queue

The no de has received a message request or dummy from every no de with send time

at least equal to the timestamp of u

Condition and the FIFOness of the channels guarantee that the no de will never receive

anything with smaller timestamp than this rst up date Furthermore anynodeeventually

succeeds in applying all up dates b ecause logical time keeps increasing at all no des Note

that we didnt strictly need Prop erty of logical time relating the times of message send

and receive here for correctness But something like that is imp ortant for p erformance

Example Banking distributed database Supp ose each no de is a bank branch and

up dates are things like dep osit withdraw addinterest withdrawfromcheckingifsucient

fundselsefromsavings etc Note that in some of these cases the order of the up dates

matters Also supp ose that all the no des wanttokeep a recent picture of all the bank

information for reading deciding on which later up dates should b e triggered at that branch

etc We can use logical time as ab ovetosimulate a centralized state machine representing

the bank information

Note that this algorithm is not very faulttolerant Also in general it do es not seem

very ecient But it can sometimes b e reasonable see the mutual exclusion algorithm

b elow

Simulating Shared Memory

In this section we consider another simplifying strategy for asynchronous networks We use

this strategy to simulate instantaneous shared memory algorithms

The basic idea is simple Lo cate each shared variable at some no de Each op eration

gets sent to the appropriate no de and the sending pro cess waits for a resp onse When the

op eration arrives at the simulated lo cation it gets p erformed p ossibly b eing queued up

b efore b eing p erformed and a resp onse is sentback When the resp onse is received by

the sending pro cess it completes its step If new inputs arrive from the outside world at a

pro cess during a waiting p erio d the pro cess only queues them These p ending inputs are

pro cessed only when the anticipated resp onse returns Wenow claim that this simulation is

correct ie that every external b ehavior is also an external b ehavior of the instantaneously

shared memory system b eing simulated also every fair b ehavior of no des and channels is

also a fair b ehavior of the instantaneous shared memory system also some kind of wait

freedom carries over To prove this claim we use the corresp onding result for atomic shared

memory simulating instantaneous shared memory and the observation that we are actually

simulating atomic ob jects using a centralized ob ject and the delays in the message system

We omit details

The problem of where to put the shared variables dep ends on the characteristics of the

application at hand Eg for a system based on singlewriter multireader shared variables

where writes are fairly frequent wewould exp ect to lo cate the variables at the writers no de

Earlier in the course we had a restriction for the atomic shared memory simulation of instantaneous

shared memory that no new inputs arriveatany no de while the no de pro cess is waiting for an invo cation

to return The reason for this was to avoid intro ducing any new interleavings It seems however that we

could have handled this by just forcing the pro cess to queue up the inputs as ab ove and so remove the need

for this restriction

Note that the ob jects have to b e simulated even while the no des dont have activere

quests

There are other ways of simulating shared memory algorithms

Caching For singlewriter multireader registers note that someone who wants to rep eat

edly test the value of a register has in the implementation describ ed ab ove to send rep eated

messages even though the variable is not changing It is p ossible to develop an alternative

strategy by which writers notify readers b efore they change the value of the variable so if

the readers havent heard ab out a mo dication they can just use their previous value This

is the basic idea of caching

Replicatedcopies Caching is one example of a general style of implementation based on

data replication Data can b e replicated for many reasons eg faulttolerance whichwe

will discuss later But often it is done only for availability Supp ose wehavea multiwriter

multireader shared register in which writes are very infrequent Then we can lo cate copies

of the register only at all the readers no des compare with the caching example A reader

to read the register can always lo ok at its lo cal copy A writer to write the register needs to

write all the copies Note that it needs to do this atomically since one at a time may cause

outoforder reads Therefore we need some extra proto col here to ensure that the copies

are written as if atomically This requires techniques from the area of database concurrency

control

Mutual Exclusion and Resource Allo cation

In this section we consider the problem of resource allo cation in a distributed messagepassing

network

Problem Denition

Wehave the same interface as b efore but now the inputs and outputs for user i o ccur at

a corresp onding no de i see Figure The pro cesses one for each i communicate via

messages over FIFOchannels

Consider the resource requirement formulation of the problem in terms of a monotone

Bo olean formula involving the needed resources Eachnode i has a static xed resource

requirement Assume that anytwo no des that haveany common resource in their two

resource requirements are neighb ors in the network graph so they can communicate to

negotiate priority for the resources We do not mo del the resources separately

In this setting wetypically drop the restriction that no des can p erform lo callycontrolled

steps only when they are b etween requests and resp onses This is b ecause the algorithms try crit exit rem

try crit exit rem

try crit exit

rem

Figure Interface sp ecication for resource allo cation on a message passing system

will typically need to resp ond to requests by their neighb ors for the resources

Th correctness conditions are now as follows As b efore we require a solution to preserve

wel lformedness cyclic b ehavior and also mutual exclusion or a more general exclusion

condition

We also require dead lockfreedom ie if there is any active request and no one in C

then some request eventually gets granted and if there is any active exit region then some

exit eventually o ccurs This time the hyp othesis is that all the no de and channel automata

exhibit fair b ehavior in the IOA sense ie all the no des keep taking steps and all the

channels continue to deliver all messages

The lockoutfreedom condition is dened as b efore under the hyp othesis that all the no de

and channel automata exhibit fair b ehavior

For the concurrency prop erty supp ose that a request is invoked at a no de iandall

neighb ors are in R and remain in R Supp ose that execution pro ceeds so that i and all its

neighb ors continue to take steps fairly and the connecting links continue to op erate fairly

Note that other pro cesses can remain in C forever Then the concurrency condition requires

that eventually i reach C

We remark that this is analogous to the weak condition we had for shared memory

systems only here we add fairness requirements on the neighb ors and the links in b etween

Mutual Exclusion Algorithms

Raynals b o ok is a go o d reference also for the shared memory mutual exclusive algorithms

Wehave many p ossible approaches now

Simulating Shared Memory

Wehave learned ab out several shared memory algorithms for mutual exclusion Wecan

simply simulate one of these in a distributed system Eg use the Peterson multiwriter

multireader shared register tournament algorithm or the variant of this that we didnt have

time to cover or some version of the bakery algorithm mayb e with unb ounded tickets

LeLann

LeLann prop osed the following simple solution The pro cesses are arranged in a logical ring

p p p p Atoken representing control of the resource is passed around the ring in

order When pro cess p receives the token it checks for an outstanding request for the

resource from user If there is no such request the token is passed to the next pro cess in

the ring If there is an outstanding request the resource is granted and the token is held

until the resource is returned and then passed to the next pro cess

The co de for pro cess p is given in Figure

Let us go over the prop erties of LeLanns algorithm

Mutual Exclusion is guaranteed in normal op eration b ecause there is only one token and

only its holder can haveany resource

Dead lockFreedom in fair executions when no one is in C the pro cess that holds the

token is either

in T and then it can go to C or

in E R and then it has to pass the token to the next pro cess

Similarly the deadlo ckfreedom for the exit region is satised

LockoutFreedom is straightforward

Lo cal variables

token fnone available in use used g initially available at p andnone else

where

region fR T C E g initially R

Actions

try

Eect region T

crit

Precondition region T

token avail

Eect region C

use token in

exit

Eect region E

rem

Precondition region E

Eect region R

token used

receive token

Eect token available

send token

W V

Precondition token used token available region T

Eect token none

Figure LeLann mutual exclusion algorithm for message passing systems

Performance First lets consider the numb er of messages It is not clear what to

measure since the messages arent naturally app ortioned to particular requests Eg we

can measure the worstcase numb er of messages sentinbetween a try and corresp onding

crit but it seems more reasonable to try to do some kind of amortized analysis Note that

each virtual link in the logical ring is really of unit length since the graph is complete

In the worst case n messages are sentbetween tr y and cr it For the amortized cost we

i i

consider the case of heavy load where there is always an active request at each no de in

this case there are only a constantnumb er of messages p er request If a request comes in

alone however we still get n messages in the worst case Also it is hard to get a go o d

b ound statement here since messages are senteven in the absence of any requests at all

For the time complexitywe assume worstcase b ounds Let c b e the time sp entinC d

b e the message delay for the rst message in anychannel and s b e the pro cess step time

The worstcase time is approximately c d O s n Note that unfortunately this time

b ound has a d n term regardless of the load and this mightbebig

Resiliency LeLann discusses various typ es of resiliency in his pap er

Process failure If a pro cess stops and announces its failure the rest of the pro cesses can

recongure the ring to bypass the failed pro cess This requires a distributed proto col

Note that a pro cess that do esnt announce its failure cant b e detected as failed since

in an asynchronous system there is no way for a pro cess to distinguish a failed pro cess

from one that is just going very slowly

Loss of token When a token loss is detected eg by timeout a new one can b e

generated by using a leaderelection proto col as studied earlier

Lamp orts algorithm

Another solution uses Lamp ort logical time In particular the state machine simulation

approach is applied The state machine here has as its state a queue of pro cess indices

which is initially empty The op erations are try and exit where try has the eect of

i i

adding i to the end of queue and exit has the eect of removing i from queue provided

it is at the front When user i initiates a try or exit event it is regarded as submitting

a corresp onding up date request Then as describ ed ab ove the state machine approach

broadcasts the up dates waits to b e able to p erform them ie a no de can p erform an up date

when that up date has the smallest timestamp of any up date in the request buer and the

no de knows it has all the smaller timestamp up dates When the no de is able to p erform the

up date it do es so The replicated ob ject state is used as follows When the queue at no de

i gets i at the front the pro cess at no de i is allowed to do a crit Whenthequeue at no de

i gets i removed the pro cess at no de i is allowed to do a rem

Let us now argue that this algorithm guarantees mutual exclusion Supp ose not say

p and p are simultaneously in C and supp ose without loss of generalitythat p has the

i j i

smaller timestamp Then when j was added to p s queue the state machine implementation

ensures that i must have already b een there ahead of j Buti has not b een removed since

it is still in C So j could not havedonea crit action a contradiction

It is a little hard to analyze the algorithm in this form since it combines three proto cols

the logical time generation proto col the replicated state machine proto col and the extra

rules for when to do crit and rem actions We can merge and optimize these proto cols

somewhat as follows Assume that the logical time generation is done according to Lamp orts

algorithm based on the given messages We combine the requestbuer and the simulated

state machines state Also the ack messages b elow are used as the dummy messages

discussed in the general approach

In this algorithm every pro cess p maintains a lo cal variable region as b efore and for each

other pro cess p a lo cal queue queue This queue contains all the messages ever received

j j

from p we remark that this can b e optimized There are three typ es of messages

trymsgi Broadcast by p to announce that it is trying

exitmsg i Broadcast by p to announce that it is exiting

ack i Sentby p to p acknowledging the receipt of a trymsgj message

i j

The co de is written so as to send all these messages at the indicated times We assume

that the sending or broadcast logical times are piggybacked on the messages and the queues

contain these logical times as well as the messages themselves Also the logical time of each

of the broadcast events for try or exit requests is used as the timestamp of the corresp onding

up date

This already tells what messages are sent and when Wewont have a separate application

step to apply the up date to the queue since wedonothave an explicit representation of the

queue All we need now is rules for p telling when to p erform crit and rem actions Below

i i i

we give these rules

Canbedoneanytime after exit o ccurs p R

i i

Must ensure that region T and that the following conditions hold p C

Mutual exclusion is preserved

There is no other request p ending with a smaller timestamp

Pro cess p can ensure that the ab ove conditions are met bychecking for each j i

Any trymsg in queue with timestamp less than the timestamp of the current

try msg of p has an exitmsg in queue with a larger timestamp than that of

j s trymsg

queue contains some message p ossibly an ack with timestamp larger than the

timestamp of the current trymsg of p

Lamp orts algorithm has the following prop erties

Mutual Exclusion This can b e proven bycontradiction as follows Assume that two

pro cesses p and p are in C at the same time and without loss of generality that the

i j

timestamp of p s request is smaller than the timestamp of p s request Then p had to

i j j

check its queue in order to enter C The second test and FIFO b ehavior of channels imply

that p had to see p s trymsgsoby the rst test it had also to see an exitmsg from p so

j i i

p must have already left C

Lockoutfreedom This prop erty results from servicing requests in timestamp order Since

eachevent in particular request event has a nite numb er of predecessors all requests will

eventually get serviced

Complexity Let us rst deal with the numb er of messages We shall use amortized

analysis here as follows Every request involves sending trymsg ack and exitmsg messages

between some pro cess and all the others thus n messages are sent p er request For

the time complexity note that when there is a single request in the system the time is

d O s where d is the communication delayand s is the lo cal pro cessing time We assume

that the broadcast is done as one atomic step if n messages are treated separately the

pro cessing costs are linear in n but these costs are still presumed to b e small compared

to the communication delay d Recall that in contrast the time complexity of Lelanns

algorithm had a dn term

JJ Distributed Algorithms December

Lecturer Nancy Lynch

Lecture

Mutual Exclusion cont

Last time wegave Lamp orts mutual exclusion algorithm based on his statemachine ap

proach Recall that wehave try and exit messages as input to the state machine and

ack messages to resp ond to try messages The following twovariants provide interesting

optimizations of this algorithm

Ricart Agrawala

Ricart and Agrawalas algorithm uses only n messages p er request It improves

Lamp orts original algorithm byacknowledging requests in a careful manner that eliminates

the need for exit messages This algorithm uses only twotyp es of messages try and OK

Pro cess p sends try i as in Lamp orts algorithm and can go critical after OK messages

have b een received from all the others So the interesting part is a rule for when to send

an OK message The idea is to use a priorityscheme Sp ecically in resp onse to a trya

pro cess do es the following

Replies with OK if it is not critical or trying

If it is critical it defers the reply until it exits and then immediately sends all the

deferred OKs

If it is trying it compares the timestamp of its own request to the timestamp of the

incoming try If its own is bigger its own is interpreted as lower priority and the pro cess

sends OK immediatelyelsehaving higher priority the pro cess defers acknowledging

the request until it nishes with its own critical region

In other words when there is some conict this strategy is to resolveitinfavor of the

earlier request as determined by the timestamps

Prop erties of the RicartAgrawala Algorithm

Mutual Exclusion Proved bycontradiction Assume that pro cesses p and p are b oth

i j

in C and without loss of generality that the timestamp of p s request is smaller than the

timestamp of p s request Then there must have b een try and OK messages sent from each

to the other prior to their entry to C at each no de the receipt of the try precedes the

sending of the matching OK But there are several p ossible orderings of the various events see Figure

Process i Process j

try−i try−j

OK−i

OK−j

Figure A p ossible scenario for the RicartAgrawala algorithm

Now the timestamp of i is less than that of j and the logical time of the receipt of j s

try by i is greater than the timestamp of j s request by the logical time prop erty So it

must b e that the receipt of p s try message at i o ccurs after p has broadcast its try Then

j i

at the time p receives p s try it is either trying or critical In either case p s rules sayit

i j i

has to defer the OK message thus p could not b e in C a contradiction

Dead lockfreedom is also proved by contradiction Assume some execution that reaches a

p oint after which no progress is achieved That is at that p oint all the pro cesses are either

in R or T none are in C and from that p oint on no pro cess changes regions By going

a little further out in the execution we can also reacha point after which no messages are

ever in transit Among all the pro cesses in T after that p oint let p b e the pro cess having

the request message with the lowest timestamp Then since p is blo cked forever it must b e

b ecause some other pro cess p has not returned an OK message to it Pro cess p could only

j j

have deferred the OK b ecause it was either

in C but since p eventually leaves C itmust eventually send the deferred OK

in T in this case p deferred the OK b ecause the timestamp of its request was smaller

than p s Since p s request has the smallest timestamp in T now p must have

i i j

completed thus after exiting C it had to send the deferred OK

In either case we get a contradiction

Fairness it can b e shown that the algorithm has lo ckoutfreedom

Complexity it is easy to see that there are n messages p er request The time

complexity is left as an exercise

Carvalho Roucairol

This algorithm improves on RicartAgrawala by giving a dierentinterpretation to the OK

message When some pro cess p sends an OK to some other pro cess p not only do es it

i j

approve p s current request but it also gives p p s p ermission to reenter C again and

j j i

again until p sends an OK to p in resp onse to a try from p

j i i

This algorithm p erforms well under light load When a single pro cess is requesting again

and again with no other pro cess interested it can go critical with no messages sent Under

heavy load however it basically reduces to Ricart and Agrawalas algorithm

This algorithm is closely related to ie is generalized by the ChandyMisra Dining

Philosophers algorithm

General Resource Allo cation

Wenow turn to consider more general resource allo cation problems in asynchronous net

works as dened earlier

BurnsLynch Algorithm

This algorithm solves conjunctive resource problems Recall the strategies studied in asyn

chronous shared memory setting involving seeking forks in order and waiting for resources

one at a time We can run the same strategies in asynchronous networks The b est waytodo

this is not via a general simulation of the shared memory mo del in the network mo del recall

that the algorithms involvebusywaiting on shared variables if simulated each busywaiting

step would involve sending messages Instead wehave each resource represented by its own

pro cess whichmust b e lo cated at some no de The user pro cess sends a message to the

resource pro cess for the rst resource it needs the resource pro cess puts the user index on

its queue the user pro cess then waits When the user index reaches the front of the queue

the resource pro cess sends a message back to that user pro cess whichthengoesontowait

for its next resource etc When a pro cess gets all its resources it tells its external user to

go critical when exit o ccurs the user pro cess sends messages to all the resource pro cesses

to remove its index from the queue

The analysis is similar to that done in the shared memory case and dep ends only on

lo cal parameters The general case has exp onential dep endence on the numb er of colors

There are some variations in the literature to improve the running time This is still a topic

of current research see eg StyerPeterson AwerbuchSaks ChoySingh

Drinking Philosophers

This is a dynamic variant of the general conjunctive static resource allo cation problem

In this setting wehave the same kind of graph as b efore where a pro cess is connected to

everyone with whom it might have a resource conict But now when a try request arrives

from the outside it contains an extra parameter a set B of resources indicating which

resources the pro cess actually needs this time This set must b e a subset of the static set for

that pro cess the twist is that now it neednt b e the whole set

As b efore wewant to get exclusion but this time based on the actual resources b eing

requested Wewould liketohave also deadlo ckfreedom and lo ckoutfreedom The concur

rency condition is now mo died as follows Supp ose that a request is invoked at no de i and

during the interval of this request there are no actual conicting requests Supp ose the

execution pro ceeds so that i and all its neighb ors in the underlying graph continue to take

steps fairly and the connecting links continue to op erate fairly note that some pro cesses

might remain in C Then eventually i reaches C

Chandy and Misra gave a solution to this whichwas based on a particular inecient

Dining Philosophers algorithm Welch and Lynch noticed that the ChandyMisra algorithm

made almostmo dular use of the underlying Dining Philosophers algorithm and so they re

did their result with the implicit mo dularity made explicit Then it was p ossible to substitute

a more ecient Dining Philosophers algorithm for the original ChandyMisra subroutine

The prop osed architecture is sketched in Figure The Dining Philosophers subroutine

executes using its own messages as usual In addition there are new messages for the

Drinking Philosophers algorithm

We assume for simplicity here that each resource is shared b etween only two pro cesses

The heart of the algorithm is the D automata First we describ e the algorithm informally

and then presenttheD automaton co de

When D enters its trying region needing a certain set of resources it sends requests

for those resources that it needs but lacks A recipient D of a request satises the request

unless D currently also wants the resource or is already using it In these two cases D

j j

defers the request so that it can satisfy it when it is nished using the resource

In order to prevent drinkers from deadlo cking a Dining Philosophers algorithm is used

as a subroutine The resources manipulated by the Dining Philosophers subroutine are User i User j

T C ER TC ER

Di send receive Dj Drinking Drinking Philosophers Philosophers algorithm receive send algorithm R T T C E C E R

Dining Philosophers algorithm

Figure Architecture for the Drinking Philosophers problem

priorities for the real resources there is one dining resource for each drinking resource

As so on as D is able to do so in its drinking trying region it enters its dining trying region

that is it tries to gain priority for its maximum set of resources If D ever enters its dining

critical region while still in its drinking trying region it sends demands for needed resources

that are still missing A recipient D of a demand must satisfy it even if D wants the

j j

resource unless D is actually using the resource In that case D defers the demand and

j j

satises it when D is through using the resource

Once D is in its dining critical region we can showthatiteventually receives all its

needed resources and never gives them up Then it mayenter its drinking critical region

Once D enters its drinking critical region it may relinquish its dining critical region since

the b enets of having the priorities are no longer needed Doing so allows some extra con

currency even if D stays in its drinking critical region forever other drinkers can continue

to make progress

A couple of p oints ab out the co de deserve explanation We can show that when a request

is received the resource is always at the recipient thus it is not necessary for the recipient

to check that it has the resource b efore satisfying or deferring the request On the other

hand it is p ossible for a demand for a missing resource to b e received so b efore satisfying

or deferring a demand the recipientmust check that it has the resource

Another p oint concerns the questions when the actions of the dining subroutine should b e

p erformed Note that some drinkers could b e lo cked out if D never relinquishes the dining

critical region The reason for this is that as long as D is in its critical region it has priority

for the resources Thus if we are not careful D could cycle through its drinking critical

region innitely many times while other drinkers wait Toavoid this situation wekeep

track using variable current of whether the dining critical region was entered on b ehalf

of the current drinking trying region ie whether the latest C o ccurred after the latest

T B If it was then D mayenter its drinking critical region assuming it has all the

i i

needed resources Otherwise D imust wait until the current dining critical region has

b een relinquished b efore continuing

The full co de is given in Figures and

Correctness pro of Exclusion is straightforward based on p ossession of the resources

since we use explicit tokens Concurrency is also straightforward Toprovelockoutfreedom

we need to rely on the DiningPhilosophers subroutine The pro of is a bit subtle First

we argue that the environment of the dining philosophers algorithm preserves diningwell

formedness this is proved by a straightforward explicit check of the co de Therefore every

execution of the system is diningwellformed since the dining subroutine also preserves

diningwellformedness Therefore the dining subroutine provides its guarantees dining

exclusion and lo ckoutfreedom for the dining subroutine in the context of the drinking

algorithm

Next we show the following lemma

Lemma In any fair drinkingwel l formedexecution if drinking users always return the

resources then dining users always return the resources

Pro of Sketch Supp ose that the dining resources are granted to D Then D sends

i i

demands Consider any recipient D of a demand If D has the resource and is not actually

j j

using it then it gives it to i b ecause by the exclusion prop erty of the dining subroutine D

cannot also b e in its dining critical region On the other hand if D is using the resource

then by the assumption that no drinker is stuck in its critical region D eventually nishes

and satises the demand Thus eventually D gets all the needed resources and enters its

drinking critical region after whichby the co de it returns the dining resources

With this we can prove the following prop erty

Lemma In any fair drinkingwel l formedexecution if drinking users always return the

resources then every drinking request is granted

Pro of Sketch Once a drinking request o ccurs sayatnodei then unless the grant o ccurs

immediately a subsequent dining request o ccurs at i Then by dining lo ckoutfreedom and

Lemma eventually the dining subroutine grants at i Again by Lemma eventually i

returns the dining resource But note that i only returns the dining resource if in the interim

it has done a drinking grant The formal pro of is byacontradiction as usual

State

drinkregion equals T if the most recent drinking action was T B for some B C if

C B etc Initially R

dineregion equals T if the most recent dining action was T C if C etc Initially R

i i

needequalsB where the most recent drinking action had parameter B Initially

empty

bottles set of tokens that have b een received and not yet enqueued to b e sent Initially

the token for a resource shared b etween D and D is in the bottles variable of exactly

i j

one of D and D with the choice b eing made arbitrarily

i j

deferred set of tokens in the set bottles for which a request or demand has b een received

since the token was received Initially empty

current Bo olean indicating whether current dining critical region is on b ehalf of current

drinking trying region Initially false

msgsj for all j i FIFO queue of messages for D enqueued but not yet sent Initially

empty

Actions for pro cess i

try B for all B B

Eect drinkregion T

need B

for all j i and b need B bottles

enqueue requestbin msgsj

send m j for all j i m frequestb tok enb demandb b B B g

i i j

Precondition m is at head of msgsj

Eect dequeue m from msgsj

try

Precondition dineregion R

drinkregion T

Eect dineregion T

Figure Co de for the mo dular drinking philosophers algorithmpart I

Actions for pro cess i cont

Receive r eq uestbj for all j i b B B

i i j

Eect if b need and drinkregion fT Cg then

deferred deferred fbg

else

enqueue tokenbin msgsj

bottles bottles fbg

crit

Eect dineregion C

if drinkregion T then

for all j i and b need B bottles enqueue demandb in msgsj

current true

Receive demandbj for all j i b B B

i i j

Eect if b bottles and b need ordrinkregion C then

enqueue tokenbin msgsj

bottles bottles fbg

deferred deferred fbg

Receive tok enbj for all j i b B B

i i j

Eect bottles bottles fbg

crit B for all B B

i i

Precondition drinkregion T

B need

need bottles

if dineregion C then current true

Eect drinkregion C

current false

exit

Precondition dineregion C

if drinkregion T then current false

Eect dineregion E

Figure Co de for the mo dular drinking philosophers algorithmpart I I

Actions for pro cess i cont

rem

Eect dineregion R

exit B for all B B

i i

Eect drinkregion E

for all j i and b deferred B

enqueue tokenbin msgsj

bottles bottles deferred

deferred

rem B for all B B

i i

Precondition drinkregion E

B need

Eect drinkregion R

Figure Co de for the mo dular drinking philosophers algorithmpart I I I

We remark that the complexity b ounds dep end closely on the costs of the underlying

dining algorithm

Stable Prop erty Detection

In this section we consider a new problem Supp ose wehave some system of IO automata as

usual and supp ose wewant to sup erimp ose on this system another algorithm that monitors

the given algorithm somehow For instance the monitoring algorithm can

detect when the given system has terminated execution

check for violation of some global invariants

detect some kind of deadlo ck where pro cesses are waiting for each other in a cycle to

do something

compute some global quantity eg the total amount of money

We shall see some p owerful techniques to solvesuch problems

Termination for Diusing Computations

First we consider the termination problem alone DijkstraScholten gave an ecient algo

rithm for the sp ecial case where the computation originates at one no de In this kind of

computation called diusing computation all no des are originally quiescent dened here to

mean that they are not enabled to do any lo cally controlled actions The algorithm starts

with an input from the outside world at one of the no des and we assume that no further

inputs o ccur According to the IOA denitions once a no de receives such a message it can

change state in sucha way that it now is enabled to do some lo callycontrolled actions eg

to send messages to other no des The no des can also do various outputs to the outside

world When other no des receive these messages they in turn might also b ecome enabled

to do some lo callycontrolled actions and so on

The termination detection problem is formulated as follows

If the entire system ever reaches a quiescent state ie all the pro cesses are

in quiescent states and there are no messages in transit then eventually the

originator no de outputs a sp ecial message done

Of course the original algorithm do es not havesuch an output The new system con

structed has to b e an augmentation of the given system where each no de automaton is

obtained from the old automaton in some allowed way

Belowwe state informally what is allowed to solve this problem The precise denition

should b e similar to the superposition denition describ ed in Chandy and Misras Unity

work

Wemay add new state comp onents but the pro jection of the start states of the aug

mented machine must b e exactly the start states of the original machine

The action set can b e augmented with new inputs outputs and internal actions The

old actions must remain with the same preconditions but they mayhave new eects

on the new comp onents they mightalsohave additional information piggybacked on

them eg send mnowdoessend m c The old actions remain in the same classes

The new input actions aect the new comp onents only

The new output and internal actions can b e preconditioned on the entire state old and

new comp onents but they may only aect the new comp onents They get group ed

into new fairness classes

The idea in the DijkstraScholten termination detection algorithm is to sup erimp ose a

kind of spanningtree construction on the existing work of the underlying algorithm Start

ing from the initial no de every message of the underlying algorithm gets a tree message

piggybacked up on it Just as in the asynchronous algorithm for constructing a spanning

tree each no de records the rst receipt of a tree message in its state as its parent Anysub

sequent receipts of tree messages are immediatelyacknowledged they are not discarded as

in the spanning tree proto col explicit acknowledgment is sentback Only the rst one gets

held and unacknowledged for now Also if the initial no de ever receives a tree message it

immediately acknowledges it So as the messages circulate around the network a spanning

tree of the no des involved in the proto col gets set up

Now we are going to let this tree converge back up on itself as in convergecast in order

to rep ort termination back to the initial no de Sp ecicallyeach node looks for when both

of the following hold at the same time a its underlying algorithm is in a quiescent state

and b all its outgoing tree messages have b een acknowledged When it nds this it closes

up the shop it sends an acknowledgment to its parent and forgets everything ab out

this proto col Note that this is dierent from what had to happ en for ordinary broadcast

convergecast There we had to allow the no des to rememb er that they had participated in

the algorithm marking themselves as done so they didnt participate another time If

we hadnt the no des would just continue broadcasting to each other rep eatedly This time

however we do allow the no des to participate anynumb er of times whether or not they do

so is controlled by where the messages of the underlying algorithm get sent

Example Supp ose the no des are and consider the following scenario see Figure

for illustration i No de wakes up sends a message to and sends a message to

ii No des and receive the messages and set their parentpointers to p ointtonode

Then they b oth wake up and send messages to each other but since each no de already has

a parent it will just send an ack Then b oth no des send messages to iii Supp ose s

message gets to no de rst then is set to b e the parent of and therefore acknowledges

immediately The four no des continue for a while sending messages to each other they

immediately acknowledge each message

Now supp ose the basic algorithm at quiesces Still do es not close up b ecause it has an

unacknowledged message to iv Supp ose now quiesces it then is ready to acknowledge

to and it forgets it ever participated in the algorithm actually it forgets everything

When receives this ack and since is also done it sends ack to and forgets everything

v Now supp ose sends out messages to and When they receive these messages they

wake up as at the b eginning continue carrying out the basic algorithm and reset their

parent p ointers this time to

This execution can continue in this fashion indenitely But if all the no des ever qui

esce and there are no messages of the basic algorithm in transit then the no des will all

succeed in converging back to the initial no de in this case When is quiesced and has (i) (ii) (iii) 2 2 2

0 3 0 3 0 3

1 1 1

(iv) (v) 2 2

0 3 0 3

1 1

Figure Example of an execution of the termination detection algorithm The arrows on

the edges indicate messages in transit and the arrows parallel to the edges indicate parent

p ointers

acknowledgments for all its outgoing messages it can announce termination

Supp ose the underlying algorithm is A In Figures and wegivecodeforthe

sup erimp osed version p

Wehave argued roughly ab ove that if the basic algorithm terminates this algorithm

eventually announces it We can also argue that this algorithm never do es announce termi

nation unless the underlying system has actually terminated We can prove this using the

following invariant cf homework

Invariant Al l nonid le nodes form a treerooted at the node with status leader with the

treeedges given by parent pointers

FrancezShavit have generalized this algorithm to the case where the computation can

initiate at several lo cations the idea there is to use several trees and wait for them all to

terminate

Complexity The numb er of messages is prop ortional to the numb er of messages of

underlying algorithm A nice prop erty of this algorithm is its lo cality if the diusing

computation only op erates for a short time in a small area of the network the termination

proto col only incurs prop ortionately small costs On the other hand if the algorithm works

for a very long time b efore terminating then the termination detection algorithm requires

considerable overhead

New state comp onents

status fid le leader nonleaderg initially id le

parent anodeornil initially nil

for eachneighbors j

send j a queue of messages initially empty

decit j an integer initially

Actions

any input action of no de i excluding a receive

New Eect

status leader

send m

New Eect decit j decit j

New action receive ack

New Eect decit j decit j

receive m where m a message of A

ji i

New Eect if status id le then

status nonleader

parent j

else add ack to send j

New action closeupshopi

Precondition status nonleader

j parent i

send k for all k

decit k forall k

state of A is quiescent

Eect add ack to send

status id le

parent nil

Figure Co de for DijkstraScholten termination detection algorithm part I

Actions cont

New action send ack

Precondition ack rst on send j

Eect remove rst message from send j

New action done

Precondition status leader

send k for all k

decit k forall k

state of A is quiescent

Eect status id le

Figure Co de for DijkstraScholten termination detection algorithm part I I

Snapshots

Problem Statement

The snapshot problem is to obtain a consistent global state of a running distributed algo

rithm By global state we mean a state for each pro cess and eachFIFOchannel The

consistency roughly sp eaking requires that the output is a global state that could have

o ccurred at a xed momentintime

We make the denition of the problem more precise using the notion of logical time

discussed in the last lecture Recall the denition of logical time it is p ossible to assign a

unique time to eachevent in a way that is increasing at anygiven no de and suchthat sends

precede receives and only nitely manyevents precede any particular event Clearly a given

execution can get many logical time assignments A consistent global state is one that arises

from any particular logical time assignment and a particular real time tFor this assignment

and this t there is a welldened notion of what has happ ened up through and including

time tWewant to include exactly this information in the snapshot states of no des after

the lo cal events up through time t and the states of channels at time t ie those messages

sentby time t and not received by time t in the order of sending See Figure for an

example

We can stretch and shrink the execution to align corresp onding times at dierent no des

and the information in snapshot is just the states of no des and channels at the tcut

Because we mighthave to shrink and stretch the execution it is not necessarily true that

the state which is found actually o ccurs at any p oint in the execution But in a sense this

Figure t is a logical time cut

do esnt matter it could have o ccurred ie no pro cess can tell lo cally that it didnt o ccur

Actuallywewould like more than just any consistent global state wewould like one

that is fairly recent eg wewant to rule out the trivial solution that always returns the

initial global state For instance one mightwant a state that reects all the events that

have actually o ccurred in real time b efore the invo cation of the snapshot algorithm So we

supp ose that the snapshot is initiated by the arrival of a snap input from the outside world

at some nonzero numb er of the no des but only once at eachnodeWe do not imp ose the

restriction that the snapshot must originate at a single no de as in the diusing computation

problem Also the underlying computation is not necessarily diusing

Applications

A simple example is a banking system where the goal is to count all the money exactly

once More generallywehave the problem of stable property detection Formallywesay

that a prop erty of global states is stable if it p ersists ie if it holds in some state then it

holds in any subsequent state For instance termination the prop erty of all the no des b eing

in quiescent states and the channels b eing empty is a stable prop erty Also unbreakable

deadlo ck where a cycle of no des are in states where they are waiting for each other is a

stable prop erty

The Algorithm

The architecture is similar to the one used ab ove for the terminationdetection algorithm

the underlying A is augmented to give p with certain additions that do not aect the

i i

underlying automatons b ehavior

Wehave already sketched one such algorithm based on an explicit logical time assign

ment Wenow give another solution by Chandy and Lamp ort The algorithm is very much

like the snapshot based on logical time only it do es away with the explicit logical time

The idea is fairly simple Each no de is resp onsible for snapping its own state plus the

states of the incoming channels When a no de rst gets a snap input it takes a snapshot of

the state of the underlying algorithm Then it immediately puts a marker message in each

of its outgoing channels and starts to record incoming messages in order to snap eachofits

incoming channels This marker indicates the dividing line b etween messages that were sent

out b efore the lo cal snap and messages sent out after it For example if this is a banking

system and the messages are money the money b efore the marker was not included in the

lo cal snap but the money after the marker was

A no de that has already snapp ed just records these incoming messages until it encounters

the marker in which case it stops recording Thus the no de has recorded all the messages

sentby the sender b efore the sender did its lo cal snap Returning to the banking system

example in this case the recipient no de has counted all the money that was sent out b efore

the sender did its snap and thus not counted by the sender

There is one remaining situation to consider supp ose that a no de receives a marker

message b efore it has done its own lo cal snap b ecause it has not received a snap input

Then immediately up on receiving the rst marker the recipient snaps its lo cal state sends

out its markers and b egins recording the incoming channels The channel up on whichithas

just received the marker is recorded as emptyhowever

Mo deling It is not completely straightforward to mo del a snapshot system The subtlety

is in the atomicity of the snapping and marker sending steps Todothismuch atomically

we seem to need to mo dify the underlying algorithm in slightly more complicated ways than

for the DijkstraScholten algorithm p ossibly b yblocking some messages from b eing sent

by A while the markers are b eing put into the channels This really means interfering with

the underlying algorithm but roughly sp eaking wedonotinterfere bytoomuch

The co de is given in Figure

Correctness We need to argue that the algorithm terminates and that when it do es it

gives a consistent global state corresp onding to some xed logical time

Termination We need strong connectivitytoprove termination As so on as the snap

input o ccurs the recipient no de do es a lo cal snapshot and sends out markers Every time

anyone gets a marker it snaps its state if it hasnt already The markers thus eventually

propagate everywhere and everyone do es a lo cal snapshot Also eventually the no des will

nish collecting the messages on all channels when they receive markers

Consistency We omit details The idea is to assign a logical time to eachevent of the

underlying system by means of such an assignment for the complete system We do this in

New state comp onents

status fstart snapping reported g initially start

snapstate a state of A or nil initially nil

for eachneighbor j channelsnappedj a Bo olean initially false and

send j snapchannelj FIFO queues of messages initially empty

Actions

New action snap

Eect if status start then

snapstate gets the state of A

status snapping

for all j neighbors add marker to send j

Replace send m actions of A with internalsend actions which put m at the end of send j

ij i

New action send m

Precondition m is rst on send j

Eect remove rst elementofsend j

receive m m a message of A

ji i

New Eect if status snapping and channel snapped j false then

add m to snapchannelj

New action receive marker

Eect if status start then

snapstate gets the state of A

status snapping

for all j neighbors add marker to send j

channelsnappedj true

New action reportsnapshots C where s is a no de state and C is the states of incoming links

Precondition status snapping

for all j neighbors channelsnappedj true

s snapstate

for all j neighbors C j snapchannelj

Eect status reported

Figure The snapshot algorithm co de for pro cess i

suchaway that the same time t with ties broken by indices is assigned to eacheventat

which a no de snaps its lo cal state The fact that we can do this dep ends on the fact that

there is no message sent at one no de after the lo cal snap and arriving at another no de b efore

its lo cal snap If a message is sent after a lo cal snap it follows the marker on the channel

then when the message arrives the marker will have already arrived and so the recipient

will have already done its snapshot It is obvious that what is returned byeachnodeis

the state of the underlying algorithm up to time tWemust also check that the channel

recordings give exactly the messages in transit at logical time t ie the messages sent

after the senders snap and received b efore the receivers snap

Some simple examples We again use the bank setting Consider a twodollar twono de

banking system where initially eachnode i and j has one dollar The underlying algorithm

has some program it do esnt matter what it is that determines when it sends out some

money Consider the following execution see Figure

# 1 # 1 # 1

1 (1)1 0 (2)1 1 (3)0 1 (4) 0

ijijijij#

1 0

ij1

Figure Execution of a twono de bank system The nal snapshot is depicted b elow

the intermediate states

snap o ccurs i records the state of the bank puts a marker in the buer to send

to j and starts recording incoming messages

i puts its dollar in the buer to send to j b ehind the marker

j sends its dollar to i i receives it and records it in its lo cation for recording incoming

messages

j receives the marker from i records its lo cal bank state puts a marker in the

buer to send to i and snaps the state of the incoming channel as empty

i receives the marker snaps the state of the channel as the sequence consisting of one

message the dollar it received b efore the marker

Put together the global snapshot obtained is one dollar at i one dollar in channel from

j to i and the other no de and channel empty This lo oks reasonable in particular the

correct total is obtained But note that this global state never actually arose during the

computation However we can show a reachability relationship Supp ose that is the

actual execution of the underlying bank algorithm Then can b e partitioned into

where indicates the part b efore the snapshot starts and indicates the part after the

snapshot ends Then the snapp ed state is reachable from the state at the end of and the

state at the b eginning of is reachable from the snapp ed state So the state could have

happ ened as far as anyone b efore or after the snapshot could tell

In terms of Lamp orts partial order consider the actual execution of the underlying

system depicted in Figure a We can reorder it while preserving the partial order so that all the states recorded in the snapshot are aligned as in Figure b

Process i Process j Process i Process j logical time−cut time−cut

(a) (b)

Figure a is the actual execution of Figure and b is a p ossible orderings of

it where the logical time cut of a is a real time cut

Applications revisited

Stable property detection

A stable prop erty is one that if it is true of any state s of the underlying system is

also true in all states reachable from s Thusifitever b ecomes true it stays true The

following is a simple algorithm to detect the o ccurrence of a stable prop erty P First the

algorithm do es a global snapshot and then either it collects the information somewhere and

returns P result or else it do es a distributed algorithm on the static information recorded

from the snapshot to determine P result The correctness conditions of the snapshot the

reachabilityversion of those conditions that is imply the following If the output is true

then P is true in the real state of the underlying algorithm at the end of the snapshot and

if the output is false then P is false in the real state of the underlying algorithm at the

b eginning of the snapshot There is some uncertainty ab out the interim

Example Termination detection Assume a system of basic pro cesses with no external

inputs but the pro cesses dont necessarily b egin in quiescent states This will execute on its

own for a while and then might quiesce terminate More precisely quiescence refers to all

no des b eing quiescent and no messages in transit anywhere Wewant to b e able to detect

this situation since it means that the system will p erform no further action since there are

no inputs Quiescence of global states is a stable prop erty assuming no inputs So here

is a simple algorithm to detect it do a snapshot send the states somewhere and checkif

anything is enabled or in transit Actuallywe dont need to send the whole state to one

pro cess we can haveeveryone check termination lo cally and fan in ie convergecast the

results to some leader along a spanning tree We only need to convergecast a bit saying if

the computation is done in the subtree It maybeinteresting to contrast this algorithm

to the DijkstraScholten termination detection the snapshot algorithm involves the whole

network ie it is not lo cal at all But on the other hand if it only has to b e executed once

then the overhead is b ounded regardless of how long the underlying algorithm has gone on

for

Example Dead lock detection Now the basic algorithm involves pro cesses that are wait

ing for other pro cesses eg it can b e determined from the state of pro cess A that it is

waiting for pro cess A say to release a resource The exact formulation varies in dierent

situations eg a pro cess mightbewaiting for resources owned by the other pro cesses etc

In all formulations however deadlo ck essentially amounts to a waiting cycle If we supp ose

that every pro cess that is waiting is stopp ed and while it is stopp ed it will never release

resources to allowanyone else to stop waiting then deadlo ck is stable Thus we can detect

it by taking snapshots sending the information to one place then doing an ordinary cen

tralized cycledetection algorithm Alternatively after taking the snapshot we can p erform

some distributed algorithm to detect cycles on the static data resulting from the snapshot

JJ Distributed Algorithms December

Lecturer George Varghese

Lecture

Designing network proto cols is complicated by three issues parallelism asynchrony and

faulttolerance This course has already covered techniques for making proto cols robust

against Byzantine faults The Byzantine fault mo del is attractive b ecause it is general the

mo del allows a faulty pro cess to continue to b ehave arbitrarily In this lecture we will study

another general fault mo del the selfstabilization mo del whic h is attractive from b oth a

theoretical and practical viewp oint

We will compare the two mo dels a little later but for now lets just saythatself

stabilization allows an arbitrary numb er of faults that stop while Byzantine mo dels allowa

limited numb er of faults that continue Stabilizing solutions are also typically muchcheap er

and have found their wayinto real networks In this lecture we will describ e selfstabilization

using twonetwork mo dels an elegan t shared memory mo del and a more practical network

mo del We will also describ e three general techniques for making proto cols stabilizing local

checking and correction counter ushing and timer ushing

The Concept of SelfStabilization

Do or Closing and Domain Restriction

Todaywe will fo cus on the abilityofnetwork proto cols to stabilize to correct b ehavior

after arbitrary initial p erturbation This prop ertywas called selfstabilization by Dijkstra

Dijkstra The self emphasizes the ability of the system to stabilize by itself without

manual intervention

A story illustrates the basic idea Imagine that you live in a house in Alaska in the

middle of winter You establish the following proto col set of rules for p eople who enter and

leaveyour house Anyb o dy who leaves or enters the house must shut the do or after them If

the do or is initially shut and nob o dy makes a mistake then the do or will eventually return

to the closed p osition Supp ose however that the do or is initially op en or that someb o dy

forgets to shut the do or after they leave Then the do or will stayopenuntil someb o dy passes

through again This can b e a problem if heating bills are exp ensive and if several hours can

go by b efore another p erson go es through the do or It is often a go o d idea to make the do or

STEP 1: ENTERING STEP 2: MIDDLE OF THE DOOR STEP 3: OUT AT LAST!

Figure Exiting through a revolving do or

closing proto col selfstabilizing This can b e done by adding a spring or automatic do or

closer that constantly restores the do or to the closed p osition

In some cases it is p ossible to trivialize the problem by hardwiring relationships b etween

variables to avoid illegal states suc h a technique is actually used in a revolving do or A

p erson enters the revolving do or Figure gets into the middle of the do or and nally

leaves It is physically imp ossible to leave the do or op en and yet there is a waytoexit

through the do or

This technique whichwe will call domain restriction is often a simple but p owerful

metho d for removing illegal states in computer systems that contain a single shared memory

Consider two pro cesses A and B that have access to a common memory as shown in the rst

part of Figure Supp ose wewant to implementmutual exclusion by passing a token

between A and B

One way to implement this is to use two boolean variables token and token Topass

A B

the token Pro cess A sets token to false and sets token to true In a selfstabilizing

A B

setting however this is not a go o d implementation For instance the system will deadlo ck

if token tok en false in the initial state The problem is that wehave some extra and

A B

useless states The natural solution is to restrict the domain to a single bit called turn such

that turn when A has the token and turn when B has the token By using domain

restriction we ensure that any possible state is also a legal state

It is often feasible to use domain restriction to avoid illegal states within a single node

of a computer network Domain restriction can b e implemented in manyways The most

In this example we are really changing the domain However we prefer the term domain restriction token A token token token A B B turn

PROCESS PROCESS PROCESS PROCESS A B A B

1. WITH SHARED MEMORY 2. WITHOUT SHARED MEMORY

Figure Token passing among two pro cesses

natural wayisby restricting the numb er of bits allo cated to a set of variables so that every

p ossible value assigned to the bits corresp onds to a legal assignmentofvalues to eachofthe

variables in the set Another p ossibility is to mo dify the co de that reads variables so that

only values within the sp ecied domain are read

Unfortunately domain restriction cannot solve all problems Consider the same two pro

cesses A and B that wish to achievemutual exclusion This time however see Figure

Part A and B areattwo dierent no des of a computer network The only way they can

communicate is by sending token messages to each other Thus we cannot use a single turn

variable that can b e read by b oth pro cesses In fact A must have at least two states a state

in which A has the token and a state in which A do es not have the token B must also have

two such states Thus we need at least four combined states of whichtwo are illegal

Thus domain restriction at eachnode cannot prevent il legal combinations across nodes

We need other techniques to detect and correct illegal states of a network It should b e no

surprise that the title of Dijkstras pioneering pap er on selfstabilization Dijkstra was

SelfStabilization in spite of Distributed Control

Figure A typical mesh Network

SelfStabilizati on is attractive for Networks

We will explore selfstabilization prop erties for computer networks and network proto cols

A computer network consists of no des that are interconnected bycommunication channels

The network top ology see Figure is describ ed by a graph The vertices of the graph

represent the no des and the edges representthechannels No des communicate with their

neighbors by sending messages along channels Many real networks such as the ARPANET

DECNET and SNA can b e mo deled in this way

A network proto col consists of a program for each network no de Each program consists

of co de and inputs as well as lo cal state The global state of the network consists of the

lo cal state of eachnodeaswell as the messages on network links We dene a catastrophic

fault as a fault that arbitrarily corrupts the global network state but not the program co de

or the inputs from outside the network

Selfstabilization formalizes the following intuitive goal for networks despite a history

of catastrophic failures oncecatastrophic failures stop the system should stabilize to correct

behavior without manual intervention Thus selfstabilization is an abstraction of a strong

faulttolerance prop erty for networks It is an imp ortant prop ertyofrealnetworks b ecause

Catastrophic faults o ccur Most network proto cols are resilient to common failures

such as no des and link crashes but not to memory corruption But memory corruption

do es happ en from time to time It is also hard to prevent a malfunctioning device from

sending out an incorrect message

Manual intervention has a high cost In a large decentralized network restoring

the network manually after a failure requires considerable co ordination and exp ense

Thus even if catastrophic faults o ccur rarely sayonceayear there is considerable

incentive to makenetwork proto cols selfstabilizing A reasonable guideline is that the

network should stabilize preferably before the user notices and at least before the user

logs a servicecal l

These issues are illustrated by the crash of the original ARPANET proto col Rosen

Perlman The proto col was carefully designed never to enter a state that contained

three conicting up dates a band c Unfortunately a malfunctioning no de injected three

such up dates into the network and crashed After this the network cycled continuously

between the three up dates It to ok days of detectivework Rosen b efore the problem was

diagnosed With hindsight the problem could havebeenavoided by making the proto col

selfstabilizing

Selfstabilization is also attractive b ecause a selfstabilizing program do es not require

initialization The concept of an initial state makes p erfect sense for a single sequential

program However for a distributed program an initial state seems to b e an articial concept

Howwas the distributed program placed in such an initial state Did this require another

distributed program Selfstabilization avoids these questions by eliminating the need for

distributed initialization

Criticisms of SelfStabili zation

Despite the claims of the previous section there are p eculiar features of the selfstabilization

mo del that need justication

The mo del allows network state to b e corrupted but not program co de However

program co de can b e protected against arbitrary corruption of memory by redundancy

since co de is rarely mo died On the other hand the state of a program is constantly

b eing up dated and it is not clear how one can prevent illegal op erations on the memory

by using checksums It is even harder to prevent a malfunctioning no de from sending

out incorrect messages

The mo del only deals with catastrophic faults that stopByzantine mo dels deal with

continuous faults However in Byzantine mo dels only a fraction of no des are allowed

to exhibit arbitrary b ehavior In the selfstabilization mo del al l no des are p ermitted

to start with arbitrary initial states Thus the two mo dels are orthogonal and can

even b e combined

A selfstabilizing program P is is allowed to make initial mistakes However the

imp ortant stabilizing proto cols that we know of are used for routing scheduling and

resource allo cation tasks For such tasks initial errors only result in a temp orary loss

of service

Denitions of Stabilization

Execution Denitions

All the existing denitions of stabilization are in terms of the states and executions of a

system We will b egin with a denition of stabilization that corresp onds to the standard

denitions for example that of Katz and Perry KP Next we will describ e another

denition of stabilization in terms of external b ehaviors Webelieve that the denition of

b ehavior stabilization is appropriate for large systems that require mo dular pro ofs However

the denition of execution stabilization given b elow is essential in order to prove results ab out

b ehavior stabilization

Supp ose we dene the correctness of an automaton in terms of a set C of legal executions

For example for a token passing system we can dene the legal executions to b e those in

which there is exactly one token in every state and in whichevery pro cess p erio dically

receives a token

What do we mean when wesay that an automaton A stabilizes to the executions in set

C Intuitivelywe mean that eventually all executions of A b egin to lo ok like an execution

in set C For example supp ose C is the set of legal executions of a token passing system

Then in the initial state of A there may b e zero or more tokens However the denition

requires that eventually there is some sux of any execution of A in which there is exactly

one token in any state

Formally

Denition Let C be a set of executions We say that automaton A stabilizes to the

executions in C if for every execution of A there is some sux of execution that is in

We can extend this denition in the natural way to dene what it means for an automaton

A to stabilize to the executions of another automaton B

Denitions of Stabilization based on External Behavior

In the IO Automaton mo del the correctness of an automaton is sp ecied in terms of its

external b ehaviors Thus we sp ecify the correctness of a token passing system without any

reference to the state of the system We can do so by sp ecifying the ways in which token

delivery and token return actions to and from some external users of the token system can

be interleaved

Thus it natural to lo ok for a denition of stabilization in terms of external b ehaviors We

would also hop e that such a denition would allow us to mo dularly comp ose results ab out

the stabilization of parts of a system to yield stabilization results ab out the whole system

NowanIOA A is said to solve a problem P if the b ehaviors of A are contained in P For

stabilization however it is reasonable to weaken this denition and ask only that an IOA

eventual ly exhibit correct b ehavior Formally

Denition Let P beaproblem ie a set of behaviors AnIOA A stabilizes to the

behaviors in P if for every behavior of A there is a sux of that is in P

Similarlywe can sp ecify that A stabilizes to the b ehaviors of some other automaton B

in the same wayThebehavior stabilization denition used in this section and V is a

sp ecial case of a denition suggested by Nancy Lynch

Finallywehave a simple lemma that ties together the execution and b ehavior stabi

lization denitions It states that execution stabilization implies b ehavior stabilization In

fact the only metho d we knowto prove abehavior stabilization result is to rst provea

corresp onding execution stabilization result and then use this lemma Thus the b ehavior

and execution stabilization denitions complementeach other in this thesis the former is

typically used for specication and the latter is often used for proofs

Lemma If IOA A stabilizes to the executions of IOA B then IOA A stabilizes to the

behaviors of B

Discussion on the Stabilization Denitions

First notice that wehave dened what it means for an arbitrary IOA to stabilize to some

target set or automaton Typicallywewillbeinterested in proving stabilization prop erties

only for a sp ecial kind of automata unrestricted automata An unrestricted IOAUIOA is

one in which all states of the IOA are also start states SuchanIOA mo dels a system that

has b een placed in an arbitrary initial state by an arbitrary initial fault

A p ossible mo dication which is sometimes needed is to only require in Denitions

and that the sux of a b ehavior execution b e a sux of a b ehavior execution of

the target set One problem with this mo died denition is that weknow of no go o d pro of

technique to prove that the b ehaviors executions of an automaton are suxes of a sp ecied

set of b ehaviors executionsBy contrast it is much easier to provethatevery b ehavior of an

automaton has a sux that is in a sp ecied set Thus we prefer to use the simpler denitions

for what follows

Examples from Dijkstras Shared Memory Mo del

In Dijkstras Dijkstra mo del a network proto col is mo deled using a graph of nite state

machines In a single move a single no de is allowed to read the state of its neighb ors

compute and then p ossibly change its state In a real distributed system such atomic

communication is imp ossible Typically communication has to pro ceed through channels

While Dijkstras original mo del is not very realistic it is probably the simplest mo del of an

asynchronous distributed system This simple mo del provided an ideal vehicle for introducing

Dijkstra the concept of stabilization without undue complexityFor this section we will

use Dijkstras original mo del to introduce two imp ortant and general techniques for self

stabilization local checking and correction and counter ushing In the next section we will

intro duce a more realistic message passing mo del

Dijkstras Shared Memory Mo del and IOA

How can we map Dijkstras mo del into the IOA mo del Supp ose each no de in Dijkstras

mo del is a separate automaton Then in the InputOutput automata mo del it is not p ossible

to mo del the simultaneous reading of the state of neighb oring no des The solution we use

is to disp ense with mo dularity and mo del the entire network as a single automaton All

actions such as reading the state of neighb ors and computing are internal actions The

asynchrony in the system which Dijkstra mo deled using a demon is naturally a part of

the IOA mo del Also we will describ e the correctness of Dijkstras systems in terms of

executions of the automaton

Formally

A shared memory network automaton N for graph G E V is an automaton in which

The state of N is the crosspro duct of a set of no de states S N one for eachnode

u V For any state s of N weusesju to denote s pro jected onto S This is also

read as the state of no de u in global state s

All actions of N are internal actions and are partitioned into sets A N one for each

no de u V

Supp ose s s is a transition of N and b elongs to A N Consider any state s

of N suchthat s ju sju and s jv sjv for all neighb ors v of u Then there is some

transition s s of N such thats jv sjv for u and all us neighb ors in G

Supp ose s s is a transition of N and b elongs to A N Then sjv sjv for all

v u

Informally the third condition requires that the transitions of a no de u V only dep end

on the state of no de u and the states of of the neighb ors of u in G The fourth condition

requires that the eect of a transition assigned to no de u V canonlybetochange the

state of u Top (Process n-1) 0 up = false 0 up = false 0 up = false Token 1 up = true 1 up = true ...... 1 up = true

Bottom (Process 0)

Figure Dijktras proto col for token passing on a line

Dijkstras Second Example as Lo cal Checking and Correc

tion

In this section we will b egin by reconsidering the second example in Dijkstra The

derivation here is based on some unpublished work I did with Anish Arora and Mohamed

Gouda at the UniversityofTexas This proto col is essentially a token passing proto col on a

line of no des with pro cess indices ranging from to n Imagine that the line is drawn

vertically so that pro cess is at the b ottom of the line and hence is called b ottom and

Pro cess n is at the top of the line and called top This is shown in Figure The

down neighb or of Pro cess i is Pro cess i and the up neighb or is Pro cess i Pro cess

n and Pro cess are not connected

Dijkstra observed that it is imp ossible without randomization to solvemutual exclusion

in a stabilizing fashion if all pro cesses have identical co de To break symmetry he made the

co de for the top and b ottom pro cesses dierent from the co de for the others

Dijkstras second example is mo deled by the automaton D shown in Figure Each

pro cess i has a b o olean variable up andabitx Roughly up is a p ointer at no de i that

i i

p oints in the direction of the token and x is a bit that is used to implement token passing

Figure shows a state of this proto col when it is working correctly First there can b e

at most two consecutive no des whose up pointers dier in value and the token is at one of

these twonodesIfthetwo bits at the two no des are dierent as in the gure then the

token is at the upp er no de else the token is at the lower no de

The state of the system consists of a b o olean variable

up andabit x one for every pro cess in the line

We will assume that up true and up false by denition

In the initial state x for i n and up false for i n

Move Up action for the b ottom pro cess only to movetoken up

Precondition x x and up false

Eect x x

Down action for top pro cess only to movetoken down Move

Precondition x x

n n

Eects

x x

n n

Move Up i n action for other pro cesses to movetoken up

Precondition x x

i i

Eects

x x

i i

up true p ointupwards in direction token was passed

Move Down i n action for other pro cesses to movetoken down

Precondition x x and up true and up false

i i i

Eect up false p ointdownwards in direction token was passed

All actions are in a separate class

Figure Automaton D a version of Dijkstras second example with initial states The

proto col do es token passing on a line using no des with at most states

For the present assume that all pro cesses start with x Also initially assume

that up false for all pro cesses other than pro cess We will remove the need for such

initialization b elow We start by understanding the correct executions of this proto col when

it has b een correctly initialized

A pro cess i is said to have the token when any action at Pro cess i is enabled As usual the

system is correct when there is at most one token in the system Now it is easy to see that

Up is enabled Once no de makes a move then Move Up in the initial state only Move

is enabled followed by Move Up and so on as the token travels up the line Finally the

token reaches no de n and we reach a state s in which x x for i n and

i i

x x Also in state s up true for i n andup false Thus in state

n n

i n

s Move Down is enabled and the token b egins to movedown the line by executing

Down followed by Move Down andsoonuntil we reach the initial state Move

n n

again Then the cycle continues Thus in correct executions the token is passed up and

down the line

We describ e these go o d states of D that o ccur in correct executions in terms of lo cal

predicates In the shared memory mo del a lo cal predicate is any predicate that only refers

to the state variables of a pair of neighbors Thus in a go o d state of D two prop erties are

true for any Pro cess i other than

If up up then x x

i i

If up true then up true

i i

First we prove that if these two lo cal predicates hold for all i n then there

is exactly one action enabled Intuitively since up false and up truewe can start

with pro cess n and go down the line until we nd a pair of no des i and i such

that up false and up true Consider the rst such pair Then the second predicate

i i

guarantees us that there is exactly one such pair The rst predicate then guarantees that

all no des ji have x x and all no des kihave x x Thus only one action is

j i k i

enabled If x x and i then only Move Down is enabled If x x and

i i i i i

i then only Move Up is enabled If x x and i n then only Move Up

i i i

is enabled If x x and i n then only Move Down is enabled

i i n

Similarly we can show that if exactly one action is enabled then all lo cal predicates hold

Thus if the system is in a bad state some pair of neighb ors will b e able to detect this fact

Hence we conclude that D islocal ly checkableHowever wewould liketogoeven further

and b e able to correct the system to a go o d state by adding extra correction actions to each

no de If we can add correction actions so that all link predicates will eventually b ecome true

then wesaythattheD is also local ly correctable Adding lo cal correction actions is tricky

b ecause the correction actions of no des mayinterfere and result in a form of thrashing

However a line is a sp ecial case of a tree with say as the ro ot eachnode i other than

can consider i tobeitsparent The tree top ology suggests a simple correction strategy

Child which is a separate class for For eachnodei we can add a new action Correct

each i Basically Correct Child checks whether the link predicate on the link b etween

i and its parent is true If not i changes its state such that the predicate b ecomes true

Child leaves the state of is parent unchanged Supp ose j is the Notice that Correct

Child will leave the lo cal predicate parentofi and k is the parentofj ThenCorrect

on link j k true if it was true in the previous state

Thus wehave an imp ortant stability prop erty correcting a link do es not aect the

correctness of links ab ove it in the tree Using this it is easy to see that eventually all links

will b e in a go o d state and so the system is in a go o d state We will ignore the details of

this pro of but the basic idea should b e clear Dijkstras actual co de do es not use an explicit

correction action However we feel that this way of understanding his example is clearer

and illustrates a general metho d The general metho d of lo cal checking and correction was

rst intro duced in APV

Dijkstras rst example as Counter Flushing

The counter ushing paradigm describ ed in this section is taken from V

Dijkstras rst example is mo deled by the automaton D shown in Figure As in

the previous example the no des once again numb ered from to n are arranged such

that no de has no de and no de as its neighbors and so on However in this case we

also assume that Pro cess and n are neighb ors In other words by making and n

adjacentwehave made the line into a ring For pro cess i let us call Pro cess i we assume

that all arithmetic on indices and counters is mo d n the counterclo ckwise neighbor of i and

i the clo ckwise neighbor of i

Each no de has a counter count in the range n that is incremented mo d n Once

again the easiest way to understand this proto col is to understand what happ ens when it

is prop erly initialized Thus assume that initially Pro cess has its counter set to while

all other pro cesses have their counter set to Pro cesses other than are only allowed

to move see Figure when their counter diers in value from that of their counter

clo ckwise neighb or in this case the pro cess is allowed to makea moveby setting its counter

to equal that of its counterclo ckwise neighbor Thus initially only Pro cess can makea

move after which Pro cess has its counter equal to next only Pro cess can move after

which Pro cess sets its counter equal to and so on until the value moves clo ckwise

The state of the system consists of an integer variable

count fng one for every pro cess in the ring

We assume that Pro cess and n are neighbors

In the initial state count for i n andcount

Move action for Pro cess only

Precondition count count equal to counterclo ckwise neighb or

Eect count count modn incrementcounter

Move i n action for other pro cesses

Precondition count count not equal to counterclo ckwise neighb or

i i

Eects

count count set equal to counterclo ckwise neighb or

i i

All actions are in a separate class

Figure Automaton D a version of Dijkstras rst example with initial states The

proto col do es token passing on a ring using no des with n states

around the ring until all pro cesses have their counter equal to

Pro cess on the other hand cannot makea moveuntil Pro cess n has the same counter

value as Pro cess Thus until Pro cess sets its counter to Pro cess cannot makea move

However when this happ ens Pro cess increments its counter mo d n Then the cycle

rep eats as now the value b egins to move across the ring assuming nandsoonThus

after prop er initialization this system do es p erform a form of token passing on a ring each

no de is again considered to have the token when the system is in a state in which the no de

can takea move

It is easy to see that the system is in a go o d state i the following lo cal predicates are

true

For i n either count count or count count

i i i i

Either count count or count count

n n

The system is lo cally checkable but it do es not app ear to b e lo cally correctable However

it do es stabilize using a paradigm that we can call counter ushingEven if the counter values

are arbitrarily initialized in the range n the system will eventually b egin executing

as some sux of a prop erly initialized execution Wewillprove this informally using three

claims

In any execution Pro cess will eventually increment its counter Supp ose

not Then since Pro cess is the only pro cess that can pro duce new counter values

the numb er of distinct counter values cannot increase If there are two or more distinct

counter values then moves by Pro cesses other will reduce the numb er of distinct

counter values to after which Pro cess will increment its counter

In any execution Pro cess will eventually reach a fresh counter value

that is not equal to the counter values of any other pro cess To see this note

that that in the initial state there are at most n distinct counter values Thus there

is some counter value say m that is not present in the initial state Since pro cess

keeps incrementing its counter Pro cess will eventually reach m and in the interim

no other pro cess can set their counter value to m

Any state in which Process has a fresh counter value m is eventual ly fol lowedbya

state in which al l processes have counter value m It is easy to see that the value m

moves clo ckwise around the ring ushing any other counter values while Pro cess

remains at m This is why I call this paradigm counter ushing

The net eect is that any execution of D eventually reaches a go o d state in whichit

remains The general counter ushing paradigm can b e stated roughly as follows

A sender in our case Pro cess p erio dically sends a message to a set of resp onders

after all resp onders have received the message the sender sends the next message this

corresp onds to one cycle around the ring in our case

To make the proto col stabilizing the sender numb ers each message with a counter

The size of the counter must b e greater than the maximum numb er of distinct counter

values that can b e present in the network in any state The sender increments its

counter after the message has reached all resp onders

The proto col must ensure that the sender counter value will always increment and

so eventually reach a fresh value not presentinthenetwork A freshly numb ered

message m from the sender must guarantee that all old counter values will b e ushed

from the network b efore the sender sends the next message

In Dijkstras rst example the ushing prop erty is guaranteed b ecause the top ology

consists of a ring of unidirectional links Similar forms of counter ushing can b e used to

implement Data Links AfekB and token passing DolevIMunb ound b etween a pair

of no des Counter ushing is however not limited to rings or pair of no des Katz and Perry

KatzP extend the use of counter ushing to arbitrary networks in an ingenious way

The stabilizing endtoend proto col APV is obtained by rst applying lo cal correction

to the Slide proto col AGRslide and then applying a variant of counter ushing to the

Ma jority proto col of AGRslide

Message Passing Mo del

Having understo o d the techniques of counter ushing and lo cal checking and correction in a

shared memory mo del wemove to a more realistic message passing mo del The work in the

next three sections is based on V which formalizes the ideas intro duced in APV To

mo del a network proto col wewillmodelthenetwork top ology the links b etween no des and

the no des themselves Our mo del is essentially the standard asynchronous message passing

mo del except for two ma jor dierences

The ma jor dierence is that links are restricted to store at most one packet at a time

We assume that for every pair of neighb ors there is some a priori way of assigning one

of the two no des as the leader of the pair

We will argue that even with these dierences our mo del can b e implemented in real

networks

Mo deling the Top ology

We will call a directed graph V E symmetric if for every edge u v E there is an edge

v u E

Denition A top ology graph G V E l is a symmetric directedgraph V E together

with a leader function l such that for every edge u v E l u v l v u and either

l u v u or l u v v

We use E Gand V G to denote the set of edges and no des in G If it is clear what

graph we mean we sometimes simply say E and V As usual if u v E we will call v a

neighbor of u

Mo deling Links

Traditional mo dels of a data link have used what we call Unbounded Storage Data Links that

can store an unb ounded number of packets Now real physical links do have b ounds on the

Each p b elongs to the packet alphab et P dened ab ove

The state of the automaton consists of a single variable Q P nil

Send p input action

Eect

If Q nil then Q p

uv uv

Free output action

Precondition Q nil

Eect None

Receive p output action

Precondition p Q nil

Eect Q nil

The Free and Receive actions are in separate classes

Figure Unit Storage Data Link automaton

numb er of stored packets However the unb ounded storage mo del is a useful abstraction in

a nonstabilizing context

Unfortunately this is no longer true in a stabilizing setting If the link can storean

unbounded number of packets it can have an unbounded number of bad packets in the

initial state It has b een shown DolevIMunb ound that almost any nontrivial task is

imp ossible in such a setting Thus in a stabilizing setting it is necessary to dene Data Links

that have b ounded storage

A network automaton for top ology graph G consists of a no de automaton for every vertex

in G and one channel automaton for every edge in GWe will restrict ourselves to a sp ecial

typ e of channel automaton a unit storage data link or UDL for short Intuitively a UDL

can only store at most one packet at any instant No de automata communicate by sending

packets to the UDLs that connect them In the next section we will argue that a UDL can

b e implemented over real physical channels

Wexapacket alphab et P We assume that P P P consists of two disjoint

data contr ol

packet alphab ets These corresp ond to what we call data packets and control packets The

sp ecication for a UDL will allow b oth data and control packets to b e sentona UDL

Denition We say that C is the UDL corresponding to orderedpair u v if C is the

uv uv

IOA dened in Figure

C is a UIOA since wehave not dened any start states for C The external interface

uv uv

to C includes an action to send a packet at no de u Send p an action to receivea

uv uv

packet at no de v Receive p and an action Free to tell the sender that the link is

uv uv

ready to accept a new packet The state of C is simply a single variable Q that stores

uv uv

a packet or has the default value of nil

Notice twopoints ab out the sp ecication of a UDL The rst is that if the UDL has a

packet stored then any new packet sent will b e dropp ed Second the Free action is enabled

continuously whenever the UDL do es not contain a packet

Mo deling Network No des and Networks

Next we sp ecify no de automata We do so using a set that contains a no de automaton for

every no de in the top ology graph For every edge incidenttoa node u a no de automaton

N must haveinterfaces to send and receivepackets on the channels corresp onding to that

edge However we will go further and require that no des ob ey a certain stylized convention

in order to receive feedback from and send packets on links

In the sp ecication for a UDL if a packet p is sent when the UDL already has a packet

stored then the new packet p is dropp ed We will prevent packets from b eing dropp ed by

requiring that the sending no de keep a corresp onding free variable for the link that records

whether or not the link is free to accept new packets The sender sets the free variable to

true whenever it receives a Free action from the link We require that the sender only

send packets on the link when the free variable is trueFinallywhenever the sender sends

a packet on the link the sender sets its free variable to false

We wish the interface to a UDL to stabilize to go o d b ehavior even when the sender

and link b egin in arbitrary states Supp ose the sender and the link b egin in arbitrary states

Then we can havetwo p ossible problems First if free true but the UDL contains a packet

then the rst packet sentby the sender can b e dropp ed However it is easy to see that all

subsequentpackets will b e accepted and delivered by the link This is b ecause after the rst

packet is sent the sender will never set free to true unless it receives a Free notication

from the link But a Free notication is delivered to the sender only when the link is empty

The second p ossible problem is deadlo ck Supp ose that initially free false but the channel

do es not contain a packet Toavoid deadlo ck the UDL sp ecication ensures that the Free

action is enabled continuously whenever the link do es not contain a packet

A formal description of no de automata can b e found in V We omit it here Finally

a network automaton is the comp osition of a set of no de and channel automata SEND (p) Physical Channel RECEIVE (p) queue

FREE Data Link Sender Data Link Receiver

Figure Implementing a UDL over a physical channel

Implementing the Mo del in Real Networks

In a real network implementation the physical channel connecting anytwo neighb oring no des

would typically not b e a UDL For example a telephone line connecting two no des can often

store more than one packet The physical channel may also not deliver a free signal Instead

an implementation can construct a Data Link proto col on top of the physical channel such

that the resulting Data Link proto col stabilizes to the b ehaviors of a UDL eg AfekB

Figure shows the structure of such a Data Link proto col over a physical link The

sender end of the Data Link proto col has a queue that can contain a single packet When

the queue is emptythe Free signal is enabled When a Sendp arrives and the queue is

empty p is placed on the queue if the queue is full p is dropp ed If there is a packet on the

queue the sender end constantly attempts to send the packet When the receiving end of

the Data Link receives a packet the receiver sends an ack to the sender When the sender

receives an ack for the packet currently in the queue the sender removes the packet from

the queue

If the physical channel is initially empty and the physical channel is FIFO ie do es

not p ermute the order of packets then a standard stop and wait or alternating bit proto col

BSW will implement a UDL However if the physical channel can initially store packets

then the alternating bit proto col is not stabilizing S There are two approaches to

creating a stabilizing stop and wait proto col Supp ose the physical channel can store at

most X packets in b oth directions Then AfekB suggest numb ering packets using a

counter that has at least X values Supp ose instead that no packet can remain on the

physical channel for more than a b ounded amount of time S exploits such a time b ound

to build a stabilizing Data Link proto col The main idea is to use either numb ered packets

counter ushing or timers timer ushing to ush the physical channel of stale packets

A stop and wait proto col is not very ecientover physical channels that havea high

transmission sp eed andor high latency It is easy to generalize a UDL to a Bounded Storage

Data Link or BDL that can store more than one packet

Lo cal Checking and Correction in our Message

Passing Mo del

In this section weintro duce formal denitions of lo cal checkability and lo cal correctability

The denitions for a message passing mo del will b e slightly more complex than corresp onding

ones for shared memory mo del b ecause of the presence of channels b etween no des

Link Subsystems and Lo cal Predicates

Consider a network automaton with graph G Roughly sp eaking a prop erty is said to b e

lo cal to a subgraph G of G if the truth of the prop erty can b e ascertained by examining

only the comp onents sp ecied by G We will concentrate on link subsystems that consist

of a pair of neighb oring no des u and v and the channels b etween them It is p ossible to

generalize our metho ds to arbitrary subsystems

In the following denitions wexa network automaton N Net G N

Denition We dene the u v link subsystem of N as the composition of N C C

u uv vu

and N

For anystate s of N sju denotes s pro jected on to no de N and sju v denotes s

pro jected onto C Thus when N is in state s the state of the u v subsystem is the

tuple sju sju v sjv usjv

A predicate L of N is a subset of the states of N Let u v b e some edge in graph G of

N Alocal predicate L of N for edge u v is a subset of the states of the u v subsystem

in N We use the word lo cal b ecause L is dened in terms of the u v subsystem

The following denition provides a useful abbreviation It describ es what it means for a

lo cal prop ertytoholdinastates of the entire automaton

Denition We say that a state s of N satises alocal predicate L of N i

sju sju v sjv usjv L

We will makefrequent use of the concept of a closed predicate Intuitively a prop ertyis

closed if it remains true once it b ecomes true In terms of lo cal predicates

Denition Alocal predicate L of network automaton N is closed if for al l transitions

s s of N if s satises L then so does s

The following denitions provide two more useful abbreviations The rst gives a name to

a collection of lo cal predicates one for each edge in the graph The second the conjunction

of a collection of lo cal prop erties is the prop erty that is true when all lo cal prop erties hold

at the same time We will require that the conjunction of the lo cal prop erties is nontrivial

ie there is some global state that satises all the lo cal prop erties

Denition L is a link predicate set for N Net G N if for each u v G thereis

some L such that

If a b c d L then d c b a L ie L and L are identical except for

uv vu uv vu

the way the states are written down

L fL u v Gg

Thereisatleast one state s of N such that s satises L for al l L L

uv uv

Denition The conjunction of a link predicate set L is the predicate fs s satises L

for al l L Lg We use Conj L to denote the conjunction of L

Note that ConjL cannot b e the null set by the denition of a link predicate set

Lo cal Checkability

Supp ose we wish a network automaton N to satisfy some prop erty An example would b e

the prop erty all no des have the same color We can often sp ecify a prop ertyofN formally

using a predicate L of N Intuitively N can b e lo cally checked for L if we can ascertain

whether L holds bychecking all link subsystems of N The motivation for intro ducing this

notion is p erformance in a distributed system wecancheck all link subsystems in parallel in

constant time We formalize the intuitive notion of a lo cally checkable prop erty as follows

Denition A network automaton N is local ly checkable for predicate L using link pred

icate set L if

Lis a link predicate set for N and L ConjL

Each L L is closed

The rst item in the denition requires that L holds if a collection of lo cal prop erties all

hold The second item is p erhaps more surprising It requires that each lo cal prop ertyalso

b e closed

We add this extra requirement b ecause in an asynchronous distributed system it app ears

to b e imp ossible to check whether an arbitrary lo cal predicate holds al l the time What

we can do is to sample the lo cal subsystem p erio dically to see whether the lo cal prop erty

holds Supp ose the network automaton consists of three no des u v and w and suchthat

v is the neighborofboth u and w Supp ose the prop erty L that we wish to check is the

conjunction of two lo cal predicates L and L Supp ose further that exactly one of the

uv vw

two predicates is always false and the predicate that is false is constantly changing Then

whenever wecheck the u v subsystem wemightnd L true Similarly whenever we

check the v w subsystem we mightnd L true Then wemay never detect the fact

that L do es not hold in this execution Weavoid this problem by requiring that L and

L b e closed

Lo cal Correctability

The motivation b ehind lo cal checking was to eciently ensure that some prop erty L holds

for network automaton N Wewould also liketo eciently correct N to make the prop erty

true Wehave already set up some plausible conditions for lo cal checking Can we nd some

plausible conditions under which N can b e local ly corrected

Tothisendwe dene a lo cal reset function f This is a function with three arguments

the rst argumentisanodesay u the second argumentisany state of no de automaton N

and the second argument is a neighbor v of u The function pro duces a state of the no de

automaton corresp onding to the rst argument Let s b e a state of N recall that sju is the

state of N Thenf u sju v is the state of N obtained by applying the lo cal reset function

u u

at u with resp ect to neighbor v We will abuse notation by omitting the rst argumentwhen

it is clear what the rst argument is Thus we prefer to write f sju v instead of the more

cumb ersome f u sju v

We will insist that f meet two requirements so that f can b e used for lo cal correction

Denition

Assume that the prop erty L holds if a lo cal prop erty L holds for every edge u v

The rst requirement is that if anyu v subsystem do es not satisfy L then applying f

to b oth u and v should result in making L hold More precisely let us assume that by

some magic wehave the ability to simultaneously

Apply f to N with resp ect to v

Apply f to N with resp ect to u

Removeany packets stored in channels C and C

uv vu

Then the resulting state of the u v subsystem should satisfy L Ofcourseinareal

distributed system such simultaneous actions are clearly imp ossible However we will achieve

essentially the same eect by applying a socalled reset proto col to the u v subsystem

We will describ e a stabilizing lo cal reset proto col for this purp ose in the next section

The rst requirement allows no des u and v to correct the u v subsystem if L do es not

hold But other subsystems may b e correcting at the same time Since subsystems overlap

correction of one subsystem mayinvalidate the correctness of an overlapping subsystem For

example the u v andv w subsystems overlap at v If correcting the u v subsystem

causes the v w subsystem to b e incorrect then the correction pro cess can thrash To

prevent thrashing we add a second requirement In its simplest form wemight require that

correction of the u v subsystem leaves the v w subsystem correct if the v w subsystem

was correct in the rst place

However there is a more general denition of a reset function f that turns out to b e useful

Recall that wewanted to avoid thrashing that could b e caused if correcting a subsystem

causes an adjacent subsystem to b e incorrect Informally let us say that the u v subsystem

dep ends on the v w subsystem if correcting the v w subsystem can invalidate the u v

subsystem If this dep endency relation is cyclic then thrashing can o ccur On the other hand

if the dep endency relation is acyclic then the correction pro cess will eventually stabilize Such

an acyclic dep endency relation can b e formalized using a partial order on unordered pairs

of no des informally the u v subsystem dep ends on the v w subsystem if fv wg fu v g

Using this notion of a partial order we present the formal denition of a lo cal reset

function

Denition We say f is a local reset function for network automaton N net G N

with respect to link predicate set L fL g and partial order if for any state s of N and

any edge u v of G

Correction f sju v nil nil fsjv u L

Stability For any neighbor w of v

If sju sju v sjv usjv L and fv wg fu v g then

sju sju v sjv ufsjv w L

Notice that in the sp ecial case where all the link subsystems are indep endent no edge is

less than any other edge in the partial order

Using the denition of a reset function we can dene what it means to b e lo cally cor

rectable

Denition A network automaton N is lo cally correctable to L using link predicate set

Llocal reset function f and partial order if

N is local ly checkable for L using L

f isalocal reset function for N with respect to L and

Intuitivelyifwehave a reset function f with partial order we can exp ect the lo cal

correction to stabilize in time prop ortional to the maximum chain length in the partial order

Recall that a chain is a sequence a a a a Thus the following piece of notation

is useful

Denition For any partial order height is the length of the maximum length chain

Lo cal Correction Theorem

Theorem Statement

In the previous section we set up plausible conditions under which a network automaton

can b e lo cally corrected to achieve a prop erty LWe claimed that these conditions could b e

exploited to yield lo cal correction In this section we make these claims precise

Intuitively the result states that if N is lo cally correctable to L using lo cal reset function

f and partial order thenwe can transform N into a new augmented automaton N such

that N satises the following prop erty eventually every b ehavior of N will lo ok like a

b ehavior of N in which L holds We use the notation NjL to denote an automaton identical

to N except that the initial states of NjL are restricted to lie in set L

Theorem Lo cal Correction Consider any network automaton N NetG N that is

lo cally correctable to L using link predicate set Llocal reset function f andpartial order

Then there exists some N that is a UIOA and a network automaton such that N

stabilizes to the behaviors of N cjL

So far wehave ignored time complexityHowever time complexity is of crucial imp or

tance to a proto col designer b ecause nob o dy would b e interested in a proto col that to ok

years to stabilize Thus V uses a version of the timed automata mo del throughout and

uses this mo del to give a formal denition of stabilization time However intuitivelywe

can measure time complexity as follows We assume that it takestimeunittosendany

message on a link and to deliver a free action we assume that no de pro cessing takes time

With this assumption we can dene the stabilization time of a b ehavior to b e time it

takes b efore the b ehavior has a sux that b elongs to the target set We dene the overall

stabilization time as the worst case stabilization time across all b ehaviors

Using this intuitive denition of stabilization time we can state a more rened version of

stabilizes to the b ehaviors the Lo cal Correction Theorem The renementwe add is that N

of N cjL in time prop ortional to height the height of the partial order Node u Node v

TIME REQUEST

RESPONSE

Figure The structure of a single phase of the lo cal snapshotreset proto col

Overview of the Transformation Co de

To transform N into N we will add actions and states to N These actions will b e used

to send and receive snapshot packets that will b e used to do lo cal checking on each link

subsystem and reset packets that will b e used to do lo cal correction on each link subsystem

For every link u v the leader l u v initiates the checking and correction of the u v

subsystem The idea is of course that the leader of eachu v subsystem will p erio dically

do a local snapshot to checkiftheu v subsystem satises its predicates if the leader

detects a violation it tries to make the lo cal predicate true by doing a local reset of the u v

subsystem

The structure of our lo cal snapshot proto col is slightly dierent from the Chandy

Lamp ort snapshot proto col ChandyL studied earlier in the course It is p ossible to

show V that the ChandyLamp ort scheme cannot b e used without mo dications over

unit storage links Prove this it will help you understand the ChandyLamp ort scheme

b etter

Our lo cal snapshotreset proto col works roughly as follows Consider a u v subsystem

Assume that l u v u ie u is the leader on link u v A single snapshot or reset phase

has the structure shown in Fig

A single phase of either a snapshot or reset pro cedure consists of u sending a request

that is received by v followed by v sending a resp onse that is received by u During a phase

no de u sets a ag check v to indicate that it is checkingcorrecting the u v subsystem

While this ag is set no packets other than request packets can be sent on link C Since

a phase completes in constanttimethisdoesnotdelay the data packets by more than a

constant factor

In what follows we will use the basic state at a no de u to mean the part of the state at

u corresp onding to automaton N To do a snapshot no de u sends a snapshot request to

u uvuv

REQ c REQ(c) TIME RESP Store c

REQ RESP(c) RESP c = c + 1

Incorrect Matching Matching using a Counter

Figure Using counter ushing to ensure that requestresp onse matching will work

correctly within a small numb er of phases

v A snapshot request is identied bya mode variable in the request packet that carries a

mode of snapshotIfv receives a request with a mode of snapshotNodev then records its

basic state say s and sends s in a resp onse to u

When u receives the resp onse it records its basic state say r No de u then records the

state of the u v subsystem as x r nil nil s If x L ie lo cal prop erty L do es

uv uv

not hold then u initiates a reset

To do a reset no de u sends a reset request to v A reset request is identied bya mode

variable in the request packet that carries a mode of reset Recall that f denotes the lo cal

reset function After v receives the request v changes its basic state to f v s u where s is

the previous value of v s basic state No de v then sends a resp onse to uWhenu receives

the resp onse u changes its basic state to f u rv where r is the previous value of us basic

state

Of course the lo cal snapshot and reset proto col must also b e stabilizing However the

proto col we just describ ed informally may fail if requests and resp onses are not prop erly

matched This can happ en for instance if there are spurious packets in the initial state

of N It is easy to see that b oth the lo cal snapshot and reset pro cedures will only work

correctly if the resp onse from v is sent following the receipt of the request at u The diagram

on the left of Figure shows a scenario in which requests are matched incorrectly to

old resp onses that were sent in previous phases Thus we need each phase to eventually

follow the structure shown in the right of Figure

To make the snapshot and reset proto cols stabilizing we use counter ushing Thus we

numb er all request and resp onse packets Thus each request and resp onse packet carries a

number count Also the leader u keeps a variable count v that u uses to numb er all requests

sentto v within a phase At the end of the phase u increments count v Similarlythe

resp onder v keeps a variable count u in which v stores the numb er of the last request it has

received from uNodev weeds out duplicates by only accepting requests whose number is

not equal count v

Clearly the count values can b e arbitrary in the initial state and the rst few phases may

not work correctlyHowever counter ushing guarantees that in constant time a resp onse

will b e prop erly matched to the correct request Because the links are unit storage there

can b e at most distinct counter values one in each link and one at the receiver stored

in the link subsystem Thus a space of numb ers is sucient to guarantee that eventually

more precisely within phases requests and resp onses are correctly matched

Besides prop erly matching requests and resp onses wemust also avoid deadlo ckwhen

the lo cal snapshotreset proto col b egins in an arbitrary state To do so when check v is

true ie u is in the middle of a phase u continuously sends requests Since v weeds out

duplicates this do es no harm and also prevents deadlo ck Similarly v continuously sends

resp onses to the last request v has received Once the resp onses b egin to b e prop erly matched

to requests this do es no harm b ecause u discards such duplicate resp onses

Intuitive Pro of of Lo cal Correction Theorem

A formal pro of of the Lo cal Correction theorem can b e found in V In this section we

provide the main intuition Wehave already describ ed how to transform a lo cally correctable

automaton N into an augmented automaton N Wehave to prove that in time prop ortional

to the height of the partial order every b ehavior of N is a b ehavior of N in which all lo cal

predicates hold

Wehave already describ ed the intuition b ehind the use of a counter to ensure prop er

requestresp onse matching Wenow describ e the intuition b ehind the lo cal snapshot and

reset pro cedures The intuition b ehind the snapshot is very similar to the intuition b ehind

the snapshot pro cedure studied earlier in the course

Intuition Behind Lo cal Snapshots

The diagram on the left of Figure shows why a snapshot works correctly if the resp onse

from v is sent following the receipt of the request at uLeta and b b e the state of no des u

b e the state of no des u and and v resp ectively just before the resp onse is sentLeta and b

v resp ectively just after the resp onse is delivered This is sketched in Figure

The snapshot proto col must guarantee that no de u do es not send anydatapackets to v

during a phase Also v cannot send another data packet to u from the time the resp onse is

sentuntil the resp onse is delivered This is b ecause the link from v to u is a UDL that will

not give a free indication until the resp onse is received Recall that nil denotes the absence

of anypacket on a link Thus the state of the u v subsystem just b efore the resp onse is

sentisa nil nil b Similarly the state of the u v subsystem just after the resp onse is

delivered is a nil nil b

We claim that it is possible to construct some other execution of the u v subsystem

which starts in state a nil nil b has an intermediate state equal to a nil nil bandhasa

nal state equal to a nil nil b This is b ecause we could have rst applied all the actions

that changed the state of no de u from a to a whichwould cause the u v subsystem to

reach the intermediate state Next we could apply all the actions that changed the state of

no de v from b to b which will cause the u v subsystem to reach the nal state Note that

this construction is only p ossible b ecause u and v do not send data packets to each other

between the time the resp onse is sentanduntil the time the resp onse is delivered

Thus the state a nil nil b recorded by the snapshot is a possible successor of the state of

u v subsystem when the resp onse is sent The recorded state is also a a possible predecessor

of the state of u v subsystem when the resp onse is delivered But L is a closed predicate

it remains true once it is true Th us if L was true just b efore the resp onse was sent

then the state recorded by the snapshot must also satisfy L Similarlyif L is false just

uv uv

after the resp onse is delivered then the state recorded by the snapshot does not satisfy L

Thus the snapshot detection mechanism wil l not produce false alarms if the lo cal predicate

holds at the start of the phase Also the snapshot mechanism wil l detect a violation if the

the lo cal predicate do es not hold at the end of the phase

Intuition Behind Lo cal Resets

The diagram on the right of Figure shows why a lo cal reset works correctly if the

resp onse from v is sent following the receipt of the request at u Let b b e the state of no de v

just before the resp onse is sent Let a and b b e the state of no des u and v resp ectively just

before the resp onse is delivered This is sketched in Figure

The co de for an augmented automaton will ensure that just after the resp onse is sent

no de v will lo cally reset its state to f b u Similarly immediately after it receives the

resp onse no de u will lo cally reset its state to f a v Using similar arguments to the ones

used for a snapshot we can show that there is some execution of the u v subsystem

which b egins in the state f a v nil nil fb u and ends in the state f a v nil nilb

But the latter state is the state of the u v subsystem immediately after the resp onse uvuv

Snapshot Request Reset Request TIME

a’ b b f(b, u) a a b’ b’ f(a, v)

Correct Snapshot Phase Correct Reset Phase

Figure Lo cal Snapshots and Resets work correctly if requests and resp onses are prop

erly matched

is delivered But we know from the correction prop erty of a lo cal reset function that

f a v nil nilfb u satises L Since L is a closed predicate we conclude that L

uv uv uv

holds at the end of the reset phase

Intuition Behind Lo cal Correction Theorem

We can now see intuitively why the augmented automaton will ensure that all lo cal predicates

hold in time prop ortional to the height of the partial order Consider a u v subsystem

where fu v g fw xg for any pair of neighbors w x ie fu v g is a minimal element in the

partial order Then within phases of the u v subsystem the requestresp onse matching

will b egin to work correctly If the sixth phase of the u v subsystem is a snapshot phase

then either L will hold at the end of the phase or the snapshot will detect a violation But

in the latter case the seventh phase will b e a reset phase which will cause L to hold at

the end of the seventh phase

But once L remains true it remains true This is b ecause L is a closed predicate of

uv uv

the original automaton N and the only extra actions wehaveaddedto N that can aect L

are actions to lo cally reset a no de using the reset function f Butby the stability prop erty

of a lo cal reset function any applications of f at u with resp ect to some neighb or other than

v cannot aect L Similarlyany applications of f at v with resp ect to some neighbor

other than u cannot aect L Thus in constant time the lo cal predicates corresp onding

to link subsystems that are minimal elements in the partial order will b ecome and remain

true

Now supp ose that the lo cal predicates for all subsystems with height i hold from some

state s onward By similar arguments we can show that in constant time after s the

i i

lo cal predicates for all subsystems with height i b ecome and remain true Once again

the argument dep ends crucially on the stability prop erty of a lo cal reset function The

intuition is that applications of the lo cal reset function to subsystems with height i do not

o ccur after state s But these are the only actions that can falsify the lo cal predicates for

subsystems with height i The net result is that all lo cal predicates b ecome and remain

true within time prop ortional to the height of the partial order

Implementing Lo cal Checking in Real Networks

Timer Flushing

In the previous section we made the snapshotreset proto col stabilizing bynumb ering snap

shot requests and resp onses We also relied on the fact that eachlinkwas a UDL In practice

however there is an even simpler way of making the snapshotreset proto col stabilizing This

can b e done using timers

Supp ose there is a known b ound on the length of time a packet can b e stored in a link

and a known b ound on the length of time b etween the delivery of a request packet and the

sending of a matching resp onse packet Then bycontrolling the interval b etween successive

snapshotreset phases it is easily p ossible to obtain a stabilizing snapshot proto col The

interval is chosen to b e large enough such that all packets from the previous phase will have

disapp eared at the end of the interval APVsigcomm We call the general metho d timer

ushing

The main idea in timer ushing is to b ound the lifetime of old state information in

the network This is done by using no de clo cks that run at approximately the same rate

and by enforcing a maximum packet lifetime over every link State information that is

not p erio dically refreshed is timed out by the no des Timer ushing has b een used in

the OSI Network Layer Proto col Perlman and the IEEE Spanning Tree proto col

Perlman Spinelli S uses timer ushing to build practical stabilizing Data Link and

virtual circuit proto cols

In most real networks each no de sends keepalive packets p erio dically on every link

in order to detect failures of adjacent links If no keepalivepacket arrives b efore a lo cal

timer expires the link is assumed to have failed Thus it is common practice to assume time

b ounds for the delivery and pro cessing of packets Note also that the snapshot and reset

packets used for lo cal checking can b e piggybacked on these keepalivepackets without

any appreciable loss in eciency

Summary

Selfstabilization was intro duced by Dijkstra in a seminal pap er Dijkstra Dijkstras

shared memory mo del is enchanting in its simplicityandmakes an ideal vehicle to describ e

nontrivial examples of selfstabilization We showed howtosimulate Dijkstras mo del as one

big IOA with a few restrictions We also used Dijkstras rst two examples to intro duce two

powerful metho ds for selfstabilization counter ushing and lo cal checking and correction

Next wemoved to a more realistic message passing mo del Weintro duced a reasonable

mo del for b ounded storage links Using this mo del we formally dened the notions of lo cal

checkability and correctability and describ ed the lo cal correction theorem This theorem

shows that any lo cally correctable proto col can b e stabilized in time prop ortional to the

height of a partial order that underlies the denition of lo cal correctability Our pro of of this

theorem was constructive weshowed how todoaugment the original proto col with snapshot

and reset actions We made the lo cal snapshot and reset proto cols stabilizing using counter

ushing While counter ushing relies on b ounding the number of packets that can b e stored

on a link we observed that a practical implementation would b e based on b ounding the time

that a message could stay on a link w e called this technique timer ushing

Together counter ushing timer ushing and lo cal checking with and without lo cal

correction can b e used to explain a numb er of selfstabilizing proto cols and can b e used to

design new proto cols While these techniques cannot b e used to solveevery problem there

are a large numb er of useful proto cols that can b e eciently stabilized using these notions

In the next lecture Boaz will provide an example of a reset proto col that somewhat

surprisingly is lo cally correctable using the trivial partial order In other places wehave used

lo cal checking to provide solutions for mutual exclusion the endtoend problem stabilizing

synchronizers and spanning tree proto cols More detailed references can b e b e found in

APV V and AVarghese

The messages used for lo cal checking can b e piggybacked on keepalive trac in real

networks without any appreciable loss of eciency As a side b enet lo cal checking also

provides a useful debugging to ol Any violations of lo cal checking can b e logged for further

examination In a trial implementation of our reset pro cedure on the Autonet Auto lo cal

checking discovered bugs in the proto col co de In the same vein lo cal checking can provide

a record of catastrophic transient faults that are otherwise hard to detect

JJ Distributed Algorithms December

Lecturer Boaz PattShamir

Lecture

SelfStabilization and Termination

Can a selfstabilizing proto col terminate This question has several p ossible answers First

consider the regular IO automata mo del and the b ehavior stabilization denition In this

setting if the problem sp ec allows for nite b ehaviors then the trivial automaton that do es

nothing is a selfstabilizing solution This is clearly unsatisfactoryWe shall use a dierent

formulation of the mo del to see that no interesting problem admits a terminating self

stabilizing proto col

Consider the inputoutput relation sp ecication of problems dened as follows In each

pro cess there are sp ecial input and output registers the problem is dened in terms of a

relation that sp ecies for each input assignment what are the p ossible output assignments

We also assume that some of the states are designated as terminating ie no action is

enabled in them This is just another formulation of a nonreactive distributed task We

saw quite a few such tasks eg MST shortestpaths leader election etc

In this setting it is easy to see that a selfstabilizing proto col can have no terminating

states if it satises the following nonlocality condition

There exist an input value v an output value v anodej andtwo input as

signments I and I suchthat v is assigned to j in b oth I and I v is a p ossible

output value for I and is not a p ossible output value for I

Intuitively the nonlo cality condition means that no de j cannot tell lo cally that having v as

input and v as output is correct or wrong We remark that all interesting distributed tasks

have this prop erty any other task amounts to a collection of unrelated pro cesses

The moral of this obvious fact is that selfstabilizing algorithms must constantly verify

whether their output or more generally their state is up dated with resp ect to the rest

of the system and therefore they cannot terminate

This can b e formalized in the IOAmodelby assuming that the input value is input to the system

innitely often and that the output action that carries the output value is continuously enabled

SelfStabilization and Finite State

Let us consider an issue whichmay seem unrelated to selfstabilization at rst glance namely

innitestate protocols In particular consider the idea of unb ounded registers Wewent

through a lot of diculty conceptually and paid a signicant price in terms of p erformance

to get b oundedregisters proto cols eg applying CTS to Lamp orts bakery

This approach is easy to criticize from the practical viewp oint to b e concrete supp ose

that the registers contain bits There is no system to day that can hit the b ound of

in any practical application However note that this argument dep ends crucially on the

assumption that the registers are initializedproperlyFor instance consider the case where

as a result of a transient fault the contents of the registers may b e corrupted Then no

matter how big we let the registers b e they may hit their b ound very quickly On the other

hand any feasible system can have only nitely many states and in many cases it is desirable

to b e able to withstand transient faults

In short unb ounded registers are usually a convenient abstraction given that the system

is initialized prop erly while selfstabilizing proto cols are useful only when they have nitely

many states This observation gives rise to two p ossible approaches The rst is to develop

only nitestate selfstabilizing proto cols tailored to the sp ecic applications The second is

to have some kind of a general transformation that enables us to design abstract unb ounded

state proto cols using realworld b oundedsize registers and whenever the physical limit is

hit to reset the system

The main to ol required for the second approachisaselfstabilizing reset protocol This

will b e the fo cus of our lecture to day

Selfstabilizing Reset Proto col

Problem statement

Informally the goal of the reset pro cedure is that whenever it is invoked to create a fresh

version of the proto col which should b e free of any p ossible corruption in the past For

mallywe require the following

We are given a user at eachnodewe sometimes identify the user with its no de The

user has input action Receive m e and output action Send m e where m is drawn from

some message alphab et and e is an incident link The meaning of these actions is the

natural one m is received from eand m is to b e sentover e resp ectively In addition the

We distinguish b etween the actual link alphab et and the user message alphab et We assume that

comprises of and the distinct Reset proto col messages User User User send(m,e) send(m,e) send(m,e) reset request reset request reset request reset signal reset signal reset signal receive(m,e) receive(m,e) receive(m,e)

Reset Reset Reset Protocol Protocol Protocol send send send receive receive receive

Asynchronous

Network

Figure Interfacespecication for reset service

user also has output action reset request and input action reset signal The problem is to

design a proto col reset service such that if one of the no des makes a reset request and

no no de makes innitely many requests then

In nite time all the no des in the connected comp onent of a requesting no de receivea

reset signal

No no de receives innitely many reset signals

Let e u v beany link in the nal top ology of the network Then the sequence

of Send m e input at u after the last reset signal at u is identical to the sequence of

Receive m output at v after the last reset signal at v

Informally and guarantee that every no de gets a last reset signal and stipu

lates that the last reset signal provides a consistent reference timep oint for the no des

In order that the consistency condition will b e satisable it is assumed that all

messages from the user to the network and viceversa are controlled by the reset proto col

see Figure

Unb oundedregisters Reset

The original problem requires us to run a new version of the users proto col Consider the

following simple solution

The reset proto col maintains a version numb er

All outgoing messages are stamp ed with the version numb er and the numb ers are

stripp ed b efore delivery of received messages

Whenever a request o ccurs version counter is incremented and signal is output im

mediately all messages will b e stamp ed with new value

Whenever a higher value is seen it is adopted and signal is output

Messages with lowvalues are ignored

This proto col works ne for the no des participating in the proto col Stabilization time

is O diam which is clearly optimal What makes this proto col uninteresting as discussed

ab ove is the need of unbounded version counter

Example Link Reset

Recall the lo cal correction theorem The basic mechanism there was applying some lo cal

correction action which reset the no des to some presp ecied state and reset the links to

empty This matches the ab ove sp ec for reset This link reset can b e implemented fairly

eciently the details are a bit messyhowever It requires the use identiers at no des

and a ushing mechanism based on either a time b ound on message delivery or a b ound on

the maximal capacity of the link

Our goal in the reset proto col is to provide a networkwide reset Wecoulddoit

using similar ushing mechanisms eg global timeout used in DECNET However global

b ounds are usually very bad as they need to accommo date simultaneous worst cases at all

the system Thus were interested in enhancing the selfstabilization prop erties of the links

as describ ed by the lo cal correction theorem to all the network without incurring the cost

of say a global timeout

Reset Proto col

Let us rst describ e how the reset proto col should haveworked if it was initialized prop erly

In the socalled Ready mo de the proto col relays using buers messages b etween the

network and the user When a reset request is made at some no de its mo de is changed to RESET REQUEST

A BORT

BORT A

ABORT

BORT A

RESET REQUEST BORT A

(a) (b)

RESET SIGNAL

CK A EADY R

ACK READY

EADY A

CK EADY R R ACK A

R EADY

CK EADY A R RESET SIGNAL

Figure An example of the run of the reset protocol In a two nodes get reset request

from the user In b the nodes in Ab ort mode arecolored black The heavy arrows represent

parent pointers and the dashedarrows represent messages in transit In c none of the

nodes is in Ready mode In d some of the nodes arein Ready mode and some reset

signals are output

Ab ort it broadcasts Abort messages to all its neighb ors sets the Ack Pend bits to true

for all incident links and waits until all the neighb oring no des send back Ack Ifanode

receives Abort message while in Ready mo de it marks the link from which the message

arrived as its parent broadcasts Abort and waits for Acks to b e received from all its

neighbors If Abort message is received by a no de not in a Ready mo de Ack is sent back

immediately When a no de receives Ack it sets the corresp onding Ack Pend bit to false

The action of a no de when it receives the last anticipated Ack dep ends on the value of its

parentIfparent nil its mo de is changed to Convergeand Ack is senton parent

If parent nil the mo de is changed to Ready the no de broadcasts Ready messages

and outputs a reset signal A no de that gets a Ready message on its parent link while in

Converge mo de changes its mo de to Readymakes its parent nil outputs a reset signal

and broadcasts Ready

While the mo de is not Ready messages input by the user are discarded and all

messages from a link e are queued on Buffere When Abort arrives on link e Buffere

is ushed While the mo de is Ready messages from the user to the network are put on

the output queue and messages in Buffer are forwarded to the user The size of Buffer

dep ends on the assumptions we make on the user

The actual co de is given in Figures A few remarks are in order

In the selfstabilization mo del we can deal with the assumption that the top ology isnt

xed in advance This is done by assuming that links maybeupordown and the

no des are constantly informed of the status of their incident links As usual we require

correctness only after the system has stabilized We mo del this by assuming that the

maximal degree d is known eg the numb er of p orts and that the state of the links

is maintained up dated in the Edges array

In the selfstabilization mo del we tend to have as little variables as p ossible less things

can get corrupted See for example the denition for mode uinFigure

The rst step in making the ab ove proto col selfstabilizing is to make it lo cally checkable

A clear problem with the existing proto col is that it will deadlo ck if in the initial state some

parent edges form a cycle We mend this awby maintaining distance variable at each

no de such that a no des distance is one greater than that of its parent Sp ecically

distance is initialized to up on reset request and its accumulated value is app ended to the

Abort messages However since all we care ab out is acyclicity there is no need to up date

distance when a link go es down

Next we list all the link predicates that are necessary to ensure correct op eration of

the Reset proto col It turns out that all the required link predicates are indep endently

stable and hence the stabilization time would b e one time unit each subsystem needs to b e

corrected at most once indep endently

Our last step is to design a lo cal correction action for links the f of the lo cal correction

theorem

The main diculty ab out designing a correcting strategy is making it lo cal ie to ensure

that when we correct a link we do not violate predicates of incident linksubsystems For the

case of dynamictop ology network the solution is simple emulate a link failure and recovery

Of course care must b e exerted when writing the co de for these events see co de for lo cal

reset in Figure

In the co de the detection and correction mechanism is written explicitly Figure

For every link wehave the ID of the no de at the other endp oint the no de with the higher

State of pro cess u

Ack Pend arrayof d Bo olean ags

parent p ointer ranging over d fnilg

distance ranges over N where N is a b ound on the numb er of pro cesses

Edges set of op erational incident links maintained by the links proto col

Queue arrayof d send buers

Buffer arrayof d receive buers

IDs arrayof d no de identiers

Timers arrayof d timers

Check arrayof d Bo olean ags

Modearrayof d Bo olean ags Snap

Shorthand for the co de of pro cess u

Pende true Ab ort if e such that Ack

mode u

Pende false Converge if parent nil and e Ack

Ready if parent nil and e Ack Pende false

Propagate e dist

if dist then parent e else parent nil

distance dist

for all edges e Edges do

Ack Pende true

enqueue Abortdistance on Queuee

Lo cal Reset e

Buffere

Queuee

Pende true or e parent and parent nil then D if e e e and Ack

if parent e then parent nil

if Ack Pende true then

Ack Pende false

if mode u Converge then enqueue Ack in Queueparent

D else if mode u Ready

Ack Pende false

parent nil

output reset signal

for all e Edges do enqueue Ready in Queuee

Figure Selfstabilizing reset protocol part I

Co de for pro cess u

Whenever reset request and mode u Ready

Q Propagate nil

Whenever Abort dist on link e

A if mode u Ready then

Propagate e dist

A if dist or mode Ready enqueue Ack in Queuee

Whenever Ack on link e and Ack Pende true

Pende false Ack

K if mode u Converge then

enqueue Ack in Queueparent

K else if mode u Ready then

output reset signal

for all e Edges do

enqueue Ready in Queuee

Whenever Ready on link e and parent e

R if mode u Converge then

parent nil

output reset signal

for all edges e Edges do

enqueue Ready in Queuee

Whenever Send m e

if mode u Ready then

enqueue m in Queuee

Whenever Queuee

send head Queuee over e

delete head Queuee

Figure Selfstabilizing reset protocol part II

Co de for pro cess u cont

Whenever m received from link e

if Checke true then

if Snap Modee snap then

record m as part of snapshot state on link e

enqueue m in Buffere

else enqueue m in Buffere

Whenever Buffere and mode uReady

output Receive head Bufferee

delete head Buffere

Whenever Abort on e

Buffere

Whenever Down on e

execute Lo cal Resete

Whenever Timerse expires for some link e

if My ID IDse then

Checke true

Modee snap then if Snap

record lo cal state

else execute Lo cal Resete

SnapSnap Modee My IDone send Start

Whenever Start Snapb id on link e

IDse id

ID then if id My

if b reset then

Resete execute Lo cal

send Resp onse Snap lo cal state

Snap S onlink e Whenever Resp onse

Checke false

if Snap Modee snap and anyinvariantdoesnotholdthen

Snap Modee reset

else execute Lo cal Reset e

Figure Selfstabilizing reset protocol part III

A Ack Pend e true i one of the following holds

Abortdistance Queue e

e mode v Ab ort and parent

Ack Queue e

B At most one of the following hold

Abort distance Queue e

e mode v Ab ort and parent

Ack Queue e

C parent e implies that mode u Converge i one of the following holds

Ack Queue e

Pend e false and mode v Ready Ack

Ready Queue e

D parent e implies that at most one of the following holds

Ack Queue e

Pend e false and mode v Ready Ack

Ready Queue e

R tail Queue e fReadyg implies

mode u Ready

P parent e implies

one of the following holds

distance distance

u v

Ready Queue e

E e Edges implies all the following

Pend e false Ack

parent e

Queue e

Buffer e

Q Queuee is a subsequence of the following

hAbort Acki

hAck Ready Aborti

hAck Ready i

Figure Reset protocol part IV link invariants for link e u v The link v u is

denoted e

ID is assumed to b e resp onsible for the correctness of the subsystem This is done as

follows Whenever the timer dedicated to that link expires the no de checks whether it is

in charge of that link if so it sends out a snapshot message that returns with the state

of the other no de Based on this state its own state and the messages received it can nd

out whether anyoftheinvariants Figure is violated If this is the case lo cal reset is

executed in a co ordinated way that resembles the way snapshots of the link are taken

The invariants of Figure express simple prop erties that turn out to b e sucientto

ensure correct op eration of the proto col Invariants A and B capture the desired b ehavior af

ab ort messages C and D deal with the converge messages analogously The ready message

is simpler since we dont exp ect anything from a no de in ready mo de and is describ ed by

invariant RInvariant P is crucial to ensure stabilization it guarantees that no cycles of the

parent p ointers can survivechecking and correction Invariant E describ es what is exp ected

from a nonexisting edge The goal of lo cal reset action brings the subsystem to that state

The last invariant Q is a technicality we need to consider the messages in the send queue

also

Analysis

Well sketch the outlines of the pro ofs just to show the ideas Details are a bit messy

We start with the statement of stabilization

Theorem Suppose that the links areboundeddelay links with delay time D and suppose

that invariants are veried by the manager at least every P time units Then any execution

of the Reset protocol regard less of the initial state in al l states from time O D P onwards

al l the invariants hold for al l the links

This theorem is proven by showing that the invariants are stable The argumentis

induction on the actions assuming that a given state is legal with resp ect to a given link

subsystem we show thatapplyingany action will not break anyinvariant in this subsystem

Thus the pro ofs are basically a straightforward somewhat tedious caseanalysis Using the

lo cal correction theorem we can deduce that the proto col is selfstabilizing

It is more interesting to analyze the executions of the algorithm We need a few deni

tions

Denition Let s a s beanexecution fragment An interval i j is a ready

k k k

interval at node u if

mode uReady in al l states s for al l i l j

If ik then mode u Ready in s andifj then mode u Ready in s

i j

Ab ort interval and converge interval are dened analogously

For these mo de intervals wehave the following lemma

Lemma Let s a s be an execution fragment and let u beanode Then at u

n n n

A ready interval may be fol lowedonlybyanabort interval

A converge interval may be fol lowed only by a ready interval

The pro of is by straightforward insp ection of the actions Next we dene the concept of

parent graph

Denition Let s be a state The parent graph Gs is denedbytheparent and the

Queue variables as fol lows The nodes of Gs arethenodes of the network and e u v

is an edge in Gs i parent e and Ready Queue e in s

From the distances invariant weknow that the parent graph is actually a forest in legal

states We also dene a sp ecial kind of ab ort intervals An ab ort interval i j atnode u is

a root interval if for some i k j parent nil in s

The next step of the pro of is to show that for anyabortinterval there exists a ro ot

interval that contains it This is done by induction on the depth of the no de in the parent

graph Using this fact we b ound the duration of ab ort intervals

Lemma Let s bealegal state let u beanode and suppose that no topological changes

occur after sIfmode uAb ort in s and depth uk in Gs then there exists a state

s in the fol lowing n k time units such that mode u Ab ort in s

Pend bit is continuously true for more This lemma is proven by showing that no Ack

than n k time units This implies the lemma since no Ack Pend bit is set to true while

the no de is in a nonready mo de

Similarlywe can proveby induction on depth u the following b ound on the duration of

the converge intervals

Lemma Let s bealegal state at time tlet u beanode and suppose that no topological

changes occur after time tIfmode uAb ort in s and depth u k in Gs then by

time t n depth u an action occurs in which mode u is changedtoReadyandReady

messages aresentby u to al l its neighbors

It is easy to derive the following conclusion

Theorem Suppose that the links including the link invariants stabilize in C time units

If the number of reset requests and topological changes is nite then in C n time units

after the last reset requesttopological change mode uReady at al l nodes u

Pro of Consider the ro ot intervals Note that by the co de each suchrootinterval can b e

asso ciated with an input request made at this no de or with a top ological change Since

each request can b e asso ciated with at most one ro ot interval and since each top ological

change can b e asso ciated with at most tworootintervals the number of root intervals is

nite Furthermore by Lemma n time units after the last reset request all ro ot intervals

terminate and Ready will b e propagated Since any nonro ot ab ort interval is contained

in some ro ot interval after n time al l ab ort intervals have terminated and hence n time

units after the last reset request the mo de of all the no des is Ready

The following theorem states the liveness prop erty of the Reset proto col

Theorem Suppose that the execution of the Reset protocol begins at time at arbitrary

state and suppose that the links including the link invariants stabilize in C time units If

the last reset requesttopological change occurs at time t C n then a RESET signal is

output at al l the nodes by time t n

Pro of Sketch The key prop ertywe need is that the time b etween successivereadyinter

vals is at most n time units

First notice that it suces to show that for anynodev there is a time in the interval

C t ninwhich mode v Ready

Supp ose now for contradiction that at time t C n o ccurs a reset request at some

no de w and that there exists a no de u whose mo de is Ready in times C t n Let v be

any no de in the network We show by induction on the distance i between u and v that

mode iReady in the time interval C n t n i This will complete the pro of since

the hyp othesis implies that mode w Ready at time t

The base case follows from the assumption that mode uReady in the time interval

C t n The inductive step follows from the fact that if for some no de v intimet Cn

wehave mode v Ready then there o ccurs an action in the time interval t n t in

which the mo de of v changed from Ready to Ab ortand Abort messages were senttoall

the neighb ors of v Hence for each neighbor v of v there is a state in the time interval

t n t n inwhich mode v Ready

Next weprove the consistency of the Reset proto col This prop erty follows from the

simple Buffer mechanism There is a buer dedicated to each link such that all messages

arriving from link e are stored in Buffere Buffere is ushed whenever a Abort message

arrives from e

Theorem Suppose no topological changes occurs after time C and that the last reset

signals occur at two adjacent nodes u and v after time C n Then the sequence Send m e

input by the user at u fol lowing the last reset signal at u is identical to the sequenceof

e output to the user at v after the last reset signal at v Receivem

Pro of Consider the last nonready intervals in u and v Supp ose that the last states in

which mode u Ready and mode v Ready are s and s resp ectively and let s and

j j i

u v u

b e the rst states in the last nonready intervals resp ectively ie for all i k j in s

u u i

s mode u Readyandins mode uReady Note that since the last reset signal

k i

is output after time C n it follows that the states s and s indeed exist Note

i i

u u

further that the change of mo de to Ab ort is accompanied by broadcasting Abort messages

by Propagate

We argue that m is delivered at v after s Consider Send m e input at u after s

i j

v u

supp ose not Then the mo de of v is changed to Ab ort after s and v broadcasts Abort to

u Hence there must b e a state s kjsuch that mode u Ready in s contradicting

k u k

is the last state in the last nonready interval Recalling that the the assumption that

manager buers messages until the mo de is Readywe conclude that the corresp onding

e is output to the user at v after s Receivem

Consider Send m e input at u b efore s Since the manager discards all messages

input while in nonready mo de we need only to consider Send m e input at u b efore s

Again we argue that Receivem e is not output at v after s This follows from the fact

that there must b e Abort message sentto u from v after m that implies that the mo de of

u is not Ready after s acontradiction

Comments

The reset proto col is a p owerful to ol but sometimes its an overkill it do esnt seem reason

able to reset the whole system whenever a minor but frequent fault o ccur eg a new no de

joins a link fails It seems that one of the b est applications of reset is when it is combined

with unb oundedregister proto cols the eect of the reset signal in this case can usually b e

dened easily eg set counters to

Finallywe note that the time complexity actually is b ounded by the length of the longest

simple path in the network which is a more rened measure than just the numb er of no des

If the network is a tree for instance the proto col works in diameter time which is clearly

optimal The space complexity is logarithmic in the b ound N As typical for the lo cal

correction metho d the communication bandwidth required for stabilization is prop ortional

to the space which is in our case logarithmic

Application Network Synchronization

Recall the network synchronization problem The basic prop erty there was that if a message

is sent in lo cal round i then no messages of rounds i orlesswillever b e received in that

no de We abstract the synchronizer as a service mo dule whose task is to provide the user

with pulse numb ers at all times If we asso ciate with eachnode v its roundpulse number

P v it can b e easily seen that a state is legal only if no two adjacent no des pulses dier by

more than ie for all no des v

u Nv jP u P v j

N v denotes the set of no des adjacentto v Call the states satisfying Eq legal

Of course we also wanttokeep progressing This can b e stated as asserting that for each

conguration there is some pulse number K such that all no des get all pulses K K

If the system initialized in a legal state we can use the following rule

P v min fP ug

uN v

The idea is that whenever the pulse number is changed the no de sends out all the

messages of previous rounds whichhavent b een sentyet

The rule ab oveisstable ie if the conguration is legal then applying the rule arbitrarily

can yield only legal conguration Notice however that if the state is not legal then applying

the rule may cause pulse numb ers to drop This is a reason to b e worried since the regular

course of the algorithm requires pulse numb ers only to grow And indeed the rule is not selfstabilizing

3 3 3 3 3 2 4 2 4 2 4 2 4 2 4 1 3 1 2 1 2 1 2 1 2 (a) (b) (c) (d) (e) 3 1 3 1 3 3 3 3 3 3 4 2 4 2 4 2 4 4 4 4 3 3 3 3 5

3 3 3 5 2 4 2 4 4 4 4 4 1 2 3 2 3 2 3 2 (f) (g) (h) (i) 2 3 2 3 2 3 2 3 4 4 4 4 4 4 4 4

5 5 5 5

Figure An execution using rule The node that movedineach step in marked

One idea that can p op into mind to try to repair the ab oveawistonever let pulse

numb ers go down Formally the rule is the following

P v max P v min fP ug

uN v

This rule can b e proven to b e selfstabilizing However it suers from a serious drawback

regarding its stabilization time Consider the conguration depicted b elow

1 1 1000000

Figure A pulse assignment for rule

A quick thought should suce to convince the reader that the stabilization time here is in

the order of time units which seems to b e unsatisfactory for such a small network

The next idea is to have a combination of rules if the neighb orho o d is legal then the

problem sp ecication requires the min rule is used But if the neighb orho o d is not legal

another rule can b e used The rst idea we consider is the following

min fP ug if u Nv jP v P uj

uN v

P v

max fP ug otherwise

uN v

It is fairly straightforward to show that if an atomic action consists of reading ones neigh

b ors and setting its own value in particular no neighbor changes its value in the meanwhile

then the max rule ab ove indeed converges to a legal conguration Unfortunately this mo del

traditionally called central demon mo del is not adequate for a truly distributed system which

is based on lo osely co ordinated asynchronous pro cesses And as one might susp ect the max

rule do es not work in a truly distributed system

So here is a solution

min fP ug if u Nv jP u P v j

uN v

P v

max fP u g if u Nv P u P v

uN v

P v otherwise

In words the rule is to apply minimum plus one when the neighb orho o d seems to b e

in a legal conguration and if the neighb orho o d seems to b e illegal to apply maximum

minus one

The intuition b ehind the mo dication is that if no des change their pulse numb ers to b e

the maximum of their neighb ors then race condition mightevolve where no des with high

pulses can run away from no des with low pulses If the correction action takes the pulse

numb er to b e one less than the maximum then the high no des are lo cked in the sense

that they cannot increment their pulse counters until all their neighborhood have reached

their pulse numb er This lo cking spreads automatically in all the infected area of the

network

We shall nowprove that this rule indeed stabilizes in time prop ortional to the diameter

of the network For this we shall need a few denitions

Denition Let v beanode in the graph The p otential of v is denotedby v and is

denedby

v maxfP u P v dist u v g

Intuitively v is a measure of howmuchis v not synchronized or alternatively the

size of the largest skew in the synchronization of v corrected by the distance Pictorially

one can think that every no de u is a p oint on a plane where the xco ordinate represents the

distance of u from v and the y co ordinate represents the pulse numb ers see Figure

for an example In this representation v is the only no de on the y axis and v isthe

maximal vertical distance of any p oint ie no de ab ove the degree line going through

pulse numbers pulse numbers

Φ (c) potentiale line potentiale line a b 6 6 0 1 5 d 5 d Φ (b) 4 4 c φ(c) φ(b) 3 3 c 3 c

6 2 2 b 5 e 1 1 b d a a 0 0 123distance from c 123distance from b

(i) (ii) (iii)

Figure On the left is an example of a graph with pulse assignment i Geometrical

representations of this conguration are shown in ii and iii The plane corresponding to

node c is in the midd le ii and the plane corresponding to node b is on the right iii As

can bereadily seen c and bAlso candb see Denition

Let us start with some prop erties of that followimmediately from the denition

Lemma For al l nodes v V v

Lemma Aconguration of the system is legal if and only if for al l v V v

Wenowshow the key prop erty of the new rule namely that the p otential of the no des

never increases under this rule

Lemma Let P be any pulse assignment and suppose that node u changes its pulse

number by applying the rule Denote the new pulse number of u by P u and the potential

of the nodes in the new conguration by Then for al l nodes v V v v

Pro of The rst easy case to consider is the p otential of u itself Since P u Pu we

have

u max fP w P u dist w ug

w V

max fP w P u dist w ug

w V

Note for later reference that the inequality in is strict if u Now consider

v u The only value that was changed in the set

fP w P v dist w v j w V g

is P u P v dist u v There are two cases to examine If u changed its pulse by applying

the min plus one part of the rule then there must b e a no de w whichisaneighbor of u

and is closer to v ie dist u v dist w v Also since min plus one was applied

wehave P u P w Now

P u P v dist u v P w P v dist w v

P w P v dist w v

and hence the v do es not increase in this case The second case to consider is when u

has changed its value by applying the max minus one part of the rule The reasoning in

this case is similar let w b e a neighbor of u with P w P u Clearly dist w v

dist u v This implies that

P u P v dist u v P w P v dist w v

P w P v dist w v

and we are done

As noted ab ove the inequality in is strict if u In other words eachtime

a no de with p ositivepotential changes its pulse numb er its p otential decreases This fact

when combined with Lemmas and immediately implies eventual stabilization However

using this argument leads to a pro of that the stabilization time is b ounded by the total

p otential of the conguration which in turn dep ends on the initial pulse assignment We

need a stronger argument in order to prove a b ound on the stabilization time that dep ends

only on the top ologyToward this end we dene the notion of wavefront

Denition Let v beanynode in the graph with potential v The wavefront of v denoted

v is denedby

v min fdist u v j P u P v dist u v v g

In the graphical representation the wavefront of a no de is simply the distance to the

closest no de of on the p otential line see Figure for an example Intuitively one

can think of v as the distance to the closest largest trouble of v The imp ortance of

the wavefront b ecomes apparent in Lemma b elow but let us rst state an immediate

prop ertyithas

Lemma Let v V Then v if and only if v

Lemma Let v beanynode with v and let v be the wavefront of v after one

time unit Then v v

Pro of Supp ose v f at some state Let u be any no de suchthatP u P v

dist u v v and dist u v f Consider a neighbor w of u which is closer to v

ie dist w v f itmay b e the case the w v From the denition of v it

follows that P w Pu Now consider the next time in which w applies Rule

If at that time v fwe are done Otherwise w must assign P w P u No

greater value can b e assigned or otherwise Lemma would b e violated At this time

P w P v dist w v v also and hence v f

Corollary Let v beanynode Then after v time units v

Wecannow prove the main theorem

Theorem Let G V E bea graph with diameter d and let P V N be a pulse

assignment Applying Rule above results in a legal conguration in d time units

Pro of By Lemma it suces to showthatafterd time units v for all v V

From Corollary ab ove we actually know that a slightly stronger fact holds for all no de

v V after v time units v The theorem follows from the facts that for all

v V v d and by the fact that v never increases by Lemma

Implementation with Bounded Pulse Numbers

The optimal rule from ab oveworks only when the pulse numb ers maygrowunb oundedly

However we can use the reset proto col to makeiswork with b oundedsize registers Supp ose

that the registers dedicated to the pulse numb ers may hold the values B for some b ound

B Then the proto col would work by pro ceeding according to the unb ounded rule Eq

and whenever a value should b e incremented ab ove B reset is invoked Notice that wemust

require that B is suciently large to enable propp er op eration of the proto col b etween

resets if B is less than the diameter of the network for instance then there is a p ossible

scenario in which the far no des never get to participate in the proto col

JJ Distributed Algorithms Decemb er

Lecturer Nancy Lynch

Lecture

This is the last lecture of the course we didnt havetimetocover a lot of material

weintended to Some of the main topics missing are faulttolerant network algorithms and

timingbased computing Well have several additional lectures in January to cover some of

this for those who are interested Todaywell do one ma jor result in faulttolerant network

algorithms namely the imp ossibility of faulttolerant consensus in asynchronous networks

Well also do two other results on ways to get around this limitation

FischerLynchPaterson Imp ossibility Result

Recall the consensus problem from the shared memory work The interface is dened by

the input actions init v and output actions decide v where i is a pro cess and v is a

i i

value Here we shall consider the same initdecide interface but the implementation is now

a distributed network algorithm The message delivery is FIFO and completely reliable In

fact we will assume that the no des have reliable broadcast actions so that when a no de

sends a message it is sent simultaneously to all the others and b ecause of the reliabilityof

message transmission it will eventually get there We consider solving this problem in the

face of pro cessor faults since we are giving an imp ossibility result we consider only a very

simple kind of faultybehavior at most one stopping failure Using suchaweak kind of

fault makes the imp ossibility result stronger

The imp ossibility of solving consensus in such a friendly environment is rather surprising

The story is that Fischer LynchandPaterson worked on this problem for some while

trying to get a p ositive result ie an algorithm Eventually they realized that there was a

very fundamental limitation getting in the way They discovered an inherent problem with

reaching agreement in an asynchronous system in the presence of any faulty pro cess

The Mo del As in any imp ossibility result it is very imp ortant to dene precisely the

assumptions made ab out the underlying mo del of computation Here we assume that the

network graph is complete and that the message system consists of fair FIFO queues Each

pro cess no de i is an IO automaton There are inputs and outputs init v and decide v

i i

resp ectively where i is a no de and v is a value There are also outputs broadcast v and

inputs receive v where i and j are no des and v is a value The eect of a broadcast v

ji i

action is to put atomically a v message in the channels leading to all the no des other than

i in the system With the broadcast action we actually dont need individual send actions

We assume without loss of generality that each pro cess has only one fairness class ie

the pro cesses are sequential and that the pro cesses are deterministic in the sense that

in each pro cess state there is at most one enabled lo cally controlled action and for each

state s and action there is at most one new state s suchthats s is a transition of the

proto col In fact we can assume furthermore without loss of generality that in each state

there is exactly one enabled lo cally controlled action since we can always add in dummy

steps

Correctness Requirement For the consensus problem we require that in any execution

such that exactly one init o ccurs for eachnode iwehave the following conditions satised

Agreement There are no two dierent decision values

Validity If all the initial values are the same value v then v is the only p ossible decision

value

In addition we need some termination conditions First we require that if there is

exactly one init for each i and if the entire system pro cesses and queues executes fairly

then all pro cesses should eventually decide This condition is easy to satisfy eg just

exchange values and take ma jority But we require more Dene fair executions to b e

those in which all but at most pro cess execute fairlyandallchannels execute fairly

We assume that a pro cess that fails still do es its input steps but it can stop p erforming

lo callycontrolled actions even if such actions remain enabled We require that in all fair

executions all pro cesses that dont stop eventually decide

It seems as though it oughttobepossibletosolve the problem for fair executions

also esp ecially in view of the p owerful broadcast primitiveyou should try to design an

algorithm Our main result in this section however is the following theorem

Theorem Thereisnoresilient consensus protocol

Henceforth we shall restrict our attention to the binary problem where the input values

are only and this is sucient to prove imp ossibility

Terminology Dene A to b e a resilient consensus protocol RCP provided that it

satises agreement validity and termination in all its fair executions Dene A to b e a

resilient consensus protocol RCP provided it is a resilient consensus proto col and if

in addition all nonstopp ed pro cesses eventually decide in fair executions

We call a nite execution valent if is the only decision value in all extensions of

Note the decision can o ccur anywhere in the extension including the initial segment

Similarlywe dene valent nite executions Wesaythat is univalent if it is either

valentorvalent and wesay that is bivalent if it is not univalent ie is bivalent

if there is one extension of in which someone decides and another extension in which

someone decides

These valency concepts capture the idea that a decision may b e determined although

the actual decision action might not yet have o ccurred As in the shared memory case

we will restrict attention in the pro of to inputrst executions ie executions in whichall

inputs arrive at the b eginning in round robin order of init events for pro cess n

Dene an initial execution to b e an inputrst execution of length exactly n that is an

initial execution just provides the initial values to all the pro cesses

We can now start proving the imp ossibility result We b egin with a statement ab out the

initial executions

Lemma In any RCP there is a bivalent initial execution

Pro of The idea is the same as in the shared memory setting Assume that the lemmais

false ie that all initial executions are univalent By validity the all input must lead to

decision of and all s to decision of By assumption anyvector of s and s leads

to a unique decision Hence there exist two input vectors with Hamming distance with

corresp onding initial executions such that one is valent and the other is valent let

and b e these two resp ective initial executions Let i b e the index of the unique pro cess

in whose initial value they dier

The idea is that the rest of the system cant tell which of the vectors was input if i never

do es anything More formally consider a fair execution that extends initial execution

in which i fails right at the b eginning ie it takes no lo callycontrolled steps In

some pro cess j eventually decides by the assumptions that is valent and that the

proto col is RCP

But then we claim that there is another execution that extends that also leads to

decide This is b ecause the only dierence b etween the states at the end of and

is in the state of pro cess inow allow the same pro cesses to take the same steps in as in

the same messages to get delivered etc see Figure The only dierence in how

and will unfold is in the state changes caused by the receipt of the same message by i

in the two executions those states can b e dierent but since i never p erforms any lo cally

controlled action in either the rest of the system cannot tell the dierence α α 0 1

α’ α’ 0 1

0 0

Figure In b oth extensions for and the same pro cesses take the same actions and

therefore cannot b e valent

Hence pro cess j must decide in to o contradicting the valence of

Examples The ma jority proto col is a RCP it has no bivalent initial execution since

an initial execution can only lead to a decision of v in case the value v is the initial value of

a ma jority of the pro cesses

Another example of an RCP is a leaderbased proto col where everyone sends their

values to pro cess and decides on the rst value received and informs everyone else ab out

the decision This do es havebivalent initial executions anything in which there are two

pro cesses other than p having dierent initial values

Wenow pro ceed with the denition of a decider which is dened somewhat dierently

from the wayitwas dened for the shared memory setting

Denition A decider for an algorithm A consists of an inputrst execution of A and

aprocess index i such that the fol lowing conditions hold see Figure

is bivalent

There exists a valent extension of such that the sux after consists of steps

of process i only These can be input output or internal steps

There exists a valent extension of such that the sux after consists of steps

of process i only

Note that the exibility in obtaining two dierent executions here arises b ecause of the

p ossible dierent orders of interleaving b etween lo cally controlled steps and messagereceipt α

α α 0 1 i i only only

0−valent 1−valent

Figure schematic diagram of a decider i

steps on the various channels Also messages could arrive in dierent orders in twodierent

executions

Let us now state a lemma ab out the existence of a decider

Lemma Let A beanyRCP with a bivalent initial execution Then there exists a decider

for A

We shall prove this lemma later Let us rst complete the imp ossibility pro of given

Lemma

Pro of of Theorem Supp ose A is a RCPSinceA is also a RCP and since it

has a bivalent initial prex we can obtain by Lemma a decider i Wenow argue a

contradiction in a way that is similar to the argument for the initial case

Consider a fair extension of in which i takes no lo callycontrolled steps ie i

fails right after Eventually in some pro cess j must decide supp ose without loss of

generality that the decision value is Also we can supp ose without loss of generalitythat

there are no message deliveries to i in the sux of after if there are any then omitting

them still leads to the same decision

Nowwe claim that the there is another execution that extends and that also leads

to decide see Figure

This is b ecause the only dierences b etween the states at the end of and are in a

the state of pro cess ibthelast elements in the channels leaving pro cess i there can b e

some extras after and c the rst elements in the channels entering pro cess i can b e

dierent there can b e some messages missing after Then weallow the same pro cesses

to take the same steps in the sux after as in the same messages to get delivered etc

Nothing that i has done will interfere with this note that wenever deliver any of the extra α

α α α 0 1 2 i no i i steps only only

0−valent 1−valent process j decides 0

α’ 2 no i steps

process j decides 0

Figure Contradiction derived from the existence of a decider

messages As b efore the only dierence in what happ ens is in the state changes caused by

the receipt of the same message by i in the two executions those states can b e dierent but

since i never p erforms any lo cally controlled action in either the rest of the system cannot

tell the dierence

contradicting the valence of Hence pro cess j decides by the end of

It remains nowtoprove Lemma

Pro of of Lemma Supp ose there is no decider Then starting from a bivalent initial

execution we construct a fair execution no failures at all in which successive prexes are

all bivalent thus violating the termination requirement

Wework in a roundrobin fashion at each phase we either let one sp ecied pro cess take

a lo cally controlled step or else we deliver the rst message in some channel Visiting all

pro cesses and all channels in the round robin order yields a fair execution if a channel is

emptywecanbypass it in the order

Wehavenow reduced the pro of to doing the following We need to consider one step at

a time where each step starts with a particular bivalent inputrst execution and we need

to either

let a particular pro cess i take a turn or

deliver the oldest message in some channel

Wemust do this while keeping the result bivalent We pro ceed by considering the p ossible

cases of the step at hand

Case Deliver message m from j to i where m is the rst message in the channel from

i to j in the state after Consider the tree of all p ossible nite extensions of in which

m is delivered to i just at the end in the interim arbitrary other steps can o ccur If any

of these leaves is bivalent we are done with this stage So assume all are univalent As in

LouiAbuAmara assume without loss of generality that delivering the message immediately

after leads to a valent state Consider a nite extension of that contains a decision of

this is p ossible b ecause is bivalent see Figure

m delivered

0−valent

Figure the delivery of m leads to a valent state

Wenow use this to get a conguration involving one step such that if m is delivered

just b efore the step the result is a valent execution whereas if it is delivered just after

the step the result is a valent execution see Figure Weshowthismust happ en by

considering cases as follows

m delivered s 0−valent m delivered

1−valent

Figure the delivery of m b efore step s leads to a valent state and the delivery of m

after step s leads to a valent state

If the extension leading to the decision of do es not contain a delivery of mthen

delivering m right at the end leads to a valent state then since attaching it anywhere gives

univalence there must b e consecutive p ositions along the chain where one gives valence

and one gives valence see Figure

α α

m delivered m delivered 0−valent 0−valent m delivered 0−valent m delivered 1−valent m delivered 1−valent 1−valent m delivered 1

1−valent 1

Figure Left the delivery of m do es not app ear in the extension to a valent state

Right m is delivered in the extension of

On the other hand if this extension do es contain a delivery eventfor m then consider the

place where this delivery o ccurs Immediately after the deliverywemust haveunivalence

which for consistency with the later decision in this execution must b e valence Then we

use the same argumentbetween this p osition and the top

Now if the intervening step step s in Figure involves some pro cess other than i

then it is also p ossible to p erform the same action after the delivery of m and the resulting

global state is the same after either order of these twoevents see Figure

m delivered s 0−valent s m delivered

1−valent

Figure if s do es not involve i then p erforming s b efore or after m contradicts valency

But this is a contradiction since one execution is supp osed to b e valent and the other

valent So it must b e that the intervening step involves pro cess i

But then this yields a decider a contradiction to the assumption

Case Process i take a local lycontrol led step

Essentially the same argumentworks in this case weonlysketch it Consider the tree of

nite extensions in whichanything happ ens not including a lo callycontrolled step of i and

then i do es a lo callycontrolled step Here we use the assumption that lo callycontrolled

steps are always p ossible Note that the tree can include message deliveries to i but not

lo callycontrolled steps of i except at the leaves

This time we get a conguration involving one step suchthatifi takes a lo cally

controlled step just b efore this step the result is a valent execution whereas if i takes its

lo callycontrolled step just after the step the result is a valent execution Again weget

this by cases If the extension do esnt contain a lo callycontrolled step of iwe can attachit

at the end and pro ceed as in Case If it do es then cho ose the rst such in the extension

and pro ceed as in Case Then we complete the pro of is as in Case by showing that

either commutativity holds or a decider exists

BenOr Randomized Proto col for Consensus

Fischer Lynch and Paterson show that consensus is imp ossible in an asynchronous system

even for simple stopping faults However this is only for deterministic algorithms It turns

out that the problem can b e solved in an asynchronous environment using randomization

with probability of eventually terminating In fact an algorithm can even b e designed to

tolerate stronger typ es of pro cess faults Byzantine faults where pro cesses can moveto

arbitrary states and send arbitrary messages at any time

The algorithm uses a large numb er of pro cesses relativetothenumberoffaultsbeing

tolerated eg n f This can b e reduced but this version is easiest to see The

algorithm pro ceeds as follows Each pro cess starts with its initial value in variable x The

algorithm works in asynchronous phases where each phase consists of two rounds Each

pro cess sends messages of the form rst rvand second rvwherer is the phase number

and v is a value in the domain b eing considered The co de is given in Figure for binary

values We assume for simplicity that the no des continue p erforming this algorithm forever

even though they only decide once

Correctness

Validity If all the pro cesses start with the same initial value v thenitiseasytoseethat

all nonfaulty pro cesses decide on v in the rst phase In the rst round of phase all the

Initially x is is initial value

For each phase r do

Round broadcast rst rx

wait for n f messages of the form rst r

if at least n f messages in this set havesamevalue v

then x v

else x nil

Round broadcast second rx

wait for n f messages of the form second r

Let v b e the value o ccurring most often break ties arbitrarily

and let m b e the numb er of o ccurrences of v

if m n f

then DECIDE v x v

else if m n f

then x v

else x random

Figure Randomized Consensus Algorithm for Asynchronous Network co de for pro cess

nonfaulty pro cesses broadcast rst v and receive rst v messages from at least n f

pro cesses Then in round of phase all nonfaulty pro cesses broadcast second v and

again receive at least n f second v messages

Agreement Weshow that the nonfaulty pro cesses cannot disagree We will also see that

all nonfaulty pro cesses terminate quickly once one pro cess decides Supp ose that pro cess i

decides v at phase r This can happ en only if i receives at least n f second rv messages

for a particular r and v which guarantees that each other nonfaulty pro cess j receives at

least n f second rv messages This is true since although there can b e some dierence

in which pro cesses messages are received by i and j pro cess j must receive messages from

at least n f of the n f senders of the second rv messages received by i Among

those senders at least n f must b e nonfaulty so they send the same message to j as to

i A similar counting argument shows that v must b e the value o ccurring most often in p s

set of messages since nf Therefore pro cess j cannot decide on a dierentvalue at

phase r Moreover j will set x v at phase r Since this holds for all nonfaulty pro cesses

j it follows as in the argumentforvalidity that all nonfaulty pro cesses that havenotyet

decided will decide v in phase r

Termination Now it remains to argue that the probability that someone eventually termi

nates is So consider anyphaser We will argue that regardless of what has happ ened at

prior rounds with probability at least which is quite small but nevertheless p ositive

all nonfaulty pro cesses will end up with the same value of x at the end of round r In this

case by the argument for agreement all nonfaulty pro cesses will decide by round r

For this phase r consider the rst time any nonfaulty pro cess say i reaches the p ointof

having received at least n f rst r messages Let M b e this set of messages and let v

b e the ma jorityvalue of the set dene v arbitrarily if there is no ma jorityvalue Wenow

claim that for any second rw message sentby a nonfaulty pro cess wemust have w v

or w nil To see that this is true supp ose w nil Now if second rwissentbya

nonfaulty pro cess j thenj receives at least n f rst rw messages Since by counting

at least n f rst rw messages app ear in M and since n f this is a ma jority

and so wehave w v

Next we claim that at any phase r the only value that can b e forced up on any

nonfaulty pro cess as its choice is v To see this note that for a pro cess to b e forced to cho ose

w the pro cess must receive at least n f second rw messages Since at least one of these

messages is from a nonfaulty pro cess it follows from the previous claim that w v

Now note that the value v is determined at the rst p oint where a nonfaulty pro cess has

received at least n f rst r messages Thus by the last claim the only forcible value

for round r is determined at that p oint But this p oint is b efore any nonfaulty pro cess tosses

a coin for round r and hence these coin tosses are independent of the choice of forcible value

for round r Therefore with probability at least all pro cesses tossing coins will cho ose

v agreeing with all those that do not toss coins This gives the claimed decision for round

The main interest in this result is that it shows a signicant dierence b etween the

randomized and nonrandomized mo dels in that the consensus problem can b e solved in the

randomized mo del The algorithm is not practical however b ecause its exp ected termination

time is very high We remark that cryptographic assumptions can b e used to improve the

probability but the algorithms and analysis are quite dicult See Feldman

ParliamentofPaxos

In this section well see Lamp orts recent consensus algorithm for deterministic asyn

chronous systems Its advantage is that in practiceitisvery tolerant to faults of the

following typ es no de stopping and recovery lost outoforder or duplicated messages link

failure and recoveryHowever its a little hard to state exactly what faulttolerance prop er

ties it has so instead well present the algorithm and then describ e its prop erties informally

We remark that the proto col always satises agreement and validity The only issue is

termination under what circumstances nonfaulty pro cesses are guaranteed to terminate

The Proto col All or some of the participating no des keep trying to conduct bal lots Each

ballot has an identier whichisatimestamp index pair where the timestamp neednt b e

a logical time but can just b e an increasing number at each no de The originator of each

ballot tries if it do esnt fail to asso ciate a value with the ballot and get the ballot

accepted by a ma jority of the no des If it succeeds in doing so the originator decides on the

value in the ballot and informs everyone else of the decision

The proto col pro ceeds in ve rounds as follows

The originator sends the identier t i to all pro cesses including itself If a no de

receives t i it commits itself to voting NO in all ballots with identier smaller

than t i on which it has not yet voted

Each pro cess sends back to the originator of t i the set of all pairs t i v indi

cating those ballots with identier smaller than t ionwhich the no de has previously

voted YES it also encloses the nal values asso ciated with those ballots If the

originator receives this information from a majorityitcho oses as the nal value of the

ballot the value v asso ciated with the largest t i t ithatarrives in any of these

messages If there is no suchvalue rep orted byanyone in the ma jority the originator

cho oses its own initial value as the value of the ballot

The originator sends out t iv where v is the chosen value from round

Each pro cess that hasnt already voted NO on ballot t ivotes YES and sends

its vote back to the originator If the originator receives YES votes from a ma jority

of the no des it decides on the value v

The originator sends out the decision value to all the no des Anynodethatreceives it

also decides

We rst claim that this proto col satises agreementandvalidityTermination will need

some more discussion

Validity Obvious since a no de can only decide on a value that is some no des initial value

Agreement Weprove agreementby contradiction supp ose there is disagreement Let b be

the smallest ballot on which some pro cess decides and say that the asso ciated value is v By

the proto col some ma joritysay M of the pro cesses vote YES on bWe rst claim that

v is the only value that gets asso ciated with any ballot strictly greater than bFor supp ose

not and let b b e the smallest ballot greater than b that has a dierentvalue say v v

In order to x the value of ballot b the originator has to hear from a ma joritysay M

of the pro cesses M must contain at least one pro cess say i that is also in M ie that

votes YES on bNow it cannot b e the case that i sends its information round for b

b efore it votes YES round on b for at the time it sends this information it commits

itself to voting NO on all smaller ballots Therefore i votes YES on b b efore sending

its information for b This in turn means that b v is included in the information i sends

for b So the originator of b sees a YES for ballot bBythechoice of b the originator

cannot see anyvalue other than v for any ballot b etween b and b Soitmust b e the case

that the originator cho oses v a contradiction

Termination Note that it is p ossible for two pro cesses to keep starting ballots and prevent

each other from nishing Termination can b e guaranteed if there exists a suciently long

time during which there is only one originator trying to start ballots in practice pro cesses

could drop out if they see someone with a smaller index originating ballots but this is

problematic in a purely asynchronous system and during which time a ma jority of the

pro cesses and the intervening links all remain op erative

The reason for the Paxos name is that the entire algorithm is couched in terms of a

ctitious Greek parliament in which legislators keep trying to pass bills Probably the

cutest part of the pap er is the names of the legislators

JJ Distributed Algorithms Handout Septemb er

Homework Assignment

Due Thursday September

Reading

Lamp ortLynch survey pap er

Exercises

Carry out careful inductive pro ofs of correctness for the LeLannChangRob erts LCR

algorithm outlined in class

For the LCR algorithm

a Give a UID assignment for whichn messages are sent

b Give a UID assignment for whichonlyO n messages are sent

c Showthattheaverage numb er of messages sentisO n log n where this average

is taken over all the p ossible orderings of the elements on the ring each assumed

to b e equally likely

Write co de for a state machine to express the Hirschb ergSinclair algorithm in the

same style as the co de given in class for the LCR algorithm

Show that the FredericksonLynchcounterexample algorithm do esnt necessarily have

the desired O n message complexity if pro cessors can wakeupatdierent times

Briey describ e a mo dication to the algorithm that would restore this b ound

Prove the b est lower bound you can for the numb er of rounds required in the worst

case to elect leader in a ring of size n Be sure to state your assumptions carefully

JJ Distributed Algorithms Handout Septemb er

Homework Assignment

Due Thursday September

Reading

The Byzantine generals problemby Lamp ort Shostak and Pease

Exercises

Prove the b est lower bound you can for the numb er of rounds required in the worst

case to elect leader in a ring of size n Be sure to state your assumptions carefully

Recall the pro of of the lower b ound on the numb er of messages for electing a reader in

a synchronous ring Prove that the bitreversal ring describ ed in class for n for

symmetric any p ositiveinteger k is

Consider a synchronous bidirectional ring of unknown size n in which pro cesses have

UIDs Give upp er and lower b ounds on the numb er of messages required for all the

pro cessors to compute n mo d output is via a sp ecial message

Write pseudo co de for the simple algorithm discussed in class for determining shortest

paths from a single source i inaweighted graph Giveaninvariant assertion pro of

of correctness

Design an algorithm for electing a leader in an arbitrary strongly connected directed

graph network assuming that the pro cesses have UIDs but that they have no knowl

edge of the size or shap e of the network Analyze its time and message complexity

Prove the following lemma If all the edges of a connected weighted graph have distinct

weights then the minimum spanning tree is unique

Consider the following variant of the generals problem Assume that the network is a

complete graph of n participants and that the system is deterministic ie non

randomized The validity requirement is the same as describ ed in class However

supp ose the agreement requirementisweakened to say that if any general decides

then there are at least two that decide That is wewant to rule out the case where

one general attacks alone

Is this problem solvable or unsolvable Prove

JJ Distributed Algorithms Handout Septemb er

Homework Assignment

Due Thursday Octob er

Exercises

Show that the generals problem with link failures is unsolvable in any nontrivial

connected graph

Consider the generals problem with link failures for two pro cesses in a setting where

each message has an indep endent probability p of getting delivered successfully As

sume that each pro cess can send only one message p er round For this setting design

an algorithm that guarantees termination has low probability of disagreementfor

any pair of inputs and has high probability L of attacking in case b oth inputs are

and there are no failures Determine and L in terms of p for your algorithm

In the pro ofs of b ounds for randomized generals algorithms we used two denitions of

information level level i k and mlevel i k Prove that for any A i and k these

A A

two are always within of each other

Be the Adversary

a Consider the algorithm given in class for consensus in the presence of pro cessor

stopping faults based on a lab eled tree Supp ose that instead of running for

f rounds the algorithm only runs for f rounds with the same decision rule

Describ e a particular execution in which the correctness requirements are violated

Hint a pro cess may stop in the middle of a round b efore completing sending

all its messages

b Now consider the algorithm given for consensus in the presence of Byzantine

faults Construct an execution to show that the algorithm gives a wrong result if

it is run with either i no des faults and rounds or ii no des faults

and rounds

Using the stopping fault consensus algorithm as a starting p oint mo dify the correctness

conditions algorithm and pro of as necessary to obtain a consensus algorithm that is

resilient to Byzantine faults assuming that authentication of messages is available

optional Consider the optimized version of the stopping fault consensus algorithm

where only the rst two messages containing distinct values are relayed In this algo

rithm it isnt really necessary to keep all the tree information around Can you reduce

the information kept in the state while still preserving the correctness of the algorithm

You should sketcha proofofwhyyour optimized algorithm works

JJ Distributed Algorithms Handout Octob er

Homework Assignment

Due Thursday Octob er

Reading

Lynch and Tuttle AnIntroduction to InputOutput Automata CWIQuarterly

Exercises

Consider the network graph G depicted in Figure

b d f

c e g

Figure the graph G is a no de connected communication graph

Show directly ie not by quoting the connectivity b ound stated in class but bya

pro of sp ecic to this graph that Byzantine agreement cannot b e solved in G when

pro cessors are faulty

In class a recursive argumentwas used to show imp ossibilityof f round consensus

in the presence of f stopping failures If the recursion is unwound the argument

essentially constructs a long chain of runs where eachtwo consecutive runs in the

chain lo ok the same to some pro cess How long is the chain of runs

Optional Can you shorten this chain using Byzantine faults rather than stopping

faults

Write co de for a correct phase commit proto col

Fill in more details in the inductive pro of of mutual exclusion for Dijkstras algorithm

Describ e an execution of Dijkstras algorithm in which a particular pro cess is lo cked

out forever

JJ Distributed Algorithms Handout Octob er

Homework Assignment

Due Thursday Octob er

Exercises

Consider the Burns mutual exclusion algorithm

a Exhibit a fair execution in which some pro cess is lo cked out

b Do a time analysis for deadlo ckfreedom That is assume that each step outside

the remainder region takes at most time unit and give upp er and lower b ounds

on the length of the time interval in which there is some pro cess in T and no

pro cess in C upp er b ound means analysis that applies to all schedules lower

b ound means a particular schedule of long duration

Why do es the Lamp ort bakery algorithm fail if the integers are replaced by the integers

mo d B for some very large value of B Describ e a sp ecic scenario

Design a mutual exclusion algorithm for a slightly dierent mo del in which there is

one extra pro cess a supervisor process that can always take steps The mo del should

use singlewritermultireader shared variables Analyze its complexity

Design an atomic readmo difywrite ob ject based on lowerlevel multiwriter multi

reader readwrite memory The ob ject should supp ort one op eration apply f where

f is a function In the serial sp ecication the eect of apply f on an ob ject with value

v is that the value of the ob ject b ecomes f v and the original value v is returned to

the user The executions can assumed to b e fair

Consider the following variantoftheunb oundedregisters atomic snapshot ob ject In

this variant the sequence numb er is also incrementedeachtimea snap is completed

rather than only in update Do es this variant preserve the correctness of the original

algorithm

JJ Distributed Algorithms Handout Octob er

Homework Assignment

Due Thursday Octob er

Exercises

Why do es the Lamp ort bakery algorithm fail if the integers are replaced by the integers

mo d B for some very large value of B Describ e a sp ecic scenario

Programmers at the Flaky Computer Corp oration designed the following proto col for

npro cess mutual exclusion with deadlo ckfreedom

Shared variables x y Initially y

Co de for p

Remainder Region

try

x i

if y then goto L

x i

if x i then goto L

crit

Critical Region

exit

rem

Do es this proto col satisfy the two claimed prop erties Sketchwhy it do es or show

that it do esnt If you quote an imp ossibility result to say that it do esnt satisfy the

prop erties then you must still go further and actually exhibit executions in which the

prop erties are violated

Reconsider the consensus problem using readwrite shared memory This time supp ose

that the typ es of faults b eing considered are more constrained than general stopping

failures in particular that the only kind of failure is stopping right at the b eginning

never p erforming any noninput steps Is the consensus problem solvable Givean

algorithm or an imp ossibility pro of

Let A be any IO automaton and supp ose that is a nite execution of A a Prove

that there is a fair execution of A ie an extension of b If is any nite or

innite sequence of input actions of A prove that there is a fair execution of A

such that the sequnce of input actions of beh isidentical to

JJ Distributed Algorithms Handout Octob er

Homework Assignment

Due ThursdayNovember

Exercises

Let A be any IO automaton Show that there is another IO automaton B with only a

single partition class that implements or solves A in the sense that fairbehs B

fairbehs A

Rewrite the Lamp ort bakery algorithm as an IO automaton in preconditioneects

notation While doing this generalize the algorithm slightly byintro ducing as much

nondeterminism in the order of execution of actions as you can The precondition

eects notation makes it easier to express nondeterministic order of actions than do es

the usual owofcontrol notation

Warning Wehavent entirely worked this one out Consider a sequential timestamp

object based on the sequential timestamp system discussed informally in class the

one with the values This is an IO automaton that has a big shared variable

label containing the latest lab els you can ignore the values for all pro cesses and that

has two indivisible actions involving the label vector An action that indivisibly

lo oks at the entire l abel vector cho oses a new lab el according to the rule based on

full comp onents and writes it in the label vector This is all done indivisibly An

label vector action that just snaps the entire

a Write this as an IOA

b Prove the following invariant For anycycleatanylevel at most two of three

comp onents are o ccupied

c Describ e a similar sequential timestamp system based on unb ounded real number

timestamps

d Use a p ossibilities mapping to showthatyour algorithm of a implements your

algorithm of c

Plenty of partial credit will b e given for go o d attempts

JJ Distributed Algorithms Handout Novemb er

Homework Assignment

Due ThursdayNovember

Exercises

Give example executions to showthatthemultiwritermultireader register implementa

tion given in class based on a concurrent timestamp ob ject is not correctly serialized

by serialization p oints placed as follows

a For a read at the p ointoftheemb edded snap op eration For a write at the

point of the emb edded update op eration

b For all op erations at the p ointoftheemb edded snap op eration

Note that the emb edded snap and update op erations are emb edded inside the atomic

snapshot out of which the concurrent timestamp ob ject is built

Consider the following simple ma jorityvoting algorithm for implementing a multi

writer multireader register in majority snapshot shared memory mo del In this mo del

the memory consists of a single vector ob ject one p osition p er writer pro cess with each

p osition containing a pair consisting of a value v and a p ositiveinteger timestamp t

Each read op eration instantaneously reads anymajority of the memory lo cations and

returns the value asso ciated with the largest timestamp Each write op eration instanta

neously reads any ma jority of the memory lo cations determines the largest timestamp

t and writes the new value to anymajority of the memory lo cations accompanied by

timestamp t

Use the lemmaproved in class Lecture to sketch a pro of that this is a correct

implementation of an atomic multiwriter multireader register

Show that every exclusion problem can b e expressed as a resource problem

Show that Lamp orts Construction when applied to atomic registers do es not nec

essarily give rise to an atomic register

JJ Distributed Algorithms Handout Novemb er

Homework Assignment

Due ThursdayNovember

Exercises

Give an explicit IO automaton with a singlewriter multireader user interface that

preserves user wellformedness alternation of requests and resp onses whose fair well

formed b ehaviors contain resp onses for all invo cations and whose wellformed external

b ehaviors are exactly those that are legal for a regular readwrite register Do this also

for a safe readwrite register

Extend the Burns lower b ound on the numb er of messages needed to elect a leader in

an asynchronous ring so that it applies to rings whose sizes are not p owers of two

Give an explicit algorithm that allows a distinguished leader no de i in an arbitrary

connected network graph G to calculate the exact total number of nodes in GSketch

a pro of that it works correctly

Consider a banking system in which eachnodeofanetwork keeps a number in

dicating an amount of moneyWe assume for simplicity that there are no external

dep osits or withdrawals but messages travel b etween no des at arbitrary times con

taining money that is b eing transferred from one no de to another The channels

preserveFIFO order Design a distributed network algorithm that allows eachnode

to decide on ie output its own balance so that the total of all the balances is the

correct amount of money in the system To rule out trivial cases we supp ose that the

initial state of the system is not necessarily quiescent ie there can b e messages in

transit initially

The algorithm should not halt or delay transfers unnecessarily Give a convincing

argument that your algorithm works

JJ Distributed Algorithms Handout Novemb er

Homework Assignment

Due Thursday December

Exercises

This question is op en ended and wehavent worked it out entirely Compare the

op eration of the GallagerHumbletSpira spanning tree algorithm to that of the syn

chronous version discussed earlier in the course Eg is there any relationship b etween

the fragments pro duced in the two cases Perhaps such a connection can b e used in

a formal pro of

Design outline an asynchronous algorithm for a network with a leader no de i that

allows i to discover and output the maximum distance d from itself to the furthest

no de in the network The algorithm should work in time O d What is the message

complexity

Consider a square grid graph G consisting of n n no des Consider a partition P

into k equalsized clusters obtained by dividing eachsideinto k equal intervals In

terms of n and k what are the time and message complexities of synchronizer based

on partition P

Obtain the b est upp er b ound you can for an asynchronous solution to the k session

problem

JJ Distributed Algorithms Handout Decemb er

Homework Assignment

Due Thursday December

Exercises

Develop a wayofsimulating singlewriter multireader shared memory algorithms in an

asynchronous network your metho d should b e based on caching where readers keep

lo cal copies of all the variables and just read their lo cal copies if p ossible The writer

however needs to carry out a more complicated proto col in order to write Describ e

appropriate proto cols for the writer and also for the reader if the lo cal copyisnot

available in order to simulate instantaneous readwrite shared memory Be sure to

guarantee that each pro cess invo cations eventually terminate

Outline whyyour algorithm works correctly

Optimize the mutual exclusion algorithm describ ed in class based on Lamp orts

state machine simulation strategy in order not to keep all the messages ever received

from all other pro cesses

Prove the following invariant for the DijkstraScholten termination detection algorithm

Al l nonid le nodes form a treerooted at the node with status leader where the tree

edges are given by parent pointers

In the shortest paths problem we are given a graph G V E with weights w e

for all e E and a distiguished no de i V The ob jective is to compute at eachnode

its weighted distance from i Consider the BellmanFord algorithm for synchronous

networks dened as follows Eachnode i maintains a nonnegative distance variable

The algorithm pro ceeds in rounds as follows Ateach round all the no des send to

their neighb ors the value of their distance variable The distinguished no de i has

distance xed and all other no des i assign at each round

distance min fdistance w j ig

i j

jiE

Assume that the state of a no de consists only of its distance variable Prove that this

algorithm is selfstabilizing

App endix

Notes for extra lectures

JJ Distributed Algorithms January

Lecturer Nancy Lynch

Lecture

A Implementing Reliable PointtoPoint Communi

cation in terms of Unreliable Channels

In our treatment of asynchronous networks wehavemostlydealtwith

the faultfree case

Exceptions selfstabilization single arbitrary failure

pro cessor stopping faults and Byzantine faults for consensus

Now emphasize failures but in a very sp ecial case a twono de

network

The problem to b e considered is reliable communication b etween users

at two no des

Messages are sentby some user at one no de and are supp osed to b e

delivered to a user at the other

Generally want this delivery to b e reliable FIFO exactly once each

The sender and receiver no des are IO automata

The twochannels inside are generally not our usual reliable channels

however

They mayhave some unreliability prop erties eg

loss of packets

duplication

reordering

However we dont normally allowworse faults suchasmanufacturing of

completely b ogus messages

We also dont allow total packet loss eg havealiveness

assumption that says something like

If innitely manypackets are sent then innitely manyof

them are delivered

Have to b e careful to state this right since we are considering

duplication of packets its not enough just to say that innitely

many sends imply innitely many receives wewant innitely many

of the dierentsendevents to have corresp onding receiveevents

We dont want innitely many receives of some early

message while innitely many new messages keep b eing inserted

Can usually formalize the channels as IO automata using IOA fairness

to express the liveness condition but it may b e more natural

just to sp ecify their allowable external b ehavior

in the form of a cause function from packet receipt events to

packet send events satisfying certain prop erties

Later we will want to generalize the conditions to utilize a

p ort structure and say that the channel exhibits FIFO b ehavior

and liveness with resp ect to each p ort separately Amounts to a

comp osition of channels one for each pair of matching p orts

A Stennings Proto col

The sender places incoming messages in a buer buer

It tags the messages in its buer with successively larger integers

no gaps starting from

The receiver assembles a buer buer also containing

messages with tags starting from

As the buer gets lled in the receiver can deliver the

initial messages in order

Alternative mo deling choice eliminate the senders buer in favor of a

handshake proto col with the user telling it when it can submit the next

message

Other than the basic liveness condition describ ed earlier this

proto col can tolerate a lot of unreliable b ehavior on the

part of the channel loss duplication and reordering

Note that only one direction of underlying channel is required

strictly sp eaking although in this case the sender never knows to

stop sending any particular message

If wewould liketoeventually stop sending each message after it has

b een delivered need to also add an acknowledgement proto col in the

receivertosender direction

The proto col can send lots of messages concurrently but just to b e

sp ecic and simple we describ e a sequential version of the

proto col in which the sender waits to know that the previous

message has b een delivered to the receiver b efore starting to send

the next version

Co de see Figures A and A

Correctness pro of

Can give a highlevel sp ec that is basically a queue MQwith

single state comp onent mqueue

A sendmessage action puts a message on the end and a receivemessage

removes it from the front

Same as our earlier channel sp ec

The liveness condition is then the same as we used b efore for

individual channels in our reliable network mo del eventual delivery

of the rst message easy sp ec as an IOA

The implementation consists of IOAs for the sender and receiver but

it is natural to use a more general mo del for the

underlying channels

However I dont want to just use axioms for external b ehavior

To do a mapping pro of

it is still convenienttohave a state machine even though the

fairness condition needed isnt so easily sp ecied using IOA

fairness

State machine has as its state a set send puts elementinto the set

can deliver a packet anynumb er of times if its in the set never

gets removed from the set

Interface

Input

msg mm M send

receive pkt ii a nonnegativeinteger

Output

send pkt m im M i a nonnegativeinteger

State

buer a nite queue of elements of M initially emptyand

integer a nonnegativeinteger initially

Steps

msg mm M send

Eect

add m to buer

send pkt m im M i a nonnegativeinteger

Precondition

m is rst on buer

i integer

Eect

None

receive pkt ii a nonnegativeinteger

Eect

if i integer then

remove rst element if any from buer

integer integer

Partition

all send pkt actions are in one class

Figure A Stenning Sender A

Interface

Input

receive pkt m in M i a nonnegativeinteger

Output

msg mm M receive

send pkt ii a nonnegativeinteger

State

buer a nite queue of elements of M initially emptyand

integer a nonnegativeinteger initially

Steps

receive msg mm M

Precondition

m is rst on buer

Eect

remove rst element from buer

pkt m im M i a nonnegativeinteger receive

Eect

if i integer then

add m to buer

integer integer

send pkt ii a nonnegativeinteger

Precondition

i integer

Eect

None

Partition

all send pkt actions

msg actions all receive

Figure A Stenning Receiver A

Liveness ie the innitesendinnitereceive condition

is sp ecied as an additional condition a liveness predicate

In order to prove that this works it is useful to havesomeinvariants

Lemma The fol lowing are true about every reachable state of Stennings

protocol

integer integer integer

r s r

All packets in queue have integer tag at most equal to

integer

All packets in queue with integer tag equal to

integer have their message components equal to the rst

message of buer

All packets in queue have integer tag at most equal to

integer

If integer appears anywhere other than the senders

state then buer is nonempty

To show the safety prop ertyshow that every b ehavior of the

implementation is also a b ehavior of the sp ec

Do this by using a forward simulation actually a renement

Sp ecically

If s and u are states of the implementation and sp ecication

resp ectively then we dene u r sprovided that

uqueue is determined as follows

If sinteger sinteger thenuqueue

s r

is obtained by rst removing the rst elementof sbuer and

then concatenating sbuer and the reduced

buer

Invariant implies that sbuer is nonempty

Otherwise uqueue is just the concatenation of

sbuer and sbuer

r s

Nowweshow that r is a renement

The corresp ondence of initial states is easy b ecause all queues are empty

Inductive step

Given s s and u r s we consider the following cases

send msg m

Let u r s

Wemust showthatu u isastepof MQ

This is true b ecause the send msg event mo dies b oth queues in the same way

msg m receive

Let u r s Wemust showu u is a step of MQ Since

is enabled in s m is rst on sbuer so m

is also rst on uqueue so is also enabled in u

Both queues are mo died in the same way

pkt send

Then wemust showthatu r s

This is true b ecause the action do esnt change the virtual queue

pkt m i receive

Weshow that u r s

The only case of interest is if the packet is accepted

This means i sinteger soby Lemma

in state s integer integer

s r

so r su is the concatenation of sbuer and

sbuer

Then in s the integers are equal and m is added to the end of

buer

So r s is determined by removing the rst elementof

s buer

and then concatenating with s buer

Then in order to see that r s isthesameas u it suces to note

that m is the same as the rst elementin sbuer

But Lemma implies that i sinteger

and that m is the same as the rst elementin sbuer as needed

receive pkt i

Again weshowthatu r s

The only case of interest is if the packet is accepted

This means i sinteger which implies by Lemma

that sinteger sinteger

s r

This means that r s is the concatenation of

sbuer and sbuer with the rst elementof

r s

sbuer removed

Then s integer s integer so r s is just

s r

the concatenation of s buer and s buer

s r

But s buer is obtained from sbuer by removing

s s

the rst element so that r s u

Since r is a renement all b ehaviors of Stennings proto col are

also b ehaviors of the sp ecication automaton and so satisfy the

safety prop erty that the sequence of messages in receive msg events is

msg events consistent with the sequence of messages in send

In particular the fair b ehaviors of Stennings proto col satisfy this safety

prop erty

Liveness

The basic idea is that we rst show a corresp ondence b etween an

execution of the impl and of the sp ec MQ

We can get this by noticing that forward simulation actually yields a

stronger relationship than just b ehavior inclusion execution

corresp ondence lemma gives a corresp ondence b etween states and

external actions throughout the execution

Then supp ose that the liveness condition of the sp ecication is

violated

That is the abstract queue is nonempty from some p oint in the

execution but no receivemessage events ever o ccur

We consider what this means in the implementation execution

If the buer is ever nonemptytheneventually deliver

message contradiction so can assume that buer is always empty

Then buer must b e nonempty throughout the execution

sender keeps sending packets for the rst message on buer

gets delivered to receiver bychannel fairness then put

in buer whichisacontradiction

A Alternating Bit Proto col

Optimized version of Stennings proto col that do esnt use sequence

numb ers but only Bo olean values

These get asso ciated with successive messages in an alternating

fashion

This only works for channels with stronger reliability conditions

assume channels exhibit loss and duplication but no reordering

Liveness again says that innitely many sends lead to receives of

innitely many of the sentpackets

Sender keeps trying to send a message with a bit attached

When it gets an ack with the same bit it switches to the opp osite bit

to send the next message

Receiver is also always lo oking for a particular bit

Co de see Figures AA

Correctness Pro of

Mo del the sender and receiver as IOAs just as b efore

Again the channels are not quite IOAs b ecause of the fairness

conditions b eing dierent from usual IOA fairness

But it is again convenienttohave a state machine with a liveness

restriction

This time the channel state machine has as its state a queue

A send puts a copy of the packet at the end of the queue

and a receive can access any element of the queue when it

do es it throws away all the preceding elements but not the

element that is delivered

This allows for message loss and duplication

Again need a liveness condition to say that anypacket that is sent

innitely many times is also delivered innitely many times

A convenient correctness pro of can b e based on a forward simulation this

one is multivalued to a restricted version of Stennings proto col

whichwe call FIFOStenning

This restricted version is correct b ecause the less restricted version is

Interface

Input

msg mm M send

receive pkt bb a Bo olean

Output

send pkt m bm M b a Bo olean

The state consists of the following comp onents

buer a nite queue of elements of M initially emptyand

ag a Bo olean initially

The steps are

msg mm M send

Eect

add m to buer

send pkt m bm M b aBoolean

Precondition

m is rst on buer

b ag

Eect

None

receive pkt bb a Bo olean

Eect

if b ag then

remove rst element if any from buer

ag ag mod

Partition

all send pkt actions are in one class

Figure A ABP Sender A

Interface

Input

receive pkt m bn M b a Bo olean

Output

msg mm M receive

send pkt bb a Bo olean

State

buer a nite queue of elements of M initially emptyand

ag a Bo olean initially

Steps

receive msg mm M

Precondition

m is rst on buer

Eect

remove rst element from buer

pkt m bm M b a Bo olean receive

Eect

if b ag then

add m to buer

ag ag mod

send pkt bb a Bo olean

Precondition

b ag

Eect

None

Partition

all send pkt actions are in one class and

msg actions are in another class all receive

Figure A ABP Receiver A

The restricted version is based on the FIFOchannels describ ed ab ove

instead of arbitrary channels

Use an additional invariant ab out this restricted version of the

algorithm

Lemma The fol lowing is true about every reachable state of the FIFOStenning

protocol

Consider the sequenceconsisting of the indices in queue

in order from rst to last on the queue

fol lowedbyinteger

fol lowed by the indices in queue

fol lowedbyinteger In other words the sequence

starting at queue and tracing the indices al l the way

back to the sender automaton

The indices in this sequencearenondecreasing furthermore the dierence

between the rst and last index in this sequence is at most

Pro of The pro of pro ceeds by induction on the numb er of steps in the nite

execution leading to the given reachable state

The base case is where there are no steps which means wehave to show

this to b e true in the initial state

In the initial state the channels are empty integer and

integer

Thus the sp ecied sequence is which has the required prop erties

For the inductive step supp ose that the condition is true in state s

and consider a step s s of the algorithm

We consider cases based on

msg or receive msg event is a send

Then none of the comp onents involved in the stated condition is changed

by the step so the condition is true after the step

is a send pkt m ievent for some m

Then queue is the only one of the four relevant comp onents of

the global state that can change

Wehave s queue equal to squeue with the addition

sr sr

of m i

But i sinteger by the preconditions of the action

Since the new i is placed consecutively in the concatenated sequence with

s integer i the prop erties are all preserved

pkt ievent is a receive

If i sinteger then the only change is to remove some elements

in the concatenated sequence so all prop erties are preserved

On the other hand if i sinteger then the inductivehyp othesis

implies that the entire concatenated

sequence in state s must consist of is

The only changes are to remove some elements from the b eginning of the

sequence and add one i to the end since s integer i

by the eect of the action Thus the new sequence consists of

all is followed by one i so the prop erty is satised

is a send pkt event

Similar to the case for send pkt

is a receive pkt m ievent

If i sinteger then the only change is to remove some

elements from the concatenated sequence so all prop erties are preserved

On the other hand if i sinteger

then the inductivehyp othesis implies that the entire concatenated

sequence in state s must consist of i s up to and including

sinteger followed entirely by is

The only changes are to remove some elements from the sequence and

change the value of integer from i

to iby the eect of the action this still has the required prop erties

Nowwe will showthattheAB P is correct

by demonstrating a mapping from AB P to FIFOStenning

This mapping will b e a multivalued p ossibilities mapping ie a

forward simulation

In particular it will ll in the integer values of tags only working

from bits

More preciselywesay that a state u of FIFOStenning is in f s

for state s of AB P provided that the following conditions hold

sbuer ubuer

s s

and sbuer ubuer

r r

sag uinteger mo d

s s

and sag uinteger mo d

r r

queue has the same number of packets in s and u

Moreover for any j ifm iisthe j packet in

queue sr in u

then m i mo d is the j packet in queue in s

Also queue has the same number of packets in s and u

Moreover for any j if i is the j packet in

queue in u

then i mo d is the j packet in queue in s

Theorem f above is a forward simulation

Pro of By induction

For the base let s b e the start state of AB P and u the start state

of FIFOStenning

First all the buers are empty

Second sag uinteger mod

s s

and sag and uinteger which is as needed

r r

Third b oth channels are empty

This suces

Now show the inductive step

Supp ose s s is a step of AB P and u f s

We consider cases based on

msg m send

Cho ose u to b e the unique state suchthatu u isastepof

FIFOStenning

Wemust showthatu f s

The only condition that is aected by the step is the rst for the

buer comp onent

and ubuer However the action aects b oth sbuer

s s

in the same way so the corresp ondence holds

receive msg m

Since is enabled in s m is the rst value on sbuer

Since u f s m is also the rst value on ubuer

which implies that is enabled in u

Nowcho ose u to b e the unique state suchthatu u is a step of

FIFOStenning

All conditions are unaected except for the rst for buer and

buer is changed

in the same way in b oth algorithms so the corresp ondence holds

send pkt m b

Since is enabled in s

b sag and m is the rst elementon sbuer

s s

Let i uinteger

Since u f s m is also the rst elementon ubuer

pkt m i is enabled in u It follows that send

Nowcho ose u so that u u is a step of FIFOStenning

Wemust showthatu f s

The only interesting condition is the third for C

By inductivehyp othesis and the fact that the two steps each insert

one packet it is easy to see that

C has the same numb er of packets in s and u

Moreover the new packet gets added with tag i in state u and with

tag b in state s

since u f s wehave sag uinteger mo d

s s

ie b i mo d

which implies the result

receive pkt m b

Since is enabled in sm b app ears in C in s

Since u g s m i app ears in the corresp onding p osition in

C in u

for some integer i with b i mo d

pkt m i then is enabled in u Let receive

Let u b e the state such that u u is a step of

FIFOStenning and such that the same number of packets is removed

from the queue

Wemust showthatu f s

It is easy to see that the third condition is preserved

since eachof and

removes the same number of packets from C

Supp ose rst that b sag

Then the eects of imply that the receiver state in s is identical

to that in s

Now since u f s sag uinteger mo d

since b i mo d this case must have

i uinteger

Then the eects of imply that the receiver state in u is

identical to that in u

It is immediate that the rst and second conditions hold

So now supp ose that b sag

The invariantabove for FIFOStenning implies that either i uinteger

or i uinteger

Since b i mo d and

since u f s sag uinteger mo d

r r

this case must have i uinteger

Then by the eect of the action u integer uinteger

r r

and s ag sag

r r

preserving the second condition

Also buer is mo died in b oth cases by adding the entry m

at the end therefore the rst condition is preserved

pkt b send

Similar to send pkt m b

receive pkt b

Since is enabled in s b is in C in s

in u Since u f s i is the corresp onding p osition in C

for some integer i with b i mo d

Let receive pkt i then is enabled in u

Let u b e the state such that u u is a step of

FIFOStenning and such that the same number of packets are removed

Wemust showthatu f s

It is easy to see that the third condition is preserved

since eachof and

removes the same numb er of messages from C

Supp ose rst that b sag

Then the eects of imply that the sender state in s

is identical to that in s

Now since u f s sag uinteger mo d

s s

since b i mo d this case must have

i uinteger

Then the eects of imply that the sender state in u is

identical to that in u

It is immediate that the rst and second conditions hold for this situation

So now supp ose that b sag

The invariantabove for FIFOStenning implies that either i uinteger

or i uinteger

Since b i mo d and

since u f s sag uinteger mo d

s s

this case must have i uinteger

Then the eect of the action implies that

u integer uinteger

s s

and s ag sag preserving the second condition

s s

Also buer is mo died in the same way in b oth cases

so the rst condition is preserved

Remark

Consider the structure of the forward simulation f of this example

In going from FIFOStenning to AB P

integer tags are condensed to their loworder bits

The multiple values of the mapping f essentially replace this

information

In this example the corresp ondence b etween AB P and FIFOStenning

can b e describ ed

in terms of a mapping in the opp osite direction a singlevalued

pro jection from the state of FIFOStenning to that of AB P that

removes information

Then f maps a state s of AB P to

the set of states of FIFOStenning whose pro jections are equal to s

While this formulation suces to describ e manyinteresting examples

it do es not always work

Consider garbage collection examples

Machine assistance should b e useful in verifying and mayb e pro ducing

such pro ofs

Liveness

As b efore the forward simulation yields a close corresp ondence

between executions of the two systems

Given any live execution of AB P construct a corresp onding

execution of FIFOStenning needs formalizing

Show that is a live execution of FIFOStenning

Includes conditions of IOA fairness for the sender and receiver

If not live in this sense then some IOA class is enabled forever but no

action in that class o ccurs easily yields the same situation in

if the corresp ondence is stated strongly enough which

contradicts the fairness of the sender and receiver in

Also includes the channel liveness conditions innitely many

sends imply innitely many receives

Again this carries over from ABP

So far we see that we can tolerate all typ es of channel faults using

Stenning with unb ounded headers for messages

With b ounded headers as in ABP can tolerate loss and duplication

but not reordering

A BoundedHeader Proto cols Tolerating Reordering

This leads us to the question of whether we can use b ounded headers

and tolerate any reordering

Consider what go es wrong with the ABP for example if we attempt to

use it with channels that reorder packets

The recipient can get fo oled into accepting an old packet with the

same bit as the one exp ected

This could cause duplicate acceptance of the same message

Here we consider the question of whether there exists a proto col that

uses b ounded headers and tolerates reordering

WangZuckprove that there is no b oundedheader proto col that tolerates

duplication and reordering and consequently none that tolerates all

three typ es of faulty b ehavior

In contrast AAFFLMWZ pro duce a b oundedheader proto col that tolerates

loss and reordering but not duplication

That pap er also contains an imp ossibility result for ecient

b oundedheader proto cols tolerating loss and reordering

Formalize the notion of b oundedheader by assuming that the external

message alphab et is nite and requiring that the packet alphab et

also b e nite

WangZuck ReoDup Imp ossibility

Supp ose there is an implementation

Starting from an initial state run the system to send

systematically copies of as many dierentpackets as p ossible

yielding execution

That is there is no extension of in whichanypacket is sent

that isnt also sentin

Supp ose there are n sendmessage events in

Let b e a live extension of that contains exactly one

additional sendmessage event

By the correcteness condition all messages in get delivered

Nowwe construct a nite execution

to the receiver just up to the p oint that lo oks like

where the n st message delivery to the user o ccurs

but in which the additional sendmessage eventnever o ccurs

Namely delay the sender immediately after also dont allow

the new sendmessage event

But let the receiver do the same thing that it do es in

receive the same packets send the same packets deliver the same

messages to the user

It can receive the same packets even though the sender do esnt send

them b ecause the earlier packets in can b e duplicated

nothing new can b e sent that do esnt app ear already in

Then as in it delivers n messages to the user

even though only n sendmessage events have o ccurred

To complete the contradiction extend to a live execution

of the system without any new sendmessage events

This has more receivemessage events than sendmessage events a

contradiction

AAFFLMWZ BoundedHeader Proto col Tolerating Loss and Reordering

It turns out that it is p ossible to tolerate reordering

and also loss of messages with b ounded headers

This assumes that duplication do es not o ccur

It is not at all obvious how to do this however it was a conjecture

for a while that this was imp ossible that tolerating reordering

and loss would require unb ounded headers

This algorithm is NOT a practical algorithm just a counterexample

algorithm to show that an imp ossibility result cannot b e proved

The algorithm can most easily b e presented in two pieces

The underlying channels are at least guaranteed not to duplicate

anything

So rst use the nodup channels to implementchannels that do not

reorder messages but can lose or duplicate them

Then use the resulting FIFOchannels to implement reliable

communication

Note that pasting these two together requires each underlying channel

to b e used for two separate purp oses to simulate channels in two

dierent proto cols one yielding FIFOcommunication in either

direction

This means that the underlying channels need to b e fair to each

proto col separately need the p ort structure mentioned earlier

The second task can easily b e done using b ounded headers eg use

the ABP

The rst is much less obvious

The sender sends a value to the receiver only in resp onse to an

explicit query packet from the receiver

The value that it sends is always the most recent message that was

given to it saved in latest

To ensure that it answers each query exactly once the

sender keeps a variable unanswered which is incremented

whenever a new query is received and decremented whenever a

value is sent

The receiver continuously sends query s to the sender keeping

track in pending of the number of unanswered queries

The receiver counts in count m the numb er of copies of each

value m received since the last time it delivered a message

or since the b eginning of the run if no message has yet b een put on

that buer

At the b eginning and whenever a new message is delivered the receiver

sets old to pending

When count m old the receiver knows that m was the

value of latest at some time after the receiver

p erformed its last receiveevent

It can therefore safely output m

The niteness of the message alphab et the fact that the sender

will always resp ond to query messages and the liveness of the

channels together imply that the receiver will output innitely

manyvalues unless there is no send event at all

Co de

Sender

Variables

latest an elementof M or nil initially nil

unanswered a nonnegativeinteger initially

send m

Eect

latest m

rcv pkt query

Eect

unanswered unanswered

send pkt m

Precondition

unanswered

m latest nil

Eect

unanswered unanswered

Receiver

Variables

pending a nonnegativeinteger initially

old a nonnegativeinteger initially

for each m M count m a nonnegativeinteger

initially

receive m

Precondition

count m old

Eect

count n for all n M

old pending

send pkt query

Eect

pending pending

rcv pkt m

Eect

pending pending

count m count m

JJ Distributed Algorithms January

Lecturer Nancy Lynch

Lecture

B Reliable Communication Using Unreliable Chan

nels

B BoundedHeader Proto cols Tolerating Reordering

We b egan considering b oundedheader proto cols tolerating reordering

Wegave a simple imp ossibility result for reordering in combination

with duplication and then showed the AAFFLMWZ proto col that

tolerates reordering in combination with loss but no duplication

Todaywe start with an imp ossibilityresultsaying that its not

p ossible to tolerate reordering and loss with an ecient proto col

AAFFLMWZ Imp ossibility Result

sr rs

Again consider the case of channels C and C

that cannot duplicate messages but can lose and reorder them

Again if innitely manypackets are sent innitely many of these packets

are required to b e delivered

The imp ossibility pro of will still work if we generalize this condition to a

corresp onding p ortbased condition

As ab ove we can get a proto col for this setting

that proto col was very inecient in the sense that more and more

packets are required to send individual messages

Nowwe will show that this is inherent such a proto col cannot b e

ecientinthatit must send more and more messages

Again assume nite message alphab et and nite packet alphab et

To capture the ineciency need a denition to capture the number

of packets needed to send a message

First a nite execution is valid

if the number of sendmsg

actions is equal to the number of recmsg actions

This means that the datalink proto col succeeded in sending all the

messages m provided by the user ieall the messages m for which

an action sendmsg m o ccurred

The following denition expresses the notion that in order to successfully

deliver any message the datalink proto col only needs in the b est case

to send a bounded numb er of packets over the channels

Denition If is a valid execution an extension is a

k extension if

In the user actions are exactly the two actions

sendmsgm

and recmsgm for some given message m

This means that exactly one message has been sent successful ly

by the protocol

All packets receivedin are sent in ieno old

packets arereceived

The number of recpkt actions in

is less than or equal to k

Aprotocol is k b ounded if thereisak extension of for

every message m and every valid execution

Theorem Thereisno k bounded datalink protocol

Assume there is such a proto col and lo ok for a contradiction

Supp ose that there is a multiset T of packets a nite execution

and a kextension such that

every packet in T is in transit from the s to r

after the execution ieT is a multiset of old packets

the multiset of packets received by the receiver in is a

submultiset of T

We could then derive a contradiction

Consider an alternative execution that b egins similarly with

but that contains no sendmsg event

All the packets in T cause the receiver to b ehave as in the

k extension and hence to generate an incorrect

recmsg action

In other words the receiver is confused by the presence of old

packets in the channel whichwere left in transit in the channel in

and are equivalent to those sentin

At the end of the alternative execution a message has b een received

without its b eing sent and the algorithm fails

Note that we are here assuming a universal channel one that

can actually exhibit all the b ehavior that is theoretically allowed

by the channel sp ec Such universal channels exist

So it remains to manufacture this bad situation

We need one further denition

Denition T T if

T T This inclusion is among multisets of packets

packet p st multp T multp T k

mult p T denotes the multiplicity of p within the multiset T

Lemma If is valid and T is a multiset of packets in transit after

then either

there exists a k extension such that the

multiset of packets

receivedbyA in is a submultiset of T or

there exists a valid execution such that

a al l packets receivedin are sent in and

b a new multiset T of packets in transit

after such that

T T

Supp ose that this lemma is true

We then show that there is some valid and a multiset

T of packets in transit after

such that case holds

As we already argued the existence of such and

T leads to the desired contradiction

For this we dene two sequences and T with

i i

equal to the empty sequence and T empty

If condition do es not hold for and T iefor i

we are in the situation of case

We then set and T T

In general assuming that case do es not hold for and

T we are then in case and derivea valid extension

of andamultiset T of packets in

i i i

transit after

T T

i i

But by denition of the

relation

the sequence T T T

k k k k

can haveatmostk jP j terms

Its last term is the T we are lo oking for

jP j is the numb er of dierent p ossible packets which is nite

b ecause we assumed that the packet alphab et and the size of the

headers are b ounded

Pro of of Lemma

Pickany m and get a k extension for mby

the k b oundedness condition

If the multiset of packets received by A is included in T we

are in case a

Otherwise there is some packet p for whichthemultiplicity

of recpkt p actions in is bigger then

multp T

We then set T T fpg

Since the extension is k b ounded the number of

of these recpktpisatmostk so that

multp T k

Wenowwant to get a valid extension of

leaving the packets from T in transit

We know that there is a sendpkt p action in

since all packets received in were sentin

Consider then a prex of ending with the

rst such sendpkt p See Figure B

send(p) rec(p) αβ

αγ

Figure B An extension of with a send pk tp action

After all packets from T are still in transit

Wewant to extend to a valid execution without

delivering any packet from T

There are three cases

First supp ose that b oth the sendmessage and receivemessage events

from app ear in then itself suces

Second supp ose that only the sendmessage but not the

receivemessage event app ears in

To get validitywe need to deliver mwhich is the only

undelivered message without delivering a packet from T

Wecanachieve this b ecause of a basic prop ertyoflive executions in

this architecture for any nite execution there is a live

extension that contains no new sendmessage eventsandthatdoesnot

deliver any old packets

Because of the correctness condition this must eventually deliver

The needed sequence stops just after the receivemessage event

Third and nally supp ose that neither the sendmessage nor the

receivemessage event app ears in

Then add a new sendmessage event just after and

extend this just as in the previous case to get the needed valid extension

B Tolerating No de Crashes

The results ab ove settle prettymuch all the cases for reliable no des

Nowwe add consideration of faulty no des

This time we consider no de crashes that lose information

We givetwo imp ossibility results then show a practical algorithm

used in the internet

Imp ossibility Results

Consider the case where each no de has an input action crash

and a corresp onding output recover

When a crash o ccurs a sp ecial crash ag just gets set and no

lo cally controlled actions other than recover can b e enabled

At some later time a corresp onding recover action is supp osed

to o ccur when this happ ens the state of the no de go es backtoan

arbitrary initial state

Thus the crash causes all information to b e lost

In such a mo del it is not hard to see that we cant solve the

reliable communication problem even if the underlying channels are

completely reliable FIFO no loss or duplication

Pro of

Consider any live execution in whicha single

send mevent o ccurs but no crashes

Then correctness implies that there is a later receive m

Let b e the prex ending just b efore this receive m

Now consider followed by the events crash recover

r r

By basic prop erties of the mo del

this can b e extended to a live execution in whichno

further sendmessage events or crashes o ccur

Since must also satisfy the correctness conditions it must

b e that contains a receive mevent corresp onding

to the given send mevent sometime after the given crash

Now take the p ortion of after the crashrecover

and splice it at the end of the nite execution

receive mcrash recover

r r

Claim that the result is another live execution

But this execution has two receive mevents a contradiction to the

correctness conditions

Thus the key idea is that the system cannot tell whether a receiver crash

o ccurred b efore or after the delivery of a message

Note that a key fact that makes this result work is the separation

between the receive meventandany other eect suchas

sending an acknowledgement

We could argue that the problem statement is to o strong for the case

of crashes

Nowweweaken the problem statement quite a lot but still obtain an

imp ossibility result though its harder to prove

Wecontinue to forbid duplication

We allow reordering however

And most imp ortantlywenow allow loss of messages sent b efore the

last recover action

Messages sent after the last recover must b e delivered however

We describ e the underlying channels in as strong a way as p ossible

Thus we insist that they are FIFO and dont duplicate

All they can do is lose messages

The only constraint on losing messages is the liveness constraint

innitely many sends lead to innitely many receives

or a p ort version of this

Nowitmakes sense to talk ab out a sequence of packets b eing

in transit on a channel

By this we mean that the sequence is a subsequence of all the packets

in the channel sent after the sending of the last packet delivered

Notation

If is an execution x fs r g k jj

then dene

in x k to b e the sequence of packets received by A

during the rst k steps of

out x k to b e the sequence of packets sentby A

during the rst k steps of

state x k to b e the state of A after exactly k

steps of and

ext x k to b e the sequence of external actions of A

o ccurring during the rst k steps of

Denex to b e the opp osite no de to x

Lemma Let be any crashfree nite execution of length nLet x beeithernode and let

th x

k n Suppose that either k or else the k action in is an action of A Then

there is a nite execution possibly containing crashes that ends in a state in which

the state of x is state x k

the state of x is state x k and

the sequenceout x k is in transit from x

Pro of Pro ceed by induction on k

k is immediate use the initial conguration only

Inductive step

Supp ose true for all values smaller than k and provefork

Case The rst k steps in are all steps of A

Then the rst k steps of suce

Case The rst k steps in include at least one step of

Then let j b e the greatest integer k such that the k

step is a step of A

Note that in fact jk since wehave assumed the k step is a

step of A

Then in x k is a subsequence of

out x j and

state x k state x j

By inductivehyp othesis we get an execution that leads the

two no des to state x j and

state x j resp ectively and that has

out x j in transit fromx to x

This already hasx in the needed state

Now run A alone let it rst crash and recover going backto

its initial state in

Now let it run on its own using the messages in

in x k which are available in the incoming channel since

they are a subsequence of out x j

This brings x to the needed state and puts the needed messages in

the outgoing channel

Nowwe nish the pro of

Cho ose length n to b e a crashfree execution containing one

send m and its corresp onding receive mmust exist

Now use to construct an execution whose nal no de states are

those of

and that has a send as the last external action

How to do this

Let k b e the last step of o ccurring at the receiver must

exist one since there is a receive eventin

Then Lemma yields an execution ending in the no de states

after k steps and with out rk in transit

Note that state rk state rn

Now crash and recover the sender and then have it run again from

the b eginning using the inputs in in s n which are a

subsequence of the available sequence out rk

This lets it reach state s n

Also there is a send step but no other external step in

this sux

This is as needed

Nowwe get a contradiction

Let b e the execution obtained ab ove whose nal no de

states are those of

and that has a send as the last external action

Now lose all the messages in transit in b oth channels then extend

to a live execution containing no further send or crash

events

By the correctness this extension must eventually deliver the last

message

Now claim we can attach this sux at the end of andit

will again deliver the message

But alread had an equal numb er of sends and receives one of

each so this extra receive violates correctness

QED

This says that we cant even solvea weak version of the problem when

wehave to contend with crashes that lose all state information

B packet HandshakeInternet Proto col

But note that it is imp ortant in practice to have a message delivery

proto col that can tolerate no de crashes

Here we give one imp ortant example the packet handshake proto col of

Belsnes used in the Internet

This do es tolerates no de crashes

It uses a small amount of stable storage in the form of unique IDs

think of this as stable storage b ecause even after a crash the system

rememb ers not to reuse anyID

Underlying channels can reorder duplicate but only nitely many

times and lose

Yields no reordering no duplication ever and always delivers

messages sent after the last recovery

The vepacket handshake proto col of Belsnes is one of a batch

he designed getting successively more reliability out of successively

more messages Used in practice

For each message that the sender wishes to send

there is an initial exchange of packets b etween the

sender and receiver to establish a commonlyagreedup on message

identier

That is the sender sends a new unique identier called jd tothe

receiver which the receiver then pairs with another new unique

identier called id

It sends this pair back to the sender who knows that the new

id is recent b ecause it was paired with its current jd

The identier jd isnt needed any longer but id is

used for the actual message communication

The sender then asso ciates this identier id with the message

The receiver uses the asso ciated identier to decide whether or not

to accept a received message it will accept a

message provided the asso ciated identier id is equal to the

receivers current id

The receiver also sends an acknowledgement

Additional packets are required in order to tell the receiver when it

can throwaway a current identier

Variable Typ e Initially Description

mode acked acked Senders mo de

needid

send rec

buf M list empty Senders list of messages to send

jd JD or nil nil jd chosen for current message by

sender

jd used S JD set empty Includes all jd sever used by sender

id D or nil nil id received from receiver

current msg M or nil nil Message ab out to b e sent to receiver

current ack Bo olean false Ack from receiver

acked buf D list empty List of id s for which the sender will is

sue acked message

mode idle idle Receivers mo de

rcvd ack

rec

buf M list empty Receivers list of messages to deliver

jd JD or nil nil jd received from sender

id D or nil nil id chosen for received jd by receiver

last D or nil nil Last id the receiver remembers

accepting

issued S D set empty Includes all id sever issued byreceiver

nack buf D list empty List of id sforwhich receiver will issue

negativeacks

Sender Actions

Input

msg m m M send

receive pkt p p P

crash

Output

Ackb b a Bo olean

pkt p p P send

recover

Internal

choose jd

grow jd used

Input

pkt p p P receive

crash

Output

receive msg m m M

send pkt p p P

recover

Internal

grow issued

msg m send

Eect

if mode rec then

add m to buf

choose jd

Precondition

mode acked

buf nonempty

jd jd used

Eect

mode needid

current msg head buf

s s

remove head buf

jd jd

add jd to jd used

receive send pkt needid jd pkt needid jd

sr sr

Precondition Eect

mode needid if mode idle then

s s

jd jd mode accept

Eect cho ose an id not in issued

none jd jd

id id

add id to issued

pkt accept jd id send pkt accept jd id receive

rs rs

Eect Precondition

if mode rec then mode accept

s r

if mode needid jd jd

and jd jd then id id

mode send Eect

id id none

else if id id then

add id to acked buf

send pkt sendmid pkt sendmid receive

sr sr

Precondition Eect

mode send if mode rec then

s r

current msg m if mode accept

id id and id id then

s r

Eect mode rcvd

none add m to buf

last id

else if id last then

add id to nack buf

msg m receive

Precondition

mode rcvd

m rst on buf

Eect

remove head buf

if buf is empty then

mode ack

pkt ack id b pkt ack id true receive send

rs rs

Eect Precondition

if mode rec then mode ack

s r

if mode send and last id

s r

id id then Eect

mode acked none

current ack b

pkt ack id false send

jd nil

Precondition

id nil

mode rec

current msg nil

id rst on nack buf

if b true then

Eect

add id to acked buf

remove head nack buf

send receive pkt acked id nil pkt acked id nil

sr sr

Precondition Eect

mode rec if mode accept

s r

id rst on acked buf and id id or

Eect mode ack

remove head acked buf and id last then

mode idle

jd nil

id nil

last nil

Ackb

Precondition

mode acked

buf is empty

b current ack

Eect

none

crash crash

s r

Eect Eect

mode rec mode rec

s r

recover recover

s r

Precondition Precondition

mode rec mode rec

s r

Eect Eect

mode acked mode idle

s r

jd nil jd nil

s r

id nil id nil

s r

empty buf last nil

current msg nil empty buf

s r

current ack false empty nack buf

empty acked buf

grow jd used grow issued

s r

Precondition Precondition

none none

Eect Eect

add some JD stojd used add some D stoissued

Explain the algorithm from mytalk overview discuss whyitworks

informally

Note the two places where msgs get added to the ackedbuer once

normally once in rep onse to information ab out an old message

The trickiest part of this co de turns out to b e the liveness argument

howdoweknow that this algorithm do es continue to make progress

There are some places where only single resp onses are sent eg a

single nack but that could get lost

However in this case the no de at the other end is continuing to

resend so eventually another nack will b e triggered

JJ Distributed Algorithms January

Lecturer Nancy Lynch

Lecture

This is the nal lecture

Its on timingbased computing

Recall that we started this course out studying synchronous

distributed algorithms sp ent the rest of the time doing

asynchronous algorithms

There is an interesting mo del that is in b etween these two really

a class of mo dels

Call these partial ly synchronous

The idea is that the pro cesses can presume some information ab out

time though the information might not b e exact

For example we mighthave b ounds on pro cess step time or message

delivery time

Note that it do esnt change things much just to include upp er b ounds

b ecause the same set of executions results

In fact wehave b een asso ciating upp er b ounds with events throughout

the course when we analyzed time complexity

Rather we will consider b oth lower and upp er b ounds on the time for

events

Nowwe do get some extra p ower in the sense that we are now

restricting the p ossible interleavings

C MMT Mo del Denition

A natural mo del to use is the one of Merritt Mo dugno and Tuttle

Here Im describing the version thats used in Lynch Attiya pap er

This is the same as IO automata but noweach class do esnt just get

treated fairly

rather havea boundmap which asso ciates lower and upp er

b ounds with each class

The lower b ound is any nonnegative real numb er while the upp er b ound

is either a p ositive real or

The intuition is that the successive actions in each class arent

supp osed to o ccur any so oner after the class is enabled or after

the last action in that class than indicated bythelower b ound

Also no later than the upp er b ound

Executions are describ ed as sequences of alternating states and

actiontime pairs

Admissibility

Ive edited the following from the Lynch Attiya mo del pap er Section

C Timed Automata

In this subsection we augment the IO automaton mo del to allow discussion of timing

prop erties The treatment here is similar to the one describ ed in AttiyaLand is a sp ecial

case of the denitions prop osed in MerrittMT A boundmap for an IO automaton A is

a a mapping that asso ciates a closed subinterval of with eac h class in part A where

the lower b ound of eachinterval is not and the upp er b ound is nonzero Intuitively

the interval asso ciated with a class C by the b oundmap represents the range of p ossible

lengths of time b etween successive times when C gets a chance to p erform an action We

sometimes use the notation b C to denote the lower b ound assigned by b oundmap b to

class C and b C for the corresp onding upp er b ound A timed automaton is a pair A b

where A is an IO automaton and b is a b oundmap for A

We require notions of timed execution timed schedule and timed b ehavior for

timed automata corresp onding to executions schedules and b ehaviors for ordinary IO

automata These will all include time information We b egin by dening the basic typ e of

sequence that underlies the denition of a timed execution

In MerrittMT the mo del is dened in a more general manner to allow b oundmaps to yield op en or

semiop en intervals as well as closed intervals This restriction is not crucial in this pap er but allows us to

avoid considering extra cases in some of the technical arguments

Denition A timed sequence for an IO automaton A is a nite or innite sequence

of alternating states and actiontime pairs

s t s t

satisfying the fol lowing conditions

The states s s are in states A

The actions areinactsA

The times t t aresuccessively nondecreasing nonnegative real numbers

If the sequence is nite then it ends in a state s

If the sequence is innite then the times are unbounded

For a given timed sequence we use the convention that t For any nite timed

sequence wedene t to b e the time of the last eventin if contains any ac

end

tiontime pairs or if contains no such pairs Also we dene s to b e the last state

end

in We denote by ord the ordinary part of the sequence

s s

ie with time information removed

If i is a nonnegativeinteger and C part A wesay that i is an initial index for C in

if s enabled A C and either i or s disabled A C or C Thus an initial

i i i

index for class C is the index of an eventatwhich C b ecomes enabled it indicates a p oint

in from whichwe will b egin measuring upp er and lower time b ounds

Denition Suppose A b is a timed automaton Then a timedsequence is a timed

execution of A b provided that or d is an execution of A and satises the fol lowing

conditions for each class C part A and every initial index i for C in

If b C then there exists jiwith t t b C such that either C or

u j i u j

s disabled A C

Theredoes not exist jiwith t t b C and in C

j i j

The rst condition says that starting from an initial index for C within time b C

either some action in C o ccurs or there is a p ointatwhichnosuch action is enabled Note

that if b C no upp er b ound requirement is imp osed The second condition says

that again starting from an initial index for C no action in C can o ccur b efore time b C

has elapsed Note in particular that if a class C b ecomes disabled and then enabled once

again the lower b ound calculation gets restarted at the p oint where the class b ecomes

reenabled

The timedschedule of a timed execution of a timed automaton A b is the subsequence

consisting of the actiontime pairs and the timedbehavior is the subsequence consisting

of the actiontime pairs for which the action is external The timed schedules and timed

behaviors of A b are just those of the timed executions of A b

We mo del each timingdep endent concurrent system as a single timed automaton A b

where A is a comp osition of ordinary IO automata p ossibly with some output actions

hidden We also mo del problem sp ecications including timing prop erties in terms of

timed automata

We note that the denition we use for timed automata may not b e suciently general to

capture all interesting systems and timing requirements It do es capture manyhowever

The MMT pap er contains a generalization of this mo del in whichwehave b ounds that

can b e op en or closed Also upp er b ound of allowed Finally dont restrict to only

nitely many pro cesses Do results ab out comp osition similar to IO automata

C Simple Mutual Exclusion Example

Just as an illustrative example Ill describ e a very basic problem we

considered within this mo del to get simple upp er and lower b ound

results for a basic problem

Started with mutual exclusion

When we did this wedidnothave a clear idea of which parameters

etc would turn out to b e the most imp ortant so we considered

everything

C Basic Pro of Metho ds

Algorithms such as the ones in this pap er quickly lead to the need for

pro of metho ds for verifying correctness of timingbased algorithms

An equivalentwayoflookingateach system is as a comp osition of timed automata An appropriate

denition for a comp osition of timed automata is develop ed in MerrittMT together with theorems

showing the equivalence of the two viewp oints

In the untimed setting wemake heavy use of invariants and

simulations so it seems most reasonable to try to extend these

metho ds to the timed setting

However it is not instantly obvious how to do this

In an asynchronous system everything imp ortant ab out the systems

state ie everything that can aect the systems future

behavior is summarized in the ordinary states of the pro cesses and

channels ie in the values of the lo cal variables and the multiset of

messages in transit on each link

In a timingbased system that is no longer the case

Now if a message is in transit it is imp ortanttoknow whether it has

just b een placed into a channel or has b een there a long time and is

therefore ab out to b e delivered

Similar remarks hold for pro cess steps

Therefore it is convenient to augment the states with some extra

predictive timing information giving b ounds on how long it will b e

b efore certain events mayormust o ccur

C Consensus

Nowwe can also revisit the problem of distributed consensus

in the timed setting

This time for simplicitywe ignore the separate clo cks and just

supp ose wehave pro cesses with upp er and lower b ounds of c c

on step time

The questions is howmuch time it takes to reach consensus if there

are f faulty pro cesses

C More Mutual Exclusion

LynchShavit mutual exclusion with pro ofs

C Clo ckSynchronization

Work from Lamp ort MelliarSmith

Lundelius Lynch

Fischer Lynch Merritt

JJ Distributed Algorithms Handout Septemb er

Course Reading List

General References

N Lync h and I Saias Distributed Algorithms Fall Lecture Notes for

MITLCSRSS Massachusetts Institute of TechnologyFebruary

N Lync h and K Goldman Distributed Algorithms MITLCSRSS Massachusetts

Institute of Technology Lecture Notes for

M Ra ynal Algorithms for Mutual Exclusion MIT Press

M Ra ynal Networks and Distributed Computation MIT Press

J M H elary and M Raynal Synchronization and Control of Distributed Systems and

Programs John Wiley Sons

K M Chandy and J Misra Paral lel program design a foundation AddisonWesley

Butler Lampson William W eihl and Eric Brewer Principles of computer sys

tems Fall MITLCSRSS Massachusetts Institute of Technology

Lecture Notes and Handouts

D Intro duction

L Lamp ort and N Lync h Distributed computing Chapter of Handb o ok on The

oretical Computer Science Also app eared as Technical Memo MITLCSTM

Lab oratory for Computer Science Massachusetts Institute of TechnologyCambridge

MA February

D Mo dels and Pro of Metho ds

D Automata

Nancy Lync handFrits Vaandrager Forward and backward simulations for timing

based systems In Proceedings of REX Workshop RealTime Theory in Practice

volume of Lecture Notes in Computer Science Mo ok The Netherlands

SpringerVerlag Also MITLCSTM

D Invariants

S Owic ki and D Gries An axiomatic pro of technique for parallel programs Acta

Informatica

L Lamp ort Sp ecifying concurren t program mo dules ACM Transactions on Program

ming Languages and Systems April

Butler Lampson William W eihl and Eric Brewer Principles of computer sys

tems Fall MITLCSRSS Massachusetts Institute of Technology

Lecture notes and Handouts

D Mapping Metho ds

Nancy Lync handFrits Vaandrager Forward and backward simulations for timing

based systems In Proceedings of REX Workshop RealTime Theory in Practice

volume of Lecture Notes in Computer Science Mo ok The Netherlands

SpringerVerlag Also MITLCSTM

Martin Abadi and Leslie Lamp ort The existence of renement mapping Theoretical

Computer Science

Nils Klarlund and F red B Schneider Proving nondeterministically sp ecied safety

prop erties using progress measures May ToappearinInformation and Compu

tation Spring

D Shared Memory Mo dels

N Lync h and M Fischer On describing the b ehavior and implementation of dis

tributed systems Theoretical Computer Science

Kenneth Goldman Nancy Lync h and KathyYelick Mo delling shared state in a shared

action mo del April Manuscript Also earlier verison in Proceedings of the th

Annual IEEE Symposium on Logic in Computer Science pages Philadelphia

PA June

D IO Automata

Nancy A Lync h and Mark R Tuttle An intro duction to inputoutput automata

CWIQuarterly

Mark T uttle Hierarchical correctness pro ofs for distributed algorithms Masters

thesis Massachusetts Institute of Technology MIT Dept of Electrical Engineering

and Computer Science April

Nic k Reingold DaWei Wang and Lenore Zuck Games IO automata play September

Technical Rep ort YALEDCSTR

DaW ei Wang Lenore Zuck and Nick Reingold The p ower of IO automata

Manuscript May

D Temp oral Logic

Z Manna and APn ueli The Temporal Logic of Reactive and Concurrent Systems

SpringerVerlag

K M Chandy and J Misra Paral lel program design a foundation AddisonWesley

Leslie Lamp ort The temp oral logic of actions T echnical Rep ort Digital Systems

ResearchCenter December

D Timed Mo dels

F Mo dugno M Merritt and M T uttle Time constrained automata In CONCUR

Proceedings Workshop on Theories of Concurrency Unication and ExtensionAms

terdam August

Nancy Lync handFrits Vaandrager Forward and backward simulations for timing

based systems In Proceedings of REX Workshop RealTime Theory in Practice

volume of Lecture Notes in Computer Science Mo ok The Netherlands

SpringerVerlag Also MITLCSTM

Ra jeev Alur and Da vid L Dill A theory of timed automata To app ear in Theoretical

Computer Science

FW V aandrager and NA Lynch Action transducers and timed automata In Pro

ceedings of CONCUR rd International Conference on Concurrency Theory Lec

ture Notes in Computer Science Stony Bro ok NY August Springer Verlag To

app ear

N Lync h and H Attiya Using mappings to prove timing prop erties Distrib Comput

To app ear

Thomas A Henzinger Zohar Manna and Amir Pn ueli Temp oral pro of metho dologies

for realtime systems September Submitted for publication An abbreviated

version in HMP

T A Henzinger Z Manna and A Pn ueli Temp oral pro of metho dologies for real

time systems In Proc ACM Symp on Principles of Programming Languages pages

January

D Algebra

CAR Hoare Communicating Sequential Processes PrenticeHall

R Milner Communication and Concurrency PrenticeHall International Englewood

Clis

D Synchronous MessagePassing Systems

D Computing in a Ring

G LeLann Distributed systems to wards a formal approach In IFIP Congress pages

Toronto

E Chang and R Rob erts An impro ved algorithm for decentralized extremanding

in circular congurations of pro cesses Communications of the ACM May

D Hirsc hb erg and J Sinclair Decentralized extremanding in circular conguarations

of pro cesses Communications of the ACM November

GL P eterson An O n log n unidirectional distributed algorithm for the circular

extrema problem ACM Transactions on Programming Languages and Systems

Octob er

G F rederickson and N Lynch Electing a leader in a synchronous ring Journal of the

ACM January Also MITLCSTM July

H A ttiya M Snir and M Warmuth Computing in an anonymous ring Journal of

the ACM Octob er

D Network Proto cols

Baruc hAwerbuch Course notes

B Aw erbuch Reducing complexities of distributed maximum ow and breadthrst

search algorithms by means of network synchronization Networks

Mic hael Luby A Simple Parallel Algorithm for the Maximal Indep endent Set Problem

SIAM J Comput Novemb er

D Consensus

Basic Results

J Gra y Notes on data base op erating systems Technical Rep ort IBM Rep ort

RJ IBM February Also in Op erating Systems An Advanced

Course SpringerVerlag Lecture Notes in Computer Science

G V arghese and N Lynch A Tradeo Between Safety and Liveness for Randomized

Co ordinated Attack Proto cols th Symposium on Principles of Distributed Systems

Vancouver BC Canada August

L Lamp ort R Shostak and M P ease The Byzantine generals problem ACM

Transactions on Programming Languages and Systems July

M P ease R Shostak and L Lamp ort Reaching agreement in the presence of faults

Journal of the ACM April

Amotz BarNo y Danny Dolev Cynthia Dwork and Raymond Strong Shifting gears

Changing algorithms on the y to exp edite Byzantine agreement Information and

Computation

D Dolev and H Strong Authen ticated algorithms for Byzantine agreement SIAM J

Computing November

T Srik anth and S Toueg Simulating authenticated broadcasts to derive simple fault

tolerant algorithms Distributed Computing

R T urpin and B Coan Extending binary Byzantine agreementtomultivalued Byzan

tine agreement Information Processing Letters Also Technical

Rep ort MITLCSTR Lab oratory for Computer Science Massachusetts Institute

Technology Cambridge MA April Also revised in B Coan Achieving consensus

in faulttolerant distributedcomputing systems PhD thesis Department of Electrical

Engineering and Computer Science Massachusetts Institute of Technology

Numb er of Pro cesses

D Dolev The Byzan tine generals strikeagain Journal of Algorithms

M Fisc her N Lynch and M Merritt Easy imp ossibility pro ofs for distributed con

sensus problems Distributed Computing

L Lamp ort The w eak Byzantine generals problem Journal of the ACM

Time Bounds

M Fisc her and N Lynch A lower b ound for the time to assure interactive consistency

Information Processing Letters June

C Dw ork and Y Moses Knowledge and common knowledge in a Byzantine environ

ment Crash failures Information and Computation Octob er

Y Moses and M T uttle Programming simultaneous actions using common knowledge

Algorithmica

Communication

BA Coan Achieving Consensus in FaultTolerant Distributed Computer Systems

Protocols Lower Bounds and Simulations PhD thesis Massachusetts Institute of

Technology June

Y oram Moses and O Waarts Co ordinated traversal tround Byzantine agreement

in p olynomial time In Proceedings of th Symposium on Foundations of Computer

Science pages Octob er To app ear in the sp ecial issue of the Jour

nal of Algorithms dedicated to selected pap ers from the th IEEE Symp osium on

Foundations of Computer Science Extended version available as Waarts MSc thesis

Weizmann Institute August

Piotr Berman and Juan Gara y Cloture voting nresilient distributed consensus in

t rounds ToappearinMathematical Systems TheoryAn International Journal

on Mathematical Computing Theory sp ecial issue dedicated to distributed agreement

Preliminary version app eared in Proc th Symp on Foundations of Computer Sci

ence pp

Approximate Agreement

D Dolev N Lync h S Pinter E Stark and W Weihl Reaching approximate agree

ment in the presence of faults Journal of the ACM

Firing Squad

JE Burns and NA Lync h The Byzantine ring squad problem Advances in Com

puting Research

B Coan D Dolev C Dw ork and L Sto ckmeyer The distributed ring squad prob

lem In Proceedings of the th ACM Symposium on Theory of Computing pages

May

Commit

C Dw ork and D Skeen The inherentcostofnonblo cking commitment In Proceedings

of the nd Annual ACM Symposium on Principles of Distributed Computing pages

August

P Bernstein V Hadzilacos and N Go o dman Concurrency Control and Recovery in

Database Systems AddisonWesley

B A Coan and J L W elch Transaction commit in a realistic timing mo del Distributed

Computing

The Knowledge Approach

JY Halp ern and Y Moses Knowledge and common knowledge in a distributed

environment In Proceedings of rdACM Symposium on Principles of Distributed

Computing pages Revised as IBM Research Rep ort IBMRJ

C Dw ork and Y Moses Knowledge and common knowledge in a Byzantine environ

ment Crash failures Information and Computation Octob er

D Asynchronous Shared Memory Systems

D Mutual Exclusion

M Ra ynal Algorithms for Mutual Exclusion MIT Press

EW Dijkstra Solution of a problem in concurren t programming control Communi

cations of the ACM September

DE Kn uth Additional comments on a problem in concurrent programming control

Communications of the ACM

JG DeBruijn Additional commen ts on a problem in concurrent programming control

Communications of the ACM

M Eisen b erg and M McGuire Further comments on Dijkstras concurrentprogram

ming control Communications of the ACM

James E Burns and Nancy A Lync h Bounds on shared memory for mutual exclusion

To app ear in Info Comput

L Lamp ort A new solution of Dijkstras concurren t programming problem Commu

nications of the ACM

L Lamp ort The mutual exclusion problem Journal of the ACM

G P eterson and M Fischer Economical solutions for the critical section problem in a

distributed system In Proceedings of th ACM Symposium on Theory of Computing

pages May

J Burns M Fisc her PJackson N Lynch and G Peterson Data requirements for

implementation of npro cess mutual exclusion using a single shared variable Journal

of the ACM

M Merritt and G T aub enfeld Sp eeding Lamp orts fast mutual exclusion algorithm

ATT Technical Memorandum May Also submitted for publication

D Resource Allo cation

M Fisc her N Lynch J Burns and A Boro din Resource allo cation with immunity

to limited pro cess failure In Proceedings of th IEEE Symosium on Foundations of

Computer Science pages Octob er

M Fisc her N Lynch J Burns and A Boro din Distributed FIFO allo cation of

identical resources using small shared space ACM Trans Prog Lang and Syst

January

D Dolev E Gafni and N Sha vit Toward a nonatomic era lexclusion as a test

case In Proceedings of th ACM Symposium on Theory of Computing pages

May

EW Dijkstra Hierarc hical ordering of sequential pro cesses Acta Informatica pages

D Atomic Registers

GL P eterson Concurrent reading while writing ACM Transactions on Programming

Languages and Systems

L Lamp ort On in terpro cess communication Distributed Computing

Digital Systems Research TM

Am buj K Singh James H Anderson and Mohamed G Gouda The elusive atomic

Bard Blo om Constructing t wowriter registers IEEE Trans Computers

Decemb er

P Vitanyi and B Awerbuch Atomic shared register access by asynchronous hard

ware In Proceedings th Annual IEEE Symposium on Theory of Computing pages

Toronto Ontario Canada May Also MITLCSTM Lab oratory

for Computer Science Massachusetts Institute of Technology Cambridge MA

Corrigenda in Proceedings of th Annual IEEE Symposium on Theory of Computing

page

R Sc haer On the correctness of atomic multiwriter registers Bachelors Thesis

June Massachusetts Institute Technology Also Technical Memo MITLCSTM

Lab for Computer Science MIT June and Revised Technical Memo

MITLCSTM January

Soma Chaudh uri and Jennifer Welch Bounds on the costs of register implementations

Manuscript Also Technical Rep ort TR University of North Carolina at

Chap el Hill June

D Snapshots

Y Afek H A ttiya D Dolev E Gafni M Merritt and N Shavit Atomic snapshots

of shared memoryInProceedings of the Annual ACM Symposium on Principles

of Distributed Computing pages Queb ec Canada August Also Technical

Memo MITLCSTM Lab oratory for Computer Science Massachusetts Institute

of TechnologyCambridge MA MayTo app ear in Journal of the ACM

Cyn thia Dwork Maurice Herlihy Serge Plotkin and Orli Waarts Timelapse snap

shots In In Israel Symposium on Theory of Computing and Systems ISTCS

Also Stanford Technical Rep ort STANCS April and Digital Equipment

Corp oration Technical Rep ort CRL

D Timestamp Systems

Amos Israeli and Ming Li Bounded timestamps In Proceedings of the th IEEE

Symposium on Foundations of Computer Science pages Octob er

R Ga wlick N Lynch and N Shavit Concurrent timestamping made simple In

In Israel Symposium on Theory of Computing and Systems ISTCS pages

SpringerVerlag May

Cyn thia Dwork and Orli Waarts Simple and ecient b ounded concurrent timestamp

ing or b ounded concurrent timestamp systems are comprehensible In In Proceedings

of the th Annual ACM Symposium on Theory of Computing STOC pages

Victoria BC Canada May Preliminary version app ears in IBM Research

Rep ort RJ Octob er

D Consensus

M Loui and H AbuAmara Memory requiremen ts for agreement among unreliable

asynchronous pro cesses Advances in Computing Research

Maurice Herlih yWaitfree synchronization ACM Transactions on Programming Lan

guages and Systems January

H A ttiya N Lynch and N Shavit Are waitfree algorithms fast Submitted for

publication Earlier versions in MITLCSTM and Proceedings of the st Annual

IEEE Symposium on Foundations of Computer Science Octob er

Karl Abrahamson On achieving consensus using a shared memoryInProceedings

of the Annual ACM Symposium on Principles of Distributed Computing pages

Toronto Canada August

D Shared Memory for Multipro cessors

Leslie Lamp ort Ho wtomakea multipro cessor computer that correctly executes mul

tipro cess programs IEEE Transactions on Computers c September

Alan Ja ySmith Cache memories Computing Surveys September

Maurice P Herlihy and Jeannette M Wing Linearizability A correctness condition

for concurrent ob jects ACM Transactions on Programming Languages and Systems

July

Maurice Herlih yWaitfree synchronization ACM Transactions on Programming Lan

guages and Systems January

Y ehuda Afek Georey Brown and Michael Merritt Lazy caching TOPLAS Octob er

To app ear Also a preliminary version app eared in SPAA

Phillip B Gibb ons Mic hael Merritt and Kourosh Gharachorlo o Proving sequential

consistency of highp erformance shared memories ATT Bell Lab oratories

TM and TM May

Phillip B Gibb ons and Mic hael Merritt Sp ecifying nonblo cking shared memories

ATT Bell Lab oratories BL T M and BL TM May

Hagit A ttiya and RoyFriedman A correctness condition for highp erformance multi

pro cessors Unpublished Manuscript November

Ric hard J Lipton and Jonathan S Sandb erg PRAM A scalable shared memory

Department of Computer Science Princeton University CSTR September

Mustaque Ahamad James E Burns Phillip W Hutto and Gil Neiger Causal memory

College of Computing Georgia Institute of Technology GITCC September

Hagit A ttiya and Jennifer Welch Sequential consistency versus linearizability In

Proceedings of the rdACM Symp on Paral lel Algorithms and Architectures SPAA

Also Technion Israel Institute Technology Dept of CS TR

D Asynchronous MessagePassage Systems

D Computing in Static Networks

Rings

G LeLann Distributed systems to wards a formal approach In IFIP Congress pages

Toronto

E Chang and R Rob erts An impro ved algorithm for decentralized extremanding

in circular congurations of pro cesses Communications of the ACM May

D Hirsc hb erg and J Sinclair Decentralized extremanding in circular conguarations

of pro cesses Communications of the ACM November

GL P eterson An O n log n unidirectional distributed algorithm for the circular

extrema problem ACM Transactions on Programming Languages and Systems

Octob er

D Dolev M Kla we and M Ro deh An O n log n unidirectional distributed algorithm

for extrema nding in a circle Research Rep ort RJ IBM July J Algorithms

J Burns A formal mo del for message passing systems Technical Rep ort TR

Computer Science Dept Indiana UniversityMay

K Abrahamson A Adler L Higham and D Kirkpatric k Tightlower b ounds for

probabilistic solitude verication on anonymous rings Submitted for publication

Complete Graphs

E Korac h S Moran and S Zaks Tightlower and upp er b ounds for some distributed

algorithms for a complete network of pro cessors In In Proceedings of rdACM Sym

posium on Principles of Distributed Computing pages

Y Afek and E Gafni Time and message b ounds for election in synchronous and

asynchronous complete networks In Proceedings of th ACM Symposium on Principles

of Distributed Computing pages Minaki Ontario August

General Networks

D Angluin Lo cal and global prop erties in net works of pro cessors In Proceedings of

th ACM Symposium on Theory of Computing pages

R Gallager P Humblet and P Spira A distributed algorithm for minimumweight

spanning trees ACM Transactions on Programming Languages and Systems

January

P Humblet A distributed algorithm for minimum weight directed spanning trees

IEEE Transactions on Computers COM MITLIDSP

Baruc hAwerbuchandDavid Peleg Sparse partitions In Proceedings of the st IEEE

Symposium on Foundations of Computer Science pages St Louis Missouri

Octob er

D Network Synchronization

L Lamp ort Time clo c ks and the ordering of events in a distributed system Com

munications of the ACM July

J M H elary and M Raynal Synchronization and Control of Distributed Systems and

Programs John Wiley Sons

B Aw erbuch Complexity of network synchronization J ACM Octo

b er Also Technical Memo MITLCSTM Lab oratory for Computer Science

Massachusetts Institute TechnologyCambridge MA January

B Aw erbuch and M Sipser Dynamic networks are as fast as static networks In

Proceedings of the th IEEE Symposium on Foundations of Computer Science IEEE

Octob er

E Arjomandi M Fisc her and N Lynch Eciency of synchronous versus asyn

chronous distributed systems Journal of the ACM July

Leslie Lamp ort Using time instead of timeout for faulttoleran t distributed systems

toplas apr

Leslie Lamp ort The parttime parliamen t Technical Memo Digital Systems Re

searchCenter Septemb er

JL W elch Simulating synchronous pro cessors Information and Computation

August

D Simulatin g Asynchronous Shared Memory

Hagit A ttiya Amotz BarNoy and Danny Dolev Sharing memory robustly in message

passing systems In Proceedings of the th Annual ACM Symposium on Principles of

Distributed Computing pages Revised version to app ear in J ACM

D Resource Allo cation

M Ra ynal Algorithms for Mutual Exclusion MIT Press

K Chandy and J Misra The drinking philosophers problem ACM Transactions on

Programming Languages and Systems Octob er

N Lync h Upp er b ounds for static resource allo cation in a distributed system Journal

Of Computer And Systems Sciences Octob er

M Rabin and D Lehmann On the adv antages of free choice a symmetric and fully

distributed solution to the dining philosophers problem In Proceedings of th ACM

Symposium on Principles of Programming Languages pages

Baruc hAwerbuch and Michael Saks A dining philosophers algorithm with p olynomial

resp onse time In Proceedings of the st IEEE Symp on Foundations of Computer

Science pages St Louis Missouri Octob er

Jennifer W elch and Nancy Lynch A mo dular drinking philosophers algorithm July

Earlier version app eared as MITLCSTM

D Delecting Stable Prop erties

Termination Detection

E Dijkstra and C Sc holten Termination detection for diusing computations Infor

mation Processing Letters August

N F rancez and N Shavit A new approach to detection of lo cally indicative stabil

ityInProceedings of the th International Col loquium on Automata Languages and

Programming ICALP pages Rennes France July SpringerVerlag

Global Snapshots

K Chandy and L Lamp ort Distributed snapshots determining global states of

distributed systems ACM Transactions on Computer Systems February

M Fisc her N Grieth and N Lynch Global states of a distributed system IEEE

Transactions on Software Engineering SE May Also in Proceed

ings of IEEE Symposium on Reliability in Distributed Software and Database Systems

Pittsburgh PA July

D Deadlo ck Detection

D Menasce and R Mun tz Lo cking and deadlo ck detection in distributed databases

IEEE Transactions on Software Engineering SE May

V Gligor and S Shattuc k On deadlo ck detection in distributed systems IEEE

Transactions on Software Engineering SE Septemb er

R Ob ermarc k Distributed deadlo ck detection algorithm ACM Transactions on

Database Systems June

G Ho and C Ramamo orth y Proto cols for deadlo ck detection in distributed database

systems IEEE Transactions on Software Engineering SE November

K Chandy J Misra and L Haas Distributed deadlo ck detection ACM Transactions

on Programming Languages and Systems May

G Brac ha and S Toueg A distributed algorithm for generalized deadlo ck detection

Distributed Computing

D Mitc hell and M Merritt A distributed algorithm for deadlo ck detection and resolu

tion In Proceedings of rdACM Symposium on Principles of Distributed Computing

pages Vancouver BC Canada August

D Consensus

M Fisc her N Lynch and M Paterson Imp ossibility of distributed consensus with

one family faulty pro cess Journal of the ACM April

S Moran and Y W olfstahl Extended imp ossibility results for asynchronous complete

networks Information Processing Letters

M Bridgland and R W atro Faulttolerant decision making in totally asynchronous

distributed systems In Proceedings of th ACM Symposium on Principles of Dis

tributed Computing pages August Revision in progress

Ofer Biran Shlomo Moran and Shm uel Zaks A combinatorial characterization of the

distributed solvable tasks Journal of Algorithms Septemb er

Earlier version in Proceedings of th ACM Symposium on Principles of Distributed

Computing pages August

Gadi T aub enfeld Shmuel Katz and Shlomo Moran Imp ossibility results in the pres

ence of multiple faulty pro cesses In CE Veni Madhavan editor Proc of the th FCT

TCS Conferencevolume of Lecture Notes in Computer Science pages

Bangalore India Springer Verlag To app ear in Information and Computation

Gadi T aub enfeld On the nonexistence of resilient consensus proto cols Information

Processing Letters March

H A ttiya A BarNoyDDolevDPeleg and R Reischuk Renaming in an asyn

chronous environment J ACM July

D Dolev C Dw ork and L Sto ckmeyer On the minimal synchronism needed for

distributed consensus Journal of the ACM

T ushar D Chandra and Sam Toueg Unreliable failure detectors for asynchronous

systems In Proceedings th ACM Symposium on Principles of Distributed Computing

pages Montreal Canada August Also Cornell University Department

of Computer Science Ithaca New York TR

T ushar D Chandra Vassos Hadzilacos and Sam Toueg The weakest failure detector

for solving consensus In Proceedings th ACM Symposium on Principles of Dis

tributed Computing pages Vancouver Canada August Also Cornell

University Department of Computer Science Ithaca New York TR

D Datalink

A Aho J Ullman A Wyner and M Y annakakis Bounds on the size and transmis

sion rate of communication proto cols Computers and Mathematics with Applications

J Y Halp ern and L D Zuc k A little knowledge go es a long way Simple knowledge

based derivations and correctness pro ofs for a family of proto cols J ACM

July Earlier version exists in Proceedings of the th Annual ACM Symposium

on Principles of Distributed Computing August

N Lync h Y Mansour and A Fekete The data link layer Two imp ossibility results In

Proceedings of th ACM Symposium on Principles of Distributed Computation pages

Toronto Canada August Also Technical Memo MITLCSTM

May

Y Afek H A ttiya A Fekete M Fischer N Lynch Y Mansour D Wang

and L Zuck Reliable communication over unreliable channels Technical Memo

MITLCSTM Lab oratory for Computer Science Massachusetts Institute Tech

nologyCambridge MA Octob er

A F ekete N Lynch Y Mansour and J Spinelli The data link layer The imp os

sibility of implementation in face of crashes Technical Memo MITLCSTMb

Massachusetts Institute of Technology Lab oratory for Computer Science August

Submitted for publication

A F ekete and N Lynch The need for headers an imp ossibility result for commu

nication over unreliable channels In G Go os and J Hartmanis editors CONCUR

Theories of Concurrency Unication and Extensionvolume of Lecture Notes

in Computer Science pages SpringerVerlag August Also Technical

Memo MITLCSTM May

Y Afek H A ttiya A Fekete M Fischer N Lynch Y Mansour D Wang and

L Zuck Reliable communication over an unreliable channel Submitted for

publication

DaW ei Wang and Lenore Zuck Tight b ounds for the sequence transmission prob

lem In Proceedings of the Annual ACM Symposium on Principles of Distributed

Computing pages August

Ew an Temp ero and Richard Ladner Tight b ounds for weakly b ounded proto cols In

Proceedings of the Annual ACM Symposium on Principles of Distributed Comput

ing pages Queb ec Canada August

Ew an Temp ero and Richard Ladner Recoverable sequence transmission proto cols

June Manuscript

D Sp ecialPurp ose Network Building Blo cks

J M H elary and M Raynal Synchronization and Control of Distributed Systems and

Programs John Wiley Sons

B Aw erbuch O Goldreich D Peleg and R Vainish A tradeo b etween information

and communication in broadcast proto cols J ACM April

Y ehuda Afek Eli Gafni and Adi Rosen The slide mechanism with applications in

dynamic networks In Proceedings of the Eleventh Annual Symp on Principles of

Distributed Computing pages Vancouver British Columbia Canada August

D SelfStabilizati on

Edsger W Dijkstra Selfstabilizing systems in spite of distributed con trol Commu

nication of the ACM November

Edsger W Dijksra A Belated Pro of of SelfStabilization Distributed Computing

January

Geoery M Bro wn and Mohamed G Gouda and ChuanLin Wu Token Systems that

SelfStabilize IEEE Trans Computers June

James E Burns and Jan P achl Uniform SelfStabilizing Ring ACM Trans Prog

Lang and Syst April

Shm uel Katz and Kenneth J Perry Selfstabilizing extensions for messagepassing

systems March To app ear in Distrib Comput Also earlier version in Proceedings

of the Annual ACM Symposium on Principles of Distributed Computing August

Y ehuda Afek and Georey Brown Selfstabilization of the alternating bit proto col In

Proceedings of the th IEEE Symposium on Reliable Distributed Systems pages

Shlomi Dolev Amos Israeli and Shlomo Moran Resource b ounds for self stabilizing

message driven proto cols In Proceedings of the th Annual ACM Symposium on

Principles of Distributed Computing August

Baruc hAwerbuch and Boaz PattShamir and George Varghese SelfStabilization by

Lo cal Checking and Correction Proceedings of the nd Annual IEEE Symposium on

Foundations of Computer Science Octob er

V arghese G Dealing with Failure in Distributed Systems PhD thesis MIT Dept of

Electrical Engineering and Computer Science In progress

D Knowledge

K Chandy and J Misra How pro cesses learn Distributed Computing

D TimingBased Systems

D Resource Allo cation

H A ttiya and N Lynch Time b ounds for realtime pro cess control in the presence of

timing uncertainty Inf Comput To app ear

N Lync h and N Shavit Timingbased mutual exclusion In Proceedings of the th

IEEE RealTime Systems Symposium Pho enix Arizona December To app ear

Ra jeev Alur and Gadi T aub enfeld Results ab out fast mutual exclusion In Proceedings

of the th IEEE RealTime Systems Symposium Pho enix Arizona Decemb er

To app ear

D Shared Memory for Multipro cessors

M Ma vronicolas Timingbased distributed computation Algorithms and imp ossibil

ity results Harvard University Cambridge MA TR

D Consensus

C Dw ork N Lynch and L Sto ckmeyer Consensus in the presence of partial syn

chrony Journal of the ACM

Hagit A ttiya Cynthia Dwork Nancy Lynch and Larry Sto ckmeyer Bounds on the

time to reach agreement in the presence of timing uncertainty JACM To

app ear Also a preliminary version app eared in rd STOC

S P onzio The realtime cost of timing uncertainty Consensus and failure detection

Masters thesis MIT Dept of Electrical Engineering and Computer Science June

Also MITLCSTR

Stephen P onzio Consensus in the presence of timing uncertainty Omission and byzan

tine failures In th ACM Symposium on Principles of Distributed Computing pages

Montreal Canada August

D Communication

DaW ei Wang and Lenore D Zuck Realtime STP In Proceedings of Tenth ACM

Symp on Principles of Distributed ComputingAugust This is also Technical

Rep ort YALEDCSTR

S P onzio Bounds on the time to detect failures using b oundedcapacity message links

In Proceedings of the th Realtime Systems Symposium Pho enix Arizona December

To app ear

S P onzio The realtime cost of timing uncertainty Consensus and failure detection

Masters thesis MIT Dept of Electrical Engineering and Computer Science June

Also MITLCSTR

Amir Herzb erg and Sha y Kutten Fast isolation of arbitrary forwardingfaults Pro

ceedings of the th Annual ACM Symposium on Principles of Distributed Computing

August

D Clo ck Synchronization

Leslie Lamp ort and P M MelliarSmith Synchronizing clo cks in the presence of faults

J ACM January

J Lundelius and N Lync h A new faulttolerant algorithm for clo cksynchronization

Information and Computation

J Lundelius and N Lync h An upp er and lower b ound for clo cksynchronization

Information and Control AugustSeptember

Dann y Dolev Joseph Y Halp ern Barbara Simons and Ray Strong Dynamic fault

tolerant clo cksynchronization January IBM Research Rep ort RJ

Also earlier version app eared in Proceedings of the rdACM Symposium on Principles

of Distributed Computing August

D Dolev J Halp ern and R Strong On the p ossibilit y and imp ossibilityofachieving

clo ck synchronization In Proceedings of th Symposium on Theory of Computing

pages May Later version in Journal of Computer and Systems Sciences

April

S Mahaney and F Sc hneider Inexact agreement accuracy precision and grace

ful degradation In Proceedings of th ACM Symposium on Principles of Distributed

Computing pages August

M Fisc her N Lynch and M Merritt Easy imp ossibility pro ofs for distributed con

sensus problems Distributed Computing

D Applications of Synchronized Clo cks

Barbara Lisk ov Practical uses of synchronized clo cks in distributed systems In Pro

ceedings of the th Annual ACM Symposium on Principles of Distributed Computing

pages Montreal Canada August

Gil Neiger and Sam T oueg Simulating synchronized clo cks and common knowledge in

distributed systems July To app ear in J ACM Earlier version exists in Pro

ceedings th ACM Symposium on Principles of Distributed Computing August

Vancouver Canada