5.1: Amortized Analysis POP(S) Frank Stajano Thomas Sauerwald 0: MULTIPOP(S,K) 1: While Not S.Empty() and K > 0 2: POP(S) U 3: K = K - 1 X T Lent 2015

What is the largest possible cost of a sequence of n stack operations?

Simple Worst-Case Bound: largest cost of an operation: n cost is at most n · n = n2 (correct, but not tight!)

0: MULTIPOP(S,k) 1: while not S.empty() and k > 0 2: POP(S) 3: k = k - 1

Motivating Example: Stack

Stack Operations X PUSH(S,x) T T pushes object x onto stack S total cost of 1 POP(S) PUSH(S,X) pops the top of (a non-empty) stack S total cost of 1 X MULTIPOP(S,k) T T pops the k top objects (S non-empty) ⇒ total cost of min{|S|, k} 5.1: Amortized Analysis POP(S) Frank Stajano Thomas Sauerwald 0: MULTIPOP(S,k) 1: while not S.empty() and k > 0 2: POP(S) U 3: k = k - 1 X T Lent 2015

MULTIPOP(S,4)

5.1: Amortized Analysis T.S. 2

Motivating Example: Stack Sequence of Stack Operations

Stack Operations X PUSH(S,x) T T pushes object x onto stack S total cost of 1 POP(S) PUSH(S,X) pops the top of (a non-empty) stack S X D total cost of 1 X B B B B MULTIPOP(S,k) T T T T T T T pops the k top objects (S non-empty) ⇒ total cost of min{|S|, k} POP(S) PUSH(T) PUSH(B) PUSH(X) POP PUSH(D) MULTIPOP(3) What is the largest possible cost of a sequence of n stack operations? U X Simple Worst-Case Bound: T largest cost of an operation: n 2 cost is at most n · n = n (correct, but not tight!) MULTIPOP(S,4)

5.1: Amortized Analysis T.S. 2 5.1: Amortized Analysis T.S. 3 T (n) ≤ TPOP (n) + TPUSH (n) ≤ 2 · TPUSH (n) ≤ 2 · n.

T (n) AggregateMULTIPOP( Analysis:k) contributes The amortized min{k, |S|} costto T perPOP operation(n) is n ≤ 2

Every item which is POPPED from Every item which is POPPED from T (n) Aggregate Analysis:the The stack amortized had to be cost PUSHED per operation earlier! is n ≤ 2 MULTIPOP(k) contributesthe stack min had{k to, |S be|} PUSHEDto TPOP (n) earlier!

≤ 2 · TPUSH (n) ≤ 2 · n.

A new Analysis Tool: Amortized Analysis Aggregate Analysis of the STACK

Data structure operations (Heap, Stack, Queue etc.) Simple Worst-Case Bound: largest cost of an operation: n Amortized Analysis · = 2 analyse a sequence of operations cost is at most n n n (correct, but not tight!) show that average cost of an operation is small concrete techniques Aggregate Analysis This is not average case analysis! D Potential Method B B T T T

Aggregate Analysis PUSH(B) MULTIPOP(3) Determine an upper bound T (n) for the total cost of any sequence of n operations Every item which is POPPED from T (n) the stack had to be PUSHED earlier! amortized cost of each operation is the average n

Even though operations may be of different types/costs

5.1: Amortized Analysis T.S. 4 5.1: Amortized Analysis T.S. 5

Aggregate Analysis of the STACK Aggregate Analysis of the STACK

Simple Worst-Case Bound: Simple Worst-Case Bound: largest cost of an operation: n largest cost of an operation: n cost is at most n · n = n2 (correct, but not tight!) cost is at most n · n = n2 (correct, but not tight!)

D D B B B B T T T T T T

PUSH(B) MULTIPOP(3) PUSH(B) MULTIPOP(3)

T (n) MULTIPOP(k) contributes min{k, |S|} to TPOP (n) Aggregate Analysis: The amortized cost per operation is n ≤ 2

T (n) ≤ TPOP (n) + TPUSH (n) T (n) ≤ TPOP (n) + TPUSH (n) ≤ 2 · TPUSH (n) ≤ 2 · n.

5.1: Amortized Analysis T.S. 5 5.1: Amortized Analysis T.S. 5 eci = ci + (Φi − Φi−1)

Φ1 − Φ0 + Φ2 − Φ1 + ··· + Φn − Φn−1 PUSH():PUSH():ci =Φ1i − Φi−1 = 1 n n POP:XPOP:c =Φ1X− Φ = −1 ecii = i (cii−+1 Φi − Φi−1) i=1 i=1 n X = ci + Φn i=1

If Φn ≥ 0 for all n, sum of amortized costs is an upper bound for the sum of actual costs!

Second Technique: Potential Method Stack as a coin-operated machine (p. 83)

Potential Method allow different amortized costs store (ﬁctitious) credit in the data structure to cover up for expensive operations

Potential of a data structure can be also thought of as amount of potential energy stored distance from an ideal state

5.1: Amortized Analysis T.S. 6 5.1: Amortized Analysis T.S. 7

Stack and Coins Potential Method in Detail

ci < eci , ci = eci or ci > eci are all possible!

X D ci is the actual cost of operation i B B B B eci is the amortized cost of operation i T T T T T Φi is the potential stored after operation i (Φ0 = 0)

PUSH(T) PUSH(B) PUSH(X) POP PUSH(D) MULTIPOP(3) Function that maps states of the data structure to some value Every operation costs at most two coins! credit 3

i 0 1 2 3 4 5 6

5.1: Amortized Analysis T.S. 8 5.1: Amortized Analysis T.S. 9 ci < eci , ci = eci or ci > eci are all possible! ci < eci , ci = eci or ci > eci are all possible!

Function that maps states of the Function that maps states of the data structure to some value data structure to some value

Φ1 − Φ0 + Φ2 − Φ1 + ··· + Φn − Φn−1 Φ1 − Φ0 + Φ2 − Φ1 + ··· + Φn − Φn−1 PUSH(): Φi − Φi−1 = 1 PUSH(): ci = 1 n n n n XPOP: ΦX− Φ = −1 POP:X c = 1X eci = i (cii−+1 Φi − Φi−1) ecii = (ci + Φi − Φi−1) i=1 i=1 i=1 i=1 n n X X = ci + Φn = ci + Φn i=1 i=1

If Φn ≥ 0 for all n, sum of amortized costs is If Φn ≥ 0 for all n, sum of amortized costs is an upper bound for the sum of actual costs! an upper bound for the sum of actual costs!

ci < eci , ci = eci or ci > eci are all possible!

Function that maps states of the data structure to some value

Amortizedn/2 PUSH, Costn/2≤ POP2 ⇒⇒TT(n(n) )≤≤2n

PUSH():PUSH():ci =Φ1i − Φi−1 = 1

POP:POP:ci =Φ1i − Φi−1 = −1

Potential Method in Detail Potential Method in Detail

ci is the actual cost of operation i ci is the actual cost of operation i eci is the amortized cost of operation i eci is the amortized cost of operation i Φi is the potential stored after operation i (Φ0 = 0) Φi is the potential stored after operation i (Φ0 = 0)

eci = ci + (Φi − Φi−1) eci = ci + (Φi − Φi−1)

PUSH(): ci = 1 PUSH(): Φi − Φi−1 = 1

POP: ci = 1 POP: Φi − Φi−1 = −1

5.1: Amortized Analysis T.S. 9 5.1: Amortized Analysis T.S. 9

Potential Method in Detail Stack: Analysis via Potential Method

Φi = # objects in the stack after ith operation

ci is the actual cost of operation i PUSH Φ eci is the amortized cost of operation i i actual cost: ci = 1 Φi is the potential stored after operation i (Φ0 = 0) potential change: Φi − Φi−1 = 1

amortized cost: cbi = ci + (Φi − Φi−1) = 1 + 1 = 2 i − 1 i PUSH eci = ci + (Φi − Φi−1) POP Φi

Φ1 − Φ0 + Φ2 − Φ1 + ··· + Φn − Φn−1 ci = 1 Φ − Φ = −1 n n i i−1 X X = + (Φ − Φ ) = − = eci = (ci + Φi − Φi−1) cbi ci i i−1 1 1 0 i − 1 i i=1 i=1 Stack is non-empty! POP n X Φi = ci + Φn MULTIPOP(k) i=1 ci = min{k, |S|}

Φi − Φi−1 = − min{k, |S|} If Φn ≥ 0 for all n, sum of amortized costs is i − 1 i cbi = ci + (Φi − Φi−1) = min{k, |S|} − min{k, |S|} = 0 an upper bound for the sum of actual costs! MULTIPOP(3)

5.1: Amortized Analysis T.S. 9 5.1: Amortized Analysis T.S. 10 n/2 PUSH, n/2 POP ⇒ T (n) ≤ n Amortized Cost ≤ 2 ⇒ T (n) ≤ 2n

0: INC(A) 1: i =What 0 is the total cost of a se- What is the total cost of a se- 2: whilequence i < k of andn INC A[i]==1 operations? quence of n INC operations? 3: A[i] = 0 4: i = i + 1 Simple Worst-Case Bound: Simple Worst-Case Bound: 5: A[i] = 1 largest cost of an operation: k largest cost of an operation: k cost is at most n · k (correct, but not tight!) cost is at most n · k (correct, but not tight!)

Stack: Analysis via Potential Method Stack: Analysis via Potential Method

Φi = # objects in the stack after ith operation Φi = # objects in the stack after ith operation

PUSH Φi PUSH Φi

actual cost: ci = 1 actual cost: ci = 1

potential change: Φi − Φi−1 = 1 potential change: Φi − Φi−1 = 1

amortized cost: cbi = ci + (Φi − Φi−1) = 1 + 1 = 2 i − 1 i amortized cost: cbi = ci + (Φi − Φi−1) = 1 + 1 = 2 i − 1 i PUSH PUSH

POP Φi POP Φi ci = 1 Amortized Cost ≤ 2 ⇒ T (n) ≤ 2n ci = 1 n/2 PUSH, n/2 POP ⇒ T (n) ≤ n

Φi − Φi−1 = −1 Φi − Φi−1 = −1

cbi = ci + (Φi − Φi−1) = 1 − 1 = 0 i − 1 i cbi = ci + (Φi − Φi−1) = 1 − 1 = 0 i − 1 i Stack is non-empty! POP Stack is non-empty! POP Φi Φi MULTIPOP(k) MULTIPOP(k)

ci = min{k, |S|} ci = min{k, |S|}

Φi − Φi−1 = − min{k, |S|} Φi − Φi−1 = − min{k, |S|} i − 1 i i − 1 i cbi = ci + (Φi − Φi−1) = min{k, |S|} − min{k, |S|} = 0 cbi = ci + (Φi − Φi−1) = min{k, |S|} − min{k, |S|} = 0 MULTIPOP(3) MULTIPOP(3)

5.1: Amortized Analysis T.S. 10 5.1: Amortized Analysis T.S. 10

Second Example: Binary Counter Second Example: Binary Counter

Binary Counter Binary Counter Array A[k − 1], A[k − 2],..., A[0] of k bits Array A[k − 1], A[k − 2],..., A[0] of k bits Use array for counting from 0 to 2k − 1 Use array for counting from 0 to 2k − 1 only operation: INC A[3]A[2]A[1]A[0] only operation: INC A[3]A[2]A[1]A[0] increases the counter by one 1 0 1 1 11 increases the counter by one 1 0 1 1 11 total cost: ≤ k total cost: number of bit ﬂips

INC 0: INC(A) INC 1: i = 0 2: while i < k and A[i]==1 A[3]A[2]A[1]A[0] 3: A[i] = 0 A[3]A[2]A[1]A[0] 1 1 0 0 12 4: i = i + 1 1 1 0 0 12 5: A[i] = 1

5.1: Amortized Analysis T.S. 11 5.1: Amortized Analysis T.S. 11 0: INC(A) 1: i = 0 2: while i < k and A[i]==1 3: A[i] = 0 4: i = i + 1 5: A[i] = 1

T (n) Aggregate Analysis: The amortized cost per operation is n ≤ 2.

k−1 k−1 X n X n 1 1 1 T (n) ≤ ≤ = n · 1 + + + ··· + ≤ 2 · n. 2i 2i 2 4 2k−1 i=0 i=0

Second Example: Binary Counter Incrementing a Binary Counter (k = 8)

Counter Total A[7] A[6] A[5] A[4] A[3] A[2] A[1] A[0] Value Cost Binary Counter 0 0 0 0 0 0 0 0 0 0 Array A[k − 1], A[k − 2],..., A[0] of k bits 1 0 0 0 0 0 0 0 1 1 Use array for counting from 0 to 2k − 1 2 0 0 0 0 0 0 1 0 3 only operation: INC A[3]A[2]A[1]A[0] 3 0 0 0 0 0 0 1 1 4 increases the counter by one 1 0 1 1 11 4 0 0 0 0 0 1 0 0 7 total cost: number of bit ﬂips 5 0 0 0 0 0 1 0 1 8 6 0 0 0 0 0 1 1 0 10 INC 7 0 0 0 0 0 1 1 1 11 What is the total cost of a se- 8 0 0 0 0 1 0 0 0 15 quence of n INC operations? 9 0 0 0 0 1 0 0 1 16 A[3]A[2]A[1]A[0] 10 0 0 0 0 1 0 1 0 18 11 0 0 0 0 1 0 1 1 19 Simple Worst-Case Bound: 1 1 0 0 12 12 0 0 0 0 1 1 0 0 22 largest cost of an operation: k 13 0 0 0 0 1 1 0 1 23 cost is at most n · k (correct, but not tight!) 14 0 0 0 0 1 1 1 0 25 15 0 0 0 0 1 1 1 1 26 16 0 0 0 1 0 0 0 0 31

5.1: Amortized Analysis T.S. 11 5.1: Amortized Analysis T.S. 12

Incrementing a Binary Counter: Aggregate Analysis Incrementing a Binary Counter: Aggregate Analysis

Counter Total Counter Total A[3] A[2] A[1] A[0] A[3] A[2] A[1] A[0] Value Cost Value Cost 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 1 0 0 0 1 1 2 0 0 1 0 3 2 0 0 1 0 3 3 0 0 1 1 4 3 0 0 1 1 4 4 0 1 0 0 7 4 0 1 0 0 7 5 0 1 0 1 8 5 0 1 0 1 8 6 0 1 1 0 10 6 0 1 1 0 10 7 0 1 1 1 11 7 0 1 1 1 11

Bit A[i] is only flipped every 2i increments Bit A[i] is only flipped every 2i increments n T (nn) In a sequence of n increments from 0, bit A[i] is flipped b i c times In a sequence of n increments from 0, bit A[i] is flipped b i c times 2 Aggregate Analysis: The amortized cost per operation is n2 ≤ 2.

k−1 k−1 X n X n 1 1 1 T (n) ≤ ≤ = n · 1 + + + ··· + ≤ 2 · n. 2i 2i 2 4 2k−1 i=0 i=0

5.1: Amortized Analysis T.S. 13 5.1: Amortized Analysis T.S. 13 Amortized Cost = 2 ⇒ T (n) ≤ 2n

Binary Counter: Analysis via Potential Function Binary Counter: Analysis via Potential Function

Φi = # ones in the binary representation of i Φi = # ones in the binary representation of i

Φ0 = 0 X Φi ≥ 0 X Φ0 = 0 X Φi ≥ 0 X

1 1 0 0 Φi 1 1 0 0 Φi Increment without Carry-Over Increment without Carry-Over Amortized Cost = 2 ⇒ T (n) ≤ 2n actual cost: ci = 1 actual cost: ci = 1 INC INC potential change: Φi − Φi−1 = 1 potential change: Φi − Φi−1 = 1 1 1 0 1 i − 1 i 1 1 0 1 i − 1 i amortized cost: cbi = ci + (Φi − Φi−1) = 1 + 1 = 2 amortized cost: cbi = ci + (Φi − Φi−1) = 1 + 1 = 2

Increment with Carry-Over Increment with Carry-Over 0 1 1 1 Φi 0 1 1 1 Φi ci = x + 1, (x lowest index of a zero) ci = x + 1, (x lowest index of a zero)

Φi − Φi−1 = −x + 1 INC Φi − Φi−1 = −x + 1 INC = + (Φ − Φ ) = + − + = = + (Φ − Φ ) = + − + = cbi ci i i−1 1 x x 1 2 1 0 0 0 i − 1 i cbi ci i i−1 1 x x 1 2 1 0 0 0 i − 1 i

5.1: Amortized Analysis T.S. 14 5.1: Amortized Analysis T.S. 14

Summary

Amortized Analysis Average costs over a sequence of n operations overcharge cheap operations and undercharge expensive operations no probability/average case analysis involved!

E.g. by bounding the number of expensive operations

Aggregate Analysis Determine an absolute upper bound T (n) T (n) T (n) every operation has amortized cost n

Full power of this method will become clear later! T (n) Potential Method use savings from cheap operations to credit compensate for expensive ones

operations may have different amortized cost i

5.1: Amortized Analysis T.S. 15 Priority Queues Overview

Operation Linked list Binary heap Binomial heap Fibon. heap

MAKE-HEAP O(1) O(1) O(1) O(1)

INSERT O(1) O(log n) O(log n) O(1)

MINIMUM O(n) O(1) O(log n) O(1)

5.2 Fibonacci Heaps EXTRACT-MIN O(n) O(log n) O(log n) O(log n)

Frank Stajano Thomas Sauerwald UNION O(n) O(n) O(log n) O(1)

DECREASE-KEY O(1) O(log n) O(log n) O(1)

DELETE O(1) O(log n) O(log n) O(log n)

Lent 2015

5.2: Fibonacci Heaps T.S. 2

Binomial Heap vs. Fibonacci Heap: Costs Actual vs. Amortized Cost

Operation Binomial heap Fibonacci heap Pn Pn i=1 eci i=1 ci actual cost amortized cost 14 ·O(1)

MAKE-HEAP O(1) O(1)

INSERT O(log n) O(1)

MINIMUM O(log n) O(1)

EXTRACT-MIN O(log n) O(log n)

UNION O(log n) O(1)

DECREASE-KEY O(log n) O(1) Potential

DELETE O(log n) O(log n) Potential ≥ 0, but should be 2 ·O(1) also as small as possible Binomial Heap: n Inserts Fibonacci Heap: n Inserts O(1) n c1 = c2 = ··· = cn = O(log n) ce1 = ce2 = ··· = cen = O(1) 0 Pn Pn Pn 1 2 14 ⇒ i=1 ci = O(n log n) ⇒ i=1 ci ≤ i=1 cei = O(n)

5.2: Fibonacci Heaps T.S. 3 5.2: Fibonacci Heaps T.S. 4 Crucial for many applications including shortest paths and minimum spanning trees!

Binomial Heap vs. Fibonacci Heap: Costs Outline

Operation Binomial heap Fibonacci heap actual cost amortized cost Structure MAKE-HEAP O(1) O(1)

INSERT O(log n) O(1)

MINIMUM O(log n) O(1) Operations

EXTRACT-MIN O(log n) O(log n)

UNION O(log n) O(1) Glimpse at the Analysis DECREASE-KEY O(log n) O(1)

DELETE O(log n) O(log n)

Can we perform EXTRACT-MIN If this was possible, then there would be a better than O(log n)? sorting algorithm with runtime o(n log n)!

5.2: Fibonacci Heaps T.S. 5 5.2: Fibonacci Heaps T.S. 6

Binomial Heap vs. Fibonacci Heap: Structure Structure of Fibonacci Heaps

Binomial Heap: Fibonacci Heap consists of binomial trees, and every order appears at most once Forest of MIN-HEAPs immediately tidy up after INSERT or MERGE Nodes can be marked 5 7 [slides/handout: roots are always unmarked, CLRS: roots can be marked, but have to be unmarked once they become a child.] 38 51 50 26 22 Tree roots are stored in a circular, doubly-linked list Min-Pointer pointing to the smallest element 63 37 30 48 min 54

Fibonacci Heap: 58 30 10 forest of MIN-HEAPs lazily defer tidying up; do it on-the-ﬂy when search for the MIN 43 41 33 70 32 37 22 5 51

54 66 82 51 50 38 26 19 7 30

66 48 How do we implement a Fibonacci Heap?

5.2: Fibonacci Heaps T.S. 7 5.2: Fibonacci Heaps T.S. 8 A single Node Magnifying a Four-Node Portion

58 30 10

Parent 58 10 43 41 33 70 32 p 30 0 3

54 66 82 51 b payload marked degree f Previous Sibling Next Sibling 0 3

c 43 0 0 41 1 2 33 0 1

One of the Children

54 82 5.2: Fibonacci Heaps T.S. 9 5.2: Fibonacci Heaps T.S. 10

Outline Fibonacci Heap: INSERT

INSERT Create a singleton tree Add to root list and update min-pointer (if necessary)

Structure 21

Operations Actual Costs: O(1) min

17 24 23 7 21 3 Glimpse at the Analysis 30 26 46 18 52 41

35 39 44

5.2: Fibonacci Heaps T.S. 11 5.2: Fibonacci Heaps T.S. 12 X X X X X

41 41

44 44

X X

X Meld childen into root list and unmark them X Consolidate so that no roots have the same degree (# children) Consolidate so that no roots have the same degree (# children) Update minimum Update minimum

Every root becomes child of another root at most once! Every root becomes child of another root at most once! degree degree 0 1 2 3 0 1 2 3 Actual Costs: O(trees(H) + d(n)) Actual Costs: O(trees(H) + d(n)) min min min 1 degree=22 0 degree=00 1 0 1 1 degree=22 0 degree=00 1 0 1

17 18 52 17 183 52

17 24 23 41 39 17 24 23 41 39

23 26 46 44 23 26 46 44

35 35

X X Update minimum Update minimum

Every root becomes child of another root at most once! Every root becomes child of another root at most once! degree 0 1 2 3 Actual Costs: O(trees(H) + d(n)) Actual Costs: O(trees(H) + d(n)) min min min min degree=2 degree=0 1 degree=22 0 degree=00 1 0 1

17 3 17 3

17 24 23 41 18 52 41 17 24 23 41 18 52 41

23 26 46 44 39 44 23 26 46 44 39 44

35 35

Fibonacci Heap: EXTRACT-MIN Fibonacci Heap: EXTRACT-MIN

EXTRACT-MIN EXTRACT-MIN Delete min Delete min X Meld childen into root list and unmark them

min

7 24 23 17 3 7 24 23 17

30 26 46 18 52 41 30 26 46 18 52 41

35 39 44 35 39 44

5.2: Fibonacci Heaps T.S. 13 5.2: Fibonacci Heaps T.S. 13

Fibonacci Heap: EXTRACT-MIN Fibonacci Heap: EXTRACT-MIN

EXTRACT-MIN EXTRACT-MIN Delete min X Delete min X Meld childen into root list and unmark them X Meld childen into root list and unmark them X Consolidate so that no roots have the same degree (# children) Consolidate so that no roots have the same degree (# children)

degree 0 1 2 3

1 2 0 0 1 0 1

7 24 23 17 18 52 41 7 24 23 17 18 52 41

30 26 46 39 44 30 26 46 39 44

35 35

5.2: Fibonacci Heaps T.S. 13 5.2: Fibonacci Heaps T.S. 13 X X

X X

X X Update minimum Update minimum

Every root becomes child of another root at most once! Every root becomes child of another root at most once!

Actual Costs: O(trees(H) + d(n)) Actual Costs: O(trees(H) + d(n)) min min min min 1 degree=22 0 degree=00 1 0 1 1 degree=22 0 degree=00 1 0 1

23 17 3 23 17 17 3

17 24 41 18 52 41 24 23 41 18 52 41

23 26 46 44 39 44 26 46 44 39 44

35 35

X X Update minimum Update minimum

Every root becomes child of another root at most once! Every root becomes child of another root at most once!

Actual Costs: O(trees(H) + d(n)) Actual Costs: O(trees(H) + d(n)) min min min min 1 degree=22 0 degree=00 1 0 1 1 degree=22 0 degree=00 1 0 1

24 23 17 17 3 24 23 17 17 3

26 46 23 41 18 52 41 26 46 23 41 18 52 41

35 44 39 44 35 44 39 44

Fibonacci Heap: EXTRACT-MIN Fibonacci Heap: EXTRACT-MIN

degree degree 0 1 2 3 0 1 2 3

7 24 17 18 52 41 7 24 18 52 41

30 26 46 23 39 44 17 30 26 46 39 44

35 23 35

5.2: Fibonacci Heaps T.S. 13 5.2: Fibonacci Heaps T.S. 13

Fibonacci Heap: EXTRACT-MIN Fibonacci Heap: EXTRACT-MIN

degree degree 0 1 2 3 0 1 2 3

7 18 52 41 7 18 52 41

17 30 24 39 44 17 30 24 39 44

23 26 46 23 26 46

35 35

5.2: Fibonacci Heaps T.S. 13 5.2: Fibonacci Heaps T.S. 13 X

Update minimum X

Every root becomes child of another root at most once! Every root becomes child of another root at most once! degree 0 1 2 3 Actual Costs: O(trees(H) + d(n)) Actual Costs: O(trees(H) + d(n)) min min min 1 degree=22 0 degree=00 1 0 1 1 degree=22 0 degree=00 1 0 1

24 23 17 17 3 41 24 23 17 17 3 41

26 46 23 18 52 41 44 26 46 23 18 52 41 44

35 39 44 35 39 44

degree Peculiar Constraint: Make sure that each non-root Degree = 3, 0 1 2 3 node loses at most one child before becoming root Nodes = 4 min 1 degree=22 0 degree=00 1 0 1

24 23 17 17 3 41 20

26 46 23 18 52 41 44 19 15 12 35 39 44 Wide and 5 shallow tree

Fibonacci Heap: EXTRACT-MIN Fibonacci Heap: EXTRACT-MIN

EXTRACT-MIN EXTRACT-MIN Delete min X Delete min X Meld childen into root list and unmark them X Meld childen into root list and unmark them X Consolidate so that no roots have the same degree (# children) X Consolidate so that no roots have the same degree (# children) X Update minimum

degree 0 1 2 3

min

7 18 52 7 18 52

17 30 24 41 39 17 30 24 41 39

23 26 46 44 23 26 46 44

35 35

5.2: Fibonacci Heaps T.S. 13 5.2: Fibonacci Heaps T.S. 13

Fibonacci Heap: EXTRACT-MIN Fibonacci Heap: DECREASE-KEY (First Attempt)

EXTRACT-MIN DECREASE-KEY of node x Delete min X Decrease the key of x (given by a pointer) Meld childen into root list and unmark them X Check if heap-order is violated Consolidate so that no roots have the same degree (# children) X If not, then done. Otherwise, cut tree rooted at x and meld into root list. Update minimum X

Every root becomes child of another root at most once!

Actual Costs: O(trees(H) + d(n)) min 7 18 38 15 5 19 12

7 18 52 24 20 17 23 21 39 41 1.D ECREASE-KEY 24 20 17 30 24 41 39 2.D ECREASE-KEY 46 15 26 46 30 52 3.D ECREASE-KEY 35 5 23 26 46 44 19 15 12 4.D ECREASE-KEY 26 19 35 5.D ECREASE-KEY 30 12 35 5

5.2: Fibonacci Heaps T.S. 13 5.2: Fibonacci Heaps T.S. 14 Peculiar Constraint: Make sure that each non-root node loses at most one child before becoming root Degree = 3, Nodes = 4

20 1.D ECREASE-KEY 24 20 20 1.D ECREASE-KEY 24 20 2.D ECREASE-KEY 46 15 2.D ECREASE-KEY 46 15 19 15 12 3.D ECREASE-KEY 35 5 19 15 12 3.D ECREASE-KEY 35 5 4.D ECREASE-KEY 26 19 4.D ECREASE-KEY 26 19 Wide and 5 5 5.D ECREASE-KEY 30 12 shallow tree 5.D ECREASE-KEY 30 12

min

Fibonacci Heap: DECREASE-KEY (First Attempt) Fibonacci Heap: DECREASE-KEY (First Attempt)

DECREASE-KEY of node x DECREASE-KEY of node x Decrease the key of x (given by a pointer) Decrease the key of x (given by a pointer) Check if heap-order is violated Check if heap-order is violated If not, then done. If not, then done. Otherwise, cut tree rooted at x and meld into root list. Otherwise, cut tree rooted at x and meld into root list.

Peculiar Constraint: Make sure that each non-root Degree = 3, node loses at most one child before becoming root Nodes = 4

7 18 38 15 5 19 12 7 18 38 15 5 19 12

24 20 17 23 21 39 41 24 20 17 23 21 39 41

26 46 30 52 26 46 30 52 19 15 12 19 15 12 Wide and 35 35 shallow tree 5 5

5.2: Fibonacci Heaps T.S. 14 5.2: Fibonacci Heaps T.S. 14

Fibonacci Heap: DECREASE-KEY Outline

DECREASE-KEY of node x Decrease the key of x (given by a pointer) (Here we consider only cases where heap-order is violated) ⇒ Cut tree rooted at x, unmark x, and meld into root list and: Check if parent node is marked If unmarked, mark it (unless it is a root) Structure If marked, unmark and meld it into root list and recurse (Cascading Cut)

Actual Cost: O(# cuts) min Operations

7 18 38 15 5 26 24 Glimpse at the Analysis 24 17 23 21 39 41

26 46 30 52 1.D ECREASE-KEY 46 15 X 15 2.D ECREASE-KEY 35 5 X 35 5

5.2: Fibonacci Heaps T.S. 15 5.2: Fibonacci Heaps T.S. 16 Amortized Analysis via Potential Method

INSERT: actual O(1) amortized O(1) EXTRACT-MIN: actual O(trees(H) + d(n)) amortized O(d(n)) DECREASE-KEY: actual O(# cuts) amortized O(1)

Φ(H) = trees(H)+2·marks(H)

7 18 38

24 17 23 21 39 41

26 46 30 52

5.2: Fibonacci Heaps T.S. 17 Outline

Amortized Analysis

5.2 Fibonacci Heaps (Analysis) Bounding the Maximum Degree Frank Stajano Thomas Sauerwald

Lent 2015

5.2: Fibonacci Heaps (Analysis) T.S. 2

Amortized Analysis of DECREASE-KEY Amortized Analysis of EXTRACT-MIN

Actual Cost Actual Cost DECREASE-KEY: O(x), where x is the number of cuts. EXTRACT-MIN: O(trees(H) + d(n))

Φ(H) = trees(H) + 2 · marks(H) Φ(H) = trees(H) + 2 · marks(H) First Coin pays cut Second Coin increase of trees(H) degrees 7 18 Change in Potential 0 1 2 3 d(n) Change in Potential marks(H0) ≤ marks(H) 24 17 23 21 39 0 trees(H0) = trees(H) + x trees(H ) ≤ d(n) + 1 0 marks(H ) ≤ marks(H) − x + 2 26 30 35 ⇒ ∆Φ = d(n) + 1 − trees(H) ⇒ ∆Φ ≤ x + 2 · (−x + 2) = 4 − x. 52 Amortized Cost 5 Scale up potential units Amortized Cost eci = ci + ∆Φ = O(trees(H) +d(n)) + d(n) + 1 − trees(H) = O(d(n))

ci = ci + ∆Φ = O(x) + 4 − x = O(1) e How to bound d(n)?

5.2: Fibonacci Heaps (Analysis) T.S. 3 5.2: Fibonacci Heaps (Analysis) T.S. 4 ?

Outline Bounding the Maximum Degree

Binomial Heap

Every tree is a binomial tree ⇒ d(n) ≤ log2 n.

Amortized Analysis

Bounding the Maximum Degree

d = 3, n = 23

Fibonacci Heap

Not all trees are binomial trees, but still d(n) ≤ logϕ n, where ϕ ≈ 1.62.

5.2: Fibonacci Heaps (Analysis) T.S. 5 5.2: Fibonacci Heaps (Analysis) T.S. 6

Lower Bounding Degrees of Children From Degrees to Minimum Subtree Sizes

We will prove a stronger statement: x A tree with degree k contains at least ϕk nodes.

y1 y2 y3 y4 yk

d(n) ≤ logϕ n

∀1 ≤ i ≤ k : di ≥ i − 2

Consider any node x (not necessarily a root) at the ﬁnal state Deﬁnition

Let y1, y2,..., yk be the children in the order of attachment Let N(k) be the minimum possible number of nodes of a subtree rooted and d1, d2,..., dk be their degrees at a node of degree k.

⇒∀ 1 ≤ i ≤ k : di ≥ i − 2 N(k) = F(k + 2)?!

x N(0) = 1 N(1) = 2 N(2) = 3 N(3) = 5 N(4) = 8 = 5 + 3 3 4 0 1 2 y y y y y 1 2 3 4 k 0 0 0 0 0 1 0 0 1 2 0 0 0 0

5.2: Fibonacci Heaps (Analysis) T.S. 7 5.2: Fibonacci Heaps (Analysis) T.S. 8 From Minimum Subtree Sizes to Fibonacci Numbers Exponential Growth of Fibonacci Numbers

∀1 ≤ i ≤ k : d ≥ i − 2 Lemma 19.3 i k For all integers k√≥ 0, the (k + 2)nd Fib. number satisﬁes Fk+2 ≥ ϕ , 1 where ϕ = (1 + 5)/2 = 1.61803 .... N(k) = ϕ2 = ϕ + 1 Fibonacci Numbers grow at least exponentially fast in k.

1 N(2 − 2) N(3 − 2) N(k − 2) Proof by induction on k: 0 Base k = 0: F2 = 1 and ϕ = 1 X 1 N(k) = 1 + 1 + N(2 − 2) + N(3 − 2) + ··· + N(k − 2) Base k = 1: F3 = 2 and ϕ < 1.619 X k−2 Inductive Step (k ≥ 2): X = 1 + 1 + N(`) Fk+2 = Fk+1 + Fk `=0 k−1 k−2 k−3 ≥ ϕ + ϕ (by the inductive hypothesis) X = 1 + 1 + N(`) + N(k − 2) = ϕk−2 · (ϕ + 1) `=0 k−2 2 2 = N(k − 1) + N(k − 2) = ϕ · ϕ (ϕ = ϕ + 1) = F(k + 1) + F(k) = F(k + 2) = ϕk

5.2: Fibonacci Heaps (Analysis) T.S. 9 5.2: Fibonacci Heaps (Analysis) T.S. 10

Putting the Pieces Together Summary

Fibonacci Heaps were developed by Fredman and Tarjan in 1984 Fibonacci Numbers were discovered >800 years ago Amortized Analysis INSERT: amortized cost O(1) Operation Linked list Binary heap Binomial heap Fibon. heap XX MAKE-HEAP O(1) O(1) O(1) O(1) EXTRACT-MIN amortized cost O(dX(nXX)) O(log n) DECREASE-KEY amortized cost O(1) INSERT O(1) O(log n) O(log n) O(1) MINIMUM O(n) O(1) O(log n) O(1)

EXTRACT-MIN O(n) O(log n) O(log n) O(log n)

UNION O(n) O(n) O(log n) O(1) k n ≥ N(k) = F(k + 2) ≥ ϕ DECREASE-KEY O(1) O(log n) O(log n) O(1) ⇒ logϕ n ≥ k DELETE O(1) O(log n) O(log n) O(log n)

DELETE = DECREASE-KEY + EXTRACT-MIN

EXTRACT-MIN = MIN + DELETE

5.2: Fibonacci Heaps (Analysis) T.S. 11 5.2: Fibonacci Heaps (Analysis) T.S. 12 Outlook: A More Efﬁcient Priority Queue

Operation Fibonacci heap Van Emde Boas Tree amortized cost actual cost INSERT O(1) O(log log u) MINIMUM O(1) O(1) EXTRACT-MIN O(log n) O(log log u) MERGE/UNION O(1) - DECREASE-KEY O(1) O(log log u) DELETE O(log n) O(log log u) SUCC - O(log log u) PRED - O(log log u) MAXIMUM - O(1)

all this requires key values to be in a universe of size u!

5.2: Fibonacci Heaps (Analysis) T.S. 13 Handle findSet(Item x) Precondition: there exists a set that contains x (given pointer to x) Behaviour: return the handle of the set that contains x Handle union(Handle h, Handle g) Handle union(Handle h, Handle g) Precondition:h 6= g Precondition:h 6= g Behaviour: merge two disjoint sets and return handle of new set Behaviour: merge two disjoint sets and return handle of new set

h2 h2 h1 h3 = h4 h3 = h4

hh451=Union(=findSet(y)h10,h23) y hh450=Union(=makeSet(x)h10,h23)

h5 h5

Outline

Disjoint Sets

5.3: Disjoint Sets Graph Representations Frank Stajano Thomas Sauerwald

Lent 2015

5.3: Disjoint Sets T.S. 2

Disjoint Sets (aka Union Find) Disjoint Sets (aka Union Find)

h0 h1 h0

h0=makeSet(x) x h1=findSet(y) y x

5.3: Disjoint Sets T.S. 3 5.3: Disjoint Sets T.S. 3 h 2 = h h1 4 h0

hh501=Union(=makeSet(x)=findSet(y)h1,h2) hh401=Union(=makeSet(x)=findSet(y)h0,h3)

h5 h5

Need to update all FIND-Operation h2 backward pointers! h1 h0 Add backward pointer to the list head from everywhere FindSet(z3)

hh401=Union(=makeSet(x)=findSet(y)h0,h3) ⇒ FIND takes constant time h4

z1 z2 z3 z4

Disjoint Sets (aka Union Find) Disjoint Sets (aka Union Find)

Disjoint Sets Data Structure Disjoint Sets Data Structure Handle makeSet(Item x) Handle makeSet(Item x) Precondition: none of the existing sets contains x Precondition: none of the existing sets contains x Behaviour: create a new set {x} and return its handle Behaviour: create a new set {x} and return its handle Handle findSet(Item x) Handle findSet(Item x) Precondition: there exists a set that contains x (given pointer to x) Precondition: there exists a set that contains x (given pointer to x) Behaviour: return the handle of the set that contains x Behaviour: return the handle of the set that contains x Handle union(Handle h, Handle g) Handle union(Handle h, Handle g) Precondition:h 6= g Precondition:h 6= g Behaviour: merge two disjoint sets and return handle of new set Behaviour: merge two disjoint sets and return handle of new set

h2 h3 h3 = h h0 h1 4

h4=Union(h0,h3) y x h5=Union(h1,h2) y x

5.3: Disjoint Sets T.S. 3 5.3: Disjoint Sets T.S. 3

Disjoint Sets (aka Union Find) First Attempt: List Implementation

Disjoint Sets Data Structure Handle makeSet(Item x) Precondition: none of the existing sets contains x UNION-Operation Union(h1, h2) Need to ﬁnd Behaviour: create a new set {x} and return its handle last element! Add extra pointer to the last h Handle findSet(Item x) element in each list 1 Precondition: there exists a set that contains x (given pointer to x) x x x Behaviour: return the handle of the set that contains x ⇒ UNION takes constant time 1 2 3

Handle union(Handle h, Handle g) h2 Precondition:h 6= g Behaviour: merge two disjoint sets and return handle of new set y1 y2

h3 = h4

h5=Union(h1,h2) y x

5.3: Disjoint Sets T.S. 3 5.3: Disjoint Sets T.S. 4 h3 h20 h10 h0

First Attempt: List Implementation First Attempt: List Implementation (Analysis)

d = DisjointSet() h0 = d.MakeSet(x0)

UNION-Operation Union(h1, h2) Need to ﬁnd last element! = .MakeSet Add extra pointer to the last h1 d (x1) h = .union , element in each list 1 h0 d (h1 h0) h2 = d.MakeSet(x2) ⇒ UNION takes constant time x1 x2 x3 h0 = d.union(h2, h0) h3 = d.MakeSet(x3) h2 h0 = d.union(h3, h0) h0 y1 y2

x3 x2 x1 x0 Need to update all FIND-Operation backward pointers! Add backward pointer to the list head from everywhere FindSet(z3) better to append shorter list to longer Weighted-Union Heuristic ⇒ FIND takes constant time h4

z1 z2 z3 z4 Pn 2 Cost for n UNION operations: i=1 i = Θ(n )

5.3: Disjoint Sets T.S. 4 5.3: Disjoint Sets T.S. 5

Weighted-Union Heuristic Analysis of Weighted-Union Heuristic

h0 h1

Weighted-Union Heuristic h2 Keep track of the length of each list x Append shorter list to the longer list (breaking ties arbitrarily)

Theorem 21.1 can be done easily without signiﬁcant overhead Using the weighted-union heuristic, any sequence of m operations, n of which are MAKE-SET operations, takes O(m + n · log n) time. Theorem 21.1 Using the Weighted-Union heuristic, any sequence of m operations, n of Proof: Can we improve on this further? which are MAKE-SET operations, takes O(m + n · log n) time. n MAKE-SET operations ⇒ at most n − 1 UNION operations Amortized Analysis: Every operation has amortized cost Consider element x and the number of updates of the backward pointer O(log n), but there may be operations with total cost Θ(n). After each update of x, its set increases by a factor of at least 2

⇒ Backward pointer of x is updated at most log2 n times Other updates for UNION, MAKE-SET & FIND-SET take O(1) time per operation

5.3: Disjoint Sets T.S. 6 5.3: Disjoint Sets T.S. 7 b h

How to Improve? Disjoint Sets via Forests

h1 {b, c, e, h} {d, f , g} {b, c, d, e, f , g, h}

h0 Rank may be just an upper bound on the height!

rank = 2 rank = 2 3 rank = 3 c f f

h e d c d

b g h e g Basic Idea: Update Backward Pointers only during FIND b

Forest Structure Doubly-Linked List Weighted-Union Heuristic Set is represented by a rooted tree with root being the representative MAKE-SET: O(1) MAKE-SET: O(1) Every node has pointer .p to its parent (for root x, x.p = x) FIND-SET: O(n) FIND-SET: O(1) UNION: Merge the two trees UNION: O(1) UNION: O(log n) (amortized) Append tree of smaller height Union by Rank

5.3: Disjoint Sets T.S. 8 5.3: Disjoint Sets T.S. 9

Path Compression during FIND-SET Combining Union by Rank and Path Compression

Data Structure is essentially optimal! (for more details see CLRS) b h c Theorem 21.14 f Any sequence of m MAKE-SET, UNION, FIND-SET operations, n of f which are MAKE-SET operations, can be performed in O(m·α(n)) time.

c d In practice, α(n) is a small constant

h e g   0 for 0 ≤ n ≤ 2,   1 for n = 3, b  α(n) = 2 for 4 ≤ n ≤ 7,   3 for 8 ≤ n ≤ 2047,  0: FIND-SET(x)  4 for 2048 ≤ n ≤ 1080 1: if x 6= x.p 2: x.p = FIND-SET(x.p) More than the number of atoms in the universe! 3: return x.p

5.3: Disjoint Sets T.S. 10 5.3: Disjoint Sets T.S. 11 Path p = (1, 2, 3, 1), which is a cycle G is not a DAG

G is not (strongly) connected

Later: edge-weighted graphs G = (V , E, w)

G is connected

If each pair of vertices has a path linking them, then G is connected

Outline

Introduction to Graphs and Graph Searching

Breadth-First Search

6.1 & 6.2: Graph Searching Depth-First Search Frank Stajano Thomas Sauerwald

Topological Sort

28 February 2014

6.1 & 6.2: Graph Searching T.S. 2

Origin of Graph Theory What is a Graph?

Directed Graph Path p = (1, 2, 3, 4) A graph G = (V , E) consists of: A 1 2 V : the set of vertices E: the set of edges (arcs) 3 4 B D Undirected Graph C V = {1, 2, 3, 4} A graph G = (V , E) consists of: E = {(1, 2), (1, 3), (2, 3), (3, 1), (3, 4)}

Source: Wikipedia Source: Wikipedia V : the set of vertices 1 2 Seven Bridges at Königsberg 1737 Leonhard Euler (1707-1783) E: the set of (undirected) edges

A Is there a tour which crosses each bridge exactly once? Paths and Connectivity 3 4 A sequence of edges between B D Is there a tour which visits every two vertices forms a path V = {1, 2, 3, 4} island exactly once? E = {{1, 2}, {1, 3}, {2, 3}, {3, 4}} 1B course: Complexity Theory C

6.1 & 6.2: Graph Searching T.S. 3 6.1 & 6.2: Graph Searching T.S. 4 Path p = (1, 2, 3, 4) Path p = (1, 2, 3, 41), which is a cycle G is not a DAG

G is not (strongly) connected

Later: edge-weighted graphs G = (V , E, w) Later: edge-weighted graphs G = (V , E, w)

G is connected

If each pair of vertices has a path linking them, then G is connected

What is a Graph? What is a Graph?

Directed Graph Path p = (1, 2, 3, 1), which is a cycle Directed Graph A graph G = (V , E) consists of: G is not a DAG 1 2 A graph G = (V , E) consists of: 1 2 V : the set of vertices V : the set of vertices E: the set of edges (arcs) E: the set of edges (arcs) 3 4 G is not (strongly) 3 4 connected Undirected Graph V = {1, 2, 3, 4} Undirected Graph V = {1, 2, 3, 4} A graph G = (V , E) consists of: E = {(1, 2), (1, 3), (2, 3), (3, 1), (3, 4)} A graph G = (V , E) consists of: E = {(1, 2), (1, 3), (2, 3), (3, 1), (3, 4)} V : the set of vertices V : the set of vertices 1 2 1 2 E: the set of (undirected) edges E: the set of (undirected) edges

Paths and Connectivity 3 4 Paths and Connectivity G is connected 3 4 A sequence of edges between A sequence of edges between two vertices forms a path V = {1, 2, 3, 4} two vertices forms a path V = {1, 2, 3, 4} E = {{1, 2}, {1, 3}, {2, 3}, {3, 4}} If each pair of vertices has a E = {{1, 2}, {1, 3}, {2, 3}, {3, 4}} path linking them, then G is connected

6.1 & 6.2: Graph Searching T.S. 4 6.1 & 6.2: Graph Searching T.S. 4

Representations590 Chapter 22 Elementary of Directed Graph Algorithms and Undirected Graphs Graph Searching

12345 1 25 1 0 1001 5 12 2 1 5 3 4 2 1 0111 8 3 2 4 3 0 1010 4 3 2 7 9 4 2 5 3 4 0 1101 5 4 5 4 1 2 5 1 1010 10 3 (a) (b) (c) 1 6

Figure 22.1 Two representations of an undirected graph. (a) An undirected graph G with 5 vertices Overview and 7 edges. (b) An adjacency-list representation of G. (c) The adjacency-matrix representation of G. Graph searching means taversing a graph via the edges in order to visit all vertices 123456 useful for identifying connected components, computing the 1 24 1 0 10100 2 5 2 0 00010 diameter etc. 123 3 6 5 3 0 00011 Two strategies: Breadth-First-Search and Depth-First-Search 4 2 4 0 10000 5 4 5 0 00100 4 5 6 6 6 6000001 (a) (b) (c) Measure time complexity in terms of the size of V and E (often write just V instead of |V |, and E instead of |E|) Figure 22.2 Two representations of a directed graph. (a) AdirectedgraphG with 6 vertices and 8 edges. (b) An adjacency-list representation of G. (c) The adjacency-matrix representation of G.

shortest-paths6.1 & 6.2: Graph algorithms Searching presented in Chapter 25 assume T.S. that their input graphs 5 6.1 & 6.2: Graph Searching T.S. 6 are represented by adjacency matrices. The adjacency-list representation of a graph G .V; E/ consists of an ar- D ray Adj of V lists, one for each vertex in V .Foreachu V ,theadjacencylist j j 2 AdjŒu contains all the vertices " such that there is an edge .u; "/ E.Thatis, 2 AdjŒu consists of all the vertices adjacent to u in G.(Alternatively,itmaycontain pointers to these vertices.) Since the adjacency lists represent the edges of a graph, in pseudocode we treat the array Adj as an attribute of the graph, just as we treat the edge set E.Inpseudocode,therefore,wewillseenotationsuchasG:AdjŒu. Figure 22.1(b) is an adjacency-list representation of the undirected graph in Fig- ure 22.1(a). Similarly, Figure 22.2(b) is an adjacency-list representation of the directed graph in Figure 22.2(a). If G is a directed graph, the sum of the lengths of all the adjacency lists is E , j j since an edge of the form .u; "/ is represented by having " appear in AdjŒu.IfG is ¡sA ¡r A w Z v¡A t£C x¡A u¡A y¡A s r w v t x u y

r s t u r s t u 1 0 2 3 1 0 2 3

2 1 2 3 2 1 2 3 v w x y v w x y

Outline Breadth-First Search: Basic Ideas

Introduction to Graphs and Graph Searching

Breadth-First Search

Basic Idea Depth-First Search Given an undirected/directed graph G = (V , E) and source vertex s BFS sends out a wave from s compute distances/shortest paths Vertex Colours: Topological Sort White = Unvisited Grey = Visited, but not all neighbors (=adjacent vertices) Black = Visited and all neighbors

6.1 & 6.2: Graph Searching T.S. 7 6.1 & 6.2: Graph Searching T.S. 8

Complete Execution of BFS (Figure 22.3) Complete Execution of BFS (Figure 22.3)

Queue: Queue: ¡sA ¡r A w Z v¡A t£C x¡A u¡A y¡A

r s t u r s t u ∞ 0 ∞ ∞ ∞1 0 ∞2 ∞3

∞ ∞ ∞ ∞ ∞2 ∞1 ∞2 ∞3 v w x y v w x y

6.1 & 6.2: Graph Searching T.S. 9 6.1 & 6.2: Graph Searching T.S. 9 w vz y xr u

s v w

1/12 2/11 13/16

4/5 3/10 14/15

x y z

7/8 6/9

u r

Breadth-First-Search: Pseudocode Outline

0: def bfs(G) 1: Run BFS on the given graph G 2: starting from source s 3: 4: assert(s in G.vertices()) 5: Introduction to Graphs and Graph Searching 6: # Initialize graph and queue 7: for v in G.vertices(): 8: v.predecessor = None From any vertex, visit all adjacent 9: v.d = Inﬁnity # .d = distance from s vertices before going any deeper 10: v.colour = "white" Breadth-First Search 11: Q = Queue() Vertex Colours: 12: 13: # Visit source vertex White = Unvisited 14: s.d = 0 15: s.colour = "grey" Grey = Visited, but not all neighbors 16: Q.insert(s) Depth-First Search 17: 18: # Visit the adjacents of each vertex in Q Black = Visited and all neighbors 19: while not Q.isEmpty(): 20: u = Q.extract() Runtime O(V + E) 21: assert (u.colour == "grey") Topological Sort 22: for v in u.adjacent() 23: if v.colour = "white" Assuming that all executions of the FOR-loop 24: v.colour = "grey" 25: v.d = u.d+1 for u takes O(|u.adj|) (adjacency list model!) 26: v.predecessor = u 27: Q.insert(v) P 28: u.colour = "black" u∈V |u.adj| = 2|E|

6.1 & 6.2: Graph Searching T.S. 10 6.1 & 6.2: Graph Searching T.S. 11

Depth-First Search: Basic Ideas Complete Execution of DFS

s v w

Basic Idea = ( , ) Given an undirected/directed graph G V E and source vertex s x y z As soon as we discover a vertex, explore from it Solving Mazes Two time stamps for every vertex: Discovery Time, Finishing Time

u r

6.1 & 6.2: Graph Searching T.S. 12 6.1 & 6.2: Graph Searching T.S. 13 w z s v y x xr u

13/16

14/15

6/ 6/

r r

Complete Execution of DFS Complete Execution of DFS

s v y w z r u

s v w s v w

1/12 2/11 1/12 2/11 13/16

4/5 3/10 4/5 3/10 14/15

x y z x y z

7/8 6/9 7/8 6/9

u r u r

6.1 & 6.2: Graph Searching T.S. 13 6.1 & 6.2: Graph Searching T.S. 13

Paranthesis Theorem (Theorem 22.7) Depth-First-Search: Pseudocode

s w 0: def dfs(G): 1: Run DFS on the given graph G 2: starting from the given source s 3: s v w 4: assert(s in G.vertices()) v z 5: We always go deeper before visiting 6: # Initialize graph 1/12 2/11 13/16 7: for v in G.vertices(): other neighbors 8: v.predecessor = None Discovery and Finish times, .d and .v 9: v.colour = "white" y 10: dfsRecurse(G,s) Vertex Colours: 4/5 3/10 14/15 White = Unvisited 0: def dfsRecurse(G,s): Grey = Visited, but not all neighbors x y z 1: s.colour = "grey" x r 2: s.d = time() # .d = discovery time Black = Visited and all neighbors 7/8 6/9 3: for v in s.adjacent() 4: if v.colour = "white" Runtime O(V + E) 5: v.predecessor = s u r 6: dfsRecurse(G,v) u 7: s.colour = "black" 8: s.f = time() # .f = ﬁnish time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 (s (v (y (x x) (r (u u) r) y) v) s) (w (z z) w)

6.1 & 6.2: Graph Searching T.S. 14 6.1 & 6.2: Graph Searching T.S. 15 Outline Topological Sort

undershorts socks watch Introduction to Graphs and Graph Searching pants shoes shirt Breadth-First Search belt tie Problem Given: a directed acyclic graph (DAG)

Depth-First Search jacket Goal: Output a linear ordering of all vertices

Topological Sort

socks undershorts pants shoes watch shirt belt tie jacket

6.1 & 6.2: Graph Searching T.S. 16 6.1 & 6.2: Graph Searching T.S. 17

Solving Topological Sort Execution of Knuth’s Algorithm

undershorts socks s v w watch 1/12 2/11 13/16 pants shoes shirt belt 4/5 3/10 14/15 tie x y z jacket 7/8 6/9

Knuth’s Algorithm (1968) u r Perform DFS’s so that all vertices are visited Output vertices in decreasing order of their ﬁnishing time

Don’t need to sort the ver- 13/16 14/15 1/12 2/11 3/10 6/9 7/8 4/5 Runtime O(V + E) tices – use DFS directly! w z s v y r u x

6.1 & 6.2: Graph Searching T.S. 18 6.1 & 6.2: Graph Searching T.S. 19 we call DFS(v) and v.f < u.f . we call DFS(v) and v.f < u.f .

2. If v is black, then v.f < u.f . ?/3/89/? 4/9/? 3. If v is white, 3. If v is white, v v ⇒ In all cases v.f < v.u, so v appears after u. ⇒ In all cases v.f < v.u, so v appears after u.

?/4/3/8 v

Correctness of Topological Sort using DFS Correctness of Topological Sort using DFS

13/16 14/15 1/12 2/11 3/10 6/9 7/8 4/5 13/16 14/15 1/12 2/11 3/10 6/9 7/8 4/5

w z s v y r u x w z s v y r u x

Theorem 22.12 Theorem 22.12 If the input graph is a DAG, then the algorithm computes a linear order. If the input graph is a DAG, then the algorithm computes a linear order.

Proof: Proof: s s Consider any edge (u, v) ∈ E(G) being explored, Consider any edge (u, v) ∈ E(G) being explored, 1/ 1/ ⇒ u is grey and we have to show that v.f < u.f ⇒ u is grey and we have to show that v.f < u.f 1. If v is grey, then there is a cycle 1. If v is grey, then there is a cycle (can’t happen, because G is acyclic!). (can’t happen, because G is acyclic!). 2. If v is black, then v.f < u.f . 7/ 4/ 7/ ?/3/8 u v u v

6.1 & 6.2: Graph Searching T.S. 20 6.1 & 6.2: Graph Searching T.S. 20

Correctness of Topological Sort using DFS Summary of Graph Searching

13/16 14/15 1/12 2/11 3/10 6/9 7/8 4/5 Breadth-First-Search w z s v y r u x vertices are processed by a queue computes distances and shortest paths Theorem 22.12 similar idea used later in Prim’s and Dijkstra’s algorithm If the input graph is a DAG, then the algorithm computes a linear order. Runtime O(V + E)

Proof: s Consider any edge (u, v) ∈ E(G) being explored, 1/ Depth-First-Search ⇒ u is grey and we have to show that v.f < u.f vertices are processed by recursive calls( ≈ stack) 1. If v is grey, then there is a cycle discovery and ﬁnishing times (can’t happen, because G is acyclic!). application: Topogical Sorting of DAGs 2. If v is black, then v.f < u.f . 7/ 9/? 3. If v is white, we call DFS(v) and v.f < u.f . Runtime O(V + E) u v ⇒ In all cases v.f < v.u, so v appears after u.

6.1 & 6.2: Graph Searching T.S. 20 6.1 & 6.2: Graph Searching T.S. 21 Outline

Minimum Spanning Tree Problem

6.3: Minimum Spanning Tree Single-Source Shortest Path Frank Stajano Thomas Sauerwald

Lent 2015

6.3: Minimum Spanning Tree T.S. 2

Minimum Spanning Tree Problem Generic Algorithm

Minimum Spanning Tree Problem 11 8 0: def minimum spanningTree(G) Given: undirected, connected 1: A = empty set of edges graph G = (V , E, w) with 6 6 2 7 2: while A does not span all vertices yet: non-negative edge weights 4 3 4 3: add a safe edge to A Goal: Find a subgraph ⊆ E of 9 minimum total weight that links 9 8 3 all vertices 5 Deﬁnition Must be necessarily a tree! A edge of G is safe if by adding the edge to A, the resulting subgraph is still a subset of a minimum spanning tree. Applications Street Networks, Wiring Electronic Components, Laying Pipes Weights may represent distances, costs, travel times, capacities, resistance etc. How to ﬁnd a safe edge?

6.3: Minimum Spanning Tree T.S. 3 6.3: Minimum Spanning Tree T.S. 4 This tree must be a spanning tree If w(e`) < w(ex ), then this spanning tree has smaller cost than T (can’t happen!) If w(e`) = w(ex ), then T ∪ e` \ ex is a MST.

This tree must be a spanning tree If w(e`) < w(ex ), then this spanning tree has smaller cost than T (can’t happen!) If w(e`) = w(ex ), then T ∪ e` \ ex is a MST.

If e` 6∈ T , then adding e` to T introduces cycle

This cycle crosses the cut through e` and ex another edge ex Consider now the tree T ∪ e` \ ex :

Consider now the tree T ∪ e` \ ex :

Finding safe edges Proof of Theorem

Theorem Let A ⊆ E be a subset of a MST of G. Then for any cut that respects A, the lightest edge of G that goes across the cut is safe. Deﬁnitions

a cut is a partition of V into at least Proof: two disjoint sets Let T be a MST containing A a cut respects A ⊆ E if no edge of Let e` be the lightest edge across the cut A goes across the cut e` If e` ∈ T , then we are done

Theorem Let A ⊆ E be a subset of a MST of G. Then for any cut that respects A, the lightest edge of G that goes across the cut is safe.

6.3: Minimum Spanning Tree T.S. 5 6.3: Minimum Spanning Tree T.S. 6

Proof of Theorem Proof of Theorem

Theorem Theorem Let A ⊆ E be a subset of a MST of G. Then for any cut that respects A, Let A ⊆ E be a subset of a MST of G. Then for any cut that respects A, the lightest edge of G that goes across the cut is safe. the lightest edge of G that goes across the cut is safe.

Proof: Proof: Let T be a MST containing A Let T be a MST containing A

Let e` be the lightest edge across the cut Let e` be the lightest edge across the cut e` e` If e` ∈ T , then we are done If e` ∈ T , then we are done

If e` 6∈ T , then adding e` to T introduces cycle If e` 6∈ T , then adding e` to T introduces cycle

This cycle crosses the cut through e` and ex This cycle crosses the cut through e` and ex another edge ex another edge ex Consider now the tree T ∪ e` \ ex : This tree must be a spanning tree If w(e`) < w(ex ), then this spanning tree has smaller cost than T (can’t happen!) If w(e`) = w(ex ), then T ∪ e` \ ex is a MST.

6.3: Minimum Spanning Tree T.S. 6 6.3: Minimum Spanning Tree T.S. 6 Glimpse at Kruskal’s Algorithm Glimpse at Kruskal’s Algorithm

Basic Strategy Basic Strategy Let A ⊆ E be a forest, intially empty Let A ⊆ E be a forest, intially empty At every step, given A, perform: At every step, given A, perform: Add lightest edge to A that does not introduce a cycle Add lightest edge to A that does not introduce a cycle

11 g 8 11 g 8

a h a h 2 2 6 6 7 6 6 7 4 4 b 3 4 f b 3 4 f c 9 c 9 3 3 9 8 9 8 Use Disjoint Sets to keep track Use Disjoint Sets to keep track 5 5 e of connected components! e of connected components! d d

6.3: Minimum Spanning Tree T.S. 7 6.3: Minimum Spanning Tree T.S. 7

Complete Run of Kruskal’s Algorithm Complete Run of Kruskal’s Algorithm

g g 11 8 11 8

a h a h 2 2 6 6 7 6 6 7 4 4

b 3 4 f b 3 4 f

c 9 c 9 3 3 9 8 9 8

5 e 5 e d d

6.3: Minimum Spanning Tree T.S. 8 6.3: Minimum Spanning Tree T.S. 8 Implementation will be based on vertices!

1126g

a h7

b63 4f

4c We computed same MST as Kruskal, but in a completely different order! 83e d95

Assign everyImplementation vertex not connected will be basedto A a onkey vertices which is! at all Assign everyImplementation vertex not connected will be basedto A a onkey vertices which is! at all stages equal to the smallest weight of an edge connecting to A stages equal to the smallest weight of an edge connecting to A

Use a Priority Queue! Use a Priority Queue! 1126g 1126g

a h7 a h7

b63 4f b63 4f

4c 4c We computed same MST as Kruskal, We computed same MST as Kruskal, but in a completely different order! but in a completely different order! 83e 83e d95 d95

Details of Kruskal’s Algorithm Prim’s Algorithm

0: def kruskal(G) Basic Strategy 1: Apply Kruskal’s algorithm to graph G 2: Return set of edges that form a MST Start growing a tree from a designated root vertex 3: At each step, add lightest edge linked to A that does not yield cycle 4: A = Set() # Set of edges of MST 5: D = DisjointSet() 6: for v in G.vertices(): Assign every vertex not connected to A a key which is at all 7: D.makeset(v) 8: E = G.edges() stages equal to the smallest weight of an edge connecting to A 9: E.sort(key=weight, direction=ascending) 10: Use a Priority Queue! 11: for edge in E: 12: startSet = D.findSet(edge.start) 13: endSet = D.findset(edge.end) 14: if startSet != endSet: 15: A.append(edge) 16: D.union(startSet,endSet) 17: return A

Time Complexity Initialization (l. 4-9): O(V + E log E) Main Loop (l. 11-16): O(E · α(n)) ⇒ Overall: O(E log E) = O(E log V ) If edges are already sorted, runtime becomes O(E · α(n))!

6.3: Minimum Spanning Tree T.S. 9 6.3: Minimum Spanning Tree T.S. 10

Prim’s Algorithm Prim’s Algorithm

Basic Strategy Basic Strategy Start growing a tree from a designated root vertex Start growing a tree from a designated root vertex At each step, add lightest edge linked to A that does not yield cycle At each step, add lightest edge linked to A that does not yield cycle

Implementation Every vertex in Q has key and pointer of least-weight edge to V \ Q At each step: ∞ 1. extract vertex from Q with smallest key ⇔ safe edge of cut (V \ Q, Q) 11 8 2. update keys and pointers of its neighbors in Q 0 ∞ 2 6 6 7 4 ∞ 3 4 ∞ 9 ∞ 3 9 8

5 ∞ ∞

6.3: Minimum Spanning Tree T.S. 10 6.3: Minimum Spanning Tree T.S. 10 Assign everyImplementation vertex not connected will be basedto A a onkey vertices which is! at all stages equal to the smallest weight of an edge connecting to A

Use a Priority Queue! 1126g

a h7

b63 4f

83e d95

Prim’s Algorithm Details of Prim’s Algorithm

Basic Strategy 0: def prim(G,r) 1: Apply Prim’s Algorithm to graph G and root r Start growing a tree from a designated root vertex 2: Return result implicitly by modifying G: At each step, add lightest edge linked to A that does not yield cycle 3: MST induced by the .predecessor fields 4: 5: Q = MinPriorityQueue() 6: for v in G.vertices(): 7: v.predecessor = None 8: if v == r: 9: v.key = 0 10: else: 11 8 11: v.key = Infinity 12: Q.insert(v) 13: 14: while not Q.isEmpty(): 2 7 15: u = Q.extractMin() 6 6 16: for v in u.adjacent(): 4 17: w = G.weightOfEdge(u,v) 18: if Q.hasItem(v) and w < v.key: 3 4 19: v.predecessor = u 20: Q.decreaseKey(item=v, newKey=v) 9 3 8 Time Complexity 9 We computed same MST as Kruskal, Init (l. 6-13): O(V ), ExtractMin (15): O(V ·log V ), DecreaseKey (16-20): O(E ·1) 5 but in a completely different order! ⇒ Overall: O(V log V + E) Amortized Cost

6.3: Minimum Spanning Tree T.S. 10 6.3: Minimum Spanning Tree T.S. 11

6.3: Minimum Spanning Tree T.S. 11 Outline

Introduction

Bellman-Ford Algorithm 6.4: Single-Source Shortest Paths Frank Stajano Thomas Sauerwald Dijkstra’s Algorithm

Lent 2015

6.4: Single-Source Shortest Paths T.S. 2

Shortest Path Problem Variants of Shortest Path Problems

a 5 d Single-source shortest-paths problem (SSSP) Shortest Path Problem 6 4 1 Bellman-Ford Algorithm Given: directed graph s b 4 e t Dijsktra Algorithm G = (V , E) with edge weights, 2 3 pair of vertices s, t ∈ V −2 All-pairs shortest-paths problem (APSP) Goal: Find a path of minimum 2 weight from s to t in G 2 5 Shortest Paths via Matrix Multiplication Johnson’s Algorithm 1 c f

6.4: Single-Source Shortest Paths T.S. 3 6.4: Single-Source Shortest Paths T.S. 4 a b

3 −1 h i

+∞ +∞ s c d g

0 5 11 −∞

+∞ v e f 8 j −∞ −∞

After relaxing (u, v): v.d ≤ u.d + w(u, v)

v 9

Distances and Negative-Weight Cycles Relaxing Edges

a b Deﬁnition −4 Fix the source vertex s ∈ V 3 −1 h i v.δ is the length of the shortest path (distance) from s to v 3 4 2 v.d is the length of the shortest path discovered so far +∞ +∞ s c d g 6 At the beginning: s.d = s.δ = 0, v.d = ∞ for v 6= s 5 8 0 5 11 −∞ −8 3 At the end: v.d = v.δ for all v ∈ V −3 2 7 +∞ Relaxing an edge (u, v) u v e f 2 3 Given estimates u.d and v.d, can we ﬁnd a 6 9 j −∞ −∞ better path from v using the edge (u, v)?

−6 ? s . > . + ( , ) v d u d w u v 0 Negative-Weight Cycle

6.4: Single-Source Shortest Paths T.S. 5 6.4: Single-Source Shortest Paths T.S. 6

Relaxing Edges Properties of Shortest Paths and Relaxations

Deﬁnition Toolkit Fix the source vertex s ∈ V Triangle inequality (Lemma 24.10) v.δ is the length of the shortest path (distance) from s to v For any edge (u, v) ∈ E, we have v.δ ≤ u.δ + w(u, v) v.d is the length of the shortest path discovered so far Upper-bound Property (Lemma 24.11) We always have v.d ≥ v.δ for all v ∈ V , and once v.d achieves the At the beginning: s.d = s.δ = 0, v.d = ∞ for v 6= s value v.δ, it never changes. At the end: v.d = v.δ for all v ∈ V Convergence Property (Lemma 24.14) If s u → v is a shortest path from s to v, and if u.d = u.δ prior to Relaxing an edge (u, v) u v relaxing edge (u, v), then v.d = v.δ at all times afterward. 2 Given estimates u.d and v.d, can we ﬁnd a 6 8 better path from v using the edge (u, v)? v.d ≤ u.d + w(u, v) ? s u v . > . + ( , ) s = u.δ + w(u, v) v d u d w u v 0 0 u.δ v.d = v.δ After relaxing (u, v): v.δ Since v.d ≥ v.δ, we have v.d = v.δ. v.d ≤ u.d + w(u, v)

6.4: Single-Source Shortest Paths T.S. 6 6.4: Single-Source Shortest Paths T.S. 7 vi

Path-Relaxation Property The Bellman-Ford Algorithm

BELLMAN-FORD(G,w,s) Path-Relaxation Property (Lemma 24.15) 0: assert(s in G.vertices()) 1: for v in G.vertices() If p = (v0, v1,..., vk ) is a shortest path from s = v0 to vk , and we relax 2: v.predecessor = None the edges of p in the order (v0, v1), (v1, v2),..., (vk−1, vk ), then vk .d = 3: v.d = Infinity vk .δ (regardless of the order of other relaxation steps). 4: s.d = 0 5: 6: repeat |V|-1 times 7: for e in G.edges() Proof: 8: Relax edge e=(u,v): Check if u.d + w(u,v) < v.d By induction on i, 0 ≤ i ≤ k: 9: if e.start.d + e.weight.d < e.end.d: 10: e.end.d = e.start.d + e.weight After the ith edge of p is relaxed, we have vi .d = vi .δ. 11: e.end.predecessor = e.start For i = 0, by the initialization s.d = s.δ = 0. 12: 13: for e in G.edges() Upper-bound Property ⇒ the value of s.d never changes after that. 14: if e.start.d + e.weight.d < e.end.d: 15: return FALSE Inductive Step (i − 1 → i): Assume vi−1.d = vi−1.δ and relax (vi−1, vi ). 16: return TRUE Convergence Property ⇒ vi .d = vi .δ (now and at all later steps)

Time Complexity v v0 v1 v2 i−1 vi A single call of line 9-11 costs O(1) In each pass every edge is relaxed ⇒ O(E) time per pass v0.δ v1.δ v2.δ vi−1.δ vi .δ Overall (V − 1) + 1 = V passes ⇒ O(V · E) time

6.4: Single-Source Shortest Paths T.S. 8 6.4: Single-Source Shortest Paths T.S. 9

Complete Run of Bellman-Ford (Figure 24.4) Complete Run of Bellman-Ford (Figure 24.4)

Pass: 1

Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y) Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y)

5 5 t x t x ∞ ∞ 6 ∞ −2 −2 6 6

s −3 s −3 0 8 7 0 8 7 2 2 −4 −4 7 7 ∞ ∞ 7 ∞ 9 9 y z y z

6.4: Single-Source Shortest Paths T.S. 10 6.4: Single-Source Shortest Paths T.S. 10 Complete Run of Bellman-Ford (Figure 24.4) Complete Run of Bellman-Ford (Figure 24.4)

Pass: 2 Pass: 3

Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y) Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y)

5 5 t x t x 6 4 2 4 −2 −2 6 6

s −3 s −3 0 8 7 0 8 7 2 2 −4 −4 7 7 7 2 7 2 9 9 y z y z

6.4: Single-Source Shortest Paths T.S. 10 6.4: Single-Source Shortest Paths T.S. 10

Complete Run of Bellman-Ford (Figure 24.4) The Bellman-Ford Algorithm: Correctness (1/2)

Pass: 4 Lemma 24.2/Theorem 24.3 Assume that G contains no negative-weight cycles that are reachable Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y) from s. Then after |V | − 1 passes, we have v.d = v.δ for all vertices v ∈ V that are reachable and Bellman-Ford returns TRUE. 5 t x Proof that v.d = v.δ 2 4 −2 Let v be a vertex reachable from s 6 Let p = (v0 = s, v1,..., vk = v) be a shortest path from s to v s −3 p is simple, hence k ≤ |V | − 1 Path-Relaxation Property ⇒ after |V | − 1 passes, v.d = v.δ 0 8 7 2 −4 Proof that Bellman-Ford returns TRUE 7 Let (u, v) ∈ E be any edge. After |V | − 1 passes: 7 2 v.d = v.δ ≤ u.δ + w(u, v) = u.d + w(u, v) 9 y z ⇒ Bellman-Ford returns TRUE

6.4: Single-Source Shortest Paths T.S. 10 6.4: Single-Source Shortest Paths T.S. 11 v v 8 9

After relaxing (u, v), regardless of whether we found a shortcut: v.d ≤ u.d + w(u, v)

The Bellman-Ford Algorithm: Correctness (2/2) Outline

Theorem 24.3 If G contains a negative-weight cycle reachable from s, then Bellman- Ford returns FALSE.

Proof:

Let c = (v0, v1,..., vk = v0) be a negative-weight cycle reachable from s Introduction If Bellman-Ford returns TRUE, then for every 1 ≤ i < k,

vi .d ≤ vi−1.d + w(vi−1, vi ) Bellman-Ford Algorithm k k k X X X ⇒ vi .d ≤ vi−1.d + w(vi−1, vi ) i=1 i=1 i=1 k X Dijkstra’s Algorithm 0 ≤ w(vi−1, vi ) i=1 This contradicts the assumption that c is a negative-weight cycle!

6.4: Single-Source Shortest Paths T.S. 12 6.4: Single-Source Shortest Paths T.S. 13

Relaxing Edges Relaxing Edges

Deﬁnition Deﬁnition Fix the source vertex s ∈ V Fix the source vertex s ∈ V v.δ is the length of the shortest path (distance) from s to v v.δ is the length of the shortest path (distance) from s to v v.d is the length of the shortest path discovered so far v.d is the length of the shortest path discovered so far

At the beginning: s.d = s.δ = 0, v.d = ∞ for v 6= s At the beginning: s.d = s.δ = 0, v.d = ∞ for v 6= s At the end: v.d = v.δ for all v ∈ V At the end: v.d = v.δ for all v ∈ V

Relaxing an edge (u, v) u v Relaxing an edge (u, v) u v 2 2 Given estimates u.d and v.d, can we ﬁnd a 6 9 Given estimates u.d and v.d, can we ﬁnd a 6 8 better path from v using the edge (u, v)? better path from v using the edge (u, v)?

? s ? s . > . + ( , ) . > . + ( , ) v d u d w u v 0 v d u d w u v 0

After relaxing (u, v), regardless of whether we found a shortcut: v.d ≤ u.d + w(u, v)

6.4: Single-Source Shortest Paths T.S. 14 6.4: Single-Source Shortest Paths T.S. 14 vi

Properties of Shortest Paths and Relaxations Path-Relaxation Property

“Propagation”: By relaxing proper edges, set of vertices with v.δ = v.d gets larger Toolkit Path-Relaxation Property (Lemma 24.15) Triangle inequality (Lemma 24.10) If p = (v0, v1,..., vk ) is a shortest path from s = v0 to vk , and we relax For any edge (u, v) ∈ E, we have v.δ ≤ u.δ + w(u, v) the edges of p in the order (v0, v1), (v1, v2),..., (vk−1, vk ), then Upper-bound Property (Lemma 24.11) vk .d = vk .δ (regardless of the order of other relaxation steps). We always have v.d ≥ v.δ for all v ∈ V , and once v.d achieves the value v.δ, it never changes. Proof: Convergence Property (Lemma 24.14) By induction on i, 0 ≤ i ≤ k: If s u → v is a shortest path from s to v, and if u.d = u.δ prior to After the ith edge of p is relaxed, we have vi .d = vi .δ. relaxing edge (u, v), then v.d = v.δ at all times afterward. For i = 0, by the initialization s.d = s.δ = 0. Upper-bound Property ⇒ the value of s.d never changes after that.

Inductive Step (i − 1 → i): Assume vi−1.d = vi−1.δ and relax (vi−1, vi ). v.d ≤ u.d + w(u, v) Convergence Property ⇒ vi .d = vi .δ (now and at all later steps) u v s = u.δ + w(u, v) 0 u.δ v.d = v.δ v0 v1 v2 vi−1 vi v.δ Since v.d ≥ v.δ, we have v.d = v.δ. v0.δ v1.δ v2.δ vi−1.δ vi .δ

6.4: Single-Source Shortest Paths T.S. 15 6.4: Single-Source Shortest Paths T.S. 16

The Bellman-Ford Algorithm Complete Run of Bellman-Ford (Figure 24.4)

BELLMAN-FORD(G,w,s) 0: assert(s in G.vertices()) 1: for v in G.vertices() 2: v.predecessor = None Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y) 3: v.d = Infinity 4: s.d = 0 5: 5 6: repeat |V|-1 times t x 7: for e in G.edges() 8: Relax edge e=(u,v): Check if u.d + w(u,v) < v.d 9: if e.start.d + e.weight.d < e.end.d: ∞ ∞ 10: e.end.d = e.start.d + e.weight −2 11: e.end.predecessor = e.start 6 12: 13: for e in G.edges() s −3 14: if e.start.d + e.weight.d < e.end.d: 15: return FALSE 0 8 7 16: return TRUE 2 −4

Time Complexity 7 A single call of line 9-11 costs O(1) ∞ ∞ In each pass every edge is relaxed ⇒ O(E) time per pass 9 Overall (V − 1) + 1 = V passes ⇒ O(V · E) time y z

6.4: Single-Source Shortest Paths T.S. 17 6.4: Single-Source Shortest Paths T.S. 18 Complete Run of Bellman-Ford (Figure 24.4) Complete Run of Bellman-Ford (Figure 24.4)

Pass: 1 Pass: 2

Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y) Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y)

5 5 t x t x

6 ∞ 6 4 −2 −2 6 6

s −3 s −3 0 8 7 0 8 7 2 2 −4 −4

7 7 7 ∞ 7 2 9 9 y z y z

6.4: Single-Source Shortest Paths T.S. 18 6.4: Single-Source Shortest Paths T.S. 18

Complete Run of Bellman-Ford (Figure 24.4) Complete Run of Bellman-Ford (Figure 24.4)

Pass: 3 Pass: 4

Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y) Relaxation Order: (t,x),(t,y),(t,z),(x,t),(y,x),(y,z),(z,x),(z,s),(s,t),(s,y)

5 5 t x t x

2 4 2 4 −2 −2 6 6

s −3 s −3 0 8 7 0 8 7 2 2 −4 −4

7 7 7 2 7 −2 9 9 y z y z

6.4: Single-Source Shortest Paths T.S. 18 6.4: Single-Source Shortest Paths T.S. 18 3. Relax all edges leaving v

9 4

DIJKSTRA(G,w,s) 0: INITIALIZE(G,s) 1: S = ∅ 2: Q = V 3: while Q 6= ∅ do 4: u = Extract-Min(Q) 5: S = S ∪ {u} 6: for each v ∈ G.Adj[u] do 7: RELAX(u, v, w) 8: end for 9: end while

Bellman-Ford Algorithm: Correctness (1/2) Bellman-Ford Algorithm: Correctness (2/2)

Theorem 24.3 Lemma 24.2/Theorem 24.3 If G contains a negative-weight cycle reachable from s, then Bellman- Assume that G contains no negative-weight cycles that are reachable Ford returns FALSE. from s. Then after |V | − 1 passes, we have v.d = v.δ for all vertices v ∈ V that are reachable and Bellman-Ford returns TRUE. Proof: Let c = (v , v ,..., v = v ) be a negative-weight cycle reachable from s Proof that v.d = v.δ 0 1 k 0 If Bellman-Ford returns TRUE, then for every 1 ≤ i < k, Let v be a vertex reachable from s

Let p = (v0 = s, v1,..., vk = v) be a shortest path from s to v vi .d ≤ vi−1.d + w(vi−1, vi ) k k k p is simple, hence k ≤ |V | − 1 X X X ⇒ v .d ≤ v .d + w(v , v ) Path-Relaxation Property ⇒ after |V | − 1 passes, v.d = v.δ i i−1 i−1 i i=1 i=1 i=1 k Proof that Bellman-Ford returns TRUE X 0 ≤ w(vi−1, vi ) Need to prove: v.d ≤ u.d + w(u, v) for all edges i=1 Let (u, v) ∈ E be any edge. After |V | − 1 passes: This contradicts the assumption that c is a negative-weight cycle! v.d = v.δ ≤ u.δ + w(u, v) = u.d + w(u, v)

Triangle inequality (holds even if w(u, v) < 0!)

6.4: Single-Source Shortest Paths T.S. 19 6.4: Single-Source Shortest Paths T.S. 20

Outline Dijkstra’s Algorithm

Overview of Dijkstra Requires that all edges have non-negative weights Use a special order for relaxing edges The order follows a greedy-strategy (similar to Prim’s algorithm): Introduction 1. Maintain set S of vertices u with u.δ = v.d 2. At each step, add a vertex v ∈ V \ S with minimal v.δ

Bellman-Ford Algorithm

1 6 2 5 ∞ 3 1 0 ∞ Dijkstra’s Algorithm 5 2 3 5 1 1 0 3 4 1 s 3 ∞ 4

6.4: Single-Source Shortest Paths T.S. 21 6.4: Single-Source Shortest Paths T.S. 22 ¨¨¨¨HH ¨¨¨HH¨¨H¨¨¨H¨H¨H ¨(¨(ss,H,HH0H0(()t()t,,t,,(10,(8t8t(,),)t)10,∞10,,((8(¨x()¨x()t,t,,(14H∞,H(814∞xH¨8(x),x),,)∞,,H(1413∞,¨(Hx9¨(xy)y),),H¨,13,H¨,(H95¨(H5y)yz),,H∞(H57(zz),),,∞(7∞(z)z),),∞∞)) t x ∞8 9

s 65 0

∞5 7 ∞9 4 y z

DIJKSTRA(G,w,s) 0: INITIALIZE(G,s) 1: S = ∅ 2: Q = V 3: while Q 6= ∅ do 4: u = Extract-Min(Q) 5: S = S ∪ {u} 6: for each v ∈ G.Adj[u] do 7: RELAX(u, v, w) 8: end for 9: end while ¨¨¨¨HH ¨¨¨HH¨¨H¨¨¨H¨H¨H ¨(¨(ss,H,HH0H0(()t()t,,t,,(10,(8t8t(,),)t)10,∞10,,((8(¨x()¨x()t,t,,(14H∞,H(814∞xH¨8(x),x),,)∞,,H(1413∞,¨(Hx9¨(xy)y),),H¨,13,H¨,(H95¨(H5y)yz),,H∞(H57(zz),),,∞(7∞(z)z),),∞∞)) t x ∞8 9

s 0

5 7 y z

Dijkstra’s Algorithm Complete Run of Dijkstra (Figure 24.6)

Overview of Dijkstra Requires that all edges have non-negative weights Priority Queue Q: Use a special order for relaxing edges (s, 0), (t, ∞), (x, ∞), (y, ∞), (z, ∞) The order follows a greedy-strategy (similar to Prim’s algorithm): t x 1. Maintain set S of vertices u with u.δ = v.d 1 2. At each step, add a vertex v ∈ V \ S with minimal v.δ ∞ ∞ 3. Relax all edges leaving v 10

s 9 1 5 2 0 2 3 4 6 5 ∞ 3 7 1 0 ∞ 5 2 1 3 5 5 1 0 3 4 1 ∞ ∞ s 3 9 2 4 y z

6.4: Single-Source Shortest Paths T.S. 22 6.4: Single-Source Shortest Paths T.S. 23

Complete Run of Dijkstra (Figure 24.6) Runtime of Dijkstra’s Algorithm

Priority Queue Q: ¨¨H ¨(x,HH9) t x 1 DIJKSTRA(G,w,s) Runtime (using Fibonacci Heaps) 8 9 0: INITIALIZE(G,s) Initialization (l. 0-2): O(V ) 10 1: S = ∅ 2: Q = V ExtractMin (l. 4): O(V · log V ) s 3: while Q 6= ∅ do 9 4: u = Extract-Min(Q) DecreaseKey (l. 7): O(E · 1) 0 2 3 4 6 5: S = S ∪ {u} ⇒ Overall: O(V log V + E) 7 6: for each v ∈ G.Adj[u] do 7: RELAX(u, v, w) 8: end for 5 9: end while 5 7 2 y z

6.4: Single-Source Shortest Paths T.S. 23 6.4: Single-Source Shortest Paths T.S. 24 There are edge cases like There are edge cases like s = x and/or y = u! s = x and/or y = u!

u.δ< ≤ y.d = y.δ u.δ< ≤ y.d = y.δ

since x.d = x.δ when x is extracted, and then since x.d = x.δ when x is extracted, and then u is extracted before y u is extracted before y (x, y) is relaxed ⇒ Convergence Property (x, y) is relaxed ⇒ Convergence Property

x y Take a shortest path from s to u and let (x, y) be the ﬁrst edge from S to V \ S ⇒ ⇒ u.d u.d

This contradicts that y is on a shortest path This contradicts that y is on a shortest path from s to u. from s to u.

There are edge cases like There are edge cases like s = x and/or y = u! s = x and/or y = u!

u.δ< = y.δ u.δ<

This contradictssince x.d = thatx.δywhenis on ax shortestis extracted, path and then This contradicts that y is on a shortest path u is extracted before y from s(xto, yu).is relaxed ⇒ Convergence Property from s to u.

Correctness of Dijkstra’s Algorithm Correctness of Dijkstra’s Algorithm

Theorem 24.6 Theorem 24.6 For any directed graph G = (V , E, w) with non-negative edge weights For any directed graph G = (V , E, w) with non-negative edge weights and source s, Dijkstra terminates with u.d = u.δ for all u ∈ V . and source s, Dijkstra terminates with u.d = u.δ for all u ∈ V .

Proof: Invariant: If v is extracted, v.d = v.δ Proof: Invariant: If v is extracted, v.d = v.δ Suppose there is u ∈ V , when extracted, Suppose there is u ∈ V , when extracted,

u.d > u.δ u u.d > u.δ u s s Let u be the first vertex with this property Let u be the first vertex with this property x y Take a shortest path from s to u and let (x, y) be the first edge from S to V \ S S S

6.4: Single-Source Shortest Paths T.S. 25 6.4: Single-Source Shortest Paths T.S. 25

Correctness of Dijkstra’s Algorithm Correctness of Dijkstra’s Algorithm

Proof: Invariant: If v is extracted, v.d = v.δ Proof: Invariant: If v is extracted, v.d = v.δ Suppose there is u ∈ V , when extracted, Suppose there is u ∈ V , when extracted,

u.d > u.δ u u.d > u.δ u s s Let u be the first vertex with this property x y Let u be the first vertex with this property x y Take a shortest path from s to u and let Take a shortest path from s to u and let (x, y) be the first edge from S to V \ S (x, y) be the first edge from S to V \ S ⇒ S ⇒ S u.d ≤ y.d u.d ≤ y.d = y.δ

since x.d = x.δ when x is extracted, and then u is extracted before y (x, y) is relaxed ⇒ Convergence Property

6.4: Single-Source Shortest Paths T.S. 25 6.4: Single-Source Shortest Paths T.S. 25 since x.d = x.δ when x is extracted, and then u is extracted before y (x, y) is relaxed ⇒ Convergence Property

Correctness of Dijkstra’s Algorithm

Theorem 24.6 For any directed graph G = (V , E, w) with non-negative edge weights and source s, Dijkstra terminates with u.d = u.δ for all u ∈ V . There are edge cases like s = x and/or y = u! Proof: Invariant: If v is extracted, v.d = v.δ Suppose there is u ∈ V , when extracted,

u.d > u.δ u s Let u be the ﬁrst vertex with this property x y Take a shortest path from s to u and let (x, y) be the ﬁrst edge from S to V \ S ⇒ S u.δ< u.d ≤ y.d = y.δ

This contradicts that y is on a shortest path from s to u.

6.4: Single-Source Shortest Paths T.S. 25 Outline

All-Pairs Shortest Path

APSP via Matrix Multiplication 6.5: All-Pairs Shortest Paths Frank Stajano Thomas Sauerwald Johnson’s Algorithm

Lent 2015

6.5: All-Pairs Shortest Paths T.S. 2

Formalizing the Problem Outline

All-Pairs Shortest Path Problem Given: directed graph G = (V , E), V = {1, 2,..., n}, with edge weights represented by a matrix W : All-Pairs Shortest Path  weight of edge (i, j) for an edge (i, j) ∈ E,  wi,j = ∞ if there is no edge from i to j,  0 if i = j. APSP via Matrix Multiplication

Goal: Obtain a matrix of shortest path weights L, that is ( weight of a shortest path from i to j, if j is reachable from i Johnson’s Algorithm li,j = ∞ otherwise.

Here we will only compute the weight of the shortest path without keeping track of the edges of the path!

6.5: All-Pairs Shortest Paths T.S. 3 6.5: All-Pairs Shortest Paths T.S. 4 A Recursive Approach Example of Shortest Path via Matrix Multiplication (Figure 25.1)

j i k 2 3 4 8 Basic Idea 1 3 1 Any shortest path from i to j of length k ≥ 2 is the concatenation of 2 −4 7 −5 a shortest path of length k − 1 and an edge `(2) = min{0 + ∞, 3 + 1, 8 + ∞, ∞ + 0, − 4 + 6} 5 4 1,4 6

    Let `(m) be min. weight of any path from i to j with at most m edges 0 3 8 ∞ −4 0 3 8 2 −4 i,j ∞ 0 ∞ 1 7 3 0 −4 1 7 ( )   (2)   (1) (1) L 1 = W =  ∞ 4 0 ∞ ∞  L =  ∞ 4 0 5 11  Then ` = wi,j , so L = W     i,j  2 ∞ −5 0 ∞   2 −1 −5 0 −2  How can we obtain L(2) from L(1)? ∞ ∞ ∞ 6 0 8 ∞ 1 6 0 (2) (1) (1)  − −   0 1 −3 2 −4  ì,j = min ì,j , min ì,k + wk,j 0 3 3 2 4 1≤k≤n Recall that w , = 0! 3 0 −4 1 −1 j j  3 0 −4 1 −1    (3)   (4)   L =  7 4 0 5 11  L =  7 4 0 5 3   2 −1 −5 0 −2    2 −1 −5 0 −2 (m) (m−1) (m−1) (m−1) 8 5 1 6 0 8 5 1 6 0 ì,j = min ì,j , min ì,k + wk,j = min ì,k + wk,j 1≤k≤n 1≤k≤n (4) `3,5 = min{7 − 4, 4 + 7, 0 + ∞, 5 + ∞, 11 + 0}

6.5: All-Pairs Shortest Paths T.S. 5 6.5: All-Pairs Shortest Paths T.S. 6

Computing L(m) Computing L(n−1) efﬁciently

(m) (m−1) ì,j = min ì,k + wk,j 1≤k≤n (m) (m−1) ì,j = min ì,k + wk,j 1≤k≤n L(n−1) = L(n) = L(n+1) = ... = L, since every shortest path uses at most n − 1 = |V | − 1 edges (assuming absence of negative-weight cycles) 3 4 Computing L(m): Takes O(n · n ) = O(n ) (m) (m) (m−1) L can be ì,j = min ì,k + wk,j 3 For, say, n = 738, we subsequently compute 1≤k≤n computed in O(n ) (1) (2) (3) (4) (737) (m−1) X (m−1) L , L , L , L ,..., L = L (L · W )i,j = ì,k × wk,j 1≤k≤n Since we don’t need the intermediate matrices, a more efficient way is The correspondence is as follows: X L(1), L(2), L(4),..., L(512), L(1024) = L min ⇔ + ⇔× ∞ ⇔ 0 We need L(4) = L(2) ·L(2) = L(3) ·L(1)! (see Ex. 25.1-4) Takes O(log n · n3). 0 ⇔ 1

6.5: All-Pairs Shortest Paths T.S. 7 6.5: All-Pairs Shortest Paths T.S. 8 k X = (w(vi−1, vi ) + vi−1.δ − vi .δ) = w(p) + v0.δ − vk .δ i=1

Pk In the original graph, the weight is i=1 w(vi−1, vi ). In the reweighted graph, the weight is

k X we(vi−1, vi ) i=1

Let p = (v0, v1,..., vk ) be any path

Outline Johnson’s Algorithm

Overview allow negative-weight edges and negative-weight cycles one pass of Bellman-Ford and |V | passes of Dijkstra after Bellman-Ford, edges are reweighted s.t. all edge weights are non-negative All-Pairs Shortest Path shortest paths are maintained Adding a constant to every edge doesn’t work!

APSP via Matrix Multiplication

3 4 1003 1004

Johnson’s Algorithm 8 1008 2 1002

−4 1 −5 996 1001 995 7 1007

6 1006

6.5: All-Pairs Shortest Paths T.S. 9 6.5: All-Pairs Shortest Paths T.S. 10

How Johnson’s Algorithm works Correctness of Johnson’s Algorithm

Johnson’s Algorithm 1. Add a new vertex s and directed edges (s, v), v ∈ V , with weight 0 w(u, v) = w(u, v) + u.δ − v.δ 2. Run Bellman-Ford on this augmented graph with source s e If there are negative weight cycles, abort Otherwise: Theorem 1) Reweight every edge (u, v) by we(u, v) = w(u, v) + u.δ − v.δ 2) Remove vertex s and its incident edges For any graph G = (V , E, w) without negative-weight cycles: 3. For every vertex v ∈ V , run Dijkstra on (G, E, we) 1. After reweighting, all edges are non-negative Runtime: O(V ·E +V ·(V log V +E)) 2. Shortest Paths are preserved 0

Proof of 1. Let u.δ and v.δ be the distances from the fake source s 0 −1 3 4 4 0 u.δ + w(u, v) ≥ v.δ (triangle inequality) 0 8 13 0 −5 ⇒ we(u, v) + u.δ + w(u, v) ≥ w(u, v) + u.δ − v.δ + v.δ s 2 2 0 ⇒ we(u, v) ≥ 0 −4 1 −5 0 0 0 7 10 −4 0 0 6 2 Direct: 7, Detour: −1 Direct: 10, Detour: 2

6.5: All-Pairs Shortest Paths T.S. 11 6.5: All-Pairs Shortest Paths T.S. 12 Correctness of Johnson’s Algorithm Comparison of all Shortest-Path Algorithms

we(u, v) = w(u, v) + u.δ − v.δ SSSP APSP negative Algorithm Theorem sparse dense sparse dense weights For any graph G = (V , E, w) without negative-weight cycles: 2 3 3 4 Bellman-Ford V V V V X 1. After reweighting, all edges are non-negative 2 2 3 2. Shortest Paths are preserved Dijkstra V log V V V log V V X 3 3 Matrix Mult. – – V log V V log V (X) Proof of 2. 2 3 Johnson – – V log V V X Let p = (v0, v1,..., vk ) be any path Pk In the original graph, the weight is i=1 w(vi−1, vi ). In the reweighted graph, the weight is

k k X X we(vi−1, vi ) = (w(vi−1, vi ) + vi−1.δ − vi .δ) = w(p) + v0.δ − vk .δ i=1 i=1

6.5: All-Pairs Shortest Paths T.S. 12 6.5: All-Pairs Shortest Paths T.S. 13 How to ﬁnd a Maximum Flow? 8/95/9

2/155/15 5/108/10 10/10 0/4 0/15 5/5 8/85/8 10/10 6 13/1510/15 0/63/6 0/4 0/15 10/10 10/3013/30 |f | = 85 + 10 + 10 = 2825

Outline

Introduction

Ford-Fulkerson

Max-Flow Min-Cut Theorem 6.6: Maximum ﬂow Frank Stajano Thomas Sauerwald Analysis of Ford-Fulkerson

Matchings in Bipartite Graphs

Lent 2015

6.6: Maximum ﬂow T.S. 2

History of the Maximum Flow Problem [Harris, Ross (1955)] Flow Network

Flow Network Abstraction for material (one commodity!) ﬂowing through the edges G = (V , E) directed graph without parallel edges distinguished nodes: source s and sink t

za every edge e has a capacity c(e)

+ SECRET Capacity function c : V × V → R c(u, v) = 0 ⇔ (u, v) 6∈ E 3 a) Maximum Flow is 163,000 tons per day! 9 2 5 On, 15 10 10

LL,, 4 15 cCfl (a _Z ~

oS 5 8 10 s 3 6 t 70 21 Z 0 0 0! E

w 0 15 0>a 6 a_ R -C = Of ~0a

a 10 'ma In cc -05 4 15 0 0 7- 0 4' 16 4 7

6.6: Maximum flow T.S. 3 6.6: Maximum flow T.S. 4 How to find a Maximum Flow? How to find a Maximum Flow?

8/99 8/99

2/1515 8/1010 2/1515 8/1010 10/1010 0/44 0/1515 10/1010 0/44 0/1515 5/55 8/88 10/1010 5/55 8/88 10/1010 6 6 13/15 13/15 15 3/66 15 3/66 0/44 0/1515 10/1010 0/44 0/1515 10/1010 13/3016 13/3016 |f | = 8 + 10 + 10 = 28 |f | = 8 + 10 + 10 = 28

0/44/4 5/99 ||ff||==1016198

5/1515 5/1010 10/1010 0/44 0/1515 0/10GreedyIs did this not optimal? succeed! ¨0/8¨ 6/109/10 5/55 5/88 10/1010 10/100/10 2/20/2 ¨0/2¨ 8/86/8 6/65/6 ¨0/6¨ 6 8/10 10/15 15 0/66 0/44 0/1515 10/1010 0/10 ¨0/98/99/9¨ 10/100/10 10/30 16 6/109/10 2/9¨ 8/10 |f | = 5 + 10 + 10 = 25 ¨

Flow Network Flow Network

Flow Flow A flow is a function f : V × V → R that satisfies: A flow is a function f : V × V → R that satisfies: For every u, v ∈ V , f (u, v) ≤ c(u, v) Flow Conservation For every u, v ∈ V , f (u, v) ≤ c(u, v) P P P P For every v ∈ V \{s, t}, (u,v)∈E f (u, v) = (v,u)∈E f (v, u) For every v ∈ V \{s, t}, (u,v)∈E f (u, v) = (v,u)∈E f (v, u) f (u, v) = −f (v, u) f (u, v) = −f (v, u) P P The value of a flow is defined as |f | = v∈V f (s, v) The value of a flow is defined as |f | = v∈V f (s, v) P P v∈V f (s, v) = v∈V f (v, t)

5/9 5/9 2 5 2 5 5/15 5/10 5/15 5/10 10/10 0/4 0/15 10/10 0/4 0/15 5/5 5/8 10/10 5/5 5/8 10/10 s 3 6 t s 3 6 t 10/15 0/6 10/15 0/6 0/4 0/15 10/10 0/4 0/15 10/10 10/30 10/30 4 7 |f | = 5 + 10 + 10 = 25 4 7 |f | = 5 + 10 + 10 = 25

6.6: Maximum ﬂow T.S. 4 6.6: Maximum ﬂow T.S. 4

Flow Network A First Attempt

Flow Greedy Algorithm : × → A flow is a function f V V R that satisfies: Start with f (u, v) = 0 everywhere For every u, v ∈ V , f (u, v) ≤ c(u, v) Repeat as long as possible: P P ( , ) = ( , ) ( , ) < ( , ) For every v ∈ V \{s, t}, (u,v)∈E f (u, v) = (v,u)∈E f (v, u) Find a s t -path p where each edge e u v has f u v c u v Augment flow along p f (u, v) = −f (v, u) P The value of a flow is defined as |f | = v∈V f (s, v) How to find a Maximum Flow? 0/4 8/9 2 4 |f | = 0 2 5 2/15 8/10 10/10 0/4 0/15 0/8 0/10 5/5 8/8 10/10 0/10 0/2 0/6 s 3 6 t 13/15 3/6 0/4 0/15 10/10 0/10 0/9 0/10 13/30 s 3 5 t 4 7 |f | = 8 + 10 + 10 = 28

6.6: Maximum ﬂow T.S. 4 6.6: Maximum ﬂow T.S. 5 0/44/4 0/44/4 ||ff||==1016190 ||ff||==161908

0/10GreedyIs did this not optimal? succeed! 0/10GreedyIs did this not optimal? succeed! 0/8 ¨0/8¨ 6/109/10 6/109/10 10/100/10 2/20/2 ¨0/2¨ 8/86/8 6/65/6 ¨0/6¨ 10/100/10 2/20/2 0/2 8/86/8 6/65/6 ¨0/6¨ 8/10 8/10

0/10 ¨0/98/99/9¨ 10/100/10 0/10 0/98/99/9 10/100/10 6/109/10 ¨2/9¨ 8/10 6/109/10 ¨2/9¨ 8/10

0/44/4 0/4 ||ff||==101908 ||ff||==101608

0/10Greedy did not succeed! 0/10 Is this optimal? ¨0/8¨ ¨0/8¨ 9/10 6/10 10/100/10 2/20/2 ¨0/2¨ 8/86/8 5/6 0/6 10/100/10 2/2 ¨0/2¨ 8/8 6/6 ¨0/6¨ 8/10 8/10

0/10 ¨0/99/9¨ 10/100/10 0/10 ¨0/98/9¨ 10/100/10 9/10 2/9 8/10 6/10 ¨2/9¨ 8/10

A First Attempt A First Attempt

Greedy Algorithm Greedy Algorithm Start with f (u, v) = 0 everywhere Start with f (u, v) = 0 everywhere Repeat as long as possible: Repeat as long as possible: Find a (s, t)-path p where each edge e = (u, v) has f (u, v) < c(u, v) Find a (s, t)-path p where each edge e = (u, v) has f (u, v) < c(u, v) Augment ﬂow along p Augment ﬂow along p

0/4 0/4 2 4 |f | = 8 2 4 |f | = 10

0/10 0/10 ¨0/8¨ 0/10 0/2 8/8 0/6 10/10 2/2 ¨0/2¨ 8/8 0/6 8/10 8/10

0/10 0/9 0/10 0/10 ¨0/9¨ 10/10 s 3 5 t s 3 5 t 8/10 2/9 8/10

6.6: Maximum ﬂow T.S. 5 6.6: Maximum ﬂow T.S. 5

A First Attempt A First Attempt

0/4 4/4 2 4 |f | = 16 2 4 |f | = 19

0/10 Is this optimal? Greedy did not succeed! 6/10 9/10 10/10 2/2 8/8 6/6 ¨0/6¨ 10/10 0/2 6/8 5/6

0/10 8/9 10/10 9/9 10/10 s 3 5 t s 3 5 t 6/10 ¨2/9¨ 9/10

6.6: Maximum flow T.S. 5 6.6: Maximum flow T.S. 5 ¨3/40/4¨ ||ff||==101618198 ¨2/4¨ 0/108/10 ¨7/80/8¨ 6/109/10 Is this a max-flow? 10/100/10 ¨2/2¨ ¨0/2¨¨8/8¨ 6/6 ¨0/6¨ 8/10 0/108/10 ¨0/98/9¨ 10/100/10 9/106/10 ¨2/99/9¨ 8/10

23 4 21

7 689 8 8 10 10 1 142 2 2 6 689 28 8 10 9 10 421 71 2

Outline Residual Graph

Original Edge Edge e = (u, v) ∈ E Introduction Graph G: ﬂow f (u, v) and capacity c(u, v) 6/17 Ford-Fulkerson u v Residual Capacity  c(u, v) − f (u, v) if (u, v) ∈ E , Max-Flow Min-Cut Theorem  cf (u, v) = f (v, u) if (v, u) ∈ E , 0 otherwise.

Analysis of Ford-Fulkerson Residual Gf : 11 Residual Graph Matchings in Bipartite Graphs u v Gf = (V , Ef , cf ), Ef := {(u, v): cf (u, v) > 0} 6

6.6: Maximum ﬂow T.S. 6 6.6: Maximum ﬂow T.S. 7

The Ford-Fulkerson Method (“Enhanced Greedy”) Illustration of the Ford-Fulkerson Method

Graph G = (V , E, c): 0/4 0: def fordFulkerson(G) 2 4 |f | = 0 1: initialize flow to 0 on all edges 2: while an augmenting path in Gf can be found: 0/8 0/10 3: push as much extra flow as possible through it 0/10 0/2 0/6

0 Augmenting path: Path If f is a flow in Gf and f a flow 0/10 0/9 0/10 0 s from source to sink in Gf in G, then f + f is a flow in G 3 5 t

Using BFS or DFS, we can ﬁnd an augmenting path in O(V + E) time. Residual Graph Gf = (V , Ef , cf ): Questions: 4 2 4 How to ﬁnd an augmenting path? Does this method terminate? 8 10 10 2 6 If it terminates, how good is the solution? 10 9 10 s 3 5 t

6.6: Maximum flow T.S. 8 6.6: Maximum flow T.S. 9 ¨3/40/4¨ ¨0/43/4¨ ||ff||==101618190 ||ff||==16181908 ¨2/4¨ ¨2/4¨ 0/108/10 0/108/10 7/80/8 ¨0/87/8¨ 6/109/10 Is this a max-flow? 6/109/10 Is this a max-flow? 10/100/10 ¨2/2¨ ¨0/2¨¨8/8¨ 6/6 ¨0/6¨ 10/100/10 ¨2/2¨ 0/2 ¨8/8¨ 6/6 ¨0/6¨ 8/10 8/10 0/108/10 ¨0/98/9¨ 10/100/10 0/108/10 ¨0/98/9¨ 10/100/10 9/106/10 ¨2/99/9¨ 8/10 6/109/10 ¨2/99/9¨ 8/10

23 23 4 4 21 21

7 689 7 689 8 10 8 8 10 10 1 142 10 1 142 2 6 2 2 6 689 28 689 8 8 10 9 10 10 9 10 421 71 421 1 2

¨0/43/4¨ 0/43/4 ||ff||==10181908 ||ff||==10161908 ¨2/4¨ ¨2/4¨ 0/108/10 0/108/10 ¨0/87/8¨ ¨0/87/8¨ 9/106/10 Is this a max-ﬂow? 6/109/10 Is this a max-ﬂow? 10/100/10 ¨2/2¨ ¨0/2¨¨8/8¨ 6/6 0/6 10/100/10 2/2 ¨0/2¨¨8/8¨ 6/6 ¨0/6¨ 8/10 8/10 0/108/10 ¨0/98/9¨ 10/100/10 0/108/10 ¨0/98/9¨ 10/100/10 9/106/10 2/99/9 8/10 6/109/10 ¨2/99/9¨ 8/10

23 3 4 4 21 1

7 89 7 69 8 8 10 8 8 10 10 1 12 10 1 14 2 2 6 2 2 6 89 28 8 69 28 8 10 9 10 10 9 10 21 71 2 41 71 2

Illustration of the Ford-Fulkerson Method Illustration of the Ford-Fulkerson Method

Graph G = (V , E, c): Graph G = (V , E, c): 0/4 0/4 2 4 |f | = 8 2 4 |f | = 10

0/10 0/10 ¨0/8¨ 0/10 0/2 8/8 0/6 10/10 2/2 ¨0/2¨ 8/8 0/6 8/10 8/10 0/10 0/9 0/10 0/10 ¨0/9¨ 10/10 s 3 5 t s 3 5 t 8/10 2/9 8/10

Residual Graph Gf = (V , Ef , cf ): Residual Graph Gf = (V , Ef , cf ): 4 4 2 4 2 4

8 8 10 8 10 10 2 2 6 2 6 8 2 10 9 10 10 s 3 5 2 t s 3 7 5 t

6.6: Maximum ﬂow T.S. 9 6.6: Maximum ﬂow T.S. 9

Illustration of the Ford-Fulkerson Method Illustration of the Ford-Fulkerson Method

Graph G = (V , E, c): Graph G = (V , E, c): 0/4 ¨0/4¨ 2 4 |f | = 16 2 4 |f | = 18 2/4 0/10 8/10 6/10 6/10 10/10 2/2 8/8 6/6 ¨0/6¨ 10/10 ¨2/2¨ 0/2 8/8 6/6

0/10 8/9 10/10 8/10 8/9 10/10 s 3 5 t s 3 5 t 6/10 ¨2/9¨ 6/10

Residual Graph Gf = (V , Ef , cf ): Residual Graph Gf = (V , Ef , cf ): 2 4 2 4 2 2 4

6 8 8 8 10 2 6 4 10 2 6 2 6 8 8 8 10 10 s 4 3 1 5 t s 2 3 1 5 t

6.6: Maximum flow T.S. 9 6.6: Maximum flow T.S. 9 ¨0/4¨ ¨0/4¨ ||ff||==10161808 ||ff||==1016181908 2/4 ¨2/4¨ 0/108/10 0/108/10 ¨0/8¨ ¨0/8¨ 6/10 6/10 Is this a max-flow? 10/100/10 ¨2/2¨ ¨0/2¨ 8/8 6/6 ¨0/6¨ 10/100/10 ¨2/2¨ ¨0/2¨¨8/8¨ 6/6 ¨0/6¨ 8/10 8/10 0/108/10 ¨0/98/9¨ 10/100/10 0/108/10 ¨0/98/9¨ 10/100/10 6/10 ¨2/9¨ 8/10 6/10 ¨2/9¨ 8/10

2 2 4 4 2 2

68 68 8 8 10 8 8 10 10 42 10 42 2 2 6 2 2 6 68 28 8 68 28 8 10 9 10 10 9 10 42 71 2 42 71 2

|f | = 16 0/4

8/8 6/10 10/10 2/2 6/6

6/10 8/9 10/10

810+ 8−−26++86==1616

Illustration of the Ford-Fulkerson Method Illustration of the Ford-Fulkerson Method

Graph G = (V , E, c): Graph G = (V , E, c): 3/4 3/4 2 4 |f | = 19 2 4 ¨2/4¨ 8/10 7/8 7/8 9/10 Is this a max-ﬂow? 9/10 10/10 0/2 ¨8/8¨ 6/6 10/10 0/2 6/6

8/10 ¨8/9¨ 10/10 10/10 s 3 5 t s 3 5 t 9/10 9/9 9/10 9/9

Residual Graph Gf = (V , Ef , cf ): Residual Graph Gf = (V , Ef , cf ): 3 3 2 1 4 2 1 4

7 9 7 9 10 2 1 6 1 10 2 1 6 1 9 9 9 10 9 10 s 1 3 5 t s 1 3 5 t

6.6: Maximum ﬂow T.S. 9 6.6: Maximum ﬂow T.S. 9

Outline From Flows to Cuts

Cut A cut (S,T) is a partition of V into S and T = V \ S such that s ∈ S and t ∈ T . Introduction The capacity of a cut (S, T ) is the sum of the capacities of the edges from S to T : X c(S, T ) = c(s, t). Ford-Fulkerson u∈S,v∈T A mininum cut of a network is a cut whose capacity is minimum over all cuts of the network. Max-Flow Min-Cut Theorem Graph G = (V , E, c): 4 Analysis of Ford-Fulkerson 2 4

8 10 10 2 6 Matchings in Bipartite Graphs 10 9 10 s 3 5 t

c({s, 3}, {2, 4, 5, t}) = 10 + 9 = 19

6.6: Maximum ﬂow T.S. 10 6.6: Maximum ﬂow T.S. 11 4 4

8 10 8 10 10 2 6 10 2 6

10 9 10 10 9 10

c({s, 3},8{+2,84,−5,6t}+)6 ==1016+ 9 = 19 c({s, 3}, {102, 4−, 52,+t}8) == 1016+ 9 = 19

8 10 10 2 6

10 9 10

c({s, 3}, {102, 4−, 52,+t}8) == 1016+ 9 = 19

From Flows to Cuts From Flows to Cuts

Flow Value Lemma (Lemma 26.4) Flow Value Lemma (Lemma 26.4) Let f be a flow with source s and sink t, and let (S, T ) be any cut of G. Let f be a flow with source s and sink t, and let (S, T ) be any cut of G. Then the value of the flow is equal to the net flow across the cut, i.e., Then the value of the flow is equal to the net flow across the cut, i.e., X X X X |f | = f (u, v) − f (v, u). |f | = f (u, v) − f (v, u). (u,v)∈E(S,T ) (v,u)∈E(T ,S) (u,v)∈E(S,T ) (v,u)∈E(T ,S)

Graph G = (V , E, c): |f | = 16 Graph G = (V , E, c): |f | = 16 0/4 0/4 2 4 2 4

8/8 6/10 8/8 6/10 10/10 2/2 6/6 10/10 2/2 6/6

s 3 5 t s 3 5 t 6/10 8/9 10/10 6/10 8/9 10/10

10 − 2 + 8 = 16 8 + 8 − 6 + 6 = 16

6.6: Maximum ﬂow T.S. 11 6.6: Maximum ﬂow T.S. 11

From Flows to Cuts Weak Duality betwen Flows and Cuts Weak Duality (Corollary 26.5) Let f be any ﬂow and (S, T ) be any cut. Then the value of f is bounded X X |f | = f (u, v) − f (v, u). from above by the capacity of the cut (S, T ), i.e., (u,v)∈E(S,T ) (v,u)∈E(T ,S) |f | ≤ c(S, T ).

X X X X |f | = f (s, w) = f (u, w) − f (w, u) X X |f | = f (u, v) − f (v, u) (s,w)∈E u∈S (u,w)∈E (w,u)∈E X X (u,v)∈E(S,T ) (v,u)∈E(T ,S) = f (u, v) − f (v, u) X ≤ f (u, v) (u,v)∈E(S,T ) (v,u)∈E(T ,S) (u,v)∈E(S,T ) X Graph G = (V , E, c): |f | = 16 ≤ c(u, v) = c(S, T ). 0/4 2 4 (u,v)∈E(S,T ) 0/4 4 8/8 6/10 2 4 2 4 6/10 10/10 2/2 6/6 8/8 8 10 10 10/10 2/2 6/6 2 6 10 9 10 s 3 5 t s 3 5 t s 3 5 t 6/10 8/9 10/10 6/10 8/9 10/10 8 + 8 − 6 + 6 = 16 |f | = 10 − 2 + 8 = 16 c(S, T )=10+9=19

6.6: Maximum ﬂow T.S. 11 6.6: Maximum ﬂow T.S. 12 X X |f | = f (u, v) − f (v, u) (u,v)∈E(S,T ) (v,u)∈E(T ,S) X = c(u, v) = c(S, T ) (u,v)∈E(S,T )

¨ 10/101/2 ¨0/2 9/10

X X |f | = f (u, v) − f (v, u) (u,v)∈E(S,T ) (v,u)∈E(T ,S) X = c(u, v) = c(S, T ) (u,v)∈E(S,T )

Max-Flow Min-Cut Theorem Key Lemma

Key Lemma (Theorem 26.6) The following three conditions are all equivalent for any ﬂow f : 1. f is a maximum ﬂow

2. There is no augmenting path in Gf 3. There exists a cut (S, T ) such that c(S, T ) = |f | Theorem The value of the max-ﬂow is equal to the capacity of the min-cut, that is Proof 1 ⇒ 2:

max |f | = min c(S, T ). For the sake of contradicion, suppose there is an augmenting path with f S,T ⊆V respect to f . Then we can improve f by increasing the ﬂow along this path. Hence f cannot be a maximum ﬂow.

0/4 4 G 2 4 Gf 2 4 6/10 6 8/8 8 10 4 10/10 2/2 6/6 2 6 6 8 10 s 3 5 t s 4 3 1 5 t 6/10 8/9 10/10

6.6: Maximum ﬂow T.S. 13 6.6: Maximum ﬂow T.S. 14

Key Lemma Key Lemma

Key Lemma (Theorem 26.6) Key Lemma (Theorem 26.6) The following three conditions are all equivalent for any flow f : The following three conditions are all equivalent for any flow f : 1. f is a maximum flow 1. f is a maximum flow

2. There is no augmenting path in Gf 2. There is no augmenting path in Gf 3. There exists a cut (S, T ) such that c(S, T ) = |f | 3. There exists a cut (S, T ) such that c(S, T ) = |f |

Proof 2 ⇒ 3: Proof 2 ⇒ 3: Let f be a ﬂow with no augmenting paths. Let f be a ﬂow with no augmenting paths.

Let S be the nodes reachable from s in Gf , T := V \ S ⇒ s ∈ S, t 6∈ S. Let S be the nodes reachable from s in Gf , T := V \ S ⇒ s ∈ S, t 6∈ S. (u, v) ∈ E(S, T ) ⇒ f (u, v) = c(u, v). (u, v) ∈ E(S, T ) ⇒ f (u, v) = c(u, v). (v, u) ∈ E(T , S) ⇒ f (v, u) = 0. (v, u) ∈ E(T , S) ⇒ f (v, u) = 0.

3 X X 3/4 |f | = f (u, v) − f (v, u) G 2 4 Gf 2 1 4 7/8 9/10 7 9 (u,v)∈E(S,T ) (v,u)∈E(T ,S) 0/2 6/6 10 2 1 6 1 X 10/10 9 = c(u, v) = c(S, T ) 9 10 s 3 5 t s 1 3 5 t (u,v)∈E(S,T ) 9/10 9/9 10/10

6.6: Maximum ﬂow T.S. 14 6.6: Maximum ﬂow T.S. 14 |ef | ≤ c(S, T ) = |f |.

X X |f | = f (u, v) − f (v, u) (u,v)∈E(S,T ) (v,u)∈E(T ,S) X = c(u, v) = c(S, T ) (u,v)∈E(S,T )

Key Lemma Proof of the Max-Flow Min-Cut Theorem

Key Lemma Key Lemma (Theorem 26.6) The following conditions are equivalent for any flow f : The following three conditions are all equivalent for any flow f : 1. There exists a cut (S, T ) such that c(S, T ) = |f | 1. f is a maximum flow 2. f is a maximum flow 2. There is no augmenting path in Gf 3. There is no augmenting path in G . 3. There exists a cut (S, T ) such that c(S, T ) = |f | f

Proof 3 ⇒ 1: Theorem The value of the max-flow is equal to the capacity of the min-cut, that is Suppose that (S, T ) is a cut with c(S, T ) = |f | max |f | = min c(S, T ). By Corollary 26.5, for any flow ef , f S,T ⊆V Hence f is a maximum flow. Proof of “≤”: 3/4 4 G 2 4 Gf 2 4 For any flow f and cut (S, T ), |f | ≤ c(S, T ) (Corollary 26.5) 9/10 7/8 8 10 ⇒ 10 10/10 0/2 6/6 2 6 max |f | ≤ min c(S, T ) 10 9 10 f S,T ⊆V s 3 5 t s 3 5 t 9/10 9/9 10/10

6.6: Maximum ﬂow T.S. 14 6.6: Maximum ﬂow T.S. 15

Proof of the Max-Flow Min-Cut Theorem Outline

Key Lemma The following conditions are equivalent for any ﬂow f : 1. There exists a cut (S, T ) such that c(S, T ) = |f | 2. f is a maximum ﬂow Introduction

3. There is no augmenting path in Gf . Ford-Fulkerson Theorem The value of the max-ﬂow is equal to the capacity of the min-cut, that is Max-Flow Min-Cut Theorem max |f | = min c(S, T ). f S,T ⊆V

Proof of “≥”: Analysis of Ford-Fulkerson

Let fmax be a maximum ﬂow

6.6: Maximum ﬂow T.S. 15 6.6: Maximum ﬂow T.S. 16 1230 /1000 1000 1000 999998997 123/ 999998997 312 123 01/1 1 1230 /1000 1000999998997 1000 123/ 999998997 123 123

Number of iterations is C := maxu,v c(u, v)!

For irrational capacities, Ford-Fulkerson may even fail to terminate!

1230 120 /1000 1000 /1000 1000 1000 999998997 1000 999998 2301/ 1000999998997 1230/ 1000999998997 312 123 312 12 10/1 1 01/1 1 1230 120 /1000 1000999998997 /1000 1000999998 1000 1000 2301/ 1000999998997 1230/ 1000999998997 123 123 12 123

Number of iterations is C := maxu,v c(u, v)!

For irrational capacities, Ford-Fulkerson may even fail to terminate!

Analysis of Ford-Fulkerson Slow Convergence of Ford-Fulkerson (Figure 26.7)

0: def FordFulkerson(G) 1: initialize flow to 0 on all edges u 0 u 2: /1000 1000 while an augmenting path in Gf can be found: 1000 3: push as much extra flow as possible through it 0/ 1000 s 0/1 t s 1 t 0 /1000 1000 Lemma 1000 v 0/ v 1000 If all capacities c(u, v) are integral, then the ﬂow at every iteration of Ford-Fulkerson is integral. G Gf Flow before iteration integral & capacities in Gf are integral ⇒ Flow after iteration integeral Theorem For integral capacities c(u, v), Ford-Fulkerson terminates after C := maxu,v c(u, v) iterations and returns the maximum ﬂow.

at the time of termination, no augmenting path ⇒ Ford-Fulkerson returns maxﬂow (Key Lemma)

6.6: Maximum ﬂow T.S. 17 6.6: Maximum ﬂow T.S. 18

Slow Convergence of Ford-Fulkerson (Figure 26.7) Slow Convergence of Ford-Fulkerson (Figure 26.7)

u 0 u u 3 u /1000 1000 /1000 1000 1000 997 1/ 999 3/ 997 1 3 3 s 1/1 t s 1 t s 0/1 t s 1 t 0 3 /1000 1000 /1000 997 1000 1000 1/ 999 3/ 997 v v 1 v 3 v 3

G Gf G Gf

Number of iterations is C := maxu,v c(u, v)!

For irrational capacities, Ford-Fulkerson may even fail to terminate!

6.6: Maximum ﬂow T.S. 18 6.6: Maximum ﬂow T.S. 18 Summary and Outlook Outline

Ford-Fulkerson Method works only for integral (rational) capacities Runtime: O(E · |f ∗|) = O(E · C) Introduction

Ford-Fulkerson Capacity-Scaling Algorithm Idea: Find an augmenting path with high capacity

Consider subgraph of Gf consisting of edges (u, v) with cf (u, v) > ∆ Max-Flow Min-Cut Theorem scaling parameter ∆, which is initially 2dlog2 Ce and 1 after termination 2 Runtime: O(E · log C) Analysis of Ford-Fulkerson

Edmonds-Karp Algorithm Matchings in Bipartite Graphs Idea: Find the shortest augmenting path in Gf Runtime: O(E 2 · V )

6.6: Maximum ﬂow T.S. 19 6.6: Maximum ﬂow T.S. 20

Application: Maximum-Bipartite-Matching Problem Matchings in Bipartite Graphs via Maximum Flows

Matching a g A matching is a subset M ⊆ E such that for all v ∈ V , at most one edge of M is incident to v. b h Bipartite Graph A graph G is bipartite if V can be partitioned into L s c i t and R so that all edges go between L and R.

d j

Given a bipartite graph G = (V ∪ L, E), ﬁnd a matching of maximum cardinality. e k L R

l Jobs Machines

6.6: Maximum ﬂow T.S. 21 6.6: Maximum ﬂow T.S. 22 Correspondence between Maximum Matchings and Max Flow From Matching to Flow

Theorem (Corollary 26.11) The cardinality of a maximum matching M in a bipartite graph G equals Given a maximum matching of cardinality k the value of a maximum flow f in the corresponding flow network Ge. Consider flow f that sends one unit along each each of k paths ⇒ f is a flow and has value k

f f max cardinality matching ≤ value of maxﬂow

a g a g

b h b h

c i s c i t s t d j d j

e k e k

l l

Graph G Graph G Graph G Graph Ge e

6.6: Maximum ﬂow T.S. 23 6.6: Maximum ﬂow T.S. 24

From Flow to Matching

Let f be a maximum flow in G0 of value k Integrality Theorem ⇒ f (u, v) ∈ {0, 1} and k integral Let M0 be all edges from L to R which carry a flow of one a) Flow Conservation ⇒ every node in L receives at most one unit b) Flow Conservation ⇒ every node in R sends at most one unit c) Cut (L ∪ {s}, R ∪ {t}) ⇒ net flow is k ⇒ M0 has k edges ⇒ By a) & b), M0 is a matching and by c), M0 has cardinality k

value of maxﬂow ≤ max cardinality matching

1/1 1/1

1/1 s t

L R L R

6.6: Maximum ﬂow T.S. 25 Outline

Introduction and Line Intersection

Convex Hull 7: Geometric Algorithms Frank Stajano Thomas Sauerwald Inside or Outside a Polygon?

Lent 2015

7: Geometric Algorithms T.S. 2

Introduction Cross Product (Area)

p1 + p2 = (3, 4) Computational Geometry y Branch that studies algorithms for p2 = (1, 3) geometric problems typically, input is a set of points, line p3 segments etc. How large is this area? p2 p1 = (2, 1) Applications p4 computer graphics x p (0, 0) computer vision 1 textile layout VLSI design Do these lines intersect? . . x1 x2 p1 × p2 = det = x1y2 − x2y1 = 2 · 3 − 1 · 1 = 5 y1 y2

p2 × p1 = y1x2 − y2x1 = −p1 × p2 = −5

7: Geometric Algorithms T.S. 3 7: Geometric Algorithms T.S. 4 p4

(p3(p−3 −p1p) 1×) ×(p2(p−2 −p1p) 1<) =0 (−3, −1) × (4, 2) = −10 p1p2 crosses p3p4 (p4(p−4 −p1p) 1×) ×(p2(p−2 −p1p) 1<) =0 (−2, −2) × (4, 2) = 4

Opposite signs ⇒p pp1p2 crossesdoes not crossOppositep signsp ⇒ p3p4 crosses (inﬁnite) line through1 p23 and p4 (inﬁnite) line3 through4 p1 and p2

Cross Product in 3D Using Cross product to determine Turns (1/2)

y p1 × p2 = (0, 0, x1y2 − x2y1) y p2 = (1, 3)

p1 × p2 > 0: left (counterclockwise) turn

p1 = (2, 1) p1 + p2 p2

p1 × xp3 < 0: right (clockwise) turn (p1 × · ) > 0 (0, 0) p1 p3 = (1, −1) (p1 × · ) < 0 x p3 Sign of cross product determines turn!

× p1 p3 Cross product equals zero iff vectors are colinear

7: Geometric Algorithms T.S. 5 7: Geometric Algorithms T.S. 6

Using Cross product to determine Turns (2/2) Solving Line Intersection (without Trigonometry and Division!)

p2 p4 y

p2 = (3, 4) (2, 1) × (1, 3) = 5

p1 (p1 − p0) × (p2 − p0) > 0: left turn

p1 = (4, 2) (2, 1) × (1, −1) = −3 p3 x p = (2, 1) 0 (p1 − p0) × (p3 − p0) < 0: right turn (0, 0) (p − p ) × (p − p ) = (3, 1) × (1, 4) = 11 x 1 3 4 3 (0, 0) p3 = (3, 0) (p2 − p3) × (p4 − p3) = (−1, 3) × (1, 4) = −7

Opposite signs ⇒ p1p2 crosses (inﬁnite) line through p3 and p4

7: Geometric Algorithms T.S. 7 7: Geometric Algorithms T.S. 8 p4 p4

p3 p3

(p3(p−1 −p1p) 3×) ×(p2(p−4 −p1p) 3<) 0= (3, 1) × (1, 4) = 11 (p3(p−13 −p1p) 31×) ×(p2(p−42 −p1p) 31<) =0 (−3,31,)−×1)(1×, 4(4) ,=2)11= −10 p1p2 crosses p3p4 (p4(p−2 −p1p) 3×) ×(p2(p−4 −p1p) 3<) =0 (−1, 3) × (1, 4) = −7 (p4(p−24 −p1p) 31×) ×(p2(p−42 −p1p) 31<) =0 (−12,,3−)2×) ×(1(,4,)2=) =−74

Opposite signs ⇒p pp1p2 crossesdoes not crossOppositep signsp ⇒ p3p4 crosses Opposite signs ⇒p pp1p2 crossesdoes not crossOppositep signsp ⇒ p3p4 crosses (infinite) line through1 p23 and p4 (infinite) line3 through4 p1 and p2 (infinite) line through1 p23 and p4 (infinite) line3 through4 p1 and p2

(p13 − p31) × (p42 − p31) = (−3,31,)−×1)(1×, 4(4) ,=2)11= −10 p1p2 crosses p3p4 (p24 − p31) × (p42 − p31) = (−12,,3−)2×) ×(1(,4,)2=) =−74

Opposite signs ⇒ p1p2 crosses Opposite signs ⇒ p3p4 crosses (inﬁnite) line through p3 and p4 (inﬁnite) line through p1 and p2

Solving Line Intersection (without Trigonometry and Division!) Solving Line Intersection (without Trigonometry and Division!)

y y

p2 p2 p4 p4

p1 p1

p3 p3 x x (0, 0) (0, 0) (p3 − p1) × (p2 − p1) = (−3, −1) × (4, 2) = −10 p1p2 crosses p3p4 (p4 − p1) × (p2 − p1) = (−2, −2) × (4, 2) = 4

Opposite signs ⇒ p1p2 crosses Opposite signs ⇒ p3p4 crosses Opposite signs ⇒ p1p2 crosses Opposite signs ⇒ p3p4 crosses (infinite) line through p3 and p4 (infinite) line through p1 and p2 (infinite) line through p3 and p4 (infinite) line through p1 and p2

7: Geometric Algorithms T.S. 8 7: Geometric Algorithms T.S. 8

Solving Line Intersection (without Trigonometry and Division!) Solving Line Intersection

y y p4 p2 p2

p3 x p4 p1 (0, 0) DIRECTION(p3, p4, p1) = (p1 − p3) × (p4 − p3) p2 p3 0: DIRECTION(pi , pj , pk ) 1: return (p − p ) × (p − p ) p2 x k i j i p4 p4 (0, 0) p3 (p3 − p1) × (p2 − p1) < 0 0: SEGMENTS-INTERSECT(pi , pj , pk ) 1: d1 = DIRECTION(p3, p4, p1) p1 2: d2 = DIRECTION(p3, p4, p2) (p4 − p1) × (p2 − p1) < 0 3: d3 = DIRECTION(p1, p2, p3) p1 4: d4 = DIRECTION(p1, p2, p4) In total 4 satisfying conditions! p3 5: If d1 · d2 < 0 and d3 · d4 < 0 return TRUE 6: ··· (handle all degenerate cases) p1p2 does not cross p3p4 Lines could toch or be colinear

7: Geometric Algorithms T.S. 8 7: Geometric Algorithms T.S. 9 Overall Runtime: O(n log n)

keep on removing recent points until point can be added

x Use Cross Product! Efﬁcient Sorting by comparing (not computing!) polar angles

Otherwise,

0: GRAHAM-SCAN(Q) 1: Let p0 be the point with minimum y-coordinate 2: Let (p1, p2,..., pn) be the other points sorted by polar angle w.r.t. p0 3: If n < 2 return false 4: S = ∅ 5: PUSH(p0,S) Takes O(n log n) time 6: PUSH(p1,S) 7: PUSH(p2,S) 8: For i = 3 to n 9: While angle of NEXT-TO-TOP(S),TOP(S),pi makes a non-left turn 10: POP(S) 11: End While Takes O(n) time, since every point is 12: PUSH(pi ,S) 13: End For part of a PUSH or POP at most once. 14: Return S

Outline Convex Hull

Vertex lies on the convex hull, but is not part of the polygon! Introduction and Line Intersection

Convex Hull Deﬁnition The convex hull of a set Q of points is the smallest convex polygon P for which each point in Q is either on the boundary of P or in its interior.

Inside or Outside a Polygon? Smallest perimeter fence enclosing the points

Convex Hull Problem Input: set of points Q in the Euclidean space Output: return points of the convex hull in counterclockwise order

7: Geometric Algorithms T.S. 10 7: Geometric Algorithms T.S. 11

Application of Convex Hull Graham’s Scan

2 Robot Motion Planning

Find shortest path from s to t which avoids a polygonal obstacle. 4 3 1

can be solved by computing the Convex hull! 0

Basic Idea Start with the point with smallest y-coordinate Sort all points increasingly according to their polar angle s Try to add next point to the convex hull t If it does not introduce non-left turn, then ﬁne X

7: Geometric Algorithms T.S. 12 7: Geometric Algorithms T.S. 13 Overall Runtime: O(n log n) Overall Runtime: O(n log n)

0 x 0 Use Cross Product! Efﬁcient Sorting by comparing (not computing!) polar angles

Otherwise, keep on removing recent points until point can be added

0: GRAHAM-SCAN(Q) 0: GRAHAM-SCAN(Q) 1: Let p0 be the point with minimum y-coordinate 1: Let p0 be the point with minimum y-coordinate 2: Let (p1, p2,..., pn) be the other points sorted by polar angle w.r.t. p0 2: Let (p1, p2,..., pn) be the other points sorted by polar angle w.r.t. p0 3: If n < 2 return false 3: If n < 2 return false 4: S = ∅ 4: S = ∅ 5: PUSH(p0,S) Takes O(n log n) time 5: PUSH(p0,S) Takes O(n log n) time 6: PUSH(p1,S) 6: PUSH(p1,S) 7: PUSH(p2,S) 7: PUSH(p2,S) 8: For i = 3 to n 8: For i = 3 to n 9: While angle of NEXT-TO-TOP(S),TOP(S),pi makes a non-left turn 9: While angle of NEXT-TO-TOP(S),TOP(S),pi makes a non-left turn 10: POP(S) 10: POP(S) 11: End While 11: End While Takes O(n) time, since every point is Takes O(n) time, since every point is 12: PUSH(pi ,S) 12: PUSH(pi ,S) 13: End For part of a PUSH or POP at most once. 13: End For part of a PUSH or POP at most once. 14: Return S 14: Return S

111214101315124368975 0 1 245 368 111297 1013 1415 y X

0 x

Graham’s Scan Graham’s Scan

2 2 y 4 4 3 1 3 1

0 0 x Use Cross Product! Efficient Sorting by comparing (not computing!) polar angles Basic Idea Basic Idea Start with the point with smallest y-coordinate Start with the point with smallest y-coordinate Sort all points increasingly according to their polar angle Sort all points increasingly according to their polar angle Try to add next point to the convex hull Try to add next point to the convex hull If it does not introduce non-left turn, then fine X If it does not introduce non-left turn, then fine X Otherwise, keep on removing recent points until point can be added

7: Geometric Algorithms T.S. 13 7: Geometric Algorithms T.S. 13

Graham’s Scan Complete Run of Graham’s Scan

2 i = 0

4 3 1 5 Overall Runtime: O(n log n) 1 4 8 2 0

6 3 0: GRAHAM-SCAN(Q) 1: Let p0 be the point with minimum y-coordinate 9 2: Let (p1, p2,..., pn) be the other points sorted by polar angle w.r.t. p0 3: If n < 2 return false 12 4: S = ∅ 7 5: PUSH(p0,S) Takes O(n log n) time 6: PUSH(p1,S) 7: PUSH(p ,S) 11 2 13 8: For i = 3 to n 9: While angle of NEXT-TO-TOP(S),TOP(S),pi makes a non-left turn 10: POP(S) 10 11: End While Takes O(n) time, since every point is 14 12: PUSH(pi ,S) 13: End For part of a PUSH or POP at most once. 15

14: Return S 0

7: Geometric Algorithms T.S. 13 7: Geometric Algorithms T.S. 14 11121410131240368975 24 36 1197 10 14

y y

x x x x x

x x

x x x y y y y y y y y y y

Complete Run of Graham’s Scan Jarvis’ March (Gift wrapping)

i = 15 X 0 1 5 8 12 13 15 Intuition 5 1 Wrapping taut paper around the points 4 8 2 1. Tape end of paper at lowest point 2. Pull paper to the right until it touches a point 3. Tape paper and go to 2

6 3

9 Algorithm 12 7 1. Let p0 be the lowest point

2. Next point the one with smallest angle w.r.t. p0 11 3. Continue until highest point p 13 k 4. Next point the one with smallest angle w.r.t. pk 10 5. Continue until p is reached 0 Here, we rotate the coordinate system by 180! 14

15 Runtime: O(n · h) Output sensitive algorithm! 0

7: Geometric Algorithms T.S. 14 7: Geometric Algorithms T.S. 15

Execution of Jarvis’ March Execution of Jarvis’ March

y y

7: Geometric Algorithms T.S. 16 7: Geometric Algorithms T.S. 16 y y y y

x x x x

x x

x x x x y y y y y y y y

Execution of Jarvis’ March Execution of Jarvis’ March

x x

y y

7: Geometric Algorithms T.S. 16 7: Geometric Algorithms T.S. 16

Computing Convex Hull: Summary Outline

Graham’s Scan natural backtracking algorithm cross-product avoids computing polar angles Runtime dominated by sorting O(n log n) Introduction and Line Intersection Jarvis’ March proceeds like wrapping a gift

Runtime O(nh) output-sensitive Convex Hull Improves Graham’s scan only if h = O(log n)

There exists an algorithm with O(n log h) runtime! Inside or Outside a Polygon? Lessons Learned cross product very powerful tool take care of degenerate cases, numerical precision

7: Geometric Algorithms T.S. 17 7: Geometric Algorithms T.S. 18 “inﬁnity”: can take Observation: p is inside if a line drawn from “inﬁnity” (max x + 1, max y + 1) Two-year old human can do it, but it’s not to p crosses the polygon an odd number of times i i i i so obvious how to program a computer... q 0

2 3

Another Problem: The Inside-or-Outside a Polygon Problem Another Problem: The Inside-or-Outside a Polygon Problem Problem Problem

Given: A polygon as list of edges (x1, y1), (x2, y2),..., (xn, yn), and Given: A polygon as list of edges (x1, y1), (x2, y2),..., (xn, yn), and another point p = (p1, p2) another point p = (p1, p2)

Question: Is the point p = (p1, p2) inside or outside the polygon? Question: Is the point p = (p1, p2) inside or outside the polygon? “inﬁnity”: can take Observation: p is inside if a line drawn from “inﬁnity” Two-year old human can do it, but it’s not (max x + 1, max y + 1) to p crosses the polygon an odd number of times i i i i so obvious how to program a computer... q 0

2 3 p p

7: Geometric Algorithms T.S. 19 7: Geometric Algorithms T.S. 19

Solution to the Inside-or-Outside a Polygon Problem Inside-or-Outside the Polygon Problem: Are we done yet? q Some extra tweaks are needed to make our program (fully) correct...

“Unusual” (Degenerate) Cases: q

line (p, q) crosses an endpoint of an edge 0 line (p, q) is parallel to an edge 0 p 1

3 But this is just the line segment intersection problem... 5 1 7 0: IsInside(point p, polygon G) 8 0 2 1: { let q be a point outside G (’infinity’) p 2: int count = 0; 3 3: for E an edge of G do p 4: if (pq intersects E) count = count+1; 5: if (count is odd) return YES; 6: return NO; 7: }

7: Geometric Algorithms T.S. 20 7: Geometric Algorithms T.S. 21 How likely are “unusual” cases? The End

First Idea p3 p2

If we take four random points in a plane, then p2 the mathematical probability of: p4 p4 two of them being identical; or any two lines connecting them being p Thank you for attending this course & parallel 1 is vashingly small (even “probability zero”)! p1 Best wishes for the rest of your Tripos!

So, I don’t have to worry about this when programming? Don’t forget to visit the online feedback page! Please send comments on the slides (typos, criticsm, praise etc.) to: Wrong! [email protected] Imagine people drawing squares or two objects together (Real examples may result in lots of horrible cases) Computer ﬂoating-point arithmetic is not exact (Problems when points are very close)

7: Geometric Algorithms T.S. 22 7: Geometric Algorithms T.S. 23