Divide-and-Conquer Algorithms

CS311H: Discrete Mathematics

Divide-and-Conquer Algorithms and The Master Theorem

I Divide-and-conquer algorithms are recursive algorithms that: Instructor: I¸sıl Dillig 1. Divide problem into k smaller subproblems of the same form

2. Solve the subproblems

3. Conquer the original problem by combining solutions of subproblems

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 1/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 2/19

Example I: Binary Search Binary Search, cont.

I Question: What is the worst-case complexity of binary search?

I Let T (n) denote # of steps taken on input array of size n

I Problem: Given sorted array of integers, is i in the array? I Write recurrence relation for T (n): I Initial condition: I Binary search algorithm:

1. Compare i with middle element m of array I How do we get a Big-O estimate from this recurrence?

2. If i > m, then recursively search right half I Idea: Solve the recurrence and then find Big-O estimate for it

3. Otherwise, recursively search left half

I Classic divide-and-conquer algorithm

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 3/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 4/19

Solving Recurrence for Binary Search Example II: Merge Sort

n T (n) = T ( ) + 1 T (1) = 1 2

I Problem: Sort elements in array I Not in a form we can immediately solve, but can massage it! I Merge sort solution: k k k 1 I Let n = 2 : T (2 ) = T (2 − ) + 1 1. Recursively sort left half of array k I Now, let ak = T (2 ): ak = ak 1 + 1 a0 = 1 − 2. Recursively sort right half of array

I What’s the solution for this recurrence? 3. Merge the two sorted arrays

k I Since n = 2 , this implies T (n) = log2n + 1

I Hence, complexity of binary search: Θ(log n)

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 5/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 6/19

1 How to Merge Two Sorted Arrays? Recurrence Relation for Merge Sort

I What is worst-case complexity of Merge Sort?

I Let T (n) be # operations performed to sort array of length n I Input: Two sorted arrays A1, A2 I What is a recurrence relation for T (n)? I Output: New sorted array that includes all elements in A1, A2 k I As before, let n = 2 : I Idea: Pointers to current elements in A1, A2 (initially first) I I Copy smaller element to output array and advance pointer

I If combined size of A1, A2 is n, merging takes 4n steps (compare, advance two pointers, copy)

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 7/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 8/19

Solving Recurrence Relation Summary

I Recurrence relations for divide-conquer algorithms look like: k ak = 2 ak 1 + 4 2 a0 = 1 · − · n T (n) = a T ( ) + f (n) I Particular solution form: · b

I Particular solution: I These are called divide-and-conquer recurrence relations

I To determine complexity of a divide-and conquer algorithm: I Solution for homogeneous recurrence: 1. Write corresponding recurrence relation 0 2 I Solve for α: α 2 + 0 2 = 1 α = 1 · · ⇒ 2. Solve it exactly I Solution: 3. Obtain Θ estimate I Plug in k = log2 n: I Can we obtain a Θ estimate without solving recurrence I Hence, algorithm is Θ(n log n) exactly? ·

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 9/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 10/19

The Master Theorem Revisiting Examples

n I Example 1: Recurrence for binary search: T (n) = T ( ) + 1 n d 2 Consider the recurrence T (n) = a T ( b ) + c n where a, c 1, · · ≥ d d 0, and b > 1. Then: I Here, a = 1, b = 2, d = 0, Hence a = b ≥ 0 1. T (n) is Θ(nd ) if a < bd I By Case 2 of Master Thm, T (n) = Θ(n log n) = Θ(log n)

n d d I Example 2: Recurrence for merge sort: T (n) = 2 T ( ) + 4n 2. T (n) is Θ(n logn) if a = b · 2 d 3. T (n) is Θ(nlogb a ) if a > bd I Here, a = 2, b = 2, d = 1, Hence a = b

I By Case 2 of Master Thm, T (n) =Θ( n log n) ·

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 11/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 12/19

2 More Examples Why is the Master Theorem True? Consider the recurrence T (n) = a T ( n ) + c nd · b ·

n I At every level of recursion, # subproblems multiplied by a I Example 3: Consider recurrence T (n) = 2 T ( ) + 3 · 2 I But size of subproblem divided by b I d I Let f (n) be c n I ·

n 2 I Example 4: Consider recurrence T (n) = T ( 2 ) + n

I

I

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 13/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 14/19

Proof of Master Theorem Proof of Master Theorem, cont.

I Total amount of work: I What is the height h of this tree? log n 1 b − n T (n) = Θ(nlogb a ) + ai c ( )d n · · bi I Since problem size is 1 in base case, = 1 h = log n i=0 bh ⇒ b X I Can be rewritten as:

i logb n log n 1 I At the i’th level, we have a subproblems, hence a leaves b − a T (n) = Θ(nlogb a ) + c ( )i nd · bd · logb a i=0 I Equal to n – verify by taking logb of both sides X

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 15/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 16/19

Proof of Master Theorem, cont. Proof of Master Theorem, cont.

log n 1 log n 1 b − b log a a i d − a T (n) = Θ(n b ) + c ( ) n T (n) = Θ(nlogb a ) + c ( )i nd · bd · · bd · i=0 i=0 X X a d I Case 1: bd < 1. In this case, T (n) is of the form: I Case 2: a = b . In this case, T (n) is of the form:

log n 1 log n 1 b − b − T (n) = Θ(nlogb a ) + c nd r i for r < 1 T (n) = Θ(nlogb a ) + c nd · · | | · i=0 i=0 X X log a d log a d I Hence: T (n) = Θ(n b ) + Θ(n ) I Hence: T (n) = Θ(n b ) + Θ(n log n) · b a d log a d d I Since < 1, we have log a d < 1. Thus T (n) = Θ(n ) I Since n b = n , this is Θ(n log n) bd b − · b

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 17/19 Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 18/19

3 Proof of Master Theorem, cont.

log n 1 b − a T (n) = Θ(nlogb a ) + c ( )i nd · bd · i=0 X d log a d I Case 3: a > b . In this case, n b > n .

I Use closed formula for geometric series to expand summation:

a logb n 1 a 1 ( d ) − c nd − b · · bd · 1 a − bd log n d I This can be rewritten to c1a b + c2n for some constants c1, c2

log n log a log a I Since , a b = n b , T (n) is Θ(n b )

Instructor: I¸sıl Dillig, CS311H: Discrete Mathematics Divide-and-Conquer Algorithms and The Master Theorem 19/19

4