Divide-and-conquer paradigm
Divide-and-conquer. 5. DIVIDE AND CONQUER I Divide up problem into several subproblems (of the same kind). Solve (conquer) each subproblem recursively. ‣ mergesort Combine solutions to subproblems into overall solution. ‣ counting inversions Most common usage. ‣ randomized quicksort Divide problem of size n into two subproblems of size n / 2. O(n) time ‣ median and selection Solve (conquer) two subproblems recursively. ‣ closest pair of points Combine two solutions into overall solution. O(n) time
Consequence. Brute force: Θ(n2).
Lecture slides by Kevin Wayne Divide-and-conquer: O(n log n). Copyright © 2005 Pearson-Addison Wesley http://www.cs.princeton.edu/~wayne/kleinberg-tardos
Last updated on 3/12/18 9:21 AM attributed to Julius Caesar 2
Sorting problem
Problem. Given a list L of n elements from a totally ordered universe, 5. DIVIDE AND CONQUER rearrange them in ascending order.
‣ mergesort ‣ counting inversions ‣ randomized quicksort ‣ median and selection ‣ closest pair of points
SECTIONS 5.1–5.2
4 Sorting applications Mergesort
Obvious applications. Recursively sort left half. Organize an MP3 library. Recursively sort right half. Display Google PageRank results. Merge two halves to make sorted whole. List RSS news items in reverse chronological order.
Some problems become easier once elements are sorted. input Identify statistical outliers. A L G O R I T H M S Binary search in a database. Remove duplicates in a mailing list. sort left half A G L O R I T H M S Non-obvious applications. sort right half Convex hull. Closest pair of points. A G L O R H I M S T Interval scheduling / interval partitioning. merge results Scheduling to minimize maximum lateness. A G H I L M O R S T Minimum spanning trees (Kruskal’s algorithm). ... 5 6
Merging Mergesort implementation
Goal. Combine two sorted lists A and B into a sorted whole C. Input. List L of n elements from a totally ordered universe. Scan A and B from left to right. Output. The n elements in ascending order. Compare ai and bj. If ai ≤ bj, append ai to C (no larger than any remaining element in B). If ai > bj, append bj to C (smaller than every remaining element in A). MERGE-SORT(L)
______IF (list L has one element)
RETURN L.
sorted list A sorted list B Divide the list into two halves A and B. 3 7 10 ai 18 2 11 bj 20 23 A ← MERGE-SORT(A). T(n / 2) B ← MERGE-SORT(B). T(n / 2)
ERGE Θ(n) merge to form sorted list C L ← M (A, B). RETURN L.
2 3 7 10 11 ______
7 8 A useful recurrence relation Divide-and-conquer recurrence: recursion tree
Def. T (n) = max number of compares to mergesort a list of length n. Proposition. If T (n) satisfies the following recurrence, then T (n) = n log2 n.
0 n =1 assuming n Recurrence. T (n)= is a power of 2 2 T (n/2) + n n>1 0 n =1
between ⎣n / 2⎦ and n − 1 compares T (n) n = n
Solution. T (n) is O(n log2 n). T (n / 2) T (n / 2) 2 (n/2) = n
T (n / 4) T (n / 4) T (n / 4) T (n / 4) 4 (n/4) = n Assorted proofs. We describe several ways to solve this recurrence. log 2 n Initially we assume n is a power of 2 and replace ≤ with = in the recurrence.
T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) 8 (n/8) = n ⋮ ⋮ T(n) = n log2 n 9 10
Proof by induction Divide-and-conquer: quiz 1
Proposition. If T (n) satisfies the following recurrence, then T (n) = n log2 n. Which is the exact solution of the following recurrence?
0 n =1 assuming n 0 n =1 T (n)= is a power of 2 T (n)= 2 T (n/2) + n n>1 T ( n/2 )+T ( n/2 )+n 1 n>1
⎣log n⎦ C. T(n) = n ⎣log2 n⎦ + 2 2 − 1 recurrence n ⎡log n⎤ D. T(n) = n ⎡log2 n⎤ − 2 2 + 1 = log k 2
T(2n) = 2 T(n) + 2n
= 2 n (log2 (2n) – 1) + 2n
= 2 n log2 (2n). ▪
11 12 Analysis of mergesort recurrence Digression: sorting lower bound
Proposition. If T(n) satisfies the following recurrence, then T(n) ≤ n ⎡log2 n⎤. Challenge. How to prove a lower bound for all conceivable algorithms?
0 n =1 Model of computation. Comparison trees. T (n) T ( n/2 )+T ( n/2 )+n n>1 Can access the elements only through pairwise comparisons.
= n ⎡log2 n2⎤ + n log n2 logl logn 2 n 1 m log2 n2 =2logd2 n e /12 Comparable[] a = ...; d 2 2ed 2 e ... ≤ n (⎡log2 n⎤ – 1) + n log2 n2 log2 n 1 can access elements only d e Arrays.sort(a); via calls to compareTo() log2 n2 log2 n 1 = n ⎡log2 n⎤. ▪ d ed e an integer 14
Comparison tree (for 3 distinct keys a, b, and c) Sorting lower bound
Theorem. Any deterministic compare-based sorting algorithm must make
a < b Ω(n log n) compares in the worst-case. height of pruned tree = yes no worst-case number of compares code between compares Pf. [ information theoretic ] (e.g., sequence of exchanges) Assume array consists of n distinct values a1 through an. Worst-case number of compares = height h of pruned comparison tree. b < c a < c Binary tree of height h has ≤ 2h leaves. yes no yes no n! different orderings ⇒ n! reachable leaves.
a b c a < c b a c b < c
yes no yes no
h
a c b c a b b c a c b a
n! leaves ≤ 2h leaves each reachable leaf corresponds to one (and only one) ordering;
exactly one reachable leaf for each possible ordering 15 16 Sorting lower bound SHUFFLING A LINKED LIST
Theorem. Any deterministic compare-based sorting algorithm must make Ω(n log n) compares in the worst-case. Problem. Given a singly linked list, rearrange its nodes uniformly at random. Assumption. Access to a perfect random-number generator. all n! permutations Pf. [ information theoretic ] equally likely Assume array consists of n distinct values a1 through an. Performance. O(n log n) time, O(log n) extra space. Worst-case number of compares = height h of pruned comparison tree. Binary tree of height h has ≤ 2h leaves. n! different orderings ⇒ n! reachable leaves.
L 2h ≥ # leaves ≥ n ! input 2♣ 3♣ 4♣ 5♣ 6♣ 7♣ ⇒ h ≥ log2 (n!) null
≥ n log2 n − n / ln 2 ▪ L shufed Stirling’s formula 5♣ 6♣ 2♣ 7♣ 3♣ 4♣ null
Note. Lower bound can be extended to include randomized algorithms. 17 18
Counting inversions
Music site tries to match your song preferences with others. 5. DIVIDE AND CONQUER You rank n songs. Music site consults database to find people with similar tastes. ‣ mergesort ‣ counting inversions Similarity metric: number of inversions between two rankings. My rank: 1, 2, …, n. ‣ randomized quicksort Your rank: a1, a2, …, an. ‣ median and selection Songs i and j are inverted if i < j, but ai > aj. ‣ closest pair of points
A B C D E
me 1 2 3 4 5 SECTION 5.3 you 1 3 4 2 5
2 inversions: 3-2, 4-2
Brute force: check all Θ(n2) pairs.
20 Counting inversions: applications Counting inversions: divide-and-conquer
Voting theory. Divide: separate list into two halves A and B. Collaborative filtering. Conquer: recursively count inversions in each list. Measuring the “sortedness” of an array. Combine: count inversions (a, b) with a ∈ A and b ∈ B. Sensitivity analysis of Google’s ranking function. Return sum of three counts. Rank aggregation for meta-searching on the Web. Nonparametric statistics (e.g., Kendall’s tau distance). input 1 5 4 8 10 2 6 9 3 7 Rank Aggregation Methods for the Web
Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Rank Aggregation Methods for the Web count inversions in left half A count inversions in right half B
Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar 1 5 4 8 10 2 6 9 3 7
ABSTRACT 5-4 6-3 9-3 9-7
ABSTRACT count inversions (a, b) with a ∈ A and b ∈ B
1 5 4 8 10 2 6 9 3 7
1.1 Motivation 4-2 4-3 5-2 5-3 8-2 8-3 8-6 8-7 10-2 10-3 10-6 10-7 10-9
1. INTRODUCTION 21 output 1 + 3 + 13 = 17 22 1.1 Motivation
1. INTRODUCTION
Counting inversions: how to combine two subproblems? Counting inversions: how to combine two subproblems?
Q. How to count inversions (a, b) with a ∈ A and b ∈ B? Count inversions (a, b) with a ∈ A and b ∈ B, assuming A and B are sorted. A. Easy if A and B are sorted! Scan A and B from left to right. Compare ai and bj. Warmup algorithm. If ai < bj, then ai is not inverted with any element left in B. Copyright is held by the author/owner. Sort A and B. WWW10, May 1-5, 2001, Hong Kong. If ai > bj, then bj is inverted with every element left in A. ACM 1-58113-348-0/01/0005.
For each element b ∈ B, 613 Append smaller element to sorted list C. Copyright is held by the author/owner. WWW10, May 1-5, 2001, Hong Kong. - binaryACM search 1-58113-348-0/01/0005. in A to find how elements in A are greater than b.
613
list A list B
7 10 18 3 14 20 23 2 11 16 count inversions (a, b) with a ∈ A and b ∈ B
sort A sort B 3 7 10 ai 18 2 11 bj 20 23
5 2 3 7 10 14 18 2 11 16 20 23
binary search to count inversions (a, b) with a ∈ A and b ∈ B merge to form sorted list C
3 7 10 14 18 2 11 16 20 23 2 3 7 10 11
5 2 1 0 0 23 24 Counting inversions: divide-and-conquer algorithm implementation Counting inversions: divide-and-conquer algorithm analysis
Input. List L. Proposition. The sort-and-count algorithm counts the number of inversions Output. Number of inversions in L and L in sorted order. in a permutation of size n in O(n log n) time.
Pf. The worst-case running time T(n) satisfies the recurrence:
SORT-AND-COUNT(L)
______IF (list L has one element) (1) n =1 T (n)= RETURN (0, L). T ( n/2 )+T ( n/2 )+ (n) n>1
Divide the list into two halves A and B.
(rA, A) ← SORT-AND-COUNT(A). T(n / 2)
(rB, B) ← SORT-AND-COUNT(B). T(n / 2)
(rAB, L) ← MERGE-AND-COUNT(A, B). Θ(n)
RETURN (rA + rB + rAB, L).
______
25 26
3-WAY PARTITIONING
5. DIVIDE AND CONQUER Goal. Given an array A and pivot element p, partition array so that: Smaller elements in left subarray L. ‣ mergesort Equal elements in middle subarray M. ‣ counting inversions Larger elements in right subarray R. ‣ randomized quicksort Challenge. O(n) time and O(1) space. ‣ median and selection ‣ closest pair of points the array A
7 6 12 3 11 8 9 1 4 10 2
SECTION 7.1–7.3 p
the partitioned array A
3 1 4 2 6 7 12 11 8 9 10
L M R
28 Randomized quicksort Randomized quicksort
Pick a random pivot element p ∈ A. Pick a random pivot element p ∈ A. 3-way partition the array into L, M, and R. 3-way partition the array into L, M, and R. Recursively sort both L and R. Recursively sort both L and R.
the array A 7 6 12 3 11 8 9 1 4 10 2
p
RANDOMIZED-QUICKSORT(A)
partition A 3 1 4 2 6 7 12 11 8 9 10 ______IF (array A has zero or one element)
RETURN. sort L 1 2 3 4 6 7 12 11 8 9 10 Pick pivot p ∈ A uniformly at random. (L, M, R) ← PARTITION-3-WAY(A, p). Θ(n)
RANDOMIZED-QUICKSORT(L). T(i) sort R 1 2 3 4 6 7 8 9 10 11 12 new analysis required (i is a random variable—depends on p) RANDOMIZED-QUICKSORT(R). T(n − i − 1)
______
the sorted array A 1 2 3 4 6 7 8 9 10 11 12
29 30
Analysis of randomized quicksort Analysis of randomized quicksort
Proposition. The expected number of compares to quicksort an array Proposition. The expected number of compares to quicksort an array of n distinct elements a1 < a2 < < an is O(n log n). of n distinct elements a1 < a2 < < an is O(n log n).
Pf. Consider BST representation of pivot elements. Pf. Consider BST representation of pivot elements. ai and aj are compared once iff one is an ancestor of the other. the original array of elements A
a7 a6 a12 a3 a11 a8 a9 a1 a4 a10 a2 a13 a5
a3 and a6 are compared first pivot first pivot (when a3 is pivot) (chosen uniformly at random) (chosen uniformly at random) first pivot in first pivot in left subarray left subarray a9 a9
a3 a11 a3 a11
a1 a8 a10 a13 a1 a8 a10 a13
a2 a4 a12 a2 a4 a12
a6 a6
a5 a7 a5 a7 31 32 Analysis of randomized quicksort Divide-and-conquer: quiz 2
Proposition. The expected number of compares to quicksort an array Given an array of n ≥ 8 distinct elements a1 < a2 < < an, what is the of n distinct elements a1 < a2 < < an is O(n log n). probability that a7 and a8 are compared during randomized quicksort?
Pf. Consider BST representation of pivot elements. A. 0 ai and aj are compared once iff one is an ancestor of the other. B. 1 / n
C. 2 / n
D. 1 if two elements are adjacent, one must be an ancestor of the other a2 and a8 are not compared first pivot (because a3 partitions them) (chosen uniformly at random) first pivot in left subarray a9 a9
a3 a11 a3 a11
a1 a8 a10 a13 a1 a8 a10 a13
a2 a4 a12 a2 a4 a12
a6 a6
a5 a7 a5 a7 33 34
Divide-and-conquer: quiz 3 Analysis of randomized quicksort
Given an array of n ≥ 2 distinct elements a1 < a2 < < an, what is the Proposition. The expected number of compares to quicksort an array of n probability that a1 and an are compared during randomized quicksort? distinct elements a1 < a2 < < an is O(n log n).
Pf. Consider BST representation of pivot elements. A. 0 ai and aj are compared once iff one is an ancestor of the other. B. 1 / n Pr [ ai and aj are compared ] = 2 / ( j - i + 1), where i < j. C. 2 / n a1 and an are compared iff first pivot is either a1 or an D. 1
Pr[a2 and a8 compared] = 2/7
compared if either a2 or a8 is chosen
as pivot before any of { a3, a4, a5, a6, a7}
a9 a9
a3 a11 a3 a11
a1 a8 a10 a13 a1 a8 a10 a13
a2 a4 a12 a2 a4 a12
a6 a6
a5 a7 a5 a7 35 36 number). 9.9 X 10 45 is used to represent infinity. Imaginary conunent I and J are output variables, and A is the array (with Analysis of randomized quicksort valuesTony of x may Hoarenot be negative and reM values of x may not be subscript bounds M:N) which is operated upon by this procedure. smaller than 1. Partition takes the value X of a random element of the array A, Values of Qd~'(x) may be calculated easily by hypergeometric and rearranges the values of the elements of the array in such a series if x is not too small nor (n - m) too large. Q~m(x) can be way that there exist integers I and J with the following properties : computed from an appropriate set of values of Pnm(X) if X is near M _-< J < I =< NprovidedM < N 1.0 or ix is near 0. Loss of significant digits occurs for x as small as A[R] =< XforM =< R _-< J Proposition. The expected number of compares to quicksort an array of n 1.1 if n is largerInvented than 10. Loss of significant quicksort digits is a major diffi-to translateA[R] = RussianXforJ < R < I into English. culty in using finite polynomiM representations also if n is larger A[R] ~ Xfor I =< R ~ N than m. However, QLEG has been tested in regions of x and n The procedure uses an integer procedure random (M,N) which distinct elements a1 < a2 < < an is O(n log n). both large [and but small; couldn’t explain hischooses algorithm equiprobably a random integeror Fimplement between M and N, and it! ] procedure QLEG(m, nmax, x, ri, R, Q); value In, nmax, x, ri; also a procedure exchange, which exchanges the values of its two real In, mnax, x, ri; real array R, Q; parameters ; begin real t, i, n, q0, s; begin real X; integer F; n :=Learned 20; Algol 60 (and recursion).F := random (M,N); X := A[F]; if nmax > 13 then I:=M; J:=N; n := nmax +7; up: for I : = I step 1 until N do Pf. Consider BST representation of pivot elements. if Implementedri = 0then quicksort. if X < A [I] then go to down; begin ifm = 0then I:=N; Q[0] := 0.5 X 10g((x + 1)/(x - 1)) down: forJ := J step --1 until M do ai and aj are compared once iff one is an ancestor of the other. else if A[J]
median and selection ‣ ‣ closest pair of points
SECTION 9.3
Brute-force solution. Compare each bolt to each nut—Θ(n 2) compares. Challenge. Design an algorithm that makes O(n log n) compares.
39 Median and selection problems Randomized quickselect
Selection. Given n elements from a totally ordered universe, find kth smallest. Pick a random pivot element p ∈ A. Minimum: k = 1; maximum: k = n. 3-way partition the array into L, M, and R. Median: k = ⎣(n + 1) / 2⎦. Recur in one subarray—the one containing the kth smallest element. O(n) compares for min or max. O(n log n) compares by sorting. O(n log k) compares with a binary heap. max heap with k smallest
Applications. Order statistics; find the “top k”; bottleneck paths, …
QUICK-SELECT(A, k)
Q. Can we do it with O(n) compares? ______A. Yes! Selection is easier than sorting. Pick pivot p ∈ A uniformly at random. (L, M, R) ← PARTITION-3-WAY(A, p). Θ(n)
IF (k ≤ | L |) RETURN QUICK-SELECT(L, k). T(i)
ELSE IF (k > | L | + | M |) RETURN QUICK-SELECT(R, k – | L | – | M |) T(n − i − 1)
ELSE IF (k = | L |) RETURN p.
______
43 44
Randomized quickselect analysis Selection in worst-case linear time
Intuition. Split candy bar uniformly ⇒ expected size of larger piece is ¾. Goal. Find pivot element p that divides list of n elements into two pieces so that each piece is guaranteed to have ≤ 7/10 n elements. T(n) ≤ T(3 n / 4) + n ⇒ T(n) ≤ 4 n
not rigorous: can’t assume E[T(i)] ≤ T(E[i]) Q. How to find approximate median in linear time? Def. T(n, k) = expected # compares to select kth smallest in array of length ≤ n. A. Recursively compute median of sample of ≤ 2/10 n elements.
Def. T(n) = maxk T(n, k).
Θ(1) if n = 1 Proposition. T(n) ≤ 4 n. T(n) = T (7/10 n) + T (2/10 n) + Θ(n) otherwise Pf. [ by strong induction on n ] can assume we always recur of Assume true for 1, 2, …, n – 1. larger of two subarrays since T(n) is monotone non-decreasing T(n) satisfies the following recurrence: two subproblems of different sizes! T(n) ≤ n + 1 / n [ 2T(n / 2) + … + 2T(n – 3) + 2T(n – 2) + 2T(n – 1) ]
≤ n + 1 / n [ 8(n / 2) + … + 8(n – 3) + 8(n – 2) + 8(n – 1) ] ⇒ T(n) = Θ(n) ≤ n + 1 / n (3n2) inductive hypothesis we’ll need to show this = 4 n. ▪ tiny cheat: sum should start at T(⎣n/2⎦)
45 46 Choosing the pivot element Choosing the pivot element
Divide n elements into ⎣n / 5⎦ groups of 5 elements each (plus extra). Divide n elements into ⎣n / 5⎦ groups of 5 elements each (plus extra). Find median of each group (except extra).
29 10 38 37 2 55 18 24 34 35 36 29 10 38 37 2 55 18 24 34 35 36
22 44 52 11 53 12 13 43 20 4 27 22 44 52 11 53 12 13 43 20 4 27
28 23 6 26 40 19 1 46 31 49 8 28 23 6 26 40 19 1 46 31 49 8
14 9 5 3 54 30 48 47 32 51 21 14 9 5 3 54 30 48 47 32 51 21
45 39 50 15 25 16 41 17 22 7 45 39 50 15 25 16 41 17 22 7
n = 54 47 n = 54 48
Choosing the pivot element Median-of-medians selection algorithm
Divide n elements into ⎣n / 5⎦ groups of 5 elements each (plus extra). Find median of each group (except extra). MOM-SELECT(A, k) Find median of ⎣n / 5⎦ medians recursively. ______ Use median-of-medians as pivot element. n ← | A |.
medians IF (n < 50) RETURN kth smallest of element of A via mergesort. median of medians 29 10 38 37 2 55 18 24 34 35 36 Group A into ⎣n / 5⎦ groups of 5 elements each (ignore leftovers). B ← median of each group of 5. 22 44 52 11 53 12 13 43 20 4 27 p ← MOM-SELECT(B, ⎣n / 10⎦) median of medians
28 23 6 26 40 19 1 46 31 49 8 (L, M, R) ← PARTITION-3-WAY(A, p).
IF (k ≤ | L |) RETURN MOM-SELECT(L, k). 14 9 5 3 54 30 48 47 32 51 21 ELSE IF (k > | L | + | M |) RETURN MOM-SELECT(R, k – | L | – | M |)
45 39 50 15 25 16 41 17 22 7 ELSE RETURN p.
______
n = 54 49 50 Analysis of median-of-medians selection algorithm Analysis of median-of-medians selection algorithm
At least half of 5-element medians ≤ p. At least half of 5-element medians ≤ p. At least ⎣⎣n / 5⎦ / 2⎦ = ⎣n / 10⎦ medians ≤ p.
medians medians ≤ p median of median of medians p 29 10 38 37 2 55 18 24 34 35 36 medians p 29 10 38 37 2 55 18 24 34 35 36
22 44 52 11 53 12 13 43 20 4 27 22 44 52 11 53 12 13 43 20 4 27
28 23 6 26 40 19 1 46 31 49 8 28 23 6 26 40 19 1 46 31 49 8
14 9 5 3 54 30 48 47 32 51 21 14 9 5 3 54 30 48 47 32 51 21
45 39 50 15 25 16 41 17 22 7 45 39 50 15 25 16 41 17 22 7
n = 54 51 n = 54 52
Analysis of median-of-medians selection algorithm Analysis of median-of-medians selection algorithm
At least half of 5-element medians ≤ p. At least half of 5-element medians ≥ p. At least ⎣⎣n / 5⎦ / 2⎦ = ⎣n / 10⎦ medians ≤ p. At least 3 ⎣n / 10⎦ elements ≤ p.
medians ≤ p medians median of median of medians p 29 10 38 37 2 55 18 24 34 35 36 medians p 29 10 38 37 2 55 18 24 34 35 36
22 44 52 11 53 12 13 43 20 4 27 22 44 52 11 53 12 13 43 20 4 27
28 23 6 26 40 19 1 46 31 49 8 28 23 6 26 40 19 1 46 31 49 8
14 9 5 3 54 30 48 47 32 51 21 14 9 5 3 54 30 48 47 32 51 21
45 39 50 15 25 16 41 17 22 7 45 39 50 15 25 16 41 17 22 7
n = 54 53 n = 54 54 Analysis of median-of-medians selection algorithm Analysis of median-of-medians selection algorithm
At least half of 5-element medians ≥ p. At least half of 5-element medians ≥ p. At least ⎣⎣n / 5⎦ / 2⎦ = ⎣n / 10⎦ medians ≥ p. At least ⎣⎣n / 5⎦ / 2⎦ = ⎣n / 10⎦ medians ≥ p. At least 3 ⎣n / 10⎦ elements ≥ p.
medians ≥ p medians ≥ p median of median of medians p 29 10 38 37 2 55 18 24 34 35 36 medians p 29 10 38 37 2 55 18 24 34 35 36
22 44 52 11 53 12 13 43 20 4 27 22 44 52 11 53 12 13 43 20 4 27
28 23 6 26 40 19 1 46 31 49 8 28 23 6 26 40 19 1 46 31 49 8
14 9 5 3 54 30 48 47 32 51 21 14 9 5 3 54 30 48 47 32 51 21
45 39 50 15 25 16 41 17 22 7 45 39 50 15 25 16 41 17 22 7
n = 54 55 n = 54 56
Median-of-medians selection algorithm recurrence Median-of-medians selection algorithm recurrence
Median-of-medians selection algorithm recurrence. Median-of-medians selection algorithm recurrence. Select called recursively with ⎣n / 5⎦ elements to compute MOM p. Select called recursively with ⎣n / 5⎦ elements to compute MOM p. At least 3 ⎣n / 10⎦ elements ≤ p. At least 3 ⎣n / 10⎦ elements ≤ p. At least 3 ⎣n / 10⎦ elements ≥ p. At least 3 ⎣n / 10⎦ elements ≥ p. Select called recursively with at most n – 3 ⎣n / 10⎦ elements. Select called recursively with at most n – 3 ⎣n / 10⎦ elements.
Def. C(n) = max # compares on any array of n elements. Def. C(n) = max # compares on any array of n elements.
C(n) C ( n/5 )+C (n 3 n/10 )+ 11 n C(n) C ( n/5 )+C (n 3 n/10 )+ 11 n 5 5 median of recursive computing median of 5 median of recursive computing median of 5 medians select (≤ 6 compares per group) medians select (≤ 6 compares per group) partitioning partitioning (≤ n compares) (≤ n compares)
Intuition. Now, let’s solve given recurrence. C(n) is going to be at least linear in n ⇒ C(n) is super-additive. Assume n is both a power of 5 and a power of 10 ? Ignoring floors, this implies that C(n) ≤ C(n / 5 + n − 3n / 10) + 11/5 n Prove that C(n) is monotone non-decreasing. = C(9n / 10) + 11/5 n C(n) ≤ 22n. ⇒ 57 58 Divide-and-conquer: quiz 4 Median-of-medians selection algorithm recurrence
Consider the following recurrence Analysis of selection algorithm recurrence. T(n) = max # compares on any array of ≤ n elements. 0 n 1 T(n) is monotone non-decreasing, but C(n) is not! C(n)= C( n/5 )+C(n 3 n/10 )+ 11 n n>1
Claim. T(n) ≤ 44 n. A. Yes, obviously. Pf. [ by strong induction ] B. Yes, but proof is tedious. Base case: T(n) ≤ 6 n for n < 50 (mergesort). C. Yes, but proof is hard. Inductive hypothesis: assume true for 1, 2, …, n – 1. Induction step: for n ≥ 50, we have either T(n) ≤ T(n – 1) ≤ 44 n or D. No. T(n) ≤ T(⎣n / 5⎦) + T(n – 3 ⎣n / 10⎦) + 11/5 n inductive ≤ 44 (⎣n / 5⎦) + 44 (n – 3 ⎣n / 10⎦) + 11/5 n C(19) = 165 hypothesis C(20) = 134.2 ≤ 44 (n / 5) + 44 n – 44 (n / 4) + 11/5 n for n ≥ 50, 3 ⎣n / 10⎦ ≥ n / 4 = 44 n. ▪ 59 60
Divide-and-conquer: quiz 5 Linear-time selection retrospective
Suppose that we divide n elements into ⎣n / r⎦ groups of r elements each, Proposition. [Blum–Floyd–Pratt–Rivest–Tarjan 1973] There exists a and use the median-of-medians of these ⎣n / r⎦ groups as the pivot. compare-based selection algorithm whose worst-case running time is O(n). JOURNAL OF COMPUTER AND SYSTEM SCIENCES 7, 448-461 (1973) For which r is the worst-case running time of select O(n) ?
Time Bounds for Selection*
MANUEL BLUM, ROBERT W. FLOYD, VAUGHAN PRATT, A. r = 3 T(n) = T(n / 3) + T(n − 2n / 6) + Θ(n) ⇒ T(n) = Θ(n log n) RONALD L. RIVEST, AND ROBERT E. TARJAN Department of Computer Science, Stanford University, Stanford, California 94305 B. r = 7 T(n) = T(n / 7) + T(n − 4n / 14) + Θ(n) ⇒ T(n) = Θ(n) Received November 14, 1972
C. Both A and B. The number of comparisons required to select the i-th smallest of n numbers is shown to be at most a linear function of n by analysis of a new selection algorithm--PICK. Specifically, no more than 5.4305 n comparisons are ever required. This bound is D. Neither A nor B. improved for extreme values of i, and a new lower bound on the requisite number of comparisons is also proved.
1. INTRODUCTION
Theory. In this paper we present a new selection algorithm, PICK, and derive by an analysis of its efficiency the (surprising) result that the cost of selection is at most a linear function of the number of input items. In addition, we prove a new lower bound Optimized versionfor the costof of BFPRT:selection. ≤ 5.4305 n compares. The selection problem is perhaps best exemplified by the computation of medians. Upper bound: [Dor–ZwickIn general, we may wish 1995] to select the i-th≤ smallest2.95 ofn a compares.set of n distinct numbers, or the element ranking closest to a given percentile level. Lower bound: [Dor–ZwickInterest in this problem 1999] may be traced ≥ to(2 the + realm 2− 80 of )sports n compares.and the design of (traditionally, tennis) tournaments to select the first- and second-best players. In 1883, Lewis Carroll published an article [1] denouncing the unfair method by which the second-best player is usually determined in a "knockout tournament" -- the loser of the final match is often not the second-best! (Any of the players who lost only Practice. Constantsto toothe best largeplayer may tobe second-best.) be useful. Around 1930, Hugo Steinhaus brought the 61 problem into the realm of algorithmic complexity by asking for the minimum number 62 of matches required to (correctly) select both the first- and second-best players from a field of n contestants. In 1932, J. Schreier [8] showed that no more than n + [logg(n)]- 2 matches are required, and in 1964, S. S. Kislitsin [6] proved this number to be necessary as well. Schreier's method uses a knockout tournament to determine the winner, followed by a second knockout tournament among the
* This work was supported by the National Science Foundation under grant GJ-992. 448 Copyright © I973 by Academic Press, Inc. All rights of reproduction in any form reserved. Closest pair of points
Closest pair problem. Given n points in the plane, find a pair of points 5. DIVIDE AND CONQUER with the smallest Euclidean distance between them.
‣ mergesort Fundamental geometric primitive. ‣ counting inversions Graphics, computer vision, geographic information systems, molecular modeling, air traffic control. ‣ randomized quicksort Special case of nearest neighbor, Euclidean MST, Voronoi. ‣ median and selection fast closest pair inspired fast algorithms for these problems ‣ closest pair of points
SECTION 5.4
64
Closest pair of points Closest pair of points: first attempt
Closest pair problem. Given n points in the plane, find a pair of points Sorting solution. with the smallest Euclidean distance between them. Sort by x-coordinate and consider nearby points. Sort by y-coordinate and consider nearby points. Brute force. Check all pairs with Θ(n2) distance calculations.
1D version. Easy O(n log n) algorithm if points are on a line.
Non-degeneracy assumption. No two points have the same x-coordinate.
65 66 Closest pair of points: first attempt Closest pair of points: second attempt
Sorting solution. Divide. Subdivide region into 4 quadrants. Sort by x-coordinate and consider nearby points. Sort by y-coordinate and consider nearby points.
8
67 68
Closest pair of points: second attempt Closest pair of points: divide-and-conquer algorithm
Divide. Subdivide region into 4 quadrants. Divide: draw vertical line L so that n / 2 points on each side. Obstacle. Impossible to ensure n / 4 points in each piece. Conquer: find closest pair in each side recursively. Combine: find closest pair with one point in each side. Return best of 3 solutions. seems like Θ(n2)
L
21 8
12
69 70 How to find closest pair with one point in each side? How to find closest pair with one point in each side?
Find closest pair with one point in each side, assuming that distance < δ. Find closest pair with one point in each side, assuming that distance < δ. Observation: suffices to consider only those points within δ of line L. Observation: suffices to consider only those points within δ of line L. Sort points in 2 δ-strip by their y-coordinate. Check distances of only those points within 7 positions in sorted list!
why?
L L
7
6
5 21 4 21
δ = min(12, 21) δ = min(12, 21) 12 12 3
2
1 δ 71 δ 72
How to find closest pair with one point in each side? Closest pair of points: divide-and-conquer algorithm
th Def. Let si be the point in the 2 δ-strip, with the i smallest y-coordinate.
CLOSEST-PAIR(p1, p2, …, pn) Claim. If ⎜j – i⎟ > 7, then the distance between
______s and s is at least δ. L i j Compute vertical line L such that half the points O(n) are on each side of the line. Pf. δ1 ← CLOSEST-PAIR(points in left half). T(n / 2) Consider the 2δ-by-δ rectangle R in strip sj δ2 ← CLOSEST-PAIR(points in right half). T(n / 2) whose min y-coordinate is y-coordinate of si. δ ← min { δ1 , δ2 }. Distance between si and any point sj R ½δ above R is ≥ δ. Delete all points further than δ from line L. O(n) diameter is δ / 2 < O(n log n) Subdivide R into 8 squares.
constant can be improved with more RETURN δ.
______refined geometric packing argument ______
2δ 73 74 Divide-and-conquer: quiz 6 Refined version of closest-pair algorithm
What is the solution to the following recurrence? Q. How to improve to O(n log n) ? A. Don’t sort points in strip from scratch each time. Each recursive call returns two lists: all points sorted by x-coordinate, (1) n =1 and all points sorted by y-coordinate. T (n)= T ( n/2 )+T ( n/2 )+ (n log n) n>1 Sort by merging two pre-sorted lists.
A. T(n) = Θ(n). Theorem. [Shamos 1975] The divide-and-conquer algorithm for finding a B. T(n) = Θ(n log n). closest pair of points in the plane can be implemented in O(n log n) time.
C. T(n) = Θ(n log2 n). (1) n =1 Pf. T (n)= D. T(n) = Θ(n2). T ( n/2 )+T ( n/2 )+ (n) n>1
75 76
Divide-and-conquer: quiz 7 Computational complexity of closest-pair problem
What is the complexity of the 2D closest pair problem? Theorem. [Ben-Or 1983, Yao 1989] In quadratic decision tree model, any algorithm for closest pair (even in 1D) requires Ω(n log n) quadratic tests.
A. Θ(n). 2 2 (x1 - x2) + (y1 - y2) B. Θ(n log* n). C. Θ(n log log n).
D. Θ(n log n).
E. Not even Tarjan knows. Theorem. [Rabin 1976] There exists an algorithm to find the closest pair of points in theVolume 8,plane number 1 whoseIhFCRMATION expected PROCESSING running LE,TTERS time isJanuary O( n1979) .
A NOTE ON RABIN’S NEAREST-NEIGHBOR ALGORITHM * not subject to Ω(n log n) lower bound because it uses the floor function Steve FORTUNE and John HOPCROFT Department of Computer Science, Cornell University,Ithaca, NY, U.S.A.
Received 20 July 1978, revised version received 21 August 1978
Probabiktic algorithms, nearest neighbor, hashing
77 78
9. introduction tion to the distance between nearest neighbors in the whole set. Using this approximation the distance be- One notion that has received some attention tween the nearest neighbor in the whole set can be recently is that of a probabilistic algorithm. An algo- found in expected linear time. The algorithm’s use of rithm is probabilistic if at certain steps it chooses a randomness in choosing the subset is crucial. number randomly to determine the next step and at Rabin’s algorithm also assumes constant time all other steps it is deterministic. The expected run- arithmetic operations. In partictilar, he assumes that ning time of such an algorithm on a given input is a special operation, described below, which is similar obtained by averaging over all possible random to hashing can be performed in constant expected choices of the algorithm on that input. The running time. It follows from [3] that the special operation time of the algorithm as a whole can be measured can indeed be implemented by a probabilistic algo- either by averaging the expected running time over rithm to run in constant expected time given that all inputs or by taking the worst case of the expected evaluating a hash function takes constant time. running time. In [l] the former analysis is used; in The fast algorithms in [ 11, [4], and [6] all have [4] and [6] the latter analysis is used. the property that for some sequence of random Rabin [4] has proposed a probabilistic algorithm choices an incorrect answer could result. Rabin’s algo- for finding the closest pair of a set of points in rithm is apparently the only known example other Euclidean space. The running time of his algorithm than hashing of an error-free probabilistic algorithm using the worst-case expected-time measure is linear. which runs faster than the deterministic equivalent. This compares with time O(n log II) for the best of In this paper we present an algorithm to find the the known determinis?ic algorithms [or this problem nearest neighbor which runs in time O(n loglog n). j2,5,7]. The algorithm does not make any random choices; Rabin’s algc-rithm works by randomly choosing a however it does assume that the special operation subset of the points and recursively using the algo- uses only constant time. The conclusion we reach is rithm to find the distance between the nearest neigh- that mclst of the speedup of Rabin’s algorithm is due bors of the subset. Ra.bin is able to show that, with to the hashing, and not to the probabilistic nature of very high probability, tht; distance between the near- the basic algorithm. It would be interesting to find est neighbors of the subset is a very good ;Ipproxima- examples of probabilistic algorithms that are faster * Research was wpported under grant number ONR N00014, than their deterministic counterparts and for which 76-C-lrO18. the speed does noi come from the capability to hash.
20 Digression: computational geometry Convex hull
Ingenious divide-and-conquer algorithms for core geometric problems. The convex hull of a set of n points is the smallest perimeter fence enclosing the points.
problem brute clever
closest pair O(n2) O(n log n)
farthest pair O(n2) O(n log n)
convex hull O(n2) O(n log n)
Delaunay/Voronoi O(n4) O(n log n)
Euclidean MST O(n2) O(n log n)
running time to solve a 2D problem with n points Equivalent definitions. Smallest area convex polygon enclosing the points. Note. 3D and higher dimensions test limits of our ingenuity. Intersection of all convex set containing all the points.
79 80
Farthest pair Delaunay triangulation
Given n points in the plane, find a pair of points with the largest Euclidean The Delaunay triangulation is a triangulation of n points in the plane distance between them. such that no point is inside the circumcircle of any triangle.
no point in the set is inside the circumcircle
point inside circumcircle of 3 points
Delaunay triangulation of 19 points
Some useful properties. Fact. Points in farthest pair are extreme points on convex hull. No edges cross. Among all triangulations, it maximizes the minimum angle. Contains an edge between each point and its nearest neighbor. 81 82 Euclidean MST Computational geometry applications
Given n points in the plane, find MST connecting them. Applications. [distances between point pairs are Euclidean distances] Robotics. VLSI design. Data mining. Medical imaging. Computer vision. Scientific computing. airflow around an aircraft wing Finite-element meshing. Astronomical simulation. Models of physical world. Geographic information systems. Computer graphics (movies, games, virtual reality).
http://www.ics.uci.edu/~eppstein/geom.html Fact. Euclidean MST is subgraph of Delaunay triangulation. Implication. Can compute Euclidean MST in O(n log n) time. Compute Delaunay triangulation. Compute MST of Delaunay triangulation. it’s planar (≤ 3n edges) 83 84