Algorithms III
1 Agenda
Sorting • Simple sorting algorithms • Shellsort • Mergesort • Quicksort • A lower bound for sorting • Counting sort and Radix sort
Randomization • Random number generation • Generation of random permutations • Randomized algorithms
2 Sorting
Informal definition: Sorting is any process of arranging items in some sequence.
Why sort?
(1) Searching a sorted array is easier than searching an unsorted array. This is especially true for people. Example: Looking up a person in a telephone book.
(2) Many problems may be solved more efficiently when input is sorted. Example: If two files are sorted it is possible to find all duplicates in only one pass.
3 Detecting duplicates in an array
If the array is sorted, duplicates can be detected by a linear-time scan of the array:
for (int i = 1; i < a.length; i++) if (a[i - 1].equals(a[i]) return true; 4 Finding the intersection of two unsorted arrays
0 i M a:
0 j N b: 0 k c:
k = 0; for (i = 0; i < M; i++) for (j = 0; j < N; j++) if (a[i].equals(b[j])) c[k++] = a[i];
Complexity: O(MN) 5 Finding the intersection of two sorted arrays
0 i M a:
0 j N b: 0 k c:
i = j = k = 0; while (i < M && j < N) if (a[i] < b[j]) i++; else if (a[i] > b[j]) j++; else { c[k++] = a[i]; i++; j++; }
Complexity: O(M + N) 6 Insertion sort
Problem: Given an array a of n elements a[0], a[1], ..., a[n-1]. Sort the array in increasing order.
Solution (by induction):
Base case: We know how to sort 1 element. Induction hypothesis: We know how to sort n-1 elements.
We can sort an array of n elements by (1) sorting its first n-1 elements, (2) insert the n’th element into its proper place among the previously sorted elements
7 Insertion sort (recursive version)
void insertionSort(int[] a, int n) { if (n > 1) { insertionSort(a, n - 1); int tmp = a[n]; int j = n; for ( ; j > 0 && a[j - 1] > tmp; j--) a[j] = a[j - 1]; a[j] = tmp; } }
Single right shift
8 9 10 A S O R T I N G E X A M P L E A S O R T I N G E X A M P L E A O S R T I N G E X A M P L E A O R S T I N G E X A M P L E A O R S T I N G E X A M P L E A I O R S T N G E X A M P L E A I N O R S T G E X A M P L E A G I N O R S T E X A M P L E A E G I N O R S T X A M P L E A E G I N O R S T X A M P L E A A E G I N O R S T X M P L E A A E G I M N O R S T X P L E A A E G I M N O P R S T X L E A A E G I L M N O P R S T X E A A E E G I L M N O P R S T X
11 Insertion sort (non-recursive version)
Only the object references are right-shifted (not the objects themselves)
12 Animation of Insertion sort
a[i]
i
Sorting the integers from 1 to 100.
13 Analysis of Insertion sort
Number of comparisons: Best case: n-1 Worst case: 1 + 2 + ... + (n-1) = n(n-1)/2, which is O(n2) Average case: n(n-1)/4, which is O(n2)
Number of moves: Best case: 0 Worst case: 1 + 2 + ... + (n-1) = n(n-1)/2, which is O(n2) Average case: n(n-1)/4, which is O(n2)
Running time is O(n) for “almost sorted” arrays
14 Shellsort Donald Shell, 1959
Idea: Insertion sort is very efficient when the array is “almost sorted”. However, it is slow for “very unsorted” arrays since it only allows exchange of neighboring elements.
Question: Can we start by exchanging elements that are far apart from each other in the array before we do an ordinary Insertion sort? Answer: Yes. We can start by sorting all subfiles consisting of every h’th element of the array, where h > 1.
15 4-sorting
1. Divide the array into 4 subfiles:
each 4’th element, starting in the first, each 4’th element, starting in the second, each 4’th element, starting in the third, and each 4’th element, starting in the fourth.
2. Sort each of these four subfiles.
The array is then said to be 4-sorted. Similarly, we can define h-sorted.
Notice that an array that is 1-sorted is sorted.
16 4-sorting Use Insertion sort with an increment of 4 A S O R T I N G E X A M P L E A S O R T I N G E X A M P L E A I O R T S N G E X A M P L E A I N R T S O G E X A M P L E A I N G T S O R E X A M P L E A I N G E S O R T X A M P L E A I N G E S O R T X A M P L E A I A G E S N R T X O M P L E A I A G E S N M T X O R P L E A I A G E S N M P X O R T L E A I A G E L N M P S O R T X E A I A G E L E M P S N R T X O
17 Implementation of h-sort
public static
The code is obtained by replacing 1 by h in insertionSort.
18 Shellsort
Shellsort uses h-sorting for a decreasing sequence of h-values, ending with h = 1.
void shellsort(AnyType[] a) { for (int h = a.length / 2; h > 0; h = h == 2 ? 1 : (int) (h / 2.2)) h_sort(a, h); }
19 20 In the figure, elements spaced five and three apart are identically shaded.
21 Animation of Shellsort
22 (milliseconds)
Shell’s increments: Start gaps at n/2 and halve until 1 is reached. Odd Gaps Only: As Shell’s increments, but add 1 whenever the gap becomes even. 23 Analysis of Shellsort
Number of comparisons (for the increment sequence 1, 4, 13, 40,...):
Best case: (n-1) + (n-4) + (n-13) + ... ≤ n log3n Worst case: less than n1.5 Average case: not known Two suggestions are O(n1.25) and O(n(log n)2)
Number of moves: Best case: 0 Worst case: as for comparisons Average case: as for comparisons
24 Bubble Sort
Repeatedly swap adjacent elements if they are in wrong order.
boolean swapped; do { swapped = false; for (int i = 1; i < a.length; i++) { if (a[i - 1] > a[i]) { swap(a, i - 1, i); swapped = true; } } } while (swapped);
Worst-case and average complexity: O(n2) Best case complexity: O(n)
25 Improved Bubble Sort
All elements after the last swap are sorted, and do not need to be checked again. int n = a.length; do { int newn = 0; for (int i = 1; i < n; i++) { if (a[i - 1] > a[i]) { swap(a, i - 1, i); newn = i; } } n = newn; } while (n > 0);
Worst-case and average complexity: O(n2) Best case complexity: O(n)
26 Mergesort
Mergesort is a divide-and-conquer based sorting algorithm.
The algorithm involves three steps:
1. If the number of items to sort is 0 or 1, then return.
2. Recursively sort the first and second halves separately.
3. Merge the two sorted halves into one sorted group.
27 Mergesort
void mergeSort(Comparable[] a, int low, int high) { if (low < high) { int mid = (low + high) / 2; mergesort(a, low, mid); mergesort(a, mid + 1, high); merge(a, low, mid, high); } }
≤ ≤
low mid high 28 Linear-time merging of two sorted arrays
0 i M a: ≤ ≤
0 j N b: ≤ ≤
0 k M+N c: ≤ for (i = j = k = 0; k < M + N; k++) if (i == M) c[k] = b[j++]; else if (j == N) c[k] = a[i++]; else c[k] = a[i].compareTo(b[j]) < 0 ? a[i++] : b[j++];
29 Merging a[low:mid] with a[mid+1:high] to a[low:high]:
a: ≤ ≤ low mid mid+1 high Merge into b: b: ≤ low high Copy b to a: a: ≤ low high void merge(Comparable[] a, int low, int mid, int high) { int i = low, j = mid + 1; for (int k = low; k <= high; k++) if (i > mid) b[k] = a[j++]; else if (j > high) b[k] = a[i++]; else b[k] = a[i].compareTo(a[j]) < 0 ? a[i++] : a[j++]; for (int k = low; k <= high; k++) a[k] = b[k]; }
30 31 32 A S O R T I N G E X A M P L E A S O R T I N G E X A M P L E A S O R T I N G E X A M P L E A O R S T I N G E X A M P L E A O R S I T N G E X A M P L E A O R S I T G N E X A M P L E A O R S G I N T E X A M P L E A G I N O R S T E X A M P L E A G I N O R S T E X A M P L E A G I N O R S T E X A M P L E A G I N O R S T A E M X P L E A G I N O R S T A E M L L P E A G I N O R S T A E M L E L P A G I N O R S T A E E L M P X A A E E G I L M N O P R S T X
33 Animation of Mergesort
34 Call trees
16
8 8
4 4 4 4
2 2 2 2 2 2 2 2
8 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
21
10 11
5 5 5 6
2 3 2 3 2 3 3 3
1 1 1 2 1 1 1 2 1 1 1 2 1 2 1 2
1 1 1 1 1 1 1 1 1 1
35 Call tree
There are about log2N levels Each level uses O(N) time
36 Evaluation of Mergesort
Advantages: • insensitive to the input order
• requires about N log2N comparisons for any array of N elements C(N) = 2C(N/2) + N, C(1) = 0 • is stable (preserves the input order of equal elements) • can be used for sorting linked lists • is suitable for external sorting
Disadvantage: • requires (in practice) extra memory proportional to N
37 Quicksort C. A. R. Hoare, 1962
Quicksort is in practice the fastest comparison-based algorithm for sorting arrays.
Idea: To sort an array, partition it into a left and a right part, such that all elements in the left part are less than or equal to all elements in the right part. ≤ left part right part
Then, recursively sort the left part and the right part.
38 Basic Quicksort
The basic algorithm Quicksort(S) consists of the following four steps:
1. If the number of elements in S is 0 or 1, then return.
2. Pick any element v in S. It is called the pivot.
3. Partition S - {v} (the remaining elements in S) into two disjoint groups L = {x ∈S − {v} | x ≤ v} and R = {x ∈S − {v} | x ≥ v}.
4. Return the result of Quicksort(L) followed by v followed by Quicksort(R).
39 40 Partitioning
The partitioning of an array a may be performed as follows:
(1) Choose a pivot v among the elements in a. (2) Traverse a from left to right until an element a[i] ≥ v is met. (3) Traverse a from right to left until an element a[j] ≤ v is met. (4) Swap a[i] and a[j]. (5) Continue traversing and swapping until the two traversals “cross”.
≤ v a[i] a[j] ≥ v ≥ v ≤ v
41 Implementation
Partitioning a[low:high] about the pivot v:
i = low; j = high; while (i <= j) { while (a[i].compareTo(v) < 0) i++; while (a[j].compareTo(v) > 0) j--; if (i <= j) { swap(a, i, j); i++; j--; } }
Result: a[low:i] ≤ a[j:high] and i > j.
May be proven by showing that a[low:i-1] ≤ v ≤ a[j+1:high] is invariant for the outermost loop.
42 Swap
void swap(Object[] a, int i, int j) { Object temp = a[i]; a[i] = a[j]; a[j] = temp; }
Only the object references are swapped (not the objects themselves)
43 Quicksort
The pivot v may be any element in a[low:high], for instance a[(low + high)/2].
void quicksort(Comparable[] a, int low, int high) { if (low < high) { Comparable v = a[(low + high)/2]; int i = low, j = high; while (true) { while (a[i].compareTo(v) < 0) i++; while (a[j].compareTo(v) > 0) j--; if (i >= j) break; swap(a, i++, j--); } quicksort(a, low, j); quicksort(a, i, high); } }
44 Alternative partitioning algorithm
Choose a[high] as pivot, partition a[low:high-1], and swap a[high] with a[i].
int partition(Comparable[] a, int low, int high) { Comparable v = a[high]; int i = low - 1, j = high; while (true) { while (a[++i].compareTo(v) < 0) ; while (v.compareTo(a[--j]) < 0) if (j == low) break; if (i >= j) break; swap(a, i, j); } swap(a, i, high); return i; } 45 Picking the pivot
46 j i
47 The method quicksort
int quicksort(Comparable[] a, int low, int high) { if (low < high) { int i = partition(a, low, high); quicksort(a, low, i - 1); quicksort(a, i + 1, high); } }
48 Animation of Quicksort
49 Number of comparisons
Let C(N) denote the number of comparisons required for executing Quicksort on an array of N elements.
Partitioning requires N comparisons. Next, the left part and the right part are sorted separately.
On average each part consists of about N/2 elements. Then we obtain the recurrence
C(N) = N + 2C(N/2) for N ≥ 2, C(0) = C(1) = 0,
which has the solution C(N) = N log2N.
50 Average number of comparisons
More precise computations give
Quicksort uses about 2N lnN comparisons on the average
where ln denotes the natural logarithm.
2N ln N ≈ 1.39 N log2N
So the average number of comparisons is only about 39% higher than in the best case.
51 Number of comparisons in the worst case
The worst case occurs when the partitioning for each N results in a part of one element and a part of N-1 elements.
In this case we have the recurrence
C(N) = N + C(N-1) for N ≥ 2, C(0) = C(1) = 0,
which has the solution C(N) = N(N+1)/2.
Picking the pivot at random or by the “median-of-three” method makes the worst case unlikely to occur.
The median of N numbers is the ⎢⎡ N / 2 ⎥⎤ th smallest number.
52 Median-of-three partitioning
53 Quicksort with median-of-three partitioning void quicksort(Comparable[] a, int low, int high) { if (low < high) { if (a[high].compareTo(a[low]) < 0) swap(a, low, high); int mid = (low + high) / 2; if (mid == low) return; if (a[mid].compareTo(a[low]) < 0) swap(a, low, mid); if (a[high].compareTo(a[mid]) < 0) swap(a, mid, high); swap(a, mid, high - 1); int i = partition(a, low + 1, high - 1); quicksort(a, low, i - 1); quicksort(a, i + 1, high); } } 54 Small array segments
Use a simple method for sorting small array segments.
The test in the beginning of quicksort:
if (low < high) is replaced by
if (high - low < CUTOFF) insertionSort(a, low, high); else where CUTOFF is in the range from 5 to 25.
55 56 Memory usage
The average extra space complexity of quicksort is O(log2N). This extra space comes from the call stack. Each recursive call will create a stack frame.
The worst case extra space complexity for a naive implementation is O(N). The worst case occurs when the partitioning for each N results in a part of one element and a part of N-1 elements, and we quicksort the largest part first.
Extra space complexity of O(log2N) is guaranteed if we always sort the array segment with the fewest elements first, and sort the other array segment using iteration (that is, eliminates the tail recursion).
57 Elimination of tail recursion
A subroutine is said to be tail-recursive, if it calls itself as its final action. Tail recursion can always be replaced by iteration.
Recursion (call: p(a)) Iteration
void p(type x) { type x = a; if (b(x)) while (!b(x)) { S1; S2; else { x = f(x); S2; } p(f(x)); S1; } }
58 int quicksort(Comparable[] a, int low, int high) { if (low < high) { int i = partition(a, low, high); quicksort(a, low, i - 1); quicksort(a, i + 1, high); } }
int quicksort(Comparable[] a, int low, int high) { while (low < high) { int i = partition(a, low, high); quicksort(a, low, i - 1); low = i + 1; } }
Smart compilers (but not Java) can detect tail recursion and convert it to iteration in order to optimize code
59 Selection
Problem: Find the k’th smallest element in an array of N elements
Example: The 3’rd smallest element in {3, 6, 5, 2, 8, 4} is 4.
Solution 1: Sort the array in increasing order. The k’th element in the sorted array is the solution to the problem.
Complexity: Depends on the sorting algorithm - using Mergesort: O(N log N).
Can we do it faster?
60 Selection by means of partition
void quickSelect(Comparable[] a, int low, int high, int k) { if (low < high) { int i = partition(a, low, high); if (k <= i) quickSelect(a, low, i - 1, k); else if (k > i + 1) quickSelect(a, i + 1, high, k); } }
Since only tail recursion is used, the recursion may be eliminated:
void quickSelect(Comparable a[], int low, int high, int k) { while (low < high) { int i = partition(a, low, high); if (k <= i) high = i - 1; else if (k >= i + 1) low = i + 1; } }
61 62 Complexity of quickSelect
O(N) on the average since N + N/2 + N/4 + N/8 + … ≤ 2N.
It is possible (but not quite easy) to achieve guaranteed linear running time.
63 A lower bound for sorting
Any sorting algorithm that sorts using comparisons must use at least ⎢⎡log(N ! ) ⎥⎤ comparisons for some input sequence.
Informal proof: Sorting is equivalent to finding a permutation of the input sequence. Thus, sorting may be modeled by a decision tree, where each internal node corresponds to a comparison, and each external node corresponds to one of the N! possible permutations. The height of the tree must at least be log2(N!).
How large is ⎢⎡ l og 2 ( N ! ) ⎥⎤ ?
N N From Stirling's formula N! ≈ 2πN( e ) we get log2(N!) ≈ Nlog2N - 1.44N.
64 Counting sort A linear sorting algorithm
Sort an array a of non-negative integers less than m.
void sort(int[] a, int m) { int[] count = new int[m]; for (int i = 0; i < a.length; i++) count[a[i]]++; for (int i = 0, j = 0; j < m; j++) for (int k = count[j]; k > 0; k--) a[i++] = j; }
65 Radix sort A linear sorting algorithm
Sort an array a of non-negative d-digit integers in radix (base) r. void sort(int[] a, int d, int r) { int[] count = new int[r]; int[] b = new int[a.length]; for (int pass = 0, pow = 1; pass < d; pass++, pow *= r) { for (int i = 0; i < r; i++) count[i] = 0; for (int i = 0; i < a.length; i++) count[a[i] / pow % r]++; for (int i = 1; i < r; i++) count[i] += count[i - 1]; for (int i = a.length - 1; i >= 0; i--) b[--count[a[i] / pow % r]] = a[i]; for (int i = 0; i < a.length; i++) a[i] = b[i]; } }
66 Radix-10 sort
31 41 59 26 53 58 97 23 93 84 a unsorted 0 1 2 3 4 5 6 7 8 9
0 2 0 3 1 0 1 1 1 1 count
0 2 2 5 6 6 7 8 9 10 count
31 41 23 53 93 84 26 97 58 59 a sorted by 1s
0 1 2 3 4 5 6 7 8 9
0 0 2 1 1 3 0 0 1 2 count
0 0 2 3 4 7 7 7 8 10 count
23 26 31 41 53 58 59 84 93 97 a sorted by 10s
0 1 2 3 4 5 6 7 8 9 (and 1s)
67 Randomization
68 Why do we need random numbers?
Many applications:
• Program testing (generation of random input data) • Sorting (e.g., determination of the pivot in Quicksort) • Simulation (e.g., generation of arrival times for bank customers) • Games (e.g., choice of opening moves in chess) • Randomized algorithms (e.g., primality testing)
69 Generation of random numbers
True randomness in a computer is impossible to achieve. Generally, it is sufficient to produce pseudorandom numbers, or numbers that appear to be random because they satisfy many of the properties of random numbers. A pseudorandom number generator must pass a number of statistical tests. One such generator is the linear congruential generator
(Lehmer, 1951) in which numbers X1, X2, ... are generated that satisfy
Xi+1 = AXi + C (mod M)
The initial value X0 is called the seed. A, C, and M must be chosen in such a way that the length of the sequence until a number is repeated (the period) becomes as large as possible. This happens when M is a large prime number.
70 Generation of random numbers in Java
Java provides the class java.util.Random.
public class Random { public Random(); public Random(long seed); public int nextInt(); public int nextInt(int n); public long nextLong(); public float nextFloat(); public double nextDouble(); public double nextBytes(byte[] bytes); public double nextGaussian(); public void setSeed(long seed); protected int next(int bits); }
71 Implementation of Lehmer's method public class Random { private long seed; private final static long multiplier = 0x5DEECE66DL; // = 25214903917 private final static long addend = 0xBL; // = 11 private final static long mask = (1L << 48) - 1;
public Random() { this(System.currentTimeMillis()); }
public Random(long seed) { this.seed = (seed ^ multiplier) & mask; }
public int nextInt() { return next(32); }
protected int next(int bits) { long nextseed = (seed * multiplier + addend) & mask; seed = nextseed; return (int) (nextseed >>> (48 - bits)); } }
72 Generation of random permutations
0≤j≤i i
a:
swap j = r.nextInt(i + 1)
void permute(Object[] a) { Random r = new Random(); for (int i = 1; i < a.length; i++) swap(a, i, r.nextInt(i + 1)); }
The algorithm runs in linear time. The number of different possible outcomes 2 . 3 . ... . N-1 . N is equal to the number of possible permutations, N!. 73 Primality testing by trial division
74 Randomized primality testing
P. de Fermat, 1601-65 Fermat’s little theorem: If P is prime and 0 < A < P, then AP-1 = 1(mod P)
If for a number N we can find a value 0 < A < N such that AN-1(mod N) is not 1, then N is not a prime. A is said to be a witness to N’s compositeness.
75 Finding a witness
Every composite number has a witness. But for some numbers it can be hard to find. The following theorem can be used to improve our chances of finding a witness.
Theorem: If P is prime and X2 = 1(mod P), then X = ±1(mod P)
Corollary: If X2 = 1(mod N) and X ≠ ±1(mod N), then N is not prime
76 Exponentiation Efficient algorithm If n is even, then n xn = (x ⋅ x)2 If n is odd, then ⎢n ⎥ ⎢ ⎥ xn = x ⋅ xn−1 = x ⋅(x ⋅ x)⎣2 ⎦
public static long power(long x, int n) { if (n == 0) return 1; long tmp = power(x * x, n / 2); if (n % 2 != 0) tmp *= x; return tmp; }
Number of multiplications < 2 log2n. 77 Modular exponentiation
78 Modular exponentiation
/** * Return a^i (mod n) * Assumes a, i >= 0, n > 0, a < n, 0^0 = 1 * Overflow may occur if n > 31 bits. */ public static long power( long a, long i, long n ) { if( i == 0 ) return 1;
long tmp = power( ( a * a ) % n, i / 2, n );
if( i % 2 != 0 ) tmp = ( tmp * a ) % n;
return tmp; }
79 80 81 Confidence of randomized primality testing
Some values of A will trick the algorithm into declaring that N is prime.
In fact, if we choose A randomly, we have at most ¼ chance of failing to detect a composite number.
However, if we independently use 20 values of A (TRIALS = 20), the chances that none of them will witness a composite number is ¼20, which is about 1 in a million million.
82