Next to storing and retrieving data, sorting of data is one of the more common algorithmic tasks, with many different ways to perform it. Whenever we perform a web search and/or view statistics at some website, the presented data has most likely been sorted in some way. In this lecture and in the following lectures we will examine several different ways of sorting. The following are some reasons for investigating several of the different algorithms (as opposed to one or two, or the “best” algorithm).
• There exist very simply understood algorithms which, although for large data sets behave poorly, perform well for small amounts of data, or when the range of the data is sufficiently small.
• There exist sorting algorithms which have shown to be more efficient in practice.
• There are still yet other algorithms which work better in specific situations; for example, when the data is mostly sorted, or unsorted data needs to be merged into a sorted list (for example, adding names to a phonebook).
1 Counting Sort
Counting sort is primarily used on data that is sorted by integer values which fall into a relatively small range (compared to the amount of random access memory available on a computer). Without loss of generality, we can assume the range of integer values is [0 : m], for some m ≥ 0. Now given array a[0 : n − 1] the idea is to define an array of lists l[0 : m], scan a, and, for i = 0, 1, . . . , n − 1 store element a[i] in list l[v(a[i])], where v is the function that computes an array element’s sorting value. The sorted list can then be obtained by scanning the lists of l one-by-one in increasing order, and placing the encountered objects in a final list. Both steps require Θ(n) steps.
Counting Sort is most commonly used on an array a of integers. One reason for this is that objects of a general type are often sorted on very large integers, which makes Counting Sort infeasible. For example, if an array of Employees are sorted based on social security number, then these values range in the tens of millions.
When using an array of integers, l can be replaced by an integer array f, where f[i] represents the frequency of the number of elements of a that are equal to i.
Example 1. Perform Counting Sort one the elements 9, 3, 3, 10, 5, 10, 3, 4, 9, 10, 1, 3, 5, 2, 4, 9, 9.
Radix Sort can be applied to an array of integers for which each integer is represented by k bits, and the time needed to access a single bit is O(1). The algorithm works in k stages. At the beginning of stage i, 1 ≤ i ≤ n, it is assumed that the integers are stored in some array. The elements are then scanned one-by-one, with elements having i th least significant bit equal to j (j = 0, 1) being placed in array bj (keeping the same order as in a). The round ends by rewriting a as the elements of b0 followed by the elements of b1.
Assuming that k is held constant, the complexity of radix sort is Θ(kn) = Θ(n).
Example 2. Perform Radix Sort on the elements 9, 13, 10, 5, 10, 3, 4, 9, 1, 5, 2.
Insertion Sort represents a doubly iterative way of sorting data in an array.
• Step 1. sort the first item in the array.
• Step i + 1: assume the first i elements of the array are sorted. Move item i + 1 left to its appropriate location so that the first i + 1 items are sorted.
Example 3. Use insertion sort on the array 43, 6, 72, 50, 44, 36, 21, 32, 47.
4 Code for Insertion Sort Applied to an Array of Integers:
//Sorting in place the elements a[left:right]. void insertion_sort(int[] array, int left, int right) { int i, j,tmp;
//Attempt to move element i to the left. for(i=left+1; i <= right; i++) { //Move left until finding the proper location of a[i]. for(j=i; j > left; j--) { if(a[j] < a[j-1]) { tmp = a[j-1]; a[j-1] = a[j]; a[j] = tmp; } else break; //found the right location for a[i] } } }
5 Average-Case Running Time of an Algorithm
When analyzing the running time of an algorithm, sometimes we are interested in its average-case running time, denoted by Tave(n), where Tave(n) is the average of all the algorithm’s running times over instances having size n. We now calculate the big-O average-case running time for Insertion Sort. We do this by assuming that each problem of size n is an array of n integers, where the integers are a permutation of the numbers 1, . . . , n. Recall that an n-permutation is simply an ordered arrangement of the first n positive integers. In other words, if σ is an n-permutation, then we write
σ = (σ(1) σ(2) ··· σ(n)), where σ(i) and σ(j) are positive integers less than or equal to n, and distinct if and only if i 6= j.
Example 4. Provide the permutation associated with the unsorted array of Example 3.
An inversion of a permutation σ is a pair (i, j) such that i < j and σ(j) < σ(i). For example, the permutation (1 4 5 3 6 2) has six inversions.
6 Observation. Every successful comparison of insertion sort reduces the number of inversions by 1. Thus, the number of steps (comparisons) needed for insertion sort is directly proportional to the number of inversions possessed by the array.
n(n−1) Theorem 1. The average number of inversions possessed by a random permutation is 4 .
n Proof of Theorem 1. Note that there are possible inversions of a permutation over n 2 numbers, and each inversion has a probability of 0.5 of appearing in a randomly generated permuta- n(n−1) tion of the numbers 1, . . . , n. Thus, we would expect a random permutation to possess 4 such inversions.
Corollary 1. The average-case running time for insertion sort is O(n2).
n(n−1) Proof of Corollary 1. Since a swap removes only one inversion, and on average there are 4 = O(n2) inversions, it follows that the average-case running is O(n2).
7 Divide and Conquer Algorithms
There exist many problems that can be solved using a divide-and-conquer algorithm. A divide-and- conquer algorithm A follows these general guidelines.
Divide Algorithm A divides original problem into one or more subproblems of a smaller size.
Conquer Each subproblem is solved by making a recursive call to A.
Combine Finally, A combines the subproblem solutions into a final solution to the original problem.
Some problems that can be solved using a divide-and-conquer algorithm:
Binary Search locating an element in a sorted array
Quicksort and Mergesort sorting an array
Order Statistics finding the k th least or greatest element of an array
Geometric Algorithms finding the convex hull of a set of points; finding two points that are closest.
Matrix Operations matrix inversion, Fast-Fourier Transform, matrix multiplication, finding the largest submatrix of 1’s in a Boolean matrix.
Maximum Subsequence Sum finding the maximum sum of any subsequence in a sequence of integers.
Minimum Positive Subsequence Sum finding the minimum positive sum of any subsequence in a sequence of integers.
Multiplication finding the product of two numbers.
8 Example 5. Using x = 5 and array
a = 1, 3, 3, 4, 6, 8.8.9.9.10, 11, 15, demonstrate how the Binary Search algorithm can be viewed as a divide-and-conquer algorithm.
9 Mergesort
The Mergesort algorithm is a divide-and-conquer algorithm for sorting an array of comparable elements. The algorithm begins by checking if input array a has two or fewer elements. If so, then the a is sorted in place by making at most one swap. Otherwise, a is divided into two (almost) equal halves aleft and aright. Both of these subarrays are sorted by making recursive calls to Mergesort. Once sorted, a merge operation merges the elements of aleft and aright into an auxiliary array. This sorted auxiliary array is then copied over to the original array.
Example 6. Demonstrate the Mergesort algorithm using the array 5, 8, 6, 2, 7, 1, 0, 9, 3, 4, 6.
Theorem 2. Mergesort has a running time of T (n) = Θ(n log n).
Proof Theorem 2. Let a be an array input of size n. The depth of Mergesort’s recursion tree is Θ(log n). Moreover, for each element x of a, and for each depth i of the recursion tree, there is a sub-array a0 at depth i containing x, and for which the merge operation is applied. During this operation, Θ(1) steps are applied to x, For a total of Θ(n) total steps applied to all elements at depth i. Therefore, the algorithm’s total steps is Θ(n)Θ(log n) = Θ(n log n).
10 Quicksort
Before introducing the Quicksort algorithm, recall that the median of an array a of n numbers a[0], . . . , a[n − 1] is the (n + 1)/2 least element of a, if n is odd, and is equal to either the n/2 or n/2 + 1 least element of a if n is even (even-length arrays have two medians).
Example 7. Determine the median of 7, 5, 7, 3, 4, 8, 2, 3, 7, 8, 2, and the medians of 4, 5, 10, 12, 6, 3.
Quicksort is considered in practice to be the most efficient sorting algorithm for arrays of data stored in local memory. Quicksort is similar to Mergesort in that the first (non base case) step is to divide the input array a into two arrays aleft and aright. However, where as Mergesort simply divides a into two equal halves, Quicksort performs the Partitioning Algorithm on a which is described below.
11 Partitioning Algorithm
Calculate Pivot The pivot M is an element of a which is used to divide a into aleft and aright. Namely all elements x ∈ aleft satisfy x ≤ M, while all elements x ∈ aright satisfy x ≥ M. A common heuristic for computing M is called median-of-three, where M is chosen as the median of the first, last, and middle elements of a; i.e. median(a[0], a[n/2], a[n − 1]).
Swap Pivot Swap the pivot element with the last element of a.
Initialize Markers Initialize a left marker to point to a[0]. Initialize a right marker to point to a[n − 2]. Let i = 0 denote the current index location of the left marker, and j = n − 2 denote the current index location of the right marker.
Examine Markers Execute one of the following cases.
• If i ≥ j, then swap a[i] with M = a[n − 1]. In this case aleft consists of the first i elements of a, while aright consists of the last n − i − 1 elements of a.Thus, a[i] = M is to the right of aleft and to the left of aright. • Else if a[i] ≥ M and a[j] ≤ M, then swap a[i] with a[j], increment i, and decrement j. • Else increment i if a[i] < M and/or decrement j if a[j] > M
Repeat Re-examine markers until i ≥ j.
Once the Partitioning algorithm has partitiioned a into aleft and aright, then Quicksort is recursively called on both these arrays, and the algorithm is complete.
Notice how Quicksort and Mergesort differ, in that Mergesort performs O(1) steps in partitioning a, but Θ(n) steps to combine the sorted subarrays, while Quicksort performs Θ(n) steps to partition a, and requires no work to combine the sorted arrays. Moreover, Quicksort has the advantage of sorting “in place”, meaning that no additional memory is required outside of the input array. Indeed, the Partitioning algorithm only requires swapping elements in the original array, and, the sorting of each subarray only uses that part of a where the elements of the subarray are located. For example, if aleft occupies locations 0 through 10 of a, then only those locations will be affected when Quicksort is called on input aleft. It is this in-place property that gives Quicksort an advantage over Mergesort.
12 Example 8. Demonstrate the quicksort algorithm using the array 5, 8, 6, 2, 7, 1, 0, 9, 3, 4, 6.
Example 9. Show that Quicksort has a worst-case running time of T (n) = O(n2).
13 Exercises
1. If Counting Sort is used to sort an array of integers that fall within the interval [−5000, 10000], then how large of an auxiliary array should one use? Explain.
2. What is the running time of Insertion Sort if all elements are equal? Explain.
3. Sort 13,1,4,5,8,7,9,11,3 using Insertion Sort.
4. Sort 13,1,4,5,8,7,9,11,3 using Radix Sort.
5. An n-permutation is an ordering of the numbers 0, 1, 2, . . . , n − 1, in which each number occurs exactly once. For example, 4, 3, 1, 2, 0 is a 5-permutation. Assume there is a function called rand int(i,j) which, on inputs i ≤ j, returns a randomly chosen integer from the set {i, i + 1, . . . , j}. Now consider the following algorithm which, on input n, generates a random n- permutation within an array a[0], . . . , a[n − 1]. To assign a[i] it calls rand int(0,n-1) until rand int returns a value that was not assigned to a[0], a[1], . . . , a[i − 1]. The code for this algorithm is provided as follows.
int a[n];
//Initialize a[] for(i=0; i < n; i++) a[i] = UNDEFINED;
for(i=0; i < n; i++) { while(a[i] == UNDEFINED) { m = rand_int(0,n-1);
//Check if m has already been used for(j=0; j < i; j++) if(a[j] == m) break;
if(i == j) //m has not previously been used a[i] == m; } }
What is the expected running time of this algorithm? Explain and show work.
6. Repeat the previous problem, but now assume that, rather than checking each a[0], . . . , a[i − 1] to see if the current random value m has already been used, an array called “used” is provided so that m has been used iff used[m] evaluates to true. In other words, the used array is initialized so that each of its values is set false, and then used[m] is set to true when m is first returned by rand int(0,n-1). Re-write the code of the previous problem and adapt it to this new algorithm, and analyze its expected running time.
14 7. This problem provides an even better approach to generating a random permutation. The algorithm starts by assigning a[i] the value i, for i = 0, 1, . . . , n − 1. It then iterates n times so that, on iteration i, i = 0, 1, . . . , n − 1, it swaps a[i] with a[k], where k is randomly chosen from the set {i, i + 1, . . . , n − 1}. Prove that this algorithm yields a random n-permutation written in a[]. What is its expected running time?
8. Draw the recursion tree that results when applying Mergesort to the array 5, −2, 0, 7, 3, 11, 2, 9, 5, 6, Label each node with the sub-problem to be solved at that point of the recursion. Assume ar- rays of size 1 and 2 are base cases. Assume that odd-sized arrays are split so that the left subproblem has one more integer than the right. Next to each node, write the solution to its associated subproblem. Perform the partitioning step of Quicksort on the array 5, −2, 0, 7, 3, 11, 2, 9, 5, 6, where the pivot is chosen using the median-of-three heuristic.
9. Write an algorithm for the function merge that takes as input two sorted integer arrays a and b of sizes m and n respectively, and returns a sorted integer array of size m + n whose elements are the union of elements from a and b.
10. Determine the medians for the array of numbers 5, −2, 0, 7, 3, 11, 2, 9, 5, 6.
11. Perform the partitioning step of Quicksort on the array 5, −2, 0, 7, 3, 11, 2, 9, 5, 6, where the pivot is chosen using the median-of-three heuristic.
12. Provide a permutation of the numbers 1-9 so that, when sorted by Quicksort using median- of-three heuristic, the aleft subarray always has one element in rounds 1,2, and 3. Note: in general, when using the median-of-three heuristic, Quicksort is susceptible to Θ(n2) worst case performance.
13. Suppose that the Quicksort pivot is chosen by randomly and uniformly selecting on the n integers. Show that its average-case running time T (n) satisfies
n 2 X T (n) = n + T (i − 1), n i=1 where we assume that the partitioning algorithm takes exactly n steps.
14. Since the right side of the equation in the previous exercise is a series of an increasing function, we may approximate the equation as 2 Z n T (n) = n + T (x)dx. n 0 Show that this equation balances for T (x) = Cx ln x, for some constant C to be determined. Conclude that, on average, Quicksort achieves an asymptotically best-case running time.
15. Consider the following algorithm called multiply for multiplying two n-bit binary numbers x and y. Let xL and xR be the leftmost dn/2e and rightmost bn/2c bits of x respectively. Define yL and yR similarly. Let P1 be the result of calling multiply on inputs xL and yL, P2 be the result of calling multiply on inputs xR and yR, and P3 the result of calling multiply on inputs n n/2 xL + xR and yL + yR. Then return the value P1 × 2 + (P3 − P1 − P2) × 2 + P2. Draw a
15 recursion tree for the algorithm using inputs x = 10011011 and y = 10111010. Assume n = 1 is the base case. Next to each tree node, write the solution that should be returned by this node, and verify that the solution is consistent with what the algorithm would return.
n 16. Verify that the algorithm always works by proving in general that xy = P1 × 2 + (P3 − P1 − n/2 P2) × 2 + P2 for arbitrary x and y. Hint: you may assume that x and y both have even lengths as binary words.
17. Given an array a[] of integers, a subsequence of the array is a sequence of the form a[i], a[i + 1], a[i + 2], . . . , a[j], where i ≤ j. Moreover, the sum of the subsequence is defined as a[i] + a[i + 1] + a[i + 2] + ··· + a[j]. Describe in words a divide-and-conquer algorithm for finding the minimum positive sum that is associated with any subsequence of the array. Make sure your description has enough detail so that someone could read it and understand how to program it.
16 Exercise Hints and Answers
1. Since there are 15,001 possible values for an array element, an auxiliary array of size 15,001 is needed. then how large of an auxiliary array should one use? Explain.
2. Linear since it requires zero swaps.
3. 1,3,4,5,7,8,9,11,13
4. After Round 1, the numbers should be ordered as 0100, 1000, 1101, 0001, 0101, 0111, 1001, 1011, 0011. After Round 2: 0100, 1000, 1101, 0001, 0101, 1001, 0111, 1011, 0011. Continue with Rounds 3, 4, and 5, using the 3rd, 4th, and 5th least significant bits of the numbers.
5. Let T be the running time of the algorithm. Let Si denote the number of calls to rand() that are needed in order to generate the i th number of the permutation. Then T = (1)S1 + 2S2 + ··· + nSn. This is true since each call to rand() when generating the i the permutation will require an average of Θ(i) steps within the inner-most loop to check if the generated number has been used. Now when generating the i th permutation number using rand(), there is a probability of pi = (i − 1)/n that a number will be generated which is already in the permutation. Then the probabiliy of not generating a repeat number is thus 1 − (n − i + 1)/n = (n − i + 1)/n, and the expected number of calls that will be needed to obtain a non-repeat is E[Si] = n/(n−i+1). Pn i Thus, by linearity of expectation, E[T ] = n i=1 n−i+1 . The asymptotic growth of this sum R n x 2 can be obtained by evaluating n 1 n−x+1 dx, which yields E[T ] = Θ(n log n).
6. Same analysis as previous problem, but now we have T = S1 + ··· + Sn, since the returned value of a call to rand() can be checked in O(1) steps as to whether or not it is a repeat. This Pn 1 yields a simplified expectation of E[T ] = n i=1 n−i+1 . But the sum in this expression is the harmonic series. Therefore E[T ] = Θ(n log n).
7. First notice that the only operations performed on a are swaps between two different array entries. Thus, if the array begins with 0, . . . , n − 1, then it will end with a permutation of 0, . . . , n−1. Moreover, all values are equally likely to be placed at position i, for all i = 0, . . . , n− 1. The expected running time is Θ(n) since the algorithm is accomplished by performing n swaps and n calls to a random-number generator.
8. We use lr-strings for the addresses of each node. For example, λ denotes the root, while lrr denotes the right child of the right child of the left child of the root. Then
λ : 5, −2, 0, 7, 3, 11, 2, 9, 5, 6
l : 5, −2, 0, 7, 3 r : 11, 2, 9, 5, 6 ll : 5, −2, 0 lr : 7, 3 rl : 11, 2, 9 rr : 5, 6 lll : 5, −2 llr : 0 rll : 11, 2 rlr : 9
Draw the recursion tree that results when applying Mergesort to the array 5, −2, 0, 7, 3, 11, 2, 9, 5, 6, Label each node with the sub-problem to be solved at that point of the recursion. Assume ar- rays of size 1 and 2 are base cases. Assume that odd-sized arrays are split so that that the left subproblem has one more integer than the right. Next to each node, write the solution to its associated subproblem.
17 9. //Merges a and b into merged. n= |a|, m = |b| void merge(int[] a, int n, int[] b, int m, int[] merged) { int i=0; int j=0; int value_a = a[0]; int value_b = b[0]; int count = 0;
while(true) { if(value_a <= value_b) { merged[count++] = value_a; i++;
if(i < n) value_a = a[i]; else break; } else { merged[count++] = value_b; j++;
if(j < m) value_b = b[j]; else break; } }
//copy remaining values if(j < m) for(; j < m; j++) merged[count++] = b[j]; else for(; i < n; i++) merged[count++] = a[i]; } 10. 5 and 5
11. Pivot = 6. aleft = 5, −2, 0, 5, 3, 2, aright = 9, 7, 11 12. 173924685 is one possible permutation. Verify! 13. Hint: consider the n equations T (n) = n + T (i − 1) + T (n − i), i = 1, . . . , n, each of which
18 represents a recurrence for the average running time T (n) in case the i th least element is chosen as the pivot. Now add all the left sides, and all the right sides, and divide each side by n.
14. Hint: use integration by parts.
19