Introduction

Chapter 7 fundamental task in data management well-studied problem in Sorting basic problem given an of items where each item contains a key, rearrange the items so that the keys appear in ascending order the key may be only of the item begin sorted e.g., the item could be an entire block of information about a student, while the search key might be only the student's name

2

Introduction Sorting

we will assume sorts the array to contains only integers defined < and > operators (comparison-based sorting) all array positions contain data to be sorted why ? the entire sort can be performed in ______Shellsort: ______number of elements is relatively small: < a few million sorts given a list of n items, we will use the notation to indicate that the search for a mergesort does not follow that of sorts that cannot be performed in main memory a lower bound on sorting by _ comparison may be performed on disk or tape sorting algorithms: count sort known as sorting string sorts

3 4 Sorting Sorting

given keys to sort, there are possible of cost model for sorting the keys! the basic operations we will count are and e.g, given and the keys , there are ______possible permutations: if there are array accesses that are not associated with comparisons or swaps, we need to count them, too programming notes if the objects being sorted are large, we should swap brute force enumeration of permutations is not ______to the objects, rather than the objects computationally once themselves polymorphism: general-purpose sorting algorithms vs templated algorithms

5 6

Insertion Sort Insertion Sort

insertion sort simple passes for (p = 1; p < N; p++) { in pass , , we move to its correct tmp = location among for (j = p; (j > 0) && (tmp < ); j--) { for passes to , insertion sort ensures that the swap and elements in positions to are in sorted order } at the end of pass , the elements in positions to are in = tmp sorted order }

7 8 Insertion Sort Insertion Sort

to avoid the full swap, we can use the following code: example

for (p = 1; p < N; p++) { tmp = for (j = p; (j > 0) && (tmp < ); j--) { = } = tmp }

9 10

Insertion Sort Insertion Sort

example analysis position 0 1 2 3 4 5 best case: the keys are already initial sequence 42 6 1 54 0 7 comparisons, no swaps p = 1 6 42 1 54 0 7 worst case: the keys are in sorted order p = 2 6 1 42 54 0 7 expected (average) case: ? 1 6 42 54 0 7 p = 3 1 6 42 54 0 7 p = 4 1 6 42 0 54 7 1 6 0 42 54 7 1 0 6 42 54 7 0 1 6 42 54 7 p = 5 0 1 6 42 7 54 0 1 6 7 42 54

11 12 Insertion Sort Insertion Sort

analysis number of inversions given that we wish to sort in ascending order, an a list can have between and inversions, the ______is any pair that is out of order relative to latter of which occurs when one another: for which but thus, counting inversions also says the worst-case behavior the list of insertion is ______what do inversions tell us about the behavior? contains the following inversions: let be the probability space of all permutations of distinct elements with equal ______Theorem: The expected number of inversions in a list swapping an adjacent pair of elements that are out of order taken from is ______exactly one inversion thus, the expected complexity of insertion sort is thus, any sort that operates by swapping adjacent terms requires as many swaps as there are inversions quadratic

13 14

Insertion Sort Insertion Sort

proof proof (cont.) observe that any pair in a list that is an inversion is in the since there are distinct pairs of lists and their correct order in the of that list reverses, there is a total of

list inversions reverse lists inversions 1, 2, 3, 4 0 4, 3, 2, 1 6 inversions among the possible lists of distinct objects 2, 1, 3, 4 1 4, 3, 1, 2 5 3, 2, 1, 4 3 4, 1, 2, 3 3 this means that the expected number of inversions in any given list is this means that if we look at a list and its reverse and count the total number of inversions, then the combined number of inversions is

15 16 Insertion Sort Selection Sort

in summary selection sort for randomly ordered arrays of length with distinct keys, find the smallest item and exchange it with the first entry insertion sort uses find the next smallest item and exchange it with the second comparisons and swaps on ______entry comparisons and swaps in the ______find the next smallest item and exchange it with the third case entry comparisons and swaps in the case

17 18

Selection Sort Selection Sort

algorithm example

for (p = 0; p < N; p++) { position 0 1 2 3 4 m = p initial sequence 42 6 9 54 0 for (j = p+1; j < 0; j++) { after p = 0 0 42 9 54 6 if ( < ) { after p = 1 0 6 9 54 42 m = j } after p = 2 0 6 9 54 42 } after p = 3 0 6 9 42 54 swap and }

19 20 Selection Sort Selection Sort

complexity proof comparisons and swaps to sort an array of there is one swap per iteration of the loop, length which is executed times the amount of work is of the input there is one comparison made in each iteration of the selection sort is no faster on input than on ______loop, which is executed random input selection sort involves a smaller number of than any of the other sorting algorithms we will consider

21 22

Bubble Sort Bubble Sort

bubble sort algorithm read the items from left to right if two items are out of order, swap them not_done = true repeat until sorted while (not_done) { a sweep with swaps means we are done not_done = false for (i = 0 to N - 2) { if ( > ) { swap and not_done = true } } }

23 24 Bubble Sort Bubble Sort

example complexity if the data are already sorted, we make only sweep initial sequence 42 6 9 54 0 through the list while-loop 6 42 9 54 0 otherwise, the complexity depends on the number of times 6 9 42 54 0 we execute the while-loop 6 9 42 0 54 since bubble sort swaps items, it will have ______worst-case and expected-case while-loop 6 9 0 42 54 complexity while-loop 6 0 9 42 54 while-loop 0 6 9 42 54

25 26

Shellsort

Shellsort Shellsort Donald L. Shell (1959), A high-speed sorting procedure, suppose we use the increment sequence 15, 7, 5, 3, 1, and Communications of the ACM 2 (7): 3032. have finished the 15-sort and 7-sort swapping only adjacent items dooms us to quadratic worst- then we know that case behavior, so swap items! Shellsort starts with an sequence

it uses insertion sort to sort we also know that every -th term starting at , then then every -th term starting at , then then etc. putting these together, we see that every term starting at , after which the array is sorted

27 28 Shellsort Shellsort

Shellsort example after we have performed the sort using increment , the array is -sorted: all elements that are terms apart are in the order:

the key to Shellsort's is the following fact: an - remains -sorted after sorting with increment

29 30

Shellsort Shellsort

complexity complexity (cont.) a good increment sequence has the the sequences property that for any element , when it is time for the - (T. H. Hibbard, 1963) sort, there are only a few elements to the left of that are ______than (V. R. Pratt, 1971) Shell's original increment sequence has ______yield worst-case complexity behavior: other sequences yield worst-case complexity ,

31 32 Heapsort Heapsort

priority queues can be used to sort in can avoid extra array by storing element in original array strategy leaves an as each smallest element build binary of elements time is deleted store element in newly opened space, which is no longer perform deleteMin operations elements (smallest first) stored in second array, then used by the heap ______back into original array time results in list of order in array requires extra , which doubles memory to achieve increasing order, change ordering of heap to requirement ______heap when complete, array contains elements in ascending order

33 34

Heapsort Heapsort

max heap after buildHeap phase max heap after first deleteMax

35 36 Heapsort Mergesort

analysis mergesort is a algorithm building of elements < comparisons in Vol. III of The Art of Computer Programming, Knuth attributes the algorithm to John von Neumann (1945) total deleteMax operations if heapsort in worst case the idea of mergesort is simple: average case extremely complex to compute divide the array in two sort each half improved to or simply the two subarrays using mergesort heapsort useful if we want to sort the or merging simple since sorted ______elements and mergesort can be implemented recursively and non- recursively runs in , worst case

37 38

Mergesort Mergesort

merging algorithm takes two arrays, and , and ______example array also uses three counters: , , initialized to beginning of respective arrays smaller of and copied to when either input array exhausted, the remainder of the other list is to

39 40 Mergesort Mergesort

example (cont.) example (cont.) another way to visualize

41 42

Mergesort Mergesort

time to merge two lists is at most analysis comparisons running time represented by relation every comparison adds an element to assume is a power of 2 so that list is always divided mergesort easy to characterize ______if , only one element to sort for , time to mergesort is ______otherwise, mergesort first half and otherwise, time to mergesort numbers is time to perform second half two recursive mergesort of size , plus the time to merge these two halves merge, which is linear problem is divided into smaller problems and solved recursively, and conquered by patching the solutions together

43 44 Mergesort Mergesort

analysis (cont.) solving mergesort using ______standard recurrence relation can be solved in at least two ways telescoping divide the recurrence through by substitution

45 46

Mergesort Mergesort

solving mergesort recurrence relation using telescoping solving mergesort recurrence relation using ______(cont.) (cont.)

47 48 Mergesort Mergesort

proof by (cont.) proof by induction (cont.)

Prove: is equivalent to I.S.: Show:

let Base: (or ) by I.H. rr: by definition cf:

I.H.: Assume By induction, we have therefore shown the original statement for some to be true.

49 50

Quicksort Quicksort

historically, quicksort has been known generic quicksort is a divide-and-conquer algorithm basic idea average running time arbitrarily select a single item worst case running time , but can be made highly form three groups: ______those than the item can be combined with to achieve those to the item average and worst case time those than the item recursively sort the first and third groups concatenate the three groups

51 52 Quicksort Quicksort

implementation performance good on most inputs if list contains many , performance is very good some issues making extra lists recursively consumes ______not much better than mergesort loop bodies too heavy can avoid category in loop

53 54

Quicksort Quicksort

classic quicksort algorithm to sort an array example if there are either 0 or 1 elements in , then ______pivot chosen randomly choose an element in to serve as the ______partition into two disjoint subsets and with the properties that if and if apply quicksort recursively to and note the for elements equal to the pivot ideally, half of the duplicates would go into each sublist

55 56 Quicksort Quicksort

many methods for selecting pivot and partitioning elements quicksort vs. mergesort very sensitive to even slight variances mergesort: is trivial; the work is in the merge in these choices quicksort: the work is in the partitioning; the is trival comparison with mergesort mergesort: requires an auxiliary array to be efficient (in- like mergesort, recursively solves two subproblems and place variants exist that are less efficient, or which sacrifice requires additional work an important property called stability) unlike mergesort, may not be of equal quicksort: faster since partitioning step can be performed size (bad) efficiently in place (with a modest amount ( ) space needed to handle the recursion) in both sorts, more efficient to switch to insertion sort once the arrays are sufficiently small to avoid the cost of the overhead of on small arrays

57 58

Quicksort Quicksort

example choosing the pivot pivots in red popular, but bad, method: choose the element in the list OK if input is ______not OK if input is presorted or in reverse order happens consistently in recursive calls results in time for presorted data for doing nothing! occurs often alternative: choose larger of first two elements could pick the pivot randomly safe, but random number generation expensive

59 60 Quicksort Quicksort

choosing the pivot (cont.) partitioning strategy -of-three partitioning first, get pivot out of the way by swapping with ______best choice would be median of sublist, but takes too element long to calculate two counters, and good estimate by picking three elements and starts at first element using middle element as pivot starts at nest-to-last element randomness not really helpful select first, middle, and last elements eliminates bad case for input reduces number of by about 15% move all the smaller elements to and all larger example: 8, 1, 4, 9, 6, 3, 5, 2, 7, 0 elements to ______from 8, 0, and , or 6, select 6

61 62

Quicksort Quicksort

partitioning strategy (cont.) partitioning strategy (cont.) while is to the left of example (cont.) move right, skipping over elements smaller than pivot move left, skipping over elements larger than pivot when and stop, is at a element and is at a element them example

after and cross, swap location with pivot

63 64 Quicksort Quicksort

partitioning strategy (cont.) implementation at this point, all positions contain smaller elements driver than , and all positions contain larger elements how to handle equal elements should stop when element to pivot? what about ? and should behave similarly to avoid all elements equal to pivot collecting in one sublist best to have and stop and perform an unnecessary swap to avoid sublists (and quadratic run time!) pass array and range (left and right) to be sorted for small arrays, and as sublists get small (< 20 elements), use insertion sort fast and avoids degenerate median-of-three cases

65 66

Quicksort Quicksort

implementation (cont.) implementation (cont.) median-of-three pivot selection median-of-three sort a[left], a[right], and a[center] in ______smallest of three ends up in location largest in last location pivot in a[right 1] can be initialized to left + 1 can be initialized to right 2 since a[left] smaller than pivot, it will act as a ______and stop from going past the beginning of the array storing pivot at a[right 1] will act as a sentinel for

67 68 Quicksort Quicksort

implementation (cont.) implementation (cont.) main quicksort main quicksort (cont.)

69 70

Quicksort Quicksort

implementation (cont.) analysis main quicksort (cont.) quicksort is interesting because its worst-case behavior and its expected behavior are very ______16: i and j start at one off 22: swap can be written inline let be the run-time needed to sort items 19-20: small inner loop very fast pivot is constant time cost of the partition is if has elements, then has elements, and

71 72 Quicksort Quicksort

worst-case analysis worst-case analysis (cont.) the worst-case occurs when or i.e., when combining these yields the is the smallest or largest element every time quicksort() is called in this case, without loss of generality we may assume that , so or

thus

quadratic! is it likely that at every recursive call to quicksort() we will choose the smallest element as the pivot? yes, if the data are already sorted

73 74

Quicksort Quicksort

best-case analysis average-case analysis in the best case, the pivot is always the of the assumption: any size is equally likely data being operated on for instance, suppose ; since we remove the , the possible sizes of the partitions are

we know from the analysis of mergesort that the solution is

in this case the value of is

75 76 Quicksort Quicksort

77 78

Quicksort Lower Bound for Pairwise Sorting

no algorithm based on comparisons can guarantee sorting items with fewer than comparisons to show this, we first abstract the behavior of such algorithms using a tree a decision tree is a in which each node represents a set of possible orderings the root consists of the possible orderings of the items to be sorted the edges represent the results of comparisons, and a node comprises the orderings consistent with the comparisons made on the path from the root to the node each consists of a single sorted ordering

79 80 Lower Bound for Pairwise Sorting Lower Bound for Pairwise Sorting

a decision tree to sort items must have leaves this requires a tree of depth by approximation thus, the best case for sorting with pairwise comparisons is

81 82

Quickselect

thus far, the best performance to select the smallest quickselect algorithm element is using a (heap) given a set , let be its cardinality quickselect quicksort can be modified to solve the selection problem if there is 1 element in , return quickselect choose an element in to serve as the pivot partition into two disjoint subsets and with the properties that if and if now the search proceeds on and if , then the smallest element must be in , so quickselect( , ) if , then the pivot is the smallest, so return otherwise, the smallest element must be in , and it is the -th element of , so return quickselect( , )

83 84 Quickselect Quickselect

example: find the median of 7 items ( ) example: find the median of 7 items ( ) red denotes pivots, while grey denotes the partition that is call quickselect ( ; partition; since , we want the ______smallest element of , so call quickselect ( ; partition, then call quickselect ( , ); call quickselect ( , ); partition; since we are inside the call once again, partition; at this point, so the pivot is quickselect ( , ), we want the 1st smallest element, so we the 4th element, and thus the answer call quickselect ( , ), which immediately exits, returning 10

85 86

Quickselect Quickselect

quickselect complexity quickselect complexity (cont.) at each recursive step quickselect ignores one partition will this expected behavior make it faster than quicksort? suppose we choose our pivot from the in the worst case, quickselect behaves like quicksort, and has terms we are searching th th complexity suppose lies between the 25 and 75 percentiles of the terms (i.e., is larger than 1/4 and smaller than 1/4 of the this occurs if the one partition is at each partitioning, terms) and we have to look at all the terms in the other partition. this means that neither partition can contain more than 3/4 of best case behavior is ______the terms, so the partitions can't be too ; occurs if each partition is ______call such a since quickselect ignores one partition at each step, its on average, how many do we need to choose before we get runtime satisfies the recurrence a good one? a randomly chosen is good with probability ½ - a good pivot lies in the middle 50% of the terms this leads to being linear choosing a good pivot is like tossing a coin and seeing heads

87 88 Quickselect Quickselect

quickselect complexity (cont.) quickselect complexity (cont.) expected behavior (cont.) expected behavior (cont.) the expected number of tosses to see a heads is two in terms of , to see this, let be the expected number of tosses expected value of expected time to before seeing a heads reduce the array toss the coin; if it's heads, we're done; if it's tails (which since each partitioning step requires work, and we occurs with probability 1/2) we have to toss it again, so expect to need 2 of them to reduce the array size to , we have , whence thus, on average, quickselect will take two ______to reduce the array to at most 3/4 of the original size

89 90

Quickselect Quickselect

quickselect complexity (cont.) quickselect complexity (cont.) expected behavior (cont.) expected behavior (cont.) consider the more general recurrence thus, the total amount of work is bounded above by , where at the level of the recursion, starting with , thus, the best case and expected case behavior of there is a single problem of size at most quickselect with a randomly chosen pivot is the amount of work done at each level is thus at most

the recursion continues until

so , or

91 92 Stability

from D. R. Musser, Introspective sorting and selection a sorting algorithm is stable if it preserves the relative order algorithms, Software: Practice and Experience 27 (8) 983- of keys 993, 1997 stable sorts: introsort is a of quicksort and heapsort insertion sort introsort starts with quicksort, but switches to heapsort mergesort whenever the number of levels of recursion exceed a unstable sorts: specified threshold (e.g., ). selection sort if it switches to heapsort, the subarrays that remain to be sorted will likely be much than the original Shell sort array quicksort this approach gives an complexity, heapsort while making the most of the efficiency of quicksort

93 94

Stability C++ Standard Library Sorts

sort(): introsort, guaranteed to be stable_sort(): mergesort, guaranteed to be stable and time, if a -sized auxiliary array can be allocated time for an in-place sort, otherwise partial_sort(): sort the largest (or smallest) items

95 96 String / String Sorting: LSD

we can use the special structure of character strings to in least-significant digit (LSD) first sorting, we sort using the devise sorting algorithms digits (characters) from least- to most-significant: here, a character should be understood in a general sense, as an element of an alphabet, which is a collection of items (presumably all requiring the same number of for their representation) the alphabet is also assumed to have an , so we may sort characters as a special case, we can think of treating base- numbers as a string of digits from the range to ( -digit base- numbers) since the base of a number system is sometimes called the ______, the sorting algorithms we will discuss are frequently called radix sorts

97 98

String Sorting: LSD String Sorting: LSD

algorithm: why not go left-to-right? for d from least- to most-significant digit { the same idea, but starting with the most significant digit, sort using on the least-significant digit doesn't work: }

observations: since counting sort is stable, LSD sorting is stable the stability of counting sort is essential to making LSD sorting work why does right-to-left digit sorting work but left-to-right does algorithm assumes all the strings have the same length not? : there are passes in the outmost loop; in each loop iteration, we apply counting sort to digits in the range to , requiring ; total work: space complexity: same as counting sort

99 100 String Sorting: LSD String Sorting: LSD

in the old days, we used Hollerith cards:

early versions of and COBOL had limits of 72 characters per line this left columns 73-80 free for the card number IBM Model 082 card sorter that way, if you dropped your deck of cards, you could go to...

101 102

String Sorting: LSD Sorting Algorithms Summary

these machines used LSD: pile the disordered cards in the input hopper and sort by the ______digit in the sequence number now take the sorted decks and stack them in order place the combined deck back in the input hopper and sort by the next-to-last digit in the sequence number repeat steps 2 and 3 until sorted

Shell sort is subquadratic with a suitable increment sequence Shell's original increment sequence is, in fact, quadratic in the worst case

103 104