Bucket Sort: Sorting Integers Sorting (Part II) • the Goal: Sort N Numbers, All Between 1 to K

Bucket Sort: Sorting Integers Sorting (Part II) • The goal: sort N numbers, all between 1 to k. CSE 373 • Example: sort 8 numbers 3,6,7,4,11,3,5,7. All between 1 to 12. Data Structures • The method: Use an array of k queues. Queue j Unit 17 (for 1 ≤ j ≤ k) keeps the input numbers whose value is j. Reading: Section 3.2.6 Radix sort • Each queue is denoted ‘a bucket’. Section 7.6 Mergesort, Section 7.7, Quicksort, Sections 7.8 Lower bound • Scan the list and put the elements in the buckets. • Output the content of the buckets from 1 to k.

Bucket Sort: Sorting Integers Radix Sort: Sorting integers

• Example: sort 8 numbers 3,6,7,4,11,3,9,7 all • Historically goes back to the 1890 census. between 1 to 12. • Radix sort = multi-pass bucket sort of integers • Step 1: scan the list and put the elements in in the range 0 to BP-1 the queues • Bucket-sort from least significant to most 1 2 3 4 5 6 7 8 9 10 11 12 significant “digit” (base B) 3 4 6 7 9 11 3 7 • Requires P(B+N) operations where P is the • Step 2: concatenate the queues number of passes (the number of base B digits 3 4 6 7 9 11 in the largest possible input number). 3 7 3,3,4,6,7,7,9,11 • If P and B are constants then O(N) time to sort! • Time complexity: O(n+k). 3 4 Radix Sort Example Radix Sort Example

After 1st pass Bucket sort Input data st nd Bucket sort After 1 pass by 10’s After 2 pass 478 721 by 1’s digit digit 3 537 3 721 3 9 9 0 1 2 3 4 5 6 7 8 9 123 0 1 2 3 4 5 6 7 8 9 721 537 123 721 721 3 537 478 9 03 721 537 67 478 123 3 123 67 38 67 537 38 478 67 09 123 38 537 123 38 478 38 67 9 38 67 9 478 This example uses B=10 and base 10 digits for simplicity of demonstration. Larger bucket counts should be used in an actual implementation.

5 6

Radix Sort Example Properties of Radix Sort

After 2nd pass Bucket sort After 3rd pass by 100’s • Not in-place 3 digit 3 9 9 › needs lots of auxiliary storage. 721 0 1 2 3 4 5 6 7 8 9 38 123 003 123 478 537 721 67 • Stable 537 009 123 › equal keys always end up in same bucket in the 38 038 478 67 067 537 same order. 478 721 • Fast › Time to sort N numbers in the range 0 to BP-1 is Invariant: after k passes the low order k digits are sorted. O(P(B+N)) (P iterations, B buckets in each)

7 8 “Divide and Conquer” Mergesort

• Very important strategy in computer science: › Divide problem into smaller parts 8 2 9 4 5 3 1 6 › Independently solve the parts › Combine these solutions to get overall solution • Idea 1: Divide array into two halves, • Divide it in two at the midpoint recursively sort left and right halves, then merge two halves Mergesort • Conquer each side in turn (by • Idea 2 : Partition array into items that are recursively sorting) “small” and items that are “large”, then • Merge two halves together recursively sort the two sets Quicksort

9 10

Mergesort Example Auxiliary Array

8 2 9 4 5 3 1 6 • The merging requires an auxiliary array. Divide 8 2 9 4 5 3 1 6 Divide 2 4 8 9 1 3 5 6 8 2 9 4 5 3 1 6 Divide 1 element 8 2 9 4 5 3 1 6 Merge 2 8 4 9 3 5 1 6 Auxiliary array Merge 2 4 8 9 1 3 5 6 Merge 1 2 3 4 5 6 8 9 11 12 Auxiliary Array Auxiliary Array

• The merging requires an auxiliary array. • The merging requires an auxiliary array.

2 4 8 9 1 3 5 6 2 4 8 9 1 3 5 6

1 Auxiliary array 1 2 3 4 5 Auxiliary array

13 14

Merging Merging

first

i j normal second i j Right completed first target target

Left completed copy i j first

target

15 16 Merging Recursive Mergesort

Merge(A[], T[] : integer array, left, right : integer) : { Mergesort(A[], T[] : integer array, left, right : integer) : { mid, i, j, k, l, target : integer; if left < right then mid := (right + left)/2; mid := (left + right)/2; i := left; j := mid + 1; target := left; Mergesort(A,T,left,mid); while i < mid and j < right do Mergesort(A,T,mid+1,right); if A[i] < A[j] then T[target] := A[i] ; i:= i + 1; Merge(A,T,left,right); else T[target] := A[j]; j := j + 1; } target := target + 1; MainMergesort(A[1..n]: integer array, n : integer) : { if i > mid then //left completed// T[1..n]: integer array; for k := left to target-1 do A[k] := T[k]; Mergesort[A,T,1,n]; if j > right then //right completed// } k : = mid; l := right; while k > i do A[l] := A[k]; k := k-1; l := l-1; for k := left to target-1 do A[k] := T[k]; } 17 18

Iterative Mergesort Iterative Mergesort

Merge by 1 Merge by 1

Merge by 2 Merge by 2 Merge by 4 Merge by 4 Merge by 8 Merge by 8 Merge by 16

Need of a last copy

19 20 Iterative Mergesort Mergesort Analysis

IterativeMergesort(A[1..n]: integer array, n : integer) : { • Let T(N) be the running time for an //precondition: n is a power of 2// i, m, parity : integer; array of N elements T[1..n]: integer array; m := 2; parity := 0; • Mergesort divides array in half and calls while m < n do for i = 1 to n – m + 1 by m do itself on the two halves. After returning, if parity = 0 then Merge(A,T,i,i+m-1); else Merge(T,A,i,i+m-1); it merges both halves using a temporary parity := 1 – parity; array m := 2*m; if parity = 1 then for i = 1 to n do A[i] := T[i]; • Each recursive call takes T(N/2) and } merging takes O(N) How do you handle non-powers of 2? How can the final copy be avoided? 21 22

Mergesort Recurrence Properties of Mergesort Relation • The recurrence relation for T(N) is: • Not in-place › T(1) < a › Requires an auxiliary array (O(n) extra • base case: 1 element array constant time space) › T(N) < 2T(N/2) + bN • Stable • Sorting N elements takes › Make sure that left is sent to target on – the time to sort the left half – plus the time to sort the right half equal values. – plus an O(N) time to merge the two halves • Iterative Mergesort reduces copying. • T(N) = O(n log n) (see Lecture 5 Slide17)

23 24 Quicksort “Four easy steps”

• Quicksort uses a divide and conquer strategy, • To sort an array S but does not require the O(N) extra space 1. If the number of elements in S is 0 or 1, that MergeSort does then return. The array is sorted. › Partition array into left and right sub-arrays 2. Pick an element v in S. This is the pivot • Choose an element of the array, called pivot • the elements in left sub-array are all less than pivot value. • elements in right sub-array are all greater than pivot 3. Partition S-{v} into two disjoint subsets, S1 › Recursively sort left and right sub-arrays ≤ ≥ = {all values x v}, and S2 = {all values x v}. › Concatenate left and right sub-arrays in O(1) time 4. Return QuickSort(S1), v, QuickSort(S2)

25 26

The steps of QuickSort Details, details

select pivot value S 81 43 31 57 13 75 • Implementing the actual partitioning 92 0 65 26 • Picking the pivot

partition S S1 0 S2 › want a value that will cause |S1| and |S2| to 43 31 75 13 65 81 26 57 92 be non-zero, and close to equal in size if possible QuickSort(S1) and S1 S2 QuickSort(S2) 0 13 26 31 43 57 65 75 81 92 • Dealing with cases where an element equals the pivot

S 0 13 26 31 43 57 65 75 81 92 Voila! S is sorted [Weiss] 27 28 Quicksort Partitioning Partitioning:Choosing the pivot

• Need to partition the array into left and right sub- • One implementation (there are others) arrays › median3 finds pivot and sorts left, center, › the elements in left sub-array are ≤ pivot right › elements in right sub-array are ≥ pivot • Median3 takes the median of leftmost, middle, and • How do the elements get to the correct partition? rightmost elements • An alternative is to choose the pivot randomly (need a › Choose an element from the array as the pivot random number generator; “expensive”) › Make one pass through the rest of the array and • Another alternative is to choose the first element (but can be very bad. Why?) swap as needed to put elements in partitions › Swap pivot with next to last element

29 30

Partitioning in-place Example

Choose the pivot as the median of three › Set pointers i and j to start and end of array 0 1 2 3 4 5 6 7 8 9 8 1 4 9 0 3 5 2 7 6 › Increment i until you hit element A[i] > pivot Median of 0, 6, 8 is 6. Pivot is 6 › Decrement j until you hit element A[j] < pivot 0 1 4 9 7 3 5 2 6 8 › Swap A[i] and A[j] i j Place the largest at the right › Repeat until i and j cross and the smallest at the left. Swap pivot with next to last element. › Swap pivot (at A[N-2]) with A[i]

31 Example Example i j

i j 0 1 4 2 7 3 5 9 6 8 0 1 4 9 7 3 5 2 6 8 i j 0 1 4 2 7 3 5 9 6 8 i j 0 1 4 9 7 3 5 2 6 8 i j 0 1 4 2 5 3 7 9 6 8 i j 0 1 4 9 7 3 5 2 6 8 i j

i j 0 1 4 2 5 3 7 9 6 8 0 1 4 2 7 3 5 9 6 8 j i 0 1 4 2 5 3 7 9 6 8 Cross-over i > j

Move i to the right up to A[i] larger than pivot. j i Move j to the left up to A[j] smaller than pivot. 0 1 4 2 5 3 6 9 7 8 Swap

pivot S1 < pivot S2 > pivot 33

Quicksort Best Case Recursive Quicksort Performance Quicksort(A[]: integer array, left,right : integer): { • Algorithm always chooses best pivot pivotindex : integer; if left + CUTOFF ≤ right then and splits sub-arrays in half at each pivot := median3(A,left,right); recursion pivotindex := Partition(A,left,right-1,pivot); Quicksort(A, left, pivotindex – 1); › T(0) = T(1) = O(1) Quicksort(A, pivotindex + 1, right); • constant time if 0 or 1 element else Insertionsort(A,left,right); › For N > 1, 2 recursive calls plus linear time } for partitioning › T(N) = 2T(N/2) + O(N) Don’t use quicksort for small arrays. • Same recurrence relation as Mergesort CUTOFF = 10 is reasonable. › T(N) = O(N log N) 35 36 Quicksort Worst Case Properties of Quicksort Performance • Algorithm always chooses the worst pivot – • Not stable because of long distance one sub-array is empty at each recursion swapping. › T(N) ≤ a for N ≤ C › T(N) ≤ T(N-1) + bN • No iterative version (without using a stack). › ≤ T(N-2) + b(N-1) + bN • Pure quicksort not good for small arrays. ≤ › T(C) + b(C+1)+ … + bN • “In-place”, but uses auxiliary storage because ≤ › a +b(C + (C+1) + (C+2) + … + N) of recursive call (O(logn) space). › T(N) = O(N2) • Fortunately, average case performance is • O(n log n) average case performance, but 2 O(N log N) (see text for proof) O(n ) worst case performance.

37 38

Folklore How fast can we sort?

• “Quicksort is the best in-memory sorting • Heapsort, Mergesort, and Quicksort all algorithm.” run in O(N log N) best case running • Truth time › Quicksort uses very few comparisons on • Can we do any better? average. › Quicksort does have good performance in • No, if sorting is comparison-based. the memory hierarchy. • We saw that radix sort is O(N) but it is • Small footprint only for integers from bounded-range. • Good locality

39 40 Sorting Model Permutations

• Recall the basic assumption: we can only • How many possible orderings can you get? compare two elements at a time › Example: a, b, c (N = 3) › we can only reduce the possible solution space by › (a b c), (a c b), (b a c), (b c a), (c a b), (c b a) half each time we make a comparison › 6 orderings = 3•2•1 = 3! (i.e., “3 factorial”) • Suppose you are given N elements › All the possible permutations of a set of 3 elements › Assume no duplicates • For N elements • How many possible orderings can you get? › N choices for the first position, (N-1) choices for the second position, …, (2) choices, 1 choice › Example: a, b, c (N = 3) › N(N-1)(N-2)3(2)(1)= N! possible orderings

41 42

Decision Tree Decision Trees

a b b < c < a c < a c a < c a > c › How many leaves for N distinct elements? a c c < a c > a • Only 1 leaf has the ordering that is the desired correctly sorted arrangement a < b < c a < c < b b < c < a b < a < c The leaves contain all the possible orderings of a, b, c 43 44 Decision Trees and Sorting Decision Tree Example

• Every comparison-based sorting algorithm a b • i.e., by making comparisons a c b < c b > c

• Run time is ≥ maximum no. of comparisons a a the longest path in the decision tree, i.e. the height b < c b > c actual order of the tree a < b < c a < c < b b < c < a b < a < c

45 46

How many leaves on a tree? Lower bound on Height

• Suppose you have a binary tree of height d . • A binary tree of height d has at most 2d leaves How many leaves can the tree have? › depth d = 1 2 leaves, d = 2 4 leaves, etc. › d = 1 at most 2 leaves, › Can prove by induction › d = 2 at most 4 leaves, etc. • Number of leaves, L < 2d

• Height d > log2 L • The decision tree has N! leaves ≥ • So the decision tree has height d log2(N!)

47 48 log(N!) is Ω(NlogN) Summary of Sorting

( ) log(N!) = log N ⋅(N −1)⋅(N − 2)3(2)⋅(1) • Sorting choices: = + − + − + + + › O(N2) – Bubblesort, Insertion Sort select just the log N log(N 1) log(N 2) 3 log 2 log1 first N/2 terms N › O(N log N) average case running time: ≥ log N + log(N −1) + log(N − 2) +3+ log • Heapsort: In-place, not stable. 2 • Mergesort: O(N) extra space, stable. each of the selected N N • Quicksort: claimed fastest in practice but, O(N2) worst terms is ≥ logN/2 ≥ log 2 2 case. Needs extra storage for recursion. Not stable. N N N › Run time of any comparison-based sorting ≈ π n ≥ (log N − log 2) = log N − algorithm is Ω(N log N) n! 2 n(n/e) 2 2 2 › O(N) – Radix Sort: fast and stable. Not Sterling’s formula = Ω(N log N) comparison based. Not in-place.

49 50