Divide and Conquer Sorting Insertion Sort

Insertion Sort

• What if first k elements of array are already Divide and Conquer Sorting sorted? – 4, 7, 12, 5, 19, 16 • We can shift the tail of the sorted elements CSE 326 list down and then insert next element into Data Structures proper position and we get k+1 sorted elements Lecture 18 – 4, 5, 7, 12, 19, 16

“Divide and Conquer” Mergesort • Very important strategy in computer science: – Divide problem into smaller parts – Independently solve the parts 8 2 9 4 5 3 1 6 – Combine these solutions to get overall solution • Idea 1: Divide array into two halves, recursively sort • Divide it in two at the midpoint left and right halves, then merge two halves Æ known as Mergesort • Conquer each side in turn (by recursively • Idea 2 : Partition array into small items and large sorting) items, then recursively sort the two sets Æ known • Merge two halves together as Quicksort

3 4

Mergesort Example Auxiliary Array

• 8 2 9 4 5 3 1 6 The merging requires an auxiliary array. Divide 8 2 9 4 5 3 1 6 Divide 2 4 8 9 1 3 5 6 8 2 9 4 5 3 1 6 Divide 1 element 8 2 9 4 5 3 1 6 Merge 2 8 4 9 3 5 1 6 Auxiliary array Merge 2 4 8 9 1 3 5 6 Merge 1 2 3 4 5 6 8 9 5 6 Auxiliary Array Auxiliary Array

• The merging requires an auxiliary array. • The merging requires an auxiliary array.

2 4 8 9 1 3 5 6 2 4 8 9 1 3 5 6

1 Auxiliary array 1 2 3 4 5 Auxiliary array

7 8

Merging Merging

first

i j normal second i j Right completed first target target

Left completed copy ij first

target

9 10

Merging Recursive Mergesort

Merge(A[], T[] : integer array, left, right : integer) : { Mergesort(A[], T[] : integer array, left, right : integer) : { mid, i, j, k, l, target : integer; if left < right then mid := (right + left)/2; mid := (left + right)/2; i := left; j := mid + 1; target := left; Mergesort(A,T,left,mid); while i < mid and j < right do Mergesort(A,T,mid+1,right); if A[i] < A[j] then T[target] := A[i] ; i:= i + 1; Merge(A,T,left,right); else T[target] := A[j]; j := j + 1; } target := target + 1; MainMergesort(A[1..n]: integer array, n : integer) : { if i > mid then //left completed// T[1..n]: integer array; for k := left to target-1 do A[k] := T[k]; Mergesort[A,T,1,n]; if j > right then //right completed// } k : = mid; l := right; while k > i do A[l] := A[k]; k := k-1; l := l-1; for k := left to target-1 do A[k] := T[k]; }

11 12 Iterative Mergesort Iterative Mergesort

Merge by 1 Merge by 1

Merge by 2 Merge by 2 Merge by 4 Merge by 4 Merge by 8 Merge by 8 Merge by 16

copy

13 14

Iterative pseudocode Mergesort Analysis

• Sort(array A of length N) • Let T(N) be the running time for an array of N – Let m = 2, let B be temp array of length N elements – While m

15 16

Mergesort Recurrence Relation Properties of Mergesort

• The recurrence relation for T(N) is: • Not in‐place – T(1) < c – Requires an auxiliary array • base case: 1 element array Æ constant time • Very few comparisons – T(N) < 2T(N/2) + dN • Sorting n elements takes • Iterative Mergesort reduces copying. – the time to sort the left half – plus the time to sort the right half – plus an O(N) time to merge the two halves • T(N) = O(N log N)

17 18 Quicksort “Four easy steps”

• Quicksort uses a divide and conquer strategy, but • To sort an array S does not require the O(N) extra space that – If the number of elements in S is 0 or 1, then MergeSort does return. The array is sorted. – Partition array into left and right sub‐arrays – Pick an element v in S. This is the pivot value. • the elements in left sub‐array are all less than pivot • elements in right sub‐array are all greater than pivot – Partition S‐{v} into two disjoint subsets, S1 = {all – Recursively sort left and right sub‐arrays values x≤v}, and S2 = {all values x≥v}. – Concatenate left and right sub‐arrays in O(1) time – Return QuickSort(S1), v, QuickSort(S2)

19 20

The steps of QuickSort Details, details

select pivot value S 81 43 31 57 • “The algorithm so far lacks quite a few of 13 75 92 0 65 26 the details” • Picking the pivot S S partition S 1 0 31 2 75 13 43 65 – want a value that will cause |S | and |S | to be 92 81 1 2 26 57 non‐zero, and close to equal in size if possible

QuickSort(S1) and S1 S2 QuickSort(S2) • Implementing the actual partitioning 0 1326 31 43 57 65 75 81 92 • Dealing with cases where the element equals the pivot S 06581921326 31 43 57 75 Presto! S is sorted [Weiss]

21 22

Alternative Pivot Rules Quicksort Partitioning

• Chose A[left] • Need to partition the array into left and right sub‐ – Fast, but too biased, enables worst‐case arrays • Chose A[random], left < random < right – the elements in left sub‐array are ≤ pivot – Completely unbiased – elements in right sub‐array are ≥ pivot – Will cause relatively even split, but slow • How do the elements get to the correct partition? • Median of three, A[left], A[right], A[(left+right)/2] – Choose an element from the array as the pivot – The standard, tends to be unbiased, and does a little sorting on the side. – Make one pass through the rest of the array and swap as needed to put elements in partitions

23 24 Example Partitioning is done In‐Place

0123456789 • One implementation (there are others) 8 1 4 9 0 3 5 2 7 6 – median3 finds pivot and sorts left, center, right – Swap pivot with next to last element 0 1 4 9 7 3 5 2 6 8 – Set pointers i and j to start and end of array i j – Increment i until you hit element A[i] > pivot Choose the pivot as the median of three. – Decrement j until you hit element A[j] < pivot – Swap A[i] and A[j] Place the pivot and the largest at the right and the smallest at the left – Repeat until i and j cross – Swap pivot (= A[N‐2]) with A[i]

25 26

Example Example

i j

i j 0 1 4 2 7 3 5 9 6 8 0 1 4 9 7 3 5 2 6 8 i j 0 1 4 2 7 3 5 9 6 8 i j 0 1 4 9 7 3 5 2 6 8 ij 0 1 4 2 5 3 7 9 6 8 i j 0 1 4 9 7 3 5 2 6 8 i j

ij 0 1 4 2 5 3 7 9 6 8 0 1 4 2 7 3 5 9 6 8 j i 0 1 4 2 5 3 7 9 6 8

Move i to the right to be larger than pivot. j i Move j to the left to be smaller than pivot. 0 1 4 2 5 3 6 9 7 8 Swap

pivot S1 < pivot S2 > pivot 27 28

Recursive Quicksort Quicksort Best Case Performance

Quicksort(A[]: integer array, left,right : integer): { • Algorithm always chooses best pivot and pivotindex : integer; if left + CUTOFF ≤ right then splits sub‐arrays in half at each recursion pivot := median3(A,left,right); – T(0) = T(1) = O(1) pivotindex := Partition(A,left,right-1,pivot); Quicksort(A, left, pivotindex – 1); • constant time if 0 or 1 element Quicksort(A, pivotindex + 1, right); – For N > 1, 2 recursive calls plus linear time for else Insertionsort(A,left,right); partitioning } – T(N) = 2T(N/2) + O(N) • Same recurrence relation as Mergesort Don’t use quicksort for small arrays. – T(N) = O(N log N) CUTOFF = 10 is reasonable.

29 30 Properties of Quicksort Quicksort Worst Case Performance • Algorithm always chooses the worst pivot –one • No iterative version (without using a stack). sub‐array is empty at each recursion • Pure quicksort not good for small arrays. – T(N) ≤ a for N ≤ C • “In‐place”, but uses auxiliary storage because – T(N) ≤ T(N‐1) + bN of recursive calls. – ≤ T(N‐2) + b(N‐1) + bN – ≤ T(C) + b(C+1)+ …+ bN • O(n log n) average case performance, but 2 – ≤ a +b(C + C+1 + C+2 + …+ N) O(n ) worst case performance. – T(N) = O(N2) • Fortunately, average case performance is O(N log N) (see text for proof)

31 32

Folklore Features of Sorting Algorithms

• “Quicksort is the best in‐memory sorting • In‐place algorithm.” – Sorted items occupy the same space as the original items. (No copying required, only O(1) extra space if any.) • Mergesort and Quicksort make different • tradeoffs regarding the cost of comparison Stable and the cost of a swap – Items in input with the same value end up in the same order as when they began.

33 34

How fast can we sort? Sorting Model • Heapsort, Mergesort, and Quicksort all run in • Recall our basic assumption: we can only compare O(N log N) best case running time two elements at a time – we can only reduce the possible solution space by half • Can we do any better? each time we make a comparison • No, if the basic action is a comparison. • Suppose you are given N elements – Assume no duplicates • How many possible orderings can you get? – Example: a, b, c (N = 3)

35 36 Permutations Decision Tree

a < b < c, b < c < a, • How many possible orderings can you get? c < a < b, a < c < b, – Example: a, b, c (N = 3) b < a < c, c < b < a

– (a b c), (a c b), (b a c), (b c a), (c a b), (c b a) a b b < c < a c < a c b < c b > c

• For N elements a c c < a c > a – N(N‐1)(N‐2)Λ(2)(1)= N! possible orderings a < b < c a < c < b b < c < a b < a < c The leaves contain all the possible orderings of a, b, c

37 38

Decision Trees Decision Tree Example

• A Decision Tree is a Binary Tree such that: a b – Each edge = 1 comparison a c b < c b > c • N!, ie, a leaf for each possible ordering a a correctly sorted arrangement b < c b > c actual order a < b < c a < c < b b < c < a b < a < c

39 40

Decision Trees and Sorting Lower bound on Height

• A binary tree of height h has at most how many • Every sorting algorithm corresponds to a decision leaves? tree L≤ 2h – Finds correct leaf by choosing edges to follow • The decision tree has how many leaves: • ie, by making comparisons L= N! – Each decision reduces the possible solution space by one half • A binary tree with L leaves has height at least: • Run time is ≥ maximum no. of comparisons h ≥ log2 L – maximum number of comparisons is the length of the longest path in the decision tree, i.e. the height of the • So the decision tree has height: tree h ≥ log2 (N!)

41 42 log(N!) is Ω(NlogN) Ω(N log N)

log(N!) = log()N ⋅(N −1)⋅(N − 2)Λ (2)⋅(1) • Run time of any comparison‐based Ω select just the = log N + log(N −1) + log(N − 2) +Λ + log 2 + log1 sorting algorithm is (N log N) first N/2 terms N • Can we do better if we don’t use comparisons? ≥ log N + log(N −1) + log(N − 2) +Λ + log 2 each of the selected N N terms is ≥ logN/2 ≥ log 2 2 N N N ≥ (log N − log 2) = log N − 2 2 2 = Ω(N logN)

43 44

BucketSort Complexity: O(n+K) BucketSort (aka BinSort) If all values to be sorted are known to be between 1 and K, create an array count of size K, increment • Case 1: K is a constant counts while traversing the input, and finally output the result. – BinSort is linear time • Case 2: K is variable Example K=5. Input = (5,1,3,4,3,2,1,1,5,4,5) – Not simply linear time count array 32 1 • Case 3: K is constant but large (e.g. 2 ) 2 – ??? 3 4 Running time to sort n items?

5 45 46

Fixing impracticality: RadixSort Radix Sort Example (1st pass)

Bucket sort by 1’s digit • st Radix = “The base of a number system” Input data After 1 pass – We’ll use 10 for convenience, but could be 478 721 537 3 anything 9 0 1 2 3 4 5 6 7 8 9 123 537 721 721 3 537 478 9 3 123 67 38 67 38 478 123 38 • Idea: BucketSort on each digit, 67 9 least significant to most significant (lsd to msd)

This example uses B=10 and base 10 digits for simplicity of demonstration. Larger bucket counts should be used in an actual implementation. 47 48 Radix Sort Example (2nd pass) Radix Sort Example (3rd pass)

Bucket sort After 1st pass After 2nd pass After 2nd pass Bucket sort After 3rd pass by 10’s by 100’s digit 721 3 3 digit 3 3 9 9 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 123 721 721 38 537 03 721 537 67 478 123 123 003 123 478 537 721 67 67 09 123 38 537 537 009 123 038 478 38 38 478 067 38 67 67 537 9 478 478 721

Invariant: after k passes the low order k digits are sorted.

49 50

Your Turn RadixSort Radixsort: Complexity BucketSort on lsd: • Input:126, 328, 636, 341, 416, 131, 328 • How many passes?

• How much work per pass? 0123456789 • Total time?

BucketSort on next-higher digit: • Conclusion?

• In practice – RadixSort only good for large number of elements with relatively small 0123456789 values – Hard on the cache compared to MergeSort/QuickSort BucketSort on msd:

0123456789 51 52

Summary of sorting

• Sorting choices: – O(N2) – Bubblesort, Insertion Sort – O(N log N) average case running time: • Heapsort: In‐place, not stable. • Mergesort: O(N) extra space, stable. • Quicksort: claimed fastest in practice, but O(N2) worst case. Needs extra storage for recursion. Not stable. – O(N) –Radix Sort: fast and stable. Not comparison based. Not in‐place.