Algorithms ROBERT SEDGEWICK | KEVIN WAYNE

2.3

‣ quicksort ‣ selection Algorithms ‣ duplicate keys FOURTH EDITION ‣ system sorts

ROBERT SEDGEWICK | KEVIN WAYNE https://algs4.cs.princeton.edu

Last updated on 3/24/20 5:34 PM Two classic sorting algorithms: mergesort and quicksort

Critical components in the world’s computational infrastructure. ・Full scientific understanding of their properties has enabled us to develop them into practical system sorts. ・Quicksort honored as one of top 10 algorithms of 20th century in science and engineering.

Mergesort. [last lecture]

...

Quicksort. [this lecture]

...

2 Quicksort t-shirt

3 A brief history

Tony Hoare. ・Invented quicksort to translate Russian into English. Learned Algol 60 (and recursion) to implement it. number). 9.9 X 10 45 is used to represent infinity. Imaginary conunent I and J are output variables, and A is the array (with values of x may not be negative and reM values of x may not be subscript bounds M:N) which is operated upon by this procedure. smaller than 1. ・ Partition takes the value X of a random element of the array A, Values of Qd~'(x) may be calculated easily by hypergeometric and rearranges the values of the elements of the array in such a series if x is not too small nor (n - m) too large. Q~m(x) can be way that there exist integers I and J with the following properties : computed from an appropriate set of values of Pnm(X) if X is near M _-< J < I =< NprovidedM < N 1.0 or ix is near 0. Loss of significant digits occurs for x as small as A[R] =< XforM =< R _-< J 1.1 if n is larger than 10. Loss of significant digits is a major diffi- A[R] = XforJ < R < I culty in using finite polynomiM representations also if n is larger A[R] ~ Xfor I =< R ~ N than m. However, QLEG has been tested in regions of x and n The procedure uses an integer procedure random (M,N) which both large and small; chooses equiprobably a random integer F between M and N, and procedure QLEG(m, nmax, x, ri, R, Q); value In, nmax, x, ri; also a procedure exchange, which exchanges the values of its two real In, mnax, x, ri; real array R, Q; parameters ; begin real t, i, n, q0, s; begin real X; integer F; n := 20; F := random (M,N); X := A[F]; if nmax > 13 then I:=M; J:=N; n := nmax +7; up: for I : = I step 1 until N do if ri = 0then if X < A [I] then go to down; begin ifm = 0then I:=N; Tony Hoare Q[0] := 0.5 X 10g((x + 1)/(x - 1)) down: forJ := J step --1 until M do else if A[J]= 0 then Programming S. L. Graham, R. L. Rivest q0 := Q[0]; I:=I+l 3:begin Techniques Editors simpler than many otherActa sorting Informatica algorithms, but 7,empir- 327--355 (1977) Q[0] := s end end; end e:= aXe;f := bXd;goto8 ical [2, ll, 13, 21] and 9 analytic by Springer-Verlag [9] studies show that 1977 if x = 1 then else if F < J tllen begin exchange (A[F], A[J]) ; end 3 ; Implementing Quicksort can be expected to be up to twice as fast as its Q[0] := 9.9 I" 45; J:=J-1 e:=bXc; nearest competitors. The method is simple enough to be R[n + 1] := x - sqrt(x X x - 1); end ; ifd ~ 0then learned by programmers who have no previous experi- for i := n step --1 until 1 do end partition 4: begin Quicksort Programs ence with sorting, and those who do know other sorting R[i] := (i + m)/((i + i + 1) X x f:=bXd; goto8 methods should also find it profitable to learn about +(m-i- 1) X R[i+l]); end 4; Robert Sedgewick ALGORITHM 61 go to the end; f:=aXd; goto8Brown University Quicksort. if m = 0 then ALGORITHMPROCEDURES 64FOR RANGE ARITHMETIC 5: end 2; Because of its prominence, it is appropriate toThe study Analysis of Quicksort Programs* begin if x < 0.5 tben QUICKSORTALLAN GIBB* ifb > 0 then how Quicksort might be improved. This subject has Q[0] := arctan(x) - 1.5707963 else University of Alberta, Calgary, Alberta, Canada 6: begin C. A. R. HOARE This paper is a practical study of how to implement received considerable attention (see, for example, [1, 4, Q[0] := - aretan(1/x)end else if d > 0 then Robert Sedgewick Elliottbegin Brothers Ltd., Borehamwood, Hertfordshire, Eng. 11, 13, 14, 18, 20]), but few real improvements have been begin t := 1/sqrt(x X x + 1); begin the Quicksort and its best variants on procedure RANGESUM (a, b, c, d, e, f); suggested beyond those described by C.A.R. Hoare, the q0 := 0; procedure quicksort (A,M,N); value M,N; e := MIN(a X reald, b computers,X c); including how to apply various code real a,b,c,d,e,f; q[0] := t; array A; integer M,N; f:= MAX(a X c,b X d); go to8 inventor of Quicksort, in his original papers [5, 6]. Hoare Received January 19, t976 comment The term "range number" was used by P. S. Dwyer, optimization techniques. A detailed implementation for i : = 2 step 1 until m do comment Quicksort is a very fast and convenient method of end 6; also showed how to analyze Quicksort and predict its Linear Computations (Wiley, 1951). Machine procedures for combining the most effective improvements to begin s := (x + x) X (i -- 1) X t X Q[0I sorting an array in the random-access store of a computer. The e:= bX c; f := aX c; go to8 running time. The analysis has since been extended to range arithmetic were developed about 1958 by Ramon Moore, Quicksort is given, along with a discussion of how to +(3i+iX i -- 2) × q0; entire contents of the store may be sorted, since no extra space is end 5; Summary. The Quicksort sorting algorithm and its best variants are presented "Automatic Error Analysis in Digital Computation," LMSD implement it in assembly language. Analytic results the improvements that he suggested, and used to indicate qO := Q[0]; required. The average number of comparisons made is 2(M--N) In f:=aXc; Report 48421, 28 Jan. 1959, Lockheed Missiles and Space Divi- describing the performance of the programs are how they may best beand implemented analyzed. [9, 15, Results 17]. The are derived which make it possible to obtain exact formulas de- Q[0] := s end end; (N--M), and the average nmnber of exchanges is one sixth this if d _-< O then sion, Palo Alto, California, 59 pp. If a _< x -< b and c ~ y ~ d, subject of the careful implementation of Quicksort has R[n + 1] := x - sqrt(x × x + 1); amount. Suitable refinements of this method will be desirable for 7: begin summarized. A variety of special situations are scribing the total expected running time of particular implementations on real com- then RANGESUM yields an interval [e, f] such that e =< (x + y) not been studied as widely as global improvements to for i := n step - 1 until 1 do its implementation on any actual computer; e:=bXd; goto8considered from a practical standpoint to illustrate f. Because of machine operation (truncation or rounding) the puters of Quick,sort and an improvement called the median-of-three modification. R[i] := (i + m)/((i -- m + 1) × R[i + 1] begin integer 1,J ; end 7 ; the algorithm, but the savings to be realized are as machine sums a -4- c and b -4- d may not provide safe end-points Quicksort's wide applicability as an internal sorting Detailed analysis of the effect of an implementation technique called loop unwrapping --(i+i+ 1) X x); if M < N then begin partition (A,M,N,I,J); e:=aXd; significant. The history of Quicksort is quite complex, of the output interval. Thus RANGESUM requires a non-local method which requires negligible extra storage. for i : = 1 step 2 until nmax do quicksort (A,M,J) ; 8: e := ADJUSTPROD (e, -1); is presented. The paper is intended not only to present results of direct practical utility, real procedure ADJUSTSUM which will compensate for the Key Words and Phrases: Quicksort, analysis of and [15] contains a full survey of the many variants Rill := -- Rill; quicksort (A, I, N) f := ADJUSTPROD (f, 1) machine arithmetic. The body of ADJUSTSUM will be de- which, have been proposed.but also to illustrate the intriguing mathematics which arises in the complete analysis the: for i : = 1 step 1 until nnmx do end end RANGEMPY; algorithms, code optimization, sorting pendent upon the type of machine for which it is written and so The purpose of this paper is to describe in detail how Q[i] := Q[i - 1] X R[i] end quieksort procedure RANGEDVD CR(a, b, Categories: c, d, e, f) ; 4.0, 4.6, 5.25, 5.31, 5.5 of this important algorithm. is not given here. (An example, however, appears below.) It Quicksort can best be implemented to handle actual end QLEG; real a, b, c, d, e, f; is assumed that ADJUSTSUM has as parameters real v and w, comment If the range divisor includes zero the program applications on real computers. A general description of * This procedure was developed in part under the sponsorship and integer i, and is accompanied by a non-local real procedure exists to a non-local label "zerodvsr". RANGEDVD assumes a the algorithm is followed by descriptions of the most of the Air Force Cambridge Research Center. CORRECTION which gives an upper bound to the magnitude ALGORITHM 65 non-local real procedure ADJUSTQUOT which is analogous effective improvements that have been proposed (as 1. Introduction of the error involved in the machine representation of a number. FIND (possibly identical) to ADJUSTPROD; The output ADJUSTSUM provides the left end-point of the demonstrated in [15]). Next, an implementation of begin In t96t-62 C.A.R. Hoare presented a new algorithm called Quicksort [7, 8] C.output A. R. interval HOARE of RANGESUM when ADJUSTSUM is called Quicksort in a typical high level language is presented, if c =< 0 A d ~ 0 then go to zer0dvsr; ALGORITHM 63 Elliottwith i Brothers= --1, and Ltd., the right Borehamwood, end-point when Hertfordshire, called with i Eng.= 1 Introduction and assembly language implementation issues are con- if c < 0 then which is suitable for putting files into order by computer. This method combines PARTITION The procedures RANGESUB, RANGEMPY, and RANGEDVD sidered. This discussion should easily translate to real procedure find (A,M,N,K); value M,N,K; C. A. R. HOARE provide for the remaining fundamental operations in range 1: begin elegance and efficiency, and it remains today the most useful general-purpose array A; integer M,N,K; ifb > 0 then One of the most widely studied practical problems in languages on real machines. Finally, a number of special Elliott Brothers Ltd., Borehamwood, Hertfordshire, Eng. arithmetic. RANGESQR gives an interval within which the comment Find will assign to A [K] the value which it would 2: begin computer science is sorting: the use of a computer to put issues are considered which may be of importance in square of a range nmnber must lie. RNGSUMC, RNGSUBC, sorting method for computers. The practical utility of the algorithm has meant procedure partition (A,M,N,I,J); value M,N; have if the array A [M:N] had been sorted. The array A will be e := b/d; go filesto3 in order. A person wishing to use a computer to sort RNGMPYC and RNGDVDC provide for range arithmetic with particular sorting applications. array A; integer M,N,1,J; partly sorted, and subsequent entries will be faster than the first; end 2; not only that it has been sfibjected to countless modifications (though few real complex range arguments, i.e. the real and imaginary parts is faced with the problem of determining which of the This paper is intended to be a self-contained overview are range numbers~ e := b/c; many available algorithms is best suited for his purpose. of the properties of Quicksortimprovements for use by those whohave need been suggested beyond those described by Hoare), but also begin Communications of the ACM 321 3: ifa -->_ 0then This task is becoming less difficult than it once was for to actually implement and use the algorithm. A compan- e := ADJUSTSUM (a, c, -1); 4: begin that it has been used .in countless applications, often to sort very large, files. f := a/c; go threeto 8 reasons. First, sorting is an area in which the ion paper [17] provides the analytical results which su- f := ADJUSTSUM (b, d, 1) mathematical has been particu- port much of the discussion presented here. end RANGESUM; end 4; Consequently, it is important to be able to estimate how long an implementation f := a/d; go to larly8 successful: we can predict the performance of many procedure RANGESUB (a, b, c, d, e, f) ; of Quicksort can be expected to run, in order to be able to compare variants or real a, b,c, d,e, f; end 1 ; sorting methods and compare them intelligently. Second, if a < 0 then comment RANGESUM is a non-local procedure; we have a great deal of experience using sorting algo- The Algofithm estimate expenses. Fortunately, as we shall see, this is an algorithm which can be begin 5: begin e := a/c; go to6 Permission to copy without fee all or part of this material is RANGESUM (a, b, -d, --c, e, f) granted provided that the copies are not made or distributed for direct Quicksort is a recursiveanalyzed. method for sorting(Hoare an array recognized this, and gave some analytic results in [8].) It is Bob Sedgewick.end RANGESUB ; end 5 ; commercial advantage, the ACM copyright notice and the title of the e := a/d; publication and its date appear, and notice is given that copying is by A[1], A[2] ..... A[N] by firstpossible "partitioning" to deriveit so that theexact formulas describing the average performance of real procedure RANGEMPY (a, b, c, d, e, f); following conditions hold: real a, b, c, d, e, f; 6: ifb > 0then permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. implementations of the algorithm. comment ADJUSTPROD, which appears at the end of this 7: begin (i) Some key v is in its final position in the array. (If it f := b/c; go to8 This work was supported in part by the Fannie and John Hertz procedure, is analogous to ADJUSTSUM above and is a non- The history of Quicksort is quite complex, and a full survey of the many variants end 7 ; Foundation and in part by NSF Grants. No. GJ-28074 and MCS75- is thejth smallest, it is in position A[j].) local real procedure. MAX and MIN find the maximum and 23738. (ii) All elements to the left of A[j] are less than or equal minimum of a set of real numbers and are non-local; f := b/d; Author's address: Division of Applied Mathematics and Computer which have been proposed is given in [t 7]. In addition, [t 7] gives analytic results begin 8: e := ADJUSTQUOTScience (e, Program,-1); f :=Brown ADJUSTQUOT University, Providence, (f,1) RI 02912. to it. (These elementsdescribing A [ 1], A [2] ..... many A [j - 1of] are the improvements which have been suggested for the purpose Refinedreal v, w; and end popularizedRANGEDVD ; © 1978 ACM 0001-0782/78/1000-0847 $00.75 quicksort.called the "left subtile.") procedure RANGESQR (a, b, e, f); if a < 0A c => 0then 847 of determining which are the most effective. There are many examples in [~ 7] 1: begin real a, b, e, f; Communications October 1978 of Volume 21 comment ADJUSTPROD is a non-10cal procedure; which illustrate that the simplicity of Quicksort is deceiving. The algorithm has ・ v:=c; c:=a; a:=v; w:=d; d:=b; b:=w the ACM Number 10 end 1; begin if a => O then if a < 0 then hidden subtleties which can have significant effects on performance. Furthermore, as we shall see, simple changes to the algorithm or its implementation can radically Analyzed manyCommunications versions of the &CM 319 of quicksort.change the analysis. In this paper, we shall consider in detail how practical implementations of the best versions of Quicksort may be analyzed. ・ In this paper, we will deal with the analysis of: (i) the basic Quicksort algo- rithm; (ii) an improvement called the "median-of-three" modification which reduces the average number of comparisons required; and (iii) an implementation Bob Sedgewick technique called "loop unwrapping" which reduces the amount of overhead per comparison. These particular methods not only represent the most effective vari- * This work was supported in part by the Fannie and John Hertz Foundation, and in part by the National Science Foundation Grants No. GJ-28074 and MCS75-23738. 22 Acta Informatica, Vol. 7 4 2.3 QUICKSORT

‣ quicksort ‣ selection Algorithms ‣ duplicate keys ‣ system sorts

ROBERT SEDGEWICK | KEVIN WAYNE https://algs4.cs.princeton.edu Quicksort overview

Step 1. Shuffle the array. Step 2. Partition the array so that, for some j ・Entry a[j] is in place. “pivot” or “partitioning item” ・No larger entry to the left of j. ・No smaller entry to the right of j. Step 3. Sort each subarray recursively.

input Q U I C K S O R T E X A M P L E shu!e K R A T E L E P U I M Q C X O S partitioning item

partition E C A I E K L P U T M Q R X O S not greater not less sort left A C E E I K L P U T M Q R X O S sort right A C E E I K L M O P Q R S T U X result A C E E I K L M O P Q R S T U X Quicksort overview 6 Quicksort partitioning demo

Repeat until i and j pointers cross. ・Scan i from left to right so long as (a[i] < a[lo]). ・Scan j from right to left so long as (a[j] > a[lo]). ・Exchange a[i] with a[j].

K R A T E L E P U I M Q C X O S

lo i j

7 Quicksort partitioning demo

Repeat until i and j pointers cross. ・Scan i from left to right so long as (a[i] < a[lo]). ・Scan j from right to left so long as (a[j] > a[lo]). ・Exchange a[i] with a[j].

When pointers cross. ・Exchange a[lo] with a[j].

E C A I E K L P U T M Q R X O S

lo j hi

partitioned! 8 The music of quicksort partitioning (by Brad Lyon)

https://learnforeverlearn.com/pivot_music

9 Quicksort partitioning: Java implementation

private static int partition(Comparable[] a, int lo, int hi) { int i = lo, j = hi+1; while (true) { while (less(a[++i], a[lo])) find item on left to swap if (i == hi) break;

while (less(a[lo], a[--j])) find item on right to swap if (j == lo) break;

if (i >= j) break; check if pointers cross exch(a, i, j); swap }

exch(a, lo, j); swap with pivot return j; return index of item now known to be in place before v } lo hi https://algs4.cs.princeton.edu/23quick/Quick.java.html before v during v Յ v Ն v

lo hi i j after before v during v Յ v Ն v Յ v v Ն v

lo hi i j lo j hi during v Յ v Ն v after Յ v v Ն v Quicksort partitioning overview 10 i j lo j hi

after Յ v v Ն v Quicksort partitioning overview

lo j hi

Quicksort partitioning overview Quicksort quiz 1

In the worst case, how many compares and exchanges does partition() make to partition a subarray of length n?

A. ~ ½ n and ~ ½ n

B. ~ ½ n and ~ n

C. ~ n and ~ ½ n

D. ~ n and ~ n scan until ≤ M

scan until ≥ M

M A B C D E V W X Y Z

0 1 2 3 4 5 6 7 8 9 10

n+1 compares in worst case (E and V are each compared with M twice)

⎡n/2⎤ exchanges in worst case (each exchange in while loop gets 2 items in final position)

11 Quicksort: Java implementation

public class Quick { private static int partition(Comparable[] a, int lo, int hi) { /* see previous slide */ }

public static void sort(Comparable[] a) { shuffle needed for StdRandom.shuffle(a); performance guarantee sort(a, 0, a.length - 1); (stay tuned) }

private static void sort(Comparable[] a, int lo, int hi) { if (hi <= lo) return; int j = partition(a, lo, hi); sort(a, lo, j-1); sort(a, j+1, hi); } }

https://algs4.cs.princeton.edu/23quick/Quick.java.html

12 Quicksort trace

lo j hi 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 initial values Q U I C K S O R T E X A M P L E random shu!e K R A T E L E P U I M Q C X O S 0 5 15 E C A I E K L P U T M Q R X O S 0 3 4 E C A E I K L P U T M Q R X O S 0 2 2 A C E E I K L P U T M Q R X O S 0 0 1 A C E E I K L P U T M Q R X O S 1 1 A C E E I K L P U T M Q R X O S 4 4 A C E E I K L P U T M Q R X O S 6 6 15 A C E E I K L P U T M Q R X O S no partition 7 9 15 A C E E I K L M O P T Q R X U S for subarrays of size 1 7 7 8 A C E E I K L M O P T Q R X U S 8 8 A C E E I K L M O P T Q R X U S 10 13 15 A C E E I K L M O P S Q R T U X 10 12 12 A C E E I K L M O P R Q S T U X 10 11 11 A C E E I K L M O P Q R S T U X 10 10 A C E E I K L M O P Q R S T U X 14 14 15 A C E E I K L M O P Q R S T U X 15 15 A C E E I K L M O P Q R S T U X

result A C E E I K L M O P Q R S T U X

Quicksort trace (array contents after each partition)

13 Quicksort animation

50 random items

algorithm position in order current subarray not in order http://www.sorting-algorithms.com/quick-sort

14 Quicksort: implementation details

Partitioning in-place. Using an extra array makes partitioning easier (and stable), but it is not worth the cost.

Loop termination. Terminating the loop is more subtle than it appears.

Equal keys. Handling duplicate keys is trickier that it appears. [stay tuned]

Preserving randomness. Shuffling is needed for performance guarantee. Equivalent alternative. Pick a random pivot in each subarray.

15 Quicksort: empirical analysis (1962)

Running time estimates: ・Algol 60 implementation. ・National Elliott 405 computer.

sorting n 6-word items with 1-word keys Elliott 405 magnetic disc (16K words)

16 Quicksort: empirical analysis

Running time estimates: ・Home PC executes 108 compares/second. ・Supercomputer executes 1012 compares/second.

(n2) mergesort (n log n) quicksort (n log n)

computer thousand million billion thousand million billion thousand million billion

home instant 2.8 hours 317 years instant 1 second 18 min instant 0.6 sec 12 min

super instant 1 second 1 week instant instant instant instant instant instant

Lesson 1. Good algorithms are better than supercomputers. Lesson 2. Great algorithms are better than good ones.

17 Quicksort quiz 2

Why do you think quicksort is faster than mergesort in practice?

A. Fewer compares.

B. Less data movement.

C. Both A and B.

D. Neither A nor B.

18 Quicksort: worst-case analysis

Worst case. Number of compares is ~ ½ n 2.

after random shuffle

19 Quicksort: worst-case analysis

Worst case. Number of compares is ~ ½ n 2.

after random shuffle

Good news. Worst case for quicksort is mostly irrelevant in practice. ・Exponentially small chance of occurring. (unless bug in shuffling or no shuffling) ・More likely that computer is struck by lightning bolt during execution.

20 Quicksort: probabilistic analysis

Proposition. The expected number of compares Cn to quicksort an array of n distinct keys is ~ 2n ln n (and the number of exchanges is ~ ⅓ n ln n ).

Recall. Any algorithm with the following structure takes Θ(n log n) time.

public static void f(int n) {

if (n == 0) return; f(n/2); solve two problems of half the size f(n/2); linear(n); do a linear amount of work }

Intuition. Each partitioning step divides the problem into two subproblems, each of approximately one-half the size.

“close enough”

21 Quicksort: probabilistic analysis

Proposition. The expected number of compares Cn to quicksort an array of n distinct keys is ~ 2n ln n (and the number of exchanges is ~ ⅓ n ln n ).

Pf. Cn satisfies the recurrence C0 = C1 = 0 and for n ≥ 2: left right partitioning

C0 + Cn 1 C1 + Cn 2 Cn 1 + C0 C =(n + 1) + + + ... + n n n n ・Multiply both sides by n and collect terms: partitioning probability

nCn = n(n + 1) + 2(C0 + C1 + ...+ Cn 1) ・Subtract from this equation the same equation for n - 1:

nCn (n 1) Cn 1 =2n +2Cn 1 Rearrange terms and divide by n (n + 1): ・ analysis beyond

Cn Cn 1 2 scope of this course = + n +1 n n +1

22 Quicksort: probabilistic analysis

・Repeatedly apply previous equation: Cn Cn 1 2 = + n +1 n n +1 Cn 2 2 2 substitute previous equation = + + n 1 n n +1 Cn 3 2 2 2 = + + + n 2 n 1 n n +1 2 2 2 2 = + + + ... + 3 4 5 n +1 ・Approximate sum by an integral: 1 1 1 1 C =2(n + 1) + + + ... + n 3 4 5 n +1 n+1 1 2(n + 1) dx x 3

・Finally, the desired result:

Cn 2(n + 1) ln n 1.39 n lg n 23 Quicksort properties

Quicksort analysis summary. 39% more than mergesort ・Expected number of compares is ~ 1.39 n lg n. [ standard deviation is ~ 0.65 n ] ・Expected number of exchanges is ~ 0.23 n lg n. less data movement than mergesort ・Min number of compares is ~ n lg n. never fewer than mergesort ・Max number of compares is ~ ½ n 2. but never happens

Context. Quicksort is a (Las Vegas) randomized algorithm. ・Guaranteed to be correct. ・Running time depends on outcomes of random coin flips (shuffle).

24 Quicksort properties

Proposition. Quicksort is an in-place sorting algorithm. Pf. ・Partitioning: Θ(1) extra space. ・Function-call stack: Θ(log n) extra space (with high probability).

can guarantee Θ(log n) depth by recurring on smaller subarray before larger subarray (but requires using an explicit stack)

Proposition. Quicksort is not stable. Pf. [ by counterexample ]

i j 0 1 2 3

B1 C1 C2 A1

1 3 B1 C1 C2 A1

1 3 B1 A1 C2 C1

0 1 A1 B1 C2 C1

25 Quicksort: practical improvements

Insertion sort small subarrays. ・Even quicksort has too much overhead for tiny subarrays. ・Cutoff to insertion sort for ≈ 10 items.

private static void sort(Comparable[] a, int lo, int hi) { if (hi <= lo + CUTOFF - 1) { Insertion.sort(a, lo, hi); return; }

int j = partition(a, lo, hi); sort(a, lo, j-1); sort(a, j+1, hi); }

26 Quicksort: practical improvements

Median of sample. ・Best choice of pivot item = median. ・Estimate true median by taking median of sample. ・Median-of-3 (random) items.

~ 12/7 n ln n compares (14% fewer) ~ 12/35 n ln n exchanges (3% more)

private static void sort(Comparable[] a, int lo, int hi) { if (hi <= lo) return;

int median = medianOf3(a, lo, lo + (hi - lo)/2, hi); swap(a, lo, median);

int j = partition(a, lo, hi); sort(a, lo, j-1); sort(a, j+1, hi); }

27 2.3 QUICKSORT

‣ quicksort ‣ selection Algorithms ‣ duplicate keys ‣ system sorts

ROBERT SEDGEWICK | KEVIN WAYNE https://algs4.cs.princeton.edu Selection

Goal. Given an array of n items, find item of rank k.

Ex. Min (k = 0), max (k = n − 1), median (k = n / 2).

Applications. ・Order statistics. ・Find the “top k.”

Use complexity theory as a guide. ・Easy O(n log n) algorithm. How? ・Easy O(n) algorithm for k = 0, 1, 2. How? ・Easy Ω(n) lower bound. Why?

Which is true? ・O(n) algorithm? [ is there a linear-time algorithm? ] ・Ω(n log n) lower bound? [ is selection as hard as sorting? ]

29 Quick-select

Partition array so that: ・Entry a[j] is in place. ・No larger entry to the left of j. ・No smaller entry to the right of j.

Repeat in one subarray, depending on j; finished when j equals k.

public static Comparable select(Comparable[] a, intbefore k) v { lo if a[k] is here if a[k] is herehi StdRandom.shuffle(a); set hi to j-1 set lo to j+1 int lo = 0, hi = a.length - 1; during v Յ v Ն v while (hi > lo) { i j int j = partition(a, lo, hi); after Յ v v Ն v if (j < k) lo = j + 1; else if (j > k) hi = j - 1; lo j hi else return a[k]; } Quicksort partitioning overview return a[k]; }

30 Quick-select: probabilistic analysis

Proposition. The expected number of compares Cn to quick-select the item of rank k in an array of length n is Θ(n).

“close enough”

Intuition. Each partitioning step approximately halves the length of the array. Recall. Any algorithm with the following structure takes Θ(n) time.

public static void f(int n) {

if (n == 0) return; n + n / 2 + n / 4 + … + 1 ~ 2n linear(n); do a linear amount of work f(n/2); solve one problem of half the size

}

Careful analysis yields: Cn ~ 2 n + 2 k ln (n / k) + 2 (n – k) ln (n / (n – k)) ≤ (2 + 2 ln 2) n max occurs for median (k = n / 2) ≈ 3.38 n 31 Quicksort quiz 3

What is the worst-case running time of quick select?

A. Θ(n)

B. Θ(n log n)

2 but never happens if you shuffle C. Θ(n ) (may happen if you don’t shuffle!) D. Θ(2n)

select(a, 14)

after random shuffle

… 32 Theoretical context for selection

Proposition. [Blum–Floyd–Pratt–Rivest–Tarjan, 1973] Compare-based

selection algorithmJOURNAL whose OF COMPUTER worst-caseAND SYSTEM SCIENCES 7, 448-461 running (1973) time is linear.

Time Bounds for Selection*

MANUEL BLUM, ROBERT W. FLOYD, VAUGHAN PRATT, RONALD L. RIVEST, AND ROBERT E. TARJAN

Department of Computer Science, Stanford University, Stanford, California 94305 Received November 14, 1972

The number of comparisons required to select the i-th smallest of n numbers is shown to be at most a linear function of n by analysis of a new --PICK. Specifically, no more than 5.4305 n comparisons are ever required. This bound is improved for extreme values of i, and a new lower bound on the requisite number of comparisons is also proved.

1. INTRODUCTION

Remark. ConstantsIn thisare paper high we present ⇒ a new not selection used algorithm, in PICK, practice. and derive by an analysis of its efficiency the (surprising) result that the cost of selection is at most a linear function of the number of input items. In addition, we prove a new lower bound for the cost of selection. The selection problem is perhaps best exemplified by the computation of medians. Use theory as a guide.In general, we may wish to select the i-th smallest of a set of n distinct numbers, or the element ranking closest to a given percentile level. Still worthwhileInterest to seekin this problem practical may be traced linear-time to the realm of sports (worst-case) and the design of algorithm. ・ (traditionally, tennis) tournaments to select the first- and second-best players. In Until one is discovered,1883, Lewis Carroll publisheduse quick-select an article [1] denouncing (ifthe unfairyou method don’t by which need a full sort). ・ the second-best player is usually determined in a "knockout tournament" -- the loser of the final match is often not the second-best! (Any of the players who lost only 33 to the best player may be second-best.) Around 1930, Hugo Steinhaus brought the problem into the realm of algorithmic complexity by asking for the minimum number of matches required to (correctly) select both the first- and second-best players from a field of n contestants. In 1932, J. Schreier [8] showed that no more than n + [logg(n)]- 2 matches are required, and in 1964, S. S. Kislitsin [6] proved this number to be necessary as well. Schreier's method uses a knockout tournament to determine the winner, followed by a second knockout tournament among the

* This work was supported by the National Science Foundation under grant GJ-992. 448 Copyright © I973 by Academic Press, Inc. All rights of reproduction in any form reserved. 2.3 QUICKSORT

‣ quicksort ‣ selection Algorithms ‣ duplicate keys ‣ system sorts

ROBERT SEDGEWICK | KEVIN WAYNE https://algs4.cs.princeton.edu Duplicate keys

Often, purpose of sort is to bring items with equal keys together. ・Sort population by age. ・Remove duplicates from mailing list. ・Sort job applicants by college attended. sorted by time sorted by city (unstable) sorted by city (stable) Chicago 09:00:00 Chicago 09:25:52 Chicago 09:00:00 Typical characteristics of such applications.Phoenix 09:00:03 Chicago 09:03:13 Chicago 09:00:59 Houston 09:00:13 Chicago 09:21:05 Chicago 09:03:13 Huge array. Chicago 09:00:59 Chicago 09:19:46 Chicago 09:19:32 ・ Houston 09:01:10 Chicago 09:19:32 Chicago 09:19:46 Small number of key values. Chicago 09:03:13 Chicago 09:00:00 Chicago 09:21:05 ・ Seattle 09:10:11 Chicago 09:35:21 Chicago 09:25:52 Seattle 09:10:25 Chicago 09:00:59 Chicago 09:35:21 Phoenix 09:14:25 Houston 09:01:10 Houston 09:00:13 NOT Chicago 09:19:32 Houston 09:00:13 Houston 09:01:10 sorted Chicago 09:19:46 Phoenix 09:37:44 sorted Phoenix 09:00:03 Chicago 09:21:05 Phoenix 09:00:03 Phoenix 09:14:25 Seattle 09:22:43 Phoenix 09:14:25 Phoenix 09:37:44 Seattle 09:22:54 Seattle 09:10:25 Seattle 09:10:11 Chicago 09:25:52 Seattle 09:36:14 Seattle 09:10:25 Chicago 09:35:21 Seattle 09:22:43 Seattle 09:22:43 Seattle 09:36:14 Seattle 09:10:11 Seattle 09:22:54 Phoenix 09:37:44 Seattle 09:22:54 Seattle 09:36:14

Stability when sorting on a second key key

35 Quicksort quiz 4

When partitioning, how to handle keys equal to partitioning key?

scan until > P scan until < P

A. P G E P A Q B P C O U P Z S

scan until ≥ P scan until ≤ P

B. P G E P A Q B P C O U P Z S

C. Either A or B.

36 War story (system sort in C)

Bug. A qsort() call in C that should have taken seconds was taking minutes to sort a random array of 0s and 1s.

Why is qsort() so slow?

A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 skip over equal keys

i j

A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 stop scan on equal keys

i j 37 Duplicate keys: partitioning strategies

Bad. Don’t stop scans on equal keys. [ Θ(n 2) compares when all keys equal ]

B A A B A B B B C C C A A A A A A A A A A A

Good. Stop scans on equal keys. [ ~ n lg n compares when all keys equal ]

B A A B A B C C B C B A A A A A A A A A A A

Better. Put all equal keys in place. How? [ ~ n compares when all keys equal ]

A A A B B B B B C C C A A A A A A A A A A A

38 DUTCH NATIONAL FLAG PROBLEM

Problem. [Edsger Dijkstra] Given an array of n buckets, each containing a red, white, or blue pebble, sort them by color.

input

sorted

Operations allowed. ・swap(i, j): swap the pebble in bucket i with the pebble in bucket j. ・getColor(i): determine the color of the pebble in bucket i.

Performance requirements. ・Exactly n calls to getColor(). ・At most n calls to swap(). ・Constant extra space. 39 3-way partitioning

Goal. Use pivot v = a[lo] to partition array into three parts so that: ・Red: smaller entries to the left of lt. ・White: equal entries between lt and gt. ・Blue: larger entries to the right of gt.

before v

lo hi before v during v

lo hi lt i gt during v after v

lt i gt lo lt gt hi after v 3-way partitioning lo lt gt hi

3-way partitioning

40 Dijkstra’s 3-way partitioning algorithm: demo

・Let v = a[lo] be pivot. ・Scan i from left to right and compare a[i] to v. – less: exchange a[lt] with a[i]; increment both lt and i – greater: exchange a[gt] with a[i]; decrement gt – equal: increment i

lt i gt

P1 D B X W P2 P3 V P4 A P5 C Y Z

lo hi

before v invariant lo hi

during v

lt i gt

after v 41

lo lt gt hi

3-way partitioning Dijkstra’s 3-way partitioning algorithm: demo

・Let v = a[lo] be pivot. ・Scan i from left to right and compare a[i] to v. – less: exchange a[lt] with a[i]; increment both lt and i – greater: exchange a[gt] with a[i]; decrement gt – equal: increment i

lo lt gt hi

D B C A P5 P2 P3 P1 P4 V W Y Z X

less equal greater

42 3-way quicksort: Java implementation

private static void sort(Comparable[] a, int lo, int hi) { if (hi <= lo) return; int lt = lo, gt = hi; Comparable v = a[lo]; int i = lo + 1; while (i <= gt) { int cmp = a[i].compareTo(v); if (cmp < 0) exch(a, lt++, i++); else if (cmp > 0) exch(a, i, gt--); else i++; } before v

sort(a, lo, lt - 1); lo hi

sort(a, gt + 1, hi); during v } lt i gt after v

lo lt gt hi

3-way partitioning 43 3-way quicksort: visual trace

equal to partitioning element

Visual trace of quicksort with 3-way partitioning

44 Quicksort quiz 5

Which is worst-case number of compares to 3-way quicksort an array of length n containing only 5 distinct values?

A. Θ(n)

B. Θ(n log n)

C. Θ(n 2)

D. Θ(2n)

input

sorted

45 Sorting summary

inplace? stable? best average worst remarks

selection ✔ ½ n 2 ½ n 2 ½ n 2 n exchanges

use for small n insertion ✔ ✔ n ¼ n 2 ½ n 2 or partially sorted arrays

n log n guarantee; merge ✔ ½ n lg n n lg n n lg n stable

improves mergesort timsort ✔ n n lg n n lg n when pre-existing order

n log n probabilistic guarantee; quick ✔ n lg n 2 n ln n ½ n 2 fastest in practice

improves quicksort 3-way quick ✔ 2 n 2 n ln n ½ n when duplicate keys

? ✔ ✔ n n lg n n lg n holy sorting grail

46 2.3 QUICKSORT

‣ quicksort ‣ selection Algorithms ‣ duplicate keys ‣ system sorts

ROBERT SEDGEWICK | KEVIN WAYNE https://algs4.cs.princeton.edu Sorting applications

Sorting algorithms are essential in a broad variety of applications: ・Sort a list of names. Organize an MP3 library. ・ obvious applications ・Display Google PageRank results. ・List RSS feed in reverse chronological order. ・Find the median. ・Identify statistical outliers. problems become easy once ・Binary search in a database. items are in sorted order ・Find duplicates in a mailing list. ・Data compression. Computer graphics. ・ non-obvious applications ・Computational biology. ・Load balancing on a parallel computer. . . .

48 Engineering a system sort (in 1993)

Bentley–McIlroy quicksort. sample 9 items ・Cutoff to insertion sort for small subarrays. ・Pivot selection: median of 3 or Tukey’s ninther. ・Partitioning scheme: Bentley–McIlroy 3-way partitioning.

similar to Dijkstra 3-way partitioning SOFTWARE—PRACTICE AND EXPERIENCE, VOL. 23(11), 1249–1265(but fewer (NOVEMBER exchanges 1993) when not many equal keys)

Engineering a Sort Function

JON L. BENTLEY M. DOUGLAS McILROY AT&T Bell Laboratories, 600 Mountain Avenue, Murray Hill, NJ 07974, U.S.A.

SUMMARY We recount the history of a new qsort function for a C library. Our function is clearer, faster and more robust than existing sorts. It chooses partitioning elements by a new sampling scheme; it partitions by a novel solution to Dijkstra’s Dutch National Flag problem; and it swaps efficiently. Its behavior was assessed with timing and debugging testbeds, and with a program to certify performance. The design techniques apply in domains beyond sorting. KEY WORDS Quicksort Sorting algorithms Performance tuning Algorithm design and implementation Testing

INTRODUCTION C libraries have long included a qsort function to sort an array, usually implemented by In the wild. Hoare’s C, C++, Quicksort. 1JavaBecause 6, existing ….qsorts are flawed, we built a new one. This paper summarizes its evolution. Compared to existing library sorts, our new qsort is faster—typically about twice as fast—clearer, and more robust under nonrandom inputs. It uses some standard Quicksort 49 tricks, abandons others, and introduces some new tricks of its own. Our approach to build- ing a qsort is relevant to engineering other algorithms. The qsort on our home system, based on Scowen’s ‘Quickersort’,2 had served faith- fully since Lee McMahon wrote it almost two decades ago. Shipped with the landmark Sev- enth Edition Unix System,3 it became a model for other qsorts. Yet in the summer of 1991 our colleagues Allan Wilks and Rick Becker found that a qsort run that should have taken a few minutes was chewing up hours of CPU time. Had they not interrupted it, it would have gone on for weeks.4 They found that it took n 2 comparisons to sort an ‘organ- pipe’ array of 2n integers: 123..nn.. 321. Shopping around for a better qsort, we found that a qsort written at Berkeley in 1983 would consume quadratic time on arrays that contain a few elements repeated many times—in particular arrays of random zeros and ones.5 In fact, among a dozen different Unix libraries we found no qsort that could not easily be driven to quadratic behavior; all were derived from the Seventh Edition or from the 1983 Berkeley function. The Seventh

0038-0644/93/111249–17$13.50 Received 21 August 1992  1993 by John Wiley & Sons, Ltd. Revised 10 May 1993 A Java mailing list post (Yaroslavskiy, September 2009)

Replacement of quicksort in java.util.Arrays with new dual-pivot quicksort

Hello All,

I'd like to share with you new Dual-Pivot Quicksort which is faster than the known implementations (theoretically and experimental). I'd like to propose to replace the JDK's Quicksort implementation by new one.

...

The new Dual-Pivot Quicksort uses *two* pivots elements in this manner:

1. Pick an elements P1, P2, called pivots from the array. 2. Assume that P1 <= P2, otherwise swap it. 3. Reorder the array into three parts: those less than the smaller pivot, those larger than the larger pivot, and in between are those elements between (or equal to) the two pivots. 4. Recursively sort the sub-arrays.

The invariant of the Dual-Pivot Quicksort is:

[ < P1 | P1 <= & <= P2 } > P2 ]

...

https://mail.openjdk.java.net/pipermail/core-libs-dev/2009-September/002630.html 50 Another Java mailing list post (Yaroslavskiy–Bloch–Bentley)

Replacement of quicksort in java.util.Arrays with new dual-pivot quicksort

Date: Thu, 29 Oct 2009 11:19:39 +0000 Subject: Replace quicksort in java.util.Arrays with dual-pivot implementation

Changeset: b05abb410c52 Author: alanb Date: 2009-10-29 11:18 +0000 URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/b05abb410c52

6880672: Replace quicksort in java.util.Arrays with dual-pivot implementation Reviewed-by: jjb Contributed-by: vladimir.yaroslavskiy at sun.com, joshua.bloch at google.com, jbentley at avaya.com

! src/share/classes/java/util/Arrays.java + src/share/classes/java/util/DualPivotQuicksort.java

https://mail.openjdk.java.net/pipermail/compiler-dev/2009-October.txt

51 Dual-pivot quicksort

Use two pivots p1 and p2 and partition into three subarrays: ・Keys less than p1. ・Keys between p1 and p2. ・Keys greater than p2.

< p1 p1 ≥ p1 and ≤ p2 p2 > p2

lo lt gt hi

Recursively sort three subarrays (skip middle subarray if p1 = p2).

degenerates to Dijkstra’s 3-way partitioning

In the wild. Java 8, Python unstable sort, Android, …

52 Quicksort quiz 6

Suppose you are the lead architect of a new programming language. Which principal sorting algorithm would you use for the system sort?

A. Mergesort (e.g., Timsort).

B. Quicksort (e.g., dual-pivot quicksort).

C. Both A and B.

D. Neither A nor B.

53 System sorts in Java 8 and Java 11

Arrays.sort() and Arrays.parallelSort(). ・Has one method for Comparable objects. ・Has an overloaded method for each primitive type. ・Has an overloaded method for use with a Comparator. ・Has overloaded methods for sorting subarrays.

Algorithms. ・Timsort for reference types. ・Dual-pivot quicksort for primitive types. ・Parallel mergesort for Arrays.parallelSort().

Q. Why use different algorithms for primitive and reference types?

Bottom line. Use the system sort!

54