Integer Sorting on the word-RAM Uri Zwick Tel Aviv University May 2015 Last updated: June 30, 2015 Integer sorting Memory is composed of 푤-bit words. Arithmetical, logical and shift operations on 푤-bit words take 푂(1) time. How fast can we sort an array of length 푛? Comparison based algorithms Θ(푛 log 푛) Time bounds dependent on 푤 Θ(푛 log 푤) - van Emde Boas trees Time bounds independent of 푤 O(푛 log 푛/ log log 푛) - Fredman-Willard (1993) O(푛 log log 푛) - Andersson et al. (1998) O 푛 log log 푛 - Han-Thorup (2002) Some of these algorithms are randomized Fundamental open problem Can we sort in 푂(푛) time, for any 푤 ≥ log 푛 ??? Sorting – three variants 푆표푟푡(푛, 푤, 푏) – Return a permutation that (stably) sorts 푛 푏-bit keys, each stored in a separate word, on a machine with 푤-bit words. 푆표푟푡′(푛, 푤, 푏) – (Stably) sort 푛 푤-bit words according to 푏-bit keys. (The keys are the first/last 푏 bits of each word.) 푆표푟푡′′(푛, 푤, 푏) – Sort 푛 푏-bit keys, each stored in a separate word, on a machine with 푤-bit words. (We will not be very precise in using these names…) Sorting – three variants Exercise: Reduce 푆표푟푡′′(푛, 푤, 푏) to 푆표푟푡′(푛, 푤, 푏) with no extra work. Exercise: Reduce 푆표푟푡′(푛, 푤, 푏) to 푆표푟푡(푛, 푤, 푏) with only 푂(푛) extra work. Exercise: Reduce 푆표푟푡(푛, 푤, 푏) to 푆표푟푡′′(푛, 푤, 푏 + log 푛) with only 푂(푛) extra work. Exercise: Reduce 푆표푟푡(푛, 푤, 푏) to 푆표푟푡′′(푛, 푤, 푏) with only 푂(푛) extra work. (Hint: hashing.) Two techniques Range reduction Reduce 푆표푟푡(푛, 푤, 푏) to 푆표푟푡(푛, 푤, 푏/2) using only 푂(푛) extra work. Packed sorting Solve 푆표푟푡′′(푛, 푤, 푏) by packing 푤/푏 keys in each word. In 푂(1) time we can perform simple operations on 푤/푏 keys. (Word-level parallelism) Backward/LSD Radix sort Stably sort according to “digits”. Starting from least significant digit. 2 8 7 1 2 8 7 1 4 5 9 1 4 5 9 1 To sort according 6 5 7 2 1 3 0 1 to a “digit” use 1 3 0 1 6 5 7 2 2 4 7 2 2 4 7 2 bucket or count sort. 3 5 5 5 7 0 2 2 7 0 2 2 8 3 9 4 8 3 9 4 4 8 4 4 Slides from 4 8 4 4 3 5 5 5 undergrad course 3 5 3 6 3 5 3 6 Backward/LSD Radix sort Stably sort according to “digits”. Starting from least significant digit. 2 8 7 1 4 5 9 1 1 3 0 1 6 5 7 2 2 4 7 2 7 0 2 2 8 3 9 4 4 8 4 4 3 5 5 5 3 5 3 6 Backward/LSD Radix sort Stably sort according to “digits”. Starting from least significant digit. 2 8 7 1 1 3 0 1 After the -th4 5pass,9 1 7 0 2 2 numbers are1 sorted3 0 1 3 5 3 6 6 5 7 2 4 8 4 4 according 2to4 the7 2 3 5 5 5 least significant7 0 2 2 2 8 7 1 digits8 3 9 4 6 5 7 2 4 8 4 4 2 4 7 2 3 5 5 5 4 5 9 1 3 5 3 6 8 3 9 4 Backward/LSD Radix sort Stably sort according to “digits”. Starting from least significant digit. 1 3 0 1 7 0 2 2 3 5 3 6 4 8 4 4 3 5 5 5 2 8 7 1 6 5 7 2 2 4 7 2 4 5 9 1 8 3 9 4 Backward/LSD Radix sort Stably sort according to “digits”. Starting from least significant digit. 1 3 0 1 7 0 2 2 7 0 2 2 1 3 0 1 3 5 3 6 8 3 9 4 4 8 4 4 2 4 7 2 3 5 5 5 3 5 3 6 2 8 7 1 3 5 5 5 6 5 7 2 6 5 7 2 2 4 7 2 4 5 9 1 4 5 9 1 4 8 4 4 8 3 9 4 2 8 7 1 Backward/LSD Radix sort Stably sort according to “digits”. Starting from least significant digit. 7 0 2 2 1 3 0 1 8 3 9 4 2 4 7 2 3 5 3 6 3 5 5 5 6 5 7 2 4 5 9 1 4 8 4 4 2 8 7 1 Backward/LSD Radix sort Stably sort according to “digits”. Starting from least significant digit. 7 0 2 2 1 3 0 1 1 3 0 1 2 4 7 2 8 3 9 4 2 8 7 1 2 4 7 2 3 5 3 6 3 5 3 6 3 5 5 5 3 5 5 5 4 5 9 1 6 5 7 2 4 8 4 4 4 5 9 1 6 5 7 2 4 8 4 4 7 0 2 2 2 8 7 1 8 3 9 4 (Adapted from Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms, Third Edition, 2009, p. 195) Backward/LSD Radix Sort in the word-RAM model Sort according to least significant log 푛 bits. Stably sort according to remaining 푤 − log 푛 bits. Least significant log 푛 bits can be sorted in 푂(푛) time using bucket sort or count sort. 푤 Total running time is 푂 푛 ⋅ . log 푛 Running time is 푂(푛) if 푤 = 푂(log 푛) . We shall revisit Radix Sort later… “Ultra” Radix Sort [Kirkpatrick-Reisch (1984)] First attempt: Split each 푤-bit word into two 푤/2-bit parts. Sort according to the 푙표푤 part. Scan the array sequentially and append each item into a bucket indexed by ℎℎ. (Each bucket is sorted.) Use a hash table to maintain the non-empty buckets. Sort the indices of the non-empty buckets. Go over the non-empty buckets, in sorted order, and concatenate the sorted lists of the buckets. “Ultra” Radix Sort [Kirkpatrick-Reisch (1984)] First attempt: The algorithm is correct. But, to sort an array of 푤-bit words, we need to sort two arrays of the same length with 푤/2-bit words. We cannot do that more than a constant number of times. To fix the problem we use a trick similar to the one used in van Emde Boas trees. “Ultra” Radix Sort [Kirkpatrick-Reisch (1984)] Working version: Scan the items in arbitrary order and throw them into buckets according to ℎℎ. In each bucket, indexed by ℎℎ, keep only the item(s) with the smallest 푙표푤 value found so far. Put all non-minimal items in a list 퐿. For each non-empty bucket, add (0, ℎℎ) to 퐿. (The length of the 퐿 is at most 푛.) ((0, ℎℎ) replaces the minimal item in bucket ℎℎ.) “Ultra” Radix Sort [Kirkpatrick-Reisch (1984)] Working version (continued): Sort 퐿 according to 푙표푤. Extract from 퐿 a sorted list of the non-empty buckets. (When we encounter the first (0, ℎℎ) pair, check whether bucket ℎℎ is non-empty.) Append each remaining item (ℎℎ, 푙표푤) in 퐿 into bucket ℎℎ. Use the sorted list of non-empty buckets to concatenate the buckets in the appropriate order. Use hashing to maintain the buckets in both phases. “Ultra” Radix Sort [Kirkpatrick-Reisch (1984)] Complexity 푇(푛, 푤) – time to sort 푛 items according to a 푤-bit key 푤 푇 푛, 푤 = 푇 푛, + 푂(푛) 2 푇 푛, 푤 = 푂(푛 log 푤) Matching van Emde Boas trees… In 푂(푛 log log 푛) time, we can reduce 푤 to w/ log 푛 2. We can then pack log2푛 keys is each word! Packed representation 푤/2 bits 0 푥푘−1 0 푥푘−2 0 … 0 푥1 0 푥0 푏 bits 푏 bits test bits 푏 bits We can easily take two such words and produce a word that contains 2푘 keys. Packed representation Useful constants: 푤 bits 퐶1 1 00…0 1 00…0 1 … 1 00…0 1 00…0 퐶2 0 2푘 − 1 0 2푘 − 2 0 … 0 1 0 0 Exercise: How quickly can we construct these constants? Packed representation Useful operation: 퐶표푝푦푇푒푠푡퐵푡푠 푤 bits 1 00…0 1 00…0 1 … 0 00…0 1 00…0 0 11…1 0 11…1 0 … 0 00…0 0 11…1 퐶표푝푦푇푒푠푡퐵푡푠 퐴 ≡ 퐴 − (퐴 ≫ 푏) Packed Sorting [Paul-Simon (1980)] [Albers-Hagerup (1997)] 푆표푟푡′′(푛, 푤, 푏) – Sort 푛 푏-bit keys on a machine with 푤-bit words. 푤 Partition the items into groups of size 푘 = . 2(푏+2) Sort each group naïvely. Pack each group into a single word. Time required for this preliminary step is 푛 푂 ⋅ 푘 log 푘 = 푂 푛 log 푘 . 푘 This is 푂(푛 log log 푛), if 푘 = log 푛 푐. (Packed) Merge Sort 푛 ⋮ ⋯ 푘 Packed Merge Sort [Paul-Simon (1980)] [Albers-Hagerup (1997)] Merge packed sorted sequences of length 푘 to sorted sequences of length 2푘, and then of length 4푘, etc., until a single sorted sequence of length 푛 is obtained. As a basic operation, use the merging of two sorted sequences of length 푘. We shall implement this basic operation in 푂(log 푘) time, by simulating a bitonic sorting network.