GPU-Quicksort: A Practical Quicksort Algorithm for Graphics Processors DANIEL CEDERMAN and PHILIPPAS TSIGAS Chalmers University of Technology In this paper we describe GPU-Quicksort, an e±cient Quicksort algorithm suitable for highly par- allel multi-core graphics processors. Quicksort has previously been considered an ine±cient sorting solution for graphics processors, but we show that in CUDA, NVIDIA's programming platform for general purpose computations on graphical processors, GPU-Quicksort performs better than the fastest known sorting implementations for graphics processors, such as radix and bitonic sort. Quicksort can thus be seen as a viable alternative for sorting large quantities of data on graphics processors. Categories and Subject Descriptors: F.2.2 [Nonnumerical Algorithms and Problems]: Sort- ing and searching; G.4 [MATHEMATICAL SOFTWARE]: Parallel and vector implementa- tions General Terms: Algorithms, Design, Experimentation, Performance Additional Key Words and Phrases: Sorting, Multi-Core, CUDA, Quicksort, GPGPU 1. INTRODUCTION In this paper, we describe an e±cient parallel algorithmic implementation of Quick- sort, GPU-Quicksort, designed to take advantage of the highly parallel nature of graphics processors (GPUs) and their limited cache memory. Quicksort has long been considered one of the fastest sorting algorithms in practice for single processor systems and is also one of the most studied sorting algorithms, but until now it has not been considered an e±cient sorting solution for GPUs [Sengupta et al. 2007]. We show that GPU-Quicksort presents a viable sorting alternative and that it can outperform other GPU-based sorting algorithms such as GPUSort and radix sort, considered by many to be two of the best GPU-sorting algorithms. GPU-Quicksort is designed to take advantage of the high bandwidth of GPUs by minimizing the amount of bookkeeping and inter-thread synchronization needed. It achieves this by i) using a two-pass design to keep the inter-thread synchronization low, ii) coa- lescing read operations and constraining threads so that memory accesses are kept Daniel Cederman was supported by Microsoft Research through its European PhD Scholarship Programme and Philippas Tsigas was partially supported by the Swedish Research Council (VR). Authors' addresses: Daniel Cederman and Philippas Tsigas, Department of Computer Sci- ence and Engineering, Chalmers University of Technology, SE-412 96 GÄoteborg, Sweden; email: fcederman,[email protected]. Permission to make digital/hard copy of all or part of this material without fee for personal or classroom use provided that the copies are not made or distributed for pro¯t or commercial advantage, the ACM copyright/server notice, the title of the publication, and its date appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists requires prior speci¯c permission and/or a fee. °c 20YY ACM 0000-0000/20YY/0000-0001 $5.00 ACM Journal Name, Vol. V, No. N, Month 20YY, Pages 1{22. 2 ¢ D. Cederman and P. Tsigas to a minimum. It can also take advantage of the atomic synchronization primitives found on newer hardware to, when available, further improve its performance. Today's graphics processors contain very powerful multi-core processors, for ex- ample, NVIDIA's highest-end graphics processor currently boasts 128 cores. These processors are specialized for compute-intensive, highly parallel computations and they could be used to assist the CPU in solving problems that can be e±ciently data-parallelized. Previous work on general purpose computation on GPUs have used the OpenGL interface, but since it was primarily designed for performing graphics operations it gives a poor abstraction to the programmer who wishes to use it for non-graphics related tasks. NVIDIA is attempting to remedy this situation by providing pro- grammers with CUDA, a programming platform for doing general purpose compu- tation on their GPUs. A similar initiative to CUDA is OpenCL, which speci¯cation has just recently been released [Khronos Group 2008]. Although CUDA simpli¯es the programming, one still needs to be aware of the strengths and limitations of the new platform to be able to take full advantage of it. Algorithms that work great on standard single processor systems most likely need to be altered extensively to perform well on GPUs, which have limited cache memory and instead use massive parallelism to hide memory latency. This means that directly porting e±cient sorting algorithms from the single pro- cessor domain to the GPU domain would most likely yield very poor performance. This is unfortunate, since the sorting problem is very well suited to be solved in parallel and is an important kernel for sequential and multiprocessing computing and a core part of database systems. Being one of the most basic computing prob- lems, it also plays a vital role in plenty of algorithms commonly used in graphics applications, such as visibility ordering or collision detection. Quicksort was presented by C.A.R. Hoare in 1961 and uses a divide-and-conquer method to sort data [Hoare 1961]. A sequence is sorted by recursively dividing it into two subsequences, one with values lower and one with values higher than the speci¯c pivot value that is selected in each iteration. This is done until all elements are sorted. 1.1 Related Work With Quicksort being such a popular sorting algorithm, there have been a lot of di®erent attempts to create an e±cient parallelization of it. The obvious way is to take advantage of its inherent parallelism by just assigning a new processor to each new subsequence. This means, however, that there will be very little parallelization in the beginning, when the sequences are few and large [Evans and Dunbar 1982]. Another approach has been to divide each sequence to be sorted into blocks that can then be dynamically assigned to available processors [Heidelberger et al. 1990; Tsigas and Zhang 2003]. However, this method requires extensive use of atomic synchronization instructions which makes it too expensive to use on graphics processors. Blelloch suggested using pre¯x sums to implement Quicksort and recently Sen- gupta et al. used this method to make an implementation for CUDA [Blelloch 1993; Sengupta et al. 2007]. The implementation was done as a demonstration of their segmented scan primitive, but it performed quite poorly and was an order of ACM Journal Name, Vol. V, No. N, Month 20YY. GPU-Quicksort: A Practical Quicksort Algorithm for Graphics Processors ¢ 3 magnitude slower than their radix-sort implementation in the same paper. Since most sorting algorithms are memory bandwidth bound, there is no surprise that there is currently a big interest in sorting on the high bandwidth GPUs. Purcell et al. [Purcell et al. 2003] have presented an implementation of bitonic merge sort on GPUs based on an implementation by Kapasi et al. [Kapasi et al. 2000]. Kipfer et al. [Kipfer et al. 2004; Kipfer and Westermann 2005] have shown an improved version of the bitonic sort as well as an odd-even merge sort. Gre¼ et al. [Gre¼ and Zachmann 2006] introduced an approach based on the adaptive bitonic sorting technique found in the Bilardi et al. paper [Bilardi and Nicolau 1989]. Govindaraju et al. [Govindaraju et al. 2005] implemented a sorting solution based on the periodic balanced sorting network method by Dowd et al. [Dowd et al. 1989] and one based on bitonic sort [Govindaraju et al. 2005]. They later presented a hybrid bitonic- radix sort that used both the CPU and the GPU to be able to sort vast quantities of data [Govindaraju et al. 2006]. Sengupta et al. [Sengupta et al. 2007] have presented a radix-sort and a Quicksort implementation. Recently, Sintorn et al. [Sintorn and Assarsson 2007] presented a hybrid sorting algorithm which splits the data with a bucket sort and then uses merge sort on the resulting blocks. The implementation requires atomic primitives that are currently not available on all graphics processors. In the following section, Section 2, we present the system model. In Section 3.1 we give an overview of the algorithm and in Section 3.2 we go through it in detail. We prove its time and space complexity in Section 4. In Section 5 we show the results of our experiments and in Section 5.4 and Section 6 we discuss the result and conclude. 2. THE SYSTEM MODEL CUDA is NVIDIA's initiative to bring general purpose computation to their graph- ics processors. It consists of a compiler for a C-based language which can be used to create kernels that can be executed on the GPU. Also included are high performance numerical libraries for FFT and linear algebra. General Architecture The high range graphics processors from NVIDIA that supports CUDA currently boasts 16 multiprocessors, each multiprocessor consisting of 8 processors that all execute the same instruction on di®erent data in lock-step. Each multiprocessor supports up to 768 threads, has 16KiB of fast local memory called the shared memory and have a maximum of 8192 available registers that can be divided between the threads. Scheduling Threads are logically divided into thread blocks that are assigned to a speci¯c multiprocessor. Depending on how many registers and how much local memory the block of threads requires, there could be multiple blocks assigned to a single multiprocessor. If more blocks are needed than there is room for on any of the multiprocessors, the leftover blocks will be run sequentially. The GPU schedules threads depending on which warp they are in. Threads with id 0::31 are assigned to the ¯rst warp, threads with id 32::63 to the next and so on. When a warp is scheduled for execution, the threads which perform the same instructions are executed concurrently (limited by the size of the multiprocessor) whereas threads that deviate are executed sequentially.