QPPT: Query Processing on Prefix Trees Thomas Kissinger, Benjamin Schlegel, Dirk Habich, Wolfgang Lehner Database Technology Group Technische Universität Dresden 01062 Dresden, Germany {firstname.lastname}@tu-dresden.de ABSTRACT Modern database systems have to process huge amounts of data and should provide results with low latency at the same time. To achieve this, data is nowadays typically hold com- pletely in main memory, to benefit of its high bandwidth and low access latency that could never be reached with disks. Current in-memory databases are usually column- stores that exchange columns or vectors between operators and suffer from a high tuple reconstruction overhead. In this paper, we present the indexed table-at-a-time process- ing model that makes indexes the first-class citizen of the database system. The processing model comprises the con- cepts of intermediate indexed tables and cooperative opera- tors, which make indexes the common data exchange format between plan operators. To keep the intermediate index ma- terialization costs low, we employ optimized prefix trees that offer a balanced read/write performance. The indexed table- at-a-time processing model allows the efficient construction of composed operators like the multi-way-select-join-group. Figure 1: Overview of QPPT’s indexed table-at-a-time Pro- Such operators speed up the processing of complex OLAP cessing Model. queries so that our approach outperforms state-of-the-art in-memory databases. the column-at-a-time [6] or the vector-at-a-time processing model [7], avoid certain drawbacks of the traditional query 1. INTRODUCTION processing (e.g., massive virtual function calls) and are thus more suitable for modern in-memory column-stores. How- Processing complex OLAP queries with low latencies as ever, the logical unit of a tuple as present in row-stores well as high throughput on ever-growing, huge amounts of is decomposed in columns resulting in an expensive tuple data is still a major challenging issue for modern database reconstruction overhead. The complexity of the tuple re- systems. With the general availability of high main memory construction procedure grows with an increasing number of capacities, in-memory database systems become more and attributes involved during query processing. more popular in the OLAP domain since often the entire To avoid the tuple reconstruction overhead, we propose to data pool fits into main memory. Hence, the superior char- return to row-oriented systems enhanced with a novel up- acteristics of main memory (e.g., low access latency, high to-date processing model. In this paper, we introduce our bandwidth) could be exploited for query processing, which new indexed table-at-a-time processing model for in-memory leads at the same time to a paradigm shift in query process- row-stores, which is depicted in Figure 1. The main char- ing models. acteristics of this processing model are: (1) intermediate in- The traditional tuple-at-a-time processing model [8] was dexed tables, (2) cooperative operators, and (3) composed found to be sufficient for row-stores as query processing was operators. limited by the I/O of disks. Newer processing models, e.g., Intermediate Indexed Tables Instead of passing plain tuples, columns, or vectors between individual operators, our indexed table-at-a-time processing model exchanges clustered indexes (a set of tuples stored This article is published under a Creative Commons Attribution License within an in-memory index) between operators. In this way, (http://creativecommons.org/licenses/by/3.0/), which permits distribution we (i) eliminate the major weakness of the volcano iterator and reproduction in any medium as well allowing derivative works, pro- vided that you attribute the original work to the author(s) and CIDR 2013. (tuple-at-a-time) model by reducing the number of next calls 6th Biennial Conference on Innovative Data Systems Research (CIDR ’13) to exactly one and (ii) enable data processing using highly January 6-9, 2013, Asilomar, California, USA. efficient operators that can exploit the indexed input data. N 0 1 … 15 4bit 4KB Physically mapped Page 4KB Virtual Page N 0 1 … 15 N 0 1 … 15 4bit 4KB 4KB 4KB 4KB … 4KB 4KB 26bit … … … … … 0 1 … 63 0 1 … 63 6bit N 0 1 … 15 N 0 1 … 15 4bit (a) Prefix Tree (k0 = 4). (b) KISS-Tree. Figure 2: Index Structures Deployed in QPPT. Cooperative Operators troduce Query Processing on Prefix Trees (QPPT), which is Based on the previous point, each operator takes a set of a realization of the indexed table-at-a-time processing model. clustered indexes as input and provides a new indexed table Figure 1 shows a QPPT overview. Leaf-operators, like as output. Our second concept is called cooperative op- the selection, access the base data via a base index and erators and enables a more efficient data handling within build an intermediate prefix tree as output. The successive a query execution plan. An operator’s output index is al- multi-way/star join operator uses the intermediate indexes ways indexed on the attribute(s) requested by the subse- of child operators and one base index. Here, the multi- quent operator. Thus, the complete set of indexed interme- way/star join as an example of an composed operator ex- diate tuples can be transfered from one operator to another ecutes a synchronous index scan, which is an efficient join by passing a single index handle; exchanging plain tuples, algorithm that works on unbalanced trees, like the prefix which would be used eventually to build internal indexes in tree. As result, the multi-way/star join builds an index that the next operator anyway, is not required anymore. A selec- contains already grouped and aggregated values. tion operator, for example, takes a base index as input that Structure of This Paper is indexed on the operators selection predicate and outputs an intermediate table that is indexed on the join predicate The following section gives a short introduction in prefix of a successive join operator. trees as prerequisite for our novel query processing model. Moreover, this section demonstrates the advantages of prefix Composed Operators trees over hash tables in terms of insert and lookup perfor- The set of operators is usually small using the first two con- mance. A general overview on our novel query processing cepts. We can omit all non-index operators and keep only on prefix trees as a realization of the indexed table-at-a-time operators that can exploit indexes as input, i.e., that make processing model is presented in Section 3. The aspects of use of efficient range scans and point accesses within the in- cooperative and composed operators are discussed in Sec- dexed data. Also some base operators like sorting/grouping tion 4, followed by an exhaustive evaluation in Section 5. and aggregation are not necessary anymore, because each The paper closes with related and future work (Section 6 operator already indexes its output. and 7) as well as a conclusion (Section 8). The indexed table-at-a-time processing model allows fur- ther composed (n-ary) operators, which additionally per- 2. QPPT PREREQUISITES form sorting/grouping or aggregation on the data being out- The performance of our indexed table-at-a-time process- putted. Using our composed operators, we are able to reduce ing model heavily depends on the index structures that are expensive materialization costs and data exchange between deployed for query processing. Traditional index structures operators to a minimum, which is highly benefical as pre- like B+-Trees [4] and its main memory optimized version, sented by recent compilation based approaches [12]. For the CSB+-Tree [14], are not suitable because they achieve this reason, composed operators form the core components only a low update performance. Instead, we employ prefix of our novel processing model. trees [5], because Query Processing on Prefix Trees 1. Prefix trees are optimized for a low number of memory One important aspect of our indexed table-at-a-time pro- accesses per key, which results in a high key lookup cessing model is the output index generation within each performance and operator. It amounts to a large fraction of an operator’s 2. They offer a balanced read/write performance, which execution time and thus must be fast. Previous research re- is essential for our indexed table-at-a-time approach. vealed that prefix trees [5] offer all the necessary character- istics (i.e., they are optimized for in-memory processing and Additionally, we deploy a modified KISS-Tree [9], which is achieve high update rates) for our approach. Additionally, a better performing version of the prefix tree, but is lim- we employ the KISS-Tree [9] as a more specialized version ited to 32bit keys. In the following, we briefly discuss both of the prefix tree, which offers a higher read/write perfor- data structures in the context of QPPT. Moreover, we intro- mance than hash tables and allows even more effective batch duce the concept of batch processing on prefix trees and an processing schemes. Based on these data structures, we in- efficient way of handling duplicates to allow an even more 350 350 PT4 GLIB BOOST KISS KISS Batched PT4 300 300 GLIB BOOST 250 250 ] ] s KISS [µ [µs KISS Batched y y 200 200 e e K K r pe per 150 150 me me Ti Ti 100 100 50 50 0 0 1M 16M 64M 1M 16M 64M Number of Keys Number of Keys (a) Insert/Update Performance. (b) Lookup Performance. Figure 3: Performance of Prefix Trees compared to Hash Tables. effective query processing on those prefix tree-based struc- 2.2 KISS-Tree tures.