Integrating Compression and Execution in Column-Oriented Database Systems
Total Page：16
File Type：pdf, Size：1020Kb
Integrating Compression and Execution in Column-Oriented Database Systems Daniel J. Abadi Samuel R. Madden Miguel C. Ferreira MIT MIT MIT [email protected] [email protected] [email protected] ABSTRACT commercial arena [21, 1, 19], we believe the time is right to Column-oriented database system architectures invite a re- systematically revisit the topic of compression in the context evaluation of how and when data in databases is compressed. of these systems, particularly given that one of the oft-cited Storing data in a column-oriented fashion greatly increases advantages of column-stores is their compressibility. the similarity of adjacent records on disk and thus opportuni- Storing data in columns presents a number of opportuni- ties for compression. The ability to compress many adjacent ties for improved performance from compression algorithms tuples at once lowers the per-tuple cost of compression, both when compared to row-oriented architectures. In a column- in terms of CPU and space overheads. oriented database, compression schemes that encode multi- In this paper, we discuss how we extended C-Store (a ple values at once are natural. In a row-oriented database, column-oriented DBMS) with a compression sub-system. We such schemes do not work as well because an attribute is show how compression schemes not traditionally used in row- stored as a part of an entire tuple, so combining the same oriented DBMSs can be applied to column-oriented systems. attribute from different tuples together into one value would We then evaluate a set of compression schemes and show that require some way to \mix" tuples. the best scheme depends not only on the properties of the Compression techniques for row-stores often employ dic- data but also on the nature of the query workload. tionary schemes where a dictionary is used to code wide val- ues in the attribute domain into smaller codes. For example, a simple dictionary for a string-typed column of colors might 1. INTRODUCTION map \blue" to 0, \yellow" to 1, \green" to 2, and so on [13, Compression in traditional database systems is known to 26, 11, 37]. Sometimes these schemes employ prefix-coding improve performance significantly [13, 16, 25, 14, 17, 37]: it based on symbol frequencies (e.g., Huffman encoding [15]) reduces the size of the data and improves I/O performance or express values as small differences from some frame of ref- by reducing seek times (the data are stored nearer to each erence and remove leading nulls from them (e.g., [29, 14, 26, other), reducing transfer times (there is less data to trans- 37]). In addition to these traditional techniques, column- fer), and increasing buffer hit rate (a larger fraction of the stores are also well-suited to compression schemes that com- DBMS fits in buffer pool). For queries that are I/O limited, press values from more than one row at a time. This al- the CPU overhead of decompression is often compensated lows for a larger variety of viable compression algorithms. for by the I/O improvements. For example, run-length encoding (RLE), where repeats of In this paper, we revisit this literature on compression the same element are expressed as (value, run-length) pairs, in the context of column-oriented database systems [28, 9, is an attractive approach for compressing sorted data in a 10, 18, 21, 1, 19]. A column-oriented database system (or column-store. Similarly, improvements to traditional com- \column-store") is one in which each attribute is stored in pression algorithms that allow basic symbols to span more a separate column, such that successive values of that at- than one column entry are also possible in a column-store. tribute are stored consecutively on disk. This is in contrast Compression ratios are also generally higher in column- to most common database systems (e.g. Oracle, IBM DB2, stores because consecutive entries in a column are often quite Microsoft SQL Server) that store relations in rows (\row- similar to each other, whereas adjacent attributes in a tu- stores") where values of different attributes from the same ple are not [21]. Further, the CPU overhead of iterating tuple are stored consecutively. Since there has been signif- through a page of column values tends to be less than that icant recent interest in column-oriented databases in both of iterating through a page of tuples (especially when all the research community [28, 9, 10, 18, 24, 34] and in the values in a column are the same size), allowing for increased decompression speed by using vectorized code that takes ad- vantage of the super-scalar properties of modern CPUs [10, 37]. Finally, column-stores can store different columns in Permission to make digital or hard copies of all or part of this work for different sort-orders [28], further increasing the potential for personal or classroom use is granted without fee provided that copies are compression, since sorted data is usually quite compressible. not made or distributed for profit or commercial advantage and that copies Column-oriented compression schemes also improve CPU bear this notice and the full citation on the first page. To copy otherwise, to performance by allowing database operators to operate di- republish, to post on servers or to redistribute to lists, requires prior specific rectly on compressed data. This is particularly true for com- permission and/or a fee. pression schemes like run length encoding that refer to mul- SIGMOD 2006, June 27–29, 2006, Chicago, Illinois, USA. Copyright 2006 ACM 1-59593-256-9/06/0006 ...$5.00. tiple entries with the same value in a single record. For ex- ample, if a run-length encoded column says the value \42" compression but that have low CPU overhead so that per- appears 1000 times consecutively in a particular column for formance is improved when taking into consideration all rel- which we are computing a SUM aggregate, the operator can evant costs. simply take the product of the value and run-length as the One way that the CPU overhead of compression has been SUM, without having to decompress. reduced over the years is by integrating knowledge about In this paper we study a number of alternative compres- compression into the query executor and allowing some amount sion schemes that are especially well-suited to column stores, of operation directly on compressed data. In early databases, and show how these schemes can easily be integrated into C- data would be compressed on disk and then eagerly decom- Store, an open-source column-oriented database [3, 28]. pressed upon being read into memory. This had the dis- In summary, in this paper we demonstrate several fun- advantage that everything read into memory had to be de- damental results related to compression in column-oriented compressed whether or not it was actually used. Graefe and database systems: Shapiro [13] (and later Goldstein et. al. [14], and Westmann et. al [29], and in the context of column-oriented DBMSs, • We overview commonly used DBMS compression algo- MonetDB/X100 [37]) cite the virtues of lazy decompression, rithms and show how they can be applied in column- where data is compressed on the attribute level and held store systems. We compare this traditional set of algo- compressed in memory, and data is decompressed only if rithms with compression algorithms especially suited needed to be operated on. This has the advantage that some for column-store systems. operations such as a hybrid hash join see improved perfor- • mance by being able to keep a higher percentage of the ta- Through experiments, we explore the trade-offs be- ble in memory, reducing the number of spills to disk. Chen tween these algorithms, varying the characteristics of et. al. [11] note that some operators can decompress tran- the data set and the query workload. We use results siently, decompressing to perform operations such as apply- from these experiments to create a decision tree to aid ing a predicate, but keeping a copy of the compressed data the database designer to decide how to compress a par- and returning the compressed data if the predicate succeeds. ticular column. The idea of decreasing CPU costs by operating directly on compressed data was introduced by Graefe and Shapiro • We introduce an architecture for a query executor that [13]. They pointed out that exact-match comparisons and allows for direct operation on compressed data while natural joins can be performed directly on compressed data minimizing the complexity of adding new compression if the constant portion of the predicate is compressed in the algorithms. We experimentally show the benefits of same way as the data. Also exact-match index lookups are operating directly on compressed data. possible on compressed data if an order-preserving (consis- • We illustrate the importance of introducing as much tent) compression scheme is used. Further, projection and order as possible into columns and demonstrate the duplicate elimination can be performed on compressed data. value of having secondary and tertiary sort orders. However, this is the extent of research on direct operation on compressed data. In particular, to the best of our knowledge, As a caveat, we note that the purpose of this paper is not there has been no attempt to take advantage of some com- to propose fundamental new compression schemes. Many of pression algorithms' ability to represent multiple values in the approaches we employ have been investigated in isolation a single field to simultaneously apply an operation on these in the context of row-oriented databases, and all are known many values at once. In essence, previous work has viewed in the data compression literature (we only propose slight each tuple as compressed or uncompressed, and when oper- variations to these schemes).