sign in sign up
<<
Home , Column (database), MonetDB

Column-oriented Systems Daniel J. Abadi Peter A. Boncz Stavros Harizopoulos Yale University CWI HP Labs New Haven, CT, USA Amsterdam, The Netherlands Palo Alto, CA, USA @cs.yale.edu [email protected] [email protected]

ABSTRACT Column-oriented database systems (column-stores) have attracted 2. PART A Fundamentals, application areas, and performance tradeoffs a lot of attention in the past few years. Column-stores, in a nutshell, store each database column separately, with The roots of column-store DBMSs can be traced in the 1970s, attribute values belonging to the same column stored when transposed files were first studied, followed by contiguously, compressed, and densely packed, as opposed to investigations on vertical partitioning as a form of a table attribute traditional database systems that store entire records (rows) one clustering technique. By the mid 1980s, the advantages of a fully after the other. Reading a subset of a table’s columns becomes decomposed storage model (DSM — a predecessor to column- faster, at the potential expense of excessive disk-head seeking stores) over NSM (traditional -based storage) were from column to column for scattered reads or updates. After documented [5] and subsequent research on and projection several dozens of research papers and at least a dozen of new indices further strengthened the advantages of DSM over NSM. column-store start-ups, several questions remain. Are these a new Despite the suitability of the DSM layout for analytical queries, breed of systems or simply old wine in new bottles? How easily market needs and non-favorable technology trends helped row- can a major row-based system achieve column-store performance? based database systems maintain their foothold. It was not until Are column-stores the answer to effortlessly support large-scale the 2000s that column-store research and commercial systems data-intensive applications? What are the new, exciting system took off. We trace the history of column-stores and examine in research problems to tackle? What are the new applications that detail those technology and application trends that lead to the can be potentially enabled by column-stores? In this tutorial, we resurgence of research and commercialization of column-stores. present an overview of column-oriented database system A columnar data layout dictates much of the basic architectural technology and address these and other related questions. design in a column-store. Each column is stored contiguously on a separate location on disk, typically using large disk pages (or 1. INTRODUCTION large read units) to amortize disk head seeks when scanning The intended audience of the tutorial is database system multiple columns on hard drives. To improve read efficiency, researchers, designers, and practitioners with a keen interest on columnar values are typically densely packed, foregoing the database system architecture and performance, and on database explicit storage of record IDs, and using light-weight compression system support for large-scale, data-intensive applications schemes whenever possible (e.g., [7]). Column scan operators (especially data warehousing and ). The differ from row-scanners in that they are responsible for expected main learning outcomes are: (a) increased understanding translating value position information into disk locations and for of column-based data management system architectures, and in combining and reconstructing (when needed) partial or entire what situations and domains they are suitable for large-scale data tuples out of different columns. Join operators can either rely on analysis, (b) in-depth coverage of advanced storage and column-scanners for receiving reconstructed tuples, or they can processing techniques (several of which have not yet been operate directly on columns by first computing a join index and transferred into commercial systems) and of promising avenues then fetching qualifying values [8]. We cover architectural for academic and industrial research, and () surveying the state- considerations and implementation of basic query processing of-the-art in today’s academic and commercial offerings, along mechanisms that are common across column-stores. with promising applications enabled by those systems. We organize the tutorial into three parts which are described next. We then provide an overview of the application areas for column- stores. We first cover the traditional areas like data warehousing, business intelligence and , and discuss the trends in these, such as personal data marts (MS Gemini) and the need for on-line updates and loads. Another promising direction is scientific data management, where we showcase some of the Permission to copy without fee all or part of this material is granted results with MonetDB on the Sloan Digital Sky Survey. Finally, provided that the copies are not made or distributed for direct commercial we describe recent work in addressing alternative data models, advantage, the VLDB copyright notice and the title of the publication and its such as object data models, array data models, XML and RDF date appear, and notice is given that copying is by permission of the Very using column-stores. Large Database Endowment. To copy otherwise, or to republish, to post on servers or to redistribute to lists, requires a fee and/or special permissions We conclude the first part with an examination of performance from the publisher, ACM. tradeoffs in row, column, and hybrid data representations, both on VLDB ’09, August 24-28, 2009, Lyon, France. Copyright 2009 VLDB Endowment, ACM 000-0-00000-000-0/00/00. disk and in main memory. The resurgence of on-disk columnar storage was preceded by interest in investigating in-memory data a data access point of , focused on exploiting column-wise layouts for addressing the growing speed disparity between CPU data processing as a way to improve the computational efficiency and RAM in the late 1990s. The main-memory advantages of of database engines, in particular as a way to better exploit the DSM were extensively demonstrated in the MonetDB and PAX features of modern processors. In the case of MonetDB, column- projects. The recent growing popularity of full-blown column- wise query processing was realized by a physical column algebra stores has sparked an interest in understanding performance — the idea being that the simplicity of columnar algebras provide tradeoffs of read-optimized that utilize columnar opportunities for more efficient execution primitives (similar to representation and efficient compression schemes (e.g., [6]). RISC vs. CISC). We explain these opportunities in detail, covering improving instruction throughput in pipelined CPUs, 3. PART B CPU branch prediction efficiency and CPU cache efficiency, and also exploiting SIMD instructions and GPU hardware. Column-store internals & advanced query processing techniques One of the most-often cited advantages of column-stores is data In X100, the benefits of the column algebras were conserved, but compression. The intuitive argument for why column-stores extended with pipelined query processing in its vectorized query compress data well is that compression perform better processing model. We elaborate on the challenges and on data with low information entropy (high data value locality). opportunities of vectorized query processing and also cover We discuss some compression algorithms that work particularly X100’s new vectorized columnar compression schemes, as well as well in column-store databases, and how one can architect a query its innovations in column-store updates. Finally, we discuss the executer to operate directly on compressed data (so that the tension between magnetic disk capabilities and data access needs system can achieve the I/O performance benefits of compression of data warehousing and column-stores, and outline work on without paying the CPU cost of decompression) [1, 9]. We also intelligent I/O scheduling for column-stores (e.g., in the area of discuss some recent work that argues that much of the data clustering and cooperative scans [10]). This discussion also compression benefits of column-stores are not necessarily unique covers the potential impact of solid state drives (SSD) technology to columns-stores; if fact, they can also be applied to row-stores. on column storage (e.g., [8]). Two of the most-often cited disadvantages of column-stores are We conclude the tutorial with a comprehensive review of write operations and tuple construction. Write operations are commercially available column-stores, including Kdb, MonetDB, generally considered problematic for two reasons: (a) inserted , Vertica/C-Store, Sybase IQ, Infobright, Exasol, tuples have to be broken up into their component attributes and ParAccel, the SAP BI accelerator, and Kickfire. each attribute must be written separately, and (b) the dense- packed data layout makes moving tuples within a page nearly 5. REFERENCES impossible. We discuss some of the techniques that column-stores [1] Abadi, D.J., Madden, S.., and Ferreira, M.: Integrating use to mitigate their fundamental write issues, such as in-memory compression and execution in column-oriented database buffering, tuple moving, and -merging. Tuple systems. In Proc. SIGMOD, 2006. construction is also considered problematic since information [2] Abadi, D.J., Myers, D.S., DeWitt, D.J., and Madden, S.R.: about a logical entity is stored in multiple locations on disk, yet Materialization strategies in a column-oriented DBMS. In most queries access more than one attribute from an entity. Proc. ICDE, 2007. Further, most database standards (e.g., ODBC and JDBC) access results entity-at-a-time (not column-at-a-time). Thus, at some [3] Abadi, D.J., Madden, S.R., and Hachem, N.: Column-stores point in a , data from multiple columns must be vs. row-stores: how different are they really? In Proc. combined into ‘rows’ of information about an entity. We discuss SIGMOD, 2008. techniques developed to reduce such construction costs (e.g., [2]). [4] Boncz, P.A. Monet: A Next-Generation DBMS Kernel For To provide a better understanding of some of the distinguishing Query-Intensive Applications. Ph.D. Thesis, Universiteit van features of the internals of column-stores, we discuss some recent Amsterdam, Amsterdam, The Netherlands, May 2002. work that attempts to simulate column-stores inside row-stores [5] Copeland, G.P., Khoshafian, S.N.: A Decomposition Storage (e.g., [3]). This work isolates some of the key techniques that Model. In Proc. SIGMOD, 1985. result in performance gains for column-store databases, and [6] Harizopoulos, S., Liang, V., Abadi, D.J., and Madden, S.: motivates an effort, both in research and industry, to build various Performance tradeoffs in read-optimized databases. In Proc. types of hybrid systems, and achieve some convergence between VLDB, 2006. column-store and row-store technologies. We also discuss some open research problems in the context of column-store systems, [7] Stonebraker, M. et al.: C-Store: A Column-oriented DBMS. including physical , indexing techniques, parallel In Proc. VLDB, 2005. query execution, replication, and load balancing. [8] Tsirogiannis, D., Harizopoulos, S., Shah, M.A., Wiener, J.L., and Graefe, G.: Query processing techniques for solid state 4. PART C drives. In Proc. SIGMOD, 2009. Case study and review of column-store products [9] Zukowski, M., Heman, S., Nes, N., and Boncz, P.A.: Super- We illustrate many of the techniques outlined, by taking an in- scalar ram- compression. In Proc. ICDE, 2006. depth look at the architecture of MonetDB/X100, which is, in [10] Zukowski, M., Heman, S., Nes, N., and Boncz, P.A. fact, two different column-stores, developed at CWI. Both X100 Cooperative Scans: Dynamic Bandwidth Sharing in a [9] and MonetDB [4], apart from pursuing columnar storage from DBMS. In Proc. VLDB, 2007.