The Impact of Columnar In-Memory Databases on Enterprise Systems
Total Page：16
File Type：pdf, Size：1020Kb
The Impact of Columnar In-Memory Databases on Enterprise Systems Implications of Eliminating Transaction-Maintained Aggregates Hasso Plattner Hasso Plattner Institute for IT Systems Engineering University of Potsdam Prof.-Dr.-Helmert-Str. 2-3 14482 Potsdam, Germany [email protected] ABSTRACT between accounts by adding debit and credit entries to the Five years ago I proposed a common database approach for accounts and updating the account balances within transac- transaction processing and analytical systems using a colum- tions. Maintaining dedicated account balances and paying nar in-memory database, disputing the common belief that the price of keeping the aggregated sums up to date on ev- column stores are not suitable for transactional workloads. ery transfer of funds was the only way we could achieve Today, the concept has been widely adopted in academia and reasonable performance as calculating the balance on de- industry and it is proven that it is feasible to run analyti- mand would require the expensive summation of all trans- cal queries on large data sets directly on a redundancy-free fers. However, this concept of maintaining pre-built aggre- schema, eliminating the need to maintain pre-built aggre- gates has three major drawbacks: (i) a lack of flexibility gate tables during data entry transactions. The resulting re- as they do not respond to organizational changes, (ii) added duction in transaction complexity leads to a dramatic simpli- complexity in enterprise applications and (iii) increased cost fication of data models and applications, redefining the way for data insertion as materialized aggregates have to be up- we build enterprise systems. First analyses of productive dated transactionally safe. applications adopting this concept confirm that system ar- In contrast, the most simplistic approach of only record- chitectures enabled by in-memory column stores are concep- ing the raw transaction data allows for simple inserts of new tually superior for business transaction processing compared account movements without the need of complex aggregate to row-based approaches. Additionally, our analyses show maintenance with in-place updates and concurrency prob- a shift of enterprise workloads to even more read-oriented lems. Balances are always calculated by aggregating all ac- processing due to the elimination of updates of transaction- count movements on the fly. This concept overcomes the maintained aggregates. drawbacks mentioned above but is not feasible using row- based databases on large enterprise data. Let us consider the thought experiment of a database with 1. INTRODUCTION almost zero response time. In such a system, we could ex- Over the last decades, enterprise systems have been built ecute the queries of all business applications, including re- with the help of transaction-maintained aggregates as on- porting, analytics, planning, etc. directly on the transac- the-fly aggregations were simply not feasible. However, the tional line items. With this motivation in mind, we started assumption that we can anticipate the right pre-aggregations out to design a common database architecture for trans- for the majority of applications without creating a transac- actional and analytical business applications [16]. The two tional bottleneck was completely wrong. The superior ap- fundamental design decisions are to store data in a columnar proach is to calculate the requested information on the fly layout [3] and keep it permanently resident in main memory. based on the transactional data. I predict that all enterprise I predicted that this database design will replace traditional applications will be built in an aggregation and redundancy row-based databases and today it has been widely adopted free manner in the future. in academia and industry [5, 9, 11, 14, 16, 19]. Consider a classic textbook example from the database lit- The dramatic response time reduction of complex queries erature for illustration purposes, the debit/credit example as processed by in-memory column stores allowed us to drop all illustrated in Figure 1. Traditionally, funds are transferred pre-aggregations and to overcome the three drawbacks men- tioned above. Although a row-based data layout is better suited for fast inserts, the elimination of maintaining aggre- This work is licensed under the Creative Commons Attribution- gates on data entry results in performance advantages of NonCommercial-NoDerivs 3.0 Unported License. To view a copy of this li- columnar-based system architectures for transactional busi- cense, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain per- ness processing [19]. An important building block to com- mission prior to any use beyond those covered by the license. Contact copyright holder by emailing [email protected]. Articles from this volume pensate for the response time of large aggregations has been were invited to present their results at the 40th International Conference on the introduction of a cache for intermediate result sets that Very Large Data Bases, September 1st - 5th 2014, Hangzhou, China. can be dynamically combined with recently entered data [15]. Proceedings of the VLDB Endowment, Vol. 7, No. 13 This caching mechanism is maintained by the database and Copyright 2014 VLDB Endowment 2150-8097/14/08. 1722 Accounts with Raw Account other workload category, but issue both analytical and trans- Transaction-Maintained Balance Transactions actional queries and benefit from unifying both systems [17, 22]. The workloads issued by these applications are referred Insert TRANSACTIONS TRANSACTIONS to as mixed workloads, or OLXP. The concept is enabled by the hardware developments Primary Index BALANCE over the recent years making main memory available in large Read capacities at low prices. Together with the unprecedented Update growth of parallelism through blade computing and multi- core CPUs, we can sequentially scan data in main memory Figure 1: Debit/Credit Example with transaction- with unbelievable performance [16, 24]. In combination with maintained balances compared to on-the-fly calcula- columnar table layouts and light-weight compression tech- tions of balances based on the raw account transac- niques, the technological hardware developments allow to tions. improve column scan speeds further by reducing the data size, splitting work across multiple cores and leveraging the optimization of modern processors to process data in sequen- completely transparent to applications. The main results tial patterns. Instead of optimizing for single point queries of an application design without pre-aggregated information and data entry by moving reporting into separate systems, are a simplified set of programs, a reduction of the data foot- the resulting scan speed allows to build different types of print and a simplified data model. For the remainder of the systems optimized for the set processing nature of business paper, we refer to the classic system architecture that keeps processes. transaction-maintained aggregates on application level and Figure 2 outlines the proposed system architecture. Data stores data in row-based disk databases as row-based ar- modifications follow the insert-only approach and updates chitectures and use column-based architectures in the sense are modeled as inserts and invalidate the updated row with- of a simplified architecture on application level removing out physically removing it. Deletes also only invalidate the transaction-maintained aggregates by leveraging in-memory deleted rows. We keep the insertion order of tuples and columnar databases. only the lastly inserted version is valid. The insert-only ap- In the following, this paper summarizes the column-based proach in combination with multi-versioning [2] allows to architecture in Section 2 and provides arguments why it is keep the history of tables and provides the ability of time- the superior solution for transactional business processing in travel queries [8] or to keep the full history due to legal Section 3. Additionally, a workload analysis of a new simpli- requirements. Furthermore, tables in the hot store are al- fied financial application without transactional-maintained ways stored physically as collections of attributes and meta- aggregates is presented in Section 4. Implications of the data and each attribute consists of two partitions: main and adoption of a column-based architecture as well as optimiza- delta partition. The main partition is dictionary-compressed tions for scalability are described in Section 5. The paper using an ordered dictionary, replacing values in the tuples closes with thoughts on future research and concluding re- with encoded values from the dictionary. In order to min- marks in Sections 6 and 7. imize the overhead of maintaining the sort order, incoming updates are accumulated in the write-optimized delta par- 2. ARCHITECTURE OVERVIEW tition [10, 21]. In contrast to the main partition, data in the delta partition is stored using an unsorted dictionary. Although the concept behind column stores is not new [3, In addition, a tree-based data structure with all the unique 21], their field of application in the last 20 years was limited uncompressed values of the delta partition is maintained per to read-mostly scenarios and analytical processing [21, 12, column [7]. The attribute vectors of both partitions are 6, 20]. In 2009, I proposed to use a columnar database as further compressed using bit-packing mechanisms [24]. Op- the backbone for enterprise systems handling analytical