• Abstract and Figures
• Public Full-text

Hybrid Transactional/Analytical Processing: A Survey Fatma Özcan Yuanyuan Tian Pınar Tözün IBM Resarch - Almaden IBM Research - Almaden IBM Research - Almaden [email protected] [email protected] [email protected] ABSTRACT To understand HTAP, we first need to look into OLTP The popularity of large-scale real-time analytics applications and OLAP systems and how they progressed over the years. (real-time inventory/pricing, recommendations from mobile Relational databases have been used for both transaction apps, fraud detection, risk analysis, IoT, etc.) keeps ris- processing as well as analytics. However, OLTP and OLAP ing. These applications require distributed data manage- systems have very different characteristics. OLTP systems ment systems that can handle fast concurrent transactions are identified by their individual record insert/delete/up- (OLTP) and analytics on the recent data. Some of them date statements, as well as point queries that benefit from even need running analytical queries (OLAP) as part of indexes. One cannot think about OLTP systems without transactions. Efficient processing of individual transactional indexing support. OLAP systems, on the other hand, are and analytical requests, however, leads to different optimiza- updated in batches and usually require scans of the tables. tions and architectural decisions while building a data man- Batch insertion into OLAP systems are an artifact of ETL agement system. (extract transform load) systems that consolidate and trans- For the kind of data processing that requires both ana- form transactional data from OLTP systems into an OLAP lytics and transactions, Gartner recently coined the term environment for analysis. Hybrid Transactional/Analytical Processing (HTAP). Many After the seminal paper of Stonebraker [34] arguing for HTAP solutions are emerging both from the industry as well multiple specialized systems, the database field has seen an as academia that target these new applications. While some influx of specialized column-oriented OLAP systems, such as of these are single system solutions, others are a looser cou- BLU[30], Vertica [23], ParAccel, GreenPlumDB, Vectorvise, pling of OLTP databases or NoSQL systems with analytical as well as many in-memory OLTP systems, including VoltDB[35], big data platforms, like Spark. The goal of this tutorial is Hekaton [13], MemSQL [24] among others. The main driver to 1-) quickly review the historical progression of OLTP and for this re-surgency in database engines is the advances in OLAP systems, 2-) discuss the driving factors for HTAP, modern hardware. This second generation of OLAP and and finally 3-) provide a deep technical analysis of existing OLTP systems take better advantage of multi-core, various and emerging HTAP solutions, detailing their key architec- levels of memory caches, and large memories. tural differences and trade-offs. At the same time, the last decade seen an explosion of many big data technologies, driven by new generation appli- cations. NoSQL or key-value stores, such as Voldemort[32], 1. INTRODUCTION Cassandra[8], RocksDB [31], offer fast inserts and lookups, In this tutorial, we plan to survey existing and emerging and very high scale out, but lack in their query capabilities, HTAP (Hybrid Transactions and Analytics Processing) so- and offer only loose transactional guarantees (see Mohan's lutions. HTAP is a term created by Gartner to describe tutorial[25]). There have been also many SQL-on-Hadoop systems that can support both OLTP (On-line transaction [10] offerings, including Hive [36], Big SQL [15], Impala [20], processing) as well as OLAP (on-line analytics processing) and Spark SQL[3], that provide analytics capabilities over within a single transaction. However, the term HTAP is large data sets, focusing on OLAP queries only, and lacking currently used more broadly, even for solutions that sup- transaction support. Although all these systems support port insertions (not necessarily ACID transactions) as well queries over text and CSV files, their focus have been on as OLAP queries. Some of these systems have the ability to columnar storage formats, like ORCFile[27], and Parquet run analytical queries over the very recent data, while others need some delay before the queries see the latest data. [1]. Recent years have seen the need for more real-time ana- lytics. In addition, mobile and Internet of Things have given Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed rise to a new generation of applications that are character- for profit or commercial advantage and that copies bear this notice and the full cita- ized by heavy ingest rates, i.e. they produce large amounts tion on the first page. Copyrights for components of this work owned by others than of data in a short time, as well as their need for more real- ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- time analysis. Enterprises are pushing for more real-time publish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. analysis of their data to drive competitive advantage, and SIGMOD ’17, May 14–19, 2017, Chicago, IL, USA. as such they need the ability to run analytics on their oper- c 2017 ACM. ISBN 978-1-4503-4197-4/17/05. $15.00 ational data as soon as possible. DOI: http://dx.doi.org/sigt013 With these developments, there is now a lot of interest, data organizations for OLTP and OLAP workloads, respec- and research focus on providing HTAP solutions over big tively. data sets. In this tutorial, we plan to provide a quick histor- On the other hand, from the beginning, HyPer [19] was ical perspective into the progression of different technologies designed to support both fast transactions and analytics us- and discuss the current set of HTAP solutions. We will ex- ing one engine. Even though, initially it used row-wise pro- amine the different architectural aspects of the current so- cessing of data for both OLTP and OLAP, today it also lutions, identifying their strengths and weaknesses. We will provides the option for choosing a columnar format to be categorize existing systems along many technological dimen- able to run the analytical requests more efficiently. sions, and provide deep dives into a few representative sys- Finally, the recent academic project Pelaton [28] aims to tems. Finally, we will discuss existing research challenges to build an autonomous in-memory HTAP system. It provides realize true HTAP, where a single transaction can contain adaptive data organization [4], which changes the data for- both insert/update/delete statements, as well as complex mat at run-time based on the type of requests. OLAP queries. All these systems require converting the data between row and columnar formats for transactions and analytics. Due 2. HTAP SOLUTIONS: DESIGN OPTIONS to these conversions, the latest committed data might not HTAP solutions today follow a variety of design practices. be available to the analytical queries right away for these This part of the tutorial goes over them to highlight their types of systems. main trade-offs while giving examples from industrial offer- ings and academic solutions. 2.1.2 Same Data Organization for both OLTP and One of the major design decisions HTAP systems have OLAP to make is whether or not to use the same engine for both H2TAP [2] is an academic project that aims to build an OLTP and OLAP requests. HTAP system focusing mainly on the hardware utilization of a single node when running on heterogeneous hardware. 2.1 Single System for OLTP and OLAP It falls under this category since the system is designed as a The traditional relational databases (e.g., DB2, Oracle, row-store. Microsoft SQL Server) have the ability to support OLTP and Among the SQL-on-Hadoop systems, there has also been OLAP in one engine using single type of data organization HTAP solutions that extend existing OLAP systems with (mainly row-stores). However, they are not very efficient for the ability to update data. Hive, since version 0.13, has in- either of these workloads. troduced the transaction support (insert, update, and delete) Therefore, following the one size doesn't fit all rule [34], at the row level [18] for ORCFile, their columnar data for- the past decade has seen the rise of specialized engines for mat. However, the primary use cases are for updating di- OLTP and OLAP exploiting the advances in modern hard- mension tables and streaming data ingest. The integration ware (larger main-memory, multicores, etc.). Various ven- of Impala [20] with the storage manager Kudu [21], also dors and academic groups have built in-memory optimized allows the SQL-on-Hadoop engine to handle updates and row-stores (e.g., VoltDB [35], Hekaton [13], MemSQL [24], deletes. The same Kudu storage is also used for running Silo [37], ...) and column-stores (e.g., MonetDB [7], Ver- analytical queries. tica [23], BLU [30], SAP HANA [14], ...) specialized for Since these systems do not require conversion from one transactional and analytical processing, respectively. These data organization to another in order to perform transac- systems departed from traditional code-bases of relational tional and analytical requests, the OLAP queries can read databases, and built leaner engines from scratch to also avoid the latest committed data. However, they might face the the large instruction footprint of the traditional engines. same shortcomings the traditional relational engines faced. However, many of the systems optimized for one type of They do not have a data organization that is optimal for processing, later started adding support for the other type in both types of processing.

Load more

  5

Copy Link