Parallel Replication across Formats in SAP HANA for Scaling Out Mixed OLTP/OLAP Workloads Juchang Lee SeungHyun Moon Kyu Hwan Kim SAP Labs Korea Pohang University of Science SAP Labs Korea Seoul, Korea and Technology (POSTECH) Seoul, Korea [email protected] Pohang, Korea [email protected] [email protected] ∗ Deok Hoe Kim Sang Kyun Cha Wook-Shin Han SAP Labs Korea Seoul National University Pohang University of Science Seoul, Korea Seoul, Korea and Technology (POSTECH) [email protected] [email protected] Pohang, Korea [email protected] ABSTRACT 1. INTRODUCTION Modern in-memory database systems are facing the need of Modern database systems need to support mixed work- efficiently supporting mixed workloads of OLTP and OLAP. loads of online transaction processing (OLTP) and online A conventional approach to this requirement is to rely on analytical processing (OLAP) workloads [11, 17, 18]. OLTP ETL-style, application-driven data replication between two workloads contain short-lived, light transactions which read very different OLTP and OLAP systems, sacrificing real- or update small portions of data, while OLAP workloads time reporting on operational data. An alternative approach contain long-running, heavy transactions which reads large is to run OLTP and OLAP workloads in a single machine, portions of data. That is, transactional and analytical be- which eventually limits the maximum scalability of OLAP haviors are mixed in today's workloads. Note that row store query performance. In order to tackle this challenging prob- formats are typically used for handling OLTP workloads, lem, we propose a novel database replication architecture while column store formats are typically used for handling called Asynchronous Parallel Table Replication (ATR). ATR OLAP workloads. supports OLTP workloads in one primary machine, while it A conventional approach to support such mixed work- supports heavy OLAP workloads in replicas. Here, row- loads is to isolate OLTP and OLAP workloads into sep- store formats can be used for OLTP transactions at the pri- arate, specialized database systems, periodically replicat- mary, while column-store formats are used for OLAP ana- ing operational data into a data warehouse for analytics. lytical queries at the replicas. ATR is designed to support Here, we can rely on an external database tool, such as elastic scalability of OLAP query performance while it min- ETL (Extraction-Transformation-Loading) [20, 21]. How- imizes the overhead for transaction processing at the pri- ever, this ETL-style, application-driven data replication be- mary and minimizes CPU consumption for replayed trans- tween two different OLTP system and OLAP systems is in- actions at the replicas. ATR employs a novel optimistic herently unable to achieve real-time reporting. Note that lock-free parallel log replay scheme which exploits charac- we may run OLTP and OLAP workloads in a single ma- teristics of multi-version concurrency control (MVCC) in chine. However, this approach requires an extremely expen- order to enable real-time reporting by minimizing the prop- sive hardware. Previous work such as Hyper [11, 18] focuses agation delay between the primary and replicas. Through on scaling up mixed workloads in a single hardware host, extensive experiments with a concrete implementation avail- which eventually limits the maximum scalability of analyti- able in a commercial database system, we demonstrate that cal query processing. ATR achieves sub-second visibility delay even for update- From analysis of our various customer workloads, we no- intensive workloads, providing scalable OLAP performance tice that one modern server machine can sufficiently handle without notable overhead to the primary. OLTP workloads while heavy OLAP workloads need to be processed in different machines. This architecture can be ∗ corresponding author realized through database replication. In this situation, we need to support 1) real-time and 2) scalable reporting on operational data. In order to support real-time reporting, we need to minimize the propagation delay between OLTP transactions and reporting OLAP queries. In order to sup- This work is licensed under the Creative Commons Attribution- port scalable reporting, query processing throughput should NonCommercial-NoDerivatives 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. For be able to increase accordingly with the increasing number any use beyond those covered by this license, obtain permission by emailing of replicas, elastically depending on the volume of the in- [email protected]. coming workloads. Proceedings of the VLDB Endowment, Vol. 10, No. 12 Copyright 2017 VLDB Endowment 2150-8097/17/08. 1598 Insert page title Application logic Application DB client lib Read/Write Read only Replication Replication Replication SQL processor log log log SQL processor generator sender replayer In-memory DB In-memory DB Primary Replicas Recovery Recovery Checkpoint Checkpoint log log Figure 1: Overall architecture. © 2016 SAP SE or an SAP affiliate company. All rights reserved. Confidential 1 Data replication is a widely studied and popular mech- ious implementation issues. Section 5 presents the results anism for achieving higher availability and higher perfor- of performance evaluations, and Section 6 gives an overview mance [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16]. However, to the of related work. Section 7 summarizes and concludes the best of our knowledge, there is little work on replication from paper. row store to column store for enhancing scalability of ana- lytical query processing. Middleware-based replication [3], which is typically used for replication across different (and 2. ARCHITECTURE heterogeneous) DBMS instances, is not directly comparable to our proposed architecture where both the primary and 2.1 Overall Architecture replicas belong to the same database schema and common Figure 1 shows the overall architecture of ATR. The ATR transaction domain. We also notice that the-state-of-the-art system consists of the primary and one or more replica parallel log replayer [9] is not scalable due to the contention servers, each of which can be connected with another by at the inter-transaction dependency checking. a commodity network interconnect without any shared stor- In this paper, we propose a novel database replication ar- age necessarily. All write requests are automatically directed chitecture called HANA Asynchronous Parallel Table Repli- to the primary server by the database client library, em- cation ( ). is designed to incur low overhead to ATR ATR bedded in the application process. During the course of transaction processing at the primary site while it supports processing a received write request, the primary server gen- elastic scalability of the analytical query performance and erates a replication log entry if the write request makes any shows less CPU consumption for replayed transactions. Thus, change to a replication-enabled table. Note that ATR can it can minimize the propagation (or snapshot) delay between be applied to only a selected list of tables, not necessar- the primary and replicas. ily replicating the entire database. The generated replica- Our contributions are summarized as follows: 1) Through tion log entry is shipped to the replicas via the network deep analysis in design requirements and decisions, we pro- interconnect and then replayed at the replicas. By replay- pose a novel database replication architecture for real-time ing the propagated replication log entries, the in-memory analytical queries on operational data. 2) We propose a database copies of the replicas are maintained in a queriable novel optimistic lock-free parallel log replay scheme which and transactionally-consistent state. The database client li- exploits characteristics of multi-version concurrency control brary transparently routes read-only queries to the replicas (MVCC) and minimizes the propagation delay. 3) We pro- if the replica database state meets the given freshness re- pose a framework for adaptive query routing depending on quirements of the queries. its predefined max acceptable staleness range. 4) Through Although ATR can also be extended for high availability extensive experiments with a concrete implementation avail- or disaster recovery purposes, the main purpose of ATR is to able in a commercial product, SAP HANA [17], we show offload OLAP-style analytical workloads from the primary that provides sub-second visibility delay even for write- ATR server which is reserved for handling OLTP-style transac- intensive workloads, achieving scalable, OLAP performance tional workloads. Additionally, by having multiple replicas without notable overhead to the primary. for the same primary table, ATR can elastically scale out the The rest of this paper is organized as follows. Section affordable volume of the OLAP-style analytical workloads. 2 shows the proposed architecture of and its design ATR Moreover, by configuring the primary table as an OLTP- choices. Section 3 presents how logs are generated at the favored in-memory row store while configuring its replicas primary and replayed at replicas. In Section 4, we present as OLAP-favored in-memory column stores in SAP HANA, a post-failure replica recovery mechanism and ATR's var- ATR can maximize the capability of processing mixed OLTP 1599 and OLAP workloads under the common database schema database, it could generate an additional overhead to ex- and under the single