AnalyticDB: Real-time OLAP Database System at Alibaba Cloud Chaoqun Zhan, Maomeng Su, Chuangxian Wei, Xiaoqiang Peng, Liang Lin, Sheng Wang, Zhe Chen, Feifei Li, Yue Pan, Fang Zheng, Chengliang Chai Alibaba Group flizhe.zcq,maomeng.smm,chuangxian.wcx,xiaoqiang.pxq,yibo.ll,sh.wang,chenzhe.cz, lifeifei,itlanger.pany,f.zheng,[email protected] ABSTRACT Alibaba Cloud from a wide range of business sectors, in- With data explosion in scale and variety, OLAP databas- cluding e-commerce, finance, logistics, public transit, mete- es play an increasingly important role in serving real-time orological analysis, entertainment, etc., as well as internal analysis with low latency (e.g., hundreds of milliseconds), business operations within Alibaba Group. especially when incoming queries are complex and ad hoc Recent works [35, 28, 29, 36, 25] have summarized the in nature. Moreover, these systems are expected to provide main challenges of designing an OLAP system as achieving high query concurrency and write throughput, and support low query latency, data freshness, flexibility, low cost, high queries over structured and complex data types (e.g., JSON, scalability, and availability. Compared to these works, ana- vector and texts). lytical workloads from our clients elevate AnalyticDB to an In this paper, we introduce AnalyticDB, a real-time O- even larger scale: 10PB+ data, hundred thousands of tables LAP database system developed at Alibaba. AnalyticDB and trillions of rows, which presents significant challenges to maintains all-column indexes in an asynchronous manner the design and implementation of AnalyticDB. with acceptable overhead, which provides low latency for The first challenge is that today's users face more com- complex ad-hoc queries. Its storage engine extends hybrid plicated analytics scenarios than before, but still have high row-column layout for fast retrieval of both structured data expectation for low query latency. Users often do not tol- and data of complex types. To handle large-scale data with erate queries that spend a long time. However, as users of high query concurrency and write throughput, AnalyticDB AnalyticDB come from various domains, their analytical de- decouples read and write access paths. To further reduce mands differ significantly and may change frequently, which query latency, novel storage-aware SQL optimizer and exe- make their diverse and complex queries hard to optimize. cution engine are developed to fully utilize the advantages The queries range from full scan, point lookup to multi- of the underlying storage and indexes. AnalyticDB has been table join, and involve conditions on many combinations of successfully deployed on Alibaba Cloud to serve numerous columns. Although indexing is a straightforward way to im- customers (both large and small). It is capable of holding prove query performance, building indexes on pre-specified 100 trillion rows of records, i.e., 10PB+ in size. At the same columns is often no longer effective. time, it is able to serve 10m+ writes and 100k+ queries per The second challenge is that emerging complex analysis second, while completing complex queries within hundreds tends to involve different types of queries and data at the of milliseconds. same time, which requires the system to have a friendly and unified data layout at the storage layer. Traditional O- PVLDB Reference Format: LAP queries and point-lookup queries require different lay- Chaoqun Zhan, Maomeng Su, Chuangxian Wei, Xiaoqiang Peng, outs, i.e., column-stores and row-stores respectively [34, 12]. Liang Lin, Sheng Wang, Zhe Chen, Feifei Li, Yue Pan, Fang Furthermore, more than half of our users' data has a com- Zheng, Chengliang Chai. AnalyticDB: Real-time OLAP Database plex data type, such as text, JSON string, vector, and other System at Alibaba Cloud. PVLDB, 12(12): 2059-2070, 2019. DOI: https://doi.org/10.14778/3352063.3352124 multi-media resources. A practical storage layout should be able to provide fast retrieval for many data types, in order to efficiently support queries involving both structured data 1. INTRODUCTION and data of complex types. AnalyticDB is an OLAP database system designed for The third challenge is that while the system is processing high-concurrency, low-latency, and real-time analytical queries real-time queries with low latency, it also needs to handle on PB scale, and has been running on 2000+ physical ma- tens of millions of online write requests per second. Tradi- chines on Alibaba Cloud [1]. It serves external clients on tional designs [6, 8, 10, 29, 5] process reads and writes in the same process path, so that reads could see newly writ- This work is licensed under the Creative Commons Attribution- ten data once it is committed. However, such designs are no NonCommercial-NoDerivatives 4.0 International License. To view a copy longer well-suited for our case, as consuming a large portion of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. For of resources to guarantee query performance will hurt write any use beyond those covered by this license, obtain permission by emailing performance, and vice versa. Careful designs to balance a- [email protected]. Copyright is held by the owner/author(s). Publication rights mong query latency, write throughput and data visibility licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 12, No. 12 should be taken into consideration. ISSN 2150-8097. DOI: https://doi.org/10.14778/3352063.3352124 2059 To address above challenges, we propose and implemen- fit for analytical queries as these queries only require a sub- t several novel designs in AnalyticDB and have made the set of columns, where row-store significantly amplifies I/Os. following contributions: Moreover, OLTP databases usually actively update indexes Efficient Index Management. AnalyticDB embeds an in the write path, which is so expensive that affects both efficient and effective index engine, which leverages two key write throughput and query latency. approaches for achieving low latency with acceptable over- OLAP databases. To improve the efficiency of analyt- head. First, indexes are built on all columns in each table for ical queries, many OLAP databases like Vertica [29], Tera- significant performance gain on ad-hoc complex queries. We data DB [10] and Greenplum [5] have been developed. Ver- further propose a runtime filter-ratio-based index path selec- tica [29] uses projection to improve query performance. In- tion mechanism to avoid performance slow-down from index stead of building conventional indexes on columns, it only abuse. Second, since it is prohibitive to update large indexes keeps information about Min/Max values, leading to high in the critical path, indexes are asynchronously built during latency from less effective pruning. Teradata DB [10] and off-peak periods. We also maintain a lightweight sorted- Greenplum [5] adopt column-store and allow users to specify index to minimize the impact of asynchronous index build- indexed columns. However, they have two main limitations: ing on queries involving incremental (i.e., newly-written af- first, they modify column indexes in the write path, which ter index building has started) data. is prohibitive for all-column indexes; second, column-store Storage layout for structured data and data of requires many random I/Os for point-lookup queries. complex types. We design the underlying storage to sup- Big Data systems. With the emergence of Map-Reduce port hybrid row-column layout. In particular, we utilize fast model [18], batch processing engines, such as Hive [35] and sequential disk IOs so that its overhead is acceptable under Spark-SQL [37, 13], become popular to process large-scale either OLAP-style or point-lookup workloads. We further data on many machines. However, the executed queries are incorporate complex data types in the storage (including in- considered \offline”. The whole execution could last for dexes) to provide the capability of searching resources (i.e., minutes or hours, and thus is not well-suited for real-time JSON, vector, text) together with structured data. queries. Impala [28] converts “offline" queries to interactive Decoupling Read/write. In order to support both queries using pipeline processing model and column-store, high-throughput writes and low-latency queries, our system reducing latency of common queries to seconds. However, follows an architecture that decouples reads and writes, i.e., Impala does not have column indexes (with Min/Max statis- they are served by write nodes and read nodes respectively. tics only), so that it can not handle complex queries well. These two types of nodes are isolated from each other and Real-time OLAP systems. Recent real-time OLAP hence can scale independently. In particular, write nodes systems include Druid [36] and Pinot [25], both adopting persist write requests to a reliable distributed storage called column store. They all build bitmap-based inverted indexes, Pangu [7]. To ensure data freshness when serving queries, a i.e., Pinot on all columns and Druid on Dimension Column- version verification mechanism is applied on read nodes, so s. A query on Druid may suffer high latency, if it involves that previous writes processed on write nodes are visible. columns not in Dimension Columns. All their indexes are Enhanced optimizer and execution engine. To fur- updated in the write path, which affects the write perfor- ther improve query latency and concurrency, we enhance the mance. Besides, they lack support for some important query optimizer and execution engine in AnalyticDB to fully uti- types such as JOIN, UPDATE, and DELETE, and are inefficient lize the advantages of our storage and indexes. Specifically, for point-lookup queries as they are column-oriented. we propose a storage-aware SQL optimization mechanism Cloud Analytical Services. A number of cloud ana- that generates optimal execution plans according to the s- lytical services such as Amazon Redshift [21] and Google torage characteristics, and an efficient real-time sampling BigQuery [33] have recently emerged.