Automatically Indexing Millions of Databases in Microsoft Azure SQL Database Sudipto Das, Miroslav Grbic, Igor Ilic, Isidora Jovandic, Andrija Jovanovic, Vivek R. Narasayya, Miodrag Radulovic, Maja Stikic, Gaoxiang Xu, Surajit Chaudhuri Microsoft Corporation ABSTRACT tools [2, 14, 46] have helped DBAs search the complex search An appropriate set of indexes can result in orders of magni- space of alternative indexes (and other physical structures tude better query performance. Index management is a chal- such as materialized views and partitioning). However, a lenging task even for expert human administrators. Fully au- DBA still drives this tuning process and is responsible for tomating this process is of significant value. We describe the several important tasks, such as: ¹iº identifying a representa- challenges, architecture, design choices, implementation, and tive workload; ¹iiº analyzing the database without impact- learnings from building an industrial-strength auto-indexing ing production instances; ¹iiiº implementing index changes; service for Microsoft Azure SQL Database, a relational data- ¹ivº ensuring these actions do not adversely affect query base service. Our service has been generally available for performance; and ¹vº continuously tuning the database as more than two years, generating index recommendations the workload drifts and the data distributions change. for every database in Azure SQL Database, automatically Cloud database services, such as Microsoft Azure SQL implementing them for a large fraction, and significantly im- Database, automate several important tasks such as provi- proving performance of hundreds of thousands of databases. sioning, operating system and database software upgrades, We also share our experience from experimentation at scale high availability, backups etc, thus reducing the total cost with production databases which gives us confidence in our of ownership (TCO). However, in most cases, performance index recommendation quality for complex real applications. tuning, e.g., index management, remains the user’s responsi- bility. Offering existing on-premises tuning tools hosted asa ACM Reference Format: cloud service still leaves a significant burden on the users, Sudipto Das, Miroslav Grbic, Igor Ilic, Isidora Jovandic, Andrija Jovanovic, Vivek R. Narasayya, Miodrag Radulovic, Maja Stikic, which is daunting for many users of cloud databases who Gaoxiang Xu, Surajit Chaudhuri. 2019. Automatically Indexing Mil- are application developers lacking expert DBA skills. lions of Databases in Microsoft Azure SQL Database. In Proceedings Another pattern we commonly observe is Software-as-a- of ACM SIGMOD conference (SIGMOD’19). ACM, New York, NY, Service (SaaS) vendors and Cloud Software Vendors (CSV) USA, Article 4, 14 pages. https://doi.org/10.475/123_4 deploying hundreds to thousands of databases for customers of their applications. Managing such a huge pool of databases 1 INTRODUCTION is a formidable task even for expert DBAs, where individual 1.1 Motivation DB instances can have different schema, queries, and data distributions [24, 33, 39]. Therefore, being able to fully au- A relational database with an appropriate set of indexes tomate indexing, i.e., not only identifying the appropriate can have orders of magnitude better performance and lower indexes, but also automating the above-mentioned steps cru- resource consumption. Choosing good indexes is challeng- cial for holistic index lifecycle management, is a significant ing even for expert Database Administrators (DBA) since step in the cloud’s promise of reducing the TCO. it is workload, schema, and data dependent. Decades of re- Azure SQL Database is a fully-managed relational database- search [2, 9–11, 13, 14, 18, 40, 46] and many commercial as-a-service with built-in intelligence [6]. Query performance Permission to make digital or hard copies of part or all of this work for is tuned through services such as automated plan correc- personal or classroom use is granted without fee provided that copies are tion [4], automated indexing [5], and intelligent insights [19]; not made or distributed for profit or commercial advantage and that copies here we focus on auto-indexing. Existing approaches that bear this notice and the full citation on the first page. Copyrights for third- party components of this work must be honored. For all other uses, contact rely on human intervention for the critical steps in indexing the owner/author(s). do not scale for Azure SQL Database automating indexing for SIGMOD’19, June 2019, Amsterdam, NL its millions of databases. Therefore, several years back, we © 2019 Copyright held by the owner/author(s). embarked on the journey to build the first industrial-strength ACM ISBN 123-4567-24-567/08/06. fully-automated indexing solution for Azure SQL Database. https://doi.org/10.475/123_4 1 SIGMOD’19, June 2019, Amsterdam, NL Das et al. 1.2 Challenges We explain the user-facing auto-indexing offering (Section 2), The first challenge is to scale the auto-indexing service to all where we expose several controls for expert users to con- databases in Azure SQL Database while meeting the statu- figure the service and provide transparency of the indexes tory and compliance requirements. Azure has more than 50 implemented and their impact on performance. We iden- regions worldwide spread over more than 140 countries [7]. tify the architecture and the key components essential to The auto-indexing service must operate with minimal or fully-automate recommendation and implementation of in- no human intervention to achieve this scale. In addition to dexes (Section 3). These components are: ¹aº control plane: ensuring the proper health and operation of this service, we the backbone of our automation that drives index lifecy- need to be able to debug the quality of the recommendations cle management and coordinates the other components in as the workloads and databases change. Moreover, the data- the system (Section 4); ¹bº index recommender: analyzes the base server bits are frequently upgraded, our service itself workload to identify an appropriate set of indexes to create goes through upgrades, and we must tolerate a wide variety or drop (Section 5); ¹cº validator: a novel component that of software and hardware failures we see at Azure-scale. Un- analyzes the impact of implementing the indexes, detects if like a DBA tuning a database who can examine the queries performance has improved or regressed, and takes corrective or understand the applications, due to compliance reasons, action if it detects a significant regression (Section 6). all of the above actions must be performed without the engi- The control plane is crucial to scale the service to millions neers having knowledge of the application, its workload, or of databases in Azure SQL Database. It is a fault-tolerant data distributions, which further amplifies the challenge! service that automatically invokes the recommender, imple- The second challenge is to automate generating the sev- ments indexes (with user’s permission), validates them, moni- eral critical inputs to the index tuner needed for high quality tors the service health, detects issues, and wherever possible recommendations. Unlike a DBA who can use additional ap- takes corrective actions. To meet the compliance require- plication context, we need to automatically identify a work- ments, any system component that accesses sensitive cus- load to tune indexes for and specify other tuning constraints, tomer data, e.g., query text or data distributions, runs within e.g., the types of indexes to recommend, storage budget, or Azure’s compliance boundaries that is guarded by multi- maximum number of indexes, etc. ple levels of security, authentication, and auditing. Health The third challenge is that the state-of-the-art index rec- signals of the service and debugging is primarily through ommenders [9, 14, 31, 46] rely on the query optimizer’s cost anonymized and aggregated data. estimates to determine the benefits of new index configu- The index recommender analyzes query execution sta- rations. However, users care about actual query execution tistics to automatically identify inputs, such as a workload cost (e.g., query’s CPU time). Due to well-known limitations to tune. To generate high quality index recommendations, of the query optimizer’s estimates [20, 21], there are many we adapt and extend two building blocks for index recom- instances where an index, which is estimated to improve mendations in SQL Server: ¹aº Missing Indexes [34]; and query performance based on the optimizer’s cost, makes (b) Database Engine Tuning Advisor (DTA) [2, 10]. These query execution costs (or performance) worse once imple- approaches provide complementary benefits which we to- mented [8, 16, 17]. Having the ability to ideally avoid or at gether leverage to support the huge diversity requirements least quickly detect and correct query performance regres- in Azure SQL Database, e.g., tuning databases with very sions due to index changes is crucial in a production setting. different workload complexity and available resources [28]. The fourth challenge is to ensure auto-indexing, which The validator compares the execution costs before and af- is a built-in service, does not significantly impact normal ter implementing an index change, detects large regressions, application workload and interfere or disrupt application and takes corrective actions (e.g., automatically reverting the processes. This has two facets: