Scalable and Highly Available Database Systems in the Cloud by Umar Farooq Minhas A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Doctor of Philosophy in Computer Science Waterloo, Ontario, Canada, 2013 c Umar Farooq Minhas 2013 I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis may be made electronically available to the public. Umar Farooq Minhas ii Abstract Cloud computing allows users to tap into a massive pool of shared computing resources such as servers, storage, and network. These resources are provided as a service to the users allowing them to “plug into the cloud” similar to a utility grid. The promise of the cloud is to free users from the tedious and often complex task of managing and provisioning computing resources to run applications. At the same time, the cloud brings several additional benefits including: a pay-as-you-go cost model, easier deployment of applications, elastic scalability, high availability, and a more robust and secure infrastructure. One important class of applications that users are increasingly deploying in the cloud is database management systems. Database management systems differ from other types of applications in that they manage large amounts of state that is frequently updated, and that must be kept consistent at all scales and in the presence of failure. This makes it difficult to provide scalability and high availability for database systems in the cloud. In this thesis, we show how we can exploit cloud technologies and relational database systems to provide a highly available and scalable database service in the cloud. The first part of the thesis presents RemusDB, a reliable, cost-effective high availability solution that is implemented as a service provided by the virtualization platform. RemusDB can make any database system highly available with little or no code modifications by exploiting the capabilities of virtualization. In the second part of the thesis, we present two systems that aim to provide elastic scalability for database systems in the cloud using two very different approaches. The three systems presented in this thesis bring us closer to the goal of building a scalable and reliable transactional database service in the cloud. iii Acknowledgements I could not be more thankful to Almighty Allah who has always blessed me with success in life. Next in line are my mother, father, six siblings, and my wife who have always extended their unconditional support, love, and prayers that have helped me throughout my personal and professional life. I would like to thank all my friends in Waterloo for making my stay very special, and my friends in Pakistan and world over for being a special part of my life. My supervisor, Professor Ashraf Aboulnaga, deserves a great deal of appreci- ation for his extensive support, both moral and financial, and his extraordinary patience. I would like to thank Professor Kenneth Salem for his contributions to the work presented in Chapter 3 and Chapter 5 of this thesis, and for his excellent advice and continued support throughout my studies at Waterloo. I would further like to thank Professor Kenneth Salem and Professor Tamer Ozsu¨ for serving on my committee, and for their critical feedback on an earlier version of this work. I would also like to thank my external examiner, Professor Donald Kossmann, and my internal external examiner, Professor Ladan Tahvildari, for serving on my thesis committee. I would like to thank Professor Andrew Warfield and my colleagues Shriram Rajagopalan and Brendan Cully, at the University of British Columbia, for their collaboration on the RemusDB work presented in Chapter 3 of this thesis. I would further like to thank my colleagues at the University of Waterloo: Jonathan Ng, Rui Liu, Sean Robertson, and Taha Rafiq for their contributions to this work. I would like to thank Dr. David Lomet and Dr. Chandramohan Thekkath for exceptional mentoring during my internships at Microsoft Research and for their encouragement and support. I would like to thank Professor Robin Cohen, Professor Jie Zhang, and Professor Thomas Tran for providing me with an excellent opportunity to work on trust modeling for vehicular ad-hoc networks (VANETs), considerably expanding my research horizons beyond this thesis. And last but not least, I would like to thank the administrative and technical staff at the University of Waterloo for always helping me and providing me with an excellent environment to conduct top quality research. iv Dedication I dedicate this work to my beloved parents, siblings, and my loving wife. v Contents List of Tables x List of Figures xi 1 Introduction 1 1.1 High Availability . 2 1.2 Elastic Scalability . 2 1.3 Challenges of Deploying Database Systems in the Cloud . 3 1.4 CloudStorageSystems ......................... 3 1.5 DatabaseasaService.......................... 4 1.6 AboutthisThesis............................ 4 1.7 Organization .............................. 6 2 Background 7 2.1 CloudComputing............................ 7 2.2 Virtualization . 9 2.2.1 Virtualization and Cloud Computing . 10 2.2.2 Deploying Database Systems in the Cloud . 11 2.3 Building Highly Available Database Systems . 12 2.3.1 High Availability Through Hardware . 12 2.3.2 High Availability for Database Systems . 13 2.3.3 High Availability Through Virtualization . 16 2.3.4 Our Approach to High Availability for Database Systems . 20 2.4 ScalingDatabaseSystems . .. .. 21 2.4.1 Replication . 21 2.4.2 Data Partitioning . 23 vi 2.4.3 Caching ............................. 24 2.4.4 DataSharing .......................... 25 2.4.5 Our Approach to Elastic Scale-out for Database Systems . 25 3 RemusDB: Transparent High Availability for Database Systems 27 3.1 BackgroundandSystemOverview. 29 3.2 RelatedWork .............................. 31 3.3 SystemDesign.............................. 33 3.4 MemoryOptimizations ......................... 34 3.4.1 SendingLessData ....................... 35 3.4.2 ProtectingLessMemory . 36 3.5 CommitProtection ........................... 38 3.5.1 CorrectnessofCommitProtection. 40 3.5.2 Implementation of Protection and Deprotection . 40 3.6 ReprotectionAfterFailure . 41 3.7 ExperimentalEvaluation . 42 3.7.1 ExperimentalEnvironment. 42 3.7.2 Behavior of RemusDB During Failover . 44 3.7.3 Reprotection After a Failure . 46 3.7.4 OverheadDuringNormalOperation . 47 3.7.5 EffectsofDBBufferPoolSize . 50 3.7.6 Effects of RemusDB Checkpoint Interval . 55 3.7.7 EffectofDatabaseSizeonRemusDB . 55 3.8 Modeling RemusDB’s Behavior for Effective Resource Scheduling . 57 3.8.1 Collecting Training Data . 59 3.8.2 Modeling Network Bandwidth Utilization . 62 3.8.3 ModelingPerformanceDegradation . 69 3.9 Summary ................................ 73 4 Chimera: Elastic Scale-out Through Data Sharing 75 4.1 Introduction............................... 75 4.1.1 DifferentDatabaseArchitectures . 75 4.1.2 OverviewofChimera ...................... 77 vii 4.2 DBMSArchitectures .......................... 79 4.2.1 TheTwoArchitectures. 79 4.2.2 ConcurrencyControl . 80 4.2.3 Recovery............................. 80 4.3 Simplified Data Sharing . 81 4.3.1 File-systemAccess ....................... 83 4.3.2 Distributed Locking . 83 4.3.3 DistributedBufferManagement . 84 4.3.4 LoggingandRecovery ..................... 85 4.4 Implementation ............................. 85 4.4.1 StoredProcedure ........................ 86 4.4.2 EnhancedBufferManager . 87 4.4.3 GlobalLockManager...................... 87 4.4.4 RecoveryIssues ......................... 88 4.5 ExperimentalEvaluation . 89 4.5.1 ExperimentalGoals....................... 89 4.5.2 ExperimentalSetup. .. .. 89 4.5.3 Scalability . 90 4.5.4 RemoteExecutionOverhead. 92 4.5.5 PrototypeOverhead . .. .. 92 4.5.6 CostofUpdates......................... 94 4.5.7 CostofReadswithUpdates . 95 4.6 RelatedWork .............................. 97 4.7 Summary ................................ 97 5 Elastic Scale-out for Partition-Based Database Systems 99 5.1 RelatedWork .............................. 101 5.2 OverviewofVoltDB .......................... 102 5.3 Enabling Elastic Scale-out with VoltDB . 104 5.3.1 Growing the Size of the Cluster . 104 5.3.2 Moving Database Partitions Between Nodes . 105 5.4 Partition Placement in an Elastically Scalable DBMS . 106 5.4.1 Problem Definition . 107 viii 5.4.2 Problem Formulation for the Offline Optimizer . 108 5.4.3 Problem Formulation for the Online Optimizer . 110 5.5 ExperimentalEvaluation . 113 5.5.1 ExperimentalSetup. 113 5.5.2 Demonstrating Elastic Scale-out and Scale-in . 113 5.5.3 Effectiveness of Optimal Partition Placement . 118 5.5.4 Scalability of the Optimizers . 123 5.6 Summary ................................ 125 6 Conclusion 126 References 128 Appendix 141 A Detailed Results Showing Optimizer Effectiveness . 141 ix List of Tables 3.1 RemusDB Source Code Modifications (lines of code) . 41 3.2 ExperimentalSettings ......................... 43 3.3 Parameters Collected During Modeling Experiments . 60 3.4 Accuracy of Non-linear Models for TPC-C (NetBand) . 65 3.5 Accuracy of Non-linear Models for TPC-H (NetBand) . 65 3.6 Robustness of Models for TPC-C (NetBand) . 67 3.7 Robustness of Models for TPC-H (NetBand) . 67 3.8 Accuracy of Non-linear Models for TPC-C (Perf) . 71 3.9 Accuracy of Non-linear Models for TPC-H (Perf) . 71 4.1 OverheadofPrototypeExecution . 93 4.2 CostofReadswithUpdates . .