High-Availability Database Systems: Evaluation of Existing Open Source Solutions
Total Page：16
File Type：pdf, Size：1020Kb
Aalto University School of Science Degree Programme of Computer Science and Engineering Tuure Laurinolli High-Availability Database Systems: Evaluation of Existing Open Source Solutions Master's Thesis Espoo, November 19, 2012 Supervisor: Professor Heikki Saikkonen Instructor: Timo L¨attil¨aM.Sc. (Tech.) Aalto University School of Science ABSTRACT OF Degree Programme of Computer Science and Engineering MASTER'S THESIS Author: Tuure Laurinolli Title: High-Availability Database Systems: Evaluation of Existing Open Source Solu- tions Date: November 19, 2012 Pages: 90 Professorship: Software Systems Code: T-106 Supervisor: Professor Heikki Saikkonen Instructor: Timo L¨attil¨aM.Sc. (Tech.) In recent years the number of open-source database systems offering high- availability functionality has exploded. The functionality offered ranges from simple one-to-one asynchronous replication to self-managing clustering that both partitions and replicates data automatically. In the thesis I evaluated database systems for use as the basis for high availability of a command and control system that should remain available to operators even upon loss of a whole datacenter. In the first phase of evaluation I eliminated systems that appeared to be unsuitable based on documentation. In the second phase I tested both throughput and fault tolerance characteristics of the remain- ing systems in a simulated WAN environment. In the first phase I reviewed 24 database systems, of which I selected six, split in two categories based on consistency characteristics, for further evaluation. Ex- perimental evaluation showed that two of these six did not actually fill my re- quirements. Of the remaining four systems, MongoDB proved troublesome in my fault tolerance tests, although the issues seemed resolvable, and Galera's slight issues were due to its configuration mechanism. This left one in each category. They, Zookeeper and Cassandra, did not exhibit any problems in my tests. Keywords: database, distributed system, consistency, latency, causality Language: English 2 Aalto-yliopisto Perustieteiden korkeakoulu DIPLOMITYON¨ Tietotekniikan tutkinto-ohjelma TIIVISTELMA¨ Tekij¨a: Tuure Laurinolli Ty¨on nimi: Korkean saavutettavuuden tietokantaj¨arjestelm¨at: Olemassa olevien avoimen l¨ahdekoodin ratkaisuiden arviointi P¨aiv¨ays: 19. marraskuuta 2012 Sivum¨a¨ar¨a: 90 Professuuri: Ohjelmistotekniikka Koodi: T-106 Valvoja: Professori Heikki Saikkonen Ohjaaja: Diplomi-insin¨o¨ori Timo L¨attil¨a Viime vuosina korkean saavutettavuuden mahdollistavat avoimen l¨ahdekoodin tietokantaj¨arjestelm¨at ovat yleistyneet. Korkean saavutettavuuden ratkaisut vaih- televat yksinkertaisesta asynkronisesta yksi yhteen -toisintamisesta dataa it- sen¨aisesti hajauttavaan ja toisintavaan ryv¨astykseen. T¨ass¨a diplomity¨oss¨a arvioin tietokantaj¨arjestelmien soveltuvuutta pohjaksi kor- kean saavutettavuuden toiminnoille komentokeskusj¨arjestelm¨ass¨a, jonka tulee pysy¨a saavutettavana my¨os kokonaisen konesalin vikaantuessa. Arvioinnin en- simm¨aisess¨a vaiheessa eliminoin dokumentaation perusteella selv¨asti soveltumat- tomat j¨arjestelm¨at. Toisessa vaiheessa testasin sek¨a j¨arjestelmien viansietoisuutta ett¨a l¨ap¨aisykyky¨a simuloidussa korkean latenssin verkossa. Ensimm¨aisess¨a vaiheessa tutustuin 24 tietokantaj¨arjestelm¨a¨an, joista valitsin kuusi tarkempaan arviointiin. Jaoin tarkemmin arvioidut j¨arjestelm¨at kahteen kategoriaan konsistenssiominaisuuksien perusteella. Kokeissa havaitsin ett¨a kaksi n¨aist¨a kuudesta ei t¨aytt¨anyt asettamiani vaatimuksia. J¨aljellej¨a¨aneist¨a nelj¨ast¨a j¨arjestelm¨ast¨a MongoDB aiheutti ongelmia viansietoisuustesteiss¨ani, joskin ongel- mat vaikuttivat olevan korjattavissa, ja Galeran v¨ah¨aiset ongelmat johtuivat sen asetusj¨arjestelm¨ast¨a. J¨aljelle j¨aiv¨at ensimm¨aisest¨a kategoriasta Zookeeper ja toi- sesta Cassandra, joiden kummankaan viansietoisuudesta en testeiss¨ani l¨oyt¨anyt ongelmia. Asiasanat: tietokanta, hajautettu j¨arjestelm¨a, ristiriidattomuus, konsis- tenssi, viive, latenssi, kausaalisuus Kieli: Englanti 3 Acknowledgements I would like to thank Portalify Ltd for offering me an interesting thesis project and ample time to work on it. At Portalify I'd especially like to thank M.Sc. Timo L¨attil¨a,my instructor, for putting me on the right track from the start. Outside Portalify, I would like to thank Professor Heikki Saikkonen for taking the time to supervise my thesis. I want to also thank my friends and family for providing me support and, perhaps even more importantly, welcome distractions. Aalto on Waves was downright disruptive, and learning to fly at Polyteknikkojen Ilmailukerho took its time too. However, constant support from old friends was the most important. Thank you, Juha and #kumikanaultimate! Helsinki, November 19, 2012 Tuure Laurinolli 4 Abbreviations and Acronyms 2PC Two-phase Commit ACID Atomicity, Consistency, Isolation, Durability API Application Programming Interface ARP Address Resolution Protocol CAS Compare And Set FMEA Failure Modes and Effects Analysis FMECA Failure Modes, Effects and Criticality Analysis FTA Fault Tree Analysis HAPS High Availability Power System HTTP Hypertext Transfer Protocol JSON JavaScript Object Notation LAN Local Area Network MII Media Independent Interface NAT Network Address Translation PRA Probabilistic Risk Assessment REST Representational State Transfer RPC Remote Procedure Call RTT Round-Trip Time SDS Short Data Service SLA Service Level Agreement SSD Solid State Drive SQL Structured Query Language TAP Linux network tap TCP Transmission Control Protocol TETRA Terrestrial Trunked Radio VM Virtual Machine WAN Wide Area Network XA X/Open Extended Architecture 5 Contents Abbreviations and Acronyms 4 1 Introduction 8 1.1 High-Availability Command and Control System . .8 1.2 Open-Source Database Systems . .9 1.3 Evaluation of Selected Databases . .9 1.4 Structure of the Thesis . 10 2 High Availability and Fault Tolerance 11 2.1 Terminology . 11 2.2 Overcoming Faults . 14 2.3 Analysis techniques . 16 3 System Architecture 24 3.1 Background . 24 3.2 Network Communications Architecture . 26 3.3 Software Architecture . 28 3.4 FMEA Analysis of System . 33 3.5 FTA Analysis of System . 37 3.6 Software Reliability Considerations . 39 3.7 Conclusions on Analyses . 40 4 Evaluated Database Systems 41 4.1 Database Requirements . 41 4.2 Rejected Databases . 42 4.3 Databases Selected for Limited Evaluation . 48 4.4 Databases Selected for Full-Scale Evaluation . 50 5 Experiment Methodology 54 5.1 Test System . 54 5.2 Test Programs . 56 6 5.3 Fault Simulation . 64 5.4 Test Runs . 65 6 Experiment Results 66 6.1 Throughput Results . 66 6.2 Fault Simulation Results . 75 7 Comparison of Evaluated Systems 84 7.1 Full-Scale Evaluation . 84 7.2 Limited Evaluation . 85 8 Conclusions 86 A Remaining throughput results 91 B Remaining fault test results 95 7 Chapter 1 Introduction In this thesis I present my research related to adoption of an existing open- source database system as the basis for high availability in a command and control system being developed by Portalify Ltd. 1.1 High-Availability Command and Control System The command and control system is designed to support operations of rescue personnel by automatically tracking status and location of field units so that dispatching operators always have correct and up-to-date view of available units. It tracks locations of TETRA handsets and vehicle radios, and handles status messages sent by field personnel in response to events such as receiving dispatch orders. The system also allows operators to dispatch a unit on a mission, and automatically sends necessary information to the unit. The system should scale to installations that span large geographical ar- eas, with dispatching operators located in multiple, geographically diverse control rooms, and thousands of controlled units spread over the geograph- ical area. Typically operators in one control room would be responsible for controlling units in a specific area, but it should be possible for another control room to take over the area in case the original control room cannot handle its tasks because it has for example lost electrical power. In this thesis I concentrate on hardware fault tolerance of the command and control system and also the database system, since studying software faults of large, existing software systems appears to be an unsolved problem. However, I touch on higher-level approaches that could be used to enhance software fault tolerance of a complex system in practice in Chapter 3. I introduce terminology and analysis methods related to availability and 8 CHAPTER 1. INTRODUCTION 9 fault tolerance in Chapter 2. In Chapter 3 I present more elaborate require- ments for the system, a system architecture based on those requirements and fault-tolerance analysis of the architecture model based on analysis methods introduced in Chapter 2. 1.2 Open-Source Database Systems The system described above must be able to share data between operators working on different workstations, located in different control rooms, dis- tributed across a country. A database system for storing the data and con- trolling access to it is required. Because of the fault tolerance requirements presented in Chapter 3, the database system must be geographically dis- tributed. Main functional requirement for the database is that it must provide atomic update primitive, preferably with causal consistency and read com- mitted visibility semantics. Main non-functional requirements are quick, au- tomatic handling of