Distributed SQL Databases Deconstructed Understanding Amazon Aurora, Google Spanner & the Spanner Derivatives Sid Choudhury Bryn Llewellyn NoCOUG Summer 2019 Conference © 2019 All rights reserved. 1 Types of Data Stores Today’s Focus OLAP OLTP Write once, Read many Mixed reads & writes Few concurrent sessions Many concurrent sessions Long running, ad-hoc queries Single-digit ms query latency Large table scans Point reads & short-range scans Petabyte-scale data storage Terabyte-scale data storage © 2019 All rights reserved. 2 Types of Data Stores Open Source Today’s Focus OLAP OLTP NoSQL SQL NoSQL SQL Amazon Aurora Google Google Google BigQuery BigTable Spanner Proprietary © 2019 All rights reserved. 3 Why Devs SQL? 1. Query Flexibility – Model data once, change queries as business changes – Balance modeling richness with performance needs 2. Rich Ecosystem – Data modeling & query examples – Developer IDEs & data visualization tools – Easy to reuse & build integrations 3. Universal Standard for Data Access – Learn once, use forever © 2019 All rights reserved. 4 Why Devs SQL? 1. Large Dataset? – No horizontal write scalability – Use manually sharded SQL or non-transactional NoSQL 2. Infrastructure Failures? – No native failover & repair, SPOF w/ Single Node DB – Use complex replication schemes 3. Multi-Region/Geo-Distributed App? – Multi-master deployment is the only option – Data inconsistency w/ Last Writer Wins conflict resolution © 2019 All rights reserved. 5 Distributed SQL = Keep & Remove 1. SQL Features – ACID, JOINs, foreign keys, serializable isolation 2. Horizontal Write Scalability – Scale write throughput by adding/removing nodes 3. Fault Tolerance With High Availability – Native failover & repair 4. Globally Consistent Writes – Lower end user latency and tolerate region failures 5. Low Read Latency – Strongly consistent (aka correct) reads © 2019 All rights reserved. 6 What’s the fastest growing service in AWS? © 2019 All rights reserved. 7 Amazon Aurora https://www.allthingsdistributed.com/2019/03/Amazon-Aurora-design-cloud-native-relational-database.html © 2019 All rights reserved. 8 Excerpts from Vogels blog post © 2019 All rights reserved. 9 What database powers Google AdWords and Google Play? © 2019 All rights reserved. 10 Google Spanner “At Google, Spanner supports tens of millions of queries per second and runs some of our most critical services, including AdWords and Google Play.” https://ai.google/research/pubs/pub39966 © 2019 All rights reserved. 11 Distributed SQL Architectures - Aurora vs Spanner Shared Storage Shared Nothing Amazon Aurora Google Cloud Spanner “A highly available MySQL and “The first horizontally scalable, strongly consistent, PostgreSQL-compatible relational database service” relational database service” Available on AWS since 2015 Available on Google Cloud since 2017 © 2019 All rights reserved. 12 #1 SQL Features and Completeness © 2019 All rights reserved. 13 Depth of SQL Support Amazon Aurora Google Spanner Subset of MySQL/PostgreSQL features ✓ MySQL and PostgreSQL-compatible © 2019 All rights reserved. 14 Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ✓ Fault Tolerance with HA ✓ Globally Consistent Writes ✓ Low Read Latency © 2019 All rights reserved. 15 #2 Horizontal Write Scalability © 2019 All rights reserved. 16 Amazon Aurora Single Node SQL on Multi-Zone Distributed Storage SQL APP INSERT ROW ❌ Add Primary Instances for Write Scaling ✓ Add Read Replicas for Read Scaling © 2019 All rights reserved. 17 Google Spanner Multi-Node SQL on Multi-Region Distributed Storage SQL APP INSERT ROW1 INSERT ROW3 ✓ Add Primary Instances for Write Scaling ✓ Add Read Replicas for Read Scaling © 2019 All rights reserved. 18 Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA Globally Consistent Writes Low Read Latency © 2019 All rights reserved. 19 #3 Fault Tolerance with HA © 2019 All rights reserved. 20 Amazon Aurora Native Failover & Repair Through Primary Auto Election SQL APP INSERT ROW ✓ HA When Primary Instance Fails ✓ HA When Read Replica Fails © 2019 All rights reserved. 21 Google Spanner Native Failover & Repair Through Shard Leader Auto Election SQL APP INSERT ROW1 INSERT ROW3 ✓ HA When Any Primary Node Fails ✓ HA When Read Replica Fails © 2019 All rights reserved. 22 Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA ✓ ✓ Globally Consistent Writes Low Read Latency © 2019 All rights reserved. 23 #4 Global Write Consistency © 2019 All rights reserved. 24 Amazon Aurora Multi-Master Last Writer Wins Conflict Resolution Leads to Inconsistencies SQL APP SQL APP SET BALANCE = BALANCE - 10 SET BALANCE = BALANCE - 100 Asynchronous Replication Region 1 Region 2 © 2019 All rights reserved. 25 Google Spanner Purpose-Built for Globally Consistent Writes SQL APP SQL APP SET BALANCE = SET BALANCE = BALANCE - 100 BALANCE - 10 © 2019 All rights reserved. 26 Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA ✓ ✓ Globally Consistent Writes ❌ ✓ Low Read Latency © 2019 All rights reserved. 27 #5 Low Read Latency © 2019 All rights reserved. 28 Amazon Aurora Strongly Consistent Reads Served By Primary Instance SQL APP READ ROW © 2019 All rights reserved. 29 Google Spanner Strongly Consistent Reads Served By Shard Leaders w/o Read Quorum SQL APP READ ROW1 © 2019 All rights reserved. 30 Aurora vs Spanner Feature Amazon Aurora Google Spanner SQL Features ✓ Horizontal Write Scalability ❌ ✓ Fault Tolerance with HA ✓ ✓ Global Write Consistency ❌ ✓ Low Read Latency ✓ ✓ © 2019 All rights reserved. 31 Battle of Architectures - Spanner Beats Aurora No Performance & Availability Bottlenecks Scale to Large Clusters while Remaining Highly Available Built for Geo-Distributed Apps Future Proofs Data Tier at Global Businesses Complex to Engineer Needs Clock Skew Tracking Across Instances © 2019 All rights reserved. 32 Analyzing Open Source Spanner-Inspired Derivatives © 2019 All rights reserved. 33 Spanner Brought to Life in Open Source No Longer Open Source as of June 2019 © 2019 All rights reserved. 34 YugaByte DB Design Principles • CP in CAP Theorem • High Performance • Consistent • All layers in C++ to ensure high perf • Partition Tolerant • Run on large memory machines • HA on failures • Optimized for SSDs (new leader elected in seconds) • ACID Transactions • Deploy Anywhere • Single-row linearizability • No IaaS specific dependencies • Multi-row ACID • No atomic clocks • Serializable & Snapshot • Bare metal, VM and Kubernetes • No bottlenecks even for geo-distributed rows © 2019 All rights reserved. 35 Functional Architecture YSQL PostgreSQL-Compatible Distributed SQL API tablet 1’ DOCDB Spanner-Inspired Distributed Document Store tablet 1’ CLOUD NEUTRAL No Specialized Hardware Needed © 2019 All rights reserved. 36 Design Follows a Layered Approach YSQL Postgres-Compatible Distributed SQL API tablet 1’ Self-Healing, Fault-Tolerant High Throughput, Low Latency ACID Transactions Auto Sharding & Rebalancing Global Data Distribution tablet 1’ tablet 1’ © 2019 All rights reserved. 37 YugaByte DB Architecture © 2019 All rights reserved. 38 Distributed SQL = Keep & Remove 1. SQL Features 2. Replication Protocol 3. Clock Skew Tracking 4. Geo-Distributed Transactions © 2019 All rights reserved. 39 Spanner vs. its Open Source Derivatives Feature Google Spanner YugaByte DB CockroachDB TiDB Expensive Free Free Free Cost Proprietary Open Source Proprietary Open Source SQL API Compatibility Replication Protocol Clock Skew Tracking Geo-Distributed Txns Tunable Read Latency Official Jepsen Tests © 2019 All rights reserved. 40 SQL API Compatibility © 2019 All rights reserved. 41 PostgreSQL Transformed into Distributed SQL © 2019 All rights reserved. 42 Depth of SQL Support - YugaByte DB • SQL Features • Advanced SQL Features • Data Types • Partial Indexes • Relational Integrity (Foreign Keys) • Stored Procedures • Built-in Functions • Triggers • Expressions • JSON Column Type • And more ... • Secondary Indexes • JOINs • Transactions • Views © 2019 All rights reserved. 43 Spanner vs. its Open Source Derivatives Feature Google Spanner YugaByte DB CockroachDB TiDB Expensive Free Free Free Cost Proprietary Open Source Proprietary Open Source PostgreSQL MySQL SQL API Compatibility Proprietary PostgreSQL No Stored Procedures No Foreign Keys Replication Protocol Clock Skew Tracking Transaction Manager Tunable Read Latency Official Jepsen Tests © 2019 All rights reserved. 44 Replication Protocol © 2019 All rights reserved. 45 Every Table is Automatically Sharded … … … … … … … … … … … … … … … SHARDING = AUTOMATIC PARTITIONING OF TABLES tablet 1’ © 2019 All rights reserved. 46 Replication Done at Shard Level Tablet #1 Tablet Peer 2 on Node Y Tablet Peer 1 on Node X Tablet Peer 3 on Node Z tablet 1’ © 2019 All rights reserved. 47 Replication uses a Consensus algorithm Raft Leader Uses Raft Algorithm First elect Tablet Leader tablet 1’ © 2019 All rights reserved. 48 Writes in Raft Consensus Write Raft Leader Writes processed by leader: Send writes to all peers Wait for majority to ack tablet 1’ © 2019 All rights reserved. 49 Reads in Raft Consensus Read Raft Leader Reads handled by leader Uses Leader Leases for performance tablet 1’ © 2019 All rights reserved. 50 Spanner vs. its Open Source Derivatives Feature Google Spanner YugaByte DB CockroachDB TiDB Expensive