Deep Dive into TiDB SunHao | PingCAP Agenda ▪ Why we need a new database ▪ The goal of TiDB ▪ Design && Architecture ▪ Storage Layer ▪ Scheduler ▪ SQL Layer ▪ Spark integration ▪ TiDB on Kubernetes Why we Need a NewSQL Database RDBMS NoSQL NewSQL ● From scratch 1970s 2010 2015 Present ● What’s wrong with the existing DBs? ● RDBMS MySQL Redis Google PostgreSQL HBase Spanner ● NoSQL & Middleware Oracle Cassandra Google F1 DB2 MongoDB TiDB ... ... ● NewSQL: F1 & Spanner What to build? ● Scalability ● High Availability ● ACID Transaction ● SQL A Distributed, Consistent, Scalable, SQL Database that supports the best features of both traditional RDBMS and NoSQL Open source, of course What problems we need to solve ▪ Data storage ▪ Data distribution ▪ Data replication ▪ Auto balance ▪ ACID Transaction ▪ SQL at scale Overview Applications BI Tools Data scientist MySQL MySQL MySQL Analytical Analytical Payload Payload Payload workload workload Stateless SQL Stateless SQL SparkSQL & RDD instances instances Spark Plugin RPC RPC RPC Distributed Transactional Key-value storage TiDB Project Overview TiDB TiSpark Stateless SQL instance Spark Plugin Distributed Transactional Key-value storage TiKV Where the data is actually stored TiKV as a KV engine A fast KV engine: RocksDB ● Good start! RocksDB is fast and stable. ● Atomic batch write ● Snapshot ● However… It’s a locally embedded KV store. ● Can’t tolerate machine failures ● Scalability depends on the capacity of the disk Let’s fix Fault Tolerance ● Use Raft to replicate data ● Key features of Raft ● Strong leader: leader does most of the work, issue all log updates ● Leader election ● Membership changes ● Implementation: ● Ported from etcd ● Replicas are distributed across machines/racks/data-centers Let’s fix Fault Tolerance Raft Raft RocksDB RocksDB RocksDB Machine 1 Machine 2 Machine 3 How about Scalability? ● What if we SPLIT data into many regions? ● We got many Raft groups. ● Region = Contiguous Keys Range Scan: Select * from t where c > 10 ● Hash partitioning or Range partitioning? and c < 100; ● Redis: Hash partitioning ● HBase: Range partitioning Region Logical Key Space ● Key: Byte Array ● A globally ordered map Meta: [Start_key, ● Can’t use hash partitioning end_key) ● Use range partitioning (-∞, +∞) ● Region 1 -> [a - d] Sorted Map ● Region 2 -> [e - h] ● … ● Region n -> [w - z] ● Data is stored/replicated/scheduled in regions How to scale? ● That’s simple ● Logical split 0 ● Just Split && Move Node 1 ● Split safely using Raft 1 Node 1 Scale-out (Add new replica in another node) 0 1 Node 1 Node 1 Node 1 Node 2 2 Scale-out (Remove old replica) 0 1 Node 1 Node 1 Node 1 Node 2 Node 1 Node 2 2 3 TiKV as a distributed KV engine Client Placement Driver RPC RPC RPC RPC PD 1 Store 1 Store 2 Store 3 Store 4 PD 2 Region 1 Region 1 Region 2 Region 1 Region 2 PD 3 Region 3 Region 2 Region 5 Region 5 Raft Region 5 Region 4 Region 3 Group Region 4 Region 4 Region 3 TiKV node 1 TiKV node 2 TiKV node 3 TiKV node 4 MVCC and Transaction ▪ MVCC ▪ Transaction ▪ Data layout ▪ Inspired by Google Percolator ▪ key1_version2 -> value ▪ ‘Almost’ decentralized 2-phase commit ▪ key1_version1 -> value ▪ key2_version3 -> value ▪ Lock-free snapshot reads TiKV: Architecture overview (Logical) ● Highly layered Transaction ● Raft for consistency and scalability MVCC ● No distributed file system RaftKV ○ For better performance and lower latency Local KV Storage (RocksDB) Replica Scheduling Placement Driver ▪ Provide the God’s view of the entire cluster ▪ Store the metadata Placement Driver ▪ Clients have cache of placement information. ▪ Maintain the replication constraint Raft Raft ▪ 3 replicas, by default ▪ Data movement for balancing the workload Placement Placement Driver Driver ▪ It’s a cluster too, of course. Raft ▪ Thanks to Raft. PD as the cluster manager PD Node 1 HeartBeat with Info Region A Cluster Info Region B Scheduling Movement Command Admin Node 2 Config Scheduling Strategy Region C Scheduling Strategy ▪ Replica number in a raft group ▪ Replica geo distribution ▪ Read/Write workload ▪ Leaders and followers ▪ Tables and TiKV instances ▪ Other customized scheduling strategy TiDB as a SQL database The SQL Layer ● SQL is simple and very productive ● We want to write code like this: SELECT COUNT(*) FROM user WHERE age > 20 and age < 30; The SQL Layer ▪ Mapping relational model to Key-Value model ▪ Full-featured SQL layer ▪ Cost-based optimizer (CBO) ▪ Distributed execution engine SQL on KV engine CREATE TABLE `t` (`id` int, `age` int, key `age_idx` (`age`)); ▪ Row ▪ Key: TableID + RowID INSERT INTO `t` VALUES (100, 35); ▪ Value: Row Value ▪ Index ▪ Key: TableID + IndexID + Index-Column-Values K1 10, 35 ▪ Value: RowID Encoded Keys: K1: tid + rowid K2 K1 K2: tid + idxid + 35 Secondary Index Index: Name Pkey Name Email Index Value Pkey 1 Edward [email protected] Edward 1 2 Tom [email protected] Tom 2 Index Value Pkey [email protected] 1 Data table [email protected] 2 Index: Email SQL on KV engine ▪ Key-Value pairs are byte arrays ▪ Row data and index data are converted into Key-Value ▪ Key should be encoded using the memory-comparable encoding algorithm ▪ compare(a, b) == compare (encode(a), encode(b)) ▪ Example: Select * from t where age > 10 Index is just not enough... ▪ Can we push down filters? ▪ select count(*) from person where age > 20 and age < 30 ● It should be much faster, maybe 100x ● Less RPC round trip ● Less transferring data Distributed Execution Engine TiDB Server age > 20 and age < 30 age > 20 and age < 30 age > 20 and age < 30 TiDB knows that Region 1 / 2 / 5 stores the data of Region 1 Region 2 person table. Region 5 TiKV Node1 TiKV Node2 TiKV Node3 What about drivers for every language? ● We just build a protocol layer that is compatible with MySQL. Then we have all the MySQL drivers. ● All the tools ● All the ORMs ● All the applications ● That’s what TiDB does. Architecture MySQL clients KV API Coprocessor Placement Load Balancer (Optional) Driver Txn, Transaction MySQL Protocol MySQL Protocol TiDB SQL Layer TiDB SQL Layer MVCC DistSQL DistSQL KV API KV API API API RawKV, Raft KV TiDB Server TiDB Server (Stateless) (Stateless) RocksDB Pluggable Storage Engine (e.g. TiKV) TiSpark (1 / 3) ▪ TiSpark = SparSQL on TiKV ▪ SparkSQL directly on top of a distributed Database Storage ▪ Hybrid Transactional/Analytical Processing(HTAP) rocks ▪ Provide strong OLAP capacity together with TiDB ▪ Spark ecosystem TiSpark (2 / 3) PD PD TSO/Data location Data location PD PD Cluster Meta data Spark Driver TiDB Application TiKV TiKV Job TiDB DistSQL API DistSQL API Worker TiDB TiKV TiKV Syncer Worker TiDB TiKV TiKV Worker TiDB ... Spark Cluster ... ... TiDB Cluster TiKV Cluster (Storage) TiSpark TiSpark (3 / 3) ▪ TiKV Connector is better than JDBC connector ▪ Index support ▪ Complex Calculation Pushdown ▪ CBO ▪ Pick up the right Access Path ▪ Join Reorder ▪ Priority & Isolation Level TiDB as a Cloud-Native Database Deploy a database on the cloud TiDB on Kubernetes Users Users Admin TiDB Operator Deployment TiDB Scheduler: TiDB Cloud Kube Scheduler + Manager TiDB Cluster Controller PD Controller Scheduler Extender RESTFul Interface DaemonSet External Service TiKV Controller TiDB Controller manager Load balancer manager GC Controller Volume Manager DaemonSet Cloud TiDB Kubernetes Core Scheduler Controller Manager API Server Cloud TiDB ？ Open Source Open Source Roadmap ▪ Multi-tenant ▪ Better Optimizer and Runtime ▪ Performance Improvement ▪ Document Store ▪ Backup & Reload & Migration Tools Thanks Q&A https://github.com/pingcap/tidb https://github.com/pingcap/tikv https://github.com/pingcap/pd https://github.com/pingcap/tispark https://github.com/pingcap/docs https://github.com/pingcap/docs-cn Contact Me: [email protected].