Database Sharding with Oracle RDBMS First Impressions Robert Bialek Principal Consultant BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Who Am I Principal Consultant and Trainer at Trivadis GmbH in Munich – MSc in Computer Engineering Focus: – Oracle Database High Availability – Database Architecture/Internals – Backup/Recovery – Troubleshooting/Performance Tuning – Linux Operating System Trainer for the following Trivadis courses: – Oracle Grid Infrastructure, RAC, Data Guard 2 17.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interacting operation of your IT systems. 3 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions With over 600 specialists and IT experts in your region. COPENHAGEN 14 Trivadis branches and more than 600 employees HAMBURG 200 Service Level Agreements Over 4,000 training participants Research and development budget: DÜSSELDORF CHF 5.0 million FRANKFURT Financially self-supporting and sustainably profitable STUTTGART Experience from more than 1,900 FREIBURG VIENNA MUNICH projects per year at over 800 BRUGG customers BASEL ZURICH BERN GENEVA LAUSANNE 4 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Technology on its own won't help you. You need to know how to use it properly. 5 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Database Scalability – Limits? For some business requirements: – extreme scalability/elasticity, global data distribution,… know database techniques might not be sufficient. The solution: Database Sharding (horizontal scaling) One Database – Not really a new concept: Cassandra, MongoDB, MySQL, … Split one … into many Only 1/N components 6 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Agenda 1. Sharded Database Architecture 2. Data Sharding Methods 3. Data High Availability/Disaster Recovery 4. Session/Statement Routing 5. Conclusions 7 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Sharded Database Architecture 8 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Distributed Partitioning Horizontally distributed data partitioning using shared-nothing database architecture – Sharded tables: data exist in one shard Up to 1000 non-CDB shards – Duplicated tables: data exist in all shards Sharded Database (SDB) Distributed Shard 1 Shard 2 Shard 3 Partitioning DB (Tables) Server B Server C Server D Unsharded Server A Don’t share physical resources No cluster software 9 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Big Picture Shard Director 1 Shard Director 2 Shard Director 3 Shard Director 4 Shard Catalog Shard Catalog Global Services: OLTP_RW, OLTP_RO, … Shards Shards Data Center (Region): Germany Data Center (Region): Switzerland 10 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Sharded Tables All tables in one table family must be equi-partitioned (sharding key) – Two methods to create a table family: reference partitioning or the PARENT clause CUSTOMERS CREATE SHARDED TABLE customers (... CUSTID FIRSTNAME … CREATE SHARDED TABLE orders (... 100 101 102 ORDERITEMS LINEITEMID CUSTID ORDERID … ORDERS 1000 100 300 Table Family Table ORDERID CUSTID ORDERDATE … 1001 100 300 300 100 1002 101 301 301 101 1003 101 301 302 102 1004 102 302 11 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Duplicated Tables Shard 1 Shard 2 Shard 3 Used to avoid cross-shard operations Use materialized view replication – The master table is created in the Shard Catalog database – R/O materialized views in all shards – All supporting objects created automatically in SDB – SHRD_DUPL_TABLE_REFRESH_RATE initialization Shard Catalog parameter controls refresh frequency Cannot coexist in a tablespace with sharded tables CREATE DUPLICATED TABLE products( productid INTEGER PRIMARY KEY, ... ) TABLESPACE PRODUCTS_TBS; 12 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – DDL Execution The application schema name as well as object names are identical in all shards DDLs are executed in the Shard Catalog database or using GDSCTL – Automatically propagated to all shards SQL> CONNECT SYS@SH_CATALOG GDSCTL> sql "CREATE USER ..." SQL> ALTER SESSION ENABLE SHARD DDL; GDSCTL> sql "CREATE TABLESPACE SQL> CREATE USER <app_name>... SET ..." SQL> GRANT CREATE TABLE TO <app_name>... ... SQL> CREATE DUPLICATED TABLE <name>... SQL> CREATE SHARDED TABLE <name>... Shard Catalog 13 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Sharded Tables (Chunks) Physical data distribution based on chunks – Each table partition is stored in different tablespace 0 CUSTOMERS_P1 Chunk #1 TBS_P1 Shard 1 ORDERS_P1 357913941 CUSTOMERS_P2 Chunk #2 TBS_P2 ORDERS_P2 Chunks: 1-6 715827882 … 2147483646 CUSTOMERS_P7 Shard 2 Chunk #7 TBS_P7 ORDERS_P7 2505397587 CUSTOMERS_P8 Chunk #8 ORDERS_P8 TBS_P8 Chunks: 7-12 2863311528 14 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Resharding (Hotspot Elimination) Adding/Removing shards or hotspot elimination requires chunk movement – Automatically or manually GDSCTL > MOVE CHUNK -CHUNK 12 -SOURCE sh01 -TARGET sh12 FAN R/O Chunk level 0 Copy files and notification to Chunk R/O on RMAN backup RMAN restore clients source Drop the chunk FAN chunk File move/ Chunk level 1 on the source up/down events restore RMAN backup A chunk can also be spit manually within a shard GDSCTL > SPLIT CHUNK -CHUNK 12 Hotspot 15 09.09.2016 Database Sharding with Oracle RDBMS 12c Release 2 - First Impressions Data Sharding Methods 16 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – System-Managed Sharding Data is automatically distributed across shards using partitioning by consistent hash – Expression: MOD(ORA_HASH('<SHARDING_KEY'),4294967295) – Useful for data distribution Automatic resharding in case of cardinality change The number of chunks is defined during CREATE SHARDCATALOG and cannot be changed later A tablespace set needs to be created in the Shard Catalog database SQL> CREATE TABLESPACE SET TBS_SET USING TEMPLATE (DATAFILE SIZE 1G AUTOEXTEND ON NEXT 512M MAXSIZE UNLIMITED EXTEND MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO); 17 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – System-Managed Sharding Shards #Chunks=12 #Shards=3 C001TS_SET customers_P1 Shard Catalog C002TS_SET customers_P2 C003TS_SET customers_P3 C004TS_SET customers_P4 CREATE SHARDED TABLE customers ( Set C005TS_SET customers_P5 custid varchar2(20) CONSTRAINT C006TS_SET customers_P6 pk_customers PRIMARY KEY, C007TS_SET customers_P7 firstname varchar2(50), C008TS_SET customers_P8 lastname varchar2(50), Tablespace ... ) TABLESPACE SET tbs_set C009TS_SET customers_P9 C00ATS_SET customers_P10 PARTITION BY CONSISTENT HASH (custid) PARTITIONS AUTO; C00BTS_SET customers_P11 C00CTS_SET customers_P12 18 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – Composite Sharding Data is first partitioned by range or list method across multiple shardspaces (set of shards), then by consistent hash in all shards within a shardspace GDSCTL> ADD SHARDSPACE germany; GDSCTL> ADD SHARDSPACE switzerland; GDSCTL> ADD SHARD –CONNECT shdb01 –SHARDSPACE germany; GDSCTL> ADD SHARD –CONNECT shdb02 –SHARDSPACE germany; GDSCTL> ADD SHARD –CONNECT shdb02 –SHARDSPACE switzerland; ... SQL> CREATE TABLESPACE SET de_tbs IN SHARDSPACE germany; SQL> CREATE TABLESPACE SET ch_tbs IN SHARDSPACE switzerland; 19 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – Composite Sharding Shards CREATE SHARDED TABLE customers C001DE_TBS customers_P1 ( C002DE_TBS customers_P2 custid varchar2(20), Shardspace country varchar2(2), GERMANY de_tbs ... C003DE_TBS customers_P3 ) C004DE_TBS customers_P4 PARTITIONSET BY LIST (country) PARTITION BY CONSISTENT HASH (custid) PARTITIONS AUTO ( C005CH_TBS customers_P5 PARTITIONSET germany VALUES ('DE') C006CH_TBS customers_P6 TABLESPACE SET de_tbs Shardspace PARTITIONSET switzerland VALUES ('CH') SWITZERLAND ch_tbs TABLESPACE SET ch_tbs C007CH_TBS customers_P7 ... C008CH_TBS customers_P8 20 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data High Availability Disaster Recovery 21 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data High Availability / Disaster Recovery – Overview Data replication with Data Guard is a crucial component in SDB environment – High availability, disaster recovery, read offloading – Replication deployment performed fully automatically – The logical unit of data replication is a shardgroup