Database Sharding with Oracle RDBMS First Impressions
Robert Bialek Principal Consultant
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Who Am I
Principal Consultant and Trainer at Trivadis GmbH in Munich – MSc in Computer Engineering Focus: – Oracle Database High Availability – Database Architecture/Internals – Backup/Recovery – Troubleshooting/Performance Tuning – Linux Operating System Trainer for the following Trivadis courses: – Oracle Grid Infrastructure, RAC, Data Guard
2 17.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Our company.
Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields:
O P E R A T I O N
Trivadis Services takes over the interacting operation of your IT systems.
3 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions With over 600 specialists and IT experts in your region.
COPENHAGEN
14 Trivadis branches and more than 600 employees HAMBURG 200 Service Level Agreements Over 4,000 training participants Research and development budget: DÜSSELDORF CHF 5.0 million
FRANKFURT Financially self-supporting and sustainably profitable STUTTGART Experience from more than 1,900 FREIBURG VIENNA MUNICH projects per year at over 800 BRUGG customers BASEL ZURICH BERN GENEVA LAUSANNE
4 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Technology on its own won't help you. You need to know how to use it properly.
5 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Database Scalability – Limits?
For some business requirements: – extreme scalability/elasticity, global data distribution,… know database techniques might not be sufficient. The solution: Database Sharding (horizontal scaling) One Database – Not really a new concept: Cassandra, MongoDB, MySQL, …
Split one … into many
Only 1/N components
6 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Agenda
1. Sharded Database Architecture 2. Data Sharding Methods 3. Data High Availability/Disaster Recovery 4. Session/Statement Routing 5. Conclusions
7 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Sharded Database Architecture
8 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Distributed Partitioning
Horizontally distributed data partitioning using shared-nothing database architecture
– Sharded tables: data exist in one shard Up to 1000 non-CDB shards – Duplicated tables: data exist in all shards Sharded Database (SDB)
Distributed Shard 1 Shard 2 Shard 3 Partitioning
DB DB (Tables)
Server B Server C Server D
Unsharded Server A Don’t share physical resources No cluster software
9 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Big Picture
Shard Director 1 Shard Director 2 Shard Director 3 Shard Director 4
Shard Catalog Shard Catalog Global Services: OLTP_RW, OLTP_RO, …
Shards Shards
Data Center (Region): Germany Data Center (Region): Switzerland
10 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Sharded Tables
All tables in one table family must be equi-partitioned (sharding key) – Two methods to create a table family: reference partitioning or the PARENT clause
CUSTOMERS CREATE SHARDED TABLE customers (... CUSTID FIRSTNAME … CREATE SHARDED TABLE orders (... 100
101 102 ORDERITEMS
LINEITEMID CUSTID ORDERID … ORDERS 1000 100 300
Table Family Table ORDERID CUSTID ORDERDATE … 1001 100 300 300 100 1002 101 301 301 101 1003 101 301 302 102 1004 102 302
11 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Duplicated Tables
Shard 1 Shard 2 Shard 3 Used to avoid cross-shard operations Use materialized view replication – The master table is created in the Shard Catalog database – R/O materialized views in all shards – All supporting objects created automatically in SDB – SHRD_DUPL_TABLE_REFRESH_RATE initialization Shard Catalog parameter controls refresh frequency Cannot coexist in a tablespace with sharded tables CREATE DUPLICATED TABLE products( productid INTEGER PRIMARY KEY, ... ) TABLESPACE PRODUCTS_TBS;
12 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – DDL Execution
The application schema name as well as object names are identical in all shards DDLs are executed in the Shard Catalog database or using GDSCTL – Automatically propagated to all shards SQL> CONNECT SYS@SH_CATALOG GDSCTL> sql "CREATE USER ..." SQL> ALTER SESSION ENABLE SHARD DDL; GDSCTL> sql "CREATE TABLESPACE SQL> CREATE USER
Shard Catalog
13 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Sharded Tables (Chunks)
Physical data distribution based on chunks – Each table partition is stored in different tablespace
0 CUSTOMERS_P1 Chunk #1 TBS_P1 Shard 1 ORDERS_P1 357913941 CUSTOMERS_P2 Chunk #2 TBS_P2 ORDERS_P2 Chunks: 1-6 715827882 … 2147483646 CUSTOMERS_P7 Shard 2 Chunk #7 TBS_P7 ORDERS_P7 2505397587 CUSTOMERS_P8 Chunk #8 ORDERS_P8 TBS_P8 Chunks: 7-12 2863311528
14 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions SDB Architecture – Resharding (Hotspot Elimination)
Adding/Removing shards or hotspot elimination requires chunk movement – Automatically or manually
GDSCTL > MOVE CHUNK -CHUNK 12 -SOURCE sh01 -TARGET sh12
FAN R/O Chunk level 0 Copy files and notification to Chunk R/O on RMAN backup RMAN restore clients source
Drop the chunk FAN chunk File move/ Chunk level 1 on the source up/down events restore RMAN backup
A chunk can also be spit manually within a shard GDSCTL > SPLIT CHUNK -CHUNK 12 Hotspot
15 09.09.2016 Database Sharding with Oracle RDBMS 12c Release 2 - First Impressions Data Sharding Methods
16 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – System-Managed Sharding
Data is automatically distributed across shards using partitioning by consistent hash – Expression: MOD(ORA_HASH('
EXTEND MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO);
17 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – System-Managed Sharding Shards #Chunks=12 #Shards=3 C001TS_SET customers_P1 Shard Catalog C002TS_SET customers_P2 C003TS_SET customers_P3 C004TS_SET customers_P4
CREATE SHARDED TABLE customers
( Set
C005TS_SET customers_P5 custid varchar2(20) CONSTRAINT C006TS_SET customers_P6 pk_customers PRIMARY KEY, C007TS_SET customers_P7 firstname varchar2(50), C008TS_SET customers_P8
lastname varchar2(50), Tablespace ... ) TABLESPACE SET tbs_set C009TS_SET customers_P9 PARTITION BY CONSISTENT HASH (custid) C00ATS_SET customers_P10 PARTITIONS AUTO; C00BTS_SET customers_P11 C00CTS_SET customers_P12
18 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – Composite Sharding
Data is first partitioned by range or list method across multiple shardspaces (set of shards), then by consistent hash in all shards within a shardspace
GDSCTL> ADD SHARDSPACE germany; GDSCTL> ADD SHARDSPACE switzerland;
GDSCTL> ADD SHARD –CONNECT shdb01 –SHARDSPACE germany; GDSCTL> ADD SHARD –CONNECT shdb02 –SHARDSPACE germany; GDSCTL> ADD SHARD –CONNECT shdb02 –SHARDSPACE switzerland; ...
SQL> CREATE TABLESPACE SET de_tbs IN SHARDSPACE germany; SQL> CREATE TABLESPACE SET ch_tbs IN SHARDSPACE switzerland;
19 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Sharding Methods – Composite Sharding Shards
CREATE SHARDED TABLE customers C001DE_TBS customers_P1 (
C002DE_TBS customers_P2 custid varchar2(20), Shardspace
country varchar2(2), GERMANY de_tbs ... C003DE_TBS customers_P3 ) C004DE_TBS customers_P4 PARTITIONSET BY LIST (country) PARTITION BY CONSISTENT HASH (custid) PARTITIONS AUTO ( C005CH_TBS customers_P5
PARTITIONSET germany VALUES ('DE') C006CH_TBS customers_P6 TABLESPACE SET de_tbs Shardspace
PARTITIONSET switzerland VALUES ('CH') SWITZERLAND ch_tbs TABLESPACE SET ch_tbs C007CH_TBS customers_P7 ... C008CH_TBS customers_P8
20 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data High Availability Disaster Recovery
21 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data High Availability / Disaster Recovery – Overview
Data replication with Data Guard is a crucial component in SDB environment – High availability, disaster recovery, read offloading – Replication deployment performed fully automatically – The logical unit of data replication is a shardgroup
Configuration Shard Director 1 Shard Director 2
– Data Guard Broker with FSFO Observers – Local replica (HA), remote replica (DR) – With or without ADG
Primary Standby
22 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Replication – For System-Managed Sharding
DC1 Shard 1 Shard 2 Shard 3 shgrp1 Server B
Shard 4 Shard 5 Shard 6 shgrp2 HA
DC2 Shard 7 Shard 8 Shard 9 shgrp3 Server B DR
23 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Replication – For System-Managed Sharding
GDSCTL> create shardcatalog -database shd01:1521:repo -chunks 12 -user mygdsadmin/
24 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Replication – For System-Managed Sharding
GDSCTL> create shard -shardgroup shgrp1 -destination sh01 -credential oracle_cred GDSCTL> create shard -shardgroup shgrp2 -destination sh02 -credential oracle_cred ... GDSCTL> create shard -shardgroup shgrp3 -destination sh03 -credential oracle_cred ... GDSCTL> deploy
25 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Replication – For Composite Sharding
DC1 Shard 1 Shard 2 shgrp1 Shard 9 Shard 10 shgrp5 Server B
DC2 Shard 3 Shard 4 shgrp2 Shard 7 Shard 8 shgrp4 Server B
DC3 Shard 5 Shard 6 shgrp3 Server B
Shardspace GERMANY Shardspace SWITZERLAND
26 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Replication – For Composite Sharding
GDSCTL> create shardcatalog -database shd01:1521:repo -user mygdsadmin/
27 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Data Replication – For Composite Sharding
GDSCTL> add shardgroup -shardgroup shgrp1 -shardspace germany -deploy_as primary -region germany GDSCTL> add shardgroup -shardgroup shgrp2 -shardspace switzerland -deploy_as active_standby -region switzerland ... GDSCTL> add shardgroup -shardgroup shgrp4 -shardspace switzerland -deploy_as primary -region switzerland
GDSCTL> create shard -shardgroup shgrp1 -destination sh01 -credential oracle_cred ... GDSCTL> deploy
28 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Session/Statement Routing
29 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Session Routing
Session routing is done based on a sharding key (super sharding key) – SQL operations are related to the specified sharding key (shard) – To work on another sharding key value, a new session needs to be created
OLTP_RW.TRIVADIS.COM = (DESCRIPTION= (CONNECT_TIMEOUT=5)(RETRY_COUNT=5)(TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST= Shard Directors (LOAD_BALANCE=ON) (ADDRESS=(PROTOCOL = TCP)(HOST=gsm01.trivadis.com)(PORT=1530)) (ADDRESS=(PROTOCOL = TCP)(HOST=gsm02.trivadis.com)(PORT=1530)) ... Only Composite (CONNECT_DATA= Sharding (SERVICE_NAME=oltp_rw.shdb.trivadis) ([email protected])(SUPER_SHARDING_KEY=germany) (REGION=germany))) Required 30 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Session Routing (GDS) [email protected] 3 1 2 1 2 4 Connection Pool Shard Shard Shard Shard Director 1 Director 2 3 Director 3 Director 4 Region: GERMANY Region: SWITZERLAND
Shard Catalog Shard Catalog
Data Replication
31 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Statement Routing/Cross-Shard Query
Client connection to the Coordinator (Catalog) Database is required – No sharding key necessary in the connect descriptor SQL excuted via DB-Link on Shards
– Partition and Shard pruning @
Shard Shard Shard Shard Director 1 Director 2 Director 3 Director 4 Region: GERMANY Region: SWITZERLAND Shard Catalog
Data Replication
32 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Global Services (GDS)
Affinity to a Local Region – Connection requests are routed to the least-loaded database only in a local region Affinity to a Local Region with Interregion Failover – Connection requests are routed only to the least-loaded database in a local region – If the global service cardinality drops to 0, client is routed to other region
GDSCTL> add service -service oltp_ro –role physical_standby -locality LOCAL_ONLY -region_failover –lag 20
Region: GERMANY Region: SWITZERLAND Global Services: OLTP_RW, OLTP_RO, …
Data Replication
33 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Conclusions
34 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Conclusions
Use case – Custom OLTP applications which require extreme scalability (global data distribution)
Existing database applications need to be adopted for database sharding – Primarily single shard access – With cross-shard operations performance may suffer
Existing database frameworks/features have been extended to allow for massive data scalability and fault isolation – Special business requirements can now also be addressed with Oracle RDBMS – But sharding is complex and requires a lot of knowledge!
35 16.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions Trivadis @ DOAG 2016
Booth: 3rd Floor – next to the escalator Know how, T-Shirts, Contest and more We look forward to your visit Because with Trivadis you always win !
36 17.11.2016 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions