How Does the Mysql Document Store Work? Architecture & Components

MySQL 8.0: Performance & Scalibility Bill Papp MySQL Solutions Architect

CopyrightCopyright © © 2016, 2017, Oracle Oracle and/or and/or its its affiliates. affiliates. All All rights rights reserved. reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Improved UTF-8 Support in MySQL 8.0 • Support for the latest Unicode 9.0 • utf8mb4 made default character set! –utf8mb4_0900_ai_ci default collation • Accent and case sensitive collations –Including 20+ language specific collations –Now also Japanese and Russian • Significantly improved performance

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 5 What Is in MySQL 5.7 and Earlier Versions? • Default charset is “latin1” and default collation is “latin1_swedish_ci” • utf8 = utf8mb3: support BMP Plane 0 only • utf8mb4 character set: –Only accent and case insensitive collations –Default collation is utf8mb4_general_ci, compares all characters beyond BMP, e.g. emojis, to be equal –20+ language specific collations –Recommend to use: utf8mb4_unicode_520_ci

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 6 New Default Character Set • No changes to existing tables • Only has effect on new tables/schemas where character set is not explicitly defined. • Separating character set/collation change from server upgrade – Upgrade first, change charset/collation afterwards • Recommend users to not mixing collations – Error “Illegal mix of collations” – Slower query because index can no longer be used

• Metadata is information about data in the RDBMS – Column definitions, Index definitions, Foreign key definitions ... • Data Dictionary collects metadata for all things in RDBMS – User table structure, Stored program definitions ...

Metadata ID NAME WEIGHT HEIGHT GENDER 3 Bob 80 185 M Data 5 Liz 55 165 F

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. MySQL Data Dictionary before MySQL 8.0 Data Dictionary Files File system FRM TRG OPT SQL System Tables (mysql.) MyISAM user events proc

InnoDB System Tables InnoDB

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 10 MySQL Data Dictionary before MySQL 8.0 Problems • Performance of INFORMATION_SCHEMA • Inconsistencies between persistent storage in files and tables • Inconsistencies between InnoDB DD and Server DD • Showstopper for crash-safe / atomic DDL • Difficult for replication • Hard to extend

• All DD tables and system tables in a DD tablespace (mysql) • Atomic DDL (this is actually a big thing!!) – Especially important for replicated topologies – Crash situations will not end up in out-of-synch information sql-layer/SE-layer • Reliability and Redundancy – Serialized Dictionary Information • Better scaling INFORMATION_SCHEMA queries – Reimplemented with views over DD tables – Scaling both on database size and query load • Improved Upgrade

Data Dictionary

DD Table DDDD Table Table SQL InnoDB

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 13 Transactional Data Dictionary in MySQL 8.0 Main features • All dictionary metadata stored in tables • Single repository of metadata for SEs and the MySQL server • Reliable, crash-safe InnoDB tables • INFORMATION_SCHEMA tables implemented as VIEWs over DD tables – Queries can be optimized by the optimizer – Improved performance – Simpler, uniform implementation, easier to maintain

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Transactional Data Dictionary in MySQL 8.0 Main features, cont’ed • Extensible –The data dictionary schema is based on the SQL standard definitions –The data dictionary is designed to be easily extended for new requirements –Designed to provide automated upgrade of dictionary data –Designed to allow plugins to add and extend system tables, I_S tables, Performance Schema tables

Archive User Table

CSV User Table

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 16 Transactional Data Dictionary API in MySQL 8.0 Design goals • The only way to deal with Data Dictionary – For the server core – For SEs – For other plugins • Less Error-prone • Internal API (non-stable) and external API (stable) • Provide a way to handle SE private data

Query Executor Parser Optimizer Plugin

Data Dictionary External API

Data Dictionary Internal API

Data Storage Dictionary Tablespace User Table Engine Tablespace InnoDB Internal SE Plugin

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Transactional Data Dictionary in MySQL 8.0 Reliability and Redundancy SDI: Serialized Dictionary Information - Used for data migration and is a redundant write-only copy in normal operation The InnoDB Data Dictionary tablespace ”mysql” is the metadata storage InnoDB User Tablespaces InnoDB Data Dictionary tablespace mysql.tables User Table ID Name 1 User Table 1 SDI 2 User Table 2

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 20 Example - DROP SCHEMA at high level MySQL 5.7 MySQL 8.0 • Delete tables • Delete tables – Metadata, TRN/TRG/FRM files – Metadata, rows in InnoDB – Data, InnoDB tables – Data, InnoDB tables • Delete stored programs Delete stored programs – Metadata, rows in MyISAM (non- • transactional) – Metadata, rows in InnoDB • Delete schema • Delete schema – Metadata, DB.OPT file – Metadata, rows in InnoDB Mix of filesystem, non- Updates to transactional storage, transactional/transactional one commit storage and multiple commits

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 21 Reliability – Disaster Recovery Use case – IMPORT using Serialized Dictionary Information • All data stored in InnoDB (transactional SE) – Metadata stored in an SDI embedded in .ibd – Data stored in .ibd • InnoDB logging DDL operations as «Dictionary Storage Engine» – After restart and recovery, the DDL operation will be complete or rolled back

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 22 Reliability – Automated Dictionary Upgrade Use case – Upgrade • The Data Dictionary definition will have a version attached • The MySQL Server executable will support upgrading dictionary tables

• Data Dictionary tables are protected against user DML – This means there will be no possibility for corruption from user DML on these tables – All information in the dictionary tables will be available through INFORMATION_SCHEMA

CREATE TABLE customers( CREATE PROCEDURE p1(v INT) id INT AUTO_INCREMENT SQL SECURITY INVOKER ... BEGIN PRIMARY KEY (id), ... INDEX ... END FOREIGN KEY ... )

Data Dictionary Table Definitions SP Definitions ACL

View Definitions Plugins

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. SELECT ... FOR UPDATE SKIP LOCKED • Common problem: – Hot row contention, multiple worker threads accessing the same rows • Solution 1: – Only read rows that are not locked – InnoDB skips a locked row, and the next one goes to the result set • Example: – Booking system: Skip orders that are pending

START TRANSACTION; SELECT * FROM seats WHERE seat_no BETWEEN 2 AND 3 AND booked = 'NO' FOR UPDATE SKIP LOCKED;

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 27 SELECT… FOR UPDATE NOWAIT • Common problem: – Hot row contention, multiple worker threads accessing the same rows • Solution 2: – If any of the rows are already locked, the statement should fail immediately – Without NOWAIT, have to wait for innodb_lock_wait_timeout (default: 50 sec) while trying to acquire lock • Usage: START TRANSACTION; SELECT * FROM seats WHERE seat_no BETWEEN 2 AND 3 AND booked = 'NO' FOR UPDATE NOWAIT; ERROR 3572 (HY000): Statement aborted because lock(s) could not be acquired …

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Invisible Index • Index is maintained by the SE, but ignored by the Optimizer • Primary key cannot be INVISIBLE • Use case: Check for performance drop BEFORE dropping an index ALTER TABLE t1 ALTER INDEX idx INVISIBLE; mysql> SHOW INDEXES FROM t1; +------+------+------+------+ | Table | Key_name | Column_name | Visible | +------+------+------+------+ | t1 | idx | a | NO | +------+------+------+------+ • To see an invisible index: set optimizer_switch='use_invisible_indexes=on';

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 30 Descending Index CREATE TABLE t1 ( a INT, b INT, INDEX a_b (a DESC, b ASC) ); • In 5.7: Index in ascending order is created, server scans it backwards • In 8.0: Index in descending order is created, server scans it forwards • Works on B-tree indexes only • Benefits: – Use indexes instead of filesort for ORDER BY clause with ASC/DESC sort key – Forward index scan is slightly faster than backward index scan

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Motivation for Improving the MySQL Cost Model • Produce more correct cost estimates – Better decisions by the optimizer should improve performance • Adapt to new hardware architectures – SSD, larger memories, caches Faster • More maintainable cost model implementation queries – Avoid hard coded “cost constants” – Refactoring of existing cost model code • Configurable and tunable • Make more of the optimizer cost-based

Memory 5 s Provide a program that could measure performance and SSD 20 - 146 s suggest good cost constant Hard disk 32 - 1465 s configuration for a running MySQL server? • Adjust cost model to support different storage technologies • Provide configurable cost constants for different storage technologies

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 34 Memory Buffer Aware Cost Estimates • Storage engines: – Estimate for how much of data and indexes are in a memory buffer Query Server executor – Estimate for hit rate for memory buffer • Optimizer cost model: – Take into account whether data is Database Storage buffer already in memory or need to be read engine from disk

Disk data

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 35 MySQL 8.0: Disk vs Memory Access • New defaults for cost constants: Cost MySQL 5.7 MySQL 8.0 Read a random disk page 1.0 1.0 Read a data page from memory buffer 1.0 0.25 Evaluate query condition 0.2 0.1 Compare keys/rows 0.1 0.05

• InnoDB reports for each table/index percentage of data cached in buffer pool • Note: Query plan may change between executions

In-memory Disk-bound In-memory Disk-bound Plan A 5.8 secs 9 min 47 secs MySQL 5.6 Plan B Plan B 77.5 secs 3 min 49 secs MySQL 5.7 Plan A MySQL 8.0 Plan A Plan B

DBT-3 Scale factor 10 In-Memory: innodb_buffer_pool_size = 32 GB Disk-bound: innodb_buffer_pool_size = 1 GB

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 39 Histograms Column statistics • Provides the optimizer with information about column value distribution • To create/recalculate histogram for a column: ANALYZE TABLE table UPDATE HISTOGRAM ON column WITH n BUCKETS; • May use sampling

– Sample size is based on available memory (histogram_generation_max_mem_size) • Automatically chooses between two histogram types: – Singleton: One value per bucket – Equi-height: Multiple value per bucket

EXPLAIN SELECT * FROM customer JOIN orders ON c_custkey = o_custkey WHERE c_acctbal < -1000 AND o_orderdate < '1993-01-01';

select key id table type possible keys key ref rows filtered extra type len i_o_orderdate, Using 1 SIMPLE orders ALL NULL NULL NULL 15000000 31.19 i_o_custkey where eq_ dbt3.orders. Using 1 SIMPLE customer PRIMARY PRIMARY 4 1 33.33 ref o_custkey where

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 41 Histograms Create histogram to get a better plan ANALYZE TABLE customer UPDATE HISTOGRAM ON c_acctbal WITH 1024 buckets; EXPLAIN SELECT * FROM customer JOIN orders ON c_custkey = o_custkey WHERE c_acctbal < -1000 AND o_orderdate < '1993-01-01';

select key id table type possible keys key ref rows filtered extra type len Using 1 SIMPLE customer ALL PRIMARY NULL NULL NULL 1500000 0.00 where dbt3. i_o_orderdate, Using 1 SIMPLE orders ref i_o_custkey 5 customer. 15 31.19 i_o_custkey where c_custkey

• Nearly all database systems rely on some variant of the First-In-First-Out (FIFO) policy. • In a nutshell, FIFO grants locks to those who requested it first (i.e., transactions that are ahead of the queue, unless they’re incompatible with the locks that are currently granted). • CATS is a new algorithm (developed by PhD Students at the University of Michigan) which can dramatically reduce latency and increase throughput when used in place of a FIFO approach.

• .

• The CATS algorithm is based on a simple intuition: not all transactions are equal, and not all objects are equal. • When a transaction already has a lock on many popular objects, it should get priority when it requests a new lock. • In other words, unblocking such a transaction will indirectly contribute to unblocking many more transactions in the system, which means higher throughput and lower latency overall.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Hints: Join Order • Hints to control table order for join execution • 5.7: STRAIGHT_JOIN to force the listed order in FROM clause • 8.0: – JOIN_FIXED_ORDER /* replacement for STRAIGHT_JOIN*/ – JOIN_ORDER /* use specified order */ – JOIN_PREFIX /* use specified order for first tables */ – JOIN_SUFFIX /* use specified order for last tables */ – No need to reorganize the FROM clause to add join order hints like you will for STRAIGHT_JOIN

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 47 Join Order Hints - Example Change join order with hint EXPLAIN SELECT /*+ JOIN_ORDER(customer, orders) */ * FROM customer JOIN orders ON c_custkey = o_custkey WHERE c_acctbal < -1000 AND o_orderdate < '1993-01-01';

select key id table type possible keys key ref rows filtered extra type len Using 1 SIMPLE customer ALL PRIMARY NULL NULL NULL 1500000 33.33 where dbt3. i_o_orderdate, Using 1 SIMPLE orders ref i_o_custkey 5 customer. 15 31.19 i_o_custkey where c_custkey Alternatives with same effect for this query: JOIN_PREFIX(customer) JOIN_SUFFIX(orders) JOIN_FIXED_ORDER()

16 14 12 10 8 6 4 2

Query Execution Time (seconds) Query Execution 0 orders → customer customer → orders

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Hints: Index Merge • Index merge: Merge rows from multiple range scans on a single table • Algorithms: union, intersection, sort union • Users can specify which indexes to use for index merge – /*+ INDEX_MERGE() */ SELECT * FROM t1 WHERE a=10 AND b=10 – /*+ NO_INDEX_MERGE() */ INDEX(a) 10

INDEX(b) 10

Intersection

Result: a=10 AND b=10

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 50 Index Merge Hint - Example Low selectivity EXPLAIN SELECT count(*) FROM users WHERE user_type=2 AND status=0 AND parent_id=0; select possible key id table type key ref rows Extra type keys len parent_id, user_type, Using intersect (user_type, index_ 1 SIMPLE users status, status, 1,1,4 NULL 2511 status, parent_id); merge user_type parent_id Using where; Using index

mysql> SELECT count(*) FROM users WHERE user_type=2 AND status=0 AND parent_id=0; ... 1 row in set (1.37 sec) mysql> SELECT /*+ INDEX_MERGE(users user_type, status) */ count(*) -> FROM users WHERE user_type=2 AND status=0 AND parent_id=0; ... 1 row in set (0.18 sec)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 51 Hints: Set session variables • Set a session variable for the duration of a single statement • Examples: SELECT /*+ SET_VAR(sort_buffer_size = 16M) */ name FROM people ORDER BY name; INSERT /*+ SET_VAR(foreign_key_checks = OFF) */ INTO t2 VALUES (1, 1), (2, 2), (3, 3); SELECT /*+ SET_VAR(optimizer_switch = 'condition_fanout_filter = off') */ * FROM customer JOIN orders ON c_custkey = o_custkey WHERE c_acctbal < 0 AND o_orderdate < '1993-01-01'; • NB! Not all session variables are settable through hints: mysql> SELECT /*+ SET_VAR(max_allowed_packet=128M) */ * FROM t1; Empty set, 1 warning (0,01 sec) Warning (Code 4537): Variable 'max_allowed_packet' cannot be set using SET_VAR hint.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Improved Support for UUID mysql> select uuid(); +------+ | uuid() | +------+ | aab5d5fd-70c1-11e5-a4fb-b026b977eb28 | +------+ • Five “-” separated hexadecimal numbers • MySQL uses version 1, the first three numbers are generated from the low, middle, and high parts of a timestamp. • 36 characters, inefficient for storage Convert to BINARY(16) datatype, only 16 bytes

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 54 Improved Support for UUID Functions to convert UUID to and from binary • UUID_TO_BIN(string_uuid, swap_flag) • BIN_TO_UUID(binary_uuid, swap_flag) Feature Request • IS_UUID(string_uuid) from Developers

From VARCHAR(36) 53303f87-78fe-11e6-a477-8c89a52c4f3b

To BINARY(16) 11e678fe53303f87a4778c89a52c4f3b

Insert Performance Optimized Original 25 26 27 28 29

– No longer truncation beyond 64 bits Feature Request from Developers

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 57 All These Features and More… • Improved Query Scan Performance • Skip index dives for FORCE INDEX • GROUPING() • Parser Refactoring • Hint: Merge/materialize derived table/view • JSON: Try it out! – Partial update – Improved performance of sorting and grouping of JSON values Give us feedback! – Path expressions: Array index ranges – JSON_MERGE_PATCH()

• New option: information_schema_stats_expiry – Default 24h – Cache dynamic values in mysql.index_stats and mysql.table_stats – Set to 0, always fetch data from SE and no «stats» will be stored – ANALYZE will update mysql.index_stats and mysql.table_stats

• Improved performance fetching dynamic data from SE (InnoDB) – Typically 4-5 times faster than 5.7 (but note that this is only relevant for a few values)

Uniform, simpler implemention makes it a lot faster

MySQL Client MySQL Client

I_S Query Results I_S Query Results

MySQL Server MySQL Server

Return rows to user. Return rows to user. Optimizer prepares Create temporary table. execution plan. File system / MyISAM / InnoDB engine InnoDB storage engine TEMP TABLE

Heuristic optimization. Executor reads metadata from data dictionary tables.

Read metadata from File system or from MyISAM/InnoDB engine.

INFORMATION_SCHEMA in 5.7 INFORMATION_SCHEMA in 8.0

I_S queries scale, both with database size and query load • Typically 30X performance improvements over MySQL 5.7 • More than 100X for some queries like: List all InnoDB table columns

List all InnoDB tables columns 5k tables

MySQL 8.0

List all InnoDB tables columns 10k tables MySQL 5.7

0 20 40 60 80 100 120 140 160 Time in Seconds (Lower is better)

100 schemas times 50 tables (5000 tables)

Count All Schemas Schema aggregate size stats All Dynamic Table Info All Static Table Info MySQL 8.0 Auto Increments Near Limit MySQL 5.7 Count All Columns Count All Indexes

0 0.5 1 1.5 2 2.5 3 3.5 4

Time in Seconds (Lower is better)

MySQL 5.5 MySQL 5.6 MySQL 5.7 MySQL 8.0

• Event Waits • Statement • Memory • Histograms Instrumentation • Mutexes Instrumentation • Indexes • Lower Overhead • Prepared Statements • Files Instrumentation • Data Locks instrumentation • Threads • Transactions Instrumentation • SQL Errors instrumentation • Scalable Memory Allocation • Variables Instrumentation • Bundled SYS schema • Table plugin • Lower Overhead • Improved Defaults

SELECT * FROM sys.session 1000 active sessions

Query Time MySQL 8.0 MySQL 5.7 0. 10. 20. 30. 40.

Time in Seconds (Lower is better)

• New mechanism to control resource usage for DML load – Can be used to assign execution threads to CPUs – Use “hints” to assign to a resource group per query • RESOURCE GROUP management – CREATE/DROP groups – Assign priority – Assign CPUs – Enable/disable • Supported in INFORMATION_SCHEMA and PERFORMANCE_SCHEMA

160,000

140,000

120,000

100,000

80,000 Select Update 60,000 Queries per Second 40,000

20,000

0 System Configuration : No Resource Group With Resource Group Oracle Linux 7, Intel(R) Xeon(R) CPU E7-4860 2.27GHz (40 Cores Shared) (40 Cores for Select) 40 cores-HT (10 Cores for Update RG)

CopyrightCopyright © © 201 2015,5Oracle, Oracle and/or and/or its its affiliates. affiliates. All All rights rights reserved. reserved. | MySQL 8.0: SysBench Read Write (Mixed) 30% Faster than MySQL 5.7

1,000,000 900,000

800,000 MySQL 8.0 700,000 600,000 500,000 MySQL 5.7 400,000

Queries per Second 300,000 MySQL 5.6 200,000 100,000 0 1 2 4 8 16 32 64 128 256 512 1,024 Users OS : Oracle Linux 7.4 CPU : 48cores-HT Intel Skylake 2.7Ghz (2CPU sockets, Intel(R) Xeon(R) Platinum 8168 CPU) RAM: 256GB Storage : x2 Intel Optane flash devices (Intel (R) Optane (TM) SSD P4800X Series)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 73 MySQL 8.0: SysBench Read Write (update nokey) 2x Faster than MySQL 5.7

300,000

250,000 MySQL 8.0

200,000 MySQL 5.7 150,000

100,000

Queries per Second MySQL 5.6

50,000

0 1 2 4 8 16 32 64 128 256 512 1,024 OS : Oracle Linux 7.4 CPU : 48cores-HT Intel Skylake 2.7Ghz Users (2CPU sockets, Intel(R) Xeon(R) Platinum 8168 CPU) RAM: 256GB Storage : x2 Intel Optane flash devices (Intel (R) Optane (TM) SSD P4800X Series)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 74 MySQL 8.0: SysBench IO Bound Read Only (Point Selects) 2x Faster than MySQL 5.7

1,200,000

1,000,000

MySQL 8.0 800,000

600,000 MySQL 5.7

400,000 Queries per Second MySQL 5.6 200,000

0 1 2 4 8 16 32 64 128 256 512 OS : Oracle Linux 7.4 Users CPU : 48cores-HT Intel Skylake 2.7Ghz (2CPU sockets, Intel(R) Xeon(R) Platinum 8168 CPU) RAM: 256GB Storage : x2 Intel Optane flash devices (Intel (R) Optane (TM) SSD P4800X Series)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 75 MySQL 8.0: SysBench OLTP Read Only (Mixed - utf8mb4) 40% Faster than MySQL 5.7

1,000,000 MySQL 8.0 900,000 800,000 MySQL 5.7 700,000

600,000 MySQL 5.6 500,000 400,000

Queries per Second 300,000 200,000 100,000 0

1 2 4 8 16 32 64 128 256 512 1,024 OS : Oracle Linux 7.4 Users CPU : 48cores-HT Intel Skylake 2.7Ghz (2CPU sockets, Intel(R) Xeon(R) Platinum 8168 CPU) RAM: 256GB Storage : x2 Intel Optane flash devices (Intel (R) Optane (TM) SSD P4800X Series)

MySQL 8.0 Overview

Bill Papp Principal Solutions Architect

CopyrightCopyright © 2018,© 2017, Oracle Oracle and/or and/or its its affiliates. affiliates. All All rights rights reserved. MySQL Innovation: 5.7 -> 8.0 MySQL 8.0 - Document Store - Data Dictionary MySQL InnoDB Cluster (GA) - Roles - Unicode - MySQL Group Replication - CTEs - MySQL Router - Window Functions MySQL 5.7 (GA) - MySQL Shell - Security - 3x Better Performance - Replication - Replication Enhancements - SysSchema - Optimizer Cost Model - GIS - JSON Support - Improved Security - Sys & Performance Schema 2 Years in Development - GIS 400+ Worklogs 5000+ Bugs Fixed 500 New Tests

Mobile First

Developer First 8.0 Data Driven 24x7 at Scale MySQL Document Store MySQL NoSQL Relational Tables JSON Documents Foreign Keys Schemaless JSON Collections X Dev API SQL CRUD

An easy, straight forward way to work with JSON documents in MySQL

• Schemaless: No centralized database schema – Data model enforcement and validation (if any) at application layer – Simpler schema updates (no ALTER TABLE penalty) • NoSQL APIs: Simpler programming interfaces – No specialized language for queries and data manipulation – Complex queries handled at application layer (no complex SELECTs, JOINs) – Document in, document out, manipulations at client side • Scalability, but some drawbacks: – Limited database features (no foreign keys, no transactions, etc.) – Weak consistency guarantees

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 92 NoSQL, MySQL, Why not… • Have both schema-less and schema in the same technology stack? • One that checks all the boxes of all stakeholders:

Developers Operations ✔ Schemaless or/and Schema ✔ Performance Management/Visibility ✔ Rapid Prototyping/Simpler APIs ✔ Robust Replication, Backup, Restore ✔ Document Model ✔ Comprehensive Tooling Ecosystem ✔ Transactions ✔ Simpler application schema upgrades Business Owner ✔ Don’t lose my data = ACID transactions ✔ Capture all my data = Extensible/Schemaless ✔ Help Comply with Regulations = ‘Make Sure It’s Secure!’ ✔ Products On Schedule/Time to Market = Rapid Development

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 93 MySQL Document Store: Components • MySQL X Plugin • X DevAPI • Introduces X Protocol for relational- and • New, modern, async developer API for CRUD document operations and SQL operations on top of X Protocol • Maps CRUD operations to standard SQL • Introduces Collections as new Schema obj. (relational tables, JSON datatype and functions) • MySQL Shell • Offers interactive X DevAPI mode for app • X Protocol prototyping • New MySQL client protocol based on top of industry standard (Protobuf) • MySQL Connectors • Works for both, CRUD and SQL operations • Support for X DevAPI for • JavaScript, Python, PHP, Java, C#, C++ • InnoDB Cluster • Read-Scaling, Write-Scaling, HA

CRUD requests + JSON

Frontend Backend MySQL

JSON

Document Store Application

DevAPI Protobuf / X Protocol / TCP/IP SQL Application Connector X Plugin MySQL

InnoDB

MySQL Document Store Read Scale-Out ✔ ✔ Relational & Document Model S1 Async Replication + Auto Failover S3

Timeline

MySQL HA Write Scale-Out ✔ Out-Of-Box HA S2 Sharding S4

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 97 MySQL Shell Interface for Development and Administration of MySQL • Scripting for Javascript, Python, and SQL mode • Supports MySQL Standard and X Protocols • Document and Relational Models • CRUD Document and Relational APIs via scripting • Traditional Table, JSON, Tab Separated output results formats • Both Interactive and Batch operations

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. MySQL Shell: What’s New • Customizable prompt • Include context and session information • Custom font and color support • Persistent command line history • Auto-complete / Content Assistance • Full Unicode support

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. JSON Support • Native File Format • Virtual Columns • 20+ Functions • New! Search Functions • New! Aggregations Functions – Query structured data and semi-structured JSON data

JSON_ARRAY_APPEND() JSON_MERGE[_PRESERVE]() JSON_PRETTY() JSON_ARRAY_INSERT() JSON_OBJECT() JSON_STORAGE_SIZE() JSON_ARRAY() JSON_QUOTE() JSON_STORAGE_FREE() JSON_CONTAINS_PATH() JSON_REMOVE() JSON_ARRAYAGG() JSON_CONTAINS() JSON_REPLACE() JSON_OBJECTAGG() JSON_DEPTH() JSON_SEARCH() JSON_MERGE_PATCH() JSON_EXTRACT() JSON_SET() JSON_TABLE() JSON_INSERT() JSON_TYPE() JSON_KEYS() JSON_UNQUOTE() JSON_LENGTH() JSON_VALID()

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. MySQL 8.0: UTF-8 Encoding • Emoji characters used as input • MySQL 8.0 defaults to utf8mb4 • Latest Unicode 9.0 Support • New collations based on DUCET, accent and case sensitive collations, Japanese, Russian

1,000,000 MySQL 8.0 900,000 800,000 MySQL 5.7 700,000

600,000 MySQL 5.6 500,000 400,000

Queries per Second 300,000 200,000 100,000 0

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 105 MySQL 8.0: Boosts Developer & DevOps Productivity • New! CTEs and Recursive CTEs • New! Window Functions • New! SKIP LOCKED and NOWAIT • UUIDs and Bitwise Functions

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 106 New! Invisible Indexes Feature Request • Indexes are “hidden” to the MySQL Optimizer from DBAs – Not the same as “disabled indexes” – Contents are fully up to date and maintained by DML • Two use cases: – Soft Delete (Recycle Bin) – Staged Rollout

Feature Request • Two of our most requested features: from Developers – Common Table Expressions (CTEs) – Window Functions

App Servers with MySQL Router “High Availability becomes a core first class feature of MySQL!” MySQL Shell Setup, Manage, Orchestrate

MySQL Group Replication

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 111 MySQL Shell: DBA Admin API • The global variable 'dba' is used to access the MySQL AdminAPI App Servers with • mysql-js> dba.help() MySQL Router • Perform DBA operations

– Manage MySQL InnoDB clusters MySQL Shell • Create clusters Setup, Manage, • Validate MySQL instances Orchestrate • Configure MySQL instances • Get cluster info MySQL Group Replication • Modify clusters • and much more ...

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 112 MySQL Group Replication: Database HA • Group Replication library – Implementation of Replicated Database State Machine • MySQL GCS is based on our home-grown Paxos implementation App Servers with – Provides virtually synchronous replication for MySQL 5.7+ MySQL Router • Guarantees eventual consistency – Automates operations MySQL Shell • Conflict detection and resolution Setup, Manage, Orchestrate • Failure detection, fail-over, recovery • Group membership management and reconfiguration MySQL Group Replication “Multi-master update anywhere replication plugin for MySQL with built-in conflict detection and resolution, automatic distributed recovery, and group membership.”

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. New! Transactional Data Dictionary • Crash-safe Database • Meta-data locking for FK constraints – Common data dictionary for the server and InnoDB –FK moved from InnoDB to server layer • Crash-safe & Atomic DDL • Scalable Information Schema – CREATE USER , DROP DATABASE with all-or-none semantic – Queries are now executed as set – Simplifies replication failure cases of SQL views on tables – Large performance improvements

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. New! Performance Improvements • Improved Query Consistency • Parallel Replication • Histograms • UTF8MB4 • Improved Cost Model • Information Schema • Faster Table/Range Scans • Performance Schema Indexes

300,000

250,000 MySQL 8.0

200,000 MySQL 5.7 150,000

100,000

Queries per Second MySQL 5.6

50,000

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 120 MySQL 8.0: SysBench IO Bound Read Only (Point Selects) 2x Faster than MySQL 5.7

1,200,000

1,000,000

MySQL 8.0 800,000

600,000 MySQL 5.7

400,000 Queries per Second MySQL 5.6 200,000

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. New! MySQL 8.0: Roles Feature Request • Improving MySQL Access Controls from DBAs • Easier to manage user and applications rights • As standards compliant as practically possible • Multiple default roles

• Export the role graph in GraphML Directly Default Role(s) Set of ACLS Indirectly Set Role(s) Set of ACLS

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 123 New! MySQL 8.0: Dynamic Privileges Provides finer grained administrative level access controls • Too often super is required for tasks when less privilege is really needed – Support concept of “least privilege” • Needed to allow adding administrative access controls – Now can come with new components – Examples • Replication • HA • Backup • Give us your ideas

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 124 MySQL Password Features • New! Password history - provides DBAs more password management – Require new passwords not reuse old ones - By number of changes and/or time. – Establish password-reuse policy globally as well as on a per-account basis. • New! SHA2 with Caching – Strong and Fast – Strong - SHA-256 password hashing (many rounds, seeds, …) – Fast - Caching • Greatly reduces latency • New! Supports for more connection protocols • New! Seamless RSA password-exchange capabilities (No linking OpenSSL)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 125 MySQL 8.0: File Encryption • New! AES 256 encryption of UNDO and REDO Logs • Super Simple to manage - Set – innodb_undo_log_encrypt=ON/OFF – innodb_redo_log_encrypt=ON/OFF • And – ON - Pages written after setting are encrypted – OFF - Pages written after setting are not.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 126 MySQL 8.0: Upgrade Checker • Quick and Easy MySQL Shell Utility – JavaScript – Python • Identifies Issues Based on Severity – No Issues – Potential Errors – Errors that must be fixed before Upgrading • Recommends Fixes – Schema, Configuration – Data on Server, etc.