Mysql High Available by Design Carsten Thalheimer

MySQL High available by design Carsten Thalheimer

Sales Consultant MySQL GBU EMEA ([email protected]) Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

• MySQL today • MySQL 5.7 ( Fall 2015) Recap • MySQL 5.7 and MySQL Cluster 7.5 ( Fall 2016) X-Protocol // Group Replication & InnoDB Cluster • MySQL 8.0 - What’s next? MySQL 8.0

Oracle ©|Nov 2016 Slide 3 2.4BN USERS 8x DATA GROWTH IN 5 YRS >1B USERS 70+ NEW DOMAINS EVERY 60 SECONDS 40% DATA GROWTH PER YEAR 600 NEW VIDEOS EVERY 60 SECONDS

$1TR BY 2014 5700 TWEETS PER SECOND $700BN IN 2011

6.7BN MOBILE SUBS IN 2012 85% HANDSETS SHIPPED WITH A BROWSER Oracle ©|Nov 2016 Slide 4 Logarithmic plotting !

Source: http://db-engines.com/en/ranking

Oracle ©|Nov 2016 Slide 5 MySQL – The view from the moon

• MySQL < 5.0/5.1 [pre Oracle] - MyISAM default, InnoDB optional Before 2010 - Very often defaults in Linux distributions

• MySQL 5.5  more ‘SQLish’ - First release under Oracle Dec 2010 - InnoDB default (MVCC, ACID), MyISAM optional - MS Windows optimized

• MySQL 5.6  Replication revamp - InnoDB+, GTID, NoSQL, Performance SchemaIn Feb 2013 - Used by some of the biggest Websites - Introduction of MySQL Fabric

• MySQL 5.7  New ways - InnoDB++, GIS Support, QRP, Instrinc Tables Oct 2015 - MultiThreaded Sls, MultiSourceRpl, Group Rpl - MySQL Fabric, SemiSync+

• MySQL 8.0  ??? - Check Labs! Two Years Cycle?

Oracle ©|Nov 2016 Slide 6 MySQL 5.7

Oracle ©|Nov 2016 Slide 7 MySQL 5.7 is GA (August 2015) Performance & Scalability Manageability

3 X Faster than MySQL 5.6 Native JSON Support

Enhanced InnoDB: faster online & bulk Improved Security: safer initialization, load operations setup & management Replication Improvements (incl. multi- Performance Schema Improvements source, multi-threaded slaves...)

New Optimizer Cost Model: greater user MySQL SYS Schema control & better query performance MySQL 5.7 http://www.thecompletelistoffeatures.com Oracle ©|Nov 2016 Slide 8 Core New JSON features in MySQL 5.7

• Native JSON data type  Native internal binary format for efficient processing & storage. Up to 10x faster than storing as text • Built-in JSON functions Allowing you to efficiently store, search, update, and manipulate Documents • Indexing of Documents using Generated Columns Automatically uses the best “functional” index available for even faster results • JSON Comparator & New inline syntax for easy SQL integration Allows for easy integration of Document data within your SQL queries

Oracle ©|Nov 2016 Slide 9 JSON Functions • 5.7 supports functions to CREATE, SEARCH, MODIFY and RETURN JSON values:

JSON_ARRAY_APPEND() JSON_INSERT() JSON_REPLACE() JSON_ARRAY_INSERT() JSON_KEYS() JSON_SEARCH() JSON_ARRAY() JSON_LENGTH() JSON_SET() JSON_CONTAINS_PATH() JSON_MERGE() JSON_TYPE() JSON_CONTAINS() JSON_OBJECT() JSON_UNQUOTE() JSON_DEPTH() JSON_QUOTE() JSON_VALID() JSON_EXTRACT() JSON_REMOVE()

https://dev.mysql.com/doc/refman/5.7/en/json-functions.html

Oracle ©|Nov 2016 Slide 10 Agenda

• MySQL today • MySQL 5.7 ( Fall 2015) Recap • MySQL 5.7 and MySQL Cluster 7.5 ( Fall 2016) X-Protocol // Group Replication & InnoDB Cluster • MySQL 8.0 - What’s next? MySQL 8.0

Oracle ©|Nov 2016 Slide 11 MySQL 5.7 [2016] - DocumentStore

Oracle ©|Nov 2016 Slide 12 Architecture

SQL API CRUD API MySQL Std Protocol X Protocol Shell

Std Protocol X Protocol X and Std 3306 33060 Protocols

MySQL

Plugins Core X Protocol Plugin Memcached Plugin

Oracle ©|Nov 2016 Slide 13 MySQL as a Document Store – a FULL Stack Store, Retrieve, Search and Managing JSON documents

• Native JSON Datatype, Indexes on JSON Documents • Native Collections with Key Value Semantics • Interactive Shell “MySQL Shell” – Javascript, Python, SQL modes • Connectors include NoSQL CRUD APIs – Java, NodeJS, NET, C++/C, PHP, Python – Method Chaining and Pipelining – Supports Combined Document and Relational

Oracle ©|Nov 2016 Slide 14 MySQL Group Replication (RC)

Oracle ©|Nov 2016 Slide 15 Group Replication Plugin, what Is It? and it is the foundation for the MySQL InnoDB Cluster! “Multi-master update anywhere replication plugin for MySQL with built-in conflict detection and resolution, automatic distributed recovery, and group membership.”

App Servers with • Group Replication library MySQL Router – Implementation of Replicated Database State Machine theory • MySQL GCS is based on Paxos (variant of Mencius) – Provides virtually synchronous replication for MySQL/InnoDB 5.7+

– Supported on all MySQL platforms MySQL Group Replication • Linux, Windows, Solaris, OSX, FreeBSD

Oracle ©|Nov 2016 Slide 16 16

Oracle/MySQL Group Replication – Distributed as part of the official MySQL binaries • from www.mysql.com, edelivery.oracle.com or support.oracle.com – Starting MySQL 5.7.17 and MySQL 8.0.1 (next MySQL) it is available in all MySQL versions by default. – Is/will be integrated with other products of the MySQL Productsuite • MySQL Workbench, MySQL Backup and MySQL Monitor – Easy to monitor (Performance Schema tables for group and node status/stat). – Predictable roadmap and backuped by Oracle MySQL Premier Support

Oracle ©|Nov 2016 Slide 17 1 7

The Theory Behind It • Implementation is based on “Replicated Database State Machines” – Group Communication primitives resemble general properties of Databases – Distributed systems meet Databases: Pedone, Guerraoui, and Schiper paper • Deferred update replication: before committing locally we certify it on all nodes – In order to implement it one needs Atomic Broadcast – the change occurs everywhere or nowhere – This is necessary to ensure data consistency across all nodes • Membership Service – Group Membership: it allows one to know that at a given moment in time all the members that are participating in the protocol are associated with the same logical identifier (view id) – View Synchrony: ensure that messages from past views are all delivered before a new view is installed

Oracle ©|Nov 2016 Slide 18 1 8 MySQL Group Replication “Multi-master update anywhere replication plugin for MySQL with built-in conflict detection and resolution, automatic distributed recovery, and group membership.” App Servers with MySQL Router • Active/Active update anywhere MySQL Shell • Distributed fault tolerance Setup, Manage, Orchestrate • Automatic server fail-over • Automatic reconfiguration MySQL Group Replication • Automatic conflict detection & resolution

Oracle ©|Nov 2016 Slide 19 1 9 MySQL Group Replication

• What does the MySQL Group Replication plugin do for the user? – Removes the need for handling server fail-over. – Provides fault tolerance. – Enables update everywhere setups. – Automates group reconfiguration (handling of crashes, failures, re-connects). – Provides a highly available replicated database. – Automatic distributed coordination (protects against split-brain and message loss). – Less admin overhead, means more fun time!

Oracle ©|Nov 2016 Slide 20 2 0 MySQL Group Replication: Architecture

Node Types R: Traffic routers/proxies: mysqlrouter, haproxy, sqlproxy, ... M: mysqld nodes participating in Group Replication

Oracle ©|Nov 2016 Slide 21 2 1

Multi-Master Update Anywhere! • Any two transactions on different servers can write to the same row • Conflicts will automatically be detected and handled – First committer wins rule

UPDATE t1 SET a=3 WHERE a=1 UPDATE t1 SET a=4 WHERE a=2 OK OK M M M M M UPDATE t1 SET a=3 WHERE a=1 UPDATE t1 SET a=2 WHERE a=1 OK M M M M M

Oracle ©|Nov 2016 Slide 22 2 2 MySQL Router ... • Motivation – Primary Master Mode by default (Single Master) App – (Mulit-Master possible) – Do not need to know which server is the master – Transparent fail-over Router

M M M

• Generic and versatile framework: Oracle ©|Nov 2016 MySQL Router. Slide 23

MySQL InnoDB Cluster (DMR)

Oracle ©|Nov 2016 Slide 24 2 4 MySQL InnoDB Cluster: Vision

“A single product — MySQL — with high availability and scaling features baked in; providing an integrated end-to-end solution that is easy to use.”

Oracle ©|Nov 2016 Slide 25 2 5 MySQL InnoDB Cluster: Goals • One Product: MySQL • Flexible and Modern – All components created together – SQL and NoSQL together – Tested together – Protocol Buffers – Packaged together – Developer friendly

• Easy to Use • Support Read/Write Scale Out – One client: MySQL Shell – Sharded clusters – Easy packaging – Federated system of N replica sets – Integrated orchestration – Each replica set manages a shard – Homogenous servers

Oracle ©|Nov 2016 Slide 26 2 6 MySQL InnoDB Cluster: Architecture – Step 2 RC

Application Application MySQL Connector MySQL Connector

MySQL Router MySQL Router

M HA

MySQL Shell M M Group Replication

Oracle ©|Nov 2016 Slide 27 Labs MySQL InnoDB Cluster: Architecture – Step 3

Application Application MySQL Connector MySQL Connector

MySQL Router MySQL Router

M HA

MySQL Shell M M Group Replication

S1 S2 S3 S4 S…

Read-Only Slaves

Oracle ©|Nov 2016 Slide 28 Labs MySQL InnoDB Cluster: Architecture – Step 4

Application Application Application Application MySQL Connector MySQL Connector MySQL Connector MySQL Connector

MySQL Router MySQL Router MySQL Router MySQL Router

M HA M HA M HA

MySQL Shell

M M M M M M

S1 S2 S3 S4 S… S1 S2 S3 S4 S… S1 S2 S3 S4 S…

ReplicaSet 2 ReplicaSet ReplicaSet 3 ReplicaSet

Oracle ©|Nov 2016 Slide 29 MySQL InnoDB Cluster: High Level Architecture

Application Application Application … Application MySQL Connector MySQL Connector MySQL Connector MySQL Connector

MySQL Router MySQL Router MySQL Router MySQL Router

MySQL MySQL Shell InnoDB MySQL Enterprise Monitor cluster

Oracle ©|Nov 2016 Slide 30 MySQL Cluster 7.5

Oracle ©|Nov 2016 Slide 31 3 1 MySQL NDB Cluster Overview

• Memory optimized tables with durability REAL-TIME • Predictable Low-Latency, Bounded Access Time • Auto-Sharding, Active-Active HIGH SCALE, READS + WRITES • ACID Compliant, OLTP + Real-Time Analytics FULLY ELASTIC • Add and remove storage and performance capacity in seconds • Fully cloud native 99.999% AVAILABILITY • Active-Active, Shared nothing, no Single Point of Failure • Self Healing + On-Line Operations • Key/Value + Complex, Relational Queries SQL + NoSQL • SQL + Memcached + JavaScript + Java + HTTP/REST & C++ • Open Source + Commercial Editions, Easy to use and deploy LOW TCO • Commodity hardware + Management, Monitoring Tools

• Node Sizes of 128TB • MySQL 5.7 • Improved Reporting • Read Optimized Tables • JSON Data Type • Improved Logging • Global Tables • Generated Columns • Improved Visibility • Records-Per-Key • Improved Debugging Optimization • Improved Restore Capacity and Improved Management Scale Out SQL

Oracle ©|Nov 2016 Slide 33 3 3 MySQL Cluster 7.5 NDB Cluster powering the world‘s fastest and most scalable Hadoop filesystem.

Scaling Hadoop 20x!

Oracle ©|Nov 2016 Slide 34 Confidential – Oracle 3 Internal/Restricted/Highly 4 Restricted MySQL Cluster 7.5 NEW! • Based on MySQL 5.7 – Native JSON datatype – Generated Columns • Function Based Indexes – Cost Based Optimizer – New Spatial engine – New hint syntax with additional hints – Memory instrumentation – Secure by default – Much more!

Oracle ©|Nov 2016 Slide 35 Copyright 2016, oracle 3 and/or its affiliates. All 5 rights reserved MySQL Cluster 7.5 NEW! • Continue Improving Supportability – Overload controls via thread idle wait time – Improve event buffer memory allocation – Better and more descriptive error messages – Dump command improvements – Improved Lock reporting • Continue Improving Performance – Faster backup & restore

Oracle ©|Nov 2016 Slide 36 Copyright 2016, oracle 3 and/or its affiliates. All 6 rights reserved MySQL Cluster 7.5 NEW! • Continue Improving Availability – Convert ndb_binlog_index to ACID storage – Make use of dynamic columns everywhere • Support for online defragmentation and space reclamation • Support for row transformations – encryption and compression • Continue Improving Quality – Refactor binlog injector in order to address many known issues – Increase the test base – More system correctness and fault injection tests

Oracle ©|Nov 2016 Slide 37 Copyright 2016, oracle 3 and/or its affiliates. All 7 rights reserved MySQL Cluster 7.5 NEW! • Continue Improving Capacity – Enable individual fragments larger than 16GB (now up to 128TB) • Improved Read Scale-Out – The introduction of “read from backup” for tables • Localized reads on every node in a node group – The introduction of “global” tables (fully replicated) • Localized reads on every node in the Cluster • Support for Increased Redundancy – Support for up to 4 nodes per node group (1-3 replicas per shard)

MySQL Group Replication MySQL NDB Cluster Storage Engine InnoDB NDBCLUSTER Distributed Architecture Shared nothing Shared nothing Consistency Model Weak Consistency Strong Consistency Sharding No Yes Arbitration No Yes Load Balancing No Yes NoSQL APIs MySQL Document Store Native NDB API Operational Complexity Medium High Administration Standard (MySQL) Custom (MySQL + NDB)

• MySQL today • MySQL 5.7 ( Fall 2015) Recap • MySQL 5.7 and MySQL Cluster 7.5 ( Fall 2016) X-Protocol // Group Replication & InnoDB Cluster • MySQL 8.0 - What’s next? MySQL 8.0

• MySQL 8.0: Native Data Dictionary – No FRM/DB.OPT/TRG/TRN/PAR - MyISAM not required • User Management DDLs Atomic ( 100% InnoDB) • Security Roles • Support for the latest Unicode 9.0 • UTF8MB4 as the default character set • Performance Schema Indexes – Implementation of indexes tricks the optimizer into better execution plan

MySQL Enterprise Edition

The Oracle MySQL business…

Licensing Support Tools Cloud

Clients and Applications

NoSQL SQL Firewall Online Backup Simple access patterns Complex queries with joins

Compromise on consistency ACID transactions for performance Audit 3rd Party Tools Monitoring 3rd Party Tools Ad-hoc data format Well defined schemas 3rd Party Tools 3rd Party Tools Encryption/TDE rd Integration Simple operation Rich set of tools 3 Party Tools

Authentication Support InnoDB Storage Engine mysqld process * All Plugins and Tools are ‚free‘ as part of the Support Subscription

Oracle ©|Nov 2016 Slide 45 Copyright © 2015, Oracle 4 and/or its affiliates. All 5 rights reserved. MySQL Enterprise and Cloud Service Support • Largest MySQL engineering and support organization • Backed by the MySQL developers • World-class support, in 29 languages • Hot fixes & maintenance releases • 24x7x365 • Unlimited incidents • Consultative support • Global scale and reach

Oracle ©|Nov 2016 Slide 46 46 Oracle MySQL …there’s much more (https://www.mysql.com/products) MySQL MySQL MySQL MySQL Community Edition Cloud Service Community Edition Cloud Service Oracle Premier Support MySQL Enterprise & Security Tools 24x7 Support √ Oracle Enterprise Manager for MySQL √ Unlimited Support Incidents √ MySQL Enterprise Monitor √ Knowledge Base √ MySQL Enterprise Dashboard √ Maintenance Releases √ MySQL Enterprise Advisors √ MySQL Consultative Support √ MySQL Query Analyzer √ MySQL Features MySQL Replication Monitor √ MySQL Database Server √ √ MySQL Enterprise Backup √ MySQL Connectors √ √ Hot backup for InnoDB √ MySQL Replication √ √ Full, Incremental, Partial, Optimistic Backups √ MySQL Fabric √ √ Full, Partial, Selective, Hot Selective restore √ MySQL Router √ √ Encryption and Compression √ MySQL Partitioning √ √ MySQL Utilities √ √ Point-In-Time-Recovery √ MySQL Workbench √ √ MySQL Enterprise Authentication √ Storage Engine: MyISAM √ √ MySQL Enterprise TDE √ Storage Engine: InnoDB √ √ MySQL Enterprise Encryption √ Storage Engine: NDB MySQL Enterprise Firewall √ Oracle Product Certifications MySQL Enterprise Audit √ Certified with Oracle Linux √ √ MySQL Enterprise Scalability √ Certified with Oracle VM √ √ Thread Pool √ Certified with Oracle Solaris √ √ MySQL Enterprise High-Availability √ Certified with Oracle Enterprise Manager √ HA using Oracle Clusterware √ Certified with Oracle GoldenGate √ HA using Solaris Clustering √ Certified with Oracle Data Integrator √ Certified with Oracle Fusion Middleware √ Certified with Oracle Secure Backup √ Certified with Oracle Audit Vault √ https://cloud.oracle.com/en_US/opc/compute/features Certified with Oracle Database Firewall √

Oracle ©|Nov 2016 Slide 47 4 7 The Winning Combination • Simple, Integrated and Enterprise Ready + – Oracle MySQL Cloud Service enables you to easily deploy modern applications powered by the World’s Most Popular Open Source Database

= – Integrated with Oracle Cloud Services for quick development and deployment

– Based on Oracle’s proven MySQL Enterprise Edition for performance, security & uptime, all MySQL Enterprise Tools, Plug-Ins, Certification and Support are included

2 Overview of MySQL architecture

3 MySQL Replication and MySQL Fabric

4 Commercial MySQL vs. „Open Source“

5 MySQL Subscription / MySQL License

6 A MySQL Reference Questions and Answers