Mysql Engineering

The State of the Dolphin

Edward Screven, Chief Corporate Architect Tomas Ulin, VP MySQL Engineering

CopyrightCopyright © © 2014 2014, Oracle, Oracle and/or and/or its its affiliates. affiliates. All All rights rights reserved. reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2x Engineering Staff 3x QA Staff 2x Support Staff

Jan 2010 Sept 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 3 Driving MySQL Innovation: 2010 - 2014 MySQL Enterprise Monitor 2.2 Oracle Products Certifications MySQL Windows Tools MySQL Cluster 7.1 MySQL Windows Installer MySQL Database 5.6 MySQL Cluster Manager 1.0 MySQL Enterprise Security MySQL Cluster 7.3 MySQL Workbench 5.2 MySQL Enterprise Scalability MySQL Enterprise Monitor 3.0 MySQL Database 5.5 MySQL Enterprise Audit MySQL Workbench 6.1 MySQL Enterprise Backup 3.5 MySQL Cluster 7.2 MySQL Enterprise Backup 3.11 MySQL Cluster Manager 1.3 MySQL Enterprise Monitor 2.3 MySQL Fabric MySQL Utilities MySQL Workbench 6.2 MySQL Cluster Manager 1.1 MySQL Workbench 6.0 MySQL Enterprise Backup 3.7 All GA! MySQL Database 5.7 DMRs* All GA! All GA! MySQL Cluster 7.4 DMR

Available Now!

Partial List of Releases Delivered *Development Milestone Release

Best Choice for Next Generation Web & Cloud Applications Strong MySQL Momentum

Leading Open Source World’s Most Popular Open Database for Web Source Database Applications

#1 Open Source Database Integrated with Hadoop in in the Cloud Big Data Platforms

Focus on Web, Cloud, Embedded

• Architecture Improvements – Improved Modularity – Refactoring Optimizer, Parser, Runtime • Web Scale Performance & Scalability – Optimizer Cost Model – InnoDB & Replication Improvements • Management & Security – Data Dictionary – MySQL Enterprise Encryption – Oracle Enterprise Manager for MySQL

Improves Mobile to Mobile Communications

Challenges Solution

. Evaluated several alternatives & selected MySQL: Very low latency . Implement new messaging variations, optimized with Flash, proven technology at Facebook. system to improve mobile to mobile communications . Highly valued the new MySQL 5.6 features: GTIDs & Multi-threaded management. Slaves improving write scalability, and leverages semi-synchronous replication, improving reliability. . User facing application with the following requirements: Benefits . Very high reliability . New IRIS application in production handling several millions of . Very low response times mobile to mobile messages/sec. . Consistency across devices . “No Data Loss” system and 10X improvement in write scalability.

Scales with MySQL Embedded

Challenges Solution

. Deliver patient privacy monitoring . MySQL as embedded database, and recently upgraded to MySQL solutions for electronic health 5.6 to benefit from improvements in InnoDB and the MySQL records Performance Schema

. Customer Average of 700K events . Leverage MySQL Enterprise Monitor & MySQL Enterprise Backup to processed daily save time, ensure optimal performance and high availability . Customers manage up to 1TB of new data each month, resulting in Benefits 14+TB deployments . Delivering high performance, extremely reliable and “zero admin” . 250 Billion events under patient privacy monitoring solutions to thousands of hospitals and management clinics . Ability to scale to meet customers’ growing demands with few . Compatibility with 250+ healthcare applications DBAs

2x Faster than MySQL 5.6 645,000 QPS 3x Faster than MySQL 5.5

MySQL 5.7: Sysbench Read Only (Point Select)

700,000 MySQL 5.7

600,000

500,000 400,000 MySQL 5.6 300,000

200,000 Queries per Second per Queries 100,000 0 MySQL 5.5 8 16 32 64 128 256 512 1,024 Connections Intel(R) Xeon(R) CPU E7-4860 x86_64 4 sockets x 10 cores-HT (80 CPU threads) 2.3 GHz, 512 GB RAM Oracle Linux 6.5

Improving Query Performance

• New cost model enables storage engines to provide – Accurate and dynamic cost estimates – Key lookups, table scans, range scans, index scans, etc… • Configurable costs for additional factors – Speed of disk I/O operations – Speed of memory operations • Improved records per key estimates • Cost values included in JSON Explain output

• Single repository of metadata for Storage Engine and MySQL server – Leveraging InnoDB Strengths – Reliable crash-safe tables – Replacing the FRMs • INFORMATION_SCHEMA tables can be simple VIEWs – Queries can be optimized – Improved performance • Extensible – Designed to provide backward-compatibility – Designed with plugins in mind

• MySQL Fabric – HA & Sharding • Provisions machines & MySQL – OpenStack Nova – Coming Soon: Trove, AWS • Server setup – Clones slave – Sets up replication

Standards Based Cryptography

• MySQL encryption libraries – Symmetric encryption AES256 – Public-key / asymmetric cryptography • Key management – Generate public and private keys – Key exchange methods: RSA, DSA, DH • Sign and verify data – Cryptographic hashing for digital signing, verification, & validation • Integrated with Oracle Key Vault

Single Dashboard to manage Oracle stack for Web & Cloud

• An estimated 70% of Oracle customers also use MySQL – For Web, custom, departmental and embedded applications – #1 Requested integration

The State of the Dolphin Tomas Ulin, VP MySQL Engineering

What’s New

• MySQL 5.7.5 Development Milestone Release (DMR) • MySQL Fabric 1.5 GA • MySQL Workbench 6.2 GA • MySQL Cluster 7.4 DMR • MySQL Enterprise Edition – MySQL Enterprise Encryption – Oracle Enterprise Manager for MySQL • Early Access Features via labs.mysql.com

• Worklogs Implemented in 5.7: 244 • New MTR Tests for 5.7: 497 • Total Bugs Fixed in 5.7: 1263 • Total Bugs Fixed since 5.5: 8259 – Since MySQL 5.5 GA Dec 15, 2010

6x Faster than MySQL 5.6 1 Million QPS Thank you, Facebook

MySQL 5.7 vs 5.6 - InnoDB & Memcached

1,200,000

1,000,000 MySQL 5.7

800,000

600,000

400,000

Queries per Second per Queries 200,000 MySQL 5.6

0 8 16 32 64 128 256 512 1,024 Connections Intel(R) Xeon(R) CPU E7-4860 x86_64 4 sockets x 10 cores-HT (80 CPU threads) 2.3 GHz, 512 GB RAM Oracle Linux 6.5

1.7x Faster than MySQL 5.6 67,000 Connections/Sec 2.5x Faster than MySQL 5.5

Connections Per Second 80,000 70,000

60,000

50,000 MySQL 5.5 40,000 MySQL 5.6 30,000 MySQL 5.7

Connections/Second 20,000

10,000

0 Intel(R) Xeon(R) CPU E7-4860 x86_64 MySQL 5.5 MySQL 5.6 MySQL 5.7 4 sockets x 10 cores-HT (80 CPU threads) 2.3 GHz, 512 GB RAM Oracle Linux 6.5

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 21 Optimizer: New Cost Model - Why? • More accurate cost estimates – Better decisions by the optimizer should improve query performance • Adapt to new hardware architectures – SSDs, larger memory sizes, improved caches • More maintainable cost model implementation – Avoid hard coded “cost constants” – Refactoring of existing cost model code • Configurable and tunable

{ "query_block": { • Expanded JSON EXPLAIN "select_id": 1, "cost_info": { "query_cost": "200.40" – Now includes all available cost info }, "table": { "table_name": "nicer_but_slower_film_list", – Used for Visual Explain In MySQL Workbench "access_type": "ALL", "rows_examined_per_scan": 992, "rows_produced_per_join": 992, "filtered": 100, "cost_info": { "read_cost": "2.00", "eval_cost": "198.40", "prefix_cost": "200.40", "data_read_per_join": "852K" }, "used_columns": [ "FID", "title", "description", "category", "price", "length", "rating", "actors" ], ...

• Query rewrite (pre and post parse) • Post-parse query plugin – Rewrite problematic queries without application changes – Add hints – Modify join order – Many more … • Improve problematic queries from ORMs, third party apps, etc.

Query Executioner Parser Optimizer Plugin Plugin Plugin

Data Dictionary External API

Data Dictionary Internal API

Data Storage Storage Dictionary Tablespace User Table Engine Engine Tablespace InnoDB Internal SE Plugin Plugin

Memory Instrumentation Statement Instrumentation Additional Information • Aggregates statistics by • Stored Procedures • Replication slave status – Type of memory used • Stored Functions • MDL lock instrumentation (caches, internal buffers, …) – Thread/account/user/host • Prepared Statements • User variables per thread indirectly performing the • Transactions • Server stage tracking memory operation • Attributes include • Track long running SQL – Memory used (bytes) • All while reducing total – Operation counts footprint and overhead – High/Low Water Marks

Helper objects for DBAs, Developers and Operations staff • Helps simplify DBA / Ops tasks - Monitor server health, user and/or host statistics, and other key metrics - Spot, diagnose, and tune performance issues • Provides easy to understand views with insights into - IO hot spots, Locking and Costly SQL statements - Schema, table and index statistics • SYS is similar to - Oracle V$ catalog views - Microsoft SQL DMVs (Dynamic Mgmnt Views)

• Transparent Page Level Compression – Happens transparently in background threads – Managed entirely within the IO layer – Uses sparse file and "hole punching" support in OS kernels and File Systems • Reduces IO – Improves MySQL performance – Improves storage efficiency – Reduces write cycles, thus increasing SSD lifespan • Applies to all InnoDB data, including the system tablespace and UNDO logs

• InnoDB support for MySQL Group Replication – High priority transactions • InnoDB Native Partitioning – Will allow us to eliminate previous limitations on partitioned tables – Removes memory usage problems when many partitions are used • Support for 32K and 64K pages • General TABLESPACE support – Store multiple tables in user defined tablespaces

Master 1 Master 2 … Master N • Consolidate updates from multiple Masters into one Slave Binlog Binlog … Binlog – Consolidated view of all shards

– More flexible topologies IO 1 IO 2 … IO N

– Centralized point for backups Relay 1 Relay 2 … Relay N • Compatible with Semi-Sync Coordinator Coordinator Coordinator Coordinator Replication & enhanced MTS W1 W2 … WX W1 W2 … WX W1 W2 … WX W1 W2 … WX • Master-specific slave filters

Slave

Slave Transactions per Second • Up to 5X performance vs single 5000 threaded slave 4500 – 4000 No changes to the application 3500

– No manipulation (delaying) of Binary 3000 Baseline log Group Commit 2500 50 clients 2000 100 clients 150 clients 1500 • GTIDs & Crash-Safe Slave enabled 200 clients 1000 • Sysbench OLTP test 500 0 – 10M rows 0 4 16 25 50 100 200 Worker Threads – SSD / 48 core HT / 512 GB RAM

• Shared-nothing virtually synchronous database system • Multi-master update anywhere – Conflict detection and resolution (transaction rollback) Application MySQL – Optimistic State Machine Replication Server • Automatic group membership management and failure API detection

MySQL Masters Replication – No need for server fail-over Plugin – Elastic scale out/in – No single point of failure Group Comms (Corosync) – Automatic reconfiguration • Well integrated – InnoDB – GTID-based replication – PERFORMANCE_SCHEMA

• OpenStack Integration Application Application • High Availability

Connector Connector – Server monitoring; auto-promotion and transparent application failover SQL MySQL Fabric • Sharding for Scale-out – Application provides shard key • Integer, date or string – Range or Hash – Tools for resharding • Fabric-aware connectors rather than proxy: Python, Java, PHP, .NET, C (labs) Read-slaves Read-slaves – Lower latency, bottleneck-free HA group HA group

MySQL MySQL Oracle VM Oracle Solaris Windows MySQL DRBD Replication Fabric Template Clusterware Cluster Cluster Cluster

App Auto-Failover ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Data Layer Auto-Failover ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Zero Data Loss MySQL 5.7 MySQL 5.7 ✔ ✔ ✔ ✔ ✔ ✔ Platform Support All All Linux Linux Solaris Windows Linux All Master + Master + Active/Pas Active/Passi Active/P Active/Pas Active/P Multi- Clustering Mode Slaves Slaves sive ve assive sive assive Master Failover Time N/A Secs Secs + Secs + Secs + Secs + Secs + < 1 Sec Scale-out Reads ✔ ✖ ✖ ✖ ✖ ✖ ✔ Cross-shard operations N/A ✖ N/A N/A N/A N/A N/A ✔ Transparent routing ✖ For HA ✔ ✔ ✔ ✔ ✔ ✔ Shared Nothing ✔ ✔ ✖ ✖ ✖ ✖ ✔ ✔ Storage Engine InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ NDB Single Vendor Support ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔

Thank you, Simon Mudd at booking.com

• Native support for syslog • Simple option to (re)direct log output to native syslog facility • Start-up server configuration option • Dynamically in the running server – System variable log_syslog (ON/OFF, defaults to OFF).

Thank you, Konstantin Osipov! Multiple User Level Locks per Connection • User-level locks can be used to organize mutual exclusion – When accessing some resource – When table or row-level locks are not appropriate • Request multiple locks by issuing a series of GET_LOCK statements • Replaces custom user-level lock implementation – With one based on the MDL lock manager – Deadlocks between different connections acquiring user-level locks, metadata locks, and those waiting for table flushes are properly detected and reported as errors.

Thank you Davi Arnaut! • Server side statement timeouts – Global for server, per session, or for individual SELECT statements

SELECT MAX_STATEMENT_TIME = 109 * FROM my_table; • Expanded to Windows and Solaris, restricted by removing USER option

• Replaced custom code – For spatial calculations – For spatial analysis • Provides OGC compliance – With improved performance • Boost.Geometry contains – Field and domain experts – Bustling and robust community • We’re also Boost.Geometry contributors!

• Server Plugin adds HTTP(S) endpoints to MySQL • Results are serialized to JSON format encoded as UTF8 • Provides 3 choices of User Endpoint Types – SQL – CRUD - Key-Value – JSON - Document

• Fabric – Add fabric node, browse, view, connect • Performance Dashboard – Performance Schema Reports & Graphs • Visual Explain • GIS Viewer • Migration – New Microsoft Access – Microsoft SQL Server, Sybase, PostgreSQL

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 55 MySQL on Windows • MySQL Installer for Windows • MySQL Workbench • MySQL Migration Wizard – Microsoft SQL Server – Microsoft Access • MySQL for Visual Studio • MySQL for Excel • MySQL Notifier • MySQL Connector/.Net • MySQL Connector/ODBC

• Simple and convenient way to install & • The latest releases of update MySQL products – MySQL Database • Supports the following distributions – MySQL Workbench – Oracle, Red Hat, CentOS – MySQL Connector/ODBC – Fedora – MySQL Connector/Python – Ubuntu, Debian – MySQL Connector/NET • Coming Soon – MySQL Utilities – SUSE – Preconfigured Containers – Improved support for popular DevOps deployment tools

MySQL Source Code Available on GitHub

• Git for MySQL Engineering – Fast, flexible and great for a distributed team – Great tooling – Large and vibrant community • GitHub for MySQL Community – Easy and fast code availability to the community and to downstream projects – Beta repo available today at https://github.com/mysql – More info at http://mysqlrelease.com

What’s New

• Performance gain over 7.3 • Faster maintenance operations – 47% (Read-Only) – Nodal & Rolling restarts – 38% (Read-Write) – Upgrades

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 60 MySQL Cluster: Active-Active Geo-Replication • Asynchronous replication between MySQL Clusters • Active-Active – Update anywhere – Conflict detection • Application notified through exception tables • Can opt to have conflicts resolved automatically – Auto-conflict-resolution • Conflicting transaction and dependent ones are rolled-back • No required changes to app schema

What’s New

Advanced Features Management Tools Support

• Scalability • Monitoring • Technical Support • High Availability • Backup • Consultative Support • Security • Development • Oracle Certifications • Audit • Administration • Encryption • Migration

• Performance & Availability Monitoring • Visually find & fix problem queries • Disk monitoring for capacity planning • Cloud friendly architecture - Policy driven configuration - No remote agent required • Start monitoring MySQL in 10 minutes

Fast, online backup & recovery • Online Backup for InnoDB (scriptable interface) • Full, Incremental, Partial Backups (with compression) • Point in Time, Full, Partial Recovery • Parallel, multi-threaded backup/recovery operations • Directly to/from Cloud Store mysqlbackup • Encryption – AES 256 • Binlog and Relay Log Backup MEB Backup MySQL • Certified with Oracle Secure Backup Files Database Files

MySQL & Oracle Product Integrations • Oracle Linux • Oracle Enterprise Manager • Oracle VM • Oracle Fusion Middleware • Oracle Solaris • Oracle GoldenGate • Oracle Clusterware • Oracle Audit Vault & Database Firewall • Oracle Secure Backup • MyOracle Online Support

MySQL Integrates into your Oracle Environment

•MySQL Performance Tuning • MySQL Cluster Latest Developments •InnoDB: What’s New • MySQL Fabric & Openstack •MySQL Replication: What’s New • MySQL Replication GTIDs •MySQL at Twitter • MySQL Workbench •MySQL at Facebook • And many more

Hands-on Labs BOFs

• High Performance Backups • MySQL Server • Securing MySQL • MySQL and GIS • Performance Schema • MySQL High Availability & Backup • And many more… • And many more…

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 70 MySQL Community Reception @ Oracle OpenWorld Mingle with the MySQL community and the MySQL team from Oracle for a fun and informative evening!

• Time: September 30 (Tue) @ 7pm • Jillian’s at Metreon 175 Fourth Street, San Francisco, CA At the corner of Howard and 4th st.; only 2-min walk from Moscone Center

Thank You!