MySQL InnoDB Cluster Management and troubleshooting using MySQL Shell Miguel Araújo Senior Software Developer MySQL Middleware and Clients September 18, 2019 Copyright © 2019 Oracle and/or its affiliates. Safe Harbor The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Copyright © 2019 Oracle and/or its affiliates. $ whoami Copyright © 2019 Oracle and/or its affiliates. Miguel Araújo • MySQL Developer since 2011 • AdminAPI Tech Lead @ MySQL Shell team • MySQL InnoDB Cluster evangelist • Living in Portugal • http://about.me/miguel-araujo • @M1guelAraujo Copyright © 2019 Oracle and/or its affiliates. What’s Ahead 1. Introduction to InnoDB cluster 2. Node Provisioning 3. Monitoring 4. Maintenance and Troubleshooting 5. Final thoughts and resources Copyright © 2019 Oracle and/or its affiliates. Introduction to InnoDB Cluster Copyright © 2019 Oracle and/or its affiliates. InnoDB cluster “A single product – MySQL – with high availability and scaling features baked in; providing an integrated end-to-end solution that is easy to use.” • MySQL Group Replication • MySQL Shell • MySQL Router Copyright © 2019 Oracle and/or its affiliates. MySQL InnoDB Cluster Goals 1 One Product: MySQL • All components developed and integrated together • Full stack testing Copyright © 2019 Oracle and/or its affiliates. MySQL InnoDB Cluster Goals 1 One Product: 2 Easy to use! MySQL • All components • One client: MySQL Shell developed and integrated together • Easy packaging • Full stack testing • Integrated Orchestration • Homogeneous servers Copyright © 2019 Oracle and/or its affiliates. MySQL InnoDB Cluster Goals 1 One Product: 2 Easy to use! 3 Flexible and MySQL Modern • All components • One client: MySQL Shell • SQL and noSQL together: developed and integrated X Protocol together • Easy packaging • Protocol Buffers • Full stack testing • Integrated Orchestration • Asynchronous API • Homogeneous servers • Developer friendly Copyright © 2019 Oracle and/or its affiliates. MySQL InnoDB Cluster Goals ! Completely Open Source --- GPL No license required to have High Availability! Copyright © 2019 Oracle and/or its affiliates. MySQL Shell • Interactive multi-language console interface that supports development and administration for the MySQL Server: • Supports JavaScript, Python and SQL • Interactive and batch operations (scripting) • Supports Document and Relational models (X and MySQL protocols) Copyright © 2019 Oracle and/or its affiliates. MySQL Shell • Exposes full Development and Administration APIs: • Scriptable “DevOps” APIS • Unified interface for developers and DBAs Copyright © 2019 Oracle and/or its affiliates. MySQL AdminAPI • Creation and Management of InnoDB clusters • Available in both JavaScript and Python; and command-line • Simple and straight-forward • Flexible, powerful and secure! Copyright © 2019 Oracle and/or its affiliates. Creating a Cluster // Configure servers for MySQL InnoDB cluster usage mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”) Copyright © 2019 Oracle and/or its affiliates. Creating a Cluster // Configure servers for MySQL InnoDB cluster usage mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”) // Connect to target instance mysqlsh-js> \c clusteradmin@mysql1:3306 Copyright © 2019 Oracle and/or its affiliates. Creating a Cluster // Configure servers for MySQL InnoDB cluster usage mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”) // Connect to target instance mysqlsh-js> \c clusteradmin@mysql1:3306 // Create the cluster mysqlsh-js> var cluster = dba.createCluster(“CodeOne19”) Copyright © 2019 Oracle and/or its affiliates. Creating a Cluster // Configure servers for MySQL InnoDB cluster usage mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”) // Connect to target instance mysqlsh-js> \c clusteradmin@mysql1:3306 // Create the cluster mysqlsh-js> var cluster = dba.createCluster(“CodeOne19”) // Add instances to the cluster mysqlsh-js> cluster.addInstance(“clusteradmin@mysql2:3306”) Copyright © 2019 Oracle and/or its affiliates. MySQL Router • Transparent client routing between the application and the backend servers: ü Application connection failover • Native support for InnoDB clusters • Stateless Copyright © 2019 Oracle and/or its affiliates. Node Provisioning Copyright © 2019 Oracle and/or its affiliates. Provisioning • Provisioning a server is the process of configuring and getting it ready to be used in an InnoDB cluster. • The server needs to have all clusters’ data to be able to join it • That’s typically done by: 1. Performing a backup of one of the cluster members 2. Restoring the image in the new member that wants to join Copyright © 2019 Oracle and/or its affiliates. Distributed Recovery • Distributed recovery is a process that runs when a new member is added to the group and has missing transactions • A donor that can provide the missing data is automatically selected • The catch-up happens transparently and the server is synchronized with the rest of the cluster Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision • Incremental recovery fetches only the missing data based on the GTIDs into - The cluster must be able to provide all the missing data • Relies on the classic Asynchronous Replication • Also provides group transactional information Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Some data is inserted in the DB Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision The data modifications are stored in the binlog. And the transactions are delivered to every cluster member Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Group Replication is eventually synchronous. Guarantees that transactions are delivered to all members in the same order. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision The binlogs keep track of all data modifications. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A new member wants to join the cluster Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A new member wants to join the cluster The missing data is fetched and the server synchronizes with the rest of the cluster Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A donor is automatically selected to fetch the missing data from, using plain Asynchronous Replication Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A donor is automatically selected to fetch the missing data from, using plain Asynchronous Replication Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A donor is automatically selected to fetch the missing data from, using plain Asynchronous Replication Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision The instance successfully joins the cluster! Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision May take a considerable amount of time ! Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision • But what if the cluster members lost all binary logs? • Or even... Just some data is missing that is not available on any binary log ? • What about space constraints? Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Some binary logs were purged. And the instance has no data yet. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Some binary logs were purged. And the instance has no data yet. Failed attempt to join the cluster ! Copyright © 2019 Oracle and/or its affiliates. MySQL Clone Plugin Available since 8.0.17 ! • Performs physical snapshots of databases ü Really fast! • Transfers the snapshots over the network to provision servers ü Can be used remotely! • Replaces all the server data Copyright © 2019 Oracle and/or its affiliates. Clone to Provision Distributed recovery = initial provisioning + online catch-up • Seamless and automated solution for provisioning • No external tools or scripting required Copyright © 2019 Oracle and/or its affiliates. Clone to Provision Common issues easily solved with clone: • Very large datasets • Incomplete binary log history • Re-provision diverged instances Copyright © 2019 Oracle and/or its affiliates. Clone to Provision Some binary logs were purged. And the