Mysql Innodb Cluster Management and Troubleshooting Using Mysql Shell

MySQL InnoDB Cluster Management and troubleshooting using MySQL Shell

Miguel Araújo Senior Software Developer MySQL Middleware and Clients September 18, 2019

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events.

• MySQL Developer since 2011 • AdminAPI Tech Lead @ MySQL Shell team • MySQL InnoDB Cluster evangelist • Living in Portugal • http://about.me/miguel-araujo • @M1guelAraujo

1. Introduction to InnoDB cluster 2. Node Provisioning 3. Monitoring 4. Maintenance and Troubleshooting 5. Final thoughts and resources

“A single product – MySQL – with high availability and scaling features baked in; providing an integrated end-to-end solution that is easy to use.”

• MySQL Group Replication

• MySQL Shell

• MySQL Router

1 One Product: MySQL

• All components developed and integrated together

• Full stack testing

1 One Product: 2 Easy to use! MySQL

• All components • One client: MySQL Shell developed and integrated together • Easy packaging

• Full stack testing • Integrated Orchestration

• Homogeneous servers

1 One Product: 2 Easy to use! 3 Flexible and MySQL Modern

• All components • One client: MySQL Shell • SQL and noSQL together: developed and integrated X Protocol together • Easy packaging • Protocol Buffers • Full stack testing • Integrated Orchestration • Asynchronous API • Homogeneous servers • Developer friendly

! Completely Open Source --- GPL No license required to have High Availability!

• Interactive multi-language console interface that supports development and administration for the MySQL Server:

• Supports JavaScript, Python and SQL

• Interactive and batch operations (scripting)

• Supports Document and Relational models (X and MySQL protocols)

• Exposes full Development and Administration APIs:

• Scriptable “DevOps” APIS

• Unified interface for developers and DBAs

• Creation and Management of InnoDB clusters

• Available in both JavaScript and Python; and command-line

• Simple and straight-forward

• Flexible, powerful and secure!

// Configure servers for MySQL InnoDB cluster usage mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”)

// Connect to target instance mysqlsh-js> \c clusteradmin@mysql1:3306

// Configure servers for MySQL InnoDB cluster usage mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”)

// Connect to target instance mysqlsh-js> \c clusteradmin@mysql1:3306

// Create the cluster mysqlsh-js> var cluster = dba.createCluster(“CodeOne19”)

// Configure servers for MySQL InnoDB cluster usage mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”)

// Connect to target instance mysqlsh-js> \c clusteradmin@mysql1:3306

// Create the cluster mysqlsh-js> var cluster = dba.createCluster(“CodeOne19”)

// Add instances to the cluster mysqlsh-js> cluster.addInstance(“clusteradmin@mysql2:3306”)

• Transparent client routing between the application and the backend servers:

ü Application connection failover

• Native support for InnoDB clusters

• Stateless

• Provisioning a server is the process of configuring and getting it ready to be used in an InnoDB cluster.

• The server needs to have all clusters’ data to be able to join it

• That’s typically done by:

1. Performing a backup of one of the cluster members 2. Restoring the image in the new member that wants to join

• Distributed recovery is a process that runs when a new member is added to the group and has missing transactions

• A donor that can provide the missing data is automatically selected

• The catch-up happens transparently and the server is synchronized with the rest of the cluster

• Incremental recovery fetches only the missing data based on the GTIDs into

- The cluster must be able to provide all the missing data

• Relies on the classic Asynchronous Replication

• Also provides group transactional information

Some data is inserted in the DB

The data modifications are stored in the binlog.

And the transactions are delivered to every cluster member

Group Replication is eventually synchronous.

Guarantees that transactions are delivered to all members in the same order.

The binlogs keep track of all data modifications.

A new member wants to join the cluster

The missing data is fetched and the server synchronizes with the rest of the cluster

A donor is automatically selected to fetch the missing data from, using plain Asynchronous Replication

The instance successfully joins the cluster!

May take a considerable amount of time !

• But what if the cluster members lost all binary logs?

• Or even... Just some data is missing that is not available on any binary log ?

• What about space constraints?

Some binary logs were purged. And the instance has no data yet.

Failed attempt to join the cluster !

• Performs physical snapshots of databases

ü Really fast!

• Transfers the snapshots over the network to provision servers

ü Can be used remotely!

• Replaces all the server data

Distributed recovery = initial provisioning + online catch-up

• Seamless and automated solution for provisioning

• No external tools or scripting required

Common issues easily solved with clone:

• Very large datasets

• Incomplete binary log history

• Re-provision diverged instances

Some binary logs were purged. And the instance has no data yet.

Clone required !

A donor is automatically selected. And a full snapshot of its data is performed.

The instance successfully joins the cluster!

• Incremental recovery is possible if:

• At least 1 member can provide the joiner with the same set of transactions it has processed. • The joiner does not have errants transactions

• Incremental recovery appropriate if:

• No purged transactions • Joiner does not have an empty GTID set • GTID support and binary logs were always enabled

Incremental recovery safe and appropriate:

Clone-based recovery required:

• Built-in support for full instance provisioning!

• AdminAPI transparently:

ü Setups the cluster to be ready for clone usage (plugin setup, user grants, etc.)

ü Detects every possible scenario and guides the user accordingly to the best provisioning method

ü Still allowing manual intervention to choose the provisioning method

• Binlogs can now be purged frequently! ! • But (at least) one cluster member should keep them for 2 days (minimum) for recovery.

The AdminAPI was extended to provide monitoring features:

• For any type of recovery method: clone or incremental

• Real-time progress information

• Detailed stage status

• Asynchronous progress information from cluster.status()

ü Clone-based recovery real-time progress info

ü Clone-based recovery progress from cluster.status()

Provides general information about the cluster and each member status

Provides information of:

• GR Group name

• Group protocol version

• Cluster members UUIDs

• Cluster members roles and states

• Read-only settings

Provides information of:

• Transactions processed by connection and applier

Provides information of:

• Detailed stats about the replication machinery of each cluster member

• Built-in reports // displays the result one time mysqlsh-js> \show query

• User-defined reports // executes repeatedly with screen refresh mysqlsh-js> \watch query

// registers a user-defined-report mysqlsh-js> shell.registerReport()

• Built-in reports // displays the result one time mysqlsh-js> \show query

• User-defined reports // executes repeatedly with screen refresh mysqlsh-js> \watch query

! // registers a user-defined-report Monitor P_S to check which mysqlsh-js> shell.registerReport() secondary instance is slower

Monitoring a cluster can provide info/hints about:

• Faulty networks

• High latencies

• Server failures

• ...

Provides information of:

• The cluster global configuration options

• Each cluster member settings

expelTimeout memberWeight Avoid frequent member expels Next primary in line when a due to faulty network failover happens

exitStateAction autoRejoinTries Action when a server is expelled Rejoin attempts after an eviction from the cluster: abort, read-only, offline-mode

// Change the member-weight value of a member mysqlsh-js> cluster.setInstanceOption(“mysql2:3307”, “memberWeight”, 75)

// Change the exit-state-action of a member mysqlsh-js> cluster.setInstanceOption(“mysql3”, “exitStateAction”, “ABORT_SERVER”)

// Change the member-weight value of a member mysqlsh-js> cluster.setInstanceOption(“mysql2:3307”, “memberWeight”, 75)

// Change the exit-state-action of a member mysqlsh-js> cluster.setInstanceOption(“mysql3”, “exitStateAction”, “ABORT_SERVER”)

// Change the expel-timeout of the cluster mysqlsh-js> cluster.setOption(“expelTimeout”, 60)

• InnoDB cluster operates, by default, in single-primary mode:

• Apps and users write to the primary instance

• What guarantees do we have that: • We always read the most up-to-date data? ? • We do not read stale data of an evicted server?

• Dirty reads do not happen?

// Create a cluster with “BEFORE” consistency level // Ensures that reads always read the most up-to-data data mysqlsh-js> cluster = dba.createCluster(“CodeOne19”, {consistency: “BEFORE”})

// Change the consistency level at any time // AFTER ensures that there are no dirty reads mysqlsh-js> cluster.setOption(“consistency”, “AFTER”)

• The cluster’s primary instance can be changed, immediately:

• The topology-mode can be changed live:

// Change the topology-mode to multi-primary mysqlsh-js> cluster.switchToMultiPrimaryMode()

// Change back to single-primary-mode mysqlsh-js> cluster.switchToSinglePrimaryMode()

• Internal recovery accounts have randomly secure generated passwords

• But security policies may enforce passwords lifetime

ü Support to reset a cluster’s recovery accounts credentials added

// Reset the recovery accounts credentials mysqlsh-js> cluster.resetRecoveryAccountsPassword()

Available in 8.0.18

Anything“ that can go wrong will go wrong.”

• Shell’s interactive behavior provides the users with valuable information, by default.

• But sometimes more information is needed for troubleshooting...

$ mysqlsh –-log-level=8

Logging levels:

• 1: none

• 2: internal

• 3: error

• 4: warning

• 5: info

• 6, 7, 8: debug

$ mysqlsh –-verbose=4

Verbose output level:

• [1, 4]

• Enables diagnostic messages output

$ mysqlsh –-dba-log-sql=2

Log AdminAPI SQL:

• 0: no logging

• 1: log all queries except SELECT and SHOW

• 2: log all queries

Available in 8.0.18

• InnoDB cluster is THE built-in HA solution for MySQL • MySQL Clone Plugin pushed InnoDB cluster to a whole new level of usability! • InnoDB Cluster features built-in support for full instance provisioning

• MySQL Shell is the unified interface for developers and DBAs and the frontend manager for InnoDB cluster

• https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb- cluster-userguide.html (InnoDB cluster userguide) https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/ (Shell APIs JS Ref manual) https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/ (Shell APIs Python Ref manual)

• https://mysqlhighavailability.com (Blogging) • http://mysqlserverteam.com/ (Blogging) • https://mysqlcommunity.slack.com (Community Slack)

• MySQL Shell: The Best DBA Tool? How to Use MySQL Shell as a Framework for DBAs [DEV2056] • What’s New in MySQL Optimizer and Executor? [DEV2077] • MySQL Connector/J 8.0 Feature Review and How to Upgrade from Connector/J 5.1 [DEV6221] • MySQL 8.0 Enterprise Backup [DEV4110] • Group Replication: How I Stopped Worrying About Consistency and Focused on Building Apps [DEV3016]

Help us make the content even better. Please complete the session survey in the Mobile App.

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.