www.fromdual.com
MySQL Servers Working as a Team - Replication or Galera Cluster
Open Source Data Center Conference, April 26th - 28th, Berlin
Jörg Brühe
Senior Support Engineer, FromDual GmbH [email protected]
CC-BY-SA
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 1 / 42 FromDual GmbH www.fromdual.com
Support Consulting
remote-DBA
Training
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 2 / 42 About Me www.fromdual.com
● Development distributed SQL-DBMS Porting mainframe -> Unix, Interface to archiver tools (ADSM, NetWorker)
● MySQL Build Team Release builds incl. tests, packaging, scripts, ...
● DBA MySQL running a web platform (master-master-replication)
● Support-Engineer (FromDual) Support + Remote-DBA for MySQL / MariaDB / Percona both with and without Galera Cluster
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 3 / 42 Contents www.fromdual.com
MySQL Server: Architecture Binlog Replication Galera Cluster Comparison Examples / When (not) to Choose Which
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 4 / 42 General Remarks www.fromdual.com
● Concepts rather than details: ”the forest, not the trees“
● MySQL 5.6 (established GA version)
● Also valid for Percona and MariaDB
● Not applicable to ”embedded“ MySQL
● Not considered: NDB = ”MySQL Cluster“
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 5 / 42 www.fromdual.com
MySQL Server: Architecture Binlog Replication Galera Cluster Comparison Examples / When (not) to Choose Which
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 6 / 42 Client-Server-DBMS www.fromdual.com
App App App Client (application) local or remote
Socket, LAN or internet
Server is separate process, Server multi-threaded: 1 thread per user session
Disk Disk / SSD, local or SAN
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 7 / 42 Inside the Server www.fromdual.com
Application / Client
Thread Connection mysqld Cache Manager Parser Optimizer User Au- thenticationMySQL ist eine multi-Thread und NICHT eine multi-Prozess Access Control CommandApplikation! Logging Table Open Dispatcher Cache (.frm, fh) Table Manager Query Query Cache Table Definition Cache Module Cache (tbl def.)
Handler Interface
MyISAM InnoDB Memory NDB PBXT Aria XtraDB Federated-X ...
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 8 / 42 www.fromdual.com
MySQL Server: Architecture Binlog Replication Galera Cluster Comparison Examples / When (not) to Choose Which
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 9 / 42 Layers + Binlog www.fromdual.com 1 2
SQL layer: d d
a a - Parser e e r r
. - Optimizer . h h . t t
- Privileges r r
e e - Query Cache s s
U U - … Handler Interface Binlog: File layer:
All data M B - Table Handler A D . . S o
and schema . I - InnoDB: n y n
I - Row Access
changes M - Row Locks - Recovery - ...
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 10 / 42 Binlog www.fromdual.com
● All data changes executed
● All schema changes executed
● Timestamps
● Essential for Point-in-Time-Recovery ”PITR“
● Independent of table handler
● Formats ”statement“, ”row“, and ”mixed“
● Segments of configurable size
● Numbered sequentially
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 11 / 42 www.fromdual.com
MySQL Server: Architecture Binlog Replication Galera Cluster Comparison Examples / When (not) to Choose Which
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 12 / 42 MySQL Replication www.fromdual.com
● Applications communicate with ”Master“
● ”Master“ logs all changes
● ”Slave“ has identical initial state
● Slave fetches all changes from master and applies them locally
● Replication is running asynchronous
● Slave stops replication on difference
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 13 / 42 Slave fetches binlog www.fromdual.com
1 2 3 1 2
d d d d d d d a a a a a a a e e r r e e e e e r r r r r h h . . t t . . h h h h h
. . t t t t t
L O r r r r r I Q e e e e e s s s s s S U U U U U
Binlog Relay Binlog M M B B A A D Log D . . . . S o S o . . I I n n y y n n I I M M
Slave: Master: “log-bin = FILE”, or else no binlog “log-bin = FILE”, or else no binlog “log_slave_updates = 1” for forwarding (no master function)
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 14 / 42 Typical Usage www.fromdual.com
● ”High Availability“
● Geographic redundancy
● Support higher read load (= ”read scale-out“)
● Read-only instance(s) e.g. for backup or reports
● Intentional delay is possible
● Filtering (by DB or table) is possible
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 15 / 42 Replication Cascade www.fromdual.com 3 1 2 1 2 2 1
d d d d d d d d d d d a a a a a a a a a a a e e e e e e e e e e e r r r r r r r r r r r . . . h h h h h h h . . . h h h h t t t t t t t . . . t t t t
r r r r r r r L L O O e e e e e e e I I s s s s s Q s s Q S S U U U U U U U B M B B M M A D A A D D ...... S o S S o o . I . . I I n n n y y y n n n I M I I M M
● Recommended: ”read-only = 1“ on slave ”log_slave_updates = 1“
● Multiple slaves per master are possible
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 16 / 42 Entries in Binlog www.fromdual.com
Original:
● Identification by file name and position
● Replication: ”change master to ...“ specifying host, port, user, password, file, position
● See also: ”mysqldump masterdata“
From MySQL 5.6 also:
● GTID = ”Global Transaction ID“
● Replication: ”change master to ...“ specifying host, port, user, password, ”auto_position = 1“
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 17 / 42 Master-Master-Replication www.fromdual.com
1 2 3 1 2
d d d d d d d d d a a a a a a a a a e e e e r r r r e e e e e r r r r r h h h h . . t t t t . . h h h h h
. t t . t t t
L L O O r r r r r I I Q Q e e e e e s s s s s S S U U U U U
Binlog Relay Binlog Relay M M B B A A D Log D
. . Log . . S S o o . . I I n n y y n n I I M M
● Overlapping changes are fatal! MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 18 / 42 Notes about Replication www.fromdual.com
● Master-Master is controversial, be careful!
● Replication increases read throughput, but not/barely write throughput
● Replication causes file IO und network load
● Format ”row“ is more efficient, but less readable
● Multi-threaded replication since MySQL 5.6, multi-master (”multi-source“) coming in MySQL 5.7
● Big installation: booking.com
● Recommended: datacharmer.blogspot.de (Giuseppe Maxia, August 2015)
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 19 / 42 www.fromdual.com
MySQL Server: Architecture Binlog Replication Galera Cluster Comparison Examples / When (not) to Choose Which
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 20 / 42 Replication Weaknesses www.fromdual.com
● Asynchronous
● Asymmetrical
● Only one write node
● Parallel writes may cause breakage
● HA needs failover after node crash
● Each node is SPOF for its slaves, breakdown requires structure change
● Dynamic changes are complicated
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 21 / 42 Better Alternative www.fromdual.com
● Synchronous transfer
● Symmetrical cluster
● Write accesses on all nodes
● Distributed conflict analysis and handling
● HA by continuity after node outage
● Dynamic entry / exit of nodes supported
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 22 / 42 Galera Cluster www.fromdual.com
App App App
Inclusiding outage detection Load balancing (LB) and redirection for HA
Node 1 Node 2 Node 3
wsrep wsrep wsrep “Working Set Replication” Galera replication Dedicated network preferred
= = Locale disks, each holding all data “shared nothing” architecture
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 23 / 42 Galera Properties (1) www.fromdual.com
+ Based on InnoDB (due to transactions and rollback) + Also transfers user definitions, privileges, ... + Quasi-synchronous transfer on commit, check for conflicts, efficient + Symmetrical, HA without server failover, quorum + No loss of transactions + Brings read scale-out, also some write increase + Dynamical entry / exit possible, synchronisation is automated
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 24 / 42 Order of Events www.fromdual.com
Graph by Vadim Tkachenko (Percona): http://www.mysqlperformanceblog.com/2012/01/19/ percona-xtradb-cluster-feature-2-multi-master-replication/ MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 25 / 42 Galera Properties (2) www.fromdual.com
- MySQL sources need patching (Codership offers binaries, ditto MariaDB and Percona) - Beware of hot spots (rows) - Conflict detection is late, full rollback (Check postponed till commit) - Minimum size is 3 nodes - Synchronisation time for large DB (mysqldump -> xtrabackup or rsync)
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 26 / 42 Certifiation at Commit www.fromdual.com
http://galeracluster.com/documentation-webpages/certificationbasedreplication.html
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 27 / 42 www.fromdual.com
MySQL Server: Architecture Binlog Replication Galera Cluster Comparison Examples / When (not) to Choose Which
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 28 / 42 Team of MySQL Servers www.fromdual.com
● Alternatives: Replication or Galera Cluster
● Redundancy of machine and storage
● HA
● Scale-out, esp. for read load
● Instances for reports, analysis, backup
● Data available locally (branch offices, ...)
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 29 / 42 Comparison (1) www.fromdual.com
Replication Galera Standard Add-on product All handlers InnoDB only Upwards compatible Same versions Minimum 2 nodes Minimum 3 nodes HA by failover HA without changes Communication: Hierarchical, chain Symmetrical, parallel Asynchronous Quasi-synchronous Delay is configurable Immediate Filtering is configurable Complete
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 30 / 42 Comparison (2) www.fromdual.com
Replication Galera Read scale-Out Read scale-Out Write unchanged Write increased 1 Master: 1* write 1* write Local conflict : Error on statement Error on statement n Master: n* write n* write Distributed conflict: Replication breakdown Rollback on commit
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 31 / 42 Comparison (3) www.fromdual.com
Replication Galera Short interruption: Replication resumes IST (incremental transfer) Long interruption: Replication resumes SST (full transfer) Structure change: Manual / separate Tool Automatic / dynamic Initial setup: Snapshot, Full transfer, master remains available donor may be blocked
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 32 / 42 CAP Theorem www.fromdual.com
”For a distributed computer system, it is impossible to simultaneously provide all three of the following guarantees:
● C = Consistency (identical data throughout)
● A = Availability (system is operational)
● P = Partition Tolerance (network outage)“ Eric Brewer, 1998 ff. https://en.wikipedia.org/wiki/CAP_theorem
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 33 / 42 Prospect: MySQL 5.7 www.fromdual.com
● MySQL 5.7 is GA (5.7.9, 2015-Oct-21)
● Replication like in MySQL 5.6, added: multi-source replication (one slave reading from several masters)
● Codership is working on adding Galera Cluster to MySQL 5.7
● Oracle is working on ”Group replication“, currently available as ”labs release“ (= ”not fit for production“)
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 34 / 42 www.fromdual.com
MySQL Server: Architecture Binlog Replication Galera Cluster Comparison Examples / When (not) to Choose Which
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 35 / 42 Communication Outage (1) www.fromdual.com
Galera Cluster:
● Isolated node has no quorum => will not serve applications
● Quorum is at risk!
● Active nodes write ”gcache“ to files, storage period?
● Switch to SST threatens
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 36 / 42 Communication Outage (2) www.fromdual.com
Replication:
● Master writes log segments to files
● IO-Thread asks to read from binlog position / GTID, retries periodical until successful
● Avoid ”purge log“!
Replication is more tolerant than Galera Cluster !
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 37 / 42 Global Production www.fromdual.com
Solution: Replication with filtering
Requirement: Head office (D) and factories (BR, CN, ...) with selective transfer
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 38 / 42 Parallel Writes + Conflict www.fromdual.com
Galera:
● Retry of autocommit statements configurable
● Transaction conflict causes rollback => Application repeats complete transaction
Replication:
● Slave detects conflict, no contact to application => Replication stops
Replication needs admin action on conflict !
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 39 / 42 Hot Spot www.fromdual.com
● Replication: frequent aborts
● Galera: frequent rollbacks
=> Agree on a single write node !
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 40 / 42 High Availability www.fromdual.com
Replication:
● Failover manual (reaction time) or automated (correct?)
● Slave lag, selection of new master
Galera:
● Symmetrical, no change of roles
● Virtually synchronous replication (no lag)
=> Advantage Galera
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 41 / 42 Q & A www.fromdual.com
Questions ?
Discussion?
● FromDual provides neutral and independent: ● Consulting ● Remote-DBA ● Support for MySQL, Galera, Percona Server and MariaDB ● Training
www.fromdual.com/presentations
MySQL Teamwork: Replication or Galera Cluster, [email protected], 2016 April, CC-BY-SA 42 / 42