Upgrading to MySQL 5.6 @ Scale

14 April 11:30AM - 12:20PM Ballroom E

#PerconaLive 2015

1

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Who am I?

• Jonah Berquist @hashtagjonah • Senior Database Administrator at Twitter

2

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Who am I?

• Tom Krouper @CaptainEyesight • Former Staff DBA at Twitter • MySQL DBA ~ 10 years

3

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Agenda

• Why Upgrade? • Infrastructure • Upgrade Process • Issues & Bugs • Status

4

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Why Upgrade?

• Online DDL • Compression • GTID • Parallel Replication • Performance Improvements • WebscaleSQL • General improvements

5

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale 6

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Infrastructure Roost Cluster: • Clusters db.roost.master • Roles db.roost.slave.app db.roost.slave.analytics • Shards db.roost.standby db.roost.hub • Replication • Gizzard TBird Cluster: db.tbird.gizzard:000 db.tbird.gizzard:001

7

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Replication

8

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Gizzard

000

001

002

003

004

00n 9

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale 10

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process

• Initial Testing/Benchmarking

11

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process

• Bake in process • Read Replicas • Gizzard Shard • Development Cluster

12

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process

http://ceilingcat.ninja • Tooling: Ansible

13

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Ansible

• http://www.ansible.com • Agentless (uses SSH)

• Upgrading multiple servers at once • Skip already upgraded boxes • Rules in place to deal with different roles • Verified upgrades

14

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process

• Replication Upgrades

15

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Replication

16

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process

• Gizzard Upgrades

17

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Gizzard

000

001

002

003

004

00n 18

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale 19

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• Replication breakage on 5.6.21 (Bug #19704825) • temporary tables were released too early

20

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• Parallel Replication • multi-threaded slaves w/large events

21

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• explicit_defaults_for_timestamp

22

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• Query regression

23

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• Buffer pool dump & restore (Twitter’s vs. 5.6)

24

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• Backups done on 5.5 using LVM snapshots

25

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• timestamp/datetime format change

26

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs

• 5.0 client errors on 5.6 server • SET OPTION SQL_QUOTE_SHOW_CREATE=1;

27

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale 28

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Status

• ~80% of the hosts upgraded • mostly replicated clusters left to upgrade

29

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Status

• Using parallel replication and GTID on new shared cluster. • Still need to upgrade GTID on existing cluster in production • GTID upgrade done in test environment • Webscale Patches • Heartbeat issues

30

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale Status

• Roll out compression

31

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale 32

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale #ThankYou

Tom Krouper Jonah Berquist @CaptainEyesight @hashtagjonah

MySQL at Twitter - 2015 15 April 4:30PM - 5:20PM Ballroom G

#Questions? 33

@TwitterDBA | Upgrading to MySQL 5.6 @ Scale