Upgrading to MySQL 5.6 @ Scale 14 April 11:30AM - 12:20PM Ballroom E #PerconaLive 2015 1 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Who am I? • Jonah Berquist @hashtagjonah • Senior Database Administrator at Twitter 2 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Who am I? • Tom Krouper @CaptainEyesight • Former Staff DBA at Twitter • MySQL DBA ~ 10 years 3 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Agenda • Why Upgrade? • Infrastructure • Upgrade Process • Issues & Bugs • Status 4 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Why Upgrade? • Online DDL • Compression • GTID • Parallel Replication • Performance Improvements • WebscaleSQL • General improvements 5 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale 6 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Infrastructure Roost Cluster: • Clusters db.roost.master • Roles db.roost.slave.app db.roost.slave.analytics • Shards db.roost.standby db.roost.hub • Replication • Gizzard TBird Cluster: db.tbird.gizzard:000 db.tbird.gizzard:001 7 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Replication 8 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Gizzard 000 001 002 003 004 00n 9 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale 10 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process • Initial Testing/Benchmarking 11 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process • Bake in process • Read Replicas • Gizzard Shard • Development Cluster 12 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process http://ceilingcat.ninja • Tooling: Ansible 13 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Ansible • http://www.ansible.com • Agentless (uses SSH) • Upgrading multiple servers at once • Skip already upgraded boxes • Rules in place to deal with different roles • Verified upgrades 14 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process • Replication Upgrades 15 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Replication 16 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Upgrade Process • Gizzard Upgrades 17 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale #Gizzard 000 001 002 003 004 00n 18 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale 19 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • Replication breakage on 5.6.21 (Bug #19704825) • temporary tables were released too early 20 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • Parallel Replication • multi-threaded slaves w/large events 21 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • explicit_defaults_for_timestamp 22 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • Query regression 23 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • Buffer pool dump & restore (Twitter’s vs. 5.6) 24 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • Backups done on 5.5 using LVM snapshots 25 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • timestamp/datetime format change 26 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Issues & Bugs • 5.0 client errors on 5.6 server • SET OPTION SQL_QUOTE_SHOW_CREATE=1; 27 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale 28 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Status • ~80% of the hosts upgraded • mostly replicated clusters left to upgrade 29 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Status • Using parallel replication and GTID on new shared cluster. • Still need to upgrade GTID on existing cluster in production • GTID upgrade done in test environment • Webscale Patches • Heartbeat issues 30 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale Status • Roll out compression 31 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale 32 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale #ThankYou Tom Krouper Jonah Berquist @CaptainEyesight @hashtagjonah MySQL at Twitter - 2015 15 April 4:30PM - 5:20PM Ballroom G #Questions? 33 @TwitterDBA | Upgrading to MySQL 5.6 @ Scale.