Mysql Cluster • Mysql, Heartbeat Plus DRBD Architectures • Setup and Configuration • Managing Failover • High Availability Services from Mysql •Q & A

High Availability Architectures with MySQL and DRBD

Jimmy Guerrero Senior Product Manager jguerrero@mysql.com

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 1 Agenda • Introduction to MySQL • Comparing MySQL HA Architectures – MySQL Replication – MySQL Replication plus Linux Heartbeat – MySQL Cluster • MySQL, Heartbeat plus DRBD Architectures • Setup and Configuration • Managing Failover • High Availability Services from MySQL •Q & A

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 2 Overview of MySQL AB • Founded in 1995 • Operations in 24 countries • World’s Most Popular Open Source Database – Over 10 million installations – 50,000 downloads each day • Reduces TCO of database management – Reliable, high performance, easy to use • Bundled by over 100 software & hardware companies • Partners HP, Dell, Novell, Red Hat, Unisys, Accenture, LogicaCMG, T-Systems and IPL

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 3 Established & Emerging Companies High Volume Websites – Web 2.0 – Dynamic content – eCommerce – Look to Book – Session Management – Gaming & Entertainment craigslist –Scale Out Enterprise – Data Warehousing – High-Volume OLTP –Scale Out Embedded – Software & Hardware

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 4 MySQL, Heartbeat and DRBD Cluster • High availability solution built on open source software – MySQL: Database – Linux Heartbeat: Virtual IP management – DRBD: Synchronous block level replication and fast data resynchronization after node failure • Low cost – Many popular Linux distributions supported – Can be built on COTS hardware components – No migration hassles • Supports MySQL “scale out” for reads • Supports transactional storage engines – InnoDB –Falcon

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 5 What is DRBD? • Block device which leverages synchronous replication – Open source Linux kernel module – Bundled by several popular Linux distributions – Project maintained by LinBit • Core functionality – The entire block device is mirrored via a dedicated network – Similar to RAID-1 over Ethernet – DRBD operates at disk I/O layer – Transparent to MySQL and the underlying file system – MySQL fails over to passive DRBD server via "virtual IP management" • Typically accomplished with the Linux Heartbeat package. – MySQL transactional storage engine independent • InnoDB, Falcon

http://www.drbd.org/ http://www.linux-ha.org/

Clustering & Geographical Redundancy MySQL Clustering Cluster Technologies

MySQL + DRBD Replication MySQL + SAN Technologies

Well-Managed MySQL Replication

Unmanaged

Small ISPs & Telco Data Banking Military Business Mainstream Centers Medical Business Defense 9 9 . 9 9 9 % 35 days 4 days 8 hours 50 mins 5 mins

Asynchronous Replication

update customer

Synchronous Replication

update customer

• Asynchronous Requirements Support

– Statement Availability – Row-based (5.1) Automated IP Fail Over No – Mixed (5.1) Automated DB Fail Over No • Read vs. Write Typical Fail Over Time Varies – Response times – Load balancing Resynchronization of Data No –Scale Out Geographic Redundancy Yes • Administration Scalability – Backups Built-in Load Balancing Scale-Out

– Upgrades Read Intensive Yes

– Testing Write Intensive No

• Geographic Redundancy Dozens for # of Nodes Reads

Master > Slave Master > Slaves

Masters > Slave (Multi-Source) Master > Slave > Slaves

Circular (Multi-Master) Master < > Master (Multi-Master)

• Write to one master • Read from many slaves • Perfect for read intensive apps Load Balancer

Writes & Reads Reads Reads

MySQL Replication

IP Management Active Passive MySQL MySQL Server Server

Asynchronous MySQL Replication

Heartbeat implements serial, UDP, and PPP/UDP heartbeats together with IP address takeover, this solves MySQL Replication’s “fail over” problem, but, data resynchronization is not automatic and can be complex.

• In-Memory Database Requirements Support – Disk Data (5.1) Availability • Shared-nothing storage Automated IP Fail Over No • Synchronous replication Automated DB Fail Over Yes • Fast-automatic data fail over Typical Fail Over Time < 3 Secs

• High performance Resynchronization of Data Yes

– NDB API Yes Geographic Redundancy • No special hardware (w/ Replication) • Best for primary key lookups Scalability • Transactional Load Balancing Scale-Out • Online backup Read Intensive Yes (Look ups) • No automated IP failover Write Intensive Yes 5.0: 64 # of Nodes 5.1: 255

Clients

NDB API Application Nodes (NDB API and/or MySQL Server)

Management Nodes (Typically 2) Data Nodes

NDB Storage Engine

PK col1 col2 col3 Partitions - Automatic or User-Defined 1 data data data - Horizontal 2 data data data - MD5 hash of Primary Key 3 data data data 4 data data data Two Data Nodes * Complete copy of database on each node

Two Replicas * Two copies of each partition Node Group One Node Group P1 - Primary P1 - Replica * One set of identical Data Nodes P2 - Replica P2 - Primary

• Distributed storage Requirements Support • Similar to network RAID 1 Availability

• Synchronous replication Automated IP Fail Over Yes

• No special networking Automated DB Fail Over Yes components Typical Fail Over Time < 30 secs • Excellent performance Resynchronization of Data Yes – Blocks vs. rows of data MySQL Geographic Redundancy • Manages inconsistencies of Replication

data during a failure Scalability • Hides the complexity of many Built-in Load Balancing No recovery actions Read Intensive Yes • Heartbeat manages fail over and virtual IPs Write Intensive Possible # of Nodes 1 Active

IP Management Active Passive DRBD DRBD Server Server

Synchronous Block Replication

IP Management Active Passive Server Server

• Data is separated into distinct, autonomous data silos or “shards” within DRBD Clusters • Applications access “shards” directly or through load balancing • Used in conjunction with MySQL Replication, can increase read & write scalability • Potential to also increase availability depending on partitioning scheme • Partitioning on different levels – Application – MySQL data partitioning in 5.1 – Reads vs Writes

Load Balancer/SQL Relay

Shard 1 Shard 2

IP Management IP Management

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 20 Block Devices • A device node is a type of special file – Used on many Unix-like operating systems • Facilitates transparent communication – Between user space applications and hardware • Correspond to resources allocated by the operating system kernel • Accessed using standard system calls and treated like regular files • Used to correspond to devices – Data is transmitted in the form of blocks. • Make use of buffered input and output routines

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 21 DRBD Basics • Each device has one of two states – Primary – Secondary • Application runs and accesses the device on the primary node • Writes are sent to the local lower level block device and to the node with the device in secondary state • Secondary device writes the data to its lower level block device • Reads are always carried out locally – Key advantage over SAN/NAS • If the primary node fails – Heartbeat switches the secondary device into a primary state and starts the application (MySQL)

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 22 DRBD Basics • When the failed node comes up – Becomes a secondary node – Must synchronize its data to the primary – Synching happens in the background without service interruption – Only the parts of the device which have changed since the failure will be resynchronized. • Total resynchronization in just several minutes – Regardless of device size – Even after the hard crash of an active node – Resynch rates may be configured per device • Device sizes currently limited to 4 TB

System Requirement Support

Hosts Minimum 2 (More for scale-out or partitioning)

COTS or SMP - 32 & 64-bit Hardware Multiple NICs for redundancy

Varies (More is generally better) Memory (32 KB per 1 GB block)

Storage SCSI, RAID 0+1 for I/O Performance

100 Mbps or better Network (Gigabit recommended, others supported)

Operating System Linux (Redhat, Suse, CentOS, Fedora, Debian)

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 24 Setup & Configuration • Multiple NICs will increase redundancy • MySQL traffic switched over public LAN • DRBD & Heartbeat over private LAN

IP Management

Active Passive DRBD DRBD Server Server

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 25 Setup & Configuration - DRBD • Additional prerequisite packages may be needed – Depending on kernel version – When building from source – Or using RPMs • Pre-built x86 and x86_64 packages for specific kernel versions available with a support subscription –SuSE9 & 10 – Redhat AS 4 • If kernel is upgraded, DRBD must be as well • Current DRBD version is 8 (Jan 24, 2007) – Better, tuneable recovery – Ability to run DRBD on top of LVM – Performance optimization (Jumbo frames)

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 26 Setup & Configuration - DRBD • Install DRBD package • Edit and copy configuration file (drbd.conf) to nodes • Choose primary node • Synchronize the underlying devices • Device is now ready – Create a file system if one does not exist • Mount DRBD on primary • Heartbeat handles – Changing of DRBD primary or secondary status – Mounting and unmounting volumes

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 27 Setup & Configuration - MySQL • Ensure all MySQL files are installed on the DRBD volume • Create ‘mysql’ group and user • Create MySQL directory • Install MySQL • Shutdown MySQL • Unmount the DRBD volume

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 28 Setup & Configuration - Heartbeat • Install Heartbeat V2 package – http://www.linux-ha.org/ – Ships with many Linux distributions • Configuration – ha.cf: describes the participating machines and their configuration – haresources: specifies virtual IP and managed services – ipfiles: detects when one of their network links has become unusable, and compensate accordingly – authkeys: three types - crc, md5, and sha1 – resource.d/mysql: start and stop script for MySQL

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 29 Initialization & Next Steps • Start DRBD • Start Heartbeat – Node becomes DRBD primary – DRBD volume is mounted –MySQL starts – IP address taken over • Next steps – Fail over testing –Benchmarking – Stress testing – Create monitoring framework – Attach MySQL Replication slaves for read scalability

IP Management

Passive X Active DRBD DRBD Server Server X X X

IP Management Passive Active DRBD DRBD Server Server

Resynchronization

Active Server is still able to process transactions during a resynchronization operation Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 32 Comparing MySQL HA Solutions MySQL MySQL, MySQL Requirements Replication + Heartbeat + MySQL Cluster Replication Heartbeat DRBD

Availability

Automated IP Fail Over No Yes Yes No

Automated DB Fail Over No No Yes Yes

Typical Fail Over Time Varies Varies < 30 secs < 3 secs

Auto Resynch of Data No No Yes Yes

Geographic Redundancy Yes Yes MySQL Replication MySQL Replication

Scalability MySQL Built-in Load Balancing MySQL Replication MySQL Replication Yes Replication

Read Intensive Yes Yes MySQL Replication Yes

Write Intensive No No Yes Yes

# of Nodes per Cluster Master/Slave(s) Master/Slave(s) Active/Passive 255

• MySQL High Availability Jumpstart – Analyze your current database infrastructure – Analyze your Scale-out and High-Availability database requirements – Identify potential issues and limitations – Install, Configure and Fine-Tune MySQL – Configure MySQL or MySQL Cluster to work with HA Technologies (e.g. DRBD, SAN, Shared SCSI, Heartbeat, etc), as required – Secure your database environment against failures and downtime – Benchmark and load test against sample queries

http://www.mysql.com/consulting/packaged/scaleout.html

Component Detail

Network Foundry Networks FastIron

Load Balancing ServerIron

Storage NetApp FAS 3020HA w/ SATA disks

Application 35-40 Dell 1955 Blades

Operating CentOS 4.4 System

Database MySQL 5.0.36

Replication 1 read-only slave Schema Details Woodstone Servers Alive, Ganglia & Monitoring Splunk 63 Tables

Backup Automysqlbackup Largest Table: 36,000,000+ rows

DB Size: 15 GB

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database 35 Summary • High availability solution built on open source software – MySQL: Database – Linux Heartbeat: Virtual IP management – DRBD: Synchronous block level replication and fast data resynchronization after node failure • Low cost – Many popular Linux distributions supported – Can be built on COTS hardware components – No migration hassles • Supports MySQL “scale out” for reads • Supports transactional storage engines – InnoDB –Falcon

Cluster – Replication – Management

Performance Tuning – Security

Administration - DRBD

Over 100 Sessions and Tutorials!

Thank You!

Jimmy Guerrero Senior Product Manager [email protected]

White Papers - Case Studies - Press Releases - News - Events - Live Webinars - Forums