High Performance MySQL Other Microsoft .NET resources from O’Reilly Related titles Managing and Using MySQL PHP Cookbook™ MySQL Cookbook™ Practical PostgreSQL MySQL Pocket Reference Programming PHP MySQL Reference Manual SQL Tuning Learning PHP Web Database Applications PHP 5 Essentials with PHP and MySQL .NET Books dotnet.oreilly.com is a complete catalog of O’Reilly’s books on Resource Center .NET and related technologies, including sample chapters and code examples. ONDotnet.com provides independent coverage of fundamental, interoperable, and emerging Microsoft .NET programming and web services technologies. Conferences O’Reilly Media bring diverse innovators together to nurture the ideas that spark revolutionary industries. We specialize in docu- menting the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches. Visit con- ferences.oreilly.com for our upcoming events. Safari Bookshelf (safari.oreilly.com) is the premier online refer- ence library for programmers and IT professionals. Conduct searches across more than 1,000 books. Subscribers can zero in on answers to time-critical questions in a matter of seconds. Read the books on your Bookshelf from cover to cover or sim- ply flip to the page you need. Try it today for free. SECOND EDITION High Performance MySQL Baron Schwartz, Peter Zaitsev, Vadim Tkachenko, Jeremy D. Zawodny, Arjen Lentz, and Derek J. Balling Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo High Performance MySQL, Second Edition by Baron Schwartz, Peter Zaitsev, Vadim Tkachenko, Jeremy D. Zawodny, Arjen Lentz, and Derek J. Balling Copyright © 2008 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (safari.oreilly.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or [email protected]. Editor: Andy Oram Indexer: Angela Howard Production Editor: Loranah Dimant Cover Designer: Karen Montgomery Copyeditor: Rachel Wheeler Interior Designer: David Futato Proofreader: Loranah Dimant Illustrators: Jessamyn Read Printing History: April 2004: First Edition. June 2008: Second Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. High Performance MySQL, the image of a sparrow hawk, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. This book uses RepKover™, a durable and flexible lay-flat binding. ISBN: 978-0-596-10171-8 [M] Table of Contents Foreword . ix Preface . xi 1. MySQL Architecture . 1 MySQL’s Logical Architecture 1 Concurrency Control 3 Transactions 6 Multiversion Concurrency Control 12 MySQL’s Storage Engines 14 2. Finding Bottlenecks: Benchmarking and Profiling . 32 Why Benchmark? 33 Benchmarking Strategies 33 Benchmarking Tactics 37 Benchmarking Tools 42 Benchmarking Examples 44 Profiling 54 Operating System Profiling 76 3. Schema Optimization and Indexing . 80 Choosing Optimal Data Types 80 Indexing Basics 95 Indexing Strategies for High Performance 106 An Indexing Case Study 131 Index and Table Maintenance 136 Normalization and Denormalization 139 Speeding Up ALTER TABLE 145 Notes on Storage Engines 149 v 4. Query Performance Optimization . 152 Slow Query Basics: Optimize Data Access 152 Ways to Restructure Queries 157 Query Execution Basics 160 Limitations of the MySQL Query Optimizer 179 Optimizing Specific Types of Queries 188 Query Optimizer Hints 195 User-Defined Variables 198 5. Advanced MySQL Features . 204 The MySQL Query Cache 204 Storing Code Inside MySQL 217 Cursors 224 Prepared Statements 225 User-Defined Functions 230 Views 231 Character Sets and Collations 237 Full-Text Searching 244 Foreign Key Constraints 252 Merge Tables and Partitioning 253 Distributed (XA) Transactions 262 6. Optimizing Server Settings . 265 Configuration Basics 266 General Tuning 271 Tuning MySQL’s I/O Behavior 281 Tuning MySQL Concurrency 295 Workload-Based Tuning 298 Tuning Per-Connection Settings 304 7. Operating System and Hardware Optimization . 305 What Limits MySQL’s Performance? 306 How to Select CPUs for MySQL 306 Balancing Memory and Disk Resources 309 Choosing Hardware for a Slave 317 RAID Performance Optimization 317 Storage Area Networks and Network-Attached Storage 325 Using Multiple Disk Volumes 326 Network Configuration 328 vi | Table of Contents Choosing an Operating System 330 Choosing a Filesystem 331 Threading 334 Swapping 334 Operating System Status 336 8. Replication . 343 Replication Overview 343 Setting Up Replication 347 Replication Under the Hood 355 Replication Topologies 362 Replication and Capacity Planning 376 Replication Administration and Maintenance 378 Replication Problems and Solutions 388 How Fast Is Replication? 405 The Future of MySQL Replication 407 9. Scaling and High Availability . 409 Terminology 410 Scaling MySQL 412 Load Balancing 436 High Availability 447 10. Application-Level Optimization . 457 Application Performance Overview 457 Web Server Issues 460 Caching 463 Extending MySQL 470 Alternatives to MySQL 471 11. Backup and Recovery . 472 Overview 473 Considerations and Tradeoffs 477 Managing and Backing Up Binary Logs 486 Backing Up Data 488 Recovering from a Backup 499 Backup and Recovery Speed 510 Backup Tools 511 Scripting Backups 518 Table of Contents | vii 12. Security . 521 Terminology 521 Account Basics 522 Operating System Security 541 Network Security 542 Data Encryption 550 MySQL in a chrooted Environment 554 13. MySQL Server Status . 557 System Variables 557 SHOW STATUS 558 SHOW INNODB STATUS 565 SHOW PROCESSLIST 578 SHOW MUTEX STATUS 579 Replication Status 580 INFORMATION_SCHEMA 581 14. Tools for High Performance . 583 Interface Tools 583 Monitoring Tools 585 Analysis Tools 595 MySQL Utilities 598 Sources of Further Information 601 A. Transferring Large Files . 603 B. Using EXPLAIN . 607 C. Using Sphinx with MySQL . 623 D. Debugging Locks . 650 Index . 659 viii | Table of Contents Foreword 1 I have known Peter, Vadim, and Arjen a long time and have witnessed their long his- tory of both using MySQL for their own projects and tuning it for a lot of different high-profile customers. On his side, Baron has written client software that enhances the usability of MySQL. The authors’ backgrounds are clearly reflected in their complete reworking in this second edition of High Performance MySQL: Optimizations, Replication, Backups, and More. It’s not just a book that tells you how to optimize your work to use MySQL better than ever before. The authors have done considerable extra work, car- rying out and publishing benchmark results to prove their points. This will give you, the reader, a lot of valuable insight into MySQL’s inner workings that you can’t eas- ily find in any other book. In turn, that will allow you to avoid a lot of mistakes in the future that can lead to suboptimal performance. I recommend this book both to new users of MySQL who have played with the server a little and now are ready to write their first real applications, and to experi- enced users who already have well-tuned MySQL-based applications but need to get “a little more” out of them. —Michael Widenius March 2008 ix Preface2 We had several goals in mind for this book. Many of them were derived from think- ing about that mythical perfect MySQL book that none of us had read but that we kept looking for on bookstore shelves. Others came from a lot of experience helping other users put MySQL to work in their environments. We wanted a book that wasn’t just a SQL primer. We wanted a book with a title that didn’t start or end in some arbitrary time frame (“...in Thirty Days,” “Seven Days To a Better...”) and didn’t talk down to the reader. Most of all, we wanted a book that would help you take your skills to the next level and build fast, reliable systems with MySQL—one that would answer questions like “How can I set up a cluster of MySQL servers capable of handling millions upon millions of queries and ensure that things keep running even if a couple of the servers die?” We decided to write a book that focused not just on the needs of the MySQL appli- cation developer but also on the rigorous demands of the MySQL administrator, who needs to keep the system up and running no matter what the programmers or users may throw at the server. Having said that, we assume that you are already rela- tively experienced with MySQL and, ideally, have read an introductory book on it. We also assume some experience with general system administration, networking, and Unix-like operating systems. This revised and expanded second edition includes deeper coverage of all the topics in the first edition and many new topics as well. This is partly a response to the changes that have taken place since the book was first published: MySQL is a much larger and more complex piece of software now. Just as importantly, its popularity has exploded. The MySQL community has grown much larger, and big corporations are now adopting MySQL for their mission-critical applications. Since the first edi- tion, MySQL has become recognized as ready for the enterprise.* People are also * We think this phrase is mostly marketing fluff, but it seems to convey a sense of importance to a lot of people.