MySQL Query Optimization Originally: Query Optimization from 0 to 10 by Jaime Crespo Agenda 1. Introduction 7. Query Profiling 2. Access Types and Basic 8. General Optimizer Indexing Techniques Improvements 3. Multi-Column Indexing 9. Computed/Virtual Columns 4. FULLTEXT Search 10. Query Rewrite Plugins 5. Joins 11. Optimizer Hints 6. Subqueries 12. SQL Mode Changes Progress bar 1. Introduction 7. Query Profiling 2. Access Types and Basic 8. General Optimizer Indexing Techniques Improvements 3. Multi-Column Indexing 9. Computed/Virtual Columns 4. FULLTEXT Search 10. Query Rewrite Plugins 5. Joins 11. Optimizer Hints 6. Subqueries 12. SQL Mode Changes MySQL Versions ‣ MySQL 5.1 no longer officially supported ‣ Results shown for MySQL 5.5-5.7 / MariaDB 10.1 Recently added features ‣ Index Condition Pushdown ‣ EXPLAIN FORMAT=JSON ‣ Subquery Optimizations ‣ EXPLAIN INSERT/UPDATE/ materialization and semijoin) DELETE ‣ ‣ IN-to-EXISTS/EXISTS-to-IN Hash Join ‣ New optimizer hints ‣ JOIN-to-WHERE ‣ New cost-based optimizer ‣ Multi-Range Read ‣ Optimizer Trace ‣ Batched Key Access ‣ Filesort optimizations ‣ Persistent InnoDB Statistics ‣ Virtual/computed columns and ‣ UNION ALL optimization “functional indexes” ‣ Improved GIS support ‣ New JSON type Example Application (I) ‣ Wiktionary (and all Wikimedia project's data) is licensed under the Creative Commons BY-SA-2.5 License and is Copyright its Contributors Example Application (II) ‣ OSM Database is licensed under the Open DataBase License and is Copyright OpenStreetMap Contributors Install sample databases to follow along ‣ Downloads and instructions at: http://speedemy.com/ppm ‣ Requirements: a MySQL or MariaDB installation (MySQL Sandbox is suggested) ‣ The wiktionary and OSM extracts ‣ Import them with: $ bunzip -c <file> | mysql <database> Progress bar 1. Introduction 7. Query Profiling 2. Access Types and Basic 8. General Optimizer Indexing Techniques Improvements 3. Multi-Column Indexing 9. Computed/Virtual Columns 4. FULLTEXT Search 10. Query Rewrite Plugins 5. Joins 11. Optimizer Hints 6. Subqueries 12. SQL Mode Changes EXPLAIN Our key tool when it comes to query optimizatin Don’t do this ‣ MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'; ‣ +------+-------------+-------+------+--------------- +------+---------+------+------+-------------+ ‣ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | ‣ +------+-------------+-------+------+--------------- +------+---------+------+------+-------------+ ‣ | 1 | SIMPLE | page | ALL | NULL | NULL | NULL | NULL | 90956 | Using where | ‣ +------+-------------+-------+------+---------------Difficult to see +------+---------+------+------+-------------+something ‣ 1 row in set (0.00 sec) Use vertical format instead MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL Use \G for rows: 90956 vertical Extra: Using where 1 row in set (0.00 sec) formatting EXPLAIN: id MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 Indicates hierarchy select_type: SIMPLE level, not execution table: page type: ALL order possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 90956 Extra: Using where 1 row in set (0.00 sec) EXPLAIN: select_type MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE Not a subquery or table: page type: ALL a UNION possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 90956 Extra: Using where 1 row in set (0.00 sec) EXPLAIN: table MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page Table scanned for type: ALL possible_keys: NULL this step key: NULL key_len: NULL ref: NULL rows: 90956 Extra: Using where 1 row in set (0.00 sec) EXPLAIN: type MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page All rows are read type: ALL possible_keys: NULL for this table (FULL key: NULL TABLE SCAN) key_len: NULL ref: NULL rows: 90956 Extra: Using where 1 row in set (0.00 sec) EXPLAIN: rows MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL Estimated number rows: 90956 Extra: Using where of rows to be read 1 row in set (0.00 sec) (all table rows) How to improve performance? MariaDB [nlwiktionary]> SELECT * FROM page WHERE page_title = 'Dutch'; 1 row in set (0.11 sec) ● Let's add an index on page.page_title: MariaDB [nlwiktionary]> ALTER TABLE page ADD INDEX page_title (page_title); Query OK, 0 rows affected (0.19 sec) Records: 0 Duplicates: 0 Warnings: 0 EXPLAIN: type - ref MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page type: ref means that an type: ref equality comparison will be possible_keys: page_title checked against an index key: page_title and several results could be key_len: 257 returned ref: const rows: 1 Extra: Using index condition 1 row in set (0.00 sec) EXPLAIN: possible_keys and key MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page type: ref possible_keys: page_title index(es) that the key: page_title optimizer considered key_len: 257 potentially useful, and ref: const final index chosen rows: 1 Extra: Using index condition 1 row in set (0.00 sec) EXPLAIN: ref MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page type: ref possible_keys: page_title key: page_title key_len: 257 ref: const Index is compared rows: 1 with a constant, not Extra: Using index conditionwith another table 1 row in set (0.00 sec) EXPLAIN: rows MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title = 'Dutch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page type: ref possible_keys: page_title key: page_title key_len: 257 ref: const rows: 1 Only 1 row read. In this Extra: Using index conditioncase, estimation is exact 1 row in set (0.00 sec) (thanks to index dive) Query time MariaDB [nlwiktionary]> SELECT * FROM page WHERE page_title = 'Dutch'; 1 row in set (0.00 sec) Query time has been reduced substantially Types of indexes ‣ B-tree ๏ B-tree in MyISAM, B+tree in InnoDB ‣ HASH ๏ Only available for MEMORY and NDB ‣ FULLTEXT ๏ Inverted indexes in MyISAM and InnoDB ‣ SPATIAL ๏ RTREEs in MyISAM and InnoDB Finding a “Dutch” within a B-tree Avast Boss Dutch Etch Freight Golf Harlem Iceberg Landscape Leek Maelstrom Nasty Rover School Walrus TABLE Finding a “Dutch” within a B-tree Avast Boss Dutch Etch Freight Golf Harlem Iceberg Landscape Leek Maelstrom Nasty Rover School Walrus TABLE Indexes don't alway work ‣ Can we use an index to make this query faster: SELECT * FROM page WHERE page_title LIKE ‘Spa%’; It’s a range lookup MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title like 'Spa%'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page Despite not being an type: range equality, we can use possible_keys: page_title the index to find the key: page_title values quickly key_len: 257 ref: NULL rows: 94 Extra: Using index condition 1 row in set (0.00 sec) B-tree index can be used for ranges too Avast Boss Dutch Etch Freight Golf Harlem Iceberg Landscape Leek Maelstrom Nasty Rover School Walrus TABLE Indexes don't alway work ‣ How about this query: SELECT * FROM page WHERE page_title like '%utch'; Let me EXPLAIN: MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM page WHERE page_title like '%utch'\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: page type: ALL No index can be possible_keys: NULL used for filtering. A key: NULL full table scan is key_len: NULL performed. ref: NULL rows: 93189 Extra: Using where 1 row in set (0.00 sec) B-tree index lookup Avast Boss Dutch Etch Freight Golf Harlem Iceberg Landscape Leek Maelstrom Nasty Rover School Walrus TABLE B-tree index usage ‣ Filtering ๏ Equality (operator '=') ๏ Ranges (BETWEEN … AND, >, <, >=, <=, like 'prefix%') ‣ “EXISTS” operators: IN, OR on the same column ๏ Ordering ๏ ORDER BY (indexed columns) ๏ GROUP BY (indexed columns) ‣ Returning values directly from the index ๏ Covering index ๏ Functions like max(), min(), etc. Type: const MariaDB [nlwiktionary]> EXPLAIN SELECT * FROM revision WHERE rev_id = 2\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: revision 'const' is a special case of 'ref', type: const when the index can assure that possible_keys: PRIMARY only 1 results can be returned key: PRIMARY (equality + primary key or key_len: 4 unique key). It is faster. ref: const rows: 1 Extra: 1 row