MySQL Database Administrator
Author: Kacper Wysocki Contact: [email protected] Date: December 2010 License: Creative Commons: CC BY-SA
Oslo, December 2010, CC BY-SA Contents
Introduction 5 Introductions everybody 5 About this course 5 Course outline 6 Course schedule 6 How to do excersies 6 MySQL: history and future 6 MySQL: the present 7 MySQL: the future 7 MySQL compared to other DBs 7 MySQL language support 8 Embedding MySQL 8 Getting help with MySQL 8 MySQL architecture 9 Modular architecture 9 The MySQL modules 9 Client/server architecture 10 Installing MySQL 10 Installation process 10 Distribution packages 11 MySQL official binaries 11 Deploying sandboxes 12 Installing from source 13 Server Startup and Shutdown 14 MySQL relevant files 15 Excersises: Installation 15 Upgrading MySQL 16 Clients: the mysql* suite 16 Client: mysql 16 Excersise: Client mysql 16 Excersise: mysql CLI 17 Further CLI fun 17 Digression: some SQL 18 Client: mysqladmin 18 Excersises: Client: mysql 18 Clients: applications and libraries 18
Oslo, December 2010, CC BY-SA migration 19 Importing data: timezones 19 Importing data 19 Excersises: importing data 20 Excersises: time zones 20 Exporting data 20 Excersises: Exporting data 21 Configuration 21 More configuration 21 Run-time Variables 22 MySQL Architecture 23 Storage Engines 23 Storage Engines 23 Storage Engines types 23 MyISAM 24 MYISAM_MRG 24 InnoDB 24 Excersises: InnoDB 24 FEDERATED 25 CSV 25 ARCHIVE 25 MEMORY 25 BLACKHOLE 25 So... which engine? 26 Engine Excersises 26 Implementing Security 26 Security consideration 26 MySQL access levels 26 User wildcards 27 MySQL Privileges 27 User management 28 The grant tables 28 mysqladmin user access 29 Resource limits 29 Exercises: User Management 29 Data Types 29 Which data type? 29
Oslo, December 2010, CC BY-SA Data Types: Numeric 30 Data Types: Floating point 30 Data Types: Fixed 30 Data Types: Bit 30 Data Types and SQL modes 30 Excersise: Numeric Data 31 Data Types: Strings 31 Excersise: Strings 31 Data Types: Binary 32 Data Types: Time and Date 32 Data Types: null 32 Indexes 32 SQL Locking 33 Transactions and locking 33 Transactions: ACID 33 Transaction isolation levels 33 Transaction operations 33 Excersises: Transactions 33 Metadata 34 Metadata: show 34 Metadata: INFORMATION_SCHEMA 34 Maintenance 34 Table Maintenance 35 Maintenance: CHECK TABLE 35 check table statement 35 checksum table 35 Maintenance: REPAIR TABLE 36 repair table 36 analyze table 36 Maintenance: OPTIMIZE TABLE 36 optimize table 36 Maintenance: ALTER TABLE 36 mysqlcheck and myisamchk 37 Excersises: Table Maintenance 37 MySQL backup options 37 Backup options: raw backups 37 Backup options: mysqldump 37
Oslo, December 2010, CC BY-SA Backup options: xtrabackup 37 Backup options: backup slave 38 Backup options: snapshots 38 Incremental backups 38 Backup recovery 38 Recovery: Binary Log 38 MySQL concepts 38 Stored routines 39 Triggers 39 Events 39 Event schedule 40 Views 40 Monitoring 40 Closing tips 40 Scaling MySQL 41 Clustering for HA or HP 41 Wrapup & Questions? 41 Introduction
• Hello everybody • About this course • Course Outline • What's covered today • Getting up and running with MySQL Introductions everybody
• Hi, my name is $name • I do $foo and am interested in $bar • After this course I will be $something MySQL • Practical stuff: coffee, breaks, food, the john. • Practical stuff: start/end times About this course
• Course outline • Schedule • How to do excersises
Oslo, December 2010, CC BY-SA • The Book(s) • Topics covered today Course outline
In this course you will learn:
• The story, installation, clients of MySQL • Import and export of data • Configuration and user management • Data types, transactions, meta data • Maintenance, backup, recovery • Storage engine fundamentals • Some best practices for performance Course schedule
Time Day 1 Day 2 Day 3 09:30-10:00 Configuration Metadata 10:00-11:00 Overview, story Storage Engines Maintenance 11:15-12:00 Server install Storage Engines Recovery 12:00-12:30 Lunch Lunch Lunch 12:30-13:30 Connect clients User Management Building blocks 13:45-14:45 Migrating:import Data Types 15:00-15:45 Migrating:export Transactions Tuning and HA How to do excersies
• After class? • While we are talking... • After each topic! • Stop me! Ask! MySQL: history and future
History
• 1994: Widenius & Axmark begat MySQL AB • 1995-2008: V.3.23, 4.0, 4.1, 5.0. Popularity! • 2008: Sun aquisition & V.5.1 • 2009: V 6.0: Falcon Maria • 2010: Oracle eclipses the Sun
Oslo, December 2010, CC BY-SA MySQL: the present
MySQL used and developed by major players!
• Goolge • Flickr • Facebook • Tripwire • Zimbra • Banking • (almost all) web developers MySQL: the future
"Prediction is very difficult, especially about the future." -- Niels Bohr 2011: The year of MySQL.
• V 5.5 • MariaDB • XtraDB • OurDelta • Drizzle
The recent year has been a controversial one for MySQL due to the uncertanity of the MySQL license status under Oracle, and there have sprung up several projects that fork the source code, expanding in its functionality with an emphasis on community maintenance under the GNU Public License. Here are presented some of the most promising forks: XtraDB, a backwards-compatible fork which includes many performance improvements, among them a fork of the InnoDB storage engine with better scalability on modern hardware, better monitoring and other features useful in high performance environments. It includes a non-blocking, online backup program called XtraBackup and it is developed and supported by Percona Inc. http://www.percona.com MariaDB is fork lead by Monty Widenius, the founder of MySQL, and aims to maintain a high level of API and binary compatibility with MySQL. MariaDB includes XtraDB as a replacement for InnoDB. http://mariadb.org OurDelta is a MySQL binary distribution aiming at providing enhanced, patched binaries of MySQL packaged for common production platforms. http://ourdelta.org Drizzle is a complete restructuring and simplification of the MySQL code after the KISS principle, which does away with authentication mechanisms and redundant data types in favour of plugin delegation and simplicity. Contributors include members from Canonical Ltd., Google, Six Apart, Sun Microsystems and Rackspace. Some of these projects may be worth looking into if your production environment has special operational or performance requirements, or if you have concerns about the license status of the Oracle MySQL distribution. For general use however the binary packages provided by your production environment are easiest to deploy and maintain, and will suffice. MySQL compared to other DBs
Oslo, December 2010, CC BY-SA • Easy setup • Lots of documentation • Fully featured + plugins! • Clients abound • From low end to high end • Unique: Storage Engines!
MySQL plugins allows extending MySQL functionality in a number of different areas, includig storage engines, information schema, parser plugins and data types. MySQL language support
Most languages wrap the libmysql C library.
• Perl (DBD::MySQL + objects) • PHP • C++ • C# • Java • Ruby, Lua, Lisp, Go • Many more!
PHP extensions: mysql, mysqli, pdo_mysqlnd(native PHP) Python: MySQLdb. JDBC driver for Java: a java native implementation of the MySQL protocol. Connector/NET: ADO.NET managed provider for the .NET framework version 1.1 and up. C++ connector mimics a subset of the JDBC 4.0 API. Many other third-party connectors are available. Embedding MySQL
Embed server into a program:
• libmysqld shared library • MySQL MXJ JAR wrapper for J2EE. Getting help with MySQL
• MySQL reference web: http://dev.mysql.com/ • Mailing list and documentation. • Training for Development, Clustering. • Commercial support from OSS vendors (Redpill Linpro, Percona etc.)
A useful feature of the MySQL web site is the built-in URL search. If you ever need to look up any particular MySQL command or feature, you may enter http://dev.mysql.com/
Oslo, December 2010, CC BY-SA automatically forwarded to the closest matches in the MySQL documentation. MySQL architecture
• Client/Server communications • Unix sockets • TCP/IP • Shared Memory/Named Pipes (windows) • Non-client programs Modular architecture
• All clients use client library • Code is segmented into modules • Modules are subsystems shared across the code • The modules have no interdependencies The MySQL modules
The core modules are
• Shared base functions • Resources: Process and thread • Memory: Caches and buffers • Comms: Networking • Logs: Logging and file rotation • Access: Access tables and grants
Oslo, December 2010, CC BY-SA Client/server architecture
Installing MySQL
• Recommended deployment • Installation alternatives • Packages • Sandbox • Compilation • Upgrading MySQL Installation process
• Install package
Oslo, December 2010, CC BY-SA • set MySQL root password • mysql_secure_installation:
• remove anonymous users • set root pw • remove test database
Distribution packages
Recommended for most uses
# Red Hat, CentOS etc.. yum install mysql mysql-server mysql-devel
# Debian and Ubuntu apt-get install mysql-server
The best reason to install from distribution packages is that it saves time and effort, and maintenance overhead is kept to a minimum. Setting up the required mysql user and group accounts as well as the mysql directories is done automatically in the distribution scripts. Upgrading mysql packages happens automatically while upgrading the rest of the distribution. This type of production environment is the easiest to reproduce and therefore the most supported. It is recommended that you always install MySQL from distribution-specific packages, unless there is a compelling reason to do otherwise. MySQL official binaries
• "The golden standard" • Approved and signed by Oracle • RPM available from mysql.com • Built and extensively tested by MySQL • Only intallation choice for Windows • Better to go with distribution packages.
Oslo, December 2010, CC BY-SA There are several steps to deploying the general binary distribution of MySQL: add mysql group and user account
groupadd mysql useradd -r -g mysql mysql
deploy binary
cd /usr/local tar zxvf /path/to/mysql-VERSION-OS.tar.gz ln -s full-path-to-mysql-VERSION-OS mysql cd mysql chown -R mysql . chgrp -R mysql .
initialize data directory
scripts/mysql_install_db --user=mysql
set correct directory ownership
chown -R root . chown -R mysql data
setup configuration and startup/shutdown script
cp /usr/local/mysql/support-files/my-small/etc/my.cnf /etc/mysql/my.cnf # Optional bin/mysqld_safe --user=mysql & # trial run cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysql.server # Optional
Deploying sandboxes
• Run (many) MySQL servers in a container • Practical for testing • Easy setup for cluster experiments • http://mysqlsandbox.net
Running several MySQL servers, each with their own separate data directory is fully possible given that each installation uses its own TCP port and unix file socket. The MySQL Sandbox is a good choice when you want to:
• set up a lab environment • expermient with different modes of replication • try out different versions of MySQL • maintain an installation with higher security isolation requirements
The MySQL Sandbox comes with ready-made scripts to do the above and more. For full installation details please refer to the mysqlsandbox.net web site.
Oslo, December 2010, CC BY-SA Installing from source
4. Source code - build your own
• Only use this if you know what you're doing • Flexible • Not recommended
Installing from source
Good reasons not to build from source are that
• it takes time to build from source • the build process is complex and may require debugging • you must rebuild to upgrade mysql • your build may not be supported by some vendors
A good reason to build from source is when you want
• to run a particular version of MySQL on a platform where packages do not exist • a feature that is not built by default • to try out a bleeding edge development build • to modify MySQL to do something specific by patching the source code.
Such a source build should not go into production without proper testing, and is easier to maintain if it results in a binary package. If you are on a RPM-based distribution and have the source RPM packages you may rebuild your own binary packages with the following commands:
yum install rpm-build rpmbuild --rebuild --clean MySQL-VERSION.src.rpm
This will check that you have all the packages needed to build MySQL and then produce an RPM package that can be subsequently installed on similar platforms. If you are running a DEB-based distribution, and have a deb-src line in your /etc/apt/sources.list file, you may perform a similar action by running the following commands
apt-get install build-essential devscripts # install builder programs apt-get build-dep mysql-server # install dependencies apt-get source -b mysql-server # this will download the source and run `debuild'
The result is a binary MySQL DEB package that can be installed on your platform. You can also build directly from source. MySQL follows the UNIX build convention of a three-step build involving configuring the build process, building the source and then installing it. First you must make sure that all the build dependencies are satisfied, either by installing dependency source packages, or by installing binary packages from your distribution.
Oslo, December 2010, CC BY-SA MySQL's build dependencies, not accounting for any special build options, are:
• libtool • libncurses5 • libwrap • zlib • readline • automake • doxygen • gawk • bison
Keep in mind that development headers must be installed with the library dependencies. The headers are included if you do a source install of the library in question, or if you install the -dev or -devel variant of the binary package. During the configure step you select the options you care about, then run :
configure --enable-profiling --prefix=/opt/mysql-5.5/ --enable-local-infile \ --with-mysqld-user=mysql --with-big-tables \ --with-plugins=partition,blackhole,federated,heap,innodb_plugin --without-doc \ --other-option-you-care-about make make test # this will run a barrage of tests on your build
make install # installs MySQL under the prefix specified above.
After successfully completing a source install you must set up the required mysql accounts and directories, as well as startup and shutdown scripts, in the same way as done with the binary distribution. Please refer to the MySQL documentation for more information on building from source. Server Startup and Shutdown
• mysqld binary
mysqld --verbose --help
• mysqld_safe : handles restarts and logging • mysqld_multi: perl script to manage many servers
service mysql start or
/etc/init.d mysql start
Oslo, December 2010, CC BY-SA The mysqld binary has many startup options, here are some of the commonly used ones:
• --datadir specifies the location of the data directory. • --port TCP port number. • --socket Unix socket file name. • --no-defaults-file disables parsing of the configuration file.
To see which parameters get passed due to the configuration file, run mysqld --print-defaults. Many instances of MySQL server can be run on the same machine at the same time but they must not share the data directory, and each have their own service name, port, socket, pipe or shared memory handle. MySQL relevant files
• /etc/my.cnf OR /etc/mysql/my.cnf - configuration file • /var/lib/mysql/ - for the database content • /var/log/mysql/ - for log-files Excersises: Installation
• install mysql • verify • check memory • stop server • start mysqld command. what args? • connect • which databases?
1. use distribution-specific tool to install mysql server. 2. verify that the server starts on bootup and you can connect. 3. how much memory does mysql consume? 4. stop the mysql server using the init script 5. start the mysql server by hand. what are the arguments absolutely required by the server? 6. connect using the client program "mysql" 7. which databases are available?
(hint: show databases; or list /var/lib/mysql/) Further excersises: install and try out GUI clients:
• MySQL Query Browser • TOra • Emma
Oslo, December 2010, CC BY-SA Upgrading MySQL
• Stop server • Back up data and conf • Install new release • Restart server • mysql_upgrade
The steps required to update MySQL are usually performed automatically by the distribution-specific upgrade procedures. Note however that you have to provide your own backups, and may need to run mysql_upgrade upon major release changes. The mysql_upgrade program makes sure all tables are in the new row format and autorepairs tables and privileges on tables that do not satisfy the conditions for upgrade. Clients: the mysql* suite
• mysql • mysqladmin • mysqldump • mysqlimport • mysqlcheck = mysqlanalyze = mysqlrepair • mysqlshow Client: mysql
Used to access the CLI. The CLI allows access to:
• Regular SQL queries • Metadata • Dynamic global and session variables • Server status variables Excersise: Client mysql
mysql [options] -h host -u user -p port (default 3306) -e command : execute command in CLI --socket=unix_files_socket --passwords=mypass (unsafe) --compress
Oslo, December 2010, CC BY-SA Excersise: mysql CLI
STATUS; SELECT VERSION(), DATABASE(); SELECT VERSION(), DATABASE()\G help contents; help SHOW;
Further CLI fun
SELECT CURRENT_DATE(),CURRENT_TIME()\c Note the prompt!
\q - quit \h - help \p - pager \e - edit query \# - rehash tee file.txt; notee;
The mysql program accepts the following command line parameters
-h host -u user -p port (default 3306) -e command : execute command in CLI --socket=unix_files_socket --passwords=mypass (unsafe) --compress
Once in the CLI, here are things to try (side excersise):
STATUS; SELECT VERSION(), DATABASE(); SELECT VERSION(), DATABASE()\G help contents; help SHOW;
SELECT CURRENT_DATE(),CURRENT_TIME()\c Note the prompt!
\q - quit \h - help \p - pager \e - edit query \# - rehash tee file.txt; notee;
Note! Control-C will quit the shell and takes some getting used to the frustration that ensues. The commands you execute are stored in ~/.mysql_history
Oslo, December 2010, CC BY-SA Unix keys: Ctrl-[AEKR] : Ahead, End, Kill, Reverse-search Output modes: --tabular, --batch, --html, --xml --safe-updates == --i-am-a-dummy Digression: some SQL
CREATE TABLE foo (id INT(5), name VARCHAR(10))
INSERT INTO foo(name) values("Beelzebub");
UPDATE foo SET id = 666 where name = "Beelzebub";
SELECT name from foo where id = 666;
Client: mysqladmin
• Perform administrative tasks. • Often faster than accessing the CLI. • Use if you're allergic to SQL. mysqladmin can be used to create and drop databases, set passwords and extract server status.
mysqladmin status variables
Excersises: Client: mysql
Add this to your ~/.my.cnf
[mysql] user=root password prompt=\u@\h [\d]>\_ tee=/home/kurs/mysql.log no-beep compress
Clients: applications and libraries
• mysql*-suite of programs • MySQL Query Browser • emma - GUI • TOra • gMySQLcc • phpMySQLAdmin
At this point we'd like to encourage you to try out some of the GUI programs, and the MySQL Query Browser in particular to see if you like it. Use the distribution-specific install command. For example, on
Oslo, December 2010, CC BY-SA Ubuntu Linux type the following:
apt-get install mysql-query-browser
If you prefer to use a GUI to execute SQL queries and MySQL commands, you may do so from now on whenever we mention using the mysql CLI. migration
• Timezone information • Importing data into MySQL • Exporting data Importing data: timezones
• 'Europe/Oslo' • Symbolic timezone tables • Empty/numeric by default • mysql_tzinfo_to_sql
MySQL ships with no timezone tables installed by default, which means symbolic timezones can not be used in SQL statements. However, you may import the system time zone information into MySQL with the script mysql_tzinfo_to_sql by executing the following:
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
This will import the system timezones into mysql. Some systems, like HP-UX and Windows, do not have time zone files and for these cases you may download a timezone distribution at the MySQL website. To install these you must stop the server before copying the files into the data directory. There are some run-time system variables that govern the MySQL server time zone settings. These are:
• system_time_zone variable and corresponding --timezone=zonename startup argument governs the system time zone as perceived by MySQL server. • time_zone global variable and --default-time-zone=zonename argument sets the current global timezone, which may be SYSTEM, same as the system time zone, and requires SUPER privileges to reset. • A per-connection variable time_zone that can be set by the user.
The current timezone affects the display and storage of zone-sensitive time values. The functions NOW() and CURTIME() as well as the storage and retrieval of TIMESTAMP values is zone-sensitive. UTC_TIMESTAMP(), DATE, TIME, DATETIME are not affected. Importing data
• Execute sql statements • SOURCE /path/to/file.sql
Oslo, December 2010, CC BY-SA • Load data infile • mysqlimport load data infile ... can be used in much the same way as select * into outfile, with the added boon that the function does not require FILE privileges.
load data infile '/root/City.txt' into table City ignore 2 lines fields terminated by ',' enclosed by '"' lines terminated by '\n';
Furthermore, we can assign input data into user variables to perform transformations on the imported data:
load data local infile 'assets.txt' [replace|ignore] into table Assets (@skip, nid, @nodename, last_seen, ip, @service, port) set name = concat(@nodename, ' ', @service);
Excersises: importing data
Create a new database called "world" Import the world database (from world.sql) Check by listing the databases Execute:
show databases; use world; show tables;
Further practice:
• create a new table smallcity with columns: name char(35), country char(52), pop int • import data into this table using variable substitution. Excersises: time zones
Import time zone information show time zone tables
SHOW TABLES LIKE 'time_zone%'; DESCRIBE time_zone\G;
Exporting data
• mysqldump • select ... into outfile
Oslo, December 2010, CC BY-SA • output from sql queries
The binary mysqldump database [tables] can be used to do a logical dump of the SQL data.
select * into outfile '/root/mytable.txt' from mytable;
This will result in a tabular file containing rows from mytable. Note that the outfile is placed on the server filesystem, is relative to the data directory, will be owned by the mysql user and will be world-readable. The operation requires the FILE privileges.
select * into outfile '/root/City.txt' fields terminated by '\s' [optionally] enclosed by '\'' escaped by '\\' lines terminated by '\n' from City;
# Escapes: \\0 (zero byte) \\N (NULL) \\b (backspace) \\t tab \\r carriage return
You can use the mysql client together with redirects to dump data into a file. A typical construct that will also compress the data is:
mysql -u root -p -e mybase 'select * from mytable;' | gzip > mytable.sql.gz
Excersises: Exporting data
Create a CSV of the City table Configuration
• my.cnf: the config file • Option groups • Logging • Run-time and startup variables More configuration
• SQL Modes • Binary logs • Server status, processes • Storage Engines • (Partitioning)
Configuration entries are grouped into sections determining the mysql program the entry applies to. The configuration file saves us from specifying long command lines. The configuration file usually resides in /etc/mysql/my.cnf and has the following format:
Oslo, December 2010, CC BY-SA [client] port = 3306 socket = /var/run/mysqld/mysqld.sock
[mysqld_safe] socket = /var/run/mysqld/mysqld.sock
[mysqld] pid_file=/var/run/mysqld/mysqld.pid user = mysql socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql
Most configuration variables have a corresponding command line option, and some also have a corresponding dynamic run-time global or session variable. Here are some of the more common configuration entries for [mysqld] bind_address - the interface address to bind the server TCP port to. log_error - specifies the location of the error log file slow_query_log - enables logging of unindexed and slow queries log-bin - binary query log, for replication and incremental backups general_log - log _everything_ - very expensive sql_mode - how should sql input be handled? innodb_buffer_pool_size - ram usage key_buffer - index pool sort_buffer_size - avoid swapping on queries myisam_sort_buffer_size - see above table_cache - MyISAM tables*threads*[0.1-1] tmp_table_size - temporary ram tables, per session max_heap_table_size - MyISAM heap usage query_cache_size - The query cache. Run-time Variables
• Can be set anytime • Global or session-specific • Some are read-only, status
Dynamic server variables are either global and affect all queries, or they are session-specific in which case the modification only applies to one client session. Setting global variables requires the SUPER privilege. Here are some variable specific commands for you to try out:
Oslo, December 2010, CC BY-SA show variables like 'bulk%'; set bul_insert_buffer=4000000
show session status; show global status;
select @@sql_mode;
MySQL Architecture
• What kind of hardware?
• RAM vs. CPU vs. disk • MySQL scales nicely in hardware • SSD or other options • ibdata, binlog and redo log on hdd... • battery backed controller cache!
• Storage engines
• MyISAM • InnoDB
• MySQL - server and client Storage Engines
• MySQL specific • Advantages and disadvantages for each
• Optimized for speed? • Locking? • ACID? • Fulltext search? • Spacial types? • Backup • ...
Storage Engines
CREATE TABLE foo (a int, b varchar) ENGINE = InnoDB; ALTER TABLE foo Engine = MyISAM;
Storage Engines types
Oslo, December 2010, CC BY-SA • MyISAM (default) - fast and simple! • InnoDB (often used) - supports transactions • NBD • MEMORY • FEDERATED • BLACKHOLE MyISAM
• Default. Very fast • No transaction support • A lot of tunable variables • Crude locking • Compression support • Three files on disk:
• the actual table specification: .frm • the table content: .MYD • the index: .MYI
MYISAM_MRG
MERGE tables, used to represent collection of identical tables treated as single large table.
• .MRG file representation • ... just a list of tables, one per line! InnoDB
• Often used • Developed by Finnish InnoBASE (now owned by Oracle) • "Slower?" • ACID • does auto-tuning inhouse • Files on disk: - table spec: .frm - innodb file-volume: ibdata1 - innodb_file_per_table : .ibd files - transaction log files: ib_logfileN Excersises: InnoDB
• find out how to rename file volume • which variables can you set for innodb?
Oslo, December 2010, CC BY-SA • list data dir • change to innodb engine • list data dir • innodb_file_per_table • default-storage-engine=innodb • which variables are important? FEDERATED
Use MySQL client API to treat remote data source as local. Upon creation, resource must exist.
ENGINE=FEDERATED CONNECTION='mysql://user@hostname/schema/table'
CSV
The CSV engine represents a table as file of comma-separated values.
• May be useful for exporting data • Does not handle corruption well • no indexes, no nulls • Excersise: try out CSV ARCHIVE
Compressed, read-only data, long-term-storage.
• No indexes • .arz. and .arm • no delete,replace,update • row-level locking • Excersise: try out ARCHIVE MEMORY
In-memory, heap-backed volatile tables.
• HASH and BTREE indexes. • Temporary tables. BLACKHOLE
• Keeps schema • sends data to /dev/null • Use for log or relay.
Oslo, December 2010, CC BY-SA So... which engine?
Pick the one that fits your application.
• If in doubt, InnoDB.
InnoDB is the new default in 5.5 over MyISAM Engine Excersises
Change the engine of your table Look at the files in the data directory Implementing Security
• Physical access • Network access • System access • Filesystem access • User access • Users • Logging • Sane configurations Security consideration
• TCP port 3306 • Remote queries? • remote root access? • Unnecessary services (Apache, X, ...) • Strong passwords • SELinux / AppArmor MySQL access levels
5-level grant table system:
Table Description user global db database tables_priv table-level columns_priv column-specific procs_priv per-procedure
Oslo, December 2010, CC BY-SA User wildcards
• MySQL looks at most specific first. • User kari@backtrack, what matches?
User Host karl % kari %ack% kari backt% MySQL Privileges
Privilege level Global Database Table Column Process
MySQL Privileges:
Privilege Level Usage global Alter db, table create db, table create temporary table db create view db delete db, table drop db, table event db grant option db, table, proc index db, table insert db, table, column select db, table, column show view db update db, table, column lock tables db trigger table insert db, table, column select db, table, column
Oslo, December 2010, CC BY-SA update db, table, column create routine proc alter routine proc execute proc
Administrative privileges:
• create temporary tables • create user • file • lock tables • process • reload (flush, reset) • replication client • replication slave • show databases • shutdown • super
Privilege changes take effect next time query reads that table. Upon startup it is possible to disable access control (for password recovery, for instance) with the --skip-grant-tables option. Usually best together with --skip-networking. User management
GRANT all on db.table to user@host [identified by 'password']
GRANT SELECT,INSERT,UPDATE,DELETE on world.* TO joe@localhost;
SHOW GRANTS FOR CURRENT_USER();
SHOW GRANTS;
SET PASSWORD for user@host = PASSWORD('RawegikRi')
RENAME user@host to newuser@newhost; REVOKE all privileges, grant option on foo.* from 'kacper'@'localhost';
The grant tables
Can UPDATE grant tables, but remember to FLUSH PRIVILEGES;
Oslo, December 2010, CC BY-SA select user,host, password from mysql.user where user=''; mysqladmin user access Resource limits
grant all on app.* to 'mickey'@'watergate' with max_user_connections 1 max_connections_per_hour 10 max_queries_per_hour 50 max_updates_per_hour 20;
Exercises: User Management
Create a new user which
• has a password • can only modify the world.* database Check security
• find anonymous users • find duplicate passwords • find host wildcards Data Types
• Numerics • Strings • Binary data • Dates and times • SQL MODE Which data type?
Enough to hold the data, no more.
CREATE TABLE IF NOT EXISTS `prads` ( `nid` int(14) unsigned NOT NULL, `nodename` varchar(255) NOT NULL, `last_seen` datetime NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `ip` decimal(39,0) unsigned DEFAULT NULL, `service` varchar(20) NOT NULL, `port` smallint(5) unsigned DEFAULT NULL, PRIMARY KEY (nid) );
Oslo, December 2010, CC BY-SA Data Types: Numeric
(signed and) unsigned types
Type Name power width TINYINT 2^8 1 byte SMALLINT 2^16 2 bytes MEDIUMINT 2^24 3 bytes INT 2^32 4 bytes BIGINT 2^64 8 bytes nodeid(11) zerofill unsigned Data Types: Floating point
FLOAT(precision) where precision 4 bytes <= 23, 8 bytes<=53
4-byte float represents 6 digits up to ±1e38
FLOAT(digits, following) digits <= 255, following <= 30
DOUBLE(digits,following)
8-byte double represents 15 digits up to ±1e308 Data Types: Fixed
DECIMAL(precision, scale)
Precision is <= 65 significant digits, Scale <= 30 <= P digits following the decimal point. Data Types: Bit
BIT(n) storage: 2^n
insert into table bit values (b'101'); select * from floating where n between .99-.0001 and .99+0.0001;
Data Types and SQL modes
SQL modes decide data trunc/conversion!
• Traditional : warning = error • ANSI : standard compliant • STRICT_TRANS_TABLES • STRICT_ALL_TABLES
Oslo, December 2010, CC BY-SA • allow_invalid_dates • ansi_quotes • DB2, ORACLE, POSTGRESQL Excersise: Numeric Data
• create a table for each data type: - small, bigint - float, double - bit • perform some inserts,selects on that table • check with different sql modes - boundry values Data Types: Strings
Type Name Properties CHAR fixed length VARCHAR variable TEXT non-indexed ENUM one out of a list SET many out of a set
Storage depends on character set. UTF8- strings max 21844 = 65532/3 in VARCHAR. Excersise: Strings
• create a table for each data type: - char, varchar - text - enum, set • perform some inserts,selects on that table • check with different sql modes - boundry values There are four sizes of TEXT data types:
• TINYTEXT • TEXT • MEDIUMTEXT • LONGTEXT
The ENUM type is used to select one element out of a list of distinct values. This can be seen for example in the Continent column of the Country table in the world database:
Continent ENUM('Asia','Europe','North America','Africa','Oceania','Antarctica','South America')
The SET type is used if a column can take on one or more distinct values:
genre SET('rock', 'heavy', 'jazz', 'funk', 'soul')
Character sets are the representable characters in a given string. You can list the available character sets and default collation:
Oslo, December 2010, CC BY-SA show character set;
In addition there is a special binary character set that does not interpret the byte values of the string.
bytes VARCHAR(127) CHARACTER SET BINARY;
Collation is the sort order for a specific character set, and can be case sensitive, case insensitive or binary:
show collation like 'latin%';
Data Types: Binary
BINARY(M) VARBINARY(M) TINYBLOB BLOB MEDIUMBLOB LONGBLOB S S+1, S+2 S+1 S+2 S+3 S+4 255B 64k 255B 64k 16M 4G Data Types: Time and Date
Type Format Size Range TIME HH:MM:SS 3 ±838.59.59 YEAR[2|4] 2011 1 1901-2155, 1970-2069 DATE 2038-01-19 3 1000-01-01-9999-12-31 DATETIME 2038-01-19 16:20:00 8 9999-12-31 23:59:59 TIMESTAMP 2010-12-14 11:23:32 4 1970 to 2037 excersise The TIMESTAMP datatype is stored as seconds since the epoch and can automatically update whenever a row is updated or inserted. Use DEFAULT CURRENT_TIMESTAMP and ON UPDATE CURRENT_TIMESTAMP, both on by default. To turn off, set a default value. Only one column can auto-update per table. Data Types: null
• N/A or unknown. • NULL is a value. • use NOT NULL to avoid. Indexes
• creating indexes • EXPLAIN query
Oslo, December 2010, CC BY-SA • PROCEDURE ANALYZE SQL Locking
• MyISAM table locks • Explicit locks
LOCK IN SHARE MODE; LOCK FOR UPDATE;
Transactions and locking
• ACID • InnoDB transactions • set autocommit = off Transactions: ACID
• Atomic • Consistent • Isolated • Durable Transaction isolation levels
(global|session).tx_isolation
• read-uncommitted (dirty reads) • read-committed (non-repeatable reads) • repeatable-read (phantom reads) • serializable
InnoDB repeatable-reads == serializable. Deadlocks and resolution Transaction operations
start transaction begin savepoint rollback rollback to savepoint release savepoint
Excersises: Transactions
Oslo, December 2010, CC BY-SA • disable autocommit • set transaction isolation level • update data, delete rows, insert • examine data in 2nd session • rollback Metadata
• SHOW • DESCRIBE • INFORMATION_SCHEMA Metadata: show
• show databases; • show [full] (tables|columns); • show index from table; • show engines\G • show table status like 'Country'\G • show processlist; • show create table • mysqlshow program Metadata: INFORMATION_SCHEMA
select TABLE_NAME from INFORMATION_SCHEMA where TABLE_SCHEMA = 'INFORMATION_SCHEMA' order by TABLE_NAME;
select COLUMN_NAME from INFORMATION_SCHEMA.columns where TABLE_SCHEMA = 'INFORMATION_SCHEMA' and TABLE_NAME = 'VIEWS';
select TABLE_NAME,ENGINE from INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA = 'world';
select TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME from INFORMATION_SCHEMA.columns where DATA_TYPE = 'set';
Maintenance
• Table Maintenance
Oslo, December 2010, CC BY-SA • Backup and Recovery • Table management • Integrity • Monitoring Table Maintenance
• check table ... • repair table ... • optimize table • null ALTER TABLE statement Maintenance: CHECK TABLE
• Does a table scan, computes checksums • Makes sure your table isn't corrupt • Checks if indexes are all right • InnoDB: it really shouldn't be check table statement
check table mytable [fast|quick|changed|medium|extended] [for upgrade];
• fast - check only improperly closed tables. • changed - check only changed tables since last check. • medium - scan rows to check link validity. • extended - full key lookup for all keys in each row. • quick - skip row scan. • for upgrade - which tables are compatible with current version? checksum table
checksum table mytable, tabletwo [quick|extended];
• quick : report existing checksum • extended: compute checksum • default: compute if not exists.
Oslo, December 2010, CC BY-SA Maintenance: REPAIR TABLE
• rebuilds table • InnoDB: not an option repair table
repair [local|no_write_to_binlog] table mytable [quick | extended | use_frm];
• QUICK: recreate index tree only • EXTENDED: create index row by row
MyISAM, ARCHIVE (and CSV) fix. Optionally recreate .MYI from .frm. analyze table
analyze [local|no_write_to_binlog] table mytable;
Update statistics on key value distribution, used by optimizer. MyISAM and InnoDB uses this. Maintenance: OPTIMIZE TABLE
• Performs table sort • Gets rid of holes • Does index optimization • Good for MyISAM • InnoDB: shouldn't need to -> null ALTER TABLE optimize table
optimize [local] table mytable;
Defragments table for memory holes. InnoDB recreates table. Maintenance: ALTER TABLE
Performing a null ALTER TABLE rebuilds the table:
ALTER TABLE foo;
Gets rid of holes. Only way to free MEMORY space.
Oslo, December 2010, CC BY-SA mysqlcheck and myisamchk client vs direct file
• -a --analyze • -c --check • -C --check-only-changed • -F --fast • -e --extended -m --medium -q --quick • --repair myisamchk: lock or stop server! --safe-recover: slower Excersises: Table Maintenance create test data create holes in data (where id between 1001 and 2000;) show table status close holes corrupt table index check, repair MySQL backup options
• mysqldump logical backups • Backup on MyISAM - but locking 'cold' directory backups • Master-slave - backup from the slave • Backup on InnoDB - non-locking, 'hot' directory backups Backup options: raw backups
• Lock or shutdown tables! • copy data directory • Remember to backup my.cnf, binary log-files and Innodb-datafile Backup options: mysqldump
mysqldump mydatabase --opt --flush-logs -a --single-transaction --master-data | gzip > mydatabase-`date +%F`.sql.gz
Backup options: xtrabackup
Oslo, December 2010, CC BY-SA • Live backup of MySQL data • innobackupex extension. • Best option if you can use it. • Fails for compressed rows and some other types. Backup options: backup slave
• Set up a read-only slave to replay logs. • Cold backups without bringing down master. Backup options: snapshots
• Have filesystem snapshotting? • Point-in-time copy of data. Example: LVM. • Database is unavailable very shortly. • Then do raw backup of snapshot. Incremental backups
set sql_log_bin = 1;
• flush binary logs on full backup • copy binary logs • point-in-time recovery! Backup recovery
• Recover backed-up data. • Re-execute stored binlog statements. • Re-import logical backup. Recovery: Binary Log
mysqlbinlog bin.000023 bin.000024 | mysql
--start-datetime, --start-position, --stop-position MySQL concepts
Routines, Triggers, Events, Views
• Stored routines
Oslo, December 2010, CC BY-SA • Triggers • Events (mysql "cron") • Views Stored routines
• show create procedure / show procedure status
• invoked through CALL, do not return a value, passes back output vars
• show create function / show function status
• invoked by name, returns a single value
Triggers
create trigger mytrigger [before|after] [update|insert|delete] on mytable for each row trigger_statement;
show triggers; drop trigger mytrigger; can access NEW (before) or OLD (after) table. Don't prepare statements, explicitly commit/rollback, flush, return results. Events
[create|alter] [ definer = (user | CURRENT_USER)] event [if not exists] myevent on schedule SCHEDULE (on completion [not] preserve) [enable | disable | disable on slave] [comment 'comment'] do sql_statement;
set global event_scheduler = on;
Oslo, December 2010, CC BY-SA Event schedule
AT timestamp [+ INTERVAL interval] | EVERY interval [STARTS timestamp [+ INTERVAL interval]] [ENDS timestamp [+ INTERVAL interval]]
INTERVAL keywords: YEAR,HOUR,WEEK etc Views
Simplified access to data. Virtual table.
create [or replace] algorithm = [merge|temptable|undefined] view myview (col1, col2, col3) as select_expression with [cascaded | local] check option; updatable if view consists of one-to-one simple references. Monitoring
• Yes, you do need to monitor your database host. • Tools:
• Trend analysis: Munin, with the dedicated MySQL plugin • Alert notification: Nagios • MySQL Query Analyzer • mytop • show engine innodb status • innotop
• performance analysis: sysbench, super-smack, phptestsuite, mysqlslap Closing tips
• INET_NTOA/ATON(ip) • REVERSE(mail) • NULL takes more space than NOT NULL • UTF16? UTF8. UTF8? latin1 • Partitioning? • Data types! • Indexes! (but not every column) • Backup slave
Oslo, December 2010, CC BY-SA Scaling MySQL
If you take "MySQL advanced DBA"
• Performance tuning • Sharding • NoSQL Socket Handler • MySQL Proxy • memcached • PrimeBase XT • Performance Schema Clustering for HA or HP
• Either high-availability or high-performance
• Master and slave(s) - heartbeat • Multi-master • Circular master - be careful! Wrapup & Questions?
• while(gets(s)?answer(s):0); • feedback welcome! http://training.redpill-linpro.com
• log in (hint: "forgot password") • Course Evaluation
Thank you!
Oslo, December 2010, CC BY-SA