SQL Server: The Transaction Log

As a general database concept, the journal or transaction log is a core part of the database functionality. As changes take place in the database, the transaction log is a core part of the DB architecture that makes disaster recover possible. Every database modification or “transactions” are written to the transaction log in a sequential fashion.

The transaction log is a physical file which is stored on the disk. The location of the log is defined when a database is created.

The log file will appear as a file type of “ldf” in the file path specified. For example, “NewDB_log.ldf”

The log files are set to autogrow, which means that they will claim more physical hard drive space as more transaction are processed by the database.

What is stored on a transaction log?

The log stores every transaction made to the database for the most part. A log is needed for the database to support ACID transactions. ACID stands for atomicity, consistency isolation and durability which means:

• Atomicity: An atomic transaction is either fully completed, or is not begun at all • Consistency: A transaction enforces consistency in the system state by ensuring that at the end of any transaction the system is in a valid state • Isolation: When a transaction runs in isolation, it appears to be the only action that the system is carrying out at one time • Durability: A transaction is durable meaning that once it has been successfully completed, all of the changes it made to the system are permanent

In theory, a log file has the following information:

– Transaction ID: In SQL Server, this is the Log Sequence Number. – Date and time of each update – Before image: A snapshot of the data before the modification took place. – After image: A snapshot of the data after the modification took place. – End of a transaction and start of a transaction – Successful completion (commit) of a transaction

Many types of operations are recorded in the transaction log. These operations include:

• The start and end of each transaction. • Every data modification (insert, update, or delete). This includes changes by system stored procedures or data definition language (DDL) statements to any table, including system tables. • Every extent and page allocation or deallocation. • Creating or dropping a table or index. • Rollback operations. Each transaction reserves space on the transaction log to make sure that enough log space exists to support a rollback that is caused by either an explicit rollback statement or if an error is encountered.

Logical Architecture

The SQL Server transaction log operates logically as if the transaction log is a string of log records.

Each log record is identified by a log sequence number (LSN). Each new log record is written to the logical end of the log with an LSN that is higher than the LSN of the record before it.

Log records are stored in a serial sequence as they are created. Each log record contains the ID of the transaction that it belongs to.

For each transaction, all log records associated with the transaction are individually linked in a chain using backward pointers that speed the rollback of the transaction.

Log records for data modifications record either the logical operation performed or they record the before and after images of the modified data.

When the logging reaches the end of the file it starts again from the beginning, but only if all the requirements has been met and the inactive parts has been truncated. The truncation process is necessary to mark all inactive parts so they can be used again and overwritten

A log record is no longer needed in the transaction log if all of the following are true:

• The transaction of which it is part has committed • The database pages it changed have all been written to disk by a checkpoint • The log record is not needed for a backup (full, differential, or log) • The log record is not needed for any feature that reads the log (such as database mirroring or replication)

Virtual Log Files

The structure of the file is internally split into “Virtual Log Files” (VLFs).

Virtual log files have no fixed size, and there is no fixed number of virtual log files for a physical log file. The Database Engine chooses the size of the virtual log files dynamically while it is creating or extending log files.

The transaction log is a wrap-around file. For example, consider a database with one physical log file divided into four virtual log files. When the database is created, the logical log file begins at the start of the physical log file.

New log records are added at the end of the logical log and expand toward the end of the physical log. Log truncation frees any virtual logs whose records all appear in front of the minimum recovery log sequence number (MinLSN).

The MinLSN is the log sequence number of the oldest log record that is required for a successful database-wide rollback. The transaction log in the example database would look similar to the one in the following illustration.

When the end of the logical log reaches the end of the physical log file, the new log records wrap around to the start of the physical log file.

This cycle repeats endlessly, as long as the end of the logical log never reaches the beginning of the logical log. If the old log records are truncated frequently enough to always leave sufficient room for all the new log records created through the next checkpoint, the log never fills.

However, if the end of the logical log does reach the start of the logical log, one of two things occurs:

• If the FILEGROWTH setting is enabled for the log and space is available on the disk, the file is extended by the amount specified in the growth_increment parameter and the new log records are added to the extension. • If the FILEGROWTH setting is not enabled, or the disk that is holding the log file has less free space than the amount specified in growth_increment, a 9002 error is generated

If the log contains multiple physical log files, the logical log will move through all the physical log files before it wraps back to the start of the first physical log file.

Log Truncation

Log truncation is essential to keep the log from filling. Log truncation deletes inactive virtual log files from the logical transaction log of a SQL Server database, freeing space in the logical log for reuse by the physical transaction log. If a transaction log were never truncated, it would eventually fill all the disk space that is allocated to its physical log files. However, before the log can be truncated, a checkpoint operation must occur. A checkpoint writes the current in-memory modified pages (known as dirty pages) and transaction log information from memory to disk.

When the checkpoint is performed, the inactive portion of the transaction log is marked as reusable. Thereafter, the inactive portion can be freed by log truncation.

The following illustrations show a transaction log before and after truncation. The first illustration shows a transaction log that has never been truncated. Currently, four virtual log files are in use by the logical log. The logical log starts at the front of the first virtual log file and ends at virtual log 4.

The MinLSN record is in virtual log 3. Virtual log 1 and virtual log 2 contain only inactive log records. These records can be truncated. Virtual log 5 is still unused and is not part of the current logical log.

The second illustration shows how the log appears after being truncated. Virtual log 1 and virtual log 2 have been freed for reuse. The logical log now starts at the beginning of virtual log 3. Virtual log 5 is still unused, and it is not part of the current logical log.

If log truncation cannot occur in time, the log file will grow and can lead to an out of space error over time.

To truncate the log, use the DBCC SHRINKFILE Useful Commands for the Transaction Log

DBCC SQLPERF(LOGSPACE)

We can use this monitor the growth of the log files:

DBCC SQLPERF (Logspace) This will give us the current lo size and how much space is currently used (in terms of percentage).

BACKUP

Used to backup the transaction log to a disk or backup device. This is typically done before shrinking the file.

USE master; GO ALTER DATABASE MyDB SET RECOVERY FULL; GO

BACKUP LOG MyDB TO DISK = 'Z:\SQLServerBackups\MyDBLog.bak' WITH FORMAT; GO

DBCC SHRINKFILE

This command can be used to shrink a log file to a specific target size. This should be done after your log file has been backed up.

-- Truncate the log by changing the database recovery model to SIMPLE. ALTER DATABASE MyDB SET RECOVERY SIMPLE; GO -- Shrink the truncated log file to 1 MB. DBCC SHRINKFILE (MyDB_Log, 1); GO -- Reset the database recovery model. ALTER DATABASE MYDB SET RECOVERY FULL; GO

Transaction Log and its role in recovery

The transaction log is a crucial component of the recoveries we can perform because it maintains the before and after image of the transactions.

Forward Recovery

Scenario:

Current prod database is damaged and is not available. Assume that we have a copy of the production database from a couple of days ago, and the full copy of the transaction log.

We can use out database backup (a couple of days old) and apply the transactions of the transaction log (because they contain the after images).

Back Recovery (Rollback)

Assume that our production database has a set of transaction which are executed so that our database is in an inconsistent state. We want to essentially do a rollback of the transactions that were committed.

We can do this because we have access to the before images of the committed transaction in the transaction log.