Data Management Systems • Transaction Processing • Concurrency control and recovery Logging • Transactions • Recovery Recovery procedures Implementations Gustavo Alonso Institute of Computing Platforms Department of Computer Science ETH Zürich Recovery 1 Dealing with failures • Recovery aims at reconstructing a consistent database state after a failure • Failures considered: • Transaction failure (transaction aborts, rollbacks, client fails in the middle of executing a transaction, transaction times out) • Recovery procedure is to undo the changes made by the transaction while it was active • System failure (machine shuts down, machine fails, OS bails out, etc.) • Recovery procedure is to bring the database back in a consistent state • Media failure (disk fails, permanent storage has errors, etc.) • Recovery procedure involves restoring the database to a known consistent state Recovery 2 Dealing with failures • Transaction failure: • Typically done using undo logs or undo records (see last section) • Transactions must create an undo record every time they modify something • The undo record is used to remove changes made by the transaction while it was active • System failure: • Typically done using redo and undo logs as well as by running a recovery procedure • Undo the changes of active transactions at the time of failure, redo the changes of committed transactions • Clan up the data in permanent storage • Media failures • Typically addressed through replication and by separating data files from log files Recovery 3 System failure • In this section we will focus on system failures: the sudden loss of the data in memory • The engine needs to be able to recover the database up to the point of its las committed state, containing the changes made by all committed transactions as of the time of the failure • This is called the recovery procedure • The recovery procedure needs to • Operate only with the data on permanent storage • Be correct even when successive failures occur in the middle of the recovery procedure Recovery 4 Logging Recovery 5 Some notation • The value that existed in the database before a transaction modifies an item is called a “before image” • The value that exists in the database after a transaction modifies an item is called “after image” • A record that registers what the transaction has done (a before image, and after image, transaction is, SCN, LSN, etc.) is a log record • In a database engine, both the data and the log are persistent; recovery procedures will combine both depending on the design • We will assume serializable and strict execution: • A transaction can be undone by restoring its before images • A transaction can be redone by applying its after images • Complication in reality: transactions update tuples, I/O system updates blocks Recovery 6 The “log” • For the moment, we will work with an idealized log • We will assume physical logging: the log entries contain the actual data modified by the transaction (before image, after image) • For each modification made by a transaction, there is a log record • Log records are ordered and reflect the logical sequence of events in the database (use SCN or LSN) • The log is made persistent (kept in stable storage) • Other data structures and information can be kept persistent as well • Which transaction have committed, aborted, active … • Flush operations force that something (a block, a log entry, some information) is written to permanent storage Recovery 7 Achieving durability Buffer cache Persistent data Transaction T I/O update x update y insert p insert q delete t In Memory Log Persistent Log …. T T T T T T T T I/O Xbi Ybi p q Xbi Ybi p q Xai Yai … Xai Yai Recovery 8 Durability • Durability ensures that the database remembers what has been committed and that it can recover the last committed state of the database • This implies writing to persistent storage (I/O), which is an expensive operation. However: • Being able to recover the last committed state implies we remember that state • Thus, when a transaction commits, we need to store in persistent state everything we might need to restore that transaction • Commit => I/O (of data, logs, or both) Recovery 9 When to write to persistent storage • Transactions modify items: • If changes form an active transaction can end up on persistent storage, recovery will involve undo of such changes -> REQUIRES UNDO • If changes form a committed transaction are not yet on persistent storage when the transaction is declared as committed (the client is notified), recovery will involve redo of such changes -> REQUIRES REDO • In both situations we need the log => when a transaction commits, all its log entries must be persistent • To undo changes: we need the before image (UNDO LOG) • To redo changes: we need the after image (REDO LOG) • In the idealized log we study now, assume the log contains everything Recovery 10 Recovery Procedures Recovery 11 Recovery manager • The recovery manager implements the recovery procedure, which will depend on how the system functions: • Requires UNDO and REDO • Requires UNDO • Requires REDO • Requires neither • The procedure will depend on when modifications are written to persistent storage with respect to when the transaction commits • We will assume updates in place and ignore blocks for the moment (assume items can be made persistent individually) Recovery 12 From the buffer cache perspective • Steal Policy • STEAL: an uncommitted transaction is allowed to overwrite in persistent storage the changes of a committed transaction • This happens by pushing a dirty block to storage before the transaction commits • Will require to be able to undo this transaction • Force Policy • FORCE: all changes made by a transaction must by in persistent storage before the transaction commits • Requires to flush all blocks with updates form the transaction • If not in place, it will require to be able to redo the transaction • Steal/no-Force = UNDO/REDO, most common approach Recovery 13 Lock tuples, update blocks => STEAL Buffer cache Persistent data Transaction T1 update x commit I/O Transaction T2 update y Transactions that do not conflict may still be updating the same block. If the block is copied to storage, it is possible that some changes are committed while others are not. If failures occur, there is no guaranteed the data in storage is 100% consistent Recovery 14 Requires UNDO and REDO • READ • Just read the value from the block on the buffer cache • WRITE • Create log entry (before image, after image) and append it to the persistent log • Write after image to block on the buffer cache • COMMIT • Write a persistent log entry indicating the transaction has committed • ABORT • For all updates, restore the before image using the log entry Recovery 15 Recovery procedure (UNDO/REDO) • Start from the end of the log and work backwards • Keep a list of UNDONE items and another for REDONE items • Procedure terminates when all items are in either the UNDONE or REDONE list or we reach the beginning of the log • For each log entry: • Look at the data item being accessed (x), if x is none of the two lists • If the log entry is of a committed transaction, apply the after image, add x to the REDONE list • If the log entry is of an aborted transaction, apply the before image, add x to the UNDONE list Recovery 16 UNDO/REDO recovery procedure • The procedure ignores the data stored on disk as it could correspond to uncommitted transactions (hence the need for UNDO), it only takes it as starting point for recovery • Another way to look at this procedure is as follows: • For every item in the database: • Find the last committed transaction that modified the item and REDO the modification • If no committed transaction modified the item, find the first aborted transaction than modified the item and UNDO the modification • If no transaction has touched the item, its value is correct (we assume we start from a consistent state) • In practice, not done like this because there are many more items than log entries, easier to process the log sequentially from end to the beginning. Recovery 17 Advantages of UNDO/REDO recovery • The only forced I/O are log records • Buffer Cache manager has a lot of freedom: • No need to flush dirty pages if there is no need to reuse the space • I/O on data is minimized and only triggered for block replacement policies • Allows to write dirty data (written by uncommitted transactions) to disk, which simplifies buffer management • Recovery is more complicated and takes more time but normal operations are only minimally affected • Queries are not affected since there is no forced I/O of data • Transactions are affected because the need to write every operation to the log Recovery 18 UNDO, no REDO • READ • Just read the value from the block in the buffer cache • WRITE • Create log entry (before image) and append it to the persistent log • Write after image to block on the buffer cache • COMMIT • Flush all dirty values modified by the transaction if still in the cache • Write a persistent log entry indicating the transaction has committed • ABORT • For all updates, restore the before image using the log entry Recovery 19 Recovery Procedure (UNDO/no REDO) • Start form the end of the log and scan backwards • Keep a list of UNDONE items • Procedure terminates when all items are in UNDONE list or we reach the beginning of the log • For each log entry: • Look at the data item being accessed (x) • if x is not in the UNDONE list and the transaction is aborted • UNDO the changes by using