Query Processing for SQL Updates Cesar´ A. Galindo-Legaria Stefano Stefani Florian Waas [email protected] [email protected] fl[email protected] Microsoft Corp., One Microsoft Way, Redmond, WA 98052 ABSTRACT tradeoffs between plans that do serial or random I/Os. This moti- A rich set of concepts and techniques has been developed in the vates the integration of update processing in the general framework context of query processing for the efficient and robust execution of query processing. To be successful, this integration needs to of queries. So far, this work has mostly focused on issues related model update processing in a suitable way and consider the special to data-retrieval queries, with a strong backing on relational alge- requirements of updates. One of the few papers in this area is [2], bra. However, update operations can also exhibit a number of query which deals with delete operations, but there is little documented processing issues, depending on the complexity of the operations on the integration of updates with query processing, to the best of and the volume of data to process. Such issues include lookup and our knowledge. matching of values, navigational vs. set-oriented algorithms and In this paper, we present an overview of the basic concepts used trade-offs between plans that do serial or random I/Os. to support SQL Data Manipulation Language (DML) by the query In this paper we present an overview of the basic techniques used processor in Microsoft SQL Server. We focus on the query process- to support SQL DML (Data Manipulation Language) in Microsoft ing aspects of the problem, how data is modeled, primitive opera- SQL Server. Our focus is on the integration of update operations tors and different execution plans. There are a number of other as- into the query processor, the query execution primitives required to pects required in a complete implementation, notably concurrency support updates, and the update-specific considerations to analyze control and locking, and algorithms for the manipulation of B-trees. and execute update plans. Full integration of update processing in Those subjects are outside of the scope of the present paper. the query processor provides a robust and flexible framework and leverages existing query processing techniques. 2. SQL UPDATES The concepts presented in this paper can be applied to a num- ber of data modification scenarios, but we focus on the basic op- 1. INTRODUCTION erations and functionality provided by SQL. This functionality is Data update operations are a necessary part of customer applica- briefly reviewed in this section. Examples throughout the paper tions, and their efficient processing is as important as that of data- use the well-known TPC-H schema [4]. retrieval operations. When data is inserted, deleted or modified in The basic data-modification statements provided by SQL are IN- a database, the DBMS (Database Management System) needs to SERT, DELETE, and UPDATE. Their target is a single table, al- validate the changes against declared constraints and maintain the though multiple tables may be modified as a result of the changes, underlying storage structures to preserve a consistent and correct as we shall see later. INSERT takes a collection of rows computed representation of the data. using an arbitrary SQL query and adds those rows to the target ta- Some aspects of update processing have been thoroughly studied ble. DELETE and UPDATE use a WHERE condition to qualify and documented in the past. Transactions and concurrency control the rows of interest, either to be removed from the table or to as- issues [1] are examples where formal models, algorithms, and per- sign values to their columns. It is valid to use subqueries in either formance analyses have been developed over time and contribute the qualifying predicate or the assignment values, so multiple tables to our understanding of the problem space. It has also led to suc- and arbitrarily complex queries are often part of data modification cessful and robust commercial implementations. statements. Depending on the complexity of the operations and the volume The following simple queries show deletion and update of a row of data to process, standard query processing issues also surface in the ORDERS table: during the execution of updates. These issues include lookup and matching of values, navigational vs. set-oriented algorithms and DELETE ORDERS WHERE O ORDERKEY = 3271 UPDATE ORDERS SET O PRIORITY = URGENT Permission to make digital or hard copies of all or part of this work for WHERE O ORDERKEY = 3271 personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage, and that copies Update operations need to be validated against declared con- bear this notice and the full citation on the first page. To copy otherwise, to straints. Several forms of constraints are supported by SQL. republish, to post on servers or to redistribute to lists, requires prior specific Check constraints are predicates that enforce the domain of a permission and/or a fee. SIGMOD 2004 June 13-18, 2004, Paris, France. column, or relationships between columns of a single table. For ex- Copyright 2004 ACM 1-58113-859-8/04/06 . $5.00. ample, a constraint can be O ORDERSTATUS IN (“O”, “F”, “P”). When inserting a row or updating column values, the DBMS re- jects the operation if the column values fail to satisfy the check constraint. A common special case of this form of constraint is to declare a column NOT NULL. UPD(S) secondary application Referential integrity (RI) constraints establish a relationship be- optional secondary ∆−Stream tween tables, indicating that the values appearing in one of the tables must be found in the key of another table. For example, the values of column O CUSTKEY in ORDERS must be found in C CUSTKEY of CUSTOMER. The column C CUSTKEY is called the referenced key, and the column O CUSTKEY of OR- UPD(T) primary application DERS is called the foreign key. primary ∆−Stream Finally, uniqueness constraints indicate that the value of one of more columns cannot have duplicates within the table, such as Read−Query C CUSTKEY in CUSTOMER. In addition to validation, update operations need to maintain the underlying storage structures. From a query processing point of T view, maintenance consists of finding all the underlying structures that represent the logical data and issuing the necessary modifica- Figure 1: General form for update plan. tions to keep them consistent and accurate for the data represented. In practice, this means updating all necessary indices and material- ized views that depend on the changed data. Due to space limita- Figure 1 shows a general template for the execution plan of up- tions, we will focus on index maintenance issues in this paper. date statements. It comprises two components. The first is read- Cascading RI constraints combine validation with maintenance only and responsible for delivering a delta stream that specifies the of structures. They specify, for example, that if the value of changes to be applied to the target table. The second component C CUSTKEY is changed then the system needs to automatically consumes the delta stream, applying the changes to the base table, change the value of the referencing O CUSTKEY to preserve the and then performs all the actions that the DML statement implicitly constraint. This is one example where multiple tables are modified fires. The list of actions to perform is determined by enumerating as a result of an update statement, even though its target is a single all the active dependencies against the target table and deciding base table. which of these are required by the current statement. 3. MODELING UPDATES IN QUERY PRO- 4. INDEX MAINTENANCE CESSING 4.1 Indices in Microsoft SQL Server The query processor of Microsoft SQL Server is based on an In Microsoft SQL Server, each base table has a “primary access algebraic, extensible model that supports the addition of new oper- path” storing the table contents. Such primary path can be either ators as necessary. Both in query optimization and in query execu- a clustered index, if organized as a B-tree, or a heap otherwise. tion, the framework provides a contract for operators and manipu- In either case, there is always a locator consisting of one or more lates trees of those operators. This general philosophy is formulated columns whose value uniquely identifies a row within the table and in [3] and it serves as the background for the work described here. can be used to locate it efficiently. A heap is not ordered; however, The following concepts make up the backbone of our update pro- a row is uniquely identified by a physical locator called RID. In a cessing approach: clustered index, a row can be uniquely identified with the values for the index keys, if the index is unique. If the clustered index is non- Delta stream. A delta stream is a collection of rows encoding unique, an extra, hidden column is added to the index keys, called changes to a particular base table. This is simply a relation uniquifier. The uniquifier column is used to distinguish between with a well-defined schema and as such it can be processed two rows with identical clustered index keys, and is automatically by any relational operator. Two different relational encod- assigned by the storage engine when populating or maintaining the ings of delta streams will be detailed later. B-tree. Secondary indices, also called non-clustered indices in Microsoft Application of a delta stream. A side-effecting operator called SQL Server, can also be created for the base table and they also StreamUpdate issues a data-modification command to the correspond to B-trees.