Including Group-By in Query Optimization Surajit Chaudhuri Kyuseok Shim* Hewlett-Packard Laboratories Palo Alto, CA 94304 [email protected], [email protected] several groups (e.g., in business sectors) and aggregate on some attributes (e.g., sum of total sales). A recent Abstract study of customer queries in DB2 [TM911 and other surveys [LCW93] h ave found that the group-by con- In existing relational database systems, processing struct occurs in a large fraction of SQL queries used of group-by and computation of aggregate func- in decision-support applications. Therefore, efficient tions are always postponed until all joins are per- processing and optimization of queries with group-by formed. In this paper, we present transformations and aggregation are of significant importance. Unfortu- that make it possible to push group-by operation nately, this problem has so far received little attention. past one or more joins and can potentially reduce For a single-block SQL query, the group-by operator the cost of processing a query significantly. There- is traditionally executed after all the joins have been fore, the placement of group-by should be decided processed. Such “two-phase” execution of aggregate based on cost estimation. We explain how the tra- queries is a consequence of the fact that the optimizers ditional System-R style optimizers can be modified concentrate on selection, projection and join operators by incorporating the greedy conservative heuristic only. Conventional relational optimizers do not exploit that we developed. We prove that applications of the knowledge about the group-by clause in a query, greedy conservative heuristic produce plans that beyond including the grouping columns in the list of are better (or no worse) than the plans gener- interesting orders during join enumeration (discussed ated by a traditional optimizer. Our experimental in Section 4). In this paper, we present significant new study shows that the extent of improvement in techniques for processing and optimization of queries the quality of plans is significant with only a mod- with group-by. est increase in optimization cost. Our technique also applies to optimization of Select Distinct queries by pushing down duplicate elimination in a cost-based fashion. 1.1 Motivating Application 1 Introduction Our examples are taken from a data warehouse appli- Decision-support systems use the SQL operation of cation that analyzes trends in order placement. We group-by and aggregate functions extensively in for- have simplified the presentation for ease of exposition. mulating queries. For example, queries that create A company has a set of business divisions. Each di- summary data are of great importance in data ware- vision belongs to a sector. Each product belongs to a house applications. These queries partition data in division. For a given product, there is a fixed process- *Work done while the author was visiting Hewlett-Packard ing overhead for every order. An order is placed by a Laboratories. Author’s current address: Division of Research dealer. For every order, the name of the product and and Statistics, Federal Reserve Board, Washington, D.C. 20551. the amount and the date of sale are registered. Finally, Permission to copy without fee all or part of this mat&al is for every dealer, the state and the street address are granted provided that the copies aTe not made OT distributed fOT recorded. Thus, the relations and the attributes in the direct commercial advantage, the VLDB copyright notice and the title of the publication and its date appea+, and notice ia schema are: given that copying is by permission of the Very Large Data Base Division(divid, sectorid) Endowment. To copy otherwise, OT to republish, requites a fee and/or special permission from the Endowment. Product(prodid, overhead, divid) Proceedings of the 20th VLDB Conference Order(orderid, prodid, dealerid, amount, date) Santiago, Chile, 1994 Dealer(dealerid, state, address) 354 1.2 Transformations orders and each division markets many products. For such statistics, the ordering of the joins in We make the key observation that since a group-by re- the traditional plan could be (from left to right) duces the cardinality of a relation, an early evaluation (Division W Product) W Order. The alternative exe of group-by could result in potential saving in the costs cution plan suggested in Example 1.2 is attractive for of the subsequent joins. We present an example that such a database since the early group-by operations illustrates a transformation based on the above obser- could result in a significant reduction in cardinality vation. An appropriate application of such a transfor- of the relations. However, the join ordering in the al- mation could result in plans that are superior to the ternative plan is different from that in the traditional plans produced by conventional optimizers by an order plan. In the former, the Order relation, after group-by, of magnitude or more. joins with Product. Subsequently, the resulting rela- tion (possibly after another group-by) is joined with Example 1.1: Let us consider the query that com- Division. This shows that the decision to push down putes the total sales for each sector of the company. group-by operations influences the join order. Also, an Traditionally, this query is computed by taking the early application of a group-by operator is not always join among Division, Product and Order, subse- optimal. Thus, if there are only a few orders for every quently doing a group-by on sectorid and computing product and each division markets a few products only, Sum(amount.). However, the following alternative plan then the alternative plan could perform worse than the is possible. First, we group-by the Order relation on traditional plan. I the attribute prodid before joining the relation with Product. In other words, we first compute the total sales for each product before the join. If each prod- The optimizer needs to identify where it can COT- uct has a large number of orders, then the above step rectly place a group-by operator that can be evaluated may lead to a significant reduction in the size of the early. Moreover, as the above example shows, its deci- relation and hence in the cost of the subsequent join sion whether or not to push the group-by operator past with Product. Next, we can group-by the resulting re- the joins must be cost-based. Since the transformations lation on divid. Intuitively, this computes the total affect the ordering of joins, their applications need to sales achieved by each division. Finally, we join the be considered in conjunction with the task of choosing resulting relation with Division and do a (residual) a join order. group-by on sectorid. I Since the optimization of queries with group-by can- not be treated in isolation, it is imperative that we In the above example, a single group-by in the tra- incorporate our techniques in the framework of con- ditional execution tree is replaced by multiple group- ventional optimizers. The System R style optimization by operators in an equivalent execution plan and the algorithm [S*79] is used in commercial systems and so grouping is done in stages, interleaved with join. In this we will use that as the prototypical conventional op- paper, we will present other transformations as well. A timizer. Integration of our optimization ideas in the simpler case of transformation is where the group-by conventional optimizer raises the following two issues. operator in the given query is moved past joins, but is First, what effect does it have on the size of the exe- not broken up in stages. In such cases, the execution cution space? Next, how do we extend the search al- plan contains only one group-by but there are multi- gorithm, taking into account the trade-off between cost ple alternatives in the execution plan where it may be of optimization and the improvement in the quality of placed. Furthermore, by maintaining the count of the plans? tuples that were in the coalesced group, we can push We note that duplicate elimination can be viewed as down the group-by when neither of the above transfor- a special case of group-by where no aggregates are com- mations apply. We will discuss such generalized trans- puted and the group-by is on all columns of the projec- formations as well. tion list. It has been recognized [DGK82, PHH92] that pushing down duplicate elimination past join could re- 1.3 Optimization sult in saving the cost of processing Select Distinct The transformations that push down the group-by op- queries for Select-Project-Join expressions. However, erator past joins must be judiciously applied depending as in pushing down group-by, the decision to push on the query and the database. The following exam- down duplicate elimination interacts with the order- ple shows that such transformations cannot be used ing of joins. Therefore, our optimization algorithm blindly. for group-by applies to the problem of placing dupli- cate elimination operators in the execution plans for Example 1.2 : Consider Example 1.1 again. As- Select Distinct queries that may not even have ex- sume that every product has a large number of plicit group-by clauses. 355 1.4 Related Work are called grouping columns of the query. The func- tions {AGGI, ..AGGn} are called the aggregating func- In a recent paper pL93], Yan and Larson identified a tions of the query. For the purposes of this paper, we transformation that enables pushing the group-by past will assume that every aggregate function has one of joins. Their approach is based on deriving two queries, the following forms: Sum(colname), Hax(colname) or one with and the other without a group-by clause, from Min(colname).