Efficient Computation of Multiple Group By Queries Zhimin Chen Vivek Narasayya Microsoft Research {zmchen, viveknar}@microsoft.com ABSTRACT Group By queries are required can also be large. A naïve Data analysts need to understand the quality of data in the approach is to execute a different Group By query for each set of warehouse. This is often done by issuing many Group By queries columns. on the sets of columns of interest. Since the volume of data in In some commercial database systems, an extension to the these warehouses can be large, and tables in a data warehouse GROUP BY construct called GROUPING SETS is available, often contain many columns, this analysis typically requires which allows the computation of multiple Group By queries using executing a large number of Group By queries, which can be a single SQL statement. The multiple Group By queries are expensive. We show that the performance of today’s database specified by providing a set of column sets. For example systems for such data analysis is inadequate. We also show that specifying GROUP BY GROUPING SETS ((A), (B), (C), (A,C)) the problem is computationally hard, and develop efficient would cause the Union All of all four Group By queries to be techniques for solving it. We demonstrate significant speedup returned in the result set. The query optimizer has the ability to over existing approaches on today’s commercial database optimize the execution of the set of queries by potentially taking systems. advantage of the commonality among them. However, as the following experiment on a commercial database system shows, 1. INTRODUCTION GROUPING SETS functionality is not optimized for the kinds of Decision support analysis on data warehouses influences data analysis scenarios discussed above. important business decisions and hence the accuracy of such Example 1. Consider a scenario where the data analyst wishes to analysis is crucial. Understanding the quality of data is an obtain single-column value distribution for each character or important requirement for a data analyst [18]. For instance, if the categorical column in a relation. For example, there are 12 such number of distinct values in the State column of a relation columns in the lineitem table of the TPC-H database (1GB) [24]. describing customers within USA is more than 50, this could We used a single GROUPING SETS query on a commercial indicate a potential problem with data quality. Other examples database system to compute results of these 12 Group By queries. include the percentage of missing (NULL) values in a column, the We compared the time taken to execute the GROUPING SETS maximum and minimum values etc. Such aggregates help a data query with the time taken using the approach presented in this analyst evaluate whether the data satisfies the expected norm. paper. Using our approach was about 4.5 times faster. This kind of data analysis can be viewed as obtaining frequency The explanation for this is that GROUPING SETS is not distributions over a set of columns of a relation; i.e., a Group By optimized for scenarios where many column sets with little query of the form: SELECT X, COUNT(*) FROM R GROUP overlap among them are requested, which is a common data BY X, where X is a set of columns on relation R. Note that X may analysis scenario. In Example 1, where we request 12 single sometimes contain derived columns, e.g., LEN(c) for computing column sets only, the plan picked by the query optimizer is to first the length distribution of a column c. For example, consider a compute the Group By of all 12 columns, materialize that result, relation Customer(Lastname, FirstName, M.I., Gender, Address, and then compute each of the 12 Group By queries from that City, State, Zip, Country). A typical scenario is understanding the materialized result. It is easy to see that such a strategy is almost distributions of values of each column, which requires issuing the same as the naïve approach of computing each of the 12 several Group By queries, one per column. In addition, it is Group By queries from the lineitem table itself. In contrast, our sometimes necessary to also understand the joint frequency solution consisted of materializing the results of the following distributions of sets of columns, e.g., the analyst may expect that Group By queries into temporary tables: (1) (l_receipdate, (LastName, FirstName, M.I., Zip) is a key (or almost a key) for l_commitdate) (2) (l_tax, l_discount, l_quantity, l_returnflag, that relation. We note that some data mining applications [15] l_linestatus); and computing the single column Group By queries also have similar requirements of computing frequency of these columns from the respective temporary tables. The Group distributions over many sets of columns of a relation. By queries for each of the remaining columns were computed This kind of data analysis can be time consuming and resource directly from the lineitem table. intensive for two reasons. First the number of rows in the relation can be large. Second, the number of sets of columns over which An important observation, illustrated by the above example, is that materializing results of queries, including queries that are not Permission to make digital or hard copies of all or part of this work for required, can speed up the overall execution of the required personal or classroom use is granted without fee provided that copies are queries. It is easy to see that the search space, i.e., the space of not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy queries that are not required, but whose results, if materialized, otherwise, or republish, to post on servers or to redistribute to lists, could speed up execution of the required queries, is very large. requires prior specific permission and/or a fee. For example, for a relation with 30 columns, if we want to SIGMOD 2005 June 14-16, 2005, Baltimore, Maryland, USA. compute all single column Group By queries, the entire space of Copyright 2005 ACM 1-59593-060-4/05/06 $5.00. relevant Group By queries to consider for materialization is 230. We note that previous solutions to the above problem, including columns, the above solution does not scale, since the step of GROUPING SETS, were designed for efficient processing of the constructing the search lattice itself would not be feasible. In (partial) datacube e.g., [2,14,16,21], as a generalization of CUBE contrast, our approach constructs only a small part of the lattice in and ROLLUP queries. Unlike the kind of data analysis scenarios a bottom-up manner, interleaved with the search algorithm presented above, these solutions were geared towards OLAP like (Section 4). This allows us to scale to a much larger inputs. We scenarios with a small set of Group By queries typically having note that once the set of queries to be materialized is determined, significant overlap in the sets of columns. Hence, most of these several physical operators, e.g., PipeHash, PipeSort [14,16], previous solutions to this problem assume that search space of Partitioned-cube, Memory-Cube [16], etc. for efficiently queries can be fully enumerated as a first step in the optimization. executing a given set of queries (without introducing any new Thus, these solutions do not scale well to the kinds of data queries) can be used. This problem is orthogonal to ours, and analysis scenario described earlier. these techniques can therefore be leveraged by our solution as Our key ideas and contributions can be summarized as follows. well. The basic ideas is to take advantage of commonality across First, we show that even a simple case of this problem, where we Group By queries using techniques such as shared scans, shared need to compute n single-column Group By queries of a relation sorts, and shared partitions [2,8,15,16,21]. We note that work on is computationally hard. Second, unlike previous solutions to this efficient computation of a CUBE (respectively ROLLUP) query is problem, our algorithm explores the search space in a bottom up a special case of the partial cube computation discussed above manner using a hill-climbing approach. Consequently, in most where the entire lattice (resp. a hierarchy) of queries needs to be cases, we do not require enumerating the entire search space for computed. In contrast our work can be thought of as introducing optimizing the required queries. This allows us to scale well to the logically equivalent rewritings of a GROUPING SETS query in a kinds of data analysis scenarios mentioned above. Third, our cost based manner. algorithm is cost based and we show how this technique can be The second category of work is creation of materialized views for incorporated into today’s commercial database systems for answering queries. The paper [17] studies the problem of how to optimizing a GROUPING SETS query. As an alternative, we speed up update/creation of a materialized view V by creating show how today’s client data analysis tools/applications can take additional materialized views. They achieve this by creating a advantage of our techniques to achieve better performance until DAG over logical operators by considering various equivalent such time that GROUPING SETS queries are better optimized for expressions for V (e.g., those generated by the query optimizer); such scenarios. Finally, we have conducted an experimental and then determining an optimal set of additional views (sub-trees evaluation of our solution on real world datasets against of the DAG) to materialize.