OLAP Expressions Are an Extremely Powerful Tool in SQL That Enable

OLAP expressions are an extremely powerful tool in SQL that enable advanced reporting features such as ranking, counting, averaging, adding, and more within a set of data processed in an SQL statement. This feature allows for data to be aggregated based upon values in a query in a manner very similar to coding control breaks in a program process. This allows for entire programs, or even applications, to be replaced by much more flexible and portable SQL statements. Reduce programming time and complexity, and improve flexibility and performance, by deploying OLAP expressions. This session will show you how!

1 2 3 If you’re keep up with IT news in recent times you’ll easily agree that analytics is a hot topic. The amount of data stored in our operational systems is increasing on a daily basis, and management is quickly learning that this information can and should be quickly harnessed in order for the business to make quick decisions concerning things such as sales directions, talent acquisition, cost containment, and more! One of the biggest challenges is to formulate answers to these questions that utilize the most current information, are inexpensive and easy to create, and can deliver the answers quickly. Many times great expense is incurred in moving data, creating data warehouses, and using specialized software to produce various reports. In addition to this, many times these reporting tools issue complex and redundant SQL to the data server that can result in excessive reporting costs.

Having OLAP functionality built into the DB2 engine can help reduce some of the operational and software costs associated with getting answers to complex questions. This functionality can be used in data warehouses, but also against OLTP databases with equal results. One more tool in the IT department’s tool box for answering complex business questions.

4 Analytics is a widely growing segment of database (and non-database) processing. DB2 has the ability to perform analytics via built-in expressions. Once again, this means that instead of purchasing an expensive product, or writing thousands of lines of code, you can simply write an SQL statement that does the processing for you and creates output that is report ready!

This type of processing is called Online Analytical Processing, OLAP. The constructs within the DB2 engine can be referred to as:

• OLAP expressions • OLAP specification • OLAP functions • Window functions

5 DB2 provides for several OLAP specific functions, as well as a host of aggregate functions in support of OLAP expressions. Each of these functions returns a scalar result to the row being processed. The operations supporting OLAP processing can process a single row, multiple rows, or an entire result set in the calculation of the scalar value returned.

A feature of this type of processing is the window. This window is a logical grouping of data within the result set, and the default window is the entire result set. Within a window OLAP processing can number or rank rows based upon an ordering. In addition, aggregation of values within an entire window or via a grouping within a window can be performed. Multiple OLAP functions can be specified in a SELECT clause mixing numbering, ranking, and aggregation. This results in some extremely powerful and flexible data analytics within the SQL language.

6 The key aspects to OLAP processing are the concepts of windowing and ordering. As stated before a window is a portion or grouping of the data in the result set. If no window is specified then the default window is the entire result set, and any ordering is applied to the entire result. If a window is specified then any ordering is within that window, and thus any calculations are based only upon the data in that window. You can specify many OLAP expressions in a single query, each of which can have its own independent windowing and ordering.

7 The first OLAP expression to explore is the numbering specification. Row numbering is the easiest concept to understand as it does exactly what its name implies, numbers rows in the output. Since windowing and ordering can be applied to row number, it is the perfect function to use to learn about these features since numbering is extremely easy to understand.

Numbering is enabled via the ROW_NUMBER() function. There are no parameters to this function. One extremely important thing to remember is that row numbering is arbitrary to the final ordering of the result. You can number within windows and you can also apply an order to the numbering. However, the numbering itself is done arbitrarily. Despite the limited functionality this function can be extremely useful for things such as determining the minimum and maximum row according to an order, data sampling, and pagination (although there are some performance implications).

8 OLAP specification is best taught by example. Let’s start first with a simple process and add to it as we go along.

OLAP specification allows for numbering of the result set. This numbering can be according to a specified order, or not. It can also be applied to something called a “partition” or “window” of the result table. The entire result set can be a window, and that’s what is happening in this example.

Here we are selecting data from the employee table, returning the lastname and salary of our employees. We’ve specified that the result will be ordered by the lastname column. We’ve also specified the ROW_NUMBER() window function in the final SELECT of the statement. The ROW_NUMBER() function tells DB2 that the output row is to be numbered according to the ordering applied to the function, starting with the number 1 and continuing by adding 1 to the number for each additional row returned. If no ORDER BY is specified in the window then the numbering is arbitrary with respect to the order of the result table. Here specifically we said:

ROW_NUMBER() OVER()

We have specified no window and no ordering, and so the rows are number arbitrarily in the result set. The ORDER BY clause of the final SELECT (the only SELECT in this example) has no meaning for the numbering. So don’t be fooled by a coincidental numbering in the order of the result.

9 In this example we have specified:

ROW_NUMBER() OVER(ORDER BY SALARY DESC)

There is no window specified and so the numbering is over the entire result set. However, we have specified the order in which the rows are to be numbered in the result set. So the rows are numbered in the entire result set in the order of the SALARY column by descending value. Each row returned gets a number one greater than the previous row. Also notice that the ORDER BY clause of the final result table is dictating an order by LASTNAME. So the numbering is in the different sequence (SALARY DESC) than the result set (LASTNAME ASC).

Already it’s becoming clear that we can create some outstanding reports simply from SQL. Cool!

10 In this example we have numbered the result over the entire result set, and so our window is the entire result table. We have numbered according to the SALARY column descending, and also ordered the result by the SALARY column descending. So our result table is in the same order as the numbers.

11 This example demonstrates a numbering of the entire result set over one order (SALARY DESC) and the ordering of that result set in a different order (WORKDEPT ASC, SALARY DESC).

12 It’s critical to the understanding of OLAP processing to understand the idea of windows, keeping in mind that windows can also be called partitions or groups. Basically a window is a logical grouping of data based upon a key value. That key value is determined by the specification of one or more expressions derived from the columns of the table or tables referenced in the FROM clause. For example:

PARTITION BY WORKDEPT

Will create one window for each department in the employee table. The window function being applied is then applied inside each window defined by each key value. Any ordering specified within the expression is applied within the scope of each window. In the following example the ordering of employees within a department will be by the date they were hired

PARTITION BY WORKDEPT ORDER BY HIREDATE

13 In this example partitioning, also called windowing, has been introduced. In the specification of what the numbering will be over is:

OVER(PARTITION BY EMP.WORKDEPT ORDER BY EMP.SALARY DESC)

This tells DB2 that the result table is to be divided up by the values of the WORKDEPT column and within each of those “windows” the numbering of the rows will be based upon the SALARY column in descending sequence. So, the numbering is no longer over the entire result set, but instead it is established afresh inside each partition or window.

The result table is also ordered by the same two columns in the same sequence as specified by the ORDER BY clause of the final SELECT (the only SELECT in this case). So the numbering of the rows appears consistent with the ordering of the output.

The numbering of the output is simply that. There is no respect to the data in the result table and the next number is simply 1 more than the previous row within the window. So, even though Nicholls and Natz have the same salary they do not receive the row number.

14 Ranking differs from numbering in that if two or more rows within the window are not distinct they will receive the same rank. So, while numbering is based upon the number of rows that precede the current row, ranking is based upon the number of rows that strictly precede the current row. That is, a rank represents the number of rows that precede the current row based upon the values as defined in the ordering within the window. Thus, if two or more rows have the same set of values (are not distinct from each other) then there will be gaps in the ranking.

In our example here within the window represented by department number C01, Nicholls and Natz have the same salary. Thus the number of rows that strictly precede them is 2 (Kwan and Quintana) and thus they both get the rank of 3. If there was another single person in the department with a lower salary than Nicholls and Natz then their rank would be 5 as the tie between Nicholls and Natz created a gap.

15 DENSE_RANK() works much like RANK(), except that it closes the gaps that RANK() would otherwise create. In this example we did not specify a window and so the entire result set is the window. The first query ranks the people working for the company by salary descending. As you can see in the result Nicholls and Natz are tied with the same salary within the window with a rank of 11, and so the next rank assigned to Jones is 13 due to the gap caused by the tie. The DENSE_RANK() window function will close these gaps, and so in the second example Jones receives a rank of 12.

16 Let’s take a look at the power of OLAP processing applied to a common business activity…to determine which employee is offered the voluntary separation package. The boss is interested in offering early retirement to the oldest employees, but is also interested in saving the company as much money as possible. So, he’d like to see first the employees that are oldest along with those that are highest paid. A complete list of employees is desired and so we can do that in a single query using OLAP expressions. This query here lists all employees and ranks them in two ways, by birthdate ascending and by salary descending. There are no windows so the application of these rankings are across the entire set of employees.

17 In the result set the employees have been ordered by their age and so the age ranking correlates to the result order. The salary ranking doesn’t correlate to the age ranking as well as the boss had hoped, and so this first run at finding the oldest and highest paid employee is not bearing enough fruit to make a decision.

18 In response to the lack of decision making information in the previous request, the boss comes back and requests more information. In addition to the rankings company wide, rankings by department are also desired. Perhaps there are highly paid older employees relative to those other employees in each department that can be offered the package. In response two new OLAP expressions are added to the existing query. These new expressions include windowing by the department so that the ordering and ranking are applied for each value of the department code. The powerful OLAP processing built into DB2 can process distinct windows within the same statement.

The boss doesn’t like a really cluttered report, and that last report had too many numbers. So, the query containing the OLAP expressions is placed in a nested table expression and the result is filtered to return only the highest paid and oldest employees in each department. While the equivalent filtered result could be returned using subqueries it would be far more complicated and potentially more expensive to do so.

19 Now the boss is excited! This looks like a pretty good list of potential candidates for the package. Some bosses would pull the trigger at this point, but the smarter ones may realize that there is a catch. How many employees are actually in these departments? We’d hate to lay off everyone in a department so we better make sure we’re covered before letting the axe fall.

20 New to DB2 10 for z/OS, but available on all currently in service versions of DB2 for LUW, aggregate OLAP functionality takes this type of processing and reporting to a new level. Common aggregate functions can be applied within windows to enable complex and diverse reporting simply by running SQL statements. To add further dimension to this type of reporting is the concept of aggregation groups which allow for further refinement of aggregation within a window.

This really opens the door to the ability to create complex reports with varying degree of analytics incorporated into the SQL statement!

21 Let’s take a look at our previous example of the company that is looking to reduce their employee headcount but do it in somewhat of an intelligent manner by looking for the older and higher paid employees in each department such that an early retirement package can be offered. The previous incarnation of this query used OLAP functionality to find the top two employees in each department ranked by ages and salary. Then it filtered to return any employee that fell into the range of the top two for age and salary. However, there was a piece of critical information lacking from that query. What if a department that had employees designation for the package had only one or two employees? This question can be easily answered by adding yet another OLAP expression to the existing query. In this example a COUNT function has been added, specifying a window based on the department column. This basically returns an employee count for each department much like if a separate statement was issued such as:

SELECT WORKDEPT, COUNT(*) FROM EMP GROUP BY WORKDEPT;

The difference is that this information is returned row by row along with the results of the other OLAP expressions (which could also be accomplished using a scalar fullselect in the SELECT list). Nonetheless, the addition of one relatively simple OLAP expression gets the needed information into the report.

22 This final report contains critical information relative to the impact of employee headcount reduction on a department level. All in one statement!

23 In the case of aggregate functions in an OLAP expression there can be further refinement of the range of values used for the computation within a window. This grouping is specified using either the RANGE or ROW keywords to specify the range over which the aggregation is applied. This enables “moving” values inside the window.

24 It’s important to understand how the aggregation group is controlled depending upon a number of rows or a range of rows. The ROWS keyword is used to designate that the set of rows to base an aggregation upon is a count of the number of rows before and after the current row being processed. It’s as simple as that. Using ROWS is most significant if the key value supplied in the ordering is distinct. That is, there is one row per unique value in the aggregation group. The RANGE keyword is used to indicate that the set of rows is not based upon counting, but instead a key value. There are significant restrictions to the key value used in that it has to be numeric and the data type comparable to the range values provided. The RANGE keyword is best if there are multiple rows per key value, as long as you can make sure that key value is numeric.

25 There are several keywords used to designation how to determine the scope of the aggregation within a window. If no grouping is defined then the scope is the entire window. If a grouping is desired than a start and end value is determined using various keywords that designate a position relative to the current row being processed. It’s best at this point to think of the processing of the SQL statement as a program loop, where that loop is processing the set of rows in a window one at a time.

The choices for determining the group are either unbounded, meaning no limit from the start or end of the window to the current row, or a certain set of rows either before or after the current position. The number of rows designated before or after the current position is dependent upon the use of RANGE or ROW, and if no start or end position is specified then the position of the current row is used.

26 I, being a home brewer and DB2 consultant, have combined both of my passions and have started recording some of my brewing activities in DB2. My first effort into this has been to set up a simple table that contains the date of a brew, the name of the beer, the style of beer, and the quantity of beer brewed in gallons. This simple record can be analyzed to determine various trends in brewing, as well as how much of each beer has been brewed. In this example I wanted to simply demonstrate the difference between using ROW and RANGE in an aggregation within a window. This query produces multiple totals of beer brewed over two different windows. The first expression totals the quantity of beer brewed by month. Since there are several years of brewing recorded each month total reflects multiple years. The second two totals are calculated over a window that is the entire result set. The first of these totals the current brew date total as well as the previous brew date. Provided that there is NOT more than one beer brewed per day then this total reflects the total of the last two brews. The last total is ordered by the month of brewing and totals the current and previous month value. RANGE is used because there are multiple rows per month and the number of those rows are unknown.

Two additional things to note here is that there is a lack of the FOLLOWING clause in the aggregation group. This means that the end point of the group aggregation is the current row. The second thing to note is that since RANGE is used the key value in the ORDER BY specification has to be numeric, as is the result of the MONTH(BREW_DATE) function invocation.

27 This result demonstrates row based aggregation versus range based aggregation. While the row based is simply a calculation based upon the current and previous row, the range based uses the value comparison to determine which rows to aggregate.

28 This example is from the DB2 for LUW sample database. It uses aggregation to calculate the average sale quantity company wide on a month-by-month basis. So, a window is established for each month of sales, and the AVG function is used to calculate the average sale quantity for each month. A SUM function is also used to calculate total sales. There is no window specified for this total, but an aggregation group has been specified designating an unbounded start (from the start of the window) and the current row. This provides a running total of sales in the result.

A variety of functions, windows, and grouping can be specified to produce all sorts of running values!

29 This result produces a list of sales from our company’s sales table. Along with the list of every sale are some important key metrics that will help our marketing team focus attention. The first is the average sale by month. That is, for each month in which my sales data span show me the average sale for that month. So, along with the list of sales is the average sale amount for the month. The second is a running total of sales over the entire result set of data. So, in one query we have detailed information along with important metrics regarding that detailed information. Sweet! This can inspire to create some extraordinary reports using SQL. Let’s break down our OLAP expressions: The first produces an average

CAST(AVG(CAST(SALES AS DEC(3,1))) OVER ( PARTITION BY YEAR(SALES_DATE), MONTH(SALES_DATE)) AS DEC(3,1)) The casting is because SALES is actually an integer and we need a decimal result.

OVER(PARTITION BY YEAR(SALES_DATE), MONTH(SALES_DATE)) tells DB2 that the average will be calculated within a window that is based on the year and month of the sales date. So, calculate the average sale for each month! No ordering is needed within the window because it’s a single value being calculated for each window. The second produces a running total:

SUM(SALES) OVER (ORDER BY SALES_DATE ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) There is no partitioning clause and so the window is the entire result table. The SUM function indicates a calculated total over the entire window ordered by the SALES_DATE column. The window-order-clause is followed by a window-aggregation- group-clause, ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW Which tells DB2 that the group of values to sum will be based upon all preceding rows up to the current row. The corresponding ORDER BY in the final SELECT makes sure that the running total makes sense.

30 This query is valid on both DB2 for z/OS and DB2 for LUW. It takes the time recorded by employees working on projects and calculates the average time entry quantity recorded each month, as well as the running total of all time recorded.

31 32 OLAP processing can be a performance gain or a detriment to performance. As with anything else the query performance is relative to the task accomplished and the alternatives to accomplishing the same tasks by other means:

1. All data returned and program loops 2. Multiple queries issued by the program each collecting a different value 3. Using more traditional SQL, including grouping, subqueries, table expressions, etc. 4. The portability of the report process (some programming languages might not be very portable across platforms) 5. The amount of time it takes to code the solution

For OLAP expressions themselves there are potentially big workfile and sort consumers within DB2. So, you need to make sure that there are adequate shared resources (workfile, CPU, I/O) to handle the OLAP queries. Using the EXPLAIN facility and running benchmarks are critical to understanding the impact of OLAP processing.

33 This explain (DB2 for z/OS) shows a relatively simple OLAP query and two sorts and workfiles allocated in support of the query. One sort is in support of the OLAP expression and the second is to order the result set.

Be mindful of the fact that on DB2 for z/OS workfiles are allocated as needed, but are not released until the query terminates (except for child tasks in parallel queries) so workfile utilization can be significant. On DB2 for LUW they workfiles can be truncated when no longer needed, as controlled by the DB2_SMS_TRUNC_TMPTABLE_THRESH environment variable. As additional OLAP expressions are added to a query, additional sorts and workfile allocations will occur.

34 As we added OLAP expressions DB2 can also add sorts and workfile allocations in response. In this query there are two sorts in support of the two OLAP expressions, and one sort in support of the final result. This is true for DB2 for z/OS and DB2 for LUW.

35 OLAP processing cannot only be a choice for business analytics and reporting, but could also be a potential performance improvement for OLTP and/or batch transaction processing. In this example here we can see two queries that return the most recent accounting history record for a history table. The original query uses a correlated subquery to find the most recent row for each primary key. Since there is no filtering by key value then the entire table is processed and the correlated subquery executed for each row in the history table. In comparison, the equivalent OLAP query in a nested table expression is used to read the entire table once and rank the rows. Subsequent filtering returns the same result as the correlated subquery example. Which is the better performer? Well, that depends of course, but in this example the OLAP query was a significant performance improvement over the subquery.

36 Here is an example of the same situation using the DB2 sample database. The queries are finding the oldest employee in each department, one via correlated subquery and the other using OLAP.

37 An EXPLAIN of the subquery solution shows the execution of the correlated subquery in a separate query block. This separate query block is executed once per row processed in the outer portion of the query (query block one), but no sorts or workfiles are utilized.

38 An EXPLAIN of the OLAP based solution shows two query blocks, but in this case there is only a single execution of each block. The second query block shows the table being read and sorted to perform the OLAP processing. The first query block shows the workfile being read to produce the final result.

Which is better? I don’t know! Try them both, explain them, and benchmark them.

The thing to keep in mind is exactly how much data is being processed. If a lot of data is going to be processed and there is little or no filtering then the OLAP solution may be better provided there are adequate workfile resources available. If there is significant filtering of the data in the outer portion of the query and/or workfile availability is limited, then the subquery solution should be better given appropriate indexing.

39 Here is another great example of using the power of OLAP processing to retrieve some sample data. In this particular case a sample of employee data is desired base upon certain rules. The rule applied here is that a sample of two employees per department is desired. Rather than running a series of queries or a single query with complicated subqueries, only one OLAP query can be used to get the desired result.

40 OLAP processing in DB2 is extremely powerful, and an important tool that can be utilized quickly to perform complex data analytics. The OLAP specification takes some time to get used to, and so you need to reserve some time for programmer education and experimentation. Once this knowledge is acquired this type of processing can be used to quickly answer complex business questions, but can also be a performance advantage for certain types of reports. This is especially true in situations where several queries can be replaced by a single query.

41 Dan Luksetich is a senior DB2 DBA consultant. He works as a DBA, application architect, presenter, author, and teacher. Dan has been in the information technology business for over 28 years, and has worked with DB2 for over 23 years. He has been a COBOL and BAL programmer, DB2 system programmer, DB2 DBA, and DB2 application architect. His experience includes major implementations on z/OS, AIX, i Series, and Linux environments. Dan's experience includes: Application design and architecture, database administration, complex SQL, SQL tuning, DB2 performance audits, replication, disaster recovery, stored procedures, UDFs, and triggers. Dan works everyday on some of the largest and most complex DB2 implementations in the world. He is a certified DB2 DBA, system administrator, and application developer, and has worked with the teams that have developed several DB2 for z/OS certification exams. He is the author of several DB2 related articles as well as co-author of the DB2 9 for z/OS Certification Guide and the DB2 10 for z/OS Certification Guide.