Linear Scalability on Decision Support Systems: Cray CS6400 Brad Carlile, Cray Research, Inc., Business Systems Division 1 INTRODUCTION The nature of DSS systems is to iteratively refine and define new queries based on the information gathered from previous Decision Support Systems (DSS) manipulate and analyze queries. These queries are ad hoc and unpredictable. It is diffi- information to highlight previously un-explored relationships in cult to pre-plan for these types of queries since they are only large Gigabyte- or Terabyte- sized databases. They are executed once and a query may access millions or billions of emerging as an area of strategic value to many customers in the rows [1]. With ad hoc queries there is no perfect data layout, merchant RDBMS market who need to explore all of their data. especially when refreshing the database with inserts and updates In the past, lack of performance has prompted users to sample imbalances the original data layout. For predictable perfor- or to summarize data for DSS processing [1], however, mance on SMP systems, fine-grain distribution of data evenly sampling can hide valuable information. Insight can be gained across the disks provides equal access time to the data. Without by knowing the detail in the large database that summary data this equal time access of data, bottlenecks can degrade perfor- obliterates. Parallel processing enables completely scanning mance by orders of magnitude and serialize the processing. large database for this accurate detailed information in a reason- Such performance problems are a defining characteristic of able time. MPPs [6] [8]. Alternatively, high-performance SMP systems New powerful Solaris SMP systems, such as the are very flexible and capable. 64-processor CRAY SUPERSERVER 6400 (CS6400), provide practical platforms that are scalable and flexible enough to A characteristic of DSS applications that take advantage of handle large databases. The CS6400 is Cray's SPARC-based parallel processing is the ability to divide a single query into SMP System that runs the Solaris 2.4 operating system and is sub-queries. Executing in parallel keeps both processors and fully SPARC binary compatible. The combination of the disks active reducing execution time. In Oracle, these CS6400 and the parallel features of Oracle7 provide scalable sub-queries execute on multiple "Query Servers" in parallel and performance for DSS operations. This paper presents results provide results to a Query coordinator that combines results as that show near-perfect linear scalability on many of the basic required by the query. Parallelism, such as on the CS6400, tasks representative of parallel DSS queries (full table scans, provides a cost-effective approach to meeting typical DSS nested loop joins, sort merge joins, index creation, etc.) using performance requirements on large databases. Oracle7 on the CS6400. These basic tasks are the components Internal and external data movement is critical to perfor- of more complex queries. Optimization of these components mance and is often an overlooked characteristic of RDBMS improves "real world" user performance. The key to high deliv- operation. This is becoming more critical as the gap between ered performance on DSS is taking advantage of the important processor speed, memory speed, and disk speed grows [3]. application characteristics and focusing on the important system Optimal disk reference patterns are quite different in OLTP parameters in the complete system. applications then they are in DSS applications. Typical OLTP 2 DSS CHARACTERISTICS disk accesses are totally random and are typically in the 2 Kbyte to 4 Kbyte size. The majority of OLTP transactions only need An understanding of an application's characteristics is impor- data from only a few rows in a few tables. The appropriate tant when implementing a balanced system. Several character- performance metric for these small reads and writes is istics of DSS dictate the need for different tuning strategies than IOs/second. In contrast, many important operations in the DSS are applied to traditional OLTP applications. Important aspects applications tend to read many consecutive rows of a particular of DSS operation are the ad hoc nature of the queries, the paral- table (table scans, aggregates, group-bys, joins, etc.). To opti- lelism that can be applied to queries and the data movement mize IO for DSS, the disk accesses issued by the RDBMS patterns within the hardware system. These aspects, when prop- should be very large, up to 1 Mbyte or more, and consecutive. erly measured, can shed additional information on obtaining Under these characteristics, the appropriate performance metric high performance on DSS operations. for these large reads is Mbytes/second. Within a single RDBMS, it is possible to tune it to implement OLTP transac- Copyright © Cray Research Inc. All rights reserved. tions with small-sized IOs and to implement DSS-style queries with large-sized IOs. For instance in the Oracle RDBMS, the CUG 1995 Fall Proceedings 61 "db_file_multiblock_read_count" parameter allows the DBA to We suggest that the best manner to look at scalability is the set a larger read size for DSS style queries. Currently, we speedup on a particular number of processors or the percentage believe that delivered application disk performance for large of parallelism. The only manner to accurately compare system DSS databases should be on the order of the hundreds of scalability is to use actual performance. The maximum number Mbytes/sec. of processors on a system does not determine its scalability. The percentage of an application that is parallel and the overheads 3 Measuring DSS Performance involved in using multiple processors limit delivered parallel performance. Parallel performance beyond a given number of During the tuning process it is necessary to establish the char- processors can be estimated using a formula based on Amdahl's acteristics of a DSS application by measuring its performance. law [3]. This estimate involves determining the percentage of a Appropriate performance metrics measure the important charac- job that is parallel and predicts the speedup for a given processor teristics of a particular operation. In addition, they provide a count using "percent parallel". reasonable prediction of performance when changing the dimen- To determine the percentage of a job that is parallel (P= sions of the database. Appropriate measures of DSS perfor- percent parallel), a one-processor run and a 40-processor run mance are MB/sec delivered during a query and the percentage (full table scan with aggregates) will be used to calculate percent of the job that is parallel. Some performance metrics do not parallel. allow accurate comparisons between different implementations and should not be used to predict performance. P = (1/observed_speedup-1)/(1/n-1) (1) The MB/s figure for a particular query will be a good predictor of the performance since a portion of most DSS queries where n is the number of processors and speedup is the observed consist of large consecutive IO operations. During these opera- speedup. For example, if a 40 processor can get a 39.083x tions, entire rows move from disk to memory even when speedup over one processor, then the percent parallel is: P = accessing a particular column of the row. The best characteriza- (1/39.083-1)/(1/40-1) P =.99939 or 99.939% parallel. To tion is the time it takes to move this data from disk and process predict other speedups, use the following formula: it (MB/s=size of the table/time to scan the table). This disk transfer time typically dominates the computation. Given a predicted_speedup=1/(P/n+(1-P)) (2) particular query type, MB/s will be roughly constant on tables of different sizes. where P is the percent parallel and n is the number of processors. An inappropriate performance metric is millions of Using the example above (P=.99939), we get the following rows/second. The problem with this measure is that the size of table, which shows good agreement with the actual results. a row is highly dependent on the table design (a row may contain Prediction using Amdahl's Law tens, hundreds, or thousands of bytes of data). For different size N Calculation Predicted Actual tables, full table scan times may be very constant in terms of Speedup Speedup 1 actual data used 1.0 1.0 Mbytes/sec whereas Mrows/sec varies by almost 3 orders of 8 1/(.99939/8+(1-.99939)) 8.0 8.7 magnitude as is illustrated in the table shown below. Mrows/sec 16 1/(.99939/16+(1-.99939)) 15.8 14.4 24 1/(.99939/24+(1-.99939)) 23.7 23.4 is an inappropriate performance metric given the inherent vari- 32 1/(.99939/32+(1-.99939)) 31.4 32.3 ability in row size. 40 actual data used 39.1 39.1 56 1/(.99939/56+(1-.99939)) 54.2 Est. On parallel systems, a useful DSS performance metric is scal- 64 1/(.99939/64+(1-.99939)) 61.6 Est. ability. Caution should be exercised when discussing scal- Potential errors in prediction may arise from the following ability. Scalability figures can be artificially inflated by areas: crippling single processor performance and optimizing parallel performance. It is important to test the application with the • Performance limitations due to application characteristics or appropriate tuning parameters for both parallel and sequential by the IO or memory bandwidth of the system. executions. It is best to be suspicious of scalability calculations • Changing the size of a problem will usually increase the time based on best parallel runs against initial (un-tuned) single spent in a parallel region (it is very difficult to use the above processor runs. estimation for a different problem size). Comparison of MRows/Sec and Mbytes/sec Rows Bytes/Row Seconds Rows/Sec Mbytes/Sec 2,000,000 2000 66 0.03 Mrows/s 61 MB/s 5,000,000 200 16 0.30 Mrows/s 63 MB/s 10,000,000 50 9 1.20 Mrows/s 57 MB/s 200,000,000 150 470 0.42 Mrows/s 64 MB/s 62 CUG 1995 Fall Proceedings • Estimates can be very low if more parallelism exists by tun- allow the RDBMS to exploit performance features of Solaris 2, ing the code (improving performance of serial section or such as the multi-threaded architecture of the Solaris kernel, making more of it parallel).