Performance Improvements to LAPACK for the Cray Scienti c Library y Mark Fahey Edward Anderson Department of Mathematics Cray Research University of Kentucky 655F Lone Oak Drive Lexington, KY 40506 Eagan, MN 55121 April 22, 1997 Abstract This rep ort details lo cal mo di cations to LAPACK routines for the Cray Scienti c Library.Performance of selected routines is evaluated and compared to that of the public domain LAPACK and the equivalent routines from LINPACK or EISPACK. Timing results from a CRAY T90 series computing system are presented. 1 Intro duction The Cray Scienti c Library [5] provides an optimized implementation of the public do- main software package LAPACK [1] for solving dense linear systems and nding eigenval- ues/eigenvectors or singular values/singular vectors of dense matrices. Most of the optimiza- tions come byway of the BLAS [8, 9, 13], a collection of Basic Linear Algebra Subprograms whichhave b een well-tuned for the Cray architectures. Many of the computationally- intensive algorithms in LAPACK, such as the LU, Cholesky, and QR factorizations and the reductions to Hessenb erg, tridiagonal, and bidiagonal form, do most of their work in the BLAS and have already b een studied in detail [3, 6, 10]. These algorithms are well-suited to the Cray architectures, and their implementation in the Cray Scienti c Library di ers little from that of the public domain LAPACK. The LAPACK routines employed after a factorization or reduction has b een completed have attracted less attention from p erformance optimizers, and it is here where most of the lo cal improvements have b een made. This rep ort discusses p erformance improvements for the Cray Scienti c Library to a wide range of LAPACK routines, including: triangular solve routines in linear system solving, auxiliary routines for computing elementary re ections and rotations, balancing for the nonsymmetric eigenvalue problem, symmetric and nonsymmetric inverse iteration, the QZ algorithm for the generalized nonsymmetric eigenvalue problem. [email protected] y [email protected] 1 Many of these mo di cations are algorithmic improvements that would b e b ene cial on archi- tectures other than those considered in this rep ort. Other more Cray-sp eci c enhancements, such as selective inlining of BLAS, will b e mentioned brie y for completeness. An interest- ing but otherwise irrelevant empirical result on the optimal blo cksize for blo ck factorization algorithms is included in an app endix. The metric for much of this optimization work was the output generated by the timing programs distributed with the LAPACK test package [4]. These programs show the p er- formance of standard LAPACK routines against that of the equivalent EISPACK routines, inviting comparison. We observed initially that some of the EISPACK routines were faster than the LAPACK routines that were supp osed to replace them. We also found instances in which the Cray Scienti c Library version of EISPACK was faster than the public do- main version of EISPACK, and worked to identify those optimizations and transfer them to LAPACK. In x 2, we describ e the computing environment and identify the di erent libraries com- pared in this rep ort. In x 3 and x 4we review all the lo cal mo di cations to LAPACK library routines. Changes to the algorithm or lo op structure are called out in separate subsections; changes that involved only inlinin g are collected at the end. We conclude with followup remarks in x 5. 2 Preliminaries 2.1 Compute environment Our compute environment consisted of a CRAY T94, a parallel-vector pro cessing PVP computing system with four pro cessors, a shared memory space of 128 MWords, and IEEE oating-p oint arithmetic. The cycle time is 2.222 nsec 450 MHz, and the vector oating- p oint units can retire four op erations p er clo ck, giving a p eak rate of approximately 1800 M ops p er pro cessor. Although the CRAY T94 is a parallel pro cessor, most of the algo- rithms weevaluated are serial, so most of our results are for a single pro cessor. 2.2 Libraries We will compare the p erformance of our enhanced version of LAPACK to the public domain version, as well as to LINPACK[7] and EISPACK[16], which are earlier collections of Fortran 77 subroutines for solving linear systems, eigenvalue problems, and singular value problems. The following conventions will b e used in referring to the di erent libraries: libsci is the name of the Cray Scienti c Library where the mo di cations describ ed here have b een implemented LAPACK in this rep ort refers to the public domain source co de for LAPACK 2.0, available from netlib at http://www.netlib.org. LINPACK refers to the public domain source co de for LINPACK. EISPACK refers to the mo di ed source co de for EISPACK supplied with the LAPACK timing routines. Further details on the mo di cations to EISPACK are given b elow. 2 libsci EISPACK is a unique hybrid of an early version of EISPACK with Cray library enhancements and the algorithmic changes from the 1983 up date to EISPACK. Al- though this software is b eing phased out of libsci, we include it for comparison in a few cases where it is still the fastest alternative. The naming convention for LAPACK routines in the Cray Scienti c Library uses a leading `S' in the routine name to indicate a subroutine op erating on 64-bit real data and a leading `C' for 64-bit complex data. On many other systems, 64-bit data is double precision, and the equivalent precision library routines would have a leading `D'or`Z'in their name. For example, the subroutine to compute an LU factorization of a 64-bit REAL array is named SGETRF in the Cray Scienti c Library but DGETRF in SGI's CompLib. All comparisons in this rep ort are for subroutines op erating on full precision 64-bit real or complex data. 2.3 Mo di cations to EISPACK One diculty in comparing LAPACK to EISPACK is that there are many test problems in the LAPACK timing suite for which EISPACK do es not converge or converges with less accuracy than LAPACK. To remedy this, the LAPACK designers provided a mo di ed version of EISPACK with the LAPACK timing suite in which the convergence criteria were changed to b e more like LAPACK. Sp eci c changes include The maximum numb er of iterations allowed for convergence was increased from 30 to 40 in IMTQL1 and IMTQL2. The test for determining if o diagonal elements are small enough for the matrix to b e split was relaxed in BISECT and TRIDIB. In EISPACK, the criterion was to split the matrix if jE ij " jD ij + jD i 1j : This condition implies that jE ij 2" max jD ij ; jD i 1j,soifbothD i and D i 1 are less than UNFL=2", where UNFL is the under ow threshhold, then E i must b e zero to split. In LAPACK, the criterion is to split the matrix if 2 2 jE ij <" jD ijjD i 1j + SAFMIN; where SAFMIN is the smallest representable numb er in the oating p oint mo del the \safe minimum". Thus jE ij may b e greater than " jD ij if it is suciently smaller p than " jD i 1j, and the matrix will always split if jE ij < UNFL, regardless of the values of D . Global matrix information was added to the splitting criterion for determining if o diagonal elements are small enough in COMQR, COMQR2, HQR, and HQR2. For instance, when lo oking for a single small sub diagonal element in HQR, the test was jH i; i 1j "S where S = jH i 1;i 1j + jH i; ij,or S = kH k if this value is zero. This test was changed to jH i; i 1j max"S; r 3 where r = maxSAFMIN;"kH k. This allows the matrix to split if H i; i 1 is small either in an absolute sense or relative to the norm of the matrix H ,even if it is not small relative to its immediate diagonal neighb ors. Previously, global matrix information was incorp orated only if H i 1;i 1 and H i; iwere zero. The absolute test for small o diagonal elements in TQLRATwas replaced with a 2 2 relative test. Instead of checking for values of E 2i <" , where E 2i= E i , the p new test checks for E 2i <"jD ij + jD i +1j. It is debatable whether or not EISPACK with all these changes is really EISPACK any more, but this is the metho dology that has b een established for comparing LAPACK to EISPACK. 2.4 Test Matrix Typ es Some of the eigenvalue routines studied in this rep ort have di erent p erformance character- istics for di erenttyp es of matrices. This is b ecause their rate of convergence dep ends on the separation of adjacent eigenvalues and on whether or not there are rep eated eigenvalues in the solution. In the timing results of x 4, we rep ort the matrix typ e along with the timing data. The enumeration of matrix typ es follows that used in the LAPACK timing program, which is describ ed in further detail in the Instal lation Guide for LAPACK [4]. If no typ e is indicated, matrices of type1were used. 3 Mo di cations to LAPACK, I: Linear System Solving The mo di cations wehave made to subroutines in the LAPACK library are extensive and so wehave divided them into two groups.