A Parallel Divide and Conquer Algorithm for the Symmetric Eigenvalue Problem on Distributed Memory Architectures F Tisseur and J Dongarra 1999 MIMS EPrint: 2007.225 Manchester Institute for Mathematical Sciences School of Mathematics The University of Manchester Reports available from: http://www.manchester.ac.uk/mims/eprints And by contacting: The MIMS Secretary School of Mathematics The University of Manchester Manchester, M13 9PL, UK ISSN 1749-9097 SIAM J. SCI. COMPUT. c 1999 Society for Industrial and Applied Mathematics Vol. 20, No. 6, pp. 2223–2236 A PARALLEL DIVIDE AND CONQUER ALGORITHM FOR THE SYMMETRIC EIGENVALUE PROBLEM ON DISTRIBUTED MEMORY ARCHITECTURES∗ FRANC¸ OISE TISSEUR† AND JACK DONGARRA‡ Abstract. We present a new parallel implementation of a divide and conquer algorithm for computing the spectral decomposition of a symmetric tridiagonal matrix on distributed memory architectures. The implementation we develop differs from other implementations in that we use a two-dimensional block cyclic distribution of the data, we use the L¨owner theorem approach to compute orthogonal eigenvectors, and we introduce permutations before the back transformation of each rank-one update in order to make good use of deation. This algorithm yields the first scalable, portable, and numerically stable parallel divide and conquer eigensolver. Numerical results confirm the effectiveness of our algorithm. We compare performance of the algorithm with that of the QR algorithm and of bisection followed by inverse iteration on an IBM SP2 and a cluster of Pentium PIIs. Key words. divide and conquer, symmetric eigenvalue problem, tridiagonal matrix, rank-one modification, parallel algorithm, ScaLAPACK, LAPACK, distributed memory architecture AMS subject classifications. 65F15, 68C25 PII. S1064827598336951 1. Introduction. The divide and conquer algorithm for the symmetric tridiag- onal eigenvalue problem was first developed by Cuppen [8], based on previous ideas of Golub [16] and Bunch, Nielsen, and Sorensen [5] for the solution of the secular equation. The algorithm was popularized as a practical parallel method by Dongarra and Sorensen [14], who implemented it on a shared memory machine. They concluded that divide and conquer algorithms, when properly implemented, can be many times faster than traditional ones, such as bisection followed by inverse iteration or the QR algorithm, even on serial computers. Later parallel implementations had mixed success. Using an Intel iPSC-1 hypercube, Ipsen and Jessup [22] found that their bisection implementation was more efficient than their divide and conquer implemen- tation because of the excessive amount of data transferred between processors and unbalanced work load after the deation process. More recently, Gates and Arbenz [15] showed that good speed-up can be achieved from distributed memory parallel implementations. However, they did not use techniques described in [18] that guar- antee the orthogonality of the eigenvectors and that make good use of the deation to speed the computation. In this paper, we describe an efficient, scalable, and portable parallel implemen- tation for distributed memory machines of a divide and conquer algorithm for the ∗Received by the editors April 6, 1998; accepted for publication (in revised form) October 15, 1998; published electronically July 16, 1999. This work was supported in part by Oak Ridge National Laboratory, managed by Lockheed Martin Energy Research Corp. for the U.S. Department of Energy under contract DE-AC05-96OR2246 and by the Defense Advanced Research Projects Agency under contract DAAL03-91-C-0047, administered by the Army Research Office. http://www.siam.org/journals/sisc/20-6/33695.html †Department of Mathematics, University of Manchester, Manchester M13 9PL, England (ftisseur@ ma.man.ac.uk). ‡Department of Computer Science, University of Tennessee, Knoxville, TN 37996-1301, and Mathematical Sciences Section, Oak Ridge National Laboratory, Oak Ridge, TN 37831 (dongarra@ cs.utk.edu). 2223 2224 FRANC¸ OISE TISSEUR AND JACK DONGARRA symmetric tridiagonal eigenvalue problem. We chose to implement the rank-one up- date of Cuppen [8] rather than the rank-two update described in [15], [18]. We see no reason why one update should be more accurate than the other or faster in general, but Cuppen’s method, as reviewed in section 2, appears to be easier to implement. Until recently, it was thought that extended precision arithmetic was needed in the solution of the secular equation to guarantee that orthogonal eigenvectors are produced when there are close eigenvalues. However, Gu and Eisenstat [18] have found a new approach that does not require extended precision; Kahan [24] showed how to make it portable and we have used it in our implementation. In section 3 we discuss several important issues to consider for parallel imple- mentation of a divide and conquer algorithm, and then we derive our algorithm. We implemented our algorithm in Fortran 77 as production quality software in the ScaLAPACK model [4] and we used LAPACK divide and conquer routines [25], [27] as building blocks. The code is well suited to compute all the eigenvalues and eigen- vectors of large matrices with clusters of eigenvalues. For these problems, bisection followed by inverse iteration as implemented in ScaLAPACK [4], [10] is limited by the size of the largest cluster that fits on one processor. The QR algorithm is less sensitive to the eigenvalue distribution but is more expensive in computation and communication and thus does not perform as well as the divide and conquer method. Examples that demonstrate the efficiency and numerical performance are presented in section 4. 2. Cuppen’s method. The spectral decomposition of a symmetric matrix is generally computed in three steps: tridiagonalization, diagonalization, and back trans- formation. Here, we consider the diagonalization T = W ΛW T of a symmetric tridi- n n agonal matrix T R × , where Λ is diagonal and W is orthogonal. Cuppen [8] introduced the decomposition∈ T 0 T = 1 + ρvvT , 0 T 2 where T1 and T2 differ from the corresponding submatrices of T only by their last and T T first diagonal coefficients, respectively. Let T1 = Q1D1Q1 ,T2 =Q2D2Q2 be spectral decompositions. Then T is orthogonally similar to the rank-one update (2.1) T = Q(D + ρzzT )QT , T where Q = diag(Q1,Q2) and z = Q v. By solving the secular equation associated with this rank-one update, we compute the spectral decomposition (2.2) D + ρzzT = UΛU T and then T = W ΛW T with W = QU. A recursive application of this strategy to T1 and T2 leads to the divide and conquer algorithm for the symmetric tridiagonal eigenvalue problem. Finding the spectral decomposition of the rank-one update D + ρzzT is the heart n of the divide and conquer algorithm. The eigenvalues λi i=1 are the roots of the secular equation { } T 1 (2.3) f(λ)=1+ρz (D λ) z, and a corresponding eigenvector u is given by 1 (2.4) u =(D λI) z. PARALLEL DIVIDE AND CONQUER EIGENSOLVER 2225 Each eigenvalue and corresponding eigenvector can be computed cheaply in O(n) ops. Unfortunately, calculation of eigenvectors using (2.4) can lead to a loss of orthogonality for close eigenvalues. Solutions to this problem are discussed in section 3.3. Dongarra and Sorensen [14] showed that the spectral decomposition (2.2) can potentially be reduced in size. If zi = 0 for some i, then di = D(i, i) is an eigenvalue with eigenvector the ith unit eigenvector ei, and if there are equal di’s, then the eigenvector basis can be rotated in order to zero out the components of z corresponding to the repeated diagonal entries. In finite precision arithmetic one needs to deate when a zi is nearly equal to zero and when there are nearly equal di’s for some suitable definitions of “nearly” that ensure numerical stability is retained [14]. With suitable deation criteria, if G is the product of all the rotations used to zero out certain components of z and if P is the accumulation of permutations used to translate the zero components of z to the bottom of z, the result is D + ρz˜z˜T 0 (2.5) PG(D+ρzzT )GT P T = +E, 0Λ e where E 2 cu, with c a constant of order unity and u the machine precision. This deationk processk ≤ is essential for the success of the divide and conquer algorithm. In practice, the dimension of D+ρz˜z˜T is usually considerably smaller than the dimension of D+ρzzt, which reduces the number of ops when computing the eigenvector matrix of T . Cuppen [8] showed thate deation is more likely to take place when the matrix is diagonally dominant When no deation is assumed, the whole algorithm requires 4 3 2 3 n + O(n ). In practice, because of deation, it appears that the algorithm takes only O(n2.3) ops on average and the cost can even be as low as O(n2) for some special cases (see [9]). 3. Parallelization issues and implementation details. Divide and conquer algorithms have been successfully implemented on shared memory multiprocessors [14], [23] but difficulties have been encountered on distributed memory machines [22]. Several issues need to be addressed. A more detailed discussion of the issues discussed below can be found in [30]. 3.1. Data distribution. The first issue, and perhaps the most critical step when writing a parallel program, is how to distribute the data. Previous implementations used a one-dimensional distribution [15], [22]. Gates and Arbenz [15] used a one- dimensional row block distribution for Q, the matrix of eigenvectors, and a one- dimensional column block distribution for U, the eigenvector matrix of the rank-one updates. This distribution simplifies their parallel matrix-matrix multiplication used for the back transformation QU. However, their matrix multiplication routine grows in communication with the number of processes, making it not scalable.