TU M ¨unchen 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 1 TU M ¨unchen 7.4. QR Iteration • Factorization methods such as the QR iteration [Francis 1961] compute all × eigenvalues of a symmetric matrix A 2 Rn n at once. • The idea is to compute a sequence of matrices fAk g that are similar to A. This sequence should converge to a matrix in triangular form, such that the diagonal of Ak is an approximation for the eigenvalues of A. • The QR iteration is based on the QR decomposition which factorizes a given matrix A such that R A = Q * with Q an orthogonal and R an upper triangular matrix. Then the matrix RQ = Q−1AQ = QT AQ is similar to A. Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 2 TU M ¨unchen • It can be shown that the sequence fA(k)g defined by A(0) = A; A(k+1) = R(k)Q(k); Q(k) orthogonal; R(k) upper triangular; A(k) = Q(k)R(k) converges to a diagonal matrix. Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 3 TU M ¨unchen QR Iteration: Algorithm • Thus, we have the following algorithm: – A(0) := A – for k = 0; 1; 2;:::: 1. Compute a factorization A(k) = QR using QR decomposition 2. A(k+1) := RQ 3. if all non-diagonal elements sufficiently small: (k+1) output λj ≈ aj;j ; j = 1;:::; n remove row and column n • (k+1) We consider the non-diagonal element aj;m “sufficiently small” if its absolute (k+1) value is ≤ " · jaj;j j. • We still need to know how to implement the QR decomposition! Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 4 TU M ¨unchen 7.4.1. QR Decomposition • Remember: We want to decompose a matrix A into an orthogonal matrix Q and an upper triangular matrix R such that A = QR. • This problem is equivalent to the following: Find an orthogonal matrix G such that GA = R, i.e. G “zeroes out” the subdiagonal part of A. Then A = GT R. T • We first have a look at the two-dimensional case: Given a vector x = (x1; x2) , find an orthogonal matrix G such that Gx = (r; 0)T for some r 2 R. • It is easy to see that (x1 ,y 2 ) 1 x1 x2 G = q θ 2 2 −x2 x1 x1 + x2 (r,0) does the job. • Geometrically, the matrix G describes a rotation by θ = arctan(x2=x1) radians in the (x1; x2) plane. • G is called a Givens rotation matrix [Givens 1958]. Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 5 TU M ¨unchen • Generalization to the n-dimensional case: When multiplied with A, 01 ··· 0 ··· 0 ··· 01 B . .. C B . C B C B0 ··· ρ · a ··· ρ · a ··· 0C B jj ij C B . C 1 Gij = B . C ; ρ = B . C q 2 2 B C aij + ajj B0 · · · −ρ · aij ··· ρ · ajj ··· 0C B C B . .. C @ . A 0 ··· 0 ··· 0 ··· 1 affects rows i and j of A and eliminates entry aij (i > j). Gij differs from the identity matrix in only four entries. Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 6 TU M ¨unchen • If a column k is already eliminated, Gij with j > k; i > j does not destroy the zeros in column k since operations between lines i and j just result in summing up zeros in column k: 0 1 a1;1 ·················· a1n B . C B . C B 0 . C B C B . C B . .. .. C B C B . C B . C B . 0 a ; ··· a ; . C B j j j i C B . C B . C B . C B C B . C B . a ··· a . C B i;j i;i C B . C B . .. C @ . A 0 ··· 0 an;j ········· an;n Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 7 TU M ¨unchen • Thus, we can eliminate the subdiagonal non-zero elements coumn by column, starting from the first column and working in columns top to bottom. This results in the G-matrix G = Gn;n−1 · Gn;n−2Gn−1;n−2 ····· Gn;2 ··· G3;2 · Gn;1 ··· G2;1; • Since the product of two orthogonal matrices is orthogonal, G is orthogonal and GA is an upper triangular matrix for any matrix A. Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 8 TU M ¨unchen • Alternatively, we can eliminate non-zeros line by line top to bottom and left to right: G = Gn;n−1 ··· Gn;1 ··· G3;2 · G3;1 · G2;1 This is possible as Gij does not change lines k with k < j and zeros in line i in a column k < j are not destroyed: 0 1 a1;1 ·················· a1n B . C B . C B 0 . C B C B . C B . .. .. C B C B . C B . .. C B . a ; ··· a ; . C B j j j i C B . C B . C B 0 ······ 0 . C B C B . C B 0 ··· 0 a ··· a . C B i;j i;i C Ba ·················· a C B i+1;1 i+1;nC B . C B . .. C @ . A an;1 ·················· an;n Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 9 TU M ¨unchen QR Iteration: Implementation • We now have a method to compute the QR decomposition of A: R = GA, Q = GT . • The naive computation of G via n(n − 1)=2 matrix multiplications costs O(n5) operations! • However, the effect of the multiplication with a Givens rotation matrix can be computed with O(n) operations due to its special structure. This leads to a total of O(n3) operations for the QR decomposition. • For QR iteration, we are not even interested in the value of G. We only need to compute the product A(k+1) = GA(k)GT ! • T T T Since matrix multiplication is associative and (M1M2) = M2 M1 , we can use the following scheme for computation of A(k+1): (k+1) (k) T T A = ::: G3;1 G2;1 A G2;1 G3;1 ::: Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 10 TU M ¨unchen Remarks • QR iteration is the method of choice if all eigenvalues of a matrix are needed. • The convergence order of the sequence fA(k)g is linear. • As with power iteration, convergence is slow if any of the eigenvalues are close together. By using a shift operation, similar to the one in Rayleigh quotient iteration, we can achieve quadratic convergence order. • Our algorithm does not calculate eigenvectors. Eigenvectors to specific eigenvalues can be retrieved using inverse iteration. • QR decomposition can also be used as an exact solver for systems of linear × equations: Let A 2 Rn n; b 2 Rn. Then Ax = b () QRx = b () Rx = QT b – Since R is upper triangular, the system Rx = QT b can be solved with O(n2) operations. – As the computation of the QR decomposition can take O(n3) operations, this method – as LU factorization – is especially useful when a set of systems with the same matrix A but many different vectors bi has to be solved. Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 11 TU M ¨unchen 7.5. Reduction Algorithms • The cost of all three algorithms presented so far depends largely on the number of non-zero entries in the matrix A: – Cost of an iteration step in power iteration is dominated by the complexity of the matrix-vector product Ax. – Cost of an iteration step in inverse iteration is dominated by the complexity of solving Ax = b. – Cost of an iteration step in QR iteration depends on the number of Givens rotations that we have to use to eliminate non-zero entries in the iteration matrix Ak . • Therefore, for numerical computation of eigenvalues, we are interested in matrices that are similar to A but have a significantly lower number of non-zero entries. • We show in the following that we can reduce A to a tridiagonal matrix at a cost comparable to one QR iteration. Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 12 TU M ¨unchen • Thus, we get the following procedure with the tridiagonalization as a kind of preprocessing step: reduction power / inverse / QR A A’ dense matrix tridiagonal matrix diagonal matrix eigenvalues (+ eigenvectors) • Depending on the method use, the result in the end is – the largest eigenvalue and the corresponding eigenvector (power iteration), – the smallest eigenvalue and the corresponding eigenvector (inverse iteration), – the complete set of eigenvalues (QR iteration). Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . QR, January 31, 2013 13 TU M ¨unchen × • We try to develop an algorithm that transforms a symmetric matrix A 2 Rn n into a similar tridiagonal symmetric matrix: 0∗ ∗ 0 0 ······ 01 B∗ ∗ ∗ 0 0 ··· 0C B C B0 ∗ ∗ ∗ 0 ··· 0C B C ! B . C A B . C B C B0 ··· 0 ∗ ∗ ∗ 0C B C @0 ··· 0 0 ∗ ∗ ∗A 0 ······ 0 0 ∗ ∗ × • Observation: Matrix A 2 Rn n is tridiagonal and symmetric if and only if it is of the form 0α ρ 0 ··· 01 Bρ C B 0 C B0 A C B C B .