7. the Symmetric Eigenvalue Problem

TU M ¨unchen

7. The Symmetric Eigenvalue Problem

A ‘must’ for engineers . . . QR

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 1 TU M ¨unchen

7.4. QR Iteration

• Factorization methods such as the QR iteration [Francis 1961] compute all × eigenvalues of a symmetric matrix A ∈ Rn n at once.

• The idea is to compute a sequence of matrices {Ak } that are similar to A. This sequence should converge to a matrix in triangular form, such that the diagonal of Ak is an approximation for the eigenvalues of A. • The QR iteration is based on the QR decomposition which factorizes a given matrix A such that

R A = Q *

with Q an orthogonal and R an upper triangular matrix. Then the matrix

RQ = Q−1AQ = QT AQ

is similar to A.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 2 TU M ¨unchen

• It can be shown that the sequence {A(k)} deﬁned by

A(0) = A, A(k+1) = R(k)Q(k), Q(k) orthogonal, R(k) upper triangular, A(k) = Q(k)R(k)

converges to a diagonal matrix.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 3 TU M ¨unchen

QR Iteration: Algorithm

• Thus, we have the following algorithm: – A(0) := A – for k = 0, 1, 2,...: 1. Compute a factorization A(k) = QR using QR decomposition 2. A(k+1) := RQ 3. if all non-diagonal elements sufﬁciently small: (k+1) output λj ≈ aj,j , j = 1,..., n remove row and column n • (k+1) We consider the non-diagonal element aj,m “sufﬁciently small” if its absolute (k+1) value is ≤ ε · |aj,j |. • We still need to know how to implement the QR decomposition!

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 4 TU M ¨unchen

7.4.1. QR Decomposition

• Remember: We want to decompose a matrix A into an orthogonal matrix Q and an upper triangular matrix R such that A = QR. • This problem is equivalent to the following: Find an orthogonal matrix G such that GA = R, i.e. G “zeroes out” the subdiagonal part of A. Then A = GT R. T • We ﬁrst have a look at the two-dimensional case: Given a vector x = (x1, x2) , ﬁnd an orthogonal matrix G such that Gx = (r, 0)T for some r ∈ R. • It is easy to see that (x1 ,y 2 ) 1 x1 x2 G = q θ 2 2 −x2 x1 x1 + x2 (r,0) does the job.

• Geometrically, the matrix G describes a rotation by θ = arctan(x2/x1) radians in the (x1, x2) plane. • G is called a Givens rotation matrix [Givens 1958].

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 5 TU M ¨unchen

• Generalization to the n-dimensional case: When multiplied with A,

1 ··· 0 ··· 0 ··· 0  ......   . . . . .    0 ··· ρ · a ··· ρ · a ··· 0  jj ij   . . . . .  1 Gij =  . . . . .  , ρ =  . . . . .  q 2 2   aij + ajj 0 · · · −ρ · aij ··· ρ · ajj ··· 0    ......   . . . . .  0 ··· 0 ··· 0 ··· 1

affects rows i and j of A and eliminates entry aij (i > j). Gij differs from the identity matrix in only four entries.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 6 TU M ¨unchen

• If a column k is already eliminated, Gij with j > k, i > j does not destroy the zeros in column k since operations between lines i and j just result in summing up zeros in column k:   a1,1 ·················· a1n  . .   . .   0 . .     . . . .   ......     . .   . .   . 0 a , ··· a , .   j j j i   ......   ......   ......     . . .   . . a ··· a .   i,j i,i   . . . .   ......   . . . . .  0 ··· 0 an,j ········· an,n

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 7 TU M ¨unchen

• Thus, we can eliminate the subdiagonal non-zero elements coumn by column, starting from the ﬁrst column and working in columns top to bottom. This results in the G-matrix

G = Gn,n−1 · Gn,n−2Gn−1,n−2 ····· Gn,2 ··· G3,2 · Gn,1 ··· G2,1,

• Since the product of two orthogonal matrices is orthogonal, G is orthogonal and GA is an upper triangular matrix for any matrix A.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 8 TU M ¨unchen

• Alternatively, we can eliminate non-zeros line by line top to bottom and left to right:

G = Gn,n−1 ··· Gn,1 ··· G3,2 · G3,1 · G2,1

This is possible as Gij does not change lines k with k < j and zeros in line i in a column k < j are not destroyed:   a1,1 ·················· a1n  . .   . .   0 . .     . . . .   ......     . .   . .. .   . . a , ··· a , .   j j j i   . .   . .   0 ······ 0 . .     .   0 ··· 0 a ··· a .   i,j i,i  a ·················· a   i+1,1 i+1,n  . .   . .. .   . . .  an,1 ·················· an,n

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 9 TU M ¨unchen

QR Iteration: Implementation

• We now have a method to compute the QR decomposition of A: R = GA, Q = GT . • The naive computation of G via n(n − 1)/2 matrix multiplications costs O(n5) operations! • However, the effect of the multiplication with a Givens rotation matrix can be computed with O(n) operations due to its special structure. This leads to a total of O(n3) operations for the QR decomposition. • For QR iteration, we are not even interested in the value of G. We only need to compute the product A(k+1) = GA(k)GT ! • T T T Since matrix multiplication is associative and (M1M2) = M2 M1 , we can use the following scheme for computation of A(k+1):

(k+1) (k) T T A = ... G3,1 G2,1 A G2,1 G3,1 ...

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 10 TU M ¨unchen

Remarks

• QR iteration is the method of choice if all eigenvalues of a matrix are needed. • The convergence order of the sequence {A(k)} is linear. • As with power iteration, convergence is slow if any of the eigenvalues are close together. By using a shift operation, similar to the one in Rayleigh quotient iteration, we can achieve quadratic convergence order. • Our algorithm does not calculate eigenvectors. Eigenvectors to speciﬁc eigenvalues can be retrieved using inverse iteration. • QR decomposition can also be used as an exact solver for systems of linear × equations: Let A ∈ Rn n, b ∈ Rn. Then Ax = b ⇐⇒ QRx = b ⇐⇒ Rx = QT b

– Since R is upper triangular, the system Rx = QT b can be solved with O(n2) operations. – As the computation of the QR decomposition can take O(n3) operations, this method – as LU factorization – is especially useful when a set of systems with the same matrix A but many different vectors bi has to be solved.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 11 TU M ¨unchen

7.5. Reduction Algorithms

• The cost of all three algorithms presented so far depends largely on the number of non-zero entries in the matrix A: – Cost of an iteration step in power iteration is dominated by the complexity of the matrix-vector product Ax. – Cost of an iteration step in inverse iteration is dominated by the complexity of solving Ax = b. – Cost of an iteration step in QR iteration depends on the number of Givens rotations that we have to use to eliminate non-zero entries in the iteration matrix Ak . • Therefore, for numerical computation of eigenvalues, we are interested in matrices that are similar to A but have a signiﬁcantly lower number of non-zero entries. • We show in the following that we can reduce A to a tridiagonal matrix at a cost comparable to one QR iteration.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 12 TU M ¨unchen

• Thus, we get the following procedure with the tridiagonalization as a kind of preprocessing step:

reduction power / inverse / QR A A’

dense matrix tridiagonal matrix diagonal matrix eigenvalues (+ eigenvectors)

• Depending on the method use, the result in the end is – the largest eigenvalue and the corresponding eigenvector (power iteration), – the smallest eigenvalue and the corresponding eigenvector (inverse iteration), – the complete set of eigenvalues (QR iteration).

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 13 TU M ¨unchen

× • We try to develop an algorithm that transforms a symmetric matrix A ∈ Rn n into a similar tridiagonal symmetric matrix:

∗ ∗ 0 0 ······ 0 ∗ ∗ ∗ 0 0 ··· 0   0 ∗ ∗ ∗ 0 ··· 0   →  .  A  .    0 ··· 0 ∗ ∗ ∗ 0   0 ··· 0 0 ∗ ∗ ∗ 0 ······ 0 0 ∗ ∗

× • Observation: Matrix A ∈ Rn n is tridiagonal and symmetric if and only if it is of the form α ρ 0 ··· 0 ρ   0  0 A     .   .  0 0 ( − )×( − ) with A ∈ R n 1 n 1 tridiagonal and symmetric. • This leads to a “dynamic programming” approach: Use a similarity transformation to transform A to a matrix in the above form, then continue with A0 recursively.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 14 TU M ¨unchen

Transformation Algorithm

0 ( − )×( − ) • If we had a matrix H ∈ R n 1 n 1 such that

a12 ρ a13 0 0     H  .  =  .   .   .  a1n 0

for some ρ ∈ R, then

1 0 ··· 0 a11 a12 ··· a1n a11 a12 ··· a1n 0  a12   ρ   0      0 H  a13   0 B  HA :=     =    .   . . .   .   .   . .. .   .  0 a1n ··· ann 0

( − )×( − ) (B ∈ R n 1 n 1 ). • We already know that such a H0 exists – for example, it can be described as the product of n − 2 Givens rotations.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 15 TU M ¨unchen

• By transposing the result, we can use the effect of multiplication with H again:

a11 ρ 0 ··· 0 a11 ρ 0 ··· 0 a12   ρ   T   0 A  H(HA)T = H a13 B  =  2     .   .   .   .   .  a1n 0

• Since H(HA)T = HAT HT = HAHT , this is a similarity transformation if and only if H is orthogonal. • It is important that the application of H from the left does not modify the ﬁrst row of A and the application of HT from the right does not modify the ﬁrst column of H · A:

A H*A H*A*HT

part of the matrix that has been modified in the last transformation step

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 16 TU M ¨unchen

Therefore, we can only eliminate everything below the subdiagonal element of the ﬁrst column and produce a triadiagonal matrix only (instead of a diagonal one).

• With basic linear algebra, one can also show that A2 is symmetric. • Our task is now to ﬁnd an appropriate matrix H0! • A ﬁrst idea could be to choose H0 as a product of Givens rotations, as previously indicated. • However, in practice Householder transformations [Householder 1958] are used more frequently.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 17 TU M ¨unchen

Householder Transformations

× • For any vector x ∈ Rn, Householder matrix H ∈ Rn n is deﬁned by H = I − 2vv T , u with v := and u := x − kxk e1, such that kuk2 2 ρ 0   Hx =  .   .  0

for some ρ ∈ R.

Geometrically, H describes a reﬂection of x in the hyperplane deﬁned by the vector v which is orthogonal to the hyperplane.

wikipedia.org

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 18 TU M ¨unchen

Remarks

• As with Givens rotation matrices, Householder matrices are usually not explicitly computed. When using an appropriate implementation, the effect of multiplication with a Householder matrix can be computed with O(n2) operations. • In that case, the transformation algorithm described above takes O(n3) steps, rather than O(n4) steps for a naive implementation with O(n) matrix multiplications. • Householder transformations can also be used to compute the QR decomposition, since the product of n − 1 Householder matrices transforms a matrix into upper triangular form. • We could therefore also use them for QR iteration! • However, it is more efﬁcient to ﬁrst reduce a matrix to tridiagonal form using Householder transformations (cost: O(n3)), then perform QR iteration using Givens rotations on the reduced matrix. • Then, in each iteration step, only O(n) elements need to be zeroed using Givens rotations, which results in a cost of O(n2) per iteration step, vs. O(n3) when using Householder transformations.

Miriam Mehl: 7. The Symmetric Eigenvalue Problem A ‘must’ for engineers . . . QR, January 31, 2013 19