Chapter 4 EIGENVALUE PROBLEM The German word eigen is cognate with the Old English word ¯agen, which became owen in Middle English and “own” in modern English. Given A ∈ Rn×n solve Ax = λx where λ ∈ C and 0 =6 x ∈ Cn. 4.1 Mathematics 4.2 Reduction to Upper Hessenberg Form 4.3 The Power Method 4.4 The QR Iteration 4.5 The Lanczos Method 4.6 Other Eigenvalue Algorithms 4.7 Computing the SVD 4.1 Mathematics The adjacency matrix of a directed graph is given by 2 3 00001 6 7 6 101007 6 7 6 000107 4 010005 00010 96 where aij is the number of arcs from i to j. The number of paths of length k joining i to j is k k given by (A )ij. Is there a closed form expression for A ? I.e., a more direct way of forming Ak? 4.1.1 Complex Values The properties of vectors and matrices of real numbers extends to vectors x ∈ Cm and matrices A ∈ Cm×n. We define the conjugate transpose by AH = A¯T. A Hermitian matrix satisfies AH = A and a unitary matrix U H = U −1. Cn is a vector space. The inner product for x, y ∈ Cn is xHy. 4.1.2 Eigenvalues and eigenvectors If Ax = λx where λ ∈ C and 0 =6 x ∈ Cn then λ is an eigenvalue1 and x is an eigenvector. Try solving for x in (A − λI)x =0. There exists a solution x =6 0 only if det(A − λI) = 0. Hence the eigenvalues λ are roots of the characteristic polynomial p(ζ):=det(ζI − A). 01 For example for A = 11 ζ −1 det(ζI − A)=det = ζ2 − ζ − 1. (4.1) −1 ζ − 1 √ λ λ2 − λ − ⇒ λ 1 ± 5 An eigenvalue satisfies 1=0 = 2 2 . In general the characteristic polynomial has n roots, counting multiplicities: det(ζI − A)=(ζ − λ1)(ζ − λ2) ···(ζ − λn) . Hence there are n eigenvalues. Because A is real, p(ζ) has real coefficients and thus each eigenvalue is either real or one of a complex conjugate pair. 1characteristic value 97 imaginary real Setting ζ = 0 in (4.1), we get n det(−A)=(−1) λ1λ2 ···λn, and since det(−A)=det((−1)A)=(−1)ndetA, we have detA = λ1λ2 ···λn, ⇒ 0 is an eigenvalue of a singular matrix. Note that for an eigenpair (λ, x) A2x = A(λx)=λAx = λ2x, A−1Ax = A−1λx ⇒ A−1x = λ−1x, Amx = λmx. Moreover, if q(ζ)isapolynomial, q(A)x = q(λ)x. For a block triangular matrix T with diagonal blocks T11,T22,...,Tpp we have det(ζI − T )=det(ζI − T11)det(ζI − T22) ···det(ζI − Tpp). example 2 3 1 24 A = 4 0 205 0 12 20 det(ζI − A)=det(ζ − 1)det(ζI − )=(ζ − 1)(ζ − 2)2 . 12 This property of determinants follows immediately if we can show A B det =detA · detC. 0 C This is shown by considering two cases: 98 (i) A singular, AB A 0 IA−1B (ii) A nonsingular, in which case = . 0 C 0 I 0 C 4.1.3 Similarity transformations Return to the problem of a simpler form for Ak. Definition If detX =0,then6 X−1AX =: B is called a similarity transformation and we say that B is similar to A. First a couple of easy-to-prove facts: 1. If B is similar to A and C is similar to B,thenC is similar to A. Therefore similarity transformations can have several steps. 2. Similar matrices have identical eigenvalues including multiplicities. Note that Ak =(XBX−1)k = ···= XBkX−1. Can we find a simple B which is similar to A? For each of the eigenvalues λi we have Axi = λixi for some xi =06 . Hence AX = XΛ where X =(x1,x2,...,xn) and Λ = diag(λ1,λ2,...,λn). If the eigenvectors can be chosen to be linearly independent, then X is nonsingular and X−1AX =Λ, in which case we say that A is diagonalizable. Otherwise A is defective. example 2 3 100 A = 4 0125 001 Eigenvalues are 1, 1, 1. Eigenvectors must satisfy (A − I)x =0. 2 3 2 3 000 x1 4 5 4 5 002 x2 =0 ⇒ x3 =0 000 x 3 2 3 0 4 5 ⇒ x = x2 x3 99 where x2 and x3 are arbitrary but not both zero. Thus the eigenvectors corresponding to 1 lie in a 2-dimensional subspace. We do not have three linearly independent eigenvectors, so this matrix is defective. Theorem: A matrix having distinct eigenvalues is diagonalizable. Proof for n = 3. Suppose Axi = λixi,xi =06 ,i=1, 2, 3. Suppose α1x1 + α2x2 + α3x3 =0. (4.2) We need to show α1 = α2 = α3 =0. Without loss of generality it is enough to show α1 =0. Multiply (4.2) by (A − λ2I)(A − λ3I), (A − λ2I)(A − λ3I)xi =(A − λ2I)(λi − λ3)xi =(λi − λ2)(λi − λ3)xi, and (4.2) becomes α1(λ1 − λ2)(λ1 − λ3)x1 =0. Distinctness ⇒ α1 =0.2 01 √ √ Example Recall that has eigenvalues 1 + 5 , 1 − 5 . The corresponding eigen- 11 2 2 2 2 √ √ " √ # 1 5 1 5 1 5 − + − − 01 + 0 − vectors are 2 2 , 2 2 . So = X 2 2 √ X 1 where 1 1 11 0 1 − 5 √ √ 2 2 − 1 + 5 − 1 − 5 X = 2 2 2 2 , so 11 " √ √ √ √ # k 01 1 ( 1+ 5 )k−1 − ( 1− 5 )k−1 ( 1+ 5 )k − ( 1− 5 )k = √ 2 √ 2√ √ 2 2√ . 11 1+ 5 k − 1− 5 k 1+ 5 k+1 − 1− 5 k+1 5 ( 2 ) ( 2 ) ( 2 ) ( 2 ) For large k, Ak is dominated by the largest eigenvalue(s). Hence the spectral radius ρ(A):=max|λi(A)|. 1≤i≤n Jordan canonical form. Any matrix can be transformed by means of a similarity trans- formation to the form diag(J1,J2,...,Jm)whereeachbasic Jordan block Jk has the form 2 3 λk 1 6 . 7 6 λ .. 7 J 6 k 7 . k = 4 . 5 .. 1 λk The Cayley-Hamilton Theorem asserts that any matrix A satisfies p(A)=0wherep(ζ) is the characteristic polynomial of A. 100 4.1.4 Unitary similarity transformations 1+ε 1 The X needed to diagonalize a matrix can be very ill-conditioned, e.g., for A = 01 1 −1/ε 1+ε 0 we need X = to get X−1AX = . (For ε =0,A is defec- 01 01 tive.) However, to get eigenvalues (and eigenvectors) it is sufficient to have X−1AX = upper triangular. And it is possible to achieve this using an X with κ2(X)=1.Fora n×n −1 H unitary matrix U ∈ C , κ2(U)=kU k2kUk2 = kU k2kUk2 =1. THEOREM Schur decomposition If A ∈ Cn×n, then there exists a unitary matrix U ∈ Cn×n such that R = U HAU is upper triangular. Proof. By induction on n. True for n = 1. Assume true for n − 1. We will seek unitary P such that λ bH P HAP = . 0 B for some λ, bH, B. In other words we want H P AP e1 = λe1 or equivalently A(Pe1)=λ(Pe1). Let λ be an eigenvalue of A and x an eigenvector with kxk2 =1. We are done if we can choose H unitary P so that Pe1 = x. Equivalently we want P x = e1. A generalized Householder reflection vvH P H = I − 2 ,v= x − e vHv 1 x eTx ≤ can be shown to work. To guard against division by zero, choose so that 1 0. There is unitary V such that V HBV =: R is upper triangular. Let U P 1 . = V then λ bH U HAU = . 2 0 R The construction in this proof is known as deflation: it depends on being able to find one eigenvalue of a matrix. If A is real and has only real eigenvalues then U and R may be chosen to be real. If A is real but has some imaginary eigenvalues (which must occur in pairs λ, λ¯), then neither U nor R can be real. Unfortunately complex arithmetic is expensive: double the storage, four times the computation. In the case of factoring polynomials we can avoid complex arithmetic by allowing the factors to be either linear or quadratic with two imaginary roots λ and λ¯. Analogously real matrices have a real Schur decomposition: 101 THEOREM Real Schur decomposition If A ∈ Rn×n, then there exists an orthogonal matrix Q ∈ Rn×n such that R = QTAQ is block upper triangular with diagonal blocks that either are 1 by 1 or are 2 by 2 with imaginary eigenvalues. 4.1.5 Symmetric matrices Theorem If A is real symmetric, then there exists a real orthogonal Q such that QTAQ = Λ= diag(λ1,λ2,...,λn), λi ∈ R. Proof. Consider the real Schur decomposition R = QTAQ. Then RT = QTATQ = QTAQ = R. Hence R is symmetric block diagonal with blocks that either are 1 by 1 or are symmetric and 2 by 2 with imaginary eigenvalues. However, a 2 by 2 symmetric matrix cannot have imaginary eigenvalues, so R must be diagonal. 2 AQ = QΛ A(Qei)=(Qei)λi Qei is an eigenvector, and λi is eigenvalue. These eigenvectors form an orthonormal set. The eigenvalues are real. If A is s.p.d., the eigenvalues are positive. 4.1.6 Matrix norms THEOREM ρ(A) ≤kAk for a consistent matrix norm. Proof. There exists an x =0and6 |λ| = ρ(A) such that Ax = λx. |λ|kxk≤kAkkxk. 2 THEOREM The spectral norm p T kAk2 = ρ(A A).