Chapter 8 Eigenvalue Problems 8.1 Introduction n n The focus of this chapter will be the eigenvalue problem: given A C ⇥ we 2 will be interested in finding pairs (λ, x) C Cn such that 2 ⇥ ? Ax = λx. Recall that, since (A λI)x = 0,wemusthavethatdet(A λI) = 0. Define − − the characteristic polynomial of the matrix A by χ (t)=det(A tI). A − This shows that λ σ(A) i↵ χ (λ) = 0. To find the eigenvalues of a matrix, 2 A it is enough to find the roots of the characteristic polynomial. However, this is never done directly – that is, finding roots of the characteristic polynomial – because it is too expensive and very sensitive to perturbations on the entries of A. The following example shows the latter point: Example 8.1.1 (instability of determinant). Suppose that 1 20 2 20 2 . 3 A = .. .. 6 7 6 9 207 6 7 6 107 6 7 4 5 243 244 CHAPTER 8. EIGENVALUE PROBLEMS It follows that σ(A)= 1, , 10 .Now,for0 <" 1, consider also { ··· } ⌧ 1 20 2 20 2 . 3 A" = .. .. 6 7 6 9 207 6 7 6" 107 6 7 4 5 In any reasonable norm, A will be close to A . However, we can choose " k "k k k so that 0 σ(A ). To see this, consider 2 " 20 2 20 . 2 20 .. .. 0 . 1 det(A")=det0 1 " det .. .. 9 20 − B C B C B 9 20 C B 10C B C B C B 10 20C @ A B C = 10! 209". @ A − Therefore, if 6 10! 3 10 6 " = ⇥ 7 10− . 209 ⇡ 5 1011 ⇡ ⇥ ⇥ the matrix A" has zero as an eigenvalue. For a linear system Ax = b, or a even least-squares problem, we showed that there are always direct methods for finding the solution vector x. These may be quite expensive, requiring, say, (n3) operations; but, nonetheless, O direct methods were available. There is another important issue associated with eigenvalue problems. Direct methods are not typically available for find- ing eigen-pairs. This fundamental difficulty can be seen from the following example. Consider the matrix c1 c2 cn 1 cn ··· − 10 00 2 . ···. 3 F = 0 .. .. 6 7 6 . .. 7 6 . 1007 6 7 6 0 0107 6 ··· 7 4 5 F has the characteristic polynomial n n n 1 χ (t)=( 1) t c t − c . F − − 1 −···− n 8.2. ESTIMATING EIGENVALUES USING GERSCHGORIN DISCS 245 In other words to any polynomial we can associate an eigenvalue problem, and vice versa, of course. Thus, since there is no general direct solution method for finding roots of polynomials (above order 5), there cannot be a direct method – an algorithmic method that completes in a finite number of steps – to find the eigenvalues of a matrix. In conclusion, the mantra of eigenvalue problems is the following: Every eigenvalue solver must be iterative. Now, you may argue that there are matrices for which the spectrum is easy to compute, namely diagonal and triangular matrices. True, for such matrices, we can simply read the eigenvalues from of the diagonal entries. The question is this: can we transform a generic matrix A into a diagonal or triangular matrix T via a similarity transformation. As guaranteed by the n n Schur factorization theorem, there exists a unitary matrix U C ⇥ such that 2 A = UTUH,whereT is triangular. So the question reduces to this one: can we find U using a direct method? The answer again is a resounding no. If there was a such a method, then we could find the roots to any polynomial of any degree in a finite number of steps. Since this is impossible, in general, we get a contradiction. Before we describe our iterative methods for finding eigen-pairs, let intro- duce some coarse approximation techniques. 8.2 Estimating Eigenvalues Using Gerschgorin Discs If all that is wanted are coarse approximations of the eigenvalues, this can be obtained easily using Gerschgorin discs. n n Definition 8.2.1 (Gershgorin discs). Let n 2 and A C ⇥ . The Gershgorin ≥ 2 discs Di of A are n Di = z C z ai,i Ri ,Ri = ai,j ,i=1,...,n. { 2 || − | } | | j=1 Xj=i 6 n n Theorem 8.2.2 (Gershgorin circle theorem). Let n 2 and A =[ai,j ] C ⇥ . ≥ 2 Then n σ(A) D . ⇢ i i[=1 246 CHAPTER 8. EIGENVALUE PROBLEMS Proof. Suppose that (λ, w) is an eigen-pair of A. Then n a w = λw ,i=1, 2, ,n. i,j j i ··· Xj=1 Suppose that n wk = w = max wi . | | k k1 i=1 | | Observe that w = 0, since w = 0. Then k 6 6 λ a w = λw a w | − k,k |·| k | | k − k,k k | n = a w a w k,j j − k,k k j=1 X n = a w k,j j j=1 Xj=k 6 n a w k,j j j=1 Xj=k 6 n a w = R w . k,j ·| k | k | k | j=1 Xj=k 6 Thus λ D . 2 k n n Theorem 8.2.3 (Gerschgorin’s second theorem). Let n 2 and A C ⇥ . ≥ 2 Suppose that 1 p n 1 and that the Gerschgorin discs of the matrix A can − be divided into disjoint subsets D(p) and D(q) containing p and q = n p discs, − respectively. Then the union of the discs in D(p) contains p eigenvalues, and the union of the discs in D(q) contains q eigenvalues, counting multiplicities. In particular, if one disc is dijoint from all the others, it contains exactly one eigenvalue. And, if all of the discs are disjoint, then each contains exactly one eigenvalue. n Proof. Define the matrix B(")=[bi,j (")]i,j=1 as follows: ai,i ,i= j bi,j (")= . "a ,i= j ( i,j 6 8.2. ESTIMATING EIGENVALUES USING GERSCHGORIN DISCS 247 Then B(1) = A and B(0) = diag(a , ,a ). Each eigenvalue of B(0) 1,1 ··· n,n is the center of one of the Gerschgorin discs of A. Thus, exactly p of the eigenvalues of B(0) lie in the union of the discs in D(p). The eigenvalues of the matrix B(") are the zeros of the characteristic polynomial of B("). The coefficients of this characteristic polynomial are con- tiuous functions of the parameter ", and, in turn, the zeros of the polynomial are continuous functions of ". As " increases from 0 to 1, the eigenvalues of B(") move in continuous paths in the complex plane. Since the order of the characteristic polynomial of B(") is always exactly n, none of the zeros of the characteristic polynomial diverge to infinity. At the same time, the radii of the Gerschgorin discs increase from 0 to Ri , respectively. In particular, it is easy to see that they increase as R = "R , 0 " 1,i=1, ,n. ",i i ··· Since p of the eigenvalues of B(") lie in the union of the discs in D(p) when " = 0, and these discs are disjoint from those in D(q),thesep eigenvalues will stay within the union of the disc in D(p), for all values of ",0 " 1. n n Theorem 8.2.4. Let n 2 and suppose that in the matrix A C ⇥ all o↵ ≥ 2 diagonal elements are smaller in modulus that ">0, i.e., a <", 1 i,j n, i = j. | i,j | 6 Suppose that for some r 1, ,n the diagonal element a satisfies 2{ ··· } r,r a a >δ, for all i = r. | r,r − i,i | 6 Then, provided δ "< , 2(n 1) − there is an eigenvalue λ σ(A) such that 2 2(n 1) δ λ a < − "2 < . | − r,r | δ 2(n 1) − n n Proof. Suppose >0. Define K =[ki,j ] C ⇥ via 2 , i = j = r ki,j = . (δi,j , otherwise 248 CHAPTER 8. EIGENVALUE PROBLEMS 1 Define the similar matrix A := KAK− . For example, if r = 3, n = 4, a1,1 a1,2 a1,3 a1,4 a2,1 a2,2 a2,3 a2,4 4 4 A = 2 3 C ⇥ , a3,1 a3,2 a3,3 a3,4 2 6a a a a 7 6 4,1 4,2 4,3 4,47 then 4 5 1 a1,1 a1,2 − a1,3 a1,4 1 a2,1 a2,2 − a2,3 a2,4 A = 2 3 . a3,1 a3,2 a3,3 a3,4 6 a a 1a a 7 6 4,1 4,2 − 4,3 4,4 7 4 5 The Gerschgorin disc of A, with respect to row r, has its center at ar,r ,and its radius is n n R,r = ai,j = ai,j j=1 j=1 Xj=r Xj=r 6 6 n < " = (n 1)". − j=1 Xj=r 6 The disc of A corresponding to row i = r has a as its center, and its radius 6 i,i is n 1 R = a + − a ,i i,j | i,r | j=1 Xj=i j=6 r 6 " < (n 2)" + . − 2" Now, pick = δ . Then, 2"2(n 1) R < − , ,r δ and δ R < +(n 2)", i = r. ,i 2 − 6 Therefore, for i = r, 6 2"2(n 1) δ R + R < − + +(n 2)" ,r ,i δ 2 − δ <"+ +(n 2)"<δ, 2 − 8.3. STABILITY 249 δ on the assumption that "< 2(n 1) . Finally, we have − δ< a a , | r,r − i,i | but R,r + R,i <δ.