Chapter 6 Eigenvalues and Eigenvectors Po-Ning Chen, Professor Department of Electrical and Computer Engineering National Chiao Tung University Hsin Chu, Taiwan 30010, R.O.C. 6.1 Introduction to eigenvalues 6-1 Motivations • The static system problem of Ax = b has now been solved, e.g., by Gauss- Jordan method or Cramer’s rule. • However, a dynamic system problem such as Ax = λx cannot be solved by the static system method. • To solve the dynamic system problem, we need to find the static feature of A that is “unchanged” with the mapping A. In other words, Ax maps to itself with possibly some stretching (λ>1), shrinking (0 <λ<1), or being reversed (λ<0). • These invariant characteristics of A are the eigenvalues and eigenvectors. Ax maps a vector x to its column space C(A). We are looking for a v ∈ C(A) such that Av aligns with v. The collection of all such vectors is the set of eigen- vectors. 6.1 Eigenvalues and eigenvectors 6-2 Conception (Eigenvalues and eigenvectors): The eigenvalue-eigenvector pair (λi, vi) of a square matrix A satisfies Avi = λivi (or equivalently (A − λiI)vi = 0) where vi = 0 (but λi can be zero), where v1, v2,..., are linearly independent. How to derive eigenvalues and eigenvectors? • For 3 × 3 matrix A, we can obtain 3 eigenvalues λ1,λ2,λ3 by solving 3 2 det(A − λI)=−λ + c2λ + c1λ + c0 =0. • If all λ1,λ2,λ3 are unequal, then we continue to derive: Av1 = λ1v1 (A − λ1I)v1 = 0 v1 =theonly basis of the nullspace of (A − λ1I) Av λ v ⇔ A − λ I v ⇔ v A − λ I 2 = 2 2 ( 2 ) 2 = 0 2 =theonly basis of the nullspace of ( 2 ) Av3 = λ3v3 (A − λ3I)v3 = 0 v3 =theonly basis of the nullspace of (A − λ3I) and the resulting v1, v2 and v3 are linearly independent to each other. • If A is symmetric,thenv1, v2 and v3 are orthogonal to each other. 6.1 Eigenvalues and eigenvectors 6-3 • If, for example, λ1 = λ2 = λ3, then we derive: A − λ I v 0 {v , v } A − λ I ( 1 ) 1 = ⇔ 1 2 =thebasis of the nullspace of ( 1 ) (A − λ3I)v3 = 0 v3 =theonly basis of the nullspace of (A − λ3I) and the resulting v1, v2 and v3 are linearly independent to each other when the nullspace of (A − λ1I)istwo dimensional. • If A is symmetric,thenv1, v2 and v3 are orthogonal to each other. – Yet, it is possible that the nullspace of (A − λ1I)isone dimensional, then we can only have v1 = v2. – In such case, we say the repeated eigenvalue λ1 only have one eigenvector. In other words, we say A only have two eigenvectors. – If A is symmetric,thenv1 and v3 are orthogonal to each other. 6.1 Eigenvalues and eigenvectors 6-4 • If λ1 = λ2 = λ3, then we derive: (A − λ1I)v1 = 0 ⇔{v1, v2, v3} =thebasis of the nullspace of (A − λ1I) • When the nullspace of (A − λ1I)isthree dimensional, then v1, v2 and v3 are linearly independent to each other. • When the nullspace of (A − λ1I)istwo dimensional, then we say A only have two eigenvectors. • When the nullspace of (A − λ1I)isone dimensional, then we say A only have one eigenvector. In such case, the “matrix-form eigensystem” becomes: λ 1 00 A v1 v1 v1 = v1 v1 v1 0 λ1 0 00λ1 • If A is symmetric, then distinct eigenvectors are orthogonal to each other. 6.1 Eigenvalues and eigenvectors 6-5 Invariance of eigenvectors and eigenvalues. A • Property 1: The eigenvectors stay the same for every power of . The eigenvalues equal the same power of the respective eigenvalues. n n I.e., A vi = λi vi. 2 2 Avi = λivi =⇒ A vi = A(λivi)=λiAvi = λi vi A • Property 2: If the nullspace of n×n consists of non-zero vectors, then 0 is the eigenvalue of A. Accordingly, there exists non-zero vi to satisfy Avi =0· vi = 0. 6.1 Eigenvalues and eigenvectors 6-6 Property 3: Assume with no loss of generality |λ1| > |λ2| > ··· > |λk|.For any vector x = a1v1 + a2v2 + ···+ akvk that is a linear combination of all P 1 A eigenvectors, the normalized mapping = λ1 (namely, • 1 1 1 P v1 = A v1 = (Av1)= λ1v1 = v1) λ1 λ1 λ1 converges to the eigenvector with the largest absolute eigenvalue (when being applied repeatedly). I.e., k 1 k 1 k k k lim P x = lim A x = lim a1A v1 + a2A v2 + ···+ akA vk k→∞ k→∞ λk k→∞ λk 1 1 1 k k k k = lim a1λ v1 + a2λ v2 + ···+ λkA vk k→∞ λk 1 2 1 steady state transient states = a1v1. 6.1 How to determine the eigenvalues? 6-7 We wish to find a non-zero vector v to satisfy Av = λv;then (A − λI)v =0, where v = 0 ⇔ det(A − λI)=0. So by solving det(A − λI) = 0, we can obtain all the eigenvalues of A. 0.50.5 Example. Find the eigenvalues of A = . 0.50.5 Solution. 0.5 − λ 0.5 det(A − λI)=det 0.50.5 − λ =(0.5 − λ)2 − 0.52 = λ2 − λ = λ(λ − 1) = 0 ⇒ λ =0 or 1. 6.1 How to determine the eigenvalues? 6-8 Proposition: Projection matrix (defined in Section 4.2) has eigenvalues either 1 or 0. Proof: • A projection matrix always satisfies P 2 = P .SoP 2v = P v = λv. • By definition of eigenvalues and eigenvectors, we have P 2v = λ2v. • Hence, λv = λ2v for non-zero vector v, which immediately implies λ = λ2. • Accordingly, λ is either 1 or 0. 2 Proposition: Permutation matrix has eigenvalues satisfying λk =1forsome integer k. Proof: • A purmutation matrix always satisfies P k+1 = P for some integer k. 001 v3 3 Example. P = 100. Then, P v = v1 and P v = v. Hence, k =3. 010 v2 • Accordingly, λk+1v = λv,whichgivesλk = 1 since the eigenvalue of P cannot be zero. 2 6.1 How to determine the eigenvalues? 6-9 Proposition: Matrix m m−1 αmA + αm−1A + ···+ α1A + α0I has the same eigenvectors as A, but its eigenvalues become m m−1 αmλ + αm−1λ + ···+ α1λ + α0, where λ is the eigenvalue of A. Proof: • Let vi and λi be the eigenvector and eigenvalue of A. Then, m m−1 m m−1 αmA + αm−1A + ···+ α0I vi = αmλ + αm−1λ + ···+ α0 vi m m−1 • Hence, vi and αmλ + αm−1λ + ···+ α0 are the eigenvector and eigen- value of the polynomial matrix. 2 6.1 How to determine the eigenvalues? 6-10 Theorem (Cayley-Hamilton): For a square matrix A,define n n−1 f(λ)=det(A − λI)=λ + αn−1λ + ···+ α0. (Suppose A has linearly independent eigenvectors.) Then n n−1 f(A)=A + αn−1A + ···+ α0I = all-zero n × n matrix. Proof: The eigenvalues of f(A)areall zeros and the eigenvectors of f(A)remain the same as A. By definition of eigen-system, we have f(λ1)0··· 0 0 f(λ2) ··· 0 f(A) v v ··· vn = v v ··· vn . 1 2 1 2 . ... 00··· f(λn) 2 Corollary (Cayley-Hamilton): (Suppose A has linearly independent eigen- vectors.) (λ1I − A)(λ2I − A) ···(λnI − A) = all-zero matrix Proof: f(λ) can be re-written as f(λ)=(λ1 − λ)(λ2 − λ) ···(λn − λ). 2 6.1 How to determine the eigenvalues? 6-11 (Problem 11, Section 6.1) Here is a strange fact about 2 by 2 matrices with eigen- values λ1 = λ2:ThecolumnsofA − λ1I are multiples of the eigenvector x2.Any idea why this should be? 00 Hint: (λ I − A)(λ I − A)= implies 1 2 00 (λ1I − A)w1 = 0 and (λ1I − A)w2 = 0 where (λ2I − A)= w1 w2 . Hence, the columns of (λ2I − A) give the eigenvectors of λ1 if they are non-zero vectors and the columns of (λ1I − A) give the eigenvectors of λ2 if they are non-zero vectors . So, the (non-zero) columns of A − λ1I are (multiples of) the eigen- vector x2. 6.1 Why Gauss-Jordan cannot solve Ax = λx? 6-12 • The forward elimination may change the eigenvalues and eigenvectors? 1 −2 Example. Check eigenvalues and eigenvectors of A = . 25 Solution. A A − λI λ − 2 . – The eigenvalues of satisfy det( )=( 3) =0 10 1 −2 – A = LU = . The eigenvalues of U apparently satisfy 21 09 det(U − λI)=(1− λ)(9 − λ)=0. – Suppose u1 and u2 are the eigenvectors of U, respectively corresponding to eigenvalues 1 and 9. Then, they cannot be the eigenvectors of A since if they were, 10 3u1 = Au1 = LUu1 = Lu1 = u1 =⇒ u1 = 0 21 2 10 3u2 = Au2 = LUu2 =9Lu2 =9 u2 =⇒ u2 = 0. 21 Hence, the eigenvalues are nothing to do with pivots (except for a triangular A). 6.1 How to determine the eigenvalues? (Revisited) 6-13 Solve det(A − λI)=0. • det(A − λI) is a polynomial of order n. a − λa a ··· a 1,1 1,2 1,3 1,n a a − λa ··· a 2,1 2,2 2,3 2,n f λ A − λI a a a − λ ··· a ( )=det( )=det 3,1 3,2 3,3 3,n . ... an,1 an,2 an,3 ··· an,n − λ =(a1,1 − λ)(a2,2 − λ) ···(an,n − λ)+terms of order at most (n − 2) (By Leibniz formula) =(λ1 − λ)(λ2 − λ) ···(λn − λ) Observations • The coefficient of λn is (−1)n,providedn ≥ 1. • λn−1 n λ n a A n ≥ The coefficient of is i=1 i = i=1 i,i = trace( ),provided 2.