AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 18: Sensitivity of Eigenvalues; Review for Midterm #2 Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 12 Outline 1 Sensitivity of Eigenvalues (MC§7.2) 2 Review for Midterm #2 Xiangmin Jiao Numerical Analysis I 2 / 12 Sensitivity of Eigenvalues Condition number of matrix X determines sensitivity of eigenvalues Theorem n×n −1 Let A 2 C be a nondefective matrix, and suppose A = X ΛX , where n×n X is nonsingular and Λ is diagonal. Let δA 2 C be some perturbation of A, and let µ be an eigenvalue of A + δA. Then A has an eigenvalue λ such that jµ − λj ≤ κp(X )kδAkp for 1 ≤ p ≤ 1. κp(X ) measures how far eigenvectors are from linear dependence For normal matrices, condition number κ2(X ) = 1 and κp(X ) = O(1), so eigenvalues of normal matrices are always well-conditioned Xiangmin Jiao Numerical Analysis I 3 / 12 Sensitivity of Eigenvalues Proof. Let δΛ = X −1 (δA) X . Then −1 kδΛkp ≤ kX kpkδAkpkX kp = κp(X )kδAkp: Let y be an engenvector of Λ + δΛ associated with µ. Suppose µ is not an eigenvalue of A, so µI − Λ is nonsingular. (Λ + δΛ)y = µy ) (µI − Λ)y = (δΛ) y ) y = (µI − Λ)−1 (δΛ) y. Thus −1 −1 k(µI − Λ) kp ≤ kδΛkp: −1 −1 k(µI − Λ) kp = jµ − λj , where λ is the eigenvalue of A closest to µ. Thus, jµ − λj ≤ kδΛkp ≤ κp(X )kδAkp: Xiangmin Jiao Numerical Analysis I 4 / 12 Left and Right Eigenvectors To analyze sensitivity of individual eigenvalues, we need to define left and right eigenvectors I Ax = λx for nonzero x then x is right eigenvector associated with λ ∗ ∗ I y A = λy for nonzero y, then y is left eigenvector associated with λ Left eigenvectors of A are right eigenvectors of A∗ Theorem n×n Let A 2 C have distinct eigenvalues λ1; λ2; : : : ; λn with associated linearly independent right eigenvectors x1;:::; xn and left eigenvectors ∗ ∗ y1;:::; yn. Then yj xi 6= 0 if i = j and yj xi = 0 if i 6= j. Proof. ∗ ∗ ∗ ∗ ∗ If i 6= j, yj Axi = λi yj xi and yj Axi = λj yj xi . Since λi 6= λj , yj xi = 0. n ∗ ∗ If i = j, since fxi g form a basis for C , yi xi = 0 together with yi xj = 0 would imply that yi = 0. This leads to a contradiction. Xiangmin Jiao Numerical Analysis I 5 / 12 Sensitivity of Individual Eigenvalues We analyze sensitivity of individual eigenvalues that are distinct Theorem n×n Let A 2 C have n distinct eigenvalues. Let λ be an eigenvalue with associated right and left eigenvectors x and y, respectively, normalized so that kxk2 = kyk2 = 1. Let δA be a small perturbation satisfying kδAk2 = , and let λ + δλ be the eigenvalue of A + δA that approximates λ. Then 1 jδλj ≤ + O(2): jy ∗xj κ = 1=jy ∗xj is condition number for eigenvalue λ A simple eigenvalue is sensitive if its associated right and left eigenvectors are nearly orthogonal Xiangmin Jiao Numerical Analysis I 6 / 12 Sensitivity of Individual Eigenvalues Proof. We know that jδλj ≤ κp(X ) = O(). In addition, δx = O() when λ is a simple eigenvalue (proof omitted). Because (A + δA)(x + δx) = (λ + δλ)(x + δx), thus (δA) x + A (δx) + O(2) = (δλ)x + λ(δx) + O(2): Left multiplying by y ∗ and using equation y ∗A = λy ∗, we obtain y ∗ (δA) x + O(2) = (δλ) y ∗x + O(2) and hence y ∗ (δA) x δλ = + O(2): y ∗x ∗ 1 2 Since jy (δA) xj ≤ kyk2 k(δA)k2 kxk2 = , jδλj ≤ jy ∗xj + O( ). Xiangmin Jiao Numerical Analysis I 7 / 12 Sensitivity of Multiple Eigenvalues and Eigenvectors Sensitivity of multiple eigenvalues is more complicated I For multiple eigenvalues, left and right eigenvectors can be orthogonal, hence very ill-conditioned I In general, multiple or close eigenvalues can be poorly conditioned, especially if matrix is defective Condition numbers of eigenvectors are also difficult to analyze I If matrix has well-conditioned and well-separated eigenvalues, then eigenvectors are well-conditioned I If eigenvalues are ill-conditioned or closely clustered, then eigenvectors may be poorly conditioned Xiangmin Jiao Numerical Analysis I 8 / 12 Outline 1 Sensitivity of Eigenvalues (MC§7.2) 2 Review for Midterm #2 Xiangmin Jiao Numerical Analysis I 9 / 12 Algorithms QR factorization I Classical and modified Gram-Schmidt I QR factorization using Householder triangularization and Givens rotation Solutions of linear least squares I Solution using Householder QR and other QR factorization I Alternative solutions: normal equation; SVD Conditioning and stability of linear least squares problems QR factorization with pivoting Xiangmin Jiao Numerical Analysis I 10 / 12 Eigenvalue Problem Eigenvalue problem of n × n matrix A is Ax = λx Characteristic polynomial is det(A − λI ) Eigenvalue decomposition of A is A = X ΛX −1 (does not always exist) Geometric multiplicity of λ is dim(null(A − λI )), and algebraic multiplicity of λ is its multiplicity as a root of pA, where algebraic multiplicity ≥ geometric multiplicity Similarity transformations preserve eigenvalues and their algebraic and geometric multiplicities Schur factorization A = QTQ∗ uses unitary similarity transformations Xiangmin Jiao Numerical Analysis I 11 / 12 Eigenvalue Algorithms Underlying concepts: power iterations, Rayleigh quotient, inverse iterations, convergence rate Schur factorization is typically done in two steps I Reduction to Hessenberg form for non-Hermitian matrices or reduction to tridiagonal form for hermitian matrices by unitary similarity transformation I Finding eigenvalues of Hessenberg or tridiagonal form Finding eigenvalue of tridiagonal forms I QR algorithm with shifts, and their interpretations as (inverse) simultaneous iterations Xiangmin Jiao Numerical Analysis I 12 / 12.