AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 18: Sensitivity of Eigenvalues; Review for Midterm #2
Xiangmin Jiao
Stony Brook University
Xiangmin Jiao Numerical Analysis I 1 / 12 Outline
1 Sensitivity of Eigenvalues (MC§7.2)
2 Review for Midterm #2
Xiangmin Jiao Numerical Analysis I 2 / 12 Sensitivity of Eigenvalues
Condition number of matrix X determines sensitivity of eigenvalues
Theorem n×n −1 Let A ∈ C be a nondefective matrix, and suppose A = X ΛX , where n×n X is nonsingular and Λ is diagonal. Let δA ∈ C be some perturbation of A, and let µ be an eigenvalue of A + δA. Then A has an eigenvalue λ such that |µ − λ| ≤ κp(X )kδAkp for 1 ≤ p ≤ ∞.
κp(X ) measures how far eigenvectors are from linear dependence
For normal matrices, condition number κ2(X ) = 1 and κp(X ) = O(1), so eigenvalues of normal matrices are always well-conditioned
Xiangmin Jiao Numerical Analysis I 3 / 12 Sensitivity of Eigenvalues
Proof. Let δΛ = X −1 (δA) X . Then
−1 kδΛkp ≤ kX kpkδAkpkX kp = κp(X )kδAkp.
Let y be an engenvector of Λ + δΛ associated with µ. Suppose µ is not an eigenvalue of A, so µI − Λ is nonsingular. (Λ + δΛ)y = µy ⇒ (µI − Λ)y = (δΛ) y ⇒ y = (µI − Λ)−1 (δΛ) y. Thus
−1 −1 k(µI − Λ) kp ≤ kδΛkp.
−1 −1 k(µI − Λ) kp = |µ − λ| , where λ is the eigenvalue of A closest to µ. Thus, |µ − λ| ≤ kδΛkp ≤ κp(X )kδAkp.
Xiangmin Jiao Numerical Analysis I 4 / 12 Left and Right Eigenvectors To analyze sensitivity of individual eigenvalues, we need to define left and right eigenvectors
I Ax = λx for nonzero x then x is right eigenvector associated with λ ∗ ∗ I y A = λy for nonzero y, then y is left eigenvector associated with λ Left eigenvectors of A are right eigenvectors of A∗
Theorem n×n Let A ∈ C have distinct eigenvalues λ1, λ2, . . . , λn with associated linearly independent right eigenvectors x1,..., xn and left eigenvectors ∗ ∗ y1,..., yn. Then yj xi 6= 0 if i = j and yj xi = 0 if i 6= j.
Proof. ∗ ∗ ∗ ∗ ∗ If i 6= j, yj Axi = λi yj xi and yj Axi = λj yj xi . Since λi 6= λj , yj xi = 0. n ∗ ∗ If i = j, since {xi } form a basis for C , yi xi = 0 together with yi xj = 0 would imply that yi = 0. This leads to a contradiction.
Xiangmin Jiao Numerical Analysis I 5 / 12 Sensitivity of Individual Eigenvalues
We analyze sensitivity of individual eigenvalues that are distinct
Theorem n×n Let A ∈ C have n distinct eigenvalues. Let λ be an eigenvalue with associated right and left eigenvectors x and y, respectively, normalized so that kxk2 = kyk2 = 1. Let δA be a small perturbation satisfying kδAk2 = , and let λ + δλ be the eigenvalue of A + δA that approximates λ. Then 1 |δλ| ≤ + O(2). |y ∗x|
κ = 1/|y ∗x| is condition number for eigenvalue λ A simple eigenvalue is sensitive if its associated right and left eigenvectors are nearly orthogonal
Xiangmin Jiao Numerical Analysis I 6 / 12 Sensitivity of Individual Eigenvalues
Proof.
We know that |δλ| ≤ κp(X ) = O(). In addition, δx = O() when λ is a simple eigenvalue (proof omitted). Because (A + δA)(x + δx) = (λ + δλ)(x + δx), thus
(δA) x + A (δx) + O(2) = (δλ)x + λ(δx) + O(2).
Left multiplying by y ∗ and using equation y ∗A = λy ∗, we obtain
y ∗ (δA) x + O(2) = (δλ) y ∗x + O(2)
and hence y ∗ (δA) x δλ = + O(2). y ∗x ∗ 1 2 Since |y (δA) x| ≤ kyk2 k(δA)k2 kxk2 = , |δλ| ≤ |y ∗x| + O( ).
Xiangmin Jiao Numerical Analysis I 7 / 12 Sensitivity of Multiple Eigenvalues and Eigenvectors
Sensitivity of multiple eigenvalues is more complicated
I For multiple eigenvalues, left and right eigenvectors can be orthogonal, hence very ill-conditioned I In general, multiple or close eigenvalues can be poorly conditioned, especially if matrix is defective Condition numbers of eigenvectors are also difficult to analyze
I If matrix has well-conditioned and well-separated eigenvalues, then eigenvectors are well-conditioned I If eigenvalues are ill-conditioned or closely clustered, then eigenvectors may be poorly conditioned
Xiangmin Jiao Numerical Analysis I 8 / 12 Outline
1 Sensitivity of Eigenvalues (MC§7.2)
2 Review for Midterm #2
Xiangmin Jiao Numerical Analysis I 9 / 12 Algorithms
QR factorization
I Classical and modified Gram-Schmidt I QR factorization using Householder triangularization and Givens rotation Solutions of linear least squares
I Solution using Householder QR and other QR factorization I Alternative solutions: normal equation; SVD Conditioning and stability of linear least squares problems QR factorization with pivoting
Xiangmin Jiao Numerical Analysis I 10 / 12 Eigenvalue Problem
Eigenvalue problem of n × n matrix A is Ax = λx Characteristic polynomial is det(A − λI ) Eigenvalue decomposition of A is A = X ΛX −1 (does not always exist) Geometric multiplicity of λ is dim(null(A − λI )), and algebraic multiplicity of λ is its multiplicity as a root of pA, where algebraic multiplicity ≥ geometric multiplicity Similarity transformations preserve eigenvalues and their algebraic and geometric multiplicities Schur factorization A = QTQ∗ uses unitary similarity transformations
Xiangmin Jiao Numerical Analysis I 11 / 12 Eigenvalue Algorithms
Underlying concepts: power iterations, Rayleigh quotient, inverse iterations, convergence rate Schur factorization is typically done in two steps
I Reduction to Hessenberg form for non-Hermitian matrices or reduction to tridiagonal form for hermitian matrices by unitary similarity transformation I Finding eigenvalues of Hessenberg or tridiagonal form Finding eigenvalue of tridiagonal forms
I QR algorithm with shifts, and their interpretations as (inverse) simultaneous iterations
Xiangmin Jiao Numerical Analysis I 12 / 12