AMS526: I (Numerical for Computational and Data Sciences) Lecture 18: Sensitivity of Eigenvalues; Review for Midterm #2

Xiangmin Jiao

Stony Brook University

Xiangmin Jiao Numerical Analysis I 1 / 12 Outline

1 Sensitivity of Eigenvalues (MC§7.2)

2 Review for Midterm #2

Xiangmin Jiao Numerical Analysis I 2 / 12 Sensitivity of Eigenvalues

Condition number of X determines sensitivity of eigenvalues

Theorem n×n −1 Let A ∈ C be a nondefective matrix, and suppose A = X ΛX , where n×n X is nonsingular and Λ is diagonal. Let δA ∈ C be some perturbation of A, and let µ be an eigenvalue of A + δA. Then A has an eigenvalue λ such that |µ − λ| ≤ κp(X )kδAkp for 1 ≤ p ≤ ∞.

κp(X ) measures how far eigenvectors are from linear dependence

For normal matrices, condition number κ2(X ) = 1 and κp(X ) = O(1), so eigenvalues of normal matrices are always well-conditioned

Xiangmin Jiao Numerical Analysis I 3 / 12 Sensitivity of Eigenvalues

Proof. Let δΛ = X −1 (δA) X . Then

−1 kδΛkp ≤ kX kpkδAkpkX kp = κp(X )kδAkp.

Let y be an engenvector of Λ + δΛ associated with µ. Suppose µ is not an eigenvalue of A, so µI − Λ is nonsingular. (Λ + δΛ)y = µy ⇒ (µI − Λ)y = (δΛ) y ⇒ y = (µI − Λ)−1 (δΛ) y. Thus

−1 −1 k(µI − Λ) kp ≤ kδΛkp.

−1 −1 k(µI − Λ) kp = |µ − λ| , where λ is the eigenvalue of A closest to µ. Thus, |µ − λ| ≤ kδΛkp ≤ κp(X )kδAkp.

Xiangmin Jiao Numerical Analysis I 4 / 12 Left and Right Eigenvectors To analyze sensitivity of individual eigenvalues, we need to define left and right eigenvectors

I Ax = λx for nonzero x then x is right eigenvector associated with λ ∗ ∗ I y A = λy for nonzero y, then y is left eigenvector associated with λ Left eigenvectors of A are right eigenvectors of A∗

Theorem n×n Let A ∈ C have distinct eigenvalues λ1, λ2, . . . , λn with associated linearly independent right eigenvectors x1,..., xn and left eigenvectors ∗ ∗ y1,..., yn. Then yj xi 6= 0 if i = j and yj xi = 0 if i 6= j.

Proof. ∗ ∗ ∗ ∗ ∗ If i 6= j, yj Axi = λi yj xi and yj Axi = λj yj xi . Since λi 6= λj , yj xi = 0. n ∗ ∗ If i = j, since {xi } form a basis for C , yi xi = 0 together with yi xj = 0 would imply that yi = 0. This leads to a contradiction.

Xiangmin Jiao Numerical Analysis I 5 / 12 Sensitivity of Individual Eigenvalues

We analyze sensitivity of individual eigenvalues that are distinct

Theorem n×n Let A ∈ C have n distinct eigenvalues. Let λ be an eigenvalue with associated right and left eigenvectors x and y, respectively, normalized so that kxk2 = kyk2 = 1. Let δA be a small perturbation satisfying kδAk2 = , and let λ + δλ be the eigenvalue of A + δA that approximates λ. Then 1 |δλ| ≤  + O(2). |y ∗x|

κ = 1/|y ∗x| is condition number for eigenvalue λ A simple eigenvalue is sensitive if its associated right and left eigenvectors are nearly orthogonal

Xiangmin Jiao Numerical Analysis I 6 / 12 Sensitivity of Individual Eigenvalues

Proof.

We know that |δλ| ≤ κp(X ) = O(). In addition, δx = O() when λ is a simple eigenvalue (proof omitted). Because (A + δA)(x + δx) = (λ + δλ)(x + δx), thus

(δA) x + A (δx) + O(2) = (δλ)x + λ(δx) + O(2).

Left multiplying by y ∗ and using equation y ∗A = λy ∗, we obtain

y ∗ (δA) x + O(2) = (δλ) y ∗x + O(2)

and hence y ∗ (δA) x δλ = + O(2). y ∗x ∗ 1 2 Since |y (δA) x| ≤ kyk2 k(δA)k2 kxk2 = , |δλ| ≤ |y ∗x|  + O( ).

Xiangmin Jiao Numerical Analysis I 7 / 12 Sensitivity of Multiple Eigenvalues and Eigenvectors

Sensitivity of multiple eigenvalues is more complicated

I For multiple eigenvalues, left and right eigenvectors can be orthogonal, hence very ill-conditioned I In general, multiple or close eigenvalues can be poorly conditioned, especially if matrix is defective Condition numbers of eigenvectors are also difficult to analyze

I If matrix has well-conditioned and well-separated eigenvalues, then eigenvectors are well-conditioned I If eigenvalues are ill-conditioned or closely clustered, then eigenvectors may be poorly conditioned

Xiangmin Jiao Numerical Analysis I 8 / 12 Outline

1 Sensitivity of Eigenvalues (MC§7.2)

2 Review for Midterm #2

Xiangmin Jiao Numerical Analysis I 9 / 12 Algorithms

QR factorization

I Classical and modified Gram-Schmidt I QR factorization using Householder triangularization and Givens rotation Solutions of

I Solution using Householder QR and other QR factorization I Alternative solutions: normal equation; SVD Conditioning and stability of linear least squares problems QR factorization with pivoting

Xiangmin Jiao Numerical Analysis I 10 / 12 Eigenvalue Problem

Eigenvalue problem of n × n matrix A is Ax = λx Characteristic is det(A − λI ) Eigenvalue decomposition of A is A = X ΛX −1 (does not always exist) Geometric multiplicity of λ is dim(null(A − λI )), and algebraic multiplicity of λ is its multiplicity as a root of pA, where algebraic multiplicity ≥ geometric multiplicity Similarity transformations preserve eigenvalues and their algebraic and geometric multiplicities Schur factorization A = QTQ∗ uses unitary similarity transformations

Xiangmin Jiao Numerical Analysis I 11 / 12 Eigenvalue Algorithms

Underlying concepts: power iterations, Rayleigh quotient, inverse iterations, convergence rate Schur factorization is typically done in two steps

I Reduction to Hessenberg form for non-Hermitian matrices or reduction to tridiagonal form for hermitian matrices by unitary similarity transformation I Finding eigenvalues of Hessenberg or tridiagonal form Finding eigenvalue of tridiagonal forms

I QR algorithm with shifts, and their interpretations as (inverse) simultaneous iterations

Xiangmin Jiao Numerical Analysis I 12 / 12