Eigenvalue Problems Existence, Uniqueness, and Conditioning Computing Eigenvalues and Eigenvectors Scientific Computing: An Introductory Survey Chapter 4 – Eigenvalue Problems Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted for noncommercial, educational use only. Michael T. Heath Scientific Computing 1 / 87 Eigenvalue Problems Existence, Uniqueness, and Conditioning Computing Eigenvalues and Eigenvectors Outline 1 Eigenvalue Problems 2 Existence, Uniqueness, and Conditioning 3 Computing Eigenvalues and Eigenvectors Michael T. Heath Scientific Computing 2 / 87 Eigenvalue Problems Eigenvalue Problems Existence, Uniqueness, and Conditioning Eigenvalues and Eigenvectors Computing Eigenvalues and Eigenvectors Geometric Interpretation Eigenvalue Problems Eigenvalue problems occur in many areas of science and engineering, such as structural analysis Eigenvalues are also important in analyzing numerical methods Theory and algorithms apply to complex matrices as well as real matrices With complex matrices, we use conjugate transpose, AH , instead of usual transpose, AT Michael T. Heath Scientific Computing 3 / 87 Eigenvalue Problems Eigenvalue Problems Existence, Uniqueness, and Conditioning Eigenvalues and Eigenvectors Computing Eigenvalues and Eigenvectors Geometric Interpretation Eigenvalues and Eigenvectors Standard eigenvalue problem : Given n × n matrix A, find scalar λ and nonzero vector x such that A x = λ x λ is eigenvalue, and x is corresponding eigenvector λ may be complex even if A is real Spectrum = λ(A) = set of eigenvalues of A Spectral radius = ρ(A) = maxfjλj : λ 2 λ(A)g Michael T. Heath Scientific Computing 4 / 87 Eigenvalue Problems Eigenvalue Problems Existence, Uniqueness, and Conditioning Eigenvalues and Eigenvectors Computing Eigenvalues and Eigenvectors Geometric Interpretation Geometric Interpretation Matrix expands or shrinks any vector lying in direction of eigenvector by scalar factor Expansion or contraction factor is given by corresponding eigenvalue λ Eigenvalues and eigenvectors decompose complicated behavior of general linear transformation into simpler actions Michael T. Heath Scientific Computing 5 / 87 Eigenvalue Problems Eigenvalue Problems Existence, Uniqueness, and Conditioning Eigenvalues and Eigenvectors Computing Eigenvalues and Eigenvectors Geometric Interpretation Examples: Eigenvalues and Eigenvectors 1 0 1 0 A = : λ = 1; x = ; λ = 2; x = 0 2 1 1 0 2 2 1 1 1 1 1 A = : λ = 1; x = ; λ = 2; x = 0 2 1 1 0 2 2 1 3 −1 1 1 A = : λ = 2; x = ; λ = 4; x = −1 3 1 1 1 2 2 −1 1:5 0:5 1 −1 A = : λ = 2; x = ; λ = 1; x = 0:5 1:5 1 1 1 2 2 1 0 1 1 i A = : λ = i; x = ; λ = −i; x = −1 0 1 1 i 2 2 1 p where i = −1 Michael T. Heath Scientific Computing 6 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Characteristic Polynomial Equation Ax = λx is equivalent to (A − λI)x = 0 which has nonzero solution x if, and only if, its matrix is singular Eigenvalues of A are roots λi of characteristic polynomial det(A − λI) = 0 in λ of degree n Fundamental Theorem of Algebra implies that n × n matrix A always has n eigenvalues, but they may not be real nor distinct Complex eigenvalues of real matrix occur in complex conjugate pairs: if α + iβpis eigenvalue of real matrix, then so is α − iβ, where i = −1 Michael T. Heath Scientific Computing 7 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Example: Characteristic Polynomial Characteristic polynomial of previous example matrix is 3 −1 1 0 det − λ = −1 3 0 1 3 − λ −1 det = −1 3 − λ (3 − λ)(3 − λ) − (−1)(−1) = λ2 − 6λ + 8 = 0 so eigenvalues are given by p 6 ± 36 − 32 λ = ; or λ = 2; λ = 4 2 1 2 Michael T. Heath Scientific Computing 8 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Companion Matrix Monic polynomial n−1 n p(λ) = c0 + c1λ + ··· + cn−1λ + λ is characteristic polynomial of companion matrix 2 3 0 0 ··· 0 −c0 61 0 ··· 0 −c1 7 6 7 60 1 ··· 0 −c2 7 Cn = 6 7 6. .. 7 4. 5 0 0 ··· 1 −cn−1 Roots of polynomial of degree > 4 cannot always computed in finite number of steps So in general, computation of eigenvalues of matrices of order > 4 requires (theoretically infinite) iterative process Michael T. Heath Scientific Computing 9 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Characteristic Polynomial, continued Computing eigenvalues using characteristic polynomial is not recommended because of work in computing coefficients of characteristic polynomial sensitivity of coefficients of characteristic polynomial work in solving for roots of characteristic polynomial Characteristic polynomial is powerful theoretical tool but usually not useful computationally Michael T. Heath Scientific Computing 10 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Example: Characteristic Polynomial Consider 1 A = 1 p where is positive number slightly smaller than mach Exact eigenvalues of A are 1 + and 1 − Computing characteristic polynomial in floating-point arithmetic, we obtain det(A − λI) = λ2 − 2λ + (1 − 2) = λ2 − 2λ + 1 which has 1 as double root Thus, eigenvalues cannot be resolved by this method even though they are distinct in working precision Michael T. Heath Scientific Computing 11 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Multiplicity and Diagonalizability Multiplicity is number of times root appears when polynomial is written as product of linear factors Eigenvalue of multiplicity 1 is simple Defective matrix has eigenvalue of multiplicity k > 1 with fewer than k linearly independent corresponding eigenvectors Nondefective matrix A has n linearly independent eigenvectors, so it is diagonalizable X−1AX = D where X is nonsingular matrix of eigenvectors Michael T. Heath Scientific Computing 12 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Eigenspaces and Invariant Subspaces Eigenvectors can be scaled arbitrarily: if Ax = λx, then A(γx) = λ(γx) for any scalar γ, so γx is also eigenvector corresponding to λ Eigenvectors are usually normalized by requiring some norm of eigenvector to be 1 Eigenspace = Sλ = fx : Ax = λxg n n Subspace S of R (or C ) is invariant if AS ⊆ S For eigenvectors x1 ··· xp, span([x1 ··· xp]) is invariant subspace Michael T. Heath Scientific Computing 13 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Relevant Properties of Matrices Properties of matrix A relevant to eigenvalue problems Property Definition diagonal aij = 0 for i 6= j tridiagonal aij = 0 for ji − jj > 1 triangular aij = 0 for i > j (upper) aij = 0 for i < j (lower) Hessenberg aij = 0 for i > j + 1 (upper) aij = 0 for i < j − 1 (lower) orthogonal AT A = AAT = I unitary AH A = AAH = I symmetric A = AT Hermitian A = AH normal AH A = AAH Michael T. Heath Scientific Computing 14 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Examples: Matrix Properties 1 2T 1 3 Transpose: = 3 4 2 4 1 + i 1 + 2iH 1 − i 2 + i Conjugate transpose: = 2 − i 2 − 2i 1 − 2i 2 + 2i 1 2 Symmetric: 2 3 1 3 Nonsymmetric: 2 4 1 1 + i Hermitian: 1 − i 2 1 1 + i NonHermitian: 1 + i 2 Michael T. Heath Scientific Computing 15 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Examples, continued p p 0 1 −1 0 2=2 2=2 Orthogonal: ; ; p p 1 0 0 −1 − 2=2 2=2 p p i 2=2 2=2 Unitary: p p − 2=2 −i 2=2 1 1 Nonorthogonal: 1 2 21 2 03 Normal: 40 1 25 2 0 1 1 1 Nonnormal: 0 1 Michael T. Heath Scientific Computing 16 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Properties of Eigenvalue Problems Properties of eigenvalue problem affecting choice of algorithm and software Are all eigenvalues needed, or only a few? Are only eigenvalues needed, or are corresponding eigenvectors also needed? Is matrix real or complex? Is matrix relatively small and dense, or large and sparse? Does matrix have any special properties, such as symmetry, or is it general matrix? Michael T. Heath Scientific Computing 17 / 87 Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Conditioning of Eigenvalue Problems Condition of eigenvalue problem is sensitivity of eigenvalues and eigenvectors to changes in matrix Conditioning of eigenvalue problem is not same as conditioning of solution to linear system