Eigenvalue Problems

Eigenvalue Problems Eigenvalue Problems Existence, Uniqueness, and Conditioning Existence, Uniqueness, and Conditioning Computing Eigenvalues and Eigenvectors Computing Eigenvalues and Eigenvectors Outline

Scientiﬁc Computing: An Introductory Survey Chapter 4 – Eigenvalue Problems 1 Eigenvalue Problems

Prof. Michael T. Heath 2 Existence, Uniqueness, and Conditioning Department of Computer Science University of Illinois at Urbana-Champaign 3 Computing Eigenvalues and Eigenvectors Copyright c 2002. Reproduction permitted for noncommercial, educational use only.

Michael T. Heath Scientiﬁc Computing 1 / 87 Michael T. Heath Scientiﬁc Computing 2 / 87

Eigenvalue Problems Eigenvalue Problems Eigenvalue Problems Eigenvalue Problems Existence, Uniqueness, and Conditioning Eigenvalues and Eigenvectors Existence, Uniqueness, and Conditioning Eigenvalues and Eigenvectors Computing Eigenvalues and Eigenvectors Geometric Interpretation Computing Eigenvalues and Eigenvectors Geometric Interpretation Eigenvalue Problems Eigenvalues and Eigenvectors

Eigenvalue problems occur in many areas of science and Standard eigenvalue problem : Given n × n matrix A, ﬁnd engineering, such as structural analysis scalar λ and nonzero vector x such that

Eigenvalues are also important in analyzing numerical A x = λ x methods λ is eigenvalue, and x is corresponding eigenvector Theory and algorithms apply to complex matrices as well as real matrices λ may be complex even if A is real

With complex matrices, we use conjugate transpose, AH , Spectrum = λ(A) = set of eigenvalues of A T instead of usual transpose, A Spectral radius = ρ(A) = max{|λ| : λ ∈ λ(A)}

Michael T. Heath Scientiﬁc Computing 3 / 87 Michael T. Heath Scientiﬁc Computing 4 / 87

1 0 1 0 A = : λ = 1, x = , λ = 2, x = 0 2 1 1 0 2 2 1 Matrix expands or shrinks any vector lying in direction of 1 1 1 1 A = : λ = 1, x = , λ = 2, x = eigenvector by scalar factor 0 2 1 1 0 2 2 1 Expansion or contraction factor is given by corresponding 3 −1 1 1 A = : λ = 2, x = , λ = 4, x = eigenvalue λ −1 3 1 1 1 2 2 −1 Eigenvalues and eigenvectors decompose complicated 1.5 0.5 1 −1 A = : λ = 2, x = , λ = 1, x = behavior of general linear transformation into simpler 0.5 1.5 1 1 1 2 2 1 actions 0 1 1 i A = : λ = i, x = , λ = −i, x = −1 0 1 1 i 2 2 1 √ where i = −1

Michael T. Heath Scientiﬁc Computing 5 / 87 Michael T. Heath Scientiﬁc Computing 6 / 87

Eigenvalue Problems Characteristic Polynomial Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Computing Eigenvalues and Eigenvectors Conditioning Characteristic Polynomial Example: Characteristic Polynomial

Equation Ax = λx is equivalent to Characteristic polynomial of previous example matrix is (A − λI)x = 0 3 −1 1 0 which has nonzero solution x if, and only if, its matrix is det − λ = singular −1 3 0 1 Eigenvalues of A are roots λi of characteristic polynomial 3 − λ −1 det = det(A − λI) = 0 −1 3 − λ in λ of degree n (3 − λ)(3 − λ) − (−1)(−1) = λ2 − 6λ + 8 = 0 Fundamental Theorem of Algebra implies that n × n matrix A always has n eigenvalues, but they may not be real nor so eigenvalues are given by distinct √ 6 ± 36 − 32 Complex eigenvalues of real matrix occur in complex λ = , or λ = 2, λ = 4 2 1 2 conjugate pairs: if α + iβ√is eigenvalue of real matrix, then so is α − iβ, where i = −1

Michael T. Heath Scientiﬁc Computing 7 / 87 Michael T. Heath Scientiﬁc Computing 8 / 87 Eigenvalue Problems Characteristic Polynomial Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Computing Eigenvalues and Eigenvectors Conditioning Companion Matrix Characteristic Polynomial, continued

Monic polynomial n−1 n p(λ) = c0 + c1λ + ··· + cn−1λ + λ Computing eigenvalues using characteristic polynomial is is characteristic polynomial of companion matrix not recommended because of   0 0 ··· 0 −c0 work in computing coefficients of characteristic polynomial 1 0 ··· 0 −c1  sensitivity of coefficients of characteristic polynomial   0 1 ··· 0 −c2  Cn =   work in solving for roots of characteristic polynomial ......  . . . . .  0 0 ··· 1 −cn−1 Characteristic polynomial is powerful theoretical tool but usually not useful computationally Roots of polynomial of degree > 4 cannot always computed in finite number of steps So in general, computation of eigenvalues of matrices of order > 4 requires (theoretically infinite) iterative process

Michael T. Heath Scientiﬁc Computing 9 / 87 Michael T. Heath Scientiﬁc Computing 10 / 87

Eigenvalue Problems Characteristic Polynomial Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Computing Eigenvalues and Eigenvectors Conditioning Example: Characteristic Polynomial Multiplicity and Diagonalizability

Consider Multiplicity is number of times root appears when 1 A = polynomial is written as product of linear factors 1 √ Eigenvalue of multiplicity 1 is simple where is positive number slightly smaller than mach Exact eigenvalues of A are 1 + and 1 − Defective matrix has eigenvalue of multiplicity k > 1 with fewer than k linearly independent corresponding Computing characteristic polynomial in ﬂoating-point eigenvectors arithmetic, we obtain Nondefective matrix A has n linearly independent 2 2 2 det(A − λI) = λ − 2λ + (1 − ) = λ − 2λ + 1 eigenvectors, so it is diagonalizable

which has 1 as double root X−1AX = D Thus, eigenvalues cannot be resolved by this method even though they are distinct in working precision where X is nonsingular matrix of eigenvectors

Michael T. Heath Scientiﬁc Computing 11 / 87 Michael T. Heath Scientiﬁc Computing 12 / 87

Eigenvalue Problems Characteristic Polynomial Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Computing Eigenvalues and Eigenvectors Conditioning Eigenspaces and Invariant Subspaces Relevant Properties of Matrices

Properties of matrix A relevant to eigenvalue problems Eigenvectors can be scaled arbitrarily: if Ax = λx, then Property Deﬁnition A(γx) = λ(γx) for any scalar γ, so γx is also eigenvector diagonal aij = 0 for i 6= j corresponding to λ tridiagonal aij = 0 for |i − j| > 1 triangular a = 0 for i > j (upper) Eigenvectors are usually normalized by requiring some ij a = 0 for i < j (lower) norm of eigenvector to be 1 ij Hessenberg aij = 0 for i > j + 1 (upper)

Eigenspace = Sλ = {x : Ax = λx} aij = 0 for i < j − 1 (lower)

n n Subspace S of R (or C ) is invariant if AS ⊆ S orthogonal AT A = AAT = I unitary AH A = AAH = I For eigenvectors x ··· x , span([x ··· x ]) is invariant 1 p 1 p symmetric A = AT subspace Hermitian A = AH normal AH A = AAH

Michael T. Heath Scientiﬁc Computing 13 / 87 Michael T. Heath Scientiﬁc Computing 14 / 87

T 1 2 1 3 √ √ Transpose: = 0 1 −1 0 2/2 2/2 3 4 2 4 Orthogonal: , , √ √ 1 0 0 −1 − 2/2 2/2 H √ √ 1 + i 1 + 2i 1 − i 2 + i i 2/2 2/2 Conjugate transpose: = Unitary: √ √ 2 − i 2 − 2i 1 − 2i 2 + 2i − 2/2 −i 2/2 1 2 Symmetric: 1 1 2 3 Nonorthogonal: 1 2 1 3 Nonsymmetric: 1 2 0 2 4 Normal: 0 1 2 1 1 + i Hermitian: 2 0 1 1 − i 2 1 1 1 1 + i Nonnormal: NonHermitian: 0 1 1 + i 2

Michael T. Heath Scientiﬁc Computing 15 / 87 Michael T. Heath Scientiﬁc Computing 16 / 87 Eigenvalue Problems Characteristic Polynomial Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Computing Eigenvalues and Eigenvectors Conditioning Properties of Eigenvalue Problems Conditioning of Eigenvalue Problems

Properties of eigenvalue problem affecting choice of algorithm and software Condition of eigenvalue problem is sensitivity of Are all eigenvalues needed, or only a few? eigenvalues and eigenvectors to changes in matrix Are only eigenvalues needed, or are corresponding Conditioning of eigenvalue problem is not same as eigenvectors also needed? conditioning of solution to linear system for same matrix Is matrix real or complex? Different eigenvalues and eigenvectors are not necessarily Is matrix relatively small and dense, or large and sparse? equally sensitive to perturbations in matrix

Does matrix have any special properties, such as symmetry, or is it general matrix?

Michael T. Heath Scientiﬁc Computing 17 / 87 Michael T. Heath Scientiﬁc Computing 18 / 87

Eigenvalue Problems Characteristic Polynomial Eigenvalue Problems Characteristic Polynomial Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Existence, Uniqueness, and Conditioning Relevant Properties of Matrices Computing Eigenvalues and Eigenvectors Conditioning Computing Eigenvalues and Eigenvectors Conditioning Conditioning of Eigenvalues Conditioning of Eigenvalues

If µ is eigenvalue of perturbation A + E of nondefective If (A + E)(x + ∆x) = (λ + ∆λ)(x + ∆x), where λ is matrix A, then simple eigenvalue of A, then

|µ − λ | ≤ cond (X) kEk kyk · kxk 1 k 2 2 |∆λ| 2 2 kEk = kEk / |yH x| 2 cos(θ) 2 where λk is closest eigenvalue of A to µ and X is nonsingular matrix of eigenvectors of A where x and y are corresponding right and left Absolute condition number of eigenvalues is condition eigenvectors and θ is angle between them number of matrix of eigenvectors with respect to solving For symmetric or Hermitian matrix, right and left linear equations eigenvectors are same, so cos(θ) = 1 and eigenvalues are Eigenvalues may be sensitive if eigenvectors are nearly inherently well-conditioned linearly dependent (i.e., matrix is nearly defective) Eigenvalues of nonnormal matrices may be sensitive For normal matrix (AH A = AAH ), eigenvectors are For multiple or closely clustered eigenvalues, orthogonal, so eigenvalues are well-conditioned corresponding eigenvectors may be sensitive

Michael T. Heath Scientiﬁc Computing 19 / 87 Michael T. Heath Scientiﬁc Computing 20 / 87

Shift : If Ax = λx and σ is any scalar, then (A − σI)x = (λ − σ)x, so eigenvalues of shifted matrix are B is similar to A if there is nonsingular matrix T such that shifted eigenvalues of original matrix B = T −1AT Inversion : If A is nonsingular and Ax = λx with x 6= 0, −1 then λ 6= 0 and A x = (1/λ)x, so eigenvalues of inverse Then are reciprocals of eigenvalues of original matrix By = λy ⇒ T −1AT y = λy ⇒ A(T y) = λ(T y) Powers : If Ax = λx, then Akx = λkx, so eigenvalues of power of matrix are same power of eigenvalues of original so A and B have same eigenvalues, and if y is matrix eigenvector of B, then x = T y is eigenvector of A Polynomial : If Ax = λx and p(t) is polynomial, then Similarity transformations preserve eigenvalues and p(A)x = p(λ)x, so eigenvalues of polynomial in matrix are eigenvectors are easily recovered values of polynomial evaluated at eigenvalues of original matrix

Michael T. Heath Scientiﬁc Computing 21 / 87 Michael T. Heath Scientiﬁc Computing 22 / 87

From eigenvalues and eigenvectors for previous example, Eigenvalues of diagonal matrix are diagonal entries, and eigenvectors are columns of identity matrix 3 −1 1 1 1 1 2 0 = −1 3 1 −1 1 −1 0 4 Diagonal form is desirable in simplifying eigenvalue problems for general matrices by similarity transformations and hence But not all matrices are diagonalizable by similarity 0.5 0.5 3 −1 1 1 2 0 = transformation 0.5 −0.5 −1 3 1 −1 0 4 Closest one can get, in general, is Jordan form, which is So original matrix is similar to diagonal matrix, and nearly diagonal but may have some nonzero entries on ﬁrst eigenvectors form columns of similarity transformation superdiagonal, corresponding to one or more multiple matrix eigenvalues

Michael T. Heath Scientiﬁc Computing 23 / 87 Michael T. Heath Scientiﬁc Computing 24 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Triangular Form Block Triangular Form

Any matrix can be transformed into triangular (Schur ) form If   by similarity, and eigenvalues of triangular matrix are A11 A12 ··· A1p diagonal entries  A22 ··· A2p A =    .. .  Eigenvectors of triangular matrix less obvious, but still  . .  straightforward to compute App If   with square diagonal blocks, then U11 u U13 T p A − λI =  0 0 v  [ λ(A) = λ(Ajj) O 0 U33 j=1 is triangular, then U11y = u can be solved for y, so that p   so eigenvalue problem breaks into smaller eigenvalue y problems x = −1 Real Schur form has 1 × 1 diagonal blocks corresponding 0 to real eigenvalues and 2 × 2 diagonal blocks is corresponding eigenvector corresponding to pairs of complex conjugate eigenvalues

Michael T. Heath Scientiﬁc Computing 25 / 87 Michael T. Heath Scientiﬁc Computing 26 / 87

Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Forms Attainable by Similarity Power Iteration ATB distinct eigenvalues nonsingular diagonal real symmetric orthogonal real diagonal Simplest method for computing one eigenvalue- complex Hermitian unitary real diagonal eigenvector pair is power iteration, which repeatedly normal unitary diagonal multiplies matrix times initial starting vector arbitrary real orthogonal real block triangular (real Schur) Assume A has unique eigenvalue of maximum modulus, arbitrary unitary upper triangular say λ1, with corresponding eigenvector v1 (Schur) arbitrary nonsingular almost diagonal Then, starting from nonzero vector x0, iteration scheme (Jordan) x = Ax Given matrix A with indicated property, matrices B and T k k−1 exist with indicated properties such that B = T −1AT converges to multiple of eigenvector v1 corresponding to If B is diagonal or triangular, eigenvalues are its diagonal dominant eigenvalue λ1 entries If B is diagonal, eigenvectors are columns of T

Michael T. Heath Scientiﬁc Computing 27 / 87 Michael T. Heath Scientiﬁc Computing 28 / 87

Ratio of values of given component of xk from one iteration To see why power iteration converges to dominant to next converges to dominant eigenvalue λ1 eigenvector, express starting vector x0 as linear 1.5 0.5 0 combination For example, if A = and x0 = , we obtain n 0.5 1.5 1 X T x0 = αivi k xk ratio i=1 0 0.0 1.0 where vi are eigenvectors of A 1 0.5 1.5 1.500 Then 2 1.5 2.5 1.667 2 k 3 3.5 4.5 1.800 xk = Axk−1 = A xk−2 = ··· = A x0 = 4 7.5 8.5 1.889 n n ! 5 15.5 16.5 1.941 X k k X k λi αivi = λ1 α1v1 + (λi/λ1) αivi 6 31.5 32.5 1.970 i=1 i=2 7 63.5 64.5 1.985 Since |λi/λ1| < 1 for i > 1, successively higher powers go 8 127.5 128.5 1.992 to zero, leaving only component corresponding to v1 Ratio is converging to dominant eigenvalue, which is 2

Michael T. Heath Scientiﬁc Computing 29 / 87 Michael T. Heath Scientiﬁc Computing 30 / 87

Power iteration can fail for various reasons Geometric growth of components at each iteration risks Starting vector may have no component in dominant eventual overﬂow (or underﬂow if λ1 < 1) eigenvector v1 (i.e., α1 = 0) — not problem in practice because rounding error usually introduces such Approximate eigenvector should be normalized at each component in any case iteration, say, by requiring its largest component to be 1 in modulus, giving iteration scheme There may be more than one eigenvalue having same (maximum) modulus, in which case iteration may converge yk = Axk−1

to linear combination of corresponding eigenvectors xk = yk/kykk∞ For real matrix and starting vector, iteration can never With normalization, kykk∞ → |λ1|, and xk → v1/kv1k∞ converge to complex vector

Michael T. Heath Scientiﬁc Computing 31 / 87 Michael T. Heath Scientiﬁc Computing 32 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Example: Normalized Power Iteration Geometric Interpretation

Repeating previous example with normalized scheme, Behavior of power iteration depicted geometrically

T k xk kykk∞ 0 0.000 1.0 1 0.333 1.0 1.500 2 0.600 1.0 1.667 3 0.778 1.0 1.800 4 0.882 1.0 1.889 5 0.939 1.0 1.941

6 0.969 1.0 1.970 Initial vector x0 = v1 + v2 contains equal components in 7 0.984 1.0 1.985 eigenvectors v1 and v2 (dashed arrows) 8 0.992 1.0 1.992 Repeated multiplication by A causes component in v1 < interactive example > (corresponding to larger eigenvalue, 2) to dominate, so sequence of vectors xk converges to v1

Michael T. Heath Scientiﬁc Computing 33 / 87 Michael T. Heath Scientiﬁc Computing 34 / 87

Convergence rate of power iteration depends on ratio |λ2/λ1|, where λ2 is eigenvalue having second largest modulus In earlier example, for instance, if we pick shift of σ = 1, (which is equal to other eigenvalue) then ratio becomes May be possible to choose shift, A − σI, such that zero and method converges in one iteration

λ2 − σ λ2 < In general, we would not be able to make such fortuitous λ1 − σ λ1 choice, but shifts can still be extremely useful in some so convergence is accelerated contexts, as we will see later

Shift must then be added to result to obtain eigenvalue of original matrix

Michael T. Heath Scientiﬁc Computing 35 / 87 Michael T. Heath Scientiﬁc Computing 36 / 87

If smallest eigenvalue of matrix required rather than largest, can make use of fact that eigenvalues of A−1 are reciprocals of those of A, so smallest eigenvalue of A is A−1 reciprocal of largest eigenvalue of Inverse iteration converges to eigenvector corresponding This leads to inverse iteration scheme to smallest eigenvalue of A Eigenvalue obtained is dominant eigenvalue of A−1, and Ayk = xk−1 hence its reciprocal is smallest eigenvalue of A in modulus xk = yk/kykk∞

which is equivalent to power iteration applied to A−1

Inverse of A not computed explicitly, but factorization of A used to solve system of linear equations at each iteration

Michael T. Heath Scientiﬁc Computing 37 / 87 Michael T. Heath Scientiﬁc Computing 38 / 87

Applying inverse iteration to previous example to compute smallest eigenvalue yields sequence As before, shifting strategy, working with A − σI for some T k xk kykk∞ scalar σ, can greatly improve convergence 0 0.000 1.0 1 −0.333 1.0 0.750 Inverse iteration is particularly useful for computing 2 −0.600 1.0 0.833 eigenvector corresponding to approximate eigenvalue, 3 −0.778 1.0 0.900 since it converges rapidly when applied to shifted matrix 4 −0.882 1.0 0.944 A − λI, where λ is approximate eigenvalue 5 −0.939 1.0 0.971 Inverse iteration is also useful for computing eigenvalue 6 −0.969 1.0 0.985 closest to given value β, since if β is used as shift, then which is indeed converging to 1 (which is its own reciprocal desired eigenvalue corresponds to smallest eigenvalue of in this case) shifted matrix < interactive example >

Michael T. Heath Scientiﬁc Computing 39 / 87 Michael T. Heath Scientiﬁc Computing 40 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Rayleigh Quotient Example: Rayleigh Quotient

Rayleigh quotient can accelerate convergence of iterative Given approximate eigenvector x for real matrix A, methods such as power iteration, since Rayleigh quotient determining best estimate for corresponding eigenvalue λ T T xk Axk/xk xk gives better approximation to eigenvalue at can be considered as n × 1 linear least squares iteration k than does basic method alone approximation problem For previous example using power iteration, value of Rayleigh quotient at each iteration is shown below xλ =∼ Ax T T T k xk kykk∞ xk Axk/xk xk From normal equation xT xλ = xT Ax, least squares 0 0.000 1.0 solution is given by 1 0.333 1.0 1.500 1.500 xT Ax λ = 2 0.600 1.0 1.667 1.800 xT x 3 0.778 1.0 1.800 1.941 This quantity, known as Rayleigh quotient, has many useful 4 0.882 1.0 1.889 1.985 properties 5 0.939 1.0 1.941 1.996 6 0.969 1.0 1.970 1.999

Michael T. Heath Scientiﬁc Computing 41 / 87 Michael T. Heath Scientiﬁc Computing 42 / 87

Given approximate eigenvector, Rayleigh quotient yields good estimate for corresponding eigenvalue Rayleigh quotient iteration is especially effective for Conversely, inverse iteration converges rapidly to symmetric matrices and usually converges very rapidly eigenvector if approximate eigenvalue is used as shift, with Using different shift at each iteration means matrix must be one iteration often sufﬁcing refactored each time to solve linear system, so cost per These two ideas combined in Rayleigh quotient iteration iteration is high unless matrix has special form that makes factorization easy T T σk = x Axk/x xk k k Same idea also works for complex matrices, for which

(A − σkI)yk+1 = xk transpose is replaced by conjugate transpose, so Rayleigh quotient becomes xH Ax/xH x xk+1 = yk+1/kyk+1k∞

starting from given nonzero vector x0

Michael T. Heath Scientiﬁc Computing 43 / 87 Michael T. Heath Scientiﬁc Computing 44 / 87

After eigenvalue λ1 and corresponding eigenvector x1 have been computed, then additional eigenvalues Using same matrix as previous examples and randomly λ2, . . . , λn of A can be computed by deﬂation, which effectively removes known eigenvalue chosen starting vector x0, Rayleigh quotient iteration

converges in two iterations Let H be any nonsingular matrix such that Hx1 = αe1, scalar multiple of ﬁrst column of identity matrix k xT σ k k (Householder transformation is good choice for H) 0 0.807 0.397 1.896 1 0.924 1.000 1.998 Then similarity transformation determined by H transforms 2 1.000 1.000 2.000 A into form λ bT HAH−1 = 1 0 B where B is matrix of order n − 1 having eigenvalues λ2, . . . , λn

Michael T. Heath Scientiﬁc Computing 45 / 87 Michael T. Heath Scientiﬁc Computing 46 / 87

Alternative approach lets u be any vector such that Thus, we can work with B to compute next eigenvalue λ2 1 T u1 x1 = λ1 Moreover, if y2 is eigenvector of B corresponding to λ2, T then Then A − x1u1 has eigenvalues 0, λ2, . . . , λn T −1 α b y2 x2 = H , where α = Possible choices for u1 include y2 λ2 − λ1 u1 = λ1x1, if A is symmetric and x1 is normalized so that is eigenvector corresponding to λ2 for original matrix A, kx1k2 = 1 provided λ 6= λ 1 2 u1 = λ1y1, where y1 is corresponding left eigenvector (i.e., AT y = λ y ) normalized so that yT x = 1 Process can be repeated to ﬁnd additional eigenvalues 1 1 1 1 1 u = AT e , if x is normalized so that kx k = 1 and kth and eigenvectors 1 k 1 1 ∞ component of x1 is 1

Michael T. Heath Scientiﬁc Computing 47 / 87 Michael T. Heath Scientiﬁc Computing 48 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Simultaneous Iteration Orthogonal Iteration

As with power iteration, normalization is needed with Simplest method for computing many eigenvalue- simultaneous iteration

eigenvector pairs is simultaneous iteration, which Each column of Xk converges to dominant eigenvector, so repeatedly multiplies matrix times matrix of initial starting columns of Xk become increasingly ill-conditioned basis vectors for span(Xk)

Starting from n × p matrix X0 of rank p, iteration scheme is Both issues can be addressed by computing QR factorization at each iteration Xk = AXk−1 ˆ QkRk = Xk−1 X = AQˆ span(Xk) converges to invariant subspace determined by k k ˆ p largest eigenvalues of A, provided |λp| > |λp+1| where QkRk is reduced QR factorization of Xk−1 Also called subspace iteration This orthogonal iteration converges to block triangular form, and leading block is triangular if moduli of consecutive eigenvalues are distinct

Michael T. Heath Scientiﬁc Computing 49 / 87 Michael T. Heath Scientiﬁc Computing 50 / 87

For p = n and X0 = I, matrices Successive matrices A are unitarily similar to each other ˆH ˆ k Ak = Qk AQk H generated by orthogonal iteration converge to triangular or Ak = RkQk = Qk Ak−1Qk block triangular form, yielding all eigenvalues of A Diagonal entries (or eigenvalues of diagonal blocks) of Ak QR iteration computes successive matrices Ak without converge to eigenvalues of A forming above product explicitly Product of orthogonal matrices Q converges to matrix of Starting with A = A, at iteration k compute QR k 0 corresponding eigenvectors factorization QkRk = Ak−1 If A is symmetric, then symmetry is preserved by QR and form reverse product iteration, so Ak converge to matrix that is both triangular and symmetric, hence diagonal Ak = RkQk

Michael T. Heath Scientiﬁc Computing 51 / 87 Michael T. Heath Scientiﬁc Computing 52 / 87

7 2 Let A = 0 2 4 Compute QR factorization Convergence rate of QR iteration can be accelerated by .962 −.275 7.28 3.02 incorporating shifts A = Q R = 0 1 1 .275 .962 0 3.30 QkRk = Ak−1 − σkI and form reverse product Ak = RkQk + σkI 7.83 .906 A1 = R1Q1 = .906 3.17 where σk is rough approximation to eigenvalue Off-diagonal entries are now smaller, and diagonal entries Good shift can be determined by computing eigenvalues of closer to eigenvalues, 8 and 3 2 × 2 submatrix in lower right corner of matrix Process continues until matrix is within tolerance of being diagonal, and diagonal entries then closely approximate eigenvalues

Michael T. Heath Scientiﬁc Computing 53 / 87 Michael T. Heath Scientiﬁc Computing 54 / 87

Repeat previous example, but with shift of σ = 4, which is 1 Efficiency of QR iteration can be enhanced by first lower right corner entry of matrix transforming matrix as close to triangular form as possible We compute QR factorization before beginning iterations .832 .555 3.61 1.66 A − σ I = Q R = Hessenberg matrix is triangular except for one additional 0 1 1 1 .555 −.832 0 1.11 nonzero diagonal immediately adjacent to main diagonal and form reverse product, adding back shift to obtain Any matrix can be reduced to Hessenberg form in finite 7.92 .615 A = R Q + σ I = number of steps by orthogonal similarity transformation, for 1 1 1 1 .615 3.08 example using Householder transformations After one iteration, off-diagonal entries smaller compared with unshifted algorithm, and eigenvalues closer Symmetric Hessenberg matrix is tridiagonal approximations to eigenvalues Hessenberg or tridiagonal form is preserved during < interactive example > successive QR iterations

Michael T. Heath Scientiﬁc Computing 55 / 87 Michael T. Heath Scientiﬁc Computing 56 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Preliminary Reduction, continued Preliminary Reduction, continued

QR iteration is implemented in two-stages Advantages of initial reduction to upper Hessenberg or symmetric −→ tridiagonal −→ diagonal tridiagonal form or Work per QR iteration is reduced from O(n3) to O(n2) for general matrix or O(n) for symmetric matrix general −→ Hessenberg −→ triangular Fewer QR iterations are required because matrix nearly Preliminary reduction requires deﬁnite number of steps, triangular (or diagonal) already whereas subsequent iterative stage continues until convergence If any zero entries on ﬁrst subdiagonal, then matrix is block triangular and problem can be broken into two or more In practice only modest number of iterations usually smaller subproblems required, so much of work is in preliminary reduction Cost of accumulating eigenvectors, if needed, dominates total cost

Michael T. Heath Scientiﬁc Computing 57 / 87 Michael T. Heath Scientiﬁc Computing 58 / 87

Krylov subspace methods reduce matrix to Hessenberg (or tridiagonal) form using only matrix-vector multiplication Approximate overall cost of preliminary reduction and QR For arbitrary starting vector x0, if iteration, counting both additions and multiplications k−1 Kk = x0 Ax0 ··· A x0 Symmetric matrices then 4 n3 for eigenvalues only −1 3 Kn AKn = Cn 9n3 for eigenvalues and eigenvectors where Cn is upper Hessenberg (in fact, companion matrix) General matrices To obtain better conditioned basis for span(Kn), compute 10n3 for eigenvalues only QR factorization Q R = K 25n3 for eigenvalues and eigenvectors n n n so that H −1 Qn AQn = RnCnRn ≡ H with H upper Hessenberg

Michael T. Heath Scientiﬁc Computing 59 / 87 Michael T. Heath Scientiﬁc Computing 60 / 87

Equating kth columns on each side of equation AQn = QnH yields recurrence x0 = arbitrary nonzero starting vector q1 = x0/kx0k2 Aqk = h1kq1 + ··· + hkkqk + hk+1,kqk+1 for k = 1, 2,... uk = Aqk relating qk+1 to preceding vectors q1,..., qk for j = 1 to k H H hjk = qj uk Premultiplying by qj and using orthonormality, uk = uk − hjkqj H end hjk = qj Aqk, j = 1, . . . , k hk+1,k = kukk2 These relationships yield Arnoldi iteration, which produces if hk+1,k = 0 then stop unitary matrix Qn and upper Hessenberg matrix Hn qk+1 = uk/hk+1,k column by column using only matrix-vector multiplication end by A and inner products of vectors

Michael T. Heath Scientiﬁc Computing 61 / 87 Michael T. Heath Scientiﬁc Computing 62 / 87

If Qk = q1 ··· qk Arnoldi iteration fairly expensive in work and storage then because each new vector qk must be orthogonalized H against all previous columns of Qk, and all must be stored Hk = Qk AQk for that purpose. is upper Hessenberg matrix So Arnoldi process usually restarted periodically with Eigenvalues of Hk, called Ritz values, are approximate carefully chosen starting vector eigenvalues of A, and Ritz vectors given by Qky, where y is eigenvector of Hk, are corresponding approximate Ritz values and vectors produced are often good eigenvectors of A approximations to eigenvalues and eigenvectors of A after relatively few iterations Eigenvalues of Hk must be computed by another method, such as QR iteration, but this is easier problem if k n

Michael T. Heath Scientific Computing 63 / 87 Michael T. Heath Scientific Computing 64 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Lanczos Iteration Lanczos Iteration, continued Work and storage costs drop dramatically if matrix is symmetric or Hermitian, since recurrence then has only three terms and Hk is tridiagonal (so usually denoted Tk) αk and βk are diagonal and subdiagonal entries of symmetric tridiagonal matrix Tk q0 = 0 β0 = 0 As with Arnoldi, Lanczos iteration does not produce x0 = arbitrary nonzero starting vector eigenvalues and eigenvectors directly, but only tridiagonal q1 = x0/kx0k2 matrix Tk, whose eigenvalues and eigenvectors must be for k = 1, 2,... computed by another method to obtain Ritz values and uk = Aqk vectors H αk = qk uk uk = uk − βk−1qk−1 − αkqk If βk = 0, then algorithm appears to break down, but in that βk = kukk2 case invariant subspace has already been identified (i.e., if βk = 0 then stop Ritz values and vectors are already exact at that point) qk+1 = uk/βk end

Michael T. Heath Scientiﬁc Computing 65 / 87 Michael T. Heath Scientiﬁc Computing 66 / 87

In principle, if Lanczos algorithm were run until k = n, Great virtue of Arnoldi and Lanczos iterations is their ability resulting tridiagonal matrix would be orthogonally similar to to produce good approximations to extreme eigenvalues A for k n

In practice, rounding error causes loss of orthogonality, Moreover, they require only one matrix-vector multiplication invalidating this expectation by A per step and little auxiliary storage, so are ideally suited to large sparse matrices Problem can be overcome by reorthogonalizing vectors as needed, but expense can be substantial If eigenvalues are needed in middle of spectrum, say near σ, then algorithm can be applied to matrix (A − σI)−1, Alternatively, can ignore problem, in which case algorithm assuming it is practical to solve systems of form still produces good eigenvalue approximations, but multiple (A − σI)x = y copies of some eigenvalues may be generated

Michael T. Heath Scientiﬁc Computing 67 / 87 Michael T. Heath Scientiﬁc Computing 68 / 87

Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Example: Lanczos Iteration Jacobi Method One of oldest methods for computing eigenvalues is Jacobi For 29 × 29 symmetric matrix with eigenvalues 1,..., 29, method, which uses similarity transformation based on behavior of Lanczos iteration is shown below plane rotations Sequence of plane rotations chosen to annihilate symmetric pairs of matrix entries, eventually converging to diagonal form Choice of plane rotation slightly more complicated than in Givens method for QR factorization To annihilate given off-diagonal pair, choose c and s so that c −s a b c s J T AJ = s c b d −s c c2a − 2csb + s2d c2b + cs(a − d) − s2b = c2b + cs(a − d) − s2b c2d + 2csb + s2a is diagonal

Michael T. Heath Scientiﬁc Computing 69 / 87 Michael T. Heath Scientiﬁc Computing 70 / 87

Transformed matrix diagonal if Consider 2 × 2 matrix c2b + cs(a − d) − s2b = 0 1 2 2 A = Dividing both sides by c b, we obtain 2 1 s (a − d) s2 1 + − = 0 Quadratic equation for tangent reduces to t2 = 1, so c b c2 t = ±1 Making substitution t = s/c, we get quadratic equation (a − d) Two roots of same magnitude,√ so we arbitrarily√ choose 1 + t − t2 = 0 b t = −1, which yields c = 1/ 2 and s = −1/ 2 for tangent t of√ angle of rotation, from which we can Resulting plane rotation J gives recover c = 1/( 1 + t2) and s = c · t √ √ √ √ 1/ 2 1/ 2 1 2 1/ 2 −1/ 2 3 0 J T AJ = √ √ √ √ = Advantageous numerically to use root of smaller −1/ 2 1/ 2 2 1 1/ 2 1/ 2 0 −1 magnitude

Michael T. Heath Scientiﬁc Computing 71 / 87 Michael T. Heath Scientiﬁc Computing 72 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Jacobi Method, continued Jacobi Method, continued

A = A Starting with symmetric matrix 0 , each iteration has Jacobi method is reliable, simple to program, and capable form of high accuracy, but converges rather slowly and difﬁcult A = J T A J k+1 k k k to generalize beyond symmetric matrices where J is plane rotation chosen to annihilate a k Except for small problems, more modern methods usually symmetric pair of entries in A k require 5 to 10 times less work than Jacobi Plane rotations repeatedly applied from both sides in One source of inefﬁciency is that previously annihilated systematic sweeps through matrix until off-diagonal mass entries can subsequently become nonzero again, thereby of matrix is reduced to within some tolerance of zero requiring repeated annihilation Resulting diagonal matrix orthogonally similar to original Newer methods such as QR iteration preserve zero entries matrix, so diagonal entries are eigenvalues, and introduced into matrix eigenvectors are given by product of plane rotations

Michael T. Heath Scientiﬁc Computing 73 / 87 Michael T. Heath Scientiﬁc Computing 74 / 87

1 0 2 Let A0 = 0 2 1 2 1 1 Next annihilate (1,2) and (2,1) entries using rotation 0.888 −0.460 0 First annihilate (1,3) and (3,1) entries using rotation J1 = 0.460 0.888 0 0.707 0 −0.707 0 0 1 J = 0 1 0 0   to obtain 0.707 0 0.707 3.366 0 0.325 to obtain T A2 = J1 A1J1 =  0 1.634 0.628  3 0.707 0  0.325 0.628 −1 T A1 = J0 A0J0 = 0.707 2 0.707 0 0.707 −1

Michael T. Heath Scientiﬁc Computing 75 / 87 Michael T. Heath Scientiﬁc Computing 76 / 87

Next annihilate (2,3) and (3,2) entries using rotation Beginning new sweep, again annihilate (1,3) and (3,1) entries using rotation 1 0 0  0.998 0 −0.070 J2 = 0 0.975 −0.221 0 0.221 0.975 J3 =  0 1 0  0.070 0 0.998 to obtain to obtain   3.366 0.072 0.317   T 3.388 0.072 0 A3 = J2 A2J2 = 0.072 1.776 0  T 0.317 0 −1.142 A4 = J3 A3J3 = 0.072 1.776 −0.005 0 −0.005 −1.164

Michael T. Heath Scientiﬁc Computing 77 / 87 Michael T. Heath Scientiﬁc Computing 78 / 87

For real symmetric matrix, can determine how many eigenvalues are less than given real number σ Process continues until off-diagonal entries reduced to as small as desired By systematically choosing various values for σ (slicing spectrum at σ) and monitoring resulting count, any Result is diagonal matrix orthogonally similar to original eigenvalue can be isolated as accurately as desired matrix, with the orthogonal similarity transformation given T by product of plane rotations For example, symmetric indeﬁnite factorization A = LDL makes inertia (numbers of positive, negative, and zero < interactive example > eigenvalues) of symmetric matrix A easy to determine By applying factorization to matrix A − σI for various values of σ, individual eigenvalues can be isolated as accurately as desired using interval bisection technique

Michael T. Heath Scientiﬁc Computing 79 / 87 Michael T. Heath Scientiﬁc Computing 80 / 87 Eigenvalue Problems Problem Transformations Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing Eigenvalues and Eigenvectors Other Methods Sturm Sequence Divide-and-Conquer Method

Another spectrum-slicing method for computing individual Express symmetric tridiagonal matrix T as eigenvalues is based on Sturm sequence property of T O symmetric matrices T = 1 + β uuT OT2 Let A be symmetric matrix and let pr(σ) denote determinant of leading principal minor of order r of A − σI Can now compute eigenvalues and eigenvectors of smaller matrices T1 and T2 Then zeros of pr(σ) strictly separate those of pr−1(σ), and number of agreements in sign of successive members of To relate these back to eigenvalues and eigenvectors of sequence pr(σ), for r = 1, . . . , n, equals number of original matrix requires solution of secular equation, which eigenvalues of A strictly greater than σ can be done reliably and efﬁciently

Determinants pr(σ) are easy to compute if A is Applying this approach recursively yields transformed to tridiagonal form before applying Sturm divide-and-conquer algorithm for symmetric tridiagonal sequence technique eigenproblems

Michael T. Heath Scientiﬁc Computing 81 / 87 Michael T. Heath Scientiﬁc Computing 82 / 87

With conventional methods, cost of computing eigenvalues Generalized eigenvalue problem has form 2 of symmetric tridiagonal matrix is O(n ), but if orthogonal Ax = λBx eigenvectors are also computed, then cost rises to O(n3) where A and B are given n × n matrices Another possibility is to compute eigenvalues ﬁrst at O(n2) cost, and then compute corresponding eigenvectors If either A or B is nonsingular, then generalized separately using inverse iteration with computed eigenvalue problem can be converted to standard eigenvalues as shifts eigenvalue problem, either −1 −1 Key to making this idea work is computing eigenvalues and (B A)x = λx or (A B)x = (1/λ)x corresponding eigenvectors to very high relative accuracy This is not recommended, since it may cause so that expensive explicit orthogonalization of eigenvectors loss of accuracy due to rounding error is not needed loss of symmetry if A and B are symmetric RRR algorithm exploits this approach to produce Better alternative for generalized eigenvalue problems is 2 eigenvalues and orthogonal eigenvectors at O(n ) cost QZ algorithm

Michael T. Heath Scientiﬁc Computing 83 / 87 Michael T. Heath Scientiﬁc Computing 84 / 87

If A and B are triangular, then eigenvalues are given by λi = aii/bii, for bii 6= 0 Finally, analogous to QR iteration, A is reduced to QZ algorithm reduces A and B simultaneously to upper triangular form while still maintaining triangular form of B, triangular form by orthogonal transformations which again requires transformations from both sides

First, B is reduced to upper triangular form by orthogonal Eigenvalues can now be determined from mutually transformation from left, which is also applied to A triangular form, and eigenvectors can be recovered from products of left and right transformations, denoted by Q Next, transformed A is reduced to upper Hessenberg form and Z by orthogonal transformation from left, while maintaining triangular form of B, which requires additional transformations from right

Michael T. Heath Scientiﬁc Computing 85 / 87 Michael T. Heath Scientiﬁc Computing 86 / 87

Eigenvalue Problems Problem Transformations Existence, Uniqueness, and Conditioning Power Iteration and Variants Computing Eigenvalues and Eigenvectors Other Methods Computing SVD

Singular values of A are nonnegative square roots of eigenvalues of AT A, and columns of U and V are orthonormal eigenvectors of AAT and AT A, respectively Algorithms for computing SVD work directly with A, without forming AAT or AT A, to avoid loss of information associated with forming these matrix products explicitly SVD is usually computed by variant of QR iteration, with A ﬁrst reduced to bidiagonal form by orthogonal transformations, then remaining off-diagonal entries are annihilated iteratively SVD can also be computed by variant of Jacobi method, which can be useful on parallel computers or if matrix has special structure

Michael T. Heath Scientiﬁc Computing 87 / 87