Lecture Notes on Numerical Methods for Eigenvalues: Part III

Jiangguo (James) Liu ∗ May 4, 2020

This part discusses some advanced numerical methods for eigenvalues. We shall ﬁrst discuss numerical eigensolvers for general matrices, then study numerical eigensolvers for symmetric (Hermitian) matrices.

1 Upper Hessenberg Form

Recall that the general Schur canonical form is an upper triangular matrix, whereas a real Schur canonical form allows 2 × 2 blocks on its diagonal. To unify them, we consider the upper Hessenberg form.

Deﬁnition (Upper Hessenberg form). A matrix H = [hij]n×n is called in the upper Hessenberg form provided that hi,j = 0 if i > j + 1. That is, its 1st sub-diagonal entries are allowed to be nonzero.

Example. The matrix Tn obtained from ﬁnite diﬀerence discretization for the 1-dim Poisson boundary value problem with a homogeneous Dirichlet condition is actually in the upper Hessenberg form.

 2 −1 0 0 ··· 0 0   −1 2 −1 0 ··· 0 0     0 −1 2 −1 ··· 0 0  Tn =   .  ·····················     0 0 0 0 ··· 2 −1  0 0 0 0 · · · −1 2

Actually it is also a symmetric tri-diagonal matrix. Deﬁnition (Unreduced upper Hessenberg form). An upper Hessenberg form is called unreduced provided that all the entries in its 1st sub-diagonal are nonzero.

Note that if an upper Hessenberg form H is not unreduced simply because of hj+1,j = 0, then we can decompose it into two unreduced upper Hessenberg matrices by considering H1 = H(1 : j, 1 : j) and H2 = H((j + 1) : n, (j + 1) : n). Then H1,H2 each is an unreduced

∗Department of Mathematics, Colorado State University, Fort Collins, CO 80523-1874, USA, [email protected]

1 upper Hessenberg matrix and the eigenvalues of H are just in the union of the eigenvalues of H1 and H2.

Our the next job is to simplify a general matrix A into an upper Hessenberg matrix H through an orthogonal or unitary similarity transformation:

QAQT = H. (1)

This can be accomplished by Householder reﬂections and left as an exercise. A good news or a bad news is that this has been implemented

• as hess(A) in Matlab;

• as hessenberg in Python library scipy.linalg.

This implies that ﬁnding the eigenvalues of a general matrix is attributed to ﬁnding the eigenvalues of an upper Hessenberg matrix. It can also be proved that when A is a real symmetric matrix, then its upper Hessenberg canonical form becomes a symmetric tridiagonal matrix.

2 Francis QR-Iterations

The eigenvalues of an upper Hessenberg matrix can be obtained via Francis QR-Iterations. But we need to check certain properties. Lemma 1. If A is upper Hessenberg and A = QR is the QR-factorization obtained via Householder reﬂections, then Q is also upper Hessenberg. Remark. The lemma indicates that if we carry out QR-factorization to a matrix in the upper Hessenberg form, then we obtain an orthogonal matrix that is also in the upper Hessenberg form. Proof. Exercise. Lemma 2. If R is an upper triangular matrix and Q is an orthogonal matrix in the upper Hessenberg form, then B = RQ is also in the upper Hessenberg form. Proof. Exercise. Francis QR-iteration algorithm. This is an iterative algorithm for eigenvalues based on QR-factorization. Shown below is a pseudo-code.

Given a square matrix A; A1 = hess(A); % Reduction to an upper Hessenberg form for k=1 to kmax do % kmax = maximal number of iterations [Qk,Rk] = qr(Ak); % QR-factorization Ak1 = Rk * Qk; Ak = Ak1; % Refreshing end

2 Here are some remarks regarding the above iterative algorithm. (i) Note that T T Ak+1 = RkQk = Qk (QkRk)Qk = Qk AkQk.

So Ak+1 is orthogonally or unitarily similar to Ak and hence all the way back to A1 and ﬁnally A. So the eigenvalues are maintained in this process.

(ii) By Lemma 1, Qk is upper Hessenberg since Ak is so;

(iii) By Lemma 2, Ak+1 is upper Hessenberg since Qk is so;

(iv) If conditions are favorable, then Ak converges to an upper triangular matrix whose diagonal entries are the eigenvalues of A. Operations count for Francis QR-iteration. Three aspects are in consideration. (i) Reduction to an upper Hessenberg for a general order-n square matrix; (ii) Operations needed for QR-factorization (using Householder reﬂections); (iii) Operations needed for RQ multiplication.

QR-iteration with shift. The shift technique can be combined with the QR-iteration algorithm. Shown below is a pseudo-code. Given A; A1 = hess(A); do while until convergence Choose a shift sigmak near an eig.val. of Ak; [Qk,Rk] = qr(Ak-sigmak*I); Ak = Rk*Qk + sigmak*I; k = k+1; enddo Lemma 3. The QR-iteration algorithm with shifts maintains the eigenvalues of the given matrix. Proof. It can be veriﬁed that T T T T Ak+1 = RkQk + σkI = Qk (QkRk)Qk + σkQk Qk = Qk (QkRk + σkI)Qk = Qk AkQk. This is actually an orthogonal similarity transformation. Shown below is a naive implementation of Francis QR-iteration function Am = FrancisQR(A,m) % Initial reduction to upper Hessenberg form Am = hess(A); for k=1:m [Q,R] = qr(Am); Am = R*Q; end % The diagonal entries of Am should be (approximation to) eigenvalues of A return;

3 >> format short; >> A = hilb(5) A = 1.0000 0.5000 0.3333 0.2500 0.2000 0.5000 0.3333 0.2500 0.2000 0.1667 0.3333 0.2500 0.2000 0.1667 0.1429 0.2500 0.2000 0.1667 0.1429 0.1250 0.2000 0.1667 0.1429 0.1250 0.1111 >> e = eig(A) e = 0.0000 0.0003 0.0114 0.2085 1.5671 >> Am = FrancisQR(A,5) Am = 1.5669 0.0124 0.0000 0.0000 -0.0000 0.0124 0.2086 0.0000 -0.0000 0.0000 0 0.0000 0.0114 -0.0000 0.0000 0 0 -0.0000 0.0003 0.0000 0 0 0 0.0000 0.0000 >> Am = FrancisQR(A,10) Am = 1.5671 0.0000 0.0000 0.0000 0.0000 0.0000 0.2085 0.0000 -0.0000 -0.0000 0 0.0000 0.0114 -0.0000 -0.0000 0 0 -0.0000 0.0003 0.0000 0 0 0 -0.0000 0.0000 >> f = diag(Am) f = 1.5671 0.2085 0.0114 0.0003 0.0000 >> f = flipud(f); >> format long; >> [e, f, e-f] ans = 0.000003287928772 0.000003287928772 -0.000000000000000 0.000305898040151 0.000305898040151 0.000000000000000 0.011407491623420 0.011407491623420 0.000000000000000 0.208534218611013 0.208534218611210 -0.000000000000196 1.567050691098231 1.567050691098035 0.000000000000197

4 Unfortunately, the Francis QR-iteration algorithm may fail. One may examine the following matrix  0 0 1  A =  1 0 0  . 0 1 0 The next question will be whether the shifting technique would be make it work. This is left as an exercise.

3 Methods for Symmetric Eigenvalue Problems

As discussed in Textbook Section 5.3, there are a variety of numerical eigen-solvers for symmetric matrices, or more choices as compared to that for general matrices. Note also that

• A general matrix can be reduced via orthogonal similarity transformations to an upper Hessenberg matrix;

• With the same spirit, a symmetric matrix can be simpliﬁed to a symmetric tridiagonal matrix, e.g., hess(A).

Therefore, for some discussion, the focus will be on how to find the eigenvalues of a symmetric tridiagonal matrix. 4 3 Note that for the reduction to a symmetric tridiagonal matrix, the initial cost is 3 n flops 8 3 if only eigenvalues are needed, and 3 n flops if eigenvectors are also desired. As listed on the textbook, there are five eigensolvers to be considered.

(i) Tridiagonal QR-iteration. – Fastest method for ﬁnding all eigenvalues and optionally eigenvectors; – Need only O(n2) operations; – Mainly for small matrices (n ≤ 25); – Used by Matlab; – Implemented in LAPACK as ssyev (dense), sstev (sparse).

(ii) Rayleigh quotient iteration. Skipped

(iii) Divide-and-conquer. – Fastest for finding all eigen-pairs of symmetric tridiagonal matrices of order ≥ 25; – Implemented in LAPACK as sstevd; – Need O(n3) flops in bad cases but O(n2) flops in good cases; – About O(n2.5) for certain experiments.

(iv) Bisection and inverse method – Mainly used to ﬁnd eigenvalues in a speciﬁed interval.

5 (v) Jacobi’s method. – The oldest (bating back to 1846); – Slow; – O(n3) operations; – But still interesting.

We shall focus on only (i) and (iii).

3.1 Symmetric Tridiagonal QR-Iteration

The Francis QR-iteration algorithm can be applied to a symmetric tridiagonal matrix T0 to produce a sequence of symmetric tridiagonal matrices (Tk)k∈N that converges to a diagonal matrix, if conditions are favorable. For this adopted algorithm:

•O (n) ﬂops are need for each iteration;

•O (n2) ﬂops for ﬁnding all n eigenvalues.

For more details, see Textbook p.213. We need to consider shifting techniques for symmetric diagonal matrices also. Let T be such a matrix in the sequence generated by the QR-iteration algorithm. Speciﬁcally,   a1 b1  b1 a2  T =   . (2)  ......  bn−1 an

There are two choices for shifting.

(i) A simple choice: shift = an; a b (ii) Wilkinson’s shift: Choose the eigenvalue of the last 2 × 2 block n−1 n−1 that bn−1 an is closer to an. This is an easy task, The two eigenvalues are surely real because of the symmetry. Finding the eigenvalues boils down to solving a quadratic equation.

Wilkinson’s Theorem. The QR-iteration algorithm with Wilkinson’s shift is globally, and at least linearly, convergent. It is asymptotically cubically convergent for almost all matrices. Proof. See B.Parlett, The symmetric eigenvalue problem, Prentice Hall, Englewood Cliﬀs, New Jersey, 1980. Example for Wilkinson’s shift. See Textbook p.214.

Here is a good exercise. Apply the symmetric tridiagonal algorithm to the Tn matrix obtained from discretization of the 1-dim Laplacian operator. Examine whether the Wilkin- son’s shift makes a diﬀerence.

6 1: % Matlab code tridiQR.m 2: % For "Applied Numerical Linear Algebra", Chapter 5, Example 5.7 3: % Written by James Demmel, Nov 13, 1995 4: % Modified Jun 6, 1997 5: % Modified slightly by James Liu, ColoState; April 2020 6: % Illustrate the cubic convergence of tridiagonal QR with Wilkinson’s shift 7: % Inputs: 8: % T = symmetric tridiagonal matrix on which to run QR 9: % m = number of QR steps to take 10: % 11: % You can also generate a random symmetric n by n matrix, 12: % and reduce it to tridiagonal form as follows: 13: % 14: A = randn(n,n); A = A+A’; % JLNote: There are many other choices for A 15: T = hess(A); 16: disp(’Initial tridiagonal matrix’); 17: T 18: n = min(size(T)); 19: disp(’pause (hit return to continue)’) 20: pause 21: % 22: % Store last subdiagonal and last diagonal entries of QR iterates in NN 23: NN = [ T(n,n-1), T(n,n) ]; 24: % Perform QR iteration with Wilkinson’s shift 25: for i=1:m 26: % JLNote: Compute the eigenvalues of the lower-right 2x2 block 27: lc = T(n-1:n,n-1:n); elc = eig(lc); % JLNote: "eig" may be replaced 28: if ( abs(T(n,n)-elc(1)) < abs(T(n,n)-elc(2)) ), 29: shift = elc(1); 30: else 31: shift = elc(2); 32: end 33: % Perform QR iteration; enforce symmetry explicitly 34: [Q,R] = qr(T-shift*eye(n)); T = R*Q + shift*eye(n); 35: T = tril(triu(T,-1),1); T = (T+T’)/2; 36: % Update NN 37: NN = [NN; [T(n,n-1),T(n,n)]]; 38: end 39: NN = [NN, NN(:,2)-NN(m+1,2)]; 40: disp(’ ’) 41: disp(’ T(n,n-1) T(n,n) T(n,n)-eigenvalue’) 42: disp(’ ------’) 43: NN 44: disp(’Final tridiagonal matrix’) 45: T

7 3.2 Divide-n-Conquer Algorithm

This is a recursive algorithm. This is the fastest algorithm for ﬁnding eigenvalues of large (n ≥ 25) symmetric tridiagonal matrices. Assume n = 2m for convenience. Let T be an order-n symmetric tridiagonal matrix and have the splitting T1 0 T T = + bmuu , 0 T2 where u = [0,..., 0, 1, 1, 0,..., 0]T . Here uuT is an outer product, or a square matrix of order 2m that has four 1s in the middle positions. You may imagine that as the Four Corners located at the intersections of Colorado, Utah, Arizona, and New Mexico.

Suppose that Ti(i = 1, 2) have eigen-decomposition

T Ti = QiDiQi .

Let D = diag(D1,D2) and Q 0 Q = 1 . 0 Q2 Then T T T = Q D + bmuu Q , T T which indicates T is similar to D + bmvv . We also deﬁne v = Q u. It can be veriﬁed that

" T # last column of Q1 v = T . last column of Q2

Therefore, this boils down to ﬁnding the eigenvalues of a matrix in the form

G = D + bvvT , (3)

where D is an order-n diagonal matrix, b ∈ R is a scalar, and v ∈ Rn is a column vector. Theorem 1. Let G be the matrix deﬁned above. If α is an eigenvalue of G but not an eigenvalue of the diagonal matrix D. Then α satisﬁes the secular equation

n X v2 f(λ) = 1 + b i = 0. (4) di − λ I=1

Proof. Assume α is an eigenvalue of G, then det(G − αI) = 0. Note that

G − αI = D + bvvT − αI = (D − αI) + bvvT = (D − αI) I + b(D − αI)−1vvT ,

which implies

det(G − αI) = det(D − αI) det I + b(D − αI)−1vvT = det(D − αI) 1 + bvT (D − αI)−1v .

8 Note for the last step we have used the beautiful identity det(I + xyT ) = 1 + yT x, which relates vectors’ outer product and inner product. Since det(D − αI) 6= 0, we must have T −1 1 + bv (D − αI) v = 0, which is exactly the secular equation. The secular equation is a nonlinear algebraic equation that can be solved by Newton’s method. In the case b 6= 0 and vi 6= 0(i = 1, 2, . . . , n), the graph of f(λ) has one horizontal asymptote y = 1 and n vertical asymptotes x = di(i = 1, 2, . . . , n), assuming di are distinct. Clearly, n X v2 f 0(λ) = b i . (5) (d − λ)2 i=1 i Shown below is a ﬁgure of f(λ) for the case (JL Note: To Be Added Later).

Theorem 2. Let G = D + bvvT be the matrix discussed in Theorem 1. Assume α is an eigenvalue of G but diﬀerent than all the diagonal entries of matrix D. Then w = (D−αI)−1v is an eigenvector associated with α. Proof. We just need to follow the deﬁnition to show that Gw = αw. Note that

Gw = (D + bvvT )w = (D − αI) + (αI + bvvT ) (D − αI)−1v = v + α(D − αI)−1v + bvvT (D − αI)−1v = αw + v + bvvT (D − αI)−1v.

We need to show v + bvvT (D − αI)−1v = 0. This is true, since 1 + bvT (D − αI)−1v = 0, which simply restates the fact that α satisﬁes the secular equation.

4 Suggestions for Further Study and Exercises

Problem 1 (Reduction to Hessenberg form). Use a 5 × 5 matrix as an example to illustrate how to reduce a matrix to an upper Hessenberg form by Householder reﬂections.

Problem 2 (Pseudo-code for Reduction to Hessenberg form). Write a pseudo-code for the Hessenberg reduction based on Householder reﬂections discussed in the previous problem. Recall that we shall not explicitly formulate the Householder matrices. Note also that it is optional to formulate the orthogonal transformation matrix.

Problem 3 (Operations count for Hessenberg reduction). Analyze computational complexity for the previously discussed procedure for reduction to Hessenberg form based on Householder reﬂections. Show that

9 10 • The operations count is n3 + O(n2); 3 14 • The operations count is n3 + O(n2) if the orthogonal matrix Q is desired. 3

Problem 4 . Practice Matlab hess command.

(i) Use rand to generate an order-5 random matrix A. Let B be the average of A with its transpose. Check hess(A) and hess(B), respectively.

(ii) Check other Matlab commands related to hess.

(iii) Explore the LAPACK subroutines used by hess. JL Note: It seems this is no longer explained in new versions of Matlab.

Problem 5 . Suppose A is an n × n upper Hessenberg matrix and we carry out QR- factorization based on Householder reﬂections. Show that the operations count for this QR-factorization (for upper Hessenberg matrix) is 3n2 + O(n).

Problem 6 . Let A = RQ, where R is an order-n right-triangular matrix and Q is an order-n upper Hessenberg matrix. Show that A is also an upper Hessenberg matrix.

Problem 7 . Implement the Francis QR-iteration method and test it on the following 3 × 3 matrix  0 0 1  A =  1 0 0  . 0 1 0 Report your ﬁndings on the numerical performance of the method. Would a shifting technique improve the method’s performance?

Problem 8 . Apply the symmetric tridiagonal algorithm to the Tn matrix obtained from discretization of the 1-dim Laplacian operator. Examine whether the Wilkinson’s shift makes a diﬀerence. D uT Problem. Let A = , where D = diag(d1, . . . , dn) has distinct diagonal entries u dn+1 and (row vector) u ∈ Rn has nonzero components. Show that (i) If α is an eigenvalue of A, then det(D − αI) 6= 0.

(ii) If α is an eigenvalue of A, then α is a root of the following equation

n X u2 f(λ) = λ − d + i = 0. n+1 d − λ i=1 i

10 Hints. Consider n = 3 as an example. Expanding det(A − λI) in its last column, we can rewrite the characteristic equation as

(d1 − λ)(d2 − λ)(d3 − λ)(d4 − λ) 2 2 2 −u1(d2 − λ)(d3 − λ) − u2(d3 − λ)(d1 − λ) − u3(d1 − λ)(d2 − λ) = 0.

Problem. Let A be a real symmetric matrix whose entries are the numerical values of the letters (a=1, b=2, ...) in the word square shown below (D. Knuth, The Art of Computer Pro- gramming, Vol.4 ). Reduce A to a symmetric tridiagonal matrix and then use the Wilkinson’s shift algorithm to ﬁnd its eigenvalues. Do you observe cubic convergence?

c l a s s l i g h t a g r e e s h e e p s t e p s