The Symmetric Eigenproblem and Singular Value Decomposition 5.1. Introduction We discuss perturbation theory (in section 5.2), algorithms (in sections 5.3 and 5.4), and applications (in section 5.5 and elsewhere) of the symmetric eigenvalue problem. We also discuss its close relative, the SVD. Since the eigendecomposition of the symmetric matrix H = [ A OT ] and the SVD of A are very simply related (see Theorem 3.3), most of the perturbation theorems and algorithms for the symmetric eigenproblem extend to the SVD. As discussed at the beginning of Chapter 4, one can roughly divide the algorithms for the symmetric eigenproblem (and SVD) into two groups: direct methods and iterative methods. This chapter considers only direct methods, which are intended to compute all (or a selected subset) of the eigenvalues and (optionally) eigenvectors, costing O(n3 ) operations for dense matrices. Iterative methods are discussed in Chapter 7. Since there has been a great deal of recent progress in algorithms and applications of symmetric eigenproblems, we will highlight three examples: • A high-speed algorithm for the symmetric eigenproblem based on divide- and-conquer is discussed in section 5.3.3. This is the fastest available al- gorithm for finding all eigenvalues and all eigenvectors of a large dense or banded symmetric matrix (or the SVD of a general matrix). It is signif- icantly faster than the previous "workhorse" algorithm, QR iteration. 17 • High-accuracy algorithms based on the dqds and Jacobi algorithms are discussed in sections 5.2.1, 5.4.2, and 5.4.3. These algorithms can find tiny eigenvalues (or singular values) more accurately than alternative 17 There is yet more recent work [201, 203] on an algorithm based on inverse iteration Downloaded 12/26/12 to 128.95.104.109. Redistribution subject SIAM license or copyright; see http://www.siam.org/journals/ojsa.php (Algorithm 4.2), which may provide a still faster and more accurate algorithm. But as of June 1997 the theory and software were still under development. 195 196 Applied Numerical Linear Algebra algorithms like divide-and-conquer, although sometimes more slowly, in the sense of Jacobi. • Section 5.5 discusses a "nonlinear" vibrating system, described by a dif- ferential equation called the Toda flow. Its continuous solution is closely related to the intermediate steps of the QR algorithm for the symmetrie eigenproblem. Following Chapter 4, we will continue to use a vibrating mass-spring system as a running example to illustrate features of the symmetrie eigenproblem. EXAMPLE 5.1. Symmetrie eigenvalue problems often arise in analyzing me- chanical vibrations. Example 4.1 presented one such example in detail; we will use notation from that example, so the reader is advised to review it now. To make the problem in Example 4.1 symmetrie, we need to assume that there is no damping, so the differential equations of motion of the mass-spring system become M(t) = —Kx(t), where M = diag(mi, ... , m) and kl + k2 — k2 — k2 k2 + k3 —k3 K = —kn-1 kn-1 + k,, —kn — kn kn Since M is nonsingular, we can rewrite this as x(t) = —M — 'Kx(t). If we seek solutions of the form x(t) = e 7tx(0), then we get eryt-y 2 x(0) = —M - 1 Keytx(0), or M -1Kx(0) = —y 2 x(0). In other words, —y 2 is an eigenvalue and x(0) is an eigenvector of M — 'K. Now M — 'K is not generally symmetrie, but we can make it symmetrie as follows. Define M 1 /2 = diag(mi/2 , ... , m/2 ), and multiply M — 'Kx(0) = — ry 2 x(0) by M l/2 on both sides to get M -1/2Kx ( 0) = M-1/2K(M-1/2M1/2) x ( 0) = — 'y 2M1/2 x ( 0) or Kx = _72 x, where x = M 1 /2 x(0) and k = M -1/2 KM-1/2 . It is easy to see that kl+k2 —k2 ml M1m2 —k2 k2+k3 —k3 mlm2 m2 ./m2m3 K= . —kom. -1 k,-1+kn —kn mn_2m n_1 m_1 m_lm-, —kn k f l Downloaded 12/26/12 to 128.95.104.109. Redistribution subject SIAM license or copyright; see http://www.siam.org/journals/ojsa.php mn_lmn Mn is symmetrie. Thus Bach eigenvalue —ry2 of k is real, and each eigenvector x = M'12 x(0) of K is orthogonal to the others. The Symmetrie Eigenproblem and SVD 197 In fact, k is a tridiagonal matrix, a special form to which any symmetrie matrix can be reduced, using Algorithm 4.6, specialized to symmetrie matrices as described in section 4.4.7. Most of the algorithms in section 5.3 for finding the eigenvalues and eigenvectors of a symmetrie matrix assume that the matrix has initially been reduced to tridiagonal form. There is another way to express the solution to this mechanical vibra- tion problem, using the SVD. Define KD = diag(kl, ... , k„) and KD 2 diag(k 1 /2 ,... , kn/2 ). Then K can be factored as K = BKDBT , where 1 -1 B = -1 1 as can be confirmed by a small calculation. Thus K = M -1/2 KM-112 = M-i/2BKDBTM-112 (M- 1 /2BK 1 /2 ) , (Kl/2 BTM-112) = (M-1/2BK1 2) . (M- 1 /2BK1 2)T GGT . (5.1) Therefore the singular values of G = M- 112 BKD 2 are the square roots of the eigenvalues of K, and the left singular vectors of G are the eigenvectors of K, as shown in Theorem 3.3. Note that G is nonzero only on the main diagonal and on the first superdiagonal. Such matrices are called bidiagonal, and most algorithms for the SVD begin by reducing the matrix to bidiagonal form, using the algorithm in section 4.4.7. Note that the factorization K = GGT implies that K is positive definite, since G is nonsingular. Therefore the eigenvalues - y2 of K are all positive. Thus ry is pure imaginary, and the solutions of the original differential equation x(t) = e^ftx(0) are oscillatory with frequency -y. For a Matlab solution of a vibrating mass-spring system, see HOMEPAGE/Matlab/massspring.m. For a Matlab animation of the vibra- tions of a similar physical system, type demo and then click on continue/ fun-extras/miscellaneous/bending. o 5.2. Perturbation Theory Suppose that A is symmetrie, with eigenvalues al >_ • • • >_ a,,, and corre- Downloaded 12/26/12 to 128.95.104.109. Redistribution subject SIAM license or copyright; see http://www.siam.org/journals/ojsa.php sponding unit eigenvectors ql, ... , qn . Suppose E is also symmetrie, and let A = A + E have perturbed eigenvalues ál > • • • â, and corresponding per- turbed eigenvectors ql, ... , qn . The major goal of this section is to bound the 198 Applied Numerical Linear Algebra differences between the eigenvalues ai and c and between the eigenvectors qi and qi in terms of the "size" of E. Most of our bounds will use 1E112 as the size of E, except for section 5.2.1, which discusses "relative" perturbation theory. We already derived our first perturbation bound for eigenvalues in Chap- ter 4, where we proved Corollary 4.1: Let A be symmetrie with eigenvalues al > • • • >_ a,. Let A + E be symmetrie with eigenvalues dl > ... >_ &. If ni is simple, then lcxi — di I c IIEII2 + O(IIEII2)• This result is weak because it assumes ni has multiplicity one, and it is useful only for sufficiently small 11E 112. The next theorem eliminates both weaknesses. THEOREM 5.1. Weyl. Let A and E be n-by-n symmetrie matrices. Let al > • • • > an be the eigenvalues of A and dl > ... &, be the eigenvalues of A = A+ E. Then lai — di j < 11E112. COROLLARY 5.1. Let G and F be arbitrary matrices (of the same size) where ul >_ • • • > un are the singular values of G and ul > • • • >_ a are the singular values of G + F. Then lui — al < h^Fl12. We can use Weyl's theorem to get error bounds for the eigenvalues com- puted by any backward stable algorithm, such as QR iteration: Such an algo- rithm computes eigenvalues di that are the exact eigenvalues of A = A + E where E 2 = O(e) IAhl2. Therefore, their errors can be bounded by ai—di ^hEil2 = O(e) hiAii2 = O(E) maxi laf l. This is a very satisfactory error bound, es- pecially for large eigenvalues (those cai near 1 Al 2 in magnitude), since they will be computed with most of their digits correct. Small eigenvalues (hniI « 11All2) may have fewer correct digits (but see section 5.2.1). We will prove Weyl's theorem using another useful classical result: the Courant—Fischer minimax theorem. To state this theorem we need to intro- duce the Rayleigh quotient, which will also play an important role in several algorithms, such as Algorithm 5.1. DEFINITION 5.1. The Rayleigh quotient of a symmetrie matrix A and nonzero vector u is p(u, A) - (uT Au)/(uT u). Here are some simple but important properties of p(u, A). First, p(lyu, A) _ p(u, A) for any nonzero scalar -y. Second, if Aqi = oigi, then p(qi, A) = ni. More generally, suppose QTAQ = A = diag(ni) is the eigendecomposition of A, with Q = [ql, ... , q,]. Expand u in the basis of eigenvectors qi as follows: u = Q(QT U) - Q = Ei qi^i. Then we can write AQe _ _T At = r_i Lei 2 eT QT p(u , A) _ S Z Downloaded 12/26/12 to 128.95.104.109.