Applied Numerical Linear Algebra. Lecture 14

Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Applied Numerical Linear Algebra. Lecture 14.

1/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) 5.3.3. Divide-and-Conquer

This method is the fastest now available if you want all eigenvalues and eigenvectors of a tridiagonal matrix whose dimension is larger than about 25. (The exact threshold depends on the computer.) It is quite subtle to implement in a numerically stable way. Indeed, although this method was ﬁrst introduced in 1981 [J. J. M. Cuppen. A divide and conquer method for the symmetric tridiagonal eigenproblem. Numer. Math., 36:177-195, 1981], the ”right” implementation was not discovered until 1992 [M. Gu and S. Eisenstat. A stable algorithm for the rank-1 modiﬁcation of the symmetric eigenproblem. Computer Science Dept. Report YALEU/DCS/RR-916, Yale University, September 1992;M. Gu and S. C. Eisenstat. A divide-and-conquer algorithm for the symmetric tridiagonal eigenproblem. SIAM J. Matrix Anal Appl, 16:172-191, 1995]).

2/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

a1 b1 0 ...... 0 b1 a2 b2 0 ... 0  ......   ... am 1 bm 1 ...... 0   − −   0 bm 1 am bm 0 0  T =  −   0 0 b a b 0   m m+1 m+1   0 0 0 b ......   m+1   ......     0 0 0 ...... bn 1   −   00 0 0 bn 1 an   −   

3/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

a1 b1 0 ...... 0 b1 a2 b2 0 ... 0  ......   ... am 1 bm 1 ...... 0  ......  − −   0 bm 1 am bm 0 0 0  ...bm bm... T =  − − +   0 0 0 a b b 0  ...b b ...  m+1 − m m+1  m m  0 0 0 b ......   ......   m+1     ......       0 0 0 ...... bn 1   −   00 0 0 bn 1 an   −   0  ...  0  T1 0...0  1  = + bm   0...0 1 1 0....0 0...0 T2 ·  1  ·    0     ....     0  4/54     Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Assume that we have eigendecomposition of T1, T2 such that T T T1 = Q1Λ1Q1 and T2 = Q2Λ2Q2 . Then we can write that

T T1 0...0 T Q1Λ1Q1 0...0 T T = + bmv v = T + bmv v 0...0 T2 · 0...0 Q2Λ2Q · 2 T Q1 0...0 Λ1 0...0 T Q1 0...0 = + bmu u T 0...0 Q2 · 0...0 Λ2 · · 0...0 Q2 Let deﬁne the diagonal matrix

Λ 0...0 D = 1 0...0 Λ 2 We observe that the eigenvalues of T are the same as of D + b u uT = D + ρu uT . m · ·

5/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

1. We want to ﬁnd eigenvalues of D + b u uT = D + ρu uT m · · 2. We assume that diagonal elements of D are sorted such that d ... d and D + λI is nonsingular 1 ≥ ≥ n 3. To ﬁnd eigenvalues we compute the characteristic polynomial D + ρu uT λI noting that · T − 1 T D + ρu u λI =(D λI )(I + ρ(D λI )− u u ) · − − − · By assumption (2) we have det(D λI ) = 0 and thus − 6 1 T det(I + ρ(D λI )− u u )=0. − ·

6/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

LEMMA 5.1. If x and y are vectors, det(I + xy T )=1+ y T x.

Thus, using this lemma we can get that 2 1 T T 1 n ui det(I + ρ(D λI )− u u )=1+ u ρ(D λI )− u =1+ ρ = i=1 di λ f (λ). − · − − We see that eigenvalues of T are roots of the secular equationPf (λ)=0. The secular equation can be solved using Newton’s method with starting point (di , di+1). T 1 LEMMA 5.2. If α is an eigenvalue of D + ρuu , then (D αI )− u is its eigenvector. Since D αI is diagonal, this costs O(n) ﬂops− to compute. −

7/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

ALGORITHM 5.2. Finding eigenvalues and eigenvectors of a symmetric tridiagonal matrix using divide-and-conquer: proc dc eig (T , Q, Λ) ..... from input T compute outputs Q and Λ where T = QΛQT if T is 1-by-1 return Q = 1, Λ= T else T 0 form T = 1 + b υυT 0 T m 2 call dc eig (T1, Q1, Λ1) call dc eig (T2, Q2, Λ2) T form D + ρuu from Λ1, Λ2, Q1, Q2 ﬁnd eigenvalues Λ and eigenvectors Q′ of D + ρuuT Q 0 form Q = 1 Q′ = eigenvectors of T 0 Q2 · return Q and Λ

end if 8/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Computing the Eigenvectors Stably

Lemma 5.2 provides formula for the computing of eigenvectors. This formula is not stable in the case when two eigenvalues are close to each other. Let αi ,αi+1 are close to each other. Then (D α )−1u and (D α )−1u are inaccurate and far − i − i+1 from orthogonal. An alternative formula was found which is based on the L¨owner’s theorem.

9/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

THEOREM 5.10. (L¨owner). Let D = diag(d1,..., dn) be diagonal with dn <...< d1. Let αn <...<α1 be given, satisfying the interlacing property

d <α < < d <α < d <α < < d <α . n n ··· i+1 i+1 i i ··· 1 1

Then there is a vectoru ˆ such that the αi are the exact eigenvalues of Dˆ D +u ˆuˆT . The entries ofu ˆ are given by ≡ n 1/2 j=1(αj di ) uˆi = n − . | | " j=1,j=i (dj di )# Q 6 − Q

10/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Proof. The characteristic polynomial of Dˆ can be written both as ˆ n ˆ T det(D λI )= j=1(αj λ) and as det(D λI ) = det(D +u ûˆ λI )= − 1 T − − 1 T − det(D(I + D− uûˆ ) λI ) = det((D λI )(I +(D λI )− uûˆ )) or Q − − − n n 2 uˆj det(Dˆ λI ) = (dj λ) 1+ −  −  ·  dj λ Yj=1 Xj=1 −    

n n 2  uˆj  = (dj λ) 1+  −  ·  dj λ j=1  j = 1 −  Y  X     j = i   6     n  2 + (dj λ) uˆi .  −  ·  j = 1   Y   j = i   6    11/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Setting λ = d in the above eq. and in det(Dˆ λI )= n (α λ), i − j=1 j − equating both expressions for det(Dˆ λI ) yield − Q n n (α d ) =u ˆ2 (d d ) j − i i · j − i Yj=1 j Y= 1 j = i 6

or n 2 j=1(αj di ) uˆi = n − . j=1,j=i (dj di ) Q 6 − Using the interlacing property,Q we can show that the fraction on the right is positive, so we can take its square root to get the desired expression for uˆi .

12/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Here is the stable algorithm for computing the eigenvalues and eigenvectors (where we assume for simplicity of presentation that ρ = 1). ALGORITHM 5.3. Compute the eigenvalues and eigenvectors of D + uuT . u2 Solve the secular equation 1+ n i = 0 to get the eigenvalues i=1 di λ α of D + uuT − i P Use Löwner’s theorem to compute uˆ so that the αi are ”exact” eigenvalues of D +u ûˆT . Use Lemma 5.2 to compute the eigenvectors of Dˆ = D +u ûˆT .

13/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) 5.3.4. Bisection and Inverse Iteration

The Bisection algorithm exploits Sylvester’s inertia theorem (Theorem 5.3) to ﬁnd only those k eigenvalues that one wants, at cost O(nk). Recall that Inertia(A)=(ν, ζ, π), where ν, ζ and π are the number of negative, zero, and positive eigenvalues of A, respectively. Suppose that X is nonsingular; Sylvester’s inertia theorem asserts that Inertia(A)= Inertia(X T AX ). Now suppose that one uses Gaussian elimination to factorize A zI = LDLT , where L is nonsingular and D diagonal. Then Inertia− (A zI )= Inertia(D). Since D is diagonal, its inertia is trivial to − compute. (In what follows, we use notation such as ”#dii < 0” to mean ”the number of values of dii that are less than zero.”)

Inertia(A zI ) = (#dii < 0, #dii = 0, #dii > 0) − = (# negative eigenvalues of A zI , # zero eigenvalues of A zI ,− # positive eigenvalues of−A zI ) = (# eigenvalues of A < z, − # eigenvalues of A = z, 14/54 # eigenvalues of A > z). Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Suppose z < z and we compute Inertia(A z I ) and Inertia(A z I ). 1 2 − 1 − 2 Then the number of eigenvalues in the interval [z1, z2) equals (# eigenvalues of A < z2) – (# eigenvalues of A < z1). To make this observation into an algorithm, deﬁne

Negcount(A, z) = # eigenvalues of A < z.

15/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

ALGORITHM 5.4. Bisection: Find all eigenvalues of A inside [a, b) to a given error tolerance tol: na = Negcount(A, a) nb = Negcount(A, b) if na = nb , quit ... because there are no eigenvalues in [a, b) put [a, na, b, nb ] onto Worklist /∗ Worklist contains a list of intervals [a, b) containing − − eigenvalues n na through n nb +1, which the algorithm will repeatedly bisect until they are narrower than tol. ∗/ while Worklist is not empty do remove [low, nlow , up, nup ] from Worklist if (up − low < tol) then print ”there are nup nlow eigenvalues in [low, up)” else mid = (low + up)/2 nmid = Negcount(A, mid) if nmid > nlow then ... there are eigenvalues in [low, mid) put [low, nlow , mid, nmid ] onto Worklist end if if nup > nmid then ... there are eigenvalues in [mid, up) put [mid, nmid , up, nup ] onto Worklist end if end if end while

16/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) From Negcount(A,z) it is easy to compute Gaussian elimination provided that

a1 z b1 ...... b− a z ......  1 2 −  A zI = ...... = LDLT −  ... bn 2 an 1 z bn 1   − − − −   ...... bn 1 an z   − −    1 ...... d1 ...... 1 l1...... l 1 ...... d ... .. 1 ... =  1   2    ...... · ...... · ...... ln 1 −  ... ln 1 1   ... .. dn   ...... 1   −   2    anda1 z = d1; d1l1 =b1; li 1di 1 + di = ai z, di li = bi .. Substitute − 2 − − − li = bi /di into li 1di 1 + di = ai z to get: − − − 2 bi 1 di =(ai z) − , − − di 1 − This is the stable procedure since A zI is tridiagonal matrix. − 17/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

LEMMA 5.3. The di computed in ﬂoating point arithmetic, using

2 bi 1 di =(ai z) − , − − di 1 −

have the same signs (and so compute the same Inertia) as the dˆi computed exactly from Aˆ, where Aˆ is very close to A:

(Aˆ) aˆ = a and (Aˆ) bˆ = b (1 + ǫ ), where ǫ 2.5ε + O(ε2). ii ≡ i i i,i+1 ≡ i i i | i |≤

18/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) 5.3.5. Jacobi’s Method

Given a symmetric matrix A = A0, Jacobi’s method produces a sequence A1, A2, ... of orthogonally similar matrices, which eventually converge to a diagonal matrix with the eigenvalues on the diagonal. Ai+1 is obtained T from Ai by the formula Ai+1 = Ji Ai Ji , where Ji is an orthogonal matrix called a Jacobi rotation. Thus

T Am = Jm 1Am 1Jm 1 T− T − − = Jm 1Jm 2Am 2Jm 2Jm 1 = T− − T − − − ··· = Jm 1 J0 A0J0 Jm 1 = JT−AJ···. ··· −

19/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

If we choose each Ji appropriately, Am approaches a diagonal matrix Λ for large m. Thus we can write Λ JT AJ or JΛJT A. Therefore, the columns of J are approximate eigenvectors.≈ ≈ T We will make J AJ nearly diagonal by iteratively choosing Ji to make T one pair of oﬀdiagonal entries of Ai+1 = Ji Ai Ji zero at a time. We will do this by choosing Ji to be a Givens rotation, j k 1 1  .  ..   j  cos θ sin θ   −   .  Ji = R(j, k,θ)  ..  , ≡    sin θ cos θ  k    .   ..     1     1      where θ is chosen to zero out the j, k and k, j entries of Ai+1. To 20/54 determine θ (or actually cos θ and sin θ), Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

write

(i+1) (i+1) T (i) (i) ajj ajk cos θ sin θ ajj ajk cos θ sin θ = − −  (i+1) (i+1)  sin θ cos θ  (i) (i)  sin θ cos θ akj akk akj akk     λ 0 = 1 , 0 λ 2

where λ1 and λ2 are the eigenvalues of

21/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

(i) (i) ajj ajk .  (i) (i)  akj akk It is easy to compute cos θ and sin θ: Multiplying out the last expression, using symmetry, abbreviating c cos θ and s sin θ, and dropping the superscript (i) for simplicity yield≡ ≡

λ 0 a c2 + a s2 + 2sca sc(a a )+ a (c2 s2) 1 = jj kk jk kk jj jk . 0 λ sc(a a )+ a (c2 s2) a s2−+ a c2 2sca− 2 kk − jj jk − jj kk − jk

22/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Setting the oﬀdiagonals to 0 and solving for θ we get 0= sc(a a )+ a (c2 s2), or kk − jj jk − a a c2 s2 cos 2θ jj − kk = − = = cot 2θ τ. 2ajk 2sc sin 2θ ≡

We now let t = s = tan θ and note that t2 + 2τt 1 = 0 to get (via the c − quadratic formula) t = sign(τ) , c = 1 and s = t c. We τ +√1+τ 2 √1+τ 2 · summarize this derivation| | in the following algorithm.

23/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

ALGORITHM 5.5. Compute and apply a Jacobi rotation to A in coordinates j, k: proc Jacobi-Rotation (A, j, k) if ajk is not too small | |τ =(a a )/(2 a ) jj − kk · jk t = sign(τ)/( τ + √1+ τ 2) c = 1/√1+ τ|2 | s = t c A = R·T (j, k,θ) A R(j, k,θ) ... where c = cos θ and s = sin θ if eigenvectors are· desired· J = J R(j, k,θ) end if · end if

24/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Jacobi’s method to ﬁnd the eigenvalues of a symmetric matrix

The cost of applying R(j, k,θ) to A (or J) is only O(n) ﬂops, because only rows and columns j and k of A (and columns j and k of J) are modiﬁed. The overall Jacobi algorithm is then as follows.

ALGORITHM 5.6. Jacobi’s method to ﬁnd the eigenvalues of a symmetric matrix: repeat choose a j, k pair call Jacobi-Rotation(A, j, k) until A is suﬃciently diagonal

25/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

We still need to decide how to pick j, k pairs. There are several possibilities. To measure progress to convergence and describe these possibilities, we deﬁne

oﬀ(A) a2 . ≡ jk 1 j

26/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

LEMMA 5.4. Let A′ be the matrix after calling Jacobi-Rotation(A, j, k) 2 2 2 for any j = k. Then oﬀ (A′)= oﬀ (A) a . 6 − jk

Proof. Note that A′ = A except in rows and columns j and k. Write

2  2  2 2 2 off (A)= aj′k′ + ajk S + ajk   ≡  1 j ′ X< k′ n   ≤ ≤   j ′ = jor k′ = k   6 6    2 2 2 2 and similarly off (A′ )= S ′ + a′jk = S ′2, since ajk′ = 0 after calling Jacobi-Rotation(A, j, k). Since X F = QX F and X F = XQ F for || || || 2|| 2 || || || || any X and any orthogonal Q, we can show S = S ′ . Thus 2 2 2 off (A′)= off (A) ajk as desired. The next algorithm− was the original version of the algorithm (from Jacobi in 1846), and it has an attractive analysis although it is too slow to use.

27/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

ALGORITHM 5.7. Classical Jacobi’s algorithm:

while oﬀ(A) > tol (where tol is the stopping criterion set by user) choose j and k so ajk is the largest oﬀdiagonal entry in magnitude call Jacobi-Rotation(A, j, k) end while

28/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

THEOREM 5.11. After one Jacobi rotation in the classical Jacobi’s 1 n(n 1) algorithm, we have off(A′) 1 off(A), where N = − = the ≤ − N 2 number of superdiagonal entriesq of A. After k Jacobi-Rotations off( ) is no more than (1 1 )k/2 off(A). · − N

2 2 2 Proof. By Lemma 5.4, after one step, off (A′)= off (A) a , where a − jk jk 2 n(n 1) 2 largest offdiagonal entry. Thus off (A) − a , or ≤ 2 jk a2 1 off2(A), so off2(A) a2 (1 1 ) off2(A) as desired. jk ≥ n(n 1)/2 − jk ≤ − N So the classical− Jacobi’s algorithm converges at least linearly with the error (measured by off(A)) decreasing by a factor of at least 1 1 at − N a time. In fact, it eventually converges quadratically. q

29/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

THEOREM 5.12. Jacobi’s method is locally quadratically convergent after N steps (i.e., enough steps to choose each ajk once). This means that for i large enough

2 oﬀ(Ai+N )= O(oﬀ (Ai )).

In practice, we do not use the classical Jacobi’s algorithm because n2 n searching for the largest entry is too slow: We would need to search 2− entries for every Jacobi rotation, which costs only O(n) ﬂops to perform, and so for large n the search time would dominate. Instead, we use the following simple method to choose j and k.

30/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) ALGORITHM 5.8. Cyclic-by-row-Jacobi: Sweep through the oﬀ diagonals of A rowwise. repeat for j = 1 to n 1 for k = j + 1 −to n call Jacobi-Rotation (A, j, k) end for end for until A is suﬃciently diagonal

A no longer changes when Jacobi-Rotation(A, j, k) chooses only c = 1 and s = 0 for an entire pass through the inner loop. The cyclic Jacobi’s algorithm is also asymptotically quadratically convergent like the classical Jacobi’s algorithm [J. H. Wilkinson. The Algebraic Eigenvalue Problem. Oxford University Press, Oxford, UK, 1965; p. 270]. The cost of one Jacobi ”sweep” (where each j, k pair is selected once) is approximately half the cost of reduction to tridiagonal form and the computation of eigenvalues and eigenvectors using QR iteration, and more than the cost using divide-and-conquer. Since Jacobi’s method often takes 5-10 sweeps to converge, it is much slower than the 31/54 competition. Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Algorithms for the Singular Value Decomposition (SVD)

All the algorithms for the eigendecomposition of a symmetric matrix A, except Jacobi’s method, have the following structure:

1 Reduce A to tridiagonal form T with an orthogonal matrix Q1: T A = Q1TQ1 .

2 T Find the eigendecomposition of T : T = Q2ΛQ2 , where Λ is the diagonal matrix of eigenvalues and Q2 is the orthogonal matrix whose columns are eigenvectors. T 3 Combine these decompositions to get A =(Q1Q2)Λ(Q1Q2) . The columns of Q = Q1Q2 are the eigenvectors of A.

32/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

All the algorithms for the SVD of a general matrix G, except Jacobi’s method, have an analogous structure:

1 Reduce G to bidiagonal form B with orthogonal matrices U1 and T V1: G = U1BV1 . This means B is nonzero only on the main diagonal and ﬁrst superdiagonal.

2 T Find the SVD of B: B = U2ΣV2 , where Σ is the diagonal matrix of singular values, and U2 and V2 are orthogonal matrices whose columns are the left and right singular vectors, respectively. T 3 Combine these decompositions to get G =(U1U2)Σ(V1V2) . The columns of U = U1U2 and V = V1V2 are the left and right singular vectors of G, respectively.

33/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

LEMMA 5.5. Let B be an n-by-n bidiagonal matrix, with diagonal a1,..., an and superdiagonal b1,..., bn 1. There are three ways to convert the problem of ﬁnding the SVD of B− to ﬁnding the eigenvalues and eigenvectors of a symmetric tridiagonal matrix.

34/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

1. 0 BT Let A = . Let P be the permutation matrix B 0 P =[e1, en+1, e2, en+2,..., en, e2n], where ei is the ith column of the T 2n-by-2n identity matrix. Then Tps P AP is symmetric tridiagonal. The subscript ”ps” stands≡ for perfect shuﬄe, because multiplying P times a vector x ”shuﬄes” the entries of x like a deck of cards. One can show that Tps has all zeros on its main diagonal, and its superdiagonal and subdiagonal is a1, b1, a2, b2,..., bn 1, an. − If Tps xi = αi xi is an eigenpair for Tps , with xi a unit vector, then 1 υi αi = σi , where σi is a singular value of B, and Pxi = √ , ± 2 ui ± where ui and υi are left and right singular vectors of B, respectively. This is by using Theorem 3.3, part 4.

35/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

T 2. Let TBBT BB . Then TBBT is symmetric tridiagonal with 2 ≡ 2 2 2 2 2 2 diagonal a1 + b1, a2 + b2,..., an 1 + bn 1, an, and superdiagonal − − and subdiagonal a2b1, a3b2,..., anbn 1. The singular values of B − are the square roots of the eigenvalues of TBBT , and the left singular vectors of B are the eigenvectors of TBBT . T 3. Let TBT B B B. Then TBT B is symmetric tridiagonal with 2 ≡2 2 2 2 2 2 diagonal a1, a2 + b1, a3 + b2,..., an + bn 1 and superdiagonal and − subdiagonal a1b1, a2b2,..., an 1bn 1. The singular values of B are − − the square roots of the eigenvalues of TBT B , and the right singular vectors of B are the eigenvectors of TBT B . TBT B contains no information about the left singular vectors of B.

Proof of these statements follows from Part 2 of Theorem 3.3.

36/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Practical algorithms for computing the SVD

1 QR iteration and its variations. This is the fastest algorithm for small matrices up to size n = 25 to ﬁnd all the singular values of a bidiagonal matrix.

2 Divide-and-conquer. This is the fastest method to ﬁnd all singular values and singular vectors for matrices larger than n = 25. However, it does not guarantee computation of tiny singular values.

3 Bisection and inverse iteration. T Lemma 5.5, part 1 is applied to Tps = P AP to ﬁnd only the singular values in a desired interval. Singular values are found by high accuracy but singular vectors can have loss of orthogonality.

4 Jacobi’s method. SVD of a dense matrix G is computd implicitly applying of Jacobi’s method (Algorithm 5.5) to GG T or G T G.

37/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) QR Iteration and Its Variations for the Bidiagonal SVD

There is a long history of variations on QR iteration for the SVD, designed to be as eﬃcient and accurate as possible; see [B. Parlett. Acta Numerica, The new qd algorithms, pages 459-491. Cambridge University Press, Cambridge, UK, 1995] for a good survey. The algorithm in the LAPACK routine sbdsqr" was originally based on [J. Demmel and W. Kahan. Accurate singular values of bidiagonal matrices. SIAM J. Sci. Statist. Comput, 11:873-912, 1990] and later updated to use the algorithm in [K. Fernando and B. Parlett. Accurate singular values and differential qd algorithms. Numer. Math., 67:191-229, 1994] in the case when singular values only are desired. This latter algorithm, called dqds" for historical reasons (dqds" is short for "differential quotient-difference algorithm with shifts" [H. Rutishauser. Lectures on Numerical Mathematics. Birkhauser, Basel, 1990]), is elegant, fast, and accurate, so we will present it. To derive dqds", we begin with an algorithm that predates QR iteration, called LR iteration, specialized to 38/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) LR iteration

ALGORITHM 5.9. LR iteration: Let T0 be any symmetric positive deﬁnite matrix. The following algorithm produces a sequence of similar symmetric positive deﬁnite matrices Ti : i = 0 repeat 2 Choose a shift τi smaller than the smallest eigenvalue of Ti . Compute the Cholesky factorization T τ 2I = BT B i − i i i (Bi is an upper triangular matrix with positive diagonal.) T 2 Ti+1 = Bi Bi + τi I i = i + 1 until convergence

39/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) LR iteration versus QR iteration

LR iteration is very similar in structure to QR iteration: We compute a factorization, and multiply the factors in reverse order to get the next iterate Ti+1. It is easy to see that Ti+1 and Ti are similar:

T 2 T T T 2 T T Ti+1 = Bi B + τ I = B− B Bi B + τ B− B i i i i i i i i (1) T T 2 T T T = Bi− (Bi Bi + τi )Bi = Bi− Ti Bi .

2 In fact, when the shift τi = 0, we can show that two steps of LR iteration produce the same T2 as one step of QR iteration.

40/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

LEMMA 5.6. Let T2 be the matrix produced by two steps of Algorithm 2 5.9 using τi = 0, and let T ′ be the matrix produced by one step of QR iteration (QR = T0, T ′ = RQ). Then T2 = T ′.

2 Proof. Since T0 is symmetric, we can factorize T0 in two ways: First, 2 T T T T0 = T0 T0 =(QR) QR = R R. We assume without loss of generality 2 that Rii > 0. This is a factorization of T0 into a lower triangular matrix RT times its transpose; since the Cholesky factorization is unique, this must in fact be the Cholesky factorization. The second factorization is 2 T T T0 = B0 B0B0 B0.

41/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

T T Now by Algorithm 5.9, T1 = B0B0 = B1 B1, so we can rewrite 2 T T T T T T0 = B0 B0B0 B0 = B0 (B1 B1)B0 =(B1B0) B1B0. This is also a 2 T factorization of T0 into a lower triangular matrix (B1B0) times its transpose, so this must again be the Cholesky factorization. By uniqueness of the Cholesky factorization, we conclude R = B1B0, thus relating two steps of LR iteration to one step of QR iteration. We exploit this relationship as follows: T0 = QR implies

1 1 1 T ′ = RQ = RQ(RR− )= R(QR)R− = RT0R− because T0 = QR T 1 T = (B1B0)(B0 B0)(B1B0)− because R = B1B0 and T0 = B0 B0 T 1 1 T 1 = B1B0B0 B0B0− B1− = B1(B0B0 )B1− T 1 T T = B1(B1 B1)B1− because B0B0 = T1 = B1 B1 T = B1B1 = T2 as desired.

42/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Remarks

Neither Algorithm 5.9 nor Lemma 5.6 depends on T0 being tridiagonal, just symmetric positive deﬁnite. Using the relationship between LR iteration and QR iteration in Lemma 5.6, one can show that much of the convergence analysis of QR iteration goes over to LR iteration. The ultimate algorithm, dqds", is mathematically equivalent to LR iteration. In dqds" alg. is formed Bi+1 directly from Bi , without ever forming the intermediate matrix Ti+1.

43/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Jacobi’s Method for the SVD

ALGORITHM 5.12. Compute and apply a one-sided Jacobi rotation to G in coordinates j, k: proc One-Sided-Jacobi-Rotation (G, j, k) T T T Compute ajj =(G G)jj , ajk =(G G)jk , and akk =(G G)kk if ajk is not too small |τ =(| a a )/(2 a ) jj − kk · jk t = sign(τ)/( τ + √1+ τ 2) c = 1/√1+ t|2 | s = c t G = G· R(j, k,θ) ... where c = cos θ and s = sin θ if right· singular vectors are desired J = J R(j, k,θ) end if · end if Note that the jj, jk, and kk entries of A = G T G are computed by procedure One-Sided-Jacobi-Rotation, after which it computes the Jacobi rotation R(j, k,θ) in the same way as procedure Jacobi-Rotation (Algorithm 5.5). 44/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

ALGORITHM 5.13. One-sided Jacobi: Assume that G is n-by-n. The outputs are the singular values σi , the left singular vector matrix U, and the right singular vector matrix V so that G = UΣV T , where Σ= diag(σi ). repeat for j = 1 to n 1 for k = j +− 1 to n call One-Sided-Jacobi-Rotation (G, j, k) end for end for until G T G is diagonal enough Let σ = G(:, i) (the 2-norm of column i of G) i || ||2 Let U =[u1,..., un], where ui = G(:, i)/σi let V = J, the accumulated product of Jacobi rotations

45/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

The following theorem shows that one-sided Jacobi can compute the SVD to high relative accuracy, despite roundoﬀ, provided that we can write G = DX , where D is diagonal and X is well-conditioned. THEOREM 5.15. Let G = DX be an n-by-n matrix, where D is diagonal and nonsingular, and X is nonsingular. Let Gˆ be the matrix after calling One-Sided-Jacobi-Rotation (G, j, k) m times in ﬂoating point arithmetic. Let σ1 ... σn be the singular values of G, and letσ ˆ1 ... σˆn be the singular≥ values≥ of Gˆ. Then ≥ ≥

σ σˆ | i − i | O(mε)κ(X ), σi ≤

1 where κ(X )= X X − is the condition number of X . In other words, the relative|| ||·|| error in|| the singular values is small if the condition number of X is small.

46/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Proof. We first consider m = 1; i.e., we apply only a single Jacobi rotation and later generalize to larger m. Examining One-Sided-Jacobi-Rotation (G, j, k), we see that Gˆ = fl(G R˜), where R˜ is a floating point Givens rotation. By construction, R˜ differs· from some exact Givens rotation R by O(ε) in norm. (It is not important or necessarily true that R˜ differs by O(ε) from the ”true” Jacobi rotation, the one that One-Sided-Jacobi-Rotation (G, j, k) would have computed in exact arithmetic. It is necessary only that it differs from some rotation by O(ε). This requires only that c2 + s2 =1+ O(ε), which is easy to verify.) Our goal is to show that Gˆ = GR(I + E) for some E that is small in norm: E2 = O(ε)κ(X ). If E were zero, then Gˆ and GR would have the same singular|| values, since R is exactly orthogonal. When E is less than one in norm, we can use Corollary 5.2 to bound the relative difference in singular values by

σi σˆi T T T | − | (I +E)(I +E) I 2 = E+E +EE 2 3 E 2 = O(ε)κ(X ), σi ≤ || − || || || ≤ || || as desired. 47/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

Now we construct E. Since R˜ multiplies G on the right, each row of Gˆ depends only on the corresponding row of G; write this in Matlab notation as Gˆ(i, :) = ﬂ(G(i, :) R˜). Let F = Gˆ GR. Then by Lemma 3.1 and the fact that G = DX ,· −

F (i, :) = Gˆ(i, :) G(i, :)R = O(ε) G(i, :)R = O(ε) d X (i, :) || ||2 || − ||2 || ||2 || ii ||2 1 1 and so dii− F (i, :) 2 = O(ε) X (i, :) 2, or D− F 2 = O(ε) X (i, :) 2. || ||1 T || 1 || || 1 || 1 1 || || Therefore, since R− = R and G − =(DX )− = X − D− ,

T 1 T 1 1 Gˆ = GR + F = GR(I + R G − F )= GR(I + R X − D− F ) GR(I + E) ≡ where

T 1 1 1 E R X − D− F = O(ε) X X − = O(ε)κ(X ) || ||2 ≤ || ||2|| ||2|| ||2 || ||2|| ||2 as desired.

48/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

To extend this result to m > 1 rotations, note that in exact arithmetic we would have Gˆ = GR = DXR = DXˆ, with κ(Xˆ)= κ(X ), so that the bound

σi σˆi T T T | − | (I +E)(I +E) I 2 = E+E +EE 2 3 E 2 = O(ε)κ(X ), σi ≤ || − || || || ≤ || || would apply at each of the m steps, yielding bound

σ σˆ | i − i | O(mε)κ(X ). σi ≤

Because of roundoﬀ, κ(Xˆ) could grow by as much as κ(I + E) (1 + O(ε)κ(X )) at each step, a factor very close to 1, which we absorb≤ into the O(mε) term.

49/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Example

We will construct a lower triangular matrix using Given’s rotation from the matrix 5 4 3 A = 4 6 1 .   3 1 7  

50/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD) Given’s matrix for j < k

function [G] = GivensMatrixLow(A, j,k)

a = A(k, k) b = A(j, k) r = sqrt(a2 + b2); c = a/r; s = b/r; − G = eye(length(A)); G(j, j)= c; G(k, k)= c; G(j, k)= s; G(k, j)= s; −

51/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

>>G1up = GivensMatrixLow(A,2,3)

1.000000000000000 0 0 G1= 0 0.989949493661166 0.141421356237310  0 0.141421356237310− 0.989949493661166    >> A1 =G1*A 5.000000000000000 4.000000000000000 3.000000000000000 A1= 3.535533905932737 5.798275605729690 0.000000000000000 3.535533905932738 1.838477631085023− 7.071067811865475   

52/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

>>G2 = GivensMatrixLow(A1,1,3)

0.920574617898323 0 0.390566732942472 G2= 0 1.000000000000000− 0 0.390566732942472 0 0.920574617898323    >> A2=G2*A1 3.222011162644131 2.964250269632601 0.000000000000000 A2= 3.535533905932737 5.798275605729690 −0.000000000000000 5.207556439232954 3.254722774520597− 7.681145747868607   

53/54 Algorithms for the Symmetric Eigenproblem: Divide-and-Conquer Algorithms for the Symmetric Eigenproblem: Bisection and Inverse Iteration Algorithms for the Symmetric Eigenproblem: Jacobi’s Method Algorithms for the Singular Value Decomposition(SVD)

>>G3 = GivensMatrixLow(A2,1,2)

0.890391914715406 0.455194725594918 0 G3= 0.455194725594918− 0.890391914715406 0  0 0 1.000000000000000   >> A3=G3*A2 1.259496302198541 0 0.000000000000000 A3= 4.614653291088246 6.512048806713364 −0.000000000000000 5.207556439232954 3.254722774520597− 7.681145747868607   

54/54