Applied Numerical Linear Algebra. Lecture 14
1/72 5.3.5.Algorithms for the Symmetric Eigenproblem: Jacobi’s Method
Given a symmetric matrix A = A0, Jacobi’s method produces a sequence A1, A2, ... of orthogonally similar matrices, which eventually converge to a diagonal matrix with the eigenvalues on the diagonal. Ai+1 is obtained T from Ai by the formula Ai+1 = Ji Ai Ji , where Ji is an orthogonal matrix called a Jacobi rotation. Thus
T Am = Jm 1Am 1Jm 1 T− T − − = Jm 1Jm 2Am 2Jm 2Jm 1 = T− − T − − − ··· = Jm 1 J0 A0J0 Jm 1 = JT−AJ···. ··· −
2/72 If we choose each Ji appropriately, Am approaches a diagonal matrix Λ for large m. Thus we can write Λ JT AJ or JΛJT A. Therefore, the columns of J are approximate eigenvectors.≈ ≈ T We will make J AJ nearly diagonal by iteratively choosing Ji to make T one pair of offdiagonal entries of Ai+1 = Ji Ai Ji zero at a time. We will do this by choosing Ji to be a Givens rotation,
j k 1 1 . .. j cos θ sin θ − . Ji = R(j, k,θ) .. , ≡ sin θ cos θ k . .. 1 1 where θ is chosen to zero out the j, k and k, j entries of Ai+1. To determine θ (or actually cos θ and sin θ), 3/72 write
(i+1) (i+1) T (i) (i) ajj ajk cos θ sin θ ajj ajk cos θ sin θ = − − (i+1) (i+1) sin θ cos θ (i) (i) sin θ cos θ akj akk akj akk λ 0 = 1 , 0 λ 2
where λ1 and λ2 are the eigenvalues of
4/72 (i) (i) ajj ajk . (i) (i) akj akk It is easy to compute cos θ and sin θ: Multiplying out the last expression, using symmetry, abbreviating c cos θ and s sin θ, and dropping the superscript (i) for simplicity yield≡ ≡
2 2 2 2 λ1 0 ajj c + akk s + 2scajk sc(akk ajj )+ ajk (c s ) = 2 2 2− 2 − . 0 λ sc(akk ajj )+ ajk (c s ) ajj s + akk c 2scajk 2 − − −
5/72 Setting the offdiagonals to 0 and solving for θ we get 2 2 0= sc(akk ajj )+ ajk (c s ), or − − 2 2 ajj akk c s cos 2θ − = − = = cot 2θ τ. 2ajk 2sc sin 2θ ≡
We now let t = s = tan θ and note that t2 + 2τt 1 = 0 to get (via the c − quadratic formula) t = sign(τ) , c = 1 and s = t c. We τ +√1+τ 2 √1+τ 2 · summarize this derivation| | in the following algorithm.
6/72 ALGORITHM 5.5. Compute and apply a Jacobi rotation to A in coordinates j, k: proc Jacobi-Rotation (A, j, k) if ajk is not too small | | τ =(ajj akk )/(2 ajk ) − · t = sign(τ)/( τ + √1+ τ 2) c = 1/√1+ τ|2 | s = t c A = R·T (j, k,θ) A R(j, k,θ) ... where c = cos θ and s = sin θ if eigenvectors are· desired· J = J R(j, k,θ) end if · end if
7/72 Jacobi’s method to find the eigenvalues of a symmetric matrix
The cost of applying R(j, k,θ) to A (or J) is only O(n) flops, because only rows and columns j and k of A (and columns j and k of J) are modified. The overall Jacobi algorithm is then as follows.
ALGORITHM 5.6. Jacobi’s method to find the eigenvalues of a symmetric matrix: repeat choose aj,k pair call Jacobi-Rotation(A, j, k) until A is sufficiently diagonal
8/72 We still need to decide how to pick j, k pairs. There are several possibilities. To measure progress to convergence and describe these possibilities, we define
off(A) a2 . ≡ jk 1 j 9/72 LEMMA 5.4. Let A′ be the matrix after calling Jacobi-Rotation(A, j, k) 2 2 2 for any j = k. Then off (A′)= off (A) a . 6 − jk The next algorithm was the original version of the algorithm (from Jacobi in 1846), and it has an attractive analysis although it is too slow to use. ALGORITHM 5.7. Classical Jacobi’s algorithm: while off(A) > tol (where tol is the stopping criterion set by user) choose j and k so ajk is the largest off-diagonal entry in magnitude call Jacobi-Rotation(A, j, k) end while 10/72 THEOREM 5.11. After one Jacobi rotation in the classical Jacobi’s 1 n(n 1) algorithm, we have off(A′) 1 off(A), where N = − = the ≤ − N 2 number of superdiagonal entriesq of A. After k Jacobi-Rotations off( ) is no more than (1 1 )k/2 off(A). · − N So the classical Jacobi’s algorithm converges at least linearly with the error (measured by off(A)) decreasing by a factor of at least 1 1 at − N a time. In fact, it eventually converges quadratically. q 11/72 THEOREM 5.12. Jacobi’s method is locally quadratically convergent after N steps (i.e., enough steps to choose each ajk once). This means that for i large enough 2 off(Ai+N )= O(off (Ai )). In practice, we do not use the classical Jacobi’s algorithm because n2 n searching for the largest entry is too slow: We would need to search 2− entries for every Jacobi rotation, which costs only O(n) flops to perform, and so for large n the search time would dominate. Instead, we use the following simple method to choose j and k. 12/72 ALGORITHM 5.8. Cyclic-by-row-Jacobi: Sweep through the off diagonals of A rowwise. repeat for j = 1 to n 1 for k = j + 1 −to n call Jacobi-Rotation (A, j, k) end for end for until A is sufficiently diagonal A no longer changes when Jacobi-Rotation(A, j, k) chooses only c = 1 and s = 0 for an entire pass through the inner loop. The cyclic Jacobi’s algorithm is also asymptotically quadratically convergent like the classical Jacobi’s algorithm [J. H. Wilkinson. The Algebraic Eigenvalue Problem. Oxford University Press, Oxford, UK, 1965; p. 270]. The cost of one Jacobi ”sweep” (where each j, k pair is selected once) is approximately half the cost of reduction to tridiagonal form and the computation of eigenvalues and eigenvectors using QR iteration, and more than the cost using divide-and-conquer. Since Jacobi’s method often takes 5-10 sweeps to converge, it is much slower than the competition. 13/72 Algorithms for the Singular Value Decomposition (SVD) All the algorithms for the eigendecomposition of a symmetric matrix A, except Jacobi’s method, have the following structure: 1 Reduce A to tridiagonal form T with an orthogonal matrix Q1: T A = Q1TQ1 . 2 T Find the eigendecomposition of T : T = Q2ΛQ2 , where Λ is the diagonal matrix of eigenvalues and Q2 is the orthogonal matrix whose columns are eigenvectors. T 3 Combine these decompositions to get A =(Q1Q2)Λ(Q1Q2) . The columns of Q = Q1Q2 are the eigenvectors of A. 14/72 All the algorithms for the SVD of a general matrix G, except Jacobi’s method, have an analogous structure: 1 Reduce G to bidiagonal form B with orthogonal matrices U1 and T V1: G = U1BV1 . This means B is nonzero only on the main diagonal and first superdiagonal. 2 T Find the SVD of B: B = U2ΣV2 , where Σ is the diagonal matrix of singular values, and U2 and V2 are orthogonal matrices whose columns are the left and right singular vectors, respectively. T 3 Combine these decompositions to get G =(U1U2)Σ(V1V2) . The columns of U = U1U2 and V = V1V2 are the left and right singular vectors of G, respectively. 15/72 Practical algorithms for computing the SVD 1 QR iteration and its variations. This is the fastest algorithm for small matrices up to size n = 25 to find all the singular values of a bidiagonal matrix. 2 Divide-and-conquer. This is the fastest method to find all singular values and singular vectors for matrices larger than n = 25. However, it does not guarantee computation of tiny singular values. 3 Bisection and inverse iteration. 4 Jacobi’s method. SVD of a dense matrix G is computd implicitly applying of Jacobi’s method (Algorithm 5.5) to GG T or G T G. 16/72 QR Iteration and Its Variations for the Bidiagonal SVD There is a long history of variations on QR iteration for the SVD, designed to be as efficient and accurate as possible; see [B. Parlett. Acta Numerica, The new qd algorithms, pages 459-491. Cambridge University Press, Cambridge, UK, 1995] for a good survey. The algorithm in the LAPACK routine sbdsqr" was originally based on [J. Demmel and W. Kahan. Accurate singular values of bidiagonal matrices. SIAM J. Sci. Statist. Comput, 11:873-912, 1990] and later updated to use the algorithm in [K. Fernando and B. Parlett. Accurate singular values and differential qd algorithms. Numer. Math., 67:191-229, 1994] in the case when singular values only are desired. This latter algorithm, called dqds" for historical reasons (dqds" is short for "differential quotient-difference algorithm with shifts" [H. Rutishauser. Lectures on Numerical Mathematics. Birkhauser, Basel, 1990]), is elegant, fast, and accurate, so we will present it. 17/72 LR iteration ALGORITHM 5.9. LR iteration: Let T0 be any symmetric positive definite matrix. The following algorithm produces a sequence of similar symmetric positive definite matrices Ti : i = 0 repeat 2 Choose a shift τi smaller than the smallest eigenvalue of Ti . 2 T Compute the Cholesky factorization Ti τ I = B Bi − i i (Bi is an upper triangular matrix with positive diagonal.) T 2 Ti+1 = Bi Bi + τi I i = i + 1 until convergence 18/72 LR iteration versus QR iteration LR iteration is very similar in structure to QR iteration: We compute a factorization, and multiply the factors in reverse order to get the next iterate Ti+1. It is easy to see that Ti+1 and Ti are similar: T 2 T T T 2 T T Ti+1 = Bi Bi + τi I = Bi− Bi Bi Bi + τi Bi− Bi I I T T 2 T T T (1) = Bi− (Bi Bi + τ|i )B{zi =} Bi− Ti Bi .| {z } Ti 2| {z } In fact, when the shift τi = 0, we can show that two steps of LR iteration produce the same T2 as one step of QR iteration. 19/72 Jacobi’s Method for the SVD ALGORITHM 5.12. Compute and apply a one-sided Jacobi rotation to G in coordinates j, k: proc One-Sided-Jacobi-Rotation (G, j, k) T T T Compute ajj =(G G)jj , ajk =(G G)jk , and akk =(G G)kk if ajk is not too small | | τ =(ajj akk )/(2 ajk ) − · t = sign(τ)/( τ + √1+ τ 2) c = 1/√1+ t|2 | s = c t G = G· R(j, k,θ) ... where c = cos θ and s = sin θ if right· singular vectors are desired J = J R(j, k,θ) end if · end if Note that the jj, jk, and kk entries of A = G T G are computed by procedure One-Sided-Jacobi-Rotation, after which it computes the Jacobi rotation R(j, k,θ) in the same way as procedure Jacobi-Rotation (Algorithm 5.5). 20/72 ALGORITHM 5.13. One-sided Jacobi: Assume that G is n-by-n. The outputs are the singular values σi , the left singular vector matrix U, and the right singular vector matrix V so that G = UΣV T , where Σ= diag(σi ). repeat for j = 1 to n 1 for k = j +− 1 to n call One-Sided-Jacobi-Rotation (G, j, k) end for end for until G T G is diagonal enough Let σi = G(:, i) (the 2-norm of column i of G) || ||2 Let U =[u1,..., un], where ui = G(:, i)/σi let V = J, the accumulated product of Jacobi rotations 21/72 The following theorem shows that one-sided Jacobi can compute the SVD to high relative accuracy, despite roundoff, provided that we can write G = DX , where D is diagonal and X is well-conditioned. THEOREM 5.15. Let G = DX be an n-by-n matrix, where D is diagonal and nonsingular, and X is nonsingular. Let Gˆ be the matrix after calling One-Sided-Jacobi-Rotation (G, j, k) m times in floating point arithmetic. Let σ1 ... σn be the singular values of G, and letσ ˆ1 ... σˆn be the singular≥ values≥ of Gˆ. Then ≥ ≥ σi σˆi | − | O(mε)κ(X ), σi ≤ 1 where κ(X )= X X − is the condition number of X . In other words, the relative|| ||·|| error in|| the singular values is small if the condition number of X is small. 22/72 Proof. We first consider m = 1; i.e., we apply only a single Jacobi rotation and later generalize to larger m. Examining One-Sided-Jacobi-Rotation (G, j, k), we see that Gˆ = fl(G R˜), where R˜ is a floating point Givens rotation. By construction, R˜ differs· from some exact Givens rotation R by O(ε) in norm. (It is not important or necessarily true that R˜ differs by O(ε) from the ”true” Jacobi rotation, the one that One-Sided-Jacobi-Rotation (G, j, k) would have computed in exact arithmetic. It is necessary only that it differs from some rotation by O(ε). This requires only that c2 + s2 =1+ O(ε), which is easy to verify.) Our goal is to show that Gˆ = GR(I + E) for some E that is small in norm: E 2 = O(ε)κ(X ). If E were zero, then Gˆ and GR would have the samek k singular values, since R is exactly orthogonal. When E is less than one in norm, we can use Corollary 5.2 to bound the relative difference in singular values by σi σˆi T T T | − | (I +E)(I +E) I 2 = E+E +EE 2 3 E 2 = O(ε)κ(X ), σi ≤ || − || || || ≤ || || as desired. 23/72 Now we construct E. Since R˜ multiplies G on the right, each row of Gˆ depends only on the corresponding row of G; write this in Matlab notation as Gˆ(i, :) = fl(G(i, :) R˜). Let F = Gˆ GR. Then using lemma 3.1 ( Gˆ = O(ε) GR ) and· the fact that G =−DX , k k k k2 F (i, :) = Gˆ(i, :) G(i, :)R = O(ε) G(i, :)R = O(ε) dii X (i, :) || ||2 || − ||2 || ||2 || ||2 1 1 and so dii− F (i, :) 2 = O(ε) X (i, :) 2, or D− F 2 = O(ε) X 2. || ||1 T || 1 || || 1 || 1 1 || || Therefore, since R− = R and G − =(DX )− = X − D− , T 1 T 1 1 Gˆ = GR +F = GR(I +R G − F )= GR(I +R X − D− F ) GR(I +E) ≡ G −1 where | {z } T 1 1 1 E R X − D− F = O(ε) X X − = O(ε)κ(X ) || ||2 ≤ || ||2|| ||2 || ||2 || ||2|| ||2 O(ε) X || ||2 as desired. | {z } 24/72 To extend this result to m > 1 rotations, note that in exact arithmetic we would have Gˆ = GR = DXR = DXˆ, with κ(Xˆ)= κ(X ), so that the bound σi σˆi T T T | − | (I +E)(I +E) I 2 = E+E +EE 2 3 E 2 = O(ε)κ(X ), σi ≤ || − || || || ≤ || || would apply at each of the m steps, yielding bound σi σˆi | − | O(mε)κ(X ). σi ≤ Because of roundoff, κ(Xˆ) could grow by as much as κ(I + E) (1 + O(ε)κ(X )) at each step, a factor very close to 1, which we absorb≤ into the O(mε) term. 25/72 Numerical Solution of Poisson’s Equation The model problem is the following Dirichlet problem for Poisson’s equation: u(x)= f (x) in Ω, −△ (2) u =0 on ∂Ω. Here, f (x) is a given function, u(x) is the unknown function, and the domain Ω is the unit square Ω = (x , x ) (0, 1) (0, 1) . To solve { 1 2 ∈ × } numerically (2) we first discretize the domain Ω with x1i = ih1 and x = jh , where h = 1/(ni 1) and h = 1/(nj 1) are the mesh sizes 2j 2 1 − 2 − in the directions x1, x2, respectively, ni and nj are the numbers of discretization points in the directions x1, x2, respectively. Usually, in computations we have the same mesh size h = h1 = h2. In this example we choose ni = nj = n with n = N + 2, where N is the number of inner nodes in the directions x1, x2, respectively. Indexes (i, j) are such that 0 i, j < n and are associated with every ≤ global node nglob of the finite difference mesh. Global nodes numbers nglob in two-dimensional case can be computed using the following formula: nglob = j + ni (i 1). (3) − 26/72 We use the standard finite difference discretization of the Laplace operator ∆u in two dimensions and obtain discrete laplacian ∆ui,j : ui+1,j 2ui,j + ui 1,j ui,j+1 2ui,j + ui,j 1 ∆u = − − + − − , (4) i,j h2 h2 where ui,j is the solution at the discrete point (i, j). Using (4), we obtain the following scheme for solving problem (2): ui+1,j 2ui,j + ui 1,j ui,j+1 2ui,j + ui,j 1 − − + − − = fi j , (5) − h2 h2 , where fi,j are the value of the function f at the discrete point (i, j). Then (5) can be rewritten as 2 (ui+1,j 2ui,j + ui 1,j + ui,j+1 2ui,j + ui,j 1)= h fi,j , (6) − − − − − or in the more convenient form as 2 ui+1,j + 4ui,j ui 1,j ui,j+1 ui,j 1 = h fi,j . (7) − − − − − − 27/72 System (7) can be written in the form Au = b. The vector b has the 2 components bi,j = h fi,j . The explicit elements of the matrix A are given by the following block matrix AN IN − .. .. IN . . A = − .. .. . . IN − IN AN − with blocks AN of order N given by 4 1 0 0 0 1− 4 1 0 ··· 0 − − ··· AN = 0 1 4 0 0 , − ··· ··· ··· ··· ··· ··· ··· 0 0 1 4 ··· ··· − which are located on the diagonal of the matrix A, and blocks with the identity matrices IN of order N on its off-diagonals. The matrix A is symmetric and positive− definite. Therefore, we can use the LU factorization algorithm without pivoting. 28/72 Suppose, that we have discretized the two-dimensional domain Ω as described above, and the number of the inner points in both directions are N = 3. We present the schematic discretization for the inner nodes of this domain and corresponding numbering for the global nodes using (3) in the following scheme: a1,1 a1,2 a1,3 n1 n2 n3 1 2 3 a a a = n n n = 4 5 6 . 2,1 2,2 2,3 ⇒ 4 5 6 ⇒ a3,1 a3,2 a3,3 n7 n8 n9 7 8 9 (8) Then the explicit form of the block matrix A will be: 4 1 0 1 0 0 0 0 0 1− 4 1 −0 1 0 0 0 0 −0 1− 4 0− 0 1 0 0 0 − − 1 0 0 4 1 0 1 0 0 − − − A = 0 1 0 1 4 1 0 1 0 . − − − − 0 0 1 0 1 4 0 0 1 − − − 0 0 0 1 0 0 4 1 0 − − 0 0 0 0 1 0 1 4 1 − − − 0 0 0 0 0 1 0 1 4 − − 29/72 Now we present the numerical solution of problem (2). We define the right hand side f (x) of (2) as 2 2 (x1 c1) (x2 c2) 1 f (x , x )= Af exp − − , (9) 1 2 − 2s2 − 2s2 a(x , x ) 1 2 1 2 The coefficient a(x1, x2) in (9) is given by the following Gaussian function: (x c )2 (x c )2 a(x , x )=1+ A exp 1 − 1 2 − 2 , (10) 1 2 − 2s2 − 2s2 1 2 Here A, Af are the amplitudes of these functions, c1, c2 are constants which show the location of the center of the Gaussian functions, and s1, s2 are constants which show spreading of the functions in x1 and x2 directions. We produce the mesh with the points (x1i , x2j ) such that x1i = ih, x2j = jh with h = 1/(N + 1), where N is the number of the inner points in x1 and x2 directions. We take the same number of points in x1 and x2 directions: ni = nj = N + 2. The linear system of equations Au = f is solved then via the LU factorization of the matrix A without pivoting. 30/72 u(x ,x ) with A = 12, n = 20 u(x ,x ) with A = 12, n = 20 1 2 × -3 1 2 ×10 -3 10 1 0.9 ×10 -3 5 5 6 0.8 5 4 0.7 4 ) 2 4 ,x 0.6 1 3 3 3 2 u(x x 0.5 2 0.4 1 2 2 0.3 0 1 0.2 1 1 0.8 1 0.6 0.1 0.4 0.5 x 0.2 2 0 0 0 0 0 x 0 0.2 0.4 0.6 0.8 1 1 x 1 u(x ,x ) with A = 12, N = 40 u(x ,x ) with A = 12, N = 40 1 2 × -3 1 2 ×10 -3 10 1 0.9 ×10 -3 5 5 6 0.8 5 4 0.7 4 ) 2 4 ,x 0.6 1 3 3 3 2 u(x x 0.5 2 0.4 1 2 2 0.3 0 1 0.2 1 1 0.8 1 0.6 0.1 0.4 0.5 x 0.2 2 0 0 0 0 0 x 0 0.2 0.4 0.6 0.8 1 1 x 1 Figure: Solution of problem (2) with the right hand side f (x) given by (9) and a(x1, x2) in (9) given by the Gaussian function (10). Figure 1 shows the results of the numerical simulations for different discretizations of the unit square with the number of the inner points N = 20, 40 and for A = 12, Af = 1, c1 = c2 = 0.5, s1 = s2 = 1 in (10) and (9). The Matlab programs of Sect.A.1 is available in Appendix for running of this test. 31/72 Introduction to Iterative Methods for Solution of Linear Systems We will discuss following basic iterative methods: 1. Jacobi. 2. Gauss-Seidel. 3. Successive overrelaxation (SOR). as well as Krylov subspace methods using conjugate gradient (CG) algorithm and preconditioning. 32/72 Basic Iterative Methods The basic iterative methods for the solution of system of linear equations Ax = b are: 1. Jacobi. 2. Gauss-Seidel. 3. Successive overrelaxation (SOR). These methods produce a sequence of iterative solutions xm which 1 converge to the solution x = A− b provided that there exist initial guess x0. To use iterative methods we will introduce a splitting: A = M K, where det M = 0. Applying this splitting to Ax = b we get: − 6 Ax = Mx Kx = b. − From the equation above we can get Mx = b + Kx and thus 1 1 1 x = M− (b + Kx)= M− b + M− Kx. 33/72 Let us define 1 1 Rx = M− Kx, c = M− b. The the iterative update for xm can be written as: xm+1 = Rxm + c, (11) where m is the number of iteration. 34/72 Lemma Rx Let R = maxx=0 k x k . If R < 1 then iterations (11) will converge for k k 6 k k k k all initial guesses x0. Theorem Rx Let R = maxx=0 k x k . If ρ(R) < 1 then iterations (11) will converge k k 6 k k for all initial guesses x0. Proof. Using (12) we have: xm+1 x = R(xm x) R xm x k − k k − k ≤ k k · k − k (12) Rm+1 x x = λm+1(x x). ≤ k k · k 0 − k 0 − Since ρ(R) < 1 then using lemma above we choose a such operator norm that R < 1. Then by lemma 1 iterations (11) will converge for all k k(R,ε) initial guesses x0. 35/72 In the iterative methods considered below we want to have efficient splitting A = M K as possible. Let us introduce following notations: If A has no zeros on− its diagonal we will write the splitting as A = D L˜ U˜ = D(I L U), (13) − − − − where D is a diagonal matrix, L˜ is the strictly lower triangular part of A such that DL = L˜, and U˜ is the− strictly upper triangular part of A such that DU = U˜. − 36/72 Jacobi Method The splitting for Jacobi method is: A = D (L˜ + U˜). (14) − Applying it to the solution of Ax = b we have: Ax = Dx (Lx˜ + Ux˜ )= b. − From the equation above we can get Dx = b + Lx˜ + Ux˜ and thus 1 1 1 1 x = D− (b + Lx˜ + Ux˜ )= D− b + D− Lx˜ + D− Ux˜ . 37/72 Let us define 1 RJ D− (L˜ + U˜)= L + U, ≡ (15) 1 cJ D− b. ≡ Then iterative update in the Jacobi method can be written as: xm+1 = RJ xm + cJ . (16) Formula (16) by mult. by D both sides can be also written as Dxm+1 = b + Lx˜ m + Ux˜ m (17) or using the definition of D on the element level as aj j xm j = bj aj k xm k (18) , +1, − , , k=j X6 38/72 Algorithms: one step in the Jacobi method for j =1 to n bj k=j aj,k xm,k xm+1,j = − 6 P aj,j end In the case of the model problem for the Poisson’s equation on a square we will have the following Jacobi’s method: for i =1 to N for j =1 to N 2 um,i 1,j + um,i+1,j + um,i,j 1 + um,i,j+1 + h fi,j u = − − m+1,i,j 4 end end 39/72 Gauss-Seidel Method To get the Gauss-Seidel method we use the same splitting (14) as for the Jacobi method. Applying it to the solution of Ax = b we have: Ax = Dx (Lx˜ + Ux˜ )= b, − we rearrange terms in the right hand side now like that: Dx Lx˜ = b + Ux˜ (19) − and thus now the solution is computed as 1 1 1 x =(D L˜)− (b + Ux˜ )=(D L˜)− b +(D L˜)− Ux˜ . − − − 40/72 We can rewrite the above equation using notations DL = L˜ and DU = U˜ as: 1 1 x =(D L˜)− b +(D L˜)− Ux˜ − − 1 1 =(D DL)− b +(D DL)− Ux˜ − − (20) 1 1 1 1 =(I L)− D− b +(I L)− D− Ux˜ − − 1 1 1 =(I L)− D− b +(I L)− Ux. − − Let us define 1 RGS (I L)− U, ≡ − 1 1 (21) cGS (I L)− D− b. ≡ − 41/72 Then iterative update in the Gauss-Seidel method can be written as: xm+1 = RGS xm + cGS . (22) Formula (20) in iterative update after multiplication by (I L)D can be also written as − (I L)Dxm = b + DUxm (23) − +1 or using the definition of D, L, U on the element level as j 1 n − aj j xm j = bj aj k xm k aj k xm k . (24) , +1, − , +1, − , , k k j X=1 X= +1 j 1 Here, terms k−=1 aj,k xm+1,k represent already updated terms with xm+1 which we can use in computations since we have computed them, and n P terms k=j+1 aj,k xm,k are with older xm which we have updated on the iteration m. P 42/72 Algorithm: one step in the Gauss-Seidel method for j =1 to n j−1 n bj k=1 aj,k xm+1,k k=j+1 aj,k xm,k xm+1,j = − − P aj,j P end 43/72 Gauss-Seidel method for the Poisson’s equation If we want apply the Gauss-Seidel method for the solution of the model problem for the Poisson’s equation we need organize ordering for the new m + 1 variables and old already computed values m. We will use such called red-black ordering based on the chessboard-like coloring. Let B nodes correspond to the black squares on a chessboard, and R nodes correspond to the weight squares. The the the Gauss-Seidel method for the solution of the two-dimensional Poisson’s equation on a square becomes the following. for all R red nodes i, j 2 um,i 1,j + um,i+1,j + um,i,j 1 + um,i,j+1 + h fi,j u = − − m+1,i,j 4 end for all B black nodes i, j 2 um+1,i 1,j + um+1,i+1,j + um+1,i,j 1 + um+1,i,j+1 + h fi,j u = − − m+1,i,j 4 end 44/72 Successive Overrelaxation SOR(ω) Method The method of successive overrelaxation improves the Gauss-Seidel method in the following way: it takes weighted average of values xm+1 and xm such that: xm j = (1 ω)xm j + ωxm j , (25) +1, − , +1, where ω is a weight called also relaxation parameter. When ω = 1, then we will get usual Gauss-Seidel method, when ω < 1 we get underrelaxation method, and when ω > 1 - overrelaxation method. To get SOR(ω) method in a matrix form, we again apply splitting (14) and obtain equation similar to (19), but only in the iterative form: (D L˜)xm = b + Ux˜ m. (26) − +1 45/72 Applying now weighted average (25) to this equation we have: (D L˜)((1 ω)xm + ωxm )= b + Ux˜ m, − − +1 (D L˜)(1 ω)xm +(D L˜)ωxm = b + Ux˜ m, − − − +1 (D L˜)ωxm = b + Ux˜ m (D L˜)(1 ω)xm, − +1 − − − (D L˜)ωxm = b + Ux˜ m D(1 ω)xm + L˜(1 ω)xm. − +1 − − − (27) From (25) we see that (1 ω)xm = xm+1 ωxm+1 = (1 ω)xm+1, and thus the last expression we− can write as − − (D L˜)ωxm = b + Ux˜ m D(1 ω)xm + L˜(1 ω)xm , − +1 − − − +1 (D L˜)ωxm L˜(1 ω)xm = b + Ux˜ m D(1 ω)xm, − +1 − − +1 − − (Dω L˜)xm = b + Ux˜ m D(1 ω)xm − +1 − − (28) 1 Denoting ω := ω we get SOR-scheme: (D ωL˜)xm = ωb + ((1 ω)D + ωU˜)xm. (29) − +1 − 46/72 Using notations DL = L˜ and DU = U˜ the equation (29) can be rewritten as 1 1 xm+1 =(D ωL˜)− ωb +(D ωL˜)− ((1 ω)D + ωU˜)xm − − − (30) 1 1 1 =(I ωL)− D− ωb +(I ωL)− ((1 ω)I + ωU)xm. − − − Now defining 1 RSOR =(I ωL)− ((1 ω)I + ωU), − − 1 1 (31) cSOR =(I ωL)− D− ωb − we can rewrite (30) in the form xm+1 = RSOR xm + cSOR . (32) 47/72 Algorithm: one step in the SOR(ω) method To get SOR(ω) for implementation, we take xm+1,j in the right hand side of (25) from the Gauss-Seidel algorithm and obtain the following algorithm: One step in the SOR(ω) method for j =1 to n j 1 n bj k−=1 aj,k xm+1,k k=j+1 aj,k xm,k xm+1,j = (1 ω)xm,j + ω − − − " P aj,j P # end 48/72 Algorithm: SOR(ω) for the solution of the Poisson’s equation To apply the SOR(ω) method for the solution of the model problem for the Poisson’s equation we will use the red-black ordering as in the Gauss-Seidel method. The SOR(ω) method will be the following. One step in the SOR(ω) method for two-dimensional Poisson’s equation for all R red nodes i, j 2 ω(um,i 1,j + um,i+1,j + um,i,j 1 + um,i,j+1 + h fi,j ) um i j = (1 ω)um i j + − − +1, , − , , 4 end for all B black nodes i, j um i j = (1 ω)um i j +1, , − , , 2 ω(um+1,i 1,j + um+1,i+1,j + um+1,i,j 1 + um+1,i,j+1 + h fi,j ) + − − 4 (33) end 49/72 Study of Convergence of SOR(ω) Theorem If the matrix A is strictly row diagonally dominant (if aii > i=j aij ), then Jacobi and Gauss-Seidel methods converge such| that| 6 | | P RGS < RJ < 1, (34) k k∞ k k∞ where RGS and RJ are defined in (21), (15), respectively. Proof. We can rewrite (34) with e = (1,..., 1)T as RGS = RGS e < RJ = RJ e < 1, (35) k k∞ k | | k∞ k k∞ k | | k∞ Using definition (21), (15) we can write (35) as 1 RGS e = (I L)− U e < RJ e = L + U e < 1, k| | k∞ k| − | k∞ k | | k∞ k | | k∞ (36) 50/72 or n 1 1 1 − i (I L)− U e (I L)− U e L U e | − | ≤| − | | | ≈| |·| |· i X=0 (37) n 1 − i 1 L U e (I L )− U e. ≤ | | ·| |· ≈ −| | | | i X=0 1 n 1 i All entries of (I L )− − L > 0. We assume that −| | ≈ i=0 | | RJ = ρ< 1 and thus k k∞ P 0 (I L U ) e ≤ −| |−| | · or RJ e =( L + U ) e e | |· | | | | · ≤ Thus, 0 L (I L U ) e =( L L 2 L U ) e ≤| | −| |−| | · | |−| | −| || | · and the following inequality is true U e (I L )( L + U ) e =( L L 2 L U + U ) e, | |· ≤ −| | | | | | · | |−| | −| || | | | · from which follows (35) and thus (34). 51/72 Theorem Let the spectral radius of RSOR is such that ρ(RSOR ) ω 1 . Then 0 <ω< 2 is required for convergence of SOR(ω). ≥| − | Proof. We write the characteristic polynomial for RSOR as ϕ(λ) = det(λI RSOR ) = det((I ωL)(λI RSOR )) − − − (38) = det((λ + ω 1)I ωλL ωU). − − − From the equation above we have n ϕ(0) = λi (RSOR )= det((ω 1)I )= (ω 1) , ± ± − ± − Y and thus max λi (RSOR ) ω 1 , i | |≥| − | from what follows that ρ(RSOR ) ω 1 . ≥| − | 52/72 Theorem If A is s.p.d matrix then ρ(RSOR ) < 1 for all 0 <ω< 2. Example. The matrix A in the model problem for the Poisson’s equation is s.p.d.. Thus, SOR(ω) for this problem will converge for all 0 <ω< 2. 53/72 Krylov Subspace Methods Krylov subspace methods are used for the solution of large system of linear equations Ax = b and for finding eigenvalues of A avoiding matrix-matrix multiplication. Instead, these methods use multiplication of matrix by the vector. Definition The Krylov subspace generated by matrix A of the size n n and vector b of the size n is the linear subspace spanned by powers of×A and multiplied by b: 2 r 1 Kr (A, b)= b, Ab, A b,..., A − b . (39) { } For the symmetric matrix A we can write the decomposition QT AQ = H, where Q is the orthogonal transformation and H is the upper Hessenberg matrix which also will be a lower Hessenberg, and thus, tridiagonal matrix. Writing Q as Q = q ,..., qn and using AQ = QH we have { 1 } j+1 Aqj = hi,j qi . (40) i X=1 54/72 We multiply both sides of the above expression by orthonormal vectors T qm and use the fact that qi are orthonormal to obtain: j+1 T T q Aqj = hi j q qi = hm j , 1 m j. (41) m , m , ≤ ≤ i X=1 We can rewrite (40) as j hj j qj = Aqj hi j qi . (42) +1, +1 − , i X=1 The formula above as well as (41) are used in the Arnoldi algorithm for the reduction of matrix A to upper Hessenberg form. Let r will be the number of columns in the matrices Q and H which we need to compute. We now formulate the Arnoldi algorithm which performs partial reduction to Hessenberg form. The vectors qj computed in this algorithm are called Arnoldi vectors. 55/72 Arnoldi algorithm Initialization: q = b 1 kbk2 for j =1 to r z = Aqj for i =1 to j T hi,j = qi z z = z hi,j qi − end hj ,j = z +1 k k2 if hj+1,j = 0 quit z qi = +1 hj+1,j end 56/72 For the case of a symmetric matrix A the Arnoldi algorithm can be simplified since the matrix H is symmetric and tridiagonal what means that α1 β1 ...... β α ...... 1 2 H = ...... (43) ...... αn 1 βn 1 − − ...... βn 1 αn − Rewriting (40) for the case of the symmetric and tridiagonal H given by (43) we have Aqj = βj 1qj 1 + αj qj + βj qj+1. (44) − − We note that columns of Q are orthonormal and thus T T qj Aqj = qj (βj 1qj 1 + αj qj + βj qj+1)= αj . (45) − − Combining (45) and (44) we get Lanczos algorithm for partial reduction to symmetric tridiagonal form. 57/72 Lanczos algorithm Initialization: q = b ,β = 0, q = 0 1 kbk2 0 0 for j =1 to r z = Aqj T αj = qj z z = z αj qj βj qj − − −1 −1 βj = z k k2 if βj = 0 quit z qi = +1 βj end 58/72 The vectors qj computed by the algorithm are called Lanczos vectors. The vectors qr computed in the Lanczos or Arnoldi algorithm create orthonormal basis of the Krylov subspace Kr defined in (39). The matrix T Hr = Qr AQr in both algorithms is called the projection of A to the Krylov subspace Kr . Our goal now to use r steps in the Lanczos or Arnoldi algorithms to solve linear system Ax = b. To do that we seek best approximation xr to the 1 exact solution x = A− b given by r xr = zj qj = Qr z, (46) j X=1 T where z =(z1,..., zr ) . 59/72 Let us define the residual as Rr = b Axr . For the case of s.p.d. matrix − T 1 1/2 A we can define the norm R −1 := (R A− R) . We note that k kA R −1 = xr x A. Thus, the the best computed solution xr will k kA k − k minimize R A−1 . The algorithm which can compute such vector xr is called thek conjugatek gradient (CG) algorithm. Theorem T Let A is a symmetric matrix, Hr = Qr AQr and residuals are defined as Rr = b Axr xr Kr . When Hr is not singular we can define − ∀ ∈ 1 xr = Qr H− e b , (47) r 1k k2 T T where e1 = (1, 0,..., 0) . Then Qr Rr = 0. Let A is also positive definite matrix. Then Hr must be nonsingular and xr defined as in (47) minimizes Rr −1 for all xr Kr , where k kA ∈ Rr = Rr qr . ±k k2 +1 60/72 Conjugate gradient algorithm Now we introduce conjugate gradients vectors pr . The pr are called gradients because in a single step of the CG algorithm we compute the approximated solution as xr = xr 1 + νpr and this solution minimizes the T −1 1/2 residual norm Rr A−1 =(Rr A− Rr ) . The vectors pr are called k k T conjugate, or more precisely A-conjugate, because p Apj =0 if j = r. r 6 61/72 Conjugate gradient algorithm Initialization: r = 0; x0 = 0; R0 = b; p1 = b; repeat r = r + 1 z = A pr · T T νr =(Rr 1Rr 1)/(pr z) − − xr = xr 1 + νr pr − Rr = Rr 1 νr z − T− T µr+1 =(Rr Rr )/(Rr 1Rr 1) − − pr+1 = Rr + µr+1pr until Rr is small enough || ||2 62/72 Preconditioning for Linear Systems Preconditioning technique is used for the reduction of the condition number of the considered problem. For the solution of linear system of equations Ax = b the preconditioner matrix P of a matrix A is a matrix 1 1 P− A such that P− A has a smaller condition number then the original matrix A. This means that instead of the solution of a system Ax = b we will consider solution of the system 1 1 P− Ax = P− b. (48) The matrix P should have the following properties: P is s.p.d. matrix; 1 P− A is well conditioned; The system Px = b should be easy solvable. The preconditioned conjugate gradient method is derived as follows. First we multiply both sides of (48) by P1/2 to get 1/2 1/2 1/2 1/2 (P− AP− )(P x)= P− b. (49) 63/72 We note that the system (49) is s.p.d. since we have chosen the matrix P such that P = QQT which is the eigendecomposition of P. Then the matrix P1/2 will be s.p.d. if it is defined as P1/2 = Q1/2QT . Defining 1/2 1/2 1/2 1/2 A˜ := P− AP− , x˜ := P x, b˜ = P− b 1 we can rewrite (49) as the system A˜x˜ = b˜. Matrices A˜ and P− A are 1 1/2 1/2 1 similar since P− A = P− AP˜ . Thus, A˜ and P− A have the same 1 1 eigenvalues. Thus, instead of the solution of P− Ax = P− b we will present preconditioned conjugate gradient (PCG) algorithm for the solution of A˜x˜ = b˜. 64/72 Preconditioned conjugate gradient algorithm 1 1 Initialization: r = 0; x0 = 0; R0 = b; p1 = P− b; y0 = P− R0 repeat r = r + 1 z = A pr · T T νr =(yr 1Rr 1)/(pr z) − − xr = xr 1 + νr pr − Rr = Rr 1 νr z −1 − yr = P− Rr T T µr+1 =(yr Rr )/(yr 1Rr 1) − − pr+1 = yr + µr+1pr until Rr is small enough || ||2 65/72 Common preconditioners Common preconditioner matrices P are: Jacobi preconditioner P =(a11,..., ann). Such choice of the 1 preconditioner reduces the condition number of P− A around factor n of its minimal value. block Jacobi preconditioner P1,1 ... 0 P = ...... (50) 0 ... Pr,r with Pi,i = Ai,i , i = 1,..., r, for the block matrix A given by A1,1 ... A1,r A = ...... (51) Ar,1 ... Ar,r with square blocks Ai,i , i = 1,..., r. Such choice of preconditioner P 1/2 1/2 minimizes the condition number of P− AP− within a factor of r. 66/72 Method of SSOR can be used as a block preconditioner as well. Incomplete Cholesky factorization with A = LLT is often used for PCG algorithm. In this case a sparse lower triangular matrix L˜ is chosen to be close to L. Then the preconditioner is defined as P = L˜L˜T . Incomplete LU preconditioner. 67/72 Example We will construct a lower triangular matrix using Given’s rotation from the matrix 5 4 3 A = 4 6 1 . 3 1 7 68/72 Given’s matrix for j < k function [G] = GivensMatrixLow(A, j,k) a = A(k, k) b = A(j, k) r = sqrt(a2 + b2); c = a/r; s = b/r; − G = eye(length(A)); G(j, j)= c; G(k, k)= c; G(j, k)= s; G(k, j)= s; − 69/72 >>G1up = GivensMatrixLow(A,2,3) 1.000000000000000 0 0 G1= 0 0.989949493661166 0.141421356237310 0 0.141421356237310− 0.989949493661166 >> A1 =G1*A 5.000000000000000 4.000000000000000 3.000000000000000 A1= 3.535533905932737 5.798275605729690 0.000000000000000 3.535533905932738 1.838477631085023− 7.071067811865475 70/72 >>G2 = GivensMatrixLow(A1,1,3) 0.920574617898323 0 0.390566732942472 G2= 0 1.000000000000000− 0 0.390566732942472 0 0.920574617898323 >> A2=G2*A1 3.222011162644131 2.964250269632601 0.000000000000000 A2= 3.535533905932737 5.798275605729690 −0.000000000000000 5.207556439232954 3.254722774520597− 7.681145747868607 71/72 >>G3 = GivensMatrixLow(A2,1,2) 0.890391914715406 0.455194725594918 0 G3= 0.455194725594918− 0.890391914715406 0 0 0 1.000000000000000 >> A3=G3*A2 1.259496302198541 0 0.000000000000000 A3= 4.614653291088246 6.512048806713364 −0.000000000000000 5.207556439232954 3.254722774520597− 7.681145747868607 72/72