Ubung¨ 4.10 Check that the inverse of L(k) is given by 1 .. 0 . . . .. .. . . 0 1 (L(k))−1 = . (4.7) . .. . −lk ,k . +1 . . 0 .. . . .. .. 0 ··· 0 −l 0 ··· 0 1 n,k Each step of the above Gaussian elimination process sketched in Fig. 4.2 is realized by a multiplication from the left by a matrix, in fact, the matrix (L(k))−1 (cf. (4.7)). That is, the Gaussian elimination process can be described as A = A(1) → A(2) =(L(1))−1A(1) → A(3) =(L(2))−1A(2) =(L(2))−1(L(1))−1A(1) → ... → A(n) =(L(n−1))−1A(n−1) = ... =(L(n−1))−1(L(n−2))−1 ... (L(2))−1(L(1))−1A(1) =:U upper triangular Rewriting this|{z yields,} the LU-factorization A = L(1) ··· L(n−1) U =:L The matrix L is a lower triangular matrix| as{z the product} of lower triangular matrices (cf. Exercise 4.5). In fact, due to the special structure of the matrices L(k), it is given by 1 . l .. L = 21 . . .. .. ln ··· ln,n 1 1 −1 as the following exercise shows: Ubung¨ 4.11 For each k the product L(k)L(k+1) ··· L(n−1) is given by 1 .. 0 . . . .. .. . . 0 1 L(k)L(k+1) ··· L(n−1) = . . . lk ,k 1 +1 . . l .. k+2,k+1 . . .. .. 0 ··· 0 l l ··· l 1 n,k n,k+1 n,n−1 40 Thus, we have shown that Gaussian elimination produces a factorization A = LU, (4.8) where L and U are lower and upper triangular matrices determined by Alg. 4.7. 4.3 LU-factorization In numerical practice, linear systems are solved by computing the factors L and U in (4.8) and the system is then solved with one forward and one back substitution: 1. compute L, U such that A = LU 2. solve Ly = b using forward substitution 3. solve Ux = y using back substitution Bemerkung 4.12 In matlab, the LU-factorization is realized by lu(A). In python one can use scipy.linalg.lu. As we have seen in Section 4.2.1, the LU-factorization can be computed with Gaussian elimi- nation. An alternative way of computing the factors L, U is given in the following section. 4.3.1 Crout’s algorithm for computing LU-factorization We seek L, U such that 1 u11 ··· ··· u1n a11 ··· ··· a1n .. .. l21 . . ! . = . . .. .. .. . ln ··· ln,n 1 unn an ··· ··· ann 1 −1 1 2 2 This represents n equations for n unknowns, i.e., we are looking for lij,uij, such that n ! aik = lij ujk, ∀i, k =1, . , n. j=1 X L is lower triangular, U is upper triangular =⇒ min(i,k) ! aik = lij ujk ∀i, k =1,...,n (4.9) j=1 X Idea: Traverse the n2 equations in (4.9) in following order: (“Crout ordering”)]] (1, 1) , (1, 2) ,..., (1, n) (2, 1) , (3, 1) ,..., (n, 1) (2, 2) , (2, 3) ,..., (2, n) (3, 2) , (4, 2) ,..., (n, 2) etc. 41 Procedure: 1. step: i =1, k =1,...,n in (4.9): ! l11 u1k = a1k =1 ⇒ U(1, :) can be computed |{z} 2. step: k =1, i =2,...,n in (4.9): ! li1u11 = ai1 ⇒ L([2 : n], 1) can be determined 3. step: i =2, k =2,...,n in (4.9): ! l21 u1k + l22 u2k = a2k for k =2,...,n is known is known =1 by 2. step by 1. step ⇒ can compute|{z}U(2, [2|{z} : n]) |{z} 4. step: k =2, i =3,...,n in (4.9): ! li1 u12 +li2 u22 = ai2 for i =3,...,n known by known by known by 2. step 1. step 3. step ⇒ can compute|{z}L([3|{z} : n], 2) |{z} . The procedure is formalized in the following Algorithmus 4.13 (Crout’s LU-factorization) Input: invertible matrix A ∈ Rn×n that has an LU-factorization Output: the non-trivial entries of the normalized LU-factorization for i = 1 : n do for k = i : n do i−1 uik := aik − lijujk j=1 end for P for k = i + 1 : n do i−1 lki := aki − lkjuji uii j=1 ! P . 42 end for end for Bemerkung 4.14 (cost when solving (4.1) with LU-factorization) • The LU-factorization dominates with O(n3) (more precisely: 2/3n3 + O(n2) floating point operations) the total cost, since the cost of back substitution and forward substitution are O(n2) • An advantage of an LU-factorization arises, when problems with multiple right-hand sides are considered: solving Ax = b for M right-hand sides b, requires only a single LU- 2 3 2 factorization, i.e., the cost are 3 n +2Mn In practice A is overwritten by its LU-decomposition: Algorithmus 4.15 (LU-factorization with overwriting A) Input: A, invertible, A has a LU-factorization Output: algorithm replaces aij with uij for j ≥ i and with lij for j < i for i = 1 : n do for k = i : n do i−1 aik := aik − aij ajk j=1 end for P for k =(i + 1) : n do i−1 aki := aki − akj aji aii j=1 ! end for P . end for 4.3.2 banded matrices n×n A matrix A ∈ R is a banded matrix with upper bandwidth q and lower bandwidth p if aik =0 for all i, k with i>k + p or k > i + q. The following theorem shows that banded matrices are of interest if p and q are small (compared to n): Satz 4.16 Let A ∈ Rn×n be a banded matrix with upper bandwidth q and lower bandwidth p. Let A be invertible and admit an LU-factorization. Then: (i) L has lower bandwidth p and U has upper bandwidth q. (ii) Cost to solve Ax = b: 43 a11 ··· a1,q+1 . .. .. . .. .. ap+1,1 . . .. .. .. . .. .. a n−q,n . .. .. a ··· a n,n−p nn Abbildung 4.3: banded matrix with upper bandwidth q and lower bandwidth p. (a) O(npq) floating point operations (flops) to determine LU-factorization (b) O(np) flops to solve Ly = b (c) O(nq) flops to solve Ux = y Beweis: (Exercise) Prove (i) for the special case of a tridiagonal matrix, i.e., p = q = 1. To that end, proceed by induction on the matrix size n: • n =1 X • for the induction step n → n+1 make the ansatz 0 . . . ! Ln 0 Un u A = An . = 0 l⊤ 1 0 ρ an,n +1 0 ··· ··· 0 a a n+1,n n+1,n+1 ⊤ and compute l , u, and ρ. Use the structure of Ln, Un given by the induction hypothesis. ✷ Folie 24 4.3.3 Cholesky-factorization A particularly important class of matrices A is that of symmetric positive definite (SPD) matrices: • A is symmetric, i.e., Aij = Aji for all i, j • A is positive definite, i.e., x⊤Ax > 0 for all x =6 0. Bemerkung 4.17 An alternative criterion for positive definiteness of a symmetric matrix is that all its eigenvalue are positive. 44 For SPD matrices, one typically employs a variant of the LU-factorization, namely, the Cholesky- factorization, i.e., A = CC⊤, (4.10) where the Cholesky factor C is lower triangular (but not normalized, i.e., the entries Cii are not necessarily 1). Ubung¨ 4.18 Formulate an algorithm to compute C. Hint: Proceed as in Crout’s method for the LU-factorization. Bemerkung 4.19 If an SPD matrix A is banded with bandwidth p = q, then the Cholesky factor C is also banded with the same bandwidth. Bemerkung 4.20 The cost of a Cholesky factorization (of either a full matrix or a banded matrix) is about half of that of the corresponding LU-factorization since only half the entries need to be computed. Bemerkung 4.21 A Cholesky factorization is computed in matlab with chol. Folie 25 4.3.4 skyline matrices Folie 26 Banded matrices are a particular case of sparse matrices, i.e., matrices with “few” non-zero entries. We note that the LU-factors have the same sparsity pattern, i.e., the zeros of A outside the band are inherited by the factors L, U. Another important special case of sparse matrices are so-called skyline matrices as depicted on the left side of Fig. 4.4. More formally, a matrix A ∈ Rn×n is called a skyline matrix, if for i =1,...,n there are numbers pi, qi ∈ N0 such that aij = 0 if j < i − pi or i < j − qj. (4.11) We have without proof: n×n Satz 4.22 Let A ∈ R be a skyline matrix, i.e., there are pi, qi with (4.11). Let A have an LU-factorization A = LU. Then the matrices L, U satisfy: lij =0 for j < i − pi, uij =0 for i < j − qj. Theorem 4.22 states that the factors L and U have the same sparsity pattern as A. Figure 4.5 illustrates this for a simple example. Obviously, this can be exploited algorithmically to econo- mize on memory requirement and computing time by simply computing the non-zero entries of L and U. Note that the matrices in Fig. 4.4 should not be treated as banded matrices as then the bands p, q would be n. The right example in Fig. 4.4 is not a skyline matrix, and the sparsity pattern of A is lost in the course of the LU-factorization: L ist in general a fully populated lower triangular matrix and U a fully populated upper triangular matrix.