Section 3.2: The LU Decomposition Jim Lambers January 25, 2021 • We have learned how to solve a system of linear equations Ax = b by applying Gaussian Elimination to the augmented matrix A~ = A b , and then performing back substitution on the resulting upper triangular matrix (row echelon form). This is kinda the \MAT 326" (intro linear algebra course) approach to solving systems of linear equations. • However, this approach is not practical if the right-hand side b of the system is changed, while A is not. • This is due to the fact that the choice of b has no effect on the row operations needed to reduce A to upper triangular form. • Therefore, it is desirable to instead apply these row operations to A only once, and then \store" them in some way in order to apply them to any number of right-hand sides. • This will be helpful when using implicit time-stepping methods for solving systems of ordinary differential equations. Derivation of the LU Decomposition To figure out how to perform Gaussian Elimination in such a way as to conveniently store row operations, we examine the process from a different perspective{matrix multiplication, rather than row operations. Elementary row Matrices To accomplish this, we first note that subtracting mij times row j from row i to eliminate aij is equivalent to multiplying A on the left by the matrix 2 1 0 ··············· 0 3 6 0 1 0 0 7 6 7 6 . 7 6 . .. .. .. 7 6 7 6 . .. .. .. 7 6 . 7 M = 6 7 ; ij 6 . .. .. .. 7 6 . −mij . 7 6 7 6 . .. .. .. 7 6 . 7 6 7 6 . 7 4 . 0 1 0 5 0 ··············· 0 1 1 where the entry −mij is in row i, column j. Each such matrix Mij is an example of an elementary row matrix, which is a matrix that results from applying a elementary row operation to the identity matrix I. Example. The matrix 2 1 2 1 −1 3 6 3 2 4 4 7 A = 6 7 4 4 4 3 4 5 2 0 1 5 can be reduced to the upper triangular matrix 2 1 2 1 −1 3 6 0 −4 1 7 7 U = 6 7 4 0 0 −2 1 5 0 0 0 −1 by performing the following row operations, or, equivalently, multiplying by the following elemen- tary row matrices. • M21: Subtracting three times the first row from the second • M31: Subtracting four times the first row from the third • M41: Subtracting two times the first row from the fourth • M32: Subtracting the second row from the third • M42: Subtracting the second row from the fourth • M43: Subtracting the third row from the fourth The elementary row matrices themselves are as follows. 2 1 0 0 0 3 2 1 0 0 0 3 6 −3 1 0 0 7 6 0 1 0 0 7 M21 = 6 7 ;M31 = 6 7 ; 4 0 0 1 0 5 4 −4 0 1 0 5 0 0 0 1 0 0 0 1 2 1 0 0 0 3 2 1 0 0 0 3 6 0 1 0 0 7 6 0 1 0 0 7 M41 = 6 7 ;M32 = 6 7 ; 4 0 0 1 0 5 4 0 −1 1 0 5 −2 0 0 1 0 0 0 1 2 1 0 0 0 3 2 1 0 0 0 3 6 0 1 0 0 7 6 0 1 0 0 7 M42 = 6 7 ;M43 = 6 7 : 4 0 0 1 0 5 4 0 0 1 0 5 0 −1 0 1 0 0 −1 1 Let b = 5 1 22 15 T . Applying the same row operations to b yields the modified right-hand side 2 5 3 6 1 7 y = 6 7 : 4 1 5 3 We then use back substitution to solve the system Ux = y. 2 2 In general, if we let A(1) = A and let A(k+1) be the matrix obtained by eliminating entries of column k in A(k), then we have, for k = 1; 2; : : : ; n − 1, A(k+1) = M (k)A(k) where 2 1 0 ·················· 0 3 6 0 1 0 0 7 6 7 6 . 7 6 . .. .. .. 7 6 7 6 . 7 6 . 0 .. .. 7 6 7 (k) 6 . .. .. 7 M = 6 . −mk+1;k . 7 ; (1) 6 7 6 . .. .. 7 6 . 0 . 7 6 7 6 . .. .. .. 7 6 . 7 6 7 6 . .. 7 4 . 1 0 5 0 ··· 0 −mnk 0 ······ 0 1 with the entries −mk+1;k;:::; −mnk occupying column k. It follows that the matrix U = A(n) = M (n−1)A(n−1) = M (n−1)M (n−2) ··· M (1)A (2) is upper triangular, and the vector y = M (n−1)M (n−2) ··· M (1)b; being the result of applying the same row operations to b, is the right-hand side for the upper triangular system that is to be solved by back substitution. Unit Lower Triangular Matrices • A matrix L is lower triangular if all of its entries above the main diagonal, that is, entries `ij for which i < j, are equal to zero. • A system of equations of the form Ly = b, where L is an n × n nonsingular lower trian- gular matrix, can be solved using a process similar to back substitution, called forward substitution. • As with upper triangular matrices, a lower triangular matrix is nonsingular (invertible) if and only if all of its diagonal entries are nonzero. We note that each matrix M (k) from (1), for k = 1; 2; : : : ; n − 1, is not only a lower triangular matrix, but a unit lower triangular matrix, because all of its diagonal entries are equal to 1. Next, we note two important properties of unit lower/upper triangular matrices: • The product of two unit lower/upper triangular matrices is unit lower/upper triangular. • A unit lower/upper triangular matrix is nonsingular, and its inverse is unit lower/upper triangular. 3 In fact, the inverse of each M (k) is easily computed. We have 2 1 0 ·················· 0 3 6 0 1 0 0 7 6 7 6 . 7 6 . .. .. .. 7 6 7 6 . 7 6 . 0 .. .. 7 6 7 (k) (k) −1 6 . .. .. 7 L = [M ] = 6 . mk+1;k . 7 : (3) 6 7 6 . .. .. 7 6 . 0 . 7 6 7 6 . .. .. .. 7 6 . 7 6 7 6 . .. 7 4 . 1 0 5 0 ··· 0 mnk 0 ······ 0 1 It follows that if we define M = M (n−1) ··· M (1), then M is unit lower triangular, and MA = U, where U is upper triangular. We then have A = M −1U = LU, where L = L(1) ··· L(n−1) = [M (1)]−1 ··· [M (n−1)]−1 is also unit lower triangular. Rearranging equation (2): [M (1)]−1 ··· [M (n−1)]−1U = [M (1)]−1 ··· [M (n−1)]−1M (n−1)M (n−2) ··· M (1)A = A Simplifying: L(1) ··· L(n−1)U = LU = A Furthermore, from the structure of each matrix L(k), it can readily be determined that 2 1 0 ······ 0 3 6 . 7 6 m21 1 0 . 7 6 7 L = 6 . .. .. 7 : (4) 6 . m32 . 7 6 7 6 . .. 7 4 . 1 0 5 mn1 mn2 ··· mn;n−1 1 That is, L stores all of the multipliers used during Gaussian Elimination. The factorization of A that we have obtained, A = LU; is called the LU Decomposition, or LU Factorization, of A. Solution of Ax = b Once the LU Decomposition A = LU has been computed, we can solve the system Ax = b by first noting that if x is the solution, then Ax = LUx = b: Therefore, we can obtain x by first solving the system Ly = b; 4 and then solving Ux = y: Then, if b should change, only these two triangular systems need to be solved in order to obtain the new solution; the LU Decomposition does not need to be recomputed. The system Ux = y can be solved by back substitution, since U is upper triangular. To solve Ly = b, we can use forward substitution, since L is unit lower triangular. Algorithm. (Forward Substitution) Given an n × n unit lower triangular matrix L and an n-vector b, the following algorithm computes an n-vector y such that Ly = b. for i = 1; 2; : : : ; n do yi = bi for j = 1; 2; : : : ; i − 1 do yi = yi − `ijyj end for end for • Like back substitution, this algorithm requires O(n2) floating-point operations. • Unlike back substitution, there is no division of the ith component of the solution by a diagonal entry of the matrix, but this is only because in this context, L is unit lower triangular, so `ii = 1. • When applying forward substitution to a general lower triangular matrix, such a division is required. Notes about efficient storage: • Because both forward and back substitution require only O(n2) operations, whereas Gaussian Elimination requires O(n3) operations, changes in the right-hand side b are best handled by computing the factors L and U once, and storing them, rather than performing Gaussian Elimination on the entire system Ax = b for each choice of b. • This storage of L and U can be accomplished quite efficiently, because L is a unit lower triangular matrix. • It follows from this that L and U can be stored in a single n × n matrix by storing U in the upper triangular part, and the multipliers mij in the strictly lower triangular part (that is, the part below the main diagonal).