The LU-Decomposition
1 The Basic LU-Decomposition
Gaussian elimination with pivoting is definitely the most efficient and numerically accurate way to solve linear systems of equations A x = b. However if, as is often the case in applications, it is required to solve several systems with the same matrix A but different right-hand sides, it is useful to avoid repetition of the algorithm. This was the motivation for Turing’s development of the LU-decomposition which factors a matrix A into a product of a lower triangular matrix L and an upper triangular matrix U, effectively summarizing the elimination procedure applied to the matrix A. For what is to come, we will assume that the matrix is non-singular and so, for arbitrary right hand side b, the system Ax = b has a unique solution. Once we have this LU-factorization, it is easy to solve the equation Ax = b by first solving the system L c = b by a forward substitution, followed by the solution of U x = c.
Example 1.1 For the matrix
1 −2 3 A = 2 −5 12 0 2 −10 we will show later that the appropriate factorization is
1 −2 3 1 0 0 1 −2 3 2 −5 12 = 2 1 0 0 −1 6 . 0 2 −10 0 −2 1 0 0 2 if we take b = (1, 0, 1)>, then the solution of A x = b is computed, first by solving
1 0 0 c1 1 2 1 0 c2 = 0 0 −2 1 c3 1 which is equivalent to the system
c1 = 1
2 c1 + c2 = 0
−2 c2 + c3 = 1 . from which we deduce, by forward substitution, that c1 = 1, c2 = −2 and c3 = −3. Now the problem is to solve, by back substitution
1 1 −2 3 x1 1 0 −1 6 x2 = −2 0 0 2 x3 −3 which is equivalent to the system
2 x3 = −3
−x2 + 6 x3 = −2
x1 − 2x2 + 3x3 = 1 . yielding x1 = −17/2, x2 = −14/2, and x3 = −3/2. We leave it to the reader to check that this is indeed the required solution.
This is an efficient way to solve the linear system, and is particularly useful if, as mentioned, when we have many different right hand sides but the same matrix A. If we think of how the elimination method proceeds, we are moving to the right through the columns of the matrix while producing zeros below each of the non-zero pivots. Those zero entries are never again touched as the process continues. From the point of view of storage, they are “wasted space”. So we can minimize storage and use those positions to record the steps we take to produce the zeros. These, of course, are recorded in the elementary matrices that reflect the elementary row operations. −1 −1 The end result of the process is a factorization of the matrix A = (Ek ··· E1 )U where U is upper triangular while the product of the inverses will be lower triangular with diagonal entries all 1. Let us give a simple example to show what it is we are suggesting.
Example 1.2 Consider the matrix 1 −2 3 A = 2 −5 12 0 2 −10 Using the (1, 1) position as pivot, we start by adding −2 times row 1 to row 2. Here is the result and the inverse of the elementary matrix which accomplishes the operation:
1 −2 3 1 0 0 −1 A1 = 0 −1 6 , E1 = 2 1 0 . 0 2 −10 0 0 1
Since there is a zero in the (3, 1) position, we now go to the (2, 2) entry as pivot and add twice row 2 to row 3 which yields the pair
2 1 −2 3 1 0 0 −1 A2 = 0 −1 6 , E2 = 0 1 0 . 0 0 2 0 −2 1
Now multiply the inverses of the elementary matrices
1 0 0 1 0 0 1 0 0 −1 −1 E1 E2 = 2 1 0 0 1 0 = 2 1 0 = L . 0 0 1 0 −2 1 0 −2 1
This latter matrix is a lower triangular matrix with ones on the diagonal which, ac- cording to our computation can be multiplies by the upper triangular matrix U which is the matrix we wrote as A2. Indeed, if we take the product of this U with L we recover A
1 0 0 1 0 0 1 −2 3 LU = 2 1 0 0 1 0 0 −1 6 0 0 1 0 −2 1 0 0 2
1 0 0 1 −2 3 1 −2 3 = 2 1 0 0 −1 6 = 2 −5 12 0 −2 1 0 0 2 0 2 −10
= A
So we see that we can factor the matrix by keeping track of the process of Gaussian elimination, placing the results in the lower triangular portion of the matrix. Those are the entries that we have boxed. Note also that the diagonal elements of U are just the pivots at each step.
Let us do one more example.
Example 1.3 We start with the matrix
2 1 1 A = 4 1 0 . −2 2 1
3 Here we need two subtractions to remove the bottom two entries of the first column. We first add -2 times row 1 to row 2, and then add 1 times row 1 to row 3. Again, we box what will be the entries of the lower triangular matrix. The matrix is then
2 1 1 2 −1 −2 −1 3 2 The final step is to remove the entry 3 in the (2, 3) position by adding 3 times row 2 to row 3. The final matrix is then 2 1 1 2 −1 −2 −1 −3 −4 From this form we can read off the factorization 1 0 0 2 1 1 L = 2 1 0 , and U = 0 −1 −2 −1 −3 1 0 0 −4
2 The LDU-Decomposition and Pivots
There is a certain asymmetry in the LU-decomposition as we have written it, namely the lower triangular matrix has unit diagonal entries while the upper triangular matrix does not. A simple thing to do, if desired, is to factor out the diagonal entries of the U matrix:
d1 0 0 ··· 0 1 u12/d1 u13/d1 ··· u1n/d1 0 d2 0 ··· 0 0 1 u23/d2 ··· u2n/d2 U = D U˜ = . ...... ...... . . . . . . . . . . 0 0 0 ··· 0 0 0 0 ··· 1
Then LU = LD U˜ . When this factorization is used, it is once again written A = LDU which can be somewhat confusing. In our last example, this modified factoriza- tion is now
1 0 0 2 0 0 1 1/2 −1/2 LDU = 2 1 0 0 −1 0 0 1 2 . −1 −3 1 0 0 −4 0 0 1
4 In fact, there can be at most only one LDU factorization of a matrix A.
Theorem 2.1 If A = L1 D1 U 1 and A = L2D2U 2 where the L’s are lower triangu- lar with unit diagonal entries, the D’s are diagonal matrices with non-zero diagonal entries, and the U’s are upper triangular matrices with unit diagonal entries, then
L1 = L2, D1 = D2 and U 1 = U 2.
Proof: The proof is simple once we recognize that the upper and lower triangular matrices have inverses which are, likewise, upper and lower triangular respectively both with unit diagonals. Hence
−1 −1 −1 −1 −1 −1 D1 L1 (L1 D1 U 1) U 2 = D1 L1 (L2 D2 U 2) U 2 , reduces to −1 −1 −1 U 1 U 2 = D1 L1 L2 D2 . So the left-hand side is the product of two upper triangular matrices with unit diagonals while the right hand side is the product of two lower triangular matrices with unit diagonals. Hence both sides reduce to the identity matrix. It follows that U 1 = U 2.
−1 −1 −1 Similarly, multiplying on the left by L2 and on the right by U 1 D1 leads to the −1 conclusion that the lower triangular matrix L2 L1 = I and so L2 = L1. Finally, with these identifications the equation L1 D1 U 1 = L2D2U 2 is L1 D1 U 1 = L1D2U 1 and so D1 = D2
In the examples above, we have been very careful to choose matrices in which we do not encounter zero pivots in the elimination process. There are two possibilities. If, say after the first column has been cleared we find that the matrix is
1 1 1 S = 0 0 3 0 0 7 then the matrix is singular, there are multiple solutions of Ax = 0, and there is no LU-decomposition. We have ruled that out at the beginning. On the other hand, if the result of the first two elimination steps is
1 1 1 1 0 0 L1A = 0 0 3 , where L1 = −2 1 0 , 0 2 4 −4 0 1
5 then it is clear what to do, we use an elementary matrix (a permutation matrix), in 1 0 0 this case P = 0 0 1 so that 0 1 0
1 1 1 PL1A = 0 2 4 0 0 3 and the elimination can proceed. Actually, in this case we are done! Here we can write PA = LU where 1 0 0 L = −4 1 0 . −2 0 1
In fact, while we will not prove the result, it is true in general that for a non-singular matrix A there is a permutation matrix P , and lower and upper triangular matrices L and U such that PA = LU. Of course, if no row exchanges are needed, the matrix P = I.
At this point, it is natural to ask if there is some criterion that can be used to rule out, ahead of time, the necessity of row exchanges and the non-singularity of the matrix. so that we know that A has an LU-decomposition. If we think carefully about how the elimination procedure unfolds, then it is possible to find such a condition. Suppose that we look at one of the 3 × 3 examples we have used above, say
2 1 1 1 0 0 2 0 0 1 1/2 1/2 A = 4 1 0 = 2 1 0 0 −1 0 0 1 2 . −2 2 1 −1 −3 1 0 0 −4 0 0 1
Look at the upper left hand 2 × 2 matrices. We see that
1 0 2 0 1 1/2 2 1 0 −1 0 1 is a product of the form LDU. Carrying out the multiplication shows that it is exactly an LDU decomposition of the 2 × 2 matrix in the upper left hand corner of A. It is a general fact that, if A = LDU then the principal minor, those k × k matrices in the upper left hand corner of A all have the factorization Ak = LkDkU k. To see this, one can partition the matrices in the LDU factorization as
6 ! ! ! ! ! L 0 D 0 U F L D 0 U F k k k = k k k B C 0 E 0 G BDk CE 0 G ! L D U L D F = k k k k k BDkU k BDkF + CEG
Clearly the upper left hand corners of A, that is, Ak, coincides with LkDkU k. From this fact, we can use determinants to determine the pivots dk which we want to be all non-zero. Since all the diagonal elements of Lk and U k are ones, and the determinant of a triangular matrix is just the product of its diagonal elements, we have
det (Ak) = det (Lk) det (Dk) det (U k) = det (Dk) = d1d2 ··· dk , k = 1, . . . , n , and so det (Ak) dk = , det (Ak−1) where we define det (Ao) = 1. We conclude that the process of LU-factorization, or, likewise, Gaussian elimination, can be carried out without exchanges of rows provided the leading minors, A1, A2,..., An−1, A are all non-singular. We will see, later that certain symmetric matrices, the ones that are positive or negative definite, will satisfy this criterion.
7