Section 3.2: The LU Decomposition

Jim Lambers

January 25, 2021

• We have learned how to solve a system of linear equations Ax = b by applying Gaussian Elimination to the augmented matrix A˜ =  A b , and then performing back substitution on the resulting upper triangular matrix (row echelon form). This is kinda the “MAT 326” (intro linear algebra course) approach to solving systems of linear equations.

• However, this approach is not practical if the right-hand side b of the system is changed, while A is not.

• This is due to the fact that the choice of b has no effect on the row operations needed to reduce A to upper triangular form.

• Therefore, it is desirable to instead apply these row operations to A only once, and then “store” them in some way in order to apply them to any number of right-hand sides.

• This will be helpful when using implicit time-stepping methods for solving systems of ordinary differential equations.

Derivation of the LU Decomposition To figure out how to perform Gaussian Elimination in such a way as to conveniently store row operations, we examine the process from a different perspective–matrix multiplication, rather than row operations.

Elementary row Matrices

To accomplish this, we first note that subtracting mij times row j from row i to eliminate aij is equivalent to multiplying A on the left by the matrix

 1 0 ··············· 0   0 1 0 0     . . . . .   ......     ......   . . . . .  M =   , ij  ......   . −mij . . . .     ......   . . . . .     .   . 0 1 0  0 ··············· 0 1

1 where the entry −mij is in row i, column j. Each such matrix Mij is an example of an elementary row matrix, which is a matrix that results from applying a elementary row operation to the identity matrix I. Example. The matrix  1 2 1 −1   3 2 4 4  A =    4 4 3 4  2 0 1 5 can be reduced to the upper triangular matrix  1 2 1 −1   0 −4 1 7  U =    0 0 −2 1  0 0 0 −1 by performing the following row operations, or, equivalently, multiplying by the following elemen- tary row matrices.

• M21: Subtracting three times the first row from the second

• M31: Subtracting four times the first row from the third

• M41: Subtracting two times the first row from the fourth

• M32: Subtracting the second row from the third

• M42: Subtracting the second row from the fourth

• M43: Subtracting the third row from the fourth The elementary row matrices themselves are as follows.  1 0 0 0   1 0 0 0   −3 1 0 0   0 1 0 0  M21 =   ,M31 =   ,  0 0 1 0   −4 0 1 0  0 0 0 1 0 0 0 1  1 0 0 0   1 0 0 0   0 1 0 0   0 1 0 0  M41 =   ,M32 =   ,  0 0 1 0   0 −1 1 0  −2 0 0 1 0 0 0 1  1 0 0 0   1 0 0 0   0 1 0 0   0 1 0 0  M42 =   ,M43 =   .  0 0 1 0   0 0 1 0  0 −1 0 1 0 0 −1 1 Let b =  5 1 22 15 T . Applying the same row operations to b yields the modified right-hand side  5   1  y =   .  1  3 We then use back substitution to solve the system Ux = y. 2

2 In general, if we let A(1) = A and let A(k+1) be the matrix obtained by eliminating entries of column k in A(k), then we have, for k = 1, 2, . . . , n − 1,

A(k+1) = M (k)A(k) where  1 0 ·················· 0   0 1 0 0     . . . . .   ......     . . . .   . 0 .. .. .    (k)  ......  M =  . . −mk+1,k . . .  , (1)    ......   . . . 0 . . .     ......   ......     ......   . . . . . 1 0  0 ··· 0 −mnk 0 ······ 0 1 with the entries −mk+1,k,..., −mnk occupying column k. It follows that the matrix

U = A(n) = M (n−1)A(n−1) = M (n−1)M (n−2) ··· M (1)A (2) is upper triangular, and the vector

y = M (n−1)M (n−2) ··· M (1)b, being the result of applying the same row operations to b, is the right-hand side for the upper triangular system that is to be solved by back substitution.

Unit Lower Triangular Matrices

• A matrix L is lower triangular if all of its entries above the main diagonal, that is, entries `ij for which i < j, are equal to zero.

• A system of equations of the form Ly = b, where L is an n × n nonsingular lower trian- gular matrix, can be solved using a process similar to back substitution, called forward substitution.

• As with upper triangular matrices, a lower triangular matrix is nonsingular (invertible) if and only if all of its diagonal entries are nonzero.

We note that each matrix M (k) from (1), for k = 1, 2, . . . , n − 1, is not only a lower triangular matrix, but a unit lower triangular matrix, because all of its diagonal entries are equal to 1. Next, we note two important properties of unit lower/upper triangular matrices:

• The product of two unit lower/upper triangular matrices is unit lower/upper triangular.

• A unit lower/upper triangular matrix is nonsingular, and its inverse is unit lower/upper triangular.

3 In fact, the inverse of each M (k) is easily computed. We have

 1 0 ·················· 0   0 1 0 0     . . . . .   ......     . . . .   . 0 .. .. .    (k) (k) −1  ......  L = [M ] =  . . mk+1,k . . .  . (3)    ......   . . . 0 . . .     ......   ......     ......   . . . . . 1 0  0 ··· 0 mnk 0 ······ 0 1

It follows that if we define M = M (n−1) ··· M (1), then M is unit lower triangular, and MA = U, where U is upper triangular. We then have A = M −1U = LU, where

L = L(1) ··· L(n−1) = [M (1)]−1 ··· [M (n−1)]−1 is also unit lower triangular. Rearranging equation (2):

[M (1)]−1 ··· [M (n−1)]−1U = [M (1)]−1 ··· [M (n−1)]−1M (n−1)M (n−2) ··· M (1)A = A

Simplifying: L(1) ··· L(n−1)U = LU = A Furthermore, from the structure of each matrix L(k), it can readily be determined that

 1 0 ······ 0   .   m21 1 0 .    L =  ......  . (4)  . m32 . . .     . . ..   . . . 1 0  mn1 mn2 ··· mn,n−1 1

That is, L stores all of the multipliers used during Gaussian Elimination. The factorization of A that we have obtained, A = LU, is called the LU Decomposition, or LU Factorization, of A.

Solution of Ax = b Once the LU Decomposition A = LU has been computed, we can solve the system Ax = b by first noting that if x is the solution, then Ax = LUx = b. Therefore, we can obtain x by first solving the system

Ly = b,

4 and then solving Ux = y. Then, if b should change, only these two triangular systems need to be solved in order to obtain the new solution; the LU Decomposition does not need to be recomputed. The system Ux = y can be solved by back substitution, since U is upper triangular. To solve Ly = b, we can use forward substitution, since L is unit lower triangular.

Algorithm. (Forward Substitution) Given an n × n unit lower triangular matrix L and an n-vector b, the following algorithm computes an n-vector y such that Ly = b.

for i = 1, 2, . . . , n do yi = bi for j = 1, 2, . . . , i − 1 do yi = yi − `ijyj end for end for

• Like back substitution, this algorithm requires O(n2) floating-point operations.

• Unlike back substitution, there is no division of the ith component of the solution by a diagonal entry of the matrix, but this is only because in this context, L is unit lower triangular, so `ii = 1.

• When applying forward substitution to a general lower triangular matrix, such a division is required.

Notes about efficient storage:

• Because both forward and back substitution require only O(n2) operations, whereas Gaussian Elimination requires O(n3) operations, changes in the right-hand side b are best handled by computing the factors L and U once, and storing them, rather than performing Gaussian Elimination on the entire system Ax = b for each choice of b.

• This storage of L and U can be accomplished quite efficiently, because L is a unit lower triangular matrix.

• It follows from this that L and U can be stored in a single n × n matrix by storing U in the upper triangular part, and the multipliers mij in the strictly lower triangular part (that is, the part below the main diagonal).

• Once the LU Decomposition has been computed, the appropriate entries of this matrix, that contains the “interesting” parts of both L and U, can be accessed by the algorithms for forward and back substitution when solving Ly = b and Ux = y, respectively.

Existence and Uniqueness • Not every nonsingular (invertible) n × n matrix A has an LU Decomposition.

• For example, if a11 = 0, then the multipliers mi1 = ai1/a11, for i = 2, 3, . . . , n, are not defined, so no multiple of the first row can be subtracted from the other rows to eliminate subdiagonal entries in the first column.

5 (j) • Even if a11 6= 0, it can happen that the (j, j) entry of A is zero, in which case a similar breakdown occurs.

• When this is the case, the LU Decomposition of A does not exist. This will be addressed by pivoting, resulting in a modification of the LU Decomposition.

It can be shown that the LU Decomposition of an n × n matrix A does exist if and only if the leading principal submatrices of A, defined by   a11 ··· a1k  . .. .  [A]1:k,1:k =  . . .  , k = 1, 2, . . . , n, ak1 ··· akk are all nonsingular. Furthermore, when the LU Decomposition exists, it is unique. Proof of uniqueness: Let A = L1U1 = L2U2, L1,L2 unit lower triangular, U1,U2 upper trian- gular. Then −1 −1 −1 −1 L2 L1U1U1 = L2 L2U2U1 or −1 −1 L2 L1 = U2U1 . Left side is unit lower triangular, right side is upper triangular. Therefore both sides are diagonal. −1 But the diagonal must be all 1’s, so both sides must equal I. Therefore from L2 L1 = I, we have L1 = L2, and similarly U1 = U2. This proves uniqueness!

Pivoting • During Gaussian Elimination, it is necessary to interchange rows of the augmented matrix whenever the diagonal entry of the column currently being processed, known as the pivot element, is equal to zero.

• However, if we examine the main step in Gaussian Elimination,

(j+1) (j) (j) aik = aik − mijajk , (5)

(j) we can see that any roundoff error in the computation of ajk is amplified by mij. • Because the multipliers can be arbitrarily large, it follows that the error in the computed solution can be arbitrarily large, meaning that Gaussian Elimination is an unstable algo- rithm.

• Therefore, it is helpful if it can be ensured that the multipliers are small. This can be accomplished by performing row interchanges, or pivoting, even when it is not absolutely necessary to do so for elimination to proceed.

6 Partial Pivoting

• In partial pivoting, when eliminating entries in column j, we seek the largest entry in column j, on or below the main diagonal, and then interchange that entry’s row with row j.

• That is, we find an integer p, j ≤ p ≤ n, such that

|a(j)| = max |a(j)|. pj j≤i≤n ij

Then, we interchange rows p and j.

(j) (j) • In view of the definition of the multiplier, mij = aij /ajj , it follows that |mij| ≤ 1 for j = 1, . . . , n − 1 and i = j + 1, . . . , n.

• Furthermore, while pivoting in this manner requires O(n2) comparisons to determine the appropriate row interchanges, that extra expense is negligible compared to the overall cost of Gaussian Elimination, which is O(n3).

• We note that when partial pivoting is used, even though the entries of L satisfy |`ij| ≤ 1, the entries of U can still grow substantially.

Complete Pivoting

• While partial pivoting helps to control the propagation of roundoff error, loss of significant (j) digits can still result if, in the main step of Gaussian Elimination (5), mijajk is much larger (j) in magnitude than aik . • Complete pivoting entails finding integers p and q such that

|apq| = max |aik|, j≤i≤n,j≤k≤n

and then using both row and column interchanges to move apq into the pivot position in row j and column j.

• It has been proven that this is an effective strategy for ensuring that Gaussian Elimination is a stable algorithm, meaning that the computed solution is the exact solution of a nearby system.

• Unfortunately, complete pivoting requires O(n3) comparisons, which is why partial pivoting is generally still preferred.

The LU Decomposition with Partial Pivoting

• If row j is interchanged with row p, for p > j, before entries in column j are eliminated, the matrix A(j) is effectively multiplied on the left by a permutation matrix P (j).

• A permutation matrix is a matrix obtained by permuting the rows (or columns) of the identity matrix I.

• In P (j), rows j and p of I are interchanged, so that multiplying A(j) on the left by P (j) interchanges these rows of A(j).

7 • It follows that the process of Gaussian Elimination with partial pivoting can be described in terms of matrix multiplications, just as it can when no pivoting is performed.

• Let i1, i2, . . . , in be a permutation of the indices 1, 2, . . . , n. Then, let the permutation matrix

P be defined by pik,k = 1 for k = 1, 2, . . . , n, with all other entries equal to zero. • We then have P T P = I. That is, P is an orthogonal matrix.

n×n T • In general, any matrix Q ∈ R is said to be orthogonal if Q Q = I.   T • The term comes from the fact that if Q = q1 ··· qn , then qi qj = 0 for i 6= j. That is, the columns of Q are orthogonal to one another.

T • Furthermore, from qi qi = 1, we see that the columns actually form an orthonormal set.

Example. Let  1 4 7  A =  2 8 5  . 3 6 9 Applying Gaussian Elimination to A, we subtract twice the first row from the second, and three times the first row from the third, to obtain

 1 4 7  (2) A =  0 0 −9  . 0 −6 −12

At this point, Gaussian Elimination breaks down, because the multiplier m32 = a32/a22 = −6/0 is undefined. Therefore, we must interchange the second and third rows, which yields the upper triangular matrix  1 4 7  (3) (2) (2) U = A = P A =  0 −6 −12  , 0 0 −9 where P (2) is the permutation matrix

 1 0 0  (2) P =  0 0 1  0 1 0 obtained by interchanging the second and third rows of the identity matrix. It follows that we have computed the factorization

PA = LU, or  1 0 0   1 4 7   1 0 0   1 4 7   0 0 1   2 8 5  =  3 1 0   0 −6 −12  . 0 1 0 3 6 9 2 0 1 0 0 −9 2

8 In the preceding example, Gaussian Elimination produces

P (2)M (1)A = U, where  1 0 0  (1) M =  −2 1 0  . −3 0 1 • This equation does not lead directly to a factorization of the form PA = LU, because the matrices involved are not in an order that is conducive to such a factorization.

• Ideally, we would like to apply all of the row interchanges to A “up front”, so that the resulting matrix PA has an LU Decomposition. To achieve such a factorization, we use the fact that a permutation matrix is orthogonal to obtain P (2)M (1)[P (2)]T P (2)A = U, which can be rewritten as  1 0 0  (1) (2) (1) (2) (1) (2) T M˜ P A = U, M˜ = P M [P ] =  −3 1 0  . −2 0 1

We then have PA = LU, where P = P (2) and L = [M˜ (1)]−1. Continuing this process, we obtain the decomposition PA = LU where

P = P (n−1)P (n−2) ··· P (1),L = [M˜ (1)]−1[M˜ (2)]−1 ··· [M˜ (n−1)]−1, where M˜ (j) = P (n−1) ··· P (j+1)M (j)[P (j+1)]T ··· [P (n−1)]T .

• Fortunately, it is easier to obtain L than it seems based on these manipulations.

• As Gaussian Elimination stores the multipliers mij in the lower triangle of A while reducing the rest of the matrix to U, it is sufficient to apply any row interchange to the entirety of the rows–including the portion containing multipliers.

• This is because M˜ (j) is obtained by applying permutations P (i) to M (j), for i > j–row interchanges that occur after the multipliers from column j are stored in M (j).

• Once the LU Decomposition PA = LU has been computed, we can solve the system Ax = b by first noting that if x is the solution, then

PAx = LUx = P b.

• Therefore, we can obtain x by first solving the system Ly = P b, and then solving Ux = y.

• Then, if b should change, only these two triangular systems need to be solved in order to obtain the solution.

• As in the case of Gaussian Elimination without pivoting, the LU Decomposition does not need to be recomputed.

9