STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2016 LECTURE 13 1. need for pivoting • last time we showed that under proper circumstances, we can write A = LU where 2 1 0 0 ··· 03 2u11 u12 ······ u1n3 6`21 1 0 ··· 07 6 0 u22 ······ u2n7 6 . .7 6 . 7 6 .. .7 6 .. 7 L = 6`31 `32 .7 ;U = 6 0 0 . 7 6 . 7 6 . 7 4 . .. .. 05 4 . .. 5 `n1 `n2 ··· `n;n−1 1 0 0 ··· 0 unn • what exactly are proper circumstances? (k) • we must have akk 6= 0, or we cannot proceed with the decomposition • for example, if 20 1 113 21 3 43 A = 43 7 2 5 or A = 42 6 45 2 9 3 7 1 2 Gaussian elimination will fail; note that both matrices are nonsingular • in the first case, it fails immediately; in the second case, it fails after the subdiagonal entries (k) in the first column are zeroed, and we find that a22 = 0 • in general, we must have det Aii 6= 0 for i = 1; : : : ; n where 2 3 a11 ··· a1i 6 . 7 Aii = 4 . 5 ai1 ··· aii for the LU factorization to exist • the existence of LU factorization (without pivoting) can be guaranteed by several conditions, 1 n×n one example is column diagonal dominance: if a nonsingular A 2 R satisfies n X jajjj ≥ jaijj; j = 1; : : : ; n; i=1 i6=j then one can guarantee that Gaussian elimination as described above produces A = LU with j`ijj ≤ 1 • there are necessary and sufficient conditions guaranteeing the existence of LU decomposition but those are difficult to check in practice and we do not state them here • how can we obtain the LU factorization for a general nonsingular matrix? • if A is nonsingular, then some element of the first column must be nonzero • if ai1 6= 0, then we can interchange row i with row 1 and proceed Date: November 12, 2016, version 1.1. Comments, bug reports: [email protected]. 1 Pn the usual type of diagonal dominance, i.e., jaiij ≥ j=1;j6=ijaij j, i = 1; : : : ; n, is called row diagonal dominance 1 • this is equivalent to multiplying A by a permutation matrix Π1 that interchanges row 1 and row i: 20 ······ 0 1 0 ··· 03 6 1 7 6 . 7 6 .. 7 6 7 6 7 6 1 7 Π1 = 6 7 61 0 ······ 0 ······ 07 6 7 6 1 7 6 . 7 4 .. 5 1 • thus M1Π1A = A2 (refer to earlier lecture notes for more information about permutation matrices) • then, since A2 is nonsingular, some element of column 2 of A2 below the diagonal must be nonzero • proceeding as before, we compute M2Π2A2 = A3, where Π2 is another permutation matrix • continuing, we obtain −1 A = (Mn−1Πn−1 ··· M1Π1) U • it can easily be shown that ΠA = LU where Π is a permutation matrix | easy but a bit of a pain because notation is cumbersome • so we will be informal but you'll get the idea • for example if after two steps we get (recall that permutation matrices or orthogonal ma- trices), −1 A = (M2Π2M1Π1) A2 T −1 T −1 = Π1M1 Π2M2 A2 T T −1 T −1 = Π1Π2(Π2M1 Π2)M2 A2 T = Π L1L2A2 then { Π = Π2Π1 is a permutation matrix −1 { L2 = M2 is a unit lower triangular matrix −1 T −1 { L1 = Π2M1 Π2 will always be a unit lower triangular matrix because M1 is of the form in 2 1 03 1 6`21 1 7 M −1 = = 6 7 (1.1) 1 ` I 6 . .. 7 4 . 0 . 5 `n1 1 whereas Π2 must be of the form 1 Π2 = Πb 2 for some (n − 1) × (n − 1) permutation matrix Πb 2 and so −1 T 1 0 Π2M1 Π2 = Πb 2` I −1 T in other words Π2M1 Π2 also has the form in (1.1) 2 • if we do one more steps we get −1 A = (M3Π3M2Π2M1Π1) A3 T −1 T −1 T −1 = Π1M1 Π2M2 Π3M3 A2 T T T −1 T T −1 T −1 = Π1Π2Π3(Π3Π2M1 Π2Π3)(Π3M2 Π3)M3 A2 T = Π L1L2L3A2 where { Π = Π3Π2Π1 is a permutation matrix −1 { L3 = M3 is a unit lower triangular matrix −1 T −1 { L2 = Π3M2 Π3 will always be a unit lower triangular matrix because M2 is of the form 21 3 2 3 60 1 7 1 6 7 −1 60 −`32 1 7 M2 = 4 1 5 = 6 7 (1.2) ` I 6. .. 7 4. 5 0 −`n2 1 whereas Π3 must be of the form 21 3 Π3 = 4 1 5 Πb 3 for some (n − 2) × (n − 2) permutation matrix Πb 3 and so 21 3 −1 T Π3M2 Π3 = 4 1 05 Πb 3` I −1 T in other words Π3M2 Π3 also has the form in (1.2) −1 T T { L1 = Π3Π2M1 Π2Π3 will always be a unit lower triangular matrix for the same reason above because Π3Π2 must have the form 21 3 1 1 Π3Π2 = 4 1 5 = Πb 2 Π32 Πb 3 for some (n − 1) × (n − 1) permutation matrix 1 Π32 = Πb 2 Πb 3 • more generally if we keep doing this, then T A = Π L1L2 ··· Ln−1An−1 where { Π = Πn−1Πn−2 ··· Π1 is a permutation matrix −1 { Ln−1 = Mn−1 is a unit lower triangular matrix −1 T T { Lk = Πn−1 ··· Πk+1Mk Πk+1 ··· Πn−1 is a unit lower triangular matrix for all k = 1; : : : ; n − 2 { An−1 = U is an upper triangular matrix { L = L1L2 ··· Ln−1 is a unit lower triangular matrix • this algorithm with the row permutations is called Gaussian elimination with partial pivoting or gepp for short; we will say more in the next section 3 2. pivoting strategies • the (k; k) entry at step k during Gaussian elimination is called the pivoting entry or just pivot for short (k) • in the preceding section, we said that if the pivoting entry is zero, i.e., akk = 0, then we (k) just need to find an entry below it in the same column, i.e., akj for some j > k, and then permute this entry into the pivoting position, before carrying on with the algorithm • but it is really better to choose the largest entry below the pivot, and not just any nonzero entry • that is, the permutation Πk is chosen so that row k is interchanged with row j, where (k) (k) akj = maxj=k;k+1;:::;njakj j • this guarantees that j`kjj ≤ 1 for all k and j • this strategy is known as partial pivoting, which is guaranteed to produce an LU factoriza- m×n tion if A 2 R has full row-rank, i.e., rank(A) = m ≤ n • another common strategy, complete pivoting, which uses both row and column interchanges (k) to ensure that at step k of the algorithm, the element akk is the largest element in absolute value from the entire submatrix obtained by deleting the first k − 1 rows and columns, i.e., (k) (k) aij = max jaij j i=k;k+1;:::;n j=k;k+1;:::;n • in this case we need both row and column permutation matrices, i.e., we get Π1AΠ2 = LU when we do complete pivoting • complete pivoting is necessary when rank(A) < minfm; ng • there are yet other pivoting strategies due to considerations such as preserving sparsity (if you're interested, look up minimum degree algorithm or Markowitz algorithm) or a tradeoff between partial and complete pivoting (e.g., rook pivoting) 3. uniqueness of the LU factorization • the LU decomposition of a nonsingular matrix, if it exists (i.e., without row or column permutations), is unique • if A has two LU decompositions, A = L1U1 and A = L2U2 −1 −1 • from L1U1 = L2U2 we obtain L2 L1 = U2U1 • the inverse of a unit lower triangular matrix is a unit lower triangular matrix, and the −1 product of two unit lower triangular matrices is a unit lower triangular matrix, so L2 L1 must be a unit lower triangular matrix −1 • similarly, U2U1 is an upper triangular matrix • the only matrix that is both upper triangular and unit lower triangular is the identity matrix I, so we must have L1 = L2 and U1 = U2 4. Gauss{Jordan elimination • a variant of Gaussian elimination is called Gauss{Jordan elimination • it entails zeroing elements above the diagonal as well as below, transforming an m × n matrix into reduced row echelon form, i.e., a form where all pivoting entries in U are 1 and all entries above the pivots are zeros • this is what you probably learnt in your undergraduate linear algebra class, e.g., 21 3 1 9 3 21 3 1 9 3 21 3 1 9 3 21 0 −2 −33 A = 41 1 −1 1 5 ! 40 −2 −2 −85 ! 40 −2 −2 −85 ! 40 1 1 4 5 = U 3 11 5 35 0 2 2 8 0 0 0 0 0 0 0 0 4 • the main drawback is that the elimination process can be numerically unstable, since the multipliers can be large • furthermore the way it is done in undergraduate linear algebra courses is that the elimination matrices (i.e., the L and Π) are not stored 5. condensed LU factorization • just like QR and svd, LU factorization with complete pivoting has a condensed form too m×n • let A 2 R and rank(A) = r ≤ minfm; ng, recall that gecp yields Π1AΠ2 = LU L 0 U U = 11 11 12 L21 Im−r 0 0 L11 = U11 U12 =: LeUe L21 r×r r×r where L11 2 R is unit lower triangular (thus nonsingular) and U11 2 R is also nonsingular m×r n×r • note that Le 2 R and Ue 2 R and so T T A = (Π1Le)(UeΠ2) is a rank-retaining factorization 6.