Stat 309: Mathematical Computations I Fall 2013 Lecture 13

STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2013 LECTURE 13

• we look at a few variants of lu factorization: condensed LU, LDU, LDLT, and Cholesky factorizations

1. condensed LU factorization • just like QR and SVD, LU factorization with complete pivoting has a condensed form too m×n • let A ∈ R and rank(A) = r ≤ min{m, n}, recall that gepp yields

Π1AΠ2 = LU L 0 U U = 11 11 12 L21 Im−r 0 0 L11 = U11 U12 =: LeUe L21 r×r r×r where L11 ∈ R is unit lower triangular (thus nonsingular) and U11 ∈ R is also nonsingular m×r n×r • note that Le ∈ R and Ue ∈ R and so T T A = (Π1 Le)(UeΠ2 ) is a rank-retaining factorization

2. LDU and LDLT factorizations n×n • if A ∈ R has nonsingular principal submatrices A1:k,1:k for k = 1, . . . , n, then there n×n n×n exists a unit lower triangular matrix L ∈ R , a unit upper triangular matrix U ∈ R , n×n and a diagonal matrix D = diag(d11, . . . , dnn) ∈ R such that       1 0 d11 1 u12 ··· u1n `21 1   d22   1 u2n       A = LDU =  . ..   ..   .. .   . .   .   . .  `n1 `n2 ··· 1 dnn 1 • this is called the LDU factorization of A • if A is furthermore symmetric, then L = U T and this called the LDLT factorization • if they exist, then both LDU and LDLT factorizations are unique (exercise) T • if a symmetric A has an LDL factorization and if dii > 0 for all i = 1, . . . , n, then A is positive definite • in fact, even though d11, . . . , dnn are not the eigenvalues of A (why not?), they must have the same signs as the eigenvalues of A, i.e., if A has p positive eigenvalues, q negative eigenvalues, and z zero eigenvalues, then there are exactly p, q, and z positive, negative, and zero entries in d11, . . . , dnn — a consequence of the Sylvester law of inertia • unfortunately, both LDU and LDLT factorizations are difficult to compute because – the condition on the principal submatrices is difficult to check in advance

Date: November 18, 2013, version 1.1. Comments, bug reports: [email protected]. 1 – algorithms for computing them are invariably unstable because size of multipliers can- not be bounded in terms of the entries of A • for example, the LDLT factorization of a 2 × 2 symmetric matrix is a c 1 0 a c 1 0 a 0 1 c/a = = c d c/a 1 0 d − (c/a)c c/a 1 0 d − (c/a)c 0 1 • so ε 1 1 0 ε 0 1 1/ε = 1 ε 1/ε 1 0 ε − 1/ε 0 1 the elements of L and D are arbitrarily large when |ε| is small • nonetheless there is one special case when LDLT factorization not only exists but can be computed in an eﬃcient and stable way — when A is positive deﬁnite

3. positive definite matrices n×n T • a symmetric matrix A ∈ R is positive definite if x Ax > 0 for all nonzero x • a symmetric positive definite matrix has real and positive eigenvalues, and its leading principal submatrices all have positive determinants • from the definition, it is easy to see that all diagonal elements are positive • to solve the system Ax = b where A is symmetric positive definite, we can compute the Cholesky factorization A = RTR where R is upper triangular • this factorization exists if and only if A is symmetric positive definite • in fact, attempting to compute the Cholesky factorization of A is an efficient method for checking whether A is symmetric positive definite • it is important to distinguish the Cholesky factorization from the square root factorization • a square root of a matrix A is defined as a matrix S such that

S2 = SS = A

• note that the matrix R in A = RTR is not the square root of A, since it does not hold that R2 = A unless A is a diagonal matrix • the square root of a symmetric positive deﬁnite A can be computed by using the fact that A has an eigendecomposition A = UΛU T where Λ is a diagonal matrix whose diagonal elements are the positive eigenvalues of A and U is an orthogonal matrix whose columns are the eigenvectors of A • it follows that A = UΛU T = (UΛ1/2U T)(UΛ1/2U T) = SS and so S = UΛ1/2U T is a square root of A

4. cholesky factorization • the Cholesky factorization can be computed directly from the matrix equation A = RTR where R is upper-triangular • while it is conventional to write Cholesky factorization in the form A = RTR, it will be more natural later when we discuss the vectorized version of the algorithm to write F = RT and A = FF T 2 • we can derive the algorithm for computing F by examining the matrix equation A = RTR = FF T on an element-by-element basis, writing       a11 a12 ··· a1n f11 f11 f21 ··· fn1 a12 a22 ··· a2n f21 f22   f22 fn2        . . .  =  . . ..   .. .   . . .   . . .   . .  a1n a2n ··· ann fn1 fn2 ··· fnn fnn 2 • from the above matrix multiplication we see that f11 = a11, from which it follows that √ f11 = a11

• from the relationship f11fi1 = ai1 and the fact that we already know f11, we obtain ai1 fi1 = , i = 2, . . . , n r11 2 2 • proceeding to the second column of F , we see that f21 + f22 = a22. Since we already know f21, we have q 2 f22 = a22 − f21

• next, we use the relation f21fi1 + f22fi2 = a2i to compute

a2i − f21fi1 fi1 = f22 • hence we get 2 a11 = f11, ai1 = f11fi1, i = 2, . . . , n . . 2 2 2 akk = fk1 + fk2 + ··· + fkk, aik = fk1fi1 + ··· + fkkfik, i = k + 1, . . . , n • the resulting algorithm that runs for k = 1, . . . , n is 1/2 Pk−1 2 fkk = akk − j=1 fkj Pk−1 aik − j=1 fkjfij fik = i = k + 1, . . . , n fkk • this algorithm requires roughly half as many operations as Gaussian elimination • instead of considering an elementwise algorithm, we can also derive a vectorized version T • we use the relationship aij = fi fj to compute rij, where fi is the ith column of F • we start by computing 1 f1 = √ a1 a11 where ai is the ith column of A • then we set A(1) = A and compute 0 0 ··· 0 0  (2) (1) T   A = A − f1f1 = .  . A2  0 3 • note that 1 0 A(1) = B BT 0 A2

where B is the identity matrix with its first column replaced by f1 −1 • writing C = B , we see that A2 is positive definite since 1 0 = CACT 0 A2 is positive definite: T T 0 1 0 0 T T T x A2x = = (C y) A(C y) > 0 x 0 A2 x for all x 6= 0 (or if you know Sylvester law of inertia, you can apply it to deduce the same thing) • so we may repeat the process on A2 h (2) (2) (2)i • we partition the matrix A2 into columns, writing A2 = a2 a3 ··· an and then compute 1 0 f2 = q (2) (2) a2 a22 • we then compute (2) T A3 = A − f2f2 and so on • note that 2 2 2 akk = fk1 + fk2 + ··· + fkk which implies that p |fki| ≤ |akk| • in other words, the elements of F are bounded • we also have the relationship T 2 2 2 2 det A = det F det F = (det F ) = f11f22 ··· fnn • is the Cholesky decompositon unique? • employing a similar approach to the one used to prove the uniquess of the lu factorization, we assume that A has two Cholesky factorizations T T A = F1F1 = F2F2 • then −1 T −T F2 F1 = F2 F1 but since F1 and F2 are lower triangular, both matrices must be diagonal • let −1 T −T F2 F1 = D = F2 F1 T T −1 T −T • so F1 = F2D and thus F1 = DF2 and we get D = F2 F1 • in other words, D−1 = D or D2 = I • hence D must have diagonal elements equal to ±1 • since we require that the diagonal elements be positive, it follows that the factorization is unique • in computing the Cholesky factorization, no row interchanges are necessary because A is positive definite, so the number of operations required to compute F is approximately n3/3 4 • a simple variant of the algorithm Cholesky factorization yields the LDLT factorization A = LDLT where L is a unit lower triangular matrix, and D is a diagonal matrix with positive diagonal elements • the algorithm is sometimes called the square-root-free Cholesky factorization since unlike in the usual Cholesky factorization, it does not require taking square roots (which can be expensive, most computer hardware and software use Newton-Raphson method to extract square roots) • the LDLT and Cholesky factorizations are related by F = LD1/2