Cholesky Decomposition Martin Licht UC San Diego Winter Quarter 2021 Cholesky Decomposition The class of symmetric positive definite matrices appears in many applications. We will see that for those matrices, the LU decomposition can always be performed without pivoting. Moreover, the LU decomposition of symmetric positive definete matrices leads to the Cholesky decomposition. Symmetric Positive Definite Matrices LU Decomposition of Symmetric Positive Definite Matrices Specialized Algorithms Symmetric Positive Definite Matrices × Let A 2 Rn n be a matrix. We call A symmetric if A = At: We call A positive definite if n 8x 2 R n f0g : hx; Axi > 0: A matrix that is both symmetric and positive definite is called symmetric positive definite. Symmetric Positive Definite Matrices × Suppose that M 2 Rn n is invertible. Let A = MtM. Then A is symmetric because At = (MtM)t = MtM = A Also, A is positive definite: for any x 2 Rn with x 6= 0 we can write y := Mx and observe y 6= 0. Then hx; Axi = hx; MtMxi = hMx; Mxi = hy; yi > 0: It follows that A is symmetric positive definite. Symmetric Positive Definite Matrices Example 01 1 05 1 05 1 01 @ 1 A ; @ 10 A ; @1 5 1A 1 0:17 0 1 5 Symmetric Positive Definite Matrices The previous example is already exhaustive for the class of symmetric positive definite matrices: all spd matrices A can be written in the form A = MtM for some invertible matrix M. In fact, we can achieve that M is upper triangular. Cholesky Decomposition Example 01 1 01 1 1 01 1 1 @1 1 A @ 1 1A = @1 2 1A 1 1 1 1 2 02 1 02 3 11 04 6 21 @3 2 A @ 2 2A = @6 13 5A 1 2 1 1 2 5 6 0p 1 0p 1 2 2 p−1 0 0 1 q 2 2 −1 0 B p−1 3 C B q 3 q 3 C B 2 C B − C = @−1 2 −1A @ 2 q A @ 2 2 A 0 − 3 p2 p2 0 −1 2 2 3 3 Symmetric Positive Definite Matrices LU Decomposition of Symmetric Positive Definite Matrices Specialized Algorithms Cholesky Decomposition × × If A 2 Rn n is invertible and R 2 Rn n is upper triangular with A = RtR, then we call R a Cholesky factor of A. We call A = RtR the Cholesky decomposition. Note that A invertible implies R invertible. Cholesky Decomposition We will show the Cholesky decomposition exists precisely for the class of symmetric positive definite matrices. The strategy will be showing that symmetric positive definite matrices allow for an LU decomposition without pivoting, and then we modify that LU decomposition to get a Cholesky decomposition. Cholesky Decomposition Lemma Suppose that A is symmetric positive definite. Then A satisfies the principle minor criterion. Proof. Let A be symmetric positive definite n × n matrix. We want to show that all principle minors Ak, 1 ≤ k ≤ n are invertible. k×k Suppose that there exists a principle minor Ak 2 R with 1 ≤ k ≤ n that is not invertible. So there exists v 2 Rk with v 6= 0 n but Akv = 0. Let ev 2 R be the extension of v to an n vector. We have ev 6= 0. We get 0 < hev; Aevi = hv; Akvi = hv; 0i = 0: This is a contradiction. So Ak must have been invertible in the first place. Cholesky Decomposition Corollary The LU decomposition without pivoting exists for symmetric definite positive matrices. Cholesky Decomposition Lemma The LU decomposition of a symmetric positive definite matrix has U with positive diagonal entries. Proof. × Let A be s.p.d. and let L; U 2 Rn n be the matrices from its LU decomposition. We want to show that Uii > 0 for all 1 ≤ i ≤ n. Suppose that instead there exists 1 ≤ i ≤ n with Uii ≤ 0. Then Uii = 0 cannot hold because U is invertible, so Uii < 0 must hold. n t Let v 2 R be the solution to L v = ei, and then let w := Uv. We obtain v by backward substitution and find that vi = 1 and that vj = 0 for all j > i. Hence wi = uiivi = uii. t 0 < hv; Avi = hv; LUvi = hL v; Uvi = hei; wi = wi = uii < 0 This is a contradiction. Hence uii > 0 must hold. Cholesky Decomposition Let A = LU be the LU decomposition of a symmetric positive × definite matrix A 2 Rn n. We now define the factors in the Cholesky decomposition.p Let n×n D 2 R be the diagonal matrix with Dii = Uii. We define new n × n matrices R := D−1U; S := LD: We get that A = LU = LDD−1U = SR: The upper triangular matrix R will be the Cholesky factor in our Cholesky decomposition. To verify that, it only remains need to show that St = R. Cholesky Decomposition To see that, we first use the symmetry of A to find SR = A = At = RtSt: Since R and S are invertible, we thus get St = R−tSR: If we can show that R−tS = I, then we get St = R. Cholesky Decomposition Since R is upper triangular, Rt is lower triangular. The inverse of a lower triangular matrix is again lower triangular. Hence R−t is lower triangular. The diagonal entries of R−t are the reciprocals of the diagonal entries of Rt. Furthermore, we recall that Rt and S have the same diagonal entries. Hence the diagonal entries of R−t are the reciprocals of the diagonal entries of S. We conclude that R−tS has only 1’s on the diagonal. Cholesky Decomposition Let us abbreviate T := R−tS. So far we have T = R−tS is a unit lower triangular matrix. Suppose that T has any non-zero entry below the diagonal, say, in row i and column j. Let i be so small that all entries to the left of ti;j are zero. Then the entry (i; j) of TR will be ti;jrj;i 6= 0, which contradicts St = TR being upper triangular. Hence T is also not only a unit lower triangular matrix but also a diagonal matrix with only 1’s on the diagonal. In other words, T is the unit matrix. So R−tS = I must hold. Putting this together, we finally get St = R. Cholesky Decomposition Theorem Every symmetric positive definite matrix has a Cholesky decomposition. But we have also established before that every matrix which has a Cholesky decomposition is symmetric positive definite. We conclude the following theorem. Theorem A matrix has a Cholesky decomposition if and only if it is symmetric positive definite. Cholesky Decomposition The above arguments also show the following procedure for computing the Cholesky decomposition of a symmetric positive definite matrix: 1: CholeskyDecomposition 2: Compute the factor U in the LU Decomposition 3: for m = 1; 2;:::; n do 4: for k = m; m + 1;:::; n do p 5: rm;k = um;k= um;m 6: end for 7: end for Specialized Algorithms × Consider the matrix A 2 R3 3 given by 04 2 0 1 @2 10 −3A : 0 −3 5 Its LU decomposition can be calculated by 04 2 0 1 0 4 2 0 1 0 4 2 0 1 @2 10 −3A @1=29 −3A @1=29 −3A ; 0 −3 5 0 −3 5 0 −1=34 that is, 0 1 0 01 04 2 0 1 04 2 0 1 @1=2 1 0A @0 9 −3A = @2 10 −3A 0 −1=3 1 0 0 4 0 −3 5 Specialized Algorithms ... that is, 0 1 0 01 04 2 0 1 04 2 0 1 @1=2 1 0A @0 9 −3A = @2 10 −3A 0 −1=3 1 0 0 4 0 −3 5 We extract the diagonal, 0 1 0 1 0 1 1 0 1 1 0 0 2 2 4 2 0 1 ··· = @1=2 1 0A @ 3 A @ 3 A @0 9 −3A − = 1 0 1 3 1 2 2 0 0 4 Specialized Algorithms We extract the diagonal, 0 1 0 1 0 1 1 0 1 1 0 0 2 2 4 2 0 1 ··· = @1=2 1 0A @ 3 A @ 3 A @0 9 −3A − = 1 0 1 3 1 2 2 0 0 4 Bringing the diagonal entries of triangular matrices to matching values, we get 02 0 01 02 1 0 1 04 2 0 1 @1 3 0A @0 3 −1A = @2 10 −3A 0 −1 2 0 0 2 0 −3 5 Note that we have divided the i-th row of U by the square root of its i-th diagonal entry, and we have multiplied the i-th column of L by the corresponding entry. Symmetric Positive Definite Matrices LU Decomposition of Symmetric Positive Definite Matrices Specialized Algorithms Specialized Algorithms The Cholesky decomposition can be calculated from the LU decomposition with only minimal effort. However, a different algorithm seems to more common in practice, as we will discuss in the next subsection. Specialized Algorithms Let A be a positive definite matrix, and let A = RtR be its Cholesky decomposition. Then R is an upper triangular matrix with positive diagonal entries. 0 1 r1;1 r1;2 r1;3 ::: r1;n B r2;2 r2;3 ::: r2;nC B C B r ::: r C R = B 3;3 3;nC : B .. C @ . A rn;n Specialized Algorithms From A = RtR, 0 1 0 1 r1;1 r1;1 r1;2 r1;3 ::: r1;n Br1;2 r2;2 C B r2;2 r2;3 ::: r2;nC B C B C Br r r C B r ::: r C A = B 1;3 2;3 3;3 C B 3;3 3;nC : B .