Cholesky Decomposition
Martin Licht
UC San Diego
Winter Quarter 2021 Cholesky Decomposition
The class of symmetric positive definite matrices appears in many applications.
We will see that for those matrices, the LU decomposition can always be performed without pivoting.
Moreover, the LU decomposition of symmetric positive definete matrices leads to the Cholesky decomposition. Symmetric Positive Definite Matrices
LU Decomposition of Symmetric Positive Definite Matrices
Specialized Algorithms Symmetric Positive Definite Matrices
× Let A ∈ Rn n be a matrix. We call A symmetric if
A = At.
We call A positive definite if
n ∀x ∈ R \{0} : hx, Axi > 0.
A matrix that is both symmetric and positive definite is called symmetric positive definite. Symmetric Positive Definite Matrices
× Suppose that M ∈ Rn n is invertible. Let A = MtM. Then A is symmetric because
At = (MtM)t = MtM = A
Also, A is positive definite: for any x ∈ Rn with x 6= 0 we can write y := Mx and observe y 6= 0. Then
hx, Axi = hx, MtMxi = hMx, Mxi = hy, yi > 0.
It follows that A is symmetric positive definite. Symmetric Positive Definite Matrices
Example
1 5 5 1 0 1 , 10 , 1 5 1 1 0.17 0 1 5 Symmetric Positive Definite Matrices
The previous example is already exhaustive for the class of symmetric positive definite matrices: all spd matrices A can be written in the form A = MtM for some invertible matrix M.
In fact, we can achieve that M is upper triangular. Cholesky Decomposition
Example
1 1 1 1 1 1 1 1 1 = 1 2 1 1 1 1 1 2 2 2 3 1 4 6 2 3 2 2 2 = 6 13 5 1 2 1 1 2 5 6 √ √ 2 2 √−1 0 q 2 2 −1 0 √−1 3 q 3 q 3 2 − = −1 2 −1 2 q 2 2 0 − 3 √2 √2 0 −1 2 2 3 3 Symmetric Positive Definite Matrices
LU Decomposition of Symmetric Positive Definite Matrices
Specialized Algorithms Cholesky Decomposition
× × If A ∈ Rn n is invertible and R ∈ Rn n is upper triangular with A = RtR, then we call R a Cholesky factor of A.
We call A = RtR the Cholesky decomposition.
Note that A invertible implies R invertible. Cholesky Decomposition
We will show the Cholesky decomposition exists precisely for the class of symmetric positive definite matrices.
The strategy will be showing that symmetric positive definite matrices allow for an LU decomposition without pivoting, and then we modify that LU decomposition to get a Cholesky decomposition. Cholesky Decomposition
Lemma Suppose that A is symmetric positive definite. Then A satisfies the principle minor criterion.
Proof. Let A be symmetric positive definite n × n matrix. We want to show that all principle minors Ak, 1 ≤ k ≤ n are invertible. k×k Suppose that there exists a principle minor Ak ∈ R with 1 ≤ k ≤ n that is not invertible. So there exists v ∈ Rk with v 6= 0 n but Akv = 0. Let ev ∈ R be the extension of v to an n vector. We have ev 6= 0. We get
0 < hev, Aevi = hv, Akvi = hv, 0i = 0.
This is a contradiction. So Ak must have been invertible in the first place. Cholesky Decomposition
Corollary The LU decomposition without pivoting exists for symmetric definite positive matrices. Cholesky Decomposition
Lemma The LU decomposition of a symmetric positive definite matrix has U with positive diagonal entries.
Proof. × Let A be s.p.d. and let L, U ∈ Rn n be the matrices from its LU decomposition. We want to show that Uii > 0 for all 1 ≤ i ≤ n. Suppose that instead there exists 1 ≤ i ≤ n with Uii ≤ 0. Then Uii = 0 cannot hold because U is invertible, so Uii < 0 must hold. n t Let v ∈ R be the solution to L v = ei, and then let w := Uv. We obtain v by backward substitution and find that vi = 1 and that vj = 0 for all j > i. Hence wi = uiivi = uii.
t 0 < hv, Avi = hv, LUvi = hL v, Uvi = hei, wi = wi = uii < 0
This is a contradiction. Hence uii > 0 must hold. Cholesky Decomposition
Let A = LU be the LU decomposition of a symmetric positive × definite matrix A ∈ Rn n.
We now define the factors in the Cholesky decomposition.√ Let n×n D ∈ R be the diagonal matrix with Dii = Uii. We define new n × n matrices
R := D−1U, S := LD.
We get that
A = LU = LDD−1U = SR.
The upper triangular matrix R will be the Cholesky factor in our Cholesky decomposition. To verify that, it only remains need to show that St = R. Cholesky Decomposition
To see that, we first use the symmetry of A to find
SR = A = At = RtSt.
Since R and S are invertible, we thus get
St = R−tSR.
If we can show that R−tS = I, then we get St = R. Cholesky Decomposition
Since R is upper triangular, Rt is lower triangular.
The inverse of a lower triangular matrix is again lower triangular.
Hence R−t is lower triangular.
The diagonal entries of R−t are the reciprocals of the diagonal entries of Rt.
Furthermore, we recall that Rt and S have the same diagonal entries.
Hence the diagonal entries of R−t are the reciprocals of the diagonal entries of S.
We conclude that R−tS has only 1’s on the diagonal. Cholesky Decomposition
Let us abbreviate T := R−tS.
So far we have T = R−tS is a unit lower triangular matrix.
Suppose that T has any non-zero entry below the diagonal, say, in row i and column j. Let i be so small that all entries to the left of ti,j are zero.
Then the entry (i, j) of TR will be ti,jrj,i 6= 0, which contradicts St = TR being upper triangular. Hence T is also not only a unit lower triangular matrix but also a diagonal matrix with only 1’s on the diagonal. In other words, T is the unit matrix.
So R−tS = I must hold.
Putting this together, we finally get St = R. Cholesky Decomposition
Theorem Every symmetric positive definite matrix has a Cholesky decomposition.
But we have also established before that every matrix which has a Cholesky decomposition is symmetric positive definite.
We conclude the following theorem. Theorem A matrix has a Cholesky decomposition if and only if it is symmetric positive definite. Cholesky Decomposition
The above arguments also show the following procedure for computing the Cholesky decomposition of a symmetric positive definite matrix: 1: CholeskyDecomposition 2: Compute the factor U in the LU Decomposition 3: for m = 1, 2,..., n do 4: for k = m, m + 1,..., n do √ 5: rm,k = um,k/ um,m 6: end for 7: end for Specialized Algorithms
× Consider the matrix A ∈ R3 3 given by 4 2 0 2 10 −3 . 0 −3 5
Its LU decomposition can be calculated by
4 2 0 4 2 0 4 2 0 2 10 −3 1/29 −3 1/29 −3 , 0 −3 5 0 −3 5 0 −1/34
that is,
1 0 0 4 2 0 4 2 0 1/2 1 0 0 9 −3 = 2 10 −3 0 −1/3 1 0 0 4 0 −3 5 Specialized Algorithms
... that is,
1 0 0 4 2 0 4 2 0 1/2 1 0 0 9 −3 = 2 10 −3 0 −1/3 1 0 0 4 0 −3 5
We extract the diagonal,
1 1 0 0 2 2 4 2 0 1 ··· = 1/2 1 0 3 3 0 9 −3 − / 1 0 1 3 1 2 2 0 0 4 Specialized Algorithms
We extract the diagonal,
1 1 0 0 2 2 4 2 0 1 ··· = 1/2 1 0 3 3 0 9 −3 − / 1 0 1 3 1 2 2 0 0 4 Bringing the diagonal entries of triangular matrices to matching values, we get
2 0 0 2 1 0 4 2 0 1 3 0 0 3 −1 = 2 10 −3 0 −1 2 0 0 2 0 −3 5
Note that we have divided the i-th row of U by the square root of its i-th diagonal entry, and we have multiplied the i-th column of L by the corresponding entry. Symmetric Positive Definite Matrices
LU Decomposition of Symmetric Positive Definite Matrices
Specialized Algorithms Specialized Algorithms
The Cholesky decomposition can be calculated from the LU decomposition with only minimal effort.
However, a different algorithm seems to more common in practice, as we will discuss in the next subsection. Specialized Algorithms
Let A be a positive definite matrix, and let A = RtR be its Cholesky decomposition. Then R is an upper triangular matrix with positive diagonal entries. r1,1 r1,2 r1,3 ... r1,n r2,2 r2,3 ... r2,n r ... r R = 3,3 3,n . .. . . . rn,n Specialized Algorithms
From A = RtR, r1,1 r1,1 r1,2 r1,3 ... r1,n r1,2 r2,2 r2,2 r2,3 ... r2,n r r r r ... r A = 1,3 2,3 3,3 3,3 3,n . . . . .. .. . . . . . . . r1,n r2,n r3,n ... rn,n rn,n
The formula for the matrix-matrix multiplication leads to several identities that relate the entries of A to the entries of R. Specialized Algorithms
From A = RtR we get the following identities for the entries of R.
Diagonal entries: for 1 ≤ i ≤ n we have
i i−1 X X 2 2 aii = rk,irk,i = rk,i + rii . k=1 k=1
Since the entries of R are positive, v u i−1 u X 2 rii = taii − rk,i. k=1
This formula for rii only depends on entries of R above rii in the i-th column of R. Specialized Algorithms
Superdiagonal entries: for 1 ≤ j < i we have
j j−1 X X aij = rk,irk,j = rk,irk,j + rj,irj,j k=1 k=1
Since rj,j is non-zero,
j−1 −1 X rj,i = rj,j aij − rk,irk,j k=1
This formula for rj,i depends only on entries of R that are in a previous column or row of R. Specialized Algorithms
This suggests that we can calculate R entry by entry from the given matrix A. We may use the following calculation: I Calculate row 1 of R I Calculate row 2 of R I ... I Calculate row n of R In a pseudocode, this reads as follows:
1: CholeskyDecomposition 2: for j = 1, 2,..., n do q Pj−1 2 3: rj,j = aj,j − k=1 rk,j 4: for i = j + 1, j + 2,..., n do −1 Pj−1 5: rj,i = rj,j ai,j − k=1 rk,irk,j 6: end for 7: end for Specialized Algorithms
Alternatively, we may use the following calculation: I Calculate column 1 of R I Calculate column 2 of R I ... I Calculate column n of R In a pseudocode, this reads as follows:
1: CholeskyDecomposition 2: for i = 1, 2,..., n do 3: for j = 1, 2,..., i − 1 do −1 Pj−1 4: rj,i = rj,j ai,j − k=1 rk,irk,j 5: end forq Pi−1 2 6: ri,i = ai,i − k=1 rk,i 7: end for