Lecture 4/26/2013 Jed Duersch Spd Lecture 4/26/2013 matrices Cholesky decomposi- tion Jed Duersch Band matrices Applied mathematics PhD candidate, physics MA UC Berkeley April 26, 2013 Jed Duersch UCB 1/19 Symmetric positive-definite matricesI Lecture 4/26/2013 Definition Jed n×n Duersch A symmetric matrix A 2 R is positive definite iff xT Ax > 0 holds 8x 6= 0 2 n. Spd R matrices Cholesky Such matrices naturally arise in a variety of math, science, decomposi- tion and engineering applications. For example, Band The Hamiltonian operator of a physical system matrices The Hessian of a convex function The covariance matrix of linearly independent random variables Jed Duersch UCB 2/19 Symmetric positive-definite matricesII Lecture 4/26/2013 Theorem Jed The following statements are equivalent: Duersch 1 A is symmetric positive definite. Spd T n matrices 2 (x; y) := x Ay is an inner product on R . Cholesky 3 decomposi- All eigenvalues are positive and eigenvectors can be tion constructed to form an orthonormal basis. Band matrices 4 A is symmetric with positive leading principal minors. 5 The Cholesky decomposition of A exists. Jed Duersch UCB 3/19 ExistenceI Lecture 4/26/2013 To prove the existence of the Cholesky decomposition for Jed symmetric positive definite matrices, we apply induction. Duersch n−1×n−1 Induction hypothesis: Given any spd An−1 2 R Spd matrices assume there exists lower-triangular Ln−1 where Cholesky decomposi- T tion An−1 = Ln−1Ln−1 Band matrices . p p T Base case: A1 = [a1;1] = [ a1;1][ a1;1] . n×n Consistency: Show that for any spd An 2 R there exists Ln such that T An = LnLn . Jed Duersch UCB 4/19 ExistenceII Lecture 4/26/2013 Let us represent An as Jed T Duersch α c An = c B Spd matrices Cholesky decomposi- where α := a1;1 is a scalar, c := a2:n;1 is a column vector, tion n−1×n−1 and B = a2:n;2:n is the remaining lower-right Band R matrices sub-matrix. Then we have p p p1 T α 0 1 0 α α c p1 c I 1 T An = α n−1 0 B − α cc 0 In−1 Jed Duersch UCB 5/19 Existence III Lecture n−1 4/26/2013 Let y be an arbitrary nonzero column vector in R . We can solve Jed Duersch p α p1 cT 0 Spd α matrices 0 In−1 x = y Cholesky decomposi- tion Band It follows matrices 1 yT (B − p ccT )y = xT A x > 0 α n 1 T Thus B − α cc is spd. From the induction hypothesis 1 B − ccT = L LT α n−1 n−1 . Jed Duersch UCB 6/19 ExistenceIV Lecture 4/26/2013 Putting it all together, we have p p Jed p1 T Duersch α 0 1 0 α α c 1 1 T An = p c In−1 0 B − cc 0 I Spd α α n−1 matrices Cholesky p p p1 T decomposi- α 0 1 0 α α c tion 1 T = p c In−1 0 L L Band α n−1 n−1 0 In−1 matrices p p p1 T α 0 α α c p1 c L T = α n−1 0 Ln−1 T = LnLn Jed Duersch UCB 7/19 ImplementationI Lecture 4/26/2013 Note that the proof of existence was constructive. That Jed means we can follow the same procedure to calculate the Duersch Cholesky decomposition. Spd 1 Loop over columns. matrices Cholesky for k=1:n-1 decomposi- tion 2 Set diagonal element of L. Band L(k,k)=sqrt(A(k,k)) matrices 3 Finish column. L(k+1:n,k)=A(k+1:n)/L(k,k) 4 Update lower right sub-matrix of A. A(k+1:n,k+1:n)=A(k+1:n,k+1:n)- L(k+1:n,k)*L(k+1:n,k)T 5 End for loop. end Jed Duersch UCB 8/19 Band matricesI Lecture 4/26/2013 We can similarly take advantage of the stucture of band Jed matrices to optimize the calculation of the LU Duersch decomposition. Spd matrices Definition Cholesky n×n decomposi- A band matrix A 2 R of band width 2m + 1 has nonzero tion entries only on the main diagonal, m super-diagonals, and m Band matrices sub-diagonals. Jed Duersch UCB 9/19 Band matricesII Lecture 4/26/2013 For example, a tridiagonal matrix (band width 3) has the form: Jed Duersch 2a1;1 a1;2 0 3 Spd matrices 6a2;1 a2;2 a2;3 7 Cholesky 6 a3;2 a3;3 a3;4 7 decomposi- 6 7 tion 6 .. 7 A = 6 a4;3 a4;4 . 7 Band 6 7 matrices 6 .. .. 7 4 . an−1;n5 0 an;n−1 an;n Jed Duersch UCB 10/19 Band matrices III Lecture 4/26/2013 We can easily show that the LU decomposition will have the form: Jed Duersch 2 1 032u1;1 u1;2 0 3 Spd . matrices l2;1 1 6 .. 7 6 76 u2;2 7 Cholesky A = 6 .. .. 76 . 7 decomposi- 4 . 54 .. u 5 tion n−1;n 0 ln;n−1 1 Band 0 un;n matrices Because we know so many entries must be zero, we can optimize the algorithm to avoid calculating them. Similarly, the procedures for forward and backward substitution can also be optimized in these cases. Jed Duersch UCB 11/19 Diagonal dominanceI Lecture 4/26/2013 Partial pivoting can be implemented, but it will increase the Jed band width of U from m + 1 to 2m + 1. However, pivoting Duersch can sometimes be avoided. Spd matrices Definition Cholesky A matrix A 2 n×n is column diagonally dominant iff decomposi- R tion X Band jaj;jj > jai;jj matrices i6=j We can show that a column diagonally dominant matrix never requires pivoting. Jed Duersch UCB 12/19 Diagonal dominanceII Lecture (i) 4/26/2013 Proof: Let ak;j represent element k; j after zeroing column i Jed the standard LU algorithm. Assume the induction Duersch hypothesis: Spd matrices n (i) X (i) Cholesky jaj;jj > jak;jj decomposi- tion k=i+1;6=j Band When column i + 1 is zeroed below the diagonal, the matrices elements of A update as follows: a(i) a(i) a(i+1) = a(i) − k;i+1 i+1;j k;j k;j (i) ai+1;i+1 Jed Duersch UCB 13/19 Diagonal dominance III Lecture 4/26/2013 (i) (i) (i+1) (i) aj;i+1ai+1;j Jed ja j = ja − j Duersch j;j j;j (i) ai+1;i+1 Spd (i) (i) matrices ja jja j ≥ ja(i)j − j;i+1 i+1;j Cholesky j;j (i) decomposi- ja j tion i+1;i+1 Band n (i) (i) X (i) jaj;i+1jjai+1;jj matrices > ja j − k;j (i) k=i+1;6=j jai+1;i+1j n (i) ! X (i) (i) jaj;i+1j = ja j + ja j 1 − k;j i+1;j (i) k=i+2;6=j jai+1;i+1j Jed Duersch UCB 14/19 Diagonal dominanceIV Lecture 4/26/2013 To make further progress we observe: Jed (i) (i) Duersch ja jja j ja(i+1)j ≤ ja(i) j + k;i+1 i+1;j k;j k;j (i) Spd ja j matrices i+1;i+1 (i) (i) Cholesky ja jja j decomposi- (i) (i+1) k;i+1 i+1;j tion jak;jj ≥ jak;j j − ja(i) j Band i+1;i+1 matrices Jed Duersch UCB 15/19 Diagonal dominanceV Lecture 4/26/2013 Together this gives: Jed 0 (i) 1 Duersch n n (i+1) X (i+1) (i) X jak;i+1j ja j > ja j + ja j 1 − j;j k;j i+1;j @ (i) A Spd ja j matrices k=i+2;6=j k=i+2 i+1;i+1 Cholesky n decomposi- X (i+1) tion > jak;j j Band k=i+2;6=j matrices This shows consistency for the induction hypothesis and the base case easily follows from the definition of diagonal dominance. The desired result immediate follows since the diagonal elements are always largest in magnitude. Jed Duersch UCB 16/19 ImplementationI Lecture 4/26/2013 We can follow the standard LU procedure and simply avoid Jed elements that do not update. If A has band width 2m + 1, Duersch we have Spd 1 Loop over columns. matrices Cholesky for k=1:n-1 decomposi- tion 2 Construct column k of L. Band L(k+1:k+m,k)=A(k+1:k+m,k)/A(k,k) matrices 3 Set column k of A to zero. A(k+1:k+m,k)=0 4 Update lower right sub-matrix of A. A(k+1:k+m,k+1:k+m)=A(k+1:k+m,k+1:k+m)- L(k+1:k+m,k)*A(k,k+1:k+m) 5 End for loop. Identify U. end; U=A Jed Duersch UCB 17/19 ImplementationII Lecture 4/26/2013 In the case of a tridiagonal matrix, the procedure becomes exceptionally simple and efficient. Let us write the diagonals Jed Duersch of A as vectors. Then L and U can be written as: Spd 2d1 e1 0 3 matrices . 6c d .. 7 Cholesky 6 1 2 7 decomposi- 6 7 tion A = .