LU-factorization and Positive Definite Matrices Tom Lyche University of Oslo Norway LU-factorization and Positive Definite Matrices – p. 1/49 Topics Today Block multiplication of matrices Basics on triangular matrices LU factorization of matrices Positive definite matrices examples criteria for positive definiteness LU factorization of positive definite matrices LU-factorization and Positive Definite Matrices – p. 2/49 Partitioned matrices A rectangular matrix A can be partitioned into submatrices by drawing horizontal lines between selected rows and vertical lines between 123 selected columns. For example, A = 456 can be partitoned as 789 h i 1 2 3 1 2 3 A11 A12 (i) = 4 5 6 , (ii) a.1, a.2, a.3 = 4 5 6 , A21 A22 7 8 9 h i 7 8 9 T a1. 1 2 3 1 2 3 T (iii) a2. = 4 5 6 , (iv) A11, A12 = 4 5 6 . aT 7 8 9 h i 7 8 9 3. The submatrices in a partition is often referred to as blocks and a partitioned matrix is sometimes called a block matrix. LU-factorization and Positive Definite Matrices – p. 3/49 Column partition Suppose A Rm,p and B Rp,n. ∈ ∈ If B = b.1,..., b.n is partitioned into columns then the partition of the producth AB intoi columns is AB = Ab.1, Ab.2,..., Ab.n . h i In particular, if I is the identity matrix of order p then A = AI = A e1, e2,..., ep = Ae1, Ae2,..., Aep h i h i and we see that column j of A can be written Aej for j = 1,...,p. LU-factorization and Positive Definite Matrices – p. 4/49 Row partition If A is partitioned into rows then T T a1. a1.B T T a2. a2.B AB = B = . . . . T T am. am.B T and taking A = Ip it follows that row i of B can be written ei B. It is often useful to write the matrix-vector product Ax as a linear combination of the columns of A Ax = x a + x a + + x a . 1 .1 2 .2 ··· p .p One way to see that this is correct is to partition A into columns and x into rows. LU-factorization and Positive Definite Matrices – p. 5/49 Rules for 2 2 blocks × p,r p,n−r If B = B , B , where B1 R and B2 R then 1 2 ∈ ∈ h i A B1, B2 = AB1, AB2 . h i h i A 1 k,p m−k,p If A = , where A1 R and A2 R then A2 ∈ ∈ A1 A1B B = . A2 A2B m,s m,p−s If A = A , A and B = B , B , where A1 R , A2 R , 1 2 1 2 ∈ ∈ B Rhs,p and iB Rp−s,nh then i 1 ∈ 2 ∈ B1 A1, A2 = A1B1 + A2B2 . B h i 2 h i LU-factorization and Positive Definite Matrices – p. 6/49 The general rule for 2 2 blocks × A11 A12 B11 B12 If A = and B = then A21 A22 B21 B22 A11 A12 B11 B12 A11B11 + A12B21 A11B12 + A12B22 = , A21 A22 B21 B22 A21B11 + A22B21 A21B12 + A22B22 provided the vertical partition line in A matches the horizontal line in B, i.e. the number of columns in A11 and A21 equals the number of rows in B11 and B12. LU-factorization and Positive Definite Matrices – p. 7/49 The general case If A11 A1s B11 B1q . ··· . ··· . A = . , B = . , Ap1 Aps Bs1 Bsq ··· ··· and if all the matrix products in s Cij = AikBkj, i = 1,...,p, j = 1,...,q kX=1 are well defined then C11 C1q . ··· . AB = . . Cp1 Cpq ··· LU-factorization and Positive Definite Matrices – p. 8/49 Block-Triangular Matrices Lemma 1. Suppose A11 A12 A = 0 A22 where A, A11 and A22 are square matrices. Then A is nonsingular if and only if both A11 and A22 are nonsingular. In that case −1 −1 −1 A11 A11 A12A22 A−1 = − (1) −1 0 A22 LU-factorization and Positive Definite Matrices – p. 9/49 Proof ⇐ If A11 and A12 are nonsingular then −1 −1 −1 A A A12A A11 A12 I 0 11 − 11 22 = = I −1 0 A22 0 A22 0 I and A is nonsingular with the indicated inverse. LU-factorization and Positive Definite Matrices – p. 10/49 Proof ⇒ Conversely, let B be the inverse of the nonsingular matrix A. We partition B conformally with A and have B11 B12 A11 A12 I 0 BA = = = I B21 B22 0 A22 0 I Using block-multiplication we find B11A11 = I, B21A11 = 0, B21A12 + B22A22 = I. The first equation implies that A11 is invertible, this in turn implies that B21 = 0 in the second equation, and then the third equation simplifies to B22A22 = I. We conclude that also A22 is invertible. LU-factorization and Positive Definite Matrices – p. 11/49 The inverse Consider now a triangular matrix. n,n Lemma 2. An upper (lower) triangular matrix A = [aij] R is ∈ nonsingular if and only if the diagonal entries aii, i = 1,...,n are nonzero. In that case the inverse is upper (lower) triangular with diagonal −1 entries aii , i = 1,...,n. Proof: We use induction on n. The result holds for n = 1: The 1-by-1 matrix A =(a11) is invertible if and only if a11 = 0 −1 −1 6 and in that case A =(a11 ). Suppose the result holds for n = k and let A = Rk+1,k+1 be upper triangular. ∈ LU-factorization and Positive Definite Matrices – p. 12/49 Proof We partition A in the form Ak ak A = 0 ak+1,k+1 and note that A Rk,k is upper triangular. By Lemma 1.1 A is k ∈ nonsingular if and only if Ak and (ak+1,k+1) are nonsingular and in that case −1 −1 −1 A A aka A−1 = k − k k+1,k+1 . −1 0 ak+1,k+1 By the induction hypothesis Ak is nonsingular if and only if the diagonal −1 entries a11,...,akk of Ak are nonzero and in that case Ak is upper −1 triangular with diagonal entries aii , i = 1,...,k. The result for A follows. LU-factorization and Positive Definite Matrices – p. 13/49 Unit Triangular Matrices Lemma 3. The product C = AB =(cij) of two upper(lower) triangular matrices A =(aij ) and B =(bij) is upper(lower) triangular with diagonal entries cii = aiibii for all i. Proof. Exercise. A matrix is unit triangular if it is triangular with 1’s on the diagonal. Lemma 4. For a unit upper(lower) triangular matrix A Rn,n: ∈ 1. A is invertible and the inverse is unit upper(lower) triangular. 2. The product of two unit upper(lower) triangular matrices is unit upper(lower) triangular. Proof. 1. follows from Lemma 1.2, while Lemma 1.3 implies 2. LU-factorization and Positive Definite Matrices – p. 14/49 LU-factorization We say that A = LU is an LU-factorization of A Rn,n if L Rn,n is ∈ ∈ lower triangular and U Rn,n is upper triangular. In addition we will ∈ assume that L is unit triangular. Example 1. The equation 2 1 1 0 2 1 A = − = − 1 2 1/2 1 0 3/2 − − gives an LU -factorization of the 2-by-2 matrix A. LU-factorization and Positive Definite Matrices – p. 15/49 Example Every nonsingular matrix has a P LU-factorization, but not necessarily an LU-factorization. 0 1 Example 2. An LU -factorization of A = 1 1 must satisfy the equation 0 1 1 0 u1 u3 u1 u3 = = 1 1 l1 1 0 u2 l1u1 l1u3 + u2 for the unknowns l1 in L and u1,u2,u3 in U . Comparing (1, 1)-elements we see that u1 = 0, which makes it impossible to satisfy the condition 1= l1u1 for the (2, 1) element. We conclude that A has no LU -factorization. LU-factorization and Positive Definite Matrices – p. 16/49 Uniqueness Theorem 5. The LU -factorization of a nonsingular matrix is unique whenever it exists. Proof. x Suppose A = L1U 1 = L2U 2 are two LU -factorizations of the nonsingular matrix A. −1 −1 The equation L1U 1 = L2U 2 can be written in the form L2 L1 = U 2U 1 , −1 −1 where by lemmas 1.2-1.4 L2 L1 is unit lower triangular and U 2 U 1 is upper triangular. But then both matrices must be diagonal with ones on the diagonal. −1 −1 We conclude that L2 L1 = I = U 1U 2 which means that L1 = L2 and U 1 = U 2. LU-factorization and Positive Definite Matrices – p. 17/49 Leading Principal Submatrices Suppose A Cn,n. The upper left k k corners ∈ × a11 a1k . ··· . Ak = . for k = 1,...,n ak1 akk ··· of A are called the leading principal submatrices of A LU-factorization and Positive Definite Matrices – p. 18/49 A Lemma The following lemma will be used for existence. Lemma 6. Suppose A = LU is the LU -factorization of A Rn,n. For k = 1,...,n ∈ let Ak, Lk, U k be the leading principal submatrices of A, L, U , respectively. Then Ak = LkU k is the LU -factorization of Ak for k = 1,...,n. Proof: We partition A = LU as follows: Ak Bk Lk 0 U k Vk A = = = LU, (2) Ck Dk Mk Nk 0 Wk where D , N ,W Rn−k,n−k. k k k ∈ LU-factorization and Positive Definite Matrices – p. 19/49 Proof Using block-multiplication we find the equations Ak = LkU k (3) Bk = LkVk (4) Ck = MkU k (5) Dk = MkVk + NkWk (6) Since Lk is unit lower triangular and U k is upper triangular we see that (3) gives the LU-factorization of Ak.