Chapter 9: Matrix Factorization

Home , Triangular matrix

• Positive Deﬁnite Matrices Deﬁnition: An n × n symmetric real matrix M is

positive deﬁnite if:

x, Mx = xT Mx> 0

for all x ∈ Rn and x = 0.

M is positive semideﬁnite if

x, Mx = xT Mx ≥ 0 ∀ x ∈ Rn

M is negative deﬁnite if

x, Mx = xT Mx< 0

for all x ∈ Rn and x = 0.

M is negative semideﬁnite if

x, Mx = xT Mx ≤ 0 ∀ x ∈ Rn.

1 Deﬁnition: A nonzero column vector x is

an eigenvector of a square matrix M

if ∃ a scalar λ such that

Mx = λx

Then λ is an eigenvalue of M.

Deﬁnition: A principal minor of M is the

determinant of any submatrix obtained

from M by deleting its last k

rows and k columns (k =0, 1, , n − 1).

2 Properties: Given an n × n symmetric real matrix M

(i) M is positive deﬁnite if and only if

all its eigenvalues are positive;

(ii) M is positive deﬁnite if and only if

all its principal minors are positive.

Some Necessary Conditions for a symmetric positive deﬁnite matrix:

(i) All diagonal elements of M must be positive. (ii) The element of M having the largest absolute value must be on the diagonal of M 2 (iii) miimjj > |mij| , ∀i = j.

3 Deﬁnition: The square root of a matrix M

is a matrix M 1/2 such that

M = M 1/2M 1/2.

Properties: If M and M 1/2 are both required

to be positive deﬁnite (or positive semideﬁnite),

then M 1/2 is unique.

4 Theorem: Let A be an m × n matrix with

full row rank (m ≤ n), then

AAT is positive deﬁnite.

Proof: (i) AAT is symmetric.

(ii) x,AAT x = xT AAT x

= AT x≥ 0, ∀x ∈ Rm.

If x,AAT x = 0, then AT x = 0.

Since A has full row rank, we have x = 0.

Hence: x,AAT x > 0 ∀x ∈ Rm and x = 0.

5 Corollary: Let A be an m × n matrix with full

row rank and D be a diagonal

matrix with all diagonal elements

being positive, then ADAT is

positive deﬁnite.

Proof: ADAT = AD1/2D1/2AT

= (AD1/2)(AD1/2)T

6 • Cholesky Factorization

Main Theorem: If M is an n × n symmetric positive deﬁnite matrix, then it has a unique triangular factorization LLT , where L is a lower triangular matrix with positive diagonal elements.

7 Proof: The proof is by induction on the order of the matrix M. The result is certainly true for one by one matrices since m11 is positive.

Suppose the assertion is true for matrices of order n − 1. Let M be a symmetric positive deﬁnite matrix of order n. It can be partitioned into the form

d vT M = . v H where d is a positive scalar and H is an n−1 by n−1 submatrix. The partitioned matrix can be written as the product

√ ! √ T ! d 0 1 0 d √v d √v I d n−1 0 H 0 In−1 vvT where H = H − d . Clearly the matrix H is symmetric. It is also positive deﬁnite since for any nonzero vector x of length n − 1, vvT xT Hx = xT (H − )x d T ! xT v d vT −x v = (− , xT ) d d v H x which implies xT Hx ≥ 0. By the induction assumption, H has a T triangular factorization LHLH with positive diagonals. Thus, M

8 can be expressed as

√ ! √ T ! d 0 1 0 1 0 d √v d √v I T d n−1 0 LH 0 LH 0 In−1 √ ! √ T ! d 0 d √v = d √v L T d H 0 LH = LLT

It is left to the reader to show that the factor L is unique.

9 • Computing Cholesky Factor

1. Outer Product Form:

T d1 v1 M = A0 = H0 = v1 H1 √ ! ! √ T ! 1 0 v1 d1 0 d1 √ = v T d1 √ 1 v1v1 In−1 0 H1 − d1 d1 0 In−1 1 0 T = L1 L1 0 H1 T = L1A1L1

  1 0 0 1 0 T A1 = =  0 d2 v2  0 H1 0 v2 H2    1 0 0  1 √ 0 =  0 d2   0 1 0   v   T  √ 2 v2v2 0 In−2 0 0 H2 − d2 d2  1 0 0  √ T v2  d2 √   d2  0 In−2 T = L2A2L2 . T An−1 = LnInLn .

10 Here, for 1 ≤ i ≤ n, di is a positive scalar, vi is a vector of length n − i, and, Hi is a n − i by n − i positive deﬁnite symmetric matrix. After N steps of the algorithm, we have T T T T M = L1L2 ··· LnLn ··· L2 L1 = LL . where it can be shown (see Exercise 2.1.6) that

L = L1 + L2 + ··· + Ln − (n − 1)In.

Thus, the i-th column of L is precisely the i-th column of Li.

In this scheme, the column of L are computed one by one. At the same time, each step involves the modiﬁcation of the submatrix T vivi Hi by the outer product to give Hi, which is simply the sub- di matrix remaining to be factored. The access to the components of M during the factorization is depicted as follows.

11 2. Bordering Method:

An alternative formulation of the factorization process is the Bor- dering Method. Suppose the matrix M is partitioned as M u M = uT s where the symmetric factorization L LT of the n − 1 by n − 1 M M leading principle submatrix M has already been obtained. (why is M positive deﬁnite?) Then factorization of M is given by L 0 LT w M = M M wT t 0 t where w = L−1u M and t = (s − wT w)1/2. (Why is s − wT w positive?) Note that the factorization L LT of the submatrix M is also ob- M M tained by the bordering technique. So, the scheme can be described as follows.

For i = 1, 2, . . . , n,       l1,1 0 li,1 ai,1 . . . . Solve  . ..   .  =  .  li−1,1 . . . li−1,i−1 li,i−1 ai,i−1 Compute i−1 X 2 1/2 li,i = (ai,i − li,k) . k=1

12 In this scheme, the rows of L are computed one at a time, the part of the matrix remaining to be factored is not accessed until the corresponding part of L is to be computed. The sequence of computations can be depicted as follows.

13 3. Inner Product Form:

The ﬁnal scheme for computing the components of L is the Inner Product Form of the algorithm. It can be described as follows.

For j = 1, 2, . . . , n,

Compute j−1 X 2 1/2 lj,j = (aj,j − lj,k) . k=1

For i = j + 1, j + 2, . . . , n,

Compute j−1 ! X li,j = ai,j − li,klj,k /lj,j. k=1

These formulae can be derived directly by equating the elements of A to the corresponding elements of the product LLT .

Like the outer product version of the algorithm, the columns of L are computed one by one, but the part of the matrix remaining to be factored is not accessed during the scheme. The sequence of computations and the relevant access to the components of M (or L) is depicted as follows.

14 The latter two formulations can be organized so that only inner products are involved. This can be used to improve the accuracy of the numerical factorization by accumulating the inner products in double precision. On some computers, this can be done at little extra cost.

15 #

• Block Cholesky

T TT MMM11 12 13 L11 LL11 21L 31 MMM = LL T T 21 22 23 21 22 L22 L32 MMM31 32 33 L31 L32L 33 T L33

M = L LT

T M11 = L11L11 ← L11

T M21 = L21L11 ← L21

T M31 = L31L11 ← L31

T T M22 = L21L21 + L22L22

△ T T S22 = M22 − L21L21 = L22L22 ← L22

T T M32 = L31L21 + L32L22

△ T T S32 = M32 − L31L21 = L32L22 ← L32

T T T M33 = L31L31 + L32L32 + L33L33

△ T T T S33 = M33 − L31L31 − L32L32 = L33L33 ← L33

" Slide#16 !

Shu-Cherng Fang #

Advantages?

1. Reduced working space for memory requirements.

2. Highly parallelizable for multiprocessors.

Clock

L11

L21

L31

S22

S32

L22

L32

L33

" Slide#17 !

Shu-Cherng Fang #

General Case

Given an m×m dimensional symmetric positive deﬁnite matrix M partitioned into p2 sub-blocks :

 M11 M1p   . . . . .  M =  . . . . .         Mp1 Mpp 

such that m = pr, where r is referred to as the block size. The Cholesky factor of M can be partitioned accordingly as

 L11 0   . . .  L =  . .. .         Lp1 Lpp 

By directly equating LLT = M with block structure, we see that

T L11L11 = M11

T Li1L11 = Mi1, for i =2, ,p.

" Slide#18 !

Shu-Cherng Fang #

Moreover, by matrix multiplication, for p ≥ i ≥ j ≥ 2,

j T Mij = k=1 LikLjk

and hence

T j−1 T LijLjj = Mij − k=1 LikLjk

j−1 T If we denote Sij = Mij − k=1 LikLjk, then, for

p ≥ i ≥ j ≥ 2,Ljj is the Cholesky factor of Sjj and

T Lij is the solution of the matrix equation ZLjj = Sij.

Hence a block Cholesky factorization scheme is obtained:

" Slide#19 !

Shu-Cherng Fang #

Algorithm C-3

compute the Cholesky factor of M11 for L11 for i =2 to p

T solve ZL11 = Mi1 for Li1 end for j =2 to p for i = j to p

S ← Mij for k =1 to j − 1

T S ← S − LikLjk end if i = j

compute the Cholesky factor of S for Ljj else

T solve ZLjj = S for Lij end end end

" Slide#20 !

Shu-Cherng Fang #

Note that Algorithm C-3 may use Algorithm C-1 or C-2 to ﬁnd the Cholesky factor for block submatrices. Also note

T T that since Ljj is upper triangular, solving ZLjj = S is relatively simple. The recursive subroutines can be used here quite eﬃciently, if the compiler supports this feature. One key factor that aﬀects the performance block Cholesky factorization is the choice of the block size r, which often needs careful thinking and experimentation. The development of block Cholesky factorization algorithms and their implementations, especially on vector/parallel processors, is an active research area.

" Slide#21 !

Shu-Cherng Fang #

• LQ Factorization

Objective: Mu = v u = M −1v

Question # 1: Don’t want to invert M !

Idea: Consider a scalar case: Mu = v

If m =1 − x for 0 < x < 1

1 then m−1 = = 1 + x + x2 + 1 − x & u = (1+ x + )v

Similarly, if M = I − B

B : symmetric p.d. with all eigenvalues being positive & less than 1

∞ k 2 then k=0 B = I + B + B + converges, and

" Slide#22 !

Shu-Cherng Fang #

−1 −1 ∞ k M =(I − B) = k=0 B

Hence u = M −1v =[I + B + B2 + ]v

Question # 2:

2 T ? M = AXk A = I − B for some B ?

Idea: Consider a special case: (m

if A = Q is orthonormal

i.e., QQT = I

△ then Q = maxx=1Qx≤ 1

T △ T and Q = maxy=1Q y≤ 1

Moreover, let

△ k 2 λmax = maxi(xi ) > 0

△ k 2 λmin = mini(xi ) > 0

" Slide#23 !

Shu-Cherng Fang #

For any α>λmax, and

α − (xk)2 0 < x¯ = i < 1 i α

We have

2 T ¯ T QXk Q = α[I − QXQ ] B

where (1) B is symmetric p.d.

(2) all eigenvalues of B are positive

(3) if kmax is the largest e.v. then kmax = B = QXQ¯ T ≤QX¯ QT α − λ ≤ (1)( min)(1) < 1 α

" Slide#24 !

Shu-Cherng Fang #

In this case,

1 (QX2QT )−1 = (I − B)−1 k α

1 = (I + B + B2 + ) α Take ﬁnite term approximation

r k Say k=0 B

Then, matrix inversion becomes matrix multiplication O(n3) O(n2)

Question # 3: In general, A is not orthonormal.

Idea: If Am×n = Lm×mQm×n (orthonormal)

" Slide#25 !

Shu-Cherng Fang #

Then AAT =(LQ)(LQ)T = LQQT LT = LLT

L is the unique Cholesky

factor of AAT !!

So L−1 exists ! and

Min cT x Min cT x (P) s.t. Ax = b ⇒ s.t. LQx = b x ≥ 0 x ≥ 0

⇓

Min cT x Min cT x (P’) s.t. Qx = b′ ⇐ s.t. Qx = L−1b x ≥ 0 x ≥ 0

" Slide#26 !

Shu-Cherng Fang #

And, everything follows to a happy ending !

Question # 4:

A =? LQ

Problem # 1: Give a detailed proof of

Theorem 10.3 (Fundamental Theorem of LQ Factorization):

Let Am×n be a matrix with full row rank

for a lower triangular matrix Lm×m and

an orthonormal matrix Qm×n.

" Slide#27 !

Shu-Cherng Fang #

Problem # 2: Write a step by step procedure,

in detail, to perform LQ factorization for a given A.

Question # 5: A is sparse ⇒? Q is sparse

Problem # 3: Provide a simple example, say

1 ≤ m

relatively sparse, but Q is very dense.

Problem # 4: Given A is sparse, Q is not,

how do you deal with the

sparsity issue ?

" Slide#28 !

Shu-Cherng Fang LQ Factorization Theorem: Let A be an m×n matrix with full row rank (m ≤ n). Then A = LQ where L is a lower triangular m × m matrix with positive diagonals and Q is an m × n orthonormal matrix, i.e. QQT = I.

Proof: Since A is of full row rank, there exists the Cholesky factor L for AAT , i.e. AAT = LLT , where L is lower triangular m × m matrix with positive diagonals. Hence L−1 exists and let −1 Qm×n = Lm×mAm×n. Then QQT = (L−1A)(L−1A)T = L−1AAT (LT )−1 = L−1LLT (LT )−1 = I.

29 Backward and Forward Solve Problem: Solve Lx = b where L is a lower triangular matrix.

Method 1 (Inner-product Form)       l11 x1 b1  l l   x   b   21 22   2   2         l31 l32 l33   x3  =  b3   . ..   .   .   . .   .   .  ln1 ··· lnn xn bn

 b1 l11x1 = b1 ⇒ x1 =  l11  b2−l21x1  l21x1 + l22x2 = b2 ⇒ x2 =  l22 . .   h Pn−1 i  ln1x1 + ··· + lnnxn = bn ⇒ xn = bn − k=1 lnkxk /lnn

Sequence of Computation

30 Method 2 (Outer-product Form)       l11 x1 b1  l21 l22   x2  =  b2  l31 l32 l33 x3 b3

 b1 l11x1 = b1 ⇒ x1 =  l11  b2  l21x1 + l22x2 = b2 ⇒ b2 ← b2 − l21x1, x2 = l22  l31x1 + l32x2 + l33x3 = b3 ⇒ b3 ← b3 − l31x1  b3  b3 ← b3 − l32x2, x3 = l33

Sequence of Computation

For i = 1, 2, . . . , n

bi xi = lii       bi+1 bi+1 li+1,i . . .  .  ←  .  −  .  xi bn bn ln,i

31 Observations

1. Solution methods have to tie with Storage Scheme: (a) Inner-product form goes with row-by-row storage of matrix L. (b) Outer-product form goes with column-by-column storage of matrix L. 2. Outer-product form leads itself to exploiting sparsity in the solution of x. If bi turns out to be zero at the beginning of the i-th step, then xi = 0 and the entire step can be skipped. Storage Schemes

32 Ordering for factorization Basic Idea: Reorder the rows and columns of a given matrix M may preserve part of its structure in L.

Example: Solving

 1      4 1 2 2 2 x1 7  1 1 0 0 0   x   3   2   2           2 0 3 0 0   x3  =  7   1 0 0 5 0   x   −4   2 8   4    2 0 0 0 16 x5 −4   2  0.5 0.5 0      L =  1 −1 1     0.25 −0.25 −0.5 0.5  1 −1 −2 −3 1

33 Reordering the variables:

x1 → xe5 x2 → xe4 x3 → xe3 x4 → xe2 x5 → xe1

Solving       16 0 0 0 2 xe1 −4  0 5 0 0 1   x   −4   8 2   e2           0 0 3 0 2   xe3  =  7   0 0 0 1 1   x   3   2   e4    1 2 2 2 1 4 xe5 7   4  0 0.791 0      Le =  0 0 1.73     0 0 0 0707  0.500 0.632 1.15 1.41 0.129

34 More Interesting Example:

35 36 37 Question: which ordering result is the best?

1. Comparisons of ordering strategies must consider the storage scheme to be used. 2. Storage schemes for sparsity involve two components: primary storage and overhead storage.

38 Example:

39 Example: