396 Chapter 7. Applied Mathematics and ATA 7.2 Positive Definite Matrices and the SVD This chapter about applications of ATA depends on two important ideas in linear algebra. These ideas have big parts to play, we focus on them now. 1. Positive definite symmetric matrices (both ATA and ATCA are positive definite) 2. Singular Value Decomposition (A U†V T gives perfect bases for the 4 subspaces) D Those are orthogonal matrices U and V in the SVD. Their columns are orthonormal eigenvectors of AAT and ATA. The entries in the diagonal matrix † are the square roots of the eigenvalues. The matrices AAT and ATA have the same nonzero eigenvalues. Section 6.5 showed that the eigenvectors of these symmetric matrices are orthogonal. I will show now that the eigenvalues of ATA are positive, if A has independent columns. Start with ATAx x. Then xTATAx xTx. Therefore Ax 2= x 2 >0 D D D jj jj jj jj I separated xTATAx into .Ax/T.Ax/ Ax 2. We don’t have 0 because ATA is invertible (since A has independent columns).D jj Thejj eigenvalues must beD positive. Those are the key steps to understanding positive definite matrices. They give us three tests on S—three ways to recognize when a symmetric matrix S is positive definite : Positive 1. All the eigenvalues of S are positive. definite 2. The “energy” xTSx is positive for all nonzero vectors x. symmetric 3. S has the form S ATA with independent columns in A. D There is also a test on the pivots .all > 0/ and a test on n determinants .all > 0/. Example 1 Are these matrices positive definite ? When their eigenvalues are positive, construct matrices A with S ATA and find the positive energy xTSx. D 4 0 5 4 4 5 (a) S (b) S (c) S D 0 1 D 4 5 D 5 4 Solution The answers are yes, yes, and no. The eigenvalues of those matrices S are (a) 4 and 1 : positive (b) 9 and 1 : positive (c) 9 and 1 : not positive. A quicker test than eigenvalues uses two determinants : the 1 by 1 determinant S11 and the 2 by 2 determinant of S. Example (b) has S11 5 and det S 25 16 9 (pass). D D D Example (c) has S11 4 but det S 16 25 9 (fail the test). D D D 7.2. Positive Definite Matrices and the SVD 397 Positive energy is equivalent to positive eigenvalues, when S is symmetric. Let me test the energy xTSx in all three examples. Two examples pass and the third fails : 4 0 x1 2 2 Œx1 x2 4x1 x2 >0 Positive energy when x 0 0 1 x2 D C ¤ 5 4 x1 2 2 Œx1 x2 5x1 8x1x2 5x2 Positive energy when x 0 4 5 x2 D C C ¤ 4 5 x1 2 2 Œx1 x2 4x1 10x1x2 4x2 Energy 2 when x .1; 1/ 5 4 x2 D C C D Positive energy is a fundamentalproperty. This is the best definition of positive definiteness. When the eigenvalues are positive, there will be many matrices A that give ATA S. One choice of A is symmetric and positive definite ! Then ATA is A2, and this choiceD A pS is a true square root of S. The successful examples (a) and (b) have S A2 : D D 4 0 2 0 2 0 5 4 2 1 2 1 and 0 1 D 0 1 0 1 4 5 D 1 2 1 2 We know that all symmetric matrices have the form S VƒV T with orthonormal eigenvectors in V . The diagonal matrix ƒ has a square root pDƒ, when all eigenvalues are positive. In this case A pS V pƒV T is the symmetric positive definite square root : D D ATA pSpS .V pƒV T/.V pƒV T/ V pƒpƒV T S because V TV I: D D D D D Starting from this unique square root pS, other choices of A come easily. Multiply pS by any matrix Q that has orthonormal columns (so that QTQ I ). Then QpS is another choice for A (not a symmetric choice). In fact all choices comeD this way : ATA .QpS/T .QpS/ pSQTQpS S: (1) D D D I will choose a particular Q in Example 1, to get particular choices of A. 0 1 Example 1 (continued) Choose Q to multiply pS. Then A QpS. D 1 0 D 0 1 2 0 0 1 4 0 A has S ATA D 1 0 0 1 D 2 0 D D 0 1 0 1 2 1 1 2 5 4 A has S ATA : D 1 0 1 2 D 2 1 D D 4 5 398 Chapter 7. Applied Mathematics and ATA Positive Semidefinite Matrices Positive semidefinite matrices include positive definite matrices, and more. Eigenvalues of S can be zero. Columns of A can be dependent. The energy xTSx can be zero—but not negative. This gives new equivalent conditions on a (possibly singular) matrix S S T. D 1 All eigenvalues of S satisfy 0 (semidefinite allows zero eigenvalues). 0 2 The energy is nonnegative for every x : xTSx 0 (zero energy is allowed). 0 T 30 S has the form A A (every A is allowed; its columns can be dependent). Example 2 The first two matrices are singular and positive semidefinite—but not the third : 0 0 4 4 4 4 (d) S (e) S (f) S . D 0 1 D 4 4 D 4 4 T 2 2 The eigenvalues are 1;0 and 8;0 and 8;0. The energies x Sx are x2 and 4.x1 x2/ 2 C and 4.x1 x2/ . So the third matrix is actually negative semidefinite. Singular Value Decomposition Now we start with A, square or rectangular. Applications also start this way—the matrix comes from the model. The SVD splits any matrix into orthogonal U times diagonal † times orthogonal V T . Those orthogonal factors will give orthogonal bases for the four fundamental subspaces associated with A. Let me describe the goal for any m by n matrix, and then how to achieve that goal. n m Find orthonormal bases v1;:::; vn for R and u1;:::; um for R so that Av1 1u1 : : : Avr r ur Avr 1 0 : : : Avn 0 (2) D D C D D The rankof A is r. Those requirementsin (4) are expressed by a multiplication AV U †. D The r nonzero singular values 1 2 : : : r >0 are on the diagonal of † : 1 0 : :: AV U † A v1 : : : vr : : : vn u1 : : : ur : : : um 2 3 (3) D 2 3 D 2 3 r 6 0 0 7 4 5 4 5 6 7 4 5 The last n r vectors in V are a basis for the nullspace of A. The last m r vectors in U are a basis for the nullspace of AT. The diagonal matrix † is m by n, withr nonzeros. 1 T n Remember that V V , because the columns v1;:::; vn are orthonormal in R : D Singular Value Decomposition AV U † becomes A U †V T : (4) D D 7.2. Positive Definite Matrices and the SVD 399 The SVD has orthogonal matrices U and V , containing eigenvectors of AAT and AT A. Comment. A square matrix is diagonalized by its eigenvectors : Axi i xi is like D Avi i ui . But even if A has n eigenvectors, they may not be orthogonal. We need two basesD—an input basis of v’s in Rn and an output basis of u’s in Rm. With two bases, any m by n matrix can be diagonalized. The beauty of those bases is that they can be chosen orthonormal. Then U TU I and V TV I . D D The v’s are eigenvectors of the symmetric matrix S ATA. We can guarantee their vTv D orthogonality, so that j i 0 for j i. That matrix S is positive semidefinite, so its 2 D ¤ eigenvalues are 0. The key to the SVD is that Avj is orthogonal to Avi : i 2 T T T T 2 i if j i Orthogonal u’s .Avj / .Avi / v .A Avi / v . vi / D (5) D j D j i D 0 if j i ¤ This says that the vectors ui Avi =i are orthonormal for i 1;:::;r. They are a basis for the column space of A.D And the u’s are eigenvectors ofD the symmetric matrix AAT, which is usually different from S ATA (but the eigenvalues 2;:::; 2 are the same). D 1 r Example 3 Find the input and output eigenvectors v and u for the rectangular matrix A : 2 2 0 A U†V T: D 1 1 0 D T 2 Solution Compute S A A and its unit eigenvectors v1; v2; v3. The eigenvalues D are 8;2;0 so the positive singular values are 1 p8 and 2 p2 : D D 5 3 0 p2 p2 0 T 1 1 A A 3 5 0 has v1 p2 ; v2 p2 ; v3 0 : D 2 3 D 2 2 3 D 2 2 3 D 2 3 0 0 0 0 0 1 4 5 4 5 4 5 4 5 The outputs u1 Av1=1 and u2 Av2=2 are also orthonormal, with 1 p8 and D D D 2 p2. Those vectors u1 and u2 are in the column space of A : D 2 2 0 v1 1 2 2 0 v2 0 u1 and u2 : D 1 1 0 p D 0 D 1 1 0 p D 1 8 2 Then U I and the Singular Value Decomposition for this 2 by 3 matrix is U†V T : D T p2 p2 0 2 2 0 1 0 p8 0 0 1 A p2 p2 0 : D 1 1 0 D 0 1 0 p2 0 2 2 3 " # 0 02 4 5 400 Chapter 7.