DEPARTMENT OF MATHEMATICS TECHNICAL REPORT SINGULAR VALUE DECOMPOSITION Andrew Lounsbury September 2018 No. 2018-2 TENNESSEE TECHNOLOGICAL UNIVERSITY Cookeville, TN 38505 Singular Value Decomposition∗ Andrew Lounsbury Department of Mathematics Tennessee Technological University Cookeville, TN, 38505 [email protected] September 28, 2018 Abstract The Singular Value Decomposition (SVD) provides a cohesive summary of a handful of topics introduced in basic linear algebra. SVD may be applied to digital photographs so that they may be approximated and transmitted with a concise computation. Mathematics Subject Classification. Primary 15A23, 15A24 Keywords. singular value, matrix factorization Contents 1 Introduction 1 2 Steps for Calculation of SVD 3 3 Theory 5 4 Examples 9 5 Maple 12 6 Conclusions 16 A Appendix 16 1 Introduction This paper begins with a definition of SVD and instructions on how to compute it, which includes calculating eigenvalues, singular values, eigenvectors, left and right singular vectors, or, alterna- tively, orthonormal bases for the four fundamental spaces of a matrix. We present two theorems ∗Completed in partial fulfillment of the requirements for MATH 4993 Mathematical Research (3 cr) in spring 2018 under supervision of Dr. R. Ablamowicz. 1 that result from SVD with corresponding proofs. We provide examples of matrices and their sin- gular value decompositions. There is also a section involving Maple that includes examples of photographs. It is demonstrated how the inclusion of more and more information from the SVD allows one to construct accurate approximations of a color image. Definition 1. Let A be an m × n real matrix of rank r ≤ min(m; n).A Singular Value Decom- position (SVD) is a way to factor A as A = UΣV T ; (1) T T where U and V are orthogonal matrices such that U U = Im and V V = In. The Σ matrix contains the singular values of A on its pseudo-diagonal, with zeros elsewhere. Thus, 2 3 σ1 0 :::::: 0 ::: 0 2 T 3 6 .. .7 v1 6 0 . .7 6 . 7 6 vT 7 T 6 . 7 6 2 7 A = UΣV = u1 u2 ··· um 6 . σr . 7 6 7; (2) 6 . 7 6 . 7 | {z } 6 . .7 4 . 5 U(m×m) 6 0 0 . .7 vT 4 . 5 n 0 0 ::: .. 0 ::: 0 | {z } T | {z } V (n×n) Σ(m×n) with u1; : : : ; um being the orthonormal columns of U; σ1; : : : ; σr being the singular values of A T satisfying σ1 ≥ σ2 ≥ · · · ≥ σr > 0, and v1; : : : ; vn being the orthonormal columns of V : Singular values are defined as the positive square roots of the eigenvalues of AT A: T 2 Note that since A A of size n × n is real and symmetric of rank r, r of its eigenvalues σi ; i = 1; : : : ; r; are positive and therefore real, while the remaining n − r eigenvalues are zero. In particular, T T T T 2 T A A = V (Σ Σ)V ;A Avi = σi vi; i = 1; : : : ; r; A Avi = 0; i = r + 1; : : : ; n: (3) T 2 Thus, the first r vectors vi are the eigenvectors of A A with the eigenvalues σi : Likewise, we have T T T T 2 T AA = U(ΣΣ )U ; AA ui = σi ui; i = 1; : : : ; r; AA ui = 0; i = r + 1; : : : ; m: (4) T 2 Thus, the first r vectors ui are the eigenvectors of AA with the eigenvalues σi : Furthermore, it can be shown (see Lemmas 1 and 2) that Avi = σiui; i = 1; : : : ; r; and Avi = 0; i = r + 1; : : : ; n: (5) If rank(A) = r < min(m; n), then there are n − r zero columns and rows in Σ; rendering the 2 r + 1; : : : ; m columns of U and r + 1; : : : ; n rows in V T unnecessary to recover A: Therefore, 2 3 2 T 3 σ1 0 :::::: 0 ::: 0 v1 6 .. .7 6 . 7 6 0 . .7 6 . 7 6 . 7 6 T 7 T 6 . 7 6 vr 7 A = UΣV = u1 ··· ur ur+1 ··· um 6 . σr . 7 6 7 6 . 7 6 vT 7 6 . .7 6 r+1 7 0 0 . 6 7 6 . 7 4 . 5 4 5 .. T 0 0 ::: 0 ::: 0 vn 2 3 σ1 0 ::: 0 2 vT 3 6 . 7 1 6 0 σ2 . 7 . = u1 ··· ur 6 7 6 . 7 6 . .. 7 4 5 | {z } . T m×r 4 5 vr 0 : : : σr | {z } | {z } r×n r×r = u1σ1v1 + ··· + urσrvr: (6) 2 Steps for Calculation of SVD Here, we provide an algorithm to calculate a singular value decomposition of a matrix. 1. Compute AT A of a real m × n matrix A of rank r. 2. Compute the singular values of AT A. T T Solve the characteristic equation ∆AT A(λ) = jA A − λIj = 0 of A A for the eigenvalues T λ1; : : : ; λr of A A. These eigenvalues will be positive. Take their square roots to obtain σ1; : : : ; σr which are the singular values of A, that is, p σi = + λi; i = 1; : : : ; r: (7) 3. Sort the singular values, possibly renaming them, so that σ1 ≥ σ2 ≥ · · · ≥ σr. 4. Construct the Σ matrix of size m × n such that Σii = σi for i = 1; : : : ; r; and Σij = 0 when i 6= j: 5. Compute the eigenvectors of AT A. T T Find a basis for Null(A A − λiI): That is, solve (A A − λiI)si = 0 for si, an eigenvector T of A corresponding to λi; for each eigenvalue λi. Since A A is symmetric, its eigenvectors corresponding to different eigenvalues are already orthogonal (but likely not orthonormal). See Lemma 1. 6. Compute the (right singular) vectors v1; : : : ; vr by normalizing each eigenvector si by multi- plying it by 1 : That is, let jjsijj 1 vi = si; i = 1; : : : ; r: (8) jjsijj 3 If n > r, the additional n − r vectors vr+1; : : : ; vn need to be chosen as an orthonormal basis in Null(A): Note that since Avi = σiui for i = 1;:::; vectors v1; : : : ; vr provide an orthonormal basis for Row(A) while the vectors u1; : : : ; ur provide an orthonormal basis for Col(A): In particular, n R = Row(A) ? Null(A) = spanfv1; : : : ; vrg ? spanfvr+1; : : : ; vr+(n−r)g: (9) 7. Construct the orthogonal matrix V = [v1j · · · jvn]. 8. Verify V T V = I. 9. Compute the (left singular) vectors u1; : : : ; ur as Avi Avi = σiui =) ui = ; i = 1 : : : r: (10) σi In this method, u1; : : : ; ur are orthogonal by Lemma 5. Alternatively, (i) Note that AAT = U(ΣΣT )U T suggests the vectors of U can be calculated as the eigen- vectors of AAT . In using this method, the vectors need to be normalized first. Namely, 1 T ui = si, where si is an eigenvector of AA : jjsijj (ii) Since ∆AT A(λ) = ∆AAT (λ) by Lemma 8, σ1; : : : ; σr are also the square roots of the eigenvalues of AAT . If m > r, the additional m − r vectors ur+1; : : : ; um need to be chosen as an orthonormal T basis in Null(A ): Note that since Avi = σiui for i = 1;:::; vectors u1; : : : ; ur provide an orthonormal basis for Col(A) while the vectors ur+1; : : : ; um provide an orthonormal basis for the left null space Null(AT ): In particular, m T R = Col(A) ? Null(A ) = spanfu1; : : : ; urg ? spanfur+1; : : : ; ur+(m−r)g: (11) 10. Construct U = [u1j · · · jum]. 11. Verify U T U = I. 12. Verify A = UΣV T . 13. Construct the dyadic decomposition1 of A, as described in Thm. 13: T T T T A = UΣV = σ1u1v1 + σ2u2v2 + ··· + urσrvr : (12) 1 T A dyad is a product of an n × 1 column vector with another 1 × n row vector, e.g., u1v1 ; resulting in a square n × n matrix whose rank is 1 by Lemma 6. 4 3 Theory In this section, we provide the two theorems related to SVD along with their proofs. Theorem 1. Let A = UΣV T be a singular value decomposition of an m × n real matrix of rank r. Then, 1. AV = UΣ and Av = σ u ; i = 1; : : : ; r Row(A) = spanfv ; : : : ; v g i i i =) 1 r Avi = 0; i = r + 1; : : : ; r + (n − r) Null(A) = spanfvr+1; : : : ; vr+(n−r)g T T T n n 2. A A = V (Σ Σ)V : R ! R 3. AT AV = V (ΣT Σ) and T 2 T A Avi = σi vi; i = 1; : : : ; r Row(A A) = spanfv1; : : : ; vrg T =) T A Avi = 0; i = r + 1; : : : ; r + (n − r) Null(A A) = spanfvr+1; : : : ; vr+(n−r)g 4. U T A = ΣV T and T T ui A = σivi ; i = 1; : : : ; r Col(A) = spanfu1; : : : ; urg T =) T ui A = 0; i = r + 1; : : : ; r + (m − r) Null(A ) = spanfur+1; : : : ; ur+(m−r)g T T T m m 5. AA = U(ΣΣ )U : R ! R 6. AAT U = U(ΣΣT ) and T 2 T AA ui = σi ui; i = 1; : : : ; r Row(AA ) = spanfu1; : : : ; urg T =) T AA ui = 0; i = r + 1; : : : ; r + (m − r) Null(AA ) = spanfur+1; : : : ; ur+(m−r)g Proof of (1). AV = (UΣV T )V = UΣ(V T V ) = UΣ So, AV = Av1 ··· Avr Avr+1 ··· Avn 2 3 σ1 0 :::::: 0 ::: 0 6 .. .7 6 0 . .7 6 7 6 . 7 = u1 ··· ur ur+1 ··· um 6 . σr . 7 6 . 7 6 . .7 6 0 0 . .7 4 . 5 0 0 ::: .. 0 ::: 0 = σ1u1 ··· σrur 0 ··· 0 : Hence, 5 1. Av1 = σ1u1; : : : ; Avr = σrur, and 2. Avr+1 = 0; : : : ; Avr+(n−r) = 0. Proof of (2). AT A = (UΣV T )T (UΣV T ) = (V ΣT U T )(UΣV T ) = V ΣT (U T U)ΣV T = V ΣT ΣV T Proof of (3).