Introduction to Numerical Linear Algebra II Petros Drineas These slides were prepared by Ilse Ipsen for the 2015 Gene Golub SIAM Summer School on RandNLA 1 / 49 Overview We will cover this material in relatively more detail, but will still skip a lot ... 1 Norms fMeasuring the length/mass of mathematical quantitiesg General norms Vector p norms Matrix norms induced by vector p norms Frobenius norm 2 Singular Value Decomposition (SVD) The most important tool in Numerical Linear Algebra 3 Least Squares problems Linear systems that do not have a solution 2 / 49 Norms General Norms How to measure the mass of a matrix or length of a vector m×n Norm k · k is function R ! R with 1 Non-negativity kAk ≥ 0, kAk = 0 () A = 0 2 Triangle inequality kA + Bk ≤ kAk + kBk 3 Scalar multiplication kα Ak = jαj kAk for all α 2 R. Properties Minus signs k − Ak = kAk Reverse triangle inequality j kAk − kBk j ≤ kA − Bk New norms m×n m×m For norm k · k on R , and nonsingular M 2 R def kAkM = kMAk is also a norm 4 / 49 Vector p Norms n For x 2 R and integer p ≥ 1 n !1=p def X p kxkp = jxi j i=1 Pn One norm kxk1 = j=1 jxj j p qPn 2 T Euclidean (two) norm kxk2 = j=1 jxj j = x x Infinity (max) norm kxk1 = max1≤j≤n jxj j 5 / 49 Exercises n 1 Determine k11kp for 11 2 R and p = 1; 2; 1 n 2 For x 2 R with xj = j, 1 ≤ j ≤ n, determine closed-form expressions for kxkp for p = 1; 2; 1 6 / 49 Inner Products and Norm Relations n For x; y 2 R Cauchy-Schwartz inequality T jx yj ≤ kxk2 kyk2 H¨olderinequality T T jx yj ≤ kxk1 kyk1 jx yj ≤ kxk1 kyk1 Relations kxk1 ≤ kxk1 ≤ n kxk1 p kxk2 ≤ kxk1 ≤ n kxk2 p kxk1 ≤ kxk2 ≤ n kxk1 7 / 49 Exercises 1 Prove the norm relations on the previous slide n p 2 For x 2 R show kxk2 ≤ kxk1 kxk1 8 / 49 2 T T T T 2 kV xk2 = (V x) (V x) = x V V x = x x = kxk2 Vector Two Norm Theorem of Pythagoras n For x; y 2 R T 2 2 2 x y = 0 () kx ± yk2 = kxk2 + kyk2 Two norm does not care about orthonormal matrices n m×n T For x 2 R and V 2 R with V V = In kV xk2 = kxk2 9 / 49 Vector Two Norm Theorem of Pythagoras n For x; y 2 R T 2 2 2 x y = 0 () kx ± yk2 = kxk2 + kyk2 Two norm does not care about orthonormal matrices n m×n T For x 2 R and V 2 R with V V = In kV xk2 = kxk2 2 T T T T 2 kV xk2 = (V x) (V x) = x V V x = x x = kxk2 9 / 49 Exercises n For x; y 2 R show 1 Parallelogram equality 2 2 2 2 kx + yk2 + kx − yk2 = 2 kxk2 + kyk2 2 Polarization identity 1 xT y = kx + yk2 − kx − yk2 4 2 2 10 / 49 Matrix Norms (Induced by Vector p Norms) m×n For A 2 R and integer p ≥ 1 def kAxkp kAkp = max = max kA ykp x6=0 kxkp kykp=1 One Norm: Maximum absolute column sum m X kAk1 = max jaij j = max kAe j k1 1≤j≤n 1≤j≤n i=1 Infinity Norm: Maximum absolute row sum n X T kAk1 = max jaij j = max kA ei k1 1≤i≤m 1≤i≤m j=1 11 / 49 Matrix Norm Properties Every norm realized by some vector y 6= 0 kAykp y kAkp = = kAzkp where z ≡ kzkp = 1 kykp kykp fVector y is different for every A and every pg Submultiplicativity m×n n Matrix vector product: For A 2 R and y 2 R kAykp ≤ kAkpkykp m×n n×l Matrix product: For A 2 R and B 2 R kABkp ≤ kAkpkBkp 12 / 49 More Matrix Norm Properties m×n m×m n×n For A 2 R and permutation matrices P 2 R ,Q 2 R Permutation matrices do not matter kPAQkp = kAkp Submatrices have smaller norms than parent matrix BA12 If P A Q = then kBkp ≤ kAkp A21 A22 13 / 49 Exercises 1 Different norms realized by different vectors Find y and z so that kAk1 = kAyk1 and kAk1 = kAzk1 when 1 4 A = 0 2 n×n 2 If Q 2 R is permutation then kQkp = 1 3 Prove the two types of submultiplicativity 4 If D = diag d11 ··· dnn then kDkp = max jdjj j 1≤j≤n n×n −1 5 If A 2 R nonsingular then kAkpkA kp ≥ 1 14 / 49 Norm Relations and Transposes m×n For A 2 R Relations between different norms 1 p p kAk ≤ kAk ≤ m kAk n 1 2 1 1 p p kAk ≤ kAk ≤ n kAk m 1 2 1 Transposes T T T kA k1 = kAk1 kA k1 = kAk1 kA k2 = kAk2 15 / 49 Proof: Two Norm of Transpose fTrue for A = 0, so assume A 6= 0g n Let kAk2 = kAzk2 for some z 2 R with kzk2 = 1 Reduce to vector norm 2 2 T T T T T kAk2 = kAzk2 = (Az) (Az) = z A Az = z A Az | {z } vector Cauchy-Schwartz inequality and submultiplicativity imply T T T T z A Az ≤ kzk2 kA Azk2 ≤ kA k2 kAk2 2 T T Thus kAk2 ≤ kA k2 kAk2 and kAk2 ≤ kA k2 T T Reversing roles of A and A gives kA k2 ≤ kAk2 16 / 49 Matrix Two Norm m×n A 2 R Orthonormal matrices do not change anything k×m T l×n T If U 2 R with U U = Im,V 2 R with V V = In T kUAV k2 = kAk2 Gram matrices m×n For A 2 R T 2 T kA Ak2 = kAk2 = kAA k2 Outer products m n For x 2 R , y 2 R T kxy k2 = kxk2 kyk2 17 / 49 Proof: Norm of Gram Matrix Submultiplicativity and transpose T T 2 kA Ak2 ≤ kAk2 kA k2 = kAk2 kAk2 = kAk2 T 2 Thus kA Ak2 ≤ kAk2 n Let kAk2 = kAzk2 for some z 2 R with kzk2 = 1 Cauchy-Schwartz inequality and submultiplicativity imply 2 2 T T T kAk2 = kAzk2 = (Az) (Az) =z A A z | {z } vector T T ≤k zk2 kA A zk2 ≤ kA Ak2 2 T Thus kAk2 ≤ kA Ak2 18 / 49 Exercises 1 Infinity norm of outer products m n T For x 2 R and y 2 R , show kx y k1 = kxk1 kyk1 n×n 2 If A 2 R with A 6= 0 is idempotent then (i) kAkp ≥ 1 (ii) kAk2 = 1 if A also symmetric n×n 3 Given A 2 R 1 T Among all symmetric matrices, 2 (A + A ) is a (the?) matrix that is closest to A in the two norm 19 / 49 n Vector If x 2 R then kxkF = kxk2 T Transpose kA kF = kAkF p Identity kInkF = n Frobenius Norm The true mass of a matrix m×n For A = a1 ··· an 2 R v v u n u n m q def uX 2 uX X 2 T kAkF = t kaj k2 = t jaij j = trace(A A) j=1 j=1 i=1 20 / 49 Frobenius Norm The true mass of a matrix m×n For A = a1 ··· an 2 R v v u n u n m q def uX 2 uX X 2 T kAkF = t kaj k2 = t jaij j = trace(A A) j=1 j=1 i=1 n Vector If x 2 R then kxkF = kxk2 T Transpose kA kF = kAkF p Identity kInkF = n 20 / 49 More Frobenius Norm Properties m×n A 2 R Orthonormal invariance k×m T l×n T If U 2 R with U U = Im,V 2 R with V V = In T kUAV kF = kAkF Relation to two norm p p kAk2 ≤ kAkF ≤ rank(A) kAk2 ≤ minfm; ng kAk2 Submultiplicativity kABkF ≤k Ak2 kBkF ≤ kAkF kBkF 21 / 49 Exercises 1 Show the orthonormal invariance of the Frobenius norm 2 Show the submultiplicativity of the Frobenius norm 3 Frobenius norm of outer products m n For x 2 R and y 2 R show T kx y kF = kxk2 kyk2 22 / 49 Singular Value Decomposition (SVD) Full SVD m×n Given: A 2 R Tall and skinny: m ≥ n 0 1 σ1 Σ A = U VT where Σ ≡ B .. C ≥ 0 0 @ . A σn Short and fat: m ≤ n 0 1 σ1 T B .. C A = U Σ 0 V where Σ ≡ @ . A ≥ 0 σm m×m n×n U 2 R and V 2 R are orthogonal matrices 24 / 49 Names and Conventions m×n A 2 R with SVD Σ A = U VT or A = U Σ 0 VT 0 Singular values (svalues): Diagonal elements σj of Σ Left singular vector matrix:U Right singular vector matrix:V Svalue ordering σ1 ≥ σ2 ≥ ::: ≥ σminfm; ng ≥ 0 Svalues of matrices B, C: σj (B), σj (C) 25 / 49 SVD Properties m×n A 2 R Number of svalues equal to small dimension m×n A 2 R has minfm; ng singular values σj ≥ 0 Orthogonal invariance m×m n×n If P 2 R and Q 2 R are orthogonal matrices then P A Q has same svalues as A Gram Product Nonzero svalues (= eigenvalues) of AT A are squares of svalues of A Inverse n×n A 2 R nonsingular () σj > 0 for 1 ≤ j ≤ n If A = U Σ VT then A−1 = V Σ−1 UT SVD of inverse 26 / 49 Exercises 1 TransposeA T has same singular values as A 2 Orthogonal matrices n×n All singular values of A 2 R are equal to 1 () A is orthogonal matrix n×n 3 If A 2 R is symmetric and idempotent then all singular values are 0 and/or 1 m×n 4 For A 2 R and α > 0 express svalues of (AT A + α I)−1 AT in terms of α and svalues of A In 5 If A 2 m×n with m ≥ n then singular values of are R A q 2 equal to 1 + σj for 1 ≤ j ≤ n 27 / 49 Singular Values m×n A 2 R with svalues σ1 ≥ · · · ≥ σp p ≡ minfm; ng Two norm kAk2 = σ1 q 2 2 Frobenius norm kAkF = σ1 + ··· + σp Well conditioned in absolute sense jσj (A) − σj (B)j ≤ kA − Bk2 1 ≤ j ≤ p Product σj (A B) ≤ σ1(A) σj (B) 1 ≤ j ≤ p 28 / 49 Matrix Schatten Norms m×n A 2 R , with singular values σ1 ≥ · · · ≥ σρ > 0, and integer p ≥ 0, the family of the Schatten p-norms is defined as ρ !1=p def X p kAkp = σi : i=1 Different than the vector-induced matrix p-norms1.