Introduction to Numerical Linear Algebra II
Petros Drineas These slides were prepared by Ilse Ipsen for the 2015 Gene Golub SIAM Summer School on RandNLA
1 / 49 Overview
We will cover this material in relatively more detail, but will still skip a lot ...
1 Norms {Measuring the length/mass of mathematical quantities} General norms Vector p norms Matrix norms induced by vector p norms Frobenius norm
2 Singular Value Decomposition (SVD) The most important tool in Numerical Linear Algebra
3 Least Squares problems Linear systems that do not have a solution
2 / 49 Norms General Norms
How to measure the mass of a matrix or length of a vector
m×n Norm k · k is function R → R with
1 Non-negativity kAk ≥ 0, kAk = 0 ⇐⇒ A = 0 2 Triangle inequality kA + Bk ≤ kAk + kBk 3 Scalar multiplication kα Ak = |α| kAk for all α ∈ R.
Properties Minus signs k − Ak = kAk Reverse triangle inequality | kAk − kBk | ≤ kA − Bk New norms m×n m×m For norm k · k on R , and nonsingular M ∈ R def kAkM = kMAk is also a norm
4 / 49 Vector p Norms
n For x ∈ R and integer p ≥ 1
n !1/p def X p kxkp = |xi | i=1
Pn One norm kxk1 = j=1 |xj | √ qPn 2 T Euclidean (two) norm kxk2 = j=1 |xj | = x x
Infinity (max) norm kxk∞ = max1≤j≤n |xj |
5 / 49 Exercises
n 1 Determine k11kp for 11 ∈ R and p = 1, 2, ∞
n 2 For x ∈ R with xj = j, 1 ≤ j ≤ n, determine closed-form expressions for kxkp for p = 1, 2, ∞
6 / 49 Inner Products and Norm Relations
n For x, y ∈ R Cauchy-Schwartz inequality
T |x y| ≤ kxk2 kyk2
H¨olderinequality
T T |x y| ≤ kxk1 kyk∞ |x y| ≤ kxk∞ kyk1
Relations
kxk∞ ≤ kxk1 ≤ n kxk∞ √ kxk2 ≤ kxk1 ≤ n kxk2 √ kxk∞ ≤ kxk2 ≤ n kxk∞
7 / 49 Exercises
1 Prove the norm relations on the previous slide
n p 2 For x ∈ R show kxk2 ≤ kxk∞ kxk1
8 / 49 2 T T T T 2 kV xk2 = (V x) (V x) = x V V x = x x = kxk2
Vector Two Norm
Theorem of Pythagoras n For x, y ∈ R
T 2 2 2 x y = 0 ⇐⇒ kx ± yk2 = kxk2 + kyk2
Two norm does not care about orthonormal matrices n m×n T For x ∈ R and V ∈ R with V V = In
kV xk2 = kxk2
9 / 49 Vector Two Norm
Theorem of Pythagoras n For x, y ∈ R
T 2 2 2 x y = 0 ⇐⇒ kx ± yk2 = kxk2 + kyk2
Two norm does not care about orthonormal matrices n m×n T For x ∈ R and V ∈ R with V V = In
kV xk2 = kxk2
2 T T T T 2 kV xk2 = (V x) (V x) = x V V x = x x = kxk2
9 / 49 Exercises
n For x, y ∈ R show 1 Parallelogram equality
2 2 2 2 kx + yk2 + kx − yk2 = 2 kxk2 + kyk2
2 Polarization identity 1 xT y = kx + yk2 − kx − yk2 4 2 2
10 / 49 Matrix Norms (Induced by Vector p Norms)
m×n For A ∈ R and integer p ≥ 1
def kAxkp kAkp = max = max kA ykp x6=0 kxkp kykp=1
One Norm: Maximum absolute column sum m X kAk1 = max |aij | = max kAe j k1 1≤j≤n 1≤j≤n i=1 Infinity Norm: Maximum absolute row sum
n X T kAk∞ = max |aij | = max kA ei k1 1≤i≤m 1≤i≤m j=1
11 / 49 Matrix Norm Properties
Every norm realized by some vector y 6= 0
kAykp y kAkp = = kAzkp where z ≡ kzkp = 1 kykp kykp
{Vector y is different for every A and every p}
Submultiplicativity m×n n Matrix vector product: For A ∈ R and y ∈ R
kAykp ≤ kAkpkykp
m×n n×l Matrix product: For A ∈ R and B ∈ R
kABkp ≤ kAkpkBkp
12 / 49 More Matrix Norm Properties
m×n m×m n×n For A ∈ R and permutation matrices P ∈ R ,Q ∈ R Permutation matrices do not matter
kPAQkp = kAkp
Submatrices have smaller norms than parent matrix BA12 If P A Q = then kBkp ≤ kAkp A21 A22
13 / 49 Exercises
1 Different norms realized by different vectors
Find y and z so that kAk1 = kAyk1 and kAk∞ = kAzk∞ when
1 4 A = 0 2
n×n 2 If Q ∈ R is permutation then kQkp = 1 3 Prove the two types of submultiplicativity 4 If D = diag d11 ··· dnn then
kDkp = max |djj | 1≤j≤n
n×n −1 5 If A ∈ R nonsingular then kAkpkA kp ≥ 1
14 / 49 Norm Relations and Transposes
m×n For A ∈ R Relations between different norms 1 √ √ kAk ≤ kAk ≤ m kAk n ∞ 2 ∞ 1 √ √ kAk ≤ kAk ≤ n kAk m 1 2 1
Transposes
T T T kA k1 = kAk∞ kA k∞ = kAk1 kA k2 = kAk2
15 / 49 Proof: Two Norm of Transpose
{True for A = 0, so assume A 6= 0}
n Let kAk2 = kAzk2 for some z ∈ R with kzk2 = 1 Reduce to vector norm
2 2 T T T T T kAk2 = kAzk2 = (Az) (Az) = z A Az = z A Az | {z } vector Cauchy-Schwartz inequality and submultiplicativity imply
T T T T z A Az ≤ kzk2 kA Azk2 ≤ kA k2 kAk2
2 T T Thus kAk2 ≤ kA k2 kAk2 and kAk2 ≤ kA k2
T T Reversing roles of A and A gives kA k2 ≤ kAk2
16 / 49 Matrix Two Norm
m×n A ∈ R Orthonormal matrices do not change anything k×m T l×n T If U ∈ R with U U = Im,V ∈ R with V V = In T kUAV k2 = kAk2
Gram matrices m×n For A ∈ R T 2 T kA Ak2 = kAk2 = kAA k2 Outer products m n For x ∈ R , y ∈ R T kxy k2 = kxk2 kyk2
17 / 49 Proof: Norm of Gram Matrix
Submultiplicativity and transpose
T T 2 kA Ak2 ≤ kAk2 kA k2 = kAk2 kAk2 = kAk2
T 2 Thus kA Ak2 ≤ kAk2
n Let kAk2 = kAzk2 for some z ∈ R with kzk2 = 1 Cauchy-Schwartz inequality and submultiplicativity imply
2 2 T T T kAk2 = kAzk2 = (Az) (Az) =z A A z | {z } vector T T ≤k zk2 kA A zk2 ≤ kA Ak2
2 T Thus kAk2 ≤ kA Ak2
18 / 49 Exercises
1 Infinity norm of outer products m n T For x ∈ R and y ∈ R , show kx y k∞ = kxk∞ kyk1
n×n 2 If A ∈ R with A 6= 0 is idempotent then
(i) kAkp ≥ 1
(ii) kAk2 = 1 if A also symmetric
n×n 3 Given A ∈ R 1 T Among all symmetric matrices, 2 (A + A ) is a (the?) matrix that is closest to A in the two norm
19 / 49 n Vector If x ∈ R then kxkF = kxk2
T Transpose kA kF = kAkF √ Identity kInkF = n
Frobenius Norm
The true mass of a matrix