MATH 532: Linear Algebra Chapter 5: Norms, Inner Products and Orthogonality Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Spring 2015 [email protected] MATH 532 1 Outline 1 Vector Norms 2 Matrix Norms 3 Inner Product Spaces 4 Orthogonal Vectors 5 Gram–Schmidt Orthogonalization & QR Factorization 6 Unitary and Orthogonal Matrices 7 Orthogonal Reduction 8 Complementary Subspaces 9 Orthogonal Decomposition 10 Singular Value Decomposition 11 Orthogonal Projections [email protected] MATH 532 2 Vector Norms Vector[0] Norms 1 Vector Norms 2 Matrix Norms Definition 3 Inner Product Spaces Let x; y 2 Rn (Cn). Then 4 Orthogonal Vectors n T X 5 Gram–Schmidt Orthogonalization & QRx Factorizationy = xi yi 2 R i=1 6 Unitary and Orthogonal Matrices n X ∗ = ¯ 2 7 Orthogonal Reduction x y xi yi C i=1 8 Complementary Subspaces is called the standard inner product for Rn (Cn). 9 Orthogonal Decomposition 10 Singular Value Decomposition 11 Orthogonal Projections [email protected] MATH 532 4 Vector Norms Definition Let V be a vector space. A function k · k : V! R≥0 is called a norm provided for any x; y 2 V and α 2 R 1 kxk ≥ 0 and kxk = 0 if and only if x = 0, 2 kαxk = jαj kxk, 3 kx + yk ≤ kxk + kyk. Remark The inequality in (3) is known as the triangle inequality. [email protected] MATH 532 5 Vector Norms Remark Any inner product h·; ·i induces a norm via (more later) p kxk = hx; xi: We will show that the standard inner product induces the Euclidean norm (cf. length of a vector). Remark Inner products let us define angles via xT y cos θ = : kxkkyk In particular, x; y are orthogonal if and only if xT y = 0. [email protected] MATH 532 6 Vector Norms Example Let x 2 Rn and consider the Euclidean norm p T kxk2 = x x n !1=2 X 2 = xi : i=1 We show that k · k2 is a norm. We do this for the real case, but the complex case goes analogously. 1 Clearly, kxk2 ≥ 0. Also, 2 kxk2 = 0 () kxk2 = 0 n X 2 () xi = 0 () xi = 0; i = 1;:::; n; i=1 () x = 0: [email protected] MATH 532 7 Vector Norms Example (cont.) 2 We have n !1=2 n !1=2 X 2 X 2 kαxk2 = (αxi ) = jαj xi = jαj kxk2: i=1 i=1 3 To establish (3) we need Lemma Let x; y 2 Rn. Then T jx yj ≤ kxk2kyk2: (Cauchy–Schwarz–Bunyakovsky) Moreover, equality holds if and only if y = αx with xT y α = : 2 kxk2 [email protected] MATH 532 8 Vector Norms Motivation for Proof of Cauchy–Schwarz–Bunyakovsky As already alluded to above, the angle θ between two vectors a and b is related to the inner product by aT b cos θ = : kakkbk Using trigonometry as in the figure, the projection of a onto b is then b aT b b aT b kak cos θ = kak = b: kbk kakkbk kbk kbk2 Now, we let y = a and x = b, so that the projection of y onto x is given by xT y αx; where α = : kxk2 [email protected] MATH 532 9 Vector Norms Proof of Cauchy–Schwarz–Bunyakovsky 2 We know that ky − αxk2 ≥ 0 since it’s (the square of) a norm. Therefore, 2 T 0 ≤ ky − αxk2 = (y − αx) (y − αx) = y T y − 2αxT y + α2xT x 2 xT y xT y = y T y − 2 xT y + xT x kxk2 kxk4 |{z} 2 =kxk2 2 xT y = k k2 − : y 2 2 kxk2 This implies 2 T 2 2 x y ≤ kxk2kyk2; and the Cauchy–Schwarz–Bunyakovsky inequality follows by taking square roots. [email protected] MATH 532 10 Vector Norms Proof (cont.) Now we look at the equality claim. T “=)”: Let’s assume that x y = kxk2kyk2. But then the first part of the proof shows that ky − αxk2 = 0 so that y = αx. “(=”: Let’s assume y = αx. Then T T 2 x y = x (αx) = jαjkxk2 2 kxk2kyk2 = kxk2kαxk2 = jαjkxk2; so that we have equality. [email protected] MATH 532 11 Vector Norms Example (cont.) 3 Now we can show that k · k2 satisfies the triangle inequality: 2 T kx + yk2 = (x + y) (x + y) = xT x +xT y + y T x + y T y |{z} |{z} |{z} =kxk2 = T 2 2 x y =kyk2 2 T 2 = kxk2 + 2x y + kyk2 2 T 2 ≤ kxk2 + 2jx yj + kyk2 CSB 2 2 ≤ kxk2 + 2kxk2kyk2 + kyk2 2 = (kxk2 + kyk2) : Now we just need to take square roots to have the triangle inequality. [email protected] MATH 532 12 Vector Norms Lemma Let x; y 2 Rn. Then we have the backward triangle inequality j kxk − kyk j ≤ kx − yk: Proof We write tri.ineq. kxk = kx − y + yk ≤ kx − yk + kyk: But this implies kxk − kyk ≤ kx − yk: [email protected] MATH 532 13 Vector Norms Proof (cont.) Switch the roles of x and y to get kyk − kxk ≤ ky − xk () − (kxk − kyk) ≤ kx − yk: Together with the previous inequality we have jkxk − kykj ≤ kx − yk: [email protected] MATH 532 14 Vector Norms Other common norms `1-norm (or taxi-cab norm, Manhattan norm): n X kxk1 = jxi j i=1 `1-norm (or maximum norm, Chebyshev norm): kxk1 = max jxi j 1≤i≤n `p-norm: n !1=p X p kxkp = jxi j i=1 Remark In the homework you will use Hölder’s and Minkowski’s inequalities to show that the p-norm is a norm. [email protected] MATH 532 15 Vector Norms Remark We now show that kxk1 = lim kxkp: p!1 Let’s use tildes to mark all components of x that are maximal, i.e.. x~1 = x~2 = ::: = x~k = max jxi j: 1≤i≤n The remaining components are then x~k+1;:::; x~n. This implies that x~ i < 1; for i = k + 1;:::; n: x~1 [email protected] MATH 532 16 Vector Norms Remark (cont.) Now n !1=p X p kxkp = jx~i j i=1 0 11=p B x~ p x~ pC ~ B k+1 n C = jx1j Bk + + ::: + C : @ x~1 x~1 A | {z } |{z} <1 <1 Since the terms inside the parentheses — except for k — go to 0 for p ! 1, (·)1=p ! 1 for p ! 1. And so kxkp ! jx~1j = max jxi j = kxk1: 1≤i≤n [email protected] MATH 532 17 Vector Norms 2 Figure: Unit “balls” in R for the `1, `2 and `1 norms. Note that B1 ⊆ B2 ⊆ B1 since, e.g., p ! p ! p ! p 2 p 2 q 2 2 p2 = 2; p2 = 1 + 1 = 1; p2 = ; 2 2 2 2 2 2 2 1 2 2 2 1 p ! p ! p ! 2 2 2 so that p2 ≥ p2 ≥ p2 : 2 2 2 2 1 2 2 2 1 In fact, we have in general (similar to HW) n kxk1 ≥ kxk2 ≥ kxk1; for any x 2 R : [email protected] MATH 532 18 Vector Norms Norm equivalence Definition Two norms k · k and k · k0 on a vector space V are called equivalent if there exist constants α; β such that kxk α ≤ ≤ β for all x(6= 0) 2 V: kxk0 Example k · k1 andp k · k2 are equivalent since from above kxk1 ≥ kxk2 and also kxk1 ≤ nkxk2 (see HW) so that kxk p α = 1 ≤ 1 ≤ n = β: kxk2 Remark In fact, all norms on finite-dimensional vector spaces are equivalent. [email protected] MATH 532 19 Matrix Norms Matrix[0] norms are special norms — they will satisfy one additional property1 Vector Norms. This property should help us measure kABk for two matrices A; B of appropriate2 Matrix Norms sizes. We3 lookInner Product at the Spaces simplest matrix norm, the Frobenius norm, defined for ; A 2 Rm n: 4 Orthogonal Vectors 1=2 0 m n 1 m !1=2 5 Gram–Schmidt OrthogonalizationX & QRX Factorization2 X 2 kAkF = @ jaij j A = kAi∗k2 6 Unitary and Orthogonal Matricesi=1 j=1 i=1 0 11=2 7 Orthogonal Reduction n q X 2 T = @ kA∗j k2A = trace(A A); 8 Complementary Subspaces j=1 9 Orthogonal Decomposition i.e., the Frobenius norm is just a 2-norm for the vector that contains all elements10 Singular Value of the Decomposition matrix. 11 Orthogonal Projections [email protected] MATH 532 21 Matrix Norms Now m 2 X 2 kAxk2 = jAi∗xj i=1 m CSB X 2 2 ≤ kAi∗k2 kxk2 i=1 | {z } 2 =kAkF so that kAxk2 ≤ kAkF kxk2: We can generalize this to matrices, i.e., we have kABkF ≤ kAkF kBkF ; which motivates us to require this submultiplicativity for any matrix norm. [email protected] MATH 532 22 Matrix Norms Definition A matrix norm is a function k · k from the set of all real (or complex) matrices of finite size into R≥0 that satisfies 1 kAk ≥ 0 and kAk = 0 if and only if A = O (a matrix of all zeros). 2 kαAk = jαjkAk for all α 2 R. 3 kA + Bk ≤ kAk + kBk (requires A; B to be of same size). 4 kABk ≤ kAkkBk (requires A; B to have appropriate sizes). Remark This definition is usually too general. In addition to the Frobenius norm, most useful matrix norms are induced by a vector norm. [email protected] MATH 532 23 Matrix Norms Induced matrix norms Theorem m n Let k · k(m) and k · k(n) be vector norms on R and R , respectively, and let A be an m × n matrix.