Chapter 6 Vector Norms and Matrix Norms 6.1 Normed Vector Spaces In order to define how close two vectors or two matrices are, and in order to define the convergence of sequences of vectors or matrices, we can use the notion of a norm. Recall that R+ = x R x 0 . { 2 | ≥ } Also recall that if z = a + ib C is a complex number, 2 with a, b R,thenz = a ib and z = pa2 + b2 ( z is the2modulus of z). − | | | | 397 398 CHAPTER 6. VECTOR NORMS AND MATRIX NORMS Definition 6.1. Let E be a vector space over a field K, where K is either the field R of reals, or the field C of com- plex numbers. A norm on E is a function : E R+, assigning a nonnegative real number u kkto any! vector u E,andsatisfyingthefollowingconditionsforallk k x, y,2 z E: 2 (N1) x 0, and x =0i↵x =0. (positivity) k k k k (N2) λx = λ x . (homogeneity (or scaling)) k k | |k k (N3) x + y x + y . (triangle inequality) k kk k k k AvectorspaceE together with a norm is called a normed vector space. kk From (N2) we get x = x , k k k k and from (N3), we get x y x y . |k kk k| k − k 6.1. NORMED VECTOR SPACES 399 Example 6.1. 1. Let E = R,and x = x ,theabsolutevalueofx. k k | | 2. Let E = C,and z = z ,themodulusofz. k k | | 3. Let E = Rn (or E = Cn). There are three standard norms. For every (x1,...,xn) E,wehavethe1-norm x ,definedsuchthat,2 k k1 x = x + + x , k k1 | 1| ··· | n| we have the Euclidean norm x ,definedsuchthat, k k2 1 x = x 2 + + x 2 2 , k k2 | 1| ··· | n| and the sup-norm x ,definedsuchthat, k k1 x =max xi 1 i n . k k1 {| || } More generally, we define the ` -norm (for p 1) by p ≥ x =(x p + + x p)1/p. k kp | 1| ··· | n| There are other norms besides the `p-norms; we urge the reader to find such norms. 400 CHAPTER 6. VECTOR NORMS AND MATRIX NORMS Some work is required to show the triangle inequality for the `p-norm. Proposition 6.1. If E is a finite-dimensional vector space over R or C, for every real number p 1, the ≥ `p-norm is indeed a norm. The proof uses the following facts: If q 1isgivenby ≥ 1 1 + =1, p q then (1) For all ↵,β R,if↵,β 0, then 2 ≥ ↵p βq ↵ + . ( ) p q ⇤ (2) For any two vectors u, v E,wehave 2 n u v u v . ( ) | i i|k kp k kq ⇤⇤ i=1 X 6.1. NORMED VECTOR SPACES 401 For p>1and1/p +1/q =1,theinequality n n 1/p n 1/q u v u p v q | i i| | i| | i| i=1 i=1 i=1 X ✓ X ◆ ✓ X ◆ is known as H¨older’s inequality. For p =2,itistheCauchy–Schwarz inequality. Actually, if we define the Hermitian inner product , h i on Cn by n u, v = u v , h i i i i=1 X where u =(u1,...,un)andv =(v1,...,vn), then n n u, v u v = u v , |h i| | i i| | i i| i=1 i=1 X X so H¨older’s inequality implies the inequality u, v u v |h i| k kp k kq also called H¨older’s inequality,which,forp =2isthe standard Cauchy–Schwarz inequality. 402 CHAPTER 6. VECTOR NORMS AND MATRIX NORMS The triangle inequality for the `p-norm, n 1/p n 1/p n 1/q ( u +v )p u p + v q , | i i| | i| | i| i=1 i=1 i=1 ✓ X ◆ ✓ X ◆ ✓ X ◆ is known as Minkowski’s inequality. When we restrict the Hermitian inner product to real vectors, u, v Rn,wegettheEuclidean inner product 2 n u, v = u v . h i i i i=1 X It is very useful to observe that if we represent (as usual) n u =(u1,...,un)andv =(v1,...,vn)(inR )bycolumn vectors, then their Euclidean inner product is given by u, v = u>v = v>u, h i and when u, v Cn,theirHermitianinnerproductis given by 2 u, v = v⇤u = u v. h i ⇤ 6.1. NORMED VECTOR SPACES 403 In particular, when u = v,inthecomplexcaseweget 2 u = u⇤u, k k2 and in the real case, this becomes 2 u = u>u. k k2 As convenient as these notations are, we still recommend that you do not abuse them; the notation u, v is more intrinsic and still “works” when our vector spaceh i is infinite dimensional. Proposition 6.2. The following inequalities hold for all x Rn (or x Cn): 2 2 x x 1 n x , k k1 k k k k1 x x 2 pn x , k k1 k k k k1 x x pn x . k k2 k k1 k k2 404 CHAPTER 6. VECTOR NORMS AND MATRIX NORMS Proposition 6.2 is actually a special case of a very impor- tant result: in a finite-dimensional vector space, any two norms are equivalent. Definition 6.2. Given any (real or complex) vector space E,twonorms and are equivalent i↵there exists kka kkb some positive reals C1,C2 > 0, such that u C u and u C u , for all u E. k ka 1 k kb k kb 2 k ka 2 Given any norm on a vector space of dimension n,for kk any basis (e1,...,en)ofE,observethatforanyvector x = x e + + x e ,wehave 1 1 ··· n n x = x e + + x e C x , k k k 1 1 ··· n nk k k1 with C =max1 i n ei and k k x = x e + + x e = x + + x . k k1 k 1 1 ··· n nk | 1| ··· | n| The above implies that u v u v C u v , |k kk k|k − k k − k1 which means that the map u u is continuous with respect to the norm . 7! k k kk1 6.1. NORMED VECTOR SPACES 405 n 1 Let S1 − be the unit sphere with respect to the norm ,namely kk1 n 1 S − = x E x =1 . 1 { 2 |k k1 } n 1 Now, S1 − is a closed and bounded subset of a finite- dimensional vector space, so by Heine–Borel (or equiva- n 1 lently, by Bolzano–Weiertrass), S1 − is compact. On the other hand, it is a well known result of analysis that any continuous real-valued function on a nonempty compact set has a minimum and a maximum, and that they are achieved. Using these facts, we can prove the following important theorem: Theorem 6.3. If E is any real or complex vector space of finite dimension, then any two norms on E are equivalent. Next, we will consider norms on matrices. 406 CHAPTER 6. VECTOR NORMS AND MATRIX NORMS 6.2 Matrix Norms For simplicity of exposition, we will consider the vector spaces Mn(R)andMn(C)ofsquaren n matrices. ⇥ Most results also hold for the spaces Mm,n(R)andMm,n(C) of rectangular m n matrices. ⇥ Since n n matrices can be multiplied, the idea behind matrix norms⇥ is that they should behave “well” with re- spect to matrix multiplication. Definition 6.3. A matrix norm on the space of kk square n n matrices in Mn(K), with K = R or K = C, ⇥ is a norm on the vector space Mn(K), with the additional property called submultiplicativity that AB A B , k kk kk k for all A, B Mn(K). A norm on matrices satisfying the above property2 is often called a submultiplicative matrix norm. Since I2 = I,from I = I2 I 2,weget I 1, for every matrix norm.k k k k k k 6.2. MATRIX NORMS 407 Before giving examples of matrix norms, we need to re- view some basic definitions about matrices. Given any matrix A =(aij) Mm,n(C), the conjugate A of A is the matrix such that2 A = a , 1 i m, 1 j n. ij ij The transpose of A is the n m matrix A> such that ⇥ A> = a , 1 i m, 1 j n. ij ji The adjoint of A is the n m matrix A⇤ such that ⇥ A⇤ = (A>)=(A)>. When A is a real matrix, A⇤ = A>. AmatrixA Mn(C)isHermitian if 2 A⇤ = A. If A is a real matrix (A Mn(R)), we say that A is symmetric if 2 A> = A. 408 CHAPTER 6. VECTOR NORMS AND MATRIX NORMS AmatrixA Mn(C)isnormal if 2 AA⇤ = A⇤A, and if A is a real matrix, it is normal if AA> = A>A. AmatrixU Mn(C)isunitary if 2 UU⇤ = U ⇤U = I. ArealmatrixQ Mn(R)isorthogonal if 2 QQ> = Q>Q = I. Given any matrix A =(aij) Mn(C), the trace tr(A)of A is the sum of its diagonal2 elements tr(A)=a + + a . 11 ··· nn It is easy to show that the trace is a linear map, so that tr(λA)=λtr(A) and tr(A + B)=tr(A)+tr(B). 6.2. MATRIX NORMS 409 Moreover, if A is an m n matrix and B is an n m matrix, it is not hard to⇥ show that ⇥ tr(AB)=tr(BA). We also review eigenvalues and eigenvectors. We con- tent ourselves with definition involving matrices. A more general treatment will be given later on (see Chapter 7).