INNER PRODUCTS and NORMS

Professor Dan A. Simovici

UMB

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 1 / 46 1 Norms

2 Inequalities

3 Metric Spaces

4 Norms on Rn

5 Norms for Matrices

6 Inner Products

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 2 / 46 Seminorms and Norms

A seminorm on an linear space V is a mapping ν : V −→ R that satisfies the following conditions: ν(x + y ) ≤ ν(x )+ ν(y ) (subadditivity), and ν(axx)= |a|ν(x ) (positive homogeneity), for x ,y ∈ V and a ∈ F .

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 3 / 46 Example

A seminorm can be defined on every linear space. Indeed, if B is a basis of V , B = {v i | i ∈ I }, J is a finite subset of I , and x = i∈I xiv i , define νJ (x ) as P 0 if x = 0, νJ (x )= ( j∈J |aj | otherwise for x ∈ V . P

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 4 / 46 If V is a real or complex linear space and ν : V −→ R is a seminorm on V , then ν(x − y ) ≥ |ν(x ) − ν(y )|, for x ,y ∈ V . If p : V −→ R is a seminorm on V , then p(x ) ≥ 0 for x ∈ V .

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 5 / 46 A norm on an F -linear space V is a seminorm ν : V −→ R such that ν(x ) = 0 implies x = 0 for x ∈ V . The pair (V ,ν) is referred to as a normed linear space.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 6 / 46 Example

The set of real-valued continuous functions defined on the interval [−1, 1] is a real linear space. The addition of two such functions f , g, is defined by (f + g)(x)= f (x)+ g(x) for x ∈ [−1, 1]; the multiplication of f by a scalar a ∈ R is (af )(x)= af (x) for x ∈ [−1, 1]. Define ν(f ) = sup{|f (x)| | x ∈ [−1, 1]}. Since |f (x)|≤ ν(f ) and |g(x)|≤ ν(g) for x ∈ [−1, 1]}, it follows that |(f + g)(x)| ≤ |f (x)| + |g(x)|≤ ν(f )+ ν(g). Thus, ν(f + g) ≤ ν(f )+ ν(g).

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 7 / 46 Preliminaries

R 1 1 Let p, q ∈ − {0, 1} such that p + q = 1. Then p > 1 if and only if q > 1. Furthermore, one of the numbers p, q belongs to the interval (0, 1) if and only if the other number is negative. R 1 1 Let p, q ∈ − {0, 1} be two numbers such that p + q = 1 and p > 1. Then, for every a, b ∈ R≥0, we have ap bq ab ≤ + , p q

− 1 where the equality holds if and only if a = b 1−p .

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 8 / 46 The H¨older Inequality

Let a1,..., an and b1,..., bn be 2n nonnegative numbers, and let p and q 1 1 be two numbers such that p + q = 1 and p > 1. We have

1 1 n n p n q p q ai bi ≤ ai · bi . i i ! i ! X=1 X=1 X=1

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 9 / 46 Theorem

Let a1,..., an and b1,..., bn be 2n numbers and let p and q be two 1 1 numbers such that p + q = 1 and p > 1. We have

1 1 n n p n q p q ai bi ≤ |ai | · |bi | . ! ! i=1 i=1 i=1 X X X

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 10 / 46 The Cauchy-Schwarz Inequality for Rn

Let a1,..., an and b1,..., bn be 2n real numbers. We have

n n n 2 2 ai bi ≤ ai · bi . v v i=1 u i=1 u i=1 X uX uX t t

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 11 / 46 Minkowski Inequality

Let a1,..., an and b1,..., bn be 2n nonnegative real numbers. If p ≥ 1, we have 1 1 1 n p n p n p p p p (ai + bi ) ≤ ai + bi . i ! i ! i ! X=1 X=1 X=1

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 12 / 46 Definition

2 A function d : S −→ R≥0 is a metric if it has the following properties: d(x, y)=0 if and only if x = y for x, y ∈ S; d(x, y)= d(y, x) for x, y ∈ S; d(x, y) ≤ d(x, z)+ d(z, y) for x, y, z ∈ S. The pair (S, d) will be referred to as a metric space. If the first property is replaced by the weaker requirement that d(x, x) = 0 for x ∈ S, then we refer to d as a semimetric

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 13 / 46 Example

2 Let S be a nonempty set. Define the mapping d : S −→ R≥0 by

1 if u 6= v, d(u, v)= (0 otherwise,

for x, y ∈ S.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 14 / 46 Example

2 R Consider the mapping d : (Seqn(S)) −→ ≥0 defined by

d(p,q)= |{i | 0 ≤ i ≤ n − 1 and p(i) 6= q(i)}|

for all sequences p,q of length n on the set S.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 15 / 46 Example

The Euclidean metric is the mapping

n d (x ,y )= (x − y )2. 2 v i i u i=1 uX t

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 16 / 46 Spheres

Let (S, d) be a metric space. The closed sphere centered in x ∈ S of radius r is the set

Bd (x, r)= {y ∈ S|d(x, y) ≤ r}.

The open sphere centered in x ∈ S of radius r is the set

Cd (x, r)= {y ∈ S|d(x, y) < r}.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 17 / 46 Definition

n For p ≥ 1, the function νp : R −→ R≥0 defined by

1 n p p νp(x1,..., xn)= |xi | , i ! X=1 n n where x = (x1,..., xn) ∈ R , is a norm on R .

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 18 / 46 Examples

The following are norms on Rn: n The mapping ν1 : R −→ R given by

ν1(x )= |x1| + |x2| + · · · + |xn|,

n for x = (x1,..., xn) ∈ R . n ν∞ : R −→ R≥0 given by

ν∞(x ) = max{|xi | | 1 ≤ i ≤ n}

n for x = (x1,..., xn) ∈ R . k x k2 is the Euclidean norm

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 19 / 46 Any norm generates a metric

Each norm ν : V −→ R≥0 on a real linear space V generates a metric on the set V defined by dν(x ,y )= ν(x − y ) for x ,y ∈ V . Note that ν can be expressed using dν as

ν(x )= dν(x ,00)

for x ∈ V .

For p ≥ 1, dp denotes the metric dνp induced by the norm νp on the linear space Rn known as the Minkowski metric. on Rn.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 20 / 46 Metrics induced by norms

p = 2: the Euclidean metric on Rn is

n n d (x ,y )= |x − y |2 = (x − y )2. 2 v i i v i i u i=1 u i=1 uX uX t t p = 1: n d1(x ,y )= |xi − yi |. i X=1 This metric is known also as the city-block metric. The norm ν∞ generates the metric d∞ given by

d∞(x ,y ) = max{|xi − yi | | 1 ≤ i ≤ n},

also known as the Chebyshev metric.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 21 / 46 If x = (x0, x1) and y = (y0, y1), then d2(x ,y ) is the length of the hypotenuse of the right triangle and d1(x ,y ) is the sum of the lengths of the two legs of the triangle.

6 y = (y0, y1)

x = (x0, x1) (y0, x1) -

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 22 / 46 Projections on Closed Sets Theorem

n n Let U be a closed subset of R such that U 6= ∅ and let x 0 ∈ R − U. Then, there exists x 1 ∈ U such that k x − x 0 k2≥k x 1 − x 0 k2 for every x ∈ U. Let p and q be two positive numbers such that p ≤ q. For every n u ∈ R , we have k u kp≥k u kq.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 23 / 46 Comparisons between norms

Let a1,..., an be n positive numbers. If p and q are two positive numbers such that p ≤ q, then

1 1 p p p q q q a1 + · · · + an ≥ a1 + · · · + an .

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 24 / 46 Spheres Bdp (00, 1) for p =1, 2, ∞

6 6 6 @ @ - '$- - @ @ (a)&% (b) (c)

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 25 / 46 Matrix Vectorization

The (m × n)-vectorization mapping is the mapping vec : Cm×n −→ Rmn defined by a11 .  .  a  m1  .  vec(A)=  .  ,   a n   1   .   .    amn   obtained by reading A column-wise.  

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 26 / 46 Example

For the matrix In we have

e 1 e 2 vec(In)=  .  . .   e n    

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 27 / 46 Vectorial Norms of Matrices

Let ν be a vector norm on the space Rmn. The vectorial matrix norm (m,n) m×n (m,n) m×n µ on R is the mapping µ : R −→ R≥0 defined by

µ(m,n)(A)= ν(vec(A)),

for A ∈ Rm×n.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 28 / 46 Consistent Families of Norms

A consistent family of matrix norms is a family of functions (m,n) m×n µ : C −→ R≥0, where m, n ∈ P that satisfies the following conditions: (m,n) µ (A)=0 if and only if A = Om,n; µ(m,n)(A + B) ≤ µ(m,n)(A)+ µ(m,n)(B) (the subadditivity property); µ(m,n)(aA)= |a|µ(m,n)(A); µ(m,p)(AB) ≤ µ(m,n)(A)µ(n,p)(B) for every matrix A ∈ Rm×n and B ∈ Rn×p (the submultiplicative property).

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 29 / 46 Example

Let P ∈ Cn×n be an idempotent matrix. If µ is a matrix norm, then either µ(P) = 0 or µ(P) ≥ 1. Indeed, since P is idempotent we have µ(P)= µ(P2). By the submultiplicative property, µ(P2) ≤ (µ(P))2, so µ(P) ≤ (µ(P))2. Consequently, if µ(P) 6= 0, then µ(P) ≥ 1.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 30 / 46 Example Some vectorial matrix norms turn out to be actual matrix norms; others fail to be matrix norms. Consider the vectorial matrix norm µ1 induced by the vector norm ν1. We n m Rm×n have µ1(A)= i=1 j=1 |aij | for A ∈ . Actually, this is a matrix norm. To prove this fact consider the matrices A ∈ Rm×p and B ∈ Rp×n. P P We have: m n p m n p µ1(AB) = aik bkj ≤ |aik bkj | i j i j X=1 X=1 Xk=1 X=1 X=1 Xk=1 p p m n ≤ |a ik0 ||bk00j | i j 0 00 X=1 X=1 kX=1 kX=1 (because we added extra non-negative terms to the sums) m p n p = |a 0 | · |b 00 | ik  k j  i 0 ! j 00 X=1 kX=1 X=1 kX=1 = µ1(A)µ1(B).   WeProfessor denote Dan A. Simovici this vectorial (UMB) matrixINNER PRODUCTS norm by and the NORMS same notation as the 31 / 46 Example The vectorial matrix norm µ2 induced by the vector norm ν2 is also a matrix norm. Indeed, using the notations as above we have: m n p 2 2 (µ2(AB)) = aik bkj i j X=1 X=1 Xk=1 m n p p 2 2 ≤ |aik | |blj | i j ! ! X=1 X=1 Xk=1 Xl=1 (by Cauchy-Schwarz inequality) 2 2 ≤ (µ2(A)) (µ2(B)) . The vectorial norm of A ∈ Cm×n, 1 n m 2 2 µ (A)= |aij | , 2   i j X=1 X=1   m×n denoted also by k A kF , is known as the Frobenius norm. For A ∈ R we have Professor Dan A. Simovici (UMB) INNER PRODUCTSn and NORMSm 32 / 46

v u uX X Frobenius Norm

For real matrices we have

2 0 0 k A kF = trace(AA ) = trace(A A).

For complex matrices:

2 H H k A kF = trace(AA ) = trace(A A).

H 2 2 Note that k A kF =k A kF for every A.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 33 / 46 Example

The vectorial norm µ∞ induced by the vector norm ν∞ is denoted by k A k∞ and is given by k A k∞= max |aij | i,j for A ∈ Cn×n. This is not a matrix norm. Indeed, let a, b be two positive numbers and consider the matrices a a b b A = and B = . a a b b    

We have k A k∞= a and k B k∞= b. However, since

2ab 2ab AB = , 2ab 2ab  

we have k AB k∞= 2ab and the submultiplicative property of matrix norms is violated.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 34 / 46 Operator Norms

m n n×m Let νm be a norm on C and νn be a norm on C and let A ∈ C be a matrix. The operator norm of A is the number

(n,m) µ (A) = sup{νn((A)x ) | νm(x ) ≤ 1}

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 35 / 46 Equivalent Conditions

n Let νn be a norm on C for n ≥ 1. The following equalities hold for µ(n,m)(A), where A ∈ C(n,m).

(n,m) m µ (A) = inf{M ∈ R≥0 | νn(Axx) ≤ Mνm(x ) for every x ∈ C }

= sup{νn(Axx) | νm(x ) ≤ 1}

= max{νn(Axx) | νm(x ) ≤ 1} = max{ν0(f (x )) | ν(x ) = 1} ν0(f (x )) = sup | x ∈ Cm − {0 } . ν(x ) m  

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 36 / 46 Example n×n To compute |||A|||1 = sup{k Axx k1 | k x k1≤ 1}, where A ∈ R , suppose that the columns of A are the vectors a1,...,an, that is

a1j a2j aj =  .  . .   a   nj  n   Let x ∈ R be a vector whose components are x1,..., xn. Then, AAxx = x1a1 + · · · + xnan, so

k Axx k1 = k x1a1 + · · · + xnan k1 n ≤ |xj | k aj k1 j X=1 n ≤ max k aj k1 |xj | j j X=1 = max k aj k1 · k x k1 . j Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 37 / 46 Thus, |||A||| ≤ max k a k . Example (cont’d)

th Let e j be the vector whose components are 0 with the exception of its j component that is equal to 1. Clearly, we have k e j k1= 1 and aj = Aeej . This, in turn implies k aj k1=k Aeej k1≤ |||A|||1 for 1 ≤ j ≤ n. Therefore, maxj k aj k1≤ |||A|||1, so

n |||A|||1 = max k aj k1= max |aij |. j j i X=1

In other words, |||A|||1 equals the maximum column sum of the absolute values.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 38 / 46 Unitarily Invariant Norms

Norms that are invariant with respect to multiplication by unitary matrices are known as unitarily invariant norms. Let U ∈ Cn×n be a unitary matrix. The following statements hold: n k Uxx k2=k x k2 for every x ∈ C ; n×p |||UA|||2 = |||A|||2 for every A ∈ C ; n×p k UA kF =k A kF for every A ∈ C .

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 39 / 46 Example

Let n S = {x ∈ R | k x k2= 1} be the surface of the sphere in Rn. The image of S under the linear transformation hU that corresponds to the unitary matrix U is S itself. Indeed, k hU (x ) k2=k x k2= 1, so hU (x ) ∈ S for every x ∈ S. Also, note that hU restricted to S is a bijection because hUH (hU (x )) = x for every x ∈ Rn.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 40 / 46 Theorem Let A ∈ Rn×n. We have |||A|||2 ≤k A kF . Let x ∈ Rn. We have r 1x . Axx =  .  , r x  n    where r 1,...,r n are the rows of the matrix A. Thus,

n 2 k Axx k (r ix ) 2 = i=1 . k x k k x k 2 pP 2 2 2 2 By Cauchy-Schwarz inequality we have (r ix ) ≤k r i k2k x k2, so

n k Axx k2 2 ≤ k r i k2 =k A kF . k x k2 v u i=1 uX t This implies |||A|||2 ≤k A kF . Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 41 / 46 What is an inner product?

Let V be a C-linear space. An inner product on V is a function f : V × V −→ C that has the following properties: f (axx + byy,z )= af (x ,z )+ bf (y ,z ) (linearity in the first argument); f (x ,y )= f (y ,x ) for y ,x ∈ V (conjugate symmetry); if x 6= 00, then f (x ,x ) is a positive real number (positivity), f (x ,x )=0 if and only if x = 0 (definiteness), for every x ,y ,z ∈ V and a, b ∈ C. The pair (V , f ) is called an inner product space. For the second argument of a scalar product we have the property of conjugate linearity, that is,

f (z , axx + byy ) =af ¯ (z ,x )+ bf¯ (z,y )

for every x ,y ,z ∈ V and a, b ∈ C.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 42 / 46 Example

n Let C be the linear space of n-tuples of complex numbers. If a1,..., an are n real, positive numbers, then the function f : Cn × Cn −→ C defined by f (x ,y )= a1x1y¯1 + a2x2y¯2 + · · · + anxny¯n is an inner product on Cn, as the reader can easily verify. If a1 = · · · = an = 1, we have the Euclidean inner product:

H f (x ,y )= x1y¯1 + · · · + xny¯n = y x .

For the linear space Rn, the Euclidean inner product is

0 0 f (x ,y )= x1y1 + · · · + xnyn = y x = x y ,

where x ,y ∈ Rn.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 43 / 46 A fundamental property

(Axx,y ) = (x , AHy ), which holds for every A ∈ Cn×n and x ,y ∈ Cn. A matrix B ∈ Cn×n is the adjoint of a matrix A ∈ Cn×n relative to the inner product (·, ·) if (Axx,y ) = (x , Byy) for every x ,y ∈ Cn.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 44 / 46 Self-adjoint Matrices

A matrix is self-adjoint if it equals its own adjoint, that is if (Axx,y ) = (x , Ayy ) for every x ,y ∈ Cn. a Hermitian matrix is self-adjoint relative to the inner product (x ,y )= x Hy for x ,y ∈ Cn. if we use the Euclidean inner product we omit the reference to this product and refer to the adjoint of A relative to this product simply as the adjoint of A.

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 45 / 46 An Inner Product on Matrices

An inner product on Cn×n, the linear space of matrices of format n × n, can be defined as (X , Y ) = trace(XY H) for X , Y ∈ Cn×n. Note that 2 H Cn×n k X kF = (X , X ) for every X ∈ .

Professor Dan A. Simovici (UMB) INNER PRODUCTS and NORMS 46 / 46