INNER PRODUCT SPACES

• Motivation and Definitions

• Inner Product Spaces as Normed Spaces

EL3370 Inner Product Spaces - 1

MOTIVATION AND DEFINITIONS

n Consider the space ! . It has:

1. Vector space (algebraic) structure: n Given !x, y ∈ , their sum !x + y and scalar multiplication !α x ( α ∈) are defined

2. Inner product structure: n (x, y)= x y x = (x ,…,x ), y = (x ,…,x )∈n (!x : complex conjugate of !x ∈) ∑ i i ! 1 n 1 n ! i=1

Many physical properties (e.g. work) can be defined in terms of inner products Also, !( ⋅ ,⋅) can define: distances (metrics), length (norms), angles, limits (topologies), ...

Goal: Extend inner products to general (possibly infinite dimensional) vector spaces

EL3370 Inner Product Spaces - 2

MOTIVATION AND DEFINITIONS (CONT.)

Definition Let l denote the vector space over  of all complex sequences x = {x } which are square ! 2 ! n ∞ 2 summable, i.e., that satisfy x < ∞, with componentwise addition and scalar !∑n=1 n multiplication:

x + y := {x + y }, x = {x }, y = { y }∈l n n n n 2 α x := {α x }, α ∈! ! n

∞ and inner product: (x, y):= x y ! ∑n=1 n n

Observation Need to verify that these operations (sum, scalar multiplication, inner product) are valid!

(we will do it later, using Cauchy-Schwarz inequality)

EL3370 Inner Product Spaces - 3

MOTIVATION AND DEFINITIONS (CONT.)

Definition (reminder) A vector space !V over a field !F (e.g.  or ) is a set with two operations, sum (!x + y ∈V , for !x, y ∈V ) and scalar multiplication (!λx ∈V , for !x ∈V and !λ ∈F ) such that for all !x, y,z ∈V , !α ,β ∈F :

1. !x + y = y + x (commutativity) 2. !(x + y)+ z = x +( y + z) (associativity) 3. There is a null vector !0∈X such that !0+ x = x 4. !α(x + y)= α x +α y 5. !(α + β)x = α x + βx (distributivity) 6. !(αβ)x = α(βx) (associativity) 7. !0α = 0, !1x = x y A field F is a set with operations + and ⋅ which are: associative and commutative; F has additive and multiplicative identities ( 0 and 1); every a ∈ F has an x additive inverse ( −a) and, if a ≠0, a multiplicative inverse x x + y too ( a−1 ); and ⋅ is distributive with respect to + : a⋅(b+c)= a⋅b+a⋅c for all a,b,c ∈ F x

EL3370 Inner Product Spaces - 4

MOTIVATION AND DEFINITIONS (CONT.)

Definition (reminder) Let !V be a vector space over !F , α ,…,α ∈ F , x ,…,x ∈V , and X ⊆V 1 n 1 n

- Linear combination of x ,…,x : an expression of the form α x +!+α x ∈V (for finite n) 1 n 1 1 n n

- lin X (span of X ): set of all linear combinations of elements of X ⊆V

- If for every linear combination α x +!+α x =0 it holds that α =!=α =0, X is linearly 1 1 n n 1 n independent (l.i.). Otherwise, X is linearly dependent (l.d.)

- Basis of V : A linearly independent set X ⊆V which spans V (i.e., lin X =V )

- dimV (dimension of V ): number of elements of some basis of V . (All bases of V have the same number of elements.)

- If dimV <∞, V is a finite-dimensional vector space (obs: V is not necessarily finite!)

EL3370 Inner Product Spaces - 5

MOTIVATION AND DEFINITIONS (CONT.)

Definition (inner products) An inner product (scalar product) on a vector space !V over  is a mapping !( ⋅ ,⋅): V ×V →  such that for all !x, y,z ∈V and λ ∈: 1. !(x, y)= ( y,x) 2. !(λx, y)= λ(x, y) 3. !(x + y,z)= (x,z)+( y,z) 4. !(x,x)> 0 when !x ≠ 0

!(V ,( ⋅ ,⋅)) is an inner product space (pre-Hilbert space)

Examples 1. Complex vector space !C[0,1]:= { f : [0,1]→ : f !continuous}, with point-wise addition and scalar multiplication (!( f + g)(t)= f (t)+ g(t), !(λ f )(t)= λ f (t) for !f , g ∈C[0,1], λ ∈ 1 and t ∈[0,1]), and inner product ( f , g)= f (t)g(t)dt ! ! ∫0 m×n H 2. Space ! of !m× n complex matrices, with inner product !(A,B)= tr(AB )

EL3370 Inner Product Spaces - 6

MOTIVATION AND DEFINITIONS (CONT.)

Proof for Example 1: Since !C[0,1] is a complex vector space (exercise!), we need to verify that !( ⋅ ,⋅) satisfies the axioms of an inner product. Let !f , g,h∈C[0,1] and λ ∈ : 1 1 1. ( f , g)= f (t)g(t)dt = g(t)f (t)dt = (g, f ) ! ∫0 ∫0 1 1 2. (λ f , g)= λ f (t)g(t)dt = λ f (t)g(t)dt = λ( f , g) ! ∫0 ∫0 1 1 1 3. ( f + g,h)= [ f (t)+ g(t)]h(t)dt = f (t)h(t)dt + g(t)h(t)dt = ( f ,h)+(g,h) ! ∫0 ∫0 ∫0 2 2 4. If f ≠ 0, then there is a t ∈[0,1] such that f (t ) =:l ≠ 0. Since f is continuous, there is ! ! 0 ! 0 ! 2 an ε > 0 such that f (t) > l 2 whenever t −t < ε ! ! ! 0 Therefore, 2✏ 1 2 l ( f , f )= f (t) dt ∫0 2 ≥ f (t) dt t [0,1]: t t l/2 2 ∫{ ∈ − 0 <ε} f | | ! ≥ εl 2> 0 ■

0 t0 1 EL3370 Inner Product Spaces - 7

MOTIVATION AND DEFINITIONS (CONT.)

Theorem For every λ ∈ and !x, y,z in an inner product space !V , (i) !(x, y + z)= (x, y)+(x,z) (ii) !(x,λ y)= λ(x, y) (iii) !(x,0)= (0,x)= 0 (iv) If !(x,z)= ( y,z) for all !z ∈V , then !x = y

Proof (i) By definition: !(x, y + z)= ( y + z,x)= ( y,x)+(z,x)= (x, y)+(x,z) (ii) Similar to (i) (iii) Notice that !(x,0)= (x,0 y), and use (ii) (iv) Since !(x,z)= ( y,z), then !(x − y,z)= 0. Since this holds for every !z , take !z = x − y , which gives !(x − y,x − y)= 0. By the last axiom of an inner product, this implies !x − y = 0 ■

EL3370 Inner Product Spaces - 8

INNER PRODUCT SPACES AS NORMED SPACES

Idea: Inner products ⇒ lengths (norms) ⇒ distances (metrics)

n T T Example: In ! , !(x, y)= x y ⇒ length!= x = x x = (x,x) ⇒ distance!= x − y

Definition In an inner product space !V , the norm of a vector !x ∈V is ! x := (x,x)

Examples 2 2 1. For x = (x ,…,x )∈n : x = x ++ x ! 1 n ! 1 n 1 2 2. For f ∈C[0,1]: f := f (t) dt ! ! ∫0

EL3370 Inner Product Spaces - 9

INNER PRODUCT SPACES AS NORMED SPACES (CONT.)

Theorem. For every !x, y in an inner product space !V , and λ ∈ : (i) ! x ≥ 0, and ! x = 0 iff !x = 0 (ii) ! λx = λ x (iii) !(x, y) ≤ x y , with equality iff !x = α y for some α ∈ (Cauchy-Schwarz inequality) (iv) ! x + y ≤ x + y (triangle inequality)

Proof. (i) Direct from last axiom of an inner product (ii) ! λx = (λx,λx) = λ(x,λx) = λ (x,x) = λ x 2 2 2 (iii) For every α ∈ : !0≤(x −α y,x −α y)= x −2Re{α(x, y)}+ α y 2 2 2 Take !α = tu, where !t ∈ and !u = exp( jarg(x, y)), which gives !0≤ x −2t (x, y) +t y 2 2 2 The minimum of this quadratic expression is ! x − (x, y) y , which must be non- negative. Furthermore, this is zero iff !x −α y = 0 for some α ∈ (iv) By (iii), 2 2 2 2 2 2 2 2 x 2Re(x, y) y x 2(x, y) y x 2 x y y x y ■ ! x + y ≤ + + ≤ + + ≤ + + = ()+

EL3370 Inner Product Spaces - 10

INNER PRODUCT SPACES AS NORMED SPACES (CONT.)

Applications of Cauchy-Schwarz inequality

Angle between vectors x (x, y) cosθ := x y ! ✓ y

Probability 2 Let !V be an inner product space of zero mean real random variables !x with !E{x } < ∞, and inner product !(x, y):= E{xy} = cov(x, y). Then Cauchy-Schwarz inequality implies

2 2 2 2 !cov(x, y) = (x, y) ≤ x y = var(x)var( y)

Exercise: Prove that the operations in l are well defined ! 2

EL3370 Inner Product Spaces - 11

INNER PRODUCT SPACES AS NORMED SPACES (CONT.)

Applications of Cauchy-Schwarz inequality (cont.)

Theorem. In an inner product space !V , the inner product is a continuous function, i.e., for every sequences {x },{ y } such that x → x and y → y , we have (x , y )→(x, y) ! n n ! n ! n ! n n

x3 x2 Proof. By Cauchy-Schwarz, xN x1 xN+1 (x, y)−(xn , yn ) = (x, y)−(xn , y)+(xn , y)−(xn , yn ) 0 x

≤ (x − xn , y) + (xn , y − yn ) ball B(x, 1) ≤ y x − x + x y − y ! n n n

Since {x } is convergent, it is also bounded (i.e., there is an !M > 0 such that x ≤ M for all ! n ! n n∈); indeed, since there is an !N > 0 such that x − x <1 (say) for !n > N , so ! ! n x = x + x − x ≤ x + x − x < x +1, we can take M = max{ x ,…, x , x +1}. Then, ! n n n ! 1 N given ε > 0, there is an N > 0 such that for n > N , x − x < ε 2 y and y − y < ε 2M , so ! ! 0 ! 0 ! n ! n (x, y)−(x , y ) ≤ y ε 2 y + M ε 2M = ε ■ ! n n () ()

EL3370 Inner Product Spaces - 12

INNER PRODUCT SPACES AS NORMED SPACES (CONT.)

Theorem (Parallelogram Law) Let !x, y be elements of an inner product space. Then,

2 2 2 2 ! x + y + x − y = 2 x +2 y

2 2 2 Proof. As ! x ± y = x ±(x, y)±( y,x)+ y , the result follows by adding these expressions ■

(See bonus slides for converse result!)

Theorem (Polarization Identity) Let !x, y be elements of an inner product space. Then, 3 1 2 2 2 2 1 2 (x, y)= x + y − x − y + j x + jy − j x − jy = jk x + jk y (complex!case) 4( ) 4 ∑ k=0 1 2 2 = ( x + y − x − y ) (real!case) ! 4

Proof. Exercise!

EL3370 Inner Product Spaces - 13

INNER PRODUCT SPACES AS NORMED SPACES (CONT.)

A more interesting example for system theory RL : space of rational functions, analytic on unit circle ∂D:= {z ∈: z = 1} ! 2 ! with usual addition and scalar multiplication, and inner product

1 dz 1 π ( f , g):= f (z)g(z) = f (e jω )g(e jω )dω ∫∂D ∫−π ! 2π j z 2π

RH : subspace of RL , of functions analytic on closed unit disc D, where D:= {z ∈: z <1} ! 2 ! 2 ! !

In engineering terms: Im Im RL consists of rational functions ! 2 without poles on !∂D (can be stable or unstable), and RH only has functions 1 1 Re 1 1 Re ! 2 with poles outside !D (stable)

RL2 RH2

EL3370 Inner Product Spaces - 14

INNER PRODUCT SPACES AS NORMED SPACES (CONT.)

A more interesting example for system theory (cont.) Exercise: Prove that RL is an inner product space ! 2

Cauchy integral formula simplifies calculations of inner products in RL : For h∈RL , ! 2 ! 2 1 h(z)dz = Res [h(z)] ∑ z=z 2π j ∫∂D i zi =poles!of!h ! !!!!!!!!!!in!D

1 1 Example: f (z)= , g(z)= !( a <1, b <1) ! z − a ! z − b 1 1 1 dz 1 1 1 ( f , g)= = − dz (since !zz = 1 in !∂D) ∫∂D ∫∂D ! 2π j z − a z − b z 2π jb z − a z −1 b 1 ⎛ h(z)⎞ 1 = − Res where h(z)= (which is analytic at !z = a) b z=a ⎜ z a⎟ z 1 b ! ⎝ − ⎠ ! − 1 1 1 1 = − h(a)= − = ! b b a−1 b 1− ab

EL3370 Inner Product Spaces - 15

BONUS: CONVERSE OF THE PARALLELOGRAM LAW

The parallelogram law can be used to show that a given norm does not come from an inner product. However, when it holds, it can also be used to derive an inner product!

Idea: Use the polarization identity! (consider the real case for simplicity) 1 2 2 (x, y)= ( x + y − x − y ) ! 4

Let us check the properties of an inner product: 1 2 2 1 2 2 1. ( y,x)= ( y + x − y − x ) = ( x + y − x − y ) = (x, y) ! 4 4 1 2 2 2 4. (x,x)= ( x + x − x − x ) = x > 0 if x ≠ 0 ! 4 3. Decompose !(x + y,z) in two different ways: 1 2 2 1 2 2 2 2 (x + y,z)= ( x + y + z − x + y − z ) = ( x + y + z + x − y + z − x + y − z − x − y + z ) 4 4 1 2 2 2 2 = ( x + y + z + x − y − z − x + y − z − x − y − z ) ! 4

EL3370 Inner Product Spaces - 16

BONUS: CONVERSE OF THE PARALLELOGRAM LAW (CONT.)

Applying the parallelogram law yields: 1 2 2 2 2 (x + y,z)= (2 x + z +2 y −2 x −2 y − z ) 4 1 2 2 2 2 = (2 x +2 y + z −2 y −2 x − z ) ! 4 Averaging these expressions and applying the polarization identity gives 1 2 2 2 2 (x + y,z)= ( x + z − y − z + y + z − x − z ) = (x,z)+( y,z) ! 4 2. From the polarization identity and (3), 1 2 2 1 2 2 (−x, y)= −x + y − −x − y = − x + y − y − x = −(x, y) 4( ) 4( ) (0, y)= (x − x, y)= (x,z)+(−x,z)= (x,z)−(x,z)= 0 !([n+1)x, y)= (nx, y)+(x, y) so by induction on n ∈! and the 1st expression, (nx, y) = n(x, y) for all n ∈!. Also, if m,n ∈! −{0}, n([m / n]x, y) = (mx, y) = m(x, y), so ([m / n]x, y) = [m / n](x, y), and (λx, y) = λ(x, y) for all λ ∈!. Since norms are continuous (because x − x! ≤ x − x! from the triangle inequality), this last expression can be extended to λ ∈! ◼︎

EL3370 Inner Product Spaces - 17