Further linear algebra. Chapter VI. Inner product spaces. Andrei Yafaev 1 Geometry of Inner Product Spaces Definition 1.1 Let V be a vector space over R and let , be a symmetric bilinear form on V . We shall call the form positive definiteh− −iif for all non-zero vectors v V we have ∈ v,v > 0. h i Notice that a symmetric bilinear form is positive definite if and only if its canonical form (over R) is In. 2 2 Rn Clearly x1 + . + xn is positive definite on . Conversely, suppose is a basis such that the matrix with respect to is the canonical form. ForB any B basis vector bi, the diagonal entry satisfies bi, bi > 0 and hence bi, bi = 1. h i h i Definition 1.2 Let V be a vector space over C. A Hermitian form on V is a function , : V V C such that: h− −i × → For all u,v,w V and all λ C, • ∈ ∈ u + λv,w = u, w + λ v,w ; h i h i h i For all u, v V , • ∈ u, v = v, u . h i h i Example 1.1 The simplest example is the following : take V = C, then <z,w>= zw¯ is a hermitian form on C. t A matrix A Mn(C) is called a Hermitian matrix if A = A¯. Here A¯ is the matrix obtained∈ from A by applying complex conjugation to the entries. 1 If A is a Hermitian matrix then the following is a Hermitian form on Cn: v,w = vtAw.¯ h i In fact every Hermitian form on Cn is one of these. To see why, suppose we are given a Hermitian form <,>. Choose a basis B =(b1, . , bn). Let v = i λibi and w = j µjbj. We calculate P P t <v,w>=< λibi, µjbj >= λiµj < bi, bj >= v Aw i j i,j X X X t where A =(< bi, bj >). Of course A = A because < bi, bj >= < bj, bi >. A matrix A satisfying At = A is called hermitian. n Example 1.2 If V = R , then <,> defined by < x1,...,xn, y1,...,yn >= i,j xiyj is called the standard inner product. Cn If V = , then <,> defined by < z1,...,zn,w1,...,wn >= i,j ziwj is calledP the standard (hermitian) inner product. P Note that a Hermitian form is conjugate-linear in the second variable, i.e. u, v + λw = u, v + λ¯ u, w . h i h i h i Note also that by the second axiom u, u R. h i∈ Definition 1.3 A Hermitian form is positive definite if for all non-zero vectors v we have v,v > 0. h i In other words, < v,v > 0 for all v and < v,v >= 0 if and only if v = 0. ≥ Clearly, the fom zw¯ is positive definite. Definition 1.4 By an inner product space we shall mean one of the follow- ing: either A finite dimensional vector space V over R with a positive definite symmetric bilinear form; 2 or A finite dimensional vector space V over C with a positive definite Hermitian form. We shall often write K to mean the field R or C, depending on which is relevant. Example 1.3 Consider the vector space V of all continuous functions [0, 1] C. −→ Then we can define 1 f,g = f(x)g(x)dx. h i Z0 This defines an inner product on V (easy exercise). Another example. Let V = Mn(R) the vector space of n n-matrices with real entries. Then × < A, B >= tr(ABt) is an inner product on V . t Similarly, if V = Mn(C) and < A, B >= tr(AB ) is an inner product. Definition 1.5 Let V be an inner product space. We define the norm of a vector v V by ∈ v = v,v . || || h i Lemma 1.4 For λ K we have λλ¯p= λ 2 for for v V we have λv = λ v . ∈ | | ∈ || || | | || || The proof is obvious. Theorem 1.5 (Cauchy-Schwarz inequality) If V is an inner product space then u, v V u, v u v . ∀ ∈ |h i| ≤ || || · || || Proof. If v = 0 then the result holds so suppose v = 0. We have for all λ K, 6 ∈ u λv,u λv 0. h − − i ≥ Expanding this out we have: u 2 λ v, u λ¯ u, v + λ 2 v 2 0. || || − h i − h i | | || || ≥ 3 u,v Setting λ = h v 2i we have: || || 2 u, v v, u u, v u 2 h i v, u h i u, v + h i v 2 0. || || − v 2 h i − v 2 h i v 2 || || ≥ || || || || || || Multiplying by v 2 we get || || 2 u 2 v 2 2 u, v 2 + u, v 0. || || || || − |h i| |h i| ≥ Hence u 2 v 2 u, v 2 . || || || || ≥ |h i| Taking the square root of both sides we get the result. Theorem 1.6 (Triangle inequality) If V is an inner product space with norm then || · || u, v V u + v u + v . ∀ ∈ || || ≤ || || || || Proof. We have u + v 2 = u + v, u + v || || h i = u 2 + 2 u, v + v 2. || || ℜh i || || Notice that (<u,v >) <u,v > hence |ℜ | ≤ | | u + v 2 u 2 + 2 <u,v > + v 2 || || ≤ || || | | || || So the Cauchy–Schwarz inequality implies that u + v 2 u 2 + 2 u v + v 2 =( u + v )5. || || ≤ || || || || || || || || || || || || Hence u + v u + v . || || ≤ || || || || Definition 1.6 Two vectors v,w in an inner product space are called or- thogonal if v,w = 0. h i 4 Theorem 1.7 (Pythagoras’ Theorem) Let (V,<,>) be an inner product space. If v,w V are orthogonal, then ∈ v 2 + w 2 = v + w 2 || || || || || || Proof. Since v + w 2 = v + w,v + w = v 2 + 2 v,w + w 2, || || h i || || ℜh i || || so we have v 2 + w 2 = v + w 2 || || || || || || if v,w = 0. h i 2 Gram–Schmidt Orthogonalisation Definition 2.1 Let V be an inner product space. We shall call a basis of B V an orthonormal basis if bi, bj = δi,j. h i Proposition 2.1 If is an orthonormal basis then for v,w V we have: B ∈ v,w = [v]t [w] . h i B B Proof. If the basis =(b1, . , bn) is orthonormal, then the matrix of <,> B in this basis is the identity In. The proposition follows. Theorem 2.2 (Gram–Schmidt Orthogonalisation) Let be any basis. Then the basis defined by B C c1 = b1 b2,c1 c2 = b2 h ic1 − c1,c1 h i b3,c1 b3,c2 c3 = b3 h ic1 h ic2 − c1,c1 − c2,c2 . h i h i . n 1 − bn,cr cn = bn h icr, − c ,c r=1 r r X h i 5 is orthogonal. Furthermore the basis defined by D 1 dr = cr, cr || || is orthonormal. Proof. Clearly each bi is a linear combination of , so spans V . As the cardinality of is dim V , is a basis. It follows alsoC thatC is a basis. We’ll C C D prove by induction that c1,...,cr is orthogonal. Clearly any one vector is { } orthogonal. Suppose c1,...,cr 1 are orthogonal. The for s < r we have { − } r 1 − br,ct cr,cs = br,cs h i ct,cs . h i h i − c ,c h i t=1 t t X h i By the inductive hypothesis we have br,cs cr,cs = br,cs h i cs,cs . = br,cs br,cs = 0. h i h i − cs,cs h i h i−h i h i (notice that < ct,cs >= 0 unless t = s). This shows that c1,...,cr are orthogonal. Hence is an orthogonal basis. It follows easily{ that } is orthonormal. C D This theorem shows in particular that an orthonormal basis always ex- ists. Indeed, take any basis and turn it into an orthonormal one by applying Gram-Schmidt process to it. Proposition 2.3 If V is an inner product space with an orthonormal basis n = b1, . , bn , then any v V can be written as v = v,ei ei. B { } ∈ i=1h i n n P Proof. We have v = λiei and v,ej = λi ei,ej = λj. i=1 h i i=1 h i Definition 2.2 Let SPbe a subspace of an innerP product space V . The or- thogonal complement of S is defined to be S⊥ = v V : w S v,w = 0 . { ∈ ∀ ∈ h i } 6 Theorem 2.4 If (V,<,>) is an inner product space and W is a subspace of V then V = W W ⊥, ⊕ and hence any v V can be written as ∈ v = w + w⊥, for unique w W and w⊥ W ⊥. ∈ ∈ Proof. We show first that V = W + W ⊥. Let = e1,...,en be an orthonormal basis for V , such that e1,...,er is a basisE for{ W . This} can be constructed by Gram-Schmidt orthogonalisa-{ } tion. (choose a basis b1, . , br for W and complete to a basis b1, . , bn of V . { } { } Then apply Gram-Schmidt process. Notice that in Gram-Schmidt pro- cess, when constructing orthonormal basis, the vectors c1,...,ck lie in the space generated by c1,...,ck 1, bk. It follows that the process will give an − orthonormal basis e1,...,en such that e1,...,er is an orthonormal basis of W .) If v V then ∈ r n v = λiei + λiei. i=1 i=r+1 X X Now r λiei W. ∈ i=1 X If w W then there exist µi R such that ∈ ∈ r w = µiei.