Contents 3 Inner Product Spaces
Total Page:16
File Type:pdf, Size:1020Kb
Linear Algebra (part 3) : Inner Product Spaces (by Evan Dummit, 2020, v. 2.00) Contents 3 Inner Product Spaces 1 3.1 Inner Product Spaces . 1 3.1.1 Inner Products on Real Vector Spaces . 1 3.1.2 Inner Products on Complex Vector Spaces . 3 3.1.3 Properties of Inner Products, Norms . 4 3.2 Orthogonality . 6 3.2.1 Orthogonality, Orthonormal Bases, and the Gram-Schmidt Procedure . 7 3.2.2 Orthogonal Complements and Orthogonal Projection . 9 3.3 Applications of Inner Products . 13 3.3.1 Least-Squares Estimates . 13 3.3.2 Fourier Series . 16 3.4 Linear Transformations and Inner Products . 18 3.4.1 Characterizations of Inner Products . 18 3.4.2 The Adjoint of a Linear Transformation . 20 3 Inner Product Spaces In this chapter we will study vector spaces having an additional kind of structure called an inner product, which generalizes the idea of the dot product of vectors in Rn, and which will allow us to formulate notions of length and angle in more general vector spaces. We dene inner products in real and complex vector spaces and establish some of their properties, including the celebrated Cauchy-Schwarz inequality, and survey some applications. We then discuss orthogonality of vectors and subspaces and in particular describe a method for constructing an orthonormal basis for any nite-dimensional inner product space, which provides an analogue of giving standard unit coordinate axes in Rn. Next, we discuss a pair of very important practical applications of inner products and orthogonality: computing least-squares approximations and approximating periodic functions with Fourier series. We close with a discussion of the interplay between linear transformations and inner products. 3.1 Inner Product Spaces • Notation: In this chapter, we will use parentheses around vectors rather than angle brackets, since we will shortly be using angle brackets to denote an inner product. We will also avoid using dots when discussing scalar multiplication, and reserve the dot notation for the dot product of two vectors. 3.1.1 Inner Products on Real Vector Spaces • Our rst goal is to describe the natural analogue in an arbitrary real vector space of the dot product in Rn. We recall some basic properties of the dot product: ◦ The dot product distributes over addition and scaling: (v1 +cv2)·w = (v1 ·w)+c(v2 ·w) for any vectors v1; v2; w and scalar c. 1 ◦ The dot product is commutative: v · w = w · v for any vectors v; w. ◦ The dot product is nonnegative: v · v ≥ 0 for any vector v. • We will use these three properties to dene an arbitrary inner product on a real vector space: • Denition: If V is a real vector space, an inner product on V is a pairing that assigns a scalar in F to each ordered pair (v; w) of vectors in V . This pairing is denoted hv; wi and must satisfy the following properties: [I1] Linearity in the rst argument: hv1 + cv2; wi = hv1; wi + chv2; wi. [I2] Symmetry: hw; vi = hv; wi. [I3] Positive-deniteness: hv; vi ≥ 0 for all v, and hv; vi = 0 only when v = 0. ◦ The linearity and symmetry properties are fairly clear: if we x the second component, the inner product behaves like a linear function in the rst component, and we want both components to behave in the same way. ◦ The positive-deniteness property is intended to capture an idea about length: namely, the length of a vector v should be the (inner) product of v with itself, and lengths are supposed to be nonnegative. Furthermore, the only vector of length zero should be the zero vector. • Denition: A vector space V together with an inner product h·; ·i on V is called an inner product space. ◦ Any given vector space may have many dierent inner products. ◦ When we say Let V be an inner product space, we intend this to mean that V is equipped with a particular (xed) inner product. • The entire purpose of dening an inner product is to generalize the notion of the dot product to more general vector spaces, so we should check that the dot product on Rn is actually an inner product: Example: Show that the standard dot product on n, dened as • R (x1; : : : ; xn)·(y1; : : : ; yn) = x1y1 +···+xnyn is an inner product. ◦ [I1]-[I2]: We have previously veried the linearity and symmetry properties. [I3]: If then 2 2 2 . Since each square is nonnegative, , and ◦ v = (x1; : : : ; xn) v · v = x1 + x2 + ··· + xn v · v ≥ 0 v · v = 0 only when all of the components of v are zero. • There are other examples of inner products on Rn beyond the standard dot product. Example: Show that the pairing on 2 is an inner product. • h(x1; y1); (x2; y2)i = 3x1x2 +2x1y2 +2x2y1 +4y1y2 R ◦ [I1]-[I2]: It is an easy algebraic computation to verify the linearity and symmetry properties. ◦ [I3]: We have h(x; y); (x; y)i = 3x2 + 4xy + 4y2 = 2x2 + (x + 2y)2, and since each square is nonnegative, the inner product is always nonnegative. Furthermore, it equals zero only when both squares are zero, and this clearly only occurs for x = y = 0. • We can dene another class of inner products on function spaces using integration: • Example: Let V be the vector space of continuous (real-valued) functions on the interval [a; b]. Show that b is an inner product on . hf; gi = a f(x)g(x) dx V ´ [I1]: We have b b b ◦ hf1 + cf2; gi = a [f1(x) + cf2(x)] g(x) dx = a f1(x)g(x) dx + c a f2(x)g(x) dx = hf1; gi + c hf2; gi. ´ ´ ´ [I2]: Observe that b b . ◦ hg; fi = a g(x)f(x) dx = a f(x)g(x) dx = hf; gi [I3]: Notice that b´ 2 is the integral´ of a nonnegative function, so it is always nonnegative. ◦ hf; fi = a f(x) dx Furthermore (since f is assumed´ to be continuous) the integral of f 2 cannot be zero unless f is identically zero. ◦ Remark: More generally, if w(x) is any xed positive (weight) function that is continuous on [a; b], b is an inner product on . hf; gi = a f(x)g(x) · w(x) dx V ´ 2 3.1.2 Inner Products on Complex Vector Spaces • We would now like to extend the notion of an inner product to complex vector spaces. ◦ A natural rst guess would be to use the same denition as in a real vector space. However, this turns out not to be the right choice! ◦ To explain why, suppose that we try to use the same denition of dot product to nd the length of a vector of complex numbers, i.e., by computing v · v. ◦ We can see that the dot product of the vector (1; 0; i) with itself is 0, but this vector certainly is not the zero vector. Even worse, the dot product of (1; 0; 2i) with itself is −3, while the dot product of (1; 0; 1 + 2i) with itself is −2 + 4i. None of these choices for lengths seem sensible. p ◦ Indeed, a more natural choice of length for the vector (1; 0; i) is 2: the rst component has absolute value 1, the second has absolute value 0, and the last has absolute value 1, for an overall length of p p 2 2 2, in analogy with the real vector which also has length . Similarly, 1 + 0 + 1 p p (1; 0; 1) 2 (1; 0; 2i) should really have length 12 + 02 + 22 = 5. ◦ One way to obtain a nonnegative function that seems to capture this idea of length for a complex vector is to include a conjugation: notice that for any complex vector v, the dot product v · v will always be a nonnegative real number. ◦ Using the example above, we can compute that (1; 0; i) · (1; 0; i) = 12 + 02 + 12 = 2, so this modied dot product seems to give the square of the length of a complex vector, at least in this one case. 1 ◦ All of this suggests that the right analogy of the Rn dot product for a pair of complex vectors v, w is v · w rather than v · w. ◦ We then lose the symmetry property of the real inner product, since now v · w and w · v are not equal in general. But notice we still have a relation between v · w and w · v: namely, the latter is the complex conjugate of the former. ◦ We can therefore obtain the right denition by changing the symmetry property hw; vi = hv; wi to conjugate-symmetry: hw; vi = hv; wi . • With this minor modication, we can extend the idea of an inner product to a complex vector space: • Denition: If V is a (real or) complex vector space, an inner product on V is a pairing that assigns a scalar in F to each ordered pair (v; w) of vectors in V . This pairing is denoted hv; wi and must satisfy the following properties: [I1] Linearity in the rst argument: hv1 + cv2; wi = hv1; wi + chv2; wi. [I2] Conjugate-symmetry: hw; vi = hv; wi (where the bar denotes the complex conjugate). [I3] Positive-deniteness: hv; vi ≥ 0 for all v, and hv; vi = 0 only when v = 0. ◦ This denition generalizes the one we gave for a real vector space earlier: if V is a real vector space then hw; vi = hw; vi since hw; vi is always a real number. ◦ Important Warning: In other disciplines (particularly physics), inner products are often dened to be linear in the second argument, rather than the rst.