Chapter 3 Inner Products and Norms The geometry of Euclidean space relies on the familiar properties of length and angle. The abstract concept of a norm on a vector space formalizes the geometrical notion of the length of a vector. In Euclidean geometry, the angle between two vectors is governed by their dot product, which is itself formalized by the abstract concept of an inner prod- uct. Inner products and norms lie at the heart of analysis, both linear and nonlinear, in both ¯nite-dimensional vector spaces and in¯nite-dimensional function spaces. It is im- possible to overemphasize their importance for both theoretical developments, practical applications, and in the design of numerical solution algorithms. We begin this chapter with a discussion of the basic properties of inner products, illustrated by some of the most important examples. Mathematical analysis is founded on inequalities. The most basic is the Cauchy{ Schwarz inequality, which is valid in any inner product space. The more familiar triangle inequality for the associated norm is then derived as a simple consequence. Not every norm arises from an inner product, and, in more general norms, the triangle inequality becomes part of the de¯nition. Both inequalities retain their validity in both ¯nite-dimensional and in¯nite-dimensional vector spaces. Indeed, their abstract formulation helps us focus on the key ideas in the proof, avoiding all distracting complications resulting from the explicit formulas. In Euclidean space Rn, the characterization of general inner products will lead us to an extremely important class of matrices. Positive de¯nite matrices play a key role in a variety of applications, including minimization problems, least squares, mechanical systems, electrical circuits, and the di®erential equations describing dynamical processes. Later, we will generalize the notion of positive de¯niteness to more general linear operators, governing the ordinary and partial di®erential equations arising in continuum mechanics and dynamics. Positive de¯nite matrices most commonly appear in so-called Gram matrix form, consisting of the inner products between selected elements of an inner product space. The test for positive de¯niteness is based on Gaussian elimination. Indeed, the associated matrix factorization can be reinterpreted as the process of completing the square for the associated quadratic form. So far, we have con¯ned our attention to real vector spaces. Complex numbers, vectors and functions also play an important role in applications, and so, in the ¯nal section, we formally introduce complex vector spaces. Most of the formulation proceeds in direct analogy with the real version, but the notions of inner product and norm on complex vector spaces requires some thought. Applications of complex vector spaces and their inner products are of particular importance in Fourier analysis and signal processing, and absolutely essential in modern quantum mechanics. 2/25/04 88 c 2004 Peter J. Olver ° v v v 3 v2 v 2 v 1 v1 Figure 3.1. The Euclidean Norm in R2 and R3. 3.1. Inner Products. The most basic example of an inner product is the familiar dot product n v ; w = v w = v w + v w + + v w = v w ; (3:1) h i ¢ 1 1 2 2 ¢ ¢ ¢ n n i i i = 1 X T T between (column) vectors v = ( v1; v2; : : : ; vn ) ; w = ( w1; w2; : : : ; wn ) lying in the Eu- clidean space Rn. An important observation is that the dot product (3.1) can be identi¯ed with the matrix product w1 w2 v w = vT w = ( v v : : : v ) 0 . 1 (3:2) ¢ 1 2 n . B C B w C B n C between a row vector vT and a column vector w. @ A The dot product is the cornerstone of Euclidean geometry. The key fact is that the dot product of a vector with itself, v v = v2 + v2 + + v2 ; ¢ 1 2 ¢ ¢ ¢ n is the sum of the squares of its entries, and hence, as a consequence of the classical Pythagorean Theorem, equal to the square of its length; see Figure 3.1. Consequently, the Euclidean norm or length of a vector is found by taking the square root: v = p v v = v2 + v2 + + v2 : (3:3) k k ¢ 1 2 ¢ ¢ ¢ n Note that every nonzero vector v = 0 has ppositive length, v 0, while only the zero vector has length 0 = 0. The dot6 product and Euclideank normk ¸ satisfy certain evident properties, and thesek servk e to inspire the abstract de¯nition of more general inner products. De¯nition 3.1. An inner product on the real vector space V is a pairing that takes two vectors v; w V and produces a real number v ; w R. The inner product is 2 h i 2 required to satisfy the following three axioms for all u; v; w V , and c; d R. 2 2 2/25/04 89 c 2004 Peter J. Olver ° (i) Bilinearity: c u + d v ; w = c u ; w + d v ; w ; h i h i h i (3:4) u ; c v + d w = c u ; v + d u ; w : h i h i h i (ii) Symmetry: v ; w = w ; v : (3:5) h i h i (iii) Positivity: v ; v > 0 whenever v = 0; while 0 ; 0 = 0: (3:6) h i 6 h i A vector space equipped with an inner product is called an inner product space. As we shall see, a given vector space can admit many di®erent inner products. Veri¯cation of the inner product axioms for the Euclidean dot product is straightforward, and left to the reader. Given an inner product, the associated norm of a vector v V is de¯ned as the positive square root of the inner product of the vector with itself: 2 v = v ; v : (3:7) k k h i The positivity axiom implies that v 0pis real and non-negative, and equals 0 if and only if v = 0 is the zero vector. k k ¸ Example 3.2. While certainly the most basic inner product on R2, the dot product v w = v1 w1 + v2 w2 is by no means the only possibility. A simple example is provided by the¢ weighted inner product v w v ; w = 2v w + 5v w ; v = 1 ; w = 1 : (3:8) h i 1 1 2 2 v w µ 2 ¶ µ 2 ¶ Let us verify that this formula does indeed de¯ne an inner product. The symmetry axiom (3.5) is immediate. Moreover, c u + d v ; w = 2(cu + dv )w + 5(cu + dv )w h i 1 1 1 2 2 2 = c(2u w + 5u w ) + d(2v w + 5v w ) = c u ; w + d v ; w ; 1 1 2 2 1 1 2 2 h i h i which veri¯es the ¯rst bilinearity condition; the second follows by a very similar computa- tion. (Or, one can use the symmetry axiom to deduce the second bilinearity identity from the ¯rst; see Exercise .) Moreover, 0 ; 0 = 0, while h i v ; v = 2v2 + 5v2 > 0 whenever v = 0; h i 1 2 6 since at least one of the summands is strictly positive, verifying the positivity requirement (3.6). This serves to establish (3.8) as an legitimate inner product on R2. The associated 2 2 weighted norm v = 2v1 + 5v2 de¯nes an alternative, \non-Pythagorean" notion of length of vectorsk andk distance between points in the plane. p A less evident example of an inner product on R2 is provided by the expression v ; w = v w v w v w + 4v w : (3:9) h i 1 1 ¡ 1 2 ¡ 2 1 2 2 2/25/04 90 c 2004 Peter J. Olver ° Bilinearity is veri¯ed in the same manner as before, and symmetry is obvious. Positivity is ensured by noticing that v ; v = v2 2v v + 4v2 = (v v )2 + 3v2 0; h i 1 ¡ 1 2 2 1 ¡ 2 2 ¸ and is strictly positive for all v = 0. Therefore, (3.9) de¯nes another inner product on R 2, 6 with associated norm v = v2 2v v + 4v2 . k k 1 ¡ 1 2 2 Example 3.3. Let c1; : :p: ; cn be a set of positive numbers. The corresponding weighted inner product and weighted norm on Rn are de¯ned by n n v ; w = c v w ; v = v ; v = c v2 : (3:10) h i i i i k k h i v i i i = 1 u i = 1 X p u X t th The numbers ci > 0 are the weights. The larger the weight ci, the more the i coordinate of v contributes to the norm. Weighted norms are particularly important in statistics and data ¯tting, where one wants to emphasize certain quantities and de-emphasize others; this is done by assigning suitable weights to the di®erent components of the data vector v. Section 4.3 on least squares approximation methods will contain further details. Inner Products on Function Space Inner products and norms on function spaces play an absolutely essential role in mod- ern analysis and its applications, particularly Fourier analysis, boundary value problems, ordinary and partial di®erential equations, and numerical analysis. Let us introduce the most important examples. Example 3.4. Let [a; b] R be a bounded closed interval. Consider the vector space C0[a; b] consisting of all con½tinuous scalar functions f(x) de¯ned for a x b. The integral of the product of two continuous functions · · b f ; g = f(x) g(x) dx (3:11) h i Za de¯nes an inner product on the vector space C0[a; b], as we shall prove below.