Lecture 2j Inner Product Spaces (pages 348-350) So far, we have taken the essential properties of Rn and used them to form general vector spaces, and now we have taken the essential properties of the standard basis and created orthonormal bases. But the definition of an or- thonormal basis was dependent on the properties of the dot product. But what was so special about the dot product? We take the essential properties of the dot product and use then to define the more general concept of an inner product. Definition: Let V be a vector space over R. An inner product on V is a function h ; i : V × V ! R such that (1) hv; vi ≥ 0 for all v 2 V, and hv; vi = 0 if and only if v = 0. (positive definite) (2) hv; wi = hw; vi for all v; w 2 V (symmetric) (3) hv; sw + tzi = shv; wi + thv; zi for all s; t 2 R and v; w; z 2 V (bilinear) Note that, if we combine properties (2) and (3), then we also have that hsv + tw; zi = shv; zi + thw; zi. In its full generality, we have that hs1x + s2v; t1w + t2zi = s1hx; t1w + t2zi + s2hv; t1w + t2zi = s1t1hx; wi + s1t2hx; zi + s2t1hv; wi + s2t2hv; zi Definition: A vector space V with an inner product is called an inner product space. Let's start our study of inner product spaces by looking in Rn. Example: Show that function h ; i defined by h~x;~yi = x1y1 + 4x2y2 + 9x3y3 is an inner product on R3. To show this, we verify that h ; i satisfies the three properties of an inner product. 3 2 2 2 (1) For any ~x 2 R , h~x;~xi = x1x1 + 4x2x2 + 9x3x3 = x1 + 4x2 + 9x3. Since 2 2 2 each of x1, x2, and x3 are greater than or equal to zero, we see that h~x;~xi ≥ 0. 2 2 2 What if h~x;~xi = 0? Then x1 + 4x2 + 9x3 = 0. But since the terms in this sum are all greater than or equal to zero, the only way their sum can equal zero is 2 2 2 if each of the terms equals zero. So we get x1 = 0, 4x2 = 0, and 9x3 = 0. This means that x1 = 0, x2 = 0, and x3 = 0, which means that ~x = ~0. And so we see that h ; i is positive definite. 1 3 (2) For any ~x;~y 2 R , h~x;~yi = x1y1 + 4x2y2 + 9x3y3 = y1x1 + 4y2x2 + 9y3x3 = h~y; ~xi, so h ; i is symmetric. (3) For any ~x;~y; ~z 2 R3 and s; t 2 R, h~x;s~y + t~zi = x1(sy1 + tz1) + 4x2(sy2 + tz2) + 9x3(sy3 + tz3) = sx1y1 + tx1z1 + 4sx2y2 + 4tx2z2 + 9sx3y3 + 9tx3z3 = s(x1y1 + 4x2y2 + 9x3y3) + t(x1z1 + 4x2z2 + 9x3z3) = sh~x;~yi + th~x;~zi So h ; i is bilinear. Example: The function h ; i defined by h~x;~yi = 2x1y1 + 3x2y2 is not an inner product on R3 because it is not positive definite. For example, *2 0 3 2 0 3+ 2 0 3 we get that 4 0 5 ; 4 0 5 = 2(0)(0) + 3(0)(0) = 0, even though 4 0 5 6= 3 3 3 2 0 3 4 0 5. Note that the textbook shows that this function does define an inner 0 product on R2, so this example is intended to show the importance of paying attention which vector space you are working in. Example: The function h ; i defined by h~x;~yi = x1y1 + x1y2 + x1y3 is not an inner product on R3 because it is not symmetric. For example, *2 1 3 2 2 3+ *2 2 3 2 1 3+ 4 2 5 ; 4 3 5 = (1)(2) + (1)(3) + (1)(4) = 9, but 4 3 5 ; 4 2 5 = 3 4 4 3 *2 1 3 2 2 3+ *2 2 3 2 1 3+ (2)(1) + (2)(2) + (2)(3) = 12, so 4 2 5 ; 4 3 5 6= 4 3 5 ; 4 2 5 . (Our 3 4 4 3 function is also not positive definite. It is bilinear.) The dot product was only defined in Rn, but an inner product can be defined in any vector space. What could be an inner product in a matrix space? Since our result is supposed to be a number, not a vector, matrix multiplication is not a contender. What about the determinant? 2 Example: The function hA; Bi = det(AB) is not an inner product on M(2; 2). 1 0 1 0 The easiest thing to see is that it is not positive definite, since det = 0 0 0 0 1 0 1 0 1 0 1 0 det = 0, so ; = 0 even though 6= 0 0 0 0 0 0 0 0 0 0 . This counterexample easily extends to show that hA; Bi = det(AB) 0 0 is not an inner product on M(n; n) for any n. It is also worth noting that the determinant is not bilinear, since in general det(sA) = sndetA 6= sdetA. So, while the determinant is useful for many things, it is not useful for defining an inner product on matrix spaces. There is another operation associated with matrices that results in a number, though, and that is the trace. Recall that the trace of a matrix is the sum of the diagonal entries. Example: The function hA; Bi = tr(BT A) is an inner product on M(m; n). To see this, let's first look at this definition in M(2; 3): a a a b b b 11 12 13 ; 11 12 13 a21 a22 a23 b21 b22 b23 02 3 1 b11 b21 a11 a12 a13 = tr @4 b12 b22 5 A a21 a22 a23 b13 b23 2 3 b11a11 + b21a21 ?? = tr 4 ? b12a12 + b22a22 ? 5 ?? b13a13 + b23a23 = b11a11 + b21a21 + b12a12 + b22a22 + b13a13 + b23a23 Note that I left some entries in the matrix BT A blank because it was not necessary to calculate them in order to compute the trace. In fact, the entries b a b a b a on the diagonal are 11 · 11 , 12 · 12 , and 13 · 13 . b21 a21 b22 a22 b23 a23 That is, the i-th diagonal entry is the dot product of the i-th column of B with the i-th column of A. As you may now be guessing, this will be true in our general case. Consider that our definition for matrix multiplication was that the ij-th entry of BA is the dot product of the i-th row of B with the j-th column of A. But our \B" is currently BT , and the i-th row of BT is the same as the i-th column of B, so the ij-th entry of BT A is the dot product of the i-th column of B and the j-th column of A. Since we are looking for the trace of BT A, we need to take the sum of the diagonal entries (the ii entries), which will be the dot product of the i-th column of B with the i-th column of A. So, h ~ ~ i if A = ~a1 ··· ~an and B = b1 ··· bn , then 3 n T X~ hA; Bi = tr(B A) = bi · ~ai i=1 With this formula in hand, let's show that h ; i is an inner product. Pn (1) hA; Ai = i=1 ~ai ·~ai. Since ~ai ·~ai ≥ 0 for all i (as the dot product is positive Pn Pn definite), we know that i=1 ~ai · ~ai ≥ 0. Moreover, if i=1 ~ai · ~ai = 0, then we need each ~ai · ~ai = 0, and again using the fact that the dot product is positive definite, this means that ~ai = 0 for all i. And since the columns of A are all the zero vector, we get that A is the zero matrix. As such, we have shown that h ; i is positive definite. Pn ~ (2) We have already shown that hA; Bi = i=1 bi ·~ai, so now we want to look at hB; Ai. By the same arguments we used for hA; Bi, we know that the ij-th entry of AT B is the dot product of the i-th column of A with the j-th column of B. T Pn ~ This means that tr(A B) = i=1 ~ai · bi. Since dot products are symmetrical, ~ ~ we know that ~ai · bi = bi · ~ai. And thus we have the following: hB; Ai = tr(AT B) Pn ~ = i=1 ~ai · bi Pn ~ = i=1 bi · ~ai = hA; Bi Pn ~ (3) hA; sB + tCi = i=1(sbi + t~ci) · ~ai Pn ~ = i=1 sbi · ~ai + t~ci · ~ai Pn ~ Pn = s i=1 bi · ~ai + t i=1 ~ci · ~ai = shA; Bi + thA; Ci Before we leave this example, let's take a further look at the formula hA; Bi = T Pn ~ ~ tr(B A) = i=1 bi ·~ai. Because the dot product bi ·~ai is the sum bi1ai1 + ··· + bimaim. But since we cycle through all possible i values when we take the inner Pn Pn product, what we end up with is that hA; Bi = i=1 j=1 bijaij. Don't let the double sum scare you! The important fact is what is inside: that we are taking the product of the corresponding entries of A and B, and adding them together. This formula is quite reminiscent of our formula for the dot product in Rn. (And, in fact, under the standard isomorphism from M(m; n) to Rmn, this is the same as the dot product.) So whenever we need to calculate tr(BT A), we Pn Pn know it is i=1 j=1 bijaij.