Inner Product, Orthogonality, and Orthogonal Projection

Home , Euclidean vector, Orthogonality, Orthogonal basis, Orthogonal complement

Math 240 TA: Shuyi Weng Winter 2017 March 2, 2017

Inner Product

The notion of inner product is important in linear algebra in the sense that it provides a sensible notion of length and angle in a vector space. This seems very natural in the Euclidean space Rn through the concept of dot product. However, the inner product is much more general and can be extended to other non-Euclidean vector spaces. For this course, you are not required to understand the non-Euclidean examples. I just want to show you a glimpse of linear algebra in a more general setting in mathematics.

Deﬁnition. Let V be a vector space. An inner product on V is a function

h−, −i: V × V → R such that for all vectors u, v, w ∈ V and scalar c ∈ R, it satisﬁes the axioms

1. hu, vi = hv, ui; 2. hu + v, wi = hu, wi + hv, wi, and hu, v + wi = hu, vi + hu, wi; 3. hcu, vi = chu, vi + hu, cvi; 4. hu, ui ≥ 0, and hu, ui = 0 if and only if u = 0.

Deﬁnition. In a Euclidean space Rn, the dot product of two vectors u and v is deﬁned to be the function u · v = uT v.

In coordinates, if we write     u1 v1  .   .  u =  .  , and v =  .  ,     un vn then   v1 T h i  .  u · v = u v = u1 ··· un  .  = u1v1 + ··· unvn.   vn The deﬁnitions in the remainder of this note will assume the Euclidean vector space Rn, and the dot product as the natural inner product.

Lemma. The dot product on Rn is an inner product.

Exercise. Verify that the dot product satisﬁes the four axioms of inner products. " # 7 2 Example 1. Let A = , and deﬁne the function 2 4

hu, vi = uT AvT

We will show that this function deﬁnes an inner product on R2. Write the vectors " # " # " # u v w u = 1 , v = 1 , and w = 1 , u2 v2 w2 then " #" # h i 7 2 v1 hu, vi = u1 u2 = 7u1v1 + 2u1v2 + 2u2v1 + 4u2v2. 2 4 v2 To verify axiom 1, we compute

hv, ui = 7v1u1 + 2v1u2 + 2v2u1 + 4v2u2.

Hence, we can conclude that hu, vi = hv, ui. To verify axiom 2, we see that

hu, v + wi = 7u1(v1 + w1) + 2u1(v2 + w2) + 2u2(v1 + w1) + 4u2(v2 + w2)

= (7u1v1 + 2u1v2 + 2u2v1 + 4u2v2) + (7u1w1 + 2u1w2 + 2u2w1 + 4u2w2) = hu, vi + hu, wi.

The other equality follows from axiom 1. Thus axiom 2 is veriﬁed. To verify axiom 3, we see that

chu, vi = c(7u1v1 + 2u1v2 + 2u2v1 + 4u2v2)

= 7(cu1)v1 + 2(cu1)v2 + 2(cu2)v1 + 4(cu2)v2 = hcu, vi.

The other equality also follows from axiom 1. Thus axiom 3 is veriﬁed. To verify axiom 4, notice that

2 2 2 2 hu, ui = 7u1u1 + 2u1u2 + 2u2u1 + 4u2u2 = 7u1 + 4u1u2 + 4u2 = 3u1 + 4(u1 + u2) .

If hu, ui = 0, that means u1 = u2 = 0. The converse is clear. Hence, axiom 4 is veriﬁed. Remark (Symmetric Bilinear Form). Let A ∈ Rn×n be a symmetric matrix. Then the function given by hu, vi = uT Av

for any vectors u, v ∈ Rn, defines an inner product on Rn. Inner products on Rn defined in this way are called symmetric bilinear form. In fact, every inner product on Rn is a symmetric bilinear form. In particular, the standard dot product is defined with the identity matrix I, which is symmetric.

2 ∞ Example 2 (` -Space*). Consider the real-valued sequences x = {xi}i=1 satisfying

∞ X 2 xi < ∞ i=1

Deﬁne the space `2(N) as the collection of all such sequences, i.e., n ∞ o 2 ∞ X 2 ` (N) = x = {xi}i=1 xi ∈ R, and xi < ∞ . i=1

It is a good exercise to verify that `2(N) is a vector space with scalars in the real numbers. We deﬁne a function by ∞ X hx, yi = xiyi, i=1 and show that this function deﬁnes an inner product on `2(N). To verify axiom 1, we see that

∞ ∞ X X hy, xi = yixi = xiyi = hx, yi. i=1 i=1 To verify axiom 2, we see that

∞ ∞ ∞ X X X hx, y + zi = xi(yi + zi) = xiyi + xizi = hx, yi + hx, zi. i=1 i=1 i=1 To verify axiom 3, we see that

∞ ∞ X X chx, yi = c xiyi = (cxi)yi = hcx, yi. i=1 i=1 To verify axiom 4, notice that ∞ X 2 hx, xi = xi i=1 It is clear from here that hx, xi = 0 if and only if x is the zero sequence. Deﬁnition. The length (or norm) of a vector v ∈ Rn, denoted by kvk, is deﬁned by √ q 2 2 kvk = v · v = v1 + ··· vn

Remark. By the last axiom of the inner product, v · v ≥ 0, thus the length of v is always a non-negative real number, and the length is 0 if and only if v is the zero vector.

Deﬁnition. A vector with length 1 is called a unit vector. If v = 0, then the vector 1 1 u = v = √ v kvk v · v is the normalization of v.

Example 3. Consider the vector 2   v = 2 . 1 Its length is √ √ √ kvk = v · v = 22 + 22 + 12 = 9 = 3, and its normalization is 2/3 1 1 u = v = v = 2/3 . kvk 3   1/3

Deﬁnition. For vectors u, v ∈ Rn, we can deﬁne the distance between them by

dist(u, v) = ku − vk. " # " # −4 1 Example 4. Let u = and v = , then the distance between them is 3 1 √ ku − vk = p(−4 − 1)2 + (3 − 1)2 = 29.

This distance is demonstrated in the following ﬁgure.

v Example 5 (`2-Distance*). Analogous to the definition of length in Rn, the `2-norm is defined by ∞ 1/2 p X 2 kxk2 = hx, xi = xi i=1 Notice that by definition, every x ∈ `2(N) has a finite norm. The `2-distance is then canonically defined to be

∞ 1/2 X 2 dist(x, y) = kx − yk2 = (xi − yi) i=1

Orthogonality

The notion of inner product allows us to introduce the notion of orthogonality, together with a rich family of properties in linear algebra.

Deﬁnition. Two vectors u, v ∈ Rn are orthogonal if u · v = 0.

Theorem 1 (Pythagorean). Two vectors are orthogonal if and only if ku+vk2 = kuk2 +kvk2.

Proof. This well-known theorem has numerous different proofs. The linear-algebraic ver- sion looks like this. Notice that

ku + vk2 = (u + v) · (u + v) = u · u + u · v + v · u + v · v = kuk2 + kvk2 + 2u · v.

The theorem follows from the fact that u and v are orthogonal if and only if u · v = 0.

The following is an important concept involving orthogonality.

Deﬁnition. Let W ⊆ Rn be a subspace. If a vector x is orthogonal to every vector w ∈ W , we say that x is orthogonal to W . The orthogonal complement of W , denoted by W ⊥, is the collection of all vectors orthogonal to W , i.e.,

⊥ n W = {x ∈ R | x · w = 0 for all w ∈ W }.

Lemma. W ⊥ is a subspace of Rn.

Exercise. Verify that W ⊥ satisﬁes the axioms of a subspace.

Exercise. Let W ⊆ Rn be a subspace. Prove that dim W + dim W ⊥ = n. [Hint: Use Rank-Nullity Theorem]

⊥ Theorem 2. If {w1,..., wk} forms a basis of W , then x ∈ W if and only if x · wi = 0 for all integers 1 ≤ i ≤ k. Proof. Let {w1,..., wk} be a basis of W . Assume that x · wi = 0. Let w ∈ W be arbitrary. Then w can be written as a linear combination

w = c1w1 + ··· + ckwk.

By the linearity of dot product, we have

x · w = c1x · w1 + ··· + ckx · wk = 0 + ··· + 0 = 0.

Thus x ∈ W ⊥. The converse is clear.

Example 6. Find the orthogonal complement of W = span{w1, w2}, where     3 0     0 2 w1 =   , and w2 =   . 1 5     1 1

By the theorem above, to find all vectors x ∈ W ⊥, we only need to make sure we find all x ∈ R4 that satisfies

x · w1 = x · w2 = 0. If we write   x1   x2 x =   , x   3 x4 we immediately obtain a system of linear equations

3x1 + x3 + x4 = 0

2x2 + 5x3 + x4 = 0

A sequence of row operation gives all solutions to this linear system       −x3/3 − x4/3 2 2       −5x3/2 − x4/2  15  3 x =   = s   + t    x  −6  0  3      x4 0 −6

Exercise. Let A be an m × n matrix. Prove that (ColA)⊥ = NulAT . [Hint: Use a similar method as demonstrated in the previous example. ] n Deﬁnition. A set of vectors {u1,..., uk} in R is an orthogonal set if each pair of distinct

vectors from the set is orthogonal, i.e., ui · uj = 0 whenever i 6= j. An orthogonal basis for a subspace W is a basis for W that is also an orthogonal set. An orthonormal basis for a subspace W is an orthogonal basis for W where each vector has length 1.

n Example 7. The standard basis {e1,..., en} forms an orthonormal basis for R .

Example 8. Consider the following vectors in R4:         1 1 1 1         1  1 −1 −1 v1 =   , v2 =   , v3 =   , v4 =   . 1 −1  1 −1         1 −1 −1 1

One can easily verify that {v1, v2, v3, v4} forms an orthogonal set. Furthermore, the matrix h i v1 v2 v3 v4

is row equivalent to the identity matrix. Thus the four vectors are linearly independent. It 4 follows that {v1, v2, v3, v4} forms an orthogonal basis for R . Furthermore, if we normalize the vectors and obtain         1/2 1/2 1/2 1/2         1/2  1/2 −1/2 −1/2 u1 =   , u2 =   , u3 =   , u4 =   , 1/2 −1/2  1/2 −1/2         1/2 −1/2 −1/2 1/2

4 then {u1, u2, u3, u4} forms an orthonormal basis for R

4 Exercise. Verify that {v1, v2, v3, v4} in Example 8 forms an orthogonal basis for R .

Deﬁnition. Let A be an n × n matrix. We say A is an orthogonal matrix if AT A = I.

Theorem 3. Let A be an n × n matrix. Then the followings are equivalent.

1. A is an orthogonal matrix. 2. The column vectors of A are orthonormal. 3. The row vectors of A are orthonormal.

4. A preserves length, that is, kAxk = kxk for all x ∈ Rn. 5. A preserves the dot product, which means (Ax) · (Ay) = x · y for all x, y ∈ Rn. Orthogonal Projection

The idea of orthogonal projection is best depicted in the following ﬁgure.

u Projuv

The orthogonal projection of v onto u gives the component vector Projuv of v in the direction of u. This fact is best demonstrated in the case that u is one of the standard basis vectors.

v Proj v e2

Proj v e1 e1

As shown in the ﬁgure above, the lengths of the orthogonal projections in the e1 and e2 directions, respectively, give the coordinates of the vector v in the standard basis. On the other hand, each coordinate can be obtained by computing the dot product of v and the corresponding standard basis vector, i.e.,

kProje1 vk = v · e1, and kProje2 vk = v · e2.

However, the orthogonal projection of v in the e1 direction should not depend on the length of the vector we use to specify the direction. Hence, the validity of the observation

above is based on the fact that e1 and e2 are “special” in some sense. The observation holds

true precisely because the vectors e1 and e2 are unit vectors. To obtain a similar conclusion in the general setting, consider vectors u and v in the first figure. We first normalize u to get u û = √ u · u Now, this unit vector û satisfies that

kProjˆuvk = v · ˆu.

Because u and û are in the same direction, we have Projûv = Projuv. Thus v · u u v · u Proj v = kProj vk û = (v · û) û = √ √ = u. u û u · u u · u u · u

Deﬁnition. Given vectors u, v ∈ Rn, where u 6= 0, then the orthogonal projection of v onto u is deﬁned to be v · u Proj v = u. u u · u Exercise. Find the orthogonal projection of v onto u in each case. " # " # " # " # −1 1 1 −3 1. u = , v = 2. u = , v = 3 −1 2 9

Exercise. Find the distance from the point (−3, 9) to the line y = 2x.