Orthogonal Projections and Orthonormal Bases
1 1 Warm-up (a) Find the angle between 2 and −1. 1 1
1 1 Solution. Let ~v = 2 and ~w = −1, and let θ be the angle between ~v and ~w. Then, 1 1 ~v · ~w = k~vkk~wk cos θ. We calculate that ~v · ~w = 1 · 1 + 2 · −1 + 1 · 1 = 0, so 0 = k~vkk~wk cos θ. Since π the lengths k~vk and k~wk are both positive, cos θ = 0, so θ = 2 . 1 0 (b) If ~v = , find k~vk, the length of ~v. 1 −2 √ √ Solution. By Definition A.6, k~vk = ~v · ~v = 6.
1 0 (c) Find a unit vector in R3 that is perpendicular to both 3 and 2. 2 0
a Solution. We are looking for a vector ~v = b which has the following three properties: c
1 a 1 • It is perpendicular to 3, so 0 = b · 3 = a + 3b + 2c. 2 c 2
0 a 0 • It is perpendicular to 2, so 0 = b · 2 = 2b. 0 c 0
• It is a unit vector, so ~v · ~v = k~vk2 is 1. That is, a2 + b2 + c2 = 1.
From the second property, we see that b = 0. Plugging this into the other 2 equations, we get 2 2 that a + 2c = 0 and a + c = 1. The former says that a = −2c; plugging√ this into the latter gives −2/ 5 5c2 = 1, so c = ± √1 . So, there are two possible answers, ± 0 . 5 √ 1/ 5
Vector Review: v1 v2 √ (a) The length of a vector ~v = is |~v| = ~vT ~v = pv2 + v2 + ··· + v2 . 1 2 n . vn
(b) If |~v| = 1, then ~v is called a unit vector.
1 (c) Let α be the angle between two vectors ~v and ~w.
~v · ~w cos α = |~v||~w|
Note: The number cos α is called the correlation coefficient. If it is positive, the vectors are positively correlated, if it is negative they are negatively correlated.
A basis {~v1,~v2, ...,~vn} is an orthonormal basis of V , if all vectors in the basis are perpendicular to each other and have length 1, which means ( 1, if i = j ~vi · ~vj = 0, if i 6= j
If they are just orthogonal, they form an orthogonal basis
1. Let V be a subspace of Rn. For any vector ~x ∈ Rn, we can write ~x = ~xk + ~x⊥ where ~xk is in V and ~x⊥ is orthogonal to V .(1) Then, ~xk is called the orthogonal projection of ~x onto V and denoted by projV (~x).
Suppose we have an orthonormal basis (~u1, . . . , ~um) of V ; that is, (~u1, . . . , ~um) is a basis of V with the property that ~u1, . . . , ~um are orthonormal.
(a) Explain why projV (~x) can be written as projV (~x) = c1~u1 +···+cm~um for some scalars c1, . . . , cm.
Solution. By definition, projV (~x) is a vector in V . On the other hand, (~u1, . . . , ~um) is a basis of V , which means that every vector in V can be expressed as a linear combination of the vectors ~u1, . . . , ~um. That is, every vector in V (including the one we’re interested in, projV (~x)) can be expressed as c1~u1 + ··· + cm~um for some scalars c1, . . . , cm.
(b) Since ~x = ~xk + ~x⊥, we can use (a) to write ⊥ ~x = (c1~u1 + ··· + cm~um) + ~x .
Express the coefficient ck in terms of ~x,~u1, . . . , ~um. Solution. A very useful technique when working with orthonormal (or even just orthogonal) ⊥ vectors is to dot with one of them. If we dot the equation ~x = (c1~u1 + ··· + cm~um) + ~x with ~uk, we get ⊥ ~x · ~uk = (c1~u1 + ··· + cm~um + ~x ) · ~uk
⊥ Now, ~uk is orthogonal to almost all of the vectors in the sum c1~u1 + ··· + cm~um + ~x . First, ⊥ ⊥ it’s perpendicular to ~x because ~x is perpendicular to all vectors in V , and ~uk ∈ V . It’s also perpendicular to all of the cj~uj except for ck~uk. So, our previous expression simplifies to:
~x · ~uk = ck(~uk · ~uk)
2 We know that ~uk · ~uk = k~ukk = 1 (the fact that ~u1, . . . , ~um are orthonormal means they all have length 1), so this simplifies even more to just:
~x · ~uk = ck
(1)When we say ~x⊥ is orthogonal to V , we mean that ~x⊥ is orthogonal to every vector in V .
2 (c) Write a formula for projV (~x) in terms of ~x,~u1, . . . , ~um. Solution. We just put together what we did in the previous two parts. We said in (a) that
projV (~x) = c1~u1 + ··· + cm~um,
and we found in (b) that ck = ~x · ~uk. So,
projV (~x) = (~x · ~u1)~u1 + (~x · ~u2)~u2 + ··· + (~x · ~um)~um
(d) In coming up with this formula for projV (~x), where was it important that (~u1, . . . , ~um) be an orthonormal basis of V ?
Solution. The fact that ~u1, . . . , ~um were orthonormal was key in (b); if the vectors had not been orthonormal, we would not have been able to come up with a simple formula for the coefficients c1, . . . , cm.
1/3 −2/3 1 2. Let V be the plane 2x + 2y + z = 0, ~u1 = −2/3, and ~u2 = 1/3. Let ~x = 4. 2/3 2/3 8
(a) Verify that (~u1, ~u2) is an orthonormal basis of V .
Solution. A basis of V consists of any two non-parallel vectors in V , so ~u1 and ~u2 clearly form a basis of V (they are both in V , and they are not parallel). To check that ~u1 and ~u2 are orthonormal, we compute some dot products:
~u1 · ~u1 = 1
~u1 · ~u2 = 0
~u2 · ~u2 = 1
So, ~u1 and ~u2 really are orthonormal.
(b) Find projV (~x). (Check that your answer is reasonable by computing the difference ~x − projV (~x). What should be true about this vector?)
Solution. We are given an orthonormal basis (~u1, ~u2) of V . Therefore, by #1(c),
projV (~x) = (~x · ~u1)~u1 + (~x · ~u2)~u2
= 3~u1 + 6~u2 −3 = 0 6
4 ⊥ Then, ~x − projV (~x) = 4; this should be orthogonal to V (it is what we called ~x in #1), and 2 it is!
3 Let {~v1,~v2, ...,~vn} be an orthonormal basis of V , and Q be the matrix containing the basis vectors as column vectors. Then the projection onto the space V is given by the matrix P = QQT , where QT is the transpose matrix.
n 3. (T/F) If ~u1, . . . , ~um are orthonormal vectors in R , then must they be linearly independent.
Solution. To decide whether ~u1, . . . , ~um are linearly independent, we should look for linear relations among them; in particular, we are wondering whether there could be nontrivial linear relations among them. Suppose we have a linear relation among them:
c1~u1 + c2~u2 + ··· + cm~um = ~0
A particularly useful technique when working with orthonormal vectors is to dot with one of the vectors, so let’s dot this equation with ~uk:
c1(~u1 · ~uk) + c2(~u2 · ~uk) + ··· + cm(~um · ~uk) = ~0 · ~uk
The right side is clearly just 0. On the left side, ~uk is orthogonal to all of the other ~ui, so there is only one non-zero term:
ck(~uk · ~uk) = 0
Since ~u1, . . . , ~um are orthonormal, ~uk has length 1, so ~uk · ~uk = 1:
ck = 0
What have we just shown? If we have a linear relation c1~u1 + ··· + cm~um, then we’ve shown that all of the ck are 0; in other words, the only linear relation among ~u1, . . . , ~um is the trivial one; this exactly says that ~u1, . . . , ~um are linearly independent.
4. Suppose that we want to fit a line to the data points (−1, 3), (0, 1), and (1, 1).
(a) Do you expect the slope of the line to be positive, negative, or zero? Solution. Looking at the data points, we see that the best-fit line should have negative slope.
(b) Find the best-fit line. (You will do a similar question in PSet12, but there you need to use a different formula) Solution. We will talk about this again later. For now you just need to know how to use the given formula as in HW12 to find this line.
5. In each part, you are given a subspace V of some Rn. Describe V ⊥. (we call V ⊥ the orthogonal complement of V )
(a) y = 3x in R2. Solution. This is a line in R2, so the orthogonal complement is the line through the origin 1 perpendicular to y = 3x, or the line y = − x . 3
4 (b) y = 3x in R3.
Solution. This is a plane in R3, so the orthogonal complement is the line through the origin per- pendicular to this plane. This plane can be expressed as 3x−y+0z = 0, so a normal vector for the 3 3 plane is −1. Therefore, the orthogonal complement of the given plane is the line span −1 . 0 0
1 (c) span 2. 3
Solution. This is a line in R3, so the orthogonal complement is the plane normal to this line, which is exactly the plane x + 2y + 3z = 0 .
n n n 6. Let V be an m-dimensional subspace of R . Consider the linear transformation projV : R → R .
(a) What is im projV ? What is rank projV ?
Solution. The image of projV is simply V itself. We know that the rank of a matrix or linear transformation is the same as the dimension of its image. So, rank projV = dim(im projV ) = dim V = m.
(b) What is ker projV ? What is its dimension? n ~ Solution. By definition, ker projV consists of all vectors ~x in R such that projV (~x) = 0. That n ~ is, it consists of all vectors ~x in R whose projection is 0. So, ker projV consists of all vectors ⊥ which are perpendicular to V (i.e., perpendicular to all vectors in V ); that is, ker projV = V .
By the rank-nullity theorem, dim(im projV ) + dim(ker projV ) = n. Therefore, dim(ker projV ) = n − dim(im projV ) = n − m .
7. See PSet12 #2 and #3 about expectation, variance, covariance, standard deviation, and the correlation coefficient, and about finding the best linear fit for some given data!
• You should understand how we use the dot product to define geometric ideas like the length of a vector and the angle between two vectors in Rn. • You should understand what it means for a collection of vectors to be orthonormal, and you should be able to determine whether given vectors are orthonormal.
• You should understand the orthogonal complement of a subspace of Rn and be able to visualize it in simple cases (in R2 or R3).
5