The Projection Matrix

David Arnold Fall 1996

Abstract In this activity you will use Matlab to project a set of vectors onto a single vector. Prerequisites. Inner product (dot product) and orthogonal vectors.

1 The Inner Product

We begin with the de nition of the inner product.

De nition 1 Let K and L be vectors from 4n. The inner product (dot prod- uct) of vectors K and L is de ned by

K ¢ L = KT L · ¸ · ¸ 1 ¡4 Example 2 If K = and L = , then 7 2

T K ¢ L = K L · ¸ £ ¤ ¡4 = 17 2 =10

Check this result in Matlab. >> u=[1;7] u=

1 7

>> v=[-4;2] v=

-4

1 2

>> u’*v ans =

10 If K, L and M are vectors from 4n and c is any real number, then it is not di¢cult to show that each of the following properties are true:

² K ¢ L = L ¢ K ² K ¢ (L + M) = K ¢ L + K ¢ M ² (cK) ¢ L = K ¢ (cL) = c(K ¢ L) ² K ¢ K ¸ 0 and K ¢ K =0if and only if K = .

Finally, orthogonal (perpendicular) vectors are de ned as follows.

De nition 3 Let K and L be vectors from 4n. Vectors K and L are orthogonal if and only if K ¢ L =0.

1.1 The Projection of One Vector Onto Another Figure 1 shows the projection of vector K onto vector L.

u

projLu v Figure 1. The projection of K onto L. In Figure 2, it is clear that the projection of K onto L is some scalar multiple of L; that is, projLK = cL.

2 u

u-cv

cv v Figure 2. The vector K ¡ cL is orthogonal to L. Note that the di¤erence vector K ¡ cL is orthogonal (perpendicular) to L. Consequently, L ¢ (K ¡ cL) =0 and we can use the previously mentioned properties to arrive at the following result:

L ¢ K ¡ L¢(cL) =0 L ¢ K ¡ c(L ¢ L) =0 c(L ¢ L) = L ¢ K L ¢ K c = (1) L ¢ L

Finally, substitute result (1) into projLK = cL. L ¢ K proj K = L (2) L L ¢ L · ¸ 1 Example 4 Use formula (2) to nd the projection of K = onto L = 7 · ¸ ¡4 : 2

L ¢ K projLK = L L·¢ L ¸ · ¸ ¡4 1 ¢ · ¸ 2 7 ¡4 = · ¸ · ¸ ¡4 ¡4 2 ¢ 2 2 · ¸ 10 ¡4 = 20 2 · ¸ ¡2 = 1

Check this result in Matlab.

3 >> (v’*u)/(v’*v)*v ans =

-2 1

1.2 The Projection Matrix Let’s rework formula (2). First, when you multiply a vector by a scalar, it doesn’t matter whether you position the scalar before or after the vector. L ¢ K proj K = L L L ¢ L L ¢ K = L L ¢ L Next, use the transpose de nition of the inner product followed by the associa- tive property of multiplication. Remember, when performing the dot product, a scalar multiplier may be placed anywhere you wish. 1 proj K = L(LT K) L LT L 1 = (LLT )K LT L LLT = K LT L The expression LLT is called an outer product (the transpose operator is outside the product versus its inside position in the inner product). If we de ne LLT P = , then the projection formula becomes LT L LLT proj K = PK, where P = : L LT L The matrix P is called the projection matrix. You can project any vector onto the vector L by multiplying by the matrix P . · ¸ · ¸ 1 ¡4 Example 5 Let K = onto L = and nd P, the matrix that will 7 2 project any matrix onto the vector L. Use the result to nd projLK. First, nd the projection matrix. LLT P = LT L · ¸· ¸ ¡4 ¡4 T 2 2 = · ¸ · ¸ ¡4 T ¡4 2 2

4 · ¸ ¡4 £ ¤ ¡42 2 = · ¸ £ ¤ ¡4 ¡42 2 · ¸ 16 ¡8 ¡84 = · 20 ¸ : 8 ¡: 4 = ¡: 4 : 2

Check this result in Matlab. >> P=(v*v’)/(v’*v)

P=

0.8000 -0.4000 -0.4000 0.2000 Use this result to nd the projection of K onto L.

proj K = PK L · ¸· ¸ : 8 ¡: 4 1 = ¡: 4 : 2 7 · ¸ ¡2 = 1

Check this last result in Matlab. >> P*u ans =

-2 1 Note that this is identical to the result in Example 4.

1.3 Projecting a Lot of Vectors onto a Single Vector First, use Matlab to generate 100 random vectors and plot them. The following commands will produce an image similar to that in Figure 3. >> U=8*rand(2,100)-4; >> x=U(1,:); >> y=U(2,:); >> plot(x,y,’o’)

5 4

2

0

-2

-4 -4 -2 0 2 4 Figure 3. The image in Figure 3 bears some explanation. Each circle in Figure 3 represents the tip of a vector whose tail begins at the origin. Next, use the projection matrix P to project each of the 100 2 £1 vectors in matrix U onto the vector L, then plot the results on the graph in the color red. >> PU=P*U; >> x=PU(1,:); >> y=PU(2,:); >> hold on >> plot(x,y,’ro’)

4

2

0

-2

-4 -4 -2 0 2 4 Figure 4.

In Figuure 4, note that each vector· in¸ the matrix U is projected onto a line ¡4 in the direction of the vector L = . 2 2 Homework

1. Create 100 random 2 £ 1 vectors in a matrix U with components between ¡2 and 2. Plot them.

6 (a) Create a matrix P which will· project¸ each of the 2 £ 1 vectors in 1 matrix U onto the vector L = . 1 (b) Use the matrix P to project each 2 £ 1 vector in matrix U onto the vector L. Plot the results in a second color. (c) Obtain a printout of your result.

2. The column space of the projection matrix P is a line in the direction of the vector L.

(a) What is the dimension of the column space of P ? Check your answer with the ATLAST distribution’s colbasis command. (b) What is the rank of P ? Check your answer with Matlab’s rank command.

3. If matrix P projects all vectors onto the vector L, what should the matrix PP do? Can you explain geometrically why the matrices PP and P are equal? Check this in Matlab.

7