Overview Last time we discussed orthogonal projection. We’ll review this today before discussing the question of how to find an orthonormal basis for a given subspace. From Lay, §6.4 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 1 / 24 Orthogonal projection Given a subspace W of Rn, you can write any vector y Rn as ∈ , y = yˆ + z = projW y + projW ⊥ y where yˆ W is the closest vector in W to y and z W . We call yˆ the ∈ ∈ ⊥ orthogonal projection of y onto W . Given an orthogonal basis u ,..., u for W , we have a formula to { 1 p} compute yˆ: y u y u yˆ = · 1 u + + · p u . u u 1 ··· u u p 1· 1 p· p If we also had an orthogonal basis u ,..., u for W , we could find z { p+1 n} ⊥ by projecting y onto W ⊥: y u y u z = · p+1 u + + · n u . u u p+1 ··· u u n p+1· p+1 n· n However, once we subtract off the projection of y to W , we’re left with z W . We’ll make heavy use of this observation today. ∈ ⊥ Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 2 / 24 Orthonormal bases In the case where we have an orthonormal basis u ,..., u for W , the { 1 p} computations are made even simpler: yˆ = (y u )u + (y u )u + + (y u )u . · 1 1 · 2 2 ··· · p p If = u ,..., u is an orthonormal basis for W and U is the matrix U { 1 p} whose columns are the ui, then UUT y = yˆ T U U = Ip Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 3 / 24 The Gram Schmidt Process The aim of this section is to find an orthogonal basis v ,..., v for a { 1 n} subspace W when we start with a basis x ,..., x that is not { 1 n} orthogonal. Start with v1 = x1. Now consider x2. If v1 and x2 are not orthogonal, we’ll modify x2 so that we get an orthogonal pair v1, v2 satisfying Span x , x = Span v , v . { 1 2} { 1 2} Then we modify x so get v satisfying v v = v v = 0 and 3 3 1 · 3 2 · 3 Span x , x , x = Span v , v , v . { 1 2 3} { 1 2 3} We continue this process until we’ve built a new orthogonal basis for W . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 4 / 24 Example 1 1 2 Suppose that W = Span x , x where x = 1 and x = 2 . Find an { 1 2} 1 2 0 3 orthogonal basis v , v for W . { 1 2} To start the process we put v1 = x1. We then find 1 2 x v 4 yˆ = proj x = 2· 1 v = 1 = 2 . v1 2 v v 1 2 1· 1 0 0 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 5 / 24 Now we define v = x yˆ; this is orthogonal to x = v : 2 2 − 1 1 2 2 0 x v v = x 2 · 1 v = x yˆ = 2 2 = 0 . 2 2 − v v 1 2 − − 1 · 1 3 0 3 So v2 is the component of x2 orthogonal to x1. Note that v2 is in W = Span x , x because it is a linear combination of v = x and x . { 1 2} 1 1 2 So we have that 1 0 v1 = 1 , v2 = 0 0 3 is an orthogonal basis forW . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 6 / 24 Example 2 4 Suppose that x1, x2, x3 is a basis for a subspace W of R . Describe an { } orthogonal basis for W . As in the previous example, we put • x v v = x and v = x 2· 1 v . 1 1 2 2 − v v 1 1· 1 Then v , v is an orthogonal basis for W =Span x , x = Span v , v . { 1 2} 2 { 1 2} { 1 2} x v x v Now proj x = 3· 1 v + 3· 2 v and • W2 3 v v 1 v v 2 1· 1 2· 2 x v x v v = x proj x = x 3· 1 v 3· 2 v 3 3 − W2 3 3 − v v 1 − v v 2 1· 1 2· 2 is the component of x3 orthogonal to W2. Furthermore, v3 is in W because it is a linear combination of vectors in W . Thus we obtain that v , v , v is an orthogonal basis for W . • { 1 2 3} Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 7 / 24 Theorem (The Gram Schmidt Process) n Given a basis x1, x2,..., xp for a subspace W of R , define { } v = x 1 1 x v v = x 2· 1 v 2 2 − v v 1 1· 1 x3 v1 x3 v2 v3 = x3 · v1 · v2 − v1 v1 − v2 v2 . · · . xp v1 xp vp 1 vp = xp · v1 ... · − vp 1 − v1 v1 − − vp 1 vp 1 − · − · − Then v ,..., v is an orthogonal basis for W . Also { 1 p} Span v ,..., v = Span x ,..., x for 1 k p. { 1 k } { 1 k } ≤ ≤ Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 8 / 24 Example 3 The vectors 3 3 − x = 4 , x = 14 1 − 2 5 7 − form a basis for a subspace W . Use the Gram-Schmidt process to produce an orthogonal basis for W . Step 1 Put v1 = x1. Step 2 x v v = x 2· 1 v 2 2 − v v 1 1· 1 3 3 3 − ( 100) = 14 − 4 = 6 . − 50 − 7 5 3 − Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 9 / 24 Then v , v is an orthogonal basis for W . { 1 2} To construct an orthonormal basis for W we normalise the basis v , v : { 1 2} 3 1 1 u1 = v1 = 4 v1 √50 − k k 5 3 1 1 1 1 u2 = v2 = 6 = 2 v2 √54 √6 k k 3 1 Then u , u is an orthonormal basis for W . { 1 2} Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 10 / 24 Example 4 1 6 6 − 3 8 3 Let A = − . Use the Gram-Schmidt process to find an 1 2 6 − 1 4 3 − orthogonal basis for the column space of A. Let x1, x2, x3 be the three columns of A. 1 − 3 Step 1 Put v = x = . 1 1 1 1 Step 2 6 1 3 − x2 v1 8 ( 36) 3 1 v2 = x2 · v1 = − − = . − v1 v1 2 − 12 1 1 · − 4 1 1 − − Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 11 / 24 Step 3 x3 v1 x3 v2 v3 = x3 · v1 · v2 − v1 v1 − v2 v2 6 · 1 · 3 − 3 12 3 24 1 = 6 − 12 1 − 12 1 3 1 1 − 1 2 = − . 3 4 Thus an orthogonal basis for the column space of A is given by 1 3 1 − 3 1 2 , , − . 1 1 3 1 1 4 − Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 12 / 24 Example 5 The matrix A is given by 1 0 0 1 1 0 A = . 0 1 1 0 0 1 Use the Gram-Schmidt process to show that 1 1 1 − 1 1 1 , , − 0 2 1 0 0 3 is an orthogonal basis for Col A. Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 13 / 24 Let a1, a2, a3 be the three columns of A. 1 1 Step 1 Put v = a = . 1 1 0 0 Step 2 0 1 1/2 − a2 v1 1 1 1 1/2 v2 = a2 · v1 = = . − v1 v1 1 − 2 0 1 · 0 0 0 1 − 1 For convenience we take v = . (This is optional, but it makes v 2 2 2 0 easier to work with in the following calculation.) Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 14 / 24 Step 3 0 1 1/3 − a3 v1 a3 v2 0 2 1 1/3 v3 = a3 · v1 · v2 = 0 = − − v1 v1 − v2 v2 1 − − 6 2 1/3 · · 1 0 1 1 1 For convenience we take v = − . 3 1 3 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 15 / 24 QR factorisation of matrices If an m n matrix A has linearly independent columns x ,..., x , then × 1 n A = QR for matrices Q is an m n matrix whose columns are an orthonormal basis for × Col(A), and R is an n n upper triangular invertible matrix. × This factorisation is used in computer algorithms for various computations. In fact, finding such a Q and R amounts to applying the Gram Schmidt process to the columns of A. (The proof that such a decomposition exists is given in the text.) Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 16 / 24 Example 6 Let 5 9 5/6 1/6 − 1 7 1/6 5/6 A = , Q = 3 5 3/6 1/6 − − − 1 5 1/6 3/6 where the columns of Q are obtained by applying the Gram-Schmidt process to the columns of A and then normalising the columns. Find R such that A = QR. As we have noted before, QT Q = I because the columns of Q are orthonormal. If we believe such an R exists, we have QT A = QT (QR) = (QT Q)R = IR = R. Therefore R = QT A. Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 17 / 24 In this case, R = QT A 5 9 5/6 1/6 3/6 1/6 1 7 = − " 1/6 5/6 1/6 3/6# 3 5 − − − 1 5 6 12 = "0 6 # An easy check shows that 5/6 1/6 5 9 − 1/6 5/6 6 12 1 7 QR = = = A.