Transpose & Dot Product Def: The transpose of an m × n matrix A is the n × m matrix AT whose columns are the rows of A. So: The columns of AT are the rows of A. The rows of AT are the columns of A. 21 43 1 2 3 Example: If A = , then AT = 2 5 : 4 5 6 4 5 3 6 Convention: From now on, vectors v 2 Rn will be regarded as \columns" (i.e.: n × 1 matrices). Therefore, vT is a \row vector" (a 1 × n matrix). Observation: Let v; w 2 Rn. Then vT w = v · w. This is because: 2 3 w1 T . v w = v1 ··· vn 4 . 5 = v1w1 + ··· + vnwn = v · w: wn Where theory is concerned, the key property of transposes is the following: Prop 18.2: Let A be an m × n matrix. Then for x 2 Rn and y 2 Rm: (Ax) · y = x · (AT y): Here, · is the dot product of vectors. Extended Example Let A be a 5 × 3 matrix, so A: R3 ! R5. ◦ N(A) is a subspace of ◦ C(A) is a subspace of The transpose AT is a matrix, so AT : ! ◦ C(AT ) is a subspace of ◦ N(AT ) is a subspace of Observation: Both C(AT ) and N(A) are subspaces of . Might there be a geometric relationship between the two? (No, they're not equal.) Hm... Also: Both N(AT ) and C(A) are subspaces of . Might there be a geometric relationship between the two? (Again, they're not equal.) Hm... Orthogonal Complements Def: Let V ⊂ Rn be a subspace. The orthogonal complement of V is the set ? n V = fx 2 R j x · v = 0 for every v 2 V g: So, V ? consists of the vectors which are orthogonal to every vector in V . Fact: If V ⊂ Rn is a subspace, then V ? ⊂ Rn is a subspace. Examples in R3: ◦ The orthogonal complement of V = f0g is V ? = R3 ◦ The orthogonal complement of V = fz-axisg is V ? = fxy-planeg ◦ The orthogonal complement of V = fxy-planeg is V ? = fz-axisg ◦ The orthogonal complement of V = R3 is V ? = f0g Examples in R4: ◦ The orthogonal complement of V = f0g is V ? = R4 ◦ The orthogonal complement of V = fw-axisg is V ? = fxyz-spaceg ◦ The orthogonal complement of V = fzw-planeg is V ? = fxy-planeg ◦ The orthogonal complement of V = fxyz-spaceg is V ? = fw-axisg ◦ The orthogonal complement of V = R4 is V ? = f0g Prop 19.3-19.4-19.5: Let V ⊂ Rn be a subspace. Then: (a) dim(V ) + dim(V ?) = n (b) (V ?)? = V (c) V \ V ? = f0g (d) V + V ? = Rn. Part (d) means: \Every vector x 2 Rn can be written as a sum x = v + w where v 2 V and w 2 V ?." Also, it turns out that the expression x = v + w is unique: that is, there is only one way to write x as a sum of a vector in V and a vector in V ?. Meaning of C(AT ) and N(AT ) Q: What does C(AT ) mean? Well, the columns of AT are the rows of A. So: C(AT ) = column space of AT = span of columns of AT = span of rows of A: For this reason: We call C(AT ) the row space of A. Q: What does N(AT ) mean? Well: x 2 N(AT ) () AT x = 0 () (AT x)T = 0T () xT A = 0T : So, for an m × n matrix A, we see that: N(AT ) = fx 2 Rm j xT A = 0T g. For this reason: We call N(AT ) the left null space of A. Relationships among the Subspaces Theorem: Let A be an m × n matrix. Then: ◦ C(AT ) = N(A)? ◦ N(AT ) = C(A)? Corollary: Let A be an m × n matrix. Then: ◦ C(A) = N(AT )? ◦ N(A) = C(AT )? Prop 18.3: Let A be an m × n matrix. Then rank(A) = rank(AT ). Motivating Questions for Reading Problem 1: Let b 2 C(A). So, the system of equations Ax = b does have solutions, possibly infinitely many. Q: What is the solution x of Ax = b with kxk the smallest? Problem 2: Let b 2= C(A). So, the system of equations Ax = b does not have any solutions. In other words, Ax − b 6= 0. Q: What is the vector x that minimizes the error kAx−bk? That is, what is the vector x that comes closest to being a solution to Ax = b? Orthogonal Projection Def: Let V ⊂ Rn be a subspace. Then every vector x 2 Rn can be written uniquely as x = v + w; where v 2 V and w 2 V ?: n n The orthogonal projection onto V is the function ProjV : R ! R given by: ProjV (x) = v. (Note that ProjV ? (x) = w.) Prop 20.1: Let V ⊂ Rn be a subspace. Then: ProjV + ProjV ? = In: Of course, we already knew this: We have x = v+w = ProjV (x)+ProjV ? (x). n Formula: Let fv1;:::; vkg be a basis of V ⊂ R . Let A be the n × k matrix 2 3 A = 4v1 ··· vk5: Then: T −1 T ProjV = A(A A) A : (∗) Geometry Observations: Let V ⊂ Rn be a subspace, and x 2 Rn a vector. (1) The distance from x to V is: kProjV ? (x)k = kx − ProjV (x)k. (2) The vector in V that is closest to x is: ProjV (x). Derivation of (∗): Notice ProjV (x) is a vector in V = span(v1;:::; vk) = C(A) = Range(A), and k therefore ProjV (x) = Ay for some vector y 2 R . ? ? T Now notice that x − ProjV (x) = x − Ay is a vector in V = C(A) = N(A ), which means that AT (x − Ay) = 0, which means AT x = AT Ay. Now, it turns out that our matrix AT A is invertible (proof in L20), so we get y = (AT A)−1AT x. T −1 T Thus, ProjV (x) = Ay = A(A A) A x. ♦ Minimum Magnitude Solution Prop 19.6: Let b 2 C(A) (so Ax = b has solutions). Then there exists T exactly one vector x0 2 C(A ) with Ax0 = b. And: Among all solutions of Ax = b, the vector x0 has the smallest length. In other words: There is exactly one vector x0 in the row space of A which solves Ax = b { and this vector is the solution of smallest length. To Find x0: Start with any solution x of Ax = b. Then x0 = ProjC(AT )(x): Least Squares Approximation Idea: Suppose b 2= C(A). So, Ax = b has no solutions, so Ax − b 6= 0. We want to find the vector x∗ which minimizes the error kAx∗ − bk. That is, we want the vector x∗ for which Ax∗ is the closest vector in C(A) to b. In other words, we want the vector x∗ for which Ax∗ − b is orthogonal to C(A). So, Ax∗ − b 2 C(A)? = N(AT ), meaning that AT (Ax∗ − b) = 0, i.e.: AT Ax∗ = AT b: Quadratic Forms (Intro) Given an m × n matrix A, we can regard it as a linear transformation T : Rn ! Rm. In the special case where the matrix A is a symmetric matrix, we can also regard A as defining a \quadratic form": Def: Let A be a symmetric n × n matrix. The quadratic form associated n to A is the function QA : R ! R given by: QA(x) = x · Ax (· is the dot product) 2 3 x1 T . = x Ax = x1 ··· xn A 4 . 5 xn Notice that quadratic forms are not linear transformations! Orthonormal Bases Def: A basis fw1;:::; wkg for a subspace V is an orthonormal basis if: (1) The basis vectors are mutually orthogonal: wi · wj = 0 (for i 6= j); (2) The basis vectors are unit vectors: wi · wi = 1. (i.e.: kwik = 1) Orthonormal bases are nice for (at least) two reasons: (a) It is much easier to find the B-coordinates [v]B of a vector when the basis B is orthonormal; (b) It is much easier to find the projection matrix onto a subspace V when we have an orthonormal basis for V . n Prop: Let fw1;:::; wkg be an orthonormal basis for a subspace V ⊂ R . (a) Every vector v 2 V can be written v = (v · w1)w1 + ··· + (v · wk)wk: (b) For all x 2 Rn: ProjV (x) = (x · w1)w1 + ··· + (x · wk)wk: T (c) Let A be the matrix with columns fw1;:::; wkg. Then A A = Ik, so: T −1 T T ProjV = A(A A) A = AA : Orthogonal Matrices Def: An orthogonal matrix is an invertible matrix C such that C−1 = CT : n Example: Let fv1;:::; vng be an orthonormal basis for R . Then the matrix 2 3 C = 4v1 ··· vn5 is an orthogonal matrix. In fact, every orthogonal matrix C looks like this: the columns of any orthogonal matrix form an orthonormal basis of Rn. Where theory is concerned, the key property of orthogonal matrices is: Prop 22.4: Let C be an orthogonal matrix. Then for v; w 2 Rn: Cv · Cw = v · w: Gram-Schmidt Process Since orthonormal bases have so many nice properties, it would be great if we had a way of actually manufacturing orthonormal bases. That is: n Goal: We are given a basis fv1;:::; vkg for a subspace V ⊂ R . We would like an orthonormal basis fw1;:::; wkg for our subspace V . Notation: We will let V1 = span(v1) V2 = span(v1; v2) . Vk = span(v1;:::; vk) = V: Idea: Build an orthonormal basis for V1, then for V2, :::; up to Vk = V .