ECE 586: Vector Space Methods Lecture 17: Best Approximation Henry D. Pfister Duke University 4.1: Best Approximation Let W be a subspace of a Banach space V and, for any v 2 V , consider finding a vector w 2 W such that kv − wk is as small as possible. Definition The vector w 2 W is a best approximation of v 2 V by vectors in W if kv − wk ≤ kv − w 0k ; for all w 0 2 W . Example If W is spanned by the vectors w 1;:::; w n 2 V , then we can write v = w + e = s1w 1 + ··· + snw n + e; where e = v − w is the approximation error. 1 / 9 Vector Projection Revisited Let u; v be vectors in an inner-product space V with inner product h·|·i. Lemma If hwjvi = 0, then kw +vk2 = kwk2 +2 Refhwjvig+kvk2 = kwk2 +kvk2. Definition (Vector Projection) The projection of w onto v is defined to be w w − u hwjvi u = v kvk2 0 u v Lemma Let u be the projection of w onto v. If hwjvi= 6 0, then kw − uk < kwk. Proof. hw − ujui = 0 implies kwk2 = k(w − u) + uk2 = kw − uk2 + kuk2. 2 / 9 4.1: Orthogonal Projection In an arbitrary Banach space, finding a best approximation can be hard. For the induced norm of a Hilbert space, orthogonal projection simplifies this! Theorem (Projection) Suppose W is a subspace of a Hilbert space V and v 2 V . Then, 1 The vector w 2 W is a best approximation of v 2 V by vectors in W if and only if v − w is orthogonal to every vector in W . 2 If a best approximation of v 2 V by vectors in W exists, it is unique. 3 If W is a closed subspace with a countable orthogonal basis w 1; w 2;:::, then the best approximation of v by vectors in W is 1 X hvjw i i w = 2 w i : i=1 kw i k Note: the implied linear mapping E : V ! W defined by E(v) = w is called the orthogonal projection of V onto W . Proof in live session. 3 / 9 Orthogonal Projection Example Example For the standard inner product space V = R3, let W be the subspace spanned by v 1 = (2; 2; 1); v 2 = (3; 6; 0): Then, the Gram-Schmidt process generates the orthogonal basis w 1 = (2; 2; 1); w 2 = (−1; 2; −2) and the orthogonal projection of v 2 V onto W is defined by 2 X hvjw i 1 1 Ev = i w = hvj(2; 2; 1)i + hvj(−1; 2; −2)i : 2 i 9 9 i=1 kw i k 4 / 9 4.1.1: Orthogonal Projection onto an Orthonormal Set Let V = Cn be the standard n-dimensional complex Hilbert space and n×m U 2 C be a matrix with orthonormal columns u1;:::; um: 2 3 U = 4u1 u2 ··· um5 Then, the best approximation of v 2 V by vectors in R(U), given by m X hvjui i w = 2 ui ; i=1 kui k can also be written as m H X H w = UU v = ui (ui v): i=1 5 / 9 4.1.1: What is a Projection (1) Definition A function F : X ! Y with Y ⊆ X is idempotent if F (F (x)) = F (x). When F is a linear transformation, this reduces to F 2 = F · F = F . Definition Let V be a vector space and T : V ! V be a linear transformation. If T is idempotent, then T is called a projection because T v = v if v 2 R(T ). Example The idempotent matrix A is a projection onto the first two coordinates. 21 0 13 A = 40 1 15 0 0 0 6 / 9 4.1.1: What is a Projection (2) Theorem Let V be a vector space and T : V ! V be a (linear) projection operator. Then, the range R(T ) and the N (T ) are disjoint subspaces of V . Proof. For a non-zero v 2 R(T ), there is a non-zero w 2 V such that v = T w. Thus, T v = T 2w = T w = v 6= 0. But, if v 2 N (T ) was also true, then one would get the contradiction T v = 0. Example Consider the linear transform T : V ! V defined by T = I − P, where P is a projection. It is easy to verify that T is a projection operator because T 2 = (I − P)(I − P) = I − P − P + P2 = I − P = T : In fact, T is a projection onto R(T ) = N (P) because Pv = 0 (i.e., v 2 N (P)) if and only if (I − P)v = v (i.e., v 2 R(T )). 7 / 9 4.1.1: Orthogonal Projection Operators Definition Let V be an inner-product space and P : V ! V be a (linear) projection operator. If R(P)?N (P), then P is called an orthogonal projection. Example Let V be an inner-product space P : V ! V be an orthogonal projection. Then, v = Pv + (I − P)v gives an orthogonal decomposition of v because Pv 2 R(P), (I − P)v 2 N (P), and R(P)?N (P). Theorem For V = F n with standard inner product, P is an orthogonal projection matrix if it is idempotent and Hermitian (i.e. P2 = P and PH = P). Proof. Since R(P) = fPuju 2 V g and N (P) = fv 2 V jPv = 0g, the general condition is hPuj(I − P)vi = 0 for all u; v 2 V . Simplifying this gives v H (I − P)H Pu = v H (P − PH P)u = v H (P − P2)u = 0: 8 / 9 Next Steps To continue studying after this video { Try the required reading: Course Notes EF 4.1 - 4.1.1 Or the recommended reading: LADR 6C Also, look at the problems in Assignment 7 9 / 9.