6 Orthogonality and Least Squares INTRODUCTORY EXAMPLE The North American Datum and GPS Navigation Imagine starting a massive project that you estimate will normal equations, which involved 928,735 equations in take ten years and require the efforts of scores of people 928,735 variables.1 to construct and solve a 1,800,000 by 900,000 system More recently, knowledge of reference points on the of linear equations. That is exactly what the National ground has become crucial for accurately determining Geodetic Survey did in 1974, when it set out to update the locations of satellites in the satellite-based Global the North American Datum (NAD)—a network of 268,000 Positioning System (GPS). A GPS satellite calculates its precisely located reference points that span the entire North position relative to the earth by measuring the time it takes American continent, together with Greenland, Hawaii, the for signals to arrive from three ground transmitters. To do Virgin Islands, Puerto Rico, and other Caribbean islands. this, the satellites use precise atomic clocks that have been The recorded latitudes and longitudes in the NAD synchronized with ground stations (whose locations are must be determined to within a few centimeters because known accurately because of the NAD). they form the basis for all surveys, maps, legal property The Global Positioning System is used both for boundaries, and layouts of civil engineering projects determining the locations of new reference points on the such as highways and public utility lines. However, ground and for finding a user’s position on the ground more than 200,000 new points had been added to the relative to established maps. When a car driver (or a datum since the last adjustment in 1927, and errors had mountain climber) turns on a GPS receiver, the receiver gradually accumulated over the years, due to imprecise measures the relative arrival times of signals from at measurements and shifts in the earth’s crust. Data least three satellites. This information, together with the gathering for the NAD readjustment was completed in transmitted data about the satellites’ locations and message 1983. times, is used to adjust the GPS receiver’s time and to The system of equations for the NAD had no solution determine its approximate location on the earth. Given in the ordinary sense, but rather had a least-squares information from a fourth satellite, the GPS receiver can solution, which assigned latitudes and longitudes to the even establish its approximate altitude. reference points in a way that corresponded best to the 1.8 1 million observations. The least-squares solution was found A mathematical discussion of the solution strategy (along with details of the entire NAD project) appears in North American Datum of 1983, in 1986 by solving a related system of so-called Charles R. Schwarz (ed.), National Geodetic Survey, National Oceanic and Atmospheric Administration (NOAA) Professional Paper NOS 2, 1989. 329 330 CHAPTER 6 Orthogonality and Least Squares Both the NAD and GPS problems are solved by finding system of equations. A careful explanation of this apparent a vector that “approximately satisfies” an inconsistent contradiction will require ideas developed in the first five sections of this chapter. WEB In order to find an approximate solution to an inconsistent system of equations that has no actual solution, a well-defined notion of nearness is needed. Section 6.1 introduces the concepts of distance and orthogonality in a vector space. Sections 6.2 and 6.3 show how orthogonality can be used to identify the point within a subspace W that is nearest to a point y lying outside of W . By taking W to be the column space of a matrix, Section 6.5 develops a method for producing approximate (“least-squares”) solutions for inconsistent linear systems, such as the system solved for the NAD report. Section 6.4 provides another opportunity to see orthogonal projections at work, creating a matrix factorization widely used in numerical linear algebra. The remaining sections examine some of the many least-squares problems that arise in applications, including those in vector spaces more general than Rn. 6.1 INNER PRODUCT, LENGTH, AND ORTHOGONALITY Geometric concepts of length, distance, and perpendicularity, which are well known for R2 and R3, are defined here for Rn. These concepts provide powerful geometric tools for solving many applied problems, including the least-squares problems mentioned above. All three notions are defined in terms of the inner product of two vectors. The Inner Product If u and v are vectors in Rn, then we regard u and v as n 1 matrices. The transpose uT is a 1 n matrix, and the matrix product uT v is a 1 1 matrix, which we write as a single real number (a scalar) without brackets. The number uT v is called the inner product of u and v, and often it is written as uv. This inner product, mentioned in the exercises for Section 2.1, is also referred to as a dot product. If u1 v1 u2 v2 u 2 : 3 and v 2 : 3 D : D : 6 7 6 7 6 u 7 6 v 7 6 n 7 6 n 7 4 5 4 5 then the inner product of u and v is v1 v2 Œ u1 u2 un 2 : 3 u1v1 u2v2 unvn : D C C C 6 7 6 v 7 6 n 7 4 5 6.1 Inner Product, Length, and Orthogonality 331 2 3 EXAMPLE 1 Compute uv and vu for u 5 and v 2 . D 2 1 3 D 2 3 3 SOLUTION 4 5 4 5 3 T uv u v Œ 2 5 1 2 .2/.3/ . 5/.2/ . 1/. 3/ 1 D D 2 3 3 D C C D 4 2 5 T vu v u Œ 3 2 3 5 .3/.2/ .2/. 5/ . 3/. 1/ 1 D D 2 1 3 D C C D 4 5 It is clear from the calculations in Example 1 why uv vu. This commutativity D of the inner product holds in general. The following properties of the inner product are easily deduced from properties of the transpose operation in Section 2.1. (See Exercises 21 and 22 at the end of this section.) THEOREM 1 Let u, v, and w be vectors in Rn, and let c be a scalar. Then a. uv vu D b. .u v/w uw vw C D C c. .cu/v c.uv/ u.cv/ D D d. uu 0, and uu 0 if and only if u 0 D D Properties (b) and (c) can be combined several times to produce the following useful rule: .c1u1 cpup/w c1.u1 w/ cp.up w/ C C D C C The Length of a Vector n If v is in R , with entries v1; : : : ; vn, then the square root of vv is defined because vv is nonnegative. DEFINITION The length (or norm) of v is the nonnegative scalar v defined by k k 2 v pvv v2 v2 v2; and v vv k k D D 1 C 2 C C n k k D q 2 a x Suppose v is in R , say, v . If we identify v with a geometric point in the 2 D b (a, b) plane, as usual, then v coincides with the standard notion of the length of the line k k √a2 + b2 segment from the origin to v. This follows from the Pythagorean Theorem applied to a |b| triangle such as the one in Fig. 1. x A similar calculation with the diagonal of a rectangular box shows that the definition |a| 0 1 of length of a vector v in R3 coincides with the usual notion of length. For any scalar c, the length of cv is c times the length of v. That is, FIGURE 1 j j Interpretation of v as length. cv c v k k k k D j jk k (To see this, compute cv 2 .cv/ .cv/ c2vv c2 v 2 and take square roots.) k k D D D k k 332 CHAPTER 6 Orthogonality and Least Squares A vector whose length is 1 is called a unit vector. If we divide a nonzero vector v by its length—that is, multiply by 1= v —we obtain a unit vector u because the length k k of u is .1= v / v . The process of creating u from v is sometimes called normalizing k k k k v, and we say that u is in the same direction as v. Several examples that follow use the space-saving notation for (column) vectors. EXAMPLE 2 Let v .1; 2; 2; 0/. Find a unit vector u in the same direction as v. D SOLUTION First, compute the length of v: 2 2 2 2 2 v vv .1/ . 2/ .2/ .0/ 9 k k D D C C C D v p9 3 k k D D Then, multiply v by 1= v to obtain k k 1 1=3 1 1 1 2 2=3 u v v 2 3 2 3 D D 3 D 3 2 D 2=3 x2 v W k k 6 0 7 6 0 7 6 7 6 7 4 5 4 5 To check that u 1, it suffices to show that u 2 1. k k D k k D 2 2 2 2 1 2 2 .0/2 1 u u u 3 3 3 x k k D D C C C 1 4 4 0 1 x1 9 9 9 1 D C C C D EXAMPLE 3 W R2 .