Appendix A Linear Algebra Overview Some of the topics in this text are approached from the point of view of generalizing fundamental notions from linear algebra. In addition, some of the examples and problems assume a certain level of familiarity with this subject. In this appendix we collect the relevant definitions and basic results for reference. A.1 Vector spaces Definition (Vector Space). A vector space V over a field of scalars F is defined to be a set on which there is a (vector) addition operation defined, together with an operation of scalar multiplication of vectors by elements of F. The operations and objects are required to satisfy the following conditions: 1 1. u C v D v C u,forallu;v 2 V. 2. There exists a vector 0 such that v C 0 D v for all v 2 V. 3. For each v there exists an additive inverse .v/ such that v C .v/ D 0. 4. For scalars ˛; ˇ 2 F .˛ C ˇ/v D ˛vC ˇv; ˛.u C v/ D ˛ u C ˛v: In many applications (including those in this text) the scalar field is either the real or complex numbers. 1There is of course a distinction between addition of scalars and addition of vectors. However, the symbol “C” is used for both, as the intent is determinable from the context. © Springer International Publishing Switzerland 2016 709 J.H. Davis, Methods of Applied Mathematics with a Software Overview, Applied and Numerical Harmonic Analysis, DOI 10.1007/978-3-319-43370-7 710 A Linear Algebra Overview Examples of vector spaces include Rn and Cn (n-tuples of real or complex numbers), sets of polynomials of degree N or less, as well as various spaces of functions defined on some domain. In the latter case, one may consider “arbitrary” real valued functions, although for applications in analysis spaces of functions satisfying some constraint of analytical regularity are of interest. Instances of the latter are spaces of continuous functions on some interval, functions continuously differentiable on some interval, or functions square-integrable on some interval. What is essential in these examples is that the desired analytical property is preserved under the operation of the formation of linear combinations of the functions in the set. Basic notions in the context of vector spaces are those of linear independence, basis, and dimension. v N Definition (Linear independence). A set of vectors f igiD1 in a vector space V is linearly independent if the equality c1 v1 C c2 v2 CCcn vn D 0 holds only when c1 D c2 DDcN D 0 (so that all the coefficients vanish). v N Definition (Span). Given a set of vectors U DfigiD1 the set of all linear combinations of elements of U, S Dfv D c1 v1 C c2 v2 CCcn vng; v N is called the span of U Df igiD1. This is a subspace of the vector space V. Definition (Basis, dimension). In the case when there exists a linearly independent v N v N set f igiD1 with the property that the vector space V is the span of f igiD1 ,thenV v N is said to have dimension N,andf igiD1 is called a basis of V. There exist vector spaces for which it is not possible to find a finite set of vectors such that every vector is expressible as a linear combination of this set. Such vector spaces are called infinite-dimensional. Typical examples are the function spaces mentioned above. Even in this context, it is useful to consider the spans of certain finite sets of vectors. These produce finite-dimensional subspaces (of an infinite- dimensional vector space) to which results of (finite-dimensional) linear algebra may be usefully applied. v N If the finite-dimensional vector space V has a basis B Df igiD1, then it is easy to see that the coefficients of the expansion of a given vector v in terms of the basis are unique. If v is expanded in a basis B, v D c1 v1 C c2 v2 CCcn vn; A.2 Linear Mappings 711 then we call the array 2 3 c1 6 7 6 7 6 c2 7 Œv 6 7 B D 6 : 7 6 : 7 4 : 5 c N B v v N the column vector of coordinates of with respect to the basis B Df igiD1. Recall that in the above it is useful to emphasize the dependence of the coordinates on the choice of basis B. A central result of linear algebra deals with the relationship of coordinate vectors v N computed with respect to two different sets of basis vectors B Df igiD1 and C D N fuigiD1. This result is the statement that the coordinates are related by multiplication by a non-singular (the so-called change of basis) matrix: in the above notation for coordinates Œv Œv B D P C is the form of the relation. A.2 Linear Mappings Linear mappings with domain the vector space U and range in the vector space V are those mappings which respect the operations of vector addition and scalar multiplication. T W U 7! V is a linear mapping from U into V if and only if T .˛u1 C ˇu2/ D ˛ T.u1/ C ˇ T.u2/ holds for all u1; u2 2 V and all scalars ˛; ˇ 2 F. The prototypical linear mapping is the operation of matrix multiplication acting on column vectors of real or complex numbers. This example is central, since it is possible to show that an arbitrary linear mapping T from one finite-dimensional vector space U to another finite-dimensional vector space V has a unique represen- tation as matrix multiplication acting on coordinate column vectors with respect to N v N bases C DfuigiD1,andB Df igiD1 for U and V. In our previous notation, this takes the form Œ Œ C Œ : T u B D T B u C Œ C The matrix T B is called the matrix of the linear transformation T with respect to the bases B and C. 712 A Linear Algebra Overview In the case that T maps the finite-dimensional vector space U into itself, it is of interest to determine the effect of a change of basis on the matrix of the transformation T. From the matrix representation of T Œ Œ B Œ T u B D T B u B and the change of basis relations Œv Œv ; B D P C we compute Œ Œ T u B D P T u C Œ C Œ D P T C u C Œ C 1 Œ D P T C P P u C Œ C 1 Œ D P T C P u B which identifies the matrix of T with respect to the new basis B as the matrix Œ B Œ C 1: T B D P T C P This matrix is said to be similar to the matrix Œ C T C computed with respect to C. A.3 Inner Products Vector spaces equipped with an inner product structure find applications in a wide variety of problems. They play a central role in the solution of boundary value problems (see Chapters 2, 3, and 4), and find application in other problems ranging from optimal control to statistical estimation of random processes. Definition. An inner product space is a vector space V (in general with complex numbers as the field of scalars) equipped with an inner product function .:; :/ defined from pairs of vectors V V and taking values in C (the complex numbers), satisfying (for all u;v;w 2 V;˛;ˇ2 C) the conditions A.5 Canonical Forms 713 1. .˛u C ˇv; w/ D ˛.u; w/ C ˇ.v;w/, 2. .u; w/ D .w; u/, 3. .u; u/>0, 4. .u; u/ D 0 if and only if u D 0. This definition allows the introduction of a distance function (or norm) according 1 to kukD.u; u/ 2 . In addition, the geometrically based intuition about mutually orthogonal vectors in R2 or R3 can be extended to general inner product spaces by declaring vectors to be orthogonal whenever .u;v/D 0. It can be shown that a finite-dimensional inner product space U always has a basis consisting of mutually orthogonal vectors of unit norm. This is called an orthonormal basis for U. Examples and further properties of inner product spaces are discussed in Chapter 2. A.4 Linear Functionals and Dual Spaces It might at first be thought that (since the real or complex numbers form a one- dimensional vector space) linear mappings from a vector space to the scalar field over which the vector space is defined would be of comparatively little interest. If has turned out (especially in the case of infinite-dimensional vector spaces arising in mathematical analysis, see Section 7.8) that consideration of such mappings is quite profitable. In the special case of inner product spaces, mappings of the form (f afixed vector) f .u/ D .u; f / are examples of such mappings. For finite-dimensional inner product spaces (and some infinite-dimensional situations) it is also true that a linear functional (a linear mapping U 7! C, the complex numbers) is representable in the indicated form for some fixed vector. For vector spaces U without an inner product, linear mappings U 7! C may still be considered. Since such objects may be added and scalar multiplied, they form a vector space themselves. This vector space of linear mappings is referred to as the dual space to the vector space U.