Linear Algebra
Total Page:16
File Type:pdf, Size:1020Kb
Carl de Boor Linear Algebra draft 25jan13 Contents Preface vi Overview viii 1 Sets, assignments, lists, and maps Sets . 1 Assignments, lists . 2 Matrices . 5 Lists of lists . 8 Maps . 8 1-1 and onto . 10 Cardinality and the pigeonhole principle . 11 Some examples . 13 Maps and their graphs . 13 Invertibility . 15 Map composition; left and right inverse . 17 The inversion of maps . 22 2 Vector spaces and linear maps Vector spaces, especially spaces of functions . 24 Linear maps . 28 Linear maps from IFn (aka column maps) . 31 Linear maps from IFT . 39 The linear equation A? = y, and ran A and null A . 39 Inverses . 42 3 The dimension of a vector space Bases . 47 Construction of a basis . 50 Dimension . 53 The dimension of IFT . 54 ii Contents iii Some uses of the dimension concept . 55 Direct sums . 60 The only matrices whose invertibility can be ascertained at a glance 63 Polynomial interpolation . 64 4 Elimination, or: The determination of null A and ran A Elimination and Backsubstitution . 66 The really reduced row echelon form and other reduced forms . 73 The basis for null A obtained from a b-form . 76 The factorization A = A(: ; b)rrrefb(A) . 79 The basis for ran A obtained from a b-form . 79 The rrref(A) and the solving of A? = y . 81 Constructing the inverse of a matrix by elimination . 83 Elimination in vector spaces . 84 5 The inverse of a basis, and interpolation Linear maps into IFn (aka row maps) . 88 A formula for the coordinate map . 89 Change of basis . 92 Interpolation and linear projectors . 93 6 Inner product spaces Definition and examples . 99 The conjugate transpose . 100 Orthogonal projectors and closest points . 103 Least-squares . 106 Orthonormal column maps . 108 ran A and null Ac form an orthogonal direct sum for tar A . 112 The inner product space IFm×n and the trace of a matrix . 113 7 Norms, map norms, and the condition of a basis How to judge the error by the residual . 115 The map norm . 118 Vector norms and their associated map norms . 120 Any linear map close enough to an invertible linear map is invertible . 125 Bounding the interpolation error: Lebesgue's inequality . 125 8 Factorization and rank The need for factoring linear maps . 127 The trace of a linear map . 130 The rank of a matrix and of its (conjugate) transpose . 131 Elimination as factorization . 132 SVD . 134 The Pseudo-inverse . 136 2-norm and 2-condition of a matrix . 137 The effective rank of a noisy matrix . 137 The polar decomposition . 139 Equivalence and similarity . 140 iv Contents 9 Duality Complementary mathematical concepts . 142 The dual of a vector space . 144 The dual of an inner product space . 147 The dual of a linear map . 148 10 The powers of a linear map and its spectrum Examples . 150 Eigenvalues and eigenvectors . 152 Diagonalizability . 155 Are all square matrices diagonalizable? . 158 Does every square matrix have an eigenvalue? . 158 Polynomials in a linear map, Krylov subspaces, and the minimal polynomials . 160 It is enough to understand the eigenstructure of matrices . 166 Every complex (square) matrix is similar to an upper triangular matrix . 167 11 Convergence of the power sequence Convergence of sequences in a normed vector space . 171 Three interesting properties of the power sequence of a linear map . 172 Splitting off the nondefective eigenvalues . 175 Three interesting properties of the power sequence of a linear map: The sequel . 179 The power method . 181 12 Canonical forms The Schur form . 183 The primary decomposition . 186 The Jordan form . 190 The Weyr form . 195 13 Localization of eigenvalues Gershgorin's circles . 197 The trace of a linear map . 199 Determinants . 200 Annihilating polynomials . 203 The multiplicities of an eigenvalue . 205 Perron-Frobenius . 206 14 Optimization and quadratic forms Minimization . 212 Quadratic forms . 213 Reduction of a quadratic form to a sum of squares . 216 Rayleigh quotient . 217 15 More on determinants Definition and basic properties . 223 Sylvester . 230 Contents v Binet-Cauchy . 232 16 Some applications The cross product in 3-space . 234 Rotation in 3-space . 235 Flats: points, vectors, barycentric coordinates, differentiation 237 An example from CAGD . 245 Tridiagonal Toeplitz matrix . 248 Markov Chains . 249 Polynomial interpolation and divided differences . 250 Linear Programming . 253 Total positivity . 262 Least-squares approximation by broken lines . 263 The B-spline basis for a spline space . 265 Frames . 265 A multivariate polynomial interpolant of minimal degree . 266 The reduced monic Gr¨obnerbasis for a zero-dimensional ideal 270 17 Background A nonempty finite subset of R contains a maximal element 271 A nonempty bounded subset of R has a least upper bound . 271 Complex numbers . 272 Convergence of a scalar sequence . 274 A real continuous function on a compact set in Rn has a maximum . 275 Groups, rings, and fields . 276 The ring of univariate polynomials . 279 Horner, or: How to divide a polynomial by a linear factor . 280 The Euclidean Algorithm . 281 18 List of Notation Rough index for this book . 284 Preface This book is motivated by the following realizations: (1) The linear maps between a vector space X over the scalar field IF and the associated coordinate spaces IFn are efficient tools for work on theoretical and practical problems involving X. Those from IFn to X share with matrices the feature of columns, hence are called column maps, while those from X to IFn share with matrices the feature of rows, hence are called row maps. Work with a linear map A usually requires its factorization into a column map and a row map. Such factorization is most efficient for the task if the particular column map is invertible as a map to the range of A, i.e., if it is a basis for that range. (2) Gauss elimination is applied to matrices for the purpose of obtaining bases for their nullspace and for their range. It results in a sequence of ma- trices all with the same nullspace, with the last matrix making the nullspace quite evident. (3) A change of basis amounts to interpolation and vice versa. (4) Since the eigenstructure of a linear map A on a vector space X over the scalar field IF is of interest in the study of the sequence A0 = id;A1 = A; A2;::: of the powers of A, its derivation and discussion is best handled in terms of polynomials p(A) in that linear map with coefficients in IF. While determinants are indispensible and powerful tools in certain situations, they do not provide the best path to understanding eigenstructure. (5) In applications, vector spaces are, by and large, spaces of maps with the vector operations defined pointwise, or derived from such spaces in a straightforward manner. The coordinate spaces IFn are merely the simplest examples of such vector spaces. These realizations led me to volunteer to teach the follow-up linear al- gebra course offered in the Mathematics Department of the University of Wisconsin-Madison and taken by undergraduate Math majors and graduate students from science and engineering departments. Each time, I produced vi Preface vii lecture notes, and this book derives from them. The numbered problems, posed throughout the book and typeset in the smaller font of this paragraph, are meant to deepen understanding of the material. While there are answers available to all the problems, the book contains only the answers to those problems that are referred to in the text for some missing proof detail or as an illustration of a point being made. The latter problems are starred. I adhere to certain notational conventions: (1) I distinguish between equalities being asserted or derived and equali- ties that hold by definition. For the latter, I use := or =: depending on which side is being defined. (2) I distinguish between terms or phrases being defined and those being emphasized. The former are set in boldface (to make them easy to find), the latter in italic. (3) I use only one numerical sequence for labeling all the equations, fig- ures and formal statements in each chapter as that seems to me more helpful for finding any particular labeled item than the more standard separate enu- meration of various classes of items. I do, however, number separately the problems given throughout. (4) I use standard symbols, like 8 (`for all') and 9 (`there exist(s)') with the subsequent `such that' not written out, and standard abbreviations, like ‘iff' for `if and only if', and use braces, f:::g, only to delimit the description of a set. (5) A question mark in an equation indicates the unknown item, the item for which to solve the equation. (6) n-vectors, i.e., elements of Rn or, more generally, of IFn, are written in boldface, like x 2 Rn, with their entries written in subscripted italics, e.g., x = (x1; : : : ; xn). In particular, n-vectors are not written as 1-column matrices. The study of Linear Algebra is incomplete without some numerical experimen- tation. I carry out such experimentation with the help of MATLAB, a program that has grown well beyond its initial purpose of being a \Matrix laboratory" into a very handy tool for experimentation in general scientific computing. Throughout this book, there are paragraphs, typeset like this one, that provide information about MATLAB essential for experimenting with the material under discussion. Some of the problems also require MATLAB, but most of these are easily adapted to other programming languages. With that proviso, any reader not interested in numerical experimentation or well familiar with MATLAB can safely skip all such paragraphs. Overview Here is a quick run-down on this book, with various terms to be learned by studying this book printed in boldface.