(FW I LEY LINEAR ALGEBRA AND ITS APPLICATIONS Second Edition aim Pure and Applied Morhemotics: A Wiley-laterscienca Series of Texts, Monographs, and Tracts Linear Algebra and Its Applications . 1 8 0 7i; :,1 9WILEY!z w 2 0 0 7 'ererrfiti.h THE WILEY BICENTENNIAL-KNOWLEDGE FOR GENERATIONS (S ach generation has its unique needs and aspirations. When Charles Wiley first opened his small printing shop in lower Manhattan in 1807. it was a generation of boundless potential searching for an identity. And we were there, helping to define a new American literary tradition. Over half a century later, in the midst of the Second Industrial Revolution, it was a generation focused on building the future. Once again, we were there, supplying the critical scientific, technical, and engineering knowledge that helped frame the world. Throughout the 20th Century, and into the new millennium, nations began to reach out beyond their own borders and a new international community was born. Wiley was there, expanding its operations around the world to enable a global exchange of ideas, opinions, and know-how. For 200 years, Wiley has been an integral part of each generation's journey, enabling the flow of information and understanding necessary to meet their needs and fulfill their aspirations. Today, bold new technologies are changing the way we live and learn. Wiley will be there, providing you the must-have knowledge you need to imagine new worlds, new possibilities, and new opportunities. Generations come and go, but you can always count on Wiley to provide you the knowledge you need, when and where you need it! WILLIAM J. PESCE PETER BOOTH WILEY PRESIDENT AND CHIEF EXECLmvE OFFICER CHAIRMAN OF THE BOARD Linear Algebra and Its Applications Second Edition PETER D. LAX New York University Courant Institute of Mathematical Sciences New York, NY Ie[Nr[N N SAL 1 8 0 7 "*WILEY2007 ! ale [Nv [NN111 LC WILEY-LNTER.SCIENCE A JOHN WILEY & SONS, INC., PUBLICATION Copyright Cl 2007 by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights reserved. Published simultaneously in Canada. No pan of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate percopy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750.4744. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 605 Third Avenue, New York, NY 10158-0012, (212) 850-6011, fax (212) 850-6008, E-Mail: PERMREQ CO WILEY.COM. For ordering and customer service, call 1-800-CALL-WILEY. Wiley Bicentennial Logo: Richard J. Pacifico Library of Congress Cataloging-in-Publication Data: Lax, Peter D. Linear algebra and its applications / Peter D. Lax. - 2nd ed. p. cm. - (Pure and applied mathematics. A Wiley-Interscience of texts, monographs and tracts) Previous ed.: Linear algebra. New York : Wiley, c1997. Includes bibliographical references and index. ISBN 978-0-471-75156-4 (cloth) 1. Algebras, Linear.1. Lax, Peter D. Linear algebra. 11. Title. QAI84.2.L38 2008 512'.5-dc22 2007023226 Printed in the United States of America 1098765432I Contents Preface Preface to the First Fditinn 1.Fundamentals Linear Space, Isomorphism Subspace Linear Dependence Basis, Dimension Quotient Space 2.Duality 13 Linear Functions 13 Dual of a Linear Space 14 Annihilator L Codimension Quadrature Formula 17 3.Linear Mappings Domain and Target Space Nullspace and Range Fundamental Theorem Underdetermined Linear Systems Interpolation Difference Equations Algebra of Linear Mappings Dimension of Nullspace and Range Transposition Similarity Projections 4. Matrices Rows and Columns Matrix Multiplication v vi CONTENTS Transposition Rank Gaussian Elimination 5.Determinant and Trace Ordered Simplices Signed Volume. Determinant Permutation Group Formula for Determinant Multiplicative Property Laplace Expansion C'ramer'c Rule Trace 6.Spectral Theory Iteration of Linear Maps Eigenvalues. Eigenvectors Fihonacci Sequence Characteristic Polynomial Trace and Determinant Revisited Spectral Mapping Theorem Cayley-Hamilton Theorem Generalized Eigenvcctors Spectral Theorem Minimal Polynomial When Are Two Matrices Similar Commuting Maps 7-Euclidean Structure Scalar Product. Distance Schwarg Inequality Onhonormal Basis Gram-Schmidt Orthogonal Complement Orthogonal Projection Adjoins Overdetermined Systems Isometry The Orthogonal Group Norm of a Linear Map Completeness Local Compactness Complex Euclidean Structure Spectral Radius Hilhert-Schmidt Norm Cross Product CONTENTS vii 8.Spectral Theory of Self-Adioint Mappings 101 Quadratic Forms 102 Law of Inertia An Spectral Resolution 105 Commuting Maps 111 Anti-Self-Adjoint Maps 112 Normal Maps 112 Rayleigh Quotient 114 Minmax Principle 116 Norm and Eigenvalues 119 9.Calculus of Vector- and Matrix-Valued Functions m Convergence in Norm 121 Rules of Differentiation 1 11 Derivative of det A(r) 126 Matrix Exponential 128 Simple Eigenvalues 129 Multiple Eigenvalues 135 Rellich's Theorem 144 Avoidance of Crossing 140 10. Matrix Inequalities 143 Positive Self-Adjoint Matrices 143 Monotone Matrix Functions 1.5.1 Gram Matrices 1.52 Schur's Theorem 1.5.3 The Determinant of Positive Matrices 154 Integral Formula for Determinants 157 Eigenvalues 16() Separation of Eigenvalues 161 Wielandt-Hoffman Theorem 1A Smallest and Largest Eigenvalue 166 Matrices with Positive Se]f-Adjoint Part 167 Polar Decomposition 169 Singular Values 170 Singular Value Decomposition 170 11. Kinematics and Dynamics 172 Axis and Angle of Rotation 172 Rigid Motion 173 Angular Velocity Vector 176 Fluid Flow 12 Curl and Divergence 179 Small Vibrations ian Conservation of Energy 182 Frequencies and Normal Modes 184 viii CONTENTS 12. Convexity 187 Convex Sets 187 Gauge Function 188 Hahn-Banach Theorem 191 Support Function 193 Caratheodorv's Theorem 195 Konig-Birkhoff Theorem 198 Helly's Theorem 199 13. The Duality Theorem 202 Farkas_Minkowski Theorem 203 Duality Theorem 206 Economics Interpretation 208 Minmax Theorem 2151 14. Normed Linear Spaces 214 Norm 214 h Nouns 215 Equivalence of Norms 217 Completeness 219 Local Compactness 219 Theorem of F. Ricsz 219 Dual Norm 222 Distance from Subspace 223 Normed Quotient Space 224 Complex Normed Spaces 226 Complex Hahn-Banach Theorem 226 Characterization of Euclidean Spaces 227 15. Linear Mappings Between Normed Linear Spaces 229 Norm of a Mapping 230 Norm of Transpose 231 Normed Algebra of Maps 232 invertible Maps 233 Spectral Radius 236 16. Positive Matrices 237 Perron's Theorem 237 Stochastic Matrices 240 Frohenius' Theorem 243 17. How to Solve Systems of Linear Equations 246 History 246 Condition Number 241E Iterative Methods 24.8. Steepest Descent 249 Chebychev Iteration 252 CONTENTS ix Three-term Chebychev Iteration 255 Optimal Three-Teen Recursion Relation 256 Rate of Convergence 261 18. How to Calculate the Eigenvalues of Seif-Adjoint Matrices 262 QR Factorization 262 Using the QR Factorization to Solve Systems of Equations 263 The QR Algorithm for Finding Eigenvalues 263 Householder Reflection for OR Factorization 266 Tridiagonal Forni 267 Analogy of QR Algorithm and Toda Flow 269 Mneer'x Theorem 223 More General Flows 276 19. Solutions 218 Bibliography 300 Appendix 1.Special Determinants 302 Appendix 2. The Pfaffian 305 Appendix 3.Symplectic Matrices 308 Appendix 4.Tensor Product 313 Appendix 5.Lattices 317 Appendix 6.Fast Matrix Multiplication 320 Appendix 7. Gershgorin's Theorem 323 Appendix 8. The Multiplicity of Eigenvalues 325 Appendix 9. The Fast Fourier Transform 328 Appendix 10. The Spectral Radius 334 Appendix 11. The Lorentz Group 342 Appendix 12. Compactness of the Unit Ball 352 Appendix 13. A Characterization of Commutators 355 Appendix 14. Liapunov's Theorem 357 Appendix 15. The Jordan Canonical Form 363 Appendix 16. Numerical Range 367 Index ,373 Preface The outlook of this second edition is the same as that of the original: to present linear algebra as the theory and practice of linear spaces and linear mappings. Where it aids understanding and calculations, I don't hesitate to describe vectors as arrays of numbers and to describe mappings as matrices. Render onto Caesar the things which are Caesar's. If you can reduce a mathematical problem to a problem in linear algebra, you can most likely solve it, provided that you know enough linear algebra. Therefore, a thorough grounding in linear algebra is highly desirable. A sound undergraduate education should offer a second course on the subject, at the senior level. I wrote this book as a suitable text for such a course. The changes made in this second edition are partly to make it more suitable as a text. Terse descriptions, especially in the early chapters, were expanded, more problems were added, and a list of solutions to selected problems has been provided. In addition, quite a bit of new material has been added, such as the compactness of the unit ball as a criterion of finite dimensionality of a normed linear space. A new chapter discusses the QR algorithm for finding the eigenvalues of a self-adjoint matrix. The Householder algorithm for turning such matrices into tridiagonal form is presented. I describe in some detail the beautiful observation of Deift, Nanda, and Tomei of the analogy between