Texts in Applied Mathematics 59 Åke Björck Numerical Methods in Matrix Computations Texts in Applied Mathematics Volume 59 Editors-in-chief Stuart Antman, College Park, MD, USA Leslie Greengard, New York, NY, USA Philip Holmes, Princeton, NJ, USA More information about this series at http://www.springer.com/series/1214 Åke Björck Numerical Methods in Matrix Computations 123 Åke Björck Department of Mathematics Linköping University Linköping Sweden ISSN 0939-2475 ISSN 2196-9949 (electronic) ISBN 978-3-319-05088-1 ISBN 978-3-319-05089-8 (eBook) DOI 10.1007/978-3-319-05089-8 Library of Congress Control Number: 2014943253 47A12, 47A30, 47A52, 47B36, 62J02, 62J05, 62J07, 62J12, 65-01, 65Fxx, 65F05, 65F08, 65F10, 65F15, 65F20, 65F22, 65F25, 65F35, 65F40, 65F50, 65F60, 65G30, 65G50, 65H04, 65H17 Springer Cham Heidelberg New York Dordrecht London Ó Springer International Publishing Switzerland 2015 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) In Memoriam George E. Forsythe and Gene H. Golub Preface Work on this book started more than 15 years ago, when I began a revision of a textbook from 1974 on numerical methods. That book devoted only about 90 pages to matrix computations compared to the more than 700 pages of the present book. This difference reflects not only a change in ambition, but also an increase in size and importance of the subject. A stunning growth in hardware performance has allowed more sophisticated mathematical models to be employed in sciences and engineering. In most of these applications, solution of systems of linear equations and/or eigenvalue problems lies at the core. Increased problem sizes and changes in computer architecture have also made the development of new methods and new implementations of old ones necessary. Although there is a link between matrix computations and linear algebra as taught in departments of mathematics, there are also several important differences. Matrices are used to represent many different objects such as networks and images, besides linear transformations. Concepts such as ill-conditioning, norms, and orthogonality, which do not extend to arbitrary fields, are central to matrix com- putations. This is the reason for not using ‘‘linear algebra’’ in the title of the book. This book attempts to give a comprehensible and up-to-date treatment of methods and algorithms in matrix computations. Both direct and iterative methods for linear systems and eigenvalue problems are covered. This unified approach has several advantages. Much of the theory and methods used to solve linear systems and eigenvalue problems are closely intertwined—it suffices to think of matrix factorizations and Krylov subspaces. It is inevitable that personal interests would to some extent influence the selection of topics. This is most obvious in Chap. 2, which gives an unusually broad coverage of least squares methods. Several nonstandard topics are treated, e.g., tensor problems, partial least squares, and least angle regression. Methods for solving discrete inverse problems are also treated. Nonlinear least squares prob- lems such as exponential fitting, nonlinear orthogonal regression, and logistic regression are covered. Parts of this chapter were originally written for a never published revised edition of my 1996 monograph entitled Numerical Methods for Least Squares. vii viii Preface The book is suitable for use in a two-semester course on matrix computations at advanced undergraduate or graduate level. The first semester could cover direct methods for linear systems and least squares using Chaps. 1 and 2; the second semester eigenvalue problems and iterative methods using Chaps. 3 and 4. But other combinations are possible. As prerequisite, a basic knowledge of analysis and linear algebra and some experience in programming and floating point com- putations will suffice. The text can also serve as a reference for graduates in engineering and as a basis for further research work. Problems and computer exercises are included after each section. It is highly recommended that a modern interactive system such as Matlab be available for working out these assignments. It should be stressed that the Matlab programs included in the text are mainly for illustration. They work, but are toy programs and not in any way close to pro- duction codes. To keep the book within reasonable bounds, complete proofs are not given for all theorems. For the pursuit of particular topics in more detail, the book contains a large comprehensive and up-to-date bibliography of historical and review papers, as well as recent research papers. Care has been taken to include references to the original research papers since these are often rewarding to read. More than 50 short biographical notes on mathematicians who have made significant con- tributions to matrix computations are given as footnotes in the text. When working on this book I soon realized that I was trying to hit a moving target. Having rewritten one chapter I invariably found that some other chapter now needed to be revised. Therefore, many draft versions have existed. At various stages of this process several colleagues, including Bo Einarsson and Tommy Elfving, read parts of early drafts and made many constructive comments. I am also greatly indebted to Michele Benzi, Nick Higham, and David Watkins, as well as several anonymous reviewers, whose suggestions led to major improvements in later versions of the text. I am indebted to Wlodek Proskurowski for his continuous encouragement and for using early versions of the book for courses at USC, Los Angeles. Michael Saunders somehow found time to proofread the last draft and pains- takingly corrected my faulty English language and other lapses. Without his help I would not have been able to get the book in shape. Finally, I thank Lynn Brandon, my editor at Springer, for her helpful and professional support during the publi- cation process. The book was written in Emacs and typeset in LATEX, the references were prepared in BibTEX, and the index with MakeIndex. Matlab was used for working out examples and generating figures. Using these great tools is always a joy. The biographical notes are based on the biographies compiled at the School of Mathematics and Statistics, Uni- versity of St Andrews, Scotland (www-history.mcs.st-andrews.ac.uk). The book is dedicated to the memory of George E. Forsythe and Gene H. Golub, who were my gracious and inspiring hosts during my postdoctoral stay at Stanford University in 1968. Their generosity with their time and ideas made Preface ix this stay into a decisive experience for my future life and work. Finally, I thank my wife Eva for her forbearance and understanding during all the time I spent on writing this book. Linköping, November 2013 Åke Björck Contents 1 Direct Methods for Linear Systems........................ 1 1.1 Elements of Matrix Theory. 1 1.1.1 Matrix Algebra . 2 1.1.2 Vector Spaces. 6 1.1.3 Submatrices and Block Matrices . 9 1.1.4 Operation Counts in Matrix Algorithms . 13 1.1.5 Permutations and Determinants. 14 1.1.6 The Schur Complement . 19 1.1.7 Vector and Matrix Norms . 22 1.1.8 Eigenvalues . 29 1.1.9 The Singular Value Decomposition . 32 1.2 Gaussian Elimination Methods . 37 1.2.1 Solving Triangular Systems . 38 1.2.2 Gaussian Elimination and LU Factorization . 39 1.2.3 LU Factorization and Pivoting . 44 1.2.4 Variants of LU Factorization . 50 1.2.5 Elementary Elimination Matrices . 54 1.2.6 Computing the Matrix Inverse . 57 1.2.7 Perturbation Analysis. 59 1.2.8 Scaling and componentwise Analysis . 64 1.3 Hermitian Linear Systems. 71 1.3.1 Properties of Hermitian Matrices . 71 1.3.2 The Cholesky Factorization . 76 1.3.3 Inertia of Symmetric Matrices . 80 1.3.4 Symmetric Indefinite Matrices . 82 1.4 Error Analysis in Matrix Computations . 88 1.4.1 Floating-Point Arithmetic. 89 1.4.2 Rounding Errors in Matrix Operations. 92 1.4.3 Error Analysis of Gaussian Elimination . 95 xi xii Contents 1.4.4 Estimating Condition Numbers . 101 1.4.5 Backward Perturbation Bounds . 105 1.4.6 Iterative Refinement of Solutions .