Numerical Linear Algebra William Layton and Myron Sussman University of Pittsburgh Pittsburgh, Pennsylvania c 2014 William Layton and Myron Sussman. All rights reserved. ISBN 978-1-312-32985-0 Contents Contents i Introduction vii Sources of Arithmetical Error . xii Measuring Errors: The trademarked quantities . xix Linear systems and finite precision arithmetic xxiii Vectors and Matrices . xxiii Eigenvalues and singular values . xxviii Error and residual . xxxvi When is a linear system solvable? . xlii When is an N×N matrix numerically singular? . xlvi I Direct Methods 1 1 Gaussian Elimination 3 1.1 Elimination + Backsubstitution . 4 1.2 Algorithms and pseudocode . 8 1.3 The Gaussian Elimination Algorithm . 10 1.4 Pivoting Strategies . 16 1.5 Tridiagonal and Banded Matrices . 23 1.6 The LU decomposition . 28 i ii CONTENTS 2 Norms and Error Analysis 41 2.1 FENLA and Iterative Improvement . 41 2.2 Vector Norms . 46 2.3 Matrix Norms .................... 52 2.4 Error, Residual and Condition Number . 59 2.5 Backward Error Analysis . 65 II Iterative Methods 79 3 The MPP and the Curse of Dimensionality 81 3.1 Derivation . 81 3.2 1-D Model Poisson Problem . 86 3.3 The 2d MPP . 96 3.4 The 3-D MPP . 105 3.5 The Curse of Dimensionality . 111 4 Iterative Methods 117 4.1 Introduction to Iterative Methods . 117 4.2 Mathematical Tools . 130 4.3 Convergence of FOR . 134 4.4 Better Iterative Methods . 145 4.5 Dynamic Relaxation . 159 4.6 Splitting Methods . 161 5 Solving Ax = b by Optimization 171 5.1 The connection to optimization . 174 5.2 Application to Stationary Iterative Methods . 181 5.3 Application to Parameter Selection . 186 5.4 The Steepest Descent Method . 191 6 The Conjugate Gradient Method 201 6.1 The CG Algorithm . 202 6.2 Analysis of the CG Algorithm . 211 CONTENTS iii 6.3 Convergence by the Projection Theorem . 212 6.4 Error Analysis of CG . 226 6.5 Preconditioning . 231 6.6 CGN for non-SPD systems . 234 7 Eigenvalue Problems 243 7.1 Introduction and Review of Eigenvalues . 243 7.2 Gershgorin Circles . 251 7.3 Perturbation theory of eigenvalues . 253 7.4 The Power Method . 256 7.5 Inverse Power, Shifts and Rayleigh Quotient Iteration261 7.6 The QR Method . 264 Bibliography 269 A An omitted proof 271 B Tutorial on basic Matlab programming 275 B.1 Objective . 275 B.2 Matlab files . 276 B.3 Variables, values and arithmetic . 279 B.4 Variables are matrices . 282 B.5 Matrix and vector Operations . 286 B.6 Flow control . 292 B.7 Script and function files . 295 B.8 Matlab linear algebra functionality . 298 B.9 Debugging . 300 B.10 Execution speed in Matlab . 303 Preface \It is the mark of an educated mind to rest satis- fied with the degree of precision that the nature of the subject permits and not to seek exactness when only an approximation is possible." - Aristotle (384 BCE) This book presents numerical linear algebra for students from a diverse audience of senior level undergraduates and beginning graduate students in mathematics, science and engineering. Typi- cal courses it serves include: A one term, senior level class on Numerical Linear Al- gebra. Typically, some students in the class will be good pro- grammers but have never taken a theoretical linear algebra course; some may have had many courses in theoretical linear algebra but cannot find the on/off switch on a computer; some have been using methods of numerical linear algebra for a while but have never seen any of its background and want to understand why methods fail sometimes and work sometimes. Part of a graduate \gateway" course on numerical meth- ods. This course gives an overview in two terms of useful methods in computational mathematics and includes a computer lab teach- ing programming and visualization connected to the methods. v vi PREFACE Part of a one term course on the theory of iterative methods. This class is normally taken by students in mathemat- ics who want to study numerical analysis further or to see deeper aspects of multivariable advanced calculus, linear algebra and ma- trix theory as they meet applications. This wide but highly motivated audience presents an interesting challenge. In response, the material is developed as follows: Every topic in numerical linear algebra can be presented algorithmically and theoretically and both views of it are important. The early sections of each chapter present the background material needed for that chapter, an essential step since backgrounds are diverse. Next methods are developed algorithmically with examples. Con- vergence theory is developed and the parts of the proofs that pro- vide immediate insight into why a method works or how it might fail are given in detail. A few longer and more technically intricate proofs are either referenced or postponed to a later section of the chapter. Our first and central idea about learning is \to begin with the end in mind". In this book the end is to provide a modern under- standing of useful tools. The choice of topics is thus made based on utility rather than beauty or completeness. The theory of al- gorithms that have proven to be robust and reliable receives less coverage than ones for which knowing something about the method can make a difference between solving a problem and not solving one. Thus, iterative methods are treated in more detail than direct methods for both linear systems and eigenvalue problems. Among iterative methods, the beautiful theory of SOR is abbreviated be- cause conjugate gradient methods are a (currently at least) method of choice for solving sparse SPD linear systems. Algorithms are given in pseudocode based on the widely used MATLAB language. The pseudocode transparently presents algorithmic steps and, at the same time, serves as a framework for computer implementation of the algorithm. The material in this book is constantly evolving. Welcome! Introduction There is no such thing as the Scientific Revolution, and this is a book about it. - Steven Shapin, The Scientific Revolution. This book presents numerical linear algebra. The presentation is intended for the first exposure to the subject for students from mathematics, computer science, engineering. Numerical linear al- gebra studies several problems: Linear Systems: Ax = b : Solve the N × N linear system. Eigenvalue Problems: Aφ = λφ : Find all the eigenvalues and eigenvectors or a selected subset. Ill-posed problems and least squares: Find a unique useful solution (that is as accurate as possible given the data errors) of a linear system that is undetermined, overdetermined or nearly singular with noisy data. We focus on the first, treat the second lightly and omit the third. This choice reflects the order the algorithms and theory are built, not the importance of the three. Broadly, there are two types of subproblems: small to medium scale and large scale. \large" in large scale problems can be defined as follows: a problem is large if memory management and turnaround time are central challenges. Thus, a problem is not large if one can simply call a canned lin- ear algebra routine and solve the problem reliably within time and vii viii INTRODUCTION resource constraints with no special expertise. Small to medium scale problems can also be very challenging when the systems are very sensitive to data and roundoff errors and data errors are sig- nificant. The latter is typical when the coefficients and RHS come from experimental data, which always come with noise. It also oc- curs when the coefficients depend on physical constants which may be known to only one significant digit. The origin of numerical linear algebra lies in a 1947 paper of von Neumann and Goldstine [VNG47]. Its table of contents, given below, is quite modern in all respects except for the omission of iterative methods: ix NUMERICAL INVERTING OF MATRICES OF HIGH ORDER JOHN VON NEUMANN AND H. H. GOLDSTINE ANALYTIC TABLE OF CONTENTS PREFACE CHAPTER I. The sources of errors in a computation 1.1. The sources of errors. (A) Approximations implied by the mathematical model. (B) Errors in observational data. (C) Finitistic approximations to transcendental and im- plicit mathematical formulations. (D) Errors of computing instruments in carrying out el- ementary operations: \Noise." Round off errors. \Analogy" and digital computing. The pseudo-op- erations. 1.2. Discussion and interpretation of the errors (A)-(D). Sta- bility 1.3. Analysis of stability. The results of Courant, Friedrichs, and Lewy 1.4. Analysis of \noise" and round off errors and their rela- tion to high speed computing 1.5. The purpose of this paper. Reasons for the selection of its problem. 1.6. Factors which influence the errors (A)-(D). Selection of the elimination method. 1.7. Comparison between \analogy" and digital computing methods x INTRODUCTION CHAPTER II. Round off errors and ordinary algebraical pro- cesses. 2.1. Digital numbers, pseudo-operations. Conventions re- garding their nature, size and use: (a), (b) 2.2. Ordinary real numbers, true operations. Precision of data. Conventions regarding these: (c), (d) 2.3. Estimates concerning the round off errors: (e) Strict and probabilistic, simple precision. Pn (f) Double precision for expressions i=1 xiyj 2.4. The approximative rules of algebra for pseudo-opera- tions 2.5. Scaling by iterated halving CHAPTER III. Elementary matrix relations. 3.1. The elementary vector and matrix operations. 3.2. Properties of jAj, jAj` and N(A). 3.3. Symmetry and definiteness 3.4. Diagonality and semi-diagonality 3.5. Pseudo-operations for matrices and vectors. The rele- vant estimates. CHAPTER IV. The elimination method. 4.1. Statement of the conventional elimination method 4.2. Positioning for size in the intermediate matrices 4.3.