Numerical Analysis Numerical Analysis L. Ridgway Scott PRINCETON UNIVERSITY PRESS PRINCETON AND OXFORD Copyright c 2011 by Princeton University Press Published by Princeton University Press, 41 William Street, Princeton, New Jersey 08540 In the United Kingdom: Princeton University Press, 6 Oxford Street, Woodstock, Oxfordshire OX20 1TW press.princeton.edu All Rights Reserved Library of Congress Control Number: 2010943322 ISBN: 978-0-691-14686-7 British Library Cataloging-in-Publication Data is available The publisher would like to acknowledge the author of this volume for type- setting this book using LATEX and Dr. Janet Englund and Peter Scott for providing the cover photograph Printed on acid-free paper ∞ Printed in the United States of America 10987654321 Dedication To the memory of Ed Conway1 who, along with his colleagues at Tulane University, provided a stable, adaptive, and inspirational starting point for my career. 1Edward Daire Conway, III (1937–1985) was a student of Eberhard Friedrich Ferdinand Hopf at the University of Indiana. Hopf was a student of Erhard Schmidt and Issai Schur. Contents Preface xi Chapter 1. Numerical Algorithms 1 1.1 Finding roots 2 1.2 Analyzing Heron’s algorithm 5 1.3 Where to start 6 1.4 An unstable algorithm 8 1.5 General roots: effects of floating-point 9 1.6 Exercises 11 1.7 Solutions 13 Chapter 2. Nonlinear Equations 15 2.1 Fixed-point iteration 16 2.2 Particular methods 20 2.3 Complex roots 25 2.4 Error propagation 26 2.5 More reading 27 2.6 Exercises 27 2.7 Solutions 30 Chapter 3. Linear Systems 35 3.1 Gaussian elimination 36 3.2 Factorization 38 3.3 Triangular matrices 42 3.4 Pivoting 44 3.5 More reading 47 3.6 Exercises 47 3.7 Solutions 50 Chapter 4. Direct Solvers 51 4.1 Direct factorization 51 4.2 Caution about factorization 56 4.3 Banded matrices 58 4.4 More reading 60 4.5 Exercises 60 4.6 Solutions 63 viii CONTENTS Chapter 5. Vector Spaces 65 5.1 Normed vector spaces 66 5.2 Proving the triangle inequality 69 5.3 Relations between norms 71 5.4 Inner-product spaces 72 5.5 More reading 76 5.6 Exercises 77 5.7 Solutions 79 Chapter 6. Operators 81 6.1 Operators 82 6.2 Schur decomposition 84 6.3 Convergent matrices 89 6.4 Powers of matrices 89 6.5 Exercises 92 6.6 Solutions 95 Chapter 7. Nonlinear Systems 97 7.1 Functional iteration for systems 98 7.2 Newton’s method 103 7.3 Limiting behavior of Newton’s method 108 7.4 Mixing solvers 110 7.5 More reading 111 7.6 Exercises 111 7.7 Solutions 114 Chapter 8. Iterative Methods 115 8.1 Stationary iterative methods 116 8.2 General splittings 117 8.3 Necessary conditions for convergence 123 8.4 More reading 128 8.5 Exercises 128 8.6 Solutions 131 Chapter 9. Conjugate Gradients 133 9.1 Minimization methods 133 9.2 Conjugate Gradient iteration 137 9.3 Optimal approximation of CG 141 9.4 Comparing iterative solvers 147 9.5 More reading 147 9.6 Exercises 148 9.7 Solutions 149 CONTENTS ix Chapter 10. Polynomial Interpolation 151 10.1 Local approximation: Taylor’s theorem 151 10.2 Distributed approximation: interpolation 152 10.3 Norms in infinite-dimensional spaces 157 10.4 More reading 160 10.5 Exercises 160 10.6 Solutions 163 Chapter 11. Chebyshev and Hermite Interpolation 167 11.1 Error term ω 167 11.2 Chebyshev basis functions 170 11.3 Lebesgue function 171 11.4 Generalized interpolation 173 11.5 More reading 177 11.6 Exercises 178 11.7 Solutions 180 Chapter 12. Approximation Theory 183 12.1 Best approximation by polynomials 183 12.2 Weierstrass and Bernstein 187 12.3 Least squares 191 12.4 Piecewise polynomial approximation 193 12.5 Adaptive approximation 195 12.6 More reading 196 12.7 Exercises 196 12.8 Solutions 199 Chapter 13. Numerical Quadrature 203 13.1 Interpolatory quadrature 203 13.2 Peano kernel theorem 209 13.3 Gregorie-Euler-Maclaurin formulas 212 13.4 Other quadrature rules 219 13.5 More reading 221 13.6 Exercises 221 13.7 Solutions 224 Chapter 14. Eigenvalue Problems 225 14.1 Eigenvalue examples 225 14.2 Gershgorin’s theorem 227 14.3 Solving separately 232 14.4 How not to eigen 233 14.5 Reduction to Hessenberg form 234 14.6 More reading 237 14.7 Exercises 238 14.8 Solutions 240 x CONTENTS Chapter 15. Eigenvalue Algorithms 241 15.1 Power method 241 15.2 Inverse iteration 250 15.3 Singular value decomposition 252 15.4 Comparing factorizations 253 15.5 More reading 254 15.6 Exercises 254 15.7 Solutions 256 Chapter 16. Ordinary Differential Equations 257 16.1 Basic theory of ODEs 257 16.2 Existence and uniqueness of solutions 258 16.3 Basic discretization methods 262 16.4 Convergence of discretization methods 266 16.5 More reading 269 16.6 Exercises 269 16.7 Solutions 271 Chapter 17. Higher-order ODE Discretization Methods 275 17.1 Higher-order discretization 276 17.2 Convergence conditions 281 17.3 Backward differentiation formulas 287 17.4 More reading 288 17.5 Exercises 289 17.6 Solutions 291 Chapter 18. Floating Point 293 18.1 Floating-point arithmetic 293 18.2 Errors in solving systems 301 18.3 More reading 305 18.4 Exercises 305 18.5 Solutions 308 Chapter 19. Notation 309 Bibliography 311 Index 323 Preface “...by faith and faith alone, embrace, believing where we cannot prove,” from In Memoriam by Alfred Lord Ten- nyson, a memorial to Arthur Hallum. Numerical analysis provides the foundations for a major paradigm shift in what we understand as an acceptable “answer” to a scientific or techni- cal question. In classical calculus we look for answers like √sin x, that is, answers composed of combinations of names of functions that are familiar. This presumes we can evaluate such an expression as needed, and indeed numerical analysis has enabled the development of pocket calculators and computer software to make this routine. But numerical analysis has done much more than this. We will see that far more complex functions, defined, e.g., only implicitly, can be evaluated just as easily and with the same tech- nology. This makes the search for answers in classical calculus obsolete in many cases. This new paradigm comes at a cost: developing stable, con- vergent algorithms to evaluate functions is often more difficult than more classical analysis of these functions. For this reason, the subject is still be- ing actively developed. However, it is possible to present many important ideas at an elementary level, as is done here. Today there are many good books on numerical analysis at the graduate level, including general texts [47, 134] as well as more specialized texts. We reference many of the latter at the ends of chapters where we suggest fur- ther reading in particular areas. At a more introductory level, the recent trend has been to provide texts accessible to a wide audience. The book by Burden and Faires [28] has been extremely successful. It is a tribute to the importance of the field of numerical analysis that such books and others [131] are so popular. However, such books intentionally diminish the role of advanced mathematics in the subject of numerical analysis. As a result, numerical analysis is frequently presented as an elementary subject. As a corollary, most students miss exposure to numerical analysis as a mathemat- ical subject. We hope to provide an alternative. Several books written some decades ago addressed specifically a mathe- matical audience, e.g., [80, 84, 86]. These books remain valuable references, but the subject has changed substantially in the meantime. We have intentionally introduced concepts from various parts of mathe- matics as they arise naturally. In this sense, this book is an invitation to study more deeply advanced topics in mathematics. It may require a short detour to understand completely what is being said regarding operator the- xii PREFACE ory in infinite-dimensional vector spaces or regarding algebraic concepts like tensors and flags. Numerical analysis provides, in a way that is accessible to advanced undergraduates, an introduction to many of the advanced concepts of modern analysis. We have assumed that the general style of a course using this book will be to prove theorems. Indeed, we have attempted to facilitate a “Moore2 method” style of learning by providing a sequence of steps to be verified as exercises. This has also guided the set of topics to some degree. We have tried to hit the interesting points, and we have kept the list of topics covered as short as possible. Completeness is left to graduate level courses using the texts we mention at the end of many chapters. The prerequisites for the course are not demanding. We assume a sophis- ticated understanding of real numbers, including compactness arguments. We also assume some familiarity with concepts of linear algebra, but we in- clude derivations of most results as a review. We have attempted to make the book self-contained. Solutions of many of the exercises are provided. About the name: the term “numerical” analysis is fairly recent. A clas- sic book [170] on the topic changed names between editions, adopting the “numerical analysis” title in a later edition [171]. The origins of the part of mathematics we now call analysis were all numerical, so for millennia the name “numerical analysis” would have been redundant. But analysis later developed conceptual (non-numerical) paradigms, and it became useful to specify the different areas by names.