D f − 0 (m+1) (m) (m) 1 (m) = f = − [ (ƒ )] ƒ i + − x x J (x ) i 1 f i− 2h 1 LECTURES IN BASIC COMPUTATIONAL NUMERICAL ANALYSIS J. M. McDonough ƒ(y,t) y′ = University of Kentucky Lexington, KY 40506 E-mail: [email protected] LECTURES IN BASIC COMPUTATIONAL NUMERICAL ANALYSIS J. M. McDonough Departments of Mechanical Engineering and Mathematics University of Kentucky c 1984, 1990, 1995, 2001, 2004, 2007 Contents 1 Numerical Linear Algebra 1 1.1 Some Basic Facts from Linear Algebra . ......... 1 1.2 SolutionofLinearSystems. ........ 5 1.2.1 Numerical solution of linear systems: direct elimination ............ 5 1.2.2 Numerical solution of linear systems: iterative methods ............ 16 1.2.3 Summary of methods for solving linear systems . .......... 24 1.3 The Algebraic Eigenvalue Problem . ......... 25 1.3.1 Thepowermethod................................ 26 1.3.2 Inverse iteration with Rayleigh quotient shifts . .............. 29 1.3.3 TheQRalgorithm ................................ 30 1.3.4 Summary of methods for the algebraic eigenvalue problem........... 31 1.4 Summary ......................................... 32 2 Solution of Nonlinear Equations 33 2.1 Fixed-Point Methods for Single Nonlinear Equations . ............... 33 2.1.1 Basic fixed-point iteration . ....... 33 2.1.2 Newtoniteration ............................... 34 2.2 Modifications to Newton’s Method . ........ 38 2.2.1 Thesecantmethod............................... 38 2.2.2 Themethodoffalseposition . ..... 39 2.3 Newton’s Method for Systems of Equations . .......... 41 2.3.1 Derivation of Newton’s Method for Systems . ......... 41 2.3.2 Pseudo-language algorithm for Newton’s method for systems ......... 43 2.4 Summary ......................................... 43 3 Approximation Theory 45 3.1 ApproximationofFunctions . ........ 45 3.1.1 Themethodofleastsquares. ..... 45 3.1.2 Lagrange interpolation polynomials . .......... 49 3.1.3 Cubic spline interpolation . ....... 55 3.1.4 Extraplotation ................................ 60 3.2 NumericalQuadrature . .. .. .. .. .. .. .. .. ...... 60 3.2.1 Basic Newton–Cotes quadrature formulas . ......... 60 3.2.2 Gauss–Legendrequadrature . ...... 65 3.2.3 Evaluation of multiple integrals . ......... 66 3.3 Finite-Difference Approximations . ........... 68 3.3.1 Basicconcepts ................................. 68 i ii CONTENTS 3.3.2 UseofTaylorseries............................. 68 3.3.3 Partial derivatives and derivatives of higher order . ............... 70 3.3.4 Differentiation of interpolation polynomials . ............. 71 3.4 Richardson Extrapolation Revisited . ........... 72 3.5 Computational Test for Grid Function Convergence . .............. 73 3.6 Summary ......................................... 76 4 Numerical Solution of ODEs 77 4.1 Initial-ValueProblems . ........ 77 4.1.1 Mathematical Background . ..... 77 4.1.2 Basic Single-Step Methods . ...... 80 4.1.3 Runge–KuttaMethods. 90 4.1.4 Multi-Step and Predictor-Corrector, Methods . ............ 94 4.1.5 SolutionofStiffEquations. ...... 99 4.2 Boundary-ValueProblemsforODEs . ........101 4.2.1 Mathematical Background . 102 4.2.2 ShootingMethods ............................... 103 4.2.3 Finite-Difference Methods . 104 4.3 Singular and Nonlinear Boundary-Value Problems . .............108 4.3.1 CoordinateSingularities . .......109 4.3.2 Iterative Methods for Nonlinear BVPs . ........110 4.3.3 TheGalerkinProcedure . 114 4.3.4 Summary ......................................118 5 Numerical Solution of PDEs 119 5.1 Mathematical Introduction . ........119 5.1.1 Classification of Linear PDEs . 120 5.1.2 Basic Concept of Well Posedness . 121 5.2 Overview of Discretization Methods for PDEs . ............122 5.3 ParabolicEquations .............................. 124 5.3.1 Explicit Euler Method for the Heat Equation . .........124 5.3.2 Backward-Euler Method for the Heat Equation . .........128 5.3.3 Second-Order Approximations to the Heat Equation . ...........130 5.3.4 Peaceman–Rachford Alternating-Direction-ImplicitScheme . .136 5.4 EllipticEquations ............................... 144 5.4.1 Successive Overrelaxation . .......145 5.4.2 The Alternating-Direction-Implicit Scheme . ............148 5.5 HyperbolicEquations ............................. 149 5.5.1 TheWaveEquation ............................... 149 5.5.2 First-Order Hyperbolic Equations and Systems . ...........155 5.6 Summary ......................................... 159 References 160 List of Figures 1.1 Sparse,bandmatrix ............................... ..... 12 1.2 Compactly-banded matrices: (a) tridiagonal, (b) pentadiagonal .. .. .. .. 12 1.3 Graphical analysis of fixed-point iteration: the convergentcase ............ 18 1.4 Graphical analysis of fixed-point iteration: the divergentcase ............. 19 2.1 GeometryofNewton’smethod . ...... 36 2.2 Newton’s method applied to F (x) = tanh x ....................... 37 2.3 Geometryofthesecantmethod . ....... 39 2.4 Geometryofregulafalsi . ....... 40 3.1 Least-squares curve fitting of experimental data . ............... 46 3.2 Linear interpolation of f(x,y): R2 R1 ......................... 53 → 3.3 Ill-behavior of high-order Lagrange polynomials . ................ 54 3.4 Discontinuity of 1st derivative in local linear interpolation . 55 3.5 GeometryofTrapezoidalRule. ........ 61 3.6 Grid-point Indexing on h and 2h Grids.......................... 65 4.1 Region of absolute stability for Euler’s method applied to u′ = λu .......... 83 4.2 Forward-Euler solutions to u′ = λu, λ< 0........................ 84 4.3 Comparison of round-off and truncation error . ............ 86 4.4 GeometryofRunge–Kuttamethods . ....... 91 4.5 Solutionofastiffsystem. .......100 4.6 Region of absolute stability for backward-Euler method ................101 4.7 Geometric representation of the shooting method . ..............103 4.8 Finite-Difference grid for the interval [0, 1]........................104 4.9 Matrix structure of discrete equations approximating (4.56) ..............107 5.1 Methods for spatial discretization of partial differential equations; (a) finite differ- ence, (b) finite element and (c) spectral. .........122 5.2 Mesh star for forward-Euler method applied to heat equation .............125 5.3 Matrix structure for 2-D Crank–Nicolson method. .............138 5.4 Implementation of Peaceman–Rachford ADI. ...........141 5.5 Matrix structure of centered discretization of Poisson/Dirichlet problem. 146 5.6 Analytical domain of dependence for the point (x,t)...................150 5.7 Numerical domain of dependence of the grid point (m,n +1)..............152 5.8 Difference approximation satisfying CFL condition. ................156 iii Chapter 1 Numerical Linear Algebra From a practical standpoint numerical linear algebra is without a doubt the single most important topic in numerical analysis. Nearly all other problems ultimately can be reduced to problems in numerical linear algebra; e.g., solution of systems of ordinary differential equation initial value problems by implicit methods, solution of boundary value problems for ordinary and partial dif- ferential equations by any discrete approximation method, construction of splines, and solution of systems of nonlinear algebraic equations represent just a few of the applications of numerical linear algebra. Because of this prevalence of numerical linear algebra, we begin our treatment of basic numerical methods with this topic, and note that this is somewhat nonstandard. In this chapter we begin with discussion of some basic notations and definitions which will be of importance throughout these lectires, but especially so in the present chapter. Then we consider the two main problems encountered in numerical linear algebra: i) solution of linear systems of equations, and ii) the algebraic eigenvalue problem. Much attention will be given to the first of these because of its wide applicability; all of the examples cited above involve this class of problems. The second, although very important, occurs less frequently, and we will provide only a cursory treatment. 1.1 Some Basic Facts from Linear Algebra Before beginning our treatment of numerical solution of linear systems we will review a few im- portant facts from linear algebra, itself. We typically think of linear algebra as being associated with vectors and matrices in some finite-dimensional space. But, in fact, most of our ideas extend quite naturally to the infinite-dimensional spaces frequently encountered in the study of partial differential equations. We begin with the basic notion of linearity which is crucial to much of mathematical analysis. Definition 1.1 Let S be a vector space defined on the real numbers R (or the complex numbers C), and let L be an operator (or transformation) whose domain is S. Suppose for any u, v S and ∈ a, b R (or C) we have ∈ L(au + bv)= aLu + bLv. (1.1) Then L is said to be a linear operator. Examples of linear operators include M N matrices, differential operators and integral operators. × It is generally important to be able to distinguish linear and nonlinear operators because prob- lems involving only the former can often be solved without recourse to iterative procedures. This 1 2 CHAPTER 1. NUMERICAL LINEAR ALGEBRA is seldom true for nonlinear problems, with the consequence that corresponding algorithms must be more elaborate. This will become apparent as we proceed. One of the most fundamental properties of any object, be it mathematical or physical, is its size. Of course, in numerical analysis we are always concerned with the size of the error in any particular