ii About this book: This introductory text is based on lectures within a multi-semester course on “Numerical Mathematics” taught by the author at Heidelberg University. The present volume treats algorithms for solving basic problems from Linear Algebra such as (large) systems of linear equations and corresponding eigenvalue problems. Theoretical as well as practical aspects are considered. Applications are found in the discretization of partial differential equations and in spectral stability analysis. As prerequisite only that prior knowledge is required as is usually taught in the introductory Analysis and Linear Algebra courses. For supporting self-study each chapter contains exercises with solutions collected in the appendix. About the author: Rolf Rannacher, retired professor of Numerical Mathematics at Heidelberg University, study of Mathematics at the University of Frankfurt/Main, doctorate 1974, postdoctorate (Habilitation) 1978 at Bonn University – 1979/1980 Vis. Assoc. Professor at the Univer- sity of Michigan (Ann Arbor, USA), thereafter Professor at Erlangen and Saarbr¨ucken, in Heidelberg since 1988 – field of interest “Numerics of Partial Differential Equations”, especially the “Finite Element Method” and its applications in the Natural Sciences and Engeneering, more than 160 scientific publications. Cover-Bild 1 (left half): Lecture Notes Mathematics LN v4 v3 v2 v1 v0 Numerical Linear Algebra ROLF RANNACHER Cover-Bild 2: Ax = b ATAx = ATb Ax = λx HEIDELBERG UNIVERSITY PUBLISHING NUMERICAL LINEAR ALGEBRA Lecture Notes Mathematics LN NUMERICAL LINEAR ALGEBRA Rolf Rannacher Institute of Applied Mathematics Heidelberg University About the author Rolf Rannacher, retired professor of Numerical Mathematics at Heidelberg Universi- ty, study of Mathematics at the University of Frankfurt/Main, doctorate 1974, post- doctorate (Habilitation) 1978 at Bonn University – 1979/1980 Vis. Assoc. Professor at the University of Michigan (Ann Arbor, USA), thereafter Professor at Erlangen ࢺHOGRILQWHUHVWّ1XPHULFVRI3DUWLDOيDQG6DDUEU¾FNHQLQ+HLGHOEHUJVLQFH LࢹHUHQWLDO(TXDWLRQVْHVSHFLDOO\WKHّ)LQLWH(OHPHQW0HWKRGْDQGLWVDSSOLFDWLRQV' LQWKH1DWXUDO6FLHQFHVDQG(QJHQHHULQJPRUHWKDQVFLHQWLࢺFSXEOLFDWLRQV Bibliographic information published by the Deutsche Nationalbibliothek 7KH'HXWVFKH1DWLRQDOELEOLRWKHNOLVWVWKLVSXEOLFDWLRQLQWKH'HXWVFKH1DWLRQDOELEOLRJUDࢺH detailed bibliographic data are available on the Internet at http://dnb.dnb.de. This book is published under the Creative Commons License 4.0 (CC BY-SA 4.0). The cover is subject to the Creative Commons License CC-BY-ND 4.0. The electronic, open access version of this work is permanently available on Heidelberg University Publishing’s website: http://heiup.uni-heidelberg.de. urn: urn:nbn:de:bsz:16-heiup-book-407-3 doi: https://doi.org/10.17885/heiup.407 Text © 2018, Rolf Rannacher ISSN 2566-4816 (PDF) ISSN 2512-4455 (Print) ISBN 978-3-946054-99-3 (PDF) ISBN 978-3-947732-00-5 (Softcover) Contents 0 Introduction 1 0.1BasicnotationofLinearAlgebraandAnalysis................ 1 0.2Linearalgebraicsystemsandeigenvalueproblems.............. 2 0.3Numericalapproaches............................. 3 0.4Applicationsandoriginofproblems...................... 4 0.4.1 Gaussianequalizationcalculus..................... 4 0.4.2 Discretization of elliptic PDEs . .................... 6 0.4.3 Hydrodynamic stability analysis .................... 10 1 Linear Algebraic Systems and Eigenvalue Problems 13 1.1 The normed Euclidean space Kn ....................... 13 1.1.1 Vectornormsandscalarproducts................... 13 1.1.2 Linearmappingsandmatrices..................... 22 1.1.3 Non-quadraticlinearsystems..................... 26 1.1.4 Eigenvaluesandeigenvectors...................... 28 1.1.5 Similaritytransformations....................... 31 1.1.6 Matrixanalysis............................. 32 1.2Spectraandpseudo-spectraofmatrices.................... 37 1.2.1 Stability of dynamical systems . .................... 37 1.2.2 Pseudo-spectrumofamatrix..................... 41 1.3Perturbationtheoryandconditioning..................... 45 1.3.1 Conditioningoflinearalgebraicsystems............... 46 1.3.2 Conditioningofeigenvalueproblems................. 48 1.4Exercises..................................... 52 2 Direct Solution Methods 55 2.1Gaussianelimination,LRandCholeskydecomposition........... 55 2.1.1 GaussianeliminationandLRdecomposition............. 55 2.1.2 Accuracyimprovementbydefectcorrection............. 64 2.1.3 InversecomputationandtheGauß-Jordanalgorithm........ 66 2.2Specialmatrices................................. 70 v vi CONTENTS 2.2.1 Bandmatrices.............................. 70 2.2.2 Diagonally dominant matrices . .................... 72 2.2.3 Positivedefinitematrices........................ 73 2.3IrregularlinearsystemsandQRdecomposition............... 76 2.3.1 Householderalgorithm......................... 78 2.4Singularvaluedecomposition......................... 82 2.5“Direct”determinationofeigenvalues..................... 88 2.5.1 Reductionmethods........................... 88 2.5.2 Hyman’smethod............................ 92 2.5.3 Sturm’smethod............................. 94 2.6Exercises..................................... 97 3 Iterative Methods for Linear Algebraic Systems 99 3.1Fixed-pointiterationanddefectcorrection.................. 99 3.1.1 Stoppingcriteria............................103 3.1.2 Constructionofiterativemethods...................105 3.1.3 Jacobi-andGauß-Seidelmethods...................108 3.2 Acceleration methods ..............................111 3.2.1 SORmethod..............................111 3.2.2 Chebyshev acceleration .........................118 3.3Descentmethods................................124 3.3.1 Gradientmethod............................126 3.3.2 Conjugategradientmethod(CGmethod)..............130 3.3.3 GeneralizedCGmethodsandKrylovspacemethods.........136 3.3.4 Preconditioning(PCGmethods)....................138 3.4Amodelproblem................................140 3.5Exercises.....................................144 4 Iterative Methods for Eigenvalue Problems 153 4.1Methodsforthepartialeigenvalueproblem..................153 4.1.1 The“PowerMethod”..........................153 4.1.2 The“InverseIteration”.........................155 4.2Methodsforthefulleigenvalueproblem....................159 CONTENTS vii 4.2.1 TheLRandQRmethod........................159 4.2.2 Computationofthesingularvaluedecomposition..........164 4.3Krylovspacemethods.............................165 4.3.1 LanczosandArnoldimethod......................167 4.3.2 Computationofthepseudo-spectrum.................172 4.4Exercises.....................................183 5 Multigrid Methods 187 5.1Multigridmethodsforlinearsystems.....................187 5.1.1 Multigridmethodsinthe“finiteelement”context..........188 5.1.2 Convergenceanalysis..........................195 5.2Multigridmethodsforeigenvalueproblems(ashortreview).........202 5.2.1 Directmultigridapproach.......................203 5.2.2 Accelerated Arnoldi and Lanczos method ...............204 5.3Exercises.....................................206 5.3.1 Generalexercises............................207 A Solutions of exercises 209 A.1Chapter1....................................209 A.2Chapter2....................................214 A.3Chapter3....................................218 A.4Chapter4....................................230 A.5Chapter5....................................238 A.5.1Solutionsforthegeneralexercises...................242 Bibliography 246 Index 250 0 Introduction Subject of this course are numerical algorithms for solving problems in Linear Algebra, such as linear algebraic systems and corresponding matrix eigenvalue problems. The emphasis is on iterative methods suitable for large-scale problems arising, e. g., in the discretization of partial differential equations and in network problems. 0.1 Basic notation of Linear Algebra and Analysis At first, we introduce some standard notation in the context of (finite dimensional) vector spaces of functions and their derivatives. Let K denote the field of real or complex numbers R or C , respectively. Accordingly, for n ∈ N ,let Kn denote the n-dimensional vector space of n-tuples x =(x1,...,xn) with components xi ∈ K,i=1,...,n.For these addition and scalar multiplication are defined by: x + y := (x1 + y1,...,xn + yn),αx:= (αx1,...,αxn),α∈ K. The elements x ∈ Kn are, depending on the suitable interpretation, addressed as “points” or “vectors” . Here, one may imagine x as the end point of a vector attached at the origin 1 of the chosen Cartesian coordinate system and the components xi as its “coordinates” with respect to this coordinate system. In general, we consider vectors as “column vec- T tors”. Within the “vector calculus” its row version is written as (x1,...,xn) .The null (or zero) vector (0,...,0) may also be briefly written as 0 . Usually, we prefer this coordinate-oriented notation over a coordinate-free notation because of its greater clearness. A set of vectors {a1,...,ak} in Kn is called “linearly independent” if k i cia =0,ci ∈ K ⇒ ci =0,i=1,...,k. i=1 Such a set of k = n linearly independent vectors is called a “basis” of Kn ,whichspans all of Kn , i. e., each element x ∈ Kn can be (uniquely) written as a linear combination of the form n i x = cia ,ci ∈ K. i=1 Each (finite dimensional) vector space, such as Kn , possesses a basis. The special “Carte- 1 n i sian basis” {e ,...,e } is formed by the “Cartesian unit vectors” e := (δ1i,...,δni), δii =1 and δij =0,for i = j, being the usual Kronecker symbol.