Math 5620 - Introduction to Numerical Analysis - Class Notes Fernando Guevara Vasquez Version 1990. Date: January 17, 2012. 3 Contents 1. Disclaimer 4 Chapter 1. Iterative methods for solving linear systems 5 1. Preliminaries 5 2. Neumann series based methods 7 3. Conjugate gradient and other Krylov subspace methods 15 na.tex 3 Rev: 1990, January 17, 2012 4 CHAPTER 0. CONTENTS 1. Disclaimer These notes are work in progress and may be incomplete and contain errors. These notes are based on the following sources: Burden and Faires, Numerical Analysis, ninth ed. ² Kincaid and Cheney, Numerical Analysis: Mathematics of scien- ² tific computing Stoer and Burlisch, Introduction to Numerical Analysis, Springer ² 1992 Golub and Van Loan, Matrix Computations, John Hopkins 1996 ² Rev: 1990, January 17, 2012 4 na.tex CHAPTER 1. CONTENTS 5 CHAPTER 1 Iterative methods for solving linear systems 1. Preliminaries 1.1. Convergence in Rn. Let be a norm defined on Rn, i.e. it sat- k ¢ k isfies the properties: i. x Rn x 0 (non-negativity) 8 2 k k ¸ ii. x 0 x 0 (definiteness) k k Æ , Æ iii. ¸ R, x Rn ¸x ¸ x (multiplication by a scalar) 8 2 2 k k Æ j jk k iv. x,y Rn x y x y (triangle inequality) 8 2 k Å k · k k Å k k Some important examples of norms in Rn are: ¡Pn 2¢1/2 (1) x 2 i 1 xi (Euclidean or `2 norm) k k Æ Pn Æ j j (2) x 1 i 1 xi (`1 norm) k k Æ Æ j j (3) x maxi 1,...,n xi (` or max norm) k k1 Æ¡Pn Æ p ¢1/j pj 1 (4) x p i 1 xi (for p 1, `p norm) k k Æ Æ j j ¸ (k) Rn Rn A sequence {v }k1 1 in converges to v if and only if Æ 2 (1) lim v(k) v . k k ¡ k !1 Since Rn is a finite dimensional vector space the notion of conver- gence is independent of the norm. This follows from the fact that in a finite dimensional vector space all norms are equivalent. Two norms k¢k and are equivalent if there are constants ®,¯ 0 such that kj ¢ jk È (2) x ® x x ¯ x . 8 kj jk · k k · kj jk Another property of Rn is that it is complete, meaning that all Cauchy Rn (k) sequences converge in . A sequence {v }k1 1 is said to be a Cauchy sequence when Æ (3) ² 0 N i, j N v(i) v(j) ². 8 È 9 8 ¸ k ¡ k Ç In English: A Cauchy sequence is a sequence for which any two iterates can be made as close as we want provided that we are far enough in the sequence. na.tex 5 Rev: 1990, January 17, 2012 6 CHAPTER 1. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS n n 1.2. Induced matrix norms. Let A R £ be a matrix (this notation 2 means that the matrix has n rows and n columns) and let be a norm k¢¢¢k on Rn. The operator or induced matrix norm of A is defined by: Ax (4) A sup k k sup Ax . k k Æ x Rn ,x 0 x Æ x 1 k k 2 6Æ k k k kÆ The induced matrix norm measures what is the maximum dilation of the image of a vector through the matrix A. It is a good exercise to show that it satisfies the axiom of a norm. Some important examples of induced matrix norms: (1) A 1 maxj 1,...,n cj 1, where cj is the j th column of A. k k Æ Æ k k ¡ (2) A maxi 1,...,n ri 1, where ri is the i th row of A. k k1 Æ Æ k k ¡ (3) A square root of largest eigenvalue of AT A. When A is sym- k k2 Æ metric we have A ¸ , with ¸ being the largest eigen- k k2 Æ j max j max value of A in magnitude. A matrix norm that is not an induced matrix norm is the Frobenius norm: Ã n !1/2 X 2 (5) A F ai j . k k Æ i,j 1 j j Æ Some properties of induced matrix norms: (1) Ax A x . k k · k kk k (2) AB A B . k k · k kk k (3) Ak A k . k k · k k 1.3. Eigenvalues. The eigenvalues of a n n matrix A are the roots of £ the characteristic polynomial (6) p(¸) det(¸I A). Æ ¡ This is a polynomial of degree n, so it has at most n complex roots. An eigenvector v associated with an eigenvalue ¸ is a nonzero vector such that Av ¸v. Sometimes it is convenient to refer to an eigenvalue ¸ and Æ corresponding eigenvector v 0 as an eigenpair of A. 6Æ The spectral radius ½(A) is defined as the magnitude of the largest eigenvalue in magnitude of a matrix A i.e. (7) ½(A) max{ ¸ det(¸I A) 0}. Æ j j j ¡ Æ The spectral radius is the radius of the smallest circle in C containing all eigenvalues of A. Two matrices A and B are said to be similar if there is an invertible matrix X such that (8) AX XB. Æ Rev: 1990, January 17, 2012 6 na.tex CHAPTER 1. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS 7 Similar matrices have the same characteristic polynomial and thus the same eigenvalues. This follows from the properties of the determi- nant since (9) 1 1 det(¸I A) det(X¡ )det(¸I A)det(X) det(¸I X¡ AX) det(¸I B). ¡ Æ ¡ Æ ¡ Æ ¡ 2. Neumann series based methods 2.1. Neumann series. We start by a fundamental theorem that is the theoretical basis for several methods for solving the linear system Ax b. Æ This theorem can be seen as a generalization to matrices of the geometric 1 2 series identity (1 x)¡ 1 x x ... for x 1. ¡ Æ Å Å Å j j Ç n n THEOREM 1. Let A R £ be such that A 1 for some induced ma- 2 k k Ç trix norm , then: k ¢ k i. I A is invertible ¡ 1 2 P k ii. (I A)¡ I A A k1 0 A . ¡ Æ Å Å Å ¢¢¢ Æ Æ PROOF. Assume for contradiction that I A is singular. This means ¡ there is a x 0 such that (I A)x 0. Taking x such that x 1, we have 6Æ ¡ Æ k k Æ (10) 1 x Ax A x A , Æ k k Æ k k · k kk k Æ k k which contradicts the hypothesis A 1. k k Ç 1 We now need to show convergence to (I A)¡ of the partial series ¡ m X (11) Ak . k 0 Æ Observe that: m m X k X k k 1 0 m 1 (12) (I A) A A A Å A A Å . ¡ k 0 Æ k 0 ¡ Æ ¡ Æ Æ Therefore m X k m 1 m 1 (13) (I A) A I A Å A Å 0 as m . k ¡ k 0 ¡ k Æ k k · k k ! ! 1 Æ Here is an application of this theorem to estimate the norm 1 X1 k X1 k 1 (14) (I A)¡ A A . k ¡ k · k 0 k k · k 0 k k Æ 1 A Æ Æ ¡ k k Here is a generalization of the Neumann series theorem. THEOREM 2. If A and B are n n matrices such that I AB 1 for £ k ¡ k Ç some induced matrix norm then i. A and B are invertible. na.tex 7 Rev: 1990, January 17, 2012 8 CHAPTER 1. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS ii. The inverses are: 1 X1 k A¡ B (I AB) , Æ k 0 ¡ (15) " Æ # 1 X1 k B¡ (I AB) A. Æ k 0 ¡ Æ PROOF. By using the Neumann series theorem, AB I (I AB) is in- Æ ¡ ¡ vertible and 1 X1 k (16) (AB)¡ (I AB) . Æ k 0 ¡ Æ Therefore 1 1 X1 k A¡ B(AB)¡ B (I AB) , Æ Æ k 0 ¡ (17) " Æ # 1 1 X1 k B¡ (AB)¡ A (I AB) A. Æ Æ k 0 ¡ Æ 2.2. Iterative refinement. Let A be an invertible matrix. The itera- tive refinement method is a method for generating successively better approximations to the solution of the linear system Ax b. Assume we 1 Æ have an invertible matrix B such that x Bb A¡ b and applying B is Æ ¼ much cheaper than applying solving a system with the matrix A (we shall see how good the approximation needs to be later). This approximate in- verse B may come for example from an incomplete LU factorization or from running a few steps of an iterative method to solve Ax b. Can we Æ use successively refine the approximations given by this method? The idea is to look at the iteration x(0) Bb (18) Æ (k) (k 1) (k 1) x x ¡ B(b Ax ¡ ). Æ Å ¡ If this iteration converges, then the limit must satisfy (19) x x B(b Ax), Æ Å ¡ i.e. if the method converges it converges to a solution of Ax b. Æ THEOREM 3. The iterative refinement method (18) generates iterates of the form m X (20) x(m) B (I AB)k b, m 0. Æ k 0 ¡ ¸ Æ Rev: 1990, January 17, 2012 8 na.tex CHAPTER 1.