Lecture Notes on Solving Large Scale Eigenvalue Problems Prof. Dr. Peter Arbenz Computer Science Department ETH Z¨urich Email: [email protected] Spring semester 2016 ii Contents 1 Introduction 1 1.1 What makes eigenvalues interesting? ...................... 1 1.2 Example 1: The vibrating string ........................ 2 1.2.1 Problem setting .............................. 2 1.2.2 The method of separation of variables ................. 5 1.3 Numerical methods for solving 1-dimensional problems ............ 6 1.3.1 Finite differences ............................. 6 1.3.2 The finite element method ........................ 7 1.3.3 Global functions ............................. 8 1.3.4 A numerical comparison ......................... 9 1.4 Example 2: The heat equation ......................... 9 1.5 Example 3: The wave equation ......................... 12 1.6 The 2D Laplace eigenvalue problem ...................... 13 1.6.1 The finite difference method ....................... 13 1.6.2 The finite element method (FEM) ................... 16 1.6.3 A numerical example ........................... 20 1.7 Cavity resonances in particle accelerators ................... 21 1.8 Spectral clustering ................................ 23 1.8.1 The graph Laplacian ........................... 24 1.8.2 Spectral clustering ............................ 25 1.8.3 Normalized graph Laplacians ...................... 27 1.9 Google’s PageRank ................................ 28 1.10 Other sources of eigenvalue problems ...................... 30 Bibliography ...................................... 31 2 Basics 33 2.1 Notation ...................................... 33 2.2 Statement of the problem ............................ 34 2.3 Similarity transformations ............................ 37 2.4 Schur decomposition ............................... 38 2.5 The real Schur decomposition .......................... 39 2.6 Normal matrices ................................. 40 2.7 Hermitian matrices ................................ 41 2.8 The Jordan normal form ............................. 43 2.9 Projections .................................... 45 2.10 The Rayleigh quotient .............................. 47 2.11 Cholesky factorization .............................. 49 2.12 The singular value decomposition (SVD) .................... 50 iii iv CONTENTS Bibliography ...................................... 52 3 Newton methods 53 3.1 Linear and nonlinear eigenvalue problems ................... 53 3.2 Zeros of the determinant ............................. 54 3.2.1 Algorithmic differentiation ........................ 55 3.2.2 Hyman’s algorithm ............................ 55 3.2.3 Computing multiple zeros ........................ 58 3.3 Newton methods for the constrained matrix problem ............. 58 3.4 Successive linear approximations ........................ 60 Bibliography ...................................... 61 4 The QR Algorithm 63 4.1 The basic QR algorithm ............................. 63 4.1.1 Numerical experiments .......................... 64 4.2 The Hessenberg QR algorithm ......................... 67 4.2.1 A numerical experiment ......................... 69 4.2.2 Complexity ................................ 70 4.3 The Householder reduction to Hessenberg form ................ 71 4.3.1 Householder reflectors .......................... 71 4.3.2 Reduction to Hessenberg form ..................... 71 4.4 Improving the convergence of the QR algorithm ................ 73 4.4.1 A numerical example ........................... 74 4.4.2 QR algorithm with shifts ........................ 75 4.4.3 A numerical example ........................... 76 4.5 The double shift QR algorithm ......................... 77 4.5.1 A numerical example ........................... 81 4.5.2 The complexity .............................. 83 4.6 The symmetric tridiagonal QR algorithm ................... 84 4.6.1 Reduction to tridiagonal form ...................... 84 4.6.2 The tridiagonal QR algorithm ...................... 85 4.7 Research ...................................... 87 4.8 Summary ..................................... 87 Bibliography ...................................... 88 5 Cuppen’s Divide and Conquer Algorithm 91 5.1 The divide and conquer idea ........................... 91 5.2 Partitioning the tridiagonal matrix ....................... 92 5.3 Solving the small systems ............................ 92 5.4 Deflation ..................................... 93 5.4.1 Numerical examples ........................... 94 5.5 The eigenvalue problem for D + ρvvT ..................... 95 5.6 Solving the secular equation ........................... 98 5.7 A first algorithm ................................. 99 5.7.1 A numerical example ...........................100 5.8 The algorithm of Gu and Eisenstat .......................103 5.8.1 A numerical example [continued] ....................104 Bibliography ......................................107 CONTENTS v 6 LAPACK and the BLAS 109 6.1 LAPACK .....................................109 6.2 BLAS .......................................110 6.2.1 Typical performance numbers for the BLAS ..............111 6.3 Blocking ......................................113 6.4 LAPACK solvers for the symmetric eigenproblems ..............114 6.5 Generalized Symmetric Definite Eigenproblems (GSEP) ...........116 6.6 An example of a LAPACK routines .......................116 Bibliography ......................................122 7 Vector iteration (power method) 125 7.1 Simple vector iteration ..............................125 7.2 Angles between vectors ..............................126 7.3 Convergence analysis ...............................127 7.4 A numerical example ...............................130 7.5 The symmetric case ...............................131 7.6 Inverse vector iteration ..............................135 7.7 The generalized eigenvalue problem .......................139 7.8 Computing higher eigenvalues ..........................139 7.9 Rayleigh quotient iteration ...........................140 7.9.1 A numerical example ...........................143 Bibliography ......................................144 8 Simultaneous vector or subspace iterations 145 8.1 Basic subspace iteration .............................145 8.2 Angles between subspaces ............................146 8.3 Convergence of basic subspace iteration ....................148 8.4 Accelerating subspace iteration .........................153 8.5 Relation between subspace iteration and QR algorithm ............158 8.6 Addendum ....................................161 Bibliography ......................................161 9 Krylov subspaces 163 9.1 Introduction ....................................163 9.2 Definition and basic properties .........................164 9.3 Polynomial representation of Krylov subspaces ................165 9.4 Error bounds of Saad ...............................168 Bibliography ......................................171 10 Arnoldi and Lanczos algorithms 173 10.1 An orthonormal basis for the Krylov space j(x) ...............173 K 10.2 Arnoldi algorithm with explicit restarts ....................175 10.3 The Lanczos basis ................................176 10.4 The Lanczos process as an iterative method ..................178 10.5 An error analysis of the unmodified Lanczos algorithm ............185 10.6 Partial reorthogonalization ...........................187 10.7 Block Lanczos ...................................190 10.8 External selective reorthogonalization .....................193 Bibliography ......................................194 vi CONTENTS 11 Restarting Arnoldi and Lanczos algorithms 195 11.1 The m-step Arnoldi iteration ..........................195 11.2 Implicit restart ..................................196 11.3 Convergence criterion ..............................198 11.4 The generalized eigenvalue problem .......................199 11.5 A numerical example ...............................201 11.6 Another numerical example ...........................206 11.7 The Lanczos algorithm with thick restarts ...................210 11.8 Krylov–Schur algorithm .............................213 11.9 The rational Krylov space method .......................214 Bibliography ......................................215 12 The Jacobi-Davidson Method 217 12.1 The Davidson algorithm .............................217 12.2 The Jacobi orthogonal component correction .................218 12.2.1 Restarts ..................................221 12.2.2 The computation of several eigenvalues ................221 12.2.3 Spectral shifts ..............................222 12.3 The generalized Hermitian eigenvalue problem ................224 12.4 A numerical example ...............................224 12.5 The Jacobi–Davidson algorithm for interior eigenvalues ............228 12.6 Harmonic Ritz values and vectors ........................229 12.7 Refined Ritz vectors ...............................231 12.8 The generalized Schur decomposition ......................233 12.9 JDQZ: Computing a partial QZ decomposition ................233 12.9.1 Restart ..................................235 12.9.2 Deflation .................................235 12.9.3 Algorithm .................................236 12.10Jacobi-Davidson for nonlinear eigenvalue problems ..............236 Bibliography ......................................239 13 Rayleigh quotient and trace minimization 241 13.1 Introduction ....................................241 13.2 The method of steepest descent .........................242 13.3 The conjugate gradient