Preconditioned inverse iteration and shift-invert Arnoldi method Melina Freitag Department of Mathematical Sciences University of Bath CSC Seminar Max-Planck-Institute for Dynamics of Complex Technical Systems Magdeburg 3rd May 2011 joint work with Alastair Spence (Bath) 1 Introduction 2 Inexact inverse iteration 3 Inexact Shift-invert Arnoldi method 4 Inexact Shift-invert Arnoldi method with implicit restarts 5 Conclusions Outline 1 Introduction 2 Inexact inverse iteration 3 Inexact Shift-invert Arnoldi method 4 Inexact Shift-invert Arnoldi method with implicit restarts 5 Conclusions Problem and iterative methods Find a small number of eigenvalues and eigenvectors of: Ax = λx, λ ∈ C,x ∈ Cn A is large, sparse, nonsymmetric Problem and iterative methods Find a small number of eigenvalues and eigenvectors of: Ax = λx, λ ∈ C,x ∈ Cn A is large, sparse, nonsymmetric Iterative solves Power method Simultaneous iteration Arnoldi method Jacobi-Davidson method repeated application of the matrix A to a vector Generally convergence to largest/outlying eigenvector Problem and iterative methods Find a small number of eigenvalues and eigenvectors of: Ax = λx, λ ∈ C,x ∈ Cn A is large, sparse, nonsymmetric Iterative solves Power method Simultaneous iteration Arnoldi method Jacobi-Davidson method The first three of these involve repeated application of the matrix A to a vector Generally convergence to largest/outlying eigenvector Shift-invert strategy Wish to find a few eigenvalues close to a shift σ σ λλ λ λ λ 3 1 2 4 n Shift-invert strategy Wish to find a few eigenvalues close to a shift σ σ λλ λ λ λ 3 1 2 4 n Problem becomes − 1 (A − σI) 1x = x λ − σ each step of the iterative method involves repeated application of A =(A − σI)−1 to a vector Inner iterative solve: (A − σI)y = x using Krylov method for linear systems. leading to inner-outer iterative method. Shift-invert strategy This talk: Inner iteration and preconditioning Fixed shifts only Inverse iteration and Arnoldi method Outline 1 Introduction 2 Inexact inverse iteration 3 Inexact Shift-invert Arnoldi method 4 Inexact Shift-invert Arnoldi method with implicit restarts 5 Conclusions The algorithm Inexact inverse iteration for i =1 to . do choose τ (i) solve (A − σI)y(i) − x(i) = d(i)≤ τ (i), y(i) Rescale x(i+1) = , y(i) H Update λ(i+1) = x(i+1) Ax(i+1), Test: eigenvalue residual r(i+1) =(A − λ(i+1)I)x(i+1). end for The algorithm Inexact inverse iteration for i =1 to . do choose τ (i) solve (A − σI)y(i) − x(i) = d(i)≤ τ (i), y(i) Rescale x(i+1) = , y(i) H Update λ(i+1) = x(i+1) Ax(i+1), Test: eigenvalue residual r(i+1) =(A − λ(i+1)I)x(i+1). end for Convergence rates If τ (i) = Cr(i) then convergence rate is linear (same convergence rate as for exact solves). The inner iteration for (A − σI)y = x Standard GMRES theory for y0 = 0 and A diagonalisable x − (A − σI)yk≤ κ(W ) min max |p(λj )|x p∈Pk j=1,...,n −1 where λj are eigenvalues of A − σI and (A − σI)= W ΛW . The inner iteration for (A − σI)y = x Standard GMRES theory for y0 = 0 and A diagonalisable x − (A − σI)yk≤ κ(W ) min max |p(λj )|x p∈Pk j=1,...,n −1 where λj are eigenvalues of A − σI and (A − σI)= W ΛW . Number of inner iterations x k ≥ C + C log 1 2 τ for x − (A − σI)yk≤ τ. The inner iteration for (A − σI)y = x More detailed GMRES theory for y0 = 0 |λ2 − λ1| x − (A − σI)yk≤ κ˜(W ) min max |p(λj )|Qx λ1 p∈Pk−1 j=2,...,n where λj are eigenvalues of A − σI. Number of inner iterations ′ ′ Qx k ≥ C + C log , 1 2 τ where Q projects onto the space not spanned by the eigenvector. The inner iteration for (A − σI)y = x Good news! (i) ′ ′ C r k(i) ≥ C + C log 3 , 1 2 τ (i) where τ (i) = Cr(i). Iteration number approximately constant! The inner iteration for (A − σI)y = x Good news! (i) ′ ′ C r k(i) ≥ C + C log 3 , 1 2 τ (i) where τ (i) = Cr(i). Iteration number approximately constant! Bad news :-( For a standard preconditioner P − − (A − σI)P 1y˜(i) = x(i) P 1y˜(i) = y(i) (i) ′′ ′′ Q˜x ′′ ′′ C k(i) ≥ C + C log = C + C log , 1 2 τ (i) 1 2 τ (i) where τ (i) = Cr(i). Iteration number increases! Convection-Diffusion operator Finite difference discretisation on a 32 × 32 grid of the convection-diffusion operator 2 −∆u + 5ux + 5uy = λu on (0, 1) , with homogeneous Dirichlet boundary conditions (961 × 961 matrix). smallest eigenvalue: λ1 ≈ 32.18560954, Preconditioned GMRES with tolerance τ (i) = 0.01r(i), standard and tuned preconditioner (incomplete LU). Convection-Diffusion operator right preconditioning, 569 iterations right preconditioning, 569 iterations 0 45 10 40 −2 10 35 −4 30 10 25 −6 10 20 inner iterations 15 residual norms \|r^{(i)}\| −8 10 10 −10 5 10 2 4 6 8 10 12 14 16 18 20 22 100 200 300 400 500 600 outer iterations sum of inner iterations Figure: Inner iterations vs outer Figure: Eigenvalue residual norms vs iterations total number of inner iterations The inner iteration for (A − σI)P −1y˜ = x How to overcome this problem Use a different preconditioner, namely one that satisfies H (i) (i) (i) (i) Pix = Ax , Pi := P +(A − P )x x minor modification and minor extra computational cost, P−1 (i) (i) [A i ]Ax = Ax . The inner iteration for (A − σI)P −1y˜ = x How to overcome this problem Use a different preconditioner, namely one that satisfies H (i) (i) (i) (i) Pix = Ax , Pi := P +(A − P )x x minor modification and minor extra computational cost, P−1 (i) (i) [A i ]Ax = Ax . Why does that work? Assume we have found eigenvector x1 −1 λ1 − σ Ax1 = Px1 = λ1x1 ⇒ (A − σI)P x1 = x1 λ1 −1 and convergence of Krylov method applied to (A − σI)P y˜ = x1 in one iteration. For general x(i) (i) ′′ ′′ C r k(i) ≥ C + C log 3 , where τ (i) = Cr(i). 1 2 τ (i) Convection-Diffusion operator Finite difference discretisation on a 32 × 32 grid of the convection-diffusion operator 2 −∆u + 5ux + 5uy = λu on (0, 1) , with homogeneous Dirichlet boundary conditions (961 × 961 matrix). smallest eigenvalue: λ1 ≈ 32.18560954, Preconditioned GMRES with tolerance τ (i) = 0.01r(i), standard and tuned preconditioner (incomplete LU). Convection-Diffusion operator right preconditioning, 569 iterations right preconditioning, 569 iterations 0 45 10 40 −2 10 35 −4 30 10 25 −6 10 20 inner iterations 15 residual norms \|r^{(i)}\| −8 10 10 −10 5 10 2 4 6 8 10 12 14 16 18 20 22 100 200 300 400 500 600 outer iterations sum of inner iterations Figure: Inner iterations vs outer Figure: Eigenvalue residual norms vs iterations total number of inner iterations Convection-Diffusion operator right preconditioning, 569 iterations right preconditioning, 569 iterations 0 10 tuned right preconditioning, 115 iterations 45 tuned right preconditioning, 115 iterations 40 −2 10 35 −4 30 10 25 −6 10 20 inner iterations 15 residual norms \|r^{(i)}\| −8 10 10 −10 5 10 2 4 6 8 10 12 14 16 18 20 22 100 200 300 400 500 600 outer iterations sum of inner iterations Figure: Inner iterations vs outer Figure: Eigenvalue residual norms vs iterations total number of inner iterations Outline 1 Introduction 2 Inexact inverse iteration 3 Inexact Shift-invert Arnoldi method 4 Inexact Shift-invert Arnoldi method with implicit restarts 5 Conclusions The algorithm Arnoldi’s method Arnoldi method constructs an orthogonal basis of k-dimensional Krylov subspace (1) (1) (1) 2 (1) k−1 (1) Kk(A, q ) = span{q , Aq , A q ,..., A q }, H AQ = Q H + q h eH = Q k k k k k+1 k+1,k k k+1 h eH k+1,k k H Qk Qk = I. The algorithm Arnoldi’s method Arnoldi method constructs an orthogonal basis of k-dimensional Krylov subspace (1) (1) (1) 2 (1) k−1 (1) Kk(A, q ) = span{q , Aq , A q ,..., A q }, H AQ = Q H + q h eH = Q k k k k k+1 k+1,k k k+1 h eH k+1,k k H Qk Qk = I. Eigenvalues of Hk are eigenvalue approximations of (outlying) eigenvalues of A H rk = Ax − θx = (AQk − QkHk)u = |hk+1,k||ek u|, The algorithm Arnoldi’s method Arnoldi method constructs an orthogonal basis of k-dimensional Krylov subspace (1) (1) (1) 2 (1) k−1 (1) Kk(A, q ) = span{q , Aq , A q ,..., A q }, H AQ = Q H + q h eH = Q k k k k k+1 k+1,k k k+1 h eH k+1,k k H Qk Qk = I. Eigenvalues of Hk are eigenvalue approximations of (outlying) eigenvalues of A H rk = Ax − θx = (AQk − QkHk)u = |hk+1,k||ek u|, at each step, application of A to qk: Aqk =q ˜k+1 Example random complex matrix of dimension n = 144 generated in Matlab: G=numgrid(’N’,14);B=delsq(G);A=sprandn(B)+i*sprandn(B) eigenvalues of A 3 2 1 0 −1 −2 −3 −4 −3 −2 −1 0 1 2 3 4 after 5 Arnoldi steps Arnoldi after 5 steps 3 2 1 0 −1 −2 −3 −4 −3 −2 −1 0 1 2 3 4 after 10 Arnoldi steps Arnoldi after 10 steps 3 2 1 0 −1 −2 −3 −4 −3 −2 −1 0 1 2 3 4 after 15 Arnoldi steps Arnoldi after 15 steps 3 2 1 0 −1 −2 −3 −4 −3 −2 −1 0 1 2 3 4 after 20 Arnoldi steps Arnoldi after 20 steps 3 2 1 0 −1 −2 −3 −4 −3 −2 −1 0 1 2 3 4 after 25 Arnoldi steps Arnoldi after 25 steps 3 2 1 0 −1 −2 −3 −4 −3 −2 −1 0 1 2 3 4 after 30 Arnoldi steps Arnoldi after 30 steps 3 2 1 0 −1 −2 −3 −4 −3 −2 −1 0 1 2 3 4 The algorithm: take σ = 0 Shift-Invert Arnoldi’s method A := A−1 Arnoldi method constructs an orthogonal basis of k-dimensional Krylov subspace −1 (1) (1) −1 (1) −1 2 (1) −1 k−1 (1) Kk(A , q ) = span{q ,A q , (A ) q ,..., (A ) q }, −1 H Hk A Qk = QkHk + qk+1hk+1,kek = Qk+1 H hk ,ke +1 k H Qk Qk = I.