Numerical Linear Algebra Contents

WS 2009/2010 Numerical Linear Algebra

Professor Dr. Christoph Pﬂaum

Contents

1 Linear Equation Systems in the Numerical Solution of PDE’s 5

1.1 ExamplesofPDE’s...... 5

1.2 Finite-Diﬀerence-Discretization of Poisson’s Equation..... 7

1.3 FD Discretization for Convection-Diﬀusion ...... 8

1.4 Irreducible and Diagonal Dominant Matrices ...... 9

1.5 FE (Finite Element) Discretization ...... 12

1.6 Discretization Error and Algebraic Error ...... 15

1.7 Basic Theory for LInear Iterative Solvers ...... 15

1.8 Eﬀective Convergence Rate ...... 18

1.9 Jacobi and Gauss-Seidel Iteration ...... 20

1.9.1 Ideas of Both Methods ...... 20

1.9.2 Description of Jacobi and Gauss-Seidel Iteration by Matrices...... 22

1.10 Convergence Rate of Jacobi and Gauss-Seidel Iteration . . . . 24

1.10.1 General Theory for Weak Dominant Matrices . . . . . 24

1.10.2 Special Theory for the FD-Upwind ...... 26

1.10.3 FE analysis, Variational approach ...... 30

1 1.10.4 Analysis of the Convergence of the Jacobi Method . . 33

1.10.5 Iteration Method with Damping Parameter ...... 34

1.10.6 Damped Jacobi Method ...... 35

1.10.7 Analysis of the Damped Jacobi method ...... 35

1.10.8 Heuristic approach ...... 37

2 Multigrid Algorithm 38

2.1 Multigrid algorithm on a Simple Structured Grid ...... 38

2.1.1 Multigrid ...... 38

2.1.2 Idea of Multigrid Algorithm ...... 39

2.1.3 Two–grid Multigrid Algorithm ...... 40

2.1.4 Restriction and Prolongation Operators ...... 41

2.1.5 Prolongation or Interpolation ...... 41

2.1.6 Pointwise Restriction ...... 41

2.1.7 Weighted Restriction ...... 42

2.2 Iteration Matrix of the Two–Grid Multigrid Algorithm . . . . 42

2.3 Multigrid Algorithm ...... 43

2.4 Multigrid Algorithm for Finite Elements ...... 44

2.4.1 ModelProblem...... 44

2.4.2 Example...... 44

2.4.3 TheNodalBasis ...... 45

2.4.4 Prolongation Operator for Finite Elements ...... 45

2 2.4.5 Restriction Operator for Finite Elements ...... 46

2.5 Fourier Analysis of the Multigrid method ...... 47

2.5.1 Local Fourier analysis ...... 47

2.5.2 Deﬁnition ...... 51

2.5.3 Local Fourier analysis of the smoother ...... 51

3 Gradient Method and cg 52

3.1 GradientMethod ...... 52

3.2 Analysis of the Gradient Method ...... 53

3.3 The Method of Conjugate Directions ...... 55

3.4 cg-Method (Conjugate Gradient Algorithm) ...... 57

3.5 Analysis of the cg algorithm ...... 59

3.6 Preconditioned cg Algorithm ...... 61

4 GMRES 63

4.1 Minimalresidualmethod ...... 64

4.2 Solution of the Minimization Problem of GMRES ...... 65

4.3 Computation of QR-Decomposition with Givens Rotation . . 66

4.4 TheGMRESAlgorithm ...... 67

4.5 Convergence of the GMRES method ...... 68

5 Eigenvalue Problems 69

5.1 RayleighQuotient ...... 69

3 5.2 Method of Conjugate Gradients ...... 71

5.3 Simple Vector Iteration ...... 74

5.4 Computation of Eigenvalues using the Rayleigh Quotient . . 76

5.5 Jacobi-Davidson-Algorithm ...... 80

5.5.1 The Jacobi-Method ...... 80

5.5.2 Motivation of Davidson’s Algorithm ...... 81

5.5.3 The concept of the Jacobi-Davidson-Algorithm . . . . 82

5.5.4 Jacobi-Davidson-Algorithm ...... 84

4 1 Linear Equation Systems in the Numerical So- lution of PDE’s

1.1 Examples of PDE’s

1. Heat Equation

temperature g at the boundary

hom. plate

Heat source f in the interior of the plate. Question: What is the temperature inside of the plate? Poisson Problem (P) Let Ω Rn open, bounded, f C(Ω), g C(δΩ). ⊂ ∈ ∈ Find u C2(Ω) such that ∈ ∆u = f on Ω − u δΩ = g ∂2 ∂2 where ∆ = + ∂x2 ∂y2

2. Convection-Diﬀusion-Problem Find u C2(Ω) such that ∈ ∆u +~b u + cu = f on Ω − ·∇ u δΩ = 0 2 where ~b (C(Ω)) , f,c C(Ω) ∈ ∈ 3. Navier-Stokes-Equation

∂u ∂p ∂(u2) ∂(uv) 1 + + + = ∆u ∂t ∂x ∂x ∂y Re ∂u ∂p ∂(uv) ∂(v2) 1 + + + = ∆v ∂t ∂y ∂x ∂y Re ∂u ∂v + = 0 ∂x ∂y

5 u

4. Laser simulation

mirror 1 mirror 2

M M2 1 r

Γ =Γ Γ M M1 ∪ M2 2 Find u CC(Ω), λ C such that ∈ ∈ ∆u k2u = λu − − u = 0 ΓM ∂u = 0 (or boundary condition third kind) ∂~n Γ rest

We apply the ansatz

−ikz˜ ikz˜ u = ure + ule

where k˜ is an average value of k. This leads to the equivalent eigenvalue problem: Find ur, ul, λ such that ∂u ∆u + 2ik˜ r + (k˜2 k2)u = λu − r ∂z − r r ∂u ∆u 2ik˜ l + (k˜2 k2)u = λu − l − ∂z − l l ∂ur ∂ul ur + ul = 0, = 0 ΓM ∂z − ∂z ΓM

∂ur ∂ul = = 0 ∂~n Γrest ∂~n Γrest

6 1.2 Finite-Diﬀerence-Discretization of Poisson’s Equation

Assume Ω =]0, 1[2 and that an exact solution of (P) exists. We are looking for an approximate solution uh of (P) on a grid Ωh of meshsize h. Choose h = 1 where m N. m ∈ Ω = (ih, jh) i, j = 1,...,m 1 h − Ωh = (ih, jh) i, j = 0,...,m

Discretization by Finite Diﬀerences: Idea: Replace second derivative by diﬀerence quotient. Let ex = (1, 0) and ey = (1, 0),

∂2u ∂2u ∆u(z)= (z)= f(z) for z Ω − −∂x2 − ∂y2 ∈ h

u (z + he ) 2u (z)+ u (z he ) h x − h h − x − h2 u (z + he ) 2u (z)+ u (z he ) h y − h h − y = f(z) − h2

and u(z) = g(z) = for z Ω Ω ≈ ∈ h\ h uh(z) = g(z)

This leads to a linear equation system L U = F where U = (u (z)) , h h h h h z∈Ωh L is Ω Ω matrix. The discretization can be described by the stencil h | h| × | h| 1 m m m − h2 −1,1 0,1 1,1 1 4 1 = m m m  − h2 h2 − h2   −1,0 0,0 1,0  1 m m m − h2 −1,−1 0,−1 1,−1    

X X X X

7 Let us abbreviate Ui,j := uh(ih, jh) and fi,j := f(ih, jh). Then, in case of g = 0, the matrix equation LhUh = Fh is equivalent to:

1 mklUi+k,j+l = fi,j k,lX=−1

1.3 FD Discretization for Convection-Diﬀusion

Let Ω, Ωh as above. du ∆u + b = f − dx Assume that b is constant.

1. Discretization by central diﬀerence:

du u (z + he ) u (z he ) (z) h x − h − x dx ≈ 2h This leads to the stencil

1 − h2   1 b 4 1 + b − h2 − 2h h2 − h2 2h      1   h2   −  unstable for large b. → 2. Upwind discretization:

du u (z) u (z he ) (z) h − h − x dx ≈ h This leads to the stencil

1 − h2   1 b 4 + b 1 − h2 − h h2 h − h2      1   h2   − 

8 1.4 Irreducible and Diagonal Dominant Matrices

Deﬁnition 1. A n n matrix A is called strong diagonal dominant, if × a > a 1 i n (1) | ii| | ij| ≤ ≤ Xi6=j A is called weak diagonal dominant, if there exists at least one i such that (1) holds and such that

a a 1 i n | ii|≥ | ij| ≤ ≤ Xi6=j Deﬁnition 2. A is called reducible, if there exists a subset J 6= 1, 2,...,n , ⊂ { } J = . such that 6 ∅ a = 0 for all i J, j J ij 6∈ ∈ A not reducible matrix is called irreducible.

Remark. An reducible matrix has the form

A11 A12 0 A 22 The equation system separates in two parts. →

Example:

1. Poisson FD: 4 diagonal: aii = h2 1 if i is N,S,W,O of j non-diagonal: a = − h2 ij 0 else A is not strong diagonal dominant, but weak diagonal dominant. • To see this, consider a point i such that j is N of i. Then 1 if i is S,W,O of j a = − h2 ij 0 else A is irreducible. • Proof: If A is reducible, then, 1, 2, ..., n is the union of two { } diﬀerent sets of colored points, where one set is J. Then, there is a point j J such that one of the points i=N,W,S,E is not ∈ contained in J, but i is contained in 1, 2, ..., n . This implies { } a = 0. contradiction. j,i 6 ⇒

9 2. Convection-Diﬀusion-Equation

centered diﬀerence • 4 a = | ii| h2 4 1 1 b 1 b a = 2 + + + | ij| h2 · h2 h2 2h h2 − 2h i6=j X 1 b 1 b = 3 + + h2 2h h2 − 2h

Thus, a a , if and only if 1 b 0. | ii|≥ i6=j | ij| h2 − 2h ≤ This shows a a , if and only if h< 2 | Pii|≥ i6=j | ij| b upwind • P 4 b a = + | ii| h2 h

4 b ≥ + a for all h,b > 0 h2 h ≥ | ij| Xi6=j Conclusion • 2 central: A is weak diagonal dominant if and only if h< b . upwind: A is weak diagonal dominant. A is irreducible in both cases.

Deﬁnition 3. Let A be an n n matrix. Consider n points P1, ..., Pn. −→ × Draw an edge between Pi, Pj if ai,j = 0. The directed graph of A is this set 6 −→ of points P1, ..., Pn with these edges Pi, Pj. Deﬁnition 4. A directed graph is called strongly connected, if for every pair of disjoint points Pi, Pj there exists a directed path in the graph. This means −→ −→ −→ there exists a path Pi0 Pi1 , Pi2 Pi3 , ..., Pir−1 Pir such that Pi0 = Pi andPir = Pj. Theorem 1. A n n matrix A is irreducible, if and only if its directed × graph is connected.

Proof. Let A be irreducible. Let 1 i n be an index. Let ≤ 0 ≤ J := j there is a directed path from P to P . . { | i0 j } J is not empty. Otherwise, a = 0 for every j and choosing J˜ = i would i0,j { 0} lead to a contradiction to A to be irreducible. Let us assume J = 1, 2, ..., n . 6 { }

10 Then, a = 0 for every i J and j J. Otherwise, there is a connected i,j ∈ 6∈ path from Pi0 to Pi and to Pj.

The above property of J is a contradiction to A irreducible.

Let the directed graph of A be connected. Assume that A is reducible. Then, there are disjoint sets J, I such that −→ −→ −→ a = 0 for every i I, j J. Let P P , P P , ..., P P be a directed i,j ∈ ∈ i0 i1 i2 i3 ir−1 ir path from i I to i J. Then, there must be a index s such that 0 ∈ r ∈ i I and i J. This implies a = 0. This is a contradiction to the s−1 ∈ s ∈ is−1,is 6 properties of J and I.

11 1.5 FE (Finite Element) Discretization

Deﬁnition 5. = T ,...,T is a conform triangulation of Ω if T { 1 M }

Ω= M T , T is triangle or square • i=1 i i T TS is either • i ∩ j – empty or – one common corner or – one common edge.

Remark.

Let us write , if the diameter h of every element T is less or • Th T ∈Th equal h:

h h. T ≤ A family of triangulations is called quasi-uniform, if there exists • {Th} a constant ρ > 0 such that the radius ρT of the largest inner ball of every triangle T satisﬁes ∈Th

ρT > ρh.

Deﬁnition.

Let be a triangulation of Ω. Then, let V be the space of linear ﬁnite • Th h elements deﬁned as follows:

V = v C0(Ω) v is linear for every T h ∈ T ∈TH 0 V = V H1(Ω) h h ∩ 0

v T is linear means that v T (x,y)= a + bx + cy. Let Ω =]0, 1[2, h = 1 and • m

= [ih, (i + 1)h] [jh, (j + 1)h] i, j = 0,...,m 1 Th × −

12 The space of bilinear ﬁnite elements on Ω is deﬁned as follows

V = v C0(Ω) v is bilinear for every T h ∈ T ∈TH

v T is bilinear means that v T (x,y)= a + bx + cy + dxy. Let V be the space of linear or bilinear ﬁnite elements on and • h Th Nh the set of corners of . Then, deﬁne the nodal basis function v V Th p ∈ h at the point p by:

1 if x = p v (x)= for x p 0 if x = p ∈Nh 6 Observe that V = span v p h p ∈Nh

This means that every function uh Vh can be represented as ∈

uh = λpvp pX∈Nh

Finite Element Discretization of Poisson’s equation:

∆u = f − u δΩ = 0

0 Thus, for every v V , we get: h ∈ h ∆u v = f v − h h ⇓ ∂u u v d(x,y)+ v d(x,y) = f v d(x,y) ∇ ∇ h ∂~n h h ZΩ ZΓ ZΩ

⇓ 0 u v d(x,y) = f v d(x,y) v V ∇ ∇ h h ∀ h ∈ h ZΩ ZΩ

0 FE Discretization: Find u V such that h ∈ h 0 u v d(x,y)= f v d(x,y) v V (2) ∇ ∇ h h ∀ h ∈ h ZΩ ZΩ

13 Stiﬀness matrix.

a := v v d(x,y), f := f v d(x,y) p.q ∇ p ∇ q q q ZΩ ZΩ 0 A := (ap,q) 0 , h:= h Ω p,q∈Nh N N ∩

uh = λp vp 0 pX∈Nh

Then, (2) implies

0 λp vp vq d(x,y) = f vq d(x,y) for all q h Ω ∇ ∇ Ω ∈N 0 Z Z pX∈Nh

⇓ 0 λ a = f q p p,q q ∀ ∈Nh 0 pX∈Nh

⇓ Uh = (λp) 0 p∈Nh A Uh = Fh where Fh = (fq) 0 q∈Nh The matrix A is called the stiﬀness matrix of the FE discretization.

14 1.6 Discretization Error and Algebraic Error

Let be a suitable norm. Then, U U is called discretization error, || · || || h − || with respect to this norm. Example 1. Poisson on a square

FD, u C4(Ω), then • ∈ 2 U U ∞ = O(h ) || h − ||L (Ωh)

FE, u H2(Ω), then • ∈ 2 U U 2 = O(h ) || h − ||L (Ω) U U 1 = O(h) || h − ||H (Ω)

Problem. The solution uh cannot be calculated exactly, since Lh (or A) is a very large matrix and A Uh = Fh. Therefore, we need iterative solvers if n> 10.000 (or n> 100.000). By such an iterative solver, we get an approximationu ˜ of u . u˜ u is called h h || h − h|| algebraic error.

1.7 Basic Theory for LInear Iterative Solvers

Let A be a non singular n n matrix and b a vector, b Rn. × ∈ Problem: Find x Rn such that Ax = b. ∈ A basic approach to construct an iterative solver is to use a decomposition

A = M N − where M is a matrix, which is easy to invert (which can be inverted by a small number of operations). Then we get

M x = N x + b

⇓ x = M −1N x + M −1b

15 By this formulas, we get the algorithm:

Algorithm: Let x0 be the start guess. Then xk+1 := M −1N xk + M −1b

Let us write the iteration formula as xk+1 = C xk + d, where C = M −1N and d = M −1b . This is the general form of a linear iterative solver.

Theorem 1. xk converges to x for every start vector x0 if and only if ρ(C) < 1 Here ρ(C) is the spectral radius of C, ρ(C) = max λ λ is eigenvalue of C | | (Observe the eigenvalues may be complex.)

Furthermore, the following convergence result holds: xk x Ck x0 x (3) || − || ≤ || || || − || If C is a normal matrix, then xk x (ρ(C))k x0 x (4) || − ||2 ≤ || − ||2 There exist start vectors x0, such that the equal sign holds in the above inequality.

Proof. By xk+1 = C xk + d and x = C x + d, we get

xk+1 x = C xk x − − This implies xk x = Ck(x0 x) (5) − − This implies (3). Let us assume, that x0 x = e is an eigenvector of C with eigenvalue λ such − that λ = ρ(C) | | Then, we get xk x = ρ(C)k x0 x || − || || − || This shows:

16 if ρ(C) 1, then xk does not converge to x. • ≥ the equal sign holds in equation (4). •

Now, let us assume that x0 is a general start vector. Let us assume ρ(C) < 1. k We want to prove limk→∞ x = x. By (3), it is enough to prove

Ck 0 for k || || → →∞ Since all norms are equivalent in a ﬁnite dimensional vector space, it is enough to show this for the -norm. k · k2

1. C is normal. Then, there exists a unitary matrix T such that

T −1D T = C

where D = diag(λ1,...,λn) is the diagonal matrix of eigenvalues. Then, we get

Ck = T −1DkT T −1 T Dk = Dk = ρ(C)k. || ||2 || ||2 ≤ || ||2 || ||2 || ||2 || ||2 This shows (4).

2. C is a general matrix. Then, we have to apply the Jordan decomposition T −1J T = C Then, we get Ck T −1 T J k || || ≤ || || || || || || Thus, it is enough to show

lim J k = 0 k→∞ || || It is enough to study an Jordan block

J˜ = λ E + N,

where E is the unit matrix and 0 1 0 . .  .. ..   N = s rows .   .. 1       0 0        

17 Since ρ(C) < 1, it follows λ < 1. A short calculation shows | | N i 1 for all i || || ≤ N s = 0

Since N E = E N it follows:

k k J˜k = (λE)k−i N i || || || i || ≤ Xi=0 s−1 k λk−i ≤ i ≤ Xi=0 s ks λk−s = ≤ = (s λ−s) ks λk 0 for k → →∞

k k(k 1) . . . (k i + 1) NR: = − · · − ki ks i i! ≤ ≤ (k + 1)sλk+1 1 s λ + 1 = 1+ λ < 1 for large k ksλk k ≤ 2

1.8 Eﬀective Convergence Rate

In several applications one would like to know, how many iterations s are needed to reduce the algebraic error by a certain factor. Let us assume that 1 this factor is 2 . Thus, we would like to know how many iterations s are needed to obtain 1 xs x x0 x . k − k≤ 2k − k To this end , let us assume that there is an estimation

xs x ρs x0 x . k − k≤ k − k In case of a linear iteration method with symmetric iteration matrix C, we can choose ρ = ρ(C).

Obviously, ln 1 s = 2 ln (ρ) s 1 since ρ(C) = 2 . s and ρ(C) are not the eﬀective convergence rate. To estimate the eﬀective convergence rate , the computational amount has to

18 be included. Let Op the number of operations for one iteration. Then, the eﬀective convergence rate is:

Op ln 1 Op G := s = 2 eff · number of unknowns ln (ρ) · n

Example 2. Gauss elimination

2 Geff = O(n )

19 1.9 Jacobi and Gauss-Seidel Iteration

The Jacobi-iteration is a one-step“ method. The Gauss-Seidel-iteration is ” a successive relaxation method.

1.9.1 Ideas of Both Methods

Relaxation of the i-th unknown xi: old new Correct xi by xi such that the i-th equation of the equation system

A x = b · is correct.

Jacobi-iteration: Calculate the relaxations simultaneously for all i = 1,...,n“ ” Thismeans: If xold = xk, then let xk+1 = xnew

Gauss-Seidel-iteration: Calculate relaxation for i = 1,...,n and use the new values“ ” This means: xold,1 = xk Iterate for i = 1,...,n: Calculate xnew,i by relaxation of the i-th component Put xold,i+1 = xnew,i xk+1 = xnew,n

Remark. Jacobi-iteration is independent of the numbering of the grid points • The convergence rate of the Gauss-Seidel iteration depends on the • numbering of the grid points

20 Example 3. Model problem, FD for Poisson

W ME 1 unew = uold + uold + uold + uold + f S M 4 N S E W M

red-black Gauss-Seidel

A four color Gauss-Seidel-relaxation is used for a 8-point stencil

1 1 1 − − − 1 8 1 − − 1 1 1 − − − - better relaxation property - after relaxation of one color all equations at those points are correct

Relaxation for the Convection-Diffusion: A convection-diffusion problem is a so-called singular perturbed problem. To see this write the convection-diffusion problem in the form: ∂u ǫ∆u + = f˜ , ǫ> 0 − ∂x

21 ǫ 0 is the diﬃcult case. → (Hackbusch’s) rule for relaxing singular perturbed problems: Construct the iteration such that it is an exact solver for ǫ = 0

For ǫ = 0 we get the stencil (for upwind FD):

0 1 1 0  − h h  0   Thus a Gauss-Seidel relaxation with a numbering of the grid points from left to right leads to an exact solver

1 2 3

4 5 6

7 8 9

This can be done also for more complicated convection directions. Excep- tion: Circles!

1.9.2 Description of Jacobi and Gauss-Seidel Iteration by Matri- ces

Let A be a n n matrix. Decompose A = D L R = × − − 0 . . . 0 0 0 . . . 0 0 . . . ∗ . . . ∗ ∗.  0 .. .   0 .   0 0 .  . ∗ . ∗......  . ..   . .. ..   . .. ..   − −   ..   ..   ..   . 0   . 0 0   . 0   ∗     ∗   0 . . . 0   . . . 0   0 . . . 0   ∗   ∗ ∗          Let x Rd be a start vector. 0 ∈ Jacobi-iteration

22 Dxk+1 (L + R) xk = b − ⇒ xk+1 = D−1 (L + R) xk + D−1b

Decomposition: A = D (L + R)= M N − − Thus, the iteration matrix is

−1 CJ = D (L + R)

Gauss-Seidel-iteration

(D L) xk+1 Rxk = b − − ⇒ xk+1 = (D L)−1 Rxk + (D L)−1 b − − Thus, the iteration matrix is

C = (D L)−1 R GS −

23 1.10 Convergence Rate of Jacobi and Gauss-Seidel Iteration

1.10.1 General theory for weak dominant matrices

1.10.2 Special theory for upwind FD

1.10.3 FE analysis, variational approach

1.10.4 Eigenvector, eigenvalue analysis: Fourier-analysis“ ”

1.10.1 General Theory for Weak Dominant Matrices

Theorem 1. 1. Assume that A is weak diagonal dominant and irreducible. Then, Ja- cobi and Gauss-Seidel iteration converge.

2. Assume that A is diagonal dominant. Then, the following estimate for the convergence rate holds: 1 ρ max aij < 1 ≤ 1≤i≤n aii | | | | Xj6=i

Proof. Let x be an eigenvector of C with eigenvalue λ, where λ = ρ(C). | | Furthermore assume x = 1. || ||∞ Assume that A is weak diagonal dominant.

In case of the Jacobi iteration: C = D−1(L + R) •

1 1 (Cx)i aij xj aij x ∞ 1 (6) | |≤ aii | | | |≤  aii | | || || ≤ | | Xj6=i | | Xj6=i  This shows Cx 1. Since xeigenvector with eigenvalue λ, it || ||∞ ≤ follows 1 Cx = λx = λ = ρ(C) ≥ || ||∞ || ||∞ | | In case of the Gauss-Seidel iteration: • C = (D L)−1R (D L)C = R C = D−1(LC + R) − ⇒ − ⇒

24 Let us prove by induction (Cx) 1 for i = 1,...,n | i|≤ 1 (Cx)i aij (Cx)j + aij xj (7) | | ≤ aii  | | | | | | | | | | Xji  1   aij 1 ≤ aii  | | ≤ | | Xj6=i  Analogously, we get ρ(C) 1.  ≤

If A is diagonal dominant, then similar calculations show

1 (Cx)i aij < 1 | |≤ aii  | | | | Xj6=i  This implies   1 Cx ∞ max aij < 1 || || ≤ i aii | | | | Xj6=i which shows 1 ρ(C) max aij < 1 ≤ i aii | | | | Xj6=i This completes the proof of 2.

Let us assume that A is irreducible and weak diagonal dominant. Let

J = i N 1 i n, x = 1 ∈ ≤ ≤ | i| Proof by contradiction. Assume ρ = 1. Then for all i J, the equal signs ∈ hold for all inequalities in (6),(7). This shows for every i J: ∈ a =0 if j / J (8) | ij| ∈ (j / J means x < 1 ) ∈ | j| By assumption, there is a i0 such that the equal sign does not hold in (6),(7). This means 1 > (Cx) = ρx = x | i0 | | i0 | | i0 | This shows that J is a real subset of 1,...,n . J is not the empty set, { } since x = 1. k k∞ This is a contradiction to A irreducible.

Example. By the examples in 1.4, Gauss-Seidel iteration and Jacobi iteration converge for Poisson problem and convection-diﬀusion problem and FD upwind. But: no estimation of the convergence rate.

25 1.10.2 Special Theory for the FD-Upwind

Deﬁnition.Let us assume, that q > 0. Then deﬁne the upwind norm

i x up,q := max q xi || || i | | Theorem 2. Assume that a | ij|qi−j < 1. aii Xji aii ρ(CGS) max a ij ≤ i=1 1 P qi−j − j

P Proof. Let us assume, that

x = 1 qix 1 i || ||up,q ⇒ | i|≤ ∀ Assume, that x is eigenvector of CGS with eigenvalue λ such that λ = i −1 | | ρ(CGS). Choose i such that q xi = 1. Then CGS = (D L) R CGS = −1 | | − ⇒ D (LCGS + R) λ q−i = λ x = (λx) = (C (x)) = (D−1(LC x + Rx)) | | | | | i| | i| | GS i| GS i ≤

1 aij (CGS x))j + aij xj ≤ aii  | || | | || | ≤ | | Xji  1   aij (λx))j + aij xj ≤ aii  | || | | || | ≤ | | Xji 

1  −j −j  aij λ q + aij q ≤ aii  | || | | |  | | Xji  ⇓   a a λ 1 | ij|qi−j | ij|qi−j | |  − aii  ≤ aii Xji | |   This completes the proof.

Example. FD - 1D convection diﬀusion

u′′ + bu′ = f on [0, 1], b> 0 − u(0) = u(1) = 0

26 Figure 1: Estimation of the spectral radius of upwind Gauss-Seidel in 1D

Stencil of upwind discretization 1 1 b 1 2 + b 1 1 = 1 s 2+ s 1 − h2 − h h2 h − h2 h2 − − − where s = bh. Normalized coeﬃcients

aii = 2+ s 1 if j = i + 1 − a = (1 + s) if j = i 1 ij  − −  0 else Let us number the grid points from left to right. Then, for 1 s, we get  − ≤ 1 q−1 1 1 ρ = 2+s = 1 1+s q 2+ s q q2 1+s − 2+s − 2+s f(q) = q q2 1+s ,f ′(q) = 1 2q 1+s =! 0 q = 1 2+s f(q ) = 1 2+s . − 2+s − 2+s ⇒ 0 2 1+s ⇒ 0 4 1+s Observe 1 1+s q = 1 > 0. For 1 s: − 2+s q=q0 2 − ≤

1 1 1+ s ρ(CGS ) 1 2+s = 4 2 ≤ 2+ s 4 1+s (2 + s) 1+ s lim 4 = 0 s→∞ (2 + s)2 1+ s 4 = 1 (2 + s)2 s=0 1+ s 4 = 0 (2 + s)2 s=−1

27 1+s The function 4 (2+s)2 is depicted in Figure 1.

2D case i (xi,j) up,(qi) := max q x(i,j) || || (i,j)

Theorem 3. Assume that

a(i,j),(k,l) | |qi−k < 1 a(i,j),(i,j) Xk

Example. FD - 2D convection diﬀusion ∂u ∆u + b = f − ∂x Stencil: 1 1 − 1 s 4+ s 1 h2  − − −  1 −  

a(i,j),(i,j) = 4+ s 1 if k i, (i, j) = (k, l) a = − ≥ 6 (i,j),(k,l) (1 + s) if k < i − Let us number the grid points ﬁrst from left to right and then from down to up. Then, for 1 s, we get: − ≤ 1 (1+1+ q−1) 1 (2 + q−1) ρ 4+s = 4+s ≤ 1 1 q(1 + s)) 1 q(1+s) − 4+s − 4+s 1 −1 4+s (2+q ) Numerically, one can calculate an optimal parameter q such that q(1+s) 1− 4+s is as small as possible. The resulting estimation of the convergence rate is depicted in Figure 2.

28 1

0.9

0.8

0.7

0.6

0.5

0 2 4 6 8 10

Figure 2: Estimation of the spectral radius of upwind Gauss-Seidel in 2D

29 1.10.3 FE analysis, Variational approach

We want to solve the following problem:

Find u V such that h ∈ h a(u , v )= f(v ) v V (9) h h h ∀ h ∈ h where Vh is the space of bilinear ﬁnite elements.

For example,

a(u , v )= u v d(x,y) and f(v )= f v d(x,y) (10) h h ∇ h∇ h h s h ΩZ ΩZ

Relaxation Let (V ) be the nodal basis of V . Now what means a relaxation step? p p∈Ωh h

Let uold be an old approximation. Then, let µ R such that, h ∈ old a(uh + µvp, vp)= f(vp) and let (11)

new old uh = uh + µvp (12)

This is the relaxation at the grid point p. Furthermore, we can calculate µ in the following way:

1 old µ = f(vp) a(uh , vp) (13) a(vp, vp) − new old 1 old uh = uh + vp f(vp) a(uh , vp) (14) a(vp, vp) − 

Implementation by EXPDE Let fs be the vector describing the right hand side and let

f(v)= f vd(x,y) and a(u, v)= u vd(x,y). s ∇ ∇ ΩZ ΩZ

The operator corresponding to a(v, v) is the Laplace operator. Now deﬁne the variables as

30 Variable f(grid); Variable u(grid); Variable f_s(grid); Variable v(grid); f=Helm_FE(f_s);

Now, for the Jacobi method, we get nu = (f-Laplace_{FE}(u)) / Diag_Laplace_FE(); u = u+nu ; and for the Gauss–Seidel method we have u = u + (f-Laplace_{FE}(u)) / Diag_Laplace_FE(); Lemma 1 (Variational approach). Let us assume that a(u, v) is symmetric positive deﬁnite. Then Find u V such that h ∈ h a(u , v )= f(v ) for all v V (15) h h h h ∈ h is equivalent to 1 u V minimizes a(v , v ) f(v ) for v = u . (16) h ∈ h 2 h h − h h h

Proof.

Let us assume that uh satisﬁes (15). Then, we have to show that 1 µ a(u + µv , u + µv ) f(u + µv )= h(µ) (17) −→ 2 h h h h − h h has a minimum at µ = 0 for every v V . We achieve this by diﬀerentiating h ∈ h (15).

0 = h′(µ) = µa(v , v )+ a(u , v ) f(v ) h h h h − h = µa(v , v )+ a(u , v ) a(u , v )) h h h h − h h = µa(vh, vh) µ = 0 ⇒

31 By diﬀerentiating the above equation again, we get h′′(µ)= a(v , v ) h′′(µ) > 0 h h ⇒ Let us assume that u satisﬁes (16), then h(µ) = 0 implies that a(u , v )= h |µ=0 h h f(vh).

A similar calculation shows that the problem

Find µ R such that a(uold + µv , v )= f(v ) ∈ h p p p is equivalent to

Minimize 1 a(uold + µv , uold + µv ) f(uold + µv ) for all µ R 2 h p h p − h p ∈

Conclusion A relaxation step is a minimizing step.

Gauss–Seidel minimizes 1 a(û , uˆ ) f(û ) in several directions. Therefore 2 h h − h divergence is very unlikely, if a(u, y) is symmetric positive definite.

Example 4. Let τh be a triangulation of a given polygon domain. Then, discretize Poisson’s equation by finite elements on this triangulation. A Gauss-Seidel iteration with respect to the nodal basis converges. However the corresponding stiffness matrix is not diagonal dominant in general. 1 Example 5 (Linear Elasticity). Let E > 0 and 0 <ν < 2 . Define the symmetric derivative 1 ∂u ∂u ǫ := i + j ij 2 ∂x ∂x j i ǫ11 ǫ22  ǫ  Du := 33  ǫ   12   ǫ   13   ǫ   23    and the matrix 1 ν ν − − ν 1 ν 0 − − 1  ν ν 1  −1 − − , C E  1+ ν     0 1+ ν     1+ ν      32 where E and ν are physical constants. The bilinear form corresponding to the problem of linear elasticity is

a : (H1(Ω))3 (H1(Ω))3 R × → (u, v) (Du)T Dv d(x,y,z) 7→ C ZΩ Using suitable boundary conditions, this matrix is symmetric positive definite. Thus, Gauss-Seidel iteration with respect to the nodal basis on the ﬁnite element grid converges.

1.10.4 Analysis of the Convergence of the Jacobi Method

Model problem : Finite difference discretization of the Poisson equation Both, Gauss–Seidel and the Jacobi method, converge if the co- efficient matrix Lh of the finite difference discretization is weak diagonally dominant and • irreducible. • Now, we want to estimate the convergence rate in more detail for Poisson’s equation.

To solve the linear system Ax = b, the iteration matrix of the Jacobi method −1 is CJ = D (L + R). Then, for the model problem, we have

h2 h2 A = D L R = C = D−1(D A)= D−1A+E+E A = E L − − ⇒ j − − − 4 − 4 h (18)

It follows from exercise (1.2) that

h2 C e = 1 λ e . (19) j νµ − 4 νµ νµ Here λν,µ are the eigenvalues 4 πνh πµh λ = sin2 + sin2 ν,µ h2 2 2 for ν,µ = 1 . . . (m 1), where h = 1 . Thus, the iteration matrix C has the − m j eigenvalues πνh πµh (ρ ) = 1 sin2 sin2 (20) J νµ − 2 − 2 33 Here, J denotes the Jacobi method. In case of ν = µ we have,

πνh πνh πνh (ρ ) = 1 sin2 sin2 = 1 2sin2 = cos(πνh) (21) J νν − 2 − 2 − 2 

The following graph depicts the eigenvalues (ρJ )νν with respect to the parameter πνh in (21).

0.5

ρ 0

-0.5

-1 π/2 π π(νh)

= Bad convergence for high and low frequencies. ⇒ = Good convergence for middle frequencies. ⇒ In particular, one can show that the spectral radius of the iteration matrix is

ρ(C) = 1 O(h2) (22) − Now, the eﬀective convergence rate for the Jacobi method is

ln( 1 ) n G = sOp(c)/n = 2 . = O(h−2)= O(n). (23) eff ln(ρ(c)) n = The convergence rate for the Jacobi method (O(n)) is better than that ⇒ of the direct solver the Gauss elimination O(n2) .

1.10.5 Iteration Method with Damping Parameter

Let us assume that xk xk+1 is an iteration. The iteration can be written −→ as xk xk + (xk+1 xk). The term (xk+1 xk) can be treated as a −→ − − correction term. Now a damped iteration is xk ω(xk+1 xk), where −→ −

34 ω is called the damping factor or the relaxation parameter and ω • ∈ ]0, 2[.

ω > 1 is called over relaxation. •

ω < 1 is called under relaxation. •

SOR(Successive Over Relaxation) method is obtained by performing the Gauss-Seidel method with over relaxation. But SOR has disadvantages for e.g like,

It is very diﬃcult to ﬁnd ω for certain class of problems. •

1.10.6 Damped Jacobi Method

The Jacobi method with relaxation parameter ω =1 is

k+1 −1 k −1 xJacobi = D (L + R)xJacobi + D b (24) The Jacobi method with damping parameter ω is

xk+1 = xk + ω(D−1(L + R)xk + D−1b xk ) ω ω ω − ω = E(1 ω)+ ωD−1(L + R) xk + ωD−1b (25) − ω = C = E(1 ω)+ ωD−1(L + R) (26) ⇒ ω − This is the iteration matrix of the damped Jacobi method.

1.10.7 Analysis of the Damped Jacobi method

The iteration matrix of the damped Jacobi method can be written as

h2 C = E(1 ω)+ wD−1(D A)= E ωD−1A = E ω A (27) J,ω − − − − 4 Furthermore, by (26), the iteration matrix of the damped Jacobi method is

C = [E + ωC ωE] = (1 ω)E + ωC (28) J,ω j − − j where Cj is the iteration matrix of the Jacobi method. The eigenvalues of

35 the iteration matrix of the Jacobi method are πνh πµh (ρ ) = 1 sin2 + sin2 J ν,µ − 2 2 Thus, the eigenvalues of the iteration matrix of the damped Jacobi method are πνh πµh (ρ ) = 1 ω sin2 + sin2 (29) J,ω ν,µ − 2 2 Now, for ν = µ, we have

πνh (ρ ) = 1 2ω sin2 (30) J,ω ν,ν − 2 1 Thus, if ω = 2 πνh (ρ ) = 1 sin2 (31) J,ω ν,ν − 2 

The following graph depicts the eigenvalues (ρJ,ω)νν with respect to the parameter πνh in (31).

0.5

ρ 0

-0.5

-1 π/2 π π(νh)

1 This shows that the damped Jacobi method with ω = 2 has the properties

Bad convergence for low frequencies. • Good convergence for high frequencies. •

The Gauss–Seidel method has similar properties as the damped Jacobi 1 method with ω = 2 .

36 1.10.8 Heuristic approach

x x x B

x x x x

A x x x

By single step methods we require O(√n) = O(h−1) operations for a correction in B due to a change in A. The idea is to achieve faster correction by using a coarser grid.

37 2 Multigrid Algorithm

2.1 Multigrid algorithm on a Simple Structured Grid

2.1.1 Multigrid

Figure 3: l=3 Figure 4: l=2 Figure 5: l=1

Let l be the number of levels such that l N and max ∈

l ml = 2 n = (m 1)2 l l − −l hl = 2

for l = 1 ...lmax.

Let us assume that a PDE (e.g. Poisson’s equation) is given. Discretize this equation by the grids Ωl := Ωhl where l = 1,...,lmax. This leads to the discrete matrix equations

Alxl = bl (32) where b ,x S and S = Rnl . The matrix A is an invertible matrix of l l ∈ l l l order n n . l × l Let an iterative solver for (32) be given as

x k+1 = C relaxx k + N b = (xk) (33) l l l l l Sl,bl l

38 2.1.2 Idea of Multigrid Algorithm

Let xl be an approximate solution for (32). The algebraic el is deﬁned as

e = x x . (34) e l l − l e e e Now el has to be calculated in order to ﬁnd xl. The following residual equation is valid for el, e e

Alel = rl (35)

e where rl is called the residual and is given by

r = b A x (36) l l − l l e The aim is to ﬁnd an approximate solution of the residual equation by solving the equation approximately on a coarse grid Ωl−1. To this end, we need the following matrix operators

Restriction operator • I l−1 : S S l l 7→ l−1 Prolongation operator • Il : S S l−1 l−1 7→ l

39 2.1.3 Two–grid Multigrid Algorithm

Two–grid Multigrid algorithm with parameters v1 and v2 k Let xl be an approximate solution of (32) and v1 and v2 the parameters of pre–smoothing and post–smoothing.

1. Step 1 (Pre–smoothing)

xk,1 = v1 xk (37) l Sl,bl l 2. Step 2 (Coarse grid correction)

Residual calculation :

r = b A xk,1 (38) l l − l l Restriction :

l−1 rl−1 = Il rl (39) Solve on coarse grid:

−1 el−1 = Al−1 rl−1 (40)

Prolongation :

l el = Il−1el−1 (41)

Correction :

k,2 k,1 xl = xl + el (42)

3. Step 3 (Post–smoothing)

xk+1 = v2 (xk,2) (43) l Sl,bl l

40 2.1.4 Restriction and Prolongation Operators

Figure 6: O–Coarse grid point and X–Fine grid point

Let us abbreviate xi,j = x(ihl−1,jhl−1) and set xi,j = 0 for i =0 or j =0 or i = ml−1 or j = ml−1.

2.1.5 Prolongation or Interpolation

The interpolation or prolongation of x given by w = Il (x) is i,j i,j { l−1 }(ihl,jhl) deﬁned by the following equations

1 w = x (44) 2i,2j 2 i,j 1 w = (x + x ) (45) 2i+1,2j 4 i,j i+1,j 1 w = (x + x ) (46) 2i,2j+1 4 i,j i,j+1 1 w = (x + x + x + x ) (47) 2i+1,2j+1 8 i,j i+1,j i,j+1 i+1,j+1

2.1.6 Pointwise Restriction

Piecewise restriction is rarely applied and deﬁned by

I˙ l−1(x) = x (48) { l }(ihl−1,jhl−1) 2i,2j The quality of this restriction operator is not very good.

41 2.1.7 Weighted Restriction

Weighted restriction or full weighting is deﬁned by 1 I l−1(x) = (x + x + x + x )+ { l }(ihl−1,jhl−1) 8 2i+1,2j+1 2i−1,2j+1 2i+1,2j−1 2i−1,2j−1 1 (x + x + x + x )+ 4 2i+1,2j 2i−1,2j 2i,2j+1 2i,2j−1 1 x 2 2i,2j

Remark

l−1 T l (Il ) = Il−1 (49)

2.2 Iteration Matrix of the Two–Grid Multigrid Algorithm

Theorem 4. The iteration matrix of a two–grid Multigrid algorithm is

v2 v1 Ctwo grid = C relax E Il (A )−1I l−1A C relax (50) l l − l−1 l−1 l l l 

Proof

The coarse grid correction is

x k,2 = x k,1 + Il (A )−1I l−1(b A x k,1) l l l−1 l−1 l l − l l = E Il (A )−1I l−1A x k,1 + Il (A )−1I l−1b − l−1 l−1 l l l l−1 l−1 l l 

Therefore the iteration matrix of the coarse grid correction of the two–grid Multigrid algorithm is

E Il (A )−1I l−1A − l−1 l−1 l l A short calculation shows that the iteration matrix of two linear iteration algorithms is the product of the iteration matrices of these algorithms.

42 2.3 Multigrid Algorithm

k Multigrid algorithm MGM(xl , bl, l) with parameters (v1,v2,µ) k Let xlmax be an approximate solution of (32). Then, k+1 k xlmax = MGM(xlmax) is the approximate solution of (32) by the multigrid algorithm with an initial k vector xlmax. The multigrid algorithm can then be described as

k −1 If l = 1 then MGM(xl , bl, l)= Al bl If l> 1 then

Step 1 (v1-pre–smoothing) xk,1 = v1 (xk) l Sl,bl l Step 2 (Coarse grid correction) k,1 Residual : rl = bl Alxl − l−1 Restriction : rl−1 = Il rl Recursive call: 0 el−1 = 0 for i = 1 ...µ i i−1 el−1 = MGM(el−1, rl−1, l 1) µ − el−1 = el−1 l Prolongation : el = Il−1el−1 k,2 k,1 Correction : xl = xl + el

Step 3 (v2-post–smoothing)

k v2 k,2 MGM(x , bl, l)= (x ) l Sl,bl l

The algorithm µ = 1 is called V-cycle. The algorithm µ = 2 is called W-cycle.

Homework: Describe the multigrid algorithm as a finite state machine, where every state is smoothing step and an operation is a restriction or prolongation. Then, the finite state machine of a V-cycle looks like a “V” and the finite state machine of a W-cycle looks like a “W”.

Let N be the number of unknowns. The computational amount of the V-cycle and W-cycle is O(N).

43 The theory of multigrid algorithms shows that there is a constant ρ such that the convergence rate of the multigrid algorithm satisﬁes ρ(C ) ρ< 1 MGM,l ≤ independent of l. Thus, the eﬀective convergence rate of the multigrid algorithm is:

Geff (MGM,µ)= O(1) for µ = 1, 2.

2.4 Multigrid Algorithm for Finite Elements

2.4.1 Model Problem

Let τh1 ,τhlmax be a sequence of quasi-uniform subdivisions, where −·l · · hl = 2 such that V V (This means V V ) hi ⊂ hi+1 2h ⊂ h

2.4.2 Example

Every triangle is divided into four triangles

44 We want to solve the problem

Find u V such that hlmax ∈ hlmax a(u , v )= f(v ) v V (51) hlmax h h ∀ h ∈ hlmax

To this end, let us study all problems of type

Find u V such that hl ∈ hl a(u , v )= f (v ) v V (52) hl h l h ∀ h ∈ hl for every l = 0, , l · · · max where fl is a suitable coarse grid right hand side.

2.4.3 The Nodal Basis

Let (vk) be the nodal basis for V . Now (52) can be deﬁned in matrix i k∈˚Ωh hi form as follows:

Aixi = bi (53) where k j Ai = (akj) ˚ , akj = a(vi , vi ) (54) kj∈Ωhi x = (xk) (55) i i k∈˚Ωh b = (bk) (56) i i k∈˚Ωh and the solution vector uh is given by k k uhi = xi vi (57) X

2.4.4 Prolongation Operator for Finite Elements

The natural inclusion is the prolongation operator u V ∈ hi ↓ u V ∈ hi+1 To implement this operator, we have to describe this operator in a matrix form.

45 ′ By V V , there are coeﬃcients γk such that hi ⊂ hi+1 k k′ k′ k vi = γk vi+1 (58) Xk

Thus, we get

k′ k′ k′ k k′ uhi = xi vi = γk vi+1xi (59) ′ ′ Xk Xk Xk k′ k′ k = γk xi vi+1 (60) ′ ! Xk Xk Now the matrix version of the prolongation operator is

i+1 ′ ′ ′ k ′ k k Ii xi ′ = k γk xi k k P ⇓ i+1 k′ Ii = (γk )(k,k′)

2.4.5 Restriction Operator for Finite Elements

Observe that F (V )′. i ∈ hi This means that F : V R is a linear mapping. The natural inclusion i hi −→ is the restriction operator.

F (V )′ i+1 ∈ hi+1 ↓ F (V )′ i ∈ hi F (w) := F (w) w V i i+1 ∀ ∈ hi The matrix version of the restriction operator can be obtained as follows

k′ k′ k′ k bi = Fi(vi )= γk Fi(vi+1) (61) Xk k′ k = γk bi+1 (62) Xk i k′ Ii+1 = γk (63) (k′,k) 

46 2.5 Fourier Analysis of the Multigrid method

2.5.1 Local Fourier analysis

A multigrid algorithm consists of several parameters that have to be properly tuned such that the algorithm converges rapidly.The parameters are,

µ : recursion parameter.

ν1,ν2 : smoothing parameter.

: choice of smoother. Sl,bl l Il−1 : choice of the prolongation operator.

l−1 Il : choice of the restriction operator

Al for l < lmax : choice of the stiﬀness matrix on the courser grid.

(Almax is determined by the discretisation.)

The following simpliﬁcation is made in order to analyze the convergence of the two–grid method more easily and exactly.

Omission of the boundary conditions – transition to an inﬁnite dimensional grid

Instead of the ﬁnite dimensional grid

1 Ωd := (j h, j h,...,j h) j , j ,...,j 0,..., (64) h 1 2 d | 1 2 d ∈ h we apply an inﬁnite dimensional grid

∞ d Ω := (j h, j h,...,j h) j , j ,...,j Z (65) h { 1 2 d | 1 2 d ∈ } The operators A , Il−1, have to be extended to the inﬁnite l l Sl,bl dimensional grid in a suitable manner.

Remark

47 The operators A etc. are stencil operators, e.g a nine point stencil. • l The operators A etc. depend on the spatial coordinates. • l

Then, we deﬁne the operators on the inﬁnite grid as follows,

d d Let Qh be a stencil operator on the grid Ωh. For every inner d point x0 in the grid Ωh of the stencil (x0),a corresponding ∞ d S stencil operator Q h is deﬁned.

Example

Let d = 1. The stiffness matrix obtained by the finite difference 2 discretization of the operator d on the grid Ω1 is − dx2 h 2 1 − 1 2 1  − −  1 2 1 1 1 − − Ah =  . . .  2 (66)  ......  h    1 2 1   − −   1 2   −    ∞ 1 The operator on the infinite grid A h is now ...... ∞ 1 1 A h =  1 2 1  2 (67) − . . − . h  ......      which implies

∞ 1 1 ∞ 1 A (u)(x) = ( u(x h) + 2u(x) u(x + h)) x Ω (68) h − − − h2 ∀ ∈ h By the extension of the operators on the inﬁnite dimensional grid, we can ∞ d construct a two–grid method on the inﬁnite dimensional grid Ω h. To analyze the convergence of the two–grid method, we need to know the iteration matrix of the method. By Lemma 3, the iteration matrix for the two–grid method is

ν2 ν1 Crelax E Ih (A )−1IH A Crelax , where H = 2h. (69) h h − H H h h h 48 where,

relax Ch : iteration matrix of the smoothening step.

Eh : extended unit matrix.

h IH : extended prolongation operator.

H Ih : extended Restriction operator.

Ah, AH : extended stiﬀness matrices on the coarser grid.

Example

49 The operators for d=1 are as follows.

...... 1 Ah =  1 2 1  2 (70) − . . − . h  ......     . . .  ...... 1 AH =  1 2 1  2 (71) − . . − . 4h  ......     .  ..  1 2 1  1 1 IH = or factor (72) h 1 2 1 4 2√2    .   ..     .  ..  1  2   1 1 IH =  1 1  or factor (73) h   2 2√2  2     1     .   ..     .  ...... relax 1 1 Ch =  2 ω 1 ω 2 ω  (74) . −. .  ......     . . .  ...... ω= 1 2 1 1 1 =  4 2 4  . . .  ......      We allow these operators to act on the following functional spaces.

x V := exp iθ d π θ π (75) h h ∞ ( x∈ Ω h | − ≤ ≤ ) x V := exp iθ d π θ π (76) H H ∞ ( x∈ Ω H | − ≤ ≤ ) For reasons of simplicity, let us restrict ourselves to the 1–D case.

50 2.5.2 Deﬁnition

x ˜x The harmonic frequency of exp iθ h is exp iθ h where, θ˜ := θ π for θ 0 − ≥ θ˜ := θ + π for θ < 0

2.5.3 Local Fourier analysis of the smoother

Deﬁnition

x The functions exp iθ h are the eigenfunctions of the iteration matrix C of the smoother with eigenvalues µ(θ). This implies that S x x C exp iθ = µ(θ)exp iθ h h We then have the smoothening factor of as S

51 3 Gradient Method and cg

3.1 Gradient Method

Let A a symmetric positive deﬁnite n n matrix and b Rn. Find x Rn × ∈ ∈ such that

A x = b (77) · Theorem 5. The solution of the linear system (77) is solution of the minimization problem: 1 min xT Ax bT x (78) x∈Rn 2 −

Proof. Let 0 = v Rn and λ R. If x solves the minimization problem, 6 ∈ ∈ then d 1 (x + λv)T A(x + λv) bT (x + λv) = 0 dλ 2 − λ=0 1 (vT Ax + xT Av) bT v = 0 ⇒ 2 − vT Ax = vT b v Rn Ax = b ⇒ ∀ ∈ ⇒ Let Ax = b and v Rn. Then, we get ∈ 1 (x + v)T A(x + v) bT (x + v)= 2 − 1 1 1 = (x + v)T b + vT Av bT (x + v)+ xT Av 2 2 − 2 1 1 1 = vT Av (x + v)T b + bT v 2 − 2 2 1 1 = vT Av xT b minimum at v = 0 2 − 2 ⇒

Gradient method

1. Choose direction for seeking

d = f(x ), k −∇ k where f(x )= 1 xT Ax bT x . This implies k 2 k k − k

d = b Ax (79) k − k

52 2. Choose α R, such that f(x ) is minimal, where k ∈ k+1 xk+1 = xk + αkdk

Theorem 6. αk of the gradient method can be computed by T dk dk αk = T (80) dk Adk

Proof.

d 1 T T (xk + αkdk) A(xk + αkdk) b (xk + αkdk) = 0 dαk 2 − 1 (dT Ax + xT Ad )+ α dT Ad bT d = 0 2 k k k k k k k − k T T dk dk + αkdk Adk = 0 − T dk dk αk = T ⇒ dk Adk

3.2 Analysis of the Gradient Method

Let x∗ the accurate solution of 1 f(x)= xT Ax bT x minimum 2 − → This implies Ax∗ = b. The energy norm is deﬁned by x = √xT Ax. || ||A Lemma 2. 1 f(x)= f(x∗)+ x x∗ 2 2|| − ||A

Proof. 1 1 f(x) f(x∗) = xT Ax bT x x∗T Ax∗ + bT x∗ = − 2 − − 2 1 1 = xT Ax x∗T Ax + x∗T Ax∗ = 2 − 2 1 = x x∗ 2 2|| − ||A Lemma 3. T 2 ∗ 2 ∗ 2 (dk dk) xk+1 x = xk x 1 || − ||A || − ||A − dT Ad dT A−1d k k k k

53 Proof. By (79) and (80), we get:

f(xk+1) = f(xk + αkdk) 1 = (x + α d )T A(x + α d ) bT (x + α d ) 2 k k k k k k − k k k 1 = f(x )+ α dT (Ax b)+ α2dT Ad k k k k − 2 k k k T 2 1 (dk dk) = f(xk)+ T 2 dk Adk Now, by Lemma 2, we obtain T 2 ∗ 2 ∗ 2 (dk dk) xk+1 x A = xk x A T || − || || − || − dk Adk By d = A(x x∗), it is k − k − x x∗ 2 = (A−1d )T A(A−1d )= dT A−1d || k+1 − ||A k k k k and so we get (dT d )2 x x∗ 2 = x x∗ 2 1 k k || k+1 − ||A || k − ||A − dT Ad dT A−1d k k k k . Lemma 4 (Inequality of Kantorowitsch). Let A symmetric positive deﬁnite and κ the condition number of A. Then, the following inequality holds (xT Ax)(XT A−1x) 1 1 2 √κ−1 + √κ (xT x)2 ≤ 2 2

Proof. Let the eigenvalues of A a = λ λ . . . λ = b and b = κ. 1 ≤ 2 ≤ ≤ n a By principal axis transformation, one gets T T −1 2 −1 2 (x Ax)(X A x) λiyi λi yi −1 T 2 = 2 2 = λizi λi zi (x x) ( yi ) P P Xi Xi 2 P yi where x = i yiei, ei are the orthogonal eigenvector, and zi = 2 . Pj yj −1 −1 Observe, thatP i zi = 1. Thus, P = ( i λizi, i λI zi)= zi i(λi, λi ) is a convex combination. P P P P Now, deﬁne P = (λ, µ)

λ = λizi Xi −1 µ = λi zi Xi 54 −1 λ1 P = (λ, µ)

−1 λn

λ1 λn

Figure 7: The convex function x x−1. →

Then, by Figure 7, we get

µ = λ−1z (λ + λ λ)λ−1λ−1, i i ≤ 1 n − 1 n Xi −1 −1 since the point P is below the line between (λ1, λ1 ) and (λn, λn ). Now, λ + λ λ λ + λ )2 max λ 1 n − = 1 n λ1≤λ≤λn λ1λn 4λ1λn completes the proof. Theorem 7. The gradient method converges as follows: κ 1 k x x∗ − x x∗ || k − ||A ≤ κ + 1 || 0 − ||A

Proof. Apply Lemma 2 to 4 and observe that 1 4κ (κ 1)2 1 2 = 1 2 = − 2 . − 1 1 −1 − (κ + 1) (κ + 1) 2 √κ + 2 √κ 

3.3 The Method of Conjugate Directions

If A is positive deﬁnite, then xT Ay deﬁnes a scalar product. Let d0, d1,...,dn−1 be A-orthogonal (conjugated) vectors. Then, the set d , d ,...,d is a basis of Rn. { 0 1 n−1}

55 Observe, that any vector y Rn can be written as: ∈ n−1 y = αkdk Xk=0 Lemma 5. Let x Rn be a start vector. Then, deﬁne 0 ∈

xk+1 = xk + αkdk (81) where

T gk dk αk = T , gk = Axk b (82) −dk Adk − This sequence leads to the exact solution after at most n iterations: −1 xn = A b.

Proof. There exists αi such that

x∗ x = α d − 0 i i Xi Thus, we get dT A(x∗ x )= α dT Ad i − 0 i i i T ∗ T di A(x x0) di (Ax0 b) αi = T − = T − di Adi − di Adi By induction we show that equation (82) holds. k = 0 follows from the upper equation. T T By xi = k

P T di (Ax0 b) αi = T − − di Adi

Lemma 6. xk in Lemma 5 minimizes 1 f(x)= xT Ax bT x 2 − on x + V , where V = span d , d ,...,d . 0 k k { 0 1 k−1} Furthermore, the following orthogonalization property holds:

T di gk = 0 for i

56 Proof. Let us show, that it is enough to prove (83).

Let d V dT g = 0 ∈ k ⇒ k 1 f(x + d) = (x + d)T A(x + d) b(x + d) k 2 k k − k 1 1 = (xT Ax + dT Ad + dT Ax bT x bT d 2 k k 2 k − k − 1 = (xT Ax + dT Ad)+ dT g bT x 2 k k k − k 1 = (xT Ax + dT Ad) bT x 2 k k − k Minimization by d V not possible! ⇒ ∈ k Proof of (83) by induction: i = k 1: − T T T gk−1 dk−1 dk−1gk = dk−1 A xk−1 T dk−1 b = 0 ( ) − dk−1Adk−1 − ∗ i < k 1 By x x = α d , it holds − k − k−1 k−1 k−1 g g = A(x x )= α Ad k − k−1 k − k−1 k−1 k−1 dT (g g )=0 for i < k 1 ⇒ i k − k−1 − By the induction hypothesis, (83) follows for i k 2. ≤ −

3.4 cg-Method (Conjugate Gradient Algorithm)

The directions d0,...,dk+1 are computed by an orthogonalization of the gradients:

d = g 0 − 0 dk+1 = gk+1 + βkdk − T gk+1Adk where βk = T if gk = 0 dk Adk 6 Theorem 8. If g = 0 then it holds k+1 6 (I) It is dk−1 = 0 6 k−1 (II) Vk = span[g0, Ag0,...,A g0]= span[g0, g1,...,gk−1]= span[d0, d1,...,dk−1] (III) The vectors d0, d1,...,dk−1 are pairwise A-orthogonal

(IV) It is f(xk) = minz∈Vk f(x0 + z) T T gk gk gk+1gk+1 (V) αk = T , βk = T dk Adk gk gk

57 Proof by Induction from (I) to (V) k = 1 : obvious k k + 1 : First observe, that →

span[g0, g1,...,gk−1] = span[d0, d1,...,dk−1] follows by d = g + β d . Thus, k+1 − k+1 k k g = g + A(x x )= g + α Ad k k−1 k − k−1 k−1 k−1 k−1 implies that

g span[g , Ag ,...,Akg ] k ∈ 0 0 0 span[g , g ,...,g ] span[g , Ag ,...,Akg ]. ⇒ 0 1 k ⊂ 0 0 0

By induction hypothesis d0, d1,...,dk−1 are linear independent. By (83), the optimization property of xk, it is

dT g =0 for i

k−1 span[g0,...,gk] = span[g0, Ag0,...,A g0].

This completes the proof of (II). Since dk−1, gk are linear independent, it follows d = 0, (I). Now, we proof (III): k 6 dT Ad = dT Ag + β dT Ad i k − I k k−1 i k−1 By the construction of β , the orthogonality (III) holds for i = k 1: k−1 − T di Adk = 0.

For i < k 1 the induction hypothesis dT Ad = 0 implies − i k−1 dT Ad = dT Ag i k − i k Since Ad span[d ,...,d ] and i + 1 < k, ( ) leads to i ⊂ 0 i+1 ∗ T di Adk = 0

(IV) is a implication from Lemma 6. Proof of (V): d = g + β d k − k k−1 k−1 and ( ) imply that ∗ T T gk dk gk gk αk = T = T . −dk Adk dk Adk

58 Now, observe that gT g = 0 follows by ( ) and (II). Thus, we get k+1 k ∗ gT Ad gT α Ad gT (g g ) gT g k+1 k k+1 k k k+1 k+1 − k k+1 k+1 βk = T = T = T = T . dk Adk gk gk gk gk gk gk

To implement the cg-algorithm in an eﬃcient way, we apply (V) in Theorem (8). Furthermore, observe that

g = A(x + α d ) b = g + α Ad k+1 k k k − k k k and let us introduce the auxiliary vector

h := Ad. cg algorithm

x = x0 g = Ax b T − δ0 = g g if δ ǫ stop 0 ≤ d = g − recursion: k = 0, 1,... h = Ad δ α = 0 dT h x := x + αd g := g + αh T δ1 = g g(= δk+1) if δ ǫ stop 1 ≤ δ δ β = 1 = β = k+1 δ δ 0 k d = g + βd − δ0 := δ1

3.5 Analysis of the cg algorithm cg is a direct and an iterative method! Lemma 7. Let p P be a polynomial such that ∈ k p(0) = 1, p(z) r for all z σ(A) | |≤ ∈

59 Then, for the cg algorithm, the following inequality holds

x x∗ r x x∗ || k − ||A ≤ || 0 − ||A

Proof. Let q(z)= p(z)−1 and y := x + q(A)g . Then, by g = A(x x∗), z 0 0 0 0 − we get

y x∗ = x x∗ + y x = x x∗ + q(A)g = p(A)(x x∗) − 0 − − 0 0 − 0 0 − y x∗ p(A) x x∗ ⇒ || − ||A ≤ || ||A · || 0 − ||A Let w = j cjej, where ej are the orthonormal eigenvectors of A such that Ae = λ e . Then, it holds j jPj p(A)w 2 = c p(λ )e 2 = λ c p(λ ) 2 || ||A || j j j||A j| j j | Xj Xj r2 λ c 2 r2 c e 2 = r2 w 2 ≤ j| j| ≤ || j j||A || ||A Xj Xj p(A) r ⇒ || ||A ≤ This shows y x∗ r x x∗ . By Theorem 8 and Lemma 2, we || − ||A ≤ || 0 − ||A conclude x x∗ r x x∗ || k − ||A ≤ || 0 − ||A Lemma 8. Let 1 T (x)= (x + x2 1)k + (x x2 1)k k 2 − − − h p p i for k = 0, 1,.... Then it holds a) T (x) is a real-valued polynomial of degree k. k ≤ b) T (x) 1 for 1 x 1. | k |≤ − ≤ ≤ c) T (x) 1 (x + √x2 1)k for x 1. k ≥ 2 − ≥ d) Tk(1) = 1

Proof. a) Using the binomial formula, one can see that the terms with odd powers cancel. Thus, of Tk is a real-valued polynomial. b) T (x) 1 x + i√1 x2 k + x i√1 x2 k 1 for x 1. | k |≤ 2 | − | | − − | ≤ | |≤ c) and d) are obvious.

Remark: Tk is called Tschebyscheﬀ polynomial. One can prove

Tk(x) = cos(k arccos x)

60 Theorem 9.

k ∗ 1 ∗ √κ 1 ∗ xk x A x0 x A 2 − x0 x A || − || ≤ κ+1 || − || ≤ √κ + 1 || − || Tk κ−1 

Proof. Let a, b be the extremal eigenvalues of A. Set

b+a−2x Tk b−a p(x)= b+a Tk b−a Then, p(0) = 1 and by b) in Lemma 8. 1 p(x) ≤ b+a Tk b−a for every x ρ(A). Furthermore, by Lemma 7: ∈

∗ 1 ∗ xk x A x0 x A || − || ≤ b+a || − || Tk b−a Furthermore, observe that b + a κ + 1 = b a κ 1 − − and κ + 1 κ + 1 2 κ +1+ √4κ √κ + 1 + 1= = . κ 1 s κ 1 − κ 1 √κ 1 − − − − Now c) in Lemma 8 completes the proof.

3.6 Preconditioned cg Algorithm

Let A be a symmetric positive deﬁnite n n-matrix and C a symmetric × positive deﬁnite n n-matrix, such that C is an approximation of A. × Example 6. C is the diagonal of A . • C is the tridiagonal part of A . Then, C−1 can be computed be a • LR-decomposition.

C−1 is the result of a suitable symmetric multigrid algorithm. •

61 Instead of the equation

Ax = b we try to solve the equation

C−1Ax = C−1b, if C−1A has a smaller condition number than A.

The problem is that in general C−1A is not symmetric positive deﬁnite. Therefore, we apply the following Lemma. Lemma 9. Deﬁne

x,y := xT Cy. h iC C−1A is symmetric positive deﬁnite with respect to , . h· ·iC

This leads to the algorithm precondition cg algorithm (bad version)

x = x0 g = C−1(Ax b) T − δ0 = g Cg if δ ǫ stop 0 ≤ d = g − recursion: k = 0, 1,... h = C−1Ad δ α = 0 dT Ch x := x + αd g := g + αh T δ1 = g Cg if δ ǫ stop 1 ≤ δ δ β = 1 = β = k+1 δ δ 0 k d = g + βd − δ0 := δ1

In several cases C−1 can be computed, but C cannot be computed. Therefore, one applies the following more eﬃcient version of the

62 precondition cg algorithm, which does not require the computation of C. In this version we introduce the new variables w and r by Cg =: r Ch =: w and omit the variable h. precondition cg algorithm (eﬃcient version) x = x0 r = Ax b − g = C−1r T δ0 = g r if δ ǫ stop 0 ≤ d = g − recursion: k = 0, 1,... w = Ad δ α = 0 dT w x := x + αd r := r + αw g := C−1r T δ1 = g r if δ ǫ stop 1 ≤ δ δ β = 1 = β = k+1 δ δ 0 k d = g + βd − δ0 := δ1

4 GMRES

Let A be an invertible n n matrix. Furthermore, let b Rn and x Rn × ∈ 0 ∈ a starting vector.

Problem: Find x Rn such that ∈ Ax = b

Let us consider the Krylov space Km deﬁned by r = b Ax , K = span r ,...,Am−1r 0 − 0 m { 0 0}

63 4.1 Minimal residual method

Find x x + K such that m ∈ 0 m b Ax is minimal. (84) || − m||2

A stable basis of Km can be obtained by the Arnoldi-algorithm. The Arnoldi-algorithm is based on the orthogonalization algorithm of Gram-Schmidt: Arnoldi Algorithm r Let q with q = 1, q = 0 . 1 || 1||2 1 r || 0||2 For j = 1,...,m 1 : − q˜j+1 = Aqj, hij =< q˜j+1,qi > for i = 1, 2,...,j j q˜ =q ˜ h q j+1 j+1 − ij i Xi=1 h = q˜ j+1,j || j+1||2 q˜j+1 qj+1 = hj+1,j

Observe, that one has to stop the Arnoldi algorithm, if hj+1,j = 0.

To analyze the properties of the Arnoldi algorithm, let us deﬁne the matrices

h11 h12 ..  h21 h22 .  ∗ ..  h32 .  Qk = (q1,...,qk) ,Hk+1,k =    ..   . hk−1,k     hk,k     h   k+1,k    Hk+1,k is a Hessenberg matrix.

Then, we get the following lemma. Lemma 10. (i) AQk = Qk+1Hk+1,k (ii) Q is an orthogonal matrix. (iii) K = span q ,...,q m { 1 m}

64 Proof: (i) :

j j+1 h q = Aq h q , Aq = h q j+1,j j+1 j − ij i j ij i Xi=1 Xi=1 for j = 1, ..., k. (ii) - (iii) is trivial.

By this lemma, we get:

x = x + Q y, y Rm m 0 m ∈

b Ax = r AQ y || − m||2 || 0 − m ||2 = r Q H y || 0 − m+1 m+1,m ||2 = Q (βξ H y) || m+1 1 − m+1,m ||2 = βξ H y , || 1 − m+1,m ||2 1 0 where β = r0 2 and ξ1 =  . . For solving the minimization problem || || .    0    (84) it is enough to resolve the problem:

Minimization Problem: Find y Rm such that ∈ βξ H y minimal (85) || 1 − m+1,m ||2 → The standard approach to solve this problem is to apply the QR-algorithm and Givens rotations.

4.2 Solution of the Minimization Problem of GMRES

min βξ1 Hm+1,my 2, (86) y∈Rm || − || where Hm+1,m is a the Hessenberg-matrix:

h11 h12 ..  h21 h22 .  ∗ ..  h32 .  Hk+1,k =    ..   . hk−1,k     hk,k     h   k+1,k    65 Problem (86) can be solved by the QR decomposition. To this end, let F be a unitary matrix (m + 1) (m + 1) matrix and R an upper × m+1,m triangular matrix, where the last row is 0 and

H Hm+1,m = F Rm+1,m

Then, one gets

min βξ1 Hm+1,my 2 = min βFξ1 Rm+1.my 2 y∈Rm || − || y∈Rm || − || The solution of this problem is:

^−1 y = Rm,m βFξ1,

^−1 where the operator˜omitsg the last row. Rm,m can easily be computed, ^ since Rm,m is an upper triangular matrix.

4.3 Computation of QR-Decomposition with Givens Rotation

F = F F F m m−1 · · · 1 I I ci si in the real case ci si Fi =   =   , s c s c − i i − i i  I   I          where ci = cosθi, si = sinθi.

It is easy to verify that

c s c s 1 0 i i i − i = s c s c 0 1 − i i i i 

So F is unitary. Construct F1,...,Fm−1 such that

∗  ∗. ∗  .. (Fm−1Fm−2 F1) Hm+1,m =   · · ·    ∗ ∗   0 d     0 h     

66 where h = hm+1,m and Fm satisﬁes the equation c s d m m = ∗ (87) s c h 0 − m m To obtain (87), consider the following to cases:

1. d = 0: cm = 0, sm = 1 h |d| 2. d = 0: sm = cm , cm = 6 d √|d|2+|h|2

Furthermore, observe h2 d2 s2 + c2 = + 1 (88) m m d2 d2 + h2 

4.4 The GMRES Algorithm

r0 (1) Let x0 be given. Compute r0 = b Ax0, q1 = . Set − ||r0||2 ξ = (1, 0,..., 0)T , β = r || 0||2 For k = 1, 2,... (2) Compute qk+1 and hi,k, i = 1,...,k + 1 by the Arnoldi Algorithm. Set H(i, k) := hi,k, i = 1,...,k + 1

(3) Apply F1,...,Fk−1 to the last column of H, that means for i = 1,...,k 1 − ci si H(i, k) H(i, k) si ci H(i + 1, k) → H(i + 1, k) − (4) Compute the rotation sk, ck to get H(k + 1, k) to 0.

(5) Compute

ξ(k) ck sk ξk ξ(k + 1) ← s c 0 − k k H(k, k) c H(k, k)+ s H(k + 1, k) ← k k

H(k + 1, k) 0 ← (6) If the residual β ξ(k + 1) is small enough, stop with the following | | solution:

67 Solve H y = βξ • k,k k k+1 x = x + Q y • k 0 k k

The computational amount of one GMRES step is

O(kn).

4.5 Convergence of the GMRES method

Theorem 10. Let A Rn×n positive deﬁnite, that means xT Ax > 0 for ∈ every x Rn,x = 0. Let r = b Ax . Then, it holds ∈ 6 m − m m 2 AT +A 2 λmin 2 rm 2 1 T r0 2 || || ≤  − λmax(A A) || ||   Proof. x = x +p ˜(A)r , wherep ˜ P m 0 0 ∈ m−1 r = b Ax = b A(x +p ˜(A)r ) = || m||2 || − m||2 || − 0 0 ||2 = r Ap˜(A)r = (1 Ap˜(A))r || 0 − 0||2 || − 0||2 = p(A)r || 0||2 whereat p P with p(0) = 1. Now, deﬁne q(A) = 1 αA, α > 0. By ∈ m − the minimization property it holds

r = p(A)r qm(A)r q(A) m r (89) || m||2 || 0||2 ≤ || 0||2 ≤ || ||2 || 0||2

(I αA)x 2 q(A) 2 = sup || − ||2 = || ||2 x 2 x6=0 || ||2 (x,Ax) (Ax, Ax) sup 1 2α 2 + α2 2 − (x,x) (x,x) x6=0 2 2 Sine A is positive deﬁnite, it follows

T (Ax, Ax)2 (x, A Ax)2 T = λmax(A A) =: λ˜max (x,x)2 (x,x)2 ≤ AT +A T (x,Ax)2 (x, 2 x)2 A + A = λmin( ) =: λ˜min > 0 (x,x)2 (x,x)2 ≥ 2 This shows:

q(A) 2 1 2αλ˜ + α2λ˜ || ||2 ≤ − min max

68 The minimization of the right hand side leads to:

λ˜min αmin = > 0 λ˜max and thus: ˜2 ˜2 2 λmin λmin 0 q(A) 2 1 2 + (90) ≤ || || ≤ − λ˜max λ˜max λ˜2 = 1 min < 1 − λ˜max (89) and (90) show the assertion.

5 Eigenvalue Problems

5.1 Rayleigh Quotient

Let A, B symmetric, positive deﬁnite n n matrices. The general × eigenvalue problem is: Find λ R and x Rn,x = 0 such that ∈ ∈ 6 Ax = λBx

Example 7. 1. Let V be a ﬁnite element space and let V H1(Ω). h h ⊂ 0 Find λ R and u V such that ∈ h ∈ h u v dz = λ u v dz ∇ h∇ h h h ZΩ ZΩ for every v V . h ∈ h 2. Eigenmodes of waveguides

2 ∆+ k0ǫ u = λu Theorem 11. Let λmin the smallest and λmax the largest eigenvalue of B−1A. Then, it holds

xT Ax min = λmin, x6=0 xT Bx xT Ax max = λmax. x6=0 xT Bx

69 xT Ax xT Bx is called Rayleigh quotient. If

x˜T Ax˜ = λ x˜T Bx˜ min then x˜ is an eigenvector with eigenvalue λmin. If

x˜T Ax˜ = λ x˜T Bx˜ max then x˜ is an eigenvector with eigenvalue λmax.

Proof. Choose the inner product

x,y := xT By. h i B−1A is symmetric with respect to , , since h· ·i T B−1Ax,y = xT AT B−1 By = xT Ay = x,B−1Ay .

Thus, there exist , -orthogonal eigenvectors e ,...,e such that h· ·i 1 n eigenvalues λ λ . . . λ . This means: 1 ≤ 2 ≤ ≤ n −1 B Aei = λiei, T ei Bej = δij.

Let x = ciei. Then, we get

xTPAx c2λ = i i . (91) xT Bx c2 P i This implies P xT Ax λ λ . min ≤ xT Bx ≤ max Furthermore, (91) implies that the Rayleigh quotient is maximal or minimal if and only if x is an eigenvector. This completes the proof.

Corollary. Let V Rn be a vector space. Then, ⊂ xT Ax min λmin x∈V,x=06 xT Bx ≥ xT Ax max λmax x∈V,x=06 xT Bx ≤

70 5.2 Method of Conjugate Gradients

Let xT Ax λ(x) := . xT Bx Lemma 11. The gradient and Hesse-matrix of λ(x) are 2 g(x) = (Ax λ(x)Bx) xT Bx − 2 H(x) = (A λ(x)B Bxg(x)T g(x)xT B). xT Bx − − −

Proof.

1 xT Ax g(x) = 2Ax 2Bx xT Bx − (xT Bx)2 2 = (Ax λ(x)Bx). xT Bx −

2 H(x) = 2Bx(Ax λ(x)Bx)T −(xT Bx)2 − 2 + (A g(x)(Bx)T Bλ(x)) xT Bx − − 2 = (A λ(x)B Bxg(x)T g(x)xT B). xT Bx − − −

−1 Lemma 12. Let λ1 be the maximal eigenvalue of B A and λn be the minimal eigenvalue of B−1A. Then H(e1) is positive deﬁnite and H(en) is negative deﬁnite.

Proof. By Theorem 11, g(e1)=0 and g(en) = 0. Then, a simple calculation completes the proof.

One can consider 1 λ˜(x + h) := λ(x)+ g(x)T h + hT H(x)h 2 as an approximation of the functional λ(x + h).

71 We are looking for an approximation of e1 and en, where we assume

λ < λ ... λ < λ . 1 2 ≤ ≤ n−1 n are the eigenvalues corresponding ei. Starting with x0, we construct a sequence (xk) which converges to e1 and en. By Theorem 11, we have to ﬁnd the extreme values of λ(x). Thus, let us deﬁne

xk+1 = xk + αksk such that

∂λ(xk+1) T = g(xk+1) sk = 0, ∂αk where sk is a search direction. T Lemma 13. The equation g(xk+1) sk = 0 leads to a quadratic equation with respect to αk.

Proof.

T g(xk+1) sk = 0

T ⇓ sk (A(xk + αksk) λ(xk + αksk)B(xk + αksk)) = 0 T− T (xk + αksk) B(xk + αksk)sk A(xk + αksk) sT B(x + α s )(x + α s )T A(x + α s ) = 0 − k k k k k k k k k k 3 The term αk in this equation cancels.

Construction of the search direction sk: Let us assume, we are looking for an approximation of λmin = λ1.

1.Choice of sk: Gradient method:

s := g(x )T . k − k

To ﬁnd a better search direction, let us construct sk such that

T sk := v + βw such that: g(xk) w = 0.

72 Now, consider the λ˜(x + αksk) 1 λ˜(x + α s ) = λ(x )+ g(x )T α s + (α s )T H(x)(α s ) k k k k k k k 2 k k k k T = λ(xk)+ g(xk) (v + βw)αk + 1 α2(v + βw)T H(x)(v + βw) 2 k T = λ(xk)+ g(xk) vαk + 1 α2 vT H(x)v + β2vT H(x)w + β2wT H(x)w 2 k To minimize λ˜(xk + αksk), let us choose β such that ∂λ˜(x + α s ) k k k = 0. ∂β This implies

vT Hw + βwT Hw = 0

⇓ (v + βw)HW = 0

⇓ vT Hw β = . −wT Hw T By Lemma 11 and g(xk) w = 0, we obtain vT (A λ(x )B)w vT g(x )xT Bw β = − k − k k . wT (A λ(x )B)w − k

By Lemma 13 we can choose v and w as follows: 2.Choice of sk: Conjugate gradient method:

sk := v + βw v = g(x ) − k w = sk−1.

3.Choice of sk: Another method:

sk := v + βw

v = r = Axk λ(xk)Bxk −T xk Bsk−1 w = sk−1 T xk. − xk Bxk T T Here: xk Bw = 0 but not g(xk) w = 0.

73 5.3 Simple Vector Iteration

Let A be a n n matrix. For reasons of simplicity, we assume that A is × diagonalizable. Similar results hold for general matrices.

Let us assume that λ1,...,λn are eigenvalues of A with orthonormal eigenvectors ei. Furthermore, let us assume that: (i) λ = . . . = λ > λ . . . λ | 1| | r| | r+1|≥ ≥ | n| (ii) λ1 = . . . = λr (iii) Let x0 be a start vector such that : x = n c e , E := c e + . . . + c e = 0. 0 i=1 i i 1 1 r r 6 P Algorithm: Vector Iteration:

xi+1 = Axi x x˜ = i i x || i|| Theorem 12. Let us assume that A is symmetric positive deﬁnite. x λ i i E r+1 x E λi − ≤ λ || − || 1 1

xi kx˜i+1k2 λr+1 limi→∞ i = E, lim i→∞ kx˜ k = λ1 and λ < 1. λ1 i 2 | | | 1 |

Proof. First, observe that x λ i λ i i = c e + . . . + c e + c r+1 e + . . . + c n e . λi 1 1 r r r+1 λ r+1 n λ n 1 1 1 This implies i i xi λr+1 λn i E = cr+1 er+1 + . . . + cn en λ1 − λ1 λ1 2 2 1 n 2 λ 2i = c 2 j  | j| λ  j=r+1 1 X   1 n 2 λ 2i c 2 r+1 ≤  | j| λ  j=r+1 1 X λ i  = r+1 x E . λ k − k2 1

Modiﬁcation of the vector iteration

74 x = A−1x leads to the smallest eigenvalue • i+1 i → x = (2Eλ A) x For symmetric positive deﬁnite matrices • i+1 max − i this also leads to the smallest eigenvalue

x = (A λI)−1 x only works, if λ is close to an eigenvalue • i+1 − i λj, that means λ λ << λ λ | j − | | k − |

One obtains numerical problems, if the eigenvalues are very close to each other (cluster). In this case, one has to ﬁnd a group of orthogonal eigenvectors.

75 5.4 Computation of Eigenvalues using the Rayleigh Quotient

n Let Vk be a sub-vector space of the R and let A be symmetric positive deﬁnite. Then,

xT Ax µ1 = min T (92) x∈Vk x x is an approximation of the smallest and

xT Ax µ2 = max T (93) x∈Vk x x an approximation of the largest eigenvalue. If k <

xT Ax min T (94) x∈Vn x x (92) can be solved by vector iteration, a direct solver, QR-algorithm, or any other direct solver. Theorem 13. Let V = span d , Ad ,...,Akd and let us assume that k { 0 0 0} the eigenvalues of A are numbered as follows

λ = λ = ... = λ < λ λ ...λ , 1 2 r−1 r ≤ r+1 ≤ n where r 2. Let e be the corresponding eigenvectors. Now, deﬁne ≥ i Z = span e ,...,e . 1 { 1 r−1} Then, it holds

2 tan φ 0 µ λ (λ λ ) 1 , 1 1 n 1  λ1  κr+1−2 ≤ − ≤ − λr Tk  κr−1      λ1 κr+1−2 where κ = λn , λr > 1, r λr κr−1 1 k T (x) x + x2 1 , k ≥ 2 − p and T d0 Az1 cos φ1 = max | | . z1∈Z1 d z || 0||A|| 1||A 76 So φ1 is the angle between d0, Z. To calculate the largest eigenvalue, let us use the following abbreviation: λr λ1 . . . λr < λr+1 = . . . = λn and κr = . Then, we get: ≤ ≤ λ1 2 tan φ1 λn µn (λn λ1) − ≤ − T 2κ−κr−1  k κ−κr  

Proof. Let ei be the normalized eigenvectors of A, Aei = λiei. Then, it follows n d0 = ciei i=1 X T T x Ax (p(A)d0) Ap(A)d0 µ1 = min T = min T = x∈Vk x x p∈Pk (p(A)d0) (p(A)d0) n 2 2 i=1 ci λip(λi) = min n 2 2 p∈Pk c p(λ ) P i=1 i i This implies P n c2(λ λ )p(λ )2 0 µ λ i=1 i i − 1 i = ≤ 1 − 1 ≤ n c2p(λ )2 P i=1 i i n c2(λ λ )p(λ )2 = i=r i i − 1P i n c2p(λ )2 ≤ P i=1 i i n c2p(λ )2 (λ Pλ ) i=r i i = ≤ n − 1 n c2p(λ )2 Pi=1 i i 1 = (λn λ1)P 2 r−1 2 − p(λ1) Pi=1 ci 1+ n 2 2 Pi=r ci p(λi) for every polynomial p. To estimate the smallest eigenvalue choose: λ + λ 2λ p(λ)= T n r − k λ λ n − r Then, it holds p(λ ) = 1, p(λ ) 1 for i = r, r + 1,...,n 1 | n | | i |≤ − Thus, we get 1 0 µ1 λ1 (λn λ1) 2 r−1 2 ≤ − ≤ − p(λ1) Pi=1 ci 1+ n 2 Pi=r ci n c2 1 (λ λ ) i=r i ≤ n − 1 r−1 2 2 i=1 ci T λn+λr−2λ1 P k λn−λr P 77 λ1 λ + λ 2λ κr + 1 2 T n r − 1 = T − λr k λ λ k κ 1 n − r r − ! n 2 i=r ci = tan φ1. qPr−1 2 i=1 ci qP To estimate the largest eigenvalue choose: 2λ λ λ p(λ)= T − r − 1 , k λ λ n − r where the eigenvalues are λ ...λ < λ = . . . = λ . Then, we get 1 ≤ r r+1 n n c2(λ λ )p(λ )2 0 λ µ i=1 i n − i i ≤ n − n ≤ n c2p(λ )2 P i=1 i i r c2 1 (λ λ ) i=1Pi ≤ n − 1 n c2 2 i=r+1 i T 2λn−λr−λ1 P k λn−λr P 1 (λ λ ) tan φ . ≤ n − 1 1 2 T 2κ−κr−1 k κ−κr Example: Poisson’s Equation Let us discretize Poisson’s equation by finite differences. Then, the eigenvalues of the matrix A are: 4 πνh πµh λ = sin2 + sin2 ν,µ h2 2 2 The smallest eigenvalue is at µ = ν = 1: 4 π2h2 λ = 2 = 2π2 1 h2 · 4 4 π2h2 π2h2 λ = + 4 = 5π2 2 h2 · 4 4 Our aim is to find the smallest eigenvalue λmin of A. There are to ways to get an approximation of λmin by minimizing the Rayleigh quotient.

1. Application of the Rayleigh quotient to A−1:

2κ κr 1 lim Tk − − = Tk(2) n→∞ κ κ − r 1 1 T (2) (2 + √3)k (3.7)k. k ≥ 2 ≥ 2

78 This implies fast convergence of the smallest eigenvalue of A, by using the inverse iteration applied to A−1.

2. Application of the Rayleigh quotient to A:

λn + λr 2λ1 lim Tk − = Tk(1) = 1. n→∞ λ λ n − r This implies low convergence of the smallest eigenvalue of A for large n! This shows that the ﬁrst approach is better!!!

79 5.5 Jacobi-Davidson-Algorithm

5.5.1 The Jacobi-Method

Let the coordinate system be transformed such that

1 e = 1 0 is a “good “ approximation of an eigenvector. We want to solve the problem 1 1 A = λ z z z Cn−1, λ C. ∈ ∈ Idea: Apply Newton method to

1 1 λ A λ =: f . z − z z Then, we have to calculate

−1 λ λ λ λ n+1 := n f ′ n f n . zn+1 zn − zn zn −1 ′ λn The calculation of f is difficult. Since, e1 is a “good “ zn approximation of an eigenvector, we can define the following approximative Newton method: −1 λ λ λ λ n+1 := n f ′ n f n . zn+1 zn − 0 zn Let us find a short formula for this iteration. To this end, let α cT A = . Then, we get b F λ 0 cT 1 0 f ′ n = 0 0 F − 0 λ E n λ p Let us abbreviate f n = . Then, let q, u be such that z w n λ q p f ′ n = . 0 u w 80 This implies

T c u q p −1 T − = u = (F λnE) w,q = c u p (F λnE)u w ⇒ − − − p λ α + cT z λ = f n = n n w zn b + F zn − λnzn p = α + cT z λ n − n ⇒ w = b + F zn λnzn = b + (F λnE)zn − − u = (F λ E)−1 (b + (F λ E)z ) = (F λ E)−1b + z − n − n n − n n z = z u = (F λ E)−1( b) n+1 n − − n − q = cT (F λ E)−1b + z α cT z + λ − n n − − n n = cT (F λ E)−1b α + λ − n − n λ = λ q = cT z + α n+1 n − n+1 SIAM Review, June 2000, Vol. 42, Number 2.

Instead of inverting F λ E exactly, one can approximate F λ E by the − n − n diagonal. This means we apply the Jacobi iteration for solving z = (F λ E)−1( b) as follows: n+1 − n − (F λ E)z = b “ ” − n n+1 − ⇒ (D λ E)z = (D F )z b − n n+1 − n − λ = cT z + α n+1 n+1

By changing the notation of λn and λn+1 this leads to

T λn = α + c zn (D λ E)z = (D F )z b − n n+1 − n −

5.5.2 Motivation of Davidson’s Algorithm

Convergence of Eigenvalues, Eigenvectors. Let V H1(Ω) be a Hilbert space and a( , ) : V V R V -koerziv. Let ⊂ · · × → Vn be spaces such that

h h lim inf u u V u Vn = 0 u V n→∞ || − || ∈ ∀ ∈ n o

81 Let e V , e = 0 and λ C such that ∈ 6 ∈ a(e, v)= λ ev dµ v V. (95) ∀ ∈ ZΩ

Let e V , λ C such that n ∈ n n ∈ a(e , v )= λ e v dµ v V . (96) n n n n n ∀ n ∈ n ZΩ Theorem 14. If λ is a single eigenvalue, then there is a constant c and a sequence (en, λn) such that

e e cd(e, V ) || − n||V ≤ n

Connection to the matrix eigenvalue problem Let A be a matrix which describes a( , ) with respect to an L2-orthogonal · · basis. Then (95) and (96), are equivalent to

A~e = λ~e (A ~e λ ~e ) ~v ~e V~ , v V~ n − n n ⊥ n n ∈ n ∀ n ∈ n λ is called Ritz value of A with Ritz vector ~e V~ . Furthermore λ is an n n ∈ n n eigenvalue of the matrix Bn = (bij), where bij = a(vi, vj) and (vi)i a basis of V~j. There is an eigenvector ξn of Bn with eigenvalue λn such that

i ~en = ξnvi. Xi

Davidson’s Idea: Choose the optimal eigenvector from the subspace V as a new approximate eigenvector. This is the Ritz vector. By increasing V one gets an approximation of the exact eigenvector.

5.5.3 The concept of the Jacobi-Davidson-Algorithm

Idea A: Compute the optimal eigenvector Ritz vector“ and Ritz value“ λ ” ” in the subspace Vk Idea B: Enlarge the subspace V V by the Idea of Jacobi“ k → k+1 ” orthogonal to the old Ritz vector“ by a Newton step on ” 1 1 A λ = 0“ ” z − z 82 Orthogonalize the new vector t with respect to Vk and build Vk+1.

In the Jacobi method we had 1 1 0 A λ = z − z 0 α cT A = b F Let us denote 1 1 α λ A λ = − = r 0 − 0 b the residual. In the Jacobi method, one has to compute

(F λ E)−1( b) − n − .

We have to describe this in suitable spaces.

Letu ˆ be an approximation of the eigenvalue θ. Let us rotate the coordinate system such that

1 0  .  ˆ=ˆu. .    0      Now,

 ∗.   .       ∗    corresponds to a space T , which is orthogonal tou ˆ. This means:

V = Cuˆ T and Cuˆ T ⊕ ⊥ Now we can describe “t˜ := (F λ E)−1( b)” in suitable spaces. We have − n − to ﬁnd a t T ,ǫ ˜C such that ∈ (A θE)t = b +ǫ ˜uˆ − − where b uˆ and r b Cuˆ. This is equivalent to t T , ǫ C, and ⊥ − ∈ ∈ ∈ (A θE)t = r + ǫuˆ (97) − −

83 This equation can approximatively be solved as follows.

Let M −1 be a preconditioner for A θE. − This means that M is an approximation of A θE. − Thus, instead of solving (97), we are looking for a tˆ T such that ∈ Mtˆ = r + ǫuˆ − This leads to tˆ= M −1r + M −1ǫuˆ. Since tˆ T , we obtain: − ∈ 0 = uMˆ −1r +uM ˆ −1ǫuˆ − ⇓ uˆ∗M −1r ǫ = uˆ∗M −1uˆ

Now, one can solve

tˆ = M −1( r + ǫuˆ). −

5.5.4 Jacobi-Davidson-Algorithm

Step 1. Start: Choose a non trivial start vector v. Calculate v1 = v/ v , w1 = Av1. ∗ k k h11 = v1w1. Set V1 = Rv1, W1 = Rw1, H1 = h11 . u = v1, θ = h11. Calculate v = w θu. 1 − Step 2. Iterate until convergence:

Step 3. Inner loop: For k = 1, ..., m 1 : −

Let M be an approximation of A θE. Calculate: • − uˆ∗M −1r ǫ = , t = M −1( r + ǫu). uˆ∗M −1uˆ −

Orthogonalize t with respect to V by Gram-Schmidt. • k This leads to the vector tortho. Extend Vk by t to obtain Vk+1.

84 Calculate w = Av and • k+1 k+1 extend Wk by wk+1 to obtain Wk+1.

∗ ∗ Calculate Vk+1wk+1 and vk+1Wk+1. • ∗ Then the whole matrix Hk+1 := Vk+1AVk+1 is computed. Calculate the largest eigenvalue of θ with eigenvector s of H • k+1 (where s = 1). k k Calculate the Ritz vector u := V s. • k+1 Calculateu ˆ := Au (this is Wk+1s) Calculate the residuum r :=u ˆ θu. − Test convergence. Stop if r is small enough.. • k k

Step 4. Restart: If r can not be reduced any more, then set: k k

Set V1 = Ru, W1 = Ruˆ, H1 = θ . Goto Step 3.