INF5620 in a nutshell

Study guide: Intro to Computing with Finite Dierence Methods Numerical methods for partial dierential equations (PDEs) How do we solve a PDE in practice and produce numbers? How do we trust the answer? Hans Petter Langtangen1,2 Approach: simplify, understand, generalize

Center for Biomedical Computing, Simula Research Laboratory1 After the course Department of Informatics, University of Oslo2 You see a PDE and can't wait to program a method and visualize a solution! Somebody asks if the solution is right and you can give a Oct 5, 2014 convincing answer.

The new ocial six-point course description More specic contents: nite dierence methods After having completed INF5620 you

can derive methods and implement them to solve frequently arising partial dierential equations (PDEs) from and mechanics. have a good understanding of nite dierence and nite ODEs element methods and how they are applied in linear and the wave equation utt = uxx in 1D, 2D, 3D nonlinear PDE problems. the diusion equation ut = uxx in 1D, 2D, 3D can identify numerical artifacts and perform mathematical write your own software from scratch analysis to understand and cure non-physical eects. understand how the methods work and why they fail can apply sophisticated programming techniques in Python, combined with Cython, C, C++, and Fortran code, to create modern, exible simulation programs. can construct verication tests and automate them. have experience with project hosting sites (GitHub), version control systems (Git), report writing (LATEX), and Python scripting for performing reproducible computational science.

More specic contents: nite element methods More specic contents: nonlinear and advanced problems

stationary diusion equations uxx = f in 1D Nonlinear PDEs time-dependent diusion and wave equations in 1D Newton and Picard iteration methods, nite dierences and PDEs in 2D and 3D by use of the FEniCS software elements perform hand-calculations, write your own software (1D) More advanced PDEs for uid ow and elasticity understand how the methods work and why they fail Parallel computing Philosophy: simplify, understand, generalize The exam

Oral exam Start with simplied ODE/PDE problems 6 problems (topics) are announced two weeks before the exam Learn to reason about the Work out a 20 min presentations (talks) for each problem Learn to implement, verify, and experiment At the exam: throw a die to pick your problem to be presented Understand the method, program, and results Aids: plots, computer programs Generalize the problem, method, and program Why? Very eective way of learning Sure? Excellent results over 15 years This is the power of ! When? Late december

Required software Assumed/ideal background

INF1100: Python programming, solution of ODEs Our software platform: Python (sometimes combined with Cython, Fortran, C, C++) Some experience with nite dierence methods Important Python packages: numpy, scipy, matplotlib, Some analytical and numerical knowledge of PDEs sympy, fenics, scitools, ... Much experience with calculus and Suggested installation: Run Ubuntu in a virtual machine Much experience with programming of mathematical problems Alternative: run a Vagrant machine Experience with mathematical modeling with PDEs (from physics, mechanics, geophysics, or ...)

Start-up example for the course Start-up example

What if you don't have this ideal background?

Students come to this course with very dierent backgrounds ODE problem First task: summarize assumed background knowledge by u0 = au, u(0) = I , t (0, T ], going through a simple example − ∈ Also in this example: where a > 0 is a constant. Some fundamental material on software implementation and software testing Everything we do is motivated by what we need as building blocks Material on analyzing numerical methods to understand why for solving PDEs! they can fail Applications to real-world problems What to learn in the start-up example; standard topics What to learn in the start-up example; programming topics

How to verify an implementation and automate verication through nose tests in Python How to think when constructing nite dierence methods, How to structure code in terms of functions, classes, and with special focus on the Forward Euler, Backward Euler, and modules Crank-Nicolson (midpoint) schemes How to work with Python concepts such as arrays, lists, How to formulate a computational algorithm and translate it dictionaries, lambda functions, functions in functions into Python code (closures), doctests, unit tests, command-line interfaces, How to make curve plots of the solutions graphical user interfaces How to compute numerical errors How to perform array computing and understand the How to compute convergence rates dierence from scalar computing How to conduct and automate large-scale numerical experiments How to generate scientic reports

What to learn in the start-up example; mathematical What to learn in the start-up example; generalizations analysis

How to uncover numerical artifacts in the computed solution Generalize the example to u0(t) = a(t)u(t) + b(t) How to analyze the numerical schemes mathematically to − Present additional methods for the general nonlinear ODE understand why artifacts occur u0 = f (u, t), which is either a scalar ODE or a system of ODEs How to derive mathematical expressions for various measures How to access professional packages for solving ODEs of the error in numerical methods, frequently by using the How our model equations like u au arises in a wide range sympy software for symbolic computation 0 = of phenomena in physics, biology, and− nance Introduce concepts such as nite dierence operators, mesh (grid), mesh functions, stability, truncation error, consistency, and convergence

Finite dierence methods Topics in the rst intro to the nite dierence method

The nite dierence method is the simplest method for solving dierential equations Contents Fast to learn, derive, and implement How to think about nite dierence discretization A very useful tool to know, even if you aim at using the nite Key concepts: element or the nite volume method mesh mesh function nite dierence approximations The Forward Euler, Backward Euler, and Crank-Nicolson methods Finite dierence operator notation How to derive an algorithm and implement it in Python How to test the implementation A basic model for exponential decay The ODE model has a range of applications in many elds

The world's simplest (?) ODE:

u0(t) = au(t), u(0) = I , t (0, T ] − ∈ Growth and decay of populations (cells, animals, human) Growth and decay of a fortune Observation Radioactive decay We can learn a lot about numerical methods, computer implementation, program testing, and real applications of these Cooling/heating of an object tools by using this very simple ODE as example. The teaching Pressure variation in the atmosphere principle is to keep the math as simple as possible while learning Vertical motion of a body in water/air computer tools. Time-discretization of diusion PDEs by Fourier techniques

See the text for details.

The ODE problem has a continuous and discrete version The steps in the nite dierence method

Continuous problem. Solving a dierential equation by a nite dierence method consists u0 = au, t (0, T ], u(0) = I (1) − ∈ of four steps: (varies with a continuous t) 1 discretizing the domain, Discrete problem. Numerical methods applied to the continuous 2 fullling the equation at discrete time points, problem turns it into a discrete problem 3 replacing by nite dierences,

4 formulating a recursive algorithm. n+1 n n u = constu , n = 0, 1,... Nt 1, u = I (2) − (varies with discrete mesh points tn)

Step 1: Discretizing the domain Step 1: Discretizing the domain

The time domain [0, T ] is represented by a mesh: a nite number of Nt + 1 points n u is a mesh function, dened at the mesh points tn, n = 0,..., Nt only.

0 = t0 < t1 < t2 < < tNt 1 < tNt = T ··· −

We seek the solution u at the mesh points: u(tn), n = 1, 2,..., Nt . Note: u0 is known as I . Notational short-form for the numerical approximation to n u(tn): u In the dierential equation: u is the exact solution In the numerical method and implementation: un is the numerical approximation, ue(t) is the exact solution What about a mesh function between the mesh points? Step 2: Fullling the equation at discrete time points

Can extend the mesh function to yield values between mesh points by linear interpolation:

un+1 un u t un t t (3) ( ) + t − t ( n) ≈ n+1 n − The ODE holds for all t (0, T ] (innite no of points) − ∈ Idea: let the ODE be valid at the mesh points only (nite no of points)

u0(tn) = au(tn), n = 1,..., Nt (4) −

Step 3: Replacing derivatives by nite dierences Step 3: Replacing derivatives by nite dierences

Now it is time for the nite dierence approximations of derivatives:

n 1 n u + u Inserting the nite dierence approximation in u0(tn) − (5) ≈ tn 1 tn + − u(t) u0(tn) = au(tn) − gives

n+1 n u u n − = au , n = 0, 1,..., Nt 1 (6) tn 1 tn − − + − (Known as discrete equation, or discrete problem, or nite dierence method/scheme)

forward

tn 1 tn tn +1 −

Step 4: Formulating a recursive algorithm Let us apply the scheme by hand

How can we actually compute the un values? Assume constant time spacing: ∆t = tn+1 tn = const −

given u0 = I compute u1 from u0 u0 = I , 2 1 0 0 compute u from u u1 = u a∆tu = I (1 a∆t), 3 2 − 2 − compute u from u (and so forth) u2 = I (1 a∆t) , 3 − 3 u = I (1 a∆t) , In general: we have un and seek un+1 . − . The Forward Euler scheme Nt Nt n+1 u = I (1 a∆t) Solve wrt u to get the computational formula: −

n+1 n n Ooops - we can nd the numerical solution by hand (in this simple u = u a(tn+1 tn)u (7) − − example)! No need for a computer (yet)... A backward dierence The Backward Euler scheme Here is another nite dierence approximation to the (backward dierence):

un un 1 Inserting the nite dierence approximation in u t au t u t − (8) 0( n) = ( n) 0( n) − yields the Backward Euler (BE) scheme: − ≈ tn tn 1 − − u(t) n n 1 u u n − − = au (9) tn tn 1 − backward − − Solve with respect to the unknown un+1:

n 1 1 n u + = u (10) 1 + a(tn 1 tn) + −

tn 1 tn tn +1 −

A centered dierence The Crank-Nicolson scheme; ideas

Idea 1: let the ODE hold at tn 1 Centered dierences are better approximations than forward or + 2 backward dierences. u(t) u0(tn 1 ) = au(tn 1 ) + 2 − + 2

Idea 2: approximate u (tn 1 by a centered dierence 0 + 2

un+1 un u0(tn 1 ) − (11) + 2 ≈ tn 1 tn + − n Problem: u(tn 1 ) is not dened, only u = u(tn) and + 2 centered n+1 u = u(tn+1) Solution:

tn tn +1 tn +1 2 1 n n+1 u(tn 1 ) (u + u ) + 2 ≈ 2

The Crank-Nicolson scheme; result The unifying θ-rule

The Forward Euler, Backward Euler, and Crank-Nicolson schemes can be formulated as one scheme with a varying parameter θ: Result:

n+1 n u u n 1 n n+1 n a u + 1 u (14) u u 1 n n 1 t − t = (θ + ( θ) ) − = a (u + u + ) (12) n+1 n − − tn 1 tn − 2 − + − 0: Forward Euler Solve wrt to un+1: θ = θ = 1: Backward Euler 1 1 2: Crank-Nicolson 1 a(tn 1 tn) θ = / un+1 2 + un (13) = − 1 − We may alternatively choose any 0 1 . 1 + a(tn 1 tn) θ [ , ] 2 + − ∈ This is a Crank-Nicolson (CN) scheme or a midpoint or centered un is known, solve for un+1: scheme.

n 1 1 (1 θ)a(tn 1 tn) u + = − − + − (15) 1 + θa(tn 1 tn) + − Constant time step Test the understanding!

Very common assumption (not important, but exclusively used for Derive Forward Euler, Backward Euler, and Crank-Nicolson schemes simplicity hereafter): constant time step tn 1 tn ∆t for Newton's law of cooling: + − ≡ Summary of schemes for constant time step T 0 = k(T Ts ), T (0) = T0, t (0, tend] − − ∈ n 1 n u + = (1 a∆t)u Forward Euler (16) − Physical quantities: n 1 1 n u + = u Backward Euler (17) 1 + a∆t T t : temperature of an object at time t 1 ( ) 1 a∆t un+1 2 un Crank-Nicolson (18) k: parameter expressing heat loss to the surroundings = − 1 1 + a∆t 2 Ts : temperature of the surroundings n 1 1 (1 θ)a∆t n u + = − − u The θ rule (19) T0: initial temperature 1 + θa∆t −

Compact operator notation for nite dierences Specic notation for dierence operators

Forward dierence:

un+1 un d + n Finite dierence formulas can be tedious to write and D u u tn (21) [ t ] = −t dt ( ) read/understand ∆ ≈ Centered dierence: Handy tool: nite dierence operator notation

Advantage: communicates the nature of the dierence in a 1 1 n+ n n u 2 u − 2 d compact way [Dt u] = − u(tn), (22) ∆t ≈ dt n [Dt−u = au] (20) Backward dierence: − n n 1 n u u − d [Dt−u] = − u(tn) (23) ∆t ≈ dt

The Backward Euler scheme with operator notation The Forward Euler scheme with operator notation

n n [Dt−u] = au − + n Common to put the whole equation inside square brackets: [Dt u = au] (25) −

n [Dt−u = au] (24) − The Crank-Nicolson scheme with operator notation Implementation

Model: Introduce an averaging operator: u0(t) = au(t), t (0, T ], u(0) = I − ∈

1 1 1 t n n n+ Numerical method: [u ] = (u − 2 + u 2 ) u(tn) (26) 2 ≈

n 1 1 (1 θ)a∆t n The Crank-Nicolson scheme can then be written as u + = − − u 1 + θa∆t

1 for 0 1 . Note t n+ θ [ , ] [Dt u = au ] 2 (27) ∈ − θ = 0 gives Forward Euler Test: use the denitions and write out the above formula to see 1 gives Backward Euler that it really is the Crank-Nicolson scheme! θ = θ = 1/2 gives Crank-Nicolson

Requirements of a program Tools to learn

n Basic Python programming Compute the numerical solution u , n = 1, 2,..., Nt at Array computing with numpy Display the numerical and exact solution ue(t) = e− Bring evidence to a correct implementation (verication) Plotting with matplotlib.pyplot and scitools Compare the numerical and the exact solution in a plot File writing and reading n computes the error ue(tn) u − computes the convergence rate of the numerical scheme Notice reads its input data from the command line All programs are in the directory src/decay.

Why implement in Python? Why implement in Python?

Python has a rich set of modules for scientic computing, and its popularity in scientic computing is rapidly growing. Python has a very clean, readable syntax (often known as Python was made for being combined with compiled languages "executable pseudo-code"). (C, C++, Fortran) to reuse existing numerical software and to reach high computational performance of new Python code is very similar to MATLAB code (and MATLAB implementations. has a particularly widespread use for scientic computing). Python has extensive support for administrative task needed Python is a full-edged, very powerful programming language. when doing large-scale computational investigations. Python is similar to, but much simpler to work with and Python has extensive support for graphics (visualization, user results in more reliable code than C++. interfaces, web applications). FEniCS, a very powerful tool for solving PDEs by the nite element method, is most human-ecient to operate from Python. Algorithm Translation to Python function

from numpy import* def solver(I, a, T, dt, theta): """Solve u'=-a*u, u(0)=I, for t in (0,T] with steps of dt.""" Nt= int(T/dt) # no of time intervals T= Nt*dt # adjust T to fit time step dt n Store u , n = 0, 1,..., Nt in an array u. u= zeros(Nt+1) # array of u[n] values t= linspace(0, T, Nt+1) # time mesh Algorithm: 1 initialize u0 u[0]=I # assign initial condition forn in range(0, Nt): # n=0,1,...,Nt-1 2 for t = tn, n = 1, 2,..., Nt : compute un using the θ-rule u[n+1]=(1-(1-theta)*a*dt)/(1+ theta*dt*a)*u[n] formula return u, t

Note about the for loop: range(0, Nt, s) generates all integers from 0 to Nt in steps of s (default 1), but not including Nt (!). Sample call:

u, t= solver(I=1, a=2,T=8, dt=0.8, theta=1)

Integer division Doc strings

First string after the function heading Python applies integer division: 1/2 is 0, while 1./2 or 1.0/2 or Used for documenting the function 1/2. or 1/2.0 or 1.0/2.0 all give 0.5. Automatic documentation tools can make fancy manuals for you A safer solver function (dt = float(dt) - guarantee oat): Can be used for automatic testing from numpy import* def solver(I, a, T, dt, theta): """Solve u'=-a*u, u(0)=I, for t in (0,T] with steps of dt.""" dt= float(dt) # avoid integer division def solver(I, a, T, dt, theta): Nt= int(round(T/dt)) # no of time intervals """ T= Nt*dt # adjust T to fit time step dt Solve u= zeros(Nt+1) # array of u[n] values t= linspace(0, T, Nt+1) # time mesh u'(t) = -a*u(t), u[0]=I # assign initial condition with initial condition u(0)=I, for t in the time interval forn in range(0, Nt): # n=0,1,...,Nt-1 (0,T]. The time interval is divided into time steps of u[n+1]=(1-(1-theta)*a*dt)/(1+ theta*dt*a)*u[n] length dt. return u, t theta=1 corresponds to the Backward Euler scheme, theta=0 to the Forward Euler scheme, and theta=0.5 to the Crank- Nicolson method. """ ...

Formatting of numbers Running the program

How to run the program decay_v1.py:

Can control formatting of reals and integers through the printf Terminal> python decay_v1.py format: Can also run it as "normal" Unix programs: ./decay_v1.py if the print 't=%6.3f u=%g'% (t[i], u[i]) rst line is

Or the alternative format string syntax: `#!/usr/bin/env python`

print 't={t:6.3f} u={u:g}'.format(t=t[i], u=u[i]) Then

Terminal> chmod a+rx decay_v1.py Terminal> ./decay_v1.py Plotting the solution Comparing with the exact solution

at Python function for the exact solution ue(t) = Ie− :

def exact_solution(t, I, a): returnI*exp(-a*t)

Basic syntax: Quick plotting:

from matplotlib.pyplot import* u_e= exact_solution(t, I, a) plot(t, u, t, u_e) plot(t, u) show() n Problem: ue(t) applies the same mesh as u and looks as a Can (and should!) add labels on axes, title, legends. piecewise linear function.

Remedy: Introduce a very ne mesh for ue.

t_e= linspace(0, T, 1001) # fine mesh u_e= exact_solution(t_e, I, a)

plot(t_e, u_e, 'b-', # blue line for u_e t, u, 'r--o') # red dashes w/circles

Add legends, axes labels, title, and wrap in a function Plotting with SciTools

from matplotlib.pyplot import* SciTools provides a unied plotting interface (Easyviz) to many def plot_numerical_and_exact(theta, I, a, T, dt): dierent plotting packages: Matplotlib, Gnuplot, Grace, VTK, """Compare the numerical and exact solution in a plot.""" u, t= solver(I=I, a=a, T=T, dt=dt, theta=theta) OpenDX, ...

t_e= linspace(0, T, 1001) # fine mesh for u_e Can use Matplotlib (MATLAB-like) syntax, or a more compact u_e= exact_solution(t_e, I, a) plot function syntax: plot(t, u, 'r--o', # red dashes w/circles t_e, u_e, 'b-') # blue line for exact sol. from scitools.std import* legend(['numerical', 'exact']) xlabel('t') plot(t, u, 'r--o', # red dashes w/circles ylabel('u') t_e, u_e, 'b-', # blue line for exact sol. title('theta=%g, dt=%g'% (theta, dt)) legend=['numerical', 'exact'], savefig('plot_%s_%g.png'% (theta, dt)) xlabel='t', ylabel='u', title='theta=%g, dt=%g'% (theta, dt), Complete code in decay_v2.py savefig='%s_%g.png'% (theta2name[theta], dt), show=True) theta=1, dt=0.8 1.0 numerical Change backend (plotting engine, Matplotlib by default): exact 0.8 Terminal> python decay_plot_st.py --SCITOOLS_easyviz_backend gnuplot Terminal> python decay_plot_st.py --SCITOOLS_easyviz_backend grace

0.6 u

0.4

Verifying the0.2 implementation Simplest method: run a few algorithmic steps by hand

0.0 0 1 2 3 4 5 6 7 8 t Use a calculator (I = 0.1, θ = 0.8, ∆t = 0.8):

1 (1 θ)a∆t Verication = bring evidence that the program works A − − = 0.298245614035 ≡ 1 + θa∆t Find suitable test problems Make function for each test problem 1 Later: put the verication tests in a professional testing u = AI = 0.0298245614035, framework 2 1 u = Au = 0.00889504462912, 3 2 u = Au = 0.00265290804728

See the function verify_three_steps in decay_verf1.py. Comparison with an exact discrete solution Making a test based on an exact discrete solution

Best verication The exact discrete solution is Compare computed numerical solution with a closed-form exact discrete solution (if possible). n n u = IA (28)

Dene Question 1 (1 θ)a∆t A = − − Understand what n in un and in An means! 1 + θa∆t Repeated use of the θ-rule: Test if 0 u = I , 1 0 n 15 u Au AI max u ue(tn) <  10− = = n | − | ∼ n n n 1 n u = A u = A I − Implementation in decay_verf2.py.

Test the understanding! Computing the numerical error as a mesh function

n n Task: compute the numerical error e = ue(tn) u − at Exact solution: ue(t) = Ie− , implemented as

def exact_solution(t, I, a): returnI*exp(-a*t) Make a program for solving Newton's law of cooling Compute en by

T 0 = k(T Ts ), T (0) = T0, t (0, tend] − − ∈ u, t= solver(I, a, T, dt, theta) # Numerical solution with the Forward Euler, Backward Euler, and Crank-Nicolson u_e= exact_solution(t, I, a) e= u_e-u schemes (or a θ scheme). Verify the implementation. Array arithmetics - we compute on entire arrays! exact_solution(t, I, a) works with t as array Must have exp from numpy (not math) e = u_e - u: array subtraction Array arithmetics gives shorter and much faster code

Computing the norm of the error Norms of mesh functions

n Problem: f = f (tn) is a mesh function and hence not dened en is a mesh function for all t. How to integrate f n? Usually we want one number for the error Idea: Apply a rule, using only the mesh Use a norm of en points of the mesh function.

The Trapezoidal rule: Norms of a function f (t):

1 2 Nt 1 / n 1 0 2 1 N 2 − n 2 T 1/2 f = ∆t (f ) + (f t ) + (f ) 2 || || 2 2 f L2 = f (t) dt (29) n=1 !! || || 0 X Z  T Common simplication yields the L2 norm of a mesh function: f L1 = f (t) dt (30) || || 0 | | Z f L max f t (31) N 1/2 ∞ = ( ) t || || t [0,T ] | | n n 2 ∈ f t f `2 = ∆ ( ) || || n 0 ! X= Implementation of the norm of the error Comment on array vs scalar computation Scalar computing of E = sqrt(dt*sum(e**2)):

m= len(u) # length of u array (alt: u.size) u_e= zeros(m) t=0 fori in range(m): u_e[i]= exact_solution(t, a, I) Nt t=t+ dt n n 2 e= zeros(m) E = e 2 = ∆t (e ) || ||` v fori in range(m): u n=0 e[i]= u_e[i]- u[i] u X s=0 # summation variable t fori in range(m): Python w/array arithmetics: s=s+ e[i]**2 error= sqrt(dt*s) e= exact_solution(t)-u E= sqrt(dt*sum(e**2)) Obviously, scalar computing

takes more code is less readable runs much slower

Rule Compute on entire arrays (when possible)!

Memory-saving implementation Memory-saving solver function

def solver_memsave(I, a, T, dt, theta, filename='sol.dat'): """ n Solve u'=-a*u, u(0)=I, for t in (0,T] with steps of dt. Note 1: we store the entire array u, i.e., u for n = 0, 1,..., Nt Minimum use of memory. The solution is stored in a file Note 2: the formula for un+1 needs un only, not un 1, un 2, ... (with name filename) for later plotting. − − """ No need to store more than un+1 and un dt= float(dt) # avoid integer division Nt= int(round(T/dt)) # no of intervals Extremely important when solving PDEs outfile= open(filename, 'w') No practical importance here (much memory available) # u: time level n+1, u_1: time level n t=0 But let's illustrate how to do save memory! u_1=I n 1 n outfile.write('%.16E %.16E\n'% (t, u_1)) Idea 1: store u + in u, u in u_1 (float) forn in range(1, Nt+1): u=(1-(1-theta)*a*dt)/(1+ theta*dt*a)*u_1 Idea 2: store u in a le, read le later for plotting u_1=u t += dt outfile.write('%.16E %.16E\n'% (t, u)) outfile.close() return u, t

Reading computed data from le Usage of memory-saving code

def read_file(filename='sol.dat'): infile= open(filename, 'r') u= []; t=[] for line in infile: words= line.split() if len(words) !=2: def explore(I, a, T, dt, theta=0.5, makeplot=True): print 'Found more than two numbers on a line!', words filename= 'u.dat' sys.exit(1) # abort u, t= solver_memsave(I, a, T, dt, theta, filename) t.append(float(words[0])) u.append(float(words[1])) t, u= read_file(filename) return np.array(t), np.array(u) u_e= exact_solution(t, I, a) e= u_e-u E= np.sqrt(dt*np.sum(e**2)) Simpler code with numpy functionality for reading/writing tabular if makeplot: data: plt.figure() ... def read_file_numpy(filename='sol.dat'): data= np.loadtxt(filename) Complete program: decay_memsave.py. t= data[:,0] u= data[:,1] return t, u

Similar function np.savetxt, but then we need all un and tn values in a two-dimensional array (which we try to prevent now!). Analysis of nite dierence equations Encouraging numerical solutions I = 1, a = 2, θ = 1, 0.5, 0, ∆t = 1.25, 0.75, 0.5, 0.1.

Method: theta-rule, theta=1, dt=1.25 Method: theta-rule, theta=1, dt=0.75 1.0 1.0 numerical numerical exact exact

0.8 0.8 Model: 0.6 0.6

u0(t) = au(t), u(0) = I (32) u u − 0.4 0.4

Method: 0.2 0.2 1 1 a t un+1 ( θ) ∆ un (33) = 0.0 0.0 − − 0 1 2 3 4 5 0 1 2 3 4 5 6 1 + θa∆t t t

Method: theta-rule, theta=1, dt=0.5 Method: theta-rule, theta=1, dt=0.1 1.0 1.0 Problem setting numerical numerical exact exact How good is this method? Is it safe to use it? 0.8 0.8

0.6 0.6 u u

0.4 0.4

0.2 0.2

0.0 0.0 0 1 2 3 4 5 0 1 2 3 4 5 t t

Discouraging numerical solutions; Crank-Nicolson Discouraging numerical solutions; Forward Euler

Method: theta-rule, theta=0.5, dt=1.25 Method: theta-rule, theta=0.5, dt=0.75 Method: theta-rule, theta=0, dt=1.25 Method: theta-rule, theta=0, dt=0.75 1.0 1.0 6 numerical numerical numerical numerical exact exact exact 1.0 exact 0.8 0.8 4

0.6 0.6 2 0.5

u 0.4 u u u

0.4 0 0.2 0.0

0.2 2 0.0

0.5 0.2 0.0 4 0 1 2 3 4 5 0 1 2 3 4 5 6 0 1 2 3 4 5 0 1 2 3 4 5 6 t t t t

Method: theta-rule, theta=0.5, dt=0.5 Method: theta-rule, theta=0.5, dt=0.1 Method: theta-rule, theta=0, dt=0.5 Method: theta-rule, theta=0, dt=0.1 1.0 1.0 1.0 1.0 numerical numerical numerical numerical exact exact exact exact

0.8 0.8 0.8 0.8

0.6 0.6 0.6 0.6 u u u u

0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2

0.0 0.0 0.0 0.0 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 t t t t

Summary of observations Problem setting

The characteristics of the displayed curves can be summarized as Goal follows: We ask the question The Backward Euler scheme always gives a monotone solution, Under what circumstances, i.e., values of the input data I , a, lying above the exact curve. and ∆t will the Forward Euler and Crank-Nicolson schemes The Crank-Nicolson scheme gives the most accurate results, result in undesired oscillatory solutions? but for ∆t = 1.25 the solution oscillates. Techniques of investigation: The Forward Euler scheme gives a growing, oscillating solution Numerical experiments for ∆t = 1.25; a decaying, oscillating solution for ∆t = 0.75; Mathematical analysis a strange solution un = 0 for n 1 when ∆t = 0.5; and a solution seemingly as accurate as≥ the one by the Backward Another question to be raised is Euler scheme for t 0 1, but the curve lies below the exact ∆ = . How does t impact the error in the numerical solution? solution. ∆ Experimental investigation of oscillatory solutions Exact numerical solution The solution is oscillatory if n n 1 u > u −

Starting with u0 = I , the simple recursion (33) can be applied repeatedly n times, with the result that

n n 1 (1 θ)a∆t u = IA , A = − − (34) 1 + θa∆t Such an exact discrete solution is unusual, but very handy for analysis.

Seems that a∆t < 1 for FE and 2 for CN.

Stability Computation of stability in this problem

A < 0 if

Since un An, 1 (1 θ)a∆t − − < 0 ∼ 1 + θa∆t n A < 0 gives a factor ( 1) and oscillatory solutions To avoid oscillatory solutions we must have A > 0 and − A > 1 gives growing solutions | | Recall: the exact solution is monotone and decaying 1 ∆t < (35) If these qualitative properties are not met, we say that the (1 θ)a − numerical solution is unstable Always fullled for Backward Euler ∆t 1/a for Forward Euler ≤ ∆t 2/a for Crank-Nicolson ≤

Computation of stability in this problem Explanation of problems with Forward Euler

Method: theta-rule, theta=0, dt=1.25 Method: theta-rule, theta=0, dt=0.75 6 numerical numerical A 1 means 1 A 1 exact 1.0 exact | | ≤ − ≤ ≤ 4

1 (1 θ)a∆t 2 0.5 1 − − 1 (36) u u − ≤ 1 + θa∆t ≤ 0 0.0 1 is the critical limit: − 2 0.5 4 0 1 2 3 4 5 0 1 2 3 4 5 6 t t 2 1 Method: theta-rule, theta=0, dt=0.5 Method: theta-rule, theta=0, dt=0.1 ∆t , θ < 1.0 1.0 1 2 a 2 numerical numerical ≤ ( θ) exact exact −2 1 0.8 0.8 ∆t , θ > ≥ (1 2θ)a 2 0.6 0.6

− u u

0.4 0.4 Always fullled for Backward Euler and Crank-Nicolson 0.2 0.2 ∆t 2/a for Forward Euler 0.0 0.0 0 1 2 3 4 5 0 1 2 3 4 5 ≤ t t

a∆t = 2 1.25 = 2.5 and A = 1.5: oscillations and growth · − a∆t = 2 0.75 = 1.5 and A = 0.5: oscillations and decay · n − ∆t = 0.5 and A = 0: u = 0 for n > 0 Smaller Deltat: qualitatively correct solution Explanation of problems with Crank-Nicolson Summary of stability

Method: theta-rule, theta=0.5, dt=1.25 Method: theta-rule, theta=0.5, dt=0.75 1.0 1.0 numerical numerical exact exact 0.8 0.8

0.6 0.6

u 0.4 u 0.4 1 Forward Euler is conditionally stable 0.2 t 2 a for avoiding growth 0.2 ∆ < / 0.0 ∆t 1/a for avoiding oscillations 0.2 0.0 ≤ 0 1 2 3 4 5 0 1 2 3 4 5 6 t t 2 The Crank-Nicolson is unconditionally stable wrt growth and

Method: theta-rule, theta=0.5, dt=0.5 Method: theta-rule, theta=0.5, dt=0.1 conditionally stable wrt oscillations 1.0 1.0 numerical numerical exact exact ∆t < 2/a for avoiding oscillations

0.8 0.8 3 Backward Euler is unconditionally stable

0.6 0.6 u u

0.4 0.4

0.2 0.2

0.0 0.0 0 1 2 3 4 5 0 1 2 3 4 5 t t

∆t = 1.25 and A = 0.25: oscillatory solution Never any growing solution−

Comparing amplication factors Plot of amplication factors

Amplification factors 1 un+1 is an amplication A of un:

0.5 n 1 n 1 (1 θ)a∆t u + = Au , A = − − 1 + θa∆t 0

The exact solution is also an amplication: A -0.5

-1 a∆t u(tn 1) = Aeu(tn), Ae = e− + exact -1.5 FE BE A possible measure of accuracy: Ae A CN − -2 0 0.5 1 1.5 2 2.5 3 a*dt

Series expansion of amplication factors Error in amplication factors

To investigate Ae A mathematically, we can Taylor expand the − expression, using p = a∆t as variable.

>>> from sympy import* >>> # Create p as a mathematical symbol with name 'p' >>>p= Symbol('p') >>> # Create a mathematical expression with p Focus: the error measure A Ae as function of ∆t (recall that − >>> A_e= exp(-p) p = a∆t): >>> >>> # Find the first 6 terms of the of A_e >>> A_e.series(p,0,6) 1+(1/2)*p**2-p-1/6*p**3-1/120*p**5+(1/24)*p**4+ O(p**6) (∆t2), Forward and Backward Euler, >>> theta= Symbol('theta') A Ae = O 3 (37) − (∆t ), Crank-Nicolson >>>A=(1-(1-theta)*p)/(1+theta*p)  >>>FE= A_e.series(p,0,4)-A.subs(theta,0).series(p,0,4) O >>>BE= A_e.series(p,0,4)-A.subs(theta,1).series(p,0,4) >>> half= Rational(1,2) # exact fraction 1/2 >>>CN= A_e.series(p,0,4)-A.subs(theta, half).series(p,0,4) >>>FE (1/2)*p**2-1/6*p**3+ O(p**4) >>>BE -1/2*p**2+(5/6)*p**3+ O(p**4) >>>CN (1/12)*p**3+ O(p**4) The fraction of numerical and exact amplication factors The true/global error at a point

Focus: the error measure 1 A/Ae as function of p = a∆t: − >>>FE=1-(A.subs(theta,0)/A_e).series(p,0,4) The error in A reects the local error when going from one >>>BE=1-(A.subs(theta,1)/A_e).series(p,0,4) >>>CN=1-(A.subs(theta, half)/A_e).series(p,0,4) time step to the next >>>FE (1/2)*p**2+(1/3)*p**3+ O(p**4) What is the global (true) error at tn? >>>BE n n atn n e = ue(tn) u = Ie IA -1/2*p**2+(1/3)*p**3+ O(p**4) − − − >>>CN Taylor series expansions of en simplify the expression (1/12)*p**3+ O(p**4)

Same leading-order terms as for the error measure A Ae. −

Computing the global error at a point Convergence

>>>n= Symbol('n') >>> u_e= exp(-p*n) # I=1 >>> u_n=A**n # I=1 >>>FE= u_e.series(p,0,4)- u_n.subs(theta,0).series(p,0,4) >>>BE= u_e.series(p,0,4)- u_n.subs(theta,1).series(p,0,4) >>>CN= u_e.series(p,0,4)- u_n.subs(theta, half).series(p,0,4) >>>FE n (1/2)*n*p**2-1/2*n**2*p**3+(1/3)*n*p**3+ O(p**4) The numerical scheme is convergent if the global error e 0 as r → >>>BE ∆t 0. If the error has a leading order term ∆t , the convergence (1/2)*n**2*p**3-1/2*n*p**2+(1/3)*n*p**3+ O(p**4) → >>>CN rate is of order r. (1/12)*n*p**3+ O(p**4)

Substitute n by t/∆t:

1 2 Forward and Backward Euler: leading order term 2 ta ∆t 1 3 2 Crank-Nicolson: leading order term 12 ta ∆t

Integrated errors Truncation error Focus: norm of the numerical error

How good is the discrete equation? Nt n n 2 e 2 = ∆t (ue(tn) u ) Possible answer: see how well ue ts the discrete equation || ||` v − u n=0 u X n t [Dt u = au] Forward and Backward Euler: − i.e.,

3 n 1 T 2 e 2 a t n+1 n ` = 4 3 ∆ u u n || || r − = au ∆t − Crank-Nicolson: Insert ue (which does not in general fulll this equation):

1 T 3 u t u t en a3 t2 e( n+1) e( n) n `2 = 12 3 ∆ − + aue(tn) = R = 0 (38) || || r ∆t 6 Summary of errors Analysis of both the pointwise and the time-integrated true errors: 1st order for Forward and Backward Euler 2nd order for Crank-Nicolson Computation of the truncation error The truncation error for other schemes

The residual Rn is the truncation error. How does Rn vary with ∆t?

Tool: Taylor expand ue around the point where the ODE is sampled Backward Euler: (here tn)

n 1 1 R ue00(tn)∆t 2 ≈ −2 ue(tn 1) = ue(tn) + ue0 (tn)∆t + ue00(tn)∆t + + 2 ··· Inserting this Taylor series in (38) gives Crank-Nicolson:

n 1 1 2 n 1 R + 2 u t 1 t R u t u t t au t e000( n+ )∆ = e0 ( n) + 2 e00( n)∆ + ... + e( n) ≈ 24 2

Now, ue solves the ODE ue = aue, and then 0 −

n 1 R ue00(tn)∆t ≈ 2 This is a mathematical expression for the truncation error.

Consistency, stability, and convergence Extension to a variable coecient; Forward and Backward Euler

Truncation error measures the residual in the dierence equations. The scheme is consistent if the truncation error goes to 0 as ∆t 0. Importance: the dierence equations u t a t u t t 0 T u 0 I (39) → 0( ) = ( ) ( ), ( , ], ( ) = approaches the dierential equation as ∆t 0. − ∈ → Stability means that the numerical solution exhibits the same The Forward Euler scheme: qualitative properties as the exact solution. Here: monotone, decaying function. n+1 n u u n − = a(tn)u (40) Convergence implies that the true (global) error ∆t − n n e = ue(tn) u 0 as ∆t 0. This is really what we want! − → → The Backward Euler scheme:

The Lax equivalence theorem for linear dierential equations: n n 1 u u − n consistency + stability is equivalent with convergence. − = a(tn)u (41) ∆t − (Consistency and stability is in most problems much easier to establish than convergence.)

Extension to a variable coecient; Crank-Nicolson Extension to a variable coecient; θ-rule

Eevaluting a(tn 1 ) and using an average for u: + 2 The θ-rule unies the three mentioned schemes,

un+1 un 1 n n+1 − = a(tn 1 ) (u + u ) (42) ∆t − + 2 2 un+1 un n n+1 − = a((1 θ)tn + θtn 1)((1 θ)u + θu ) (44) ∆t + Using an average for a and u: − − − or, n 1 n un+1 un u + u 1 n n+1 a t un a t un+1 (43) = (1 θ)a(tn)u θa(tn 1)u (45) − = ( ( n) + ( n+1) ) −t + ∆t −2 ∆ − − − Extension to a variable coecient; operator notation Extension to a source term

u0(t) = a(t)u(t) + b(t), t (0, T ], u(0) = I (46) + n − ∈ [Dt u = au] , − n [Dt−u = au] , − t n 1 D u au + 2 + n [ t = ] , [Dt u = au + b] , − 1 t n+ − n [Dt u = au ] 2 [Dt−u = au + b] , − − 1 t n+ [Dt u = au + b] 2 , − t 1 n+ [Dt u = au + b ] 2 −

Implementation of the generalized model problem Implementations of variable coecients; functions

n+1 n n n+1 n n+1 1 u = ((1 ∆t(1 θ)a )u +∆t(θb +(1 θ)b ))(1+∆tθa )− − − − (47)

Implementation where a(t) and b(t) are given as Python functions (see le decay_vc.py): Plain functions:

def solver(I, a, b, T, dt, theta): defa(t): """ return a_0 ift< tp elsek*a_0 Solve u'=-a(t)*u + b(t), u(0)=I, for t in (0,T] with steps of dt. defb(t): a and b are Python functions of t. return1 """ dt= float(dt) # avoid integer division Nt= int(round(T/dt)) # no of time intervals T= Nt*dt # adjust T to fit time step dt u= zeros(Nt+1) # array of u[n] values t= linspace(0, T, Nt+1) # time mesh

u[0]=I # assign initial condition forn in range(0, Nt): # n=0,1,...,Nt-1 u[n+1]=((1- dt*(1-theta)*a(t[n]))*u[n]+\ dt*(theta*b(t[n+1])+(1-theta)*b(t[n])))/\ (1+ dt*theta*a(t[n+1])) return u, t

Implementations of variable coecients; classes Implementations of variable coecients; lambda function

Quick writing: a one-liner lambda function

a= lambda t: a_0 ift< tp elsek*a_0

Better implementation: class with the parameters a0, tp, and k as In general, attributes and a special method __call__ for evaluating a(t): f= lambda arg1, arg2, ...: expressin classA: def __init__(self, a0=1, k=2): is equivalent to self.a0, self.k= a0, k deff(arg1, arg2, ...): def __call__(self, t): return expression return self.a0 ift< self.tp else self.k*self.a0 One can use lambda functions directly in calls: a= A(a0=2, k=1) # a behaves as a function a(t) u, t= solver(1, lambda t:1, lambda t:1, T, dt, theta)

for a problem u = u + 1, u(0) = 1. 0 − A lambda function can appear anywhere where a variable can appear. Verication via trivial solutions Verication via trivial solutions; test function

def test_constant_solution(): """ Test problem where u=u_const is the exact solution, to be reproduced (to machine precision) by any relevant method. Start debugging of a new code with trying a problem where """ def exact_solution(t): u = const = 0. return u_const 6 Choose u C (a constant). Choose any a t and set = ( ) defa(t): b = a(t)C and I = C. return 2.5*(1+t**3) # can be arbitrary "All" numerical methods will reproduce u =const exactly defb(t): (machine precision). return a(t)*u_const Often u = C eases debugging. u_const= 2.15 theta= 0.4; I= u_const; dt=4 In this example: any error in the formula for un+1 make u = C! Nt=4 # enough with a few steps 6 u, t= solver(I=I, a=a, b=b, T=Nt*dt, dt=dt, theta=theta) printu u_e= exact_solution(t) difference= abs(u_e- u).max() # max deviation tol= 1E-14 assert difference< tol

Verication via manufactured solutions Linear manufactured solution n u = ctn + I fullls the discrete equations! First, Choose any formula for u(t). t t Fit I , a(t), and b(t) in u = au + b, u(0) = I , to make the + n n+1 n 0 [Dt t] = − = 1, (48) chosen formula a solution of− the ODE problem. ∆t n tn tn 1 Then we can always have an analytical solution (!). [Dt−t] = − − = 1, (49) ∆t Ideal for verication: testing convergence rates. 1 1 tn 1 tn 1 n t n t n + 2 2 ( + 2 )∆ ( 2 )∆ Called the method of manufactured solutions (MMS) [Dt t] = − − = − − = 1 (50) ∆t ∆t Special case: u linear in t, because all sound numerical methods will reproduce a linear u exactly (machine precision). Forward Euler: u(t) = ct + d. u(0) = 0 means d = I . n ODE implies c = a(t)u + b(t). [D+u = au + b] − − Choose a(t) and c, and set b(t) = c + a(t)(ct + I ). an a t , bn c a t ct I , and un ct I results in Any error in the formula for un+1 makes u = ct + I ! = ( n) = + ( n)( n + ) = n + 6

c = a(tn)(ctn + I ) + c + a(tn)(ctn + I ) = c −

Test function for linear manufactured solution Extension to systems of ODEs

def test_linear_solution(): """ Sample system: Test problem where u=c*t+I is the exact solution, to be reproduced (to machine precision) by any relevant method. """ def exact_solution(t): returnc*t+I u0 = au + bv (51)

defa(t): v 0 = cu + dv (52) returnt**0.5 # can be arbitrary

defb(t): returnc+ a(t)*exact_solution(t) The Forward :

theta= 0.4; I= 0.1; dt= 0.1; c=-0.5 T=4 Nt= int(T/dt) # no of steps n+1 n n n u, t= solver(I=I, a=a, b=b, T=Nt*dt, dt=dt, theta=theta) u = u + ∆t(au + bv ) (53) u_e= exact_solution(t) n+1 n n n difference= abs(u_e- u).max() # max deviation v = u + ∆t(cu + dv ) (54) print difference tol= 1E-14 # depends on c! assert difference< tol The Backward Euler method gives a system of algebraic Generic form equations The standard form for ODEs:

u f u t u 0 I (59) The Backward Euler scheme: 0 = ( , ), ( ) =

u and f : scalar or vector. n 1 n n 1 n 1 u + = u + ∆t(au + + bv + ) (55) Vectors in case of ODE systems: v n+1 v n t cun+1 dv n+1 (56) = + ∆ ( + ) (0) (1) (m 1) u(t) = (u (t), u (t),..., u − (t)) which is a 2 2 linear system: ×

0 0 m 1 n 1 n 1 n f u t f ( ) u( ) u( ) (1 ∆ta)u + + bv + = u (57) ( , ) = ( ( ,..., − ) 1 0 m 1 n−1 n 1 n f ( ) u( ) u( ) cu + + (1 ∆td)v + = v (58) ( ,..., − ), − . . Crank-Nicolson also gives a 2 2 linear system. f (m 1) u(0) t u(m 1) t × − ( ( ),..., − ( )))

The θ-rule Implicit 2-step backward scheme

un+1 un n 1 n n 1 n+1 n 3u + 4u + u = θf (u , tn 1) + (1 θ)f (u , tn) (60) u t − −t + 0( n+1) − ∆ − ≈ 2∆t Bringing the unknown un+1 to the left-hand side and the known terms on the right-hand side gives Scheme: 4 1 2 n+1 n n 1 n+1 u = u u − + ∆tf (u , tn+1) n+1 n+1 n n 3 − 3 3 u ∆tθf (u , tn+1) = u + ∆t(1 θ)f (u , tn) (61) − − n 1 This is a nonlinear equation in un+1 (unless f is linear in u)! Nonlinear equation for u + .

The Leapfrog scheme The ltered Leapfrog scheme

Idea: n+1 n 1 u u − n u0(tn) − = [D2t u] (62) ≈ 2∆t Scheme: After computing un+1, stabilize Leapfrog by D u f u t n [ 2t = ( , )] n n n 1 n n+1 u u + γ(u − 2u + u ) (64) or written out, ← − n+1 n 1 n u = u − + ∆tf (u , tn) (63)

Some other scheme must be used as starter (u1). Explicit scheme - a nonlinear f (in u) is trivial to handle. Downside: Leapfrog is always unstable after some time. 2nd-order Runge-Kutta scheme 4th-order Runge-Kutta scheme

Forward-Euler + approximate Crank-Nicolson: The most famous and widely used ODE method u un tf un t (65) ∗ = + ∆ ( , n), 4 evaluations of f per time step 1 un+1 un t f un t f u t (66) Its derivation is a very good illustration of numerical thinking! = + ∆ 2 ( ( , n) + ( ∗, n+1))

2nd-order Adams-Bashforth scheme 3rd-order Adams-Bashforth scheme

1 n+1 n n n 1 1 u = u + ∆t 3f (u , tn) f (u − , tn 1) (67) un+1 un 23f un t 16f un 1 t 5f un 2 t 2 − = + ( , n) ( − , n 1) + ( − , n 2) − 12 − − − (68)  

The Odespy software Example: Runge-Kutta methods

Odespy features simple Python implementations of the most fundamental schemes as well as Python interfaces to several famous packages for solving ODEs: ODEPACK, Vode, rkc.f, rkf45.f, Radau5, as well as the ODE solvers in SciPy, SymPy, and odelab. solvers= [odespy.RK2(f), Typical usage: odespy.RK3(f), odespy.RK4(f), # Define right-hand side of ODE odespy.BackwardEuler(f, nonlinear_solver='Newton')] def f(u, t): return -a*u for solver in solvers: solver.set_initial_condition(I) import odespy u, t= solver.solve(t) import numpy as np # + lots of plot code... # Set parameters and time mesh I = 1; a = 2; T = 6; dt = 1.0 Nt = int(round(T/dt)) t_mesh = np.linspace(0, T, Nt+1) # Use a 4th-order Runge-Kutta method solver = odespy.RK4(f) solver.set_initial_condition(I) u, t = solver.solve(t_mesh) Plots from the experiments Example: Adaptive Runge-Kutta methods Adaptive methods nd "optimal" locations of the mesh points to ensure that the error is less than a given tolerance. Downside: approximate error estimation, not always optimal location of points. "Industry standard ODE solver": Dormand-Prince 4/5-th order Runge-Kutta (MATLAB's famous ode45).

The 4-th order Runge-Kutta method (RK4) is the method of choice!