FINITE DIFFERENCE

AND SPECTRAL METHODS

FOR

ORDINARY AND PARTIAL

DIFFERENTIAL EQUATIONS

Lloyd N Trefethen

Cornell University

c

Copyright by Lloyd N Trefethen

Note

A trap that academics sometimes fall into is to b egin a b o ok and fail to nish it This

has happ ened to me This book was intended to b e published several years ago Alas

two other pro jects jump ed the queue Numerical Linear Algebra with David Bau

to be published in by SIAM and Spectra and Pseudospectra to be completed

thereafter AtthispointI do not know whether this numerical ODEPDE b o ok will

ever be prop erly nished

This is a shameas p eople keep telling me Electronically by email and in p erson at

conferences I am often asked when that PDE book is going to get done Anyone

who has b een in such a situation knows how attering such questions feel at rst

The book is based on graduate courses taught to mathematicians and engineers since

at the Courant Institute MIT and Cornell Successively more complete versions

have b een used by me in these courses half a dozen times and by a handful of colleagues

at these universities and elsewhere Much of the b o ok is quite p olished esp ecially the

rst half and there are numerous exercises that have b een tested in the classro om

But there are many gaps to o and undoubtedly a few errors I have repro duced here

the version of the text I used most recently in class in the spring of

I am proud of Chapter which I consider as clean an intro duction as any to the

classical theory of the numerical solution of ODE The other main parts of the b o ok are

Chapter which emphasizes the crucial relationship b etween smo othness of a function

and decay of its Fourier transform Chapters and which present the classical theory

of numerical stability for nite dierence approximations to linear PDE and Chapters

and which oer a handson intro duction to Fourier and Chebyshev sp ectral metho ds

This book is largely linear and for that reason and others there is much more to the

numerical solution of dierential equations than you will nd here On the other hand

Chapters represent material that every numerical analyst should know These

chapters alone can b e the basis of an app ealing course at the graduate level

Please do not repro duce this text all rights are reserved Copies can be obtained for

or each including shipping by contacting me directly Professors who are

using the book in class may contact me to obtain solutions to most of the problems

This is no longer true the book is freely available online at

httpwebcomlaboxacu kou cl work nic ktr efe then pde text htm l

Nick Trefethen

July

Department of Computer Science and Center for

Upson Hall Cornell University Ithaca NY USA

fax LNTcscornelledu

CONTENTS TREFETHEN v

Contents

Preface viii

Notation x

BigO and related symb ols

Rounding errors and discretization errors

Ordinary dierential equations

Initial value problems

Linear multistep formulas

Accuracy and consistency

Derivation of linear multistep formulas

Stability

Convergence and the Dahlquist Equivalence Theorem

Stability regions

Stiness

RungeKutta metho ds

Notes and references

Fourier analysis

The Fourier transform

The semidiscrete Fourier transform

Interp olation and sinc functions

The discrete Fourier transform

Vectors and multiple space dimensions

Notes and references

Finite dierence approximations

Scalar mo del equations

Finite dierence formulas

Spatial dierence op erators and the metho d of lines

Implicit formulas and linear algebra

Fourier analysis of nite dierence formulas

Fourier analysis of vector and multistep formulas

Notes and references

CONTENTS TREFETHEN vi

Accuracy stability and convergence

An example

The Lax Equivalence Theorem

The CFL condition

The von Neumann condition for scalar onestep formulas

Resolvents pseudosp ectra and the Kreiss matrix theorem

The von Neumann condition for vector or multistep formulas

Stabilityofthe metho d of lines

Notes and references

Dissipation disp ersion and group velo city

Disp ersion relations

Dissipation

Disp ersion and group velo city

Mo died equations

p

norms Stabilityin

Notes and references

Boundary conditions

Examples

Scalar hyp erb olic equations

Systems of hyp erb olic equations

Absorbing b oundary conditions

Notes and references

Fourier sp ectral metho ds

An example

Unb ounded grids

Perio dic grids

Stability

Notes and references

Chebyshev sp ectral metho ds

Polynomial interp olation

Chebyshev dierentiation matrices

Chebyshev dierentiation by the FFT

Boundary conditions

Stability

Some review problems

Two nal problems

Notes and references

CONTENTS TREFETHEN vii

Multigrid and other iterations

Jacobi GaussSeidel and SOR

Conjugate gradients

Nonsymmetric matrix iterations

Preconditioning

The multigrid idea

Multigrid iterations and Fourier analysis

Notes and references

APPENDIX A Some formulas of complex analysis APP

APPENDIX B Vector matrix and op erator norms APP

REFERENCES REFS

PREFACE TREFETHEN viii

Preface

Many books have b een written on numerical metho ds for partial dier

ential equations ranging from the mathematical classic by Richtmyer and

Morton to recent texts on computational uid dynamics But this is not an

easy eld to set forth in abook It is to o active and to o large touching a b e

wildering variety of issues in mathematics physics engineering and computer

science

My goal has been to write an advanced textb o ok fo cused on basic prin

ciples Two remarkably fruitful ideas p ervade this eld numerical stability

and Fourier analysis I want the reader to think at length ab out these and

other fundamentals b eginning with simple examples and build in the pro cess

a solid foundation for subsequent work on more realistic problems Numerical

metho ds for partial dierential equations are a sub ject of subtlety and b eauty

but the appreciation of them may be lost if one views them to o narrowly as

to ols to be hurried to application

This is not a b o ok of pure mathematics Anumb er of theorems are proved

but my purp ose is to present a set of ideas rather than a sequence of theorems

The book should be accessible to mathematically inclined graduate students

and practitioners in various elds of science and engineering so long as they

are comfortable with Fourier analysis basic partial dierential equations some

numerical metho ds the elements of complex variables and linear algebra in

cluding vector and matrix norms At MIT and Cornell successive drafts of

this text have formed the basis of a large graduate course taught annually

since

One unusual feature of the book is that I have attempted to discuss the

numerical solution of ordinary and partial dierential equations in parallel

There are numerous ideas in common here well known to professionals but

usually not stressed in textb o oks In particular I have made extensive use of

the elegantideaofstability regions in the complex plane

Another unusual feature is that in a single volume this b o ok treats sp ec

tral as well as the more traditional nite dierence metho ds for discretization

dierential equations The solution of discrete equations by multi of partial

grid iterations the other most conspicuous development in this eld in the

past twenty years is also intro duced more briey

The unifying theme of this b o ok is Fourier analysis Tobesure it is well

PREFACE TREFETHEN ix

known that Fourier analysis is connected with numerical metho ds but here we

shall consider at least four of these connections in detail and p oint out analo

gies between them stability analysis sp ectral metho ds iterative solutions of

elliptic equations and multigrid metho ds Indeed a reasonably accurate title

would have b een Fourier Analysis and Numerical Metho ds for Partial Dif

ferential Equations Partly b ecause of this emphasis the ideas covered here

are primarily linear this is by no means a textb o ok on computational uid

dynamics But it should prepare the reader to read such b o oks at high sp eed

Besides these larger themes anumber of smaller topics app ear here that

are not so often found outside research pap ers including

Order stars for stabilityand accuracy analysis

Modied equations for estimating discretization errors

p

Stabilityin norms

Absorbing boundary conditions for wave calculations

Stability analysis of nonnormal discretizations via pseudospectra

Group velocity eects and their connection with the stability of b oundary

conditions

Like every author I have inevitably given greatest emphasis to those topics I

know through my own work

The reader will come to know my biases so on enough I hop e he or she

shares them One is that exercises are essential in any textb o ok even if the

treatment is advanced The exercises here range from co okb o ok to philo

sophical those marked with solid triangles require computer programming

Another is that almost every idea can and should b e illustrated graphically A

third is that numerical metho ds should b e studied at all levels from the most

formal to the most practical This b o ok emphasizes the formal asp ects saying

relatively little ab out practical details but the reader should forget at his or

her p eril that a vast amount of practical exp erience in numerical computation

has accumulated over the years This exp erience is magnicently represented

in a wealth of highquality mathematical software available around the world

such as EISPACK LINPACK LAPACK ODEPACK and the IMSL and NAG

mention such interactive systems as Matlab Bo eing Fortran libraries not to

and Mathematica Half of every applied problem can and should b e handled

by otheshelf programs nowadays This book will tell you something ab out

what those programs are doing and help you to think pro ductively ab out the

other half

For online acquisition of numerical software every reader of this b o ok should b e sure to b e familiar

with the Netlib facility maintained at Bell Lab oratories and Oak Ridge National Lab oratory For

information send the email message help to netlibresearchattcom or netlibornlgov

NOTATION TREFETHEN x

Notation

R C Z real numb ers complex numb ers integers

marks anumber contaminated by rounding errors

O o order relations analogous to

x y t space and time variables

x wave numb er y wave number frequency

T initialvalue problem is p osed for t T

N dimension of system of equations

d number of space dimensions

u dep endent variable scalar or N vector

h k space and time step sizes h x k t

i j n space and time indices

rs integers dening size of nite dierence stencil

n

v numerical approximation to uih j h nk

ij

u v Fourier transforms

mesh ratios kh k h

n n

K Z space and time shift op erators K v v Z v v

j j

i h i k

z space and time amplication factors e z e

r forward and backward dierence op erators K r K

z characteristic p olynomials for a linear multistep formula z

c c phase and group velo cities c c d d

g g

A A sp ectrum and pseudosp ectrum of A

AND TREFETHEN

The following little sections and contain essential background material

for the remainder of the text They are not chapters just tidbits Think of

them as app etizers

BIGO AND RELATED SYMBOLS TREFETHEN

BigO and related symbols

How fast is an algorithm How accurate are the results it pro duces

Answering questions like these requires estimating op eration counts and errors

and we need a convenient notation for doing so This book will use the six

symbols o O and Four of these are standard throughout the

mathematical sciences while and ha ve recently b ecome standard at least

in theoretical computer science

Here are the rst four denitions They are written for functions f xand

g xas x but analogous denitions apply to other limits suc h as x

O f xO g x as x if there exist constants C and x such

that jf xjCgx for all x x If g x this means jf xjg x is

b ounded from ab ove for all suciently large x In words f is O of g

o f xog x as x if for any constant C there exists a constant

x such that jf xjCgx for all x x If g x this means

jf xjg x approaches as x In words f is littleO of g

f xg x as x if there exist constants C and x such

that f x Cgx for all x x If g x this means f xg x is

b ounded from below for all suciently large x In words f is omega

of g



f xg x as x if there exist constants C C and x



means such that Cgx f x C g x for all x x If g x this

f xg x is b ounded from ab ove and b elow In words f is theta of g

The use of the symb ols O o and is a bit illogical for prop erly

sp eaking O g x for example is the set of all functions f x with a certain

prop erty but why then dont we write f x O g x The answer is sim

ply that the traditional usage illogical or not is well established and v ery

convenient

The nal two denitions are not so irregular

Thanks to the delightful note Big omicron and big omega and big theta byDEKnuth ACM

SIGACT News

BIGO AND RELATED SYMBOLS TREFETHEN

f x g xas x if f x g xojg xj If g x this is equiva

lent to lim f xg x In words f is asymptotic to g

x

This symbol is easy it has no precise denition The statement f x

g x is qualitative only and the argument x plays no role Often in this

b o ok but not always we shall use to indicate that a real number

has b een rounded or chopp ed to a nite number of digits eg or

 x 

EXAMPLE True statements as x sin x O log x ox e x



x x

sin x cosh x e o e False statements as x x ox sin x

x

e

The relationship b etween these denitions can b e summarized by the Venn

diagram of Figure Let X denote the set of all pairs of functions ff gg The

diagram illustrates for example that the subset of X consisting of pairs ff gg

with f xog x is contained in the subset of pairs with f xO g x In

other words f xog x implies f xO g x

Figure Venn diagram showing the relationship b etween o O and

BIGO AND RELATED SYMBOLS TREFETHEN

EXERCISES

True or False

 

a x sinx x as x



b x sinx x as x

 

c sin x sinx asx

 

d log x ologx as x

n

e nne as n

Supp ose g x asx Find interesting examples of functions f g o ccupying

e distinct p ositions in the Venn diagram of Figure all v

True or False



a A V asV where A and V are the surface area and volume of a sphere

measured in square miles and cubic microns resp ectively

b lim a b a b o as n

n n n

c A set of N words can be sorted into alphab etical order by an algorithm that makes

O N log N pairwise comparisons

d Solving an N N matrix problem Ax b by Gaussian elimination on a serial computer



takes N op erations

ROUNDING AND TRUNCATION ERRORS TREFETHEN

 Rounding and truncation errors

Throughout this b o ok we shall b e concerned with computed inexact approximations

to ideal quantities Twothingskeep them from b eing exact rounding errors and truncation

errors

Rounding errors are the errors intro duced by computers in simulating real or com

plex arithmetic by oatingp oint arithmetic For most purp oses they can be treated as

ubiquitous and random and it is p ossible to b e quite precise ab out how they b ehave Given

a particular machine with a particular system of oatingp oint arithmetic let machine ep

silon b e dened as the smallest p ositive oatingp ointnumber suchthat Here

denotes oatingp oint addition of oatingp ointnumb ers analogous notations apply for

and Then on welldesigned computers if a and b are arbitrary oatingp oint

numb ers the following identities hold

a b a b











a b a b

for some with j j



a b a b









a b a b

the quantity is of course dierent from one app earance to the next In other words each

y

oatingp oint op eration intro duces a relative error of magnitude less than

Truncation errors or discretization errors are the errors intro duced by algo

rithms in replacing an innite or innitesimal quantityby something nite they have noth

ing to do with oatingp oint arithmetic For example the error intro duced in truncating an

innite series at some nite term N is a truncation error and so is the error intro duced in

replacing a derivative dudt by the nite dierence ut t ut tt Truncation

errors do not app ear in all areas of numerical computation they are absent for example

in solving a linear system of equations by Gaussian elimination They are usually unavoid

able however in the numerical solution of dierential equations Unlike rounding errors

truncation errors are generally not at all random but mayhave a great deal of regularity

Rounding errors would vanish if computers were innitely accurate Truncation errors

would vanish if computers were innitely fast and had innitely large memories so that there

were no need to settle for nite approximations Further discussion of this distinction can

e

Floatingp ointnumb ers are real numb ers of the form x f  where is the base usually

or f is the fraction represented to t digits in base and e is an adjustable

exp onent Floatingp oint arithmetic refers to the use of oatingp ointnumb ers together with

approximate arithmetic op erations dened up on them Early computers sometimes used xed

point arithmetic instead but this is unheardof nowadays except in sp ecialpurp ose machines For

an intro duction to oatingp oint arithmetic see G B Forsythe M A Malcolm C B Moler

Computer Metho ds for Mathematical Computations PrenticeHall and for a table of the

oatingp ointcharacteristics of existing computers as of see W J Co dy ACM Trans Math

w Soft

y

This description ignores the p ossibility of underoworoverow

ROUNDING AND TRUNCATION ERRORS TREFETHEN

b e found in many b o oks on numerical analysis suchasP Henrici Essentials of Numerical

Analysis Wiley

In this b o ok we shall frequently ask how a particular computation may b e aected by

rounding or truncation errors Which source of error dominates Most often rounding errors

are negligible and the accuracy is determined by truncation errors When an instabilityis

present however the situation is sometimes reversed We shall pay careful attention to

such matters for there is no other way to understand mathematical computations fully

Truncation errors dep end only on the algorithm but rounding errors dep end also on

the machine We shall use the following notation throughout the b o ok anumb er preceded

by a star is signicantly contaminated by rounding errors and hence machinedep endent

x

EXAMPLE  For example supp ose weapproximate e by the term series

 

x x

f xx

evaluated on the computer from left to right as written On a Sun Workstation in double

 

precision with we obtain

  

e f

  

e f

  

e f

Thus f is accurate but f is inaccurate b ecause of truncation errors and f

is inaccurate b ecause of rounding errors Why See Exercise This straightforward

x

summation of the Taylor series for e is a classic example of a bad algorithm it is sometimes

unstable and always inecient unless jxj is small

Some p eople consider the study of rounding errors a tedious business and

there are those who assume that all of numerical analysis must b e tedious for

isnt it mainly the study of rounding errors Here are three reasons whythese

views are unjustied

First thanks to the simple formulas the mo deling of rounding errors

is easy and even elegant Rounding errors are by no means capricious or

unpredictable and one can understand them well without worrying ab out

machinedep endent engineering details

Second the existence of inexact arithmetic should b e blamed not on com

puters but on mathematics itself It has b een known for a long time that many

welldened quantities cannot be calculated exactly by any nite sequence of

elementary op erations the classic example is the unsolvability of the ro ots

of p olynomials of degree Thus approximations are unavoidable even in

principle

Mathematics also rests on approximations at a more fundamental level The set of real numb ers is

uncountably innite but as was rst p ointed out byTuring in his famous pap er on Computable

ROUNDING AND TRUNCATION ERRORS TREFETHEN

Finally and most imp ortant numerical analysis is not just the study of

rounding errors Here is a b etter denition

Numerical analysis is the study of algorithms

for mathematical problems involving continuous variables

Thus it is a eld devoted to algorithms in which the basic questions take the

form What is the b est algorithm for computing suchandsuch Insensitivity

to rounding errors is only a part of what makes a numerical algorithm go o d

sp eed for example is certainly as imp ortant

EXERCISES

Figure out mac hine epsilon exactly on your calculator or computer What are the

base and the precision t Explain your reasoning carefully

Continuation of Example

a Explain the mechanism bywhich rounding errors aected the computation of f

b If we compute f in the same way will there b e a star in front of the result

x

c Suggest an algorithm for calculating e more quickly and accurately in oatingp oint

arithmetic

Numb ers in only a countable subset of them can ever b e sp ecied byany nite description

The vast ma jority of real numb ers can only b e describ ed in eect by listing their innite decimal

expansions which requires an innite amountof time Thus most real numb ers are not merely

unrepresentable on computers they are unmentionable even by pure mathematicians and exist

only in a most shadowy sense

CHAPTER TREFETHEN

Chapter

Ordinary Dierential Equations

Initial value problems

Linear multistep formulas

Accuracy and consistency

Derivation of linear multistep formulas

Stability

Convergence and the Dahlquist Equivalence Theorem

Stability regions and absolute stability

Stiness

RungeKutta metho ds

Notes and references

Just as the twig is b ent the trees inclined

ALEXANDER POPE Moral Essays

CHAPTER TREFETHEN

The central topic of this b o ok is timedep endent partial dierential equa

tions PDEs Even when we come to discuss elliptic PDEs which are not

timedep endent we shall nd that some of the best numerical metho ds are

iterations that b ehavevery much as if there were a time variable That is why

elliptic equations come at the end of the book rather than the b eginning

Ordinary dierential equations ODEs embody one of the two essential

ingredients of this central topic variation in time but not in space Because

the state at any moment is determined b y a nite set of numb ers ODEs are

far easier to solve and far more fully understo o d than PDEs In fact an ODE

usually has to be nonlinear b efore its behavior b ecomes very interesting

which is not the case at all for a PDE

The solution of ODEs by discrete metho ds is one of the oldest and most

successful areas of numerical computation A dramatic example of the need

for such computations is the problem of predicting the motions of planets or

other b o dies in space The governing dierential equations have b een known

since Newton and for the case of only two b o dies such as a sun and a planet

Newton showed how to solve them exactly In the three centuries since then

however no one has ever found an exact solution for the case of three or more

b o dies Yet the numerical calculation of such orbits is almost eortless by

mo dern standards and is a routine comp onentofthe control of spacecraft

The most imp ortant families of numerical metho ds for ODEs are

linear multistep metho ds

RungeKutta metho ds

This chapter presents the essential features of linear multistep metho ds with

emphasis on the fundamental ideas of accuracy stability and convergence An

imp ortant distinction is made between classical stability and eigenvalue

stability All of these notions will prove indisp ensable when we move on to

discuss PDEs in Chapter Though we do not describ e RungeKutta metho ds

except briey in x most of the analytical ideas describ ed here apply with

minor changes to RungeKutta metho ds to o

In fact there are theorems to the eect that such an exact solution can never be found This

unpredictability of manyb o dy motions is connected with the phenomenon of chaos

INITIAL VALUE PROBLEMS TREFETHEN

Initial value problems

An ordinary dierential equationorODE is an equation of the form

u tf utt

t

where t is the time variable u is a real or complex scalar or vector function

N N

of t ut C N and f is a function that takes values in C We

The sym also say that is a system of ODEs of dimension N

bol u denotes dudt and if N it should be interpreted comp onentwise

t

N

N T T

u u u u Similarly u denotes d udt and so on

t tt

t t

Where clarity p ermits we shall leave out the arguments in equations like

which then b ecomes simply u f

t

The study of ODEs go es back to Newton and Leibniz in the s and

likesom uch of mathematics it owes a great deal also to the work of Euler in

the th century Systems of ODEs were rst considered by Lagrange in the

s but the use of vector notation did not b ecome standard until around

If f u t tu t for some functions t and t the ODE is

linear and if t it is linear and homogeneous In the vector case t

is an N N matrix and t is an N vector Otherwise it is nonlinear If

f u t is indep endentoft the ODE is autonomous If f u t is indep endent

of uthe ODE reduces to an indenite integral

To make into a fully sp ecied problem we shall provide initial

data at t and lo ok for solutions on some interval t T T The

choice of t as a starting point intro duces no loss of generality since any

other t could b e treated by the change of variables t t t

N

Initial Value Problem Given f as describ ed ab ove T and u C

nd a dierentiable function ut dened for t T such that

a u u

b u tf utt for all t T

t

N

C is the space of complex column vectors of length N In practical ODE problems the variables

N

are usually real so that for many purp oses we could write R instead When wecometoFourier

analysis of linear partial dierential equations however the use of complex variables will b e very

convenient

INITIAL VALUE PROBLEMS TREFETHEN

Numerical metho ds for solving ODE initialvalue problems are the sub ject of

this chapter We shall not discuss b oundaryvalue problems in which various

comp onents of u are sp ecied at two or more distinct p oints of time see Keller

and Ascher et al

EXAMPLE The scalar initialvalue problem u Au u a has the solution

t

tA

utae whichdecays to as t provided that A or Re Aif A is complex

This ODE is linear homogeneous and autonomous

EXAMPLE The example ab ove b ecomes a vector initialvalue problem if u and

tA

a are N vectors and A is an N N matrix The solution can still b e written ute a

tA

if e now denotes the N N matrix dened by applying the usual Taylor series for the

exp onen tial to the matrix tA For generic initial vectors a this solution ut decays to the

zero vector as t ie lim kutk if and only if each eigenvalue of A satises

t

Re

EXAMPLE The scalar initialvalue problem u u cos t u has the solution

t

sin t

ute One can derive this by separation of variables by integrating the equation

duu costdt

EXAMPLE The nonlinear initialvalue problem u u u u has the solution

t

t

ut e which is valid until the solution b ecomes innite at t log

This ODE is an example of a Bernoulli dierential equation One can derive the solution by

the substitution w tut which leads to the linear initialvalue problem w w

t

t

w with solution w te

EXAMPLE The Lorenz equations with the most familiar choice of parameters

can b e written

w uv u u v v u v uw w

t t t

This is the classical example of a nonlinear system of ODEs whose solutions are chaotic

The solution cannot b e written in closed form

Equation is an ODE of rst order for it contains only a rst

derivative with resp ect to t Many ODEs that o ccur in practice are of sec

ond order or higher so the form we havechosen may seem unduly restrictive

However any higherorder ODE can be reduced to an equivalent system of

rstorder ODEs in a mechanical fashion by intro ducing additional variables

representing lowerorder terms The following example illustrates this reduc

tion

EXAMPLE Supp ose that a mass m at the end of a massless spring of length

 

y exp eriences a force F K y y where K is the spring constant and y is the rest

p osition Ho okes Law By Newtons First Law the motion is governed by the autonomous

equation

K



y y y y a y b

tt t m

INITIAL VALUE PROBLEMS TREFETHEN

for some a and b a scalar secondorder initialvalue problem Now let u be the vector

with u y u y Then the equation can b e rewritten as the rstorder system

t

u u

t

u a

K



u b

u y u

t

m

It should be clear from this example and Exercise below how to

reduce an arbitrary collection of higherorder equations to a rstorder system

More formal treatments of this pro cess can be found in the references

Throughout the elds of ordinary and partial dierential equationsand

their numerical analysisthere are a variety of pro cedures in which one prob

lem is reduced to another We shall see inhomogeneous terms reduced to

initial data initial data reduced to b oundary values and so on But this re

duction of higherorder ODEs to rstorder systems is unusual in that it is not

just a convenient theoretical device but highly practical In fact most of the

generalpurp ose ODE software currently available assumes that the equation is

written as a rstorder system One pays some price in eciency for this but

it is usually not to o great For PDEs on the other hand such reductions are

less often practical and indeed there is less generalpurp ose software available

of an y kind

It may seem obvious that should have a unique solution for all

t after all the equation tells us exactly how u changes at each instant

But in fact solutions can fail to exist or fail to be unique and an example of

nonexistence for t log app eared already in Example see also Exercises

and To ensure existence and uniqueness we must make some

assumptions concerning f The standard assumption is that f is continuous

with resp ect to t and satises a uniform Lipschitz condition with resp ect

N

to u This means that there exists a constant Lsuc h that for all u v C

and t T

kf u t f v tkLku v k

where kk denotes some norm on the set of N vectors For N ie a scalar

system of equations kk is usually just the absolute value jj For N the

most imp ortant examples of norms are kk kk and kk and the reader

should make sure that he or she is familiar with these examples See App endix

B for a review of norms A sucient condition for Lipschitz continuityisthat

the partial derivativeoff u t with resp ect to u exists and is b ounded in norm

N

by L for all u C and t T

The following result go es back to Cauchy in

INITIAL VALUE PROBLEMS TREFETHEN

EXISTENCE AND UNIQUENESS FOR THE INITIAL VALUE PROBLEM

Theorem Let f u t be continuous with resp ect to t and uniformly

Lipschitz continuous with resp ect to u for t T Then there exists a

unique dierentiable function ut that satises the initialvalue problem

The standard pro of of Theorem nowadays which can b e found in many

b o oks on ODEs is based on a construction called the metho d of successive

approximations or the Picard iteration This construction can be realized as

a numerical metho d but not one of the form we shall discuss in this book

Following Cauchys original idea however a more elementary pro of can also

b e based on more standard numerical metho ds such as Eulers formula which

is mentioned in the next section See Henrici or Hairer Nrsett

Wanner

Theorem can b e strengthened byallowing f to b e Lipschitz continuous

N

not for all u but for u conned to an op en subset D of C Unique solutions

then still exist on the interval T where T is either the rst p ointatwhich

the solution hits the b oundary of D or T whichever is smaller

EXERCISES

Reduction to rstorder system Consider the system of ODEs

t

u u v v t u sinv e u v

t

ttt tt t t t

with initial data

u u u v v

t tt t

Reduce this initialvalue problem to a rstorder system in the standard form

The planets M planets or suns or spacecraft orbit ab out each other in three

space dimensions according to Newtons laws of gravitation and acceleration What are

the dimension and order of the corresp onding system of ODEs a When written in their

most natural form b When reduced to a rstorder system This problem is intended

to be straightforward do not attempt clever tricks such as reduction to centerofmass

co ordinates

Existence and uniqueness Apply Theorem to show existence and uniqueness of

the solutions wehavegiven for the following initialvalue problems stating explicitly your

choices of suitable Lipschitz constants L

a Example

b Example

c Example First by considering u u u itself explain why Theorem do es

t

not guarantee existence or uniqueness for all t Next by considering the transformed

equation w w show that Theorem does guarantee existence and uniqueness t

INITIAL VALUE PROBLEMS TREFETHEN

until such time as u may b ecome innite Exactly how do es the pro of of the theorem

fail at that p oint

d Example

Nonexistence and nonuniqueness Consider the scalar initialvalue problem

u u u u

t

for some constant

a For which do es Theorem guarantee existence and uniqueness of solutions for all

t

b For and u no solution for all t exists Verify this informally byndingan

explicit solution that blows up to innity in a nite time Of course suchanexample

by itself do es not constitute a pro of of nonexistence

c For and u there is more than one solution Find one of them an obvious

one Then nd another obvious one Now construct an innite family of distinct

solutions

Continuity with resp ect to t Theorem requires continuity with resp ect to t as well

as Lipschitz continuity with resp ect to u Show that this assumption cannot b e disp ensed

with byndinganinitialvalue problem indep endentofT inwhich f is uniformly Lipschitz

continuous with resp ect to u but discontinuous with resp ect to t and for which no solution

exists on anyinterval T with T Hint consider the degenerate case in which f u t

is indep endentofu

LINEAR MULTISTEP FORMULAS TREFETHEN

Linear multistep formulas

Most numerical metho ds in every eld are based on discretization For the solution

of ordinary dierential equations one of the most p owerful discretization strategies go es by

the name of linear multistep metho ds

Supp ose we are given an initialvalue problem that satises the hyp otheses of

Theorem on some interval T Then it has a unique solution ut on that interval

but the solution can rarely be found analytically Let k be a real numb er the time

step and let t t b e dened by t nk Our goal is to construct a sequence of values

n

v v such that

n

v ut n

n

The sup erscripts are not exp onents we are leaving ro om for subscripts to accommo date

n

spatial discretization in later chapters We also sometimes write v t instead of v Let

n

n

f b e the abbreviation

n n

f f v t

n

n

Alinearmultistep metho d is a formula for calculating eachnew value v from some of

n n

the previous values v v and f f Equation b elow will makethismore

precise

We shall take the attitude standard in this eld that the aim in solving an initial

value problem numerically is to achieve a prescrib ed accuracy with the use of as few function

n

evaluations f v t as p ossible In other words obtaining function values is assumed to b e

n

so exp ensive that all subsequent manipulationsall overhead op erationsare essentially

free For easy problems this assumption may b e unrealistic but it is the harder problems

that matter more For hard problems values of f may be very exp ensive to determine

particularly if they are obtained by solving an inner algebraic or dierential equation

Linear multistep metho ds are designed to minimize function evaluations by using the

n n

approximate values v and f rep eatedly

The simplest linear multistep metho d is a onestep metho d the Euler formula dened

by

n n n

v v kf

The motivation for this formula is linear extrap olation as suggested in Figure a If

v is given presumably set equal to the initial value u it is a straightforward matter to

apply to compute successivevalues v v Eulers metho d is an example of an

explicit onestep formula

A related linear multistep metho d is the backward Euler or implicit Euler for

mula also a onestep formula dened by

n n n

v v kf

n n

The switch from f to f is a big one for it makes the backward Euler formula im

n n

Therefore it would plicit According to f is an abbreviation for f v t

n

n n N

In general v and f are vectors in C but it is safe to think of them as scalars the extension to

systems of equations is easy

LINEAR MULTISTEP FORMULAS TREFETHEN

o

* n

v

n v

*o

n

v

n

o v o

* * n

*o v

t t t t t

n n n n n

a Euler b midp oint

Figure One step of the Euler and midp oint formulas The solid curves

represent not the exact solution to the initialvalue problem but the exact solu

n

tion to the problem u f ut v

t n

n

app ear that v cannot b e determined from unless it is already known In fact to

implement an implicit formula one must employaniterationofsomekindtosolve for the

n

unknown v and this involves extra work not to mention the questions of existence and

uniqueness But as we shall see the advantage of is that in some situations it may

b e stable when is catastrophically unstable Throughout the numerical solution of

dierential equations there is a tradeo b etween explicit metho ds which tend to b e easier

to implement and implicit ones which tend to b e more stable Typically this tradeo tak es

the form that an explicit metho d requires less work p er time step while an implicit metho d

is able to take larger and hence fewer time steps without sacricing accuracy to unstable

oscillations

An example of a more accurate linear multistep formula is the trap ezoid rule

k

n n n n

v v f f

also an implicit onestep formula Another is the midp oint rule

n n n

v v kf

an explicit twostep formula Figure b The fact that involves multiple time

levels raises a new diculty Wecansetv u but to compute v with this formula where

In socalled predictorcorrector metho ds not discussed in this b o ok the iteration is terminated

b efore convergence giving a class of metho ds intermediate b etween explicit and implicit

But dont listen to p eople who talk ab out conservation of diculty as if there were never any

clear winners We shall see that sometimes explicit metho ds are vastly more ecient than implicit

ones large nonsti systems of ODEs and sometimes it is the reverse small sti systems

LINEAR MULTISTEP FORMULAS TREFETHEN



shall we get v Or if we wantto begin by computing v where shall weget v This

initialization problem is a general one for linear multistep formulas and it can b e addressed

in several ways One is to calculate the missing values numerically by some simpler formula

such as Eulers metho dp ossibly applied several times with a smaller value of k to avoid

loss of accuracy Another is to handle the initialization pro cess by RungeKutta metho ds

to b e discussed in Section

In Chapter we shall see that the formulas have imp ortant analogs for

y

partial dierential equations For the easier problems of ordinary dierential equations

however they are to o primitive for most purp oses Instead one usually turns to more

complicated and more accurate formulas such as the fourthorder AdamsBashforth

formula

k

n n n n n n

v v f f f f

an explicit fourstep formula or the fourthorder AdamsMoulton formula

k

n n n n n n

v v f f f f

an implicit threestep formula Another implicit threestep formula is the thirdorder

backwards dierentiation formula

n

n n n n

v v v v kf

whose advantageous stability prop erties will b e discussed in Section

EXAMPLE Let us p erform an exp eriment to compare some of these metho ds The

initialvalue problem will b e

u u t u

t

t

whose solution is simply ute Figure compares the exact solution with the nu

merical solutions obtained by the Euler and midp oint formulas with k and k

For simplicity wetookv equal to the exact value ut to start the midp oint formula

A solution with the fourthorder AdamsBashforth formula was also calculated but is in

distinguishable from the exact solution in the plot Notice that the midp oint formula do es

the Euler formula and that cutting k in half improves b oth To make much b etter than

this precise Table compares the various errors u v The nal column of ratios

suggests that the errors for the three formulas have magnitudes k k and k as

k The latter gure accounts for the name fourthorder see the next section

Up to this pointwe have listed a few formulas some with rather compli

cated co ecients and shown that they work at least for one problem Fortu

nately the sub ject of linear multistep metho ds has a great deal more order in

it than this A number of questions suggest themselves

What is the general form of a linear multistep metho d

y

Euler Backward Euler CrankNicolson and Leap Frog

LINEAR MULTISTEP FORMULAS TREFETHEN

o o

o *

o * o * o * * o o * o * *

o o * * * o o o * * o * o o * * o * o o * * o * o o * * *o * *o o *o *o o *o *

*o *o *

a k b k

Figure Solution of by the Euler and midp ointo formulas

k k k ratio k tok

Euler

Midp oint

AB

Table Errors u v for the exp eriment of Figure

Can appropriate co ecients be derived in a systematic way

How accurate are these metho ds

Can anything go wrong

In the following pages we shall see that these questions have very interesting

answers

We can take care of the rst one rightawayby dening the general linear

multistep formula

LINEAR MULTISTEP FORMULAS TREFETHEN

An sstep linear multistep formula is a formula

s s

X X

nj nj

f v k

j j

j j

for some constants f g and f g with and either or

s

j j

If the formula is explicitandif it is implicit

s s

Readers familiar with electrical engineering may notice that lo oks

like the denition of a recursive or I IR digital lter Opp enheim Sc hafer

Linear multistep formulas can indeed be thought of in this way and

many of the issues to b e describ ed here also come up in digital signal pro cess

ing such as the problem of stability and the idea of testing for it by lo oking for

zeros of a p olynomial in the unit disk of the complex plane Alinearmultistep

formula is not quite a digital lter of the usual kind however In one resp ect

it is more general since the function f dep ends on u and t rather than on t

alone In another resp ect it is more narrow since the co ecients are chosen

so that the formula has the eect of integration Digital lters are designed

to accomplish a much wider variety of tasks such as highpass or lowpass

ltering smo othing dierentiation or channel equalization

What ab out the word linear Do linear multistep formulas apply only

to linear dierential equations Certainly not Equation is called

n n

linear b ecause the quantities v and f are related linearly f may very well

be a nonlinear function of u and t Some authors refer simply to multistep

formulas to avoid this p otential source of confusion

EXERCISES

gand f g for formulas What are s f

j j

Linear multistep formula for a system of equations One of the fo otnotes ab ove

claimed that the extension of linear multistep metho ds to systems of equations is easy

Verify this by writing down exactly what the system of Example b ecomes when

it is approximated by the midp oint rule

Exact answers for lowdegree p olynomials If L is a nonnegativeinteger the initial

value problem

L

u t ut u

t

t

L

has the unique solution utt Supp ose we calculate an approximation v bya

linear multistep formula using exact values where necessary for the initialization

a For which L do es repro duce the exact solution b c

Extrap olation metho ds

LINEAR MULTISTEP FORMULAS TREFETHEN

a The values V v u computed by Eulers metho d in Figure are V

k

and V Use a calculator to compute the analogous value V

b It can b e shown that these quantities V satisfy

k

V u C k C k O k

k

for some constants C C as k In the pro cess known as Richardson extrap ola

tion one constructs the higherorder estimates



V V V V u O k

k k k k

and

   

V V V V u O k

k k k k



Apply these formulas to compute V for this problem How accurate is it This is

an example of an extrap olation metho d for solving an ordinary dierential equation

Gragg Bulirsch Sto er The analogous metho d for calculating integrals

is known as Romb erg integration Davis Rabinowitz

ACCURACY AND CONSISTENCY TREFETHEN

Accuracy and consistency

In this section we dene the consistency and order of accuracy of a linear multistep for

mula as well as the asso ciated characteristic p olynomials z and z and showhow these

denitions can b e applied to derive accurate formulas by a metho d of undetermined co e

cients We also show that linear multistep formulas are connected with the approximation

of the function log z by the rational function z z atz

With f g and f g as in the characteristic p olynomials or generating

j j

p olynomials for the linear multistep formula are dened by

s s

X X

j j

z z z z

j j

j j

The p olynomial has degree exactly s and has degree s if the formula is implicit or s

if it is explicit Sp ecifying and is obviously equivalent to sp ecifying the linear multistep

formula by its co ecients Weshallseethat and are convenient for analyzing accuracy

and indisp ensable for analyzing stability

EXAMPLE Here are the characteristic p olynomials for the rst four formulas of

the last section

Euler s z z z

Backward Euler s z z z z

Trap ezoid s z z z z

Midp oint s z z z z

Now let Z denote a time shift op erator that acts both on discrete

functions according to

n n

Zv v

and on continuous functions according to

Zutut k

n n

In principle we should write Zv and Zut but expressions like Zv and

n

even Z v are irresistibly convenient The powers of Z have the obvious

n n

meanings eg Z v v and Z utut k

Equation can be rewritten compactly in terms of Z and

n n

Z v k Z f

When a linear multistep formula is applied to solve an ODE this equation is

n

satised exactly since it is the denition of the numerical approximation fv g

ACCURACY AND CONSISTENCY TREFETHEN

If the linear multistep formula is a go o d one the analogous equation ought

n n

to be nearly satised when fv g and ff g are replaced by discretizations of

anywellb ehaved function ut and its derivative u t With this in mind let

t

us dene the linear multistep dierence op erator L acting on the set of

continuously dierentiable functions ut by

L Z k D Z

where D is the time dierentiation op erator that is

Lut Z ut k Z u t

n n t n

s s

X X

u t ut k

j t nj j nj

j j

If the linear multistep formula is accurate Lut ought to be small and we

n

make this precise as follows Let a function ut and its derivative u t be

t

expanded formally in Taylor series ab out t

n

ut ut jku t jk u t

nj n t n tt n

u t u t jku t jk u t

t nj t n tt n ttt n

Inserting these formulas in gives the formal lo cal discretization

error or formal lo cal truncation error for the linear multistep formula

Lut C ut C ku t C k u t

n n t n tt n

where

C

s

s C

s s

C s s

s

s

s s

m m

X X

j j

C

m j j

m m

j j

We now dene

A linear multistep formula has order of accuracy p if

p

Lut k as k

n

ie if C C C but C The error constant is C

p p p

The formula is consistent if C C ie if it has order of accuracy

p

ACCURACY AND CONSISTENCY TREFETHEN

EXAMPLE CONTINUED First let us analyze the lo cal accuracy of the Euler

formula in a lowbrow fashion that is equivalent to the denition ab ove we supp ose that

n n

v v are exactly equal to ut ut and ask how close v will then b e to ut

n

n

n

If v ut exactly then by denition

n

n n n

v v kf ut ku t

n t n

while the Taylor series gives

k

ut ku t ut u t O k

n n t n tt n

Subtraction gives

k

n

u t O k ut v

tt n n

as the formal lo cal discretization error of the Euler formula

Now let us restate the argument in terms of the op erator L By combining

and or by calculating C C C C from the values

in we obtain

Lut ut ut ku t

n n n t n

k k

u t u t

tt n ttt n

Thus again we see that the order of accuracy is and evidently the error constantis

Similarly the trap ezoid rule has

k

u t u t Lut ut ut

t n t n n n n

k k

u t u t

ttt n tttt n

with order of accuracy and error constant and the midp oint rule has

k u t k u t Lut

ttt n tttt n n

with order of accuracy and error constant

The idea b ehind the denition of order of accuracy is as follows Equation

suggests that if a linear multistep formula is applied to a problem with a

p

d u

p

suciently smo oth solution ut an error of approximately C k t

p

p n

dt

p

O k will be intro duced lo cally at step n This error is known as the

lo cal discretization or truncation error at step n for the given initial

value problem as opp osed to the formal lo cal discretization error which is a

As usual the limit asso ciated with the BigO symbol k is omitted here b ecause it is obvious

ACCURACY AND CONSISTENCY TREFETHEN

formal expression that do es not dep end on the initialvalue problem Since

there are Tk k steps in the interval T these lo cal errors can be

p

exp ected to add up to a global error O k We shall make this argument

more precise in Section

With hindsight it is obvious by symmetry that the trap ezoid and mid

pointformulas had to haveeven orders of accuracy Notice that in the

nj nj

terms involving v are antisymmetric ab out t and those involving f

n

are symmetric In the eect of these symmetries is disguised b ecause

t was shifted to t for simplicity of the formulas in passing from to

n

n

and However if we had expressed Lut as a Taylor series

n

ab out t instead we would have obtained

n

Lut k u t k u t O k

n ttt ttttt

n n

with all the evenorder derivatives in the series vanishing due to symmetry

Now it can be shown that to leading order it do esnt matter what p oint one

expands Lut ab out C C are indep endent of this p oint though the

n

p

subsequent co ecients C C are not Thus the vanishing of the even

p p

order terms ab ove is a valid explanation of the secondorder accuracy of the

midp oint formula For the trap ezoid rule we can make an analogous

argument involving an expansion of Lut ab out t

n

n

As a rule symmetry arguments do not go very far in the analysis of

discrete metho ds for ODEs since most of the formulas used in practice are of

high order and fairly complicated They go somewhat further with the simpler

formulas used for PDEs

The denition of order of accuracy suggests a metho d of undetermined

co ecients for deriving linear multistep formulas having decided somehow

which and are p ermitted to b e nonzero simply adjust these parameters

j j

to make p as large as p ossible If there are q parameters not counting the

xed value then the equations with m q constitute a

s

q q linear system of equations If this system is nonsingular as it usually is

it must have a unique solution that denes a linear multistep formula of order

at least p q See Exercise

The metho d of undetermined co ecients can also b e describ ed in another

equivalentway that was hinted at in Exercise It can b e shown that any

consistent linear multistep formula computes the solution to an initialvalue

problem exactly in the sp ecial case when that solution is a p olynomial pt

of degree L provided that L is small enough The order of accuracy p is the

largest such L see Exercise To derive a linear multistep formula for a

particular choice of available parameters and onecancho ose the param

j j

eters in suchaway astomake the formula exact when applied to p olynomials

of as high a degree as p ossible

ACCURACY AND CONSISTENCY TREFETHEN

At this p oint it may app ear to the reader that the construction of linear

multistep formulas is a rather uninteresting matter Given s why not sim

ply pick and so as to achieve order of accuracy s The

s

s

answer is that for s the resulting formulas are unstable and consequently

useless see Exercise In fact as we shall see in Section a famous

theorem of Dahlquist asserts that a stable sstep linear multistep formula can

have order of accuracy at most s Consequently nothing can b e gained by

p ermitting all s co ecients in an sstep formula to be nonzero Instead

the linear multistep formulas used in practice usually have most of the or

j

most of the equal to zero In the next section we shall describ e several

j

imp ortant families of this kind

And there is another reason why the construction of linear multistep for

mulas is interesting It can be made exceedingly slick We shall now describ e

how the formulas ab ove can b e analyzed more compactly and the question of

order of accuracy reduced to a question of rational approximation by manip

ulation of formal p ower series

Taking j in gives the iden tity

ut ut ku t k u t

n n t n tt n

Since ut Zut here is another way to express the same fact

n

n

k D

Z k D k D k D e

Like this formula is to be interpreted as a formal identity the idea

is to use it as a to ol for manipulation of terms of series without making any

claims ab out convergence Inserting in and comparing with

gives

k D k D

L e k D e C C k D C k D

In other words the co ecients C of are nothing else than the Taylor

j

k D k D

series co ecients of the function e k D e with resp ect to the argu

ment k D If we let be an abbreviation for k D this equation becomes even

simpler

L e e C C C

With the aid of a symb olic algebra system such as Macsyma Maple or Mathe

matica it is a trivial matter to make use of to compute the co ecients

C for a linear multistep formula if the parameters and are given See

j j j

Exercise

By denition a linear multistep formula has order of accuracy p if and

p

only if the the term between the equal signs in is as

ACCURACY AND CONSISTENCY TREFETHEN

Since e is an analytic function of if it is nonzero at we can divide

through to conclude that the linear multistep formula has order of accuracy p

if and only if

e

p

as

e

The following theorem restates this conclusion in terms of the variable z e

LINEAR MULTISTEP FORMULAS AND RATIONAL APPROXIMATION

Theorem A linear multistep formula with has order of

accuracy p if and only if

z

p

log z z

z

i h

p

z z z z

as z It is consistent if and only if

and

Pro of To get from to the rst equality of we make the

p

change of variables z e log z noting that as has the same

p

meaning as z as z since e and de d at The

second equality of is just the usual Taylor series for log z

To prove let b e written in the form

p

z z z O z z

or by expanding z and z ab out z

p

z z O z z

Matching successive powers of z we obtain p and p

p Thus is equivalent to p which is the

denition of consistency

In Theorem the ODE context of a linear multistep formula has van

ished entirelyleaving a problem in the mathematical eld known as approxi

mation theory Questions of approximation underlie most discrete metho ds

for dierential equations b oth ordinary and partial

ACCURACY AND CONSISTENCY TREFETHEN

EXAMPLE The trap ezoid rule has z z and z z Since



and the formula is consistent Comparing with the expan

sion

z z z z z

z

z

z z

conrms that the trap ezoid rule has order and error constant

This approach to linear multistep formulas by rational approximation is

closely related to the metho ds of generating functions describ ed in several

of the references It is also the basis of the metho d of analysis by order stars

describ ed later in this chapter For ordinary dierential equations with sp ecial

structure such as highly oscillatory b ehavior it is sometimes advantageous to

approximate log z atoneormorepoints other than z this is the idea b ehind

the frequencytted or mo dedep endent formulas discussed by Gautschi

Liniger and Willoughby Kuo and Levy and others See

Exercise

EXERCISES

Showby the metho d of undetermined co ecients that

n n n n n

v v v k f f

is the most accurate step explicit linear multistep formula with order of accuracy p

In Section we shall see that is unstable and hence useless in practice

Consider the thirdorder backwards dierentiation formula

a What are z and z

b Apply Theorem to verify consistency

c Apply Theorem to verify that the order of accuracy is

Optimal formulas with nite step size The concept of order of accuracy is based

on the limit k but one can also devise formulas on the assumption of a nite step size

k For example an Eulerlikeformula might b e dened by

n n n

v v k kf

for some function k with k ask

a What choice of k makes exact when applied to the equation u u

t

b ODEs of practical interest contain various time scales so it is not really appropriate

to consider just u u Supp ose the goal is to approximate all problems u au with

t t

a as accurately as p ossible State a denition of as accurately as p ossible



based on the L norm of the error over a single time step and determine the resulting

function k

ACCURACY AND CONSISTENCY TREFETHEN

Numerical exp eriments Consider the scalar initialvalue problem

costut

u te for t u

t

In this problem you will test four numerical metho ds i Euler ii Midp oint iii Fourth

order AdamsBashforth and iv Fourthorder RungeKutta dened by

n

a kf v t

n

n

b kf v at k

n

n

c kf v bt k

n

n

k d kf v c t

n

n n

v v a b c d

a Write a computer program to implement in high precision arithmetic digits

or more Run it with k until you are condentthatyou have a computed

value v accurate to at least digits This will serveasyour exact solution Make

a computer plot or a sketchof v t Store appropriate values from your RungeKutta

computations for use as starting values for the multistep formulas ii and iii

b Mo dify your program so that it computes v byeach of the metho ds iiv for the

sequence of time steps k Make a table listing v and v u

for each metho d and each k

c Draw a plot on a loglog scale of four curves representing j v uj as a function of

the number of evaluations of f for each of the four metho ds Makesureyou calculate

n

each f only once and count the numb er of function evaluations for the RungeKutta

formula rather than the numb er of time steps

d What are the approximate slop es of the lines in c and why If you cant explain

them there may be a bug in your programvery likely in the sp ecication of initial

conditions Which of the four metho ds is most ecient

e If you are programming in Matlab solve this same problem with the programs ode

and ode with eight or ten dierent error tolerances Measure howmany time steps

are required for each run and howmuch accuracy is achieved in the value u and add

these new results to your plot of c What are the observed orders of accuracy of these

adaptivecodes How do they compare in eciency with your nonadaptive metho ds

iiv

p

Statistical eects It was stated ab ove that if lo cal errors of magnitude O k are

 p

made at eachofk steps then the global error will have magnitude O k However

one might argue that more likely the lo cal errors will b ehave like random numbers of

p

and will accumulate in a squarero ot fashion according to the principles of order O k

p

a random walk giving a smaller global error O k Exp eriments show that for most

problems including those of Example and Exercise this optimistic prediction is

invalid What is the fallacy in the random walk argument Be sp ecic citing an equation

in the text to supp ort your answer

Prove the lemma alluded to on p An sstep linear multistep formula has order of

accuracy p if and only if when applied to an ordinary dierential equation u q titgives

t

exact results whenever q is a p olynomial of degree p but not whenever q is a p olynomial

ACCURACY AND CONSISTENCY TREFETHEN

of degree p Assume arbitrary continuous initial data u and exact numerical initial

s

data v v

If you have access to a symbolic computation system carry out the suggestion on

p write a short program which given the parameters and for a linear multistep

j j

formula computes the co ecients C C Use your program to verify the results of

Exercises and and then explore other linear multistep formulas that interest

you

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

Derivation of linear multistep formulas

The oldest linear multistep formulas are the Adams formulas

s

X

ns ns nj

v v k f

j

j

which date to the work of J C Adams as early as In the notation of we

have and and the rst characteristic p olynomial

s s s

s s

is z z z For each s the sstep AdamsBashforth and AdamsMoulton

formulas are the optimal explicit and implicit formulas of this form resp ectively Optimal

means that the available co ecients f g are chosen to maximize the order of accuracy and

j

in b oth Adams cases this choice turns out to b e unique

Wehave already seen the step AdamsBashforth and AdamsMoulton formulas they

are Eulers formula and the trap ezoid formula resp ectively The fourthorder

AdamsBashforth and AdamsMoulton formulas with s and s resp ectively were

listed ab ove as and The co ecients of these and other formulas are listed in

Tables on the next page The stencils of various families of linear multistep

formulas are summarized in Figure which should b e selfexplanatory

To calculate the co ecients of Adams formulas there is a simpler and more enlight

ening alternative to the metho d of undetermined co ecients mentioned in the last section

n ns n ns

Think of the values f f AB or f f AM as discrete samples of a

continuous function f tf uttthatwewanttointegrate

Z Z

t t

ns ns

ut ut u t dt f t dt

ns ns t

t t

ns ns

n ns

as illustrated in Figure a Of course f f will themselves be inexact due to

earlier errors in the computation but we ignore this for the moment Let q t b e the unique

p olynomial of degree at most s AB or s AM that interp olates these data and set

Z

t

ns

ns ns

v v q tdt

t

ns

nj

Since the integral is a linear function of the data ff g with co ecients that can be

computed once and for all implicitly denes a linear multistep formula of the Adams

typ e

EXAMPLE Let us derive the co ecients of the ndorder AdamsBashforth for

n

mula which are listed in Table In this case the data to b e interp olated are f and the same Adams who rst predicted the existence of the planet Neptune

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

Adams Adams Generalized Backwards

Bashforth Moulton Nystrom MilneSimpson Dierentiation

j

j j j j j j j j j j

n s

n s

n s

n

Figure Stencils of various families of linear multistep formulas

number

of steps s order p

s s s s s

EULER

Table Co ecients f g of AdamsBashforth formulas

j

number

of steps s order p

s

s s s s

BACKWARD EULER

TRAPEZOID

Table Co ecients f g of AdamsMoulton formulas

j

number

of steps s order p

s s

s s s s

BACKWARD EULER

Table Co ecients f g and of backwards dierentiation formulas

s j

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

n n  n n

f and the interp olant is the linear p olynomial q tf k f f t t

n

Therefore b ecomes

Z

t

n

n  n n n n

f k f f t t dt v v

n

t

n

Z

t

n

n  n n

t tdt kf k f f

n

t

n

n  n n

k kf k f f

n n

kf kf

n

ns

f v *o *o *o

o

*o *

q t

s

n o

o f *

s

* n n

o f

v

t q *

*o *o

t t t

ns ns ns

a Adams b backwards dierentiation

Figure Derivation of Adams and backwards dierentiation

formulas via p olynomial interp olation

More generally the co ecients of the interp olating p olynomial q can be

determined with the aid of the Newton interp olation formula To b egin

n

with consider the problem of interp olating a discrete function fy g in the

points by a p olynomial q tofdegree at most Let and r denote

the forward and backward dierence op erators

Z r Z

where represents the identity op erator For example

n n n n n n n

y y y y y y y

a

Also let denote the binomial co ecient a c ho ose j

j

a aa a a j

j j

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

dened for integers j and arbitrary a C The following is a standard result

that can be found in many books of numerical analysis and approximation

theory

NEWTON INTERPOLATION FORMULA

Theorem The p olynomial

t t t

q t y

is the unique p olynomial of degree at most that interp olates the data

y y in the points

t

Pro of First of all from it is clear that is a monomial of degree

j

j and since describ es a linear combination of such terms with j

q tis evidently a p olynomial of degree at most

We need to show that q t interp olates y y To this end note that

Z and therefore

j j j

j j j

Z

j

for any integer j the binomial formula If j we may equally well

extend the series to term

j j j

j

Z

j

for mj By taking t j in this identity implies that since

m

j

q j Z y for j In other words q tinterp olates the data as required

Finally uniqueness of the interp olating p olynomial is easy to prove If

q t and q t are two p olynomials of degree that interp olate the data

then q q is p olynomial of degree that vanishes at the interp olation

points which implies q q identically since a nonzero p olynomial of

degree can have at most zeros

We want to apply Theorem to the derivation of AdamsBashforth

formulas To do this we need a version of the theorem that is normalized

dierently and considerably uglier though equivalent Let the points

be replaced by the points t t Then from Theorem or by a pro of

from scratch one can readily show that the p olynomial

tk tk tk

q t r r r y

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

is the unique p olynomial of degree at most that interp olates the data y

y in the p oints t t Note that among other changes has b een

replaced by r

Now let us replace y by f by s and t by t t hence t t

ns

by t t Equation then b ecomes

n ns

t tk t tk

n s ns

s s ns

q t r r f

s

Inserting this expression in gives

s

X

j ns ns ns

r f v v k

j

j

where

Z Z

j

t

ns

t tk

ns

j

dt d

j

j j k

t

ns

The rst few values are

j

The following theorem summarizes this derivation and some related facts

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

ADAMSBASHFORTH FORMULAS

Theorem For any s the sstep AdamsBashforth formula has

order of accuracy sand is given by

Z

s

X

j ns j ns ns

r f v v k d

j j

j

j

For j the co ecients satisfy the recurrence relation

j

j j j

j

Pro of We have already shown ab ove that the co ecients corre

sp ond to the linear multistep formula based on p olynomial interp olation To

prove the theorem we must show three things more that the order of accu

racy is as high as p ossible so that these are indeed AdamsBashforth formulas

that the order of accuracy is s and that holds

The rst claim follows from the lemma stated in Exercise If f u t

is a p olynomial in t of degree sthenq tf u t in our derivation ab ove so

the formula gives exact results and its order of accuracy is accordingly

s On the other hand any other linear multistep formula with dierent

co ecients would fail to integrate q exactly since p olynomial interp olants are

unique and accordingly would have order of accuracy s Thus is

indeed the AdamsBashforth formula

The second claim can also b e based on Exercise If the sstep Adams

Bashforth formula had order of accuracy sitwould b e exact for any problem

u q t with q t equal to a p olynomial of degree s But there are nonzero

t

s

p olynomials of this degree that interp olate the values f f from

whichwe can readily derivecounterexamples in the form of initialvalue prob

lems with v t but u t

s s

Finally for a derivation of the reader is referred to Henrici

or Hairer Nrsett Wanner

EXAMPLE CONTINUED To rederive the ndorder AdamsBashforth formula

directly from we calculate

n n n n n n

n n

v v k f k f f v k f f

EXAMPLE To obtain the thirdorder AdamsBashforth formula we increment n

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

n

to n in the formula ab ove and then add one more term k r f to get

n n

n n n n n

v v k f f k f f f

n

n n n

f f f v k

which conrms the result listed in Table

For AdamsMoulton formulas the derivation is entirely analogous We

have

s

X

ns ns j ns

v v k r f

j

j

where

Z Z

j

t

ns

t tk

ns

j

dt d

j

k j j

t

ns

are and the rst few values

j

Notice that these numb ers are smaller than b efore an observation which is

related to the fact that AdamsMoulton formulas generally have smaller error

constants than the corresp onding AdamsBashforth formulas

The analog of Theorem is as follows

ADAMSMOULTON FORMULAS

Theorem For any s the sstep AdamsMoulton formula has

order of accuracy s and is given by

Z

s

X

ns ns j ns j

v v k r f d

j j

j

j

For j the co ecients satisfy the recurrence relation

j

j j j

j



The step AdamsMoulton formula of this theorem actually has s as indicated in Table

b ecause of the nonzero co ecient

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

Pro of Analogous to the pro of of Theorem

Both sets of co ecients f g and f g can readily b e converted into co ef

j j

cients f g of our standard representation and the results for s

j

are listed ab ove in Tables and

Besides Adams formulas the most imp ortant family of linear multistep

formulas dates to Curtiss and Hirschfelder in and is also asso ciated with

the name of C W Gear The sstep backwards dierentiation formula

is the optimal implicit linear multistep formula with An

s

ards dif example was given in Unlike the Adams formulas the backw

ferentiation formulas allo cate the free parameters to the f g rather than the

j

f g These formulas are maximally implicit in the sense that the function

j

f enters the calculation only at the level n For this reason they are the

hardest to implement of all linear multistep formulas but as we shall see in

xx they are also the most stable

To derive the co ecients of backwards dierentiation formulas one can

again make use of p olynomial interp olation Now however the data are sam

ples of v rather than of f as suggested in Figure b Let q be the unique

n ns ns

p olynomial of degree s that interp olates v v The number v is

unknown but it is natural to dene it implicitly by imp osing the condition

ns

q t f

t ns

Like the integral in the derivative in represents a linear func

tion of the data with co ecients that can be computed once and for all and

so this equation constitutes an implicit denition of a linear multistep formula

The co ecients for s were listed ab ove in Table

To convert this prescription into numerical co ecients one can again

apply the Newton interp olation formula see Exercise However the

pro of b elowisaslicker one based on rational approximation and Theorem

BACKWARDS DIFFERENTIATION FORMULAS

Theorem For any s the sstep backwards dierentiation formula

has order of accuracy s and is given by

s

X

j ns ns

r v kf

j

j

Note that is not quite in the standard form it must be

ns

normalized by dividing by the co ecient of v

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

s

Pro of Let z and z z be the characteristic p olynomials corre

sp onding to the sstep backwards dierentiation formula Again we have

normalized dierently from usual By Theorem since log z log z

the order of accuracy is p ifandonlyif

z

p

log z z

s

z

i h

p

z z z z

that is

h i

s p

z z z z z z

By denition z is a p olynomial of degree at most s with equiv

s

alently it is z times a p olynomial in z of degree exactly s Since z

maximizes the order of accuracy among all such p olynomials the last formula

makes it clear that we must have

s s

z z z z

s

with order of accuracy s This is precisely

These three families of linear multistep formulasAdamsBashforth Ad

amsMoulton and backwards dierentiationare the most imp ortant for prac

tical computations Other families however havealsobeendevelop ed over the

years The sstep Nystrom formula is the optimal explicit linear multistep

s s

formula with z z z that is and otherwise

s

s

j

The sstep generalized MilneSimpson formula is the optimal implicit lin

ear multistep formula of the same typ e Co ecients for these formulas can b e

obtained by the same pro cess describ ed in Figure a except that now q t

is integrated from t to t Like the Adams and backwards dierenti

ns

ns

ation formulas the sstep Nystrom and generalized MilneSimpson formulas

have exactly the order of accuracy one would exp ect from the number of free

parameters s and s resp ectively with one exception the generalized

MilneSimpson formula with s has order not This formula is known

as the Simpson formula for ordinary dierential equations

n n n n n

v v k f f f

DERIVATION OF LINEAR MULTISTEP FORMULAS TREFETHEN

EXERCISES

Secondorder backwards dierentiation formula Derive the co ecients of the step

backwards dierentiation formula in Table

a By the metho d of undetermined co ecients

b By Theorem making use of the expansion z z z

c By interp olation making use of Theorem

Backwards dierentiation formulas Derive from Theorem

Thirdorder Nystrom formula Determine the co ecients of the thirdorder Nystrom

formula byinterp olation making use of Theorem

Quadrature formulas What happ ens to linear multistep formulas when the function

f u t is indep endent of u To be sp ecic what happ ens to Simpsons formula

v in an integration of Comment on the eect of various strategies for initializing the p oint

such a problem by Simpsons formula

STABILITY TREFETHEN

Stability

It is time to intro duce one of the central themes of this b o ok stability Before the

word stability rarely if ever app eared in pap ers on numerical metho ds but by at which

time computers were widely distributed its imp ortance had b ecome universally recognized

Problems of stability aect almost every numerical metho d for solving dierential equations

and they must b e confronted headon

For b oth ordinary and partial dierential equations there are twomainstability ques

tions that haveproved imp ortantover the years

Stabilityy If t is held xed do the computed values v t remain b ounded as k

Eigenvalue stability If k is held xed do the computed values v t remain

b ounded as t

These two questions are related but distinct and each has imp ortant applications We shall

wing section and the second in xx consider the rst in this and the follo

The motivation for all discussions of stability is the most fundamental question one

could ask ab out a numerical metho d will it give the rightanswer Of course one can never

exp ect exact results so a reasonable way to make the question precise for the initialvalue

problem is to ask if t is a xed numb er and the computation is p erformed with

various step sizes k in exact arithmetic will v tconverge to utask

A natural conjecture might be that for any consistent linear multistep formula the

answer must b e yes After all as p ointed out in x such a metho d commits lo cal errors of

p 

size O k withp and there are a total of k timesteps But a simple argument

shows that this conjecture is false Consider a linear multistep formula based purely on

n

extrap olation of previous values fv gsuchas

n n n

v v v

This is a rstorder formula with z z and z and in fact we can construct

s

extrap olation formulas of arbitrary order of accuracy by taking z z Yet sucha

formula cannot p ossibly converge to the correct solution for it uses no information from the

dierential equation

Thus lo cal accuracy cannot b e sucient for convergence The surprising fact is that

accuracy plus an additional condition of stability is sucient and necessary to o In fact this

is a rather general principle of numerical analysis rst formulated precisely by Dahlquist for

ordinary dierential equations and by Lax and Richtmyer for partial dierential equations

b oth in the s Strang calls it the fundamen tal theorem of numerical analysis

See G Dahlquist years of numerical instability BIT

y Stabilityisalsoknown as zerostability or sometimes Dstability for ODEs and Lax stability

or LaxRichtmyer stability for PDEs Eigenvalue stability is also known as weak stability or

absolute stability for ODEs and timestability practical stability or Pstability for PDEs

The reason for the word eigenvalue will b ecome apparentinxx

p

Where exactly did the argument suggesting global errors O k break down Try to pinp ointit

yourself the answer is in the next section

STABILITY TREFETHEN

Theorem in the next section gives a precise statement for the case of linear multistep

formulas and for partial dierential equations see Theorem

We b egin with a numerical exp eriment

EXAMPLE In the spirit of Example supp ose we solve the initialvalue

problem

u u t u

t

by three dierent step explicit metho ds the extrap olation formula the second

order AdamsBashforth formula and the optimal step formula Again we

nk

take exact quantities e where needed for starting values Figure shows the computed

functions v tfork and In the rst plot b oth of the higherorder formulas app ear

to be p erforming satisfactorily In the second however large oscillations have begun to

develop in the solution based on Obviously is useless for this problem

Table makes this b ehavior quantitativeby listing the computed results v for k

As k decreases the solution based on converges to an incorrect

solution namely the function t and the solution based on diverges explosively

Notice that none of the numb ers are preceded bya In this example the instabilityhas

b een excited by discretization errors not rounding errors

Figure gives a fuller picture of what is going on in this example by plotting the

error jv ej as a function of k on a log scale with smaller values of k to the right for the

same three linear multistep formulas as well as the fourthorder AdamsBashforth formula

The two AdamsBashforth formulas exhibit clean secondorder and fourthorder

convergence as one would exp ect showing in the plot as lines of slop e approximately and

resp ectively The extrap olation formula exhibits zerothorder convergencein

other words div ergence with the error approaching the constant e The formula

diverges explosively

It is not dicult to see what has gone wrong with For simplicity assume k

is negligible Then b ecomes

n n n

v v v

n n

a secondorder recurrence relation for fv g It is easy to verify that both v and

n n n

v are solutions of Caution the n in v is a sup erscript but the n in

n

is an exp onent Since an arbitrary solution to is determined by two initial

values v and v itfollows that any solution can b e written in the form

n n n

v a b

for some constants a and b What has happ ened in our exp erimentisthatb has ended up

n

nonzerothere is some energy in the mo de and an explosion has taken place In

n

fact if the computation with k is carried out to one more time step v takes the

value whichis times the nal quantitylistedin the table So the

assumption that k was negligible was not to o bad

Although Example is very simple it exhibits the essential mechanism

of instability in the numerical solution of ordinary dierential equations by

linear multistep formulas a recurrence relation that admits an exp onentially

STABILITY TREFETHEN

e e

AB

t t

a k b k

Figure Solution of by three explicit twostep linear multistep for

mulas The highorder formula lo oks go o d at rst but b ecomes unstable

when the time step is halved

k k k k

Extrap olation

ndorder AB

optimal

Table Computed values v e for the initialvalue problem

 

k

AB

AB



Figure Error jv ej as a function of time step k for the solution of

by four explicit linear multistep formulas

STABILITY TREFETHEN

n

growing solution z for some jz j Such a solution is sometimes known

as a parasitic solution of the numerical metho d since it is intro duced by

the discretization rather than the ordinary dierential equation itself It is a

general principle that if such a mo de exists it will almost always b e excited by

either discretization or rounding errors or by errors in the initial conditions Of

course in principle the co ecient b in might turn out to b e identically

zero but in practice this p ossibility can usually b e ignored Even if b were zero

at t itwould so on b ecome nonzero due to variable co ecients nonlinearity

n

or rounding errors and the z growth would then take over so oner or later

The analysis of Example can be generalized as follows Given any

linear multistep formula consider the asso ciated recurrence relation

s

X

nj n

v Z v

j

j

obtained from with k We dene

n

A linear multistep formula is stable if all solutions fv g of the recurrence

relation are b ounded as n

n

This means that for any function fv g that satises there exists a con

n

stant M such that jv j M for all n We shall refer to the recurrence

relation itself as stable as well as the linear multistep formula

There is an elementary criterion for determining stability of a linear mul

tistep formula based on the characteristic p olynomial

ROOT CONDITION FOR STABILITY

Theorem A linear multistep formula is stable if and only if all the

ro ots of z satisfy jz j and any ro ot with jz j is simple

A simple ro ot is a ro ot of multiplicity

First pro of To prove this theorem we need to investigate all p ossible

n

solutions fv g n of the recurrence relation Since any such solu

s

tion is determined by its initial values v v the set of all of them is a

A general observation is that when any linear pro cess admits an exp onentially growing solution in

theorythatgrowth will almost invariably app ear in practice Only in nonlinear situations can the

existence of exp onentially growing mo des fail to make itself felt A somewhat farung example

comes from numerical linear algebra In the solution of an N N matrix problems by Gaussian

elimination with partial pivotinga highly nonlinear pro cessan explosion of rounding errors at

N 

the rate can o ccur in principle but almost never do es in practice Trefethen Schreib er

Averagecase stability of Gaussian elimination SIAM J Matrix Anal Applics

STABILITY TREFETHEN

vector space of dimension s Therefore if we can nd s linearly indep endent

n

solutions v these will form a basis of the space of all p ossible solutions and

the recurrence relation will b e stable if and only if each of the basis solutions

is b ounded as n

It is easy to verify that if z is any ro ot of z then

n n

v z

is one solution of Again on the left n is a sup erscript and on the

right it is an exp onent If z we dene z If has s distinct ro ots

then these functions constitute a basis Since each function is b ounded

if and only if jz j this proves the theorem in the case of distinct ro ots

On the other hand supp ose that z has a ro ot z of multiplicity m

Then it can readily be veried that each of the functions

n n n n n m n

v nz v n z v n z

is an additional solution of and clearly they are all linearly indep endent

since degreem p olynomial interp olants in m points are unique to say

j n

nothing of points If z we replace n z by the function that takes the

value at n j and elsewhere These functions are b ounded if and only if

jz j and this nishes the pro of of the theorem in the general case

Alternative pro of based on linear algebra The pro of ab ove is simple and

complete but there is another way of lo oking at Theorem that involves a

technique of general imp ortance Let us rewrite the sstep recurrence relation

as a step matrix op eration on vectors v of length s

n n

v v

C B

B C B C

n n

C B

B C B C

v v

C B

B C B C

C B

B C B C

C B

B C B C

C B

B C B C

C B

B C B C

C B

B C B C

C B

B C B C

C B

A A

A

ns ns

v v

s s

That is

n n

v Av

or after n steps

n n

v A v

n

where A denotes the nth p ower of the matrix A This is a discrete analog of

the reduction of higherorder ODEs to rstorder systems describ ed in x

STABILITY TREFETHEN

n

The scalar sequence fv g will b e bounded as n if and only if the vector

n n

sequence fv g is b ounded and fv g will in turn b e b ounded if and only if the

n

elements of A are b ounded Thus we have reduced stabilityto a problem of

growth or b oundedness of the powers of a matrix

A matrix A of the form is known as a companion matrix and

where z is dened by one can verify that detzI Az for any z

as usual In other words the characteristic p olynomial of the matrix

A is the same as the characteristic p olynomial of the linear multistep formula

Therefore the set of eigenvalues of A is the same as the set of ro ots of

n

and these eigenvalues determine how the powers A b ehave asymptotically

as n To make the connection precise one can lo ok at the similarity

transformation that brings A into Jordan canonical form

A SJ S

Here S is an s s nonsingular matrix and J is an s s matrix consisting of all

zeros except for a set of Jordan blo cks J along the diagonal with the form

i

z

i

B C

z

B C

i

B C

B C

B C

B C

B C

J

i

B C

B C

B C

B C

B C

A

z

i

z

i

where z is one of the eigenvalues of A Every matrix has a Jordan canonical

i

form and for matrices in general each eigenvalue may app ear in several Jordan

blo cks For a companion matrix however there is exactly one Jordan blo ckfor

each eigenvalue with dimension equal to the multiplicity m of that eigenvalue

i

Pro of zI A has rank s forany z since its upp errights s

blo ckisobviously nonsingular Such a matrix is called nonderogatory Now

the powers of A are

n n

A SJ S SJ S SJ S

so their growth or b oundedness is determined by the growth or b oundedness

s T

One waytoverify it is to lo ok for eigenvectors of the form zz where z is the eigenvalue

STABILITY TREFETHEN

n

of the powers J which can be written down explicitly

nm

n n

n

n

i

z z z

i

i i

m

C B

i

C B

C B

n

n

n

C B

z z

i

i

C B

C B

C B

C B

C B

n

J C B

i

C B

C B

C B

C B

n

n

n

C B

z z

i

i

C B

C B

A

n

z

i

If jz j these elements approach as n If jz j they approach

i i

If jz j they are b ounded in the case of a blo ck but unb ounded if

i

m

i

The reader should study Theorem and b oth of its pro ofs until he or she

is quite comfortable with them These tricks for analyzing recurrence relations

come up so often that they are worth rememb ering

EXAMPLE Let us test the stabilityofvarious linear multistep formulas considered

s s

up to this point Any AdamsBashforth or AdamsMoulton formula has z z z

with ro ots f g so these metho ds are stable The Nystrom and generalized Milne

s s

Simpson formulas have z z z with ro ots f g so they are stable to o

The scheme that caused trouble in Example has z z z with ro ots

f g so it is certainly unstable As for the less dramatically unsuccessful formula

it has z z z z withamultiple ro ot f gsoitcounts as unstable to o

n

since it admits the growing solution v n The higherorder extrap olation formula dened

s s

by z z z admits the additional solutions n n n making for an

instability more pronounced but still algebraic rather than exp onential

Note that by Theorem any consistent linear multistep formula has a

ro ot of z at z and if the formula is stable then Theorem ensures

that this ro ot is simple It is called the principal ro ot of for it is the

one that tracks the dierential equation The additional consistency condition

of Theorem amounts to the condition that if z is p erturb ed

away from the principal ro ot b ehaves correctly to leading order

In x an analogy was mentioned b etween linear multistep formulas and

In digital signal pro cessing one demands jz j for recursive digital lters

stability why then do es Theorem contain the weaker condition jz j

The answer is that unlike the usual lters an ODE integrator must remember

the past even for values of t where f is zero u is in general nonzero If all

ro ots z satised jz j the inuence of the past would die away exp onentially

STABILITY TREFETHEN

Theorem leaves us in a remarkable situation summarized in Figure

By Theorem consistency is the condition that z z matches

log z to at least second order at z By Theorem stability is the con

dition that all ro ots of z lie in jz j with simple ro ots only p ermitted

on jz j Thus the two crucial prop erties of linear multistep formulas hav e

b een reduced completely to algebraic questions concerning a rational function

The pro ofs of many results in the theory of linear multistep formulas such as

Theorems and b elow consist of arguments of pure complex analysis of

rational functions having nothing sup ercially to do with ordinary dierential

equations

STABILITY zeros of z in unit disk simple if on unit circle

ORDER OF ACCURACY p

z

p

log z z

z

CONSISTENCY p

Figure Stability consistency and order of accuracy as alge

braic conditions on the rational function z z

The following theorem summarizes the stability of the standard families

of linear multistep formula that we have discussed

STABILITY OF STANDARD LINEAR MULTISTEP FORMULAS

Theorem The sstep AdamsBashforth AdamsMoulton Nystrom

and generalized MilneSimpson formulas are stable for all s The sstep

backwards dierentiation formulas are stable for s but unstable for

s

Pro of The results for the rst four families of linear multistep formulas

follow easily from Theorem as describ ed in Example The analy

sis of backwards dierentiation formulas is a more complicated matter since

STABILITY TREFETHEN

the p olynomials z are no longer trivial Instability for s was recog

nized numerically in the s Mitchell and Craggs but not proved

mathematically until Cryer An elegant short pro of of instability

for s was devised by Hairer and Wanner in and can be found on

pp of Hairer Nrsett Wanner

To close this section we shall present an imp ortant result that is known as

the rst Dahlquist stability barrier In x we mentioned that an sstep

linear multistep formula can have order s why not simply use that high

order formula and forget sp ecial classes like Adams and Nystrom metho ds

Dahlquists famous theorem conrms that the answer is an impassable barrier

of stability

FIRST DAHLQUIST STABILITY BARRIER

Theorem The order of accuracy p of a stable sstep linear multistep

formula satises

s if s is even

p s if s is o dd

s if the formula is explicit

Pro of Various pro ofs of Theorem have b een published b eginning with

the original one by Dahlquist in More recent pro ofs havebeenbased on

the b eautiful idea of order stars intro duced byWanner Hairer and Nrsett

BIT The following argument is adapted from A pro of of the rst

Dahlquist barrier by order stars by A Iserles and S P Nrsett BIT

Though fundamentally correct it is somewhat casual ab out geometric details

for a more detailed treatment see that pap er or the book Order Stars by the

same authors

Let z and z be the characteristic p olynomials corresp onding to a

stable sstep linear multistep formula of order of accuracy p The k ey idea is

to lo ok at level lines of the function wehave b een dealing with since

z

z log z

z

This function is analytic throughout the complex plane except near zeros of

z provided one intro duces a branch cut going to the point z Alter

z

natively one can work with e and eliminate the need for a branch cut

In particular z is analytic at z If z and z have a

STABILITY TREFETHEN

Figure Order star dened by for the thorder Adams

Bashforth formula The zeros of z at z are marked by o and

the zeros of z are marked by The fold daisy at z reects

the order of accuracy of this formula If there were a b ounded

shaded nger that did not intersect the unit disk dashed the

formula would b e unstable

common factor z and the linear multistep formula is not in simplest form

By Theorem its behavior near there is

p p

z C z O z

for C C Now let A the order star for the linear multistep

p

formula b e the set dened by

A fz C Re z g

In other words it is the inverse image of the right halfplane under Figure

shows the order star for the thorder AdamsBashforth formula

The power of order stars comes from their ability to couple lo cal and

global prop erties of a function by way of geometric arguments The global

STABILITY TREFETHEN

prop erty that concerns us is stability the zeros of z must lie in the unit

disk The lo cal prop erty that concerns us is order of accuracy from

it follows that near z A must lo ok like a daisy with p evenly spaced

p etalsor ngers as they are sometimes called In Figure the number

of ngers is

Since the linear multistep formula is stable all the zeros of z must lie

in A or p ossibly on its boundary in the case of a zero with jz j This

follows from and since the zeros have to satisfy jz j hence

Re log z At this p oint some reasoning by the argument principle of com

plex analysis is needed which will be lled in later The conclusion is this

any b ounded nger of A ie not extending to must contain one of the

zeros of z Consequence for stability every boundedngerhas to intersect

the unit disk

To nish the argument let us now assume that the linear multistep for

mula is explicit similar arguments apply in the implicit case Exercise

Our goal is then to prove p s To do this we shall count zeros of Re z on

the unit circle jz j Let M b e this numb er of zeros counted with multiplic

ity obviously M must be even How big can M be Note rst that

z z z z z z

Rez

z

z z z

on the unit circle jz j since Re log z there Since the linear multistep

formula is explicit z has degree s so the numerator is a polynomial

of degree s which implies that M being even satises

M s

Now howmany zeros are implied by order of accuracy p First there is a zero

of multiplicity p at z In addition there is a zero wherever the b oundary

of A crosses the unit circlein Figure four such zeros altogether To

count these we note that p ngers of A begin at z outside the

unit circle One of these may be unb ounded and go to innity but the other

p must intersect the unit disk and cross the unit circle on the way Two

of those may cross the unit circle just once one each in the upp er and lower

halfplanes but all the remaining p ngers are trapp ed inside those

ngers and must cross the unit circle twice All told we count

M p p p

Combining and gives p sas required

Similar arguments apply if the linear multistep formula is implicit

STABILITY TREFETHEN

The restrictions of Theorem are tight In eect they show half of the

available parameters in a linear multistep formula are wasted

Theorems and imply that the AdamsBashforth and Nystrom

formulas are all optimal in the sense that they attain the b ound p s for sta

ble explicit formulas and the AdamsMoulton and generalized MilneSimpson

formulas of odd step number s are also optimal with p s Simpsons

rule with p s is an example of an optimal implicit formula with even

step numb er It can b e shown that for any optimal implicit formula with even

step number s the ro ots of z all lie on the unit circle jz j Henrici

p Thus the stability of these formulas is alw ays of a b orderline kind

EXERCISES

Backwards dierentiation formulas Show that the following backwards dierentia

tion formulas from Table are stable a s b s

Prove

a Any consistent step linear multistep formula is stable

b Any consistent linear multistep formula with z z is unstable

Consider the sstep explicit linear multistep formula of optimal order of accuracy of

P

s

nj ns n

f the form v v k

j

j

a For which s is it stable

b Derive the co ecients for the case s

c Likewise for s

Padeapproximation The typ e Padeapproximant to log z at z is dened

as the unique rational function z z oftyp e ie with numerator of degree

and denominator of degree that satises

z

logz O z as z

z

By Theorem taking s gives the maximally accurate implicit sstep formula with

order of accuracy at least p s and taking s s gives the maximally accurate

explicit sstep formula with order of accuracy at least p s

Without p erforming any calculations but app ealing only to the uniqueness of Padeapprox

imants and to theorems stated in this text determine whether each of the following Pade

schemes is stable or unstable and what it is called if wehavegiven a name for it In each

case state exactly what theorems you have used

a s explicit b s implicit

c s explicit d s implicit

e s explicit f s implicit

g If you have access to a symb olic calculator such as Macsyma Maple or Mathematica

use it to calculate the co ecients of the linear multistep formulas af and conrm

that the names you haveidentied ab ove are correct

STABILITY TREFETHEN

Linear combinations of linear multistep formulas In Example weshowed that

and the midp ointformula has error constant the trap ezoid formula has error constant

a Devise a linear combination of these two formulas that has order of accuracy higher

than What is the order of accuracy

b Show that the formula of a is stable

c In general is a convex linear combination of two stable linear multistep formulas always

stable A convex linear combination has the form a formula a formula with

a Prove it or give a counterexample

Order stars Write a program to generate order star plots like that of Figure

This may b e reasonably easy in a higherlevel language like Matlab Plot order stars for the

following linear multistep formulas among others

a thorder AdamsBashforth

b thorder AdamsMoulton

c thorder backwards dierentiation

d the unstable form ula

The pro of of Theorem in the text covered only the case of an explicit linear

multistep formula Show what mo dications are necessary for the implicit case

THE DAHLQUIST EQUIVALENCE THEOREM TREFETHEN

Convergence and the Dahlquist

equivalence theorem

Up to this p ointwehave talked ab out accuracy consistency and stability but wehave

yet to establish that a linear multistep formula with these admirable prop erties will actually

work After one more denition we shall b e able to remedy that To set the stage let us

p

return to the fo otnote on p If a linear multistep formula has lo cal accuracy O k

p

how can it fail to have global accuracy O k The answer has nothing to do with the fact

that f may b e nonlinear for wehave assumed that f is Lipschitz continuous and that is

enough to keep the b ehavior of small p erturbations essentially linear so long as they remain

small

p

The awinthe O k argument is as follows Even though a discretization error may

b e small when it is rst intro duced from that point on it may growoften exp onentially

The global error at step n consists of the sup erp osition not simply of the lo cal errors at

all previous steps but of what these lo cal errors have become at step n Consistency is the

condition that the lo cal errors are small at the time that they are intro duced provided that

the function b eing dealt with is smo oth The additional condition of stability is needed to

make sure that they do not b ecome bigger as the calculation pro ceeds

The purp ose of this section is to describ e the ideas related to the Dahlquist Equivalence

Theorem that have b een built to describ e these phenomena The rigorous statementofthe

argumentabove app ears in the pro of of Theorem b elow If f is linear the argument

b ecomes simpler see Exercise

To b egin wemust dene the notion of convergence The standard denition requires

the linear multistep formula to b e applicable not just to a particular initialvalue problem

but to an arbitrary initialvalue problem with Lipschitz continuous data f It must also

s

work for any starting v alues v v that satisfy the consistency condition

n

kv u k o as k n s

Equivalent statements of the same condition would b e

n n n

lim kv u k lim v u or v u o

as k for n s see Exercise b

A linear multistep formula is convergent if for all initialvalue problems satisfy

s

ing the conditions of Theorem on an interval T and all starting values v v

n

satisfying the solution v satises

kv t utk o as k

uniformly for all t T

The choice of the norm kk do esnt matter since all norms on a nitedimensional space are

equivalent For a scalar ODE the norm can b e replaced by the absolute value

THE DAHLQUIST EQUIVALENCE THEOREM TREFETHEN

Uniformly means that kv t utk is b ounded by a xed function k o as k

indep endent of t To put it in words a convergent linear multistep formula is one that

is guaranteed to get the rightanswer in the limit k for each t in a bounded interval

T assuming there are no rounding errors

Some remarks on this denition are in order First is a statementaboutthe

limit k for each xed value of t Since v t is dened only on a discrete grid this means

that k is implicitly restricted to values tn in this limit pro cess However it is p ossible to

lo osen this restriction by requiring only t t as k

n

k

Second to be convergent a linear multistep formula must work for al l wellp osed

initialvalue problems not just one This may seem an unnaturally strict requirement but

it is not A formula that worked for only a restricted class of initialvalue problemsfor

example those with suciently smo oth co ecientswould b e a fragile ob ject sensitiveto

p erturbations

n

Third v refers to the exact solution of the linear multistep formula rounding errors

are not accounted for in the denition of convergence This is a reasonable simplication

b ecause discretization errors and errors in the starting values are included via and

that govern these various sources of error are nearly the same the stability phenomena

Alternatively the theory can b e broadened to include rounding errors explicitly

Finally note that condition is quite weak requiring only o accuracy of start

ing values or O k if the errors happ en to followanintegral p ower of k This is in contrast

to the O k accuracy required at subsequent time steps by the denition of consistency

The reason for this discrepancy is simple enough starting errors are intro duced only at s



time steps while subsequent errors are intro duced k times Therefore one can get

away with one order lower accuracy in starting values than in the time integration F or

partial dierential equations we shall nd that analogously one can usually get awaywith

one order lower accuracy in discretizing b oundary conditions

We come now to a remarkable result that might be called the fundamental theorem

of linear multistep formulas Likemuch of the material in this and the last section it rst

app eared in a classic pap er by Germund Dahlquist in

DAHLQUIST EQUIVALENCE THEOREM

Theorem Alinearmultistep formula is convergent if and only if it is consistent

and stable

Before proving this theorem let us pause for a moment to consider what it says It

seems obvious that a linear multistep formula that admits unstable solutions is unlikely to

b e useful but it is not so obvious that instabilityisthe only thing that can go wrong For

example clearly the extrap olation formula of Example must b e useless since it

n

ignores ff g but why should that haveanythingtodowithinstability Yet Theorem

asserts that any useless consistentformula must b e unstable And indeed wehaveveried

this prediction for in Example

G Dahlquist Convergence and stability in the numerical integration of ordinary dierential equa

tions Math Scand There are imp ortant earlier references in this area to o

notably an inuential article by Richard von Mises in who proved convergence of Adams

metho ds The classic reference in b o ok form on this material is P Henrici Discrete Variable Meth

o ds in Ordinary Dierential Equations Wiley A mo dern classic is the twovolume series by

Hairer Nrsett and Wanner

THE DAHLQUIST EQUIVALENCE THEOREM TREFETHEN

An indication of the p ower of Theorem is that here as in all of the developments of

this chapter the initialvalue problems under consideration may b e linear or nonlinear For

partial dierential equations we shall see that the b est known analog of Theorem the

Lax Equivalence Theoremrequires linearity

Pro of Theorem is not mathematically deep the art is in the denitions Toprove

it we shall establish three implications

a convergent stable

b convergent consistent

c consistent stable convergent

In outline a and b are proved by applying the linear multistep formula to particular

initialvalue problems and c is proved byverifying that so long as one has stability

lo cal errors cannot add up to o much

ergent stable If the linear multistep formula is convergent then a Conv

must hold in particular for the initialvalue problem u u whose solution is

t

ut Now supp ose the formulaisunstable Then by the pro of of Theorem it admits

n n n n

a particular solution V z with jz j or V nz with jz j In either case supp ose

the starting values for the time integration are taken as

p

n n

v kV n s

where k is as always the time step Then the computed solution for all n will b e precisely

p p

n

kV But whereas the factor k ensures that the starting values approach u as

p

n

k as required by j kV j approaches for any t nk Thus do es

not hold and the formula is not convergent

b Convergent consistent To prove consistency by Theorem wemust show



and For the rst consider the particular initialvalue problem u

t

s

u whose solution is ut and the particular initial values v v

Since f u t the linear multistep formula reduces to and convergence implies

that the solutions to this recurrence relation for the given initial data satisfy v as

n n

k Since v do es not dep end on k this is the same as saying v asn for xed

k and by this implies



To show consider the particular initialvalue problem u u

t

with exact solution ut t Since f u t the linear multistep formula reduces

n

to Z v k NowsinceZ is a p olynomial of degree s with it has a nite

Taylor expansion

 s s 

Z Z Z Z

s

n   n

LetusapplyZ to the particular function V kn t Since V is

n

n  n

linear in n all but the rst term in the series are and wegetZ V Z V

n

which reduces to k In other words V is a solution to the linear multistep formula if

n n

the prescrib ed initial values are v V for n s Obviously this solution satises

condition for the given initial data u For the linear multistep formula to b e

n 

convergent it follows that V must satisfy and thus w emust have as

claimed

c Consistent stable convergent This part of the pro of is not yet written see

the references such as pp of Hairer Nrsett Wanner Following work of

THE DAHLQUIST EQUIVALENCE THEOREM TREFETHEN

Butcher and Skeel it is now standard to carry out the pro of by rst reducing

the linear multistep formula to a onestep recursion as in The underlying idea is

backward error analysisie one shows that the numerical metho d for a xed time step

computes the exact solution to a problem with a slightly p erturb ed function f u t

Besides convergence it is desirable to know something ab out the accuracy of solutions

computed with linear multistep formulas To ensure p thorder accuracy condition

may b e strengthened as follows

n p

kv ut k O k as k n s

n

The following theorem conrms that for a stable linear multistep formula applied to an

initialvalue problem with suciently smo oth co ecients the lo cal errors add up as ex

p ected

TH

GLOBAL p ORDER ACCURACY

Theorem Consider an initialvalue problem that satises the conditions

of Theorem on an interval T and in addition assume that f u t is p times

continuously dierentiable with resp ect to u and t Let an approximate solution be

computed byaconvergentlinearmultistep formula of order of accuracy p with starting

values satisfying Then this solution satises

p

kv t utk O k as k

uniformly for all t T

EXERCISES

Which of the following linear multistep formulas are convergent Are the noncon

vergent ones inconsistent or unstable or b oth

n n n n

v v kf a v

n n

b v v

n n n n n

c v v k f f f

n n n n n

k f f f d v v

n n n n n n n n

v v v k f f f f e v

n n n n n n

f v v v v k f f

Borderline cases

a Consider the unstable extrap olation formula What is the order of magnitude

p ower of k of the errors intro duced lo cally at each step What is the order of mag



nitude factor by which these are amplied after k time steps Verify that this

factor is large enough to cause nonconvergence

s

b What are the corresp onding numb ers for the sstep generalization with z z

z

THE DAHLQUIST EQUIVALENCE THEOREM TREFETHEN

c On the other hand devise another unstable linear multistep formulainwhich the lo cal

discretization errors are of a high enough order of accuracy relative to the unstable

amplication factors that they will not cause nonconvergence

d Why do es the example of c not contradict Theorem Why is it appropriate to

consider this example unstable

Convergence for scalar linear initialvalue problems Prove as briey and elegantly

as you can that stability and consistency imply convergence for the sp ecial case in which u

is scalar and the function f u t is scalar and linear

STABILITY REGIONS TREFETHEN

Stability regions

The results of the last two sections concerned the b ehavior of linear multistep formulas

in the limit k But for the imp ortant class of sti ODEs whichinvolve widely varying

time scales it is impractical to take k small enough for those results to apply Analysis of

behavior for nite k b ecomes indisp ensable In Chapter we shall see that consideration of

nite k is also essential for the analysis of discretizations of partial dierential equations

Stiness is a subtle idea and our discussion of it will b e deferred to the next section

Here we shall simply consider the problem of nite k analysis of linear multistep formulas

The key idea that emerges is the idea of a stability region

EXAMPLE Let us b egin by lo oking again at the unstable thirdorder formula

In Example we applied this formula to the initialvalue problem u u

t

u with time step k and observed the oscillatory instabilityshown in Figure

b From one time step to the next the solution grew by a factor ab out and

to explain this welooked at the recurrence relation

n n n n

Z v v v v

n n n

The zeros of z areand corresp onding to solutions v and v to

and the second of these solutions explains the factor at least approximately

nj

For this scalar linear example we can do much b etter by retaining the terms f in

the analysis The function f is simply f uu and thus an exact rather than approximate

mo del of the calculation is the recurrence relation

n n n

Z v kZ f Z kZ v

that is

n n n n

Z k Z k v v k v k v

Setting k gives the zeros

z z

And now we have a virtually exact explanation of the ratio of Exercise

accurate as it happ ens to ab out digits of accuracy see Exercise

For an arbitrary ODE f is not just a multiple of u and nite k analysis

is not so simple as in the example just considered Wewanttotake the terms

nj

f into consideration but we certainly dont want to carry out a separate

j

analysis for each function f Instead let us assume that f u tau for some

constant a C In other words we consider the linear model equation

u au t

STABILITY REGIONS TREFETHEN

In the next section we shall see that a nonlinear system of ODEs can b e reduced

to a set of problems of the form by linearization freezing of co ecients

and diagonalization

If a linear multistep formula is applied to the mo del equation

it reduces to the recurrence relation

s s

X X

nj nj

v k v

j j

j j

or equivalently

h i

n

Z kZ v

where we have dened

k ak

Now let z be the stability p olynomial

k

s

X

j

z z kz k z

j j

k

j

whose co ecients dep end on the parameter k Then the solutions to

exactly as the solutions to were related are related to the zeros of

k

to the zeros of z In analogy to the developments of x w e dene

A linear multistep formula is absolutely stable for a particular value

n

k ak if all solutions fv g of the recurrence relation are bounded

as n

Just as in Theorem it is easy to characterize those linear multistep formulas

that are absolutely stable

ROOT CONDITION FOR ABSOLUTE STABILITY

Theorem A linear multistep formula is absolutely stable for a par

z satisfy jz j and ticular value k ak if and only if all the zeros of

k

any zero with jz j is simple

What is dierent here from what we did in x is that everything dep ends

on k For some k a linear multistep formula will b e absolutely stable and for

others it will b e absolutely unstable Here now is the key denition

Other terms are weakly stable and timestable

STABILITY REGIONS TREFETHEN

The stability region S of a linear multistep formula is the set of all k C

for which the formula is absolutely stable

Note that according to this denition a linear multistep formula is stable if

and only if the point b elongs to its stabilityregion

Let us now derive the four most familiar examples of stability regions

illustrated in Figure

EXAMPLE For the Euler formula b ecomes

z z k z k

k

with zero k Therefore S is the set of k C with jk j that is the disk jk j

Figure a plots this region

EXAMPLE For the backward Euler formula the stability p olynomial is

z z kz k z

k

 

with zero k S is the set of k C with j k j that is the exterior of the disk

jk j See Figure b

EXAMPLE For the trap ezoid formula wehave

z z k z k z k

k

k k k k S is the set of p oints in C that are no farther with zero

from than from ie Re z the left halfplane See Figure c

EXAMPLE The midp ointformula has

z z kz

k

which has two zeros z satisfying

z k

z

Obviously there is always one zero with jz j and another with jz j so for absolute

stability we must have jz j jz j b oth zeros on the unit circle which will occur if

and only if k lies in the closed complex interval i i The two extreme values k i give

as double zeros z z i so we conclude that S is the op en complex interval i i

shown in Figure d

STABILITY REGIONS TREFETHEN

a Euler b Backward Euler

i

i

c Trap ezoid d Midp oint

Figure Stability regions shaded for four linear multistep

formulas In case d the stability region is the op en complex interval

i i

As mentioned on p the four linear multistep formulas just considered

are the bases of four imp ortant classes of nite dierence formulas for partial

dierential equations Euler backward Euler trap ezoid and leap frog We

shall refer often to Figure in examining the stability prop erties of these

nite dierence formulas

The b oundaries of the stability regions for various AdamsBashforth Ad

amsMoulton and backwards dierentiation formulas are plotted in Figures

Note that the scales in these three gures are very dierent

the AdamsMoulton stability regions are much larger than those for Adams

Bashforth Note also that for all of the higherorder Adams formulas the

stability regions are b ounded whereas for the backwards dierentiation for

STABILITY REGIONS TREFETHEN

mulas they are unb ounded As will be explained in the next section this is

why backwards dierentiation formulas are imp ortant

How do es one compute stability regions like these One approach would

b e to calculate the zeros of for a large number of values of k C and draw

k

a contour plot but there is a simpler and more accurate metho d By

if z is a zero of z for some k and z then

k

z

k

z

cf To determine the b oundary of the stability region rst calculate

i

the curveofvalues k corresp onding to z e with This ro ot lo cus

curve has the prop erty that at each point k on the curve one zero of just

k

touches the unit circle It follows that the b oundary of S is a subset of the ro ot

lo cus curveonly a subset in general b ecause the other zeros of might lie

k

either in the disk or outside By the principle of the argument of complex

analysis one can determine which comp onents are whichbychecking just one

value k in each lo op enclosed by the ro ot lo cus curv e See Exercise

EXERCISES

Prove that for the unstable linear multistep formula the stability region S

is the empty set

Find exact formulas for the b oundaries of S for a the secondorder AdamsBashforth

formula b the thirdorder backwards dierentiation formula

Write a computer program to plot ro ot lo cus curves In a highlevel language like



Matlab it is most convenient to dene a variable r z where z is a vector of or

so p oints on the unit circle and then work directly with formulas such as

rather than with the co ecients and

j j

a Repro duce Figure and then generate the ro ot lo cus curve for p What is S

Be careful

b Repro duce Figure and then generate the ro ot lo cus curvefor p What is S

c Plot the ro ot lo cus curve for Lab el each comp onentby the numb er of zeros of

outside the unit disk and explain how this pictures relates to Exercise

k

What is the maximum time step k for which the thirdorder AdamsBashforth

formula is absolutely stable when applied to i u u ii u iu

t t

a First estimate the time step limits with the aid of Figure and a ruler

b Now derive the exact answers For ii this is hard but it can b e done

Explain the remark ab out digit accuracy at the end of Example Approxi

mately where do es the gure of digits come from

True or False the stability region for anylinearmultistep formula is a closed subset of the complex plane

STABILITY REGIONS TREFETHEN

i

p

i

Figure Boundaries of stability regions for AdamsBashforth

formulas of orders

i

p

i

Figure Boundaries of stability regions for AdamsMoulton

formulas of orders Orders and were displayed already in

Figure bc Note that the scale is very dierent from that of

the previous gure

STABILITY REGIONS TREFETHEN

i

p

i

Figure Boundaries of stability regions for backwards dier

entiation formulas of orders exteriors of curves shown

Simpsons formula Determine the stability region for Simpsons formula

and drawasketch

Prove that any convergent linear multistep formula is absolutely unstable for all

suciently small p ositive k

STIFFNESS TREFETHEN

Stiness

In Henricis Discrete Variable Metho ds in Ordinary Dierential Equations ap

p eared This landmark book presenting the material of the previous sections in great

detail made the Dahlquist theory of linear multistep formulas widely known With the

availabilityofformulas of arbitrarily high order of accuracy and a general convergence the

ory to provethat they worked it may have seemed that little was left to do except turn

theory into software

As ODE computations b ecame commonplace in the s however it b ecame clear

that certain problems were handled badly by the usual metho ds Unreasonably small time

steps would b e required to achieve the desired accuracy What was missing in the standard

theory was the notion of stiness As it happ ens the missing piece had b een supplied

a decade earlier in an eightpage pap er by a pair of chemists at the University of Wis

consin CF Curtiss and JO Hirschfelder Pro c Nat Acad Sci Curtiss and

Hirschfelder had identied the phenomenon of stiness coined the term and invented back

wards dierentiation formulas to cop e with it However their pap er received little attention

for a number of years and for example it do es not app ear among Henricis three hundred

references

What is a sti ODE The following are the symptoms most often mentioned

The problem contains widely varying time scales

Stability is moreofaconstraint on k than accuracy

Explicit methods dont work

Each of these statements has b een used as a characterization of stiness by one author or

another In fact they are all correct and they are not indep endent statements but part of a

coherent whole After presenting an example we shall make them more precise and explain

the logical connections b etween them

EXAMPLE The linear initialvalue problem

u u cost sin t u

t

has the unique solution ut cost for if ut cost the rst term on the right b ecomes

so that the large co ecient drops out of the equation That co ecient has a dominant

eect on nearby solutions of the ODE corresp onding to dierent initial data however as

illustrated in Figure A typical tra jectory ut of a solution to this ODE begins by

sho oting rapidly towards the curve cost on a time scale This is the hallmark of

stiness rapidly changing comp onents that are present in an ODE even when they are

absent from the solution b eing tracked

This last item is quoted from p of the book by Hairer and Wanner For all kinds of

information on numerical metho ds for sti ODEs including historical p ersp ectives and lighthearted

humor that is the b o ok to turn to Another earlier reference worth noting is L F Shampine and C

W Gear A users view of solving sti ordinary dierential equations SIAM Review

STIFFNESS TREFETHEN

t

Figure utcost and nearby solutions of the initialvalue problem

k AB BD

cos

Table Computed values v for the same problem

o o o o

o AB

o o o o o o o o o o o o o

o

  

o

o o o k o * * * BD * * * * * ** * * * * * * * * * * * * o * * * *o *o o o o * * * *o *o o o o * * * *o *o o o o

* * * *o *o



Figure Computed errors jv cosj as a function of step size k for the same problem loglog scale

STIFFNESS TREFETHEN

Table indicates the curious eect that this prop erty of the ODE has on numerical

computations For six values of k the table compares the results at t computed bythe

secondorder AdamsBashforth and backwards dierentiation formulas Since the ODE is

linear implementing the backwards dierentiation formula is easy The dierence b etween

the two columns of numb ers is striking The backwards dierentiation formula behaves

b eautifully converging smo othly and quadratically to the correct answer but the Adams

Bashforth formula generates enormous and completely erroneous numb ers for mo derate k

Yet when k b ecomes small enough it settles down to be just as accurate as backwards

dierentiation

This b ehavior is shown graphically in Figure which is a loglog plot of the error

j as a function of k The AdamsBashforth formula is obviously useless for jv cos

k What if weonlywanttwo or three digits of accuracy With the AdamsBashforth

formula that request cannot b e satised seven digits is the minimum

Since the example just considered is linear one can analyze what went

wrong with the AdamsBashforth formula exactly In the notation of the last

n

section the trouble is that there is a ro ot of the recurrence relation Z v

k

that lies outside the unit disk We make the argument precise as follows

EXAMPLE CONTINUED If utisany solution to u u cost sint

t

then utut cost satises the equation

u u

t

This is a linear scalar constantco ecient ODE of the form of the last section with

a If we apply the ndorder AdamsBashforth formula to it we get the recurrence

relation

n n

n n

v v v v k

that is

n n n

v k v kv

with characteristic p olynomial

z z k z k z k z k

k

For k one of the two ro ots of this p olynomial lies in the negative real interval

whereas for k that ro ot crosses into the interval This is why

k is the critical value for this problem as is so strikingly evident in Figure

Figure shows how this analysis relates to the stability region of the secondorder

AdamsBashforth formula The shaded region in the gure is the stability region see Figure

and the crosses mark the quantities k k corresp onding to the values k except

k in Table When k b ecomes as small as ie k the crosses moveinto

the stability region and the computation is successful

STIFFNESS TREFETHEN

k

k

xxxxx

k k k

k k k

Figure The stability region for the ndorder AdamsBash

forth formula with crosses marking the values k k from Table

With this example in mind let us turn to a more careful discussion of

statements on p The following summary describ es what it means to

say that the ODE u f u t is sti with resp ect to the solution u t for

t

times t t

Widely varying time scales Stiness arises when the time scale that

characterizes the evolution of u t for t t is much slower than the time

scale that characterizes the evolution of small p erturbations ut on that

solution for t t

Stability is more of a constraint than accuracy If widely varying time

scales in this sense are present and if the discrete ODE formula has a b ounded

stability region then there is a mismatch between the relatively large time

steps that are adequate to resolve u t and the relatively small time steps

that are necessary to prevent unstable growth of small p erturbations ut

A successful computation will necessitate the use of these small time steps

Explicit methods dont work In particular since explicit metho ds

always have bounded stability regions Exercise the solution of sti

ODEs by explicit metho ds necessitates the use of small time steps For b etter

results one must usually turn to implicit formulas with unb ounded stability

regions

If the real world supplied ODEs to be solved at random then widely

varying time scales might not be a common concern In actuality the appli

cations that the real world presents us with are often exceedingly sti One

example is the eld of chemical kinetics Here the ordinary dierential equa

tions describ e reactions of various chemical sp ecies to form other sp ecies and

the stiness is a consequence of the fact that dierent reactions take place

on vastly dierent time scales Time scale ratios of or more are common

Two other elds in which sti ODEs arise frequently are control theorywhere

controlling forces tend to b e turned on and o again suddenly and circuit sim

ulation since dierent circuit comp onents tend to have widely diering natural

STIFFNESS TREFETHEN

frequencies A fourth imp ortant class of sti ordinary dierential equations

are the systems of ODEs that arise after discretizing timedep endent partial

dierential equations particularly parab olic problems with resp ect to their

space variablesthe metho d of lines This example will be discussed at

length in Chapter

To be able to determine whether a particular problem is sti we need

to ols for quantifying the notion of the time scales present in an ODE The

way this is generally done is by reducing an arbitrary system of ODEs to a

collection of scalar linear mo del problems of the form to which the

idea of stability regions will be applicable This is achieved by the following

sequence of three steps a sequence that has been familiar to mathematical

scientists for a century

u f u t

t

LINEARIZE

u J tu

t

FREEZE COEFFICIENTS

u Au

t

DIAGONALIZE

fu ug A

t

In A denotes the sp ectrum of A ie the set of eigenvalues

We b egin with a system of N rstorder ODEs with u t denoting

the particular solution that we are interested in If we make the substitution

utu tut

as in then stabilityand stiness dep end on the evolution of ut

The rst step is to linearize the equation by assuming u is small If f is

dierentiable with resp ect to each comp onent of uthen we have

f u t f u tJ tutokuk

where J t is the N N Jacobian matrix of partial derivatives of f with

resp ect to u

i

f

u tt i j N J t

ij

j

u

See A SangiovanniVincentelli and J K White Relaxation Techniques for the Solution of VLSI

Circuits Kluwer

STIFFNESS TREFETHEN

This means that if u is small the ODE can be accurately approximated by

a linear problem

u f u tJ tu

t

If we subtract the identity u f u t from this equation we get

t

u J tu

t

One can think of this result as approximate if u is small or exact if u is

innitesimal Rewriting u as anew variable u gives

The second step is to freeze coecients by setting

A J t

for the particular value t of interest The idea here is that stability and

stiness are lo cal phenomena which may app ear at some times t and not

others The result is the constantco ecient linear problem

Finally assuming A is diagonalizable we diagonalize it In general any

system of N linear ordinary dierential equations with a constant diagonal

izable co ecient matrix A is exactly equiv alent to N scalar equations

with values a equal to the eigenvalues of A To exhibit this equivalence let V

be an N N matrix such that V AV is diagonal The columns of V are eigen

vectors of A Then u Au can be rewritten as u V V AV V u ie

t t

V u V AV V u a diagonal system of equations in the variables

t

V u This diagonal system of ODEs can be interpreted as a representation

of the original system in the basis of eigenvectors of A dened by the

columns of V And thus since a diagonal system of ODEs is the same as a

collection of indep endent scalar ODEs we end up with the N problems

Having reduced to a collection of N scalar linear constantco ef

cient mo del problems we now estimate stability and stiness as follows

RULE OF THUMB For a successful computation of u t for t t k must

be small enough that for each eigenvalue of the Jacobian matrix J t

k k lies inside the stabilityregion S or at least within a distance O k

of S

This Rule of Thumb is not a theorem exceptions can be found But if it

is violated some numerical mo de will very likely blow up and obliterate the

correct solution Regarding the gure O k see Exercise

It is interesting to consider what keeps the Rule of Thumb from b eing a rigorous statement The

various gaps in the argument can b e asso ciated with the three reductions The step

may fail if the actual discretization errors or rounding errors present in the problem

are large enough that they cannot b e considered innitesimal ie the b ehavior of u is nonlinear

STIFFNESS TREFETHEN

A sti problem is one for which satisfying the conditions of the Rule of

Thumb is more of a headache than resolving the underlying solution u t

Here are some examples to illustrate the reduction

EXAMPLE If the equation in Example had b een

u sinu cost sin t

t

then linearization would have b een called for Setting utut costgives

u sinu

t

and if ut cost then ut is small and the linear mo del b ecomes again

EXAMPLE For the initialvalue problem

u u t sinu cost sin t

t

we end up with a mo del equation with a timedep endent co ecient

u cos tu

t

These are scalar examples However the usual reason that an ODE has

widely varying time scales is that it is a system of dierential equations If the

solution has N comp onents it is natural that some of them mayevolvemuch

more rapidly than others Here are three examples

EXAMPLE Consider the linear initialvalue problem

u u u

A A A A A

t

u u u

t



Supp ose we decide that an acceptable solution at t isanyvector with jv e j



and jv e j for some If is solved by a p th order linear multistep

formula with time step k the u and u comp onents decouple completely so the results

The step may fail if the density of time steps is not large compared with the time

scale on which J tvaries DJ Higham The nite size of the time steps also implies that if

an eigenvalue drifts outside the stability region for a short p erio d the error growth it causes may

b e small enough not to matter Finally the step may fail in the case where the

matrix A is far from normal ie its eigenvectors are far from orthogonal In this case the matrix

V is illconditioned and instabilities may arise even though the sp ectrum of A lies in the stability

region it b ecomes necessary to consider the pseudospectra of A as well DJ Higham and LNT

These eects of nonnormality will b e discussed in xx and turn out to b e particularly

imp ortant in the numerical solution of timedep endentPDEs by sp ectral metho ds discussed in Chapter

STIFFNESS TREFETHEN

in each comp onentwill b e those for the corresp onding mo del equation To obtain

p

v suciently accuratelyweneedk O But to obtain v suciently accurately

if the formula has a stability region of nite size like the Euler formula we need k to b e on



the order of Most likelythisisamuchtighter restriction

EXAMPLE The linear problem

u u u

A A A A A

u u u

t

is only sup ercially less trivial The eigenvalues of the matrix are approximately

and if and are changed to and they b ecome approximately

and As a result this system will exp erience a constrainton k just like that of

Example or worse

EXAMPLE The nonlinear ODE

u u u

A A

u cosu expu

t

has Jacobian matrix

u u

A

J

sinu exp u

Nearapoint t with u t and u t the matrix is diagonal with widely diering

eigenvalues and the b ehavior will probably b e sti

In general a system of ODEs u f u t is likely to b e sti at a p oint u

t

u t t if the eigenvalues of J u t dier greatly in magnitude esp ecially

if the large eigenvalues have negative real parts so that the corresp onding

solution comp onents tend to die away rapidly

To solve sti problems eectively one needs discrete ODE formulas with

large stability regions The particular shap e of the stability region required

will dep end on the problem but as a rule the most imp ortant prop erty one

would like is for the stability region to include a large part of the left half

plane Why the left halfplane See Exercise The following denitions

are standard

A linear multistep formula is Astable if the stability region contains the

entire left halfplane Re k It is A stable if the stability

region contains the innite sector j arg k j It is Astable if it is

A stable for some

STIFFNESS TREFETHEN

Roughly sp eaking an Astable formula will p erform well on almost any sti

problem and of course also on nonsti problems at some cost in extra work

per time step An Astable formula will p erform well on sti problems

whose comp onent mo des exhibit exp onential decay negativerealeigenvalues

but not oscillation imaginary eigenvalues Discretizations of parab olic and

hyp erb olic partial dierential equations resp ectivelywillprovide go o d exam

ples of these two situations later in the b o ok

Figure showed that b oth the backward Euler and trap ezoid formulas

are Astable This seems encouraging p erhaps to play it safe we can simply

use Astable formulas all the time But the following famous theorem shows

that that p olicy is to o limiting

SECOND DAHLQUIST STABILITY BARRIER

Theorem The order of accuracy of an implicit Astable linear mul

tistep formula satises p An explicit linear multistep formula cannot

be Astable

Pro of Dahlquist BIT Not yet written

For serious computations secondorder accuracy is often not go o d enough

This is why backwards dierentiation formulas are so imp ortant For all of

the higherorder Adams formulas the stability regions are b ounded but the

backwards dierentiation formulas are Astable for p Most of the

software in use to day for sti ordinary dierential equations is based on these

formulas though RungeKutta metho ds are also contenders see the next sec

tion Because of Theorem the usable backwards dierentiation formulas

are only those with p In practice some co des restrict attention to p

Of course there are a dozen practical issues of computation for sti ODEs

that we have not touched up on here For example how do es one solve the

nonlinear equations at each time step The answer is invariably some form of

Newtons metho d see the references The state of software for sti problems

is highly advanced as is also true for nonsti problems A nonsti solver

applied to a sti problem will typically get the right answer but it will take

excessively small time steps Many such co des incorp orate devices to detect

that stiness is present and alert the user to that fact or in some cases to

trigger an automatic change to a sti solution metho d

EXERCISES

Prove that the stability region of any explicit linear multistep formulaisbounded

   

For p the values of involved are approximately See Figure

STIFFNESS TREFETHEN

Supp ose Figure had b een based on the error jv cosj instead of jv

cosj Qualitatively sp eaking howwould the plot have b een dierent

What is sp ecial ab out the left halfplane that makes it app ear in the denition of

Astability Avoid an answer full of wae write two or three sentences that hit the nail on

the head

The Rule of Thumbofpmentions the distance O k from the stability region

Explain where this gure comes from p erhaps with the aid of an example WhyisitO k

rather than say O k orO k

Here is a secondorder initialvalue problem

t ut t u u u t costu

t tt t

a Rewrite it as a rstorder initialvalue problem of dimension

n

b Write down the precise formula used to obtain v at each step if this initialvalue

problem is solved by the secondorder AdamsBashforth formula

c Write down the Jacobian matrix J for the system a

An astronomer decides to use a nonsti ODE solver to predict the motion of Jupiter

around the sun over a p erio d of years Saturn must certainly be included in the

calculation if high accuracy is desired and Neptune and Uranus mightaswell b e thrown in

for go o d measure Now what ab out Mars Earth Venus and Mercury Including these inner

planets will improve the accuracy slightly but it will increase the cost of the computation

Estimate as accurately as you can the ratio bywhich it will increase the computation time

on a serial computer Hint your favorite almanac or encyclop edia maycomeinhandy

Alinearmultistep formula is Astable What can you conclude ab out the ro ots of

z

Van der Pol oscillator The equation u u u u known as the Van der Pol

tt t

equation represents a simple harmonic oscillator to which has b een added a nonlinear term

that intro duces p ositive damping for juj and negative damping for juj Solutions

to this equation approach limit cycles of nite amplitude and if is small the oscillation

is characterized by p erio ds of slowchange punctuated by short intervals during which ut

swings rapidly from p ositive to negativeorbackagain

a Reduce the Van der Pol equation to a system of rstorder ODEs and compute the

Jacobian matrix of this system

b What can you say ab out the stiness of this problem

Given supp ose you solve an initialvalue problem involving the linear equation

 

u Au f t where A is a constant matrix with eigenvalues and and

t

f tisaslowlyvarying vector function of dimension You solve this equation by a linear

multistep program that is smart enough to vary the step size adaptively as it integrates and

your goal is to compute the solution u accurate to within an absolute error on the order

of Howmanytime steps order of magnitude functions of and will b e needed

if the program uses a the secondorder AdamsMoulton metho d and b the thirdorder

AdamsMoulton metho d Hint b e careful

A nonlinear sti ODE To app ear

RUNGEKUTTA METHODS TREFETHEN

RungeKutta metho ds

Linear multistep formulas represent one extremeone function evaluation per time

stepand RungeKutta metho ds represent the other They are onestep multistage meth

o ds in which f u tmaybeevaluated at anynumber of stages in the pro cess of getting from

n n

v to v but those stages are intermediate evaluations that are never used again As a

result RungeKutta metho ds are often more stable than linear multistep formulas but at

the price of more work p er time step They tend to b e easier to implement than linear multi

step formulas since starting values are not required but harder to analyze These metho ds

were rst develop ed a century ago by Runge Heun and Kutta

In this b o ok we will not discuss RungeKutta metho ds in any detail wemerelylista

few of them below and state two theorems without pro of This is not b ecause they are

less imp ortant than linear multistep metho ds but merely to keep the scale of the book

manageable Fortunately the fundamental concepts of stability accuracy and convergence

are much the same for RungeKutta as for linear multistep formulas The details are quite

dierent however and quite fascinating involving a remarkable blend of ideas of combina

torics and graph theory For information see the b o oks by Butcher and by Hairer Nrsett

and Wanner

RungeKutta formulas tend to b e not very unique If we dene

s numb ers of stages p order of accuracy

then for most values of s there are innitely many RungeKutta formulas with the maximal

order of accuracy p Here are some of the b estknown examples which should prop erly b e

presented in tableau form but I havent gotten around to that

Mo died Euler or improved p olygon formula s p

n

a kf v t

n

n

b kf v at k

n

n n

v v b

Improved Euler or Heun formula s p

n

a kf v t

n

n

b kf v a t k

n

n n

a b v v

Heuns thirdorder formula s p

n

a kf v t

n

n

b kf v at k

n

n

c kf v bt k

n

n n

v v a c

RUNGEKUTTA METHODS TREFETHEN

Fourthorder RungeKutta formula s p

n

a kf v t

n

n

b kf v at k

n

n

c kf v bt k

n

n

d kf v c t k

n

n n

a b c d v v

If f is indep endent of u the rst twoof these formulas reduce to the midp oint and

trap ezoid formulas resp ectively and the fourthorder formula reduces to Simpsons rule

This last formula is sometimes called the RungeKutta formula and is very widely known

How accurate can a RungeKutta formula b e Here is what was known as of

ORDER OF ACCURACY OF EXPLICIT RUNGEKUTTAFORMULAS

Theorem An explicit RungeKutta formula of order of accuracy p has the fol

lowing minimum numb er of stages s

Order p Minimum numb er of stages s

resp ectively

precise minimum unknown

precise minimum unknown

Pro of See Butcher

Figure shows the stability regions for the explicit RungeKutta formulas with

s Since these formulas are not unique the reader may wonder which choice

the gure illustrates As it turns out it illustrates al l the choices for they all have the

same stability regions whicharethelevel curves jpz j for the truncated Taylor series

z

approximations to e This situation changes for s

The classical RungeKutta formulas were explicit but in recent decades implicit Runge

Kutta formulas have also b een studied extensively Unlike linear multistep formulas Runge

Kutta formulas are not sub ject to an Astability barrier The following result should be

compared with Theorems and

IMPLICIT RUNGEKUTTAFORMULAS

Theorem An sstage implicit RungeKutta formula has order of accuracy p s

For each s there exists an Astable implicit RungeKutta formula with p s

already given in Exercise

RUNGEKUTTA METHODS TREFETHEN

i s

s

s

s

i

Figure Boundaries of stability regions for RungeKutta formulas with

s

Theorem lo oks like a panacea why not use implicit RungeKutta metho ds all

the time for sti problems The answer is that they lead to large systems of equations

to b e solved For an initialvalue problem involving N variables an implicit linear multi

step formula requires the solution of a system of N equations at each step whereas in the

RungeKutta case the dimension b ecomes sN Since the work involved in solving a system

of equations usually dep ends sup erlinearly on its size it follows that implicit RungeKutta

form ulas tend to be more advantageous for small systems than for the larger ones that

arise for example in discretizing partial dierential equations On the other hand many

sp ecial tricks have b een devised for those problems esp ecially to takeadvantage of sparsity

so no judgment can b e considered nal The b o ok by Hairer and Wanner contains exp eri

mental comparisons of explicit and implicit multistep and RungeKutta co des reaching the

conclusion that they all work well

NOTES AND REFERENCES TREFETHEN

Notes and References

This chapter has said little ab out the practical side of numerical solution of ordinary

dierential equations One topic barely mentioned is the actual implementation of implicit

metho dsthe solution of the asso ciated equations at each time step by Newtons metho d

and its variants Another gap is that we have not discussed the idea of adaptive step

size and order controlwhich together with the development of sp ecial metho ds for sti

problems are the most imp ortant developments in the numerical solution of ODEs during

the computer era The software presently available for solving ODEs is so powerful and

reliable that there is little reason other than educational to write ones own program except

for very easy problems or sp ecialized applications Essentially all of this software makes use

of adaptive step size control and some of it controls the order of the formula adaptively

to o Trustworthy co des can b e found in the IMSL NAG and Harwell libraries in Matlab

and in numerous other sources The b estknown programs are p erhaps those develop ed at

Sandia Lab oratories and the Lawrence Livermore Lab oratorywhich can b e obtained from

the Netlib facility describ ed in the Preface

Although this b o ok do es not discuss b oundaryvalue problems for ODEs the reader

should be aware that in that area to o there is excellent adaptive software which is far

more ecient and reliable than the program a user is likely to construct bycombining an

initialvalue problem solver with the idea of sho oting Two wellknown programs are

PASVA and its various derivatives which can b e found in the NAG Library and COLSYS

which has been published in the Transactions on Mathematical Software and is available

from Netlib A standard textb o ok on the sub ject is U M Ascher R M M Mattheij and

R D Russell Numerical Solution of Boundary Value Problems for Ordinary Dierential

Equations Prentice Hall

Some applications lead to systems of dierential equations that must b e solved in con

junction with additional nondierential conditions Such systems are called dierential

algebraic equations DAEs and have b een investigated by Gear Petzold and many

others Software is available for these problems to o including a wellknown program by

Petzold known as DASSL

The scale of ODEs that o ccur in practice can b e enormous One example of engineering

interest arises in the simulation of VLSI circuits where one encounters sparse sti systems

it is of paramount containing thousands or tens of thousands of variables In such cases

imp ortance to exploit the sp ecial prop erties of the systems and one sp ecial metho d that

has b een devised go es by the name of waveform relaxation see the fo otnote on p

CHAPTER TREFETHEN 

Chapter

Fourier Analysis

The Fourier transform

The semidiscrete Fourier transform

Interp olation and sinc functions

The discrete Fourier transform

Vectors and multiple space dimensions

Notes and references

Untwisting all the chains that tie

the hidden soul of harmony

JOHN MILTON LAllegro

CHAPTER TREFETHEN 

The last chapter dealt with time dep endence and this one is motivated

by space dep endence Later chapters will combine the two

Fourier analysis touches almost every asp ect of partial dierential equa

tions and their numerical solution Sometimes Fourier ideas enter into the

analysis of a numerical algorithm derived from other principlesesp ecially in

the stability analysis of nitedierence formulas Sometimes they underlie

the design of the algorithm itselfsp ectral metho ds And sometimes the situ

ation is a mixture of b oth as with iterativeandmultigrid metho ds for elliptic

equations For one reason or another Fourier analysis will app ear in all of the

remaining chapters of this b o ok

The impact of Fourier analysis is also felt in many elds b esides dier

ential equations and their numerical solution such as quantum mechanics

crystallography signal pro cessing statistics and information theory

There are four varieties of Fourier transform dep ending on whether the

spatial domain is unb ounded or bounded continuous or discrete

Name Space variable Transform variable

Fourier transform unb ounded continuous continuous unb ounded

Fourier series bounded continuous discrete unb ounded

semidiscrete Fourier transform unb ounded discrete continuous b ounded

or z transform

discrete Fourier transform b ounded discrete discrete bounded

DFT

The second and third varieties are mathematically equivalent This chapter

will describ e the essentials of these op erations emphasizing the parallels be

tween them In discrete metho ds for partial dierential equations one lo oks

for a representation that will converge to a solution of the continuous problem

as the mesh is rened Our denitions are chosen so that the same kind of

convergence holds also for the transforms

Rigorous Fourier analysis is a highly technical and highly develop ed area

of mathematics which dep ends heavily on the theory of Leb esgue measure and

integration We shall make use of L and spaces but for the most part this

chapter avoids the technicalities In particular a number of statements made

in this chapter hold not at every point of a domain but almost everywhere

everywhere but onaset of measure zero

THE FOURIER TRANSFORM TREFETHEN 

The Fourier transform

If ux is a Leb esguemeasurable function of x R the L norm of u

is the nonnegative or innite real number

Z

h i

kuk juxj dx

The symbol L Ltwo denotes the set of all functions for whichthisintegral

is nite

L fu kuk g

and L are the sets of functions having nite L andL norms Similarly L

dened by

Z

juxjdx kuk sup juxj kuk

x

Note that since the L norm is the norm used in most applications b ecause

of its many desirable prop erties we have reserved the symbol kk without a

subscript for it

The of two functions u v is the function u v dened by

Z Z

u v x u v x ux y v y dy uy v x y dy

assuming these integrals exist One way to think of u v is as a weighted

moving average of values uxwithweights dened by v x or vice versa

u L the Fourier transform of u is the function u dened For any

by

Z

i x

u F u e uxdx R

The quantity is known as the wavenumber the spatial analog of frequency

this integral converges in the usual sense for all For many functions u L

R but there are situations where this is not true and in these cases one

must interpret the integral as a limit in a certain L norm sense of integrals

R

M

as M The reader interested in such details should consult the

M

various b o oks listed in the references

If u L then u exists for every and is continuous with resp ect to According to the

RiemannLeb esgue Lemma it also satises ju jas

THE FOURIER TRANSFORM TREFETHEN 

x

Figure Space and wavenumb er domains for the Fourier trans

form compare Figures and

The following theorem summarizes some of the fundamental prop erties of

Fourier transforms

THE FOURIER TRANSFORM

Theorem If u L then the Fourier transform

Z

i x

u F u e uxdx R

b elongs to L also and u can b e recovered from u bythe inverse Fourier

transform

Z

i x

ux F u x e u d x R

The L norms of u and u are related by Parsevals equality

p

ku k kuk

d

If u L and v L or vice versa then u v L and u v satises

d

u v u v

These four equations are of such fundamental imp ortance that they are

worth commenting on individually although it is assumed the reader has al

ready been exp osed to Fourier analysis

As mentioned in the intro duction to this chapter some of these prop ertiesnamely equations

and hold merely for almost every value of x or In fact even if f z is a



continuous function in L its Fourier transform may fail to converge at certain p oints x To ensure

pointwise convergence one needs additional assumptions such as that f is of b ounded variation

dened b elow b efore Theorem and b elongs to L These assumptions also ensure that at any

point x where f has a jump discontinuity its Fourier transform converges to the average value



f x f x

THE FOURIER TRANSFORM TREFETHEN 

First of all indicates that u is a measure of the correlation of

i x

ux with the function e The idea behind Fourier analysis is to interpret

i x

uxas a sup erp osition of mono chromatic waves e with various wave num

b ers andu represents the complex amplitude more precisely amplitude

density with resp ect to of the comp onentofu at wave number

Conversely expresses the synthesis of u x as a sup erp osition of its

i x

comp onents e each multiplied by the appropriate factor u The factor

is a nuisance that could have been put in various places in our formulas

but is hard to eliminate entirely

Equation Parsevals equality is a statement of energy conserva

tion the L energy of any signal ux is equal to the sum of the energies of

p

By energy we mean its comp onent vibrations except for the factor

the square of the L norm

Finally the convolution equation is p erhaps the most subtle of

d

the four The left side u v represents the strength of the wave number

comp onent that results when u is convolved with v in other words the degree

to which u and v beat in and out of phase with each other at wave number

when multiplied together in reverse order with a varying oset Such b eating

is caused b y a quadratic interaction of the wave number comp onent in u

with the same comp onentofv hence the righthand side u v

All of the assertions of Theorem can be veried in the following ex

ample which the reader should study carefully

EXAMPLE Bsplines Supp ose u is the function

for x



ux

otherwise

p

Figure Then by wehave kuk and gives

Z

i x

sin e

i x

e dx u



i

This functionu is called a sinc function more on these in x From and the

indisp ensable identity

Z



sin s

ds



s

p

which can b e derived by complex contour integration we calculate kuk which conrms

From the denition it is readily veried that in this example

jxj for x



u ux

otherwise

worth memorizing

THE FOURIER TRANSFORM TREFETHEN 

u

u

x

d

u u

u u

x

d

u u u

u u u

x

Figure The rst three Bsplines of Example and their Fourier trans

forms

and





x for x

 



u u ux

jxj x for jxj



otherwise

and by and the corresp onding Fourier transforms must b e

 

sin sin

d d

u u u u u

 

See Figure In general a convolution u of p copies of u has the Fourier transform

p

p

sin

ud Ffu u ug

p

Note that whenever u or any other function is convolved with the function u of

p

it b ecomes smo other since the convolution amounts to a lo cal moving average In

particular u itself is piecewise continuous u u is continuous and has a piecewise continuous

rst derivative u u u has a continuous derivative and a piecewise continuous second

derivative and so on In general u is a piecewise p olynomial of degree p with a

p

con tinuous p nd derivative and a piecewise continuous p st derivative and is known

as a Bspline See for example C de Bo or A Practical Guide to Splines Springer

Thus convolution with u makes a function smo other while the eect on the Fourier

transform is to multiply it bysin and therebymake it decay more rapidly This

relationship is evident in Figure

For applications to numerical metho ds for partial dierential equations

there are two prop erties of the Fourier transform that are most imp ortant

THE FOURIER TRANSFORM TREFETHEN 

One is equation the Fourier transform converts convolution into mul

tiplication The second can be derived by

Z Z

i x i x

c

u e u xdx i e uxdx i u

x x

the Fourier transform assuming ux is smo oth and decays at That is

converts dierentiation into multiplication by i This result is rigorously

valid for any absolutely continuous function u L whose derivative b elongs

to L Note that dierentiation makes a function less smo oth so the fact that

it makes the Fourier transform decay less rapidly ts the pattern mentioned

ab ove for convolution

Figure

EXAMPLE The function

for x



ux

for x



otherwise

illustrated in Figure has Fourier transform

Z Z



i x i x

u e dx e dx

 





i sin

i i i i 

e e e e

i i

whichisi times the Fourier transform of the triangular hat function In

keeping with is the derivative of

The following theorem collects together with a number of addi

tional prop erties of the Fourier transform

THE FOURIER TRANSFORM TREFETHEN

PROPERTIES OF THE FOURIER TRANSFORM

Theorem Let u v L have Fourier transforms u F u v F v

Then

a Linearity Ffu v g u v Ffcug cu

i x

b Translation If x R then Ffux x g e u

i x

c Mo dulation If R then Ffe uxg u

d Dilation If c R with c then Ffucxg u cjcj

u g u e Conjugation Ff

f Dierentiation If u L then Ffu g i u

x x

g Inversion F fug u

Pro of See Exercise

In particular taking c in part d ab ove gives Ffuxg u

Combining this result with part e leads to the following elementary but useful

results Denitions ux is even odd real or imaginary if uxux

ux ux uxux or uxux resp ectively ux is hermitian

uxor uxux resp ectively or skewhermitian if ux

SYMMETRIES OF THE FOURIER TRANSFORM

Theorem Let u L have Fourier transform u F u Then

a ux is even o dd u is even o dd

b ux is real imaginary u is hermitian skewhermitian

and therefore

c ux is real and even u is real and even

d ux is real and odd u is imaginary and odd

is imaginary and even e ux is imaginary and even u

f ux is imaginary and odd u is real and odd

Pro of See Exercise

THE FOURIER TRANSFORM TREFETHEN

In the discussion ab ovewehavetwice observed the following relationships

between the smo othness of a function and the decay of its Fourier transform

u ux

smo oth decays rapidly as j j

decays rapidly as jxj smo oth

Of course since the Fourier transform is essentially the same as the inverse

Fourier transform by Theorem g the tworows of this summary are equiv

alent The intuitive explanation is that if a function is smo oth then it can

be accurately represented as a sup erp osition of slowlyvarying waves so one

do es not need much energy in the high wavenumb er comp onents Conversely

a nonsmo oth function requires a considerable amplitude of high wavenumber

comp onents to b e represented accurately These relationships are the b edro ck

of analog and digital signal pro cessing where all kinds of smo othing op erations

are eected b y multiplying the Fourier transform by a windowing function

that decays suitably rapidly

The following theorem makes these connections between u and u precise

This theorem may seem forbidding at rst but it is worth studying carefully

Each of the four parts of the theorem makes a stronger smo othness assumption

on u than the last and reaches a corresp ondingly stronger conclusion ab out

the rate of decay of u as j j Parts c and d are known as the

PaleyWiener theorems

First a standard denition A function u dened on R is said to hav e

b ounded variation if there is a constant M such that for any nite m and

P

m

jux ux jM any p oints x x x

j m

j j

THE FOURIER TRANSFORM TREFETHEN

SMOOTHNESS OF u AND DECAYOFu

Theorem Let u be a function in L

a If u has p continuous derivatives in L for some p and a pth

derivativeinL that has bounded variation then

p

u O j j as j j

b If u has innitely many continuous derivatives in L then

M

u O j j as j j for all M

and conversely

c If u can b e extended to an analytic function of z x iy in the complex

strip j Im z j a for some a with kux iy k const uniformly for each

constant ay a then

aj j

e u L

and conversely

d If u can be extended to an entire function of z x iy with juz j

ajz j

O e as jz j z C for some a then u has compact supp ort

contained in a a ie

u for all j j a

and conversely

Pro of See for example xVI of Y Katznelson An Intro duction to

Harmonic Analysis Dover Also see Rudin p Paley Wiener

Reed Simon v Hormander v p entire functions b o oks

A function of the kind describ ed in d is said to b e bandlimited since

only a nite band of wave numb ers are represented in it

Since the Fourier transform and its inverse are essentially the same by

Theorem g Theorem also applies if the roles of ux and u are inter

changed

EXAMPLE CONTINUED The square wave u of Example Figure

satises condition a of Theorem with p so its Fourier transform should satisfy

An entire function is a function that is analytic throughout the complex plane C

THE FOURIER TRANSFORM TREFETHEN 

ju j O j j as is veried by On the other hand supp ose we interchange

the roles of u and u and apply the theorem again The function u sin is entire

i i j j

and since sin e e iit satises u O e as j j with now taking

complex values By part d of Theorem it follows that ux must have compact

supp ort contained in as indeed it do es

Rep eating the example for u u condition a now applies with p and the Fourier



transform is indeed of magnitude O j j as required Interchanging u and u



 j j

we note that sin is an entire function of magnitude O e asj j and u u has

supp ort contained in

EXERCISES

Show that the twointegrals in the denition of u v are equivalent

Derive conditions ag of Theorem Do not worry ab out justifying the usual

op erations on integrals

Prove Theorem



a Which functions u L L satisfy u u

b Howabout u u u

Integration

a What do es part f of Theorem suggest should be the Fourier transform of the

R

x

usds function U x

R



b Obviously U x cannot b elong to L unless uxdx soby Theorem this is

R



a necessary condition for U to b e in L also Explain how the condition uxdx

relates to your formula of a for U in terms of u



a Calculate the Fourier transform of ux x Hint use a complex contour

integral if you knowhow Otherwise lo ok the result up in a table of integrals

b How do es this example t into the framework of Theorem Which parts of the

theorem apply to u

u are interchanged how do es the example now t Theorem c If the roles of u and

Which parts of the theorem apply to u



The auto correlation function of a function u L L may b e dened by

Z

c uxux cdx



kuk

Find an expression for casaninverse Fourier transform of a pro duct of Fourier transforms

involving u This expression is the basis of some algorithms for computing c

THE FOURIER TRANSFORM TREFETHEN 

Without evaluating any integrals use Theorem and to determine the

Fourier transform of the following function

The uncertainty principle Showby using Theorem that if uxandu b oth



have compact supp ort with u L thenux

THE SEMIDISCRETE FOURIER TRANSFORM TREFETHEN 

The semidiscrete Fourier transform

The semidiscrete Fourier transform is the reverse of the more familiar

instead of a b ounded continuous spatial domain and an unb ounded discrete transform

domain it involves the opp osite This is just what is needed for the analysis of numerical

metho ds for partial dierential equations where we are p erp etually concerned with functions

dened on discrete grids For many analytical purp oses it is simplest to think of these grids

as innite in extent

Let h b e a real numb er the space stepandlet x x x b e dened by

x jh Thus fx g hZwhereZ is the set of integers We are concerned now with spatial

j j

grid functions v fv gwhichmayorma y not b e approximations to a continuous function

j

u

v ux

j j

As in the last chapter it will b e convenient to write v x sometimes for v

j j

********oooooooo x

h h

h

Figure Space and wave number domains for the semidiscrete Fourier

transform

For functions dened on discrete domains it is standard to replace the upp ercase letter

L bya lowercase script letter Both symb ols honor Henri Leb esgue the mathematician

who laid the foundations of mo dern early in the twentieth century The



norm of a grid function v is the nonnegative or innite real number

h



X



kv k h jv j

j

j

Notice the h in front of the summation One can think of as a discrete approximation

to the integral by the trap ezoid rule or the rectangle rule for quadrature On an



unb ounded domain these two are equivalen t The symbol little Ltwosub h denotes

h

the set of grid functions of nite norm



fv kv k g

h



and similarly with and In contrast to the situation with L L and L these

h h

spaces are nested



h h h See Exercise

THE SEMIDISCRETE FOURIER TRANSFORM TREFETHEN 

The convolution v w of two functions v w is the function v w dened by

X X

v w h v w h v w

m mj j j mj

j j

provided that these sums exist This formula is a trap ezoid or rectangle rule approximation

to



For any v the semidiscrete Fourier transform of v is the functionv dened

h

by

X

i x

j

v h h v F v h e

j h

j

a discrete approximation to A priori this sum denes a functionv for all R

imx h

imj

j

However notice that for anyinteger m the exp onential e e is exactly at

all of the grid points x More generally any wave number is indistinguishable on the

j

grid from all other wavenumbers mh where m is anyintegera phenomenon called

p erio dic on R Tomake sense of the aliasing This means that the function v ish

idea of analyzing v into comp onent oscillations we shall normally restrict attention to one

period ofv by lo oking only at wavenumb ers in the range h h and it is in this sense

that the Fourier transform of a grid function is dened on a bounded domain But the

reader should b ear in mind that the restriction of to any particular interval is a matter

ij

of lab eling not mathematics in principle e and e are equally valid representations of

the grid function v

j

Thus for discretized functions v the transform v inhabits a b ounded domain On

the other hand the domain is still continuous This reects the fact that arbitrarily ne

gradations of wavenumb er are distinguishable on an unb ounded grid

Since x and belong to dierent sets it is necessary to dene an additional vector



norm of a functionv is the number space for functionsv The L

h

Z

h

i h





jv j d kvk

h

One can think of this as an approximation to in whichthewavenumber components



denotes the set of Leb esgue with j j h have b een replaced by zero The symbol L

h

measurable functions on h h of nite norm



g L fv kvk

h

Nowwe can state a theorem analogous to Theorem

THE SEMIDISCRETE FOURIER TRANSFORM TREFETHEN 

THE SEMIDISCRETE FOURIER TRANSFORM



Theorem If v then the semidiscrete Fourier transform

h

X

i x

j

v h h v F v h e

j h

j



b elongs to L and v can be recovered from v by the inverse semidiscrete Fourier

h

transform

Z

h

i x

j

v F vx v d j Z e

j

h

h

 

The norm of v and the L norm of v are related by Parsevals equality

h h

p

kvk kv k

 

d

If u and v or vice versa then v w andv w satises

h h h

d

v w v w

As in the continuous case the following prop erties of the semidiscrete Fourier transform

will b e useful In c and throughout this b o ok wherever convenient wetake advantage of

the fact thatv can b e treated as a p erio dic function dened for all R

THE SEMIDISCRETE FOURIER TRANSFORM TREFETHEN 

PROPERTIES OF THE SEMIDISCRETE FOURIER TRANSFORM



Theorem Let v w haveFourier transforms v w Then

h

a Linearity F fv w g v w F fcv g cv

h h

i x

j

b Translation If j Z then F fv g e v

h j j

i x

j

g v v c Mo dulation If R then F fe

j h

d Dilation If m Z with m then F fv g v mjmj

h mj

e Conjugation F f v g v

h

The symmetry prop erties of the Fourier transform summarized in Theorem apply

to the semidiscrete Fourier transform to o we shall not rep eat the list here

We come now to a fundamental result that describ es the relationship of the Fourier

transform of a continuous function u to that of a discretization v of uor if x and are

interchanged the relationship of Fourier series to Fourier transforms Recall that b ecause

of the phenomenon of aliasing all wavenumb ers j h j Z are indistinguishable



on the grid hZ Supp ose that u L is a suciently smo oth function dened on R and



be the discretization obtained let v by sampling ux at the points x The aliasing

j

h

principle implies that v must consist of the sum of all of the values u j h This

result is known as the Poisson summation formula or the aliasing formula

ALIASING FORMULA



Theorem Let u L b e suciently smo oth with Fourier transform u and let



b e the restriction of u to the grid hZ Then v

h

X

v u j h h h

j

Pro of Not yet written See P Henrici Applied and Computational Complex Analysis

v Wiley

In applications we are very often concerned with functions v obtained by discretization

and it will be useful to know how much the Fourier transform is aected in the pro cess

Theorems and combine to give the following answers to this question

THE SEMIDISCRETE FOURIER TRANSFORM TREFETHEN 

EFFECT OF DISCRETIZATION ON THE FOURIER TRANSFORM



Theorem Let v b e the restriction to the grid hZ of a function u L The following

estimates hold uniformly for all h h or a forteriori for in anyxedinterval

A A



a If u has p continuous derivatives in L for some p and a pth derivativein



L that has b ounded variation then

p

jv u j O h as h



b If u has innitely many continuous derivatives in L then

M

j v u j O h as h for all M

c If u can b e extended to an analytic function of z x iy in the complex strip j Im z j a

for some awithku iy kconst uniformly for each ay a then for any

ah

jv u j O e as h

ajz j

d If u canbeextendedtoanentire function of z x iy with uz O e as jz j

z C for some a then

v u provided h a

In part c u iy denotes a function of x namely ux iy with x interpreted as a variable

and y as a xed parameter

Pro of In each part of the theorem ux is smo oth enough for Theorem to apply

whichgives the identity

X

v u u j hu j h

j

Note that since h h the arguments of u in this series have magnitudes at least

hhh

p

ju jC j j for some constant C For part a Theorem a asserts that

With this implies

X X

p p p

jv u jC jh C h j



j j

Since p this sum converges to a constant which implies as required

Part b follows from part a

For part c

us reduces to for all For part d note that if ha then h a Th

h h as claimed

Note that part d of Theorem asserts that on a grid of size h the semidiscrete

Fourier transform is exact for bandlimited functions containing energy only at wavenumb ers

THE SEMIDISCRETE FOURIER TRANSFORM TREFETHEN

j j smaller than hthe Nyquist wave number corresp onding to two grid p oints p er

wavelength This twop ointsp erwavelength restriction is famous among engineers and has

practical consequences in everything from ghter planes to compact disc players When we

come to discretize solutions of partial dierential equations twopoints p er wavelength will

b e the coarsest resolution we can hop e for under normal circumstances

EXERCISES



a Prove

h h h

 

b Give examples to show that these inclusions are prop er and

h h h h

e examples to show that neither inclusion in a holds for functions on continuous c Giv

 

domains L L and L L

 

Let b e the discrete dierentiation and smo othing op erators dened by

h h

v v v v v v

j j  j j j j 



h



a Showthat and are equivalent to with appropriate sequences d m

h

Be careful with factors of h

b Compute the Fourier transforms d and m How do es d compare to the transform of

the exact dierentiation op erator for functions dened on R Theorem f Illustrate

this comparison with a sketchofd against

c Compute kd k kd k kmkandkm k and verify Parsevals equality

d Compute the F ourier transforms of the convolution sequences corresp onding to the

p p

iterated op erators and p Discuss how these results relate to the rule of

thumb discussed in the last section the smo other the function the more rapidly its

Fourier transform decays as j j What imp erfection in do es this analysis bring

to light

Continuation of Exercise Let v be the discretization on the grid hZ of the



function uxx

a Determinev Hint calculating it from the denition is very dicult

b How fast do es v approach u ash Give a precise answer based on athen

compare your answer with the prediction of Theorem

c What would the answer to b have b een if the roles of u andu had b een interchanged

that is if v had b een the discretization not of ux but of its Fourier transform



Integration by the trap ezoid rule A function u L L can b e integrated approxi

mately by the trap ezoid rule

Z

X

I ux dx I h ux

h j

j

This is an innite sum but in practice one might delete the tails if u decays suciently

rapidly as jxj This idea leads to excellent quadrature algorithms even for nite

intervals which are rst transformed to the real axis byachange of variables for a survey

THE SEMIDISCRETE FOURIER TRANSFORM TREFETHEN

see M Mori Quadrature formulas obtained byvariable transformation and the DErule

J Comp Appl Math

As h how go o d an approximation is I to the exact integral I Of course the answer

h

will dep end on the smo othness of ux

a State how I is related to the semidiscrete Fourier transform

h

b Give a b ound for jI I j based on the theorems of this section

h



x

c In particular what can you sayabout jI I j for the function uxe

h

d Showthatyour b ound can b e improved in a certain sense by a factor of

integers not the Draw a plot of sin n as a function of n where n ranges over the

real numb ers from to That is your plot should contain dots in Matlab this

can b e done in one line Explain why the plot lo oks the wayitdoestothehuman eye and

what this has to do with aliasing Makeyour explanation precise and quantitative See G

Strang WellesleyCambridge Press

INTERPOLATION AND SINC FUNCTIONS TREFETHEN

Interp olation and sinc functions

This section is not written yet but heres a hintastowhatwillbeinit

If is the Kronecker delta function

j

if j

j

if j

then gives the semidiscrete Fourier transform

h for all

j

If wenow apply the inverse transform formula we nd after a little algebra

sinx h

j

j

x h

j

at least for j Since x h is a nonzero integer for each j the sines are zero and this

j

formula matches

Supp ose however that weevaluate not just for x x but for all values x R

j

Then weve got a sinc function again one that can b e called a grid sinc function

sinxh

S x

h

xh

The plot of S x is the same as the upp erright plot of Figure except scaled so that

h

the zeros are on the grid ie at integer multiples of h Obviously S x is a continuous

h

interp olant to the discrete delta function Which one It is the unique bandlimited

j

c

interp olant bandlimited in the sense that its Fourier transform S is zero for

h

h h Pro of by construction its bandlimited in that way and uniqueness can b e

proved via an argumentbycontradiction making use of Parsevals equality

More generally supp ose wehave an arbitrary grid function v well not quite arbitrary

j

w Then the band well need certain integrability assumptions but lets forget that for no

limited interp olant to v is the unique function v x dened for x R with v x v

j j j

andv for h h It can b e derived in two equivalentways

Method Fourier transform Given v compute the semidiscrete Fourier transform

j

v Then invert that transform and evaluate the resulting formula for all x rather than

just on the grid

Method linear combination of sinc functions Write

X

v v

m mj j

m

INTERPOLATION AND SINC FUNCTIONS TREFETHEN 

and then set

X

v x v S x x

m h m

m

The equivalence of Metho ds and is trivial it follows from the linearity and translation

invariance of all the pro cesses in question

The consideration of bandlimited interp olation is a go o d way to get insightinto the

Aliasing Formula presented as Theorem In fact mayb e that should go in this section

The following schema summarizes everything Study it

FT

ux u

ALIASING

DISCRETIZE l

FORMULA

FT

v v

j

ZERO HIGH BANDLIMITED

WAVE NOS INTERPOLATION

FT

v x v

The Gibbs phenomenon is a particular phenomenon of bandlimited interp olation

that has received much attention After an initial discovery by Wilbraham in it was

made famous by Michelson in in a letter to Nature and then by an analysis byGibbs

in Nature the next year Gibbs showed that if the step function

x

ux

x

is sampled on a grid and then interp olated in the bandlimited manner then the resulting

function v x exhibits a ringing eect it oversho ots the limits by ab out achieving

amaximum amplitude

Z

siny

dy

y

The ringing is scaleinvariant it do es not go awayas h In the nal text I will illustrate

the Gibbs phenomenon and include a quick derivation of

THE DISCRETE FOURIER TRANSFORM TREFETHEN 

The discrete Fourier transform

Note although the results of the last two sections will b e used throughout the remain

der of the b o ok the material of the present section will not b e needed until Chapters and

For the discrete Fourier transform b oth x and inhabit discrete b ounded domains

or if your prefer they are p erio dic functions dened on discrete unb ounded domains Thus

there is a pleasing symmetry here as with the Fourier transform that was missing in the

semidiscrete case

N N

x x x

N N N N

 

   

*********ooooooooo *********ooooooooo

x

h

Figure Space and wavenumb er domains for the discrete Fourier trans

form

For the fundamental spatial domain we shall take as illustrated in Figure

Let N b e a p ositiveeven integer set

N even h

N

and dene x jh for any j The grid p oints in the fundamental domain are

j

x x x h

N N

An invaluable identitytokeep in mind is this

N

h



denote the set of functions on fx g that are N p erio dic with resp ect to j ie Let

j

N

p erio dic with resp ect to x with the norm

N

h i

X





kv k h jv j

j

j N

Since the sum is nite the norm is nite so every function of the required typ e is guaranteed



to b elong to and to and The discrete Fourier transform DFT of a

N N N



function v is dened by

N

N

X

i x

j

v Z e h v F v

j N

j N

THE DISCRETE FOURIER TRANSFORM TREFETHEN 

Since the spatial domain is p erio dic the set of wavenumbers is discrete and in fact

ranges precisely over the set of integers Z Thus it is natural to use as a subscript

N

X

i j h

v F v h e v Z

N j

j N

and since h N v is N p erio dic as a function of We shall takeNN as the



avenumb ers and let L denote the set of N p erio dic functions fundamental domain of w

N

on the grid Z with norm

N

h i

X





kvk jv j

N

This is nonstandard notation for an upp er case L is normally reserved for a family of

functions dened on a continuum We use it here to highlight the relationship of the discrete

Fourier transform with the semidiscrete Fourier transform



The convolution of two functions in is dened by

N

N N

X X

v w h v w h v w

m mj j j mj

j N j N

Again since the sum is nite there is no question of convergence

Here is a summary of the discrete Fourier transform

THE DISCRETE FOURIER TRANSFORM TREFETHEN 

THE DISCRETE FOURIER TRANSFORM



Theorem If v thenthediscreteFourier transform

N

N

X

N N

i j h

v F v h e v

N j

 

j N



b elongs to L and v can b e recovered from v by the inverse discrete Fourier trans

N

form

N

X

i j h

e v v F v

j j

N

N

 

The norm of v and the L norm of v are related by Parsevals equality

N N

p

kvk kv k



d

If v w then v w satises

N

d

v w v w

As with the other Fourier transforms wehave considered the following prop erties of

the discrete Fourier transform will b e useful Once again wetake advantage of the fact that

v can b e treated as a p erio dic function dened for all Z

PROPERTIES OF THE DISCRETE FOURIER TRANSFORM



have discrete Fourier transforms v w Then Theorem Let v w

N

a Linearity F fv w g v w F fcv g cv

N N

i x

j

g e v b Translation If j Z then F fv

N j j

i x

j

c Mo dulation If Z then F fe v g v

N j

v g v e Conjugation F f

N



v g Inversion F fv g

N

h

An enormously imp ortant fact ab out discrete Fourier transforms is that they can b e

computed rapidly by the recursive algorithm known as the fast Fourier transform FFT



A direct implementation of or requires N arithmetic op erations but the

The fast Fourier transform was discovered by Gauss in at the age of but although he wrote

a pap er on the sub ject he did not publish it and the idea was more or less lost until its celebrated

THE DISCRETE FOURIER TRANSFORM TREFETHEN 

FFT is based up on a recursion that reduces this gure to N log N We shall not describ e

the details of the FFT here but refer the reader to various b o oks in numerical analysis signal

pro cessing or other elds However to illustrate how simple an implementation of this idea

may b e Figure repro duces the original Fortran program that app eared in a pap er

by Co oley Lewis and Welchy Assuming that N is a p ower of it computes F in

N

our notation the vector A N representsv v on input and v v on

N N

output

subroutine fftam

complex auwt

n m do l m

nv n le l

le le nm n

j u

do i nm ang le

if igej goto w cmplxcosangsinang

t aj do j le

aj ai do i jnle

ai t ip ile

k nv t aipu

if kgej goto aip ait

j jk ai ait

k k u uw

goto return

j jk end

Figure Complex inverse FFT program of Co oley Lewis and Welch

As mentioned ab ov e this program computes the inverse Fourier transform according

to our denitions times The same program can b e used for the forward transform by

making use of the following identity

rediscovery byCooleyandTukey in See M T Heideman et al Gauss and the history

of the fast Fourier transform IEEE ASSP Magazine Octob er Since then fast Fourier

transforms havechanged prevailing computational practices in many areas

y Before publication p ermission to print this program will b e secured

THE DISCRETE FOURIER TRANSFORM TREFETHEN 

v F fvg hF fvg

N N

These equalities follow from parts e and g of Theorem resp ectively

VECTORS AND MULTIPLE SPACE DIMENSIONS TREFETHEN 

Vectors and multiple space dimensions

Fourier analysis generalizes with surprising ease to situations where the indep endent

variable x andor the dep endentvariable u are vectors We shall only sketch the essentials

which are based on the following two ideas

If x is a dvector then the dual variable is a dvector to o and the Fourier integral is

amultiple integral involving the inner pro duct x

If u is an N vector then its Fourier transform u is an N vector to o and is dened

comp onentwise

As these statements suggest our notation will b e as follows

T

x x x d numb er of space dimensions

d

T

N numb er of dep endentvariables u u u

N

Both and u b ecome vectors of the same dimensions

T T

u u u

d N

and x b ecomes the dot pro duct x x x The formulas for the Fourier

d d

transform and its inverse read

Z

i x

u F u e ux dx

Z Z

i x

e ux dx dx

d

d

for R and

Z

d i x

uxF ux e u d

Z Z

d i x

e u d d

d

d

for x R In other words u and u are related comp onentwise

 N  T

u u u



If the vector L norm is dened by

Z Z Z

  

kuk kuxk dx kuxk dx dx

d

VECTORS AND MULTIPLE SPACE DIMENSIONS TREFETHEN 

where the symbol kk in the integrand denotes the norm on vectors of length N then

Parsevals equality for vector Fourier transforms takes the form

d

kuk kuk

 N

The set of vector functions with b ounded vector norms can b e written simply as L

Before sp eaking of convolutions wehave to go a step further and allow uxandu

to be M N matrices rather than just N vectors The denitions ab ove extend to this

further case unchanged if the symbol kk in the integrand of now represents the

norm largest singular value of a matrix If uxisanM P matrix and v xisaP N

matrix then the convolution u v is dened by

Z

u v x ux y v y dy

Z

uy v x y dy

Z Z

ux y v y dy dy

d

and it satises

d

u v u v

Since matrices do not commute in general it is no longer p ossible to exchange u and v as

in

This generalization of Fourier transforms and convolutions to matrix functions is far

from idle for we shall need it for the Fourier analysis of multistep nite dierence approxi

mations such as the leap frog formula

Similar generalizations of our scalar results hold for semidiscrete and discrete Fourier

transforms

EXERCISES

What is the Fourier transform of the vector function

T

sin x sin x

ux

x x

dened for x R

What is the Fourier transform of the scalar function

 



x x 

 



uxe



T

dened for x x x R



CHAPTER TREFETHEN

Chapter

Finite Dierence Approximations

Scalar mo del equations

Finite dierence formulas

Spatial dierence op erators and the metho d of lines

Implicit formulas and linear algebra

Fourier analysis of nite dierence formulas

Fourier analysis of vector and multistep formulas

Notes and references

By a small sample wemay judge of the whole piece

MIGUEL DE CERVANTES Don Quixote de la Mancha Chap

CHAPTER TREFETHEN

This chapter b egins our study of timedep endent partial dierential equa

tions whose solutions vary b oth in time as in Chapter and in space as

in Chapter The simplest approach to solving partial dierential equations

numerically is to set up a regular grid in space and time and compute approx

imate solutions on this grid by marching forwards in time The essential p oint

is discretization

Finite dierence mo deling of partial dierential equations is one of several

elds of science that are concerned with the analysis of regular discrete struc

tures Another is digital signal pro cessing already mentioned in Chapter

where continuous functions are discretized in a similar fashion but for quite

dierent purp oses A third is crystallography whichinvestigates the b ehav

ior of physical structures that are themselves discrete The analogies b etween

these three elds are close and we shall o ccasionally point them out The

reader who wishes to pursue them further is referred to DiscreteTime Signal

Pro cessing by A V Opp enheim and R V Schafer and to An Intro duction

to Solid State Physics by C Kittel

This chapter will describ e ve dierent ways to lo ok at nite dierence

formulasas discrete approximations to derivatives as convolution lters as

To eplitz matrices as Fourier multipliers and as derivatives of p olynomial in

terp olants Each of these points of view has its adv antages and the reader

should b ecome comfortable with all of them

The eld of partial dierential equations is broad and varied as is in

evitable b ecause of the great diversityofphysical phenomena that these equa

tions mo del Much of the variety is intro duced by the fact that practical

problems usually involve one or more of the following complications

multiple space dimensions

systems of equations

b oundaries

variable co ecients

nonlinearity

To b egin with however we shall concentrate on a simple class of problems

pure nite dierence mo dels for linear constantco ecient equations on

innite onedimensional domain The fascinating phenomena that emerge an

from this study turn out to be fundamental to an understanding of the more

complicated problems to o

SCALAR MODEL EQUATIONS TREFETHEN

Scalar mo del equations

Partial dierential equations fall roughly into three great classes which

can be lo osely describ ed as follows

elliptic timeindep endent

parab olic timedep endent and diusive

hyp erb olic timedep endent and wavelike nite sp eed of propagation

In some situations this trichotomy can be made mathematically precise but

not always and we shall not worry ab out the rigorous denitions The reader

is referred to various books on partial dierential equations such as those by

John Garab edian or Courant and Hilb ert There is a particularly careful dis

cussion of hyp erb olicity in G B Whithams b o ok Linear and Nonlinear Waves

tial equations in general the state of the art among For linear partial dieren

pure mathematicians is set forth in the fourvolume work by L Hormander

The Analysis of Linear Partial Dierential Op erators

Until Chapter we shall consider only timedep endent equations

The simplest example of ahyp erb olic equation is

u u

t x

the onedimensional rstorder wave equation which describ es advection

of a quantity ux t at the constant velo city Given suciently smo oth

initial data ux u x has the solution

u x t u x t

ascanbeveried by inserting in see Figure b This solution

is unique The propagation of energy at a nite sp eed is characteristic of

hyp erb olic partial dierential equations but this example is atypical in having

all of the energy propagate at exactly the same nite sp eed

The simplest example of a parab olic equation is

u u

t xx

the onedimensional heat equation which describ es diusion of a quan

tity such as heat or salinity In this book u denotes the partial derivative

xx

u and so on For an initialvalue problem u x and similarly with u

xxx xt

SCALAR MODEL EQUATIONS TREFETHEN

a

b

c

d

Figure Evolution to t of a hatshap ed initial data under

b the wave equation c the heat equation and d

the Schrodinger equation the real part is shown

dened by and suciently wellb ehaved initial data ux u x the

solution

Z

i x t

ux t e u d



Z

xs t

p

e u s ds



t

can be derived by Fourier analysis Physically asserts that the os

cillatory comp onent in the initial data of wave number decays at the rate

 t

e b ecause of diusion which is what one would exp ect from See

Figure c Incidentally is not the only mathematically valid solu

omake it unique restrictions on tion to the initialvalue problem for T

ux t must be added such as a condition of b oundedness as jxj This

phenomenon of nonuniqueness is typical of parab olic partial dierential equa

tions and results from the fact that is of lower order with resp ect to t

than x so that u x constitutes data on acharacteristic surface

A third mo del equation that we shall consider from time to time is the

onedimensional Schrodinger equation

u iu

t xx

In fact it was Joseph Fourier who rst derived the heat equation equation in He then invented

Fourier analysis to solve it

SCALAR MODEL EQUATIONS TREFETHEN

which describ es the propagation of the complex state function in quantum

mechanics and also arises in other elds such as underwater acoustics Just

as with the heat equation the solution to the Schrodinger equation can be

expressed by an integral

Z

i xi t

ux t e u d



Z

ixs t

p

e u s ds



it

but the behavior of solutions to this equation is very dierent Schrodingers

equation is not diusive but disp ersive which means that rather than de

caying as t increases solutions tend to break up into oscillatory wavepackets

See Figure d

Of course and can all be mo died to incorp orate

constant factors other than so that they b ecome u au u au u

t x t xx t

iau This aects the sp eed of advection diusion or disp ersion but not the

xx

essential mathematics The constant can be eliminated by a rescaling of x or

t so we omit it in the interests of simplicity Exercise

The b ehavior of our three mo del equations for a hatshap ed initial function

is illustrated in Figure The three waves shown there are obviously

very dierent In b nothing has happ ened except advection In c strong

dissipation or diusion is evident sharp corners have been smo othed The

Schrodinger result of d exhibits disp ersion oscillations have app eared in

an initially nonoscillatory problem These three mechanisms of advection

dissipation and disp ersion are central to the behavior of partial dierential

equations and their discrete mo dels and together account for most linear

phenomena We shall fo cus on them in Chapter

Since many of the pages ahead are concerned with Fourier analysis of

nite dierence and sp ectral approximations to and we

should say a few words here ab out the Fourier analysis of the partial dierential

equations themselves The fundamental idea is that when an equation is linear

and has constant co ecients it admits plane wave solutions of the form

ixt

ux t e R C

where is again the wave number and is the frequency Another wayto

i x

putitistosay that if the initial data ux e are supplied to an equation

of this kind then there is a solution for t consisting of ux multiplied by

i t

an oscillatory factor e The dierence between various equations lies in the

dierentvalues of they assign to eachwavenumber and this relationship

SCALAR MODEL EQUATIONS TREFETHEN

is known as the disp ersion relation for the equation For rstorder examples

it might b etter be called a disp ersion function but higherorder equations

typically provide multiple values of for each and so the more general term

relation is needed See Chapter

It is easy to see what the disp ersion relations are for our three mo del

ixt

scalar equations For example substituting e into gives i i

or simply Here are the three results

u u

t x

u u i

t xx

u iu

t xx

Notice that for the wave and Schrodinger equations R for x R these

equations conserve energy in the L norm For the heat equation on the

other hand the frequencies are complex every nonzero R has Im

which by corresp onds to an exp onential decay and the L energy is

not conserved

The solutions and can b e derived byFourier synthesis from

the disp ersion relations and For example for the heat equa

tion and imply

Z

i x t

e u d ux t



Z Z



i x t i x  

e e ux dx d

 

From here to is just a matter of algebra

Equations and will serve as our basic mo del equa

tions for investigating the fundamentals of nitedierence and sp ectral meth

ods This may seem o dd since in all three cases the exact solutions are known

so that numerical metho ds are hardly called for Yet the study of numerical

metho ds for these equations will reveal many of the issues that come up rep eat

edly in more dicult problems In some instances the reduction of complicated

problems to simple mo dels can be made quite precise For example a hyp er

b olic system of partial dierential equations is dened to be one that can be

lo cally diagonalized into a collection of problems of the form see Chap

ter In other instances the guidance given by the mo del equations is more

heuristic

The backwards heat equation u u has the disp ersion relation i and its solutions

xx

t

blow upatanunb ounded rate as as t increases unless the range of wavenumb ers present is limited

The initialvalue problem for this equation is illp osed in L

SCALAR MODEL EQUATIONS TREFETHEN

EXERCISES

Showby rescaling x andor t that the constants a and b can b e eliminated in a

u au b u bu c u au bu

t x t xx t x xx

Consider the secondorder wave equation u u

tt xx

a What is the disp ersion relation Plot it in the real plane and b e sure to show all

values of for each

R

xt

f x tf x t g s ds is the solution b Verify that the function ux t

xt

corresp onding to initial data ux f x u x g x

t

a Verify that and represent solutions to and b oth dieren

tial equation and initial conditions

b Fill in the derivation of ie justify the second equals sign

Derive a Fourier integral representation of the solution to the initialvalue

problem u u ux u x

t x

a If is written as a convolution ux tu x h x what is h x This

t t

function is called the

b Prove that if u x is a continuous function with compact supp ort then the resulting

solution ux t to the heat equation is an entire function of x for each t

c Outline a pro of of the Weierstrass approximation theorem if f is a continuous

function dened on an interval a b then for any there exists a p olynomial px

such that jf x pxj for x a b

Metho d of characteristics Supp ose u ax tu and ux u x for x R

t x

and t where ax t is a smo othly varying p ositive function of x and t Then ux tis

constant along characteristic curves with slop e ax t

t

ux t

Figure

x

a Derive a representation for u as the solution to an ODE initialvalue problem

tcos x

b Find u to vedigit accuracy for the problem u e u ux x

t x

Plot the appropriate characteristic curve

c Find u to vedigit accuracy for the same equation dened on the interval x

with righthand b oundary condition utt Plot the appropriate charac

teristic curve

FINITE DIFFERENCE FORMULAS TREFETHEN

Finite dierence formulas

Let h and k b e a xed space step and time step resp ectively and set x jh

j

and t nk for anyintegers j and n The p oints x t dene a regular grid or mesh in

n n

j

two dimensions as shown in Figure formally the subset hZ k Z of R For the rest

n

of this b o ok our aim is to approximate continuous functions ux tby grid functions v

j

n

v ux t

j j n

n n

v x t v will also b e convenient and we shall sometimes write v or v t The notation

n n

j j

n

to represent the spatial grid function fv g j Z for a xed value n

j

l k

h

Figure Regular nite dierence grid in x and t

The purp ose of discretization is to obtain a problem that can be solved by a nite

pro cedure The simplest kind of nite pro cedure is an sstep nite dierence formula

n

which is a xed formula that prescrib es v as a function of a nite numberofothergrid

j

values at time steps n s through n explicit case or n implicit case To compute

n s

an approximation fv g to ux t we shall begin with initial data v v and then

j

s s

compute values v v in succession by applying the nite dierence formula This

pro cess is sometimes known as marching with resp ect to t

A familiar example of a nite dierence mo del for the rstorder wave equation

is the leap frog LF formula

n n

n n

v v v v LF

j j 

j j

k h

This equation can b e obtained from by replacing the partial derivatives in x and t by

centered nite dierences The analogous leap frog typ e approximation to the heat equation

is

n n

n n n

LF v v v v v

xx j j j 

j j

k h

However we shall see that this formula is unstable A b etter approximation is the Crank

Nicolson CN formula

n n n n

n n n n

v v CN v v v v v v

j  j j j

j j j j 

k h h

Sp elling note the name is Nicolson not Nicholson

FINITE DIFFERENCE FORMULAS TREFETHEN

n

which is said to b e implicit since it couples together the values v at the new time step

j

and therefore leads to a system of equations to b e solved In contrast leap frog formulas

are explicit One can also dene a CN formula for namely

n n n

n n n

CN v v v v v v

x j j j 

j j j 

k h h

but we shall see that since explicit formulas such as LF are stable for and easier to

implement an implicit formula like has little to recommend it in this case Another

famous and extremely imp ortant explicit approximation for u u is the LaxWendro

t x

formula discovered in

k

n

n n n n n n

v v v v v v v LW

j j j  j j j 

j

h

k h

The second term on the right is the rst wehave encountered whose function is not imme

diately obvious we shall see later that it raises the order of accuracy from to We shall

see also that although the leap frog formula may b e suitable for linear hyp erb olic problems

such as arise in acoustics the nonlinear hyp erb olic problems of uid mechanics generally

require a formula like LaxWendro that dissipates energy at high wavenumbers

We shall often use acronyms such as LF CN and LW to abbreviate the names of

standard nite dierence formulas as ab ove and subscripts x or xx will b e added sometimes

to distinguish b etween a mo del of the wave equation and a mo del of the heat equation For

the formulas that are imp ortant in practice we shall usually manage to avoid the subscripts

Of the examples ab ove as already mentioned LF and CN are imp ortant in practice

while LF and CN are not so imp ortant

xx x

Before intro ducing further nite dierence formulas we need a more compact notation

Chapter intro duced the time shift op erator Z

n

n

v Zv

j

j

Similarlylet K denote the space shift op erator

n n

v Kv

j j

and let I or represent the identity op erator

n n n

v v Iv

j j j

We shall make regular use of the following discrete op erators acting in the space direction

SPATIAL DIFFERENCE AND AVERAGING OPERATORS

 

I K K I K K



 

K I I K K K



h h h



K I K



h

FINITE DIFFERENCE FORMULAS TREFETHEN

and are known as forward backward and centered spatial averaging



op erators and are the corresp onding spatial dierence op erators of rst

order and is a centered spatial dierence op erator of the second order For discretization



in time we shall use exactly the same notation but with sup erscripts instead of subscripts

TEMPORAL DIFFERENCE AND AVERAGING OPERATORS

  

I Z Z I Z Z

  

Z I I Z Z Z

k k k

 

Z I Z

k

In this notation for example the LF and CN formulas and can be

rewritten

v v CN v v LF



Note that since Z and K commute ie ZK KZ the order of the terms in any pro duct

of these discrete op erators can b e p ermuted at will For example wemight have written

ab ove instead of

 

Since all of these op erators dep end on h or on k a more complete notation would b e

h h h etc For example the symbol h is dened by





hv K K v v v

j j j j 

h h

and similarly for h etc Here and in subsequentformulas subscripts or sup erscripts

are omitted when they are irrelevant to the discrete pro cess under consideration

In general there maybemanyways to write a dierence op erator For example

h

   

As indicated ab ove a nite dierence formula is explicit if it contains only one nonzero

term at time level n eg LF and implicit if it contains several eg CN As in the

ODE case implicit formulas are typically more stable than explicit ones but harder to

implement On an unb ounded domain in space in fact an implicit formula would seem

n n

to require the solution of an innite system of equations to get from v to v This is

essentially true and in practice a nite dierence formula is usually applied on a b ounded

mesh where a nite system of equations must b e solved Thus our discussion of unb ounded

meshes will b e mainly a theoretical devicebut an imp ortant one for many of the stability

and accuracy phenomena that need to b e understo o d have nothing to do with b oundaries

In implementing implicit nite dierence formulas there is a wide gulf b etween one

dimensional problems which lead to matrices whose nonzero entries are concentrated in a

The notations are reasonably common if not quite standard The other

 

notations of this section are not standard

FINITE DIFFERENCE FORMULAS TREFETHEN

narrow band and multidimensional problems whichdonot The problem of howtosolve

such systems of equations eciently is one of great imp ortance to whichwe shall return in

x and in Chapters

We are now equipp ed to presentanumberofwellknown nite dierence formulas for

the wave and heat equations These are listed in Tables wave equation and

heat equation and the reader should takethe time to b ecome familiar with them The

tables make use of the abbreviations

k k

h h

whichwill app ear throughout the b o ok The diagram to the rightof eachformula in the

tables whose meaning should b e selfevident is called the stencil of that formula More

extensive lists of formulas can b e found in a number of books For the heat equation for

example see Chapter of the b o ok byRichtmyer and Morton

Of the formulas mentioned in the tables the ones most often used in practice are

probably LF UW upwind and LWLaxWendro for hyp erb olic equations and CN

and DF DuFortFrankel for parab olic equations However computational problems vary

enormously and these judgments should not b e taken to o seriously

As with linear multistep formulas for ordinary dierential equations it is useful to

ve a notation for an arbitrary nite dierence formula for a partial dierential equation ha

The following is an analog of equation

An sstep linear nite dierence formula is a scalar formula

s r

X X

n

v

j



for some and for some for some constants f g with

r





If for all the formula is explicit whereas if for some

it is implicit Equation also describ es a vectorvalued nite dierence formula

n

in this case each v is an N vector each is an N N matrix and the conditions

j

b ecome det

The analogy between linear nite dierence formulas and linear

n

multistep formulas is imp erfect What has b ecome of the quantities ff g in

The answer is that was a general formula that applied to any ODE dened bya

function f u t p ossibly nonlinear the word linear there referred to the way f enters into

the formula not to the nature of f itself In by contrast wehave assumed that

the terms analogous to f u t in the partial dierential equation are themselves linear and

have b een incorp orated into the discretization Thus is more precisely analogous to

EXERCISES

Computations for Figure The goal of this problem is to calculate the curves of

w your mesh should extend over Figure by nite dierence metho ds In all parts b elo

FINITE DIFFERENCE FORMULAS TREFETHEN

an interval M M large enough to b e eectively innite At the b oundaries it is simplest

to imp ose the b oundary conditions uM tuM t

It will probably b e easiest to program all parts together in a single collection of subroutines

which accepts various input parameters to control h k M choice of nite dierence formula

and so on Note that parts c f and g involve complex arithmetic

The initial function for all parts is u x maxf jxjg and the computation is carried

to t

Please make your output compact by combining plots and numb ers on a page wherever

appropriate

a LaxWendro for u u Write a program to solve u u by the LWformula with

t x t x

k h h Make a table of the computed values v and the

error in these values for each h Make a plot showing the sup erp osition of the results

ie v x for various h and comment

b Euler for u u Extend the program to solve u u bytheEU formula with

t xx t xx xx

k h h Make a table listing v for each h Plot the results

and comment on them

c Euler for u iu Nowsolve u iu bytheEU formula mo died in the obvious

t xx t xx xx

waywith k h h can you go further Make a table listing v for

each h Your results will be unstable Explain why this has happ ened by drawing a

sketch that compares the stability region of a linear multistep formula to the set of

eigenvalues of a spatial dierence op erator This kind of analysis is discussed in the

next section

d Tridiagonal system of equations T o compute the answer more eciently for the heat

equation and to get any answer at all for Schrodingers equation it is necessary to use

an implicit formula whichinvolves the solution of a tridiagonal system of equations at

each time step Write a subroutine TRDIAGn c d e b x to solvethe linear system

of equations Ax b where A is the n n tridiagonal matrix dened by a c

ii i

a d a e The metho d to use is Gaussian elimination without pivoting of

ii i ii i

rows or columns if you are in doubt ab out howto do this you can nd details in

many books on numerical linear algebra or numerical solution of partial dierential

equations Test TRDIAG carefully and rep ort the solution of the system

x

x

B C B C B C

A A A

x

x

e CrankNicolson for u u Write down carefully the tridiagonal matrix equation that

t xx

is involved when u u is solved by the formula CN Apply TRDIAG to carry out

t xx

this computation with k h h Make a table listing v for

each h Plot the results and comment on them

f CrankNicolson for u iu Now write down the natural mo dication of CN for

t xx

h h solving u iu Making use of TRDIAG again solve this equation with k

t xx

The avoidance of pivoting is justiable provided that the matrix A is diagonally dominant as it

will b e in the examples we consider Otherwise Gaussian elimination may b e unstable see Golub

Van Loan Matrix Computations nd ed Johns Hopkins

FINITE DIFFERENCE FORMULAS TREFETHEN

Make tables listing v and v for each h Plot the results

b oth Re v x and jv x j sup erimp osed on a single graphand commentonthem

Howfaraway do es the b oundary at x M have to b e to yield reasonable answers

g Articial dissipation In part f you mayhave observed spurious wiggles contaminating

the solution These can be blamed in part on unwanted reections at the numerical

b oundaries at x M and we shall havemoretosay ab out them in Chapters and

To suppress these wiggles try adding a articial dissipation term to the righthand

side of the nite dierence formula suchas

a

n

v hu x t

 j xx j n

h

for some a What choices of M and a b est repro duce Figure d Do es it help

to apply the articial dissipation only near the b oundaries x M

Mo del equations with nonlinear terms Our mo del equations develop some inter

esting solutions if nonlinear terms are added Continuing the ab ove exercise mo dify your

programs to compute solutions to the following partial dierential equations all dened in

the interval x and with b oundary conditions u Devise whatever strategies

you can think of to handle the nonlinearities successfully such problems are discussed more

systematically in

a Burgers equation u u u Consider a LaxWendro typeofformula

t x xx

with say and initial data the same as in Figure Howdoesthenumerical

solution b ehave as t increases How do you think the exact mathematical solution

should b ehave

u

b Nonlinear heat equation u u e ux For this problem you will need a

t xx

variant of the CrankNicolson formula or p erhaps the backward Euler formula With

the aid of a simple adaptive timestepping strategy generate a p ersuasive sequence of

plots illustrating the blowup of the solution that o ccurs Make a plot of kutk



the maximum value of u as a function of t What is your best estimate based on

comparing results with several grid resolutions of the time at which kutk b ecomes



innite

c Nonlinear heat equation u u u ux cosx Rep eat part b for this

t xx

new nonlinearity Again with the aid of a simple adaptive timestepping strategy

generate a p ersuasive sequence of plots illustrating the blowup of the solution and

make a plot of kutk as a function of t What is your b est estimate of the time at



k b ecomes innite which kut



d Nonlinear Schrodinger equation u iu juj u Take the initial data from

t xx

Figure again How do es the solution b ehave as a function of t and howdoesthe

b ehavior dep end on Again try to generate a go o d set of plots and estimate the

critical value of t if there is one

Sp elling note the name is Burgers so one may write Burgers equation but never Burgers equation

THE TREFETHEN

EU Euler

x

n

n n n

v v v v v v

j j j

j

BE Backward Euler

x

n n n

n

v v v v v v

j

j j j

CN CrankNicolson

x

n n n

n n n

v v v v v v v v

j j j

j j j

LF Leap frog

n n

n n

v v v v v v

j j

j j

BOX Box

x

n n

n n

v v v v v v

j j

j j

LF Fourthorder Leap frog

n n

n n n n

v v v hv hv v v v v

j j j j

j j

LXF LaxFriedrichs

n

n n n n

Z v v v v v v v

j j j j

j

k

UW Upwind

n

n n n

v v v v v v

j j j

j

LW LaxWendro

n

n n n n n n

v v v k v v v v v v v



j j j j j j

j

Table Finite dierence approximations for the rstorder

wave equation u u with kh For the equation u au

t x t x

replace by a in each formula Names in parenthesis mark formu

las that are not normally useful in practice Orders of accuracy and

stability restrictions are listed in Table

THE METHOD OF LINES TREFETHEN

EU Euler

xx

n

n n n n

v v v v v v v



j j j j

j

BE Backward Euler Laasonen

xx

n n n n

n

v v v v v v v



j

j j j j

CN CrankNicolson

n n n n

n n n n

v v v v v v v v v v



j j j j

j j j j

LF Leap frog

xx

n n

n n n

v v v v v v v



j j j

j j

BOX Box

xx

I v v



CN Fourthorder CrankNicolson

v h hv

 

DF DuFortFrankel

n n n n

n n

v h K K v v v v v v v

j j

j j j j

SA Saulev

Table Finite dierence approximations for the heat equation

u u with kh For the equation u au replace by

t xx t xx

a in each formula Names in parenthesis mark formulas that are

not normally useful in practice Orders of accuracy and stability

restrictions are listed in Table

THE METHOD OF LINES TREFETHEN

Spatial dierence op erators and the

metho d of lines

In designing a nite dierence metho d for a timedep endent partial dif

ferential equation it is often useful to divide the pro cess into two steps rst

discretize the problem with resp ect to space thereby generating a system of

ordinary dierential equations in t next solve this system by some discrete

metho d in time Not all nite dierence metho ds can be analyzed this way

t but many can and it is a p oint of view that b ecomes increasingly imp ortan

as one considers more dicult problems

EXAMPLE For example supp ose u u is discretized in space by the approxi

t x

mation x Then the PDE b ecomes

v v

t

where v represents an innite sequence fv tg of functions of t This is an innite system

j

of ordinary dierential equations each one of the form

v

j

v v

j j 

t h

On a b ounded domain the system would b ecome nite though p ossibly quite large

A system of ordinary dierential equations of this kind is known as a

semidiscrete approximation to a partial dierential equation The idea of

constructing a semidiscrete approximation and then solving it by anumerical

metho d for ordinary dierential equations is known as the metho d of lines

The explanation of the name is that one can think of fv tg as an approxima

j

tion to ux t dened on an array of parallel lines in the xt plane as suggested

in Figure

v t

v t

v t



x

Figure The metho d of linessemidiscretization of a time

dep endent PDE

THE METHOD OF LINES TREFETHEN

EXAMPLE CONTINUED Several of the formulas of the last section can b e in

terpreted as timediscretizations of by linear multistep formulas The Euler and

Backward Euler discretizations and give the Euler and Backward Euler for

mulas listed in Table The trap ezoid rule gives the CrankNicolson formula of

and the midp oint rule gives the leap frog formula of On the other

hand the upwind formula comes from the Euler discretization like the Euler formula but

with the spatial dierence op erator instead of The LaxWendro and LaxFriedrichs

formulas do not t the semidiscretization framework

The examples just considered were rst and secondorder accurate ap

proximations with resp ect to t the denition of order of accuracy will come

in x Higherorder time discretizations for partial dierential equations

have also b ecome p opular in recent years although one would rarely go so

far as the sixth or eighthorder formulas that app ear in many adaptive ODE

co des The advantage of higherorder metho ds is of course accuracy One

disadvantage is complexity both of analysis and of implementation and an

other is computer storage For an ODE involving a few dozen variables there

is no great diculty if three or four time levels of data must be stored but

for a largescale PDEfor example a system of ve equations dened on a

mesh in three dimensionsthe storage requirements b ecome

quite large

The idea of semidiscretization fo cuses attention on spatial dierence op

erators as approximations of spatial dierential op erators It happ ens that

just as in Chapter many of the approximations of practical interest can be

derived by a pro cess of interp olation Given data on a discrete mesh the idea

is as follows

Interp olate the data

Dierentiate the in terp olantat the mesh p oints

In step one dierentiates once for a rstorder dierence op erator twice

for a secondorder dierence op erator and so on The spatial discretizations

of many nite dierence and sp ectral metho ds t the scheme the vari

ations among them lie in in the nature of the grid the choice of interp olating

functions and the order of dierentiation

EXAMPLE

First order of accuracy For example supp ose data v v are interp olated by a

j j

p olynomial q x of degree Then v q x See Figure a

x

j j

Unfortunatelytheword order customarily refers b oth to the order of a dierential or dierence

op erator and to the order of accuracy of the latter as an approximation to the former The reader is advised to b e careful

THE METHOD OF LINES TREFETHEN

x

*o *o q

q x

*o *o

v v

j j

*o *o

v v j *o *o j

o o v



* * j

x x x x x

j j j  j j

a b

Figure Derivation of spatial dierence op erators via p olyno

mial interp olation

Second order of accuracy Let data v v v be interp olated by a p olynomial q x

j  j j

of degree Then v q x and v q x See Figure b

x  xx

j j j j

Fourth order of accuracy Let v v v v v be interp olated by a p olynomial

j  j  j j j

hv hv the fourthorder approximation listed q x of degree Then q x

x

j j j

in Table

To pro ceed systematicallyletx x x b e a set of arbitrary distinct

n

max

points of R not necessarily uniformly spaced Supp ose we wish to derive the

m

co ecients c of all of the spatial dierence op erators centered at x of

nj

orders m m based on any initial subset x x x of these p oints

max n

That is we want to derive all of the approximations

n

m

X

d f

m

c f x m m m n n

nj j max max max

m

dx

j

in a systematic fashion The following surprisingly simple algorithm for this

purp ose was published by Bengt Fornb erg in Generation of nite dierence

formulas on arbitrarily spaced grids Math Comp

THE METHOD OF LINES TREFETHEN

FINITE DIFFERENCE APPROXIMATIONS ON AN ARBITRARY GRID

Theorem Given m and n m the following algorithm

max max max

computes co ecients of nite dierence approximations in arbitrary distinct

m m

to x m m at x as describ ed ab ove p oints x x

max n

max

c

for n to n

max

for j to n

x x

n

j

n

if n m then c

max

nj

for m to min n m

max

m

m m

c x c mc x x

n n

nj nj j

nj

for m to minn m

max

m

m m

c mc x c

nn n nn

nn



The undened quantities c app earing for m maybe taken to be

nj

Pro of Not yet written

From this single algorithm one can derive co ecients for centered onesided

and much more general approximations to all kinds of derivatives A number

are listed in Tables at the end of this section see also Exercise

If the grid is regular then simple formulas can be derived for these nite

dierence approximations In particular let D denote the discrete rst

p

order spatial dierence op erator obtained by interp olating v v by a

j p j p

m

p olynomial q x of degree p and then dierentiating q once at x and let D

j

p

be the analogous higherorder dierence op erator obtained by dierentiating

m times Then we have for example

D h D h



and

D h h D h h

 

THE METHOD OF LINES TREFETHEN

and

D h h h

D h h h

  

The corresp onding co ecients app ear explicitly in Table

The following theorem gives the co ecients for rst and secondorder

formulas of arbitrary order of accuracy

CENTERED FINITE DIFFERENCE APPROXIMATIONS ON A REGULAR GRID

Theorem For each integer p there exist unique rstorder and

of order of accuracy p that secondorder dierence op erators D and D

p

p

utilize the points v v namely

j p j p

p p

X X

D jh D jh

p j j 

p

j j

where

j

p p j p

j

j

p j p p j p j

Pro of Not yet written

As p and have the following formal limits

D h h h

D h h h

  

These series lo ok b oth innite and nonconvergentunimplementable and p os

sibly even meaningless However that is far from the case In fact they are

precisely the rstorder and secondorder sp ectral dierentiation op era

for data dened on the innite grid hZ The corresp onding interp olation tors

pro cesses involve trigonometric or sinc functions rather than p olynomials

Interp olate the data by sinc functions as in x

Dierentiate the interp olant at the mesh points

As was describ ed in x such a pro cedure can b e implemented by a semidis

crete Fourier transform and it is well dened for all data v The uses of

h

these op erators will b e the sub ject of Chapter

THE METHOD OF LINES TREFETHEN

One useful way to interpret spatial dierencing op erators such as D is

p

in terms of convolutions From it is easy to verify that the rstorder

op erators mentioned ab ove can be expressed as

D v v

h

v D v

h

D v v

h

recall Exercise In each of these formulas the sequence in parentheses

indicates a grid function w fw g with the zero in the middle representing

j

w Since w has compact supp ort except in the case D there is no problem

of convergence asso ciated with the convolution

Any convolution can also be thought of as multiplication by a To eplitz

matrixthat is a matrix with constant entries along each diagonal a

ij

For example if v is interpreted as an innite column vector v v a



ij

T

v then v b ecomes the leftmultiplication of v bytheinnite matrix

of the form

B C

B C

B C

B C

B C

B C

B C

h

B C

B C

A

All together there are at least ve distinct ways to interpret the construc

tion of spatial dierence op erators on a regular gridall equivalent but each

having its own advantages

Approximation of dierential op erators To the classical numerical

analyst a spatial dierence op erator is a nite dierence approximation to a

dierential op erator

Interp olation To the data tter a spatial dierence op erator is an ex

act dierential op erator applied to an interp olatory approximant as describ ed

ab ove This p oint of view is basic to sp ectral metho ds which are based on

global rather than lo cal interp olants

Convolution Tothe signal pro cessor a dierence op erator is a convo

lution lter whose co ecients happ en to b e chosen so that it has the eect of

dierentiation

To eplitz matrix multiplication To the linear algebraist it is mul

tiplication by a To eplitz matrix This point of view becomes central when

THE METHOD OF LINES TREFETHEN

problems of implementation of implicit formulas come up where the matrix

denes a system of equations that must be solved

Fourier multiplier Finally to the Fourier analyst a spatial dier

ence op erator is the multiplication of the semidiscrete Fourier transform by a

trigonometric function of which happ ens to approximate the polynomial

corresp onding to a dierential op erator

Going back to the metho d of lines idea of the b eginning of this section if

we view a nite dierence mo del of a partial dierential equation as a system

of ordinary dierential equations whichissolved numericallywhatcanwesay

ab out the stability of this system This viewp oint amounts to taking h

xed but letting k vary From the results of Chapter we would exp ect

meaningful answers in the limit k so long as the discrete ODE formula is

stable On the other hand if k is xed as well as h the question of absolute

stability comes up as in x Provided that the innite size of the system of

ODEs can safely b e ignored we exp ect timestability whenever the eigenvalues

of the spatial dierence op erator lie in the stability region of the ODE metho d

In subsequent sections we shall determine these eigenvalues byFourier analysis

and show that their lo cation often leads to restrictions on k as a function of h

EXERCISES

j

Nonuniform grids Consider an exp onentially graded mesh on with x hs

j

s Apply to derive a p ointcentered approximation on this grid to the rstorder

dierentiation op erator

x

Fornb ergs algorithm Write a brief program either numerical or b etter symb olic

to implementFornb ergs algorithm of Theorem Run the program in suchawayasto

repro duce the co ecients of backwards dierentiation formulas in Table and equiva

lently Table What are the co ecients for onesided halfwaypoint approximation

of zeroth rst and second derivatives in the p oints

LaxWendro formula Derive the LaxWendro formula via interp olation of

n n n

v v and v by a polynomial q x followed b y evaluation of q x at an appropriate

j  j j

point

THE METHOD OF LINES TREFETHEN

Table Co ecients for centered nite dierence approxima

tions from Fornb erg

THE METHOD OF LINES TREFETHEN

Table Co ecients for centered nite dierence approxima

tions at a halfway p oint from Fornb erg

THE METHOD OF LINES TREFETHEN

Table Co ecients for onesided nite dierence approxima

tions from Fornb erg

IMPLICIT FORMULAS AND LINEAR ALGEBRA TREFETHEN

Implicit formulas and linear algebra

This section is not yet written but here is a sketch

Implicit nite dierence formula lead to systems of equations to solve If

the PDE is linear this b ecomes a linear algebra problem Ax b while if it is

nonlinear an iteration will p ossibly be required that involves a linear algebra

problem at each step Thus it is hard to overstate the imp ortance of linear

algebra in the numerical solution of partial dierential equations

For a nite dierence grid involving N points in each of d space dimen

d d

sions A will have dimension N and thus N entries Most of these are

zero the matrix is sparse If there is just one space dimension A will have a

narrow bandwidth and Ax b can b e solved in N op erations by Gaussian

elimination or related algorithms Just a few remarks ab out solutions of this

kind First if A is symmetric and p ositive denite one normally preserves

this form by using the Cholesky decomp osition Second unless the matrix

is p ositive denite or diagonally dominant pivoting of the rows is usually

essential to ensure stability

When there are two or more space dimensions the bandwidth is larger

and the number of op erations go es up so algorithms other than Gaussian

elimination b ecome imp ortant Here are some typical op eration counts orders

of magnitude for the canonical problem of solving the standard vep oint

Laplacian nitedierence op erator on a rectangular domain For the iterative

metho ds denotes the accuracy of the solution typically log log N

and we have assumed this in the last line of the table

Algorithm D D D

Gaussian elimination N N N

banded Gaussian elimination N N N

Jacobi or GaussSeidel iteration N log N log N log

SOR iteration N log N log N log

conjugate gradient iteration N log N log N log

preconditioned CG iteration N log N log N log

nested dissection N N N log

fast Poisson solver N log N N log N N log N

multigrid iteration N log N log N log

full multigrid iteration N N N

IMPLICIT FORMULAS AND LINEAR ALGEBRA TREFETHEN

These algorithms vary greatly in howwell they can b e generalized to vari

able co ecients dierent PDEs and irregular grids Fast Poisson solvers are

the most narrowly applicable and multigrid metho ds despite their remarkable

sp eed the most general Quite a bit of programming eort may be involved

in multigrid calculations however

Two observations may b e made ab out the state of linear algebra in scien

tic computing nowadays First multigrid metho ds are extremely imp ortant

and b ecoming ever more so Second preconditioned conjugate gradient CG

metho ds are also extremely imp ortant as well as other preconditioned itera

tions such as GMRES BCG and QMR for nonsymmetric matrices These are

often very easy to implement once one nds a go o d preconditioner and can

be sp ectacularly fast See Chapter

FOURIER ANALYSIS OF FINITE DIFFERENCE FORMULAS TREFETHEN

Fourier analysis of

nite dierence formulas

In x we noted that a spatial dierence op erator D can b e interpreted as

a convolution Dv a v for some a with compact supp ort By Theorem

d

d

then Dv a v a v This fact is the basis it follows that if v

h

of Fourier analysis of nite dierence metho ds In this section we shall work

out the details for scalar onestep nite dierence formulas s in

next section will treating rst explicit formulas and then implicit ones The

extend these developments to vector and multistep formulas

To b egin in the simplest setting consider an explicit scalar onestep

nite dierence formula

r

X

n n n

Sv v v

j j

j



where f g are xed constants The symbol S denotes the op erator that

n n

maps v to v In this case of an explicit formula S is dened for arbitrary

sequences v and by we have

Sv a v a



h

which To be able to apply Fourier analysis however let us assume v

h

implies Sv also since S is a nite sum Then Theorem gives

h

c

d

Sv a v a v

EXAMPLE Upwind formula for u u The upwind formula Table is

t x

dened by

n

n n n n

v Sv v v v

j j j j

j

where kh By or this is equivalentto Sv a v with

if j

h

a

j

if j

h

otherwise

A go o d reference on the material of this section is the classic monograph byRDRichtmyer and

K W Morton Dierence Metho ds for InitialValue Problems Chapters and

FOURIER ANALYSIS OF FINITE DIFFERENCE FORMULAS TREFETHEN

By the Fourier transform is

i h i x i x



a e a e a he



The interpretation of a is simple it is the amplication factor by

which the comp onent in v of wave number is amplied when the nite dif

ference op erator S is applied Often we shall denote this amplication factor

simply by g a notation that will prove esp ecially convenient for later gen

eralizations

To determine g for a particular nite dierence formula one can always

apply the denition ab ove nd the sequence a then compute its Fourier

complicated however because of a transform This pro cess is unnecessarily

factor h that is divided out and then multiplied in again and also a pair

of factors in the exp onent and the subscripts that cancel Instead as a

practical matter it is simpler and easier to remember to insert the trial

n i j h n

g e in the nite dierence formula and see what expression solution v

j

for g g results

EXAMPLE CONTINUED To derive the amplication factor for the upwind

n n i j h

formula more quickly insert v g e in to get

j

n i j h n i j h i j h i j h

g e g e e e

n i j h

or after factoring out g e

i h

g e

EXAMPLE LaxWendro formula for u u The LaxWendro formula Table

t x

is dened by

n

n n n n n n n

v v Sv v v v v v

j j j j j  j j 

j

n n i j h n i j h

Inserting v g e and dividing by the common factor g e gives

j

i h i h i h i h

e e e e g

The two expressions in parentheses come up so often in Fourier analysis of nite dierence

formulas that it is worth recording what they are equivalentto

i h i h

e e i sin h

h

i h i h

e e cos h sin

The amplication factor function for the LaxWendro formula is therefore

h

g i sin h sin

FOURIER ANALYSIS OF FINITE DIFFERENCE FORMULAS TREFETHEN

EXAMPLE Euler formula for u u As an example involving the heat equation

t xx

consider the Euler formula of Table

n

n n n n n

v Sv v v v v

j j j j j 

j

n n i j h

where kh By insertion of v g e gives

j

h

g sin

Now let us return to generalities Since g a isbounded as a func

c

tion of implies the b ound kSvk kav kka k kv k by hence

kSvkkak kv k by where ka k denotes the supnorm

ka k max ja j

 h h

to Moreover since v Thus S is a b ounded linear op erator from

h h

could be chosen to be a function arbitrarily narrowly peaked near a wave

number with ja j kak this inequality cannot b e improved Therefore

kS k ka k

norm of S that is the norm on The symbol kS k denotes the op erator

h

the op erator S see App endix induced by the usual norm on

h h h

B

kSvk

kS k sup

kv k

v 

h

n

Rep eated applications of the nite dierence formula are dened by v

c

n n

n

c

S v and if v v Since a is just a scalar then v a

h

function of we have

n n n n

ka k ka k maxja j max ja j

and therefore by the same argument as ab ove

n n

kv kkak kv k

and

n n

kS k ka k

A comparison of and reveals that if S is the nite dif

ference op erator for an explicit scalar onestep nite dierence formula then

n n

op erators satisfy only the kS k kS k In general however b ounded linear

FOURIER ANALYSIS OF FINITE DIFFERENCE FORMULAS TREFETHEN

n n

inequality kS k kS k and this will be the b est we can do when we turn

to multistep nite dierence formulas or to systems of equations in the next

section

The results ab ove can b e summarized in the following theorem

FOURIER ANALYSIS OF

EXPLICIT SCALAR ONESTEP FINITE DIFFERENCE FORMULAS

Theorem The scalar nite dierence formula denes a b ound

ed linear op erator S with

h h

n n n

kS k ka k ka k for n

n n

and v S v then If v

h

n c

n

c

v a v

Z

h

n

c i x n

j

a v d e v

j

h

and

n n

kv kka k kv k

Pro of The discussion ab ove together with Theorem

Now let us generalize this discussion by considering an arbitrary onestep

scalar nite dierence formula which may be explicit or implicit This is the

sp ecial case of with s dened as follows

A onestep linear nite dierence formula is a formula

r r

X X

n n

v v

j

j

 

for some constants f g and f g with If for the

formula is explicit while if for some it is implicit

Equation is the sp ecial case of with for

n n

Again we wish to use to dene an op erator S v v but now

n

we have to be more careful Given any sequence v amounts to an

FOURIER ANALYSIS OF FINITE DIFFERENCE FORMULAS TREFETHEN

n

innite system of linear equations for the unknown sequence v In the

terms of the last section we must solve

n n

Bv Av

n

for v where A and B are innite To eplitz matrices But b efore the op erator

S will b e welldened we must make precise what it means to do this

n

The rst observation is easy Given any sequence v the righthand side

of is unambiguously dened since only nitely many numbers are

n

nonzero Likewise given any sequence v the lefthand side is unambigu

n n

ously dened Thus for any pair v v there is no ambiguity ab out whether

or not they satisfy it is just a matter of whether the two sides are

equal for all j We can write equivalently as

n n

b v a v

for sequences a b there is no ambiguity ab out what it

 

h h

n n

means forapairv v to satisfy

n

The diculty comes when we ask given a sequence v do es there exist

n

a unique sequence v satisfying In general the answer is no as is

shown by the following example

EXAMPLE CrankNicolson for u u The CrankNicolson formula is

t xx

n n n n

n n n n

v v v v v v v v

j j j j 

j j j j 

n

where kh Supp ose v is identically zero Then the formula reduces to

n n n

v v v

j j  j

n

One solution of this equation is v forall j and this is the right solution as far as

j

applications are concerned But is a secondorder recurrence relation with resp ect

n

j

to j and therefore it has two linearly indep endent nonzero solutions to o namely v

j

where is either ro ot of the characteristic equation

Thus solutions to implicit nite dierence formulas on an innite grid may

b e nonunique In general if the nonzero co ecients at level n extend over

a range of J grid points there will be a J dimensional space of p ossible

solutions at each step In a practical computation the grid will b e truncated

by b oundaries and the nonuniqueness will usually disapp ear However from

FOURIER ANALYSIS OF FINITE DIFFERENCE FORMULAS TREFETHEN

a conceptual p oint of view there is a more basic observation to b e made which

has relevance even to nite grids the nite dierence formula has a unique

solution in the space

h

Tomake this precise we need the following assumption which is satised

for the nite dierence formulas used in practice Let b denote as usual

the Fourier transform of the sequence b

Solvability Assumption for implicit scalar onestep nite dierence for

mulas

b for h h

Since b is hp erio dic this is equivalent to the assumption that b

for all R It is also equivalent to the statementthatnoroot of the char

acteristic equation analogous to lies on the unit circle in the complex

plane

n n

Now supp ose v and v are two sequences in that satisfy

h

Then Theorem implies

d

n n

c

b v a v

or by the solvability assumption

d

n n

c

v g v

where

a

g

b

Since g isa continuous function on the compact set h h it has

a nite maximum

a

kg k max

 h h

b

is uniquely determined by its Fourier transform There Now a function in

h

n n

fore implies that for any v there is at most one solution v

h h

to On the other hand obviously such a solution exists since

tells how to construct it

Wehave proved that if b satises the solvability assumption then

n n

for any v there exists a unique v satisfying In other

h h

n n

words denes a b ounded linear op erator S v v

This and related conclusions are summarized in the following generaliza

tion of Theorem

FOURIER ANALYSIS OF FINITE DIFFERENCE FORMULAS TREFETHEN

FOURIER ANALYSIS OF

IMPLICIT SCALAR ONESTEP FINITE DIFFERENCE FORMULAS

Theorem If the solvability assumption holds then the im

plicit nite dierence formula denes a bounded linear op erator

with S

h h

n n n

kS k kg k kg k for n

n n

and v S v then where g a b If v

h

n c

n

c

v g v

Z

h

i x

n n

j

e g v d v

j

h

and

n n

kv kkg k kv k

In principle the op erator S might b e implemented by computing a semi

discrete Fourier transform multiplying by a b and computing the in

verse transform In practice an implicit formula will be applied on a nite

grid and its implementation will usually be based on solving a nite linear

system of equations But as will b e seen in later sections sometimes the best

metho ds for solving this system are again based on Fourier analysis

EXAMPLE CONTINUED As with explicit formulas the easiest waytocalculate

n i j h n

g e amplication factors of implicit formulas is by insertion of the trial solution v

j

For the implicit CrankNicolson mo del of by this leads to

h h

g sin g sin

that is

h

sin

a

g

h

sin

b

where again kh Since the denominator b is p ositiveforall the CrankNicolson

formula satises the solvability assumption regardless of the value of

EXERCISES

Amplication factors Calculate the amplication factors for the a Euler b Crank

Nicolson c Box and d LaxFriedrichs mo dels of u u For the implicit formulas verify

t x

that the solvability assumption is satised

VECTOR AND MULTISTEP FORMULAS TREFETHEN

Fourier analysis of

vector and multistep formulas

It is not hard to extend the developments of the last section to vector or multistep

nite dierence formulas Both extensions are essentially the same for we shall reduce

a multistep scalar formula to a onestep vector formula in analogy to the reduction of

higherorder ordinary dierential equations in x and of linear multistep formulas in x

It is easiest to b egin with an example and then describ e the general situation

EXAMPLE Leap frog for u u The leap frog formula is

t x

n n

n n

v v v v

j j 

j j

n n

Let w fw g be the vectorvalued grid function dened by

j

n

v

j

n

w

j

n

v

j

Then the leap frog formula can b e rewritten as

n n n

n

v v v

v

j  j j

j

n

n  n n

v

v v v

j

j  j j

that is

n

n n n

w w w w

 j  j j

j

where



Equivalently

n n

w a w

 n N

where a is the innite sequence of matrices with a h x For w



h

wethenhave the set of N vector sequences with each comp onent sequence in

h

d

n n

c

w a w

As describ ed in x all of these transforms are dened comp onentwise From we

get

i sin h

a

VECTOR AND MULTISTEP FORMULAS TREFETHEN

This is the amplication matrix a orG for leap frog and values at later time steps

are given by

n

n

c

w a wc

In general an arbitrary explicit or implicit scalar or vector multistep

formula can be reduced to the onestep form where each v

j

is an N vector and each or is an N N matrix for a suitable value N

The same formula can also be written in the form if B and A are

innite To eplitz matrices whose elements are N N matrices ie A and B

are tensors or in the form if a and b are sequences of N N matrices

The condition b ecomes

Solvability Assumption for implicit vector onestep nite dierence for

mulas

det b for h h

The amplication matrix for the nite dierence formula is



G b a

and as b efore the nite dierence formula denes a b ounded linear op erator

N

on sequences of N vectors in h

VECTOR AND MULTISTEP FORMULAS TREFETHEN

FOURIER ANALYSIS OF

IMPLICIT VECTOR ONESTEP FINITE DIFFERENCE FORMULAS

Theorem If the solvability assumption holds the implicit

vector nite dierence formula denes a bounded linear op erator

N N

with S

h h

n n n

kS k kG k kGk for n

N n n 

and v S where G b a If v v then

h

n c

n

c

v G v

Z

h

i x

n n

j

v e G v d

j

h

and

n n

kv kkGk kv k

In and kGk is the natural extension of to the matrix

valued case it denotes the maximum

kGk max kG k

 h h

where the norm on the right is the matrix norm largest singular value of

n n

an N N matrix The formula kS k kS k is no longer valid in general for

vector nite dierence formulas

EXERCISES

Amplication matrices Calculate the amplication matrices for the a DuFort

Frankel and b fourthorder leap frog formulas of x Be sure to describ e precisely what

onestep vector nite dierence formula your amplication matrix is based up on

CHAPTER TREFETHEN 

Chapter

Accuracy Stability and Convergence

An example

The Lax Equivalence Theorem

The CFL condition

The von Neumann condition

Resolvents pseudosp ectra and the Kreiss Matrix Theorem

The von Neumann condition for vector or multistep formulas

Stability of the metho d of lines

Notes and references

Mighty oaks from little acorns grow

ANONYMOUS

CHAPTER TREFETHEN 

The problem of stability is p ervasive in the numerical solution of par

tial dierential equations In the absence of computational exp erience one

would hardly b e likely to guess that instabilitywas an issue at all yet it is a

dominant consideration in almost every computation Its impact is visible in

the nature of algorithms all across this sub jectmost basically in the central

imp ortance of linear algebra since stability so often necessitates the use of im

plicit or semiimplicit formulas whose implementation involves large systems

of discrete equations

The relationship between stability and convergence was hinted at by

Courant Friedrichs and Lewy in the s identied more clearly by von

Neumann in the s and broughtinto organized form byLaxandRichtmyer

in the sthe Lax Equivalence Theorem After presenting an example

we shall begin with the latter and then relate it to the CFL and von Neu

mann conditions After that we discuss the imp ortant problem of determining

stability of the metho d of lines For problems that lead to normal matrices it

is enough to make sure that the sp ectra of the spatial discretization op erators

lie within a distance O k of the stability region of the timestepping formula

but if the matrices are not normal one has to consider pseudosp ectra instead

The essential issues of this chapter are the same as those that came up

for ordinary dierential equations in Sections For partial dierential

er the details are more complicated and more interesting equations howev

In addition to Richtmyer and Morton a go o d reference on the material of

this chapter is V Thomee Stability theory for partial dierence op erators

SIAM Review

In particular L F Richardson the originator of nitedierence metho ds for partial dierential

equations did not discover instability see his b o ok Weather Prediction by Numerical Pro cesses

Cambridge University Press reprinted byDover in

AN EXAMPLE TREFETHEN 

An example

Consider the mo del partial dierential equation

u u x R t

t x

together with initial data

cos x jxj

ux

jxj

Let us solve this initialvalue problem numerically by the leap frog formula

with space and time steps

h k h

where is a constant known as the mesh ratio Thus the leap frog formula

takes the form

n n n n

v v v v

j j

j j

with the bump in the initial function represented by grid points The

starting values at t and k will b oth be taken from the exact solution

ux tux t

Figure shows computed results with and and they

are dramatically dierent For the leap frog formula is stable generat

ing a leftpropagating wave as exp ected For it is unstable The errors

intro duced at each step are not much bigger than b efore but they grow ex

ponentially in subsequent time steps until the wave solution is obliterated by

a sawto oth oscillation with points per wavelength This rapid blowup of a

sawto oth mo de is typical of unstable nite dierence formulas

Although rounding errors can excite an instability more often it is dis

cretization errors that do so and this particular exp eriment is quite typical in

this resp ect Figure would havelooked the same even if the computation

had been carried out in exact arithmetic

This chapter is devoted to understanding instability phenomena in a gen

eral way Let us briey mention how each of the sections to follow relates to

the particular example of Figure

x presents the celebrated Lax Equivalence Theorem a consistent First

nite dierence formula is convergent if and only if it is stable Our example

AN EXAMPLE TREFETHEN 

3

2

1

0 3 5 2

1 0

0 −5

a

3

2

1

0 3 5 2

1 0

0 −5

b

Figure Stable and unstable leap frog approximations to u u

t x

AN EXAMPLE TREFETHEN 

x

i i o x

oo

x

i i

a largest value i sin h b corresp onding amplication factors z

Figure Righthand sides and corresp onding amplication fac

tors z of The circles corresp ond to Figure a and the

crosses to Figure b

is consistent for any but stable only for Thus as Figure suggests

the numerical results would converge to the correct solution as h k in case

a but not in case b

Next x presents the CFL condition a nite dierence formula can be

stable only if its numerical domain of dep endence is at least as large as the

mathematical domain of dep endence In the spacetime grid of Figure b

information travels under the leap frog mo del at sp eed at most which

is less than the propagation sp eed for the PDE itself Thus something had

to go wrong in that computation

Section presents the von Neumann approach to stability Fourier anal

ysis and amplication factors This is the workhorse of stability analysis and

the foundations were given already in xx For our leap frog mo del

inserting the trial solution

i x

n n

j

v z e R

j

leads to the equation

i h i h

z z e e

that is

z z i sin h

This is a quadratic equation in z with two complex ro ots in general As

varies the righthand side ranges over the complex interval i i For

jj this interval is a subset of i i and so the ro ots z lie in symmetric

AN EXAMPLE TREFETHEN 

p ositions on the unit circle jz j Figure For jj on the other hand

some values of lead to righthand side values not contained in i i and

the ro ots z then move o the unit circleone inside and one outside The ro ot

z outside the circle amounts to an amplication factor greater than and

causes instability The largest z o ccurs for sin h which explains why

the instabilityofFigureb had p oints p er wavelength

Finally xx discuss stability analysis via stability regions for prob

lems in the form of the metho d of lines Our leap frog example can be in

terpreted as the midp oint rule in time coupled with the centered dierence

op erator in space Figure a shows the stability region in the aplane

not the k ka plane for the midp oint rule rep eated from Figure it is

the complex op en interval ik ik on the imaginary axis Figure b

i x

j

shows the eigenvalues of its eigenfunctions are the functions v e

j

R with corresp onding eigenvalues

i

i h i h

e e sin h

h h

Thus the eigenvalues cover the complex interval ih ih For absolute sta

bility of the system of ODEs that arise in the metho d of lines these eigenvalues

must lie in the stability region leading once again to the condition h k

that is kh Section shows that this condition relates to true stabilityas

well as to absolute stability

i x

j

do not b elong Weare b eing careless with the term eigenvalue Since the functions v e

j

to they are not true eigenfunctions and the prop er term for the quantities is sp ectral

h

values However this technicality is of little imp ortance for our purp oses and go es awaywhen

one considers problems on a b ounded interval or switches to the norm so we shall ignore it and

h

sp eak of eigenvalues anyway

AN EXAMPLE TREFETHEN 

i

k

i

k

i

h

i

h

a Stability region for midp ointrule b Eigenvalues of

Figure Absolute stability analysis of Example

EXERCISES

Determine the unstable mo de that dominates the behavior of Figure b In

particular what is the factor bywhich the unstable solution is amplied from one step to the next

THE LAX EQUIVALENCE THEOREM TREFETHEN 

The Lax Equivalence Theorem

This section is rather sketchy at the moment omitting some essential p oints of rigor as well

as explanation esp ecially in twoareas the application of the op erator A on a dense subspace rather

than the whole space B and the relationship b etween continuous and discrete norms For a more

precise discussion see Richtmyer and Morton

The essential idea of the Lax Equivalence Theorem is this for consistent linear nite

dierence mo dels stability is a necessary and sucient condition for convergence This is an

analog of the Dahlquist Equivalence Theorem for ordinary dierential equations Theorem

except that the latter is valid for nonlinear problems to o

Aside from the assumption of linearity the formulation of the Lax Equivalence Theo

remisvery general Let B be a a complete normed vector space with norm

denoted by kk In applications of interest here each elementofB will b e a function of one

or more space variables x Let A BB b e a linear op erator on this space Here A will b e

a dierential op erator We are given the initial v alue problem

u tAut t T u u

t 

where A is xed but u may range over all elements of B Actually A only has to be



dened on a dense subset of B This initial value problem is assumed to b e wellp osed

which means that a unique solution ut exists for any initial data u and ut dep ends



continuously up on the initial data

EXAMPLE Supp ose we wish to solve u u for x t with initial

t x

data f x and we wish to lo ok for solutions in the space L In this case each ut in

is a function of x namely ux t Technically we should not use the same symbol u in

b oth places The Banach space B is L A is the rstorder dierentiation op erator and

x

u f



EXAMPLE Supp ose we wish to solve u u for x t with initial

t xx

data f x and b oundary conditions utut Again each ut is a function of

x Now an appropriate Banach space mightbeC the set of continuous functions of



x with value at x together with the supremum norm

Abstractly ut is nothing more than an element in a BanachspaceB and

this leaves ro om for applications to a wide variety of problems in dierential

equations As in the examples ab ove B might be L and A might be or

x

and homogeneous b oundary conditions could be included by restricting

x

B appropriately More generally ut could be an vectorvalued function of

multiple space variables

The next step is to dene a general nite dierence formula In the

setting this is a family of b ounded linear op erators abstract

S BB k

THE LAX EQUIVALENCE THEOREM TREFETHEN 

where the subscript k indicates that the co ecients of the nite dierence

formula dep end on the time step Weadvance from one step to the next bya

single application of S

k

n n n n

v S v hence v S v

k k

n n

abbreviates S Watch out for the usual confusion of notation where S

k k

n n

it is an exp onent For simplicity the n in v is a sup erscript while in S

k

but no more essential reason we are assuming that the problem and

hence S have no explicit dep endence on t But S do es potentially dep end

k k

on k and this is an imp ortant p oint On the other hand it do es not explicitly

dep end on the space step h for we adopt the following rule

h is a xed function hk of k

q

For example we might have h k constant or h k constant

If there are several space dimensions each may have its own function h k

j

More generally what we really need is gridk not hk there is no need at

all for the grid to b e regular in the space dimensions

EXAMPLE Lowerorder terms For the UW mo del of u u the discrete solution

t x

n n n n

op erator is dened by S v v v v and if is held constant as k this

j j j  j

k

formula happ ens to b e indep endentofk The natural extension of UW to u u uon

t x

n n n n n

the other hand is S v v v v kv and here there is an explicit dep endence

j j j  j j

k

on k This kind of k dep endence app ears whenever the op erator A involves derivatives of

dierent orders

Implicit or multistep nite dierence formulas are not excluded by this for

mulation As explained in x an implicit formula may still dene a b ounded

and a multistep formula can op erator S on an appropriate space such as

h k

be reduced to an equivalent onestep formula by the intro duction of a vector

n n ns

w v v

Let us now b e a bit more systematic in summarizing how the setup for the

Lax Equivalence Theorem do es or do es not handle the various complications

that make real problems dier from u u and u u

t x t xx

THE LAX EQUIVALENCE THEOREM TREFETHEN 

Nonlinearity

The restriction here is essential the LaxRichtmyer theory do es not han

dle nonlinear problems However see various more recent pap ers by

SanzSerna and others

Multiple space dimensions

Implicit nite dierence formulas

Both of these are included as part of the standard formulation

Timevarying co ecients

Initialvalue problems with timevarying co ecients are not covered in the

description given here or in Richtmyer and Morton but this restriction

is not essential The theory can be straightforwardly extended to such

problems

Boundary conditions

Spacevarying co ecients

Lowerorder terms

All of these are included as part of the standard formulation and they

have in common the prop erty that they all lead to nitedierence ap

proximations S that dep end on k as illustrated in Example ab ove

k

Systems of equations

Higherorder initialvalue problems

Multistep nite dierence formulas

These are covered by the theory if we make use of the usual device of

reducing a onestep vector nite dierence approximation to a rstorder

initialvalue problem

As in Chapter we begin a statement of the LaxRichtmyer theory by

dening the order of accuracy and consistency of a nite dierence formula

fS g has order of accuracy p if

k

p

kut k S utk O k as k

k

for any t T where ut is any suciently smo oth solution to the initial

value problem It is consistent if it has order of accuracy p

There are dierences between this denition and the denition of order of ac

curacy for linear multistep formulas in x Here the nite dierence formula

is applied not to an arbitrary function u but to a solution of the initial value

THE LAX EQUIVALENCE THEOREM TREFETHEN 

problem In practice however one still calculates order of accuracy by substi

tuting formal Taylor expansions and determining up to what order the terms

cancel Exercise

Another dierence is that in the case of linear multistep formulas for

ordinary dierential equations the order of accuracy was always an integer

and so consistency amounted to p Here nonintegral orders of accuracy

are p ossible although they are uncommon in practice

EXAMPLE Nonintegral orders of accuracy The nite dierence approximation

to u u

t x

n n n n p

S v v v v k constant

k j j j  j

is a contrived example with order of accuracy pifp is any constant in the range A

slightly less contrived example with order of accuracy p is

k

n n p n n

v v with h k S v v

j  j  k j j

h

for any p

As with ordinary dierential equations a nite dierence formula for a

partial dierential equation is dened to be convergent if and only if it con

verges to the correct solution as k for arbitrary initial data

fS g is convergent if

k

n

kS u utk

lim

k

k 

nk t

for any t T where ut is the solution to the initialvalue problem

for any initial data u

Note that there is a big change in this denition from the denition of

convergence for linear multistep formulas in x There a xed formula had

to apply successfully to any dierential equation and initial data Here the

dierential equation is xed and only the initial data vary

The denition of stability is slightly changed from the ordinary dierential

equation case b ecause of the dep endence on k

fS g is stable if for some C

k

n

kS kC

k

for all n and k such that nk T

THE LAX EQUIVALENCE THEOREM TREFETHEN 

n

k is equivalentto This bound on the op erator norms kS

k

n n

kv k kS v kC kv k

k

for all v B and nk T

Here is the Lax Equivalence Theorem compare Theorem

LAX EQUIVALENCE THEOREM

Theorem Let fS g be a consistent approximation to a wellp osed

k

linear initialvalue problem Then fS g is convergent if and only if

k

it is stable

Pro of Not yet written

The following analog to Theorem establishes that stable discrete for

mulas have the exp ected rate of convergence

GLOBAL ACCURACY

Theorem Let a convergent approximation metho d of order of accu

racy p be applied to a wellp osed initialvalue problem with some

additional smo othness assumptions Then the computed solution satis

es

p

kv t utk O k as k

uniformly for all t T

Pro of Not yet written

A number of remarks should be made ab out the developments of this

section

The denitions of convergence and of consistency make reference to the

initialvalue problem but the denition of stabilitydoes not One can

ask whether a nite dierence formula is stable or unstable without having

any knowledge of what partial dierential equation if any it approximates

No assumption has b een made that the initialvalue problem is hyp er

b olic or parab olic so long as it is wellp osed Indeed as far as the theory is

e a dierential op erator or concerned the initialvalue problem maynotinvolv

an x variable at all There are some useful applications of the Lax Equiva

lence Theorem of this kind one of whichinvolves the socalled Trotter pro duct

formula of mathematical physics

THE LAX EQUIVALENCE THEOREM TREFETHEN 

As in x we are dealing with the limit in which t is xed and k

The situation for t with k xed will b e discussed in x

The denition of the nite dierence formula dep ends up on the

mesh function h hk Consequently whether the formula is stable or not

may dep end on hk to o Examples are given in x

As in x it is quite p ossible for an unstable nite dierence formula to

give convergent results for some initial data For example this mighthappen

if the initial data were particularly smo oth But the denition of convergence

requires go o d results for all p ossible initial data

Relatedly the theory assumes exact arithmetic discretization errors are

included but not rounding errors The justication for this omission is that

the same phenomena of stability and instabilitygovern the propagation of b oth

kinds of errors so that in most situations a prediction based on discretization

errors alone will b e realistic On the other hand the fact that rounding errors

o ccur in practice is one motivation for requiring convergence for all initial data

The initial data prescrib ed mathematically for a particular computation might

be smo oth but the rounding errors sup erimp osed on them will not be

Consistency convergence and stability are all dened in terms of a norm

kkand it must be the same norm in each case

It is quite p ossible for a nitedierence mo del to be stable in one norm

and unstable in others see x This may sound like a defect in the theory

but in such cases the instabilityis usually so weak that the behavior is more

or less stable in practice

We close this section by mentioning a general theorem that follows from

the denition of stability

PERTURBATIONS OF A STABLE FAMILY

Theorem Let fS g be a stable family of op erators and let T be a

k k

family of op erators satisfying kT k O k as k Then fS T g is also

k k k

a stable family

Pro of not yet written see Richtmyer Morton x

Theorem has an imp ortant corollary that generalizes Example

LOWER ORDER TERMS

Theorem Let fS g b e a consistent nite dierence approximation to

k

a wellp osed linear initialvalue problem in which A is a dierential

op erator acting on one or more space variables The stability of fS g is

k

determined only by the terms that relate to spatial derivatives

THE LAX EQUIVALENCE THEOREM TREFETHEN 

Pro of sketch If the nite dierence approximation is consistent then

lowerorder terms mo dify the nite dierence formula only by terms of order

O k so by Theorem they do not aect stability

EXAMPLE If fS g is a stable nite dierence approximation of u u on

t x

k

then the approximation remains stable if additional terms are added so that

it b ecomes consistent with u u f x u for any function f x u The same is true for

t x

the equation u u A consistentchange from u u to u u f x u u on the

t xx t xx t xx x

other hand might destroy stability

References

Chapters and of R D Richtmyer and K W Morton Dierence

Metho ds for InitialValue Problems Wiley

P D Lax and R D Richtmyer Survey of the stabilityof linear nite

dierence equations Comm Pure Appl Math

EXERCISES

Order of accuracy of the LaxWendro formula Consider the LaxWendro mo del

of u u with kh constant In analogy to the developments of x insert a formal

t x

power series for ux t to obtain a formula for the leadingorder nonzero term of

j j nn

the discretization error Verify that the order of accuracy is

THE CFL CONDITION TREFETHEN 

The CFL condition

In Richard Courant Kurt Friedrichs and Hans Lewy of the UniversityofGottingen

in Germany published a famous pap er entitled On the partial dierence equations of math

ematical physics This pap er was written long b efore the invention of digital computers

and its purp ose in investigating nite dierence approximations was to apply them to prove

existence of solutions to partial dierential equations But the CFL pap er laid the the

oretical foundations for practical nite dierence computations to o and in particular it

identied a fundamental necessary condition for convergence of anynumerical scheme that

has subsequently come to b e known as the CFL condition

What Courant Friedrichs and Lewy p ointed out was that a great deal can b e learned

by considering the domains of dep endence of a partial dierential equation and of its

discrete approximation As suggested in Figure a consider an initialvalue problem for

a partial dierential equation and let x t b e some p ointwitht Despite the picture

the spatial grid need not b e regular or onedimensional The mathematical domain of

dep endence of ux t denoted by X x t is the set of all p oints in space where the initial

data at t mayhave some eect on the solution ux ty

h

x t x t *o x x x x x x x x x x x x x x x

x x x x x x x x x x x x x x x

l k x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

x x x x x x x x x x x x x x x

X x t X x t

k

a b

Figure Mathematical and numerical domains of dep endence

For example for u u or any other parab olic partial dierential equation in one

t xx

space dimension X x t will be the entire real axis b ecause under a parab olic equation

information travels innitely fast The magnitude of the inuence of faraway data may

decay exp onentially with distance but in the denition of the domain of dep endence it

matters only whether this inuence is zero or nonzero The same conclusion holds for the

Schrodinger equation u iu

t xx

On the other hand for u u u u u u u orany other hyp erb olic partial dif

t x t x tt xx



ferential equation or system of equations including nonlinear equations suchas u u

t x

In German Ub er die partiellen Dierenzengleichungen der mathematischen Physik Math Ann

An English translation app eared much later in IBM Journal

y More precisely for a problem in d space dimensions X x t is the intersection of all closed sets

d d

E R with the prop erty that the data on R nE has no eect on ux t

THE CFL CONDITION TREFETHEN 

X x t is nite for each x and t The reason is that in hyp erb olic problems information

travels at a nite sp eed Figure a suggests a problem of this sort since the domain

of dep endence shown there is nite For the mo del problem u u X x t is the single

t x

point fx tg but the more typical situation for hyp erb olic problems is that the domain of

dep endence covers a b ounded range of values of x In onedimensional problems the curves

that b ound this range as in Figure a are the characteristic curves for the partial dif

ferential equation and these are straight lines in simple examples but usually more general

curves in problems containing variable co ecients or nonlinearity

Anumerical approximation also has a domain of dep endence and this is suggested in

n

Figure b With an implicit nite dierence formula eachvalue v dep ends on all the

j

values at one or more earlier steps and the domain of dep endence is unb ounded On the

n

other hand with an explicit formula v dep ends on only a nite range of values at previous

j

steps For any xed k and h the domain of dep endence will then fan out in a triangle

that go es backwards in time The triangle will be symmetrical for a threep oint formula

like LaxWendro or leap frog symmetrical and twice as wide for a vep ointformula like

fourthorder leap frog asymmetrical for a onesided formula likeupwind and so on

The numerical domain of dep endence for a xed value k denoted by X x t is dened

k



to b e the set of p oints x whose initial data v enter into the computation of v x t For

j j

each time step k this set is discrete but what really matters is the limit k We dene

the limiting numerical domain of dep endence X x t to b e the set of all limit p oints



of the sets X x task This will b e a closed subset of the spatial domaintypically

k

an interval if there is one space variable or a parallelepip ed if there are several

From the p oint of view of domain of dep endence there are three general classes of dis

crete approximations In the case of an implicit nite dierence mo del X x tisunb ounded

k

for each k and therefore provided only that the spatial grid b ecomes ner everywhere as

k X x twillbetheentire spatial domain Sp ectral metho ds are also essentially of



this typ e although the time stepping may b e explicit their stencils cover the entire spatial

domain and so X x tisunb ounded for each k

k

At the other extreme is the case of an explicit nite dierence formula with a spatial

grid that scales in prop ortion to k sothatX x tandX x t are b ounded sets for each x



k

and t In the particular case of a regular grid in space with mesh ratio kh constant

their size is prop ortional to the width of the stencil and inversely prop ortional to The

latter statement should b e obvious from Figure bif is cut in half it will taketwice

as many time steps to get to x t for a xed h and so the width of the triangle will double

Between these two situations lies the case of an explicit nite dierence formula whose

spatial grid is rened more slowly than the time step as k for example a regular

grid with k oh Here X x t is b ounded for each k but X x t is unb ounded This



k

inb etween situation shows that even an explicit formula can haveanunb ounded domain of

dep endence in the limiting sense

The observation made by Courant Friedrichs and Lewy is b eautifully simple anu

merical approximation cannot con verge for arbitrary initial data unless it takes all of the

necessary data into account Taking the data into account means the following

d

More precisely for a problem in d space dimensions X x t is the set of all p oints s R every



op en neighb orho o d of whichcontains a p ointofX x t for all suciently small k k

THE CFL CONDITION TREFETHEN 

THE CFL CONDITION For each x t the mathematical domain of dep endence is

contained in the numerical domain of dep endence

X x t X x t



Here is their conclusion

CFL THEOREM

Theorem The CFL condition is a necessary condition for the convergence of a

numerical approximation of a partial dierential equation linear or nonlinear

The justication of Theorem is so obvious that weshallnot attempt to state or

prove it more formally But a few words are in order to clarify what the terms mean First

the theorem is certainly valid for the particular case of the linear initialvalue problem

with the denition of convergence provided in the last section In particular unlike

the von Neumann condition of the next section it holds for anynormkk and any partial

dierential equation including problems with b oundary conditions variable co ecients or

nonlinearity But one thing that cannot b e changed is that Theorem must always be

interpreted in terms of a denition of convergence that involves arbitrary initial data The

reason is that for sp ecial initial data a numerical metho d mightconverge even though it had

a seemingly inadequate domain of dep endence The classic example of this is the situation in

which the initial data are so smo oth that they form an analytic function Since an analytic

function is determined globally by its behavior near any point a nite dierence mo del

might sample to o little initial data in an analytic case and still converge to the correct

solutionat least in the absence of rounding errors

A priori the CFL condition is a necessary condition for convergence But for linear

problems the Lax Equivalence Theorem asserts that convergence is equivalent to stability

From Theorems and we therefore obtain

CFL CONDITION AND STABILITY

Theorem Let fS g be a consistent approximation to awellp osed linear initial

k

value problem Then the CFL condition is a necessary condition for the stability

of fS g

k

Unlike Theorem Theorem is valid only if its meaning is restricted to the linear

formulations of the last section The problem of stability of nite dierence mo dels had

not yet b een identied when the CFL pap er app eared in but Theorem is the form

in whichthe CFL result is now generally rememb ered The reason is that the connection

between convergence and stabilityissouniversally recognized now that one habitually thinks

of stability as the essential matter to b e worried ab out

The reader may have noticed a rather strange feature of Theorem In the last

section it was emphasized that the stability of fS g has nothing to do with what initial

k

value problem if any it approximates Yet Theorem states a stability criterion based on

THE CFL CONDITION TREFETHEN 

an initialvalue problem This is not a logical inconsistency for nowhere has it b een claimed

that the theorem is applicable to every nite dierence mo del fS gan initialv alue problem

k

is broughtinto the act only when a consistency condition happ ens to b e satised For more

on this p oint see Exercise

Before giving examples we shall state a fundamental consequence of Theorem

EXPLICIT MODELS OF PARABOLIC PROBLEMS

Theorem If an explicit nite dierence approximation of a parab olic initialvalue

problem is convergent then the time and space steps must satisfy k oh as k

Pro of This assertion follows from Theorem and the fact that an explicit nite

dierence mo del with k oh has a b ounded numerical domain of dep endence X x t for



eachx t whereas the mathematical domain of dep endence X x tisunb ounded

The impact of Theorem is farreaching parab olic problems must always b e solved

by implicit formulas or by explicit formulas with small step sizes This would makethem

generally more dicult to treat than hyp erb olic problems were it not that hyp erb olic prob

lems tend to feature sho ckwaves and other strongly nonlinear phenomenaa dierent source

of diculty that evens the score somewhat

In computations involving more complicated equations with b oth convective and dif

fusive terms such as the NavierStokes equations of uid dynamics the considerations of

Theorem often lead to numerical metho ds in which the time iteration is based on a

splitting into an explicit substep for the convective terms and an implicit substep for the

diusive terms See any b o ok on computational uid dynamics

EXAMPLE Approximations of u u For the equation u u all information

t x t x

propagates leftward at sp eed exactly and so the mathematical domain of dep endence for

eachx t is the single p oint X x tfx tg Figure suggests how this relates to the

domain of dep endence for various nite dierence formulas As always the CFL condition is

necessary but not sucient for stability it can prove a metho d unstable but not stable For

the nite dierence formulas of Table with kh constant we reach the following

conclusions

LF unstable for

UW LF LW EU LXF unstable for

x

Downwind formula unstable for all

BE CN BOX no restriction on

x x x

We shall see in the next section that these conclusions are sharp except in the cases of LF

and EU and LF marginally whose precise condition for instabilityis rather than

x

EXAMPLE Approximations of u u Since u u is parab olic Theorem

t xx t xx

asserts that no consistent explicit nite dierence approximation can be stable unless

k ohask Thus for the nite dierence formulas of Table it implies

EU LF unstable unless k oh

xx xx

BE CN BOX CN no restriction on hk

xx xx

THE CFL CONDITION TREFETHEN 

a LWorEU b LW or EU c UW

Figure The CFL condition and u u The dashed line represents the

t x

characteristic of the PDE and the solid line represents the stencil of the nite

dierence formula

The next section will show that these conclusions for LF and EU are not sharp In fact

xx xx



h for example EU is stable only if k

xx

The virtue of the CFL condition is that it is extremely easy to apply

Its weakness is that it is necessary but not sucient for convergence As

a practical matter the CFL condition often suggests the correct limits on

stability but not always and therefore it must be supplemented by more

careful analysis

EXERCISES

Multidimensional wave equation Consider the secondorder wave equation in d space

dimensions

u u u

tt x x x x

d d

the ddimensional analog of the simple secondorder nite dierence ap a Write down

proximation

n n

n n n n

v v v v v v

j  j j  j

j j

for a regular grid with space step h in all directions and kh constant

b What do es the CFL condition tell you ab out values of for which this mo del must b e

unstable Hint if you are in doubt ab out the sp eed of propagation of energy under

the multidimensional wave equation consider the fact that the equation is isotropic

ie energy propagates in the same manner regardless of direction Another hint be careful

THE VON NEUMANN CONDITION TREFETHEN 

The von Neumann condition

for scalar onestep formulas

Von Neumann analysis is the analysis of stability by Fourier metho ds In principle

this restricts its applicabilitytoanarrow range of linear constantco ecient nite dier

ence formulas on regular grids with errors measured in the norm In practice the von

h

Neumann approach has something to say ab out almost every nite dierence mo del It is

the mainstay of practical stability analysis

The essential idea is to combine the ideas of x sp ecialized to the norm with those

h

of xx For onestep scalar mo dels the ensuing results give a complete characterization

of stability in terms of the amplication factors intro duction in x which for stability

must lie within a distance O k of the unit disk as k For multistep or vector problems

one works with the asso ciated amplication matrices intro duced in x and a complete

analysis of stability requires lo oking not only at the sp ectra of these matrices which amount

to amplication factors but also at their resolvents or pseudosp ectra We shall treat the

onestep scalar case in this section and the general case in the next two sections

We begin with two easy lemmas The rst is a generalization of the wellknown in



equality e

Lemma For anyrealnumbers a and b

b ab

a e

Pro of Both sides are nonnegative so taking the b th ro ot shows that it is enough to

a

provea e This inequality is trivial just draw a graph

The second lemma deals with arbitrary sets of numbers s which in our applica

k

tions will b e norms suchaskS k

k

Lemma Let fs g b e a set of nonnegativenumb ers indexed by k and let

k

n

T b e xed Then s C for some C uniformly for all k and n with kn T if

 

k

and only if s O k as k

k

n n C kn



Pro of If s C k for each k thenby Lemma s C k e

k k

C T n n



e Converselyifs C for all kn T then in particular for each k s C for

 

k k

T T n

k Tn

O k C C some value of n with nk T This implies s C

  

k

Now we are ready for von Neumann analysis Supp ose that is a wellp osed

linear initialvalue problem in which ut is a scalar function of x and B is the space

h

with kk denoting the norm Supp ose also that for each k the op erator S of

h k

denotes an explicit or implicit onestep nite dierence formula with co ecients

f g and f g that are constant except for the p ossible dep endence on k If S is implicit

k

it is assumed to satisfy the solvability condition which ensures that it has a b ounded

b amplication factor function g a

k k k

THE VON NEUMANN CONDITION TREFETHEN 

By the formula is stable if and only if

n

kS kC for nk T

k

for some constant C By Theorem this is equivalent to the condition

n

kg k C for nk T

k

or equivalently

n

jg j C for nk T

k

where C is a constant indep endentof By Lemma this is equivalentto

jg jO k

k

k uniformly in What this means is that there exists a constant C such that for as

all n and k with nk T andall h h

jg jC k

k

We have obtained the following complete characterization of stable nite dierence

formulas in our scalar problem

h

VON NEUMANN CONDITION

FOR SCALAR ONESTEP FINITE DIFFERENCE FORMULAS

Theorem A linear scalar constantco ecient onestep nite dierence formula as

describ ed ab ove is stable in if and only if the amplication factors g satisfy

h k

jg jO k

k

as k uniformly for all h h

With Theorem in hand we are equipp ed to analyze the stability of manyof the

nite dierence formulas of x

EXAMPLE Upwind formula for u u In we computed the amplication

t x

factor for the upwind formula as

i h

g e

assuming that kh is a constant For any this formula describ es a circle in the

complex plane shown in Figure as ranges over h h The circle will lie in

the closed unit disk as required by if and only if which is accordingly the

stability condition for the upwind formula This matches the restriction suggested bythe CFL condition Example

THE VON NEUMANN CONDITION TREFETHEN 

a stable b unstable

Figure Amplication factors for the upwind mo del of u u solid

t x

curve The shaded region is the unit disk

EXAMPLE CrankNicolson formulas for u u u u u iu In

t x t xx t xx

we found the amplication factor for the CrankNicolson formula to b e

h

k

sin



h

g

h

k

sin



h

Here jg j for all regardless of k and h Therefore the CrankNicolson formula is

stable as k no matter how k and h are related It will b e consistent hence convergent

so long as hk o as k

For the CrankNicolson mo del of u u the corresp onding formula is

t x

ik

sin h

h

g

ik

sin h

h

Now jg j for all so the formula is again unconditionally stable The same is true of

the CrankNicolson mo del of u iu whose amplication factor function is

t xx

h

k

i sin



h

g

h

k

i sin



h

EXAMPLE Euler formulas for u u and u u The amplication factor for

t x t xx

the Euler mo del of u u was given in as

t xx

h k

sin g

h

As illustrated in Figure a this expression describ es the interval kh in the

complex plane as ranges over h h If kh is held constant as k we



This is a tighter restriction conclude that the Euler formula is stable if and only if

than the one provided by the CFL condition which requires only k oh

THE VON NEUMANN CONDITION TREFETHEN 

ik h

a u u b u u

t xx t x

Figure Amplication factors for the Euler mo dels of u u and u u

t xx t x

The amplication factor for the Euler mo del of u u is

t x

ik

sin h g

h

which describ es a line segment tangent to the unit circle in the complex plane Figure

b Therefore the largest amplication factor is

r

k

h

If kh is held xed as k then this nite dierence formula is therefore unstable

regardless of On the other hand the square ro ot will have the desired magnitude O k

if k h O k ie k O h and this is accordingly the stability condition for arbitrary

mesh relationships h hk Thus in principle the Euler formula is usable for hyp erb olic

equations but it is not used in practice since there are alternatives that p ermit larger time

steps k O h as well as having higher accuracy

EXAMPLE LaxWendro formula for u u The amplication factor for the

t x

endro formula was given in as LaxW

h

g i sin h sin

if kh is a constant Therefore jg j is

h h





jg j sin sin sin h



cos sin sin to the last term converts this Applying the identity sin sin

expression to

h h h h

  

jg j sin sin sin sin

h

 

sin





If is xed it follows that the LaxWendro formula is stable provided that

This is true if and only if which is accordingly the stability condition

Tables and summarize the orders of accuracy and the stability

limits for the nite dierence formulas of Tables and The results

listed for multistep formulas will b e justied in x

THE VON NEUMANN CONDITION TREFETHEN 

order of CFL stability Exact stability

Formula accuracy restriction restriction

EU Euler unstable

x

BE Backward Euler none none

x

CN CrankNicolson none none

x

LF Leap frog

BOX Box none none

x

LFFourthorder Leap frog

LXF LaxFriedrichs

UW Upwind

LW LaxWendro

Table Orders of accuracy and stability limits for various

nite dierence approximations to the wave equation u u with

t x

kh constant see Table

order of CFL stability Exact stability

Formula accuracy restriction restriction

EU Euler y none

xx

BE Backward Euler none none

xx

CN CrankNicolson none none

LF Leap frog none unstable

xx

BOX Box none none

xx

CN Fourthorder CN none none

DF DuFortFrankel none none

Table Orders of accuracy and stability limits for various

nite dierence approximations to the heat equation u u with

t xx

kh constant see Table The orders of accuracy are

with resp ect to h not k as in

See Exercise

y See Exercise See Exercise

THE VON NEUMANN CONDITION TREFETHEN 

EXERCISES

Generalized CrankNicolson or theta metho d Let the heat equation u u be

t xx

mo deled by the formula

k k

n n n n

n n n n

v v v v v v v v

j  j j  j

j  j j  j

h h



with For this is Euler CrankNicolson or backward Euler formula

resp ectively

a Determine the amplication factor function g

b Supp ose kh is held constantask For which and is stable

c Supp ose kh is held constantas k For which and is stable

d Your b oss asks you to solve a heat conduction problem involving a space interval of

length and a time interval of length She wants an answer with errors on the

Roughly sp eaking order of magnitude in howmany order of some number



oatingp oint op erations will you have to p erform if you use and

The downwind formula The downwind approximation to u u is

t x

k

n

n n n n

v v v S v v

j j  k j j

j

h

In this problem do not assume that kh is held constantask let hk b e a completely

arbitrary function of k

a For what functions hk if any is this formula stable Use Theorem and be

careful

b For what functions hk if any is it consistent

c For what functions hk if any is it convergent Use the Lax Equivalence Theorem

d How do es the result of c match the prediction by the CFL condition

The CFLstability link This problem was originally worked out by G Strang in

the s Bernsteins inequality asserts that if

m

X

i

g e

m

for some constants f g then

kg k mkg k

where kk denotes the maximum over and g is the derivative dg d

Derive from this the CFL condition if an explicit nite dierence formula

m

X

n

n

v v

j 

j

m

is stable as h with kh constant and consistent with u u then m Hint

t x

what do es consistency imply ab out the b ehavior of g near

THE VON NEUMANN CONDITION TREFETHEN 

A thought exp eriment Supp ose the LaxWendro mo del of u u is applied on an

t x

innite grid with h to compute an approximate solution at t The initial data are

u x cos x or u xmaxf jxjg



and the time step is k h with

or

a b

Thus wehave four problems a b a b Let E E E E denote the corresp onding

a a

b b

maximum errors over the grid at t

h

One of the four numb ers E E E E dep ends signicantly on the machine precision

a a

b b

and thus deserves a star in front of it see x the other three do not Which one

Explain carefully whyeachofthemdoesordoesnotdependon In the pro cess give order

 

ofmagnitude predictions for the four numb ers such as E E E

Explain your reasoning

If you wish you are welcome to turn this thought exp erimentinto a computer exp eriment

The Euler formula for the heat equation Showthatif the order of accuracy

of the Euler formula for u u increases from to Note Such bits of go o d fortune

t xx

are not always easy to take advantage of in practice since co ecients and hence eective

mesh ratios mayvary from p ointtopoint

Weak inequalities In Tables and the stability restrictions for onestep

formulas all involve weak rather than strong inequalities Prove that this

is a general phenomenon the stabilit y condition for a formula of the typ e considered in

Theorem may b e of the form k f h but never kfh

PSEUDOSPECTRA AND THE KREISS MATRIX THEOREM TREFETHEN 

Resolvents pseudosp ectra and the

Kreiss matrix theorem

Powers of matrices app ear throughout numerical analysis In this book they arise

mainly from equation which expresses the computed solution at step n for a numer

ical mo del of a linear problem in terms of the solution at step

n n 

v S v

k

In this formula S represents a b ounded op erator on a BanachspaceB p ossibly dierent

k

for each time step k Thus we are dealing with a family of op erators fS g indexed by

k

k According to the theory presented in xakey question to ask ab out such a family of

op erators is whether it is stable ie whether a b ound

n

kS kC

k

holds for all n and k with nk T

This question ab out op erators may be reduced to a question ab out matrices in two

distinct ways The rst is byFourier or von Neumann analysis applicable in cases where

one is dealing with a regular grid constant co ecients and the norm For onestep scalar

formulas Fourier analysis reduces S to a scalar the amplication factor g treated in the

k

last section For vector formulas the sub ject of this and the next section we get the more

interesting Fourier analogue

n

b

n 

c

v G v

k

n

Here G is the nth p ower of the amplication matrix dened in x and the equivalence

k

n

of and implies that the norm of G determines the norm of S

k k

n n

kS k sup kG k

k k

hh

n

The question of stability b ecomes are the norms kG k bounded by a constant for all

k

nk T uniformly with resp ect to Note that here the dimension N of G is xed

k

indep endentofk and h

For problems involving variable co ecients variable grids or translationdep endent

discretizations such as Chebyshev sp ectral metho ds Chapter Fourier analysis cannot b e

tforward reduction to matrices b ecomes relevant applied Here a second and more straigh

One can simply work with S itself as a matrixthat is work in space itself not Fourier

k

space If the grid is unb ounded then the dimension of S is innite but since the grids one

k

computes with are usually b ounded S is usually nite in practice The dimension increases

k

to however as k

In summary the stability analysis of numerical metho ds for partial dierential equa

tions leads naturally to questions of whether the p owers of a family of matrices have uni

formly b ounded norms or as the problem is often put whether a family of matrices is

powerb ounded Dep ending on the circumstances the matrices in the family maybeof

xed dimension or varying dimensions

PSEUDOSPECTRA AND THE KREISS MATRIX THEOREM TREFETHEN 

10

k

k

n

kA k

k

k

k

k

0 n

0 10 20 30

Figure Norms of p owers of the matrix A of for various k Each A

k k

for k is individually p owerb ounded but the family fA g is not p ower

k

k

b ounded since the p owers come arbitrarily close to those of the limiting defective

matrix A dashed



In x we have already presented an algebraic criterion for powerb oundedness of

n

a matrix A According to the alternative pro of of Theorem the powers kA k are

b ounded if and only if the eigenvalues of A lie in the closed unit disk and anyeigenvalues on

the unit circle are nondefective For families of matrices however matters are not so simple

The eigenvalue condition is still necessary for powerb oundedness but it is not sucient

To illustrate this consider the two families of matrices



k

A B

k k

k

for k For each k A and B are individually powerb ounded but neither fA g

k k k

For fB g the explanation is quite obvious the nor fB g is powerb ounded as a family

k k

upp errightentry diverges to as k and thus even the rst p ower kB k is unb ounded

k

as k For fA g the explanation is more interesting As k these matrices come closer

k

and closer to a defective matrix whose p owers increase in norm without b ound Figure

illustrates this eect

In the next section we shall see that an unstable family muchlike fA g arises in the

k

von Neumann analysis of the leap frog discretization of u u with mesh ratio In

t x

most applications however the structure of the matrices that arise is far from obvious by

insp ection and one needs a general to ol for determining p owerb oundedness

What is needed is to consider not just the sp ectra of the matrices but also their

pseudosp ectra The idea of pseudosp ectra is as follows An eigenvalue of a matrix A is a



number z C with the prop erty that the resolvent matrix zI A is undened By



convention we may write kzI A k in this case A pseudoeigenvalue of A is a

PSEUDOSPECTRA AND THE KREISS MATRIX THEOREM TREFETHEN 



number z where kzI A k while not necessarily innite is very large Here is the

denition

Given thenumber C is an pseudoeigenvalue of A if either of the following

equivalent conditions is satised

 

i kI A k

N N

ii is an eigenvalue of A E for some E C with kE k

The pseudosp ectrum of A denoted by A is the set of all of its pseudo

eigenvalues

Condition i expresses the idea of an pseudoeigenvalue just describ ed Condition ii is

a mathematical equivalent with quite a dierent avor though an pseudoeigenvalue of

A need not b e near to any exact eigenvalue of A it is an exact eigenvalue of some nearby

matrix The equivalence of conditions i and ii is easy to prove Exercise

If a matrix is normal which means that its eigenvectors are orthogonal then the



norm of the resolventisjust kzI A k dist z A where dist z A denotes

the distance between the point z and the set A Exercise Therefore for each

A is equal to the union of the closed balls of radius ab out the eigenvalues of

Aby condition ii the eigenvalues of A are insensitive to p erturbations The interesting

cases arise when A is nonnormal where the pseudosp ectra maybemuch larger and the

eigenvalues may b e highly sensitive to p erturbations Figure illustrates this comparison

rather mildly by comparing the pseudosp ectra of the two matrices

A A

motivated by for These two matrices both have the

sp ectrum f g but the pseudosp ectra of A are considerably larger than those of A

Roughly sp eaking matrices with larger pseudosp ectra tend to have larger p owers even

if the eigenvalues are the same Figure illustrates this phenomenon for the case of the

two matrices A and A of Asymptotically as n the powers decrease in b oth

n

cases at the rate determined by the sp ectral radius AA The transient

n

behavior is noticeably dierent however with the curve for kA k showing a hump by a

factor of ab out centered around n b efore the eventual decay

Factors of are of little consequence for applications but then A is not a very highly

nonnormal matrix In more extreme examples the hump in a gure like Figure may

b e arbitrarily high Tocontrol it wemust have a b ound on the pseudosp ectra and this is

the sub ject of the Kreiss matrix theorem rst proved in

More precisely a matrix is normal if there exists a complete set of orthogonal eigenvectors

PSEUDOSPECTRA AND THE KREISS MATRIX THEOREM TREFETHEN 

1 1

0 *o *o 0 *o *o

-1 -1

0 1 2 0 1 2

a A normal b A nonnormal

Figure Boundaries of pseudosp ectra of the matrices A and A of

for The dashed curveis the righthalf of the unit circle

whose lo cation is imp ortant for the Kreiss matrix theorem The solid dots are

the eigenvalues

3

n

kA k

2

1

n

kA k

0 n

0 10 20 30

n n

Figure kA k and kA k for the matrices A and A of

PSEUDOSPECTRA AND THE KREISS MATRIX THEOREM TREFETHEN 

KREISS MATRIX THEOREM

Theorem Let A b e a matrix of dimension N If

n

kA k C n

for some constant C then

j j C A

or equivalently

C



kzI A k z C jz j

jz j

Conversely imply

n

kA k eC minfN n g n

Theorem is stated for an individual matrix but since the b ounds asserted are

explicit and quantitative the same result applies to families of matrices A satisfying a

n

uniform b ound kA kC provided the dimensions N are all the same If the dimensions N

vary unb oundedly then in only the factor n remains meaningful The inequality

is sometimes called the Kreiss condition and the constant C there is the Kreiss

constant

Pro of The equivalence of and follows trivially from denition i of

A The pro of that implies is also straightforward if one considers the

power series expansion

 

zI A z I z A z A

Thus the real substance of the Kreiss Matrix theorem lies in the assertion that implies

This is really two assertions one involving a factor N the other involving a factor

n

n

The starting p oint for b oth pro ofs is to write the matrix A as the Cauchyintegral

Z

n n 

A z zI A dz

i

where is any curve in the complex plane that encloses the eigenvalues of A Toprovethat

n n

kA k satises the b ound it is enough to showthatjv A uj satises the same b ound

for anyvectors u and v with kuk kv k Thus let u and v b e arbitrary N vectors of this

kind Then implies

Z

n n

v A u z q z dz

i



where q z v zI A u It can be shown that q z is a rational function of order N

ie a quotientoftwo p olynomials of degrees at most N andby it satises

C

jq z j

jz j

PSEUDOSPECTRA AND THE KREISS MATRIX THEOREM TREFETHEN 



Letustake to b e the circle fz C jz j n g which certainly encloses the

eigenvalues of A if holds On this circle implies jq z jC n Therefore

yields the b ound

Z

n n

jv A uj jz j C n jdz j

 n 

n C n n

 n

n C n eCn

In the last of these inequalities wehave used Lemma This proves the part of

involving the factor n

The other part of involving the factor N is more subtle Integration by parts

of gives

Z

n n

z q z dz v A u

in

n

Using the same contour of integration as b efore and again the estimate jz je we

obtain

Z

e

n

jv A uj jq z jjdz j

n

The integral in this formula can b e interpreted as the arc length of the image of the circle

under the rational function q z Now according to a result known as Spijkers Lemma

if q z is a rational function of order N the arc length of the image of a circle under q z

can b e at most N times the maximum mo dulus that q z attains on that circle whichin

this case is at most C n Thus weget

e

n

jv A uj N C n eCN

n

This completes the pro of of the Kreiss matrix theorem

We close this section with a gure to further illustrate the idea of pseudosp ectra

Consider the matrix of the form

B C

B C

B C

A

B C

B C

A

Spijkers Lemma was conjectured in byLeVeque and Trefethen BIT and proved up to

a factor of The sharp result was proved bySpijker in BIT Shortly thereafter it was

pointed out byEWegert that the heart of the pro of is a simple estimate in integral geometry that

generalizes the famous Buon needle problem of See Wegert and Trefethen From the

Buon needle problem to the Kreiss matrix theorem Amer Math Monthly pp

PSEUDOSPECTRA AND THE KREISS MATRIX THEOREM TREFETHEN 

whose only eigenvalue is The upp er plot of Figure depicts the b oundaries of

  

the pseudosp ectra of this matrix for Even for the

pseudosp ectrum of this matrix is evidently quite a large set covering a heartshap ed region

of the complex plane that extends far from the sp ectrum fg Thus by condition ii of

the denition of the pseudosp ectrum it is evident that the eigenvalues of this matrix are

exceedingly sensitive to p erturbations The lower plot of Figure illustrates this fact

more directly It shows a sup erp osition of the eigenvalues of matrices A E where

each E is a random matrix of norm kE k The elements of each E are taken as

indep endent normally distributed complex numb ers of mean and then the whole matrix is

scaled to achieve this norm Thus there are dots in Figure b whichby denition

must lie within in secondlargest of the curves in Figure a

For further illustrations of matrices with interesting pseudosp ectra see L N Trefethen

Pseudosp ectra of matrices in D F Griths and G A Watson eds Numerical Analysis

Longman pp

EXERCISES

Equivalent denitions of the pseudosp ectrum

a Prove that conditions i and ii on p are equivalent

b Prove that another equivalent condition is

n

iii u C kuk suchthat kA uk

Suchavector u is called an pseudoeigenvector of A

c Prove that if kk kk then a further equivalent condition is

iv I A

N

where I A denotes the smallest singular value of I A

N



Provethatif A is a normal matrix then kzI A k dist z A for all z C

Hint if A is normal then it can b e unitarily diagonalized

a Making use of Figure a a ruler and the Kreiss matrix theorem derivelower and

n

upp er b ounds as sharp as you can manage for the quantitysup kA k for the matrix

n

A of

b Find the actual actual numb er with Matlab How do es it compare with your b ounds

VECTOR AND MULTISTEP FORMULAS TREFETHEN 

2

1

0

-1

-2

-2 -1 0 1 2

 

a Boundaries of A for

2

...... 1 ...... 0 ...... -1 ......

-2

-2 -1 0 1 2

b Eigenvalues of randomly p erturb ed matrices A E kE k

Figure Pseudosp ectra of the matrix A of

VECTOR AND MULTISTEP FORMULAS TREFETHEN 

The von Neumann condition

for vector or multistep formulas

Just as x followed x Fourier analysis applies to vector or multistep nite dierence

formulas as well as to the scalar onestep case The determination of stability b ecomes more

complicated however b ecause one must estimate norms of p owers of matrices

Consider a linear constantco ecient nitedierence formula on a regular grid where

the dep endentvariable is an N vector By intro ducing new variables as necessarywemay

n n

assume that the formula is written in onestep form v S v It may be explicit or

k

implicit provided that in the implicit case the solvability condition is satised

If kk is the norm then the condition for stability is equivalent to the condition

n

kG k C

k

for all h h and n k with nk T Here G denotes the amplication

k

matrix an N N function of as describ ed in x Stabilityisthus a question of p ower

b oundedness of a family of N N matrices a family indexed bythetwo parameters and

k This is just the question that was addressed in the last section

n

The simplest estimates of the p owers kG k are based on the norm kG k or the

k k

sp ectral radius G that is the largest of the mo duli of the eigenvalues of G These

k k

two quantities provide a lower and an upp er b ound on according to the easily proved

inequalities

n n n

G kG kkG k

k k k

Combining and yields

VON NEUMANN CONDITION

FOR VECTOR FINITE DIFFERENCE FORMULAS

Theorem Let fS g be a linear constantco ecient nite dierence formula as

k

describ ed ab ove Then

a G O k is necessary for stability and

k

b kG kO k is sucient for stability

k

Both a and b are assumed to hold as k uniformly for all h h

Condition a is called the von Neumann condition For the record let us give it a

formal statement

VON NEUMANN CONDITION The sp ectral radius of the amplication matrix

satises

G O k

k

as k uniformly for all h h

VECTOR AND MULTISTEP FORMULAS TREFETHEN 

In summaryforvector or multistep problems the von Neumann condition is a state

ment ab out eigenvalues of amplication matrices and it is necessary but not sucient for

stability

Obviously there is a gap between conditions and To eliminate this

gap one can apply the Kreiss matrix theorem For any matrix A and constant let

us dene the pseudosp ectral radius A of A to b e the largest of the mo duli of its

pseudoeigenvalues that is

A sup j j

 A

From condition ii of the denition of A it is easily seen that an equivalent denition

is

A sup A E

kE k

Theorem can b e restated in terms of the pseudosp ectral radius as follows a matrix

A is p owerb ounded if and only if

A O

as For the purp ose of determining stabilitywe need to mo dify this condition slightly

in recognition of the fact that only p owers n with nk T are of interest Here is the result

STABILITY VIA THE KREISS MATRIX THEOREM

Theorem A linear constantco ecient nite dierence formula fS g is stable in

k

the norm if and only if

G O O k

k

as and k

The O symb ols in this theorem are understo o d to apply uniformly with resp ect to

h h

Pro of A more explicit expression of is

j jC C k



for all in the pseudosp ectrum G all all k and all h h

k

Equivalently

C



kzI G k

jz j C k

for all jz j C k From here its easyto b e completed later

Theorem is a p owerful result giving a necessary and sucient condition for sta

bility of a wide variety of problems but it has two limitations The rst is that determining

resolvent norms and pseudosp ectra is not always an easy matter and that is whyitisconve

nienttohave Theorem available as well The second is that not all numerical metho ds

satisfy the rather narrow requirements of constant co ecients and regular unb ounded grids

that makeFourier analysis applicable This diculty will b e addressed in the next section

VECTOR AND MULTISTEP FORMULAS TREFETHEN 

2 2

1 1 *o **oo

0 0

-1 -1

-2 -2

-2 -1 0 1 2 -2 -1 0 1 2

a b

Figure Boundaries of pseudosp ectra of the leap frog amplication ma

trix G with For there is a defective

eigenvalue at z i the condition G O fails and the formula is

unstable

EXAMPLE For the leap frog mo del of u u the amplication matrix

t t

i sin h

G

was derived in Example For this family of matrices is powerb ounded but

for the matrix G is defective and not powerb ounded The pseudosp ectra for

G for and are illustrated and Figure See Exercise

In practice how do p eople test for stability of nite dierence metho ds

Usually by computing eigenvalues and checking the von Neumann condition

If it is satised the metho d is often stable but sometimes it is not It is

probably accurate to say that when instability o ccurs the problem is more

often in the b oundary conditions or nonlinearities than in the gap between

Theorems and asso ciated with nonnormality of the amplication

matrices In Chapter we shall discuss additional tests that can be used to

check for instabilityintro duced by b oundary conditions

VECTOR AND MULTISTEP FORMULAS TREFETHEN 

EXERCISES

Multidimensional wave equation Consider again the secondorder wave equation in

d space dimensions

u u u

tt x x x x

d d

and the nite dierence approximation discussed in Exercise Use ddimensional

Fourier analysis to determine the stability b ound on You do not have to use matrices

and do it rigorously whichwould involve an amplication matrix with a defective eigenvalue

under certain conditionsjust plug in the appropriate F ourier mo de solution Ansatz and

compute amplication factors You need not worry ab out keeping track of strong vs weak

inequalities Is it the same as the result of Exercise

u with kh Stability of leap frog Consider the leap frog mo del of u

x t

constant Example and Figure

a Compute the eigenvalues and sp ectral radius of G and verify that the von Neumann

k

condition do es not reveal leap frog to b e unstable

b Compute the norm kG kandverify that the condition whichwould guar

k

antee that leap frog is stable do es not hold

c In fact leap frog is stable for Prove this byany means you wish but be sure

that you haveshown b oundedness of the p owers G uniformly in not just for each

k

individually One way to carry out the pro of is to argue rst that for each xed

n

kG kM for all n for an appropriate function M and then argue that M

must be b ounded as a function of Another metho d is to compute the eigenvalue

decomp osition of G

The fourthorder leap frog formula In Table the stability restriction for the

fourthorder leap frog formula is listed as What is this number

The DuFortFrankel formula The DuFortFrankel mo del of u u listedinTable

t xx

on p has some remarkable prop erties

a Derive an amplication matrix for this formula This was done already in Exercise

b Show that it is unconditionally stable in

h

c What do es the result of b together with the theorems of this chapter imply ab out

ortFrankel formula with u u Be precise and state the consistency of the DuF

t xx

exactly what theorems you have app ealed to

d By manipulating Taylor series derivethe precise consistency restriction involving k

and h for the DuFortFrankel formula Is it is same as the result of c

STABILITY OF THE METHOD OF LINES TREFETHEN 

Stability of the metho d of lines

This section is not yet written Most of its results can b e found in S C Reddy and L N

Trefethen Stability of the metho d of lines Numerische Mathematik

The Kreiss matrix theorem Theorem asserts that a family of matrices fA g is

powerb ounded if and only if its pseudosp ectra A lie within a distance O of the



unit disk or equivalently if and only if its resolventnorms kzI A k increase at most

inverselinearly as z approaches the unit disk D from the outside If the matrices all havea

xed dimension N then this statementisvalid exactly as it stands and if the dimensions

N are variable one loses a factor minfN n g thatisO N orO n

The Kreiss matrix theorem has many consequences for stabilityofnumerical metho ds

for timedep endent PDEs To apply it to this purp ose we rst make a small mo dication

so that we can treat stability on a nite interval T as in x rather than the innite

interval All that is involved is to replace O by O O k as in Theorem

It turns out that there are four particularly imp ortant consequences of the Kreiss

matrix theorem which can be arranged in a twobytwo table according to the following

two binary choices Theorem was one of these four consequences

First one can work either with the op erators fS g as matrices or with the amplication

k

matrices fG g The latter choice is only p ossible when Fourier analysis is applicable ie

k

under the usual restrictions of constant co ecients regular grids no b oundaries etc It

has the great advantage that the dimensions of the matrices involved are xed so there is

no factor O norO N toworry ab out and indeed G is often indep endentofk When

k

Fourier analysis is inapplicable however one always has the option of working directly with

the matrices fS g themselvesin space space instead of Fourier space This is customary

k

for example in the stability analysis of sp ectral metho ds on b ounded domains

Thus our rst pair of theorems are as follows

STABILITY

Theorem again A linear constantco ecient nite dierence formula fS g is

k

stableinthenorm if and only if the pseudoeigenvalues G of the ampli

k

cation matrices satisfy

dist DO O k

STABILITY IN FOURIER SPACE

Theorem A linear nite dierence formula fS g is stable up to a factor minfN n

k

g if and only if the pseudoeigenvalues S satisfy

k

dist DO O k

STABILITY OF THE METHOD OF LINES TREFETHEN 

In these theorems the order symbols O O k should b e understo o d to apply uniformly

for all k and where appropriate as k and

As a practical matter the factor minfN n g is usually not imp ortant b ecause most

often the instabilities that cause trouble are exp onential One derivative of smo othness

in the initial and forcing data for a timedep endent PDE is generally enough to ensure that

such a factor will not prevent convergence as the mesh is rened In a later draft of the

b o ok this p oint will b e emphasized in Chapter

The other pair of theorems comes when we deal with the metho d of lines discussed

previously in x Following the standard formulation of the LaxRichtmyer stabilitytheory

in Chapter supp ose wearegiven an autonomous linear partial dierential equation

u Lu

t

where ut is a function of one or more space variables on a b ounded or unb ounded domain

and L is a dierential op erator indep endentoft For each suciently small time step k

let a corresp onding nite or innite spatial grid b e dened and let rst b e discretized

with resp ect to the space variables only so that it b ecomes a system of ordinary dierential

equations

v L v

t k

where v tisavector of dimension N and L is a matrix or b ounded linear op erator

k k

With the space discretization determined in this way let then be discretized with

resp ect to t by a linear multistep or RungeKutta formula with time step k We assume

that the stability region S is b ounded by a cuspfree curve Then one can show

STABILITY OF THE METHOD OF LINES

Theorem The metho d of lines discretization describ ed ab ove is stable up to a

factor minfN n g if and only if the pseudoeigenvalues kL satisfy

k

dist SO O k

STABILITY OF THE METHOD OF LINES IN FOURIER SPACE

Theorem A linear constantco ecient metho d of lines discretization as describ ed

ab ove is stable in the norm if and only if the pseudoeigenvalues k L satisfy

k

dist SO O k

When the matrices S or G orL or L app earing in these theorems are normal

k k k k

one can simplify the statements by replacing pseudoeigenvalues by eigenvalues and O by

In particular for a metho d of lines calculation in which the space discretization matrices

L are normal a necessary and sucient condition for stability is that the eigenvalues of

k

fkL g lie within a distance O k of the stability region S as k k

CHAPTER TREFETHEN 

Chapter

Dissipation Disp ersion and Group Velo city

Disp ersion relations

Dissipation

Disp ersion and group velo city

Mo died equations

p

Stabilityin norms

Notes and references

Things fall apart the center cannot hold

Mere anarchy is loosed upon the world

WBYEATS The Second Coming

CHAPTER TREFETHEN 

It would be a ne thing if discrete mo dels calculated solutions to partial

dierential equations exactly but of course they do not In fact in general they

could not even in principle since the solution dep ends on an innite amount

of initial data Instead the b est we can hop e for is that the errors intro duced

by discretization will be small when those initial data are reasonably well

b ehaved

This chapter is devoted to understanding the b ehavior of numerical errors

From truncation analysis wemayhave a b ound on the magnitude of discretiza

tion errors dep ending on the step sizes h and k but much more can be said

for the b ehavior of discretization errors exhibits great regularitywhichcanbe

quantied by the notions of numerical dissipation and disp ersion Rounding

errors to o though intro duced essentially at random propagate in the same

predictable ways

So long as we can estimate the magnitude of the discretization and round

ing errors what is the p oint in trying to investigate their behavior in more

detail There are several answers to this question One is that it is a go o d idea

to train the eye a practitioner familiar with articial numerical eects is less

likely to mistake spurious features of a numerical solution for mathematical or

physical reality Another is that in certain situations it may be advantageous

to design schemes with sp ecial prop ertieslow dissipation for example or

low disp ersion A third is that in more complicated circumstances the mag

nitude of global errors may dep end on the behavior of lo cal errors in ways

that ordinary analysis of discretization and rounding errors cannot predict In

particular we shall see in the next chapter that the stability of b oundary con

ditions for hyp erb olic partial dierential equations dep ends up on phenomena

of numerical disp ersion

One mightsay that this chapter is built around an irony nite dierence

approximations have a more complicated physics than the equations they are

designed to simulate The irony is no paradox however for nite dierences

are used not b ecause the numbers they generate have simple prop erties but

b ecause those numb ers are simple to compute

DISPERSION RELATIONS TREFETHEN 

Disp ersion relations

Any timedep endent scalar linear partial dierential equation with con

stant co ecients on an unb ounded space domain admits plane wave solutions

ixt

ux te R

where is the wave number and is the frequency Vector dierential

equations admit similar mo des multiplied by a constant vector the extension

to multiple space dimensions is describ ed at the end of this section For each

not all values of can be taken in Instead the PDE imp oses a

relationship b etween and

which is known as the disp ersion relation mentioned already in x In

general each wave number corresp onds to m frequencies where m is the

order of the dierential equation with resp ect to t and that is why is

called a relation rather than a function For most purp oses it is appropriate

to restrict attention to values of that are real in whichcase mayberealor

complex dep ending on the PDE The wave decays as t if Im

and grows if Im

For example here again are the disp ersion relations for the mo del equa

tions of x and also for the secondorder wave equation

u u

t x

u u ie

tt xx

u u i

t xx

u iu

t xx

These relations are plotted in Figure Notice the doublevaluedness of

the disp ersion relation for u u and the dashed curve indicating complex

tt xx

values for u u

t xx

More general solutions to these partial dierential equations can be ob

tained by sup erimp osing plane waves so long as each comp onent sat

ises the disp ersion relation the mathematics behind such Fourier synthesis

DISPERSION RELATIONS TREFETHEN 

a u u b u u

t x tt xx

i

c u u d u iu

t xx t xx

Figure Disp ersion relations for four mo del partial dierential

equations The dashed curve in c is a reminder that is complex

DISPERSION RELATIONS TREFETHEN 

was describ ed in Chapter and examples were given in x For a PDE of

rst order in t the result is

Z

ix t

ux t e u d

Since most partial dierential equations of practical imp ortance have variable

co ecients nonlinearity or b oundary conditions it is rare that this integral

representation is exactly applicable but it may still provide insightinto lo cal

b ehavior

Discrete approximations to dierential equations also admit plane wave

solutions at least if the grid is uniform and so they to o have disp ersion

relations To b egin with let us discretize in x only so as to obtain a semidis

crete formula Here are the disp ersion relations for the standard centered

semidiscretizations of

u u sin h

t

h

h

u u sin

tt

h

h

u u i sin

t

h

h

u i u sin

t

h

These formulas are obtained by substituting into the nite dierence

formulas with x x In keeping with the results of x each disp ersion

j

relation is hp erio dic in and it is natural to take h h as

a fundamental domain The disp ersion relations are plotted in Figure

sup erimp osed up on dotted curves from Figure for comparison

Stop for a moment to compare the continuous and semidiscrete curves in

Figure In each case the semidiscrete disp ersion relation is an accurate

approximation when is smal l which corresp onds to many grid points per

wavelength The numberofpoints p er spatial wavelength for the wave

is h In general the disp ersion relation for a partial dierential equation

is a p olynomial relation between and while a discrete mo del amounts to

a trigonometric approximation Although other design principles are p ossible

the standard discrete approximations are chosen so that the trigonometric

function matches the p olynomial to as high a degree as p ossible at the origin

To illustrate this idea Figure plots disp ersion relations for the

standard semidiscrete nite dierence approximations to u u and u iu

t x t xx

of orders and The formulas were given in x

DISPERSION RELATIONS TREFETHEN 

a u u b u u

t tt

i

c u u d u i u

t t

Figure Disp ersion relations for centered semidiscrete approx

imations to the four mo del partial dierential equations Each func

tion is hp erio dic in the plots show the fundamental domain

h h

DISPERSION RELATIONS TREFETHEN 













a u u b u iu

t x t xx

Figure Disp ersion relations for semidiscrete centered dier

ence approximations to u u and u iu of orders

t x t xx

Now let us turn to fully discrete nite dierence formulas discrete in time

as well as space The p ossibilities b ecome numerous For example substituting

the plane wave

ix t

n ij hnk

n

j

v e e

j

into the leap frog approximation of u u yields the disp ersion relation

t x

i k i k i x i x

e e e e

where khthat is

sin k sin h

Similarly the CrankNicolson approximation of u u has the disp ersion

t xx

relation

i k

h i

e

i h i h i k

e e e

which reduces to

k h

i tan sin

DISPERSION RELATIONS TREFETHEN 

These disp ersion relations are h p erio dic in and k p erio dic in

With the use of inverse trigonometric functions it is p ossible to solve such

equations for so as to exhibit the functional dep endence explicitlybut the

resulting formulas are less app ealing and often harder to work with Equations

like and have a certain eleganceone sees at a glance that

the time and space derivatives have been replaced by trigonometric analogs

Tables and consider once again the various nite dierence

approximations to u u and u u that app eared in Tables

t x t xx

and In each case the disp ersion relation is b oth listed and plotted

Since h and k are indep endent parameters there is now a range of p ossible

plots wehave arbitrarily taken kh in the rst table and kh

in the second That is whyeach plot o ccupies a rectangle of asp ect ratio

in wavenumb er spacey Notice that the multistep leap frog formulas contain

two branches of values

For partial dierential equations in several space dimensions the notion of

disp ersion relation generalizes just as x would suggest a plane wave takes

the form

i xt

ux te R

where and x are vectors and b ecomes a scalar function or relation

of a vector argument For example the wave equation

u u u

tt xx yy

has the disp ersion relation

if the vector is written so that the lines of constant in the

plane are concentric circles On the other hand the leap frog approximation

n n n n n n n n

v v v v v v v v

ij ij ij ij ij ij

ij ij

appropriate to a uniform grid with h x y has the disp ersion relation

i h

h h k

sin sin sin

which is plotted in Figure for Once again the disp ersion relation

is accurate for small wave numb ers but diverges dramatically elsewhere

EXERCISES

What are the co ecients as trigonometric functions of the disp ersion relations plotted

in Figure

Sketch the disp ersion relation for the leap frog mo del of u u with say

t x

How is the instability of this nite dierence formula reected in your sketch

Not yet written

y analogous to a Brillouin zone in crystallography C Kittel Intro duction to Solid State Physics

Wiley

DISSIPATION TREFETHEN 

i k

BE Backward Euler i e sin h

x

k

sin h CN CrankNicolson tan

x

LF Leap Frog sin k sin h

h

k

BOX Box tan tan

x

LF thorder Leap Frog sin k sin h sin h

i k

LXF LaxFriedrichs e cos h i sin h

i k i h

UW Upwind e e

h

i k

LW LaxWendro ie sin h i sin

Table Disp ersion relations for various nite dierence approx

imations to u u with kh See Tables and

t x

The dashed lines indicate the slop e d d at isolated p oints where

is real

DISSIPATION TREFETHEN 

Figure Disp ersion relation for the leap frog mo del of u

tt

u u in the limit The region shown is the domain h

xx yy

h of the plane The concentric curves are lines of constant

for h

Dissipation

Even though a partial dierential equation may conserve energy in the L norm its

nite dierence mo dels will often lose energy as t increases esp ecially in the wavenumb ers

comparable to the grid size This prop ertyisnumerical dissipation and it is often advan

tageous since it tends to combat instabilityandunwanted oscillations In fact articial

dissipation is often added to otherwise nondissipativeformulas to achieve those ends An

example of this kind app eared in Exercise

To make the matter quantitative supp ose wehave a linear partial dierential equation

or nite dierence approximation that admits waves with given by a disp ersion

relation Since is assumed to b e real it follows that the wave has absolute value

ixt t Im

je j e

In computational uid dynamics one encounters the more dramatic term articial viscosity

DISSIPATION TREFETHEN 

as a function of tandthus decays exp onentiallyifIm By Parsevals equalitytheL

norm of a sup erp osition of waves is determined by a sup erp osition of such factors

t Im

e u kutk

As an extreme case the heat equation u u with disp ersion relation i dissipates

t xx

nonzero wave numbers stronglyand indeed it do es nothing else at time t only wave

p

numbers O t remain with close to their initial amplitudes But it is principally the

dissipation intro duced by nite dierence formulas that we are concerned with here

The following denitions are standard

A nite dierence formula is nondissipative if Im for all It is dissipative if

Im for all It is dissipative of order r if satises

r ik r

Im k h ie je j h

for some constants In each of these statements varies over the interval

j

h h and represents all p ossible values corresp onding to a given For

problems in multiple space dimensions h is replaced by khk in any norm

For example the leap frog and CrankNicolson mo dels of u u are nondissipative

t x

while the upwind and LaxWendro formulas are dissipative

Dissipative and nondissipative are mutually exclusive but not exhaustive a nite

dierence formula can b e neither dissipative nor nondissipative See Exercise

According to the denition no consistent nite dierence approximation of u u u

t x

could b e dissipative but customary usage would probably use the term dissipative sometimes

for such a problem anyway One could mo dify the denition to account for this by including

a term O k in

A more serious problem with these standard denitions arises in the case of multistep

formulas for which each corresp onds to several values of In such cases the denition

of dissipative for example should b e interpreted as requiring Im for every value of

that corresp onds to some The diculty arises b ecause for multistep formulas that

condition ensures only that the formula dissipates oscillations in space not in time For

n

example the leap frog mo del b ecomes dissipativeif a small term such as k v is added



to it according to our denitions yet the resulting formula still admits the wave

with h whichissawto othed in time To exclude p ossibilities of that kind it is

sometimes desirable to use a stronger denition

A nite dierence formula is totally dissipative if it is dissipative and in addition

Im implies

EXERCISES

Determine whether each of the following mo dels of u u is nondissipative dissi

t x

pative or neither If it is dissipative determine the order of dissipativity

a LaxWendro b Backward Euler c Fourthorder leap frog d Box e Upwind

DISPERSION AND GROUP VELOCITY TREFETHEN  

Disp ersion and group velo city

This section is not yet prop erly written The next few pages contain a few remarks followed by

an extract from my pap er Disp ersion dissipation and stability

The general idea Whereas dissipation leads to decayofawave form disp ersion leads

to its gradual separation into a train of oscillations This phenomenon is a go o d deal less

obvious intuitively for it dep ends up on constructive and destructiveinterference of Fourier

comp onents It is of central imp ortance in nite dierence mo deling b ecause although many

partial dierential equations are nondisp ersive their discrete approximations are almost

invariably disp ersive Sp ectral metho ds are an exception

Caution Phase and group velo city analysis dep end up on the problem b eing linear and

nondissipativeie must be real when is real However similar predictions hold if

there is a suciently slight amount of dissipation

Phase velo city Supp ose that a PDE or nite dierence formula admits a solution

itx

e Its then a triviality to see that any individual wave crest of this wave ie a

pointmoving in sucha way that the quantity inside the parentheses has a constantvalue

the phase moves at the velo city

Phase velo city c

Group velo city However early in the twentieth century it was realized that wave

energy propagates at a dierentvelo city

d

Group velo city c

g

d

The algebraic meaning of this expression is that we dierentiate the disp ersion relation with

resp ect to In a plot in space c is minus the slop e of the line through and

the origin and c is minus the slop e of the line tangent to the disp ersion relation at

g

The physical meaning is that for example a wave packeta smo oth envelop e times an

oscillatory carrier wave with parameters will moveapproximately at the velo city c

g

The same go es for a wave front and for any other signal that can carry information

Disp ersive vs nondisp ersive systems If the disp ersion relation is linear ie

const then and are equal and the system is nondisp ersive If the disp ersion

relation is nonlinear the system is disp ersive Finite dierence formulas are almost always

disp ersive since their disp ersion relations are p erio dic and therefore nonlinear However

see Exercise

Precise meaning of group velo city The meaning of group velo city can b e made precise

in various asymptotic fashions for example by considering the limit t The mathemat

ics b ehind this is usually a stationary phase or steep est descent argument

Simple explanations of group velo city There are a numberofintuitiveways to under

stand where the derivative comes from One is to sup erimp ose twowaves with nearby

DISPERSION AND GROUP VELOCITY TREFETHEN  

parameters and It is then readily seen that the sup erp osition consists of a

and wavenumber smo oth envelop e times a carrier wave at frequency

and the envelop e moves at velo city which approaches in the

itx

limit Another approachis to take a pure exp onential e with

and real and change slightly to a complex value i in order to visualize which

way the envelop e is moving If the eect on is to makeitc hange to i it is readily

calculated that the resulting evanescent wavehasan envelop e that moves laterally at the

velo city and this again approaches in the limit Athird

more PDEstyle explanation is based up on advection of lo cal wavenumb er according to

a simple hyp erb olic equation with co ecient c see Lighthill

g

Group velo cityinmultiple dimensions If there are several space dimensions the group

velo city b ecomes the gradientof with resp ect to the vector ie c r

g

Phase and group velo cities on a grid There is another sense in which group velo city

has more physical meaning than phase velo city on a nite dierence grid the former is

welldened but the latter is not On a p erio dic grid anyFourier mo de can b e represented

in terms of innitely many p ossible choices of and that are indistinguishable physically

and according to eachchoice gives a dierent phase velo city Whats going on here

is that naturally one cant tell how fast a pure complex exp onential waveismoving if one

sees it at only intermittentpoints in space or time for one wave crest is indistinguishable

from another By contrast the group velo cityiswelldened since it dep ends only on the

slop e which is a lo cal prop erty formula has the same p erio dicity as the disp ersion

relation itself

Computation of a group velo city on a grid To compute the group velo city for a

nite dierence formula dierentiate the disp ersion relation implicitly and then solve for

c forall For the leap c d d For example the wave equation u u has c

g g t x

frog approximation the disp ersion relation is sin k sin which implies k cos k d

h cos d and since k h c cos h cos k

g

Parasitic waves Many nite dierence formulas admit parasitic waves as solutions ie

waves that are sawto othed with resp ect to space or time These corresp ond to h

k or b oth It is common for suchwaves to have group velo cities opp osite in sign

to what is correct physically In the example of the leap frog formula all four parasitic

n j n j

and are p ossible with group velo cities and mo des

resp ectively

Spurious wiggles near interfaces and b oundaries It is common to observe spurious

wiggles in a nite dierence calculation and they app ear most often near b oundaries in

terfaces or discontinuities in the solution itself The explanation of where they app ear is

usually a matter of group velo city Typically a smo oth wave has passed through the discon

tinuity and generated a small reected wave of parasitic form which propagates backwards

into the domain b ecause its group velo city has the wrong sign More on this in the next

chapter

Waves in crystals Disp ersion relations for vibrations in crystals are also p erio dic with

resp ect to As a result sound waves in crystals exhibit disp ersive eects muchlike those

asso ciated with nite dierence formulas including the existence of p ositive and negative

group velo cities

DISPERSION AND GROUP VELOCITY TREFETHEN  

Figure Disp ersion under the leap frog mo del of u u with The

t x

lower mesh is twice as ne as the upp er

DISPERSION AND GROUP VELOCITY TREFETHEN  

EXERCISES

The Boxformula

n n

a Write out the BOX formulaofTable in terms of v v etc

x

j j

b Determine the disp ersion relation expressed in as simple a form as p ossible

c Sketch the disp ersion relation

d Determine the group velo city as a function of and

Schrodinger equation

a Calculate and plot the disp ersion relation for the CrankNicolson mo del of u iu of

t xx

Exercise f

b Calculate the group velo city How do es it compare to the group velo city for the equation

u iu itself

t xx

A paradox Find the resolution of the following apparent paradox and b e precise in

stating where the mistake is Drawasketch of an appropriate disp ersion relation to explain

your answer

u bytheleapfrogformula with the results will One the one hand if wesolve u

x t

b e exact and in particular no disp ersion will takeplace

On the other hand as discussed ab ove the disp ersion relation on any discrete grid must b e

p erio dic hence nonlinearand so disp ersion must take place after all

CHAPTER TREFETHEN 

Chapter

Boundary Conditions

Examples

Scalar hyp erb olic equations

Systems of hyp erb olic equations

Absorbing b oundary conditions

Notes and references

OGod I could b e b ounded in a nutshell

and countmyself a king of innite space

were it not that I have bad dreams

WSHAKESPEARE Hamlet I I ii

CHAPTER TREFETHEN 

The diculties caused by b oundary conditions in scientic computing

would be hard to overemphasize Boundary conditions can easily make the

dierence between a successful and an unsuccessful computation or between

a fast and a slow one Yet in many imp ortant cases there is little agreement

ab out what the prop er treatmentofthe b oundary should be

One of the sources of diculty is that although some numerical b oundary

conditions come from discretizing the b oundary conditions for the continuous

problem other articial or numerical b oundary conditions do not The

reason is that the numb er of b oundary conditions required by a nite dierence

formula dep ends on its stencil not on the equation b eing mo deled Thus even

a complete mathematical understanding of the initial b oundary v alue problem

to be solvedwhich is often lackingis in general not enough to ensure a

successful choice of numerical b oundary conditions This situation reaches an

extreme in the design of what are variously known as op en radiation

absorbing nonreecting or fareld boundary conditions which are

numerical artifacts designed to limit a computational domain in a place where

the mathematical problem has no b oundary at all

Despite these remarks p erhaps the most basic and useful advice one can

oer concerning numerical b oundary conditions is this b efore worrying ab out

discretization make sure to understand the mathematical problem If the

IBVP is illp osed no amount of numerical cleverness will lead to a successful

computation This principle may seem to o obvious to deserve mention but it

is surprising how often it is forgotten

Only p ortions of this chapter havebeen written so far The preceding typ ewritten pages from

Chapter however include some of the material that b elongs here

SCALAR HYPERBOLIC EQUATIONS TREFETHEN 

Scalar hyp erb olic equations

This section is not yet prop erly written but some of the essential ideas are summarized b elow

z and The following are standard abbreviations

n n j ixt i k i h

v z e z e e

j

Thus z is the temp oral amplication factor and is the spatial amplica

ixt

tion factor for a wave mo de e We shall often write x and t as here

even though the application mayinvolve only their discretizations x and t

n

j

History The four pap ers listed rst in the references for this chapter t

the following neat pattern The pap er by Kreiss is the classic reference

on wellp osedness of initial boundary value problems and the pap er by

Higdon presents an interpretation of this theory in terms of disp ersive wave

propagation The GKS pap er by Gustafsson Kreiss and Sundstrom is

the classic reference on stability of nite dierence mo dels of initial b oundary

value problems although there are additional imp ortant references by Strang

Osher and others and the pap er by Trefethen presents the disp ersiv e

waves interpretation for that

Leftgoing and rightgoing waves For any real frequency ie jz j

a threep oint nite dierence formula typically admits two wave numb ers

ie and often one will have c and the other c This is true

g g

for example for the leap frog and CrankNicolson mo dels of u u We shall

t x

lab el these wave numb ers and for leftgoing and rightgoing

L R

Interactions at b oundaries and interfaces If a plane wave hits a b oundary

or interface then typically a reected wave is generated that has the same

ie z but dierent ie The reected value must also satisfy the

nite dierence formula for the same so for the simple formulas mentioned

ab ove it will simply b e the other value eg at a lefthand b oundary

R

Reection co ecients Thus at a b oundary it makes sense to lo ok for

singlefrequency solutions of the form

j j

n n i t i x i x

L R

v z e e e

j L R L R

L R

If there is such a solution then it makes sense to dene the reection co ef

cient R by

R

R

L

SCALAR HYPERBOLIC EQUATIONS TREFETHEN

Stable and unstable b oundary conditions Very roughlythe GKS theory

asserts that a lefthand b oundary condition is stable if and only if it admits

no solutions with The idea is that such a solution corresp onds

L

to spurious energy radiating into the domain from nowhere Algebraically

one checks for stabilityby checking whether there are any mo des that

satisfy three conditions

n

satises the interior nite dierence formula v

j

n

satises the discrete b oundary conditions v

j

n

is either v is a rightgoing mo de that

j

a jz j jj and c or

g

b jz j jj

Finite vs innite reection co ecients If R for some the

b oundary condition must be unstable because that implies ab ove

L

The converse however do es not hold That is R may be nite if

R

and happ en to b e zero simultaneously and this is a situation that comes

L

up fairly often A b oundary instability tends to be more pronounced if the

asso ciated reection co ecientisinnite However a pronounced instabilityis

sometimes less dangerous than a weak one for it is less likely to go undetected

The eect of dissipation Adding dissipation or shifting from centered

to onesided interior or b oundary formulas often makes an unstable mo del

stable Theorems to this eect can be found in various pap ers by Goldb erg

Tadmor see the references However dissipation is not a panacea for if

a slight amount of dissipation is added to an unstable formula the resulting

formula maybetechnically stable but still sub ject to oscillations large enough

to b e troublesome

Hyp erb olic problems with tw o b oundaries A general theorem by Kreiss

shows that in the case of a nite dierence mo del of a linear hyp erb olic system

of equations on an interval such as x the twob oundary problem is

stable if and only if eachofthetwo b oundary conditions is individually stable

Hyp erb olic problems in corners Analogously one might exp ect that if a

hyp erb olic system of equations in two space variables x and y is mo deled with

stable b oundary conditions for x and for y then the same problem

would be stable in the quarterdomain x y However examples by

Osher and Sarason Smoller show that this is not true in general

SCALAR HYPERBOLIC EQUATIONS TREFETHEN 

n n n n

a v v b v v

Figure Stable and unstable lefthand b oundary conditions for

the leap frog mo del of u u with v at the righthand

t x

b oundary initial data exact

SCALAR HYPERBOLIC EQUATIONS TREFETHEN 

EXERCISES

Interpretation of Figure In Figure bvarious waves are evidently prop

agating back and forth b etween the two b oundaries In each of the time segments marked

a b c and d one particular mo de dominates What are these four mo des Explain

your answer with reference to a sketch of the disp ersion relation

Exp eriments with CrankNicolson

Go back to your program of Exercise and mo dify it now to implement CN the

x

CrankNicolson mo del of u u This is not a metho d one would use in practice since

t x

yp erb olic case but theres no harm in lo oking theres no need for an implicit formula in this h

at it as an example

The computations b elow should b e p erformed on the interval with h and

Let f x b e dened by

x

f xe

n n

and let the b oundary conditions at the endp oints b e v v except where otherwise



h

sp ecied

a Write the down the disp ersion relation for CN mo del of u u andsketchit In the

x t x

problems b elow you should refer to this sketch rep eatedly Also write down the group

velo city formula

b Plot the computed solutions v x t at time t for the initial data v x

j j

i f x ii f x iii Refi f xg

and explain your results In particular explain whymuch more disp ersion is apparent

in iii than in i or ii

c Plot the computed solutions at time t for initial data v x f x and lefthand

b oundary conditions

n n n

n

i v v ii v v



  

n n n n n n

v v v iv v v iii v

     

ivyou will havetomake a small mo dication in your tridiagonal solver To implement

d Rep eat c for the initial data v x g x maxf jx j g

e On the basis of c and dwhich b oundary conditions would you say app ear stable

Rememb er stability is what we need for convergence and for convergence the error

must decay to zero as k If you are in doubt try doubling or halving k to see what

the eect on the computed solution is

f Prove that b oundary condition i is stable or unstable whichever is correct

g Likewise ii

h Likewise iii

i Likewise iv

ABSORBING BOUNDARY CONDITIONS TREFETHEN 

Absorbing b oundary conditions

A recurring problem in scientic computing is the design of articial b oundaries

How can one limit a domain numerically to keep the scale of the computation within

reasonable b ounds yet still end up with a solution that approximates the correct result

for an unb ounded domain For example in the calculation of the transonic owover an

airfoil what b oundary conditions are appropriate at an articial b oundary downstream In

an acoustical scattering problem what b oundary conditions are appropriate at a spherical

articial b oundary far away from the scattering ob ject

Articial b oundary conditions designed to achieve this kind of eect go bymany names

such as absorbing nonreecting op en radiation invisible or fareld b oundary

sp ecial situations a p erfect articial b oundary cannot be designed conditions Except in

even in principle After all in the exact solution of the problem b eing mo deled interactions

might o ccur outside the b oundary which then propagate backinto the computational domain

at a later time In practice however articial b oundaries can often do very well

Figure The problem of articial b oundaries

There are three general metho ds of coping with unbounded domains in numerical

simulations

Stretched grids change of variables By a change of variables a innite

domain can be mapp ed into a nite one which can then be treated by a nite grid Of

course the equation b eing solved changes in the pro cess A closely related idea is to stay

in the original physical domain but use a stretched grid that has only nitely manygrid

Exp erimentalists have the same problemhow can the walls of a wind tunnel be designed to

inuence the ow as little as p ossible

ABSORBING BOUNDARY CONDITIONS TREFETHEN 

points Metho ds of this kind are app ealing but for most problems they are not the b est

approach

Sp onge layers Another idea is to surround the region of physical interest bya

layer of grid p oints in which the same equations are solved except with an extra dissipation

term added to absorb energy and therebyprevent reections This sp onge layer might

have a width of saytogridpoints

Oneway equations matched solutions A third idea is to devise more sophis

ticated b oundary conditions that allow propagation of energy out of the domain of interest

but not into it In some situations such a b oundary condition can be obtained as a dis

cretization of a oneway equation of some kind More generally one can think of matching

the solution in the computational domain to some known outer solution that is valid near

innity and then design b oundary conditions analytically that are consistent with the latter

See eg pap ers by H B Keller and T Hagstrom

Of these three metho ds the rst and second have a go o d deal in common Both involve

padding the computational domain by wasted p oints whose only function is to prevent

reections and in b oth cases the padding must be handled smo othly if the absorption is

to b e successful This means that a stretched grid should stretch smo othly and an articial

dissipation term should b e turned on smo othly Consequently the region of padding must b e

fairly thick and hence is p otentially exp ensive esp ecially in two or three space dimensions

On the other hand these metho ds are quite general

The third idea is quite dierent and more problemdep endent When appropriate one

way b oundary conditions can b e devised their eect may b e dramatic As a rule of thumb

p erhaps it is safe to say that eective oneway b oundary conditions can usually b e found if

the b oundary is far enough out that the physics in that vicinity is close to linear Otherwise

some kind of a sp onge layer is probably the b est idea

Of course various combinations of these three ideas have also b een considered See

for example S A Orszag and M Israeli J Comp Phys

The remainder of this section will describ e a metho d for designing oneway b oundary

conditions for acoustic wave calculations whichwas develop ed by Lindman in J Comp

Phys and by Engquist and Ma jda in Math Comp and Comm Pure Appl

Math Closely related metho ds w ere also devised byBayliss and Turkel in Comm

Pure Appl Math

For the secondorder wave equation

u u u

tt xx yy

itxy   

the disp ersion relation for wave mo des ux y te is or equivalently

p

DISPERSION RELATION



s

FOR WAVE EQUATION

with

 

s sin

This is the equation of a circle in the plane corresp onding to plane waves propagating

in all directions The wave with wave numbers has velo city c c

g

cos sin where is the angle counterclo ckwise from the negative x axis See Figure

ABSORBING BOUNDARY CONDITIONS TREFETHEN

v

y

v

x

a space b Fourier space

Figure Notation for the onewaywave equation

p

**oo

 s

*o *o

r s

s

 

p



Figure The approximation problem r s s

By taking the plus or minus sign in onlywe can restrict attention to leftgoing

 

j j or rightgoing j j waves resp ectively For deniteness we shall cho ose the

former course which is appropriate to a lefthand b oundary and write

p

DISPERSION RELATION



s

FOR IDEAL OWWE

See Figure again

Because of the square ro ot is not the disp ersion relation of any partial dieren

tial equation but of a pseudo dierential equation The idea b ehind practical onewaywave

equations is to replace the square ro ot by a rational function r softyp e m n for some m

ABSORBING BOUNDARY CONDITIONS TREFETHEN

and n that is the ratio of a p olynomial p of degree m and a p olynomial q of degree n

m n

p s

m

r s

q s

n

Then b ecomes

DISPERSION RELATION FOR

rs

APPROXIMATE OWWE

for the same range of s and See Figure By clearing denominators we can

transform into a p olynomial of degree maxfm n g in and and this is the

disp ersion relation of a true dierential equation

The standard choice of the approximation r sr s is the Padeapproximant to

mn

p



s which is dened as that rational function of the prescrib ed typ e whose Taylor series

p



at s matches the Taylor series of s to as high an order as p ossible Assuming that

m and n are even the order will b e

p

mn



r s s O s

mn

p



For example the typ e Pade approximant to s is r s Taking this

choice in gives

ie u u

x t

This simple advection equation is thus a suitable loworder absorbing b oundary condition

for the wave equation at a lefthand b oundary For a numerical simulation one would

discretize it by a onesided nite dierence formula In computations is much b etter

than a Neumann or Dirichlet b oundary condition at absorbing outgoing waves

p







At the next order the typ e Padeapproximantto s is r s s Taking



this choice in gives

 





that is

 

 

ie u u u

xt tt yy

 

This b oundary condition absorbs waves more eectively than and is the most com

monly used absorbing b oundary condition of the EngquistMa jda typ e

p



As a higher order example consider the typ e Padeapproximantto s





s

r s





s

Then b ecomes

 

 

 

or

   

   

ie u u u u

xtt xy y ttt ty y

This thirdorder b oundary condition is harder to implement than but provides ex

cellent absorption of outgoing waves

ABSORBING BOUNDARY CONDITIONS TREFETHEN 

Why didnt welookatthePade approximation of typ e There are twoanswers

First although that approximation is of the the same order of accuracy as the typ e

approximation it leads to a b oundary condition of order instead of which is even

harder to implement Second that b oundary condition turns out to b e illp osed and is thus

useless in any case Exercise In general it can b e shown that the EngquistMa jda

b oundary conditions are wellp osed if and only if m n or n that is for precisely those

approximations r s taken from the main diagonal and rst sup erdiagonal in the Pade

table L N Trefethen L Halp ern Math Comp pp

Figure taken from Engquist and Ma jda illustrates the eectiveness of their

absorbing b oundary conditions The upp erleft plot A shows the initial data a quarter

circular wave radiating out from the upp erright corner If the b oundaries have Neumann

b oundary conditions the wave reects with reection co ecient at the lefthand b oundary

B and again at the righthand b oundary C Dirichlet b oundary conditions are equally

bad except that the reection co ecient b ecomes D Figure E shows the typ e

absorbing b oundary condition and the reected wave amplitude immediately cuts

to ab out In Figure F with the typ e b oundary condition the amplitude

is less than

EXERCISES

Illp osed absorbing b oundary conditions

p



a What is the typ e Padeapproximant r sto s

b Find the co ecients of the corresp onding absorbing b oundary condition for a lefthand

boundary

itxy

c Show that this b oundary condition is illp osed by nding a mo de ux y te

that satises b oth the wave equation and the b oundary condition with R Im

Im Explain why the existence of such a mo de implies that the initial b oundary

value problem is illp osed In a practical computation would you exp ect the illp osed

mo de to show up as mild or as explosive

Absorbing b oundary conditions for oblique incidence Sometimes it is advantageous

to tune an absorbing b oundary condition to absorb not waves that are normally incidentat

the b oundary but waves at some sp ecied angle or angles Use this idea to derive absorbing



b oundary conditions that are exact for plane waves traveling at in the southwest direction

as they hit a lefthand b oundary

a Typ e b Typ e

Dont worry ab out rigor or ab out wellp osedness This problem concerns partial dierential

equations only not nite dierence approximations

This gure will app ear in the published version of this b o ok only with p ermission

ABSORBING BOUNDARY CONDITIONS TREFETHEN 

Figure Absorbing b oundary conditions for u u u from Engquist

tt xx yy Ma jda

CHAPTER TREFETHEN 

Chapter

Fourier sp ectral metho ds

An example

Unb ounded grids

Perio dic grids

Stability

Notes and references

Think globally Act lo cally

BUMPER STICKER

CHAPTER TREFETHEN 

Finite dierence metho ds like nite element metho ds are based on lo cal

representations of functionsusually by loworder p olynomials In contrast

sp ectral metho ds make use of global representations usually by highorder

p olynomials or Fourier series Under fortunate circumstances the result is a

degree of accuracy that lo cal metho ds cannot match For largescale compu

tations esp ecially in several space dimensions this higher accuracy may be

most imp ortant for p ermitting a coarser mesh hence a smaller numb er of data

values to store and op erate up on It also leads to discrete mo dels with little

or no articial dissipation a particularly v aluable feature in high Reynolds

number uid ow calculations where the small amountofphysical dissipation

maybeeasilyoverwhelmed byany dissipation of the numerical kind Sp ectral

metho ds have achieved dramatic successes in this area

Some of the ideas b ehind sp ectral metho ds have been intro duced several

times into numerical analysis One early prop onentwas Cornelius Lanczos in

the s who showed the p ower of Fourier series and Chebyshev p olynomials

in a variety of problems where they had not b een used b efore The emphasis

was on ordinary dierential equations Lanczoss work has been carried on

esp ecially in Great Britain byanumb er of colleagues suchasCWClenshaw

More recently sp ectral metho ds were intro duced again by Kreiss and

Oliger Orszag and others in the s for the purp ose of solving the par

tial dierential equations of uid mechanics Increasingly they are becoming

viewed within some elds as an equal comp etitor to the better established

nite dierence and nite element approaches At present however they are

less well understo o d

Sp ectral metho ds fall into various categories and one distinction often

made is between Galerkin tau and collo cation or pseudosp ectral

sp ectral metho ds In a word the rst twowork with the co ecients of a global

expansion and the latter with its values at p oints The discussion in this b o ok

is entirely conned to collo cation metho ds which are probably used the most

often chiey b ecause they oer the simplest treatment of nonlinear terms

Sp ectral metho ds are aected far more than nite dierence metho ds by

the presence of b oundaries which tend to intro duce stability problems that are

illundersto o d and sometimes highly restrictive as regards time step Indeed

diculties with b oundaries direct and indirect are probably the primary rea

son why sp ectral metho ds have not replaced their loweraccuracy comp etition

CHAPTER TREFETHEN 

in most applications Chapter considers sp ectral metho ds with b oundaries

but the presentchapter assumes that there are none This means that the spa

tial domain is either innitea theoretical device not applicable in practice

or p erio dic In those cases where the physical problem naturally inhabits a

p erio dic domain sp ectral metho ds maybe strikingly successful Conspicuous

examples are the global circulation mo dels used by meteorologists Limited

area meteorological co des since they require b oundaries are often based on

nite dierence formulas but as of this writing almost all of the global circula

tion co des in use which mo del ow in the atmosphere of the entire spherical

earth are sp ectral

AN EXAMPLE TREFETHEN 

An example

Sp ectral metho ds have b een most dramatically successful in problems with

p erio dic geometries In this section we presenttwo examples of this kind that

involve elastic wave propagation Both are taken from B Fornb erg The

pseudosp ectral metho d comparisons with nite dierences for the elastic wave

equation Geophysics Details and additional examples

can be found in that pap er

Elastic waves are waves in an elastic medium such as an iron bar a build

ing or the earth and they come in two varieties P waves pressure or

primary characterized by longitudinal vibrations and S waves shear or

secondary characterized by transverse vibrations The partial dierential

equations of elasticity can be written in various forms such as a system of

two secondorder equations involving displacements For his numerical simu

lations Fornb erg choseaformulation as a system of ve rstorder equations

Figures and show the results of calculations for two physical

problems In the rst a P wave propagates uninterruptedly through a p erio dic

uniform medium In the second an oblique P wave oriented at hits a

horizontal interface at whichthewave sp eeds abruptly cut in half The result

is reected and transmitted P and S waves For this latter example the actual

computation was p erformed on a domain of twice the size shown which is

a hint of the trouble one may be willing to go to with sp ectral metho ds to

avoid coping explicitly with b oundaries

The gures show that sp ectral metho ds may sometimes decisively outp er

form secondorder and fourthorder nite dierence metho ds In particular

sp ectral metho ds are nondispersive and in a wave calculation that prop erty

can be of great imp ortance In these examples the accuracy achieved by the

sp ectral calculation on a grid is not matched by fourthorder nite dif

ferences on a grid or by secondorder nite dierences on a

grid The corresp onding dierences in work and storage are enormous

Fornberg picked his examples carefully sp ectral metho ds do not always

p erform so convincingly Nevertheless sometimes they are extremely impres

sive Although the reasons are not fully understo o d their advantages often

hold not just for problems involving smo oth functions but even in the presence

of discontinuities

The gures in this section will app ear in the published version of this b o ok only with p ermission

UNBOUNDED GRIDS TREFETHEN 

a Schematic initial and end states

b Computational results

Figure Sp ectral and nite dierence simulations of a P wave

propagating through a uniform medium from Fornb erg

UNBOUNDED GRIDS TREFETHEN 

a Schematic initial and end states

b Computational results

Figure Sp ectral and nite dierence simulations of a P wave

incident obliquely up on an interface from Fornb erg

UNBOUNDED GRIDS TREFETHEN 

Unb ounded grids

We shall b egin our study of sp ectral metho ds by lo oking at an innite un

b ounded domain Of course real computations are not carried out on innite

domains but this simplied problem contains many of the essential features

of more practical sp ectral metho ds

Consider again the set of squareintegrable functions v fv g on the

j

h

unb ounded regular grid hZ As mentioned already in x the foundation

of sp ectral metho ds is the sp ectral dierentiation op erator D

h h

ays One is by means of the which can be describ ed in several equivalent w

Fourier transform

SPECTRAL DIFFERENTIATION BY THE SEMIDISCRETE FOURIER TRANS

Compute v

Multiply by i

Inverse transform



Dv F i F v

h

h

Another is in terms of bandlimited interp olation As describ ed in x

one can think of the interp olantasaFourier integral of bandlimited complex

exp onentials or equivalentlyasan innite series of sinc functions

SPECTRAL DIFFERENTIATION BY SINC FUNCTION INTERPOLATION

P



Interp olate v by a sum of sinc functions q x v S x x

k 

k h k

Dierentiate the interp olant at the grid points x

j



x Dv q

j j

Recall that the sinc function S x dened by

h

sin xh

S x

h xh

UNBOUNDED GRIDS TREFETHEN 

is the unique function in L that interp olates the discrete delta function e

j

j

e

j

j

and that moreover is bandlimited in the sense that its Fourier transform has

compact supp ort contained in h h

For higher order sp ectral dierentiation we multiply F v by higher

h

powers of i or equivalently dierentiate q x more than once

tal reason is Why are these two descriptions equivalent The fundamen

that S x is not just any interp olant to the delta function e but the band

h

limited interp olant For a precise argument note that both pro cesses are

obviously linear and it is not hard to see that b oth are shiftinvariantin the

m m

sense that D K v K Dv for any m The shift op erator K was dened

can be written as a convolution in Since an arbitrary function v

h

P



sum v v e it follows that it is enough to prove that the two

k 

j

k j k

pro cesses give the same result when applied to the particular function e That

equivalence results from the fact that the Fourier transform of e is the constant

function h whose inverse Fourier transform is in turn precisely S x

h

Since sp ectral dierentiation constitutes a linear op eration on it can

h

also b e viewed as multiplication by a biinnite To eplitz matrix

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

D

B C

h

B C

B C

B C

B C

B C

B C

B C

B C

A

As discussed in x this matrix is the limit of banded To eplitz matrices

corresp onding to nite dierence dierentiation op erators of increasing orders

of accuracy see Table on p In this chapter we drop the subscript

on the symbol D used in x We shall be careless in this text ab out



the distinction between the op erator D and the matrix D that represents it

Another way to express the same thing is to write

Dv a v a

h

UNBOUNDED GRIDS TREFETHEN 

S x

*o h *o

S x

x h

*o o * *o ****oooo ****oooo *o o * *o *o

*o

a b

Figure The sinc function S x and its derivativeS x

x

h h

as in

The co ecients of can be derived from either the Fourier

transform or the sinc function interpretation Let us b egin with the latter

The sinc function has derivative

cos xh sinxh

S x

h x

x x h

with values

if j

S x

h x j

j



if j

jh

at the grid points See Figure This is precisely the zeroth column

of D since that column must by denition contain the values on the grid of

the sp ectral derivative of the delta function The other columns corresp ond

ing to delta functions centered at other points x contain the same entries

j

appropriately shifted

Now let us rederive by means of the Fourier transform If

d

Dv a v then Dv a v and for sp ectral dierentiation we want

a i Therefore by the inverse semidiscrete Fourier transform

Z

h

i j h

i e d a

j

h

For j the integral is giving a while for j integration by parts

Think ab out this Make sure you understand why represents a column rather than a rowof

D

UNBOUNDED GRIDS TREFETHEN 

yields

h

Z

i j h i j h

h

e ie

a i d

j

ij h ij h

h

h

The integral here is again zero since the integrand is a periodic exp onential

and what remains is

i j i j

a e e

j

j h h h

j

i j i j

e e

jh jh

as in

The entries of D are suggestiveoftheTaylor expansion of log x and

this is not a coincidence In the notation of the spatial shift op erator K of

D can be written

  

D K K K K K K

h h

which corresp onds formally to



log K log K D

h h

K

log log K



h h

K

hD

Therefore formally e K and this makes sense byintegrating the deriva

tive over a distance h one gets a shift See the pro of of Theorem

i j h

If v e for some h h then Dv i v Therefore i is an

j

eigenvalue of the op erator D On the other hand if v has the same form with

h h then will be indistinguishable on the grid from some alias

 

wave number h h with h for some integer and the



result will b e Dv i v In other words in Fourier space the spatial dieren

tiation op erator b ecomes multiplication by a p erio dic function thanks to the

discrete grid and in this sense is only an approximation to the exact dier

entiation op erator for continuous functions Figure shows the situation

graphically For bandlimited data however the sp ectral dierentiation op

erator is exact in contrast to nite dierence dierentiation op erators which

are exact only in the limit dashed line in the Figure

i j h

Actually this is not quite true by denition an eigenvector must b elong to and e do es not

Strictly sp eaking i is in the sp ectrum of D but is not an eigenvalue However this technicalityis

unimp ortant for our purp oses and will b e ignored in the present draft of this b o ok

UNBOUNDED GRIDS TREFETHEN 

eigenvalue

i

h h h h

Figure Eigenvalue of D divided by i corresp onding to the

i x

eigenfunction e as a function of The dashed line shows corre

sp onding eigenvalues for the nite dierence op erator D

ih

C C

ih

ih

ih

a D nite dierence b D sp ectral

Figure Eigenvalues of nite dierence and sp ectral rstorder

dierentiation matrices as subsets of the complex plane

Figure compares the sp ectrum of D to that of the secondorder nite

dierence op erator D of x

D is a b ounded linear op erator on withnorm

h

kD k max ji j

h

 h h

see xx Notice that the norm increases to innity as the mesh is rened

UNBOUNDED GRIDS TREFETHEN 

This is inevitable as the dierentiation op erator for continuous functions is

unb ounded

So far wehave describ ed the sp ectral approximation to the rst derivative

op erator x but it is an easy matter to approximate higher derivatives to o

For the second derivative the co ecients turn out to b e

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

D

B C

h

B C

B C

B C

B C

B C

B C

B C

B C

A

To derive the entries of this matrix one can simply square D this leads to

innite series to be summed One can dierentiate a second time Or

one can compute the inverse Fourier transform ofa asfollows Two

integrations by parts are involved and terms that are zero have b een dropp ed

For j

Z

h

i j h

a e d

j

h

Z

i j h

h

e

d

ij h

h

h

j ij ij i j h

e e e

ij h j h j h

h

For j the integral is simply

a

h h

i j h

The eect of D on a function v e is to multiply it by the square

j

of the factor asso ciated with D as illustrated in Figure Again one has

a periodic multiplier that is exactly correct for h h The dashed

line shows the analogous curve for the standard centered threep oint nite

UNBOUNDED GRIDS TREFETHEN 

eigenvalue

h h h h

Figure Eigenvalue of D corresp onding to the eigenfunction

i x

e as a function of The dashed line shows corresp onding eigen

values for the nite dierence op erator



C C

h h

a nite dierence b D sp ectral



Figure Eigenvalues of nite dierence and sp ectral second

order dierentiation matrices as subsets of the complex plane

dierence op erator of x The sp ectrum of D must be real since D is



symmetric it is the interval h

The developments of this section are summarized and generalized to the

mthorder case in the following theorem

UNBOUNDED GRIDS TREFETHEN 

SPECTRAL DIFFERENTIATION ON AN UNBOUNDED REGULAR GRID

m

Theorem The mthorder sp ectral dierentiation op erator D is a

with norm b ounded linear op erator on

h

m m

kD k

h

m m m

If m is odd D has the imaginary sp ectrum ih ih and can

be represented by an innite skewsymmetric To eplitz matrix with entries

m

a a h for j

j

m m m

If m is even D has the real sp ectrum h and can be

represented by an innite symmetric To eplitz matrix with entries

m

a a h for j

j

The purp ose of all of these sp ectral dierentiation matrices is to solve

partial dierential equations In a sp ectral collo cation computation this is

done in the most straightforward way p ossible one discretizes the continuous

problem as usual and integrates forward in time by a discrete formula usually

by nite dierences Spatial derivatives are approximated by the matrix

D This same prescription holds regardless of whether the partial dierential

equation has variable co ecients or nonlinear terms For example to solve

u axu by sp ectral collo cation one approximates axu at each time step

t x x

by ax Dv For u u one uses D v where v denotes the p ointwise

t x

j

square v v Alternative discretizations may also be used for b etter

i i

stability prop erties see This is in contrast to Galerkin or tau sp ectral

metho ds in which one adjusts the co ecients in the Fourier expansion of v to

satisfy the partial dierential equation globally

Sp ectral approximations with resp ect to time can also sometimes b e used see H TalEzer Sp ectral

metho ds in time for hyp erb olic equations SIAM J Numer Anal pp

UNBOUNDED GRIDS TREFETHEN 

EXERCISES



Co ecients of D Determine the matrix co ecients of the thirdorder sp ectral



dierentiation matrix D Compare your result with the co ecients of Table

R

a Compute the integral S xdx of the sinc function One could do this by

h



complex contour integration but instead be cleverer than that and nd the answer

by considering the Fourier transform The argument is quite easy b e sure to state it

precisely

b By considering the trap ezoid rule for integration Exercise explain why the

answer ab ove had to come out as it did Hint what is the integral of a constant function

PERIODIC GRIDS TREFETHEN 

Perio dic grids

To b e implemented in practice a sp ectral metho d requires a b ounded domain In this

section we consider the case of a p erio dic domainor equivalentlyanunb ounded domain

on which we p ermit only functions with a xed p erio dicity The underlying mathematics

of discrete Fourier transforms was describ ed in x The next chapter will deal with more

general b ounded problems

N N

x x x

N N N N

 

 

*********ooooooooo *********ooooooooo

x

h

Figure Space and wavenumb er domains for the discrete Fourier trans

form

To rep eat some of the material of x our fundamental spatial domain will now be

as illustrated in Figure Let N b e a p ositiveeven integer set

h N even

N

and dene x jh for any j The grid p oints in the fundamental domain are

j

x x x h



N N

and the invaluable identity is this

N

h

were dened in x as was the discrete Fourier transform The norm kk and space

N

N

X

i j h

v F v h e v

N j

j N

the inverse discrete Fourier transform

N

X



i j h

v F v e v

j j

N

N

and the discrete convolution

N

X

v w v w h

k j j k

j N

PERIODIC GRIDS TREFETHEN 

Recall that since the spatial domain is p erio dic the set of wave numbers is discrete

namely the set of integers Zand this is whywehavechosen to use itself as a subscript

in and We take NN as our fundamental wave number

domain The prop erties of the discrete Fourier transform were summarized in Theorems

and recall in particular the convolution formula

d

v w v w

As was describ ed in x the discrete Fourier transform can b e computed with great

eciency b y the fast Fourier transform FFT algorithm for which a program was given on

p The discovery of the FFT in was an imp etus to the development of sp ectral

metho ds for partial dierential equations in the s Curiouslyhowever practical imple

mentations of sp ectral metho ds do not always make use of the FFT but instead sometimes

p erform an explicit matrix multiplication The reason is that in largescale computations

whichtypically involvetwo or three space dimensions the grid in each dimension mayhave

as few as or p oints or even fewer in socalled sp ectral element computations and

these numb ers are low enough that the costs of an FFT and of a matrix multiplication may

b e roughly comparable

Now we are ready to investigate D the sp ectral dierentiation op erator

N N N

for N p erio dic functions As usual D can b e describ ed in various ways One is by means

N

of the discrete Fourier transform

SPECTRAL DIFFERENTIATION BY THE DISCRETE FOURIER TRANSFORM

Compute v

Multiply by i except that v is multiplied by

N

Inverse transform



D v F for N i v otherwise

N

N

The sp ecial treatment of the value v is required to maintain symmetry and app ears

N

only in sp ectral dierentiation of odd order

Another is in terms of interp olation by a nite series of complex exp onentials or

equivalently p erio dic sinc functions

PERIODIC SPECTRAL DIFFERENTIATION BY SINC INTERPOLATION

Interp olate v by a sum of p erio dic sinc functions

N

X

v S x x q x

k N k

k N

Dierentiate the interp olant at the grid p oints x

j



D v q x

N j j

PERIODIC GRIDS TREFETHEN 

In the second description wehave made use of the p erio dic sinc function on the N p oint

grid

x

sin

h

S x

N

x

tan

h

which is the unique p erio dic function in L that interp olates the discrete delta function

e on the grid

j

j

e

j

j N N

and which is bandlimited in the sense that its Fourier transform has compact supp ort

contained in h h and furthermore satisesv v Compare

N N

For higher order sp ectral dierentiation on the p erio dic grid wemultiplyv by higher

powers of i or equivalentlydierentiate q x more than once If the order of dierentiation

is o ddv is multiplied by the sp ecial valuetomaintain symmetry

N

As in the last section the sp ectral dierentiation pro cess can b e viewed as multiplica

tion byaskewsymmetric To eplitz matrix D compare N

PERIODIC GRIDS TREFETHEN 

 h  h  h

cot cot cot

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

 h  h  h  h

cot cot cot cot D

B C

N

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

A

h  h  h 

cot cot cot

In contrast to the matrix D of D is nite it applies to vectors v v

N

N N

and has dimension N N D is not only a To eplitz matrix but is in fact circulant

N

This means that its entries D wrap around dep ending not merely on i j but on

ij

N

i j mo d N

As in the last section the entries of can b e derived either by the inverse discrete

Fourier transform or by dierentiating a sinc function The latter approach is illustrated in



Figure which shows S and S for N Since N is even symmetry implies that

N N



S x for x as well as for x Dierentiation yields

N

x x

sin cos



h h

x S

N

x

x 

tan

sin

h

and at the grid p oints the values are

if j



S x

N j

jh



j

cot if j

Notice that for jjhj these values are approximately the same as in Thus the

i j entry of D as indicated in is

N

if i j

D

N ij

x x

i j



ij

cot if i j

Any circulant matrix describ es a convolution on a p erio dic grid and is equivalent to a p ointwise

multiplication in Fourier space In the case of D that multiplication happ ens to b e by the function

N

i

PERIODIC GRIDS TREFETHEN 

S x

o N

* *o

S x N

*o *o ****oooo ****oooo *o *o *o *o *o

*o

a b



Figure The p erio dic sinc function S x and its derivative S x

N N

To derive by the Fourier transform one can make use of summation by parts

the discrete analog of integration by parts See Exercise

i x

The eigenvectors of D are the vectors e with Z and the eigenvalues are the

N

quantities i with N N These are precisely the factors i in the denition

of D by the Fourier transform formula The numb er is actually a double eigenvalue

N

of D corresp onding to two distinct eigenvectors the constant function and the sawto oth

N

What ab out the sp ectral dierentiation op erator of second order Again there are

various ways to describ e it This time b ecause is an even function no sp ecial treatment

of N is required to preserve symmetry in the Fourier transform description

Computev

Multiply by

Inverse transform





D v F v

N N

The sinc interp olation description follows the usual pattern

Interp olate v by a sum of p erio dic sinc functions

N

X

v S x x q x

k N k

k N

terp olanttwice at the grid p oints x Dierentiate the in

j





D v q x

j j

N

The matrix lo oks like this compare

Now that D is nite they are truly eigenvalues there are no technicalities to worry ab out as in

N the fo otnote on p

PERIODIC GRIDS TREFETHEN 

h h h    

csc csc csc

h

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C



h  h   h  h 

D

B C

csc csc csc csc

N

h

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

B C

A

 h  h  h 

csc csc csc

h



Note that b ecause N has b een treated dierently in the two cases D is not the

N

square of D which is why we have put the sup erscript in parentheses In general the

N



mthorder sp ectral dierentiation op erator can b e written as a p ower of D if m is even

N



and as a p ower of D or as D times a p ower of D ifm is o dd See Exercise

N N

N

Figures are the analogs of Figures for a p erio dic grid

We summarize and generalize the developments of this section in the following theorem

PERIODIC GRIDS TREFETHEN 

SPECTRAL DIFFERENTIATION ON A PERIODIC GRID

Theorem Let N be even If m is o dd the mthorder sp ectral dierentiation matrix

m

isaskewsymmetric matrix with entries D

N

a a for j

 j

m m

eigenvalues i i and norm

h h

m

m

kD k

N

h

m

If m is even D is a symmetric matrix with entries

N

a a for j

 j

m m

eigenvalues and norm

h

m

m

kD k

N h

PERIODIC GRIDS TREFETHEN 

eigenvalue i

*o *o *o *o *o *o *o *o *o *o *o *o *o *o *o *o *o. . . *o. . . *o. . . *o. . . . *o. . *o. . *o. . . *o. . *o. . *o. . o. o. o. o. o. o. o.

* * * * * * *

. *o. . *o. . *o. .

h h h h

...... *o . . . *o . . . *o . . . *o *o *o *o *o *o *o *o *o *o *o *o *o *o *o *o

*o *o *o *o

Figure Eigenvalue of D divided by i corresp onding to the eigenfunc

N

i x

tion e as a function of for N The smaller dots show corresp onding

eigenvalues for the nite dierence op erator

 ih *o

o

C C * *o *o o *o *o ih *o *o *o *o *o *o *o *o *o o

o * ih * * *o *o *o *o

*o

ih

a nite dierence b DN sp ectral



Figure Eigenvalues of nite dierence and sp ectral rstorder dierenti

ation matrices on a p erio dic grid as subsets of the complex plane for N

PERIODIC GRIDS TREFETHEN 

eigenvalue

. . . *o. *o *o. *o. *o *o. *o. *o *o.

*o. *o. *o. *o. *o. *o.

h h h h o. o. o. o. o. o. o. o. * . . * * . . * * . . * * . . * *o . . *o *o . . *o *o . . *o *o . . *o ...... *o . . . *o *o . . . *o *o . . . *o *o . . . *o

*o *o *o *o *o *o *o *o

*o *o *o *o *o *o *o *o

*o *o *o *o



i x

Figure Eigenvalue of D corresp onding to the eigenfunction e asa

N

function of for N The smaller dots show corresp onding eigenvalues for

the nite dierence op erator



C C

h h

***ooo*o * o * o * o * o * o * **ooo **********oooooooooo*o *o *o *o *o *o *o

a nite dierence b D sp ectral



N

Figure Eigenvalues of secondorder nite dierence and sp ectral dieren

tiation matrices on a p erio dic grid as subsets of the complex plane for N

PERIODIC GRIDS TREFETHEN 

EXERCISES

Fourier transform derivation of D Not yet written

N



D D For most values of N the matrix D would serve as quite a go o d discrete

N N

N



secondorder dierentiation op erator but as mentioned in the text it is not identical to D

N



a Determine D D andD for N and conrm that the latter two are not equal

N N

N

b Explain the result of a by considering sinc interp olation as in Figure

c Explain it again by considering Fourier transforms as in Figure and

J

for arbitrary J d Give an exact formula for the eigenvalues of D

N

tiation Sp ectral dieren

Making use of a program for computing the FFT in Matlab such a program is builtin in

Fortran one can use the program of p write a program DERIV that computes the

m thorder sp ectral derivativeofanN p oint data sequence v representing a function dened

on or

N length of sequence p ower of input

m order of derivativeinteger input

v sequence to b e dierentiated real sequence of length N input

w m th sp ectral derivativeofv real sequence of length N output

Although a general FFT co de deals with complex sequences make v and w real variables

in your program since most applications concern real variables Allow m to b e anyinteger

m and make sure to treat the distinction prop erly b etween even and o dd values of m

Test DERIV with N and N on the functions



u x expsin x and u xj sin xj



for the values m and hand in two tablesone for each functionof the resulting

errors in the discrete norm Put a star where appropriate and explain your results

Plot the computed derivatives with m

Sp ectral integration Mo dify DERIV to accept negativeaswell as p ositivevalues of

m For m DERIV should return a scalar representing the denite integral of v over one

period together with a function w representing the jmjth indenite integral Explore the

behavior of DERIV with various v and m

Hamming window Write a program FILTER that takes a sequence v and smo oths

v multiplying the transform by the Hamming window it by transforming to

k

w cos v

k k

N

and inverse transforming Apply FILTER to the function j sin xj and hand in a plot of the result

PERIODIC GRIDS TREFETHEN 

Fourier transform derivation of D Not yet written

N



D D For most values of N the matrix D would serve as quite a go o d discrete

N N

N



secondorder dierentiation op erator but as mentioned in the text it is not identical to D

N



aDetermine D D andD for N and conrm that the latter two are not equal

N N

N

bExplain the result of aby considering sinc interp olation as in Figure

cExplain it again by considering Fourier transforms as in Figures and

Sp ectral dierentiation Typ e the program FFT of Figure into your computer

t you understand how to compute b oth a and exp eriment with it until you are conden

discrete Fourier transform and an inverse discrete Fourier transform For example try as

input a sine wave and a sinc function and make sure the output you get is what you exp ect

Then makesureyou can get the input backagainbyinversion

Making use of FFT write a program DERIVN m v w which returns the m thorder sp ec

tral derivativeofthe N p oint data sequence v representing a function dened on or

N length of sequence p ower of input

m order of derivativeinteger input

v sequence to b e dierentiated real sequence of length N input

w m th sp ectral derivativeofv real sequence of length N output

Although FFT deals with complex sequences make v and w real variables in your program

since most applications concern real variables Allow m to b e anyinteger m and make

sure to treat the distinction prop erly b etween even and o dd values of m

Test DERIV with N and N on the functions





u x exp sin x and u xj sin xj



for the values m and hand in two tablesone for each functionof the result

ing errors in the discrete norm Explain your results If p ossible plot the computed

derivatives with m

Sp ectral integration Mo dify DERIV to accept negativeaswell as p ositivevalues of

m For m DERIV should return a scalar representing the denite integral of v over one

period together with a function w representing the jmjth indenite integral Explore the

behavior of DERIV with various v and m

Hamming window Write a program FILTERN v w whichtakes a sequence v and

smo oths it by transforming tov multiplying the transform by the Hamming window

k

w cos v

k k

N

and inverse transforming Apply FILTER to the function j sin xj and hand in the result A

plot would b e nice

STABILITY TREFETHEN 

Stability

Just a few results so far The nished section will b e more substantial

Sp ectral metho ds are commonly applied to timedep endent problems according to the

metho d of lines prescription of x rst the problem is discretized with resp ect to space

and then the resulting system of ordinary dierential equations is solved by a nite dierence

metho d in time As usual we can investigate the eigenvalue stability of this pro cess by

examining under what conditions the eigenvalues of the sp ectral dierentiation op erator are

contained in the stability region of the time discretization formula The separate question of

stability in the sense of the Lax Equivalence Theorem is rather dierent and involves some

subtleties that were not present with nite dierence metho ds these issues are deferred to

the next section

In xx wehaveintro duced two families of sp ectral dierentiation matrices D and

m

m

not exactly its p owers D for an innite grid and D and its higherorder analogs D

N

N

powers for a p erio dic grid The sp ectra of all of these matrices lie in the closed left half of

the complex plane and that is the same region that comes up in the denition of Astability

We conclude that if any equation

m

u

u

t

m

x

is mo deled on a regular grid by sp ectral dierentiation in space and an Astable formula in

time the result is eigenvalue stable regardless of the time step

By Theorem an Astable linear multistep formula must b e implicit For a mo del

problem as simple as the system of equations involved in the implicit formula can b e

solved quickly by the FFT but in more realistic problems this is often not true Since sp ec

tral dierentiation matrices are dense unlike nite dierence dierentiation matrices the

implementation of implicit formulas can b e a formidable problem Therefore it is desirable

to lo ok for explicit alternatives

On a regular grid satisfactory explicit alternatives exist For example supp ose we

solve u u by sp ectral dierentiation in space and the midp ointformula in time

t x

The stability region for the midp oint formula is the complex interval ik ik From

Theorem or Figure we conclude that the timestability restriction is h k

ie

h

k

This is stricter by a factor than the timestability restriction k h for the leap frog

formula which is based on secondorder nite dierence dierentiation The explanation

go es back to the fact that the sawto oth curve in Figure is times taller than the

dashed one

On a p erio dic grid Theorem or Figure lo osens slightly to

h

k

h N

Other explicit timediscretization formulas can also b e used with u u so long as

t x

their stability regions include a neighb orho o d of the imaginary axis near the origin Figure

STABILITY TREFETHEN 

reveals that this is true for example for the AdamsBashforth formulas of orders

The answer to Exercise b can readily b e converted to the exact stability b ound for

the rdorder AdamsBashforth discretization

For u u the eigenvalues of D or D b ecome real and negative so we need a

t xx

N

stability region that contains a segment of the negative real axis Thus the midp oint rule

will b e unstable On the other hand the Euler formula whose stability region was drawn in

Figure and again in Figure leads to the stability restriction h k ie

h

k

On an innite grid this is stricter than the stability restriction for the nite dierence

forward Euler formula considered in Example The cusp ed curve in Figure is

times deep er than the dashed one By Theorem exactly the same restriction is

also valid for a p erio dic grid

h

k

N

As another example the answer to Exercise a can be converted to the exact

stability b ound for the rdorder AdamsBashforth discretization of u u

t xx

In general sp ectral metho ds on a p erio dic grid tend to have stability restrictions that

are stricter by a constant factor than their nite dierence counterparts This is opp osite

to what the CFL condition might suggest the numerical domain of dep endence of a sp ectral

metho d is unb ounded so there is no CFL stability limit The constant factor is usually

not much of a problem in practice for sp ectral metho ds p ermit larger values of h than nite

dierence metho ds in the rst place b ecause of their high order of spatial accuracy In other

words relatively small time steps k are needed anyway to avoid large timediscretization

errors

EXERCISES

A simple sp ectral calculation

Write a program to solve u u on with periodic b oundary conditions by the

t x

pseudosp ectral metho d The program should use the midp oint time integration formula

and spatial dierentiation obtained from the program DERIV of Exercise

a Run the program up to t with k h N and N and initial data

 

v f x v f x k with b oth

cos x for jxj mo d

f x cos x and f x



otherwise

List the four errors you obtain at t Plot the computed solutions v x if

N

p ossible

b Rerun the program with k h and list the same four errors as b efore

c Explain the results of a and b What order of accuracy is observed How mightit

b e improved

Why is the ratio not Because h not h



 

STABILITY TREFETHEN 

Sp ectral calculation with ltering

n

a Mo dify the program ab ove so that instead of computing v at each step with DERIV

alone it uses DERIV followed by the program FILTER of Exercise Take k h

again and print and plot the same results as in the last problem Are the errors smaller

than they were without FILTER

b Run the program again in DERIVFILTER mo de with k h Print and plot the

same results as in c

c What is the exact theoretical stability restriction on k for the DERIVFILTER metho d

You may consider the limit N for simplicity

Note Filtering is an imp ortant idea in sp ectral metho ds but this simple linear problem is

not a go o d example of a problem where ltering is needed

Inviscid Burgers equation

u u on with p erio dic b oundary conditions by the Write a program to solve

t x

pseudosp ectral metho d The program should use forward Euler time integration formula

and spatial dierentiation implemented with the program DERIV



a Run the program up to t with k h N and initial data v

sin x Plot the computed results sup erimp osed on a single plot at times t

b Explain the results of part a as well as you can

c Optional Can you nd a way to improve the calculation

CHAPTER TREFETHEN

Chapter

Chebyshev sp ectral metho ds

Polynomial interp olation

Chebyshev dierentiation matrices

Chebyshev dierentiation by the FFT

Boundary conditions

Stability

Legendre points and other alternatives

Implicit metho ds and matrix iterations

Notes and references

CHAPTER TREFETHEN

This chapter discusses sp ectral metho ds for domains with b oundaries

The eect of b oundaries in sp ectral calculations is great for they often in

tro duce stability conditions that are both highly restrictive and dicult to

analyze Thus for a rstorder partial dierential equation solved on an N

point spatial grid by an explicit timeintegration formula a sp ectral metho d

typically requires k O N for stability in contrast to k O N for

nite dierences For a secondorder equation the disparityworsens to O N

vs O N To make matters worse the matrices involved are usually non

normal and often very far from normal so they are dicult to analyze as well

as troublesome in practice

Sp ectral metho ds on b ounded domains typically employ grids consisting

of zeros or extrema of Chebyshev p olynomials Chebyshev p oints zeros or

extrema of Legendre p olynomials Legendre points or some other set of

points related to a family or orthogonal p olynomials Chebyshev grids have

the advantage that the FFT is available for an O N log N implementation

of the dierentiation pro cess and they also have slightadvantages connected

their ability to approximate functions Legendre grids havevarious theoretical

and practical advantages b ecause of their connection with Gauss quadrature

Atthispoint one cannot saywhichchoice will win in the long run but in this

book in keeping with out emphasis on Fourier analysis most of the discussion

is of Chebyshev grids

Since explicit sp ectral metho ds are sometimes troublesome implicit sp ec

tral calculations are increasingly p opular Sp ectral dierentiation matrices are

dense and illconditioned however so solving the asso ciated systems of equa

tions is not a trivial matter even in one space dimension Currently p opular

metho ds for solving these systems include preconditioned iterative metho ds

and multigrid metho ds These techniques are discussed briey in x

POLYNOMIAL INTERPOLATION TREFETHEN

Polynomial interp olation

Sp ectral metho ds arise from the fundamental problem of approximation

of a function by interp olation on an interval Multidimensional domains of a

rectilinear shap e are treated as pro ducts of simple intervals and more compli

cated geometries are sometimes divided into rectilinear pieces In this section

therefore we restrict our attention to the fundamental interval The

question to b e considered is what kinds of interp olants in what sets of p oints

are likely to b e eective

Let N be an integer even or odd and let x x or sometimes

N

x x be a set of distinct points in For deniteness let the num

N

b ering b e in reverse order

x x x x

N N

The following are some grids that are often considered

j

Equispaced p oints x j N

j

N

j

Chebyshev zero p oints x cos j N

j

N

j

Chebyshev extreme p oints x cos j N

j

N

Legendre zero p oints x j th zero of P j N

j N

Legendre extreme p oints x j th extremum of P j N

j N

where P is the Legendre p olynomial of degree N Chebyshev zeros and ex

N

treme p oints can also b e describ ed as zeros and extrema of Chebyshev p olyno

mials T more on these in x Chebyshev and Legendre zero p oints are also

N

called GaussChebyshev and GaussLegendre p oints resp ectively and Cheby

shev and Legendre extreme p oints are also called GaussLobattoChebyshev

These names originate in and GaussLobattoLegendre p oints resp ectively

the eld of numerical quadrature

Such sub division metho ds havebeendevelop ed indep endently by I Babushka and colleagues for

structures problems who call them p nite element metho ds and byAPatera and colleagues

for uids problems who call them sp ectral element metho ds

POLYNOMIAL INTERPOLATION TREFETHEN

It is easy to remember how Chebyshev points are dened they are the

pro jections onto the interval of equallyspaced points ro ots of unity

along the unit circle jz j in the complex plane

Figure Chebyshev extreme p oints N

To the eye Legendre p oints lo ok much the same although there is no

elementary geometrical denition Figure illustrates the similarity

a N

b N

Figure Legendre vs Chebyshev zeros

As N equispaced points are distributed with density

N

x Equally spaced

POLYNOMIAL INTERPOLATION TREFETHEN

and Legendre or Chebyshev pointseither zeros or extremahave density

N

p

Legendre Chebyshev x

x

Indeed the density function applies to point sets asso ciated with any

Jacobi p olynomials of which Legendre and Chebyshev p olynomials are sp ecial

cases

Why is it a good idea to base sp ectral metho ds up on Chebyshev Legen

dre and other irregular grids We shall answer this question by addressing

a second more fundamental question why is it a go o d idea to interp olate a

function f x dened on by a p olynomial p x rather than a trigono

N

metric p olynomial and why is it a go o d idea to use Chebyshev or Legendre

points rather than equally spaced points

The remainder of this section is just a sketch details to b e supplied later

PHENOMENA

Trigonometric interp olation in equispaced points suers from the Gibbs

phenomenon due to Michelson and Gibbs at the turn of the twentieth cen

tury kf p k O as N even if f is analytic One can try to get

N

around the Gibbs phenomenon by various tricks such as doubling the domain

and reecting but the price is high

Polynomial interp olation in equally spaced p oints suers from the Runge

N

phenomenon due to Meray and Runge Figure kf p k O

N

much worse

Polynomial interp olation in Legendre or Chebyshev points kf p k

N

N

O constant if f is analytic for some constant greater than Even if

f is quite rough the errors will still go to zero provided f is say Lipschitz

continuous

POLYNOMIAL INTERPOLATION TREFETHEN

Figure The Runge phenomenon

POLYNOMIAL INTERPOLATION TREFETHEN

FIRST EXPLANATIONEQUIPOTENTIAL CURVES

Think of the limiting p oint distribution x ab ove as a charge density

distribution a charge at p osition x is asso ciated with a p otential log jz xj

Lo ok at the equip otential curves of the resulting p otential function z

R

xlogjz xj dx

CONVERGENCE OF POLYNOMIAL INTERPOLANTS

Theorem

In general kf p k as N in the largest region b ounded by an

N

equip otential curve in which f is analytic In particular

For Chebyshev or Legendre p oints or any other typ e of GaussJacobi p oints

convergence is guaranteed if f is analytic on

For equally spaced points convergence is guaranteed if f is analytic in a

particular lensshap ed region containing Figure

Figure Equip otential curves

POLYNOMIAL INTERPOLATION TREFETHEN

SECOND EXPLANATIONLEBESGUE CONSTANTS

Denition of Leb esgue constant

kI k

N N

where I is the interp olation op erator I f p A small Leb esgue constant

N N N

means that the interp olation pro cess is not much worse than b est approxima

tion

k kf p k kf p

N N N

where p is the best minimax equiripple approximation

N

LEBESGUE CONSTANTS

Theorem

N

Equispaced points e N log N

N

p

N Legendre points const

N

Chebyshev points const log N

N

THIRD EXPLANATIONNUMBER OF POINTS PER WAVELENGTH

Consider approximation of say f x cos N x as N Thus f

N N

changes but the number of points per wavelength remains constant Will the

error kf p k go to zero The answer to this question tells us something

N N

ab out the abilityofvarious kinds of sp ectral metho ds to resolve data

POINTS PER WAVELENGTH

Theorem

Equispaced p oints convergence if there are at least points p er wavelength

Chebyshev p oints convergence if there are at least points p er wavelength

on average

We have to say on average b ecause the grid is nonuniform In fact it

is times less dense in the middle than the equally spaced grid with the

same number of points N see and Thus the second part of

the theorem says that we need at least points per wavelength in the center

of the gridthe familiar Nyquist limit See Figure The rst part of

the theorem is mathematically valid but of little value in practice b ecause of rounding errors

CHEBYSHEV DIFFERENTIATION MATRICES TREFETHEN

a Equally spaced points

b Chebyshev p oints

Figure Error as a function of N in interp olation of cos N x

with hence the number of p oints per wavelength held xed

CHEBYSHEV DIFFERENTIATION MATRICES TREFETHEN

Chebyshev dierentiation matrices

Just a sketch

From now on Chebyshev points means Chebyshev extreme points

Multiplication by the rstorder Chebyshev dierentiation matrix D

N

transforms a vector of data at the Chebyshev points into approximate deriva

tives at those points

v w

D

N

v w

N N

usual the implicit denition of D is as follows As

N

CHEBYSHEV SPECTRAL DIFFERENTIATION BY POLYNOMIAL INTERPOLA

TION

Interp olate v by a p olynomial q xq x

N

Dierentiate the interp olantat the grid p oints x

j

w D v q x

j N j j

Higherorder dierentiation matrices are dened analogously From this

denition it is easy to work out the entries of D in sp ecial cases For N

N

D x

For N

D x

CHEBYSHEV DIFFERENTIATION MATRICES TREFETHEN

For N

D x

These three examples illustrate an imp ortant fact mentioned in the intro duc

tion to this chapter Chebyshev sp ectral dierentiation matrices are in general

not symmetric or skewsymmetric A more general statement is that they are

not normal This is why stability analysis is dicult for sp ectral metho ds

The reason they are not normal is that unlikenite dierence dierentiation

sp ectral dierentiation is not a translationinvariant pro cess but dep ends in

stead on the same global interp olantatallpoints x

j

The general formula for D is as follows First dene

N

for i or N

c

i

for i N

and of course analogously for c Then

j

CHEBYSHEV SPECTRAL DIFFERENTIATION

Theorem Let N be anyinteger The rstorder sp ectral dieren

tiation matrix D has entries

N

N N

D D

N N NN

x

j

D for j N

N jj

x

j

ij

c

i

for i j D

N ij

c x x

j i j

Analogous formulas for D can b e found in Peyret Ehrenstein

N

Peyret ref and in Zang Streett and Hussaini ICASE Rep ort

See also Canuto Hussaini Quarteroni Zang

T T

Recall that a normal matrix A is one that satises AA A A Equivalently A p ossesses an

n n

orthogonal set of eigenvectors which implies many desirable prop erties suchas A kA k

n

kAk for any n

CHEBYSHEV DIFFERENTIATION MATRICES TREFETHEN

A note of caution D is rarely used in exactly the form describ ed in

N

Theorem for b oundary conditions will mo dify it slightly and these dep end

on the problem

EXERCISES

n

Prove that for any N D is nilp otent D for a suciently high integer n

N N

CHEBYSHEV DIFFERENTIATION BY THE FFT TREFETHEN

Chebyshev dierentiation by the FFT

Polynomial interp olation in Chebyshev p oints is equivalent to trigonometric interp o

lation in equally spaced p oints and hence can b e carried out by the FFT The algorithm

describ ed b elow has the optimal order O N log N but wedonotworry ab out achieving

the optimal constant factor For more practical discussions see App endix B of the b o ok by

Canuto et al and also PNSwarztraub er Symmetric FFTs Math Comp

Valuable additional references are the b o ok The by Rivlin

and Chapter of P Henrici Applied and Computational Complex Analysis

Consider three indep endent variables R x and z S where S is the

complex unit circle fz jz j g They are related as follows

i

z e x Rez z z cos



which implies

p

dx



sin x

d

See Figure Note that there are two conjugate values z S for each x and

an innite numb er of p ossible choices of z

*o

*o

x

*o

z

Figure z xand

optimal that is so far as anyone knows as of

CHEBYSHEV DIFFERENTIATION BY THE FFT TREFETHEN

n

In generalization of the fact that the real part of z is x the real part of z n

is T x the Chebyshev p olynomial of degree n This statement can be taken as a

n

denitionofChebyshev p olynomials

n n n

T xRez z z cos n

n



where x and z and are as always implicitly related by It is clear that

denes T xtobesome function of x but it is not obvious that the function is a p olynomial

n

However a calculation of the rst few cases makes it clear what is going on

 

z z T x





T x z z x



   

T x z z z z x



 

  





z z z z z z x x T x



  

In general the Chebyshev p olynomials are related by the threeterm recurrence relation

n n

T x z z

n



n n n n

z z z z z z

 

xT x T x

n n

By and the derivativeof T xis

n

n sin n d



T xn sin n

n

dx sin

n

Thus just as x z and are equivalent so are T x z and cos n By taking linear

n

combinations we obtain three equivalent kinds of p olynomials A trigonometric p olyno

mial q of degree N isa p erio dic sum of complex exp onentials in or equivalently

sines and cosines Assuming that q isaneven function of it can b e written

N N

X X

in in

a e e a cos n q

n n



n n

A Laurent p olynomial q z of degree N is a sum of negative and p ositivepowers of z up

to degree N Assuming q z q z for z S it can b e written

N

X

n n

q z a z z

n



n

An algebraic p olynomial q x of degree N is a p olynomial in x of the usual kind and we

can express it as a linear combination of Chebyshev p olynomials

N

X

q x a T x

n n

n

Equivalently the Chebyshev p olynomials can b e dened as a system of p olynomials orthogonal on

 

with resp ect to the weight function x

CHEBYSHEV DIFFERENTIATION BY THE FFT TREFETHEN

The use of the same co ecients a in is no accident for all three of the

n

p olynomials ab ove are identical

q q z q x

where again x and z and are implicitly related by For this reason wehopetobe

forgiven the sloppy use of the same letter q in all three cases

Finally for anyinteger N we dene regular grids in the three variables as follows

j

i

j

x Rez z e z z cos

j j j j j j

j



N

for j N The p oints fx g and fz g were illustrated already in Figure And now

j j

we are ready to state the algorithm for Chebyshev dierentiation by the FFT

CHEBYSHEV DIFFERENTIATION BY THE FFT TREFETHEN

ALGORITHM FOR CHEBYSHEV DIFFERENTIATION

Given data fv g dened at the Chebyshev p oints fx g j N think of the same

j j

data as b eing dened at the equally spaced p oints f g in

j

FFT Find the co ecients fa g of the trigonometric p olynomial

n

N

X

q a cos n

n

n

that interp olates fv g at f g

j j

FFT Compute the derivative

N

X

dq

na sin n

n

d

n

Change variables to obtain the derivative with resp ect to x

N N

X X

dq na sin n na sin n d dq

n n

p



dx d dx sin

x

n n

At x ie LHopitals rule gives the sp ecial values

N

X

dq

n 

n a

n

dx

n

Evaluate the result at the Chebyshev p oints

dq

w x

j j

dx

Note that by equation can b e interpreted as a linear combination of

Chebyshev p olynomials and by equation is the corresp onding linear com

bination of derivatives But of course the algorithmic content of the description ab ove

relates to the variable for in Steps and wehave p erformed Fourier sp ectral dierenti

ation exactly as in x discrete Fourier transform multiply by i inverse discrete Fourier

transform Only the use of sines and cosines rather than complex exp onentials and of n

instead of has disguised the pro cess somewhat

orofChebyshev p olynomials U xofthe second kind n

CHEBYSHEV DIFFERENTIATION BY THE FFT TREFETHEN

EXERCISES

Fourier and Chebyshev sp ectral dierentiation

Write four brief elegant Matlab programs for rstorder sp ectral dierentiation

FDERIVM CDERIVM construct dierentiation matrices

FDERIV CDERIV dierentiate via FFT

In the Fourier case there are N equally spaced points x x N even in

N N

and no b oundary conditions In the Chebyshev case there are N Chebyshev p oints

x x in N arbitrary with a zero b oundary condition at x The eect of

N

this b oundary condition is that one removes the rst row and rst column from D leading

N

to a square matrix of dimension N instead of N

You do not havetoworry ab out computational eciency such as using an FFT of length

rather than N in the Chebyshev case but you are welcome to worry ab out it if you N

like

Exp erimentwithyour programs to makesuretheydierentiate successfully Of course the

matrices can b e used to check the FFT programs

a Turn in a plot showing the function ux cosx and its derivative computed by

FDERIV for N Discuss the results

b Turn in a plot showing the function uxcosx and its derivative computed by

CDERIV again for N Discuss the results

c Plot the eigenvalues of D for Fourier and Chebyshev sp ectral dierentiation with

N

N

STABILITY TREFETHEN

Stability

This section is not yet written What follows is a copy of a pap er of mine from K

W Morton and M J Baines eds Numerical Metho ds for Fluid Dynamics I I I Clarendon

Press Oxford

Because of stability problems like those describ ed in this pap er more and more atten

tion is currently b eing devoted to implicit timestepping metho ds for sp ectral computations

The asso ciated linear algebra problems are generally solved by preconditioned matrix iter

ations sometimes including a multigrid iteration

This pap er was written b efore I was using the terminology of pseudosp ectra Iwould

now summarize Section of this pap er bysaying that although the sp ectrum of the Legendre



sp ectral dierentiation matrix is of size N asN the pseudosp ectra are of size N

for any The connection of pseudosp ectra with stability of the metho d of lines was discussed in Sections

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

STABILITY TREFETHEN

SOME REVIEW PROBLEMS TREFETHEN

Some review problems

EXERCISES

TRUE or FALSE Give each answer together with at most two or three

sentences of explanation The b est p ossible explanation is a pro of a counterexample

or the citation of a theorem in the text from which the answer follows If you cant

do quite that well try at least to give a convincing reason why the answer you have

chosen is the rightone In some cases a wellthoughtout sketch will suce

a The Fourier transform of f x expx has compact supp ort

b When you multiply a matrix by a vector on the right ie Ax the result is a

bination of the columns of that matrix linear com

c If an ODE initialvalue problem with a smo oth solution is solved by the fourth

order AdamsBashforth formula with step size k and the missing starting values

v v v are obtained by taking Euler steps with some step size k then in

general we will need k O k tomaintain overall fourthorder accuracy

d If a consistent nite dierence mo del of a wellp osed linear initialvalue problem

violates the CFL condition it must b e unstable

e If you Fourier transform a function u L four times in a row you end up with

u again times a constant factor



f If the function f xx x is interp olated by a p olynomial q x

N

in N equally spaced p oints of then kf q k as N



N

x x

g e O xe asx

h If a stable nitedierence approximation to u u with real co ecients has

t x

order of accuracy then the formula must b e dissipative

i If

A

n

then kA k C n for some constant C

j If the equation u A u is solved by the fourthorder AdamsMoulton

t

formula where ux t is a vector and A is the matrix ab ove then k is

a suciently small time step to ensure timestability

k Let u u on with p erio dic b oundary conditions b e solved byFourier

t xx

pseudosp ectral dierentiation in x coupled with a fourthorder RungeKutta

SOME REVIEW PROBLEMS TREFETHEN

formula in t For N k is a suciently small time step to ensure

timestability

l The ODE initialvalue problem u f u tcos u u t is

t

wellp osed

m In exact arithmetic and with exact starting values the numerical approxima

tions computed by the linear multistep formula

n

n n n n n n

v v v k f f f v

are guaranteed to converge to the unique solution of a wellp osed initialvalue

problem in the limit k

n If computers did not make rounding errors wewould not need to study stability

o The solution at time t to u u u x R initial data ux f x is

t x xx

the same as what you would get by rst diusing the data f x according to the

equation u u then translating the result leftward by one unit according to

t xx

the equation u u

t x

p The discrete Fourier transform of a threedimensional p erio dic set of data on an

N log N op erations N N N grid can b e computed on a serial computer in O

q The addition of numerical dissipation may sometimes increase the stability limit

of a nite dierence formula without aecting the order of accuracy

r For a nondissipative semidiscrete nitedierence mo del ie discrete space but

continuous time phase velo cityaswell as group velo cityisawelldened quan

tity

n

n

is a stable lefthand b oundary condition for use with the leap frog s v v

mo del of u u with kh

t x

t If a nite dierence mo del of a partial dierential equation is stable with kh

for some then it is stable with kh for any

u To solve the system of equations that results from a standard secondorder

discretization of Laplaces equation on an N N N grid in three dimensions

by the obvious metho d of banded Gaussian elimination without any clever

tricks requires N op erations on a serial computer

v If ux t is a solution to u iu for x R then the norm kutk is inde

t xx

pendentoft

w In a metho d of lines discretization of a wellp osed linear IVP having the ap

propriate eigenvalues t in the appropriate stability region is sucient but not

necessary for Laxstability

x Supp ose a signal thats bandlimited to frequencies in the range kHz kHz

is sampled times a second ie fast enough to resolve frequencies in the

range kHz kHz Then although some aliasing will o ccur the information

in the range kHz kHz remains uncorrupted

TWO FINAL PROBLEMS TREFETHEN

Two nal problems

EXERCISES

Equip otential curves Write a short and elegant Matlab program to plot

equip otential curves in the plane corresp onding to avector of p ointcharges inter

P



log jz x j p olation p oints x x Your program should simply sample N

j

N

on a grid then pro duce a contour plot of the result See meshdom and contour

Turn in b eautiful plots corresp onding to a equispaced p oints b Chebyshev

yshev p oints By all means play around points c equispaced p oints d Cheb

with D graphics convergence and divergence of asso ciated interp olation pro cesses

or other amusements if youre in the mo o d

Fun with Chebyshev sp ectral metho ds The starting point of this problem

is the Chebyshev dierentiation matrix of Exercise It will be easiest to use

a program like CDERIVM from that exercise whichworks with an explicit matrix

rather than the FFT Be careful with b oundary conditions you will want to square

the N N matrix rst b efore stripping o anyrows or columns

x

a Poisson equation in D The function ux x e satises u and

x

has second derivative u x x x e Thus it is the solution to the

b oundary value problem

x

u x x e x u

xx

Write a little Matlab program to solve bya Chebyshev sp ectral metho d and

pro duce a plot of the computed discrete solution values N discrete p oints in

sup erimp osed up on exact solution a curve Turn in the plot for N

and a table of the errors u u for N What can you

exact

computed

say ab out the rate of convergence

y cosx y b Poisson equation in D Similarly the function ux y x

is the solution to the b oundary value problem

u u sorry illegible x y ux uy

xx yy

Write a Matlab program to solve byaChebyshev sp ectral metho d involving

a grid of N interior points You may nd that the Matlab command

KRON comes in handy for this purp ose You dont have to pro duce a plot of

the computed solution but do turn in a table of u u for

exact

computed

N How do es the rate of convergence lo ok

c Heat equation in D BacktoD now Supp ose you have the problem

x

u u ut ux x e

t xx

TWO FINAL PROBLEMS TREFETHEN

At what time t do es max ux t rst fall b elow Figure out the answer

x

c

to at least digits of relative precision Then describ e what you would do if I

asked for digits

GENERAL REFERENCES TREFETHEN REF

General References

D A Anderson J C Tannehill and R H Pletcher Computational Fluid Mechanics

and Heat Transfer McGrawHill

J PBoyd Chebyshev Fourier Sp ectral Metho ds SpringerVerlag

J C Butcher The Numerical Analysis of Ordinary Dierential Equations Wiley

C Canuto M Y Hussaini A Quarteroni and T A Zang Sp ectral Metho ds in

Fluid Dynamics Springer

R Courant and D Hilb ert Metho ds of Mathematical Physics v I and II Wiley

Interscience

C W Gear Numerical Initial Value Problems in Ordinary Dierential Equations

PrenticeHall

W Hackbusch Multigrid Metho ds and Applications SpringerVerlag

W Hackbusch and U Trottenberg Multigrid Metho ds Lect Notes in Math v

SpringerVerlag

G H Golub and C F Van Loan Matrix Computations Johns Hopkins Univ Press

E Hairer S P Nrsett and G Wanner Solving Ordinary Dierential Equations I

SpringerVerlag

F John Partial Dierential Equations SpringerVerlag

H Kreiss and J Oliger Metho ds for the Approximate Solution of Time Dep endent

Problems Global Atm Res Prog Publ GARP

J D Lambert Computational Metho ds in Ordinary Dierential Equations Wiley

R J LeVeque Numerical Metho ds for Conservation Laws BirkhauserVerlag to

app ear

A R Mitchell and D F Griths Computational Metho ds in Partial Dierential

Equations Wiley

P M Morse and H Feshbach Metho ds of Mathematical Physics v I and II

McGrawHill

R Peyret and T D Taylor Computational Metho ds for Fluid Flow Springer

Verlag

R D Richtm yer and K W Morton Dierence Metho ds for InitialValue Problems

nd ed WileyInterscience

G A So d Numerical Metho ds in Fluid Dynamics Cambridge Univ Press

JOURNALS TREFETHEN REF

Journals

ACM Transactions on Mathematical Software

AIAA Journal

BIT

Communications on Pure and Applied Mathematics

IMA Journal of Numerical Analysis

Journal of Computational Physics

Journal of Computational and Applied Mathematics

Journal of Scientic Computing

Mathematics of Computation

Numerische Mathematik

SIAM Journal on Numerical Analysis

SIAM Journal on Scientic and Statistical Computing SIAM Review

CHAPTER REFERENCES TREFETHEN REF

References for Chapter

The following list comprises most of the the standard b o oks on the numerical

solution of ordinary dierential equations Henrici Stetter and Dekker Verwer

represent the theoretical end of the sp ectrum Shampine Gordon the practical

The pamphlet by Sand and sterbyisadelightful comp endium of plots of stability

regions Co ddington Levinson and Bender Orszag present the mathematical

theory of ordinary dierential equations from the pure and applied p oints of view

resp ectively Lamb ert and Gear are p erhaps the b est general intro ductions to numer

ical metho ds for odes while Butchers b o ok is a more detailed treatise containing

pages of references up to The b o oks by Keller and by Ascher Mattheij

Russell treat b oundaryvalue ode problems not covered in this b o ok For p er

cannot do b etter than Hairer sp ective historical notes and sense of humor one

Nrsett and Wanner

U M Ascher R M M Mattheij R D Russell Numerical Solution of Boundary

Value Problems for Ordinary Dierential Equations PrenticeHall

C M Bender and S A Orszag Advanced Mathematical Metho ds for Scientists and

Engineers McGrawHill

J C Butcher The Numerical Analysis of Ordinary Dierential Equations Wiley

G D Byrne and A C Hindmarsh Sti ODE solvers A review of current and

coming attractions J Comp Phys

E A Co ddington and N Levinson Theory of Ordinary Dierential Equations

McGrawHill

K Dekker and J G Verwer Stability of RungeKutta Metho ds for Sti Nonlinear

Dierential Equations NorthHolland

C W Gear Numerical Initial Value Problems in Ordinary Dierential Equations

PrenticeHall

E Hairer S P Nrsett and G Wanner Solving Ordinary Dierential Equations I

Springer

P Henrici Discrete Variable Metho ds in Ordinary Dierential Equations Wiley

H B Keller Numerical Solution of Two Point Boundary Value Problems SIAM

J D Lambert Computational Metho ds in Ordinary Dierential Equations Wiley

CHAPTER REFERENCES TREFETHEN REF

L Lapidus and J H Seinfeld Numerical Solution of Ordinary Dierential Equa

tions Academic Press

J Sand and O sterby Regions of Absolute Stability Rep PB Comp Sci

Dept Aarhus Univ Denmark

L F Shampine and C W Gear A users view of solving sti ordinary dierential

equations SIAM Review

L F Shampine and M K Gordon Computer Solution of Ordinary Dierential

Equations W H Freeman

H J Stetter Analysis of Discretization Metho ds for Ordinary Dierential Equa

tions SpringerVerlag

TREFETHEN  CHAPTER  REFERENCES

References for Chapter

Mathematical treatments of Fourier analysis app ear in many b o oks of analy

sis partial dierential equations and applications as well as in books devoted to

Fourier analysis itself Fourier analysis is also discussed in b o oks in many other elds

esp ecially digital signal pro cessing The following references include a sampling of

various kinds For rigorous theory the b o oks by Katznelson and bySteinandWeiss

are both excellent and readable the classic is by Zygmund For discrete FFTs as

they app ear in complex analysis the b est source is Henrici

Bracewell

Brigham

P L Butzer and R J Nessel Fourier Analysis and Approximation

Churchill

H Dym and H P McKean Fourier Series and Integrals Academic Press

G Edwards Fourier Series

P Henrici Applied and Computational Complex Analysis I I I Wileychap

Y Katznelson An Intro duction to Harmonic Analysis Dover

A V Opp enheim and R W Schafer Digital Signal Pro cessing PrenticeHall

W H Press et al Numerical Recip es Cambridge Univ Press

W Rudin Real and Complex Analysis McGrawHill

E M Stein and G Weiss Introuction to Fourier Analysis on Euclidean Spaces

Princeton Univ Press

I N Sneddon Fourier Transforms McGrawHill

PNSwarztraub er Symmetric FFTs Math Comput

A Zygmund Trigonometric Series Cambridge University Press

TREFETHEN  CHAPTER  REFERENCES

References for Chapter

P Brenner V Thomee and L Wahlbin Besov Spaces and Applications to Dier

ence Metho ds for Initial Value Problems Springer Lect Notes in Math v

R C Y Chin and G W Hedstrom A disp ersion analysis for dierence schemes

tables of generalized Airy functions Math Comp

M B Giles and W T Thompkins Jr Propagation and stabilityof wavelike so

lutions of nite dierence equations with variable co ecients J Comp Phys

J Lighthill Waves in Fluids Cambridge University Press

Shokin b o ok on mo died equations etc

L N Trefethen Group velo city in nite dierence schemes SIAM Review

L N Trefethen Disp ersion dissipation and stability in D F Griths and G

A Watson eds Numerical Analysis Longman

R Vichnevetsky and J B Bowles Fourier Analysis of Numerical Approximations

of Hyp erb olic Equations SIAM

R F Warming and B J Hyett The mo died equation approach to the stability

and accuracy analysis of nitedierence metho ds J Comp Phys

G B Whitham Linear and Nonlinear Waves WileyInterscience

TREFETHEN  CHAPTER  REFERENCES

References for Chapter

Four basic pap ers on wellp osedness and stability for hyp erb olic IBVPs

HO Kreiss Initial b oundary value problems for hyp erb olic systems Comm Pure

and Appl Math Wellp osedness for hyp erb olic IBVPs

R L Higdon Initialb oundary value problems for linear hyp erb olic systems

SIAM Review Wave propagation interpretation of ab ove

B Gustafsson HO Kreiss and A Sundstrom Stability theory of dierence

approximations for initial b oundary value problems II Math Comput

Stability for hyp erb olic nite dierence mo dels

L N Trefethen Instability of dierence mo dels for hyp erb olic initial b oundary

value problems Commun Pure and Appl Math Wave

propagation interpretation of ab ov e

Additional references

B Engquist and A Ma jda Absorbing b oundary conditions for the numerical sim

ulation of waves Math Comp

M Golb erg and E Tadmor Convenient stability criteria for dierence approxi

mations of hyp erb olic initialb oundary value problems II Math Comp

G A So d Numerical Metho ds in Fluid Dynamics Cambridge University Press

TREFETHEN  CHAPTER  REFERENCES

References for Chapter

C Canuto M Y Hussaini A Quarteroni and T A Zang Sp ectral Metho ds in

Fluid Dynamics SpringerVerlag

B Fornberg On a Fourier metho d for the integration of hyp erb olic equations

SIAM J Numer Anal

D Gottlieb and S A Orszag Numerical Analysis of Sp ectral Metho ds Theory and

Applications SIAM Philadelphia

R G Voigt D Gottlieb and M Y Hussaini Sp ectral Metho ds for Partial Dier

ential Equations SIAM Philadelphia

TREFETHEN  CHAPTER  REFERENCES

References for Chapter

C Canuto M Y Hussaini A Quarteroni and T A Zang Sp ectral Metho ds in

Fluid Dynamics SpringerVerlag

D Gottlieb and S A Orszag Numerical Analysis of Sp ectral Metho ds Theory and

Applications SIAM Philadelphia

T J Rivlin The Chebyshev Polynomials Wiley

R G Voigt D Gottlieb and M Y Hussaini Sp ectral Metho ds for Partial Dier

ential Equations SIAM Philadelphia

TREFETHEN  CHAPTER  REFERENCES

References for Chapter

C Canuto M Y Hussaini A Quarteroni and T A Zang Sp ectral Metho ds in

Fluid Dynamics SpringerVerlag

B Fornberg On a Fourier metho d for the integration of hyp erb olic equations

SIAM J Numer Anal

D Gottlieb and S A Orszag Numerical Analysis of Sp ectral Metho ds Theory and

Applications SIAM Philadelphia

R G Voigt D Gottlieb and M Y Hussaini Sp ectral Metho ds for Partial Dier

ential Equations SIAM Philadelphia