CS321-001

Introduction to Numerical Methods

Lecture 3

Interpolation and Numerical Differentiation

Professor Jun Zhang

Department of Computer Science University of Kentucky Lexington, KY 40506-0633 Interpolation

Given a set of discrete values, how can we estimate other values between these data

The method that we will use is called .

We assume the data we had are from the evaluation of a smooth function. We may be able to use a polynomial p(x) to approximate this function, at least locally.

A condition: the polynomial p(x) takes the given values at the given points (nodes), i.e., p(xi) = yi with 0 ≤ i ≤ n. The polynomial is said to interpolate the table, since we do not know the function.

2 Polynomial Interpolation

Note that all the points are passed through by the curve 3 Polynomial Interpolation

We do not know the original function, the interpolation may not be accurate 4 Order of Interpolating Polynomial

A polynomial of degree 0, a constant function, interpolates one set of data

If we have two sets of data, we can have an interpolating polynomial of degree 1, a linear function  x  x1   x  x0  p(x)    y0    y1  x0  x1   x1  x0 

 y1  y0   y0   (x  x0 )  x1  x0  Review carefully if the interpolation condition is satisfied

Interpolating can be written in several forms, the most well known ones are the Lagrange form and form. Each has some advantages

5 Lagrange Form

For a set of fixed nodes x0, x1, …, xn, the cardinal functions, l0, l1,…, ln, are defined as 0 if i  j li (x j )   ij   1 if i  j

We can interpolate any function f(x) by the Lagrange form of the interpolating polynomial of degree ≤ n n pn (x)   li (x) f (xi ) i0

Note that li(x) is of order n, so pn(x) is of order ≤ n, and n pn (x j )  li (x j )f(xi )  l j (x j )f(x j )  f(x j ) i0 The (point exact) interpolation condition is satisfied

6 Cardinal Functions

The cardinal function is n  x  x   j  li (x)     (0  i  n) ji, j0  xi  x j  What it looks like  x  x0  x  x1  li (x)      xi  x0  xi  x1 

 x  xi1  x  xi1      xi  xi1  xi  xi1 

 x  xn     xi  xn 

Note that li (x j )  0, for i  j and li (xi )  1 7 An Example

Find the interpolating polynomial for this table

Lagrange form (x  1)(x  1) l (x)   (x  1)(x  1) 0 (0  1)(0  1) (x  0)(x  1) 1 l (x)   x(x  1) 1 (1 0)(1 1) 2 (x  0)(x  1) 1 l (x)   x(x  1) 2 (1  0)(1 1) 2 The interpolating polynomial is 3 15 p (x)  5(x  1)(x  1)  x(x  1)  x(x  1) 2 2 2

8 Five Lagrange Basis Polynomials

These are the 5 Lagrange basis polynomials for N=5 9 Representation by Lagrange Polynomials

Five weighted polynomials and their sum (red line) for a set of 5 random samples (red points) 10 Step by Step Construction

For any table of data, we can construct a Lagrange interpolating polynomial. Its evaluation is a little bit costly, but we can always do that. The existence of the interpolating polynomial is guaranteed

Can we construct the interpolating polynomial step by step, or if we discover some new data, can we add those data to make the interpolation more accurate?

We can use Newton form of the interpolating polynomial

Let pk(x) be an interpolating polynomial for the data set {(xi,yi)} with 0 ≤ i ≤ k such that pk(xi) = yi

11 Newton Form - Cont.

We want to add another data (xk+1, yk+1) to have a new interpolating polynomial pk+1(x) such that pk+1(xi) = yi for 0 ≤ i ≤ (k + 1). Let

pk1 (x)  pk (x)  c(x  x0 )(x  x1 )(x  xk ) where c is an undetermined constant

Sincepk1(xk1 )  yk1 , we have

pk (xk1 )  c(xk1  x0 )(xk1  x1 )

(xk1  xk )  yk1

We can solve this equation for c, with the condition that x0, x1, …, xk+1 are all distinct y  p (x ) c  k1 k k1 (xk1  x0 )(xk1  x1 )(xk1  xk )

12 Newton Polynomial Interpolation

13 Uniqueness of Polynomial

Is the interpolating polynomial unique?

If p and q are interpolating polynomials for the data set {(xi,yi)} for 0 ≤ i ≤ n such that p(xi) = q(xi) = yi

Then the polynomial r(x) = p(x) – q(x) of degree at most n is zero at x0, x1, …, xn. Note that a polynomial of degree n can have at most n roots, we must have r(x) = 0, or p – q = 0. Hence p = q

The interpolating polynomial is unique

It may be written in different forms

14 An Example

Find the interpolating polynomial for this table

Lagrange form (x  1)(x  1) l (x)   (x  1)(x  1) 0 (0  1)(0  1) (x  0)(x  1) 1 l (x)   x(x  1) 1 (1 0)(1 1) 2 (x  0)(x  1) 1 l (x)   x(x  1) 2 (1  0)(1 1) 2 The interpolating polynomial is 3 15 p (x)  5(x  1)(x  1)  x(x  1)  x(x  1) 2 2 2

15 Newton Form

The zeroth order polynomial is

p0 (x)  5 Let the 1st order interpolating polynomial be

p1 (x)  p0  c(x  x0 )  5  c(x  0)

We want p1(x1) = -3, hence -5 + c(1 – 0) = -3, we have c = 2, it follows that

p1 (x)  5  2x Let the 2nd order interpolating polynomial be

p2 (x)  p1(x)  c(x  x0 )(x  x1 )

Put p2(-1) = -15, i.e., -5+2(-1) + c(-1 – 0)(-1 – 1) = -15. We have c = -4. The Newton form of the interpolating polynomial is

p2 (x)  5  2x  4x(x  1)

16 Nested Form

For easy programming and efficient computation, we can write Newton form of the interpolating polynomial in nested form

p(x)  a0  a1[(x  x0 )] a2[(x  x0 )(x  x1 )]  a3[(x  x0 )(x  x1 )(x  x2 )]  an[(x  x0 )(x  x1 )(x  xn1 )] Or, using standard product notations as n  i1  p(x)  a0  ai   (x  x j ) i1  j1  Using successive factorization, the nested form is p(x)  a0  (x  x0 )(a1  (x  x1 )(a2 

 (x  xn1 )an )))

 (((an (x  xn1 )  an1 )(x  xn2 )

 an2 ))(x  x0 )  a0 17 Computation Procedure

To evaluate p(x) for a given x, we start from the innermost parentheses, forming successively some intermediate quantities

v0  an v1  v0 (x  xn1 )  an1 v2  v1(x  xn2 )  an2  vi  vi1(x  xni )  ani  vn  vn1 (x  x0 )  a0

A pseudocode is

real array (ai )0:n , (xi )0:n integer i, n real x, v v  an for i  n  1 to 0 step -1 do v  v(x  xi )  ai end for 18 Divided Difference

The coefficients ai in Newton form of the interpolating polynomial need to be computed. A notation is introduced to facilitate such computation

ak  f[x0,x1,,xk ] which is called the divided difference of order k for f

Newton form interpolating polynomial is

pn (x)  a0  a1 (x  x0 )  a2 (x  x0 )(x  x1 )

  an (x  x0 )(x  xn1 ) Or written in a compact form n i1 pn (x)   ai (x  x j ) i0 j0 with the convention 1  (x  x j )  1 j0 19 Computing Coefficients ai

We want pn(xi) = f(xi). So we have f (x0 )  a0

f (x1)  a0  a1(x1  x0 )

f (x2 )  a0  a1(x2  x0 )  a2 (x2  x0 )(x2  x1)   The solution of this system is a0  f (x0 )

f (x1 )  a0 a1  x1  x0 f (x )  f (x )  1 0 x1  x0 The divided difference of order 1 is f (x1 )  f (x0 ) f [x0 , x1 ]  x1  x0 k Note that f[x0, x1,…, xk] is the coefficient of x in the polynomial pk of degree ≤ k 20 Computing Coefficients – Cont.

f (x2 )  a0  a1 (x2  x0 ) a2  (x2  x0 )(x2  x1 ) f (x )  f [x ] f [x , x ](x  x )  2 0 0 1 2 0 (x2  x0 )(x2  x1 )  f [x0 , x1 , x2 ] In general, we have f [x0 , x1 ,, xk ]  k1 i1 f (xk )  i0 f [x0 , x1 ,, xi ]  j0 (xk  x j ) k1  j0 (xk  x j )

Computational algorithm

•Set f[x0] = f(x0)

•For k = 1, 2,…, n, compute f[x0, x1,…, xk] using the above equation 21 Recursive Formula

The divided difference has a recursive formula

f[x0 , x1,, x k ] 

f[x1, x 2 ,, x k ] f[x0, x1,, x k1]

x k  x 0 Proof:

k f[x0, x1,…, xk] is the coefficient of x in the polynomial pk of degree ≤ k, which interpolates f at x0, x1,…, xk

k-1 f[x1, x2,…, xk] is the coefficient of x in the polynomial qk-1 of degree ≤ (k – 1), which interpolates f at x1, x2,…, xk

k-1 f[x0, x1,…, xk-1] is the coefficient of x in the polynomial pk-1of degree ≤ (k – 1), which interpolates f at x0, x1,…, xk-1

22 Recursive Formula - Proof

We have

x  xk pk (x)  qk1(x)  [qk1 (x)  pk1(x)] xk  x0 To prove this identity, it suffices to show that it holds at (k + 1) different points, since the left-hand side and the right-hand side are polynomials of degree ≤ k. Note that the left-hand side is pk(xi) = f(xi) for i = 0, 1,…, k

Check the right-hand side at point x0

x0  xk qk1 (x0 )  [qk1 (x0 )  pk1(x0 )] xk  x0

 qk1 (x0 ) [qk1(x0 )  pk1(x0 )]

 pk1(x0 )  f (x0 )

23 Formula Proof - Cont.

Check for points 1 ≤ i ≤ (k – 1), xi  xk qk1 (xi )  [qk1(xi )  pk1(xi )] xk  x0

xi  xk  f (xi )  [ f (xi )  f (xi )]  f (xi ) xk  x0

Check the right-hand side at point xk xk  xk qk1 (xk )  [qk1(xk )  pk1(xk )] xk  x0

 qk1 (xk )  f (xk ) Hence the said identity holds

We take the coefficients of xk on both sides, which yields the desired recursive formula

24 Invariance Theorem

The divided difference f[x0, x1,…, xk] is invariant under all permutations of the arguments x0, x1,…, xk

k This is because f[x0, x1,…, xk] is the coefficient of x of the polynomial pk(x) of degree ≤ k that interpolates f at x0, x1,…, xk. f[x1, x0,…, xk] is the k coefficient of x of the polynomial pk(x) of degree ≤ k that interpolates f at x1, x0,…, xk. These two polynomials are the same

The generic recursive formula is

f [xi , xi1 ,, x j1 , x j ] 

f xi1 , xi2 ,, x j  f xi , xi1 ,, x j1 

x j  xi

25 Divided Difference Table

We can construct a divided difference table for f to facilitate computation of the coefficients of the interpolating polynomial

The coefficients along the top diagonal are the ones needed to form the Newton form of the interpolating polynomial 20 Divided Difference Table

20 Divided Difference Algorithm

A pseudocode for computing divided difference is real array (aij )0:n0:n , (xi )0:n integer i, j, n for i  0 to n do ai0  f (xi ) end for for j  1 to n do for i  0 to n  j do aij  (ai1, j1  ai, j1 ) /(xi j  xi ) end for end for

This algorithm computes and stores all components of the divided difference. The coefficients of the Newton interpolating polynomial are stored in the first row of the array (aij)0:n×0:n, i.e., in a(0 : n, 0)

28 Computing the Coefficients only

If we compute divided difference only for constructing Newton interpolating polynomial, there is no need to store the unnecessary divided difference terms. (But they will be computed, used, and discarded)

real array (ai )0:n , (xi )0:n integer i, j, n for i  0 to n do

ai  f (xi ) end for for j 1 to n do for i  n to j step 1 do

ai  (ai  ai1) /(xi  xi j ) end for end for

The two algorithms assume the same computational cost

29 Memory Allocations

Here we show how the memory is occupied and updated in computing coefficients of Newton interpolating polynomial

Computation must be done backward to avoid erasing needed memory locations

30 Divided Difference Table

20 Newton Polynomial i 0 1 234 0 1 2 3 4 0 1 8 27 64

20 Inverse Interpolation

It is also possible to use a polynomial to approximate the inverse of a function y = f(x). Given a table

An interpolation polynomial n i1 p( y)  ci  ( y  y j ) i0 j0

Can be constructed such that p(yi) = xi. This interpolating polynomial is useful to find the approximate location of a root of a function f(x). E.g., there is a root in [4.0,5.0] for this table

33 Neville’s Algorithm

Neville proposed a different scheme to construct interpolation polynomial step by step. Start with zero degree polynomials Pi(x) = f(xi), we construct higher degree interpolation polynomials by the recurrence relation  x  x   i j  Sij (x)   Si, j1(x)  xi  xi j     xi  x    Si1, j1 (x)  xi  xi j 

With Si0(x) = Pi(x) = f(xi). The relation table can be written as

34 Interpolation Property

(0) Redefine constant polynomials as Pi (x)  yi for 0  i  n, we can define higher order polynomial as  x  x  P ( j) (x)  i j P ( j1) (x) i    i  xi  xi j     xi  x  ( j1)   Pi1 (x)  xi  xi j  The range of j is 1 ≤ j ≤ n and that of i is j ≤ i ≤ n

The interpolation properties of these polynomials are:

(j) The polynomials Pi defined above interpolate the original table as follows (see p. 153 for a proof) ( j) Pi (xk )  yk (0  i  j  k  i  n)

35 Higher Dimensional Interpolation

It is possible to define interpolation polynomials of several variables. The tensor-product interpolation is used on rectangular domain [a,b] × [α,β]. Select n nodes in [a,b] and define the Lagrange polynomials as n x  x j li (x)   (1  i  n). ji, j1 xi  x j Select m nodes in [α,β] and define n y  y j hi (y)   (1 i  m). ji, j1 yi  y j

Then function n m P(x, y)  f (xi , y j )li (x)h j (y) i1 j 1 Interpolates the two dimensional table with data

(xi , y j , f (xi , y j ))

36 2D Interpolation

20 Errors of Interpolation

If p is the polynomial of degree at most n that interpolates f at the n+1 () distinct nodes belonging to an interval [a,b] and if is continuous for each x in [a,b], there is a in (a, b) for which

n 1 (n1) f (x)  p(x)  f ( )(x  xi ) (n 1)! i0

What kind of function may have a small interpolation error?

38 Computing First Derivative

The first derivative can be approximated as 1 f '(x)  [ f (x  h)  f (x)] (1) h For accurate approximation, h should be small. Thus f(x + h) and f(x) are close to each other. This may cause loss of significant digits in finite precision computation

Using Taylor’s theorem, we have 1 f (x  h)  f (x)  hf '(x)  h2 f "( ) 2 For ξ between x and x + h. It follows that 1 1 f '(x)  [ f (x  h)  f (x)] hf "( ) h 2  1 hf "( ), The approximation error of (1) is2 or of order O(h). This is a first order (or sided) approximation of first derivative. The error goes to 0 as fast as h → 0 39 1st Order Approximation

20 Higher Order Approximation

It is desirable to have some higher order (faster) approximation schemes 1 f (x  h)  f (x)  hf '(x)  h2 f "(x) 2! 1 1  h3 f '''(x)  h4 f (4) (x)  3! 4! 1 f (x  h)  f (x)  hf '(x)  h2 f "(x) 2! 1 1  h3 f '''(x)  h4 f (4) (x)  3! 4! Subtracting these two equations, we have f (x  h)  f (x  h)  2hf '(x) 2 2  h3 f '''(x)  h5 f (5) (x)  3! 5!

41 2nd Order Approximation

It follows that 1 f '(x)  [ f (x  h)  f (x  h)] 2h h2 h4  f '''(x)  f (5) (x)  3! 5! After dropping the higher order terms, we have a second order approximation formula as 1 f '(x)  [ f (x  h)  f (x  h)] 2h

2 The leading truncated terms of this approximation scheme is  h f '''(x). 6 Hence the approximation is of O(h2). The approximation error goes to 0 as fast as h2 → 0. The exact truncation error is 1  f '''( )  f '''( ) 1  h2 1 2   h2 f '''( ) 6  2  6 42 2nd Order Approximation

20 Central DIfference Approximation

20 Richardson Extrapolation (I)

The first derivative can be approximated as f '(x)  1 [ f (x  h)  f (x  h)] 2h 2 4 6  a2h  a4h  a6h 

In which the constants a2,a4,… depend on the higher order derivatives of f and the value of x. When such information is available, it is possible to construct much more accurate approximation schemes

Define a function  (h)  1 [ f (x  h)  f (x  h)] 2h which is an approximation to f’(x) with error of order O(h2). This approximation becomes accurate as h → 0. So we can study the quantity limh → 0 ψ(h)

45 Richardson Extrapolation (II)

Richardson extrapolation estimates the value of ψ(0) from some computed values of ψ(h) near 0

2 4 6  (h)  f '(x)  a2h  a4h  a6h  2 4 6  h  h  h  h     f '(x)  a2    a4    a6     2   2   2   2  Multiply the 2nd equation by 4 and subtract it from the 1st equation

 h 3 4 15 6  (h)  4    3 f '(x)  a4h  a6h   2  4 16 Hence

 h  1   h   a4 4 5 6         (h)  f '(x)  h  a6h   2  3   2   4 16 f’(x) can be computed as accurate as O(h4)

46 General Approach

A general Richardson extrapolation form  2k  (h)  L   a2k h k1 where we assume that ψ(h) is computable for any h > 0 and we want to approximate L as accurately as possible h Choose a special sequence define 2n  h  D(n,0)    (n  0)  2n  Then, we have 2k   h  D(n,0)  L   A(k,0) n  k1  2 

With A(k,0) = -a2k. D(n,0) is a rough approximate of L = limx→0ψ(x) 47 Richardson Theorem The extrapolation formula is 4m D(n,m)  D(n,m  1)  4m  1 1 D(n  1,m  1) (1  m  n) 4m  1 Richardson Extrapolation Theorem: 2k   h  D(n,m)  L   A(k,m) n  km1  2  For 0 ≤ m ≤ n

The proof of this theorem is based on induction on m, see p. 175 of the Book.

Not that D(n,m) approximates L at the order of O(h2m). The convergence rate is fast 48 Computational Procedure

Richardson extrapolation computational procedure:

1.) write a procedure to compute ψ(h)

2.) decide on suitable values for n and h

3.) for i = 0, 1,…, n, compute

D(i,0)  (h / 2i )

4.) for 0 ≤ i ≤ j ≤ n, compute

D(i, j)  D(i, j  1)  (4 j  1)1[D(i, j  1)  D(i  1, j  1)]

49 Using Interpolation Polynomial

We can approximate the function f(x) by a polynomial pn(x) of order n, such that pn(x) ≈ f(x)

To compute f’(x), we use the approximation f’(x) ≈ p’n(x)

Higher order polynomials are avoided because of oscillation

Let p interpolates f at two points, x0 and x1

p1(x)  f (x0 )  f [x0 , x1](x  x0 )

The first derivative of p1(x) is

f (x1 )  f (x0 ) p'1 (x)  f [x0 , x1 ]   f '(x) x1  x0

50 1st and 2nd Order Approx.

Let x0 = x and x1 = x + h, we have 1 f '(x)  [ f (x  h)  f (x)] h This is just the O(h) order sided approximation formula

2 Put x0 = x – h and x1 = x + h, we have the O(h ) approximation scheme 1 f '(x)  [ f (x  h)  f (x  h)] 2h A three point polynomial interpolation is

p2 (x)  f (x0 )  f [x0 , x1 ](x  x0 )

 f [x0 , x1 , x2 ](x  x0 )(x  x1 ) We have corrected approximation

p'2 (x)  f [x0 , x1 ] f [x0 , x1 , x2 ](2x  x0  x1 )

51 Second Derivative

If we have first derivative, we can use 1 f "(x)  [ f '(x  h)  f '(x  h)] 2h To approximate the second derivative to O(h2)

A direct approximation would be using Taylor expansion f (x  h)  f (x  h)   1  2 f (x)  h2 f "(x)  2 h4 f (4) (x)  4!  Hence, we have 1 f "(x)  [ f (x  h)  2 f (x)  f (x  h)] h2 This approximation is of O(h2) accuracy

52