Part I Models, Errors, and Numbers 1 Part 1: Table of Contents i Python, Numpy, and Matplotlib Making Models with Polynomials: Taylor Series Making Models with Polynomials: Interpolation Making Models with Monte Carlo Error, Accuracy and Convergence Floating Point 2 Python, Numpy, and Matplotlib Programming Language: Python/numpy i • Reasonably readable • Reasonably beginner-friendly • Mainstream (top 5 in ‘TIOBE Index’) • Free, open-source • Great tools and libraries (not just) for scientific computing • Python 2/3? 3! • numpy: Provides an array datatype Will use this and matplotlib all the time. • See class web page for learning materials 3 • Demo: Python • Demo: numpy • In-class activity: Image Processing 4 Making Models with Polynomials: Taylor Series Why polynomials? i 3 2 a3x + a2x + a1x + a0 How do we write a general degree n polynomial? n X i aix i=0 Why polynomials and not something else? • We can add, multiply, maybe divide (grade school, computer HW) 5 Why polynomials? ii p • More complicated functions (ex, sin x, x) have to be built from those parts ! at least approximately • Easy to work with as a building block. General recipe for numerics: Model observation with a polynomial, perform operation on that, evaluate. (e.g. calculus, root finding) 6 Reconstructing a Function From Derivatives i 0 00 Given f (x0); f (x0); f (x0);::: can we reconstruct a polynomial f ? f (x) = ??? + ???x + ???x2 + ··· For simplicity, let us consider x0 = 0 and a degree 4 polyno- mial f 2 3 4 f (x) = a0 + a1x + a2x + a3x + a4x Note that a0 = f (0). Now, take a derivative 0 2 3 f (x) = a1 + 2a2x + 3a3x + 4a4x 7 Reconstructing a Function From Derivatives ii 0 we see that a1 = f (0), differentiating again 00 2 f (x) = 2a2 + 3 · 2a3x + 4 · 3a4x 00 yields a2 = f (0)=2, and finally 000 f (x) = 3 · 2a3 + 4 · 3 · 2a4x 000 (i) yields a3 = f (0)=3!. In general, ai = f (0)=i!. 8 Reconstructing a Function From Derivatives i Found: Taylor series approximation. f 00(0) f (0 + x) ≈ f (0) + f 0(0)x + x2 + ··· 2 The general Taylor expansion with center x0 = 0 is 1 X f (i)(0) f (x) = xi i! i=0 Demo: Polynomial Approximation with Derivatives (click to visit) (Part I) 9 Shifting the Expansion Center i Can you do this at points other than the origin? In this case, 0 is the center of the series expansion. We can obtain a Taylor expansion of f centered at x0 by defining a shifted function g(x) = f (x + x0) The Taylor expansion g with center 0 is as before 1 X g(i)(0) g(x) = xi i! i=0 10 Shifting the Expansion Center ii This expansion of g yields a Taylor expansion of f with center x0 1 X g(i)(0) f (x) = g(x − x ) = (x − x )i 0 i! 0 i=0 (i) (i) It suffices to note that f (x0) = g (0) to get the general form 1 1 X f (i)(x ) X f (i)(x ) f (x) = 0 (x − x )i or f (x + h) = 0 hi i! 0 0 i! i=0 i=0 11 Errors in Taylor Approximation (I) i Can’t sum infinitely many terms. Have to truncate. How big of an error does this cause? Demo: Polynomial Approximation with Derivatives (click to visit) (Part II) Suspicion, as h ! 0, we have n X f (i)(x ) f (x + h) − 0 hi C · hn+1 0 i! 6 i=0 | {z } Taylor error for degree n or n X f (i)(x ) f (x + h) − 0 hi = O(hn+1) 0 i! i=0 | {z } Taylor error for degree n 12 Errors in Taylor Approximation (I) ii As we will see, there is a satisfactory bound on C for most functions f . 13 Making Predictions with Taylor Truncation Error i p Suppose you expand x − 10 in a Taylor polynomial of degree 3 about the center x0 = 12. For h1 = 0:5, you find that the Taylor truncation error is about 10−4. What is the Taylor truncation error for h2 = 0:25? Error(h) = O(hn+1), where n = 3, i.e. 4 Error(h1) ≈ C · h1 4 Error(h2) ≈ C · h2 14 Making Predictions with Taylor Truncation Error ii While not knowing C or lower order terms, we can use the ratio of h2=h1 4 4 4 4 h2 h2 Error(h2) ≈ C · h2 = C · h1 ≈ Error(h1) · h1 h1 Can make prediction of the error for one h if we know another. Demo: Polynomial Approximation with Derivatives (click to visit) (Part III) 15 Taylor Remainders: the Full Truth i Let f : R ! R be (n + 1)-times differentiable on the interval (n) (x0; x) with f continuous on [x0; x]. Then there exists a ξ 2 (x0; x) so that n X f (i)(x ) f (n+1)(ξ) f (x + h) − 0 hi = ·(ξ − x )n+1 0 i! (n + 1)! 0 i=0 | {z } “C” and since jξ − x0j 6 h n (i)( ) f (n+1)(ξ) X f x0 i n+1 f (x0 + h) − h 6 ·h : i! (n + 1)! i=0 | {z } “C” 16 Intuition for Taylor Remainder Theorem i 0 Given the value of a function and its derivative f (x0); f (x0), prove the Taylor error bound. We express f (x) as Z x 0 f (x) = f (x0) + f (w0)dw0 x0 17 Intuition for Taylor Remainder Theorem ii 0 00 We then express the integrand f (w0) for w0 2 [x0; x] using f in the same way Z x Z w0 0 00 f (x) = f (x0) + f (x0) + f (w1)dw1 dw0 x0 x0 Z x Z w0 0 00 = f (x0) + f (x0)(x − x0) + f (w1)dw1 dw0 | {z } x0 x0 Taylor expansion t1(x) 18 Intuition for Taylor Remainder Theorem iii We can bound the error by finding the maximizer ξ = argmax j 00(ξ)j ξ2[x0;x] f Z x Z w0 00 jf (x) − t1(x)j ≤ jf (ξ)jdw1 dw0 x0 x0 jf 00(ξ)j = (x − x )2 2! 0 jf 00(ξ)j jf (x + h) − t (x + h)j ≤ h2 for x = x + h 0 1 0 2! 0 In-class activity: Taylor series 19 Proof of Taylor Remainder Theorem i We can complete the proof by induction of the Taylor degree expansion n Z x Z w0 Z wn−1 (n) f (x) = tn−1(x) + ··· f (wn)dwn ··· dw0 x0 x0 x0 Given the formula above, it suffices to expand Z wn (n) (n) (n+1) f (wn) = f (x0) + f (wn+1)dwn+1 x0 20 Proof of Taylor Remainder Theorem ii Inserting this back in gives us the next inductive hypothesis (n) f (x0) n f (x) = t − (x) + (x − x ) n 1 n! 0 | {z } Taylor expansion tn(x) Z x Z w0 Z wn (n+1) + ··· f (wn+1)dwn+1 ··· dw0 x0 x0 x0 21 Proof of Taylor Remainder Theorem iii The bound follows as for n = 1, with ξ = argmax j (n+1)(ξ)j ξ2[x0;x] f Z x Z w0 Z wn (n+1) jf (x) − tn(x)j ≤ ··· jf (ξ)jdwn+1 ··· dw0 x0 x0 x0 jf 00(ξ)j = (x − x )n+1 (n + 1)! 0 jf 00(ξ)j jf (x + h) − t (x + h)j ≤ hn+1 for x = x + h 0 n 0 (n + 1)! 0 22 Using Polynomial Approximation i Suppose we can approximate a function as a polynomial: 2 3 f (x) ≈ a0 + a1x + a2x + a3x : How is that useful? E.g.: What if we want the integral of f ? Easy: Just integrate the polynomial: Z t Z t 2 3 f (x)dx ≈ a0 + a1x + a2x + a3x dx s s Z t Z t Z t Z t 2 3 = a0 1dx + a1 x · dx + a2 x dx + a3 x dx s s s s 23 Using Polynomial Approximation ii Even if you had no idea how to integrate f using calculus, you can approximately integrate f anyway, by taking a bunch of derivatives (and forming a Taylor polynomial). Demo: Computing Pi with Taylor (click to visit) 24 Making Models with Polynomials: Interpolation Reconstructing a Function From Point Values i If we know function values at some points f (x1); f (x2);:::; f (xn), can we reconstruct the function as a polynomial? f (x) = ??? + ???x + ???x2 + ··· In particular, we’d like to obtain a1; a2;::: to satisfy equations: 2 a0 + a1 · x1 + a2 · x1 + ··· = f (x1) . 2 a0 + a1 · xn + a2 · xn + ··· = f (xn) 25 Reconstructing a Function From Point Values ii Q: How many ai can we (uniquely) determine this way? (Answer: n–same number of unknowns as equations.) The equations for these n-unkowns are 2 n−1 a0 + a1 · x1 + a2 · x1 + ··· + an−1 · x1 = f (x1) . 2 n−1 a0 + a1 · xn + a2 · xn + ··· + an−1 · xn = f (xn) We can rewrite them in matrix form 0 n−1 1 0 1 0 1 1 x1 ··· x1 a0 f (x1) B n−1 C B C B C B 1 x2 ··· x2 C B a1 C B f (x2) C B C B C = B C : B . .. C B . C B . C @ . A @ . A @ . A n−1 1 xn ··· xn an−1 f (xn) | {z } | {z } | {z } V a f 26 Reconstructing a Function From Point Values iii Thus, to find (a0;:::; an) we need to solve a linear system. This gives us a degree n − 1 polynomial interpolant satisfying ~ f (xi) = f (xi) of the form ~ 2 n−1 f (x) = a0 + a1 · xn + a2 · xn + ··· + an−1 · xn 27 Vandermonde Linear Systems i Polynomial interpolation is a critical component in many nu- merical models.