Steven R. Dunbar Department of Mathematics 203 Avery Hall University of Nebraska-Lincoln Lincoln, NE 68588-0130 http://www.math.unl.edu Voice: 402-472-3731 Fax: 402-472-8466 Selected Topics in Probability and Stochastic Processes Steve Dunbar Question of the Day How can we approximate a continuous function with a polynomial? How do we measure the quality of the approximation? If the function is differentiable, do the derivatives of the polynomials also approximate the derivatives of the function? Key Concepts 1. Weierstrass approximation theorem n j n−j 2. Bernstein polynomials bn(x) = j x (1 − x) 3. Chebyshev’s inequality Vocabulary n j n−j 1. Bernstein polynomials bn(x) = j x (1 − x) 2. Chebyshev’s inequality 1 Mathematical Ideas Bernstein Polynomials The Bernstein polynomials are defined as n b xj(1 − x)n−j n,j j for j = 0, 1, . , n. Weierstrass Approximation Theorem This section is adapted from: “Bernstein Polynomials and Brownian Mo- tion”, by Emmanuel Kowalski, American Mathematical Monthly, December 2006, pages 865-886. Theorem 1 (Weierstrass Approximation Theorem) Let d be a posi- tive integer. If f : [0, 1]d → C is a continuous function and if i i n n X 1 d Y ik n−ik Bn[f](x) = f ,..., xk (1 − xk) n n ik 0≤i1...id≤n k=1 d for n = 1, 2,... and x = (x1, . , xn) in [0, 1] , then Bn[f] → f uniformly on [0, 1]d. The Big Idea of the proof is the following: Let Xij, for 1 ≤≤ d and integers j ≥ 1 be a Bernoulli random variable so that PXij = 1 = xi and Pn PXij = 0 = 1 − xj. Let Sin = j=1 Xij Let Sn = (S1,...,Sn). By either the Weak Law or the Strong Law of Large Numbers we can assert that Sn/n ≈ x. By continuity, f(Sn/n) ≈ f(x). Then all the more so ESn/n ≈ f(x). But this is Bn[f](x) ≈ f(x), and we can see that Bn[f](x) is a polynomial. d Proof: Consider a positive integer n and x = (x1, . , xd) in [0, 1] . Let Xi = (X1i,...Xdi) be a finite sequence of random vectors defined on some probability space such that Xji are all independent and each takes values 0 and 1 according to a Bernoulli distribution for which PXji = 1 = xj and PXji = 0 = 1 − xj. (Note that the first index j is the coordinate and the Pn second indexi is the sequence index.) Write Sjn = i=1 Xji and Sn = (S1n,...,Sdn). Then as binomial random variable, n ij n−ij PSjn/n = ij/n = xj (1 − xj) ij 2 Then i i n n X 1 d Y ik n−ik Bn[f](x) f ,..., xk (1 − xk) n n ik 0≤i1...id≤n k=1 noting the use of the independence and the definition of the expectation. Note also that x (1 − x ) 1/4n σ2 = Var S /n = j j ≤ j,n jn n . (Note that the source article has a typo here.) Note that also ESn = nx, whence S S S B [f](x) − f(x) = f n − f(x) = f n − f( n ). n E n E n E n d Set kxk = maxj{|xj|} for x ∈ R . The event that the maximum of the coordinates is greater than a value is contained in the union of events where each coordinate is greater than the value. Then by Chebyshev’s inequality applied component-wise, and independence, we obtain that for any δ > 0 the upper bound 2 Sn X Sjn dσjn d P − x ≥ δ ≤ P − xj ≥ δ ≤ ≤ . n n δ2 4nδ2 1≤j≤d Jamie Radcliffe also pointed out that Sn Sn P − x ≥ δ = 1 − P − x < δ n n ! Y Sjn = 1 − P − xj < δ n 1≤j≤d ! Y Sjn = 1 − 1 − P − xj ≥ δ n 1≤j≤d X Sjn X Sjn Skn = P − xj ≥ δ − P − xj ≥ δ P − xk ≥ δ + ... n n n 1≤j≤d 1≤j,k≤d X Sjn ≤ P − xj ≥ δ n 1≤j≤d 3 This gives a sharper estimate. Now let > 0 be given. By the uniform continuity of f on [0, 1]d, there is a positive δ > 0 such that |f(x) − f(y)| < , whenever kx − yk < δ. Then Z Sn f − f(x) dP ≤ . kSn/n−xk<δ n (This is not sophisticated, we are integrating something of magnitude at most over a measure of at most 1.) Then on the complementary event, using the triangle inequality and the previous estimate of the probability of the complementary event: Z Sn Sn f − f(x) dP ≤ 2 max |f(x)|P − x ≥ δ x∈[0,1]d kSn/n−xk≥δ n n 2d ≤ max |f(x)|. 4nδ2 x∈[0,1]d Putting together the integrals over the two complementary events, for suffi- ciently large values of n, Sn Sn Ef − f E ≤ 2 n n which holds independently of n, and the theorem is established. Lemma 1 If f : [0, 1] → C is a continuous function and Bn[f] is the nth Bernstein polynomial of f, then n−1 X n − 1 j + 1 j B [f]0(x) = f − f xj(1 − x)n−1−j n j n n j=0 Proof: The proof is purely computational: n X n B [f]0(x) = jxj−1(1 − x)n−j − (n − j)xj(1 − x)n−j−1 n j j=0 n n−1 X n X n = j xj−1(1 − x)n−j − (n − j) xj(1 − x)n−j−1 j j j=1 j=0 4 Note the slight shift in indices on the summand in the last line, reflecting that the end terms disappear. Now making the change of summation j = k − 1 in the first summand and then relabeling k = j, and noting that n n − 1 j = n j j − 1 and n n (n − j) = n j j − 1 we regroup to get n−1 X n − 1 j + 1 j B [f]0(x) = f − f xj(1 − x)n−1−j n j n n j=0 2 Lemma 2 If f : [0, 1] → C is a continuous function and Bn[f] is the nth Bernstein polynomial of f, then n−1 X X n − 1n j + 1 k j k ∂ B [f](x, y) = n n f , − f , xj(1−x)n−1−jyk(1−y)n−k x n j k n n n n j=0 k=0 Proof: In the two-variable Bernstein polynomial for f, grouping all the terms with x = j/n and factoring we can see that X n j B [f](x, y) = n G , y xj(1 − x)n−j n j n n j=0 where Gn(x, y) = Bn[g](y) the Bernstein polynomial of the function of y given by g(y) = f(x, y) with x fixed. Now applying the previous lemma in the one-dimensional case, n−1 X n − 1 j + 1 j ∂ B [f](x, y) = n G , y − G , y xj(1 − x)n−j. x n j n n n n j=0 Now expand, and unfactor, and regroup to get the result of the lemma. 5 Theorem 2 If f : [0, 1] → C is a continuous function and x0 is a point of 0 [0, 1] such that the derivative f (x0) exists then 0 lim Bn[f](x) f (x) n→∞ Proof: The case x0 = 0 follows immediately from evaluating n−1 X n − 1 j + 1 j B [f]0(x) = f − f xj(1 − x)n−1−j n j n n j=0 at x0 = 0 to obtain only the j = 0 summand, leading to f(1/n) − f(0) B [f]0(0) = . n 1/n Likewise x0 = 1 leads to f(1) − f(1 − 1/n) B [f]0(1) = . n 1/n So now assume that x0(1 − x0) 6= 0. For the binomial random variables, this means that Var Sn = nx0(1 − x0) 6= 0. 0 Writing f(x) = f(x0) + (x − x0) ∗ f (x0) + g(x) with g(x0) differen- 0 tiable at x = x0 and g (x0) = 0. Since the operation of finding the Bern- stein polynomial is linear and the derivative is linear, it is enough to show 0 that Bn[g] (x0) → 0 as n → ∞. We can write g(x) = (x − x0)(x) where limx→x0 (x) = 0. Pn We claim that the following probabilistic formula holds, with Sn = j=0 Xj is the sum of Bernoulli random variables as before with PXj = 1 = x0 and Tn = Sn/n is the sequence of empirical averages: 0 E(Tn − x)g(TN ) Bn[g] (x) = . V (Tn) Proof of claim: Go back to the derivative expression for the Bernstein ap- proximation, and apply it to g: n 1 X n j B [g]0(x) = g (j − nx)xj(1 − x)n−j n x(1 − x) j n j=0 n n X n j = g (j/n − x)xj(1 − x)n−j. x(1 − x) j n j=0 6 After expanding j j j j j g − x = g − xg n n n n n we recognize the expression n B [g]0(x) = (B [g ](x) − xB [g ](x)) n x(1 − x) n 1 n 2 in which the Bernstein polynomials of g1(y) = yg(y) and g2(y) = xg(y) occur. Writing both of these functions in expectation Bernstein polynomial form, Bn[g1](x) = Eg1(Sn/n) = (Sn/n) · g(Sn/n) = Tn · g(Tn), and Bn[g2](x) = Exg(Sn/n) = Exg(Tn), using the linearity of the expectation, and Var Tn = x(1 − x)/n and the claim is established.