DOCSLIB.ORG

Approximation Error and Approximation Theory

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Approximation Error and Approximation Theory Approximation Error and Approximation Theory Federico Girosi Center for Basic Research in the Social Sciences Harvard University and Center for Biological and Computational Learning MIT [email protected] 1 Plan of the class • Learning and generalization error • Approximation problem and rates of convergence • N-widths • “Dimension independent” convergence rates 2 Note These slides cover more extensive material than what will be presented in class. 3 References The background material on generalization error (first 8 slides) is explained at length in: 1. P. Niyogi and F. Girosi. On the relationship between generalization error, hypothesis complexity, and sample complexity for Radial Basis Functions. Neural Computation, 8:819–842, 1996. 2. P. Niyogi and F. Girosi. Generalization bounds for function approximation from scattered noisy data. Advances in Computational Mathematics, 10:51–80, 1999. [1] has a longer explanation and introduction, while [2] is more mathematical and also contains a very simple probabilistic proof of a class of “dimension independent” bounds, like the ones discussed at the end of this class. As far as I know it is A. Barron who first clearly spelled out the decomposition of the generalization error in two parts. Barron uses a different framework from what we use, and he summarizes it nicely in: 3. A.R. Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning, 14:115–133, 1994. The paper is quite technical, and uses a framework which is different from what we use here, but it is important to read it if you plan to do research in this field. The material on n-widths comes from: 4. A. Pinkus. N-widths in Approximation Theory, Springer-Verlag, New York, 1980. Although the book is very technical, the first 8 pages contain an excellent introduction to the subject. The other great thing about this book is that you do not need to understand every single proof to appreciate the beauty and significance of the results, and it is a mine of useful information. 5. H.N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural Computation, 8:164–177, 1996. 4 6. A.R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transaction on Information Theory, 39:3, 930–945, 1993. 7. F. Girosi and G. Anzellotti. Rates of convergence of approximation by translates A.I. Memo 1288, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1992. For a curious way to prove dimension independent bounds using VC theory see: 8. F. Girosi. Approximation error bounds that use VC-bounds. In Proc. International Conference on Artificial Neural Networks, F. Fogelman-Souli`eand P. Gallinari, editors, Vol. 1, 295–302. Paris, France, October 1995. 5 Notations review • I[f] = X×Y V(f(x), y)p(x, y)dxdy R 1 l • Iemp[f] = l i=1 V(f(xi), yi) P • f0 = arg minf I[f] , f0 ∈ T • fH = arg minf∈H I[f] • f^H,l = arg minf∈H Iemp[f] 6 More notation review • I[f0] = how well we could possibly do • I[fH] = how well we can do in space H • I[f^H,l] = how well we do in space H with l data • |I[f] − Iemp[f]| ≤ Ω(H, l, δ) ∀f ∈ H (from VC theory) • I[f^H,l] is called generalization error • I[f^H,l] − I[f0] is also called generalization error ... 7 A General Decomposition I[f^H,l] − I[f0] = I[f^H,l] − I[fH] + I[fH] − I[f0] generalization error = estimation error + approximation error When the cost function V is quadratic: 2 I[f] = kf0 − fk + I[f0] and therefore 2 2 kf0 − fH,lk = I[f^H,l] − I[fH] + kf0 − fHk generalization error = estimation error + approximation error 8 A useful inequality If, with probability 1 − δ I[f] − Iemp[f] ≤ Ω(H, l, δ) ∀f ∈ H then ^ I[fH,l] − I[fH] ≤ 2Ω(H, l, δ) You can prove it using the following observations: • I[fH] ≤ I[f^H,l] (from the definition of fH) • Iemp[f^H,l] ≤ Iemp[fH] (from the definition of f^H,l) 9 Bounding the Generalization Error 2 2 kf0 − fH,lk ≤ 2Ω(H, l, δ) + kf0 − fHk Notice that: • Ω has nothing to do with the target space T , it is studied mostly in statistics; • kf0 − fHk has everything to do with the target space T , it is studied mostly in approximation theory; 10 Approximation Error We consider a nested family of hypothesis spaces Hn: H0 ⊂ H1 ⊂ ...Hn ⊂ ... and define the approximation error as: T (f, Hn) ≡ inf kf − hk h∈Hn T (f, Hn) is the smallest error that we can make if we approximate f ∈ T with an element of Hn (here k · k is the norm in T ). 11 Approximation error For reasonable choices of hypothesis spaces Hn: lim T (f, Hn) = 0 n This means that we can→∞ approximate functions of T arbitrarily well with elements of {Hn}n=1 Example: T = continuous functions∞ on compact sets, and Hn = polynomials of degree at most n. 12 The rate of convergence The interesting question is: How fast does T (f, Hn) go to zero? • The rate of convergence is a measure of the relative complexity of T with respect to the approximation scheme H. • The rate of convergence determines how many samples we need in order to obtain a given generalization error. 13 An Example • In the next slides we compute explicitly the rate of convergence of approximation of a smooth function by trigonometric polynomials. • We are interested in studying how fast the approximation error goes to zero when the number of parameters of our approximation scheme goes to infinity. • The reason for this exercise is that the results are representative: more complex and interesting cases all share the basic features of this example. 14 Approximation by Trigonometric Polynomials Consider the set of functions \ C2[−π, π] ≡ C[−π, π] L2[−π, π] Functions in this set can be represented as a Fourier series: π ikx −ikx f(x) = cke , ck ∝ dxf(x)e ∞ −π kX=0 Z The L2 norm satisfies the equation: kfk2 = c 2 L2 k ∞ kX=0 15 We consider as target space the following Sobolev space of smooth functions: s 2 d f Ws,2 ≡ f ∈ C2[−π, π] | < + dxs L2 ∞ The (semi)-norm in this Sobolev space is defined as: s 2 2 d f 2s 2 kfk ≡ = k c Ws,2 s k dx L ∞ 2 kX=1 If f belongs to Ws,2 then Fourier coefficients ck must go to zero at a rate which increases with s. 16 We choose as hypothesis space Hn the set of trigonometric polynomials of degree n: n ikx p(x) = ake kX=1 Given a function of the form ikx f(x) = cke ∞ kX=0 the optimal hypothesis fn(x) is given by the first n terms of its Fourier series: n ikx fn(x) = cke kX=1 17 Key Question For a given f ∈ Ws,2 we want to study the approximation error: [f] ≡ kf − f k2 n n L2 • Notice that n, the degree of the polynomial, is also the number of parameters that we use in the approximation. • Obviously n goes to zero as n + , but the key question is: how fast? → ∞ 18 An easy estimate of n [f] ≡ kf − f k2 = c2 = c2k2s 1 < n n L2 k=n+1 k k=n+1 k k2s P∞ P∞ 2 kfkW < 1 c2k2s < 1 c2k2s = s,2 n2s k=n+1 k n2s k=1 k n2s P∞ P∞ kfk2 Ws,2 n[f] <⇓ n2s More smoothness faster rate of convergence But what happens in⇒ more than one dimension? 19 Rates of convergence in d dimensions It is enough to study d = 2. We proceed in full analogy with the 1-d case: i(kx+my) f(x, y) = ckme ∞ k,mX=1 s 2 s 2 2 d f d f 2s 2s 2 kfk ≡ + = (k + m )c Ws,2 s s km dx L dy L ∞ 2 2 k,mX=1 Here Ws,2 is defined as the set of functions such that kfk2 < + Ws,2 ∞ 20 We choose as hypothesis space Hn the set of trigonometric polynomials of degree l: l (ikx+imy) p(x) = akme k,mX=1 A trigonometric polynomial of degree l in d variables has a number of coefficients n = ld. We are interested in the behavior of the approximation error as a function of n. The approximating function is: l (ikx+imy) fn(x, y) = ckme k,mX=1 21 An easy estimate of n (k2s + m2s) [f] ≡ kf − f k2 = c2 = c2 < n n L2 km km 2s 2s ∞ ∞ k + m k,mX=l+1 k,mX=l+1 1 1 < c2 (k2s + m2s) < c2 (k2s + m2s) = 2s km 2s km 2l ∞ 2l ∞ k,mX=l+1 k,mX=1 2 kfkW = s,2 2l2s d 1 Since n = l , then l = nd (with d = 2), and we obtain: kfk2 Ws,2 n < 2s 2n d 22 (Partial) Summary The previous calculations generalizes easily to the d-dimensional case. Therefore we conclude that: if we approximate functions of d variables with s square integrable derivatives with a trigonometric polynomial with n coefficients, the approximation error satisfies: C n < 2s n d More smoothness s faster rate of convergence d Higher dimension ⇒ slower rate of convergence ⇒ 23 Another Example: Generalized Translation Networks Consider networks of the form: n f(x) = akφ(Akx + bk) kX=1 d m where x ∈ R , bk ∈ R , 1 ≤ m ≤ d, Ak are m × d matrices, ak ∈ R and φ is some given function.
Load more

Recommended publications
  • Fen Fakültesi Matematik Bölümü Y
    İZMİR INSTITUTE OF TECHNOLOGY GRADUATE SCHOOL OF ENGINEERING AND SCIENCES DEPARTMENT OF MATHEMATICS CURRICULUM OF THE GRADUATE PROGRAMS M.S. in Mathematics (Thesis) Core Courses MATH 596 Graduate Seminar (0-2) NC AKTS:9 MATH 599 Scientific Research Techniques and Publication Ethics (0-2) NC AKTS:8 MATH 500 M.S. Thesis (0-1) NC AKTS:26 MATH 8XX Special Studies (8-0) NC AKTS:4 In addition, the following courses must be taken. MATH 515 Real Analysis (3-0)3 AKTS:8 MATH 527 Basic Abstract Algebra (3-0)3 AKTS:8 *All M.S. students must register Graduate Seminar course until the beginning of their 4th semester. Total credit (min.) :21 Number of courses with credit (min.) : 7 M.S. in Mathematics (Non-thesis) Core Courses MATH 516 Complex Analysis (3-0)3 AKTS:8 MATH 527 Basic Abstract Algebra (3-0)3 AKTS:8 MATH 533 Ordinary Differential Equations (3-0)3 AKTS:8 MATH 534 Partial Differential Equations (3-0)3 AKTS:8 MATH 573 Modern Geometry I (3-0)3 AKTS:8 MATH 595 Graduation Project (0-2) NC AKTS:5 MATH 596 Graduate Seminar (0-2) NC AKTS:9 MATH 599 Scientific Research Techniques and Publication Ethics (0-2) NC AKTS:8 Total credit (min.) :30 Number of courses with credit (min.) :10 Ph.D. in Mathematics Core Courses MATH 597 Comprehensive Studies (0-2) NC AKTS:9 MATH 598 Graduate Seminar in PhD (0-2) NC AKTS:9 MATH 599 Scientific Research Techniques and Publication Ethics* (0-2) NC AKTS:8 MATH 600 Ph.D.
    [Show full text]
  • RESOURCES in NUMERICAL ANALYSIS Kendall E
    RESOURCES IN NUMERICAL ANALYSIS Kendall E. Atkinson University of Iowa Introduction I. General Numerical Analysis A. Introductory Sources B. Advanced Introductory Texts with Broad Coverage C. Books With a Sampling of Introductory Topics D. Major Journals and Serial Publications 1. General Surveys 2. Leading journals with a general coverage in numerical analysis. 3. Other journals with a general coverage in numerical analysis. E. Other Printed Resources F. Online Resources II. Numerical Linear Algebra, Nonlinear Algebra, and Optimization A. Numerical Linear Algebra 1. General references 2. Eigenvalue problems 3. Iterative methods 4. Applications on parallel and vector computers 5. Over-determined linear systems. B. Numerical Solution of Nonlinear Systems 1. Single equations 2. Multivariate problems C. Optimization III. Approximation Theory A. Approximation of Functions 1. General references 2. Algorithms and software 3. Special topics 4. Multivariate approximation theory 5. Wavelets B. Interpolation Theory 1. Multivariable interpolation 2. Spline functions C. Numerical Integration and Differentiation 1. General references 2. Multivariate numerical integration IV. Solving Differential and Integral Equations A. Ordinary Differential Equations B. Partial Differential Equations C. Integral Equations V. Miscellaneous Important References VI. History of Numerical Analysis INTRODUCTION Numerical analysis is the area of mathematics and computer science that creates, analyzes, and implements algorithms for solving numerically the problems of continuous mathematics. Such problems originate generally from real-world applications of algebra, geometry, and calculus, and they involve variables that vary continuously; these problems occur throughout the natural sciences, social sciences, engineering, medicine, and business. During the second half of the twentieth century and continuing up to the present day, digital computers have grown in power and availability.
    [Show full text]
  • A Short Course on Approximation Theory
    A Short Course on Approximation Theory N. L. Carothers Department of Mathematics and Statistics Bowling Green State University ii Preface These are notes for a topics course offered at Bowling Green State University on a variety of occasions. The course is typically offered during a somewhat abbreviated six week summer session and, consequently, there is a bit less material here than might be associated with a full semester course offered during the academic year. On the other hand, I have tried to make the notes self-contained by adding a number of short appendices and these might well be used to augment the course. The course title, approximation theory, covers a great deal of mathematical territory. In the present context, the focus is primarily on the approximation of real-valued continuous functions by some simpler class of functions, such as algebraic or trigonometric polynomials. Such issues have attracted the attention of thousands of mathematicians for at least two centuries now. We will have occasion to discuss both venerable and contemporary results, whose origins range anywhere from the dawn of time to the day before yesterday. This easily explains my interest in the subject. For me, reading these notes is like leafing through the family photo album: There are old friends, fondly remembered, fresh new faces, not yet familiar, and enough easily recognizable faces to make me feel right at home. The problems we will encounter are easy to state and easy to understand, and yet their solutions should prove intriguing to virtually anyone interested in mathematics. The techniques involved in these solutions entail nearly every topic covered in the standard undergraduate curriculum.
    [Show full text]
  • Zonotopal Algebra and Forward Exchange Matroids
    Zonotopal algebra and forward exchange matroids Matthias Lenz1,2 Technische Universit¨at Berlin Sekretariat MA 4-2 Straße des 17. Juni 136 10623 Berlin GERMANY Abstract Zonotopal algebra is the study of a family of pairs of dual vector spaces of multivariate polynomials that can be associated with a list of vectors X. It connects objects from combinatorics, geometry, and approximation theory. The origin of zonotopal algebra is the pair (D(X), P(X)), where D(X) denotes the Dahmen–Micchelli space that is spanned by the local pieces of the box spline and P(X) is a space spanned by products of linear forms. The first main result of this paper is the construction of a canonical basis for D(X). We show that it is dual to the canonical basis for P(X) that is already known. The second main result of this paper is the construction of a new family of zonotopal spaces that is far more general than the ones that were recently studied by Ardila–Postnikov, Holtz–Ron, Holtz–Ron–Xu, Li–Ron, and others. We call the underlying combinatorial structure of those spaces forward exchange matroid. A forward exchange matroid is an ordered matroid together with a subset of its set of bases that satisfies a weak version of the basis exchange axiom. Keywords: zonotopal algebra, matroid, Tutte polynomial, hyperplane arrangement, multivariate spline, graded vector space, kernel of differential operators, Hilbert series 2010 MSC: Primary 05A15, 05B35, 13B25, 16S32, 41A15. Secondary: 05C31, arXiv:1204.3869v4 [math.CO] 31 Mar 2016 41A63, 47F05. Email address: [email protected] (Matthias Lenz) 1Present address: D´epartement de math´ematiques, Universit´ede Fribourg, Chemin du Mus´ee 23, 1700 Fribourg, Switzerland 2The author was supported by a Sofia Kovalevskaya Research Prize of Alexander von Humboldt Foundation awarded to Olga Holtz and by a PhD scholarship from the Berlin Mathematical School (BMS) 22nd November 2018 1.
    [Show full text]
  • Math 541 - Numerical Analysis Approximation Theory Discrete Least Squares Approximation
    Math 541 - Numerical Analysis Approximation Theory Discrete Least Squares Approximation Joseph M. Mahaffy, [email protected] Department of Mathematics and Statistics Dynamical Systems Group Computational Sciences Research Center San Diego State University San Diego, CA 92182-7720 http://jmahaffy.sdsu.edu Spring 2018 Discrete Least Squares Approximation — Joseph M. Mahaffy, [email protected] (1/97) Outline 1 Approximation Theory: Discrete Least Squares Introduction Discrete Least Squares A Simple, Powerful Approach 2 Application: Cricket Thermometer Polynomial fits Model Selection - BIC and AIC Different Norms Weighted Least Squares 3 Application: U. S. Population Polynomial fits Exponential Models 4 Application: Pharmokinetics Compartment Models Dog Study with Opioid Exponential Peeling Nonlinear least squares fits Discrete Least Squares Approximation — Joseph M. Mahaffy, [email protected] (2/97) Introduction Approximation Theory: Discrete Least Squares Discrete Least Squares A Simple, Powerful Approach Introduction: Matching a Few Parameters to a Lot of Data. Sometimes we get a lot of data, many observations, and want to fit it to a simple model. 8 6 4 2 0 0 1 2 3 4 5 Underlying function f(x) = 1 + x + x^2/25 Measured Data Average Linear Best Fit Quadratic Best Fit PDF-link: code. Discrete Least Squares Approximation — Joseph M. Mahaffy, [email protected] (3/97) Introduction Approximation Theory: Discrete Least Squares Discrete Least Squares A Simple, Powerful Approach Why a Low Dimensional Model? Low dimensional models (e.g. low degree polynomials) are easy to work with, and are quite well behaved (high degree polynomials can be quite oscillatory.) All measurements are noisy, to some degree.
    [Show full text]
  • Arxiv:1106.4415V1 [Math.DG] 22 Jun 2011 R,Rno Udai Form
    JORDAN STRUCTURES IN MATHEMATICS AND PHYSICS Radu IORDANESCU˘ 1 Institute of Mathematics of the Romanian Academy P.O.Box 1-764 014700 Bucharest, Romania E-mail: [email protected] FOREWORD The aim of this paper is to offer an overview of the most important applications of Jordan structures inside mathematics and also to physics, up- dated references being included. For a more detailed treatment of this topic see - especially - the recent book Iord˘anescu [364w], where sugestions for further developments are given through many open problems, comments and remarks pointed out throughout the text. Nowadays, mathematics becomes more and more nonassociative (see 1 § below), and my prediction is that in few years nonassociativity will govern mathematics and applied sciences. MSC 2010: 16T25, 17B60, 17C40, 17C50, 17C65, 17C90, 17D92, 35Q51, 35Q53, 44A12, 51A35, 51C05, 53C35, 81T05, 81T30, 92D10. Keywords: Jordan algebra, Jordan triple system, Jordan pair, JB-, ∗ ∗ ∗ arXiv:1106.4415v1 [math.DG] 22 Jun 2011 JB -, JBW-, JBW -, JH -algebra, Ricatti equation, Riemann space, symmet- ric space, R-space, octonion plane, projective plane, Barbilian space, Tzitzeica equation, quantum group, B¨acklund-Darboux transformation, Hopf algebra, Yang-Baxter equation, KP equation, Sato Grassmann manifold, genetic alge- bra, random quadratic form. 1The author was partially supported from the contract PN-II-ID-PCE 1188 517/2009. 2 CONTENTS 1. Jordan structures ................................. ....................2 § 2. Algebraic varieties (or manifolds) defined by Jordan pairs ............11 § 3. Jordan structures in analysis ....................... ..................19 § 4. Jordan structures in differential geometry . ...............39 § 5. Jordan algebras in ring geometries . ................59 § 6. Jordan algebras in mathematical biology and mathematical statistics .66 § 7.
    [Show full text]
  • Feynman Quantization
    3 FEYNMAN QUANTIZATION An introduction to path-integral techniques Introduction. By Richard Feynman (–), who—after a distinguished undergraduate career at MIT—had come in as a graduate student to Princeton, was deeply involved in a collaborative effort with John Wheeler (his thesis advisor) to shake the foundations of field theory. Though motivated by problems fundamental to quantum field theory, as it was then conceived, their work was entirely classical,1 and it advanced ideas so radicalas to resist all then-existing quantization techniques:2 new insight into the quantization process itself appeared to be called for. So it was that (at a beer party) Feynman asked Herbert Jehle (formerly a student of Schr¨odinger in Berlin, now a visitor at Princeton) whether he had ever encountered a quantum mechanical application of the “Principle of Least Action.” Jehle directed Feynman’s attention to an obscure paper by P. A. M. Dirac3 and to a brief passage in §32 of Dirac’s Principles of Quantum Mechanics 1 John Archibald Wheeler & Richard Phillips Feynman, “Interaction with the absorber as the mechanism of radiation,” Reviews of Modern Physics 17, 157 (1945); “Classical electrodynamics in terms of direct interparticle action,” Reviews of Modern Physics 21, 425 (1949). Those were (respectively) Part III and Part II of a projected series of papers, the other parts of which were never published. 2 See page 128 in J. Gleick, Genius: The Life & Science of Richard Feynman () for a popular account of the historical circumstances. 3 “The Lagrangian in quantum mechanics,” Physicalische Zeitschrift der Sowjetunion 3, 64 (1933). The paper is reprinted in J.
    [Show full text]
  • Faster Algorithms Via Approximation Theory
    Foundations and Trends R in Theoretical Computer Science Vol. 9, No. 2 (2013) 125–210 c 2014 S. Sachdeva and N. K. Vishnoi DOI: 10.1561/0400000065 Faster Algorithms via Approximation Theory Sushant Sachdeva Nisheeth K. Vishnoi Yale University Microsoft Research [email protected] [email protected] Contents Introduction 126 I APPROXIMATION THEORY 135 1 Uniform Approximations 136 2 Chebyshev Polynomials 140 3 Approximating Monomials 145 4 Approximating the Exponential 149 5 Lower Bounds for Polynomial Approximations 153 6 Approximating the Exponential using Rational Functions 158 7 Approximating the Exponential using Rational Functions with Negative Poles 163 ii iii II APPLICATIONS 171 8 Simulating Random Walks 174 9 Solving Equations via the Conjugate Gradient Method 179 10 Computing Eigenvalues via the Lanczos Method 185 11 Computing the Matrix Exponential 191 12 Matrix Inversion via Exponentiation 196 References 205 Abstract This monograph presents ideas and techniques from approximation theory for approximating functions such as xs; x−1 and e−x, and demonstrates how these results play a crucial role in the design of fast algorithms for problems which are increasingly relevant. The key lies in the fact that such results imply faster ways to compute primitives such as Asv; A−1v; exp(−A)v; eigenvalues, and eigenvectors, which are fundamental to many spectral algorithms. Indeed, many fast algorithms reduce to the computation of such primitives, which have proved useful for speeding up several fundamental computations such as random walk simulation, graph partitioning, and solving systems of linear equations. S. Sachdeva and N. K. Vishnoi. Faster Algorithms via Approximation Theory.
    [Show full text]
  • Publications by Lloyd N. Trefethen
    Publications by Lloyd N. Trefethen I. Books I.1. Numerical Conformal Mapping, editor, 269 pages. Elsevier, 1986. I.2. Finite Difference and Spectral Methods for Ordinary and Partial Differential Equations, x+315 pages. Graduate textbook, privately published at http://comlab.ox.ac.uk/nick.trefethen/, 1996. I.3. Numerical Linear Algebra, with David Bau III, xii+361 pages. SIAM, 1997. I.4. Spectral Methods in MATLAB, xviii+165 pages. SIAM, 2000. I.5. Schwarz-Christoffel Mapping, with Tobin A. Driscoll, xvi+132 pages. Cambridge U. Press, 2002. I.6. Spectra and Pseudospectra: The Behavior of Nonnormal Matrices and Operators, with Mark Embree, xviii+606 pages. Princeton U. Press, 2005. I.7. Trefethen’s Index Cards: Forty years of Notes about People, Words, and Mathematics, xv + 368 pages. World Scientific Publishing, 2011. I.8. Approximation Theory and Approximation Practice. SIAM, to appear in 2013. In addition to these a book has been published by other authors about the international problem solving challenge I organised in 2002 (item VIII.8 below): F. Bornemann, D. Laurie, S. Wagon and D. Waldvogel, The SIAM 100-Digit Challenge, SIAM, 2004, to appear in German translation as Abenteuer Numerik: Eine Führung entlang der Herausforderung von Trefethen, Springer, 2006. II. Finite difference and spectral methods for partial differential equations II.1. Group velocity in finite difference schemes. SIAM Review 24 (1982), 113-136. II.2. Group velocity interpretation of the stability theory of Gustafsson, Kreiss, and Sundström. J. Comp. Phys. 49 (1983), 199-217. II.3. On Lp-instability and dispersion at discontinuities in finite difference schemes.
    [Show full text]
  • Approximation in High-Dimensional and Nonparametric Statistics: a Unified Approach
    Approximation in high-dimensional and nonparametric statistics: a unified approach Yanjun Han Department of Electrical Engineering, Stanford University Joint work with: Jiantao Jiao Stanford EE Rajarshi Mukherjee Stanford Stats Tsachy Weissman Stanford EE July 4, 2016 Outline 1 Problem Setup 2 High-dimensional Parametric Setting 3 Infinite Dimensional Nonparametric Setting Upper bound Lower bound General Lr norm 2 / 49 1 Problem Setup 2 High-dimensional Parametric Setting 3 Infinite Dimensional Nonparametric Setting Upper bound Lower bound General Lr norm 3 / 49 Problem: estimation of functionals Given i.i.d. samples X1; ··· ; Xn ∼ P, we would like to estimate a one-dimensional functional F (P) 2 R: Parametric case: P = (p1; ··· ; pS ) is discrete, and S X F (P) = I (pi ) i=1 High dimensional: S & n Nonparametric case: P is continuous with density f , and Z F (P) = I (f (x))dx 4 / 49 Parametric case: when the functional is smooth... When I (·) is everywhere differentiable... H´ajek{Le Cam Theory The plug-in approach F (Pn) is asymptotically efficient, where Pn is the empirical distribution 5 / 49 Nonparametric case: when the functional is smooth... When I (·) is four times differentiable with bounded I (4), Taylor expansion yields Z Z 1 I (f (x))dx = I (f^) + I (1)(f^)(f − f^) + I (2)(f^)(f − f^)2 2 1 + I (3)(f^)(f − f^)3 + O((f − f^)4) dx 6 where f^ is a \good" estimator of f (e.g., a kernel estimate) Key observation: suffice to deal with linear (see, e.g., Nemirovski'00), quadratic (Bickel and Ritov'88, Birge and Massart'95) and cubic terms (Kerkyacharian and Picard'96) separately.
    [Show full text]
  • Topics in High-Dimensional Approximation Theory
    TOPICS IN HIGH-DIMENSIONAL APPROXIMATION THEORY DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of the Ohio State University By Yeonjong Shin Graduate Program in Mathematics The Ohio State University 2018 Dissertation Committee: Dongbin Xiu, Advisor Ching-Shan Chou Chuan Xue c Copyright by Yeonjong Shin 2018 ABSTRACT Several topics in high-dimensional approximation theory are discussed. The fundamental problem in approximation theory is to approximate an unknown function, called the target function by using its data/samples/observations. Depending on different scenarios on the data collection, different approximation techniques need to be sought in order to utilize the data properly. This dissertation is concerned with four different approximation methods in four different scenarios in data. First, suppose the data collecting procedure is resource intensive, which may require expensive numerical simulations or experiments. As the performance of the approximation highly depends on the data set, one needs to carefully decide where to collect data. We thus developed a method of designing a quasi-optimal point set which guides us where to collect the data, before the actual data collection procedure. We showed that the resulting quasi-optimal set notably outperforms than other standard choices. Second, it is not always the case that we obtain the exact data. Suppose the data is corrupted by unexpected external errors. These unexpected corruption errors can have big magnitude and in most cases are non-random. We proved that a well-known classical method, the least absolute deviation (LAD), could effectively eliminate the corruption er- rors.
    [Show full text]
  • Function Approximation and the Remez Algorithm 11
    FUNCTION APPROXIMATION AND THE REMEZ ALGORITHM ABIY TASISSA Abstract. The Remez exchange algorithm is explained. An implementation in Python is tested on different test functions. We make a comparison with SLSQP(Sequential Least Squares Programming) optimizer. Key words. Remez exchange, Minimax polynomial, polynomial interpolation 1. Overview. Given a set of data points and values measured at these points, it is of interest to to determine the function that does a `good' approximation. This problem belongs to the realm of function approximation or interpolation theory and finds its use in scientific computations[7]. In the past, these problems were handled using complicated tables and the evaluations were done using calculators or just by hand[4]. To increase the number of significant digits, it meant more manual work to be done. With the coming of computers, the computations were very fast and there was no need to store values as things could be computed on the fly. Approximating functions by rational functions is one class of problems in approx- imation theory. We are interested in the approximation of a function by a rational function in the L1 norm. That is we want to minimize the maximum vertical dis- tance between the function in consideration and a rational function. This is called the Minimax Approximation[7]. The theoretical framework for this was originally done by Chebyshev[16]. He stated what is called equioscillation theorem that gave a necessary and sufficient condition for a minimax polynomial/rational function[7]. By 1915, all the theoretical results regarding Minimax approximations had been es- tablished [19]. However, as we will see in later sections, the theoretical tools are useful but don't lead to any feasible way to find the Minimax rational function.
    [Show full text]