DOCSLIB.ORG

A New Perspective on Machine Learning: How to Do Perfect Supervised Learning

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A New Perspective on Machine Learning: How to Do Perfect Supervised Learning A New Perspective on Machine Learning: How to do Perfect Supervised Learning Hui Jiang [email protected] iFLYTEK Laboratory for Neural Computing and Machine Learning (iNCML) Department of Electrical Engineering and Computer Science York University, 4700 Keele Street, Toronto, Ontario, M3J 1P3, Canada http://wiki.cse.yorku.ca/user/hj/ Abstract other hand, since the boom of deep learning in the past decade, the landscape of machine learning practices has In this work, we introduce the concept of ban- changed dramatically. A variety of artificial neural net- dlimiting into the theory of machine learning works (ANN) have been successfully applied to all sorts because all physical processes are bandlimited of real-world applications, ranging from speech recogni- by nature, including real-world machine learn- tion and image classification to machine translation. The ing tasks. After the bandlimiting constraint is list of success stories in many diverse application domains taken into account, our theoretical analysis has is still growing year after year. The superhuman perfor- shown that all practical machine learning tasks mance has even been claimed in some tasks, which were are asymptotically solvable in a perfect sense. originally thought to be very hard. The divergence between Furthermore, the key towards this solvability al- the theories and the practices has equally puzzled both ML most solely relies on two factors: i) a suffi- theorists and ML practitioners. At this point, we desper- ciently large amount of training samples beyond ately need to answer a series of serious questions in order a threshold determined by a difficulty measure- to further advance the field as a whole. For instance, why ment of the underlying task; ii) a sufficiently do the ANN-type models significantly overtake other ex- complex and bandlimited model. Moreover, for isting ML methods on all of these practical applications? some special cases, we have derived new error What is the essence to the success of the ANN-type models bounds for perfect learning, which can quantify on these ostensibly challenging tasks? Where is the limit of the difficulty of learning. These generalization these ANN-type models? Why does horrific overfitting, as bounds are not only asymptotically convergent predicted by the current ML theory, never happen in these but also irrelevant to model complexity. Our real-world tasks even when some shockingly huge models new results on generalization have provided a are used? (Zhang et al., 2016) new perspective to explain the recent successes of large-scale supervised learning using complex In this paper, we develop a new ML theory to shed some models like neural networks. light on these questions. The key to our new theory is the concept of bandlimiting. Not all processes may actually ex- arXiv:1901.02046v3 [cs.LG] 19 Mar 2019 ist in the real world and all physically realizable processes 1. Introduction must be bandlimited. Much of the previous efforts in ma- chine learning theory have been spent in studying some ex- The fundamental principles and theories for machine learn- tremely difficult problems that are over-generalized in the- ing (ML) were established a few decades ago, such as the ory but may not actually exist in practice. After the ban- No-Free-Lunch theory (Wolpert, 1995), statistical learning dlimiting constraint is taken into account, our theoretical theory (Vapnik, 2000), and probably approximately correct analysis has shown that all practical machine learning tasks (PAC) learning (Valiant, 1984). These theoretical works are asymptotically solvable in a perfect sense. Our theo- have successfully explained which problems are learnable retical results suggest that the roadmap towards successful and how to achieve effective learning in principle. On the supervised learning consists of several steps: (a) collecting sufficient labelled in-domain data; (b) fitting a complex and This work is partially supported by a research donation from iFLY- bandlimited model to the large training set. The amount of TEK Co., Hefei, China and a Discovery Grant from Natural Sci- ences and Engineering Research Council (NSERC) of Canada. data needed for perfect learning depends on the difficulty of Perfect Machine Learning each underlying task. For some special cases, we have de- 2.2. The Bandlimiting Property rived some new error bounds to quantitatively measure the In engineering, it is a well-known fact that all physically difficulty of learning. As the amount of training data grows, realizable processes must satisfy the so-called bandlimit- we need a complicated model to complement step (b). The ing property. Bandlimiting is a strong constraint imposed universal approximation theory in (Cybenko, 1989; Hornik, on the smoothness and growth of functions, which corre- 1991) makes neural networks an ideal candidate for perfect sponds to the mathematic concept of finite exponent type learning since similar model structures can be fitted to any of entire functions in mathematical analysis (Levin, 1964; large training set if we keep increasing the model size. The Levinson, 1940). As shown in Figures1 to4, several 1-D highly-criticized engineering tricks used in the training of functions with different bandlimiting constraints are plot- neural networks are just some empirical approaches to en- ted as an illustration, which clearly show that the various sure that a complicated model is effectively fit to a very bandlimiting contraints heavily affect the smoothness of a large training set in step (b) (Jiang, 2019). However, there function. is no evidence to show that neural networks are the only possible models that are able to achieve perfect learning. In practice, if a supervised learning problem arises from a real-world task or a physical process, the above target 2. Problem Formulation function y = f(x) will satisfy the bandlimiting property as constrained by the physical world. The central idea in In this work, we study the standard supervised learning this paper is to demonstrate that the bandlimiting property, problem in machine learning. Given a finite training set largely overlooked by the machine learning community in of N samples of input and output pairs, denoted as DN = the past, is essential in explaining why real-world machine f(x1; y1); (x2; y2); ··· ; (xN ; yN )g, the goal is to learn a learning problems are not as hard as speculated by statisti- model from input to output over the entire feature space: cal learning theory (Vapnik, 2000; Shalev-Shwartz & Ben- x ! y, which will be used to predict future inputs. David, 2014). The theory proposed in this paper further suggests that under certain conditions we may even solve 2.1. Machine Learning as Stochastic Function Fitting many real-world supervised learning problems perfectly. Instead of starting our analysis from the joint probabilis- First of all, let’s give the definition of bandlimiting1.A tic distribution of inputs and outputs p(x; y) as in normal function f(x) 2 L1(RK ) is called to be strictly bandlim- statistical learning theory, we adopt a more restricted for- ited if its multivariate Fourier transform (Stein & Weiss, mulation in this paper. Here, we assume all inputs x are 1971), F (!), vanishes to zero beyond a certain finite spa- random variables following a probabilistic density func- tial frequency range. If there exists B > 0, such that tion, i.e. x ∼ p(x), in the input feature space (without losing generality, we may assume x 2 K ). The rela- Z Z +1 R −ix·! tion between input x (x 2 K ) and output y (y 2 ) F (!) = ··· f(x)e dx = 0 if k!k > B; R R −∞ is deterministic, which may be represented by a function (1) K f : x ! y (x 2 R ; y 2 R), denoted as the target func- then f(x) is called a strictly bandlimited function by B. tion y = f(x). In this setting, the goal of machine learning 1 K is to learn a model f^(x) to minimize the expected error Similarly, we may define a function f(x) 2 L (R ) is approximately bandlimited if its Fourier transform F (!) between f^(x) and f(x) as measured by p(x). satisfies: Most of interesting and meaningful learning problems in the real world can be easily accommodated by the above Z Z lim ··· kF (!)k2 d! = 0: deterministic function between inputs and outputs. For ex- B!+1 k!k>B ample, we may define y = f(x) = arg maxy p(yjx) if the conditional distribution p(yjx) is sharp and unimodal. If p(yjx) is sharp but not unimodal, we may decompose the In other words, for any arbitrary small > 0, 9B > 0, the learning problem into several sub-problems, each of which out-of-band residual energy satisfies is represented by one deterministic function as above. If Z Z p(yjx) is not sharp, it means that the relation between in- 2 2 puts and outputs are fairly weak. In these cases, either it ··· kF (!)k d! < (2) k!k>B is usually not a meaningful learning problem in practice, or we may improve input features x to further enhance the where B is called the approximate band of f(x) at . relation between x and y. 1Also known as wavenumber-limited. Here, we prefer the term “bandlimiting” as it is better known in engineering. Perfect Machine Learning in the sense of mean squared error (MSE): Figure 1. A function without band-limiting h i No Band-limiting ^ ^ 2 R(f j DN ) = Ep(x) kf(xjDN ) − f(x)k 300 200 Z Z +1 2 100 ^ = ··· f(x) − f(xjDN ) p(x) dx: (3) 0 output: y −∞ -100 -200 Usually the above expected risk is not practically achiev- -300 20 40 60 80 100 120 140 160 180 200 input: x able since it requires two unknown functions, f(x) and p(x).
Load more

Recommended publications
  • Sampling Signals on Graphs from Theory to Applications
    1 Sampling Signals on Graphs From Theory to Applications Yuichi Tanaka, Yonina C. Eldar, Antonio Ortega, and Gene Cheung Abstract The study of sampling signals on graphs, with the goal of building an analog of sampling for standard signals in the time and spatial domains, has attracted considerable attention recently. Beyond adding to the growing theory on graph signal processing (GSP), sampling on graphs has various promising applications. In this article, we review current progress on sampling over graphs focusing on theory and potential applications. Although most methodologies used in graph signal sampling are designed to parallel those used in sampling for standard signals, sampling theory for graph signals significantly differs from the theory of Shannon–Nyquist and shift-invariant sampling. This is due in part to the fact that the definitions of several important properties, such as shift invariance and bandlimitedness, are different in GSP systems. Throughout this review, we discuss similarities and differences between standard and graph signal sampling and highlight open problems and challenges. I. INTRODUCTION Sampling is one of the fundamental tenets of digital signal processing (see [1] and references therein). As such, it has been studied extensively for decades and continues to draw considerable research efforts. Standard sampling theory relies on concepts of frequency domain analysis, shift invariant (SI) signals, and bandlimitedness [1]. Sampling of time and spatial domain signals in shift-invariant spaces is one of the most important building blocks of digital signal processing systems. However, in the big data era, the signals we need to process often have other types of connections and structure, such as network signals described by graphs.
    [Show full text]
  • Lecture19.Pptx
    Outline Foundations of Computer Graphics (Fall 2012) §. Basic ideas of sampling, reconstruction, aliasing CS 184, Lectures 19: Sampling and Reconstruction §. Signal processing and Fourier analysis §. http://inst.eecs.berkeley.edu /~cs184 Implementation of digital filters §. Section 14.10 of FvDFH (you really should read) §. Post-raytracing lectures more advanced topics §. No programming assignment §. But can be tested (at high level) in final Acknowledgements: Thomas Funkhouser and Pat Hanrahan Some slides courtesy Tom Funkhouser Sampling and Reconstruction Sampling and Reconstruction §. An image is a 2D array of samples §. Discrete samples from real-world continuous signal (Spatial) Aliasing (Spatial) Aliasing §. Jaggies probably biggest aliasing problem 1 Sampling and Aliasing Image Processing pipeline §. Artifacts due to undersampling or poor reconstruction §. Formally, high frequencies masquerading as low §. E.g. high frequency line as low freq jaggies Outline Motivation §. Basic ideas of sampling, reconstruction, aliasing §. Formal analysis of sampling and reconstruction §. Signal processing and Fourier analysis §. Important theory (signal-processing) for graphics §. Implementation of digital filters §. Also relevant in rendering, modeling, animation §. Section 14.10 of FvDFH Ideas Sampling Theory Analysis in the frequency (not spatial) domain §. Signal (function of time generally, here of space) §. Sum of sine waves, with possibly different offsets (phase) §. Each wave different frequency, amplitude §. Continuous: defined at all points; discrete: on a grid §. High frequency: rapid variation; Low Freq: slow variation §. Images are converting continuous to discrete. Do this sampling as best as possible. §. Signal processing theory tells us how best to do this §. Based on concept of frequency domain Fourier analysis 2 Fourier Transform Fourier Transform §. Tool for converting from spatial to frequency domain §.
    [Show full text]
  • And Bandlimiting IMAHA, November 14, 2009, 11:00–11:40
    Time- and bandlimiting IMAHA, November 14, 2009, 11:00{11:40 Joe Lakey (w Scott Izu and Jeff Hogan)1 November 16, 2009 Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting sampling theory and history Time- and bandlimiting, history, definitions and basic properties connecting sampling and time- and bandlimiting The multiband case Outline Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting Time- and bandlimiting, history, definitions and basic properties connecting sampling and time- and bandlimiting The multiband case Outline sampling theory and history Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting connecting sampling and time- and bandlimiting The multiband case Outline sampling theory and history Time- and bandlimiting, history, definitions and basic properties Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting The multiband case Outline sampling theory and history Time- and bandlimiting, history, definitions and basic properties connecting sampling and time- and bandlimiting Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting Outline sampling theory and history Time- and bandlimiting, history, definitions and basic properties connecting sampling and time- and bandlimiting The multiband case Joe Lakey (w Scott Izu and Jeff Hogan) Time- and bandlimiting R −2πitξ Fourier transform: bf (ξ) = f (t) e dt R _ Bandlimiting:
    [Show full text]
  • Information Theory
    Information Theory Professor John Daugman University of Cambridge Computer Science Tripos, Part II Michaelmas Term 2016/17 H(X,Y) I(X;Y) H(X|Y) H(Y|X) H(X) H(Y) 1 / 149 Outline of Lectures 1. Foundations: probability, uncertainty, information. 2. Entropies defined, and why they are measures of information. 3. Source coding theorem; prefix, variable-, and fixed-length codes. 4. Discrete channel properties, noise, and channel capacity. 5. Spectral properties of continuous-time signals and channels. 6. Continuous information; density; noisy channel coding theorem. 7. Signal coding and transmission schemes using Fourier theorems. 8. The quantised degrees-of-freedom in a continuous signal. 9. Gabor-Heisenberg-Weyl uncertainty relation. Optimal \Logons". 10. Data compression codes and protocols. 11. Kolmogorov complexity. Minimal description length. 12. Applications of information theory in other sciences. Reference book (*) Cover, T. & Thomas, J. Elements of Information Theory (second edition). Wiley-Interscience, 2006 2 / 149 Overview: what is information theory? Key idea: The movements and transformations of information, just like those of a fluid, are constrained by mathematical and physical laws. These laws have deep connections with: I probability theory, statistics, and combinatorics I thermodynamics (statistical physics) I spectral analysis, Fourier (and other) transforms I sampling theory, prediction, estimation theory I electrical engineering (bandwidth; signal-to-noise ratio) I complexity theory (minimal description length) I signal processing, representation, compressibility As such, information theory addresses and answers the two fundamental questions which limit all data encoding and communication systems: 1. What is the ultimate data compression? (answer: the entropy of the data, H, is its compression limit.) 2.
    [Show full text]
  • Lecture 16 Bandlimiting and Nyquist Criterion
    ELG3175 Introduction to Communication Systems Lecture 16 Bandlimiting and Nyquist Criterion Bandlimiting and ISI • Real systems are usually bandlimited. • When a signal is bandlimited in the frequency domain, it is usually smeared in the time domain. This smearing results in intersymbol interference (ISI). • The only way to avoid ISI is to satisfy the 1st Nyquist criterion. • For an impulse response this means at sampling instants having only one nonzero sample. Lecture 12 Band-limited Channels and Intersymbol Interference • Rectangular pulses are suitable for infinite-bandwidth channels (practically – wideband). • Practical channels are band-limited -> pulses spread in time and are smeared into adjacent slots. This is intersymbol interference (ISI). Input binary waveform Individual pulse response Received waveform Eye Diagram • Convenient way to observe the effect of ISI and channel noise on an oscilloscope. Eye Diagram • Oscilloscope presentations of a signal with multiple sweeps (triggered by a clock signal!), each is slightly larger than symbol interval. • Quality of a received signal may be estimated. • Normal operating conditions (no ISI, no noise) -> eye is open. • Large ISI or noise -> eye is closed. • Timing error allowed – width of the eye, called eye opening (preferred sampling time – at the largest vertical eye opening). • Sensitivity to timing error -> slope of the open eye evaluated at the zero crossing point. • Noise margin -> the height of the eye opening. Pulse shapes and bandwidth • For PAM: L L sPAM (t) = ∑ai p(t − iTs ) = p(t)*∑aiδ (t − iTs ) i=0 i=0 L Let ∑aiδ (t − iTs ) = y(t) i=0 Then SPAM ( f ) = P( f )Y( f ) BPAM = Bp.
    [Show full text]
  • Frames and Other Bases in Abstract and Function Spaces Novel Methods in Harmonic Analysis, Volume 1
    Applied and Numerical Harmonic Analysis Isaac Pesenson Quoc Thong Le Gia Azita Mayeli Hrushikesh Mhaskar Ding-Xuan Zhou Editors Frames and Other Bases in Abstract and Function Spaces Novel Methods in Harmonic Analysis, Volume 1 Applied and Numerical Harmonic Analysis Series Editor John J. Benedetto University of Maryland College Park, MD, USA Editorial Advisory Board Akram Aldroubi Gitta Kutyniok Vanderbilt University Technische Universität Berlin Nashville, TN, USA Berlin, Germany Douglas Cochran Mauro Maggioni Arizona State University Duke University Phoenix, AZ, USA Durham, NC, USA Hans G. Feichtinger Zuowei Shen University of Vienna National University of Singapore Vienna, Austria Singapore, Singapore Christopher Heil Thomas Strohmer Georgia Institute of Technology University of California Atlanta, GA, USA Davis, CA, USA Stéphane Jaffard Yang Wang University of Paris XII Michigan State University Paris, France East Lansing, MI, USA Jelena Kovaceviˇ c´ Carnegie Mellon University Pittsburgh, PA, USA More information about this series at http://www.springer.com/series/4968 Isaac Pesenson • Quoc Thong Le Gia Azita Mayeli • Hrushikesh Mhaskar Ding-Xuan Zhou Editors Frames and Other Bases in Abstract and Function Spaces Novel Methods in Harmonic Analysis, Volume 1 Editors Isaac Pesenson Quoc Thong Le Gia Department of Mathematics School of Mathematics and Statistics Temple University University of New South Wales Philadelphia, PA, USA Sydney, NSW, Australia Azita Mayeli Hrushikesh Mhaskar Department of Mathematics Institute of Mathematical
    [Show full text]
  • Ideal C/D Converter
    Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.341: Discrete-Time Signal Processing OpenCourseWare 2006 Lecture 4 DT Processing of CT Signals & CT Processing of DT Signals: Fractional Delay Reading: Sections 4.1 - 4.5 in Oppenheim, Schafer & Buck (OSB). A typical discrete-time system used to process a continuous-time signal is shown in OSB Figure 4.15. T is the sampling/reconstruction period for the C/D and the D/C converters. When the input signal is bandlimited, the effective system in Figure 4.15 (b) is equivalent to the continuous-time system shown in Figure 4.15 (a). Ideal C/D Converter The relationship between the input and output signals in an ideal C/D converter as depicted in OSB Figure 4.1 is: Time Domain: x[n] = xc(nT ) j! 1 P1 ! 2¼k Frequency Domain: X(e ) = T k=¡1 Xc(j( T ¡ T )); j! where X(e ) and Xc(jΩ) are the DTFT and CTFT of x[n], xc(t). These relationships are illustrated in the figures below. See Section 4.2 of OSB for their derivations and more detailed descriptions of C/D converters. Time/Frequency domain representation of C/D conversion 1 In the frequency domain, a C/D converter can be thought of in three stages: periodic 1 repetition of the spectrum at 2¼, normalization of frequency by , and scaling of the amplitude T 1 ! by . The CT frequency Ω is related to the DT frequency ! by Ω = . If x (t) is not T T c ­s ­s bandlimited to ¡ 2 < Ωmax < 2 , repetition at 2¼ will cause aliasing.
    [Show full text]
  • Focm 2017 Foundations of Computational Mathematics Barcelona, July 10Th-19Th, 2017 Organized in Partnership With
    FoCM 2017 Foundations of Computational Mathematics Barcelona, July 10th-19th, 2017 http://www.ub.edu/focm2017 Organized in partnership with Workshops Approximation Theory Computational Algebraic Geometry Computational Dynamics Computational Harmonic Analysis and Compressive Sensing Computational Mathematical Biology with emphasis on the Genome Computational Number Theory Computational Geometry and Topology Continuous Optimization Foundations of Numerical PDEs Geometric Integration and Computational Mechanics Graph Theory and Combinatorics Information-Based Complexity Learning Theory Plenary Speakers Mathematical Foundations of Data Assimilation and Inverse Problems Multiresolution and Adaptivity in Numerical PDEs Numerical Linear Algebra Karim Adiprasito Random Matrices Jean-David Benamou Real-Number Complexity Alexei Borodin Special Functions and Orthogonal Polynomials Mireille Bousquet-Mélou Stochastic Computation Symbolic Analysis Mark Braverman Claudio Canuto Martin Hairer Pierre Lairez Monique Laurent Melvin Leok Lek-Heng Lim Gábor Lugosi Bruno Salvy Sylvia Serfaty Steve Smale Andrew Stuart Joel Tropp Sponsors Shmuel Weinberger 2 FoCM 2017 Foundations of Computational Mathematics Barcelona, July 10th{19th, 2017 Books of abstracts 4 FoCM 2017 Contents Presentation . .7 Governance of FoCM . .9 Local Organizing Committee . .9 Administrative and logistic support . .9 Technical support . 10 Volunteers . 10 Workshops Committee . 10 Plenary Speakers Committee . 10 Smale Prize Committee . 11 Funding Committee . 11 Plenary talks . 13 Workshops . 21 A1 { Approximation Theory Organizers: Albert Cohen { Ron Devore { Peter Binev . 21 A2 { Computational Algebraic Geometry Organizers: Marta Casanellas { Agnes Szanto { Thorsten Theobald . 36 A3 { Computational Number Theory Organizers: Christophe Ritzenhaler { Enric Nart { Tanja Lange . 50 A4 { Computational Geometry and Topology Organizers: Joel Hass { Herbert Edelsbrunner { Gunnar Carlsson . 56 A5 { Geometric Integration and Computational Mechanics Organizers: Fernando Casas { Elena Celledoni { David Martin de Diego .
    [Show full text]
  • Aliasing Reduction in Clipped Signals
    This is an electronic reprint of the original article. This reprint may differ from the original in pagination and typographic detail. Author(s): Fabián Esqueda, Stefan Bilbao and Vesa Välimäki Title: Aliasing reduction in clipped signals Year: 2016 Version: Author accepted / Post print version Please cite the original version: Fabián Esqueda, Stefan Bilbao and Vesa Välimäki. Aliasing reduction in clipped signals. IEEE Transactions on Signal Processing, Vol. 64, No. 20, pp. 5255-5267, October 2016. DOI: 10.1109/TSP.2016.2585091 Rights: © 2016 IEEE. Reprinted with permission. In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of Aalto University's products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink. This publication is included in the electronic version of the article dissertation: Esqueda, Fabián. Aliasing Reduction in Nonlinear Audio Signal Processing. Aalto University publication series DOCTORAL DISSERTATIONS, 74/2018. All material supplied via Aaltodoc is protected by copyright and other intellectual property rights, and duplication or sale of all or part of any of the repository collections is not permitted, except that material may be duplicated by you for your research use or educational purposes in electronic or print form. You must obtain permission for any other use. Electronic or print copies may not be offered, whether for sale or otherwise to anyone who is not an authorised user.
    [Show full text]
  • Bandlimited Signal Extrapolation Using Prolate Spheroidal Wave Functions
    IAENG International Journal of Computer Science, 40:4, IJCS_40_4_09 ______________________________________________________________________________________ Bandlimited Signal Extrapolation Using Prolate Spheroidal Wave Functions Amal Devasia 1 and Michael Cada 2 reconstruction of signals. Further to this, in [11,12], they Abstract— A simple, efficient and reliable method to showed signal reconstruction using non-uniform sampling extrapolate bandlimited signals varying from lower to higher and level-crossing sampling with Slepian functions. frequencies is proposed. The orthogonal properties of linear Since the primary focus of this paper is on bandlimited prolate spheroidal wave functions (PSWFs) are exploited to signal extrapolation and not just reconstruction or form an orthogonal basis set needed for synthesis. A significant step in the process is the higher order piecewise polynomial interpolation, we will concentrate more on the recent approximation of the overlap integral required for obtaining advancements in this regard. While we consider the signals the expansion coefficients accurately with very high precision. to be bandlimited in the Fourier transform domain, much Two sets of PSWFs, one relatively with lower Slepian attention has recently been on extrapolation of signals frequency and the other with higher Slepian frequency, are bandlimited in linear canonical transform (LCT) domain, considered. Numerical results of extrapolation of some this being a four-parameter family of linear integral standard test signals using the two sets are presented, compared, discussed, and some interesting inferences are transform [13,14] that generalizes Fourier transform as one made. of its special cases. For extrapolation of LCT bandlimited signals, several iterative and non-iterative algorithms have Index Terms—Signal extrapolation; Slepian series; been proposed [15-18].
    [Show full text]
  • Alias-Free Digital Synthesis of Classic Analog Waveforms
    Alias-Free Digital Synthesis of Classic Analog Waveforms Tim [email protected] rd.edu Julius Smith [email protected] u CCRMA http://www-ccrma.stanford.edu/ Music Department, Stanford University Abstract Techniques are reviewed and presented for alias-free digital synthesis of classical analog synthesizer waveforms such as pulse train and sawtooth waves. Techniquesdescribed include summation of bandlim- ited primitive waveforms as well as table-lookup techniques. Bandlimited pulse and triangle waveforms are obtained by integrating the difference of two out-of-phase bandlimited impulse trains. Bandlimited impulse trains are generated as a superposition of windowed sinc functions. Methods for more general bandlimited waveform synthesis are also reviewed. Methods are evaluated from the perspectives of sound quality, computational economy, and ease of control. 1 Introduction impulse train Any analog signal with a discontinuity in the wave- 1 form (such as pulse train or sawtooth) or in the wave- 0.8 form slope (such as triangle wave) must be bandlim- 0.6 ited to less than half the sampling rate before sampling 0.4 to obtain a corresponding discrete-time signal. Simple methods of generating these waveforms digitally con- 0.2 tain aliasing due to having to round off the discontinuity 0 0 5 10 15 20 time to the nearest available sampling instant. The sig- nals primarily addressed here are the impulse train, rect- Box Train, and Sample Positions angular pulse, and sawtooth waveforms. Because the 1 latter two signals can be derived from the ®rst by inte- 0.8 gration, only the algorithm for the impulse train is de- 0.6 veloped in detail.
    [Show full text]
  • Analytical Model for GNSS Receiver Implementation Losses*
    © The MITRE Corporation. All rights reserved. Analytical Model for GNSS Receiver Implementation Losses* Christopher J. Hegarty, The MITRE Corporation intensive and do not provide much insight into the various BIOGRAPHY1 loss mechanisms. Dr. Christopher J. Hegarty is a director with The MITRE Earlier analytical studies of DSSS receiver SNR losses Corporation, where he has worked mainly on aviation due to quantization include derivations in [1] for applications of GNSS since 1992. He is currently the quantization losses of 1.96 dB and 0.55 dB, respectively, Chair of the Program Management Committee of RTCA, for 1- and 2-bit uniform quantizers. These derivations are Inc., and co-chairs RTCA Special Committee 159 simplified in that they assume that the received signal is (GNSS). He served as editor of NAVIGATION: The sampled, but with independent noise upon each sample Journal of the Institute of Navigation from 1997 – 2006 and ignoring bandlimiting effects on the desired signal and as president of the Institute of Navigation in 2008. He component. A more thorough treatment of bandlimiting, was a recipient of the ION Early Achievement Award in sampling, and quantization effects in digital matched 1998, the U.S. Department of State Superior Honor filters is provided in [2]. The results in [2] were later Award in 2005, the ION Kepler Award in 2005, and the inferred to apply to GNSS receiver losses in [3, 4]. As Worcester Polytechnic Institute Hobart Newell Award in detailed later in this paper, however, the results in [2] are 2006. He is a Fellow of the ION, and co-editor/co-author truly not directly applicable to GNSS or other DSSS of the textbook Understanding GPS: Principles and receivers because the losses derived therein presume Applications, 2nd Ed.
    [Show full text]