DOCSLIB.ORG

Gradient-Based Learning Applied to Document Recognition

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Gradient-Based Learning Applied to Document Recognition PROC OF THE IEEE NOVEMBER GradientBased Learning Applied to Do cument Recognition Yann LeCun Leon Bottou Yoshua Bengio and Patrick Haner Abstract I Introduction Multilayer Neural Networks trained with the backpropa gation algorithm constitute the b est example of a successful Over the last several years machine learning techniques GradientBased Learning technique Given an appropriate particularly when applied to neural networks haveplayed network architecture GradientBased Learning algorithms an increasingly imp ortant role in the design of pattern can b e used to synthesize a complex decision surface that can recognition systems In fact it could be argued that the classify highdimensional patterns such as handwritten char acters with minimal prepro cessing This pap er reviews var availability of learning techniques has b een a crucial fac ious metho ds applied to handwritten character recognition tor in the recent success of pattern recognition applica and compares them on a standard handwritten digit recog tions suchascontinuous sp eech recognition and handwrit nition task Convolutional Neural Networks that are sp ecif ically designed to deal with the variability of D shap es are ing recognition shown to outp erform all other techniques The main message of this pap er is that b etter pattern Reallife do cument recognition systems are comp osed recognition systems can b e built by relying more on auto of multiple mo dules including eld extraction segmenta tion recognition and language mo deling A new learning matic learning and less on handdesigned heuristics This paradigm called Graph Transformer Networks GTN al is made p ossible by recent progress in machine learning ultimo dule systems to b e trained globally using lows suchm and computer technology Using character recognition as GradientBased metho ds so as to minimize an overall p er a case study we show that handcrafted feature extrac formance measure Two systems for online handwriting recognition are de tion can b e advantageously replaced by carefully designed scrib ed Exp eriments demonstrate the advantage of global learning machines that op erate directly on pixel images training and the exibility of Graph Transformer Networks Using do cument understanding as a case study we show A Graph Transformer Network for reading bank checkis that the traditional way of building recognition systems by also describ ed It uses Convolutional Neural Network char acter recognizers combined with global training techniques manually integrating individually designed mo dules can b e to provides record accuracy on business and p ersonal checks replaced by a unied and wellprincipled design paradigm It is deployed commercially and reads several million checks called Graph Transformer Networks that allows training per day all the mo dules to optimize a global p erformance criterion Keywords Neural Networks OCR Do cument Recogni tion Machine Learning GradientBased Learning Convo Since the early days of pattern recognition it has been lutional Neural Networks Graph Transformer Networks Fi known that the variability and richness of natural data nite State Transducers be it sp eech glyphs or other typ es of patterns make it almost imp ossible to build an accurate recognition system entirely by hand Consequen tly most pattern recognition Nomenclature systems are built using a combination of automatic learn GT Graph transformer ing techniques and handcrafted algorithms The usual GTN Graph transformer network metho d of recognizing individual patterns consists in divid HMM Hidden Markov mo del ing the system into twomain mo dules shown in gure HOS Heuristic oversegmentation The rst mo dule called the feature extractor transforms KNN Knearest neighb or the input patterns so that they can b e represented bylow NN Neural network dimensional vectors or short strings of symb ols that a can OCR Optical character recognition b e easily matched or compared and b are relatively in PCA Principal comp onent analysis variant with resp ect to transformations and distortions of RBF Radial basis function the input patterns that do not change their nature The RSSVM Reducedset supp ort vector metho d feature extractor contains most of the prior knowledge and SDNN Space displacement neural network is rather sp ecic to the task It is also the fo cus of most of SVM Supp ort vector metho d the design eort b ecause it is often entirely handcrafted TDNN Time delay neural network The classier on the other hand is often generalpurp ose VSVM Virtual supp ort vector metho d and trainable One of the main problems with this ap proach is that the recognition accuracy is largely deter The authors are with the Sp eech and Image Pro mined by the ability of the designer to come up with an cessing Services Research Lab oratory ATT Labs appropriate set of features This turns out to b e a daunt Research Schulz Drive Red Bank NJ Email fyannleonbyoshuahanergresearchattcom Yoshua Bengio ing task which unfortunatelymust b e redone for eachnew is also with the Departement dInformatique et de Recherche problem A large amount of the pattern recognition liter Operationelle UniversitedeMontreal CP Succ CentreVille Chemin de la Tour Montreal Qu eb ec Canada HC J ature is devoted to describing and comparing the relative PROC OF THE IEEE NOVEMBER Class scores manipulate directed graphs This leads to the concept of ransformer Network GTN also intro trainable Graph T in Section IV Section V describ es the now clas TRAINABLE CLASSIFIER MODULE duced sical metho d of heuristic oversegmentation for recogniz words or other character strings Discriminative and Feature vector ing nondiscriminative gradientbased techniques for training a recognizer at the word level without requiring manual FEATURE EXTRACTION MODULE segmentation and lab eling are presented in Section VI Sec tion VI I presents the promising SpaceDisplacementNeu Raw input ral Network approach that eliminates the need for seg mentation heuristics by scanning a recognizer at all pos Fig Traditional pattern recognition is p erformed with twomod sible lo cations on the input In section VIII it is shown ules a xed feature extractor and a trainable classier that trainable Graph Transformer Networks can be for mulated as multiple generalized transductions based on a general graph comp osition algorithm The connections b e merits of dierent feature sets for particular tasks tween GTNs and Hidden Markov Mo dels commonly used Historically the need for appropriate feature extractors in sp eech recognition is also treated Section IX describ es was due to the fact that the learning techniques used by a globally trained GTN system for recognizing handwrit the classiers were limited to lowdimensional spaces with ing entered in a p en computer This problem is known as easily separable classes Acombination of three factors online handwriting recognition since the machine must have changed this vision over the last decade First the pro duce immediate feedback as the user writes The core of availabilityoflowcost machines with fast arithmetic units the system is a Convolutional Neural Network The results allows to rely more on bruteforce numerical metho ds clearly demonstrate the advantages of training a recognizer than on algorithmic renements Second the availability at the word level rather than training it on presegmented of large databases for problems with a large market and handlab eled isolated characters Section X describ es a wide interest such as handwriting recognition has enabled complete GTNbased system for reading handwritten and designers to rely more on real data and less on handcrafted machineprinted bank checks The core of the system is the feature extraction to build recognition systems The third Convolutional Neural Network called LeNet describ ed in and very imp ortantfactoristheavailabilityofpowerful ma Section I I This system is in commercial use in the NCR chine learning techniques that can handle highdimensional Corp oration line of check recognition systems for the bank inputs and can generate intricate decision functions when ing industry It is reading millions of checks p er month in fed with these large data sets It can b e argued that the several banks across the United States recent progress in the accuracy of sp eech and handwriting recognition systems can b e attributed in large part to an A Learning from Data increased reliance on learning techniques and large training data sets As evidence to this fact a large prop ortion of There are several approaches to automatic machine mo dern commercial OCR systems use some form of multi learning but one of the most successful approaches p op layer Neural Network trained with backpropagation ularized in recentyears by the neural network community In this studywe consider the tasks of handwritten char can b e called numerical or gradientbased learning The p p acter recognition Sections I and I I and compare the p er learning machine computes a function Y F Z W p formance of several learning techniques on a benchmark where Z is the pth input pattern and W represents the data set for handwritten digit recognition Section I I I collection of adjustable parameters in the system In a p While more automatic learning is b enecial no learning pattern recognition setting the output Y may be inter p technique can succeed without a minimal amount of prior preted as the recognized class lab el of pattern Z or as knowledge ab out the task In the case of multilayer neu scores or probabilities asso ciated with each
Load more

Recommended publications
  • Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos
    Noname manuscript No. (will be inserted by the editor) Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos Hugo Jair Escalante∗ · Heysem Kaya∗ · Albert Ali Salah∗ · Sergio Escalera · Ya˘gmur G¨u¸cl¨ut¨urk · Umut G¨u¸cl¨u · Xavier Bar´o · Isabelle Guyon · Julio Jacques Junior · Meysam Madadi · Stephane Ayache · Evelyne Viegas · Furkan G¨urpınar · Achmadnoer Sukma Wicaksana · Cynthia C. S. Liem · Marcel A. J. van Gerven · Rob van Lier Received: date / Accepted: date ∗ Means equal contribution by the authors. Hugo Jair Escalante INAOE, Mexico and ChaLearn, USA E-mail: [email protected] Heysem Kaya Namık Kemal University, Department of Computer Engineering, Turkey E-mail: [email protected] Albert Ali Salah Bo˘gazi¸ci University, Dept. of Computer Engineering, Turkey and Nagoya University, FCVRC, Japan E-mail: [email protected] Furkan G¨urpınar Bo˘gazi¸ciUniversity, Computational Science and Engineering, Turkey E-mail: [email protected] Sergio Escalera University of Barcelona and Computer Vision Center, Spain E-mail: [email protected] Meysam Madadi Computer Vision Center, Spain E-mail: [email protected] Ya˘gmur G¨u¸cl¨ut¨urk,Umut G¨u¸cl¨u,Marcel A. J. van Gerven and Rob van Lier Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, the Netherlands E-mail: fy.gucluturk,u.guclu,m.vangerven,[email protected] Xavier Bar´oand Julio Jacques Junior Universitat Oberta de Catalunya and Computer Vision Center, Spain arXiv:1802.00745v3 [cs.CV] 29 Sep 2019 E-mail: fxbaro,[email protected] Isabelle Guyon UPSud/INRIA, Universit´eParis-Saclay, France and ChaLearn, USA 2 H.J.
    [Show full text]
  • NVIDIA CEO Jensen Huang to Host AI Pioneers Yoshua Bengio, Geoffrey Hinton and Yann Lecun, and Others, at GTC21
    NVIDIA CEO Jensen Huang to Host AI Pioneers Yoshua Bengio, Geoffrey Hinton and Yann LeCun, and Others, at GTC21 Online Conference to Feature Jensen Huang Keynote and 1,300 Talks from Leaders in Data Center, Networking, Graphics and Autonomous Vehicles NVIDIA today announced that its CEO and founder Jensen Huang will host renowned AI pioneers Yoshua Bengio, Geoffrey Hinton and Yann LeCun at the company’s upcoming technology conference, GTC21, running April 12-16. The event will kick off with a news-filled livestreamed keynote by Huang on April 12 at 8:30 am Pacific. Bengio, Hinton and LeCun won the 2018 ACM Turing Award, known as the Nobel Prize of computing, for breakthroughs that enabled the deep learning revolution. Their work underpins the proliferation of AI technologies now being adopted around the world, from natural language processing to autonomous machines. Bengio is a professor at the University of Montreal and head of Mila - Quebec Artificial Intelligence Institute; Hinton is a professor at the University of Toronto and a researcher at Google; and LeCun is a professor at New York University and chief AI scientist at Facebook. More than 100,000 developers, business leaders, creatives and others are expected to register for GTC, including CxOs and IT professionals focused on data center infrastructure. Registration is free and is not required to view the keynote. In addition to the three Turing winners, major speakers include: Girish Bablani, Corporate Vice President, Microsoft Azure John Bowman, Director of Data Science, Walmart
    [Show full text]
  • On Recurrent and Deep Neural Networks
    On Recurrent and Deep Neural Networks Razvan Pascanu Advisor: Yoshua Bengio PhD Defence Universit´ede Montr´eal,LISA lab September 2014 Pascanu On Recurrent and Deep Neural Networks 1/ 38 Studying the mechanism behind learning provides a meta-solution for solving tasks. Motivation \A computer once beat me at chess, but it was no match for me at kick boxing" | Emo Phillips Pascanu On Recurrent and Deep Neural Networks 2/ 38 Motivation \A computer once beat me at chess, but it was no match for me at kick boxing" | Emo Phillips Studying the mechanism behind learning provides a meta-solution for solving tasks. Pascanu On Recurrent and Deep Neural Networks 2/ 38 I fθ(x) = f (θ; x) ? F I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Supervised Learing I f :Θ D T F × ! Pascanu On Recurrent and Deep Neural Networks 3/ 38 ? I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Supervised Learing I f :Θ D T F × ! I fθ(x) = f (θ; x) F Pascanu On Recurrent and Deep Neural Networks 3/ 38 Supervised Learing I f :Θ D T F × ! I fθ(x) = f (θ; x) ? F I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Pascanu On Recurrent and Deep Neural Networks 3/ 38 Optimization for learning θ[k+1] θ[k] Pascanu On Recurrent and Deep Neural Networks 4/ 38 Neural networks Output neurons Last hidden layer bias = 1 Second hidden layer First hidden layer Input layer Pascanu On Recurrent and Deep Neural Networks 5/ 38 Recurrent neural networks Output neurons Output neurons Last hidden layer bias = 1 bias = 1 Recurrent Layer Second hidden layer First hidden layer Input layer Input layer (b) Recurrent
    [Show full text]
  • Generative Adversarial Networks (Gans)
    Generative Adversarial Networks (GANs) The coolest idea in Machine Learning in the last twenty years - Yann Lecun Generative Adversarial Networks (GANs) 1 / 39 Overview Generative Adversarial Networks (GANs) 3D GANs Domain Adaptation Generative Adversarial Networks (GANs) 2 / 39 Introduction Generative Adversarial Networks (GANs) 3 / 39 Supervised Learning Find deterministic function f: y = f(x), x:data, y:label Generative Adversarial Networks (GANs) 4 / 39 Unsupervised Learning "Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we do not know how to make the cake. We need to solve the unsupervised learning problem before we can even think of getting to true AI." - Yann Lecun "You cannot predict what you cannot understand" - Anonymous Generative Adversarial Networks (GANs) 5 / 39 Unsupervised Learning More challenging than supervised learning. No label or curriculum. Some NN solutions: Boltzmann machine AutoEncoder Generative Adversarial Networks Generative Adversarial Networks (GANs) 6 / 39 Unsupervised Learning vs Generative Model z = f(x) vs. x = g(z) P(z|x) vs P(x|z) Generative Adversarial Networks (GANs) 7 / 39 Autoencoders Stacked Autoencoders Use data itself as label Generative Adversarial Networks (GANs) 8 / 39 Autoencoders Denosing Autoencoders Generative Adversarial Networks (GANs) 9 / 39 Variational Autoencoder Generative Adversarial Networks (GANs) 10 / 39 Variational Autoencoder Results Generative Adversarial Networks (GANs) 11 / 39 Generative Adversarial Networks Ian Goodfellow et al, "Generative Adversarial Networks", 2014.
    [Show full text]
  • Arxiv:2008.08516V1 [Cs.LG] 19 Aug 2020
    Automated Machine Learning - a brief review at the end of the early years Hugo Jair Escalante Computer Science Department Instituto Nacional de Astrof´ısica, Optica´ y Electronica,´ Tonanzintla, Puebla, 72840, Mexico E-mail: [email protected] Abstract Automated machine learning (AutoML) is the sub-field of machine learning that aims at automating, to some extend, all stages of the design of a machine learning system. In the context of supervised learning, AutoML is concerned with feature extraction, pre processing, model design and post processing. Major contributions and achievements in AutoML have been taking place during the recent decade. We are there- fore in perfect timing to look back and realize what we have learned. This chapter aims to summarize the main findings in the early years of AutoML. More specifically, in this chapter an introduction to AutoML for supervised learning is provided and an historical review of progress in this field is presented. Likewise, the main paradigms of AutoML are described and research opportunities are outlined. 1 Introduction Automated Machine Learning or AutoML is a term coined by the machine learning community to refer to methods that aim at automating the design and development of machine learning systems and appli- cations [33]. In the context of supervised learning, AutoML aims at relaxing the need of the user in the loop from all stages in the design of supervised learning systems (i.e., any system relying on models for classification, recognition, regression, forecasting, etc.). This is a tangible need at present, as data are be- arXiv:2008.08516v1 [cs.LG] 19 Aug 2020 ing generated vastly and in practically any context and scenario, however, the number of machine learning experts available to analyze such data is overseeded.
    [Show full text]
  • Robust Deep Learning: a Case Study Victor Estrade, Cécile Germain, Isabelle Guyon, David Rousseau
    Robust deep learning: A case study Victor Estrade, Cécile Germain, Isabelle Guyon, David Rousseau To cite this version: Victor Estrade, Cécile Germain, Isabelle Guyon, David Rousseau. Robust deep learning: A case study. JDSE 2017 - 2nd Junior Conference on Data Science and Engineering, Sep 2017, Orsay, France. pp.1-5, 2017. hal-01665938 HAL Id: hal-01665938 https://hal.inria.fr/hal-01665938 Submitted on 17 Dec 2017 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Robust deep learning: A case study Victor Estrade, Cecile Germain, Isabelle Guyon, David Rousseau Laboratoire de Recherche en Informatique Abstract. We report on an experiment on robust classification. The literature proposes adversarial and generative learning, as well as feature construction with auto-encoders. In both cases, the context is domain- knowledge-free performance. As a consequence, the robustness quality relies on the representativity of the training dataset wrt the possible perturbations. When domain-specific a priori knowledge is available, as in our case, a specific flavor of DNN called Tangent Propagation is an effective and less data-intensive alternative. Keywords: Domain adaptation, Deep Neural Networks, High Energy Physics 1 Motivation This paper addresses the calibration of a classifier in presence of systematic errors, with an example in High Energy Physics.
    [Show full text]
  • Yoshua Bengio and Gary Marcus on the Best Way Forward for AI
    Yoshua Bengio and Gary Marcus on the Best Way Forward for AI Transcript of the 23 December 2019 AI Debate, hosted at Mila Moderated and transcribed by Vincent Boucher, Montreal AI AI DEBATE : Yoshua Bengio | Gary Marcus — Organized by MONTREAL.AI and hosted at Mila, on Monday, December 23, 2019, from 6:30 PM to 8:30 PM (EST) Slides, video, readings and more can be found on the MONTREAL.AI debate webpage. Transcript of the AI Debate Opening Address | Vincent Boucher Good Evening from Mila in Montreal Ladies & Gentlemen, Welcome to the “AI Debate”. I am Vincent Boucher, Founding Chairman of Montreal.AI. Our participants tonight are Professor GARY MARCUS and Professor YOSHUA BENGIO. Professor GARY MARCUS is a Scientist, Best-Selling Author, and Entrepreneur. Professor MARCUS has published extensively in neuroscience, genetics, linguistics, evolutionary psychology and artificial intelligence and is perhaps the youngest Professor Emeritus at NYU. He is Founder and CEO of Robust.AI and the author of five books, including The Algebraic Mind. His newest book, Rebooting AI: Building Machines We Can Trust, aims to shake up the field of artificial intelligence and has been praised by Noam Chomsky, Steven Pinker and Garry Kasparov. Professor YOSHUA BENGIO is a Deep Learning Pioneer. In 2018, Professor BENGIO was the computer scientist who collected the largest number of new citations worldwide. In 2019, he received, jointly with Geoffrey Hinton and Yann LeCun, the ACM A.M. Turing Award — “the Nobel Prize of Computing”. He is the Founder and Scientific Director of Mila — the largest university-based research group in deep learning in the world.
    [Show full text]
  • Mlcheck– Property-Driven Testing of Machine Learning Models
    MLCheck– Property-Driven Testing of Machine Learning Models Arnab Sharma Caglar Demir Department of Computer Science Data Science Group University of Oldenburg Paderborn University Oldenburg, Germany Paderborn, Germany [email protected] [email protected] Axel-Cyrille Ngonga Ngomo Heike Wehrheim Data Science Group Department of Computer Science Paderborn University University of Oldenburg Paderborn, Germany Oldenburg, Germany [email protected] [email protected] ABSTRACT 1 INTRODUCTION In recent years, we observe an increasing amount of software with The importance of quality assurance for applications developed machine learning components being deployed. This poses the ques- using machine learning (ML) increases steadily as they are being tion of quality assurance for such components: how can we validate deployed in a growing number of domains and sites. Supervised ML whether specified requirements are fulfilled by a machine learned algorithms “learn” their behaviour as generalizations of training software? Current testing and verification approaches either focus data using sophisticated statistical or mathematical methods. Still, on a single requirement (e.g., fairness) or specialize on a single type developers need to make sure that their software—whether learned of machine learning model (e.g., neural networks). or programmed—satisfies certain specified requirements. Currently, In this paper, we propose property-driven testing of machine two orthogonal approaches can be followed to achieve this goal: (A) learning models. Our approach MLCheck encompasses (1) a lan- employing an ML algorithm guaranteeing some requirement per guage for property specification, and (2) a technique for system- design, or (B) validating the requirement on the model generated atic test case generation.
    [Show full text]
  • Hello, It's GPT-2
    Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems Paweł Budzianowski1;2;3 and Ivan Vulic´2;3 1Engineering Department, Cambridge University, UK 2Language Technology Lab, Cambridge University, UK 3PolyAI Limited, London, UK [email protected], [email protected] Abstract (Young et al., 2013). On the other hand, open- domain conversational bots (Li et al., 2017; Serban Data scarcity is a long-standing and crucial et al., 2017) can leverage large amounts of freely challenge that hinders quick development of available unannotated data (Ritter et al., 2010; task-oriented dialogue systems across multiple domains: task-oriented dialogue models are Henderson et al., 2019a). Large corpora allow expected to learn grammar, syntax, dialogue for training end-to-end neural models, which typ- reasoning, decision making, and language gen- ically rely on sequence-to-sequence architectures eration from absurdly small amounts of task- (Sutskever et al., 2014). Although highly data- specific data. In this paper, we demonstrate driven, such systems are prone to producing unre- that recent progress in language modeling pre- liable and meaningless responses, which impedes training and transfer learning shows promise their deployment in the actual conversational ap- to overcome this problem. We propose a task- oriented dialogue model that operates solely plications (Li et al., 2017). on text input: it effectively bypasses ex- Due to the unresolved issues with the end-to- plicit policy and language generation modules. end architectures, the focus has been extended to Building on top of the TransferTransfo frame- retrieval-based models.
    [Show full text]
  • Persian Handwritten Digit Recognition Using Combination of Convolutional Neural Network and Support Vector Machine Methods
    572 The International Arab Journal of Information Technology, Vol. 17, No. 4, July 2020 Persian Handwritten Digit Recognition Using Combination of Convolutional Neural Network and Support Vector Machine Methods Mohammad Parseh, Mohammad Rahmanimanesh, and Parviz Keshavarzi Faculty of Electrical and Computer Engineering, Semnan University, Iran Abstract: Persian handwritten digit recognition is one of the important topics of image processing which significantly considered by researchers due to its many applications. The most important challenges in Persian handwritten digit recognition is the existence of various patterns in Persian digit writing that makes the feature extraction step to be more complicated.Since the handcraft feature extraction methods are complicated processes and their performance level are not stable, most of the recent studies have concentrated on proposing a suitable method for automatic feature extraction. In this paper, an automatic method based on machine learning is proposed for high-level feature extraction from Persian digit images by using Convolutional Neural Network (CNN). After that, a non-linear multi-class Support Vector Machine (SVM) classifier is used for data classification instead of fully connected layer in final layer of CNN. The proposed method has been applied to HODA dataset and obtained 99.56% of recognition rate. Experimental results are comparable with previous state-of-the-art methods. Keywords: Handwritten Digit Recognition, Convolutional Neural Network, Support Vector Machine. Received January 1, 2019; accepted November 11, 2019 https://doi.org/10.34028/iajit/17/4/16 1. Introduction years, they are good alternatives to handcraft feature extraction method. Optical Character Recognition (OCR) is one of the attractive topics of Artificial Intelligence [3, 6, 15, 23, 24].
    [Show full text]
  • Hierarchical Multiscale Recurrent Neural Networks
    Published as a conference paper at ICLR 2017 HIERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS Junyoung Chung, Sungjin Ahn & Yoshua Bengio ∗ Département d’informatique et de recherche opérationnelle Université de Montréal {junyoung.chung,sungjin.ahn,yoshua.bengio}@umontreal.ca ABSTRACT Learning both hierarchical and temporal representation has been among the long- standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural network, that can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that the proposed model can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence generation. 1 INTRODUCTION One of the key principles of learning in deep neural networks as well as in the human brain is to obtain a hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015; Schmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize the target task, make deep neural networks entertain advantages such as generalization to unseen examples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering disentangling factors of variation (Kingma & Welling, 2013).
    [Show full text]
  • Dynamic Factor Graphs for Time Series Modeling
    Dynamic Factor Graphs for Time Series Modeling Piotr Mirowski and Yann LeCun Courant Institute of Mathematical Sciences, New York University, 719 Broadway, New York, NY 10003 USA {mirowski,yann}@cs.nyu.edu http://cs.nyu.edu/∼mirowski/ Abstract. This article presents a method for training Dynamic Fac- tor Graphs (DFG) with continuous latent state variables. A DFG in- cludes factors modeling joint probabilities between hidden and observed variables, and factors modeling dynamical constraints on hidden vari- ables. The DFG assigns a scalar energy to each configuration of hidden and observed variables. A gradient-based inference procedure finds the minimum-energy state sequence for a given observation sequence. Be- cause the factors are designed to ensure a constant partition function, they can be trained by minimizing the expected energy over training sequences with respect to the factors’ parameters. These alternated in- ference and parameter updates can be seen as a deterministic EM-like procedure. Using smoothing regularizers, DFGs are shown to reconstruct chaotic attractors and to separate a mixture of independent oscillatory sources perfectly. DFGs outperform the best known algorithm on the CATS competition benchmark for time series prediction. DFGs also suc- cessfully reconstruct missing motion capture data. Key words: factor graphs, time series, dynamic Bayesian networks, re- current networks, expectation-maximization 1 Introduction 1.1 Background Time series collected from real-world phenomena are often an incomplete picture of a complex underlying dynamical process with a high-dimensional state that cannot be directly observed. For example, human motion capture data gives the positions of a few markers that are the reflection of a large number of joint angles with complex kinematic and dynamical constraints.
    [Show full text]