DOCSLIB.ORG

Deep Learning - Review Yann Lecun, Yoshua Bengio & Geoffrey Hinton

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

DEEP LEARNING - REVIEW YANN LECUN, YOSHUA BENGIO & GEOFFREY HINTON. OUTLINE • Deep Learning - History, Background & Applications. • Recent Revival. • Convolutional Neural Networks. • Recurrent Neural Networks. • Future. WHAT IS DEEP LEARNING? • A particular class of Learning Algorithms. • Rebranded Neural Networks : With multiple layers. • Inspired by the Neuronal architecture of the Brain. • Renewed interest in the area due to a few recent breakthroughs. • Learn parameters from data. • Non Linear Classification. SOME CONTEXT HISTORY • 1943 - McCulloch & Pitts develop computational model for neural network. Idea: neurons with a binary threshold activation function were analogous to first order logic sentences. • 1949 - Donald Hebb proposes Hebb’s rule Idea: Neurons that fire together, wire together! • 1958 - Frank Rosenblatt creates the Perceptron. • 1959 - Hubel and Wiesel elaborate cells in Visual Cortex. • 1975 - Paul J. Werbos develops the Backpropagation Algorithm. • 1980 - Neocognitron, a hierarchical multilayered ANN. • 1990 - Convolutional Neural Networks. APPLICATIONS • Predict the activity of potential drug molecules. • Reconstruct Brain circuits. • Predict effects of mutation on non-coding regions of DNA. • Speech/Image Recognition & Language translation MULTILAYER NEURAL NETWORK COMMON NON- LINEAR FUNCTIONS: 1) F ( Z ) = MAX(0,Z ) 2) SIGMOID 3)LOGISTIC COST FUNCTION: 1/2[(YL - TL)]2 Source : Deep learning Yann LeCun, Yoshua Bengio, Geoffrey Hinton Nature 521, 436–444 (28 May 2015) doi:10.1038/nature14539 STOCHASTIC GRADIENT DESCENT • Analogy: A person is stuck in the mountains and is trying to get down (i.e. trying to find the minima). • SGD: The person represents the backpropagation algorithm, and the path taken down the mountain represents the sequence of parameter settings that the algorithm will explore. The steepness of the hill represents the slope of the error surface at that point. The instrument used to measure steepness is differentiation (the slope of the error surface can be calculated by taking the derivative of the squared error function at that point). The direction he chooses to travel in aligns with the gradient of the error surface at that point. The amount of time he travels before taking another measurement is the learning rate of the algorithm. Source: http://sebastianraschka.com/images/faq/closed-form-vs-gd/ball.png; Wikipedia. BACK PROPAGATION COST FUNCTION: Error 2 =1/2[(yl - tl)] Source : Deep learning Yann LeCun, Yoshua Bengio, Geoffrey Hinton Nature 521, 436–444 (28 May 2015) doi:10.1038/nature14539 WHY ALL THE BUZZ? ImageNet: • ~5M labeled high resolution images. • Roughly 22K categories. • Collected from web & labeled by Amazon Mechanical Turk http://vision.stanford.edu/teaching/cs231b_spring1415/slides/alexnet_tugce_kyunghee.pdf DIMINISHING ERROR RATES CONVOLUTION NEURAL NETWORKS- CORE IDEA • Color Image - 32 x 32 pixels on 3 color palettes. • Pixel Intensity - 0 - 255. • Image Representation : 32 * 32 * 3 array of numbers with each pixel ranging between 0 and 255. • Idea : Feed the numerical array to a ConvNet and obtain probabilities for each class of objects as an N dimensional vector, where N is the number of classes. CONVOLUTIONAL NEURAL NETS • Multi stage Neural Nets that model V1,V2,V3 areas of the visual cortex. • (Convolutional Layer + NL Layer + Pooling Layer)^n + Fully Connected Layer. • Highly correlated local values are easily detected. • Ideal for volumetric data that come in multiple arrays. e.g., Color images. • Learn the ‘essence' of images well. • Applications in Computer Vision INITIAL WORK - YAN LECUN • Primitive recognition without hand coded features. • Adaptive, yet constrained architecture. • Hand written digit recognition served as a simple and powerful model. • Training Sample : 9298 zip codes on mails passing through Buffalo, NY. CONCEPTUAL OVERVIEW Source : Deep learning Yann LeCun, Yoshua Bengio, Geoffrey Hinton Nature 521, 436–444 (28 May 2015) doi:10.1038/nature14539 WHAT IS A CONVOLUTION? • Several meanings depending on the area of application. • Convolution - Operation of applying filters/kernels through overlapping regions of the image. • Stride - Extent of overlap during convolution. • Each filter has the same set of weights and biases. This minimizes the number of parameters. Sources: http://ufldl.stanford.edu/ http://www.kdnuggets.com/2016/09/beginners-guide-understanding-convolutional-neural- networks-part-1.html FILTERS/KERNELS • Filters - Carefully designed feature detectors (matrices) to detect edges, curves, colors etc. • Receptive field - Area covered by a single filter. • 3*3 and 5*5 are the common sizes. • Alexnet used 96 Kernels on the input layer. http://web.pdx.edu/~jduh/courses/Archive/geog481w07/Students/Ludwig_ImageConvolution.pdf RELU & POOLING LAYERS • RELU - Applies non-linear activation function MAX(0,x) to every pixel. Other common functions include tanh and sigmoid. • RELU - Addresses the ‘vanishing gradient problem’. • Pooling - Reduces the spatial size and minimizes overfitting. • MAX 2x2 is the most common pooling operation. • Dropout - Random elimination of neurons to minimize overfitting. • Pooling Vs. Larger Strides. NON LINEARITIES- TREND https://arxiv.org/pdf/1606.02228.pdf ALEXNET - PERSPECTIVE http://vision.stanford.edu/teaching/cs231b_spring1415/slides/alexnet_tugce_kyunghee.pdf CONVNETS - THE BIG PICTURE RECURRENT NEURAL NETWORKS - RNN • RNN - Neural Nets with feedback loops. Multiple copies of the same network, each passing a message to a successor • Used to train sequential inputs. i.e., Speech, DNA sequences etc. • Operate over sequences of vectors. Predict next character, word etc. http://colah.github.io/posts/2015-08-Understanding-LSTMs/ LONG SHORT TERM MEMORY (LSTM) • Sequences have long term dependencies. Why? “the clouds are in the sky” vs. “I grew up in France… I speak fluent French.” • Problem: Hard to store information for very long. • Solution: LTSTM http://colah.github.io/posts/2015-08-Understanding-LSTMs/ EXAMPLE BY ANDREJ KARPATHY • Source : 474MB of C code from Github • Multiple 3-layer LSTMs; • Few days of training on GPUs • Parameters - 10 Million. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ RNNS AND BEYOND • RNNs augmented with Memory Networks: 1. Improved Performance. 2. Applications in question answering systems. • ConvNets + RNNs = Novel Applications FUTURE - DEEP LEARNING • Extension of recent successes from to Unsupervised Learning. • End to End Integration :Reinforcement + Convnets + RNNs • Natural language understanding. • Complex systems that combine learning, memory and reasoning. Source: https://developer.amazon.com/alexaprize REFERENCES • Andrej Karpathy’s Course: http://cs231n.stanford.edu/ • DeepLearning.tv : https://www.youtube.com/channel/UC9OeZkIwhzfv-_Cb7fCikLQ • Wikipedia!.
Recommended publications
  • Backpropagation and Deep Learning in the Brain

    Backpropagation and Deep Learning in the Brain

    Backpropagation and Deep Learning in the Brain Simons Institute -- Computational Theories of the Brain 2018 Timothy Lillicrap DeepMind, UCL With: Sergey Bartunov, Adam Santoro, Jordan Guerguiev, Blake Richards, Luke Marris, Daniel Cownden, Colin Akerman, Douglas Tweed, Geoffrey Hinton The “credit assignment” problem The solution in artificial networks: backprop Credit assignment by backprop works well in practice and shows up in virtually all of the state-of-the-art supervised, unsupervised, and reinforcement learning algorithms. Why Isn’t Backprop “Biologically Plausible”? Why Isn’t Backprop “Biologically Plausible”? Neuroscience Evidence for Backprop in the Brain? A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: How to convince a neuroscientist that the cortex is learning via [something like] backprop - To convince a machine learning researcher, an appeal to variance in gradient estimates might be enough. - But this is rarely enough to convince a neuroscientist. - So what lines of argument help? How to convince a neuroscientist that the cortex is learning via [something like] backprop - What do I mean by “something like backprop”?: - That learning is achieved across multiple layers by sending information from neurons closer to the output back to “earlier” layers to help compute their synaptic updates. How to convince a neuroscientist that the cortex is learning via [something like] backprop 1. Feedback connections in cortex are ubiquitous and modify the
  • NVIDIA CEO Jensen Huang to Host AI Pioneers Yoshua Bengio, Geoffrey Hinton and Yann Lecun, and Others, at GTC21

    NVIDIA CEO Jensen Huang to Host AI Pioneers Yoshua Bengio, Geoffrey Hinton and Yann Lecun, and Others, at GTC21

    NVIDIA CEO Jensen Huang to Host AI Pioneers Yoshua Bengio, Geoffrey Hinton and Yann LeCun, and Others, at GTC21 Online Conference to Feature Jensen Huang Keynote and 1,300 Talks from Leaders in Data Center, Networking, Graphics and Autonomous Vehicles NVIDIA today announced that its CEO and founder Jensen Huang will host renowned AI pioneers Yoshua Bengio, Geoffrey Hinton and Yann LeCun at the company’s upcoming technology conference, GTC21, running April 12-16. The event will kick off with a news-filled livestreamed keynote by Huang on April 12 at 8:30 am Pacific. Bengio, Hinton and LeCun won the 2018 ACM Turing Award, known as the Nobel Prize of computing, for breakthroughs that enabled the deep learning revolution. Their work underpins the proliferation of AI technologies now being adopted around the world, from natural language processing to autonomous machines. Bengio is a professor at the University of Montreal and head of Mila - Quebec Artificial Intelligence Institute; Hinton is a professor at the University of Toronto and a researcher at Google; and LeCun is a professor at New York University and chief AI scientist at Facebook. More than 100,000 developers, business leaders, creatives and others are expected to register for GTC, including CxOs and IT professionals focused on data center infrastructure. Registration is free and is not required to view the keynote. In addition to the three Turing winners, major speakers include: Girish Bablani, Corporate Vice President, Microsoft Azure John Bowman, Director of Data Science, Walmart
  • The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design

    The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design

    The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design Jeffrey Dean Google Research [email protected] Abstract The past decade has seen a remarkable series of advances in machine learning, and in particular deep learning approaches based on artificial neural networks, to improve our abilities to build more accurate systems across a broad range of areas, including computer vision, speech recognition, language translation, and natural language understanding tasks. This paper is a companion paper to a keynote talk at the 2020 International Solid-State Circuits Conference (ISSCC) discussing some of the advances in machine learning, and their implications on the kinds of computational devices we need to build, especially in the post-Moore’s Law-era. It also discusses some of the ways that machine learning may also be able to help with some aspects of the circuit design process. Finally, it provides a sketch of at least one interesting direction towards much larger-scale multi-task models that are sparsely activated and employ much more dynamic, example- and task-based routing than the machine learning models of today. Introduction The past decade has seen a remarkable series of advances in machine learning (ML), and in particular deep learning approaches based on artificial neural networks, to improve our abilities to build more accurate systems across a broad range of areas [LeCun et al. 2015]. Major areas of significant advances ​ ​ include computer vision [Krizhevsky et al. 2012, Szegedy et al. 2015, He et al. 2016, Real et al. 2017, Tan ​ ​ ​ ​ ​ ​ and Le 2019], speech recognition [Hinton et al.
  • On Recurrent and Deep Neural Networks

    On Recurrent and Deep Neural Networks

    On Recurrent and Deep Neural Networks Razvan Pascanu Advisor: Yoshua Bengio PhD Defence Universit´ede Montr´eal,LISA lab September 2014 Pascanu On Recurrent and Deep Neural Networks 1/ 38 Studying the mechanism behind learning provides a meta-solution for solving tasks. Motivation \A computer once beat me at chess, but it was no match for me at kick boxing" | Emo Phillips Pascanu On Recurrent and Deep Neural Networks 2/ 38 Motivation \A computer once beat me at chess, but it was no match for me at kick boxing" | Emo Phillips Studying the mechanism behind learning provides a meta-solution for solving tasks. Pascanu On Recurrent and Deep Neural Networks 2/ 38 I fθ(x) = f (θ; x) ? F I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Supervised Learing I f :Θ D T F × ! Pascanu On Recurrent and Deep Neural Networks 3/ 38 ? I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Supervised Learing I f :Θ D T F × ! I fθ(x) = f (θ; x) F Pascanu On Recurrent and Deep Neural Networks 3/ 38 Supervised Learing I f :Θ D T F × ! I fθ(x) = f (θ; x) ? F I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Pascanu On Recurrent and Deep Neural Networks 3/ 38 Optimization for learning θ[k+1] θ[k] Pascanu On Recurrent and Deep Neural Networks 4/ 38 Neural networks Output neurons Last hidden layer bias = 1 Second hidden layer First hidden layer Input layer Pascanu On Recurrent and Deep Neural Networks 5/ 38 Recurrent neural networks Output neurons Output neurons Last hidden layer bias = 1 bias = 1 Recurrent Layer Second hidden layer First hidden layer Input layer Input layer (b) Recurrent
  • Generative Adversarial Networks (Gans)

    Generative Adversarial Networks (Gans)

    Generative Adversarial Networks (GANs) The coolest idea in Machine Learning in the last twenty years - Yann Lecun Generative Adversarial Networks (GANs) 1 / 39 Overview Generative Adversarial Networks (GANs) 3D GANs Domain Adaptation Generative Adversarial Networks (GANs) 2 / 39 Introduction Generative Adversarial Networks (GANs) 3 / 39 Supervised Learning Find deterministic function f: y = f(x), x:data, y:label Generative Adversarial Networks (GANs) 4 / 39 Unsupervised Learning "Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we do not know how to make the cake. We need to solve the unsupervised learning problem before we can even think of getting to true AI." - Yann Lecun "You cannot predict what you cannot understand" - Anonymous Generative Adversarial Networks (GANs) 5 / 39 Unsupervised Learning More challenging than supervised learning. No label or curriculum. Some NN solutions: Boltzmann machine AutoEncoder Generative Adversarial Networks Generative Adversarial Networks (GANs) 6 / 39 Unsupervised Learning vs Generative Model z = f(x) vs. x = g(z) P(z|x) vs P(x|z) Generative Adversarial Networks (GANs) 7 / 39 Autoencoders Stacked Autoencoders Use data itself as label Generative Adversarial Networks (GANs) 8 / 39 Autoencoders Denosing Autoencoders Generative Adversarial Networks (GANs) 9 / 39 Variational Autoencoder Generative Adversarial Networks (GANs) 10 / 39 Variational Autoencoder Results Generative Adversarial Networks (GANs) 11 / 39 Generative Adversarial Networks Ian Goodfellow et al, "Generative Adversarial Networks", 2014.
  • ARCHITECTS of INTELLIGENCE for Xiaoxiao, Elaine, Colin, and Tristan ARCHITECTS of INTELLIGENCE

    ARCHITECTS of INTELLIGENCE for Xiaoxiao, Elaine, Colin, and Tristan ARCHITECTS of INTELLIGENCE

    MARTIN FORD ARCHITECTS OF INTELLIGENCE For Xiaoxiao, Elaine, Colin, and Tristan ARCHITECTS OF INTELLIGENCE THE TRUTH ABOUT AI FROM THE PEOPLE BUILDING IT MARTIN FORD ARCHITECTS OF INTELLIGENCE Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Acquisition Editors: Ben Renow-Clarke Project Editor: Radhika Atitkar Content Development Editor: Alex Sorrentino Proofreader: Safis Editing Presentation Designer: Sandip Tadge Cover Designer: Clare Bowyer Production Editor: Amit Ramadas Marketing Manager: Rajveer Samra Editorial Director: Dominic Shakeshaft First published: November 2018 Production reference: 2201118 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK ISBN 978-1-78913-151-2 www.packt.com Contents Introduction ........................................................................ 1 A Brief Introduction to the Vocabulary of Artificial Intelligence .......10 How AI Systems Learn ........................................................11 Yoshua Bengio .....................................................................17 Stuart J.
  • Yoshua Bengio and Gary Marcus on the Best Way Forward for AI

    Yoshua Bengio and Gary Marcus on the Best Way Forward for AI

    Yoshua Bengio and Gary Marcus on the Best Way Forward for AI Transcript of the 23 December 2019 AI Debate, hosted at Mila Moderated and transcribed by Vincent Boucher, Montreal AI AI DEBATE : Yoshua Bengio | Gary Marcus — Organized by MONTREAL.AI and hosted at Mila, on Monday, December 23, 2019, from 6:30 PM to 8:30 PM (EST) Slides, video, readings and more can be found on the MONTREAL.AI debate webpage. Transcript of the AI Debate Opening Address | Vincent Boucher Good Evening from Mila in Montreal Ladies & Gentlemen, Welcome to the “AI Debate”. I am Vincent Boucher, Founding Chairman of Montreal.AI. Our participants tonight are Professor GARY MARCUS and Professor YOSHUA BENGIO. Professor GARY MARCUS is a Scientist, Best-Selling Author, and Entrepreneur. Professor MARCUS has published extensively in neuroscience, genetics, linguistics, evolutionary psychology and artificial intelligence and is perhaps the youngest Professor Emeritus at NYU. He is Founder and CEO of Robust.AI and the author of five books, including The Algebraic Mind. His newest book, Rebooting AI: Building Machines We Can Trust, aims to shake up the field of artificial intelligence and has been praised by Noam Chomsky, Steven Pinker and Garry Kasparov. Professor YOSHUA BENGIO is a Deep Learning Pioneer. In 2018, Professor BENGIO was the computer scientist who collected the largest number of new citations worldwide. In 2019, he received, jointly with Geoffrey Hinton and Yann LeCun, the ACM A.M. Turing Award — “the Nobel Prize of Computing”. He is the Founder and Scientific Director of Mila — the largest university-based research group in deep learning in the world.
  • Hello, It's GPT-2

    Hello, It's GPT-2

    Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems Paweł Budzianowski1;2;3 and Ivan Vulic´2;3 1Engineering Department, Cambridge University, UK 2Language Technology Lab, Cambridge University, UK 3PolyAI Limited, London, UK [email protected], [email protected] Abstract (Young et al., 2013). On the other hand, open- domain conversational bots (Li et al., 2017; Serban Data scarcity is a long-standing and crucial et al., 2017) can leverage large amounts of freely challenge that hinders quick development of available unannotated data (Ritter et al., 2010; task-oriented dialogue systems across multiple domains: task-oriented dialogue models are Henderson et al., 2019a). Large corpora allow expected to learn grammar, syntax, dialogue for training end-to-end neural models, which typ- reasoning, decision making, and language gen- ically rely on sequence-to-sequence architectures eration from absurdly small amounts of task- (Sutskever et al., 2014). Although highly data- specific data. In this paper, we demonstrate driven, such systems are prone to producing unre- that recent progress in language modeling pre- liable and meaningless responses, which impedes training and transfer learning shows promise their deployment in the actual conversational ap- to overcome this problem. We propose a task- oriented dialogue model that operates solely plications (Li et al., 2017). on text input: it effectively bypasses ex- Due to the unresolved issues with the end-to- plicit policy and language generation modules. end architectures, the focus has been extended to Building on top of the TransferTransfo frame- retrieval-based models.
  • Fast Neural Network Emulation of Dynamical Systems for Computer Animation

    Fast Neural Network Emulation of Dynamical Systems for Computer Animation

    Fast Neural Network Emulation of Dynamical Systems for Computer Animation Radek Grzeszczuk 1 Demetri Terzopoulos 2 Geoffrey Hinton 2 1 Intel Corporation 2 University of Toronto Microcomputer Research Lab Department of Computer Science 2200 Mission College Blvd. 10 King's College Road Santa Clara, CA 95052, USA Toronto, ON M5S 3H5, Canada Abstract Computer animation through the numerical simulation of physics-based graphics models offers unsurpassed realism, but it can be computation­ ally demanding. This paper demonstrates the possibility of replacing the numerical simulation of nontrivial dynamic models with a dramatically more efficient "NeuroAnimator" that exploits neural networks. Neu­ roAnimators are automatically trained off-line to emulate physical dy­ namics through the observation of physics-based models in action. De­ pending on the model, its neural network emulator can yield physically realistic animation one or two orders of magnitude faster than conven­ tional numerical simulation. We demonstrate NeuroAnimators for a va­ riety of physics-based models. 1 Introduction Animation based on physical principles has been an influential trend in computer graphics for over a decade (see, e.g., [1, 2, 3]). This is not only due to the unsurpassed realism that physics-based techniques offer. In conjunction with suitable control and constraint mechanisms, physical models also facilitate the production of copious quantities of real­ istic animation in a highly automated fashion. Physics-based animation techniques are beginning to find their way into high-end commercial systems. However, a well-known drawback has retarded their broader penetration--compared to geometric models, physical models typically entail formidable numerical simulation costs. This paper proposes a new approach to creating physically realistic animation that differs Emulation for Animation 883 radically from the conventional approach of numerically simulating the equations of mo­ tion of physics-based models.
  • Hierarchical Multiscale Recurrent Neural Networks

    Hierarchical Multiscale Recurrent Neural Networks

    Published as a conference paper at ICLR 2017 HIERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS Junyoung Chung, Sungjin Ahn & Yoshua Bengio ∗ Département d’informatique et de recherche opérationnelle Université de Montréal {junyoung.chung,sungjin.ahn,yoshua.bengio}@umontreal.ca ABSTRACT Learning both hierarchical and temporal representation has been among the long- standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural network, that can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that the proposed model can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence generation. 1 INTRODUCTION One of the key principles of learning in deep neural networks as well as in the human brain is to obtain a hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015; Schmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize the target task, make deep neural networks entertain advantages such as generalization to unseen examples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering disentangling factors of variation (Kingma & Welling, 2013).
  • Neural Networks for Machine Learning Lecture 4A Learning To

    Neural Networks for Machine Learning Lecture 4A Learning To

    Neural Networks for Machine Learning Lecture 4a Learning to predict the next word Geoffrey Hinton with Nitish Srivastava Kevin Swersky A simple example of relational information Christopher = Penelope Andrew = Christine Margaret = Arthur Victoria = James Jennifer = Charles Colin Charlotte Roberto = Maria Pierro = Francesca Gina = Emilio Lucia = Marco Angela = Tomaso Alfonso Sophia Another way to express the same information • Make a set of propositions using the 12 relationships: – son, daughter, nephew, niece, father, mother, uncle, aunt – brother, sister, husband, wife • (colin has-father james) • (colin has-mother victoria) • (james has-wife victoria) this follows from the two above • (charlotte has-brother colin) • (victoria has-brother arthur) • (charlotte has-uncle arthur) this follows from the above A relational learning task • Given a large set of triples that come from some family trees, figure out the regularities. – The obvious way to express the regularities is as symbolic rules (x has-mother y) & (y has-husband z) => (x has-father z) • Finding the symbolic rules involves a difficult search through a very large discrete space of possibilities. • Can a neural network capture the same knowledge by searching through a continuous space of weights? The structure of the neural net local encoding of person 2 output distributed encoding of person 2 units that learn to predict features of the output from features of the inputs distributed encoding of person 1 distributed encoding of relationship local encoding of person 1 inputs local encoding of relationship Christopher = Penelope Andrew = Christine Margaret = Arthur Victoria = James Jennifer = Charles Colin Charlotte What the network learns • The six hidden units in the bottleneck connected to the input representation of person 1 learn to represent features of people that are useful for predicting the answer.
  • Dynamic Factor Graphs for Time Series Modeling

    Dynamic Factor Graphs for Time Series Modeling

    Dynamic Factor Graphs for Time Series Modeling Piotr Mirowski and Yann LeCun Courant Institute of Mathematical Sciences, New York University, 719 Broadway, New York, NY 10003 USA {mirowski,yann}@cs.nyu.edu http://cs.nyu.edu/∼mirowski/ Abstract. This article presents a method for training Dynamic Fac- tor Graphs (DFG) with continuous latent state variables. A DFG in- cludes factors modeling joint probabilities between hidden and observed variables, and factors modeling dynamical constraints on hidden vari- ables. The DFG assigns a scalar energy to each configuration of hidden and observed variables. A gradient-based inference procedure finds the minimum-energy state sequence for a given observation sequence. Be- cause the factors are designed to ensure a constant partition function, they can be trained by minimizing the expected energy over training sequences with respect to the factors’ parameters. These alternated in- ference and parameter updates can be seen as a deterministic EM-like procedure. Using smoothing regularizers, DFGs are shown to reconstruct chaotic attractors and to separate a mixture of independent oscillatory sources perfectly. DFGs outperform the best known algorithm on the CATS competition benchmark for time series prediction. DFGs also suc- cessfully reconstruct missing motion capture data. Key words: factor graphs, time series, dynamic Bayesian networks, re- current networks, expectation-maximization 1 Introduction 1.1 Background Time series collected from real-world phenomena are often an incomplete picture of a complex underlying dynamical process with a high-dimensional state that cannot be directly observed. For example, human motion capture data gives the positions of a few markers that are the reflection of a large number of joint angles with complex kinematic and dynamical constraints.