Curriculum Vitae
Total Page：16
File Type：pdf, Size：1020Kb
Load more
Recommended publications
-
NVIDIA CEO Jensen Huang to Host AI Pioneers Yoshua Bengio, Geoffrey Hinton and Yann Lecun, and Others, at GTC21
NVIDIA CEO Jensen Huang to Host AI Pioneers Yoshua Bengio, Geoffrey Hinton and Yann LeCun, and Others, at GTC21 Online Conference to Feature Jensen Huang Keynote and 1,300 Talks from Leaders in Data Center, Networking, Graphics and Autonomous Vehicles NVIDIA today announced that its CEO and founder Jensen Huang will host renowned AI pioneers Yoshua Bengio, Geoffrey Hinton and Yann LeCun at the company’s upcoming technology conference, GTC21, running April 12-16. The event will kick off with a news-filled livestreamed keynote by Huang on April 12 at 8:30 am Pacific. Bengio, Hinton and LeCun won the 2018 ACM Turing Award, known as the Nobel Prize of computing, for breakthroughs that enabled the deep learning revolution. Their work underpins the proliferation of AI technologies now being adopted around the world, from natural language processing to autonomous machines. Bengio is a professor at the University of Montreal and head of Mila - Quebec Artificial Intelligence Institute; Hinton is a professor at the University of Toronto and a researcher at Google; and LeCun is a professor at New York University and chief AI scientist at Facebook. More than 100,000 developers, business leaders, creatives and others are expected to register for GTC, including CxOs and IT professionals focused on data center infrastructure. Registration is free and is not required to view the keynote. In addition to the three Turing winners, major speakers include: Girish Bablani, Corporate Vice President, Microsoft Azure John Bowman, Director of Data Science, Walmart -
On Recurrent and Deep Neural Networks
On Recurrent and Deep Neural Networks Razvan Pascanu Advisor: Yoshua Bengio PhD Defence Universit´ede Montr´eal,LISA lab September 2014 Pascanu On Recurrent and Deep Neural Networks 1/ 38 Studying the mechanism behind learning provides a meta-solution for solving tasks. Motivation \A computer once beat me at chess, but it was no match for me at kick boxing" | Emo Phillips Pascanu On Recurrent and Deep Neural Networks 2/ 38 Motivation \A computer once beat me at chess, but it was no match for me at kick boxing" | Emo Phillips Studying the mechanism behind learning provides a meta-solution for solving tasks. Pascanu On Recurrent and Deep Neural Networks 2/ 38 I fθ(x) = f (θ; x) ? F I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Supervised Learing I f :Θ D T F × ! Pascanu On Recurrent and Deep Neural Networks 3/ 38 ? I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Supervised Learing I f :Θ D T F × ! I fθ(x) = f (θ; x) F Pascanu On Recurrent and Deep Neural Networks 3/ 38 Supervised Learing I f :Θ D T F × ! I fθ(x) = f (θ; x) ? F I f = arg minθ Θ EEx;t π [d(fθ(x); t)] 2 ∼ Pascanu On Recurrent and Deep Neural Networks 3/ 38 Optimization for learning θ[k+1] θ[k] Pascanu On Recurrent and Deep Neural Networks 4/ 38 Neural networks Output neurons Last hidden layer bias = 1 Second hidden layer First hidden layer Input layer Pascanu On Recurrent and Deep Neural Networks 5/ 38 Recurrent neural networks Output neurons Output neurons Last hidden layer bias = 1 bias = 1 Recurrent Layer Second hidden layer First hidden layer Input layer Input layer (b) Recurrent -
Generative Adversarial Networks (Gans)
Generative Adversarial Networks (GANs) The coolest idea in Machine Learning in the last twenty years - Yann Lecun Generative Adversarial Networks (GANs) 1 / 39 Overview Generative Adversarial Networks (GANs) 3D GANs Domain Adaptation Generative Adversarial Networks (GANs) 2 / 39 Introduction Generative Adversarial Networks (GANs) 3 / 39 Supervised Learning Find deterministic function f: y = f(x), x:data, y:label Generative Adversarial Networks (GANs) 4 / 39 Unsupervised Learning "Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we do not know how to make the cake. We need to solve the unsupervised learning problem before we can even think of getting to true AI." - Yann Lecun "You cannot predict what you cannot understand" - Anonymous Generative Adversarial Networks (GANs) 5 / 39 Unsupervised Learning More challenging than supervised learning. No label or curriculum. Some NN solutions: Boltzmann machine AutoEncoder Generative Adversarial Networks Generative Adversarial Networks (GANs) 6 / 39 Unsupervised Learning vs Generative Model z = f(x) vs. x = g(z) P(z|x) vs P(x|z) Generative Adversarial Networks (GANs) 7 / 39 Autoencoders Stacked Autoencoders Use data itself as label Generative Adversarial Networks (GANs) 8 / 39 Autoencoders Denosing Autoencoders Generative Adversarial Networks (GANs) 9 / 39 Variational Autoencoder Generative Adversarial Networks (GANs) 10 / 39 Variational Autoencoder Results Generative Adversarial Networks (GANs) 11 / 39 Generative Adversarial Networks Ian Goodfellow et al, "Generative Adversarial Networks", 2014. -
Yoshua Bengio and Gary Marcus on the Best Way Forward for AI
Yoshua Bengio and Gary Marcus on the Best Way Forward for AI Transcript of the 23 December 2019 AI Debate, hosted at Mila Moderated and transcribed by Vincent Boucher, Montreal AI AI DEBATE : Yoshua Bengio | Gary Marcus — Organized by MONTREAL.AI and hosted at Mila, on Monday, December 23, 2019, from 6:30 PM to 8:30 PM (EST) Slides, video, readings and more can be found on the MONTREAL.AI debate webpage. Transcript of the AI Debate Opening Address | Vincent Boucher Good Evening from Mila in Montreal Ladies & Gentlemen, Welcome to the “AI Debate”. I am Vincent Boucher, Founding Chairman of Montreal.AI. Our participants tonight are Professor GARY MARCUS and Professor YOSHUA BENGIO. Professor GARY MARCUS is a Scientist, Best-Selling Author, and Entrepreneur. Professor MARCUS has published extensively in neuroscience, genetics, linguistics, evolutionary psychology and artificial intelligence and is perhaps the youngest Professor Emeritus at NYU. He is Founder and CEO of Robust.AI and the author of five books, including The Algebraic Mind. His newest book, Rebooting AI: Building Machines We Can Trust, aims to shake up the field of artificial intelligence and has been praised by Noam Chomsky, Steven Pinker and Garry Kasparov. Professor YOSHUA BENGIO is a Deep Learning Pioneer. In 2018, Professor BENGIO was the computer scientist who collected the largest number of new citations worldwide. In 2019, he received, jointly with Geoffrey Hinton and Yann LeCun, the ACM A.M. Turing Award — “the Nobel Prize of Computing”. He is the Founder and Scientific Director of Mila — the largest university-based research group in deep learning in the world. -
Hello, It's GPT-2
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems Paweł Budzianowski1;2;3 and Ivan Vulic´2;3 1Engineering Department, Cambridge University, UK 2Language Technology Lab, Cambridge University, UK 3PolyAI Limited, London, UK [email protected], [email protected] Abstract (Young et al., 2013). On the other hand, open- domain conversational bots (Li et al., 2017; Serban Data scarcity is a long-standing and crucial et al., 2017) can leverage large amounts of freely challenge that hinders quick development of available unannotated data (Ritter et al., 2010; task-oriented dialogue systems across multiple domains: task-oriented dialogue models are Henderson et al., 2019a). Large corpora allow expected to learn grammar, syntax, dialogue for training end-to-end neural models, which typ- reasoning, decision making, and language gen- ically rely on sequence-to-sequence architectures eration from absurdly small amounts of task- (Sutskever et al., 2014). Although highly data- specific data. In this paper, we demonstrate driven, such systems are prone to producing unre- that recent progress in language modeling pre- liable and meaningless responses, which impedes training and transfer learning shows promise their deployment in the actual conversational ap- to overcome this problem. We propose a task- oriented dialogue model that operates solely plications (Li et al., 2017). on text input: it effectively bypasses ex- Due to the unresolved issues with the end-to- plicit policy and language generation modules. end architectures, the focus has been extended to Building on top of the TransferTransfo frame- retrieval-based models. -
Hierarchical Multiscale Recurrent Neural Networks
Published as a conference paper at ICLR 2017 HIERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS Junyoung Chung, Sungjin Ahn & Yoshua Bengio ∗ Département d’informatique et de recherche opérationnelle Université de Montréal {junyoung.chung,sungjin.ahn,yoshua.bengio}@umontreal.ca ABSTRACT Learning both hierarchical and temporal representation has been among the long- standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural network, that can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that the proposed model can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence generation. 1 INTRODUCTION One of the key principles of learning in deep neural networks as well as in the human brain is to obtain a hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015; Schmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize the target task, make deep neural networks entertain advantages such as generalization to unseen examples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering disentangling factors of variation (Kingma & Welling, 2013). -
Radford M. Neal Curriculum Vitae, 25 April 2021
Radford M. Neal Curriculum Vitae, 25 April 2021 Biographical Information Personal Data Date of Birth: 12 Sept 1956 Citizenship: Canadian Home address: University address: 21 Shaftesbury Avenue, #106 Department of Statistical Sciences Toronto, Ontario M4T 3B4 University of Toronto M5S 3G3 U of T web page: http://www.cs.utoronto.ca/∼radford/ Blog (not affiliated with U of T): https://radfordneal.wordpress.com E-mail addresses: [email protected], [email protected] Degrees Awarded BSc (Honours) in Computer Science, University of Calgary, 1977 MSc in Computer Science, University of Calgary, 1980 Thesis title: An Editor for Trees Supervisor: David Hill Ph.D in Computer Science, University of Toronto, 1995 Thesis title: Bayesian Learning for Neural Networks Supervisor: Geoffrey Hinton Employment and Other Relevant Experience Present position: Professor Emeritus, Department of Statistical Sciences and Department of Computer Science, University of Toronto, Appointed as Lecturer 1 July 1994. Appointed as Associate Member (Limited Term) of the graduate faculty 13 January 1995. Promoted to Assistant Professor 1 July 1995. Promoted to Associate Professor and full member of the graduate faculty 1 July 1999. Promoted fo Full Professor 1 July 2001. Awarded a Canada Research Chair in Statistics and Machine Learning 1 July 2003 (held to 31 December 2016). Cross-appointed to the Dalla Lana School of Public Health 14 February 2006. Retired as of 1 January 2017. Earlier teaching experience: Sessional instructor in computer science at the University of Calgary for the following courses: 1 Summers 1979-1981 Taught half of the core second-year course on machine archi- tecture and programming. -
Dynamic Factor Graphs for Time Series Modeling
Dynamic Factor Graphs for Time Series Modeling Piotr Mirowski and Yann LeCun Courant Institute of Mathematical Sciences, New York University, 719 Broadway, New York, NY 10003 USA {mirowski,yann}@cs.nyu.edu http://cs.nyu.edu/∼mirowski/ Abstract. This article presents a method for training Dynamic Fac- tor Graphs (DFG) with continuous latent state variables. A DFG in- cludes factors modeling joint probabilities between hidden and observed variables, and factors modeling dynamical constraints on hidden vari- ables. The DFG assigns a scalar energy to each configuration of hidden and observed variables. A gradient-based inference procedure finds the minimum-energy state sequence for a given observation sequence. Be- cause the factors are designed to ensure a constant partition function, they can be trained by minimizing the expected energy over training sequences with respect to the factors’ parameters. These alternated in- ference and parameter updates can be seen as a deterministic EM-like procedure. Using smoothing regularizers, DFGs are shown to reconstruct chaotic attractors and to separate a mixture of independent oscillatory sources perfectly. DFGs outperform the best known algorithm on the CATS competition benchmark for time series prediction. DFGs also suc- cessfully reconstruct missing motion capture data. Key words: factor graphs, time series, dynamic Bayesian networks, re- current networks, expectation-maximization 1 Introduction 1.1 Background Time series collected from real-world phenomena are often an incomplete picture of a complex underlying dynamical process with a high-dimensional state that cannot be directly observed. For example, human motion capture data gives the positions of a few markers that are the reflection of a large number of joint angles with complex kinematic and dynamical constraints. -
An Empirical Study on Evaluation Metrics of Generativeadversarialnetworks
Under review as a conference paper at ICLR 2018 AN EMPIRICAL STUDY ON EVALUATION METRICS OF GENERATIVE ADVERSARIAL NETWORKS Anonymous authors Paper under double-blind review ABSTRACT Despite the widespread interest in generative adversarial networks (GANs), few works have studied the metrics that quantitatively evaluate GANs’ performance. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the important problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. Then with a series of carefully designed experiments, we are able to comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far these state-of-the-art GANs are from perfect. 1 INTRODUCTION Generative adversarial networks (GANs) (Goodfellow et al., 2014) have been studied extensively in recent years. Besides producing surprisingly plausible images of faces (Radford et al., 2015; Larsen et al., 2015) and bedrooms (Radford et al., 2015; Arjovsky et al., 2017; Gulrajani et al., 2017), they have also been innovatively applied in, for example, semi-supervised learning (Odena, 2016; Makhzani et al., 2015), image-to-image translation (Isola et al., 2016; Zhu et al., 2017), and simulated image refinement (Shrivastava et al., 2016). -
Generative Linguistics and Neural Networks at 60: Foundation, Friction, and Fusion*
Generative linguistics and neural networks at 60: foundation, friction, and fusion* Joe Pater, University of Massachusetts Amherst October 3, 2018. Abstract. The birthdate of both generative linguistics and neural networks can be taken as 1957, the year of the publication of foundational work by both Noam Chomsky and Frank Rosenblatt. This paper traces the development of these two approaches to cognitive science, from their largely autonomous early development in their first thirty years, through their collision in the 1980s around the past tense debate (Rumelhart and McClelland 1986, Pinker and Prince 1988), and their integration in much subsequent work up to the present. Although this integration has produced a considerable body of results, the continued general gulf between these two lines of research is likely impeding progress in both: on learning in generative linguistics, and on the representation of language in neural modeling. The paper concludes with a brief argument that generative linguistics is unlikely to fulfill its promise of accounting for language learning if it continues to maintain its distance from neural and statistical approaches to learning. 1. Introduction At the beginning of 1957, two men nearing their 29th birthdays published work that laid the foundation for two radically different approaches to cognitive science. One of these men, Noam Chomsky, continues to contribute sixty years later to the field that he founded, generative linguistics. The book he published in 1957, Syntactic Structures, has been ranked as the most influential work in cognitive science from the 20th century.1 The other one, Frank Rosenblatt, had by the late 1960s largely moved on from his research on perceptrons – now called neural networks – and died tragically young in 1971. -
Energy-Based Generative Adversarial Network
Published as a conference paper at ICLR 2017 ENERGY-BASED GENERATIVE ADVERSARIAL NET- WORKS Junbo Zhao, Michael Mathieu and Yann LeCun Department of Computer Science, New York University Facebook Artificial Intelligence Research fjakezhao, mathieu, [email protected] ABSTRACT We introduce the “Energy-based Generative Adversarial Network” model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discrimi- nator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the dis- criminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images. 1I NTRODUCTION 1.1E NERGY-BASED MODEL The essence of the energy-based model (LeCun et al., 2006) is to build a function that maps each point of an input space to a single scalar, which is called “energy”. The learning phase is a data- driven process that shapes the energy surface in such a way that the desired configurations get as- signed low energies, while the incorrect ones are given high energies. -
Neural Networks and the Distorted Automation of Intelligence As Statistical Inference Matteo Pasquinelli
JOURNAL > SITE 1: LOGIC GATE, THE POLITICS OF THE ARTIFACTUAL MIND | 2017 Machines that Morph Logic: Neural Networks and the Distorted Automation of Intelligence as Statistical Inference Matteo Pasquinelli Perceptrons [artificial neural networks] are not intended to serve as detailed copies of any actual nervous system. They are simplified networks, designed to permit the study of lawful relationships between the organization of a nerve net, the organization of its environment, and the “psychological” performances of which the network is capable. Perceptrons might actually correspond to parts of more extended networks in biological systems… More likely, they represent extreme simplifications of the central nervous system, in which some properties are exaggerated, others suppressed. —Frank Rosenblatt1 No algorithm exists for the metaphor, nor can a metaphor be produced by means of a computer’s precise instructions, no matter what the volume of organized information to be fed in. —Umberto Eco2 The term Artificial Intelligence is often cited in popular press as well as in art and philosophy circles as an alchemic talisman whose functioning is rarely explained. The hegemonic paradigm to date (also crucial to the automation of labor) is not based on GOFAI (Good Old-Fashioned Artificial Intelligence that never succeeded at automating symbolic deduction), but on the neural networks designed by Frank Rosenblatt back in 1958 to automate statistical induction. The text highlights the role of logic gates in the Machines that Morph Logic: Neural Networks and the Distorted Automation of Intelligence as Statistical Inference | Matteo Pasquinelli distributed architecture of neural networks, in which a generalized control loop affects each node of computation to perform pattern recognition.