An Introduction to Recurrent Neural Networks
Total Page:16
File Type:pdf, Size:1020Kb
Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients An Introduction to Recurrent Neural Networks Alex Atanasov1 1Dept. of Physics Yale University April 24, 2018 Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Overview 1 Introduction to RNNs 2 Historical Background 3 Mathematical Formulation 4 Unrolling 5 Computing Gradients Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Throughout this presentation I will be using the notation from the book by Ian Goodfellow, Yoshua Bengio, and Aaron Courville Alex Atanasov VFU An Introduction to Recurrent Neural Networks Modeling of Neuronal Connectivity Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Alex Atanasov VFU An Introduction to Recurrent Neural Networks Modeling of Neuronal Connectivity Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Sequential Processing Alex Atanasov VFU An Introduction to Recurrent Neural Networks Modeling of Neuronal Connectivity Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Sequential Processing Alex Atanasov VFU An Introduction to Recurrent Neural Networks Modeling of Neuronal Connectivity Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Sequential Processing Alex Atanasov VFU An Introduction to Recurrent Neural Networks Modeling of Neuronal Connectivity Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Sequential Processing Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Modeling of Neuronal Connectivity Most human brains don't look like this: Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Modeling of Neuronal Connectivity Most human brains don't look like this: Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Modeling of Neuronal Connectivity Most human brains don't look like this1: Alex Atanasov VFU An Introduction to Recurrent Neural Networks 1Obligatory political reference here Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Modeling of Neuronal Connectivity Instead, we have neurons connecting in a dense web called a recurrent network Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Motivation Two motivations for recurrent neural network models: Modeling of Neuronal Connectivity Instead, we have neurons connecting in a dense web called a recurrent network Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients RNN Examples We can also combine RNNs with other networks we've seen before Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients RNN Examples Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients RNN Examples Alex Atanasov VFU An Introduction to Recurrent Neural Networks Designed for processing sequential data Like CNNs, motivated by biological example Unlike CNNs and deep neural networks in that their neural connections can contain cycles A very general form of neural network Turing complete Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Recurrent neural networks are Alex Atanasov VFU An Introduction to Recurrent Neural Networks Like CNNs, motivated by biological example Unlike CNNs and deep neural networks in that their neural connections can contain cycles A very general form of neural network Turing complete Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Recurrent neural networks are Designed for processing sequential data Alex Atanasov VFU An Introduction to Recurrent Neural Networks Unlike CNNs and deep neural networks in that their neural connections can contain cycles A very general form of neural network Turing complete Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Recurrent neural networks are Designed for processing sequential data Like CNNs, motivated by biological example Alex Atanasov VFU An Introduction to Recurrent Neural Networks A very general form of neural network Turing complete Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Recurrent neural networks are Designed for processing sequential data Like CNNs, motivated by biological example Unlike CNNs and deep neural networks in that their neural connections can contain cycles Alex Atanasov VFU An Introduction to Recurrent Neural Networks Turing complete Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Recurrent neural networks are Designed for processing sequential data Like CNNs, motivated by biological example Unlike CNNs and deep neural networks in that their neural connections can contain cycles A very general form of neural network Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Recurrent neural networks are Designed for processing sequential data Like CNNs, motivated by biological example Unlike CNNs and deep neural networks in that their neural connections can contain cycles A very general form of neural network Turing complete Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Visual Representation From a neuroscience paper1: 1Song et al. 2014, Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Visual Representation From a neuroscience paper1: 1Song et al. 2014, Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework Alex Atanasov VFU An Introduction to Recurrent Neural Networks Very simple form of neural network Build in the context of theoretical neuroscience First order attempt at understanding the mechanism underlying associative memory Still an important object of study, primarily in neuroscience Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Hopfield Networks The first formulation of a \recurrent-like" neural network was made by John Hopfield (1982) Alex Atanasov VFU An Introduction to Recurrent Neural Networks Build in the context of theoretical neuroscience First order attempt at understanding the mechanism underlying associative memory Still an important object of study, primarily in neuroscience Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Hopfield Networks The first formulation of a \recurrent-like" neural network was made by John Hopfield (1982) Very simple form of neural network Alex Atanasov VFU An Introduction to Recurrent Neural Networks First order attempt at understanding the mechanism underlying associative memory Still an important object of study, primarily in neuroscience Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Hopfield Networks The first formulation of a \recurrent-like" neural network was made by John Hopfield (1982) Very simple form of neural network Build in the context of theoretical neuroscience Alex Atanasov VFU An Introduction to Recurrent Neural Networks Still an important object of study, primarily in neuroscience Introduction to RNNs Historical Background Mathematical Formulation Unrolling Computing Gradients Hopfield Networks The first formulation of a \recurrent-like" neural network was made by John Hopfield (1982) Very simple form of neural network Build in the context of theoretical neuroscience First order attempt at understanding the mechanism underlying associative memory Alex Atanasov VFU An Introduction to Recurrent Neural Networks Introduction to RNNs Historical