DOCSLIB.ORG

Backpropagation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Backpropagation Machine Learning Srihari Backpropagation Sargur Srihari 1 Machine Learning Srihari Topics in Backpropagation 1. Forward Propagation 2. Loss Function and Gradient Descent 3. Computing derivatives using chain rule 4. Computational graph for backpropagation 5. Backprop algorithm 6. The Jacobian matrix 2 Machine Learning Srihari A neural network with one hidden layer D input variables x1,.., xD M hidden unit activations D (1) (1) a w x w where j 1,..,M j = ∑ ji i + j 0 = i=1 Hidden unit activation functions z j =h(aj) K output activations M (2) (2) a w x w where k 1,..,K k = ∑ ki i + k 0 = i=1 Output activation functions Augmented network yk =σ(ak) ⎛ M ⎛ D ⎞ ⎞ No. of weights in w: y ( , ) w(2)h w(1)x w(1) w(2) k x w = σ ⎜ ∑ kj ⎜ ∑ ji i + j 0 ⎟ + k 0 ⎟ j =1 ⎝ i=1 ⎠ T=(D+1)M+(M+1)K ⎝ ⎠ =M(D+K+1)+K 3 Machine Learning Srihari Matrix Multiplication: Forward Propagation • Each layer is a function of layer that preceded it • First layer is given by z =h (W(1)T x + b(1)) • Second layer is y = σ (W(2)T x + b(2)) • Note that W is a matrix rather than a vector • Example with D=3, M=3 ⎪⎧ T T T ⎪ W (1) = ⎡W W W ⎤ ,W (1) = ⎡W W W ⎤ ,W (1) = ⎡W W W ⎤ T ⎪ 1 ⎣⎢ 11 12 13 ⎦⎥ 2 ⎣⎢ 21 22 23 ⎦⎥ 3 ⎣⎢ 31 32 33 ⎦⎥ x=[x1,x2,x3] w = ⎨ ⎪ T T T ⎪ W (2) = ⎡W W W ⎤ ,W (2) = ⎡W W W ⎤ ,W (2) = ⎡W W W ⎤ ⎪ 1 ⎣⎢ 11 12 13 ⎦⎥ 2 ⎣⎢ 21 22 23 ⎦⎥ 3 ⎣⎢ 31 32 33 ⎦⎥ ⎩⎪ First Network layer Network layer output In matrix multiplication notation 4 Machine Learning Srihari Loss and Regularization y=f (x,w) 1 N E = E f(x (i),w),t N ∑ i ( i ) i=1 x Forward: y Loss + Ei Backward: Gradient of Ei+R R(W) 5 Machine Learning Srihari Gradient Descent • Goal: determine weights w from labeled set of training samples • Learning procedure has two stages 1. Evaluate derivatives of loss ∇E(w) with respect to weights w1,..wT 2. Use derivative vector to compute adjustments to weights ⎡ ∂E ⎤ ⎢ ⎥ (τ+1) (τ) (τ) ∂w E( ) ⎢ 0 ⎥ w = w − η∇ w ⎢ ∂E ⎥ ⎢ ⎥ ∇E w = w ( ) ⎢ ∂ 1 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ∂E ⎥ ⎢ ⎥ ∂w ⎢ T ⎥ ⎣ ⎦ Machine Learning Srihari Derivative of composite function with one weight g (y) y=f (w) p=g(y) w f (w) k(p,q) o Input Output weight z=f (w) q=h(z) h(z) Composite Function o = k(p,q)=k(g(f(w)),h(f(w)) !" = !" !% !& + !" !' !( ð$ ð% ð& ð$ ð' ð( ð$ !" !)(%.') !)(%,') = -′(/) 0′(1)+ ℎ4 5 0′(1) ð$ ð% ð' 7 Path 1 Path 2 Machine Learning Srihari Derivative of a composite function with four inputs E (a,b,c,d) = e =a.b + c log d Derivatives by inspection: e !" !" !" 1 =1 1 =1 =b ð$ ð& ð% !" =a ð) u v !" =log d ð' !$ !$ !& !& 3 =b 2 =a 5 = c 1 =log d ð% ð) ð( ð' !" = + c ð* * a c Computational graph b !" !" !" e=u+v, u=a.b, v=c.t , t=log d =3 =2 =1 ð% ð) t ð' We want to compute derivatives !( + of output wrt the input values 0.1 = ð* * a = 2, b = 3, c = 5, d =10 ⎡ ∂E ⎤ ⎢ ⎥ ∂w d ⎢ 0 ⎥ 3 ⎢ E ⎥ ⎢ ∂ ⎥ !" ∇E w = w 2 ( ) ⎢ ∂ 1 ⎥ =0.5 ⎢ ⎥ ð* ⎢ ⎥ 1 ⎢ ∂E ⎥ ⎢ ⎥ ∂w 0.5 ⎢ T ⎥ 8 ⎣ ⎦ Machine Learning Srihari Example of Derivative Computation 9 Machine LearningDerivatives of f =(x+y)z wrt x,y,z Srihari Machine Learning Srihari Derivatives for a neuron: z=f(x,y) Machine Learning Srihari Composite Function • Consider a composite function f (g (h (x))) • i.e., an outer function f, an inner function g and a final inner function h(x) sin(x 2 ) • Say f ( x ) = e we can decompose it as: f (x)=ex g(x)=sin x and h(x)=x2 or f (g(h(x)))=e g(h(x)) • Its computational graph is • Every connection is an input, every node is a function or operation DeepMachine Learning Learning Srihari Derivatives of Composite function • To get derivatives of f (g (h (x)))= e g(h(x)) wrt x df df dg dh 1. We use the chain rule = ⋅ ⋅ where dx dg dh dx df g(h(x )) = e g(h(x)) x dg since f (g(h(x)))=e & derivative of e is e dg = cos(h(x)) since g(h(x))=sin h(x) & derivative sin is cos dh dh because h x x2 & its derivative is x = 2x ( )= 2 dx df = eg(h(x )) ⋅cos h(x)⋅2x = esinx**2 ⋅cosx 2 ⋅2x • Therefore dx • In each of these cases we pretend that the inner function is a single variable and derive it as such 2 2. Another way to view it sin(x ) f(x) = e • Create temp variables u=sin v, v=x2, then f (u)=eu with computational graph: DeepMachine Learning Learning Srihari Derivative using Computational Graph • All we need to do is get the derivative of each node wrt each of its inputs With u=sin v, v=x2, f (u)=eu • We can get whichever derivative we want by multiplying the ‘connection’ derivatives df dh dg = eg(h(x )) = 2x = cos(h(x)) dg dx dh df df dg dh df Since f (x)=ex, g(x)=sin x and = ⋅ ⋅ = eg(h(x )) ⋅cos h(x)⋅2x dx dg dh dx h(x)=x2 dx sinx 2 2 = e ⋅cosx ⋅2x 14 Machine Learning Srihari Evaluating the gradient • Goal of this section: • Find an efficient technique for evaluating gradient of an error function E(w) for a feed-forward neural network: • Gradient evaluation can be performed using a local message passing scheme • In which information is alternately sent forwards and backwards through the network • Known as error backpropagation or simply as backprop Machine Learning Srihari Back-propagation Terminology and Usage • Backpropagation means a variety of different things • Computing derivative of the error function wrt weights • In a second separate stage the derivatives are used to compute the adjustments to be made to the weights • Can be applied to error function other than sum of squared errors • Used to evaluate other matrices such as Jacobian and Hessian matrices • Second stage of weight adjustment using calculated derivatives can be tackled using variety of optimization schemes substantially more powerful than gradient descent Machine Learning Srihari Overview of Backprop algorithm • Choose random weights for the network • Feed in an example and obtain a result • Calculate the error for each node (starting from the last stage and propagating the error backwards) • Update the weights • Repeat with other examples until the network converges on the target output • How to divide up the errors needs a little calculus 17 Machine Learning Srihari Evaluation of Error Function Derivatives • Derivation of back-propagation algorithm for • Arbitrary feed-forward topology • Arbitrary differentiable nonlinear activation function • Broad class of error functions • Error functions of practical interest are sums of errors associated with each training data point N E( ) E ( ) w = ∑ n w n=1 • We consider problem of evaluating ∇E (w) n • For the nth term in the error function • Derivatives are wrt the weights w1,..wT • This can be used directly for sequential optimization or accumulated over training set (for batch) 18 Machine LearningSimple Model (Multiple Linear Regression) Srihari • Outputs yk are linear combinations of inputs xi yk y w x wki k = ∑ ki i i xi • Error function for a particular input xn is 1 2 Where summation is E = y − t n 2 ∑( nk nk ) over all K outputs For a particular input x and k weight w , squared error is: • where ynk=yk(xn,w) 1 2 E = (y(x,w) −t ) • Gradient of Error function wrt a weight wji: 2 ∂E ∂E n = y − t x = (y(x,w) −t )x = δ ⋅x ∂w ( nj nj ) ni ∂w ji • a local computation involving product of yj • error signal ynj-tnj associated with output end of link wji wji t • variable xni associated with input end of link j xi ∂E = (yj −tj )xi = δj ⋅ xi ∂wji Machine Learning Srihari Extension to more complex multilayer Network a w z • Each unit computes a weighted sum of its inputs j = ∑ ji i i zj=h(aj) aj=∑iwjizi zi wji • zi is activation of a unit (or input) that sends a connection to unit j and wji is the weight associated with the connection • Output is transformed by a nonlinear activation function zj=h(aj) • The variable zi can be an input and unit j could be an output • For each input xn in the training set, we calculate activations of all hidden and output units by applying above equations • This process is called forward propagation Machine Learning Srihari Evaluation of Derivative En wrt a weight wji • The outputs of the various units depend on particular input n • We shall omit the subscript n from network variables • Note that En depends on wji only via the summed input aj to unit j. • We can therefore apply chain rule for partial derivatives to give ∂E ∂E ∂a n = n j ∂w ∂a ∂w ji j ji • Derivative wrt weight is given by product of derivative wrt activity and derivative of activity wrt weight ∂E δ ≡ n • We now introduce a useful notation j ∂a j • Where the δs are errors as we shall see ∂aj a = w z = zi • Using j ∑ ji i we can write ∂w ji i ∂E n = δ z • Substituting we get ∂w j i ji • i.e., required derivative is obtained by multiplying the value of δ for the unit at the output end of the weight by the the value of z at the input end of the weight • This takes the same form as for the simple linear model 21 Machine Learning Srihari ∂ E Summarizing evaluation of Derivative n ∂w ji • By chain rule for partial derivatives ∂E ∂E ∂a n = n j a =∑ w z ∂w ∂a ∂w j i ji i ji j ji a = w z Define ∂E j ∑ ji i δ ≡ n i zi wji j ∂a ∂a j we have j = z ∂w i ji • Substituting we get ∂E n = δ z ∂w j i ji • Thus required derivative is obtained by multiplying 1.
Load more

Recommended publications
  • Backpropagation and Deep Learning in the Brain
    Backpropagation and Deep Learning in the Brain Simons Institute -- Computational Theories of the Brain 2018 Timothy Lillicrap DeepMind, UCL With: Sergey Bartunov, Adam Santoro, Jordan Guerguiev, Blake Richards, Luke Marris, Daniel Cownden, Colin Akerman, Douglas Tweed, Geoffrey Hinton The “credit assignment” problem The solution in artificial networks: backprop Credit assignment by backprop works well in practice and shows up in virtually all of the state-of-the-art supervised, unsupervised, and reinforcement learning algorithms. Why Isn’t Backprop “Biologically Plausible”? Why Isn’t Backprop “Biologically Plausible”? Neuroscience Evidence for Backprop in the Brain? A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: How to convince a neuroscientist that the cortex is learning via [something like] backprop - To convince a machine learning researcher, an appeal to variance in gradient estimates might be enough. - But this is rarely enough to convince a neuroscientist. - So what lines of argument help? How to convince a neuroscientist that the cortex is learning via [something like] backprop - What do I mean by “something like backprop”?: - That learning is achieved across multiple layers by sending information from neurons closer to the output back to “earlier” layers to help compute their synaptic updates. How to convince a neuroscientist that the cortex is learning via [something like] backprop 1. Feedback connections in cortex are ubiquitous and modify the
    [Show full text]
  • Implementing Machine Learning & Neural Network Chip
    Implementing Machine Learning & Neural Network Chip Architectures Using Network-on-Chip Interconnect IP Implementing Machine Learning & Neural Network Chip Architectures USING NETWORK-ON-CHIP INTERCONNECT IP TY GARIBAY Chief Technology Officer [email protected] Title: Implementing Machine Learning & Neural Network Chip Architectures Using Network-on-Chip Interconnect IP Primary author: Ty Garibay, Chief Technology Officer, ArterisIP, [email protected] Secondary author: Kurt Shuler, Vice President, Marketing, ArterisIP, [email protected] Abstract: A short tutorial and overview of machine learning and neural network chip architectures, with emphasis on how network-on-chip interconnect IP implements these architectures. Time to read: 10 minutes Time to present: 30 minutes Copyright © 2017 Arteris www.arteris.com 1 Implementing Machine Learning & Neural Network Chip Architectures Using Network-on-Chip Interconnect IP What is Machine Learning? “ Field of study that gives computers the ability to learn without being explicitly programmed.” Arthur Samuel, IBM, 1959 • Machine learning is a subset of Artificial Intelligence Copyright © 2017 Arteris 2 • Machine learning is a subset of artificial intelligence, and explicitly relies upon experiential learning rather than programming to make decisions. • The advantage to machine learning for tasks like automated driving or speech translation is that complex tasks like these are nearly impossible to explicitly program using rule-based if…then…else statements because of the large solution space. • However, the “answers” given by a machine learning algorithm have a probability associated with them and can be non-deterministic (meaning you can get different “answers” given the same inputs during different runs) • Neural networks have become the most common way to implement machine learning.
    [Show full text]
  • Training Autoencoders by Alternating Minimization
    Under review as a conference paper at ICLR 2018 TRAINING AUTOENCODERS BY ALTERNATING MINI- MIZATION Anonymous authors Paper under double-blind review ABSTRACT We present DANTE, a novel method for training neural networks, in particular autoencoders, using the alternating minimization principle. DANTE provides a distinct perspective in lieu of traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convex optimization techniques to cast autoencoder training as a bi-quasi-convex optimiza- tion problem. We show that for autoencoder configurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform the alternations very effectively. DANTE effortlessly extends to networks with multiple hidden layers and varying network configurations. In experiments on standard datasets, autoencoders trained using the proposed method were found to be very promising and competitive to traditional backpropagation techniques, both in terms of quality of solution, as well as training speed. 1 INTRODUCTION For much of the recent march of deep learning, gradient-based backpropagation methods, e.g. Stochastic Gradient Descent (SGD) and its variants, have been the mainstay of practitioners. The use of these methods, especially on vast amounts of data, has led to unprecedented progress in several areas of artificial intelligence. On one hand, the intense focus on these techniques has led to an intimate understanding of hardware requirements and code optimizations needed to execute these routines on large datasets in a scalable manner. Today, myriad off-the-shelf and highly optimized packages exist that can churn reasonably large datasets on GPU architectures with relatively mild human involvement and little bootstrap effort.
    [Show full text]
  • Predrnn: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal Lstms
    PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs Yunbo Wang Mingsheng Long∗ School of Software School of Software Tsinghua University Tsinghua University [email protected] [email protected] Jianmin Wang Zhifeng Gao Philip S. Yu School of Software School of Software School of Software Tsinghua University Tsinghua University Tsinghua University [email protected] [email protected] [email protected] Abstract The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical frames, where spatial appearances and temporal vari- ations are two crucial structures. This paper models these structures by presenting a predictive recurrent neural network (PredRNN). This architecture is enlightened by the idea that spatiotemporal predictive learning should memorize both spatial ap- pearances and temporal variations in a unified memory pool. Concretely, memory states are no longer constrained inside each LSTM unit. Instead, they are allowed to zigzag in two directions: across stacked RNN layers vertically and through all RNN states horizontally. The core of this network is a new Spatiotemporal LSTM (ST-LSTM) unit that extracts and memorizes spatial and temporal representations simultaneously. PredRNN achieves the state-of-the-art prediction performance on three video prediction datasets and is a more general framework, that can be easily extended to other predictive learning tasks by integrating with other architectures. 1 Introduction
    [Show full text]
  • Q-Learning in Continuous State and Action Spaces
    -Learning in Continuous Q State and Action Spaces Chris Gaskett, David Wettergreen, and Alexander Zelinsky Robotic Systems Laboratory Department of Systems Engineering Research School of Information Sciences and Engineering The Australian National University Canberra, ACT 0200 Australia [cg dsw alex]@syseng.anu.edu.au j j Abstract. -learning can be used to learn a control policy that max- imises a scalarQ reward through interaction with the environment. - learning is commonly applied to problems with discrete states and ac-Q tions. We describe a method suitable for control tasks which require con- tinuous actions, in response to continuous states. The system consists of a neural network coupled with a novel interpolator. Simulation results are presented for a non-holonomic control task. Advantage Learning, a variation of -learning, is shown enhance learning speed and reliability for this task.Q 1 Introduction Reinforcement learning systems learn by trial-and-error which actions are most valuable in which situations (states) [1]. Feedback is provided in the form of a scalar reward signal which may be delayed. The reward signal is defined in relation to the task to be achieved; reward is given when the system is successfully achieving the task. The value is updated incrementally with experience and is defined as a discounted sum of expected future reward. The learning systems choice of actions in response to states is called its policy. Reinforcement learning lies between the extremes of supervised learning, where the policy is taught by an expert, and unsupervised learning, where no feedback is given and the task is to find structure in data.
    [Show full text]
  • CS 189 Introduction to Machine Learning Spring 2021 Jonathan Shewchuk HW6
    CS 189 Introduction to Machine Learning Spring 2021 Jonathan Shewchuk HW6 Due: Wednesday, April 21 at 11:59 pm Deliverables: 1. Submit your predictions for the test sets to Kaggle as early as possible. Include your Kaggle scores in your write-up (see below). The Kaggle competition for this assignment can be found at • https://www.kaggle.com/c/spring21-cs189-hw6-cifar10 2. The written portion: • Submit a PDF of your homework, with an appendix listing all your code, to the Gradescope assignment titled “Homework 6 Write-Up”. Please see section 3.3 for an easy way to gather all your code for the submission (you are not required to use it, but we would strongly recommend using it). • In addition, please include, as your solutions to each coding problem, the specific subset of code relevant to that part of the problem. Whenever we say “include code”, that means you can either include a screenshot of your code, or typeset your code in your submission (using markdown or LATEX). • You may typeset your homework in LaTeX or Word (submit PDF format, not .doc/.docx format) or submit neatly handwritten and scanned solutions. Please start each question on a new page. • If there are graphs, include those graphs in the correct sections. Do not put them in an appendix. We need each solution to be self-contained on pages of its own. • In your write-up, please state with whom you worked on the homework. • In your write-up, please copy the following statement and sign your signature next to it.
    [Show full text]
  • Double Backpropagation for Training Autoencoders Against Adversarial Attack
    1 Double Backpropagation for Training Autoencoders against Adversarial Attack Chengjin Sun, Sizhe Chen, and Xiaolin Huang, Senior Member, IEEE Abstract—Deep learning, as widely known, is vulnerable to adversarial samples. This paper focuses on the adversarial attack on autoencoders. Safety of the autoencoders (AEs) is important because they are widely used as a compression scheme for data storage and transmission, however, the current autoencoders are easily attacked, i.e., one can slightly modify an input but has totally different codes. The vulnerability is rooted the sensitivity of the autoencoders and to enhance the robustness, we propose to adopt double backpropagation (DBP) to secure autoencoder such as VAE and DRAW. We restrict the gradient from the reconstruction image to the original one so that the autoencoder is not sensitive to trivial perturbation produced by the adversarial attack. After smoothing the gradient by DBP, we further smooth the label by Gaussian Mixture Model (GMM), aiming for accurate and robust classification. We demonstrate in MNIST, CelebA, SVHN that our method leads to a robust autoencoder resistant to attack and a robust classifier able for image transition and immune to adversarial attack if combined with GMM. Index Terms—double backpropagation, autoencoder, network robustness, GMM. F 1 INTRODUCTION N the past few years, deep neural networks have been feature [9], [10], [11], [12], [13], or network structure [3], [14], I greatly developed and successfully used in a vast of fields, [15]. such as pattern recognition, intelligent robots, automatic Adversarial attack and its defense are revolving around a control, medicine [1]. Despite the great success, researchers small ∆x and a big resulting difference between f(x + ∆x) have found the vulnerability of deep neural networks to and f(x).
    [Show full text]
  • Face Recognition: a Convolutional Neural-Network Approach
    98 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 1, JANUARY 1997 Face Recognition: A Convolutional Neural-Network Approach Steve Lawrence, Member, IEEE, C. Lee Giles, Senior Member, IEEE, Ah Chung Tsoi, Senior Member, IEEE, and Andrew D. Back, Member, IEEE Abstract— Faces represent complex multidimensional mean- include fingerprints [4], speech [7], signature dynamics [36], ingful visual stimuli and developing a computational model for and face recognition [8]. Sales of identity verification products face recognition is difficult. We present a hybrid neural-network exceed $100 million [29]. Face recognition has the benefit of solution which compares favorably with other methods. The system combines local image sampling, a self-organizing map being a passive, nonintrusive system for verifying personal (SOM) neural network, and a convolutional neural network. identity. The techniques used in the best face recognition The SOM provides a quantization of the image samples into a systems may depend on the application of the system. We topological space where inputs that are nearby in the original can identify at least two broad categories of face recognition space are also nearby in the output space, thereby providing systems. dimensionality reduction and invariance to minor changes in the image sample, and the convolutional neural network provides for 1) We want to find a person within a large database of partial invariance to translation, rotation, scale, and deformation. faces (e.g., in a police database). These systems typically The convolutional network extracts successively larger features return a list of the most likely people in the database in a hierarchical set of layers. We present results using the [34].
    [Show full text]
  • Can Temporal-Difference and Q-Learning Learn Representation? a Mean-Field Analysis
    Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Analysis Yufeng Zhang Qi Cai Northwestern University Northwestern University Evanston, IL 60208 Evanston, IL 60208 [email protected] [email protected] Zhuoran Yang Yongxin Chen Zhaoran Wang Princeton University Georgia Institute of Technology Northwestern University Princeton, NJ 08544 Atlanta, GA 30332 Evanston, IL 60208 [email protected] [email protected] [email protected] Abstract Temporal-difference and Q-learning play a key role in deep reinforcement learning, where they are empowered by expressive nonlinear function approximators such as neural networks. At the core of their empirical successes is the learned feature representation, which embeds rich observations, e.g., images and texts, into the latent space that encodes semantic structures. Meanwhile, the evolution of such a feature representation is crucial to the convergence of temporal-difference and Q-learning. In particular, temporal-difference learning converges when the function approxi- mator is linear in a feature representation, which is fixed throughout learning, and possibly diverges otherwise. We aim to answer the following questions: When the function approximator is a neural network, how does the associated feature representation evolve? If it converges, does it converge to the optimal one? We prove that, utilizing an overparameterized two-layer neural network, temporal- difference and Q-learning globally minimize the mean-squared projected Bellman error at a sublinear rate. Moreover, the associated feature representation converges to the optimal one, generalizing the previous analysis of [21] in the neural tan- gent kernel regime, where the associated feature representation stabilizes at the initial one. The key to our analysis is a mean-field perspective, which connects the evolution of a finite-dimensional parameter to its limiting counterpart over an infinite-dimensional Wasserstein space.
    [Show full text]
  • CNN Architectures
    Lecture 9: CNN Architectures Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 1 May 2, 2017 Administrative A2 due Thu May 4 Midterm: In-class Tue May 9. Covers material through Thu May 4 lecture. Poster session: Tue June 6, 12-3pm Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 2 May 2, 2017 Last time: Deep learning frameworks Paddle (Baidu) Caffe Caffe2 (UC Berkeley) (Facebook) CNTK (Microsoft) Torch PyTorch (NYU / Facebook) (Facebook) MXNet (Amazon) Developed by U Washington, CMU, MIT, Hong Kong U, etc but main framework of Theano TensorFlow choice at AWS (U Montreal) (Google) And others... Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 3 May 2, 2017 Last time: Deep learning frameworks (1) Easily build big computational graphs (2) Easily compute gradients in computational graphs (3) Run it all efficiently on GPU (wrap cuDNN, cuBLAS, etc) Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 4 May 2, 2017 Last time: Deep learning frameworks Modularized layers that define forward and backward pass Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 5 May 2, 2017 Last time: Deep learning frameworks Define model architecture as a sequence of layers Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 6 May 2, 2017 Today: CNN Architectures Case Studies - AlexNet - VGG - GoogLeNet - ResNet Also.... - NiN (Network in Network) - DenseNet - Wide ResNet - FractalNet - ResNeXT - SqueezeNet - Stochastic Depth Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 7 May 2, 2017 Review: LeNet-5 [LeCun et al., 1998] Conv filters were 5x5, applied at stride 1 Subsampling (Pooling) layers were 2x2 applied at stride 2 i.e.
    [Show full text]
  • Introduction to Machine Learning
    Introduction to Machine Learning Perceptron Barnabás Póczos Contents History of Artificial Neural Networks Definitions: Perceptron, Multi-Layer Perceptron Perceptron algorithm 2 Short History of Artificial Neural Networks 3 Short History Progression (1943-1960) • First mathematical model of neurons ▪ Pitts & McCulloch (1943) • Beginning of artificial neural networks • Perceptron, Rosenblatt (1958) ▪ A single neuron for classification ▪ Perceptron learning rule ▪ Perceptron convergence theorem Degression (1960-1980) • Perceptron can’t even learn the XOR function • We don’t know how to train MLP • 1963 Backpropagation… but not much attention… Bryson, A.E.; W.F. Denham; S.E. Dreyfus. Optimal programming problems with inequality constraints. I: Necessary conditions for extremal solutions. AIAA J. 1, 11 (1963) 2544-2550 4 Short History Progression (1980-) • 1986 Backpropagation reinvented: ▪ Rumelhart, Hinton, Williams: Learning representations by back-propagating errors. Nature, 323, 533—536, 1986 • Successful applications: ▪ Character recognition, autonomous cars,… • Open questions: Overfitting? Network structure? Neuron number? Layer number? Bad local minimum points? When to stop training? • Hopfield nets (1982), Boltzmann machines,… 5 Short History Degression (1993-) • SVM: Vapnik and his co-workers developed the Support Vector Machine (1993). It is a shallow architecture. • SVM and Graphical models almost kill the ANN research. • Training deeper networks consistently yields poor results. • Exception: deep convolutional neural networks, Yann LeCun 1998. (discriminative model) 6 Short History Progression (2006-) Deep Belief Networks (DBN) • Hinton, G. E, Osindero, S., and Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554. • Generative graphical model • Based on restrictive Boltzmann machines • Can be trained efficiently Deep Autoencoder based networks Bengio, Y., Lamblin, P., Popovici, P., Larochelle, H.
    [Show full text]
  • Neural Network in Hardware
    UNLV Theses, Dissertations, Professional Papers, and Capstones 12-15-2019 Neural Network in Hardware Jiong Si Follow this and additional works at: https://digitalscholarship.unlv.edu/thesesdissertations Part of the Electrical and Computer Engineering Commons Repository Citation Si, Jiong, "Neural Network in Hardware" (2019). UNLV Theses, Dissertations, Professional Papers, and Capstones. 3845. http://dx.doi.org/10.34917/18608784 This Dissertation is protected by copyright and/or related rights. It has been brought to you by Digital Scholarship@UNLV with permission from the rights-holder(s). You are free to use this Dissertation in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s) directly, unless additional rights are indicated by a Creative Commons license in the record and/or on the work itself. This Dissertation has been accepted for inclusion in UNLV Theses, Dissertations, Professional Papers, and Capstones by an authorized administrator of Digital Scholarship@UNLV. For more information, please contact [email protected]. NEURAL NETWORKS IN HARDWARE By Jiong Si Bachelor of Engineering – Automation Chongqing University of Science and Technology 2008 Master of Engineering – Precision Instrument and Machinery Hefei University of Technology 2011 A dissertation submitted in partial fulfillment of the requirements for the Doctor of Philosophy – Electrical Engineering Department of Electrical and Computer Engineering Howard R. Hughes College of Engineering The Graduate College University of Nevada, Las Vegas December 2019 Copyright 2019 by Jiong Si All Rights Reserved Dissertation Approval The Graduate College The University of Nevada, Las Vegas November 6, 2019 This dissertation prepared by Jiong Si entitled Neural Networks in Hardware is approved in partial fulfillment of the requirements for the degree of Doctor of Philosophy – Electrical Engineering Department of Electrical and Computer Engineering Sarah Harris, Ph.D.
    [Show full text]