DOCSLIB.ORG

Neural Networks

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Neural Networks Neural Networks Nicholas Ruozzi University of Texas at Dallas Handwritten Digit Recognition • Given a collection of handwritten digits and their corresponding labels, we’d like to be able to correctly classify handwritten digits – A simple algorithmic technique can solve this problem with 95% accuracy • This seems surprising, in fact, state- of-the-art methods can achieve near 99% accuracy (you’ve probably seen these in action if you’ve deposited a Digits from the MNIST check recently) data set 2 Neural Networks • The basis of neural networks was developed in the 1940s -1960s – The idea was to build mathematical models that might “compute” in the same way that neurons in the brain do – As a result, neural networks are biologically inspired, though many of the algorithms developed for them are not biologically plausible – Perform surprisingly well for the handwritten digit recognition task 3 Neural Networks • Neural networks consist of a collection of artificial neurons • There are different types of neuron models that are commonly studied – The perceptron (one of the first studied) – The sigmoid neuron (one of the most common, but many more) – Rectified linear units • A neural network is typically a directed graph consisting of a collection of neurons (the nodes in the graph), directed edges (each with an associated weight), and a collection of fixed binary inputs 4 The Perceptron • A perceptron is an artificial neuron that takes a collection of binary inputs and produces a binary output – The output of the perceptron is determined by summing up the weighted inputs and thresholding the result: if the weighted sum is larger than the threshold, the output is one (and zero otherwise) 푥1 푥2 푦 푥3 1 푤 푥 + 푤 푥 + 푤 푥 > 푡ℎ푟푒푠ℎ표푙푑 푦 = ቊ 1 1 2 2 3 3 0 표푡ℎ푒푟푤푖푠푒 5 The Perceptron 푥1 푥2 푦 푥3 1 푤 푥 + 푤 푥 + 푤 푥 > 푡ℎ푟푒푠ℎ표푙푑 푦 = ቊ 1 1 2 2 3 3 0 표푡ℎ푒푟푤푖푠푒 • The weights can be both positive and negative • Many simple decisions can be modeled using perceptrons 6 Perceptron for NOT 푥 ⌐ 푦 • Choose 푤 = −1, threshold = −.5 1 −푥 > −.5 • 푦 = ቊ 0 −푥 ≤ −.5 7 Perceptron for OR 8 Perceptron for OR 푥1 ᴠ 푦 푥2 • Choose 푤1 = 푤2 = 1, threshold = 0 1 푥 + 푥 > 0 • 푦 = ቊ 1 2 0 푥1 + 푥2 ≤ 0 9 Perceptron for AND 10 Perceptron for AND 푥1 ᴧ 푦 푥2 • Choose 푤1 = 푤2 = 1, threshold = 1.5 1 푥 + 푥 > 1.5 • 푦 = ቊ 1 2 0 푥1 + 푥2 ≤ 1.5 11 Perceptron for XOR 12 Perceptron for XOR • Need more than one perceptron! 푥1 ᴠ ᴧ 푦 푥2 ᴧ ⌐ • Weights for incoming edges are chosen as before – Networks of perceptrons can encode any circuit! 13 Perceptrons • Perceptrons are usually expressed in terms of a collection of input weights and a bias 푏 (which is the negative threshold) 푥1 푥2 푦 푥3 1 푤 푥 + 푤 푥 + 푤 푥 + 푏 > 0 푦 = ቊ 1 1 2 2 3 3 0 표푡ℎ푒푟푤푖푠푒 • A single node perceptron is just a linear classifier – This is actually where the “perceptron algorithm” comes from 14 Neural Networks • Gluing a bunch of perceptrons together gives us a neural network • In general, neural nets have a collection of binary inputs and a collection of binary outputs Inputs Outputs 15 Beyond Perceptrons • Given a collection of input-output pairs, we’d like to learn the weights of the neural network so that we can correctly predict the ouput of an unseen input – We could try learning via gradient descent (e.g., by minimizing the Hamming loss) • This approach doesn’t work so well: small changes in the weights can cause dramatic changes in the output • This is a consequence of the discontinuity of sharp thresholding (same problem we saw in SVMs) 16 The Sigmoid Neuron • A sigmoid neuron is an artificial neuron that takes a collection of inputs in the interval [0,1] and produces an output in the interval [0,1] – The output is determined by summing up the weighted inputs plus the bias and applying the sigmoid function to the result 푥1 푥2 푦 푥3 푦 = 휎(푤1푥1 + 푤2푥2 + 푤3푥3 + 푏) where 휎 is the sigmoid function 17 The Sigmoid Function • The sigmoid function is a continuous function that approximates a step function 1 휎 푧 = 1 + 푒−푧 18 Rectified Linear Units • The sigmoid neuron approximates a step function as a smooth function • The relu approximates a hinge loss max(0, 푥) as a smooth continuous function ln(1 + 푒푥) 19 Multilayer Neural Networks from Neural Networks and Deep Learning by Michael Nielson 20 Multilayer Neural Networks NO intralayer connections from Neural Networks and Deep Learning by Michael Nielson 21 Neural Network for Digit Classification from Neural Networks and Deep Learning by Michael Nielson 22 Neural Network for Digit Classification Why 10 instead of 4? from Neural Networks and Deep Learning by Michael Nielson 23 Expressiveness of NNs • Boolean functions • Every Boolean function can be represented by a network with a single hidden layer consisting of possibly exponentially many hidden units • Continuous functions • Every bounded continuous function can be approximated up to arbitrarily small error by a network with one hidden layer • Any function can be approximated to arbitrary accuracy with two hidden layers 24 Training Neural Networks • To do the learning, we first need to define a loss function to minimize 1 퐶 푤, 푏 = ෍ 푦푚 − 푎(푥푚, 푤, 푏) 2 2푀 푚 • The training data consists of input output pairs (푥1, 푦1), … , (푥푀, 푦푀) • 푎(푥푚, 푤, 푏) is the output of the neural network for the 푚푡ℎ sample • 푤 and 푏 are the weights and biases 25 Gradient of the Loss • The derivative of the loss function is calculated as follows 휕퐶(푤, 푏) 1 휕푎(푥푚, 푤, 푏) = ෍ 푦푚 − 푎(푥푚, 푤, 푏) 휕푤푘 푀 휕푤푘 푚 – To compute the derivative of 푎, use the chain rule and the derivative of the sigmoid function 푑휎(푧) = 휎 푧 ⋅ (1 − 휎 푧 ) 푑푧 – This gets complicated quickly with lots of layers of neurons 26 Stochastic Gradient Descent • To make the training more practical, stochastic gradient descent is used instead of standard gradient descent • Recall, the idea of stochastic gradient descent is to approximate the gradient of a sum by sampling a few indices and averaging 푛 퐾 1 훻 ෍ 푓 (푥) ≈ ෍ 훻 푓 푘(푥) 푥 푖 퐾 푥 푖 푖=1 푘=1 here, for example, each 푖푘 is sampled uniformly at random from {1, … , 푛} 27 Computing the Gradient • We’ll compute the gradient for a single sample 퐶 푤, 푏 = 푦 − 푎(푥, 푤, 푏) 2 • Some definitions: – 퐿 is the number of layers 푙 푡ℎ 푡ℎ – 푎푗 is the output of the 푗 neuron on the 푙 layer 푙 푡ℎ 푡ℎ – 푧푗 is the input of the 푗 neuron on the 푙 layer 푙 푙 푙−1 푙 푧푗 = ෍ 푤푗푘푎푘 + 푏푗 푘 푙 휕C – 훿푗 is defined to be 푙 휕푧푗 28 Computing the Gradient For the output layer, we have the following partial derivative 휕푎퐿 휕C 퐿 푗 퐿 = − 푦푗 − 푎푗 퐿 휕푧푗 휕푧푗 휕휎 푧퐿 퐿 푗 = − 푦푗 − 푎푗 퐿 휕푧푗 퐿 퐿 퐿 = − 푦푗 − 푎푗 휎 푧푗 1 − 휎 푧푗 퐿 = 훿푗 • For simplicity, we will denote the vector of all such partials for each node in the 푙푡ℎ layer as 훿푙 29 Computing the Gradient For the 퐿 − 1 layer, we have the following partial derivative 퐿 휕C 휕푎푗 = ෍ 푎퐿 − 푦 휕푧퐿−1 푗 푗 휕푧퐿−1 푘 푗 푘 퐿 휕휎 푧푗 = ෍ 푎퐿 − 푦 푗 푗 휕푧퐿−1 푗 푘 퐿 휕푧푗 = ෍ 푎퐿 − 푦 휎 푧퐿 1 − 휎 푧퐿 푗 푗 푗 푗 휕푧퐿−1 푗 푘 퐿 퐿−1 퐿 휕 σ푘′ 푤푗푘′푎푘′ + 푏푗 = ෍ 푎퐿 − 푦 휎 푧퐿 1 − 휎 푧퐿 푗 푗 푗 푗 휕푧퐿−1 푗 푘 퐿 퐿 퐿 퐿−1 퐿−1 퐿 = ෍ 푎푗 − 푦푗 휎 푧푗 1 − 휎 푧푗 휎 푧푘 1 − 휎 푧푘 푤푗푘 푗 퐿 푇 퐿 퐿−1 퐿−1 = (훿 ) 푤∗푘 1 − 휎 푧푘 휎 푧푘 30 Computing the Gradient • We can think of 푤푙 as a matrix • This allows us to write 훿퐿−1 = (훿퐿)푇푤퐿 1 − 휎 푧퐿−1 휎 푧퐿−1 퐿−1 푡ℎ 퐿−1 where 휎 푧 is the vector whose 푘 component is 휎 푧푘 • Applying the same strategy, for 푙 < 퐿 훿푙 = (훿푙+1)푇푤푙+1 1 − 휎 푧푙 휎 푧푙 31 Computing the Gradient • Now, for the partial derivatives that we care about 휕푧푙 휕퐶 휕퐶 푗 푙 푙 = 푙 ⋅ 푙 = 훿푗 휕푏푗 휕푧푗 휕푏푗 휕푧푙 휕퐶 휕퐶 푗 푙 푙−1 푙 = 푙 ⋅ 푙 = 훿푗 푎푘 휕푤푗푘 휕푧푗 휕푤푗푘 • We can compute these derivatives one layer at a time! 32 Backpropagation: Putting it all together • Compute the inputs/outputs for each layer by starting at the input layer and applying the sigmoid functions • Compute 훿퐿 for the output layer 퐿 퐿 퐿 퐿 훿 = − 푦푗 − 푎푗 휎 푧푗 1 − 휎 푧푗 • Starting from 푙 = 퐿 − 1 and working backwards, compute 훿푙 = (훿푙+1)푇푤푙+1 휎 푧푙 1 − 휎 푧푙 • Perform gradient descent 푙 푙 푙 푏푗 = 푏푗 − 훾 ⋅ 훿푗 푙 푙 푙 푙−1 푤푗푘 = 푤푗푘 − 훾 ⋅ 훿푗 푎푘 33 Backpropagation • Backpropagation converges to a local minimum (loss is not convex in the weights and biases) – Like EM, can just run it several times with different initializations – Training can take a very long time (even with stochastic gradient descent) – Prediction after learning is fast – Sometimes include a momentum term 훼 in the gradient update 푤 푡 = 푤 푡 − 1 − 훾 ⋅ 훻푤퐶 푡 − 1 + 훼 −훾 ⋅ 훻푤퐶(푡 − 2) 34 Overfitting 35 Overfitting 36 Neural Networks in Practice • Many ways to improve weight learning in NNs – Use a regularizer! (better generalization) – Try other loss functions – Initialize the weights of the network more cleverly • Random initializations are likely to be far from optimal – etc.
Load more

Recommended publications
  • Training Autoencoders by Alternating Minimization
    Under review as a conference paper at ICLR 2018 TRAINING AUTOENCODERS BY ALTERNATING MINI- MIZATION Anonymous authors Paper under double-blind review ABSTRACT We present DANTE, a novel method for training neural networks, in particular autoencoders, using the alternating minimization principle. DANTE provides a distinct perspective in lieu of traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convex optimization techniques to cast autoencoder training as a bi-quasi-convex optimiza- tion problem. We show that for autoencoder configurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform the alternations very effectively. DANTE effortlessly extends to networks with multiple hidden layers and varying network configurations. In experiments on standard datasets, autoencoders trained using the proposed method were found to be very promising and competitive to traditional backpropagation techniques, both in terms of quality of solution, as well as training speed. 1 INTRODUCTION For much of the recent march of deep learning, gradient-based backpropagation methods, e.g. Stochastic Gradient Descent (SGD) and its variants, have been the mainstay of practitioners. The use of these methods, especially on vast amounts of data, has led to unprecedented progress in several areas of artificial intelligence. On one hand, the intense focus on these techniques has led to an intimate understanding of hardware requirements and code optimizations needed to execute these routines on large datasets in a scalable manner. Today, myriad off-the-shelf and highly optimized packages exist that can churn reasonably large datasets on GPU architectures with relatively mild human involvement and little bootstrap effort.
    [Show full text]
  • Neural Network in Hardware
    UNLV Theses, Dissertations, Professional Papers, and Capstones 12-15-2019 Neural Network in Hardware Jiong Si Follow this and additional works at: https://digitalscholarship.unlv.edu/thesesdissertations Part of the Electrical and Computer Engineering Commons Repository Citation Si, Jiong, "Neural Network in Hardware" (2019). UNLV Theses, Dissertations, Professional Papers, and Capstones. 3845. http://dx.doi.org/10.34917/18608784 This Dissertation is protected by copyright and/or related rights. It has been brought to you by Digital Scholarship@UNLV with permission from the rights-holder(s). You are free to use this Dissertation in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s) directly, unless additional rights are indicated by a Creative Commons license in the record and/or on the work itself. This Dissertation has been accepted for inclusion in UNLV Theses, Dissertations, Professional Papers, and Capstones by an authorized administrator of Digital Scholarship@UNLV. For more information, please contact [email protected]. NEURAL NETWORKS IN HARDWARE By Jiong Si Bachelor of Engineering – Automation Chongqing University of Science and Technology 2008 Master of Engineering – Precision Instrument and Machinery Hefei University of Technology 2011 A dissertation submitted in partial fulfillment of the requirements for the Doctor of Philosophy – Electrical Engineering Department of Electrical and Computer Engineering Howard R. Hughes College of Engineering The Graduate College University of Nevada, Las Vegas December 2019 Copyright 2019 by Jiong Si All Rights Reserved Dissertation Approval The Graduate College The University of Nevada, Las Vegas November 6, 2019 This dissertation prepared by Jiong Si entitled Neural Networks in Hardware is approved in partial fulfillment of the requirements for the degree of Doctor of Philosophy – Electrical Engineering Department of Electrical and Computer Engineering Sarah Harris, Ph.D.
    [Show full text]
  • Population Dynamics: Variance and the Sigmoid Activation Function ⁎ André C
    www.elsevier.com/locate/ynimg NeuroImage 42 (2008) 147–157 Technical Note Population dynamics: Variance and the sigmoid activation function ⁎ André C. Marreiros, Jean Daunizeau, Stefan J. Kiebel, and Karl J. Friston Wellcome Trust Centre for Neuroimaging, University College London, UK Received 24 January 2008; revised 8 April 2008; accepted 16 April 2008 Available online 29 April 2008 This paper demonstrates how the sigmoid activation function of (David et al., 2006a,b; Kiebel et al., 2006; Garrido et al., 2007; neural-mass models can be understood in terms of the variance or Moran et al., 2007). All these models embed key nonlinearities dispersion of neuronal states. We use this relationship to estimate the that characterise real neuronal interactions. The most prevalent probability density on hidden neuronal states, using non-invasive models are called neural-mass models and are generally for- electrophysiological (EEG) measures and dynamic casual modelling. mulated as a convolution of inputs to a neuronal ensemble or The importance of implicit variance in neuronal states for neural-mass population to produce an output. Critically, the outputs of one models of cortical dynamics is illustrated using both synthetic data and ensemble serve as input to another, after some static transforma- real EEG measurements of sensory evoked responses. © 2008 Elsevier Inc. All rights reserved. tion. Usually, the convolution operator is linear, whereas the transformation of outputs (e.g., mean depolarisation of pyramidal Keywords: Neural-mass models; Nonlinear; Modelling cells) to inputs (firing rates in presynaptic inputs) is a nonlinear sigmoidal function. This function generates the nonlinear beha- viours that are critical for modelling and understanding neuronal activity.
    [Show full text]
  • Dynamic Modification of Activation Function Using the Backpropagation Algorithm in the Artificial Neural Networks
    (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 10, No. 4, 2019 Dynamic Modification of Activation Function using the Backpropagation Algorithm in the Artificial Neural Networks Marina Adriana Mercioni1, Alexandru Tiron2, Stefan Holban3 Department of Computer Science Politehnica University Timisoara, Timisoara, Romania Abstract—The paper proposes the dynamic modification of These functions represent the main activation functions, the activation function in a learning technique, more exactly being currently the most used by neural networks, representing backpropagation algorithm. The modification consists in from this reason a standard in building of an architecture of changing slope of sigmoid function for activation function Neural Network type [6,7]. according to increase or decrease the error in an epoch of learning. The study was done using the Waikato Environment for The activation functions that have been proposed along the Knowledge Analysis (WEKA) platform to complete adding this time were analyzed in the light of the applicability of the feature in Multilayer Perceptron class. This study aims the backpropagation algorithm. It has noted that a function that dynamic modification of activation function has changed to enables differentiated rate calculated by the neural network to relative gradient error, also neural networks with hidden layers be differentiated so and the error of function becomes have not used for it. differentiated [8]. Keywords—Artificial neural networks; activation function; The main purpose of activation function consists in the sigmoid function; WEKA; multilayer perceptron; instance; scalation of outputs of the neurons in neural networks and the classifier; gradient; rate metric; performance; dynamic introduction a non-linear relationship between the input and modification output of the neuron [9].
    [Show full text]
  • Two-Steps Implementation of Sigmoid Function for Artificial Neural Network in Field Programmable Gate Array
    VOL. 11, NO. 7, APRIL 2016 ISSN 1819-6608 ARPN Journal of Engineering and Applied Sciences ©2006-2016 Asian Research Publishing Network (ARPN). All rights reserved. www.arpnjournals.com TWO-STEPS IMPLEMENTATION OF SIGMOID FUNCTION FOR ARTIFICIAL NEURAL NETWORK IN FIELD PROGRAMMABLE GATE ARRAY Syahrulanuar Ngah1, Rohani Abu Bakar1, Abdullah Embong1 and Saifudin Razali2 1Faculty of Computer Systems and Software Engineering, Universiti Malaysia Pahang, Malaysia 2Faculty of Electrical and Electronic Engineering, Universiti Malaysia Pahang, Malaysia [email protected] ABSTRACT The complex equation of sigmoid function is one of the most difficult problems encountered for implementing the artificial neural network (ANN) into a field programmable gate array (FPGA). To overcome this problem, the combination of second order nonlinear function (SONF) and the differential lookup table (dLUT) has been proposed in this paper. By using this two-steps approach, the output accuracy achieved is ten times better than that of using only SONF and two times better than that of using conventional lookup table (LUT). Hence, this method can be used in various applications that required the implementation of the ANN into FPGA. Keywords: sigmoid function, second order nonlinear function, differential lookup table, FPGA. INTRODUCTION namely Gaussian, logarithmic, hyperbolic tangent and The Artificial neural network (ANN) has become sigmoid function has been done by T. Tan et al. They a growing research focus over the last few decades. It have found that, the optimal configuration for their controller been used broadly in many fields to solve great variety of can be achieved by using the sigmoid function [12].In this problems such as speed estimation [1], image processing paper, sigmoid function is being used as an activation [2] and pattern recognition [3].
    [Show full text]
  • Information Flows of Diverse Autoencoders
    entropy Article Information Flows of Diverse Autoencoders Sungyeop Lee 1,* and Junghyo Jo 2,3,* 1 Department of Physics and Astronomy, Seoul National University, Seoul 08826, Korea 2 Department of Physics Education and Center for Theoretical Physics and Artificial Intelligence Institute, Seoul National University, Seoul 08826, Korea 3 School of Computational Sciences, Korea Institute for Advanced Study, Seoul 02455, Korea * Correspondence: [email protected] (S.L.); [email protected] (J.J.) Abstract: Deep learning methods have had outstanding performances in various fields. A fundamen- tal query is why they are so effective. Information theory provides a potential answer by interpreting the learning process as the information transmission and compression of data. The information flows can be visualized on the information plane of the mutual information among the input, hidden, and output layers. In this study, we examine how the information flows are shaped by the network parameters, such as depth, sparsity, weight constraints, and hidden representations. Here, we adopt autoencoders as models of deep learning, because (i) they have clear guidelines for their information flows, and (ii) they have various species, such as vanilla, sparse, tied, variational, and label autoen- coders. We measured their information flows using Rényi’s matrix-based a-order entropy functional. As learning progresses, they show a typical fitting phase where the amounts of input-to-hidden and hidden-to-output mutual information both increase. In the last stage of learning, however, some autoencoders show a simplifying phase, previously called the “compression phase”, where input-to-hidden mutual information diminishes. In particular, the sparsity regularization of hidden activities amplifies the simplifying phase.
    [Show full text]
  • Compositional Distributional Semantics with Long Short Term Memory
    Compositional Distributional Semantics with Long Short Term Memory Phong Le and Willem Zuidema Institute for Logic, Language and Computation University of Amsterdam, the Netherlands p.le,zuidema @uva.nl { } Abstract any continuous function (Cybenko, 1989) already make them an attractive choice. We are proposing an extension of the recur- In trying to answer the second question, the ad- sive neural network that makes use of a vari- vantages of approaches based on neural network ar- ant of the long short-term memory architec- ture. The extension allows information low chitectures, such as the recursive neural network in parse trees to be stored in a memory reg- (RNN) model (Socher et al., 2013b) and the con- ister (the ‘memory cell’) and used much later volutional neural network model (Kalchbrenner et higher up in the parse tree. This provides a so- al., 2014), are even clearer. Models in this paradigm lution to the vanishing gradient problem and can take advantage of general learning procedures allows the network to capture long range de- based on back-propagation, and with the rise of pendencies. Experimental results show that ‘deep learning’, of a variety of efficient algorithms our composition outperformed the traditional neural-network composition on the Stanford and tricks to further improve training. Sentiment Treebank. Since the first success of the RNN model (Socher et al., 2011b) in constituent parsing, two classes of extensions have been proposed. One class is to en- 1 Introduction hance its compositionality by using tensor product Moving from lexical to compositional semantics in (Socher et al., 2013b) or concatenating RNNs hor- vector-based semantics requires answers to two dif- izontally to make a deeper net (Irsoy and Cardie, ficult questions: (i) what is the nature of the com- 2014).
    [Show full text]
  • Long Short-Term Memory 1 Introduction
    LONG SHORTTERM MEMORY Neural Computation J urgen Schmidhuber Sepp Ho chreiter IDSIA Fakultat f ur Informatik Corso Elvezia Technische Universitat M unchen Lugano Switzerland M unchen Germany juergenidsiach ho chreitinformatiktumuenchende httpwwwidsiachjuergen httpwwwinformatiktumuenchendehochreit Abstract Learning to store information over extended time intervals via recurrent backpropagation takes a very long time mostly due to insucient decaying error back ow We briey review Ho chreiters analysis of this problem then address it by introducing a novel ecient gradientbased metho d called Long ShortTerm Memory LSTM Truncating the gradient where this do es not do harm LSTM can learn to bridge minimal time lags in excess of discrete time steps by enforcing constant error ow through constant error carrousels within sp ecial units Multiplicative gate units learn to op en and close access to the constant error ow LSTM is lo cal in space and time its computational complexity p er time step and weight is O Our exp eriments with articial data involve lo cal distributed realvalued and noisy pattern representations In comparisons with RTRL BPTT Recurrent CascadeCorrelation Elman nets and Neural Sequence Chunking LSTM leads to many more successful runs and learns much faster LSTM also solves complex articial long time lag tasks that have never b een solved by previous recurrent network algorithms INTRODUCTION Recurrent networks can in principle use their feedback connections to store representations of recent input events in form of activations
    [Show full text]
  • 1 Activation Functions in Single Hidden Layer Feed
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE Selçuk-Teknik Dergisi ISSN 1302-6178 Journal ofprovided Selcuk by- Selçuk-TeknikTechnic Dergisi (E-Journal - Selçuk Üniversiti) Özel Sayı 2018 (ICENTE’18) Special Issue 2018 (ICENTE'18) ACTIVATION FUNCTIONS IN SINGLE HIDDEN LAYER FEED-FORWARD NEURAL NETWORKS Ender SEVİNÇ University of Turkish Aeronautical Association, Ankara Turkey [email protected] Abstract Especially in the last decade, Artificial Intelligence (AI) has gained increasing popularity as the neural networks represent incredibly exciting and powerful machine learning-based techniques that can solve many real-time problems. The learning capability of such systems is directly related with the evaluation methods used. In this study, the effectiveness of the calculation parameters in a Single-Hidden Layer Feedforward Neural Networks (SLFNs) will be examined. We will present how important the selection of an activation function is in the learning stage. A lot of work is developed and presented for SLFNs up to now. Our study uses one of the most commonly known learning algorithms, which is Extreme Learning Machine (ELM). Main task of an activation function is to map the input value of a neural network to the output node with a high learning or achievement rate. However, determining the correct activation function is not as simple as thought. First we try to show the effect of the activation functions on different datasets and then we propose a method for selection process of it due to the characteristic of any dataset. The results show that this process is providing a remarkably better performance and learning rate in a sample neural network.
    [Show full text]
  • Rethinking the Role of Activation Functions in Deep Convolutional Neural Networks for Image Classification
    Engineering Letters, 28:1, EL_28_1_11 ______________________________________________________________________________________ Rethinking the Role of Activation Functions in Deep Convolutional Neural Networks for Image Classification Qinghe Zheng, Mingqiang Yang, Xinyu Tian, Xiaochen Wang and Deqiang Wang and development of artificial neural networks. Back Abstract—Deep convolutional neural network used for image propagation algorithm [12] makes deep learning model no classification is an important part of deep learning and has longer remote, and eventually brings the revival of deep great significance in the field of computer vision. Moreover, it learning research based on statistical model [13]. helps humans to simulate the human brain more realistically, pointing out the direction for the development of artificial Activation functions are the core of deep neural network intelligence. In fact, the rapid development and its application structure. It is just a node added to the output of the neural of deep neural networks are due to the improvements of various network, also known as a transfer function. It can also activation functions. The activation function is one of the most connect two layers of neural network models. It is used to critical parts of the neural networks, which provides the determine whether the neural network output is yes or no, possibility of strong nonlinear fitting ability of the deep neural mapping the output values between 0 and 1 or between -1 and network. In this paper, we analyze how the activation function affects the deep neural network, and then analyzes and 1 (depending on the activation functions between different summarizes the development status and the performance of two layers). At present, the popular and commonly used different activation functions.
    [Show full text]
  • Chapter 10: Artificial Neural Networks
    Chapter 10: Artificial Neural Networks Dr. Xudong Liu Assistant Professor School of Computing University of North Florida Monday, 9/30/2019 1 / 17 Overview 1 Artificial neuron: linear threshold unit (LTU) 2 Perceptron 3 Multi-Layer Perceptron (MLP), Deep Neural Networks (DNN) 4 Learning ANN's: Error Backpropagation Overview 2 / 17 Differential Calculus In this course, I shall stick to Leibniz's notation. The derivative of a function at an input point, when it exists, is the slope of the tangent line to the graph of the function. 2 0 dy Let y = f (x) = x + 2x + 1. Then, f (x) = dx = 2x + 2. A partial derivative of a function of several variables is its derivative with respect to one of those variables, with the others held constant. 2 2 @z Let z = f (x; y) = x + xy + y . Then, @x = 2x + y. The chain rule is a formula for computing the derivative of the composition of two or more functions. dz dz dy 0 0 Let z = f (y) and y = g(x). Then, dx = dy · dx = f (y) · g (x). 1 Nice thing about the sigmoid function f (x) = 1+e−x : f 0(x) = f (x) · (1 − f (x)). Some Preliminaries 3 / 17 Artificial Neuron Artificial neuron, also called linear threshold unit (LTU), by McCulloch and Pitts, 1943: with one or more numeric inputs, it produces a weighted sum of them, applies an activation function, and outputs the result. Common activation functions: step function and sigmoid function. Artificial Neuron 4 / 17 Linear Threshold Unit (LTU) Below is an LTU with the activation function being the step function.
    [Show full text]
  • On the Derivatives of the Sigmoid
    Neural Networks. Vol. 6, pp. 845-853, 1993 0893-6080/93 $6.00 + .00 Printed in the USA. All fights reserved. Copyright © 1993 Pergamon Press Ltd. ORIGINAL CONTRIBUTION On the Derivatives of the Sigmoid ALI A. MINAI AND RONALD D. WILLIAMS University of Virginia, Charlottesville (Received 6 Februa~. 1992; revised and accepted 13 January 1993) Abstract--The sigmoid fimction & very widely used as a neuron activation fimction in artificial neural networks. which makes its attributes a matter of some interest. This paper presents some general results on the derivatives of the sigmoid. These results relate the coefficients of various derivatives to standard number sequences from combi- natorial theory, and thus provide a standard efficient way of calculating these derivatives. Such derivatives can be used in approximating the effect of input perturbations on the output of single neurons and could also be usefid in statistical modeling applications. Keywords--Neuron activation function, Sigmoid function, Logistic function, Eulerian numbers, Stifling numbers. 1. INTRODUCTION one input. However, it can also be given a more general interpretation. Let i be a neuron with n inputs A), 1 < The sigmoid function has found extensive use as a non- j < n, and let the output y of the neuron be given by: linear activation function for neurons in artificial neural 1 networks. Thus, it is of some interest to explore its y - -- (2.1) characteristics. In this paper, we study the derivatives 1 +e--" of the l-dimensional sigmoid function z= ~ woxj + O = wx + O, (2.2) J I y = a(x; w) 1 + e -w" ' ( 1 ) where 0 is a bias value, x = [x~, x: ....
    [Show full text]