DOCSLIB.ORG

A REVIEW and PERFORMANCE ANALYSIS of NON-LINEAR ACTIVATION FUNCTIONS in DEEP NEURAL NETWORKS Sushma Priya Anthadupula *1, Dr

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A REVIEW and PERFORMANCE ANALYSIS of NON-LINEAR ACTIVATION FUNCTIONS in DEEP NEURAL NETWORKS Sushma Priya Anthadupula *1, Dr e-ISSN: 2582-5208 International Research Journal of Modernization in Engineering Technology and Science Volume:03/Issue:03/March-2021 Impact Factor- 5.354 www.irjmets.com A REVIEW AND PERFORMANCE ANALYSIS OF NON-LINEAR ACTIVATION FUNCTIONS IN DEEP NEURAL NETWORKS Sushma Priya Anthadupula *1, Dr. Manasi Gyanchandani*2 *1 M.Tech Scholar, Department of Computer Science Engineering, Maulana Azad National Institute of Technology, Bhopal, M.P, India. *2 Professor, Department of Computer Science Engineering, Maulana Azad National Institute of Technology, Bhopal, M.P, India. ABSTRACT Activation Functions are mathematical equations which perform a nonlinear transformation on the input such that the neuron can be used to learn and perform complex tasks efficiently. A weighted sum of the input is calculated and a bias is added to it to produce an output. The decision to fire a neuron is made after applying the activation function to the output. An analysis of activation functions used in ANN is presented. Our paper explores the efficiency of activation functions on the MNIST and Fashion MNIST datasets. The convergence speed during the backpropagation after applying the activation functions is also shown in this work. Keywords: Activation Function, Neural Network, Hidden Unit, Backpropagation, Deep Neural Networks. I. INTRODUCTION The brain is an intelligent system. Artificial Neural Networks (ANN) tries to mimic this behavior. The output of the neuron is produced by summing up the products of weight and inputs and added with a bias. The output value can range anywhere between -∞ to +∞. Activation function is defined as any function which can bound the output of a neuron in the range of (0,1) or (-1,1) to fire a neuron. Different activation functions like sigmoid, Hyperbolic Tangent, ReLU, ELU, Softplus, or LeakyReLU can approximate arbitrary continuous functions. Sigmoid function has vanishing gradient problem. As a result, the Rectified Linear Unit (ReLU) activation function was suggested, which quickly gained popularity, but the Dying ReLU problem arose. Later Leaky ReLU, Parametric ReLU, Exponential Linear Unit (ELU), etc. have been introduced. Automation will help you find the right activation functions in [1]. A neural network does the same job as Linear Regression without an activation function. The number of hidden layers has shown no effect on the output without an activation function in the output layer. Real world data such as videos, images, music is nonlinear in nature. A nonlinear activation function must be continuous and differentiable between 0 and 1. This produces a nonlinear decision boundary. The network architecture is decided by variables called hyper parameters. These help in determining the accuracy or improving the training time of the neural network. Batch size, momentum, learning rate, and number of epochs are some of the trainable hyper parameters. The number of hidden neurons could be increased, which would enhance prediction accuracy, training speed, and stability[2] [3] [4]. Adaptive batch sizes for non-convex problems are seen to produce good results [5]. In this paper, we compare different activation functions on the MNIST Handwritten Digits dataset and the MNIST Fashion Dataset. Number of evaluation images are 10,000, validation images are 10,000, and training images are 50,000 in this dataset, each with a 28x28 greyscale pixel representation of one of the ten digits. On the MNIST , the performance of the activation functions is investigated using two hidden layers, each with 100 units. II. REVIEW OF ACTIVATION FUNCTIONS The advantages and implementations of activation functions, which are commonly used in deep learning algorithms, are discussed in this paper. Table 1 discusses the uses and limitations of various activation functions. The equations of all the activation functions can be seen in Table 2. 1) Sigmoid Activation Function The Sigmoid Function is widely used since several years. Squashing function or logistic function are other names of Sigmoid function [6]. It has a smooth S-shaped curve. It is a differentiable function and is bounded continuously with positive derivatives everywhere [7]. It has applications in modeling logistic www.irjmets.com @International Research Journal of Modernization in Engineering, Technology and Science [1168] e-ISSN: 2582-5208 International Research Journal of Modernization in Engineering Technology and Science Volume:03/Issue:03/March-2021 Impact Factor- 5.354 www.irjmets.com regression tasks, binary classification problems [8]. Sharp gradients from deep hidden layers to the input layer are propagated during backpropagation. Figure 1 shows the shape of sigmoid function. The following are the variants of Sigmoid activation function a) Hard Sigmoid Function The equation of this variant of sigmoid function is given by ( ) ( ( )) Hard sigmoid has a lower computation cost and shows positive results on binary classification tasks based on deep learning [9]. b) Sigmoid-Weighted Linear Units (SiLU) The equation for this activation function is given by ( ) ( ) Where s= input vector, ∑ c) Sigmoid-Weighted Linear Units Derivative (dSiLU) The equation for dSiLU is given by ( ) ( ) ( ( ( ))) dSiLU which looks like an overshooting sigmoid function outperformed the standard Sigmoid function significantly [10]. Figure 1 : Sigmoid Activation Function 2) Hyperbolic Tangent Activation Function The tanh function suffers from dead neurons problem because when x = 0 then the tanh function attains a gradient of 1. tanh is extensively used in speech recognition tasks and in recurrent neural networks for natural language processing [11]. Hard Hyperbolic Tangent is a variant of tanh. Figure 2 shows the shape of tanh function. www.irjmets.com @International Research Journal of Modernization in Engineering, Technology and Science [1169] e-ISSN: 2582-5208 International Research Journal of Modernization in Engineering Technology and Science Volume:03/Issue:03/March-2021 Impact Factor- 5.354 www.irjmets.com a) Hard Hyperbolic Tangent The Hard tanh is a variant of tanh that is less costly and more computationally powerful. The -1 to 1 range is covered by the Hard tanh feature. The equation is as follows: -1 if x <- 1 x if x = -1 1 if x > 1 Hard tanh applications include usage in natural language processing to improve speed and accuracy [12]. Figure 2 : Hyperbolic Tangent Activation Function 3) Rectified Linear Unit For deep neural networks, ReLU is the most commonly used activation function. Its ease of use and linear property that preserves non-linear behavior aid in achieving cutting-edge results. To trigger neurons, it uses a threshold operation. Its performance is better compared to Sigmoid and tanh functions with respect to generalization and training. The property of ReLU to squish the values between maximum and zero, helps to introduce sparsity in the hidden units. However, in some cases during backpropagation, the value of gradient is zero, due to which the weights and biases are not updated. Dying neurons problem is seen in ReLU due to which the weight updates cannot activate other neurons and thus hinders the learning. To solve this issue Leaky ReLU has been proposed [16]. ReLU and its variants are used effectively in Restricted Boltzmann machines (RBM) and in CNN [13]. Figure 3 shows the shape of ReLU function. Figure 3 : ReLU Activation Function a) Leaky ReLU Function (LReLU) To change the weights during backpropagation, the LReLU function adds a slight negative slope to the ReLU. www.irjmets.com @International Research Journal of Modernization in Engineering, Technology and Science [1170] e-ISSN: 2582-5208 International Research Journal of Modernization in Engineering Technology and Science Volume:03/Issue:03/March-2021 Impact Factor- 5.354 www.irjmets.com To make sure the gradients are not zero during the training parameter has been introduced. This ensures the dead neurons problem is rectified. As compared to hyperbolic tangent and regular ReLU, LReLU ensures improved sparsity and dispersion [14] [15]. b) Randomized Leaky ReLU (RReLU) RReLU is a vigorous form of leaky ReLU in which the network is trained using a number drawn at random from a uniform distribution U (l; u). If the average of all aj is taken during training without any dropout parameter, the test output is given by It has been discovered that RReLU outperforms ReLU in classification tasks [16] [17] . c) S-shaped ReLU (SReLU) Both convex and nonconvex functions are learned by the S-shaped ReLU. The need of less parameters helps SReLU in training deeper networks easily. SReLU has shown performance improvements on most popular datasets with and without data augmentation for regularization. d) Parametric ReLU (PReLU) In Leaky ReLU the slope is fixed for all the negative inputs. However, in PReLU a parameter α is considered as slope. The neural network figures out the value of α through gradient descent. 4) Softplus Function The softplus function enhances the performance and stabilization of deep neural networks. It is considered to have nonzero gradient properties. When the Softplus function was compared to the ReLU and Sigmoid functions, the Softplus function performed better with less epochs to convergence during testing. 5) Exponential Linear Units (ELUs) ELUs are used to accelerate deep neural network training. Since ELU uses identity for positive values, it avoids the vanishing gradient problem. When compared to ReLU, ELU effectively eliminates bias shifts. 6) Swish Function The Swish function is a hybrid
Load more

Recommended publications
  • Improvements on Activation Functions in ANN: an Overview
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by CSCanada.net: E-Journals (Canadian Academy of Oriental and Occidental Culture,... ISSN 1913-0341 [Print] Management Science and Engineering ISSN 1913-035X [Online] Vol. 14, No. 1, 2020, pp. 53-58 www.cscanada.net DOI:10.3968/11667 www.cscanada.org Improvements on Activation Functions in ANN: An Overview YANG Lexuan[a],* [a]Hangzhou No.2 High School of Zhejiang Province; Hangzhou, China. An ANN is comprised of computable units called * Corresponding author. neurons (or nodes). The activation function in a neuron processes the inputted signals and determines the output. Received 9 April 2020; accepted 21 May 2020 Published online 26 June 2020 Activation functions are particularly important in the ANN algorithm; change in activation functions can have a significant impact on network performance. Therefore, Abstract in recent years, researches have been done in depth on Activation functions are an essential part of artificial the improvement of activation functions to solve or neural networks. Over the years, researches have been alleviate some problems encountered in practices with done to seek for new functions that perform better. “classic” activation functions. This paper will give some There are several mainstream activation functions, such brief examples and discussions on the modifications and as sigmoid and ReLU, which are widely used across improvements of mainstream activation functions. the decades. At the meantime, many modified versions of these functions are also proposed by researchers in order to further improve the performance. In this paper, 1. INTRODUCTION OF ACTIVATION limitations of the mainstream activation functions, as well as the main characteristics and relative performances of FUNCTIONS their modifications, are discussed.
    [Show full text]
  • Eis- a Family of Activation Functions Combining Exponential,Isru, and Softplus
    EIS - A FAMILY OF ACTIVATION FUNCTIONS COMBINING EXPONENTIAL, ISRU, AND SOFTPLUS APREPRINT Koushik Biswas Sandeep Kumar Department of Computer Science Department of Computer Science, IIIT Delhi IIIT Delhi & New Delhi, India, 110020 Department of Mathematics, Shaheed Bhagat Singh College, [email protected] University of Delhi. New Delhi, India. [email protected], [email protected] Shilpak Banerjee Ashish Kumar Pandey Department of Mathematics Department of Mathematics IIIT Delhi IIIT Delhi New Delhi, India, 110020 New Delhi, India, 110020 [email protected] [email protected] ABSTRACT Activation functions play a pivotal role in the function learning using neural networks. The non- linearity in the learned function is achieved by repeated use of the activation function. Over the years, numerous activation functions have been proposed to improve accuracy in several tasks. Basic functions like ReLU, Sigmoid, Tanh, or Softplus have been favorite among the deep learning community because of their simplicity. In recent years, several novel activation functions arising from these basic functions have been proposed, which have improved accuracy in some challenging datasets. We propose a five hyper-parameters family of activation functions, namely EIS, defined as, x(ln(1 + ex))α : pβ + γx2 + δe−θx We show examples of activation functions from the EIS family which outperform widely used x ln(1+ex) activation functions on some well known datasets and models. For example, x+1:16e−x beats ReLU arXiv:2009.13501v2 [cs.LG] 12 Oct 2020 by 0.89% in DenseNet-169, 0.24% in Inception V3 in CIFAR100 dataset while 1.13% in Inception x ln(1+ex) V3, 0.13% in DenseNet-169, 0.94% in SimpleNet model in CIFAR10 dataset.
    [Show full text]
  • Arxiv:2102.02694V2 [Stat.ML] 19 May 2021
    Invertible DenseNets with Concatenated LipSwish Yura Perugachi-Diaz Jakub M. Tomczak Sandjai Bhulai Vrije Universiteit Amsterdam Vrije Universiteit Amsterdam Vrije Universiteit Amsterdam [email protected] [email protected] [email protected] Abstract We introduce Invertible Dense Networks (i-DenseNets), a more parameter efficient extension of Residual Flows. The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce invertibility of the network by satisfying the Lipschitz constant. Furthermore, we propose a learnable weighted concatenation, which not only improves the model performance but also indicates the importance of the concatenated weighted representation. Additionally, we introduce the Concatenated LipSwish as activation function, for which we show how to enforce the Lipschitz condition and which boosts performance. The new architecture, i-DenseNet, out-performs Residual Flow and other flow-based models on density estimation evaluated in bits per dimension, where we utilize an equal parameter budget. Moreover, we show that the proposed model out- performs Residual Flows when trained as a hybrid model where the model is both a generative and a discriminative model. 1 Introduction Neural networks are widely used to parameterize non-linear models in supervised learning tasks such as classification. In addition, they are also utilized to build flexible density estimators of the true distribution of the observed data [25, 33]. The resulting deep density estimators, also called deep generative models, can be further used to generate realistic-looking images that are hard to separate from real ones, detection of adversarial attacks [9, 17], and for hybrid modeling [27] which have the property to both predict a label (classify) and generate.
    [Show full text]
  • Deep Learning Fall 2020 Last Time: Deep Learning
    CPSC 340: Machine Learning and Data Mining More Deep Learning Fall 2020 Last Time: Deep Learning https://en.wikipedia.org/wiki/Neuron https://www.youtube.com/watch?v=aircAruvnKk ImageNet Challenge • Millions of labeled images, 1000 object classes. http://www.image-net.org/challenges/LSVRC/2014/ ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. • 2015: Won by Microsoft Asia – 3.6% error.
    [Show full text]
  • Tion Function for Deep Neural Networks
    Under review as a conference paper at ICLR 2019 ON THE SELECTION OF INITIALIZATION AND ACTIVA- TION FUNCTION FOR DEEP NEURAL NETWORKS Anonymous authors Paper under double-blind review ABSTRACT The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propaga- tion and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demon- strated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the ‘edge of chaos’ can lead to good performance. We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos. By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions. This class includes the Swish activation, φswish(x) = x · sigmoid(x), used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and Ramachandran et al. (2017). This provides a theoretical grounding for the excellent empirical performance of φswish observed in these contributions. We complement those previous results by illustrating the benefit of using a random initialization on the edge of chaos in this context. 1 INTRODUCTION Deep neural networks have become extremely popular as they achieve state-of-the-art performance on a variety of important applications including language processing and computer vision; see, e.g., LeCun et al.
    [Show full text]
  • Activation Functions: Comparison of Trends in Practice and Research for Deep Learning
    This is a peer-reviewed, accepted author manuscript of the following paper: Nwankpa, C. E., Ijomah, W., Gachagan, A., & Marshall, S. (2021). Activation functions: comparison of trends in practice and research for deep learning. 124 - 133. 2nd International Conference on Computational Sciences and Technology, Jamshoro, Pakistan. Activation Functions: Comparison of Trends in Practice and Research for Deep Learning Chigozie E. Nwankpa Winifred l.Ijomah Anthony Gachagan Stephen Marshall Dept. of DMEM Dept. of DMEM Dept. of EEE Dept. of EEE University of Strathclyde University of Strathclyde University of Strathclyde University of Strathclyde Glasgow, UK Glasgow, UK Glasgow, UK Glasgow, UK [email protected] [email protected] [email protected] [email protected] Abstract—Deep neural networks (DNN) have been successfully and detection. Deep learning, a sub-field of machine learning used in diverse emerging domains to solve real-world complex (ML), is more recently being referred to as representation problems with may more deep learning(DL) architectures, learning in some literature [3]. The typical artificial neural being developed to date. To achieve this state-of-the-art (SOTA) networks (ANN) are biologically inspired computer pro- performances, the DL architectures use activation functions grammes, designed by the inspiration of the workings of (AFs), to perform diverse computations between the hidden the human brain. These ANNs are called networks because layers and the output layers of any given DL architecture. they are composed of different functions [4], which gathers This paper presents a survey on the existing AFs used in knowledge by detecting the relationships and patterns in deep learning applications and highlights the recent trends data using past experiences known as training examples in in the use of the AFs for DL applications.
    [Show full text]
  • Activation Functions in Neural Networks
    International Journal of Engineering Applied Sciences and Technology, 2020 Vol. 4, Issue 12, ISSN No. 2455-2143, Pages 310-316 Published Online April 2020 in IJEAST (http://www.ijeast.com) ACTIVATION FUNCTIONS IN NEURAL NETWORKS Siddharth Sharma, Simone Sharma Anidhya Athaiya UG Scholar, Dept. of Computer Science and Assistant Professor, Dept. of Computer Science and Engineering, Global Institute of Technology, Jaipur Engineering, Global Institute of Technology, Jaipur Abstract—Artificial Neural Networks are inspired from linear activation function is used where the output is similar the human brain and the network of neurons present in as the input fed along with some error. A linear activation the brain. The information is processed and passed on function’s boundary is linear and the if they are used, then from one neuron to another through neuro synaptic the network can adapt to only the linear changes of the junctions. Similarly, in artificial neural networks there input are different layers of cells arranged and connected to but, in real world the errors possess non-linear each other. The output/information from the inner characteristics which in turn with the neural networks layers of the neural network are passed on to the next ability to learn about erroneous data. Hence non-linear layers and finally to the outermost layer which gives the activation functions are preferred over linear activation output. The input to the outer layer is provided non- functions in a Neural Network. linearity to inner layers’ output so that it can be further The most appealing property of Artificial Neural processed. In an Artificial Neural Network, activation Networks is the ability to adapt their behavior according to functions are very important as they help in learning the changing characteristics of the system.
    [Show full text]
  • A New Activation Function for Training Deep Neural Networks to Avoid Local Minimum
    A New Activation Function for Training Deep Neural Networks to Avoid Local Minimum Abhinav Sagar Vellore Institute of Technology, Vellore, Tamil Nadu, India ABSTRACT Activation functions have a major role to play and hence are very important while training neural networks. Understanding various activation functions and their advantages and disadvantages is crucial to achieve better results. This paper will first introduce common types of non linear activation functions and then evaluate their characteristics with their pros and cons. We will be focussing only on deep neural networks as it has proven to be much more difficult to train while avoiding overfitting. We have proposed a new activation function named - Abhinav which adds a non linearity with parametric coefficients to the Swish activation function. This has proven to give better results on the MNIST dataset. We reason this is because the model avoids getting stuck in local minima due to only the sigmoidal term present in the Swish function. The coefficients are automatically adjusted in each and every iteration i.e. coefficient values are reduced if the error is large and also sometimes reducing it to zero thus removing coefficients present in the polynomial. This new activation function can be made to generalize to other tasks including multi class and multi label image classification also. Index Terms—Neural Networks, Deep learning, Image classification, Activation function, Swish ​ 1. INTRODUCTION Deep neural network algorithms are multi layered learning techniques that allows non-linear nodes to transform representations from the raw input into the higher levels of abstract representations, with many of these transformations producing learned complex functions.
    [Show full text]
  • Activation Functions: Comparison of Trends in Practice and Research for Deep Learning
    1 Activation Functions: Comparison of Trends in Practice and Research for Deep Learning Chigozie Enyinna Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall Abstract Deep neural networks have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning(DL) architectures, being developed to date. To achieve these state-of-the-art performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing AFs used in deep learning applications and highlights the recent trends in the use of the activation functions for deep learning applications. The novelty of this paper is that it compiles majority of the AFs used in DL and outlines the current trends in the applications and usage of these functions in practical deep learning deployments against the state-of-the-art research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate activation function for any given application, ready for deployment. This paper is timely because most research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the research results from literature, found in deep learning research to date. Index Terms activation function, activation function types, activation function choices, deep learning, neural networks, learning algorithms I. INTRODUCTION Deep learning algorithms are multi-level representation learning techniques that allows simple non-linear modules to transform representations from the raw input into the higher levels of abstract representations, with many of these transformations producing learned complex functions.
    [Show full text]
  • Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks
    Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks Tomasz Szandała​1 1Wroclaw​ University of Science and Technology Wroclaw, Poland The primary neural networks decision-making units are activation functions. Moreover, they evaluate the output of networks neural node; thus, they are essential for the performance of the whole network. Hence, it is critical to choose the most appropriate activation function in neural networks calculation. Acharya et al. (2018) suggest that numerous recipes have been formulated over the years, though some of them are considered deprecated these days since they are unable to operate properly under some conditions. These functions have a variety of characteristics, which are deemed essential to successfully learning. Their monotonicity, individual derivatives, and finite of their range are some of these characteristics (Bach 2017). This research paper will evaluate the commonly used additive functions, such as swish, ReLU, Sigmoid, and so forth. This will be followed by their properties, own cons and pros, and particular formula application recommendations. deep learning, neural networks, activation function, classification, regression Introduction There exist many applications of deep learning neural network, such as voice analysis, speech or pattern recognition, and object classification. This has been demonstrated in its excellent performance in many fields. Moreover, it comprises a structure with hidden layers; this implies that the layers are more than one [1], [2]. However, from the study of deep learning structure, it can adapt in terms of real-world application with natural scenes due to its salient features. There exist only a few layers in the first deep learning model used for classification tasks; for example, there were only five layers in LeNet5 model[3].
    [Show full text]
  • Improvements on Activation Functions in ANN: an Overview
    ISSN 1913-0341 [Print] Management Science and Engineering ISSN 1913-035X [Online] Vol. 14, No. 1, 2020, pp. 53-58 www.cscanada.net DOI:10.3968/11667 www.cscanada.org Improvements on Activation Functions in ANN: An Overview YANG Lexuan[a],* [a]Hangzhou No.2 High School of Zhejiang Province; Hangzhou, China. An ANN is comprised of computable units called * Corresponding author. neurons (or nodes). The activation function in a neuron processes the inputted signals and determines the output. Received 9 April 2020; accepted 21 May 2020 Published online 26 June 2020 Activation functions are particularly important in the ANN algorithm; change in activation functions can have a significant impact on network performance. Therefore, Abstract in recent years, researches have been done in depth on Activation functions are an essential part of artificial the improvement of activation functions to solve or neural networks. Over the years, researches have been alleviate some problems encountered in practices with done to seek for new functions that perform better. “classic” activation functions. This paper will give some There are several mainstream activation functions, such brief examples and discussions on the modifications and as sigmoid and ReLU, which are widely used across improvements of mainstream activation functions. the decades. At the meantime, many modified versions of these functions are also proposed by researchers in order to further improve the performance. In this paper, 1. INTRODUCTION OF ACTIVATION limitations of the mainstream activation functions, as well as the main characteristics and relative performances of FUNCTIONS their modifications, are discussed. Activation functions are functions that determine the key words: Activation functions; ANN; Piecewise- output of the nodes (neurons) in ANNs.
    [Show full text]
  • Deep Learning Fall 2019 Last Time: Deep Learning
    CPSC 340: Machine Learning and Data Mining More Deep Learning Fall 2019 Last Time: Deep Learning https://en.wikipedia.org/wiki/Neuron https://www.youtube.com/watch?v=aircAruvnKk ImageNet Challenge • Millions of labeled images, 1000 object classes. http://www.image-net.org/challenges/LSVRC/2014/ ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. Syberian Husky Canadian Husky https://ischlag.github.io/2016/04/05/important-ILSVRC-achievements/ http://arxiv.org/pdf/1409.0575v3.pdf http://arxiv.org/pdf/1409.4842v1.pdf ImageNet Challenge • Object detection task: – Single label per image. – Humans: ~5% error. • 2015: Won by Microsoft Asia – 3.6% error.
    [Show full text]