VOL. 11, NO. 7, APRIL 2016 ISSN 1819-6608 ARPN Journal of Engineering and Applied Sciences ©2006-2016 Asian Research Publishing Network (ARPN). All rights reserved. www.arpnjournals.com TWO-STEPS IMPLEMENTATION OF SIGMOID FUNCTION FOR ARTIFICIAL NEURAL NETWORK IN FIELD PROGRAMMABLE GATE ARRAY Syahrulanuar Ngah1, Rohani Abu Bakar1, Abdullah Embong1 and Saifudin Razali2 1Faculty of Computer Systems and Software Engineering, Universiti Malaysia Pahang, Malaysia 2Faculty of Electrical and Electronic Engineering, Universiti Malaysia Pahang, Malaysia [email protected] ABSTRACT The complex equation of sigmoid function is one of the most difficult problems encountered for implementing the artificial neural network (ANN) into a field programmable gate array (FPGA). To overcome this problem, the combination of second order nonlinear function (SONF) and the differential lookup table (dLUT) has been proposed in this paper. By using this two-steps approach, the output accuracy achieved is ten times better than that of using only SONF and two times better than that of using conventional lookup table (LUT). Hence, this method can be used in various applications that required the implementation of the ANN into FPGA. Keywords: sigmoid function, second order nonlinear function, differential lookup table, FPGA. INTRODUCTION namely Gaussian, logarithmic, hyperbolic tangent and The Artificial neural network (ANN) has become sigmoid function has been done by T. Tan et al. They a growing research focus over the last few decades. It have found that, the optimal configuration for their controller been used broadly in many fields to solve great variety of can be achieved by using the sigmoid function [12].In this problems such as speed estimation [1], image processing paper, sigmoid function is being used as an activation [2] and pattern recognition [3]. ANN have been function. It is an important component of ANN, mainly to implemented mostly in software, however hardware limit down the neuron output. Since the hardware is version are gaining importance [4]. Software unable to calculate the floating number as compared to implementation offers flexibility for emulating wide range software especially when involving the activation of neural networks model while the hardware function, new method is needed to do the calculation and implementation is vital in taking advantages of inherent at the same time produce the output as similar as software parallelism [5]. The implementation of artificial neural calculation. network in field programmable gate array is attracting more attention now due to two reasons. Firstly, though the Table-1. Some common activation function. performance of conventional processor is continually improved, the fastest sequential processor is still unable to provide a real-time response for the ANN with a large number of neurons. Secondly, most implementations have been done in the form of simulations running on a personal computer (PC) or workstation owing to the lack of portable software for running the ANN [6]. Alternatively, the ANN can be run on field programmable gate array (FPGA) since it provides more portable platform for numerous real-world applications. FPGA is completely programmable and efficient device to replace the microprocessor yet faster to develop the dedicated ships [7]. At the same time, parallel structure of ANN makes it potentially fast for the computation of certain task and resulted reduce the processing time [4][8][9]. ARTIFICIAL NEURAL NETWORK (ANN) However, implementing the ANN into FPGA is not as NNs is one of the techniques in the area of simple as implementing in software especially when artificial intelligence known as nature inspired. It can be involving a large number of neuron and the calculation of defined as a model of reasoning based on the human brain. complex equation such as activation function[9]. There are It resembles the human brain in two aspects i) the acquired several common types of activation function used in ANN knowledge by the network is gain from the environment define, respectively as linear, bipolar threshold, sigmoidal through a learning process and ii) the strength of (sigmoid function) and hyperbolic tan [7][10][11]. Table-1 connection between neuron (weight connection) are used lists a few of linear and nonlinear activation functions. to store the acquired knowledge. The ANNs consists of a Comparison between various nonlinear activation function huge class of different architectures. There are several 4882 VOL. 11, NO. 7, APRIL 2016 ISSN 1819-6608 ARPN Journal of Engineering and Applied Sciences ©2006-2016 Asian Research Publishing Network (ARPN). All rights reserved. www.arpnjournals.com factors need to be considered before implementing the hardware. Both problem can be solved by using the FPGA ANN in solving certain problems. This is because ANN [18]. will only perform excellently if the selections are perfectly However, in order to implement the ANN on matched to the problems [13]. The general structure of FPGA, a special attention must be given on the activation ANN model is depicted in Figure-1. This model is function, particularly for the nonlinear activation function. composed of input, hidden and output layers. The number A high accuracy of ANN output requires a high precision of neurons in the input and output layers are determined in the output of the activation function. Since the output of by system itself. For example, let used Figure-1 for the activation function will become the input for the next positioning the object. Figure-1 have 4 neurons since it layer, any error that exists at the output of activation need to receive 4 distance enter from the user/system and function are accumulated and amplified by the ANN. 2 neuron in output layer to produce 2 output of coordinate Hence, the final results of the ANN will contain a very x and y. Figure-2 depicts one of the neurons where the large error. calculation of activation function is involved. As aforementioned, the sigmoid function as the activation function is one of the most frequently used in back-propagation neural networks applications. Sigmoid function provides smooth transition between input and output thereby improving the neural response when compared to other activation function like step function or saturating linear activation function [19]. It can be described by the following mathematical equation: (1) Figure-3 shows the output of the eq.1 (curve of the sigmoid function). Nevertheless, this function is not suitable for direct hardware implementation. A straightforward implementation of the sigmoid function is Figure-1. General structure of neural network. unfeasible, since both division and exponentiation are very demanding operations in terms of area of resources and long clock cycle [20]. A more practical solution is needed to overcome this problem. The used of approximation approach/method is one of work has been done on the hardware implementation of the sigmoid function [21][22]. Figure-2. Abstracted model of neuron with connection. HARDWARE IMPLEMENTATION OF ANN ANN has been mostly implemented in the software [14][15]. The benefit is that the designer does not need to understand how the neural network elements work. The parallel structure of the ANN makes it potentially fast for hardware implementation. Basically, the Figure-3. Sigmoid function curve. implementation of the ANN into hardware can be classified into three categories [15] which are digital a) Conventional sigmoid function implementation signal processing (DSP) [16], application specific It has been reported that several methods can be integrated circuits (ASIC)[17] and FPGA. ANN can be employed for implementing the sigmoid function in performed in parallel, but this feature cannot be exploited hardware, such as look-up table (LUT), coordinate rotation in DSP since DSP chips perform the tasks given in serial digital computer (CORDIC) algorithm, piecewise linear manner. On the other hand, if the ANN is implemented in approximation (PWL) and piecewise nonlinear ASIC, the network configuration cannot be updated or approximation[23][24][25]. All these methods will be modified to any further extent. It is fixed into ASIC discussed in details in the next sub-section. 4883 VOL. 11, NO. 7, APRIL 2016 ISSN 1819-6608 ARPN Journal of Engineering and Applied Sciences ©2006-2016 Asian Research Publishing Network (ARPN). All rights reserved. www.arpnjournals.com i. Look-up table (LUT) accuracy and the time required to complete the calculation The simplest implementation of sigmoid function by one or more clock cycles [25]. The 11-stage CORDIC is the use of a LUT. Many implementations used a LUT function has a latency of 50 clock cycles. The deviation for the approximation of the sigmoid function[26][27][28]. resulted from the CORDIC function ranges from -0.005 to The value of the sigmoid function curve are divided into 0.005. several segments and stored into table as shown in Figure-4. However the amount of hardware memory iii. Piecewise linear approximation (PWL) required for LUT implementation can be quite large There are few PWL methods proposed by especially if one requires higher accuracy in researches such a-law based companding technique [33], approximation since the value of the sigmoid function PWL of a nonlinear function (PLAN) [34] and recursive curve need to be divided into more segments. To create a algorithm based on the centred recursive interpolation LUT, a limited operating range should be determined. (CRI) [21]. Basically, all the proposed PWL methods Based on Figure-4, the sigmoid function is nearly obtain the low value of both maximum and average error 1 for x > 8 and 0 for x < -8. The non-saturation range is with low computational complexity. They are different in between -8 and 8. Since the sigmoid function has a numbers and locations of start and end point of the symmetry point at (0, 0.5), only half of the x-y pairs have approximating lines. A-Law based technique used a to be computed [20] and can be represented by: modified curve so that the gradient of each linear segment is expressed by the power of two.