Advances in Science, Technology and Engineering Systems Journal Vol. 2, No. 1, 69-75 (2017) ASTESJ www.astesj.com ISSN: 2415-6698 Special Issue on Computer Systems, Information Technology, Electrical and Electronics Engineering Recent Trends in ELM and MLELM: A review R. Manju Parkavi*,1, M. Shanthi1, M.C. Bhuvaneshwari2 1Department of ECE, Kumaraguru College of Technology, 641035, India 2Department of EEE, PSG College of Technology, 641004, India A R T I C L E I N F O A B S T R A C T Article history: Extreme Learning Machine (ELM) is a high effective learning algorithm for the single Received: 08 December, 2016 hidden layer feed forward neural networks. Compared with the existing neural network Accepted: 15 January, 2017 learning algorithm it solves the slow training speed and over-fitting problems. It has been Online: 28 January, 2017 used in different fields and applications such as biomedical engineering, computer vision, remote sensing, chemical process and control and robotics. It has better generalization Keywords : stability, sparsity, accuracy, robustness, optimal control and fast learning rate This paper Extreme learning machine (ELM) introduces a brief review about ELM and MLELM, describing the principles and latest Artificial Neural Network (ANN) research progress about the algorithms, theories and applications. Next, Multilayer MLELM Extreme Learning Machine (MLELM) and other state-of-the-art classifiers are trained on Deep Learning this suitable training feature vector for classification of data. Deep learning has the advantage of approximating the complicated function and mitigating the optimization difficulty associated with deep models. Multilayer extreme learning machine is a learning algorithm of an Artificial Neural Network (ANN) which takes to be good for deep learning and extreme learning machine. This review presents a comprehensive view of these advances in ELM and MLELM which may be worthy of exploring in the future. 1. Introduction artificial intelligence is based on the coexistence of three necessary conditions: superlative computing environments, rich As early as in the 1940s, arithmetician Pitts and psychologist and large data, and very effective learning techniques McCulloch have put forward neurons mathematical model (MP (algorithms). The Extreme Learning Machine as an evolving model) from the arithmetical logic view (McCulloch and Pitts learning technique gives efficient unified solutions to generalize 1943) which opened the prelude of artificial neural network feed-forward networks including but not limited to (each supports experimental work. Neural network with parallel and distributed single- and multi-hidden-layer) neural networks, Radial Basis information processing network framework has a strong nonlinear Function (RBF) networks and kernel learning. Single hidden layer mapping ability and robust self-learning, adaptive and fault feed forward networks (SLFNs), is one of the most accepted feed tolerance characteristics. Due to the recent popularity of deep forward neural networks, have been greatly studied from each learning, two of the most widely studied artificial neural theoretical and application aspect for their learning capabilities networks these days are auto encoders and Boltzmann machines. and fault-tolerant dependability [1–6]. However, most accepted An auto encoder with a single hidden layer as well as a structurally learning algorithms for training SLFNs are still relatively not restricted version of the Boltzmann machine, called a restricted quick since all the parameters of SFLNs need to be tuned through Boltzmann machine, have become popular due to their application repetitive procedures and these algorithms may also stuck in a in layer-wise pretraining of deep Multilayer perceptron’s. Table.1 local minimum. describes about Difference in Conventional learning method and Biological learning method. The progress of machine learning and Recently, a new rapid learning neural algorithm for SLFNs, named extreme learning machine (ELM) [7, 8], was developed to *R.ManjuParkavi, 4 Mahendrakumar building, Burgur road, anthiyur, erode. 9894768880, [email protected] enhance the efficiency of SLFNs. Different from the existing www.astesj.com 69 https://dx.doi.org/10.25046/aj020108 M. Parkavi et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 2, No. 1, 69-75 (2017) learning algorithms for neural networks (such as Back Propagation (BP) algorithms), which may face problems in manually tuning control parameters (learning rate, learning epochs, etc.) and/or local minima, ELM is automatically employed without iterative tuning, and in theory, no intervention is required from users. Furthermore, the learning speed of ELM is exceptionally fast compared to other traditional methods. Unlike current single-hidden-layer ELM methods, in this paper, we indicate that with general hidden nodes (or called sub network nodes) added to existing networks, MLELM can be used for classification, dimensions reduction, and feature learning. Figure 1.Structure of single layer feed-forward neural network In addition, Autoencoder (AE) based on deep network also can get better features and classification results. From this point of There are two training phases in ELM algorithm: feature view, deep learning is suitable for feature extraction of images. mapping and output weights solving. ELM feature mapping: However, the learning speed of deep learning is not fast. Due to Given input data x, the output function of ELM for generalized this reason, similar to deep learning, MLELM stacks Extreme SLFNs is Learning Machine Autoencoder (ELM-AE) to create a Multilayer neural network which has much faster learning speed with better 푓(푥) = ∑퐿 훽 ℎ (푥) = ℎ(푥)훽 (1) performance [14-16]. 푖=1 푖 푖 Where ℎ(푥) = [ℎ (푥), … , ℎ (푥)]is the output vector of the Table 1. Difference in Conventional learning method and Biological learning 1 퐿 푇 method hidden layer and 훽 = [훽1, … , 훽퐿] denotes the output weights between the hidden layer of L hidden nodes and the output layer. Conventional Learning Biological Learning The procedure of getting h is called ELM feature mapping which Methods maps the input data from RD to the feature space RL. In real Very sensitive to network size Stable in a wide range (tens to thousands of neurons in each applications, h can be described as module) ( ) ( ) 퐷 Difficult for parallel Parallel implementation ℎ푖 푥 = 푔 푎푖, 푏푖, 푋 , 푎푖€푅 , 푏푖€푅 (2) implementation Where g(a,b,x) is an activation function satisfying ELM Difficult for hardware “Biological” implementation implementation universal approximation capability theorems [11]. In fact, any nonlinear piecewise continuous functions (e.g. Gaussian, Sigmoid, Very sensitive to user Free of user specified specified parameters parameters etc.) can be used as activation function h. In ELM, the parameters of h are randomly produced based on a continuous probability Different network types for One module possibly for several distribution. different type of applications types of applications ELM output weights solving: In the second phase, given a Time consuming in each Fast in micro learning point training sample set, (X , t )n = 1 with 푡 = learning point i i i 푖 [0, … 0, 1 , 0, . … , 0 ]Tthe class indicator of xi, ELM aims to Difficult for online sequential Nature in online sequential i m learning learning minimize both the training error and the Frobenius norm of “Greedy” in best accuracy High speed and high accuracy output weights. This objective function used for both binary and multi-class classification tasks, can be expressed as follows, “Brains (devised by Brains are built before conventional learning applications 푛 methods)” are chosen after 휔 2 1 2 applications are present Min ∑‖ξ ‖2 + ‖훽‖퐹, 훽,ξ 2 2 푖=1 푠. 푡. 훽ℎ(푋푖) = 푡푖 − ξ푖, ∀∈ 1,2, … , n (3) 2. Extreme Learning Machine where n is the number of samples and denotes the training error of the i-th sample, ω is a regularization parameter which trades off Extreme learning machine was planned for generalized single the average of output weights and training error and denotes the hidden layer feed forward neural network [9] where the hidden Frobenius norm. The optimization problem in (3) can be layer need not be neuron alike. Unlike other neural networks with efficiently solved. Specifically, according to the Wood bury back propagation [10], the hidden nodes in ELM are randomly identity [12] the optimal β can be analytically obtained as generated, whereas the activation functions of the neurons are nonlinear piecewise continuous. The weights between the hidden 퐼 layer and the output layer are calculated analytically. The general (퐻푇퐻 + 퐿) 퐻푇푇 푖푓 퐿 ≤ 푘 ∗ 휔 architecture of single layer ELM is shown in Figure 1. 훽 = { 퐼 (4) 퐻푇(퐻퐻푇 + 푛)−1푇 표푡ℎ푒푟푤푖푠푒 휔 www.astesj.com 70 M. Parkavi et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 2, No. 1, 69-75 (2017) Where In and IL are identity matrices, and H is the hidden [21] have been proposed. OS-ELM can decrease the requirement layer output matrix (randomized matrix) which is defined in (5) of runtime memory because the model is trained based on each chunk. I-ELM can diagnose memory problem by training a basic ℎ (푥 ) … ℎ (푥 ) network with some hidden nodes and then adding hidden node to 1 1 퐿 1 the conventional network one by one. To rest the computational ℎ(푥 ) 1 . cost incurred by these operations, many variants containing . partitioned ELM [22] and parallel ELM [23], have been 퐻 = . = (5) . recommended. Recently, a variant of OS-ELM also called parallel . OS-ELM (POS-ELM) [24] and parallel regularized ELM (PR- [ℎ(푥 )] ELM) [25] are recommended to reduce training time and memory 푛 requirement instantaneously. [ℎ1(푥푛) … ℎ퐿(푥푛) ] 3. Multilayer Extreme Learning Machine 2.1. ELM-Based Autoencoder Multilayer ELM is an advanced machine learning approach Apart from the ELM-based SLFNs, the ELM theory has also based on the architecture of artificial neural network and is been applied to build an autoencoder for an MLP.