Expert Systems With Applications 56 (2016) 368–384

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Optimization of neural networks through and a

∗ Lídio Mauro Lima de Campos a, , Roberto Célio Limão de Oliveira a, Mauro Roisenberg b a Faculty of Computer Engineering, Federal University of Pará, Rua Augusto Corrêa 01, Guamá, CEP:66075-100, Belém Caixa postal 479, Pará Brazil b Informatic and Statistic Department, Federal University of Santa Catarina, PO. Box 476, Trindade, Florianópolis 88040-900, Santa Catarina, Brazil

a r t i c l e i n f o a b s t r a c t

Article history: This paper proposes a hybrid neuro-evolutive algorithm (NEA) that uses a compact indirect encoding Received 17 August 2015 scheme (IES) for representing its genotypes (a set of ten production rules of a Lindenmayer System with

Revised 18 January 2016 memory), moreover has the ability to reuse the genotypes and automatically build modular, hierarchi- Accepted 8 March 2016 cal and recurrent neural networks. A genetic algorithm (GA) evolves a Lindenmayer System (L-System) Available online 16 March 2016 that is used to design the neural network’s architecture. This basic neural codification confers scalability Keywords: and search space reduction in relation to other methods. Furthermore, the system uses a parallel genome scan engine that increases both the implicit parallelism and convergence of the GA. The fitness function Neural networks of the NEA rewards economical artificial neural networks (ANNs) that are easily implemented. The NEA Grammatical evolution was tested on five real-world classification datasets and three well-known datasets for time series fore- casting (TSF). The results are statistically compared against established state-of-the-art algorithms and various forecasting methods (ADANN, ARIMA, UCM, and Forecast Pro). In most cases, our NEA outper- formed the other methods, delivering the most accurate classification and time series forecasting with the least computational effort. These superior results are attributed to the improved effectiveness and efficiency of NEA in the decision-making process. The result is an optimized neural network architecture for solving classification problems and simulating dynamical systems. ©2016 Elsevier Ltd. All rights reserved.

1. Introduction Neuro-evolutive algorithms (NEAs) design and/or train ANNs through evolutionary algorithms (EAs). Bio-inspired algorithms Artificial Neural Networks (ANNs) are applied in classification (BIOAs) have gained popularity as efficient solvers of nonlinear op- ( Rivero, Dorado, Rabual, & Pazos, 2010 ), control systems ( Li, Jia, timization problems ( Krömer, Platoš, & Snášel, 2014 ). In this paper, Liu, & Ding, 2014 ), prediction ( Donate, Sanchez, & de Miguel, 2012 ), we investigate whether a new biologically inspired NEA can pro- and many other problems ( Czajkowski, Patan, & Szyma nski,´ 2014 ). vide an efficient automatic design tool for ANNs. As the network architecture depends on the class of problem to As discussed by ( Stanley & Miikkulainen, 2003 ) , the increas- be solved (classification, time series forecasting, or reinforcement ing complexity of evolutionary computation demands more so- learning), the full potential of an ANN can be exploited by optimiz- phisticated methods than direct mapping from genotype to phe- ing its architecture and training. The architecture is designed by notype , ( Dasgupta & McGregor, 1992; Donate et al., 2012; Miller, heuristic trial-and-error methods that select the transfer function Todd, & Hegde, 1989 ). An indirect encoding system (IES) allows and training algorithm for adjusting the synaptic weights. These a more compact and scalable representation than a direct encod- heuristics affect the learning and generalization capacities of the ing scheme (DES) ( Ahmadizar, Soltanian, & AkhlaghianTab, 2015; network and are usually performed by experts, who sequentially Hornby & Pollack, 2002; Lee, Seo, & Sim, 2007; Soltanian, Tab, Zar, train different topologies to determine the best architecture for a & Tsoulos, 2013; Stanley, D’Ambrosio, & Gauci, 2009 ). specific problem. To mitigate these deficiencies, researchers have Building an efficient encoding scheme for repetitive and proposed automatic procedures for these processes. recurrent structures is a challenging task for IES-based NEAs. The studies of ( Ahmadizar et al., 2015; Dasgupta & McGregor, 1992; Donate et al., 2012; Lee et al., 2007; Miller et al., 1989;

∗ Niska, Hiltunen, Karppinen, Ruuskanen, & Kolehmainen, 2004; Corresponding author. Tel.: +55 91 3201 7901.

E-mail addresses: [email protected] , [email protected] (L.M.L. de Campos), Sanchez & Melin, 2014; Soltanian et al., 2013; Tsoulos, Gavrilis, [email protected] (R.C.L. de Oliveira), [email protected] (M. Roisenberg). & Glavas, 2008 ) were limited to multilayer perceptron neural http://dx.doi.org/10.1016/j.eswa.2016.03.012 0957-4174/© 2016 Elsevier Ltd. All rights reserved. L.M.L. de Campos et al. / Expert Systems With Applications 56 (2016) 368–384 369 networks (MLPs). However, dynamical systems cannot be precisely enables the reuse of phenotypic structures (rewrite of nodes and estimated by MLPs, and they require recurrent neural networks connections) at different stages of development. Such reuse is an (RNNs). NEAs evolving RNNs are rarely reported in the literature important capability of NEAs. ( Beer & Gallagher, 1992; Hornby & Pollack, 2002 ). The present Our ADEANN also utilizes expert knowledge of the problem to paper proposes that the brain manipulates recurrent subnetworks more efficiently search the infinite space of topologies, thus min- with specialized functions. Such architecture is more similar to imizing the expert’s effort in the optimization. The penalty ap- biological neural networks than feedforward neural networks and proach implemented by the fitness function ( Eq. 5 ) automatically realizes more complex computation. rewards the economical ANNs with stronger generalization and Our approach is inspired by two natural biological mechanisms: extrapolation capacities. Our L-system generates ANN topologies genetic encoding and the evolution of genetic coding. As is well- without requiring additional substrate configurations for the given known, neuron development is governed by the genetic informa- problem. In this sense, our model automates the methodology of tion encoded in deoxyribonucleic acid (DNA), and ultimately gen- ( Stanley et al., 2009 ). erates the final shape of the brain. During biological development, In addition, we investigate whether increasing or decreasing in the complex process that links the DNA code to its phenotype, the chromosome length affects the crossover rate, and conse- the same gene is used in many contexts. This compactness min- quently the number of valid production rules. Our experiments imizes the information required to describe complex individuals. confirm interference between the chromosome length, crossover On the other hand, evolution describes the temporal changes in rate, and production rules. The variable associations are discussed the genetic code (DNA). Among the several mechanisms underlying in Section 6.1 . these evolutionary changes, natural selection is very important. The next section presents state-of-the-art methodologies and The two natural processes described above are hybridized such the findings of other authors. Section 3 presents the theoretical that the DNA contained in cells can also spawn cells. On the other foundation of our work. The components of ADEANN and system hand, the changes in DNA are passed onto later generations. Mo- operation are detailed in Section 4 . Section 5 describes the materi- tivated by these natural processes, we propose an artificial hybrid als and methods and Section 6 presents the general experimental system that abstracts these natural mechanisms at an acceptable design and analyzes the computational results. A general discus- level of complexity. sion and final conclusions are presented in Sections 7 and 8 . To this end, we propose a new NEA, a biologically inspired ar- tificial hybrid system called Artificial Development and Evolution 2. State-of-the-art techniques of ANNs (ADEANN). The ADEANN integrates two components. The first is a generative representation that represents genotypes (a This section discusses existing research on NEAs. Most NEAs set of production rules of a Lindenmayer system) by a compact use DESs, which specify every connection and node in the geno- IES. The IES also conducts and controls the process of mapping type that will appear in the phenotype (ANN) . In the approaches the genotypes to the phenotypes (complex neural morphologies). proposed by Dasgupta and McGregor (1992) ; Miller et al. (1989) a To mimic the DNA encoding scheme and enable scalability, our multilayer perceptron is encoded as a directed graph, and its geno- IES leverages the phenotype representation to a smaller genotype. type is represented by an adjacency matrix . The ANN architecture Thus, the search process is carried out in a lower-dimensional solu- is evolved by the ANN itself. In other words, the chromosome en- tion space. In addition, our IES implements the organizational prin- codes all details of the network architecture. The weights of the ciples of hierarchy, modularity, and gene reuse (allowing compact neural network can then be optimized by gradient-based search representation of complex phenotypes). The second component is a approaches. These methods are simply implemented, but the size genetic algorithm (GA), a simplified representation of natural evo- of the connectivity matrix scales as the square of the number of lution. In local search problems based on GAs, a bit string is called nodes. Thus, this representation blows up as the number of nodes a chromosome (the genotype). Each bit on the chromosome is a increases. gene, and a gene set represents the parameters of a function to be Stanley and Miikkulainen (2002) proposed neuro-evolution of optimized. Each string is assigned a fitness that indicates the qual- augmenting topologies (NEAT). In NEAT, the DES incorporates a few ity of its encoded solution (the phenotype). To improve the biolog- biologically plausible entities, and alters both the weighting pa- ical realism of GA, the GA in our approach evolves the generative rameters and structures of the networks. The flexible genetic en- representation. The evolutionary process can be regarded as the coding of NEAT describes a network by a list of edge genes. Each temporal genetic changes in the hypothetical DNAs of a popula- edge gene denotes an edge between two node genes, and specifies tion of individuals, regulated by an artificial selection mechanism. the in-node, out-node, and weight of the edge. The data structure The above biological inspiration underlies the originality of our ap- of NEAT, which represents the genotype, grows linearly with the proach. To our knowledge, we report the first attempt to generate number of edges between two nodes. recurrent neural networks from combined metaphors. Donate et al. (2012) proposed a new approach for auto- The main contribution of our method is the genotype represen- matic ANN design, and applied it to time-series forecasting us- tation by our proposed IES. Using a compact DNA encoding, we ing a GA. Their method alters both the weighting parameters codify a parametric Lindenmayer system (L-system) with memory, and network structures. The objective was to improve the accu- which implements the principles of organization, modularity, repe- racy of time-series forecasting. However, they restricted their ap- tition, and hierarchy to achieve complex neural architectures (mul- proach to MLP, which is unsuitable for simulating temporal series tilayer and recurrent networks).( Hornby & Pollack, 2002; Lee et al., forecasting. 2007 ) adopted L-systems. Although Lee et al. (2007) , used DNA Sanchez and Melin (2014) developed a novel multi-objective encoding, their study was restricted to feedforward neural net- optimization for a hierarchical genetic algorithm (MOHGA) based works, whereas our approach is extended to recurrent networks. on the micro-GA approach. Their method, which optimizes modu- In the IES used by Hornby and Pollack (2002) , the genotypes en- lar neural networks (MNNs), is used in iris recognition. The MO- code twenty rewrite rules of an L-system. Our DNA encoding sys- HGA automatically divides the data into granules or submodules, tem encodes a parametric L-system with memory using 10 pro- and selects the data for the training and testing phases. It deter- duction rules. Therefore, our IES is more compact than Hornby and mines the number of granules or submodules and the percentage Pollack (2002) ’s method, and reduces the search space of all feasi- of the training data to improve the results. The applicability of MO- ble solutions. In addition, the memory mechanism in our approach HGA was confirmed in human recognition. The MMNs delivered Download English Version: https://daneshyari.com/en/article/382346

Download Persian Version:

https://daneshyari.com/article/382346

Daneshyari.com