mathematics Article Optimization of the ANNs Predictive Capability Using the Taguchi Approach: A Case Study Andrea Manni 1,2, Giovanna Saviano 2 and Maria Grazia Bonelli 3,4,* 1 Chemical Research 2000 S.r.l., Via S. Margherita di Belice 16, 00133 Rome, Italy; [email protected] or [email protected] 2 Department of Chemical, Materials and Environmental Engineering (DICMA), “La Sapienza” University of Rome, Via Eudossiana 18, 00184 Rome, Italy; [email protected] 3 Programming and Grant Office Unit (UPGO), Italian National Research Council (CNR), Piazzale Aldo Moro 7, 00185 Rome, Italy 4 InterUniversity Consortium Georesources Engineering (CINIGeo), Corso Vittorio Emanuele II 244, 00186 Rome, Italy * Correspondence: [email protected] Abstract: Artificial neural networks (ANNs) are a valid alternative predictive method to the tradi- tional statistical techniques currently used in many research fields where a massive amount of data is challenging to manage. In environmental analysis, ANNs can analyze pollution sources in large areas, estimating difficult and expensive to detect contaminants from other easily measurable pollutants, especially for screening procedures. In this study, organic micropollutants have been predicted from heavy metals concentration using ANNs. Sampling was performed in an agricultural field where organic and inorganic contaminants concentrations are beyond the legal limits. A critical problem of a neural network design is to select its parametric topology, which can prejudice the reliability of the model. Therefore, it is very important to assess the performance of ANNs when applying different types of parameters of the net. In this work, based on Taguchi L12 orthogonal array, turning Citation: Manni, A.; Saviano, G.; experiments were conducted to identify the best parametric set of an ANNs design, considering Bonelli, M.G. Optimization of the different combinations of sample number, scaling, training rate, activation functions, number of ANNs Predictive Capability Using hidden layers, and epochs. The composite desirability value for the multi-response variables has the Taguchi Approach: A Case Study. been obtained through the desirability function analysis (DFA). The parameters’ optimum levels Mathematics 2021, 9, 766. https:// doi.org/10.3390/math9070766 have been identified using this methodology. Academic Editor: Daniel-Ioan Curiac Keywords: artificial neural network; Design of Experiment (DoE); parametric design; forecasting; environmental pollution Received: 23 February 2021 Accepted: 29 March 2021 Published: 1 April 2021 1. Introduction Publisher’s Note: MDPI stays neutral Artificial neural networks (ANNs) are a machine learning technique that is widely with regard to jurisdictional claims in used as an alternative forecasting method to traditional statistical approaches in many sci- published maps and institutional affil- entific disciplines [1], such as marketing, meteorology, finance, or environmental research, iations. when a massive amount of data is challenging to manage. Unlike conventional methodologies, ANNs are data-driven self-adaptive, and non- linear methods that do not require specific assumptions about the underlying model [2]. ANNs are data-driven because they can process the available data (input) and produce a Copyright: © 2021 by the authors. target variables vector (output) through a feed-forward algorithm. Moreover, the method Licensee MDPI, Basel, Switzerland. is self-adaptive. In fact, a neural network can approximate a wide range of statistical This article is an open access article models without hypothesizing in advance on any relationships between the dependent distributed under the terms and and independent variables. Instead, the form of the relationship is determined during the conditions of the Creative Commons network learning process. If a linear relationship between the dependent and independent Attribution (CC BY) license (https:// variables is appropriate, the neural network results should closely approximate those of creativecommons.org/licenses/by/ the linear regression model. If a non-linear relationship is more appropriate, the neural 4.0/). Mathematics 2021, 9, 766. https://doi.org/10.3390/math9070766 https://www.mdpi.com/journal/mathematics Mathematics 2021, 9, x FOR PEER REVIEW 2 of 17 Mathematics 2021, 9, 766 2 of 16 approximate those of the linear regression model. If a non-linear relationship is more appropriate, the neural network will automatically match the “correct” model structure. networkTherefore, will this automatically algorithm is matchsuitable the for “correct” approximating model structure. complex relationships Therefore, this between algorithm the isinput suitable and foroutput approximating variables with complex a non-linear relationships optimization between [3]. the input and output variables with aANNs non-linear algorithms optimization simulate [3 how]. the human brain processes information through the nerveANNs cells, algorithms or neurons, simulate connected how theto humaneach other brain in processes a complex information network, through within the a nervecomputational cells, or neurons, model [4]. connected The similarity to each otherbetween in a an complex artificial network, neural within network a compu- and a tationalbiological model neural [4 ].network The similarity relies on between the network’s an artificial acquirement neural network of knowledge and a biologicalthrough a neural“learning network process” relies [5]. on From the network’sthe initial data acquirement (input), it of is knowledge possible to throughdetermine a “learningthe target process”variable (output) [5]. From through the initial the datacomplex (input), system it is of possible cause–effect to determine relationships the target that the variable model (output)discovers. through the complex system of cause–effect relationships that the model discovers. TheThe neuronneuron (node)(node) isis thethe primaryprimary processingprocessing unitunit inin neuralneural networks.networks. EachEach artificialartificial neuronneuron hashas weightedweighted inputs,inputs, transfertransfer functions,functions, andand oneone output.output. AnAn exampleexample ofof neuralneural propagationpropagation isis thethe McCulloch–PittsMcCulloch–Pitts modelmodel (Figure(Figure1 1),), which which combines combines its its inputs inputs (typically (typically asas aa weightedweighted sum)sum) toto generategenerate aa singlesingle output.output. FigureFigure 1.1. ExampleExample ofof thethe modelmodel ofof neuralneural propagation.propagation. InIn thisthis model, model, the the inputs inputs xi arexi are multiplied multiplied by theby weightsthe weights wi, with wi, wwith0 as w bias.0 as Biasbias. shifts Bias theshifts activation the activation function function by adding by aadding constant a constant to the input to the to better input fit to the better data fit prediction. the data Biasprediction. in neural Bias networks in neural can networks be thought can ofbe asthou analogousght of as toanalogous the role of to athe constant role of ina constant a linear function,in a linear whereby function, the whereby constant the value consta transposesnt value transposes the line: without the line: w 0without, the line w would0, the line go throughwould go the through origin (0,the 0), origin and the(0, 0), fit and could the be fi poor.t could If thebe poor. weighted If the sum weighted of the sum inputs of “a”the overcomesinputs “a” overcomes a threshold a valuethreshold T, neurons value T, release neurons its release output its y, output which isy, awhich function is a function of (a-T). Inof other(a-T). words, In other the arrivingwords, the signals arriving (inputs), signal multiplieds (inputs), by themultiplied connection by weights, the connection are first summedweights, are and first then summed passed and through then anpassed activation through function an activation (θ) to produce function the(θ) outputto produce [6]. Athe unit output feeds [6]. its A output unit feeds to all its the output nodes to on all the the next nodes layer, on but the there next islayer, no feedback but there to is the no previousfeedback layer to the (feed-forward previous layer network). (feed-forwar Weightsd representnetwork). the Weights system memoryrepresent and the indicate system thememory importance and indicate of each inputthe impo neuronrtance in theof each output input generation neuron processing. in the output The generation activation functionprocessing. is usedThe activation to introduce function non-linearity is used into theintroduce network’s non-linearity modeling in capabilities the network’s [7]. Activationsmodeling capabilities are typically [7]. a Activations number within are thetypi rangecally a of number 0 to 1, and within the weightthe range is aof double, 0 to 1, e.g.,and 2.2,the weight−1.2, 0.4. is a double, e.g., 2.2, −1.2, 0.4. InIn aa neuralneural network,network, therethere areare manymany parametersparameters forfor buildingbuilding aa model.model. ItIt isis possiblepossible toto changechange thethe numbernumber ofof layers,layers, thethe numbernumber ofof neuronsneurons forfor eacheach layer,layer, thethe typetype activationactivation functionfunction toto useuse inin eacheach layer,layer, thethe training training duration,duration, andand thethe learning learning rule. rule. TheThe optimizationoptimization of of neural neural network network architecture architecture is a is reccurring a reccurring topic topic of several of several stud- ies.studies.