              ARTIFICIAL NEURAL NETWORKS: A REVIEW OF TRAINING TOOLS   Darío Baptista, Fernando Morgado-Dias   Madeira Interactive Technologies Institute and Centro de Competências de Ciências Exactas e da Engenharia, Universidade da Madeira Campus da Penteada, 9000-039 Funchal, Madeira, Portugal. Tel:+351 291-705150/1, Fax: +351 291-705199     Abstract: Artificial Neural Networks became a common solution for a wide variety of problems in many fields. The most frequent solution for its implementation consists of building and training the Artificial Neural Network within a computer. For implementing a network in an efficient way, the user can access a large choice of software solutions either commercial or prototypes. Choosing the most convenient solution for the application according to the network architecture, training algorithm, and price can be a complex task. This paper helps the Artificial Neural Network user by providing a large list of solution available and explaining their characteristics and terms of use. The paper is confined to reporting the software products that have been developed for Artificial Neural Networks. The features considered important for this kind of software to have in order to accommodate its users effectively are specified. The development of software that implements Artificial Neural is a rapidly growing field driven by strong research interests as well as urgent practical, economical and social needs. Copyright CONTROLO2012  Keywords: Artificial Neural Networks, Training Tools, Training Algorithms, Software.   1. INTRODUCTION commercialization of new ANN tools. With the  purpose to inform which tools are available at present Nowadays, in different areas, it is important to and make the choice of which tool to use, this paper analyse nonlinear data to do prediction, classification contains the description of software that has been or to build models. For this purpose, a nonlinear developed specifically for ANN, independently of model such as the Artificial Neural Network (ANN) the language used. It is important to refer that some can be used. An ANN is an interconnection of tools were created and used for academic studies and neurons to build a network, whose inspiration comes others are commercial and sophisticated tools, which from the natural neural networks that compose the help the mainstream business users take decisions brain. The processing capacity of the ANN is stored about their company. in the weights of the connections and they are The paper does not cover ANN such as spiking obtained, in the most common case of supervised neurons and other models that closely biologically training, by a learning process that uses examples or inspired. training patterns [1].  Each year more ANN tools are being used for 2. ARTIFICIAL NEURAL NETWORKS TOOLS data collecting, not only for academic studies, but  also to manage decisions in some large and mid- Most of the ANN’s tools are documented through range companies because this data holds valuable scientific papers [2-12] or websites [13-58] with information, e.g., trends and patterns, which could be enough information for the user to make an informed used to improve business decisions and optimize decision of whether to test and use the proposed tool success. This need for an automated extraction of or to search for another one that fits its needs. useful knowledge from enormous quantity of data is  widely recognized, which leads to creations and 1.1 System Requirement

Table 1 List the operating system for each tool. The most common set of requirements defined by OS Mac Window Unix Linux Sun Others any tool is the operating system. To be used Tools OS X efficiently, each tool needs an appropriate Operating AiNet  (3.1 to NT)      System. Table 1 shows the recommended operating  system for the proper functioning. It also shows that Annie      (XP or later) the majority of the tools use Windows as the Aspirin/Migraines       operating system. Outside the Microsoft universe, the

Basis-of-AI-NN  operating system most often used for the realization (3.1)      of an ANN’s tool is the Mac OS. The operating EasyNN     (all version)   system requirements of some of the tools are often Encog  accompanied by hardware and software minimum (all version)      characteristics. These requirements correspond to the

FANN    (all version)    elements that are necessary to be installed on the FastICA any OS (need Matlab platform - tool constituted by m files) computer to provide optimal functioning of tool. It is FuNeGen  important to remind that these software pre-requisites (all version)      Fuzzy ART/ Fuzzy are not included in the tool installation package and any OS (need Matlab platform - tool constituted by m files) ARTMAP need to be installed separately before installing the GENESIS 2.3       required tool. JATTON       Table 2 shows the various aspects of software and (all version) hardware requirements necessary for each tool. It is

Java NNS   (NT or later)    (OS 7)  important to remind the user that exceeding by far  these requirements does not guarantee to the user that Joone      (all version) everything will run with absolute smoothness and

Lens  (all version)      look its best.

LM-MLP       (95 or later) Table 2. List of software requirements Multiple     HD Video   Tools RAM Processor JRE Free Space Resolution Color  NNFit    AIX; IRIX  (OS 4.1.4) AiNet 4MB 5MB     NNSYSID any OS (need Matlab platform - tool constituted by m files) 1.5 or Encog NNT any OS (need Matlab platform - tool constituted by m files)      later Intel and GENESIS 2.3 NetLab any OS (need Matlab platform - tool constituted by m files)     AMD 64 bit  1.6 or  JATTON NeuroModeler           later (95 to XP) 1.3 or Java NNS Neural Network      later Toolbox any OS (need Matlab platform - tool constituted by m files) Joone 256MB       NeuroIntelligence      LM-MLP 32MB Pentium I or II (98or later)     Neuroph       Nenet    16 bits    NetLab  456KB     NeuroSolutions      Processor (XP or later) NeuroModeler 128MB    Speed: 1GHz  AIX ; Nest      NeuroIntelligence 128MB 15MB 800x600 8 bits Pentium II  (10.3) (Solaris) SGI; 1.6 or Neuroph NevProp            later NeuroSolutions 100MB 1024x768 Pentium IV Nico Toolkit          1.4 or Nico Toolkit      later Nuclass       (NT or later) Nuclass 32MB 25MB     Numap       Numap 32MB 25MB     (NT or later) Pythia 32 MB 1.39 MB Pentium PDP++ (known as         SGI Emergent) (95 or later) SOM 64 MB 1 MB     Processor Statistica 1GB 172.84MB Pythia         Speed: 2GHz  (95 or later) 1.5 or Weka Ultrix;      later SNNS      (OS 4;5) AIX; IRIX  There is not any specification. SOM any OS (need Matlab platform - tool constituted by m files) SOM_PAK - LVQ_PAK       1.2. Network architectures for each neural tools.  Statistica       (NT or later) Table 3 and Table 4 show the networks architecture Torch (*)     FreeBSD and the training algorithm for each tool. It can be seen that the networks more usual are Radial Basis Trajan       Function, Multiplayer and Self- Uts      IRIX (OS 5.3) Organizing Map and the training algorithm more usual is the Backpropagation. WEKA       (10.6)  AIX;Ultrix;  XNBC      (95 to XP) Hpux  (*) is not yet Windows compatible, coming soon;   There is;  There is not. 

Table 3. List the network architecture for each tool. Legend: Tools Arquitecture  N1 – Arbitrarily connected neuron; AiNet N38 N2 – Adaline Linear Neuron; N19; N20; N22; N27; N32; N38; Annie N47 N3 – Autoassociative Memory; Aspirin/Migraines N13; N43 N4 – Adaptive Resonance Theory; Basis-of-AI-NN N4; N6; N9; N20; N22; N27; N29 N5 – Adaptive Resonance Theory Mapping N6 – Bidirectional Associative Memory; EasyNN N38 N7 – BayesNet/Bayesian Network; Encog N2; N4; N6; N9; N16; N19; N27; N31; N32; N38; N42; N47; N50 N8 – Bipolar network; FANN N20 N9 – Boltzmann Machine FastICA N30 N10 – Bridged Multiplayer Perceptron; N11 – Back-Propagation network; FuNeGen N12 N12 – Coactive Network – Fuzzy Interference Fuzzy ART/ Fuzzy ARTMAP N4; N5 System; GENESIS 2.3 N20; N26 N13 – Canonical Discriminants Analysis JATTON N11; N16; N32 N14 – Convolutional Network; Java NNS N20 N15 – Competitive neural network; N16 – Counter-Propagation neural network; Joone N20; N32; N39; N43; N52 N17 – Deep Boltzmann Machines/Mean-field; Lens N9; N20; N22; N29; N32 N18 – Dynamic network; LM-MLP N38 N19 – Elman Network; Multiple Backpropagation N20; N38 N20 – Feed-Forward neural network; NNFit N38 N21 – Functional Link Network; N22 – Recurrent network; NNSYSID N20; N38 N23 – Generalized regression neural network; NNT N1; N8; N10; N38 N24 – Hebbian Network; NetLab N20; N24; N35; N38; N47; N50 N25 – Hybrid Models NeuroModeler N2; N4; N6; N9; N16; N38; N42 N26 – Hodgkin & Huxley Neural Network; N27 – ; Neural Network Toolbox N2; N18; N19; N22; N27; N33; N38; N47; N50; N52 N28 – Integrate and fire NeuroIntelligence N11; N38 N29 – Interactive Activation Network; N2; N6; N15; N24; N27; N32; N34; N30– Independent component analysis; Neuroph N38; N47 N31 – Jordan Recurrent NeuroSolutions N12; N19; N20; N22; N23; N38; N39; N46; N47; N50; N51 N32 – Kohonen networks; Nest N4; N28 N33 – Learning Vector Quantization nets; NevProp N20; N38 N34 – Maxnet; N35 – Mixture Density Network; Nico Toolkit N18; N38; N47 N36 – Multidimensional Scaling; Nuclass N38; N45; N50 N37 – Mixture of Gaussians Networks; Numap N38; N45; N50 N38 – Multiplayer Perceptron; N9; N17; N20; N24; N27; N27; N39 – Modular neural Network; PDP++ (also known as Emergent) N36; N40; N43; N49; N50 N40 – Mixture of Experts; N11; N38 Pythia N41 – Neocognitron Neural Network; N3; N4; N6; N16; N19; N27; N33; SNNS N44; N50; N22; N52 N42 – Neuroevolution of Augmenting Topologies SOM N43; N50 N43 – Principal Component Analysis; N44 – Pruned Cascade-Correlation; SOM_PAK - LVQ_PAK N33; N50 N45 – Piecewise linear network Statistica N23; N38; N41; N46; N47; N50 N46 – Probabilistic Neural Network; Torch N14; N38; N40; N47; N52 N47 – Radial Basis Function; Trajan N23; N38; N46; N47; N50 N48 – Space Mapping Uts N37; N50 N49 – Stochastic Neural Network; N50 – Self-Organizing Map; WEKA N7; N38; N47 N51 – Support Vector Machine; XNBC N26; N28 N52– Time-Delay neural Network;                 

Table 4. List the network training algorithm for each Legend: tool.  A1 – Back propagation Tools Arquitecture A2 – Batch training AiNet – A3 – Batch version of the Back-Propagation Annie A1; A8 A4 – Batch training with weight and bias learning Aspirin/Migraines A1; A7; A44; A49 rules Basis-of-AI-NN A1; A10;A25; A27; A37 A5 –Bayesian regularization

EasyNN A1 A6 – Conjugate Gradient A7 – Conjugate Gradient Descent Encog A1; A6; A8; A16; A29 A8 – Competitive Learning FANN A1; A2; A22; A37; A39 A9 – Cyclical order incremental update FastICA A13 A10 – Delta-Bar-Delta A11 – Enhanced Self Aware algorithm FuNeGen A1; A7 A12 – Evolutionary Gradient Fuzzy ART/ Fuzzy ARTMAP A1 A13 – fast fixed-point algorithm GENESIS 2.3 A1 A14 – fixed global adaptive JATTON A1; A8 A15 – Fletcher-Powell conjugate gradient back Java NNS A1 propagation A16 – Genetic Algorithm Joone A1; A2;A7; A10; A39 A17 – Gradient descent back propagation Lens A1; A8; A10; A31 A18 – Gradient descent with adaptive learning rule LM-MLP A29 back propagation Multiple Backpropagation A1 A19 – Gradient descent with momentum and NNFit A35 adaptive learning rule Backpropagation A20 –Gradient descent with momentum back NNSYSID A1; A3; A21; A23; A29; A30; A38 propagation NNT A1; A11; A12; A32; A40 A21 – Recursive prediction error (Gauss-Newton). NetLab A1; A6; A29 A22 – Incremental training NeuroModeler A1; A6; A16; A35; A46 A23 – Iterated generalized least squares A1; A2; A3; A4; A5; A9; A15; A17; A24 – k-Means Neural Network Toolbox A18; A19; A20; A22; A29; A33; A34; A35; A41 A25 – k-Nearest neighbour A26 – Lazy Bayesian Rules NeuroIntelligence A3; A6; A7; A22; A29; A35; A36; A37 A27 – Learning Vector Quantization Neuroph A1; A8 A28 – Leabra A29 – Levenberg-Marquardt NeuroSolutions A1; A6; A10; A29; A31; A37; A43 A30 – Memory-saving implementation of the LM. Nest – A31 – Momentum NevProp A14; A37 A32 – Neuron by Neuron algorithm Nico Toolkit – A33 – Polak-Ribiére conjugate gradient back propagation Nuclass A24; A25 A34 – Powell-Beale conjugate gradient back Numap A24; A25 propagation PDP++ (also known as Emergent) A1; A8; A28; A42 A35 – Quasi-Newton A36 – Quasi-Newton (Limited Memory) Pythia – A37 – Quick propagation SNNS A1; A37 A38 – Recursive (/incremental) version of back- SOM A24; A25; A27 propagation SOM_PAK - LVQ_PAK A27 A39 – Resilient Propagation (RPROP) A40 – Self Aware algorithm Statistica A1; A6; A10; A29; A35; A37 A41 – Scaled conjugate gradient Backpropagation Torch A7 A42 – Soft Competitive Learning Trajan A1; A6; A10; A29; A35; A37 A43 – Step by step Uts A6 A44 – Stability Issues

WEKA A1; A26 A45 – Sequential minimal optimization algorithm A46 – Simulated Annealing XNBC – A47 – Weight perturbation  A48 – Weight update driven node splitting

A49 – Weight Update Issues        

3. CONCLUSIONS Software: An Update; SIGKDD Explorations,  Volume 11, Issue 1. As has been pointed out, ANNs are models with a very wide application. More and more practitioners SHANKAR, Asim. Annie - Artificial Neural from different fields of study use these models to Network Library. Last Update: Jun 18th, 2002. solve problems that would otherwise be more Available at: difficult to solve. For most of these users developing software for their ANN is not considered and they LEIGHTON, Russell. Elegant Software. need fast and reliable solutions. Aspirin/Migraines. Last accessed: 13rh Out 2010. In this paper an overview of the different software Available at: as a way to simplify the practitioner job. The facts related to licenses, requirements, additional TVETER, Don. The Pattern Recognition Basis of AI software, training algorithms and network Neural Networking Software. Last accessed: 14th architectures are pointed out throughout the paper for Out 2010. the easy selection of the user. Available at:

ACKNOWLEDGMENTS Neural Planner Software. EasyNN Plus. Last The authors would like to acknowledge the accessed: 12th Out 2010. Portuguese Foundation for Science and Technology Available at: for their support for this work through project PEst- OE/EEI/LA0009/2011. Laboratory of Computer and Information Science. The FastICA package for MATLAB. Last accessed: 12th Out 2010. REFERENCE Available at: R. Morgado F. Dias, Fernando (2005). Técnicas de controlo não-linear baseadas em Redes GenAlgo Team. FuNeGen 1.0. Last accessed: 13th Neuronais: do algoritmo à implementação. PhD Out 2010. thesis, Departamento de Electrónica e Available at:

Huxhold, W.L.; Henson, T.F.; Bowman, J.D.; IBM Aaron Garrett. MatLab Central: Fuzzy ART and Corp., Houston, TX (1992). ANNIE: a simulated Fuzzy ARTMAP Neural Networks. Last neural network for empirical studies and accessed: 16th Out 2010. application prototyping. Annie. Simulation Available at: < http://www.mathworks.com/ Symposium. Proceedings. 25th Annual USA, pp. matlabcentral/fileexchange/4306> 2-8. GENESIS. Last accessed: 15th Out 2010. Aapo, H. (1999) Fast and Robust Fixed-Point Available at: < http://www.genesis-sim.org/ Algorithms for Independent Component GENESIS/> Analysis. FastICA. IEEE Transactions on Neural Networks, Vol. 10, nº3, 626-63. João Aires de Sousa. JATOON - Java Tools for Neural Networks. Last accessed: 13rd Out 2010. Yu, H. and Wilamowski, B. M (2009). Efficient and Available at: Neural Network Trainer. IEEE Human System Interaction Conference, Italy, May 21-23, pp. Prof. Dr. Andreas Zell, JavaNNS – Java Neural 109-115. Network Simulator. Last accessed: 11rd Out 2010. Available at: Recognition. UK: Springer. Java Tips, Joone. Last accessed: 15th Out 2010. Lopes, N. and Ribeiro, B. (2009), GPU Available at: Propagation Algorithms In Proceedings of Intelligent Data Engineering and Automated Douglas Rohde, Lens the light, efficient network Learning, Springer-Verlag, pp. 449-456. simulator. Last accessed: 14th Out 2010. Available at: Mark H., Eibe F., Geoffrey H., Bernhard P., Peter R., Ian H. W. (2009), The WEKA Neural Decision Lab LLC, Software. LM_MLP, Nuclass and Numap.Last accessed: 13rd Out 2010.

Available at: download/pythia-the-neural-network-designer- v1.00-2523.html> Noel J. M. Lopes. Multiple Back-Propagation. Last accessed: 16th Out 2010. Dr. Andreas Zell, Stuttgart Neural Network Available at: Simulator. Last accessed: 11rd Out 2010. Available at: Grandjean; Jules Thibault. NNFit (Neural Network Fitting). Last accessed: 11rd Out 2010. Laboratory of Computer and Information Science. Available at: Available at: < http://www.cis.hut.fi/projects/ somtoolbox/> Ole Ravn; Magnus Nørgaard. The NNSYSID Toolbox for use with MatLab. Last accessed: 12nd Jussi Hynninen. SOM_PAK and LVQ_PAK. Last Out 2010. accessed: 14th Out 2010. Available at: -research/nnrc-programs.shtml>

B. M. Wilamowski. Neural network training Statsoft. Statistica. Last accessed: 17th Out 2010. software for networks with arbitrarily connected Available at: < http://www.statsoft.com/> neurons. Last accessed: 15th Out 2010. Available at: 2010. Available at: < http://www.torch.ch/> MathWorks. Neural Network Toolbox – Design and simulate neural networks. Last accessed: 11th Out TrajanSoftware. TRAJAN 6.0 PROFESSIONAL. 2010. Last accessed: 9th Out 2010. Available at: co.uk/>

Alyuda Research. ALYUDA. NeuroSolutions. Last Q. J. Zhang, Carleton University. NeuroModeler. accessed: 12nd Out 2010. Last accessed: 6th Jan 2012. Available at: networks-software.htm> Heaton Research and Encog. Encog Java and DotNet Softpedia. Neuroph is lightweight Java neural Neural Network Framework. Encog. Last network framework.Last accessed: 15th Out 2010. accessed: 6th Jan 2012. Available at: encog>

NeuroDimension. NeuroSolutions–Premier Neural J-F Vibert and F Alvarez. XNBC: a software package Network Development Environment. Last to simulate biological neural networks for accessed: 15th Out 2010. research and education. XNBC. Last accessed: Available at: 17th Out 2010. Available at:< http://www.b3e.jussieu.fr/xnbc/> Phil Goodman. Easy to use Feed-Forward Backpropagation program. Nevada Marc-Oliver Gewaltig.Nest Initiative. Nest. Last Backpropagation (NevProp). Last accessed: 10th accessed: 17th Out 2010. Out 2010. Available at:< http://www.nest-initiative.org> Available at:

KTH. The NICO Toolkit. Last Update: –. Last accessed: 14th Out 2010.Available at:

The PDP++ Software Home Page. Last accessed: 10th Out 2010. Available at:

Runtime Software. Pythia - The Neural Network Designer. Last accessed: 16th Out 2010.