Artificial Neural Networks· T\ Briet In!Toquclion

Home , Artificial neuron, Soma (biology)

GENERAL I ARTICLE

Artificial Neural Networks· t\ BrieT In!TOQuclion

Jitendra R Raol and Sunilkumar S Mankame Artificial neural networks are 'biologically' inspired net works. They have the ability to learn from empirical datal information. They find use in computer science and control engineering fields.

In recent years artificial neural networks (ANNs) have fascinated scientists and engineers all over the world. They have the ability J R Raol is a scientist with to learn and recall - the main functions of the (human) brain. A NAL. His research major reason for this fascination is that ANNs are 'biologically' interests are parameter inspired. They have the apparent ability to imitate the brain's estimation, neural networks, fuzzy systems, activity to make decisions and draw conclusions when presented genetic algorithms and with complex and noisy information. However there are vast their applications to differences between biological neural networks (BNNs) of the aerospace problems. He brain and ANN s. writes poems in English.

A thorough understanding of biologically derived NNs requires knowledge from other sciences: biology, mathematics and artifi cial intelligence. However to understand the basics of ANNs, a knowledge of neurobiology is not necessary. Yet, it is a good idea to understand how ANNs have been derived from real biological neural systems (see Figures 1,2 and the accompanying boxes). The soma of the cell body receives inputs from other neurons via Sunilkumar S Mankame is adaptive synaptic connections to the dendrites and when a neuron a graduate research student from Regional is excited, the nerve impulses from the soma are transmitted along Engineering College, an axon to the synapses of other neurons. The artificial neurons Calicut working on direct are called neuron cells, processing elements or nodes. They neural network based attempt to simulate the structure and function of biological (real) adaptive control. neurons. Artificial neural models are loosely based on biology since a complete understanding of the behaviour of real neuronal systems is lacking. The point is that only a part of the behaviour of real neurons is necessary for their information capacity. Also,

------RESONANCE ·I February. 1996 47 GENERAL ! ARTICLE

input signals ~ndritiC spine l where synapse n ~+axons ~ is possible p -----~ f t u ____-.J/ S P soma t summation u synapses soma threshold t (weights)

Model of Artificial Neuron

Figure 1 Biological and ar flflclal neuron models have Biological Neuron System certain features in common. Weights in ANN modeVrep • Dendrites input branching tree of fibres - connect to a set of other resent synapses of BNN. neurons-receptive surfaces for input signals • Soma cell body -all the logical functions ofthe neurons are realised here • Synapse specialized contacts on a neuron - interfaces some axons to the spines of the input dendrites - can increase/dampen the neuron excitation • Axon nerve fibre - final output channel - signals converted into nerve pulses (spikes) to target cells

Artificial Neuron System

• Input layer the layer of nodes for data entering on ANN • Hidden layer the layer between input and output layers • Output layer the layer of nodes that produce the network's output Artjfjcial neural responses networks have the • Weights strength or the (gain) value of the connection between appamnt obility to nodes imitate the brain's activity to make it is easier to implement/simulate simplified models rather than decisions and draw complex ones. conclusions when presented with The first model of an elementary neuron was outlined by complex and noisy McCulloch and Pitts in 1943. Their model included the elements information. needed to perform necessary computations, but they could not

\..-~------.-- --- 48 RESONANCE I February 1996 GENERAL I ARTICLE ------

SignallInfonnation ------> flow output bias

inputs

input layer I hidden layer I output layer

Feed Forward Neural Network Non-linear activation' f ' realise the model using the bulky vacuum tubes of that era. FIguf'fI2 A fyplCtlI arllfldtll McCulloch and Pitts must nevertheless be regarded as the pio neural network /fHd for neers in the field of neural networks. ward} does nofusIHII/y have fHdbtlck - fltIW of slglJQV Information Is Dnly In the ANNs are designed to realise very specific computational tasks/ folWtlrd dll'fldlon. A multi problems. They are highly interconnected, parallel computa layer fHd foIward neural tional structures with many relatively simple individual process nlltwork /FFNN} Is shown ing elements (Figure 2). A biological neuron either excites or here. The non-linear char inhibits all neurons to which it is connected, whereas in ANNs acterIsNcs of ANNs aM dUll to thll non-linear adMlflDn either excitatory or inhibitory neural connections are possible. fundlon f. This Is USBfuI for There are many kinds of neurons in BNNs, whereas in ANNs only accurate mtHllllllng ofnon certain types are used. It is possible to implement and study linear sysffIms.. various ANN architectures using simulations on a personal com puter (PC). Characteristics of non-linear systems Types of ANN s In non-linear systems, the output variables do not There are mainly two types of ANNs: feed forward neural net depend on the input vari works (FFNNs) and recurrent neural networks (RNN s). In FFNN ables in a linear manner. there are no feedback loops. The flow of signals/information is The dynamic characteris only in the forward direction. The behaviour of FFNN does not tics of the system itself depend on past input. The network responds only to its present would depend on either input. In RNN there are feedback loops (essent~ally FFNN with one or more of the follow output fed back to input). Different types of neural network ing: amplitude of the input architectures are briefly described next. signal. its wave form. its frequency. This is not so in • Single-layer feed forward networks: It has only one layer of the case of a linear system .

49 GENERAL I ARTICLE

Figure 3 Recun-ent neural network has feedback loops unit with output fed back to in delay put.

computational nodes. It is a feed forward network since it does not have any feedback.

• Multi-layer feed forward networks: It is a feed forward network with one or more hidden layers. The source nodes in the input layer supply inputs to the neurons of the first hidden layer. The outputs of the first hidden layer neurons are applied as inputs to the neurons of the second hidden layer and so on (Figure 2). If The first model of an every node in each layer of the network is connected to every other elementary neuron node in the adjacent forward layer, then the network is called fully was outlined by connected. If however some of the links are missing, the network McCulloch and Pitts is said to be partially connected. Recall is instantaneous in this in 1943. The model type of network (we will discuss this later in the section on the included the uses of ANNs). These networks can be used to realise complex elements to perform input/output mappings. necessary computations , but • Recurrent neural networks: A recurrent neural network is one in they could not which there is at least one feedback loop. There are different kinds

realisE ~ the model of recurrent networks depending on the way in which the feed using the bulky back is used. In a typical case it has a single layer of neurons with vacuum tubes of each neuron feeding its output signal back to the inputs of all that era. McCulloch other neurons. Other kinds of recurrent networks may have self and Pitts must feedback loops and also hidden neurons (Figure 3). nevertheless be rega rded as the • Lattice networks: A lattice network is a feed forward network pioneer s in the with the output neurons arranged in rows and columns. It can fie id of neural have one-dimensional, two-dimensional or higher dimensional netWorks. arrays of neurons with a corresponding set of source nodes that

------~-=~------50 RESONANCE I February 1996 GENERAL ! ARTICLE

Input_yer of source :rerinput nodes source nodes

One Dimensional Two-Dimensional supply the input signals to the array. A one-dimensional network Figure 4 tafflce networks of three neurons fed from a layer of three source nodes and a two are FFNNs with output neu rons a"anged In rows and dimensional lattice of 2-by-2 neurons fed from a layer of two columns. The one-dimen source nodes are shown in Figure 4. Each source node is connected sional nefwDrk offhree neu to every neuron in the lattice network. rons shown here Is fed from a layer of three source There are also other types of ANNs: Kohonen nets, adaptive nodes. The fwD-dimensional resonance theory (ART) network, radial basis function (RBF) lafflce Is fed from a layer of network, etc., which we will not consider now. two source nodes.

Training an ANN

A learning scheme for updating a neuron's connections (weights) was proposed by Donald Hebb in 1949. A new powerful learning law called the Widrow-H offleaming rule was developed by Bernard Widrow and Marcian Hoff in 1960.

How does an ANN achieve 'what it achieves'? In general, an ANN structure is trained with known samples of data. As an example, if a particular pattern is to be recognised, then the ANN is first trained with the known pattern/information (in the form of digital signals). Then the ANN is ready to recognise a similar ANNs are used for pattern when it is presented to the network. If an ANN is trained pattern recognition, with a character 'H' , then it must be able to recognise this image processing, character when some noisy/fuzzy H is presented to it. A Japanese signal processing/ optical character recognition (OCR) system has an accuracy of prediction, adaptive about 99% in recognising characters from thirteen fonts used to control and other train the ANN. The network learns (updates its weights) from the related problems.

~~~~~_AAnAA_~ ___ __~_~ ____ ~~ __~ RESONANCE I February 1996 'V V VV V v' 51 GENERAL : ARTICLE

How does on ANN given data while trying to minimise some cost function of the achieve 'what it 'error' between the training set data (output) and its own output. achieves'? In This learning is accomplished by the popular back-propagation general, an ANN (BPN) algorithm, which is slow in convergence. It is a learning structure is trained method in which an output error is reflected to the hidden layers with known (for updating the hidden layer weights). In essence, the numerical samples of data. values of the weights (strengths ofsynapses; see the box and Figure For example, if (] 2) of the network are updated using some rule of the type: particular pattern is to be recognised, new weights = old (previous) weights + learning rate times gradient then the ANN is first of the cost function (with respect to the weights), trained with the known pattern/ Once the ANN is trained, the 'learned information' is captured in information (in the these weights in a condensed (and yet complex) way. In general, form of digital these weights have no direct physical significance or meaning signals). Then the related to the process or phenomenon which is described or ANN is ready to modelled by the ANN. However, the overall function of the ANN recognise a similar is relevant to a given task/application. pattern when it is presented to Although the basic processing element is simple, when several of the network. them are connected (and also needed in many applications), the size of an ANN could be very large in terms of the number of neurons (hundreds) in a given layer. A maximum of one hidden layer is often sufficient for many tasks, to attain the required accuracy. The larger the ANN, the more time it takes to train the network. Despite the simplicity ofthe basic units, the mathemati cal theory to study and analyse the various structures and schemes based on ANNs and their (non-linear) behaviour can be very involved.

Uses of ANNs

The ANNs are used for pattern recognition, image processing, signal processing/prediction, adaptive control and other related problems. Due to the use of non-linear activation functions, the ANNs are highly suitable for very accurate mapping (modelling) of non-linear systems based on the input/output data. This non-

52 ------~-~-~------R-E-S-O-N-A-N-C-E--I-Fe-b-r-Ua-r-Y-1-9-9-6 GENERAL I ARTICLE linear activation function is also useful in reducing the adverse effect of outliers/spikes in data, and in improving the accuracy of RNNs estimates obtained by RNN (see the box). RNNs are specially suit able for estimation of pa Some of the uses of ANNs (FFNNs and RNNs) are briefly rameters of dynamic sys mentioned here: tems in an explicit way. The RNN-based schemes Information storage/recall- The recall is a process ofdecoding • for parameter estimation the previously stored information by the network. have been shown (by the Pattern recognition/classification - to recognise a specific • first author) to be the pattern from a cluster of data! to classify sets of data/informa generalisation of some of tion. the conventional param N on-linear mapping between high -dimensional spaces (math • eter estimation methods. ematical modelling of non-linear behaviour of systems). These methods are useful • Time-series prediction (like weather forecasting), modelling for estimation of param of non-linear aerodynamic phenomena, detection of faults/ eters of dynamic systems failures in systems like power plants and aircraft sensors. in explicit ways, e.g. math ematical modelling of air Some of the specific applications (based on open literature)/ craft dynamics. When possibilities are: realised in hardware, the RNN architectures are • Assisting IC-CIM (computer integrated manufacturing of such that they are natu integrated circuits). rally adapted to obtain fast • Japanese-OCR (optical character recognition). solutions toparameter es • Financial forecasting (the Neuro-Forecasting Centre in timation problems (de London). pending on the speed of • Process control (Fujitsu Ltd., Kawasaki, Japan). the basic processing • Analysis of medical tests. hardware elements). • Target tracking and recognition (multi-sensor data fusion). RNNs are thus specially suited for arriving at non In the field of artificial intelligence, we have heard of expert linear state space models systems (ES). In essence an ES is a software-based system that of dynamic systems. describes the behaviour of (human) experts in some field by capturing/collecting the knowledge in the form of rules and symbols. If fuzzy or noisy data is given to an ES, it might give wrong answers. Since the ANN-based system can be trained with some fuzzy or noisy data, the combination of ES and ANN

53 GENERAL I ARTICLE

The days or neu ral (hybrid) might be very useful to devise powerful systems called network- based expert networks (ENs). (parallel) computers may not be too far Conc1uding Remarks off. Thus brain to bread By now it must be clear that in ANNs physiological or chemical (industrial processes play no role. ANNs can be called massively parallel applications) neural adaptive filters/circuits (MAP AFS). They are more like electrical! computers wlil electronic (EE) circuits with some useful adaptive properties. In gradua,ly become fact ANN s can be realised using EE hardware components. With a reality very large scale integration (VLSI) technology, it might be fea sible to fabricate microelectronic networks of high complexity for solving many optimisation and control problems using artificial neural networks. The days of neural network- based (parallel) computers may not be too far off. Thus brain to bread ( -industrial applications) neural computers will gradually become a reality. As of now the field is fascinating, sometimes intriguing, and offers great challenges and promises to scientists and engineers.