Table of Contents CHAPTER III - MULTILAYER PERCEPTRONS.........................................................................................3 1. ARTIFICIAL NEURAL NETWORKS (ANNS) ..........................................................................................4 2. PATTERN RECOGNITION ABILITY OF THE MCCULLOCH-PITTS PE........................................................6 3. THE PERCEPTRON ........................................................................................................................27 4. ONE HIDDEN LAYER MULTILAYER PERCEPTRONS ............................................................................39 5. MLPS WITH TWO HIDDEN LAYERS...................................................................................................53 6. TRAINING STATIC NETWORKS WITH THE BACKPROPAGATION PROCEDURE .........................................60 7. TRAINING EMBEDDED ADAPTIVE SYSTEMS.......................................................................................72 8. MLPS AS OPTIMAL CLASSIFIERS.....................................................................................................77 9. CONCLUSIONS ..............................................................................................................................81 SEPARATION SURFACES OF THE SIGMOID PES ....................................................................................85 PROBABILISTIC INTERPRETATION OF SIGMOID OUTPUTS .......................................................................85 VECTOR INTERPRETATION OF THE SEPARATION SURFACE ....................................................................86 PERCEPTRON LEARNING ALGORITHM ..................................................................................................87 ERROR ATTENUATION ........................................................................................................................88 OPTIMIZING LINEAR AND NONLINEAR SYSTEMS ....................................................................................89 DERIVATION OF LMS WITH THE CHAIN RULE ........................................................................................89 DERIVATION OF SENSITIVITY THROUGH NONLINEARITY .........................................................................90 WHY NONLINEAR PES? .....................................................................................................................91 MAPPING CAPABILITIES OF THE 1 HIDDEN LAYER MLP .........................................................................91 BACKPROPAGATION DERIVATION ........................................................................................................92 MULTILAYER LINEAR NETWORKS .........................................................................................................95 REDERIVATION OF BACKPROP WITH ORDERED DERIVATIVES .................................................................95 ARTIFICIAL NEURAL NETWORKS ..........................................................................................................96 TOPOLOGY........................................................................................................................................96 FEEDFORWARD .................................................................................................................................97 SIGMOID ...........................................................................................................................................97 F. ROSENBLATT ................................................................................................................................97 SENSITIVITY ......................................................................................................................................97 GLOBAL MINIMUM ..............................................................................................................................97 NONCONVEX .....................................................................................................................................97 SADDLE POINT...................................................................................................................................97 LINEARLY SEPARABLE PATTERNS........................................................................................................97 GENERALIZE......................................................................................................................................98 LOCAL ERROR ...................................................................................................................................98 MINSKY ............................................................................................................................................98 MULTILAYER PERCEPTRONS...............................................................................................................98 BUMP................................................................................................................................................98 BACKPROPAGATION...........................................................................................................................99 INVENTORS OF BACKPROPAGATION ....................................................................................................99 ORDERED DERIVATIVE .......................................................................................................................99 LOCAL MAPS .....................................................................................................................................99 DATAFLOW........................................................................................................................................99 TOPOLOGY......................................................................................................................................100 A POSTERIORI PROBABILITY .............................................................................................................100 LIKELIHOOD.....................................................................................................................................100 PROBABILITY DENSITY FUNCTION......................................................................................................100 EQ2................................................................................................................................................100 ADALINE .........................................................................................................................................100 EQ.1 ..............................................................................................................................................101 EQ.6 ..............................................................................................................................................101 EQ.8 ..............................................................................................................................................101 EQ.10 ............................................................................................................................................101 CONVEX..........................................................................................................................................101 1 EQ.9 ..............................................................................................................................................101 EQ.12 ............................................................................................................................................101 EQ.13 ............................................................................................................................................102 EQ.14 ............................................................................................................................................102 LMS ..............................................................................................................................................102 EQ.7 ..............................................................................................................................................102 EQ.21 ............................................................................................................................................102 EQ.20 ............................................................................................................................................102 EQ.11 ............................................................................................................................................103 EQ.23 ............................................................................................................................................103 EQ.33 ............................................................................................................................................103 EQ.30 ............................................................................................................................................103 EQ.31 ............................................................................................................................................103 EQ.38 ............................................................................................................................................103 EQ.26 ............................................................................................................................................104