MATHEMATICS OF NEURAL NETWORKS Models, Algorithms and Applications OPERATIONS RESEARCH/COMPUTER SCIENCE INTERFACES SERIES Ramesh Sharda, Series Editor ConocolDuPont Chair of Management of Technology Oklahoma State University Stillwater, Oklahoma U.S.A. Other published titles in the series: Brown, Donald/Scherer, William T. University of Virginia Intelligent Scheduling Systems Nash, Stephen G'/Sofer, Ariela George Mason University The Impact ofEmerging Technologies on Computer Science and Operations Research Barth, Peter Max-Planck-Institut fur Infonnatik, Gennany Logic-Based 0-/ Constraint Programming Jones, Christopher V. University of Washington Visualization and Optimization Barr, Richard S./ Helgason, Richard V./ Kennington, Jeffery L. Southern Methodist University Interfaces in Computer Science and Operations Research: Advances in Metaheuristics, Optimization, and Stochastic Modeling Technologies MATHEMATICS OF NEU RAL N ETWORKS Models, Aigorithms and Applications EDITED SY Stephen W ELLACOTT University of Srighton United Kingdom • John C MASON University of Huddersfield United Kingdom • lain J ANDERS ON University of Huddersfield United Kingdom SPRINGER SCIENCE+BUSINESS MEDIA, LLC Library of Congress Cataloging-in-Publication Data A C.I.P. Catalogue record for this book is available from the Library of Congress. ISBN 978-1-4613-7794-8 ISBN 978-1-4615-6099-9 (eBook) DOI 10.1007/978-1-4615-6099-9 © Springer Science+Business Media New York 1997 Originally published by Kluwer Academic Publishers 1997 Softcover reprint of the hardcover 1st edition 1997 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo­ copying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC Printed on acid-free paper. CONTENTS PREFACE XXI Part I INVITED PAPERS 1 1 N-TUPLE NEURAL NETWORKS N. M. Allinson and A. R. Kolcz 3 2 INFORMATION GEOMETRY OF NEURAL NETWORKS -AN OVERVIEW- Shun-ichi Amari 15 3 Q-LEARNING: A TUTORIAL AND EXTENSIONS George Cybenko, Robert Gray and Katsuhiro Moizumi 24 4 ARE THERE UNIVERSAL PRINCIPLES OF BRAIN COMPUTATION? Stephen Grossberg 34 5 ON-LINE TRAINING OF MEMORY-DRIVEN ATTRACTOR NETWORKS Morris W. Hirsch 41 6 MATHEMATICAL PROBLEMS ARISING FROM CONSTRUCTING AN ARTIFICIAL BRAIN J. G. Taylor 47 Part II SUBMITTED PAPERS 59 7 THE SUCCESSFUL USE OF PROBABILITY DATA IN CONNECTIONIST MODELS J. R. Alexander Jr. and J. P. Coughlin 61 8 WEIGHTED MIXTURE OF MODELS FOR ON-LINE LEARNING P. Edgar An 67 9 LOCAL MODIFICATIONS TO RADIAL BASIS NETWORKS I. J. Anderson 73 v VI MATHEMATICS OF NEURAL NETWORKS 10 A STATISTICAL ANALYSIS OF THE MODIFIED NLMS RULES E.D. Aved'yan, M. Brown and C.J. Harris 78 11 FINITE SIZE EFFECTS IN ON-LINE LEARNING OF MULTI-LAYER NEURAL NETWORKS. David Barber, Peter Sollich and David Saad 84 12 CONSTANT FAN-IN DIGITAL NEURAL NETWORKS ARE VLSI-OPTIMAL V. Beiu 89 13 THE APPLICATION OF BINARY ENCODED 2ND DIFFERENTIAL SPECTROMETRY IN PREPROCESSING OF UV- VIS ABSORPTION SPECTRAL DATA N Benjathapanun, W J 0 Boyle and K T V Grattan 95 14 A NON-EQUIDISTANT ELASTIC NET ALGORITHM Jan van den Berg and Jock H. Geselschap 101 15 UNIMODAL LOADING PROBLEMS Monica Bianchini, Stefano Fanelli, Marco Gori and Marco Protasi 107 16 ON THE USE OF SIMPLE CLASSIFIERS FOR THE INITIALISATION OF ONE-HIDDEN- LAYER NEURAL NETS Jan C. Bioch, Robert Carsouw and Rob Potharst 113 17 MODELLING CONDITIONAL PROBABILITY DISTRIBUTIONS FOR PERIODIC VARIABLES Christopher M Bishop and Ian T Nabney 118 18 INTEGRO-DIFFERENTIAL EQUATIONS IN COMPARTMENTAL MODEL NEURODYNAMICS Paul C. Bressloff 123 19 NONLINEAR MODELS FOR NEURAL NETWORKS Susan Brittain and Linda M. Haines 129 20 A NEURAL NETWORK FOR THE TRAVELLING SALESMAN PROBLEM WITH A WELL BEHAVED ENERGY FUNCTION Marco Budinich and Barbara Rosario 134 Contents Vll 21 SEMIPARAMETRIC ARTIFICIAL NEURAL NETWORKS Enrico Capobianco 140 22 AN EVENT-SPACE FEEDFORWARD NETWORK USING MAXIMUM ENTROPY PARTITIONING WITH APPLICATION TO LOW LEVEL SPEECH DATA D.K. Y. Chiu, D. Bockus and J. Bradford 146 23 APPROXIMATING THE BAYESIAN DECISION BOUNDARY FOR CHANNEL EQUALISATION USING SUBSET RADIAL BASIS FUNCTION NETWORK E.S. Chng, B. Mulgrew, S. Chen and G. Gibson 151 24 APPLICATIONS OF GRAPH THEORY TO THE DESIGN OF NEURAL NETWORKS FOR AUTOMATED FINGERPRINT IDENTIFICATION Carol G. Crawford 156 25 ZERO DYNAMICS AND RELATIVE DEGREE OF DYNAMIC RECURRENT NEURAL NETWORKS A. Delgado, C. Kambhampati and K. Warwick 161 26 IRREGULAR SAMPLING APPROACH TO NEUROCONTROL: THE BAND-AND SPACE-LIMITED FUNCTIONS QUESTIONS Andrzej Dzielinski and Rafal Zbikowski 166 27 UNSUPERVISED LEARNING OF TEMPORAL CONSTANCIES BY PYRAMIDAL-TYPE NEURONS. Michael Eisele 171 28 NUMERICAL ASPECTS OF MACHINE LEARNING IN ARTIFICIAL NEURAL NETWORKS S. W. Ellacott and A. Easdown 176 29 LEARNING ALGORITHMS FOR RAM-BASED NEURAL NETWORKS Alistair Ferguson, Laurence C Dixon and Hamid Bolouri 181 30 ANALYSIS OF CORRELATION MATRIX MEMORY AND PARTIAL MATCH- IMPLICATIONS FOR COGNITIVE Vlll MATHEMATICS OF NEURAL NETWORKS PSYCHOLOGY Richard Filer and James Austin 186 31 REGULARIZATION AND REALIZABILITY IN RADIAL BASIS FUNCTION NETWORKS Jason A.S. Freeman and David Saad 192 32 A UNIVERSAL APPROXIMATOR NETWORK FOR LEARNING CONDITIONAL PROBABILITY DENSITIES D. Husmeier, D. Allen and J. G. Taylor 198 33 CONVERGENCE OF A CLASS OF NEURAL NETWORKS Mark P. Joy 204 34 APPLICATIONS OF THE COMPARTMENTAL MODEL NEURON TO TIME SERIES ANALYSIS S. K asderidis and J. G. Taylor 209 35 INFORMATION THEORETIC NEURAL NETWORKS FOR CONTEXTUALLY GUIDED UNSUPERVISED LEARNING Jim Kay 215 36 CONVERGENCE IN NOISY TRAINING Petri K oistinen 220 37 NON-LINEAR LEARNING DYNAMICS WITH A DIFFUSING MESSENGER Barl Krekelberg and John G. Taylor 225 38 A VARIATIONAL APPROACH TO ASSOCIATIVE MEMORY Abderrahim Labbi 230 39 TRANSFORMATION OF NONLINEAR PROGRAMMING PROBLEMS INTO SEPARABLE ONES USING MULTILAYER NEURAL NETWORKS Bao-Liang Lu and Koji Ito 235 40 A THEORY OF SELF-ORGANISING NEURAL NETWORKS S P Luttrell 240 41 NEURAL NETWORK SUPERVISED TRAINING BASED ON A DIMENSION REDUCING METHOD G.D. Magoulas, M.N. Vrahatis, T.N. Grapsa and G.S. Androulakis 245 Contents IX 42 A TRAINING METHOD FOR DISCRETE MULTILAYER NEURAL NETWORKS G.D. Magoulas, M.N. Vrahatis, T.N. Grapsa and G.S. Androulakis 250 43 LOCAL MINIMAL REALISATIONS OF TRAINED HOPFIELD NETWORKS S. Manchanda and G.G.R. Green 255 44 DATA DEPENDENT HYPERPARAMETER ASSIGNMENT Glenn Marion and David Saad 259 45 TRAINING RADIAL BASIS FUNCTION NETWORKS BY USING SEPARABLE AND ORTHOGONALIZED GAUSSIANS J. C. Mason, 1. J. Anderson, G. Rodriguez and S. Seatzu 265 46 ERROR BOUNDS FOR DENSITY ESTIMATION BY MIXTURES Ronny Meir and Assaf J. Zeevi 270 47 ON SMOOTH ACTIVATION FUNCTIONS H. N. Mhaskar 275 48 GENERALISATION AND REGULARISATION BY GAUSSIAN FILTER CONVOLUTION OF RADIAL BASIS FUNCTION NETWORKS Christophe Molina and Mahesan Niranjan 280 49 DYNAMICAL SYSTEM PREDICTION: A LIE ALGEBRAIC APPROACH FOR A NOVEL NEURAL ARCHITECTURE Yves Moreau and Joos Vandewalle 285 50 STOCHASTIC NEURODYNAMICS AND THE SYSTEM SIZE EXPANSION Toru Ohira and Jack D. Cowan 290 51 AN UPPER BOUND ON THE BAYESIAN ERROR BARS FOR GENERALIZED LINEAR REGRESSION Cazhaow S. Qazaz, Christopher K. I. Williams and Christopher M. Bishop 295 52 CAPACITY BOUNDS FOR STRUCTURED NEURAL NETWORK ARCHITECTURES Peter Rieper, Sabine Kroner and Reinhard Moratz 300 53 ON-LINE LEARNING IN MULTILAYER NEURAL NETWORKS David Saad and Sara A. Solla 306 x MATHEMATICS OF NEURAL NETWORKS 54 SPONTANEOUS DYNAMICS AND ASSOCIATIVE LEARNING IN AN ASSYMETRIC RECURRENT RANDOM NEURAL NETWORK M. Samuelides, B. Doyon, B. Cessac and M. Quoy 312 55 A STATISTICAL MECHANICS ANALYSIS OF GENETIC ALGORITHMS FOR SEARCH AND LEARNING Jonathan L. Shapiro, Adam Prugel-Bennett and Magnus Rattray 318 56 VOLUMES OF ATTRACTION BASINS IN RANDOMLY CONNECTED BOOLEAN NETWORKS Sergey A. Shumsky 323 57 EVIDENTIAL REJECTION STRATEGY FOR NEURAL NETWORK CLASSIFIERS A. Shustorovich 328 58 DYNAMICS APPROXIMATION AND CHANGE POINT RETRIEVAL FROM A NEURAL NETWORK MODEL J. Smid and P. Vol! 333 59 QUERY LEARNING FOR MAXIMUM INFORMATION GAIN IN A MULTI-LAYER NEURAL NETWORK Peter Sollich 339 60 SHIFT, ROTATION AND SCALE INVARIANT SIGNATURES FOR TWO-DIMENSIONAL CONTOURS, IN A NEURAL NETWORK ARCHITECTURE David McG. Squire and Terry M. Caelli 344 61 FUNCTION APPROXIMATION BY THREE- LAYER ARTIFICIAL NEURAL NETWORKS Shin Suzuki 349 62 NEURAL NETWORK VERSUS STATISTICAL CLUSTERING TECHNIQUES: A PILOT STUDY IN A PHONEME RECOGNITION TASK G. Tambouratzis, T. Tambouratzis and D. Tambouratzis 355 63 MULTISPECTRAL IMAGE ANALYSIS USING PULSED COUPLED NEURAL NETWORKS G L Tarr, X Clastres, L Freyss, M Samuelides, C Dehainaut and W Burckel 361 Contents Xl 64 REASONING NEURAL NETWORKS Rua-Huan R. Tsaih 366 65 CAPACITY OF THE UPSTART ALGORITHM Ansgar H. L. West and David Saad 372 66 REGRESSION WITH GAUSSIAN PROCESSES Christopher K. 1. Williams 378 67 STOCHASTIC FORWARD-PERTURBATION, ERROR SURFACE AND PROGRESSIVE LEARNING IN NEURAL NETWORKS Li-Qun Xu 383 68 DYNAMICAL STABILITY OF A HIGH- DIMENSIONAL SELF -ORGANIZING MAP Howard Hua Yang 389 69 MEASUREMENTS OF GENERALISATION BASED ON INFORMATION GEOMETRY Huaiyu Zhu and Richard Rohwer 394 70 TOWARDS AN ALGEBRAIC THEORY OF NEURAL NETWORKS: SEQUENTIAL COMPOSITION R. Zimmer 399 CONTRIBUTORS J. R. Alexander Jr. James Austin Towson State University, Advanced Computer Architecture Group, Towson, MD 21032, USA. Department of Computer Science, Email: e7 [email protected] University of York, Heslington, Y01 5DD, UK. D. Allen Centre for Neural Networks, E.D. Aved'yan Department of Mathematics, Russian Academy of Sciences, King's College London, Institute of Control Sciences, London WC2R 1LS UK. 65 Profsoyuznaya str., Moscow 117342, Russia. N. M. Allinson Email: [email protected] Department of Electrical Engineering and Electronics, David Barber UMIST, Manchester M60 1QD, UK. Dept. Comp, Sci. and Appl. Maths., Email: [email protected] Aston University, Birmingham B4 7ET, UK. Shun-ichi Amari Web: http://www.ncrg.aston.ac.uk/ University of Tokyo, Tokyo, Japan, V. Beiu RIKEN Frontier Research Program, Wako-City, Saitama, Japan.