Table of Contents CHAPTER V- FUNCTION APPROXIMATION WITH MLPS, RADIAL BASIS FUNCTIONS, AND SUPPORT VECTOR MACHINES ..........................................................................................................................................3 1. INTRODUCTION................................................................................................................................4 2. FUNCTION APPROXIMATION ...........................................................................................................7 3. CHOICES FOR THE ELEMENTARY FUNCTIONS...................................................................................12 4. PROBABILISTIC INTERPRETATION OF THE MAPPINGS-NONLINEAR REGRESSION .................................23 5. TRAINING NEURAL NETWORKS FOR FUNCTION APPROXIMATION ......................................................24 6. HOW TO SELECT THE NUMBER OF BASES ........................................................................................28 7. APPLICATIONS OF RADIAL BASIS FUNCTIONS................................................................................38 8. SUPPORT VECTOR MACHINES........................................................................................................42 9. PROJECT: APPLICATIONS OF NEURAL NETWORKS AS FUNCTION APPROXIMATORS ...........................52 10. CONCLUSION ..............................................................................................................................59 CALCULATION OF THE ORTHONORMAL WEIGHTS ..................................................................................63 SINC DECOMPOSITION........................................................................................................................64 FOURIER FORMULAS..........................................................................................................................65 EIGENDECOMPOSITION ......................................................................................................................65 WEIERSTRASS THEOREM ..................................................................................................................68 MULTI-HIDDEN-LAYER MLPS ..............................................................................................................69 OUTLINE OF PROOF ...........................................................................................................................69 LOCAL MINIMA FOR GAUSSIAN ADAPTATION.........................................................................................70 APPROXIMATION PROPERTIES OF RBF ...............................................................................................71 MDL AND BAYESIAN THEORY.............................................................................................................72 DERIVATION OF THE CONDITIONAL AVERAGE .......................................................................................73 PARZEN WINDOW METHOD.................................................................................................................74 RBF AS KERNEL REGRESSION ...........................................................................................................75 L1 VERSUS L2 ..................................................................................................................................75 FUNCTION APPROXIMATION ................................................................................................................76 FUNCTIONAL ANALYSIS ......................................................................................................................76 WEIERSTRASS ..................................................................................................................................76 SERIES .............................................................................................................................................77 SAMPLING THEOREM..........................................................................................................................77 SINC .................................................................................................................................................77 FOURIER SERIES ...............................................................................................................................77 DELTA FUNCTION...............................................................................................................................77 LINEAR SYSTEMS THEORY..................................................................................................................78 EIGENFUNCTIONS ..............................................................................................................................78 SHIFT-INVARIANT...............................................................................................................................78 COMPLEX NUMBER ............................................................................................................................78 STATISTICAL LEARNING THEORY .........................................................................................................78 MANIFOLD .........................................................................................................................................78 POLYNOMIALS ...................................................................................................................................79 SCIENTIFIC METHOD ..........................................................................................................................79 VOLTERRA EXPANSIONS ....................................................................................................................79 SQUARE INTEGRABLE ........................................................................................................................79 JORMA RISSANEN .............................................................................................................................79 AKAIKE .............................................................................................................................................79 TIKONOV ..........................................................................................................................................80 ILL-POSED.........................................................................................................................................80 INDICATOR FUNCTION ........................................................................................................................80 SPLINES............................................................................................................................................80 FIDUCIAL...........................................................................................................................................80 CODE................................................................................................................................................80 VC DIMENSION..................................................................................................................................80 COVER THEOREM .............................................................................................................................81 1 LEARNING THEORY ............................................................................................................................81 A. BARRON.......................................................................................................................................81 PARK AND SANDBERG, ......................................................................................................................81 BISHOP ............................................................................................................................................81 VLADIMIR VAPNIK..............................................................................................................................81 PARZEN E. .......................................................................................................................................82 SIMON HAYKIN..................................................................................................................................82 EQ.1 ................................................................................................................................................82 EQ.4 ................................................................................................................................................82 EQ.11 ..............................................................................................................................................82 EQ.2 ................................................................................................................................................82 EQ.14 ..............................................................................................................................................82 EQ.30 ..............................................................................................................................................83 EQ.25 ..............................................................................................................................................83 EQ.7 ................................................................................................................................................83