Table of Contents CHAPTER XI- TRAINING AND USING RECURRENT NETWORKS ................................................................3 1. INTRODUCTION................................................................................................................................4 2. SIMPLE RECURRENT TOPOLOGIES ..................................................................................................5 3. ADAPTING THE FEEDBACK PARAMETER .............................................................................................8 4. UNFOLDING RECURRENT NETWORKS IN TIME...................................................................................11 5. THE DISTRIBUTED TLFN TOPOLOGY...............................................................................................24 6. DYNAMICAL SYSTEMS....................................................................................................................31 7. RECURRENT NEURAL NETWORKS ..................................................................................................34 8. LEARNING RULES FOR RECURRENT SYSTEMS ................................................................................36 9. APPLICATIONS OF DYNAMIC NETWORKS TO SYSTEM IDENTIFICATION AND CONTROL...........................43 10. HOPFIELD NETWORKS .................................................................................................................50 11. GROSSBERG’S ADDITIVE MODEL...................................................................................................59 12. BEYOND FIRST ORDER DYNAMICS: FREEMAN’S MODEL...................................................................62 13. CONCLUSIONS ............................................................................................................................68 BACKPROPAGATION VERSUS BPTT....................................................................................................72 VECTOR SPACE INTERPRETATION OF TLFNS ......................................................................................72 ADVANTAGE OF LINEAR MEMORY PES ................................................................................................75 TRAINING FOCUSED TLFNS ...............................................................................................................75 TRAINING THE GAMMA FILTER ............................................................................................................77 TRAINING ALTERNATE MEMORIES........................................................................................................78 TLFN ARCHITECTURES .....................................................................................................................79 DYNAMIC BACKPROPAGATION.............................................................................................................83 DERIVATION OF ENERGY FUNCTION.....................................................................................................85 FULLY RECURRENT ............................................................................................................................86 TLRN ..............................................................................................................................................86 TRAJECTORY.....................................................................................................................................87 FIXED POINT......................................................................................................................................87 HOPFIELD .........................................................................................................................................87 EQ.1 ................................................................................................................................................87 EQ.3 ................................................................................................................................................87 UNFOLDING.......................................................................................................................................87 EQ.6 ................................................................................................................................................88 EQ.8 ................................................................................................................................................88 EQ.9 ................................................................................................................................................88 EQ.10 ..............................................................................................................................................88 EQ.12 ..............................................................................................................................................88 EQ.4 ................................................................................................................................................88 EQ.15 ..............................................................................................................................................89 EQ.11 ..............................................................................................................................................89 EQ.24 ..............................................................................................................................................89 EQ.5 ................................................................................................................................................89 ATTRACTOR ......................................................................................................................................89 DKFL ................................................................................................................................................89 FREEMAN .........................................................................................................................................89 LUIS ALMEIDA ...................................................................................................................................90 EQ.46 ..............................................................................................................................................90 EQ. 48 .............................................................................................................................................90 EQ.22 ..............................................................................................................................................90 EQ.25 ..............................................................................................................................................90 EQ.31 ..............................................................................................................................................90 EQ.34 ..............................................................................................................................................91 EQ.33 ..............................................................................................................................................91 EQ.38 ..............................................................................................................................................91 EQ.45 ..............................................................................................................................................91 1 LEE GILES ........................................................................................................................................91 EQ.35 ..............................................................................................................................................91 EQ.32 ..............................................................................................................................................92 NARENDRA .......................................................................................................................................92 WAN ................................................................................................................................................92 EQ.14 ..............................................................................................................................................92 EQ.31 ..............................................................................................................................................92 BENGIO ............................................................................................................................................92 FELDKAMP........................................................................................................................................92 2 Chapter XI- Training and Using Recurrent Networks Version 2.0 This Chapter is Part of: Neural and Adaptive Systems: Fundamentals Through Simulation© by Jose C. Principe Neil R. Euliano W. Curt Lefebvre Copyright 1997 Principe The goal of this chapter is to introduce the following concepts: • Why backpropagation can not train recurrent systems. • Develop the backpropagation through time algorithm. • Introduce and train distributed TLFNs. • Provide the basic theory to study the gamma model • Introduce and train fully recurrent networks. • Explain fixed point learning.