Exploring Fractional Order Calculus as an Artificial Neural Network Augmentation by Samuel Alan Gardner A project document submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science MONTANA STATE UNIVERSITY Bozeman, Montana April, 2009 c Copyright by Samuel Alan Gardner 2009 All Rights Reserved ii APPROVAL of a project document submitted by Samuel Alan Gardner This project document has been read by each member of the project committee and has been found to be satisfactory regarding content, English usage, format, citations, bibliographic style, and consistency, and is ready for submission to the Division of Graduate Education. Dr. John Paxton Approved for the Department of Computer Science Dr. John Paxton Approved for the Division of Graduate Education Dr. Carl A. Fox iii STATEMENT OF PERMISSION TO USE In presenting this project document in partial fulfullment of the requirements for a master's degree at Montana State University, I agree that the Library shall make it available to borrowers under rules of the Library. If I have indicated my intention to copyright this project document by including a copyright notice page, copying is allowable only for scholarly purposes, consistent with \fair use" as prescribed in the U.S. Copyright Law. Requests for permission for extended quotation from or reproduction of this project document in whole or in parts may be granted only by the copyright holder. Samuel Alan Gardner April, 2009 iv TABLE OF CONTENTS 1. INTRODUCTION .................................................................................... 1 2. BACKGROUND....................................................................................... 4 Artificial Neural Networks .........................................................................4 Neurons................................................................................................4 Topologies ............................................................................................5 Learning Algorithms..............................................................................8 Backpropagation.......................................................................................8 Evolutionary Algorithms ........................................................................... 10 Genetic Algorithms ................................................................................... 12 GNARL ................................................................................................... 15 Initialization ......................................................................................... 16 Selection............................................................................................... 16 Mutation .............................................................................................. 17 Parametric Mutation ......................................................................... 17 Structural Mutation........................................................................... 18 Fractional Order Calculus.......................................................................... 18 3. SYSTEM DESCRIPTION......................................................................... 22 Neural Network Implementation ................................................................ 22 Discrete Differintegral Computation ........................................................... 22 Fractional Calculus Neuron Augmentation.................................................. 25 Learning Algorithm................................................................................... 26 Initialization ......................................................................................... 27 Selection............................................................................................... 27 Crossover.............................................................................................. 27 Mutation .............................................................................................. 28 Parametric Mutation ......................................................................... 28 Structural Mutation........................................................................... 29 Statistic Collection.................................................................................... 30 4. EXPERIMENTS AND RESULTS.............................................................. 31 Artificial Life Simulation ........................................................................... 31 Common Experimental Parameters ............................................................ 33 Baseline.................................................................................................... 36 Non-Resetting........................................................................................... 40 Hidden Layer Size Comparison................................................................... 41 Fixed Q-Value Comparison........................................................................ 44 v GA Evolution ........................................................................................... 46 Sensed Walls............................................................................................. 48 Track ....................................................................................................... 50 Intercept .................................................................................................. 51 Hide and Seek........................................................................................... 54 Structural Evolution.................................................................................. 57 Feed-Forward ........................................................................................ 58 Recurrent ............................................................................................. 60 Feed-Forward GA.................................................................................. 61 Summary of Results .................................................................................. 63 5. FUTURE WORK ..................................................................................... 65 More Experimental Data ........................................................................... 65 Structural Evolution.................................................................................. 66 Backpropagation....................................................................................... 66 Simulation Complexity .............................................................................. 67 6. CONCLUSIONS ....................................................................................... 69 REFERENCES.............................................................................................. 72 APPENDICES .............................................................................................. 75 Appendix A: Software ............................................................................. 76 Appendix B: Code Listings ...................................................................... 99 vi LIST OF TABLES Table Page 1 Link Restriction Options.................................................................... 26 2 Mutable Parameter Types.................................................................. 28 3 Sensor Cell Type Values .................................................................... 33 4 Base Experimental Evolution Parameters............................................ 34 5 Base Experimental Simulation Parameters .......................................... 34 6 Base Experimental Neural Network Parameters................................... 35 7 Baseline { 95% Confidence Intervals for Maximum Fitness................... 37 8 Baseline { Simulation Data From Peak Fitness Individuals................... 39 9 Non-Resetting { 95% Confidence Intervals for Maximum Fitness.......... 41 10 Hidden Layer Size { NN 95% Confidence Intervals for Maximum Fitness 43 11 Hidden Layer Size { FNN 95% Confidence Intervals for Maximum Fitness 43 12 Fixed Q Value { 95% Confidence Intervals for Maximum Fitness.......... 45 13 GA { 95% Confidence Intervals for Maximum Fitness.......................... 48 14 Sensed Walls { 95% Confidence Intervals for Maximum Fitness............ 49 15 Track { 95% Confidence Intervals for Maximum Fitness....................... 50 16 Intercept { 95% Confidence Intervals for Maximum Fitness.................. 53 17 Hide and Seek { 95% Confidence Intervals for Maximum Fitness.......... 56 18 Feed-Forward Structural { 95% Confidence Intervals for Maximum Fitness.............................................................................................. 59 19 Recurrent Structural { 95% Confidence Intervals for Maximum Fitness. 61 20 Feed-Forward Structural GA { 95% Confidence Intervals for Maximum Fitness.............................................................................................. 63 vii LIST OF FIGURES Figure Page 1 Anatomy of a Neuron ........................................................................4 2 A Feed-forward, Multi-layer Neural Network .......................................6 3 An Example of a Fully-connected, Feed-forward, Multi-layer Neural Network............................................................................................7 4 Roulette Selection Illustration with Population Size = 10..................... 14 5 Single Point Crossover ......................................................................