AHYBRID NEURAL NETWORK AND GENETIC PROGRAMMING APPROACH TO THE AUTOMATIC CONSTRUCTION OF COMPUTER VISION SYSTEMS Cameron P.Kyle-Davidson A thesis submitted for the degree of Master of Science by Dissertation School of Computer Science and Electronic Engineering University of Essex October 2018 ii Abstract Both genetic programming and neural networks are machine learning techniques that have had a wide range of success in the world of computer vision. Recently, neural networks have been able to achieve excellent results on problems that even just ten years ago would have been considered intractable, especially in the area of image classification. Additionally, genetic programming has been shown capable of evolving computer vision programs that are capable of classifying objects in images using conventional computer vision operators. While genetic algorithms have been used to evolve neural network structures and tune the hyperparameters of said networks, this thesis explores an alternative combination of these two techniques. The author asks if integrating trained neural networks with genetic pro- gramming, by framing said networks as components for a computer vision system evolver, would increase the final classification accuracy of the evolved classifier. The author also asks that if so, can such a system learn to assemble multiple simple neural networks to solve a complex problem. No claims are made to having discovered a new state of the art method for classification. Instead, the main focus of this research was to learn if it is possible to combine these two techniques in this manner. The results presented from this research in- dicate that such a combination does improve accuracy compared to a vision system evolved without the use of these networks. i ii Acronyms ANN Artificial Neural Network. CGP Cartesian Genetic Programming. CNN Convolutional Neural Network. DRS2 Dynamic Range Selection 2. EA Evolutionary Algorithm. ES Evolutionary Strategy. GA Genetic Algorithm. GP Genetic Programming. LGP Linear Genetic Programming. LSTM Long Short-Term Memory. MLP Multilayer Perceptron. ROI Region of Interest. STGP Strongly Typed Genetic Programming. iii Acronyms iv Acknowledgements I would like to thank Dr. Adrian Clark for his continued support throughout the course of this research project, without which I am certain that I would not have got nearly as far. v Acronyms vi Contents Abstract i Acronyms iii Acknowledgementsv List of Figures xiii 1 Introduction1 1.1 Forms of Machine Learning...............................1 1.2 Computer Vision......................................2 1.3 Motivation and Goal...................................3 1.4 Challenges.........................................4 1.5 Work Conducted......................................4 1.6 Thesis Overview......................................5 2 Overview of Evolutionary Algorithms7 2.1 Evolutionary Algorithm Operations..........................8 2.1.1 Selection......................................8 2.1.2 Genotype and Phenotype............................ 10 2.1.3 Fitness Evaluation................................ 10 2.1.4 Crossover..................................... 10 2.1.5 Mutation...................................... 11 2.2 Types of Evolutionary Algorithms........................... 12 2.2.1 Evolutionary Strategies............................. 12 2.2.2 Genetic Algorithms............................... 13 vii CONTENTS 2.2.3 Neuroevolution.................................. 13 2.3 Genetic Programming.................................. 14 2.3.1 Fitness Evaluation................................ 14 2.3.2 Population Instantiation............................ 14 2.3.3 Linear Genetic Programming......................... 15 2.3.4 Cartesian Genetic Programming....................... 16 2.3.5 Structure, Crossover, and Mutation...................... 16 2.4 Typed Genetic Programming.............................. 17 2.4.1 Individual Representation........................... 19 2.4.2 Building the Initial Population......................... 19 2.4.3 Fitness Evaluation................................ 20 2.4.4 Crossover and Mutation............................ 20 2.4.5 Performance.................................... 20 2.5 Limitations and Drawbacks of Evolutionary Algorithms.............. 21 2.6 Evolving Vision Systems................................. 22 2.7 Conclusion......................................... 23 3 Overview of Neural Networks 24 3.1 A Brief History of Neural Networks.......................... 25 3.2 Common Neural Networks............................... 27 3.2.1 Multilayer Perceptrons............................. 28 3.2.2 Recurrent Neural Networks.......................... 29 3.2.3 Convolutional Neural Networks........................ 31 3.3 Neural Networks in Detail................................ 33 3.3.1 Artificial Neurons................................ 34 3.3.2 Activation Functions............................... 35 3.3.3 Loss Functions.................................. 37 3.3.4 The Backpropagation Algorithm....................... 37 3.4 Limitations of Neural Networks............................ 38 3.4.1 Blackbox Nature................................. 38 3.4.2 Hyperparameter Tuning............................ 38 3.4.3 Available Datasets................................ 39 3.4.4 Computational Power.............................. 39 3.5 Conclusion......................................... 39 viii CONTENTS 4 Combining Evolutionary Algorithms and Neural Networks 40 4.1 Existing Work....................................... 40 4.1.1 Neuroevolution.................................. 41 4.1.2 Conclusion..................................... 44 4.2 This Research Project................................... 44 4.2.1 The Problem................................... 44 4.2.2 A Possible Solution................................ 45 4.2.3 Aims and Objectives............................... 46 4.2.4 Challenges..................................... 46 4.3 Conclusion......................................... 48 5 The Implemented System 49 5.1 An Introduction to GPT................................. 49 5.1.1 The Main Component.............................. 51 5.1.2 The GP Component............................... 54 5.1.3 Ancillary Components.............................. 55 5.2 Genetic Components for Computer Vision...................... 56 5.3 Cost Functions....................................... 57 5.3.1 High Level Features Cost Function...................... 58 5.3.2 Cost Regions................................... 59 5.3.3 Comparing Cost Functions........................... 60 5.4 Extracting Structure from Neural Network Filters................. 62 5.5 Classifying Regions with Neural Network Filters.................. 65 5.6 Higher Level Feature Extraction............................ 66 5.7 Separating Feature Extraction and Classification.................. 67 5.8 Binary Decomposition.................................. 69 5.8.1 A Final Cost Function.............................. 70 5.9 Integrating Neural Networks.............................. 73 5.9.1 Network Structure................................ 73 5.9.2 Decomposed Neural Networks........................ 75 5.9.3 Integration.................................... 77 5.10 Performance and Optimisation............................. 79 5.10.1 Memoisation................................... 79 5.10.2 Multiprocessing.................................. 80 ix CONTENTS 6 Results and Analysis 81 6.1 Overview of Datasets Used............................... 81 6.1.1 Datasets Description............................... 82 6.1.2 Dataset Generation............................... 84 6.2 GPT............................................. 86 6.3 Neural Filters........................................ 88 6.4 Baseline and No Neural Components......................... 91 6.5 Multiclass Neural Networks as Components..................... 94 6.5.1 MNIST....................................... 94 6.5.2 Fashion MNIST.................................. 98 6.6 Decomposed Neural Networks............................. 101 6.6.1 Analysis...................................... 102 6.7 Conclusion......................................... 103 7 Conclusion 104 7.1 Assessment......................................... 105 7.1.1 Multiclass Neural Networks.......................... 106 7.1.2 Decomposed Neural Networks........................ 106 7.1.3 Advantages and Drawbacks.......................... 107 7.2 Future Research...................................... 108 7.3 Closing Remarks...................................... 109 Appendices 111 A Diagrams 112 B Functions and Terminals 114 B.1 Math Functions...................................... 114 B.2 Boolean Functions.................................... 115 B.3 Basic Image Functions.................................. 116 B.3.1 Simple & Numeric................................ 116 B.3.2 Thresholding................................... 116 B.3.3 Morphological.................................. 117 B.4 Advanced Image Functions............................... 117 B.5 Neural Functions..................................... 118 x CONTENTS B.6 Terminals.......................................... 118 xi CONTENTS xii List of Figures 2.1 An example of a valid and an invalid strongly typed program tree....... 19 3.1 Diagram of a simple multilayer perceptron...................... 29 3.2 Simplified diagram of an LSTM Module........................ 30 3.3 An Artificial Neuron................................... 34 3.4 Graph of sigmoid function...............................