A Thesis Submitted for the Degree of PhD at the University of Warwick Permanent WRAP URL: http://wrap.warwick.ac.uk/94683 Copyright and reuse: This thesis is made available online and is protected by original copyright. Please scroll down to view the document itself. Please refer to the repository record for this item for information to help you to cite it. Our policy information is available from the repository home page. For more information, please contact the WRAP Team at: [email protected] warwick.ac.uk/lib-publications Imitation Learning in Artificial Intelligence by Alexandros Gkiokas Thesis Submitted to the University of Warwick for the degree of Computer Science. Supervisor: Alexandra I. Cristea Doctor of Philosophy Department of Computer Science September 2016 Contents Acknowledgments v Declarations vi Abstract vii Abbreviations viii List of Tables xi List of Figures xii Chapter 1 Introduction 1 1.1 Human Intelligence . 1 1.2 Imitation in Humans . 2 1.3 Imitation in Artificial Intelligence . 3 1.4 Cognitive Artificial Intelligence . 4 1.5 Icarus Engine . 5 1.6 Research Scope & Biological Plausibility . 6 1.7 Contributions . 8 1.8 Thesis Overview . 9 Chapter 2 Background and Literature Review 10 2.1 Alan Turing and the intelligent machines . 10 2.2 Imitation in Nature . 11 2.2.1 What is imitation? . 11 2.2.2 How does imitation work? . 12 2.3 Symbolic Artificial Intelligence . 13 2.3.1 Knowledge Representation . 14 2.3.2 Criticism and Limitations of Symbolic Artificial Intelligence . 18 i 2.4 Connectionistic Artificial Intelligence . 19 2.4.1 Artificial Neural Networks . 19 2.4.2 General Purpose Computing on Graphic Processing Units . 24 2.4.3 Deep Learning . 24 2.4.4 Reinforcement Learning . 28 2.4.5 Deep Reinforcement Learning . 31 2.4.6 Connectionism Criticism and Limitations . 32 2.5 Cognitive or Synthetic Artificial Intelligence . 33 2.5.1 Bach and Synthetic Intelligence . 33 2.5.2 Haikonen and Cognitive Intelligence . 34 2.5.3 Five Cognitive Agent Criteria . 36 2.5.4 AI Architectures . 37 2.6 Programming by Example . 40 2.6.1 PBE: Theory and Models . 40 2.6.2 PBE: Application Domains and Criticism . 43 2.6.3 PBE: Differences from Imitation Learning . 44 2.7 Parsing and Understanding . 44 2.7.1 Semantics . 44 2.7.2 Distributional Semantics . 45 2.7.3 Relational Semantics . 46 2.7.4 Part of Speech Tagging . 48 2.7.5 Semantic and Syntactic Parsing . 48 2.7.6 Implementing Parsing and NLU . 50 2.7.7 Models and Algorithms in NLU . 52 2.7.8 NLU Performance and Issues . 53 2.8 Background Conclusion . 54 Chapter 3 Theory and Agent Design 56 3.1 MDP as a Template for Learning . 57 3.2 Paradigm Decomposition and Training . 58 3.3 Rewarding and Evaluation . 62 3.4 Episode Iteration and Inference . 63 3.5 Decision Making and Policy Approximation . 64 3.6 Statistical and Probabilistic Approximation . 67 3.7 Semantic and Heuristic Approximation . 68 3.8 Neural Approximation and Distributed Encoding . 70 3.9 Deep Neural Approximation and Sparse Encoding . 72 ii 3.10 Semantic Approximation and Sparse Encoding . 74 3.11 Conceptual Graph Output . 75 3.12 Metalearning and Knowledge Compression . 76 3.12.1 Metalearning on Learnt Knowledge . 76 3.12.2 Grouping by Similarity . 76 3.12.3 Generalising Cluster Graphs . 77 3.12.4 Optimisation by Belief Evaluation . 78 3.13 Conclusion . 78 3.13.1 Bandura and Imitation in Humans . 79 3.13.2 Haikonen and Cognitive AI . 79 3.13.3 Bach and Synthetic Intelligence . 80 3.13.4 Five Cognitive Agent Criteria . 81 3.13.5 Icarus and Cognitive AI . 81 3.13.6 Discussion on Icarus Implementation . 82 Chapter 4 Conceptual Graph Dataset 83 4.1 Datasets for NLU and NLP . 84 4.1.1 Creating a New Dataset . 85 4.1.2 Partitioning the Dataset . 86 4.1.3 Translating and Converting Datasets . 87 4.2 Conceptual Graph Complexity . 88 4.2.1 Graph Columns and Linearity . 89 4.2.2 Graph Branching and Grouping . 90 4.2.3 Graphs and Operators . 90 4.3 Dataset Conclusion . 91 Chapter 5 Experiments, Methodology and Results 92 5.1 Methodology and Experiment Design . 92 5.1.1 Randomised Block Design . 93 5.1.2 Experiment Logs . 93 5.1.3 Accuracy Measures . 95 5.2 Semantic-Heuristic Experiments . 97 5.2.1 Implementation . 97 5.2.2 Results and Discussion . 99 5.3 Probability-based Experiments . 101 5.3.1 Implementation . 101 5.3.2 Probability Space Analysis . 103 5.3.3 Results . 106 iii 5.3.4 Discussion & Conclusion . 109 5.4 Shallow Neural Experiments . 110 5.4.1 Implementation . 110 5.4.2 Results . 114 5.5 Deep Learning Experiments . 118 5.5.1 Implementation . 118 5.5.2 Results . 122 5.6 Experiment Conclusions . 126 Chapter 6 Conclusions and Future Work 129 6.1 Conclusions . 129 6.2 Criticism and Limitations . 132 6.3 Future Work . 133 Appendix A Penn Treebank POS tags 135 Appendix B Conceptual Graph examples 137 iv Acknowledgments I would like to express my sincere gratitude to my supervisor Alexandra Cristea, without whom this thesis and research would never have taken place. My parents George and Maria, for never giving up on me and always believing. My current employer Stratos Arampatzis and Ortelio Ltd, as well as my previous employer Matthew Sewell and Athium Ltd, for their support, funding and understanding. Karen Stepanyan and Panagiotis Petridis for putting up with my questions, offering their support and advice, and guiding me through this adventure. Finally, Mina Makridi for putting up with me and for the past four years, and entertaining the notion that one day we'll witness true AI. v Declarations This thesis is submitted to the University of Warwick in support of my application for the degree of Doctor of Philosophy. All experimental data presented and sim- ulations were carried out by the author, except in the following cases: Daedalus experiments were done on-line after being approved by the BSREC with refer- ence REGO-2015-1529. All experiments for Daedalus were carried out on the web and they represent anonymous data. Chapter 3 contains theoretical formulations [Gkiokas et al., 2014] carried out in cooperation with Matthew Thorpe from the Mathematics Institute in Warwick. Parts of this thesis have been published by the author, including submitted papers: - Training a Cognitive Agent to Acquire and Represent Knowledge from RSS feeds onto Conceptual Graphs [Gkiokas and Cristea, 2014a]. - Unsupervised neural controller for Reinforcement Learning action-selection: Learning to represent knowledge [Gkiokas and Cristea, 2014b]. - Self-reinforced meta learning for belief generation [Gkiokas et al., 2014]. - Cognitive Agents and Machine Learning by Example: Representation with Conceptual Graphs [Gkiokas and Cristea, 2016a]. - Deep Learning and Encoding in Natural Language Understanding: Sparse and Dense Encoding Schemes for Neural-based Parsing [Gkiokas and Cristea, 2016b] vi Abstract Acquiring new knowledge often requires an agent or a system to explore, search and discover. Yet us humans build upon the knowledge of our forefathers, as did they, using previous knowledge; there does exist a mechanism which allows transference of knowledge without searching, exploration or discovery. That mechanism is known as imitation and it exists everywhere in nature; in animals, insects, primates, and humans. Enabling artificial, cognitive and software agents to learn by imitation could potentially be crucial to the emergence of the field of autonomous systems, robotics, cyber-physical and software agents. Imitation in AI implies that agents can learn from their human users, other AI agents, through observation or using physical interaction in robotics, and therefore learn a lot faster and easier. Describing an imitation learning framework in AI which uses the Internet as the source of knowledge requires a rather unconventional approach: the procedure is a temporal-sequential process which uses reinforcement based on behaviouristic Psychology, deep learning and a plethora of other Algorithms. Ergo an agent using a hybrid simulating-emulating strategy is formulated, implemented and experimented with. That agent learns from RSS feeds using examples provided by the user; it adheres to previous research work and theoretical foundations and demonstrates that not only is imitation learning in AI possible, but it compares and in some cases outperforms traditional approaches. vii Abbreviations ADABOOST Adaptive Boosting AGI Artificial General Intelligence ANN Artificial Neural Networks AI Artificial Intelligence BFGS Broyden-Fletcher-Goldfarb-Shanno BPROP Back Propagtion CA Cognitive Agent CE Cross Entropy CG Conceptual Graph CNN Convolutional Neural Networks CRF Conditional Random Field DARPA Defense Advanced Research Projects Agency DNN Deep Neural Networks FOL First Order Logic FSM Finite State Machine GP2U General Purpose Compute on Graphics Processing Units viii GOFAI Good Old Fashioned Artificial Intelligence HOL Higher Order Logic KR Knowledge Representation LBFGS Limited storage Broyden-Fletcher-Goldfarb-Shanno LMA Levenberg-Marquardt Algorithm LSTM Long Short Term Memory MBSGD Mini Batch Stochastic Gradient Descent MDP Markov Decision Process ML Machine Learning MRL Meaning Representation Language MSE Mean Squared Error NLP Natural Language Processing NLU Natural Language Understanding PBD Programming by Demonstration PBE Programming by Example POS Part Of Speech RBM Restricted Boltzmann Machine ReLU Rectified Linear Unit RNN Recurrent Neural Network RPROP Resilient Propagation SMT Statistical Machine Translation ix SVM Support Vector Machine UI User Interface x List of Tables 2.1 State of the Art NLU Software Tools. 50 4.1 Common NLP Datasets . 84 5.1 Accuracy Metrics Equivalence. 97 5.2 Statistic & Probability Oriented Research. 109 xi List of Figures 2.1 Simple MRL structure. 15 2.2 MRL with Edges. 16 2.3 Annotated MRL with Edges and Meta-data. 16 2.4 Conceptual Graph example.