MEng Individual Project Imperial College London Department of Computing Neural Logic Framework for Digital Assistants Supervisor: Prof. Alessandra Russo Author: Nuri Cingillioglu Second Marker: Dr. Krysia Broda June 16, 2017 Abstract Digital assistants are becoming ubiquitous with consumers across mobile platforms help- ing with everyday tasks. The natural language interface of most assistants are built on machine learning based intent parsing techniques. This design cannot handle higher level abstract reasoning such as defaults while logic programs can incorporate them. In this project we present Kevin, a digital personal assistant with a logical framework built on top of neural networks to provide a flexible execution environment while har- nessing the capabilities of machine learning at a lower level. Kevin demonstrates natural language based logical constructs such as unification and resolution with integrated neural network information retrieval. Acknowledgements I would like to thank: • my parents for their wholehearted support, endless love and perpetual encourage- ment. • my supervisor Prof. Alessandra Russo for giving me this opportunity by accepting the project proposal and allowing me to explore the topic as well as her PhD student Mark Law for his feedback. • my personal tutor Dr. Alistair Donaldson for his support and advice for the years. • my friends for their companionship and solace throughout my degree. Contents 1 Introduction 11 1.1 Motivation.................................... 11 1.2 Objectives.................................... 12 1.3 Challenges.................................... 12 1.4 Contributions.................................. 13 2 Related Work 14 2.1 Natural Language Interactions......................... 14 2.1.1 Turing Test............................... 14 2.1.2 Conversational Agents......................... 15 2.2 Domain-specific Assistants........................... 17 2.2.1 PANDA: Virtual assistant for in-car child entertainment...... 18 2.2.2 Radar: Assistant agent for email and task management....... 18 2.2.3 SimCity game assistant......................... 19 2.2.4 Creating Conversation Flows...................... 20 2.3 Towards Intelligent Assistants......................... 21 2.3.1 Domain Problem............................ 21 2.3.2 Adding Knowledge........................... 22 2.3.3 Retaining Knowledge.......................... 22 2.4 Virtual Assistant Architectures........................ 23 2.4.1 Attorney 209: Adviser for family-based cases............. 23 2.4.2 Project Execution Assistant (PExA)................. 24 2.4.3 Cloud Scale Assistant Architectures.................. 26 2 CONTENTS CONTENTS 2.5 Personalisation & Gender Issues........................ 28 2.5.1 Interaction design............................ 28 2.5.2 Social environment and expectations................. 29 2.5.3 Becoming human-like.......................... 30 2.6 Language Representations........................... 30 2.6.1 Combinatory Categorical Grammar (CCG).............. 31 2.6.2 Abstract Meaning Representation (AMR).............. 31 3 Background 33 3.1 From Conversations to Assistants....................... 33 3.1.1 Personal Digital Assistant, the Definition............... 33 3.1.2 Pragmatism in Agents......................... 34 3.2 Extracting Structured Data.......................... 35 3.3 Logic Programming............................... 36 3.3.1 Negation by Failure........................... 37 3.3.2 SLDNF Resolution........................... 37 3.4 Deep NLP.................................... 39 3.4.1 word2vec................................. 39 3.4.2 Global Vectors (GloVe)......................... 41 3.4.3 Long Short-Term Memory (LSTM).................. 42 3.4.4 Bi-Directional Attention Flow (BiDAF) Model............ 43 4 Logic Framework 46 4.1 Scripting..................................... 46 4.1.1 Tracing an Answer........................... 47 4.2 Facts....................................... 49 4.3 Constraints................................... 50 4.4 Intent parsing.................................. 51 4.4.1 Interactive Rule Constructing..................... 53 4.5 Template IR................................... 54 4.6 Template deduction............................... 55 3 CONTENTS CONTENTS 4.7 Negation by failure............................... 57 4.8 Free Form Integrations............................. 59 4.8.1 Query Redirecting........................... 60 4.8.2 Neural Network Extraction....................... 61 5 Similarity & Unification 64 5.1 Types of Similarity............................... 64 5.1.1 Document Average Similarity..................... 65 5.1.2 Bag-of-words Similarity......................... 66 5.1.3 Maximal Token Similarity....................... 67 5.1.4 Weighted Maximal Token (Fuzzy) Similarity............. 68 5.1.5 Tree Edit Similarity........................... 69 5.1.6 Hybrid Similarity............................ 70 5.2 Sentence Unification.............................. 70 5.2.1 Strict Unification............................ 71 5.2.2 Fuzzy Unification............................ 72 6 Deep Question Answering 74 6.1 Similar Selection................................ 74 6.2 Guided BiDAF................................. 76 6.2.1 Network Architecture.......................... 77 6.2.2 Model Differences............................ 78 6.2.3 Training and Limitations........................ 79 6.2.4 Output Examples............................ 81 7 Implementation 85 7.1 NLP Back End................................. 85 7.1.1 Spacy.................................. 85 7.1.2 CoreNLP................................ 87 7.2 Internal Representation............................. 88 7.2.1 Parse Trees (PTree)........................... 89 4 CONTENTS CONTENTS 7.2.2 Expressions (Expr)........................... 90 7.2.3 Rules (Rule)............................... 91 7.2.4 Knowledge Bases (KB)......................... 92 7.3 Query Pipeline................................. 93 7.3.1 Input Module.............................. 93 7.3.2 Command Module........................... 94 7.3.3 Basic Co-reference........................... 94 7.3.4 Query Evaluation............................ 95 7.3.5 Proof Engine.............................. 96 7.3.6 Resolution (Display).......................... 97 7.3.7 Output Module............................. 98 8 Evaluation 99 8.1 Similarity Metrics................................ 99 8.1.1 Average Based.............................. 100 8.1.2 Token Based............................... 101 8.1.3 Tree Structure Based.......................... 102 8.2 Sentence Extraction............................... 104 8.2.1 Performance............................... 105 8.3 User Interactions................................ 106 8.3.1 Sample Queries............................. 107 8.3.2 Wikipedia Application......................... 108 8.4 Limitations................................... 110 8.4.1 Co-reference resolution......................... 110 8.4.2 Follow-up Queries............................ 111 8.4.3 Single Sentence Selection........................ 113 9 Conclusion 115 9.1 Summary of Work................................ 115 9.2 Future Work................................... 117 9.2.1 Neural Logic Amalgamation...................... 117 5 CONTENTS CONTENTS 9.2.2 Multi Sentence Comprehension.................... 118 A NLP 119 A.1 Part-of-speech Tags (POS)........................... 119 A.2 Dependency Tags (DEP)............................ 119 6 List of Figures 2.1 The Radar system pipeline overview...................... 19 2.2 The SimCity assistant architecture....................... 20 2.3 Recast AI conversation flow representation................... 20 2.4 Attorney 209 architecture overview....................... 24 2.5 Project Execution Assistant (PExA) architecture overview.......... 25 2.6 Project Execution Assistant (PExA) follow up dialogue interaction..... 26 2.7 A generic cloud-based personal assistant architecture............. 27 2.8 Sirius architecture overview with mobile and server components....... 27 2.9 CCG proof of "Marcel proved completeness" using lambda calculus..... 31 2.10 PENNMAN representation of AMR on "The boy wants to go."....... 32 2.11 Graphical representation of AMR on "The boy wants to go."........ 32 3.1 Entity recognition and variable assignment in Wit AI............. 35 3.2 SLDNF operational semantics of goal oriented proof............. 38 3.3 Example search space of SLDNF on list membership............. 38 3.4 Continuous bag-of-words and skip-gram models presented in word2vec [1].. 39 3.5 Word2vec results from TensorFlow [2] example implementation....... 41 3.6 Superlative linear structures in learnt vectors from GloVe.......... 42 3.7 A representation of an LSTM unit....................... 43 3.8 BiDAF model architecture overview...................... 44 3.9 BiDAF model contextual embedding visualisation............... 45 4.1 Graphical representation of a basic rule in logic framework.......... 47 4.2 Query rules for music playing intent in API AI................ 52 7 LIST OF FIGURES LIST OF FIGURES 4.3 Graphical representation of negation by failure................ 58 4.4 Graphical representation of overriding expression proofs with neural networks. 63 5.1 Fuzzy unification example on slightly different sentence structures...... 72 6.1 The architecture of the network used in guided BiDAF............ 77 6.2 Factual answer extraction using guided BiDAF................ 81 6.3 Causal answer extraction using guided BiDAF................. 82 6.4 Failed