Towards Understanding Natural Language: Semantic Parsing, Commonsense Knowledge Acquisition, Reasoning Framework and Applications by Arpit Sharma A Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy Approved June 2019 by the Graduate Supervisory Committee: Chitta Baral, Chair Joohyung Lee Paolo Papotti Yezhou Yang ARIZONA STATE UNIVERSITY August 2019 ABSTRACT Reasoning with commonsense knowledge is an integral component of human behavior. It is due to this capability that people know that a weak person may not be able to lift some- one. It has been a long standing goal of the Artificial Intelligence community to simulate such commonsense reasoning abilities in machines. Over the years, many advances have been made and various challenges have been proposed to test their abilities. The Winograd Schema Challenge (WSC) is one such Natural Language Understanding (NLU) task which was also proposed as an alternative to the Turing Test. It is made up of textual question answering problems which require resolution of a pronoun to its correct antecedent. In this thesis, two approaches of developing NLU systems to solve the Winograd Schema Challenge are demonstrated. To this end, a semantic parser is presented, vari- ous kinds of commonsense knowledge are identified, techniques to extract commonsense knowledge are developed and two commonsense reasoning algorithms are presented. The usefulness of the developed tools and techniques is shown by applying them to solve the challenge. i DEDICATION To mom, dad and my wife Megha ii ACKNOWLEDGMENTS I would like to thank my advisor Dr. Chitta Baral for providing me the opportunity to work on this research and guiding me throughout. I would like to thank Dr. Joohyung Lee, Dr. Paolo Papotti and Dr. Yezhou Yang for being a part of my thesis defense committee and providing me their valuable feedback on my work. I would like to thank my collaborators Dr. Davy Weissenbacher and Dr. Nguyen Vo Ha for their valuable guidance and support in various projects. I would also like to thank Dr. Saadat Anwar for his technical and moral support when I was facing difficult times in my research. I would like to thank my peers in Dr. Chitta Baral’s lab at ASU for their valuable feedback on my works. Finally, I would like to thank my family and friends for their continuous support thro- ughout the years. iii TABLE OF CONTENTS Page LIST OF TABLES . ix LIST OF FIGURES . xi CHAPTER 1 INTRODUCTION . 1 1.1 Motivation . 1 1.2 Contributions of the Research . 3 1.3 Dissertation Organization . 7 2 SOLVING THE WINOGRAD SCHEMA CHALLENGE: A GLIMPSE OF OUR TWO APPROACHES . 9 2.1 Introduction . 9 2.2 Related Challenges . 11 2.3 Approach 1: Semantic Parsing and Graphical Reasoning Approach . 14 2.3.1 Step 1: Formal Representation of a WSC Problem . 14 2.3.2 Step 2: Commonsense Knowledge Extraction and Representation 14 2.3.3 Step 3: Reasoning with Commonsense Knowledge . 16 2.4 Approach 2: Natural Language Inference Based Approach . 19 2.4.1 Step 1: Knowledge Hunting . 20 2.4.2 Step 2: Entity Alignment. 20 2.4.3 Step 3: Answer Retrieval. 21 3 KNOWLEDGE TYPES IDENTIFICATION . 22 3.1 Introduction . 22 3.2 Commonsense Knowledge Categorization . 24 3.2.1 Knowledge Type 1: A Property May Prevent an Action . 25 3.2.2 Knowledge Type 2: An Action May Cause an Action . 26 iv CHAPTER Page 3.2.3 Knowledge Type 3: A Property May Cause an Action. 26 3.2.4 Knowledge Type 4: An Action May Cause a Property. 27 3.2.5 Knowledge Type 5: An Action May Prevent an Action . 27 3.2.6 Knowledge Type 6: An Action May be Followed By an Action . 28 3.2.7 Knowledge Type 7: An Action May be Followed by a Property . 28 3.2.8 Knowledge Type 8: A Property May be followed by an Action . 29 3.2.9 Knowledge Type 9: A Property May Cause a Property . 29 3.2.10 Knowledge Type 10: A Co-occurring Set of Actions and Prop- erties . 30 3.2.11 Additional Knowledge Categories . 31 4 REASONING WITH THE COMMONSENSE KNOWLEDGE . 34 4.1 Formal Representation of a WSC Problem . 34 4.1.1 Abstract Meaning Representation (AMR) . 34 4.1.2 Knowledge Parser (K-Parser) Representation . 35 4.1.3 TRIPS Parser Representation . 36 4.2 Representation of the Knowledge Types . 43 4.2.1 Representation of Knowledge Types 1 through 10 . 43 4.3 Reasoning Algorithm for Knowledge Types 1-10 . 47 4.3.1 Winograd Schema Challenge Reasoning (WiSCR) Algorithm . 49 4.3.2 Implementation of the WiSCR Algorithm . 53 4.3.3 Adding New Constraints . 61 4.4 Empirical Evaluation of the WiSCR Algorithm . 64 4.5 Representation of the Knowledge Types: Revisited . 66 v CHAPTER Page 4.5.1 Representation of the Knowledge Type 11: Statement 1 is more likely than Statement 2 . 66 4.5.2 Representation of the Knowledge Type 12: Multiple Knowl- edge Pieces . 67 4.6 Reasoning : Revisited . 68 4.6.1 Reasoning Algorithm to Handle “is more likely than” Knowledge 68 4.6.2 Reasoning Algorithm to Handle Multiple Knowledge Pieces. 72 5 KNOWLEDGE HUNTING AND KNOWLEDGE BASE CREATION . 73 5.1 Introduction & Motivation . 73 5.2 Events-Based Conditional Commonsense Knowledge Extraction . 75 5.2.1 Background . 76 5.2.2 Knowledge Extraction Algorithm . 77 5.2.3 Storage and Retrieval of the Knowledge . 81 5.2.4 Evaluation . 82 5.3 Extraction of Commonsense Knowledge about Actions . 84 5.3.1 Background . 84 5.3.2 The Extraction Algorithm . 85 5.3.3 Evaluation of the Extracted Knowledge . 91 5.4 Automatic Extraction of Knowledge For the Winograd Schema Chal- lenge Problems. 93 5.5 Related Works . 95 5.6 Conclusion . 97 6 PARSING TEXT TO SEMANTIC GRAPHS: THE KNOWLEDGE PARSER 99 vi CHAPTER Page 6.1 Knowledge Parser (K-Parser): An Events Centered Semantic Represen- tation System . 99 6.1.1 Semantic Representation . 101 6.1.2 Semantic Parsing . 106 6.1.3 Evaluation . 110 6.1.4 Related Works . 113 6.2 Improvements in the K-Parser System . 115 6.3 K-Parser Wrapper to Automatically Represent WSC Sentences . 117 6.4 K-Parser Wrapper to Automatically Represent Commonsense Knowledge119 7 End-To-End Systems . 121 7.1 System 1: Graphical Reasoning Based System . 121 7.1.1 Overview of the System. 121 7.1.2 Representing WSC Problems . 122 7.1.3 Extraction and Representation of Commonsense Knowledge . 123 7.1.4 Reasoning Algorithm . 124 7.1.5 Evaluation . 124 7.1.6 Conclusion . 126 7.2 System 2: Entailment Based Reasoning System . 127 7.2.1 Problem Formulation . 130 7.2.2 Knowledge Extraction . 131 7.2.3 Alignment Algorithm . 134 7.2.4 Experiments . 139 7.2.5 Conclusion . 147 vii CHAPTER Page 7.3 Updated System 2: Combining Automatic Knowledge Extraction and Neural Language Models. 148 7.3.1 Knowledge Extraction . ..