University of Pennsylvania ScholarlyCommons Publicly Accessible Penn Dissertations 2019 Reasoning-Driven Question-Answering For Natural Language Understanding Daniel Khashabi University of Pennsylvania, [email protected] Follow this and additional works at: https://repository.upenn.edu/edissertations Part of the Artificial Intelligence and Robotics Commons Recommended Citation Khashabi, Daniel, "Reasoning-Driven Question-Answering For Natural Language Understanding" (2019). Publicly Accessible Penn Dissertations. 3271. https://repository.upenn.edu/edissertations/3271 This paper is posted at ScholarlyCommons. https://repository.upenn.edu/edissertations/3271 For more information, please contact [email protected]. Reasoning-Driven Question-Answering For Natural Language Understanding Abstract Natural language understanding (NLU) of text is a fundamental challenge in AI, and it has received significant attention throughout the history of NLP research. This primary goal has been studied under different tasks, such as Question Answering (QA) and Textual Entailment (TE). In this thesis, we investigate the NLU problem through the QA task and focus on the aspects that make it a challenge for the current state-of-the-art technology. This thesis is organized into three main parts: In the first part, we explore multiple formalisms to improve existing machine comprehension systems. We propose a formulation for abductive reasoning in natural language and show its effectiveness, especially in domains with limited training data. Additionally, to help reasoning systems cope with irrelevant or redundant information, we create a supervised approach to learn and detect the essential terms in questions. In the second part, we propose two new challenge datasets. In particular, we create two datasets of natural language questions where (i) the first one requires reasoning over multiple sentences; (ii) the second one requires temporal common sense reasoning. We hope that the two proposed datasets will motivate the field to address more complex problems. In the final part, we present the first formal framework for multi-step reasoning algorithms, in the presence of a few important properties of language use, such as incompleteness, ambiguity, etc. We apply this framework to prove fundamental limitations for reasoning algorithms. These theoretical results provide extra intuition into the existing empirical evidence in the field. Degree Type Dissertation Degree Name Doctor of Philosophy (PhD) Graduate Group Computer and Information Science First Advisor Dan Roth Keywords Language Understanding, Natural Language Processing Subject Categories Artificial Intelligence and Robotics | Computer Sciences This dissertation is available at ScholarlyCommons: https://repository.upenn.edu/edissertations/3271 REASONING-DRIVEN QUESTION-ANSWERING FOR NATURAL LANGUAGE UNDERSTANDING Daniel Khashabi A DISSERTATION in Computer and Information Sciences Presented to the Faculties of the University of Pennsylvania in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy 2019 Supervisor of Dissertation Dan Roth, Professor of Computer and Information Science Graduate Group Chairperson Rajeev Alur, Professor of Computer and Information Science Dissertation Committee Dan Roth, Professor, Computer and Information Science, University of Pennsylvania Mitch Marcus, Professor of Computer and Information Science, University of Pennsylvania Zachary Ives, Professor of Computer and Information Sciences, University of Pennsylvania Chris Callison-Burch, Associate Professor of Computer Science, University of Pennsylvania Ashish Sabharwal, Senior Research Scientist, Allen Institute for Artificial Intelligence REASONING-DRIVEN QUESTION-ANSWERING FOR NATURAL LANGUAGE UNDERSTANDING c COPYRIGHT 2019 Daniel Khashabi This work is licensed under the Creative Commons Attribution NonCommercial-ShareAlike 3.0 License To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/ Dedicated to the loving memory of my gramma, An'nah Your patience and kindness will forever stay with me. iii ACKNOWLEDGEMENT I feel incredibly lucky to have Dan Roth as my advisor. I am grateful to Dan for trusting me, especially when I had only a basic understanding of many key challenges in natural language. It took me a while to catch up with what is important in the field and be able to communicate the challenges effectively. During these years, Dan's vision has always been the guiding principle to many of my works. His insistence on focusing on the long-term progress, rather than \easy" wins, shaped the foundation of many of the ideas I pursued. This perspective pushed me to think differently than the popular trends. It has been a genuine privilege to work together. I want to thank my thesis committee at UPenn, Mitch Marcus, Zach Ives and Chris Callison- Burch for being a constant source of invaluable feedback and guidance. Additionally, I would like to thanks the many professors who have touched parts of my thinking: Jerry DeJong, for encouraging me read the classic literature; Chandra Chekuri and Avrim Blum, for their emphasis on intuition, rather than details; and my undergraduate advisor Hamid Sheikhzadeh Nadjar, for encouraging me to work on important problems. A huge thank you to the Allen Institute for Artificial Intelligence (AI2) for much support during my PhD studies. Any time I needed any resources (computing resources, crowdsourc- ing credits, engineering help, etc), without any hesitation, AI2 has provided me what was needed. Special thanks to Ashish Sabhwaral and Tushar Khot for being a constant source of wisdom and guidance, and investing lots of time and effort. They both have always been present to listen to my random thoughts, almost on a weekly basis. I am grateful to other members of AI2 for their help throughout my projects: Oren Etzioni, Peter Clark, Oyvind Tafjord, Peter Turney, Ingmar Ellenberger, Dirk Groeneveld, Michael Schmitz, Chandra Bhagavatula and Scott Yih. Moreover, I would like to remember Paul Allen (1953-2018): his vision and constant generous support has tremendously changed our field (and my life, in particular). iv My collaborators, especially past and present CogComp members, have been major con- tributors and influencers throughout my works. I would like to thank Mark Sammons, Vivek Srikumar, Christos Christodoulopoulos, Erfan Sadeqi Azer, Snigdha Chaturvedi, Kent Quanrud, Amirhossein Taghvaei, Chen-Tse Tsai, and many other CogComp members. Furthermore, I thank Eric Horn and Jennifer Sheffield for their tremendous contributions to many of my write-ups. And thank you to all the friends I have made at Penn, UIUC, and elsewhere, for all the happiness you've brought me. Thanks to Whitney, for sharing many happy and sad moments with me, and for helping me become a better version of myself. Last, but never least, my family, for their unconditional sacrifice and support. I wouldn't have been able to go this far without you. v ABSTRACT REASONING-DRIVEN QUESTION-ANSWERING FOR NATURAL LANGUAGE UNDERSTANDING Daniel Khashabi Dan Roth Natural language understanding (NLU) of text is a fundamental challenge in AI, and it has received significant attention throughout the history of NLP research. This primary goal has been studied under different tasks, such as Question Answering (QA) and Textual Entailment (TE). In this thesis, we investigate the NLU problem through the QA task and focus on the aspects that make it a challenge for the current state-of-the-art technology. This thesis is organized into three main parts: In the first part, we explore multiple formalisms to improve existing machine comprehension systems. We propose a formulation for abductive reasoning in natural language and show its effectiveness, especially in domains with limited training data. Additionally, to help reasoning systems cope with irrelevant or redundant information, we create a supervised approach to learn and detect the essential terms in questions. In the second part, we propose two new challenge datasets. In particular, we create two datasets of natural language questions where (i) the first one requires reasoning over multiple sentences; (ii) the second one requires temporal common sense reasoning. We hope that the two proposed datasets will motivate the field to address more complex problems. In the final part, we present the first formal framework for multi-step reasoning algorithms, in the presence of a few important properties of language use, such as incompleteness, ambiguity, etc. We apply this framework to prove fundamental limitations for reasoning algorithms. These theoretical results provide extra intuition into the existing empirical evidence in the field. vi TABLE OF CONTENTS ACKNOWLEDGEMENT . iv ABSTRACT . vi LIST OF TABLES . xv LIST OF ILLUSTRATIONS . xx PUBLICATION NOTES . xxi CHAPTER 1 : Introduction . 1 1.1 Motivation . 1 1.2 Challenges along the way to NLU . 1 1.3 Measuring the progress towards NLU via Question Answering . 4 1.4 Thesis outline . 6 CHAPTER 2 : Background and Related Work . 8 2.1 Overview . 8 2.2 Terminology . 8 2.3 Measuring the progress towards NLU . 10 2.3.1 Measurement protocols . 10 2.4 Knowledge Representation and Abstraction for NLU . 14 2.4.1 Early Works: \Neats vs Scruffies”1 ................... 14 2.4.2 Connectionism . 16 2.4.3 Unsupervised representations . 16 2.4.4 Grounding of meanings . 17 1Terms originally made by Roger Schank to characterize two different camps: the first group that repre- sented commonsense knowledge