Automated Reasoning Embedded in Question Answering Bj¨ornPelzer Mai 2013 Vom Promotionsausschuss des Fachbereichs 4: Informatik der Universit¨atKoblenz-Landau zur Verleihung des akademischen Grades Doktor der Naturwissenschaften (Dr. rer. nat.) genehmigte Dissertation. Vorsitzender des Promotionsausschusses: Prof. Dr. R¨udigerGrimm Vorsitzende der Promotionskommission: Prof. Dr. Karin Harbusch 1. Berichterstatter: Prof. Dr. Ulrich Furbach 2. Berichterstatter: Prof. Dr. Peter Baumgartner Datum der wissenschaftlichen Aussprache: 03.05.2013 Ver¨offentlicht als Dissertation an der Universit¨atKoblenz-Landau. ii Abstract This dissertation investigates the usage of theorem provers in auto- mated question answering (QA). QA systems attempt to compute correct answers for questions phrased in a natural language. Commonly they utilize a multitude of methods from computational linguistics and knowl- edge representation to process the questions and to obtain the answers from extensive knowledge bases. These methods are often syntax-based, and they cannot derive implicit knowledge. Automated theorem provers (ATP) on the other hand can compute logical derivations with millions of inference steps. By integrating a prover into a QA system this reasoning strength could be harnessed to deduce new knowledge from the facts in the knowledge base and thereby improve the QA capabilities. This in- volves challenges in that the contrary approaches of QA and automated reasoning must be combined: QA methods normally aim for speed and robustness to obtain useful results even from incomplete of faulty data, whereas ATP systems employ logical calculi to derive unambiguous and rigorous proofs. The latter approach is difficult to reconcile with the quantity and the quality of the knowledge bases in QA. The dissertation describes modifications to ATP systems in order to overcome these ob- stacles. The central example is the theorem prover E-KRHyper which was developed by the author at the Universit¨atKoblenz-Landau. As part of the research work for this dissertation E-KRHyper was embedded into a framework of components for natural language processing, information retrieval and knowledge representation, together forming the QA system LogAnswer. Also presented are additional extensions to the prover im- plementation and the underlying calculi which go beyond enhancing the reasoning strength of QA systems by giving access to external knowledge sources like web services. These allow the prover to fill gaps in the knowl- edge during the derivation, or to use external ontologies in other ways, for example for abductive reasoning. While the modifications and exten- sions detailed in the dissertation are a direct result of adapting an ATP system to QA, some of them can be useful for automated reasoning in gen- eral. Evaluation results from experiments and competition participations demonstrate the effectiveness of the methods under discussion. iii iv Zusammenfassung Die vorliegende Dissertation behandelt den Einsatz von Theorembe- weisern innerhalb der automatischen Fragebeantwortung (question answe- ring - QA). QA-Systeme versuchen, naturlichsprachliche¨ Fragen korrekt zu beantworten. Sie verwenden eine Vielzahl von Methoden aus der Com- puterlinguistik und der Wissensrepr¨asentation, um menschliche Sprache zu verarbeiten und die Antworten aus umfangreichen Wissensbasen zu beziehen. Diese Methoden sind allerdings meist syntaxbasiert und k¨onnen kein implizites Wissen herleiten. Die Theorembeweiser der automatischen Deduktion dagegen k¨onnen Folgerungsketten mit Millionen von Inferenz- schritten durchfuhren.¨ Die Integration eines Beweisers in ein QA-System er¨offnet die M¨oglichkeit, aus den Fakten einer Wissensbasis neues Wissen herzuleiten und somit die Fragebeantwortung zu verbessern. Herausfor- derungen liegen in der Uberwindung¨ der gegens¨atzlichen Herangehens- weisen von Fragebeantwortung und Deduktion: W¨ahrend QA-Methoden normalerweise darauf abzielen, auch mit unvollst¨andigen oder fehlerhaf- ten Daten robust und schnell zu halbwegs annehmbaren Ergebnissen zu kommen, verwenden Theorembeweiser logische Kalkule¨ zur Gewinnung exakter und beweisbarer Resultate. Letzterer Ansatz erweist sich sich aber als schwer vereinbar mit der Quantit¨at und der Qualit¨at der im QA- Bereich ublichen¨ Wissensbest¨ande. Die Dissertation beschreibt Anpassun- gen von Theorembeweisern zur Uberwindung¨ dieser Hurden.¨ Zentrales Beispiel ist der an der Universit¨at Koblenz-Landau entwickelte Beweiser E-KRHyper, der im Rahmen dieser Dissertation in das QA-System Log- Answer integriert worden ist. Außerdem vorgestellt werden zus¨atzliche Er- weiterungsm¨oglichkeiten auf der Implementierungs- und der Kalkulebene,¨ die sich aus dem praktischen Einsatz bei der Fragebeantwortung ergeben haben, dabei aber generell fur¨ Theorembeweiser von Nutzen sein k¨onnen. Uber¨ die reine Deduktionsverbesserung der QA hinausgehend beinhalten diese Erweiterungen auch die Anbindung externer Wissensquellen wie et- wa Webdienste, mit denen der Beweiser w¨ahrend des Deduktionsvorgangs gezielt Wissenslucken¨ schließen kann. Zudem erm¨oglicht dies die Nutzung externer Ontologien beispielsweise zur Abduktion. Evaluationsergebnis- se aus eigenen Versuchsreihen und aus Wettbewerben demonstrieren die Effektivit¨at der diskutierten Methoden. v vi For Ania vii viii Acknowledgements While this dissertation is the result of several years of my work, this work would have been impossible without the aid and advice by many people I am fortunate to know. Here I would like to express my gratitude towards these supporters. First of all I thank my thesis supervisor Ulrich Furbach for his guidance and encouragement in my work, and for allowing me the chance at this project in the first place. His tutelage provided me with manifold experience in the world of science, and I am indebted both to his willingness to explore promising areas, and to his pragmatism at not losing sight of the main goals. Likewise I thank Peter Baumgartner, who let me take my first steps in AI, who also worked with me on various occasions over the years, allowing me to broaden the scope of my experiences, and who took on the task of being the second reviewer of this thesis. My gratitude goes out to the members of the AGKI for the great work environment. Here I would like to single out Markus Bender for his contributions to LogAnswer, and Beate K¨ornerfor always knowing everything and for taking care of bureaucracy. I am also obliged to former AGKI member Christoph Wernhard, whose work provided the foundation for my own. Regarding the LogAnswer project I am also most thankful to my colleagues in Hagen, notably Ingo Phoenix n´eGl¨ockner, for the good cooperation over many years. My research was backed by the Deutsche Forschungsgemeinschaft (DFG) and of course the Universit¨atKoblenz-Landau, which I appreciate greatly. I thank my family and friends for their support and patience - in particular my parents Anna-Lena and Heinz-Leo Pelzer, for proofreading and constructive criticism, for always being there for me and for helping out in countless instances. I hope I can make up for this somehow. In the same way I am indebted to Anna Sto_zekfor her moral support and for bearing with me. For a special kind of inspiration and motivation I am thankful to Iain M. Banks, one of the few authors who have shown AI contributing to a better future. Finally I am grateful to Hans de Nivelle for bringing CADE to Wroc law in 2011, a decision that has enriched my life in so many ways. ix x Preliminary Remarks The work presented in this dissertation is part of the LogAnswer research project which involved several scientists, see Section 5.4. For a proper consideration of my own contributions a broader understanding of LogAnswer is required. As some aspects of LogAnswer were dealt with in cooperative work while others were the responsibility of individuals, generally the author's \we" will be used for the sake of consistency throughout the remainder of the dissertation. How- ever, on some occasions I will refer to myself explicitly when clarifying which parts are my own contributions. Also, some sections of this dissertation describe work, some of it my own, which was not conducted during my research for this thesis. These aspects are therefore not new contributions, but they are nevertheless included for clar- ity, as my dissertation builds upon them and would be difficult to understand otherwise. This will be clearly mentioned when introducing these subjects. References to my own publications have numeric labels ([1], [2], . ), whereas other references use mnemonic labels consisting of abbreviations of the authors' names and the year of publication, for example [Wer03]. A collection of materials relevant for this dissertation, including theorem prover implementations and evaluation logs, is available online.1 1http://userpages.uni-koblenz.de/~bpelzer/dissertation/materials.tar.gz xi xii Contents 1 Introduction 5 2 Formal Preliminaries 11 2.1 First-Order Logic . 11 2.1.1 Terms and Substitutions . 11 2.1.2 Formulas . 12 2.1.3 Satisfiability . 12 2.1.4 Multisets . 13 2.1.5 Clauses . 13 2.1.6 Calculi . 14 2.2 Equality . 14 2.2.1 Equations . 15 2.2.2 Positions . 15 2.2.3 Term Ordering . 16 2.2.4 Rewrite Systems . 17 3 Automated Reasoning 19 3.1 Automated Theorem Proving . 19 3.2 Theorem Prover Implementation . 22 3.3 Theorem Prover Evaluation . 27 4 Question Answering 31 4.1 QA System Implementation . 34 4.2 QA System Evaluation . 36 5 Combining Question Answering and Automated Reasoning 39 5.1 Advantages and Problems of Conventional QA Methods . 39 5.2 Advantages and Problems of AR Methods . 42 5.2.1 Logical Knowledge Base Representation . 43 5.2.2 Size of the Knowledge Base . 44 5.2.3 Brittleness of Precision . 47 5.3 Issues and Goals of the Combination . 48 5.4 The LogAnswer Research Project . 48 6 The Deductive Basis - Hyper Tableaux 51 6.1 The Hyper Tableaux Calculus . 51 6.1.1 Trees and Tableaux . 52 6.1.2 Hyper Tableaux . 52 1 6.1.3 Redundancy and Model Generation . 53 6.1.4 Hyper Tableaux Derivations . 54 6.1.5 Hyper Tableaux Derivation Example . 54 6.2 The E-Hyper Tableaux Calculus . 55 6.2.1 Inference Rules .