NISTIR 7310 Evaluating Reasoning Systems Conrad Bock Michael Gruninger Don Libes Joshua Lubell Eswaran Subrahmanian NISTIR 7310 Evaluating Reasoning Systems Conrad Bock Michael Gruninger1 Don Libes Joshua Lubell 2 Eswaran Subrahmanian Manufacturing Systems Integration Division Manufacturing Engineering Laboratory 1Also at the University of Toronto, and during some of the preparation of this report, at the University of Maryland, College Park. 2Also at Carnegie Mellon University. May 2006 U.S. DEPARTMENT OF COMMERCE Carlos M. Gutierrez, Secretary TECHNOLOGY ADMINISTRATION Robert Cresanti, Under Secretary of Commerce for Technology NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY William Jeffrey, Director Executive Summary A review of the literature on evaluating reasoning systems reveals that it is a very broad area with wide variation in depth and breadth of research on metrics and tests. Consolidation is hampered by nonstandard terminology, differing methodologies, scattered application domains, unpublished algorithmic details, and the effects of domain content and context on the choice of metric and tests. The field of information metrology, which applies to reasoning as a kind of information processing, is still emerging from ad hoc experience in evaluating narrow kinds of information systems. This report begins to bring order to the area by categorizing reasoning systems according to their capabilities. The characteristics of each category can be used as a basis for evaluating and testing reasoning systems claiming to be in that category. Capabilities are analyzed along several dimensions, including representation languages, inference, and user and software interfaces. The report groups representation languages by their relation to first-order logic, and model-theoretic properties, such as soundness and completeness. Inference procedures are divided into deduction, induction, abduction, and analogical reasoning. Capabilities of user and software interfaces are described as they apply to reasoning systems. The report introduces information metrology, model theory, and inference to facilitate understanding of the reasoning categories presented. It concludes with recommendations for future work. Applying the results of this report to evaluation of specific reasoning systems requires further refinement of reasoning categories as needed to support development of test cases of interest to reasoning system users. Then development can proceed on generic test cases for reasoning categories. These are independent of the application, reasoning tool, and computational platform. Finally, based on the generic tests, development can start on specific test cases that are dependent on application, reasoning system tool, and computational platform. These steps can also be applied to user interface and software interface capabilities. This work can only be completed in partnership with users of reasoning systems and reasoning tool providers, to focus effort and reach the level of completeness necessary to put these metrics into practice. 2 Table of Contents 1 Introduction .................................................................................................................. 5 2 Information Metrology................................................................................................. 5 3 Reasoning..................................................................................................................... 9 3.1 Introduction to Reasoning ................................................................................... 10 3.1.1 Model Theory................................................................................................ 10 3.1.2 Inference ....................................................................................................... 13 3.2 Representation Languages................................................................................... 15 3.2.1 First-Order Logic .......................................................................................... 16 3.2.2 Restrictions of First-Order Logic.................................................................. 17 3.2.2.1 Monadic First-Order Logic........................................................................ 17 3.2.2.2 Universal-Existential Sentences ................................................................ 17 3.2.2.3 Universal Sentences................................................................................... 17 3.2.2.4 Horn Clauses.............................................................................................. 18 3.2.2.5 Datalog....................................................................................................... 19 3.2.2.6 Existential Sentences................................................................................. 19 3.2.3 Beyond First-Order Logic............................................................................. 19 3.2.3.1 Transitive Closure Logic ........................................................................... 19 3.2.3.2 Least Fixpoint Logic.................................................................................. 19 3.2.3.3 Partial Fixpoint Logic................................................................................ 20 3.2.3.4 Second-Order Logic .................................................................................. 20 3.2.3.5 Monadic Second-Order Logic ................................................................... 20 3.2.3.6 Infinitary Logic.......................................................................................... 21 3.2.3.7 CycL .......................................................................................................... 21 3.2.4 Reified First-Order Logics............................................................................ 22 3.2.4.1 Common Logic.......................................................................................... 22 3.2.5 Description Logics........................................................................................ 23 3.2.6 Web Languages............................................................................................. 24 3.2.6.1 RDF/S ........................................................................................................ 24 3.2.6.2 OWL .......................................................................................................... 25 3.2.7 Modal Logics ................................................................................................ 26 3.2.7.1 Propositional Modal Logics....................................................................... 26 3.2.7.2 Quantified Modal Logics........................................................................... 28 3.2.8 Nonmonotonic Logics................................................................................... 28 3.2.8.1 Reiter’s Default Logic ............................................................................... 28 3.2.8.2 Model Preference Default Logic ............................................................... 29 3.2.8.3 Circumscription ......................................................................................... 29 3.2.8.4 Autoepistemic Logic.................................................................................. 30 3.2.9 Additional Terminology................................................................................ 30 3.3 Inference .............................................................................................................. 32 3.3.1 Deduction...................................................................................................... 33 3.3.1.1 First order deduction.................................................................................. 33 3.3.1.2 Rule Systems ............................................................................................. 36 3.3.1.2.1 Intersection of Inference and Programming ........................................ 36 3.3.1.2.2 Basic Rule System Semantics.............................................................. 38 3 3.3.1.2.3 Rule System Inference ......................................................................... 40 3.3.1.2.4 Rule System Semantics Continued ...................................................... 44 3.3.1.3 Temporal Reasoning.................................................................................. 53 3.3.1.3.1 Types of Temporal Reasoning System ................................................ 53 3.3.1.3.2 Evaluation of temporal systems........................................................... 54 3.3.2 Induction ....................................................................................................... 55 3.3.2.1 Definitions of Induction............................................................................. 56 3.3.2.2 Measurement for Machine Learning Systems........................................... 57 3.3.2.2.1 Evaluating Machine learning systems ................................................. 57 3.3.2.2.2 Rule Quality Measures for Rule Induction Systems............................ 60 3.3.3 Abduction...................................................................................................... 62 3.3.3.1 Best explanation Model of Abduction....................................................... 63 3.3.3.1.1 Josephson best explanation model....................................................... 63 3.3.3.1.2 Konolige Model for Abduction...........................................................