SPATIAL PROBLEM SOLVING FOR DIAGRAMMATIC REASONING DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Bonny Banerjee, M.S. * * * * * The Ohio State University 2007 Dissertation Committee: Approved by Dr. Balakrishnan Chandrasekaran, Adviser Dr. John R. Josephson Dr. Tamal K. Dey _________________________________ Adviser Graduate Program in Computer Science and Engineering Copyright by Bonny Banerjee 2007 ABSTRACT Diagrammatic reasoning (DR) is pervasive in human problem solving as a powerful adjunct to symbolic reasoning based on language-like representations. However, Artificial Intelligence is overwhelmingly based on symbolic representations, with proportionately scant attention to diagrams. This dissertation is a contribution to building artificial agents that can create and use diagrams as part of their problem solving. The work is in a framework in which DR is modeled as a process in which subtasks are solved, as appropriate, either by inference from symbolic representations or by information perceived from a diagram, and subtasks may also act on the diagram, i.e., create or modify objects in the diagram. The perceptions and actions are in fact domain- and task-specific 2D spatial problems defined in terms of properties and relations involving diagrammatic objects. Most DR systems built so far are task-specific, and their developers as a rule have hand-crafted the required perceptions and actions. Our goal is the development of a general, i.e., domain- and task-independent, capability that takes specifications of perceptions and actions and automatically executes them. Thus, the purpose of this dissertation is to investigate: ii 1. A language for a human problem solver to communicate a wide variety of 2D spatial problems relevant to DR, and 2. A general domain-independent framework of underlying representations and reasoning strategies suitable for efficiently solving spatial problems without human intervention. This dissertation will present a high-level language that is extensible, human- usable, and expressive enough to describe a wide variety of spatial problems in terms of constraints. The constraints are specified in first-order logic over the real domain using a vocabulary of objects, properties, relations and actions. Two general and independent strategies -- constraint satisfaction and spatial search -- are developed for automatically solving the spatial problems specified in that language. Several ideas about how to make these strategies computationally efficient are proposed and illustrated by examples. A traditional AI problem solver is augmented with this spatial problem solver for reasoning with diagrams in different domains for real-world applications. The utility of the framework is judged by the expressiveness of the language, and generality and efficiency of the two strategies. iii Dedicated to my wonderful family iv ACKNOWLEDGMENTS I gratefully acknowledge the guidance and support rendered to me by my adviser Prof. B. Chandrasekaran (Chandra) in walking me through the problem, developing the concepts related to the subject of diagrammatic reasoning and representation, providing valuable suggestions and preparing this dissertation. I am also indebted to Dr. John R. Josephson for helping me with discussions and valuable suggestions related to my work. Thanks are due to Prof. Tamal K. Dey for being in my dissertation committee and helping me with research issues from time to time. Thanks are also due to Unmesh Kurup and Vivek Bharathan for helping me in many different ways, especially through lively intellectual discussions. Several others in the Department of Computer Science & Engineering, whose names have not been mentioned, have also been very helpful. I wish to thank each of them and extend my sincerest apologies for overlooking their contribution. Finally, I will always remain deeply indebted to my family for providing the support that made this dissertation possible. The research reported in this dissertation was supported by participation in the Advanced Decision Architectures Collaborative Technology Alliance sponsored by the U.S. Army Research Laboratory under Cooperative Agreement DAAD19-01-2- 0009. v VITA November, 1977 Born – Calcutta (Kolkata), India July, 2000 B.E. Electronics & Telecommunication Engineering Jadavpur University, India. August, 2002 M.S. Electrical Engineering The Ohio State University, Columbus, USA October, 2001 – June, 2007 Graduate Research Associate Laboratory for Artificial Intelligence Research, Department of Computer Science & Engineering, The Ohio State University, Columbus, USA. PUBLICATIONS B. Banerjee, ”String tightening as a self-organizing phenomenon.” IEEE Transactions on Neural Networks, 18(5):1463-1471, (2007). B. Banerjee and B. Chandrasekaran, ”A constraint satisfaction framework for visual problem solving.” Trends in Constraint Programming, F. Benhamou, N. Jussien and B. OSullivan, Editors, Hermes Science, Chapter 26, (2007). B. Banerjee, ”A layered abductive inference framework for diagramming group motions.” Special Issue of Logic Journal of IGPL: Abduction, Practical Reasoning, and Creative Inferences in Science, L. Magnani, Editor, 14(2):363-378, Oxford University Press, (2006). B. Chandrasekaran, U. Kurup, B. Banerjee, J. R. Josephson and R.Winkler, ”An architecture for problem solving with diagrams.” Diagrammatic Representation and Inference, A. Blackwell, K. Marriott and A. Shimojima, Editors, Lecture Notes in Artificial Intelligence 2980:151-165, Berlin: Springer-Verlag, (2004). vi B. Banerjee, ”Recognition of partially occluded shapes using a neural optimization network.” Machine Graphics & Vision, Institute of Computer Science of the Polish Academy of Sciences, 13(1/2):3-23, (2004). FIELDS OF STUDY Major Field: Computer Science and Engineering Specialization: Artificial Intelligence Minor Fields: Cognitive Science, Computer Graphics vii TABLE OF CONTENTS Abstract…………………………………………………………………………..… ii Dedication………………………………………………………………………..… v Acknowledgments………………………………………………………………..…vi Vita……………………………………………………………………………….… viii List of Figures……………………………………………………………………… xii Chapters: 1. Introduction………………………………………………………………..…… 1 1.1 Diagrammatic reasoning as a problem solving activity.....………………… 1 1.2 What do we mean by a diagram? ………....................…………………….. 5 1.3 Perceptions and actions in diagrammatic reasoning…………………..…… 6 1.4 The problem…………………………………………………………..……. 13 1.5 Contributions………………………………………………………..……....14 1.6 Organization of the dissertation…………………………………………….16 2. A specification language..……………………………………………………....18 2.1 Vocabulary...…………………………………………………………..…....18 2.2 Specification language………………………………………………..…….20 2.3 Discussion……………………………………………………………..…....26 3. A constraint satisfaction framework…………………………………………....27 3.1 Overall algorithm………………...………………………………………....27 3.2 Modeling language……………………………………………………….....31 3.3 Mapping to a similar problem…………………………………………........32 3.4 Memory organization………………………………………….....................35 3.5 Computational complexity………………………………………….............37 3.6 An example…………………………………………………………….........39 3.7 Discussion……………….…………………………………………..............41 viii 4. A spatial search framework…………………………………………..................42 4.1 The overall idea...………………...…………………………………………43 4.2 Underlying representation……………………………….……………….....44 4.3 The core algorithm ……………………………………………………........44 4.4 Enhancing the efficiency of visual search …………………….....................51 4.5 Computational complexity………………………………………….............55 4.6 Discussion……………….………………………………………….............56 5. Applications………………………………………………………………..…...58 5.1 Entity re-identification……………………………………………………...58 5.2 Ambush analysis…………………………………………………………… 62 5.3 Euclidean geometry theorem proving……………………………………… 64 5.4 Discussion………………………………………………………………..… 68 6. Conclusions………………………………………………………………..……70 6.1 Evaluations………………………………………………………………….71 6.2 Contributions………………………………………………………………..73 6.3 Future research……………………………………………………………...76 Bibliography…….……………………………………………………………..……78 Index…….……………………………………………………………..……………81 ix LIST OF FIGURES 1.1 A general purpose diagrammatic reasoning architecture……………………… 2 1.2 Diagrammatic reasoning by an army commander…………………………….. 4 1.3 Examples of graphs understood by SKETCHY……………………………….. 7 1.4 An example of a deflected frame analysis by REDRAW……………………... 7 1.5 An example of a geometry theorem demonstrated by ARCHIMEDES………. 9 1.6 An example of a mathematical theorem proven by DIAMOND……………… 10 1.7 An example of ambush analysis by GeoRep……………………………………11 1.8 An example of route planning in urban scenario by Chandrasekaran, et. al’s diagrammatic reasoning architecture…………………………………... 12 2.1 The BehindCurve as a decision problem ……………………………………… 24 2.2 The BehindCurve as a function problem………………………………………. 24 2.3 The FurthestBehindCurve as an optimization problem……………………….. 25 3.1 Flow diagram of the spatial problem solver using constraint satisfaction…….. 30 3.2 Hierarchical problem classification in memory……………………………….. 36 3.3 Parse tree for a subproblem of the BehindCurve problem……………………... 40 4.1 An example of abstraction for efficient computation…………………………. 52 4.2 Solving the BehindCurve problem by spatial search………………………….. 54 5.1 Problem solving for entity re-identification…………………………………… 61 5.2 Ambush analysis………………………………………………………………. 63 5.3 Diagrammatic proof of Pythagoras