Appears in Diagrammatic Representation and Inference, Alan Blackwell, Kim Marriott and Atsushi Shimojima, Editors, Lecture Notes in Artificial Intelligence 2980, Berlin: Springer- Verlag, 2004, pp. 151-165. An Architecture for Problem Solving with Diagrams1 B. Chandrasekaran+, Unmesh Kurup+, Bonny Banerjee+, John R. Josephson+ and Robert Winkler++ +Dept of Computer & Information Science The Ohio State University Columbus, OH 43210, USA {Chandra|kurup|banerjee|jj}@cis.ohio-state.edu, [email protected] ++U. S. Army Research Lab 2800 Powder Mill Rd Adelphi, MD 20783 USA Abstract. In problem solving a goal/subgoal is either solved by generating needed information from current information, or further decomposed into addi- tional subgoals. In traditional problem solving, goals, knowledge, and problem states are all modeled as expressions composed of symbolic predicates, and in- formation generation is modeled as rule application based on matching of sym- bols. In problem solving with diagrams on the other hand, an additional means of generating information is available, viz., by visual perception on diagrams. A subgoal is solved opportunistically by whichever way of generating information is successful. Diagrams are especially effective because certain types of infor- mation that is entailed by given information is explicitly available – as emer- gent objects and emergent relations – for pickup by visual perception. We add to the traditional problem solving architecture a component for representing the diagram as a configuration of diagrammatic objects of three basic types, point, curve and region; a set of perceptual routines that recognize emergent objects and evaluate a set of generic spatial relations between objects; and a set of ac- tion routines that create or modify the diagram. We discuss how domain- specific capabilities can be added on top of the generic capabilities of the dia- gram system. The working of the architecture is illustrated by means of an ap- plication scenario. 1 Introduction Diagrams are ubiquitous as aids to human reasoning. They are potentially useful rep- resentations either when the domain is intrinsically spatial, as in the case of maps for route planning, or when there is a mapping from domain information to two- dimensional space that preserves certain properties, as in the case of pie charts, where 1 This paper was prepared through participation in the Advanced Decision Architectures Col- laborative Technology Alliance sponsored by the U.S. Army Research Laboratory under Co- operative Agreement DAAD19-01-2-0009. 151 the ratios of wedge angles are the same as ratios of certain quantities in the target do- main. Some of the ways diagrams help are not unique to diagrams as spatial repre- sentation. For example, like many external representations, diagrams are a memory aid. They also enable certain abstractions to stand out, by omitting details that are not relevant. Maps work this way – in contrast to an aerial photo of the same region of space, only certain objects are shown, and many of them are iconized rather than spa- tially veridical, such as a church icon to represent the location of a church, instead of the actual spatial extent of the church. But diagrams enable deeper use of their spatial- ity in comparison to textual forms external representation. Diagrams help by enabling perception to pick up information corresponding to what one might call emergent ob- jects and emergent relations. An emergent relation is exemplified in Fig.1. Simi- larly, if one were to draw a diagram consisting of two intersecting curves, visual per- ception can pick up a number of emergent objects, such as their intersection points and the curve segments. If the curves represent roads, a problem solver can make use of these new objects in, say, route planning. To the extent that recognition of such objects and relations is easy for the perceptual system of a particular agent, the agent can avoid complex sequences of reasoning or calculation that might otherwise be re- quired in deliberative problem solving to obtain the same information. A B C Fig. 1. The figure is generated from description, “A is to the left of B, and B is to the left of C,” and relation “A is to the left of C” is emergent, and can be picked up human perception. We distinguish the work in this paper from two other research streams in dia- grammatic reasoning. In one stream, the focus is on algorithms that operate on repre- sentations of diagrams to produce new diagrams that represent some information of interest. The authors of such algorithms usually contrast them with symbolic reason- ing, and propose that their algorithms exploit the structure of space in their data struc- tures, e.g., local operations on pixel arrays. In another stream, the focus is on the logi- cal aspects of diagrams, in particular on how proofs may be thought of as transformations of diagrams e.g., Venn diagrams as they are often used in informal set theory. The goal is to give diagrams and their manipulations the same logical status as sentences and their transformations in traditional logic. Our aim, however, is to un- derstand reasoning and problem solving activities that combine representation and in- ferences in both diagrammatic and symbolic modes so that different subproblems are solved by the mode that is best suited for them. Research in this area may make use of algorithms and ideas in the other streams. For example, some of the inferences from diagrams – we call them Perceptual Routines and discuss them later in the paper –- may make use of algorithms in the first stream, and some decisions to transform diagrams in certain ways may relate to symbolic reasoning. However, the focus in our research is an architecture that best deploys each of the modes. Our goal in this paper is to propose and develop a representation for diagrams that is as domain-independent as possible. The goal is to provide a diagrammatic represen- 152 tation that can help a computational problem solver, much as external diagrams help human problem solvers. We intend to present a certain intuition about a diagram as an abstract, but still spatial, notion. We also extend existing goal-directed problem- solving frameworks in such a way that diagrammatic representations as we conceive them can be used as a basic part of problem state representations, complementing the traditional predicate representations. Though we hope that much of what we say is consistent with important aspects of the human cognitive architecture, our motivation is not to propose a psychological theory, but rather to expand the options available to designers of AI and human-machine systems. 2 Problem Solving: Information Generation for Goals and Subgoals Problem solving as goals and subgoals. On a standard account in AI of problem solving (see [1] for the most evolved version), problem solving is a means-ends, or a subgoaling, process. The agent uses general and domain knowledge to decompose the goal into subgoals, subgoals into further subgoals, and so on, as needed. The task of problem solving then is to solve enough subgoals so that the main problem can be solved. Each important step in problem solving then is to solve a (sub)goal, or to de- compose it into additional subgoals. For our current purpose, it is useful to think of goals in information terms: achieving a goal (whether the main or a subgoal) corre- sponds to having information of a certain description. Solving a (sub)goal corre- sponds to inferring the information needed from background general and domain knowledge, information defining the problem, and information generated in earlier steps in problem solving. Thus, as each subgoal is solved, it may provide information that may help infer solutions to other subgoals, and finally the main goal. As infor- mation is generated, it results in change in the representation of problem state. Let us consider an information generation step in the above problem solving proc- ess, say I2 from I1 . For I1 to be support the generation of information I2 , I1 must en- tail I2 in some sense. Information generation usually involves representing informa- tion in some representation framework. In general, if I1 is represented by a representation R in a representation system R , not all the information entailed by I1 I will be explicit in R . The task then becomes one of transforming R into a repre- 1 I1 I1 sentation R such that R makes I explicit. However, as has been often remarked I2 I2 2 in the literature on diagrammatic reasoning (e.g., [2]), representation systems differ in the degree to which the representation of information I also explicitly represents other information entailed by I . In diagrammatic reasoning, two different representa- tion systems are employed: • Information generation by rule-based transformation of symbol strings. In most standard accounts, the goals, subgoals, and information are all represented in terms of symbolic predicates, such as the familiar On(A,B) in the blocks domain. In this representation, only I is explicit in RI . Entailed information is generated by apply- 153 ing rules to such expressions composed of predicates. For our purposes, AI repre- sentations such as frames, rules, etc., still belong in this category. The relevant point is that the representation is composed of predicates, and explicit rules deter- mine what information can be generated from what predicate expressions. • Information generation by perception from spatial representations. Diagrams (more generally visual representations) are a class of representation whose in- stances automatically make some of the entailed information available, provided suitable pickup capabilities are present. The medium constrains the representation in such a way that it is not possible to represent I without representing a range of entailed information. In human use of diagrams, visual perception can pick up some of the entailed information, as in the example in Fig.