Articles Reasoning with Diagrammatic Representations A Report on the Spring Symposium
B. Chandrasekaran, N. Hari Narayanan, and Yumi Iwasaki
■ We report on the spring 1992 symposium on dia- debates about the reality and nature of grammatic representations in reasoning and mental images and visual representations problem solving sponsored by the American have also raged in psychology. Design theo- Association for Artificial Intelligence. The sympo- rists have always been interested in the role of sium brought together psychologists, computer sketches and diagrams as design aids. Howev- scientists, and philosophers to discuss a range of er, logicians have traditionally disdained dia- issues covering both externally represented dia- grams as merely heuristic aids to be discarded grams and mental images and both psychology- and AI-related issues. In this article, we develop a once the correct path to the real proof is framework for thinking about the issues that obtained. Finally, although AI researchers were the focus of the symposium as well as report flirted with diagrams in the early decades, on the discussions that took place. We anticipate especially in the work on geometric theorem that traditional symbolic representations will proving, they have ridden on the wave of the increasingly be combined with iconic representa- so-called discrete–symbol-processing view or tions in future AI research and technology and the logic view, and the representation and the that this symposium is simply the first of many use of diagrams have not been the center of that will be devoted to this topic. attention for the last couple of decades. he American Association for Artificial Intelligence (AAAI) held the Spring Diagrammatic Reasoning: TSymposium on Reasoning with Dia- A New Emphasis in AI grammatic Representations from 25–27 March 1992. The emphasis of this symposium Lately, there has been a ground swell of inter- was diagrammatic (or pictorial) representa- est in AI on working with different types of tions in problem solving and reasoning. The representations and seeing how perceptual issues were not about how the raw sensory and motor components of intelligence can be information in the visual modality is pro- integrated with the explicitly cognitive com- cessed to form percepts; this process is the ponents. Correspondingly, there has also subject of image-processing and perception been an interest in treating the external world theories. Rather, the issues related to repre- itself as a representation and in studying the sentations of diagrams and mental images multiplicity of types of representations that and their functions in problem solving. The are available in it for an agent working in the symposium attracted a large, diverse, and world. Figure 1 shows the emerging picture of multidisciplinary audience: Philosophers, these concerns in AI. Because visual modality cognitive psychologists, design theorists, logi- is one area that seems so rich in its participa- cians, and AI researchers participated in 2-1/2 tion in problem solving and reasoning, dia- days of presentations and intense discussions. grammatic representation has been one Philosophers have been interested in the subject of interest in this emerging milieu for nature of mental imagery for a long time, and research.
Copyright © 1993, AAAI. All rights reserved. 0738-4602-1993 / $2.00 SUMMER 1993 49 Articles
appear in the working notes of the sympo- sium,1 which contain sections on diagram understanding and processing spatial expres- sions in addition to sections corresponding to symposium sessions.
Internal versus External Diagrammatic Representations In discussing the role of diagrams in reason- ing, we find it useful to make the following distinction about where the representations reside: The external world is the world in all its detail and form as sensed in various modalities. Figure 1. Multiple Types of Representations External diagrammatic representations for a Situated Agent. are constructed by the agent in a The representations include not only the tradition- medium in the external world (paper, al conceptual ones but also iconic and motor repre- and so on) but are meant as representa- sentations. The external world can also be used as tions by the agent. an additional source of representations. Internal diagrams or images make up the (controversial) internal representa- tions that are posited to have some pic- torial properties. Organization Some of the questions that arise in this context are, Are there any continuities in the This symposium was organized by B. Chan- mechanisms of processing and use regarding drasekaran, Yumi Iwasaki, N. Hari Narayanan, these representations? What are their and Herbert Simon. More than 70 researchers common functional roles in problem solving from 9 countries attended. The symposium and reasoning? consisted of five moderated sessions (modera- tors are given in parentheses): Imagery and Mental Images and Their Status Inference (Nancy Nersessian, Princeton Uni- versity), Human Diagrammatic There has been a tradition in psychology and Reasoning—Analyses and Experimental Stud- philosophy that dismisses mental images as ies and Sketching (Irvin Katz, Educational epiphenomenal; that is, they do not causally We anticipate Testing Service), Logic and Visual Reasoning participate in reasoning or problem solving. that (Pat Hayes, Stanford University), Computa- Some even dismiss claims of mental imagery traditional tional and Cognitive Models of Diagrammatic with the assertion that people who think Representation and Reasoning (Brian Funt, they have mental images are just imagining symbolic Simon Fraser University), and Qualitative things. No one at the symposium denied the representa- Reasoning (Leo Joskowicz, IBM T. J. Watson phenomenal reality of mental images, but Labs). Each session was structured as a series there were discussions about the sense in tions will of presentations followed by moderator’s which they were pictures (P. Slezak, Universi- increasingly comments and open discussion. The sympo- ty of New South Wales, Australia). Obviously, sium concluded with a one-hour open discus- this point is the heart of the imagery debate. be combined sion. However, the discussion in the field about with iconic In what follows, we give an overview of the this issue has been stymied by a lack of con- field as we see it, interspersed with remarks sensus on what various terms mean. We hope representa- about the contributions from presentations the following discussion is helpful in clarify- tions in that were made. (The names of the ing the relevant issues. researchers are given in parentheses, as is Let us refer to whatever it is that people future their affiliation the first time a name is men- have when they say they have a mental AI research tioned.) These presentations constituted only image as the phenomenal image and denote it and a subset of the many excellent papers that by I. The agent can describe its content as were accepted. Space limitation precludes a propositions, but phenomenally, it is an technology discussion of all the papers here. The papers image for the agent. Let R refer to the pattern
50 AI MAGAZINE Articles of activation in the neural structure in long- as we discuss in the next section, some image- It is term memory that contained the information processing theories propose that the neural- that gave rise to I. Let P refer to the pattern of activation patterns corresponding to seeing possible activation in the neural structure when the have a systematic spatial array character. To that agent reports having I. The neural pattern P the extent that P shared the patterns with occurs in the part of the cognitive architec- those that occur in perception, there would scanning a ture that corresponds to current awareness or be a sense in which P would be picturelike. At mental deliberation. the symposium, a proposal was made (B. image is What is the appropriate way to talk about Chandrasekaran and N. H. Narayanan, The R? We can talk abstractly about the content of Ohio State University) about P and what fea- scanning R. We can certainly have a debate about tures it shared with the pattern that arises only in a which language type is the best to talk about during perception. or describe the contents of visual memory. Another dimension to how mental images metaphoric We can draw pictures to describe the content; arise is the degree to which typical mental sense. after all, we are talking about information images are the result of composition and con- that directly gave rise to the phenomenal struction operations at run time as opposed image. Logicists in AI have argued that predi- to the retrieval process of complete images. cate logic can also describe the content well, When people are asked to recall images of and there is no need to assign a privileged objects that they are familiar with (for exam- and unique role to images as a language for ple, the map of South America), they often describing this content. However, other logi- produce images in which different parts were cians (J. Barwise, Indiana University at moved and rotated to standard positions in Bloomington, and J. Etchemendy, Stanford) relation to each other (B. Tversky, Stanford). argue that there is something fundamentally Thus, support is given to the constructivist different about pictures as representational hypothesis. However, the issue of how the categories. Whatever the resolution of this elementary pieces are represented is left open debate, our point here is that the debate is (that is, how far down does the constructivist about the appropriate descriptive language. It hypothesis go? Are there any visual primitives makes no sense to ask if R is a picture or a set in representation?). of propositions: R is neural pattern. It does make sense to ask if there is a consistent interpretation of R’s content as an image, but The World, Diagrams, such an interpretation is not in opposition to and Mental Images: R having a consistent interpretation as a set of Some Distinctions propositions. In this sense, image versus We use the term visual information to refer to propositions is a false opposition about the information that humans can extract by contents of R. With advances in neuroscience, inspection from an image or from the world we might discover that the neural pattern R by directing visual attention to it. What can has a preferred interpretation as the content be extracted visually by inspection varies of a picture, but still talking about R as a pic- ture would be a category error. somewhat from person to person because of How about the quality for P of being pic- training and other personal factors; hence, an turelike? P is the neural activation that corre- exhaustive list cannot be given of what this sponds to the agent having a phenomenal information consists of, but shapes, certain image. All the points that we made about the simple spatial relations, color, texture, and so content of R are applicable to the content of on, are the kind of information we have in P as well, but we can also ask what it is about mind for this term. P that gives the agent the experience of the It is useful to make a number of distinc- phenomenal image. One hypothesis is that tions so that the issue of mental images and the activation pattern of P has commonalities their representation and use is not conflated in both the shape and the location of the acti- with the issue of diagrammatic reasoning in vation patterns with some part of the archi- general. tecture that is involved in seeing. Specifically, Seeing: Seeing is perceiving the three- if the architecture under discussion is the part dimensional (3D) reality of the visual that is involved in the final stages of percep- world. As we mentioned, most of this is tion, having the activation pattern P would in the realm of image processing, from correspond to the experience of image I. So sensory information in the retina to the far, there is nothing specifically picturelike construction of, say, 3D shape descrip- about P other than it gave rise to I. However, tions.
SPRING 1993 51 Articles
What kinds Problem solving by using vision on array) is being performed on a mental image. the world: Here, as the problem-solving (To suppose that the full range of image-pro- of process sets up subgoals that require cessing operations that are performed during representations visually obtainable information from the the seeing of objects in the world or external world, visual attention can be shifted to diagrams is also done on mental images is can support the relevant part of the world, and infor- truly to fall into the vortex of infinite the range of mation can be extracted. regress!) However, a repeatedly proposed hypothesis is that there is a visual buffer in Drawing: An external representation behaviors which the final stage of visual perception is constructed, often from memory, of associated resides, and this visual buffer is shared relevant aspects of some visual domain, between imaging and seeing. In Marr’s view, with the use possibly imaginary. There are interesting one can think of image processing as taking issues about how the pieces of the draw- of images place in layers. Each layer preserves some ing are conceived, related, annotated or aspects of the pictorial quality of the visual and otherwise modified, and so on. world, the lower layers corresponding to the diagrams? Using drawings for problem solving: results of early image-processing operations Using drawings for problem solving and the final layer corresponding to the involves many of the same processes as visual buffer whose elements are perceptually using vision on the world for this task interpreted elements of the scene. The buffer (scanning by attention-driven mecha- preserves the spatial relations between the nisms, using perception for extracting elements, each of which is already, in some needed information, and so on), but the sense, interpreted, not requiring low-level drawings are almost always much more visual processing. Only the elements of the simplified, though they preserve some of visual buffer are available to the agent for the visual information about the world access (in perception) or construction (in that is necessary for problem solving. imaging). Also, many drawings are abstract and are Similarly, some of the motor and perceptu- not meant as veridical visual representa- al mechanisms are shared between seeing the tions of the real world, but they are used world, seeing a drawing (which is a represen- to represent information about some tation of some possible world), and using an problem in a form that can visually be image during problem solving. The extraction extracted easily. of certain classes of visual predicates seems to Imaging: Imaging is making a mental be shared by all these processes. Scanning is image of some aspect of a real or imagi- also generally thought of as an activity that nary visual world. Here, some of the can be performed on the world, the drawing, issues are similar to drawing, but there and the mental image, although it is doubtful are open issues about how the images are that an agent has to engage the same muscu- generated and processed (for example, lar actions in scanning a mental image as what mechanisms are shared between he/she would in scanning an external image processing an external diagram and an of the world. It is possible that scanning a internal one). mental image is scanning only in a metaphoric sense, but instead, what really Using mental images for problem happens is the reconstruction of the image to solving: Some of the issues here are sim- correspond to the results of scanning. ilar to those for drawings in problem In the constructivist view, images are con- solving, but it is not clear what scanning structed in the visual buffer from pieces that really means here or what role percep- are set in a certain relation. The pieces them- tion plays. selves tend to be stereotypical in character, A point that repeatedly comes up in discus- resulting in relations between the elements of sions on imaging is the degree to which the the image that violate verisimilitude. In this hardware is shared between seeing and imag- sense, imaging is partly like drawing, with all ing. In the previous section, we talked about the errors and biases that are features of most the location and shape of activation patterns people’s drawings. The constructivist view being shared between mental images and per- still leaves open the issue of how the elemen- ception. It is unlikely that early visual pro- tary pieces of the image are stored. cessing (that is, to use David Marr’s [vision A persistent issue about images is the researcher and author of Vision] language, degree to which additional direct work is extraction of the primal sketch, 2-1/2–dimen- done on the images by visual modality–spe- sional sketch, and so on, from the retinal cific operations to yield further information.
52 AI MAGAZINE Articles
This issue can be illustrated by considering Simon, Carnegie Mellon University [CMU], What two examples, one owed to R. Lindsay (Uni- and P. Cheng, CMU), economic reasoning (H. versity of Michigan) and the other attributed Tabachneck and Simon, CMU), architecture properties of to Z. Pylyshyn (who was not present at the (J. G. Wickham, University of Toronto), cre- images symposium): ative and conceptual design (V. Goel, Univer- make Take one step north, one step east, and sity of California at Berkeley, and B. Faltings, one step south. How do you get back to Swiss Federal Institute of Technology), mathe- them so the starting point? matical proofs (Barwise; Etchemendy; S-J. Shin, University of Notre Dame; K. Stenning useful for Imagine a vat of blue paint, and imag- and J. Oberlander, University of Edinburgh; problem ine pouring yellow paint into it and stir- D. Barker-Plummer, Swarthmore College; and solving? ring it. What do you see happening to S. C. Bailin, CTA Inc.), biological reasoning the paint in the blue vat? (A. Kindfield, UC Berkeley), molecular scene To the extent that articulations of cognitive analysis (Glasgow and Papadias), geometric strategies that people consciously adopt for theorem proving (K. Koedinger, CMU), plan- solving a given problem can be relied on, it ning (Faltings and P. Pu, University of Con- seems that a common strategy for the first necticut), college physics (G. Novak, problem is to construct a mental map and University of Texas at Austin), and procedure extract from it, by inspection, the relation of instructions (B. Pedell and H. Kuwahara, the starting point to the ending point. It is Michigan Technological University). What also possible to directly use prior factual properties of images make them so useful for knowledge to conclude that the starting point problem solving? is to the west of the ending point. A delibera- In the following discussion, we use the tive syllogistic reasoning account, distinct word diagram to stand for both mental images from both the image-based strategy and the and externally drawn diagrams. (We already direct application of factual knowledge, is provided some arguments for the hypothesis theoretically possible but rarely reported. that the processes that are involved in However, in the paint example, a strong argu- extracting visual information from diagrams ment can be made that the conclusion that and images share many elements. At any rate, one sees the paint in the blue vat turning this hypothesis is a working hypothesis for green is not given by the image, but rather, it the rest of the discussion in this section.) The is retrieved from one’s store of factual knowl- following argument suggests why and under edge and imparted to the image. what conditions diagrammatic representa- tions might be helpful. Representational Issues Diagrams preserve or directly represent locality information. A number of visual What kinds of representations can support predicates are efficiently computed by the the range of behaviors associated with the use visual architecture from this information, for of images and diagrams? At the symposium, example, neighborhood relations, relative there were proposals based on perceptual size, and intersections. This ability makes cer- primitives (Chandrasekaran and Narayanan) tain types of inferences easy or relatively for mental images and on a distinction direct. (It should be emphasized that only between spatial and visual representations as some visual predicates are easily computed by the basis for computational imagery (J. Glas- the visual architecture. The visual system is gow and D. Papadias, Queens University, not good at making many inferences for Kingston, Ontario). Two-dimensional pixel which information is directly available in the representations and pixel-pattern-to-pixel- diagram. For example, given a large circle and pattern inference rules were presented as a smaller circle, the visual system can directly components of a deductive system that uses tell that one is smaller than the other, but only picturelike representations (G. Furnas, given two complicated shapes, where one of Bellcore). the shapes has a smaller area than the other, the visual architecture cannot compare them Diagrams and Images directly or easily without additional measure- in Problem Solving ments and calculation.) This ability to get some of the needed The symposium gave rich evidence that information visually explains the role of dia- images and diagrams are used extensively in grams in problems that are essentially spatial, reasoning and problem solving in all sorts of such as geometry problems. Even here, there domains: scientific discovery (Y. Qin and H. are interesting strategies by which the dia-
SPRING 1993 53 Articles
gram is additionally manipulated. Additional one can effectively use the visual representa- constructions are made on the diagram (Lind- tion for the original problem. say), which enable the detection of new All the assumptions in this scenario might visual information (or emergent properties, make it sound as if it is a rare event for such according to Koedinger) in the next cycles of mappings to be found, and it would only be inspection. Also, symbolic annotations can infrequently that a nonvisual problem might be made on the diagram, which enable a new be aided by such visual analogs. In fact, how- round of inferences to be made, not by the ever, for commonsense reasoning phenome- visual architecture but by use of information na, over a period of time, we learn a whole in the conceptual modality. The result of this repertoire of such analogs constructed by the inference can be represented in the diagram, culture. Think of how common mappings which can then enable the extraction of addi- from temporal phenomena to spatial phe- tional visual information. This highly inter- nomena are. We often represent time as a line laced sequence of extraction of visual and reason with lengths of lines when we information and symbolic inference making want to compare durations. is what gives this whole approach its power: In general, mapping to visual representa- Each modality obtains information that it is tions is a possible option for nonvisual prob- best suited for and then sets up additional lems whose properties can be mapped to information that makes it possible for the spatial properties that are easy to visually rec- other modality to arrive at additional infor- ognize. The phrase easily recognizable visually mation for which it is best suited (K. Myers is important here because many properties and K. Konolige, SRI International; are spatial but not easily recognizable visual- Narayanan and Chandrasekaran; and Barwise ly. Properties that are easily recognizable visu- and Etchemendy). Cognitive models for such ally include such things as larger and smaller, visual interactions were proposed and dis- longer and shorter, more to the right (left, cussed (E. Rogers, Georgia Institute of Tech- above, below), thinner and thicker, and nology). diagrams brighter and darker. In these types of proper- Now, the previous discussion explains the aid in the ties, what is important seems to be the role that diagrams play in problems that have notion of (total) ordering in the one or two organization an intrinsic spatial content. What about dimensions in which the properties of inter- problems that are not spatial? What explains of cognitive est vary. Another type of spatial property that the ubiquity of diagrams in such problems as is often used to represent nonvisual proper- activity. well? The key is the existence of mappings ties is containment, as in the case of Venn with the property shown in figure 2. diagrams. Spatial containment of the sort used in Venn diagrams is also easily recog- nized visually. However, containment does not necessarily impose a total ordering among represented objects. Diagrammatic representations play another important role (Novak, Koedinger, Simon, and co-workers). In this role, diagrams help in selecting methods to solve a problem, not Figure 2. Mapping from a Nonvisual so much in making immediate inferences. to a Visual Problem. That is, diagrams aid in the organization of R is a nonvisual representation from which it is cognitive activity. However, there are also limitations to the desired to compute a predicate P. VR is a visual rep- resentation of R such that PR and, hence, P can diagrammatic form of reasoning. Because dia- quickly be computed by the visual architecture. grams are concrete representations (that is, they are models in the sense of Johnson- Suppose that R is some (nonvisual) repre- Laird’s mental models), when inferences can sentation and that P is a predicate that we are be made, they are fast and direct. This con- interested in computing about this represen- creteness enables such representations to tation. Suppose a mapping is available such avoid some of the problems associated with that R goes to VR, a visual representation. the frame problem in inference (S. Huffman Suppose also that some visual predicate PR and J. Laird, University of Michigan). Howev- can be extracted from VR efficiently by the er, unless special techniques are employed, visual architecture and that P can be obtained they cannot be used as such to perform rea- from PR by the reverse of the mapping from R soning involving universal quantification. to VR. If these conditions are satisfied, then Disjunctive reasoning (especially when the
54 AI MAGAZINE Articles numbers of disjunctions are large) becomes (echoed by Lindsay). hard in a purely visual mode. These limita- As mentioned earlier, normative logic (that tions of a purely diagrammatic reasoning are is, logic as an approach for deriving sound why most nontrivial problems involve the conclusions) has long treated diagrams as integration of both visual and nonvisual mere heuristic aids in the organization of modes of reasoning (Barwise and thought and as objects that should not Etchemendy). appear in the proofs that are finally produced. Quite a bit of discussion was devoted to the A number of papers that were presented at use of sketches in the task of design (Goel and the symposium challenged this tradition in Faltings). Sketches are often intended to be logic. The general point that united all of vague in some aspects; that is, the person them was that it is possible to devise logical who draws one is not committed to all the systems that take diagrams seriously and that dimensions and relations as drawn. Several have their inference rules organized partly as speakers argued that this vagueness plays a diagrams with symbolic annotations. Some functional role: It helps the designer avoid interesting systems based on this approach overcommitment to those aspects of the were discussed or demonstrated at the sympo- design to which he/she is not yet ready to sium (Barwise and Etchemendy; Shin, Sten- make a precise commitment but, at the same ning, and Oberlander; and Barker-Plummer time, still take advantage of the visual mode and Bailin). for organizing problem-solving activity and inference making. In fact, what makes sketch- Concluding Remarks es especially useful is the fact that a sketch is not so much vague as it is something that The symposium was successful in many stands for a family of precise models. dimensions. It fostered many interdisci- what many The role of generic visual forms in facilitat- plinary interactions: Psychologists, philoso- AI researchers ing creative discoveries, demonstrated by phers, and computer scientists listened to experiments in which subjects constructed each other about problems of common inter- considered to mental images of such forms and then est. The discussions were intense. The sympo- be a explored various ways of interpreting them, sium revealed an emerging consensus on the given, was a topic of discussion (R. Finke, Texas need to expand AI’s representational reper- A&M University). A pilot study of cognitive toire to include diagrammatic representations researchers difficulty in display-based problem solving, in and integrate their use with traditional repre- in psychology which subjects solved Tower of Hanoi iso- sentations. Furthermore, it highlighted con- morphs by directly manipulating computer tributions that researchers in other disciplines considered displays of problem objects, was also present- can make to this endeavor. Interestingly, the to be ed (M. Lewis and J. Toth, University of Pitts- fact that researchers in different disciplines burgh). bring different assumptions to bear on their problematic Some of the discussion related to the role work was brought into sharp focus. of images and diagrams in qualitative reason- For example, what many AI researchers ing about the physical world. It was suggested considered to be a given (S. M. Kosslyn’s (Narayanan and Chandrasekaran) that much imagery theory), researchers in psychology of our commonsense knowledge about how considered to be problematic. A number of objects in the world behave under various ideas and systems were placed on the table for forces and collisions is actually in the form of use in diagrammatic representations by both abstract perceptual chunks that directed the humans and computers and for a wide variety reasoning activity. Representation of object- of problem-solving tasks. An electronic mail- configuration diagrams and predictive knowl- ing list was set up for continued discussions edge indexed by shapes, as well as visual and information exchange.2 events, were emphasized as central problems. Following the symposium, an informal This work contrasts with much of the current survey of its impact was conducted among work in qualitative physics, which empha- the participants. An interesting picture sizes symbolic and axiomatic reasoning in emerged from the responses. Most respon- this task. The use of visually based analog dents said that their main goals in attending simulations for predicting the behavior of liq- the symposium were to learn about cognitive uids to augment traditional symbolic meth- and computational research concerning the ods was also proposed (J. Decuyper, D. representation and use of diagrams, meet Keymeulen, and L. Steels, Free University, other researchers, and publicize their own Brussels). Using visual representation for ana- work, and they agreed that the meeting defi- logic simulation was, in fact, a general theme nitely facilitated these goals. Many were
SPRING 1993 55 Articles
pleasantly surprised by the diversity of the through an earlier draft of this article and work presented, but some lamented the lack making his customary friendly but thorough of specific applications and connectionist critiques. The final version has benefited models. Although the program devoted enormously from his critique, but we do not almost a third of the total time to discussion imagine that it resolves all his criticisms satis- periods, respondents felt that more time was factorily. needed. This comment is an indicator of the enthusiasm generated, and the mailing list is, Notes indeed, serving as a forum for continued dia- 1. The working notes were intended only for distri- logues. This enthusiasm foretells of a bright bution to the attendees. However, interested read- ers can contact Hari Narayanan at future for research in this area.
56 AI MAGAZINE