A Case-Based Approach to Diagrammatic Reasoning

Dale E. Fish and Robert McCartney Department of ComputerScience and Engineering University of Connecticut Storrs, CT 06269-3155 [email protected] robert @eng2.uconn.edu

Abstract diagrammatic reasoning (DR) but little work using case- Weare exploring the possibility of using case-based based reasoning (CBR) as an approach to DR. We think reasoning as an approachto diagrammaticreasoning. This this maybe a natural fit, reasoning with diagrams about paper describes a work in progress on a system that diagrams. This work explores the possible advantages of ’understands’a diagramof a problemsituation by finding using a store of case diagramsto interpret newdiagrams. correspondencesbetween it and similar diagramsin its case This paper describes a workin progress on a system that memory.The input to the system is a diagram of the ’understands’ a diagram of a problemsituation by finding problemsituation consisting of somenumber of simple correspondencesbetween it and similar diagrams in its case elementssuch as circles and lines. Thegoal of the systemis memory. The input to the system is a diagram of the to determine the significance of the diagram and its constituent elementsby recognizingthe structure of the problem situation consisting of some number of simple diagramand structures withinit. Theapproach is to use high elementssuch as circles and lines. The goal of the systemis level perception(recognition of instancesof categoriesand to determine the significance of the diagram and its relations) to build a representationof the problemsituation constituent elements by recognizing the structure of the (a diagram), and infer additional informationby finding diagram and structures within it. The approach is to use correspondencesbetween this representation and stored high level perception (recognition of instances of cases. Theoutput is the problemsituation diagramannotated categories and relations) to build a representation of the with descriptions of the elementsand relationships between problem situation (a diagram), and infer additional them and any inferences the system maybe able to make information by finding correspondences between this alongwith their justifications. representation and stored cases. The output is the problem situation diagram (or just problemdiagram) annotated with descriptions of the elements and relationships between Introduction them and any inferences the system may be able to make There are a numberof problem domains where information alongwith their justifications. is more easily and naturally represented using diagrams. In the next section, we state the problem we are While humans are adept at using diagrams, automated addressing with this workand follow that with a discussion reasoning systems typically use pre-distilled codified of the approach we are taking. The following section representations. This means that in domains where elaborates on the approach by providing an example. We diagrammatic representations are the norm, the diagrams then discuss someother work related to our approach, some must first be interpreted by humans, which (in extreme problems to be addressed and possible extensions to our cases) results in either simplified representations where work, and we conclude with a brief discussion of the determinations of relevancy and saliency have already been contribution we believe this workprovides. made, or representations that (attempt to) include all the information present in the original diagrams. The former reduces the system’s reasoning to a form of unification and Problem Statement the latter can be very difficult if not impossible given the Weare interested in looking at the utility of using diagrams density and complexity of information inherent even in we knowand understand to reason about new diagrams. In simple diagrams. An automated system that reasons general terms, the question we are dealing with is can directly from diagrams both precludes the necessity of diagrams be useful in understanding other diagrams. For spoon feeding the system interpreted representations and humansworking in domains where information is routinely retains the original information-rich diagrams. Using the represented diagrammatically, the answer seems to be an actual diagrams in reasoning is especially important when obvious yes. Architects represent plans with diagrams and there is limited domainknowledge (i.e. the first principles understand these representations because of their of a domainare unknown)or flexibility is required, both of experience in working with them. The military uses whichare characteristics of learning systems. diagrams to represent tactical scenarios, sports teams use There has been considerable interest in the area of diagrams to represent plays, and directors and actors use

6 diagrams to represent sets and stage movement. The terms of offensive plays with whichit is familiar. Figure 2 usefulness of diagrammatic representations in these and shows a diagram of a problem situation as it would be other domains is the efficiency with which they convey presented to the system. The diagramconsists of 11 circles information, and the efficiency with which the people representing wherethe players on offense line up relative involved understand the diagrams is due in large part to to one another for the given play. their experience in workingwith them. We would like to develop a CBRsystem that uses diagrams of situations as cases and reasons from these cases about new diagrammatically represented problem situations. Weare assumingthat a problemsituation can be represented using some number of simple pictorial elements such as basic geometric figures, and that the identity of these elements along with their location and orientation is given, sidestepping the issue of low level recognition. (Whatthis really meansis that the system is ! given instructions on howto draw the problemsituation as opposed to giving it a bit map.) What these elements represent at some basic level may also be knowngiven somedomain, but the roles the individual elements play in a particular diagram must be inferred by recognizing the structures and relationships inherent in the diagram. This seems to be a difficult problem. Take a simple relationship like grouping: if an element can be in at most Figure 1: Sample play diagram showing player alignment one group, the numberof possible groupings is given by and routes of receivers. the size of the powerset of the elements. Evenwith a small number of elements, a small number of relationships between elements and/or groups, and a small number of possible categories to assign to the elements, the problem space gets large in a hurry. Onerole of the cases then is to focus attention, sort of like saying look for somethinglike this around here. The structures discovered in the problem O O diagram may be mappedto the structures in the case or O cases that prompted their discovery along with the corresponding elements, allowing inferences to be formed 000 00 0 about the elementsin the problemdiagram (i.e. if this maps ! to that, thenthis is like that). Figure 2: A problemsituation. The domain we have chosen is the game of football which is a domain where diagrams are routinely used to Whatwe would like to do is take a diagram such as this represent plans called plays. These play diagrams showthe and makeuseful inferences to achieve somegoal. The goal initial locations of the players and their assignmentsduring maybe something as nebulous as ’understand the picture’ the play. Figure 1 shows an example of a play diagram. or somethingas specific as ’identify the primary receiver’. This domain can be quite complicated although the For now, our concern is to be able to makegood judgments diagrams themselves are very simple, using a small set of about similarity between the problem situation and cases symbols, mostly easily recognizable geometric shapes. The and find reasonable correspondences, or mappings, simplicity of these diagrams makes it a good domain for between the elements of two similar diagrams. Webelieve initial investigation (since we are more interested in that this is useful since missing information about an conceptual analysis of a diagram rather than low level element in the problem diagram maybe inferred from the recognition of complexelements within a diagram) while information knownabout its corresponding element in a the complexity inherent in the domain should provide similar case. The desired output of the system is the fertile ground for interesting extensions. Weare of course problem diagram annotated with labels identifying the very interested in provingclaims of generality by using this types of the player elements and the discovered structures approach on other domains. and relationships; the mappings are included as For the present we are ignoring the action part of the justifications for the inferences madeand to indicate which diagrams (the lines showing player movementduring the groups in the problemdiagram are inferred to share roles play) and concentrating on the initial positioning, or with which groups in the case diagram. In other words, the alignment, of the players. This is the problemsituation the system’sgoal is to flesh out the diagram. system is to understand. The perspective is that of the defense trying to recognize a novel offensive alignment in

7 concepts)and avoid the restricting specificity of superficial similarity by allowing near concepts to mapto one another. Approach Specific domainexperience is represented as a collection of cases in the case memorywhere cases are annotated diagrams. The conceptual network and case memoryare High Level Description discussedin moredetail later. We are taking a CBRapproach to DR. Wesee this as an An important aspect of the way our approach works is interesting extension of the CBRparadigm, where cases that the various types of processing are interleaved. This and problem situations are represented diagrammatically, means that all types of structure building (describing and as a natural approach to the tough DRproblem. The elements, identifying relations betweenthem, and mapping basic goal of this workis to develop a general approachfor structures in the problemdiagram to structures in the cases) automated diagram understanding. By ’general’ we do not go on simultaneously. There are no separate phases. The mean to preclude the need for domain knowledge, rationale behind this (mentionedin the problemstatement) especially since we are using CBR,but instead that the is that it is impractical and (we believe) unnecessary approachitself is not weddedto a particular domain. completely describe a situation in order to make useful In any CBRsystem, cases are used to encapsulate inferences about it. Better to take a quick look, see what specific information. This information maybe specific in types of things you find, look for more of the sorts of terms of a given domain, such as recipes in CHEF things you are successful at finding, and use the cases to (Hammond,1986) or in terms of more abstract goals, such help focus the attention on the problem diagram as case adaptation in DIAL(Leake et al., 1997). The extensionally (i.e. does the problem diagram have utilization of specific knowledge(experiences in human somethinglike this over in that area?). Anotheradvantage terms) is a strength of CBR.In a real sense, the ability to of this approachis scaling: in domainswith moreconcepts, reason from cases means that a system’s domain more complexrelations, and complicated diagrams, finding knowledgerequirement is simplified; the system does not a complete consistent description of the problem diagram need knowledge of first principles which are at best would be inefficient. This interleaving of recognition difficult to apprehend and at worst lacking consensus. The processes is an idea borrowed from Tabletop (French, other advantage of CBRis that it is naturally a machine 1995; Hofstadter, 1995) which is discussed briefly in the learning paradigm; new cases are acquired and reasoning section on related work. (hopefully) improvesas the likelihood of having cases that This interleaving is accomplishedby encoding each type more closely resemble the problem diagram increases. of recognition and structure building as a separate chunk, Understanding the problem diagram involves finding a or method, which is added to a code queue. These methods consistent set of reasonable structures. (We will use maycause other methods to be added to the queue, either ’structures’ as a generic term to refer to descriptions, because they were successful (in which case similar relationships, and mappings: basically any type of methods would be added) or because they require some additional information the system attaches to play preprocessing (in which case a different kind of method diagrams.) Consistent simply meansthat their are no two would be added). These methods are chosen to run based contradictory structures. For example, an element cannot on priority, wherepriority is a function of dynamicfactors, have two player type descriptions attached to it, or an most important of which are the activation levels of the element in the problem diagram cannot be mappedto more particular concepts involved. All recognition methods than one element in a particular case. Reasonableis more involve concepts and when instances of concepts are difficult. For our approach, reasonableness is a measureof recognized in the problem diagram, the corresponding the quality of the mappings between structures in the nodes in the conceptual network receive some activation problem diagram and structures in a case (i.e. how much which also spreads to neighboring nodes. Methods in the conceptual slippage is required to makea mappingfit). For code queue that recognize more activated concepts are example, mapping a group of two WIDE-RECEIVER chosen over methods whose corresponding nodes are less elements to a group of two WIDE-RECEIVERelements is activated. To begin, the code queue is primed with several better than mapping a group of two WIDE-RECEIVER methodsreflecting domainpredispositions. elements to a group of five LINEMANelements. The system first applies general domain knowledge to Our system uses three types of knowledge: general the problemdiagram in order to begin building descriptions knowledge, general domain knowledge, and specific of the elements of the diagram. This involves recognizing domain experience. The first two are represented in the elements in terms of the concepts they represent and system’s conceptual network which includes general recognizing relationships between elements or other concepts (e.g. spatial relationships such as BEHINDand structures. The problem diagram is annotated with the NEXT-TO)and domain concepts (e.g. player types such structures as they are recognized and the concept nodes in QUARTERBACKand WIDE-RECEIVER). A network the conceptual networkrepresenting the types of structures representation is used to encode associations among that have been recognizedare activated. This activation has concepts. These associations enable the system to focus two effects: the system tries to recognize more of the moreon structural similarity (structures involving related activated types of structures (by giving priority to related

8 methodsin the code queue), and cases containing instances contained. For example, the spatial relation concept NEXT- of the same(or similar) structures are activated¯ TO has a method for recognizing if two elements are This phase of the process does not actually finish since adjacent¯ Domain knowledge concepts always involve there maybe any number of possible structures, few of other concepts¯ For example, recognizing the whichwould actually be correct and or useful¯ Instead, the QUARTERBACKrequires recognizing the player element system begins to consider cases that share similarity with BEHINDthe CENTER,where BEHINDand CENTERare the problem diagram in terms of the descriptions and concept nodes with their own recognition methods structures as soon as any similarities are identified. The (BEHINDis a spatial relation and CENTERis a player type) system tries to recognize correspondences between cases ¯ If the system tries to identify the QUARTERBACK,it and the problem diagram. A correspondence is a structure scans the descriptions of the player objects to see if the that indicates a mappingbetween elements or structures¯ CENTERhas been identified, meaningthat a player element Each case that is considered will have its own set of description has been augmentedby the system identifying correspondences with the problem diagram and for each it as the CENTER.If the CENTERhas been recognized, then case, there will be a maximal consistent set of the methodfor recognizing the BEHINDrelationship is used correspondences where maximalis in terms of the quality to identify the QUARTERBACK.If the CENTERhas not of the mappingsand the extent of the structures involved¯ been identified, then the methodfor recognizing it is added Recognizing new elements of the problem diagram to the code queue along with another copy of the method brings new cases to the foreground and causes new for recognizing the QUARTERBACK.This is one way that structures to be built. Formingcorrespondences causes the concepts makeuse of other concepts. system to look for other elements or structures in order to Given a network sort of representation, with related form similar correspondences. concepts as neighbors, we find a spreading activation Cases are ranked on the basis of their maximalconsistent approach can be used for a number of purposes. As sets of correspondences with the problem diagram. These mentionedin the high level description, nodes are activated sets change as new structures and correspondences are by successful recognition methods¯ The links between formed which means cases come in and out of favor. At nodes provide a measure of the conceptual distance some point, the system becomes more stable; fewer new betweenconcepts, the shorter the distance the moresimilar structures are found and the ranking of the cases remains the concepts. Activation ’flows’ along these links so that the same. Whenthis stability reaches somethreshold, the near nodes receive somefraction of the activation level of system stops¯ The ’answer’ is the current description of the the source node. The amountof activation that spreads is a problem diagram and its correspondences with the highest function of the distance betweenthe nodes¯ For example, if rated case. a GROUPstructure of THREEelements is recognized, some activation is added to the GROUPconcept and THREE The Conceptual Network concept nodes, and this activation will spread to the TWO and FOURconcept nodes, and so on. Activation levels are The conceptual network consists of nodes which represent an indication of the importanceof specific concepts in the the ’center’ of particular concepts connectedto one another problem diagram in terms of what has been discovered up via directional links which represent the relationships to that point¯ Whenthe activation level of a node rises between concepts¯ There are two main types of links: the above some threshold, a recognition method for that ISA and HAS-MEMBERlinks which represent hierarchical concept is added to the code queue. In this way the class relationships, and LABELEDlinks which represent spreading activation causes recognition methodssimilar to relationships described by other concepts in the conceptual successful recognition methods to be added to the code network¯ For example, the TIGHT-ENDnode representing a queue. Since we are interested in similar things and not just specific class of player is connected to the ELIGIBLE- specific things, spreading activation is used so that similar RECEIVERnode representing a more general class of concepts are included. player by an ISA link, and there is a HAS-MEMBERlink The next purpose the conceptual network serves is that connecting the ELIGIBLE-RECEIVERnode to the TIGHT- of a kind of index into case memory. This also takes ENDnode; the nodes for the concepts ON-LEVI"and ON- advantageof the spreading activation. Each concept node is RIGHTare connected by a LABELEDlink where the label is connectedto occurrencesof itself in the individual cases in the OPPOSITEnode. In this way, a particular concept is case memory. Activating concept nodes results in represented by the node for that concept and to a lesser activating cases. The activation level of a particular extent its neighboringnodes. concept node is in a sense an indication of the relevancy of The conceptual network serves multiple purposes. The that concept to the problem diagram. Cases that involve first purpose it serves is providing the methods for activated concepts are relevant to the problemdiagram. recognizing structures in the problem diagram. Each The conceptual network also provides a basis for concept node contains a methodor methodsfor recognizing mappingsimilar concepts onto one another. Whenlooking an instance of the concept it represents. These methods for similar cases or forming correspondences between may be entirely self contained or may involve the situations, we do not want to restrict the indexing or recognition methods of other concepts. The recognition mapping to exact matches; this would put too much methods for general knowledge concepts are often self emphasison superficial similarity. Instead, we wouldlike in the problem diagram to elements and structures in the to allow for slippage, allowing a particular concept to case. The sameelement or structure cannot participate in ’remind’ the system of similar concepts and similar more than one correspondence, but for any element or concepts to mapto each other. Of course, everything else structure, there may be several reasonable mappings, so being equal, it seems logical to prefer correspondences there must be a means for preferring one correspondence involving moresimilar concepts over less similar concepts. over another. The quality of correspondencesare evaluated The conceptual distance between concepts provides a accordingto certain predispositions (e.g. preferring to map mechanismfor preferring correspondences between closer groups similar in numberand element composition). These matches. mappings work with the conceptual network as a focusing mechanismfor the system. The system looks for instances Case Memory of highly activated concepts and structures involving highly activated relational concepts in the problem Case memoryconsists of a numberof situations from the diagram. Concept nodes receive additional activation when problem domain. A case includes a diagram (location and their concepts participate in correspondencesbetween cases orientation of each player element) and whateveradditional and the problem diagram, so attention on the problem structures the system may have formed or discovered diagramis in a sense directed by its relations to cases. during the same process of understanding that the problem Cases are scored in order to makea determination as to diagram is subjected to. These descriptions are not assumed which cases are most relevant to the problem diagram. The to be complete because the system does not attempt to score of a case is a function of the activation levels of the discover all possible information or form all possible concepts in the case. Activated concepts that participate in structures when analyzing a situation. Figure 3 shows a structures within a case (as opposedto concepts that occur samplecase diagram of a passing play. The player elements in isolation) are weighted in order to reflect the system’s are annotated with labels for the types of players they preference for structural similarity versus superficial represent (football is a game of specialization). For similarity. example, the element that represents the QUARTERBACK is annotated with the label QB.These labels comefrom the corresponding concepts in the conceptual network. There A Sample Problem are also 4 group structures shownin the case diagram. These are the rectangles marked G1--G4. The primary Given a problem situation described by the diagram in receiver for this particular play is the SPLIT-END(the Figure 2, the system first attempts to recognize what element markedSE) and this is indicated by the bold circle. concepts are represented by the elements of the diagram. There wouldbe other structures as well, such as the spatial The rules of the game of football and certain relationships between the groups and between some standardizations regarding the way the gameis played are elements as well as additional information describing group used to identify the classes of players represented by the compositions. These are omitted from the figure for circles in the problem diagram by recognizing certain simplicity. spatial relationships betweenthe elements in the diagram. For example, recognizing which circle represents the QUARTERBACKinvolves identifying which circle stands in the relation BEHINDto the circle representing the CENTER,which in turn is recognized by identifying which circle stands in the relation BEHINDto the element representing the BALLwhich is recognized by its unique G3 shape, location, etc. in the diagram. As the system recognizes elements, it builds descriptions of them using concepts from the conceptual network. This in turn activates the corresponding concept nodes in the network Figure 3: Anannotated case diagram. which has the effect of activating cases in case memory containing occurrences of the activated concepts. The In considering a case, the system opens a window spreading activation in the conceptual network meansthat showing the problem diagram (upper half of the window) cases containing similar concepts will also be activated. and the particular case (lower half of the window). The At the same time, the system identifies higher level system chooses structures in the case and adds recognition structures in the problemdiagram. These structures consist methodsfor that type of structure to the code queue. These of groups of elements and relationships between elements methodsmay include an argumentfor a specific area of the and other structures. The system may try a number of problem diagram. For example, if there is a GROUP different structures, some of which maybe inconsistent structure in the lower right of the case (e.g. G4in Figure3), with one another. For example, in forming groups, the the system may add a method to the code queue for system is predisposed to favor groups containing similar recognizing a group in the lower right of the problem elements (i.e. elements sharing a superclass) and groups diagram. The system mapselements and structures it finds containing neighboring elements. Since elements are not

10 allowed to be in more than one group (unless one group is Figure 4 had not been recognized (and therefore proper subset of the other), including one grouping in the correspondence M5had not been found), the existence of description of the problem diagram may prevent the the unmappedgroup G4 in the case prompts the system to inclusion of another. The system tries to find a consistent look for a group of two elements of type SLOT-BACKand set of structures, favoring the set that has the moststructure WIDE-RECEIVER.Similarly, the correspondence marked involving the concepts with the most activation. M4 prompts the system to look for such a group on the As the system proceeds with building a description of opposite side of the formation by activating the the problemdiagram, it begins to consider cases, preferring DIAGONAL-SYMMEI’RYconcept which is a particular type cases with similar structures found in the problemdiagram. of mapping. The group G1 is then formed since it has the Considering cases means finding correspondences between right numberof elements, the types share a superclass with structures or elements in the case and the problemdiagram. the elements of group G4 in the case, and it is where it These processes are interleaved. This means that in should be. The correspondence M5can then be recognized. considering a certain case, if a number of good The system considers multiple cases simultaneously, correspondenceshave been identified but somestructure in concentrating on the most promising cases. Rating a case the case has no corresponding structure in the problem involves scoring the structures within the case and its diagram, the systemwill look for such a structure. The idea correspondences with the problem diagram. Structures is to avoid trying to completely describing the problem involving more concepts with more activation are scored diagramand instead let partial mappingswith similar cases higher than structures with fewer concepts having less indicate whatto look for. activation, and correspondences involving structures with Figures 4 and 5 are possible results of the system, higher scores are scored higher than correspondences showing correspondences between the problem diagram involving structures with lower scores or correspondences (the upper half of each figure) and two cases. For each betweenisolated elements. Also, concepts participating in the two cases, other correspondencesare possible but these structures and correspondences receive additional maybe preferred by the system for a variety of reasons. For activation. This helps bias the system toward structural example, in Figure 4 the correspondences marked M4and similarity as opposedto superficial similarity. M5 suggest a degree of diagonal symmetry between the diagrams. The system prefers this mappingover the mirror symmetry mapping (G1 to G3 and G4 to G4) because the G2 correspondences as showninvolve more similar groups in terms of the numberof elements.

G2 {)t:T:)®@1 @---1 G1 ® ®i Mf M4 i® ’ r G1 ! I’

MI M

5,®1 ! °--@--- I@ _ I Figure 5: Correspondencesbetween another case and the I_QIIC)( @©®®1I ®1 problem diagram. 0 The mapping in Figure 4 scores higher than the one in Figure 4: Correspondencesbetween a case and the problem Figure 5. All the groups mappedto each other in Figure 4 diagram. have the same numberand type of elements. Also there are five elements in the problemdiagram that are recognized as As mentioned,building descriptions and structures in the representing concepts that are subclasses of the F.LIGIBLE- problem diagram is interleaved with forming RECEIVERconcept. Thus the node for this concept is correspondences with similar cases. The correspondences highly activated whichcontributes to the high score for the help focus the system’s attention by suggesting where to case in Figure 4 since it contains five instances of concepts look and what to look for in the problem diagram. For with ISA relationships to ELIGIBLE-RECEIVER. example, if the grouping G1 in the problem diagram in Eventually the system begins to stabilize. This means

11 that fewer new descriptions, relationships, and similarities with our approach, especially the idea that correspondences are found, the maximalconsistent set of features discovered in the problem diagram are used to descriptions in the problemdiagram remains the same, and prompt the activation of methodsto look for certain other the ordering of the cases remains the same. Whenthis features, instead of trying to describe the diagram stabilization reaches somethreshold (or activity falls below completely. Animportant difference is that the search plans some threshold) the system is done. The ’answer’ is the are not diagrams. They are methods for finding more annotated diagram of the problem diagram and the features based on the features that have already been correspondences with the highest rated case. The identified. Using actual diagrams similar to the problem descriptions for the problem diagram may include diagrams to guide processing may make our approach inferences drawn from its correspondences with the best better suited to weaktheory domains and it may be more case. For example, the case in Figure 4 may include a general. But another important difference is that POLYAis description of the play as a PASS-PLAY(versus working, so we will have to wait to see if these claims are RUNNING-PLAY)so the problem diagram would likewise valid. be inferred to represent a PASS-PLAY.Similarly, the Anderson (Anderson & McCartney, 1996) describes element in the case in Figure 4 marked SE (SPLIT-END) efficient use of stored diagrams applicable to a numberof mayinclude in its description the designation PRIMARY- domains. A syntax and semantics of inter-diagrammatic RECEIVER, meaning the QUARTERBACKwill look to reasoning is developed along with operators that allow throw the pass first to him, so the element in the problem retrieval of cases using diagrammatic matching. Our diagram mapping to the SPLIT-END(also marked SE) approachis different in that retrieval is based on similarity inferred to be the PRIMARY-RECEIVERin the problem of concepts and structures identified in the diagrams(i.e. diagram. what the diagrams represent) as opposed to similarity of planar tessellations. Related Work COACH Tabletop COACH(Collins, 1989) is a CBRsystem that works in the football domain.The emphasisof the workis plan creation. Tabletop is a system developed by French (French, 1995) COACHgenerates new plays by recognizing the bug in a that models analogical reasoning. It is presented with a failed play and applying abstract strategies for modifying source and a target and attempts to identify analogous plans whichare indexed by generalized descriptions of plan structures betweenthem. Tabletop’s problemis a table set failures. Cases in COACHare primarily a starting point; (sometimeshaphazardly) for two; an object on one side planning from scratch is expensive and difficult in weak- the table is ’touched’ and Tabletop must decide what the theory domains. The differences between COACHand our analogousobject on the other side of the table wouldbe. It approach are the system goals (planning versus is included here mainly because we borrowed some recognition) and COACHdoes not use diagrams. important aspects of it for our approach, namely using a conceptual network to model associations and focus system resources on activated concepts, and the idea of Future Work interleaving building representations and forming correspondences.It is also included here because, although Weare in the process of implementinga system using the it is not purported to be a diagrammaticreasoning system, approachdescribed in this paper that applies this approach it could easily be one, and it is a motivationfor our work. to limited aspects of the football domain. We have Weoriginally thought of our work as an attempt to use implementedsmall test versions of the main componentsof CBRto do the sort of high-level perception that Tabletop the system but have not yet put the pieces together. The does, but it soon becameobvious that what we were doing conceptual network consists of a player type hierarchy and was DR. Tabletop’s knowledge is represented in its several relational concepts. A handful of the recognition conceptual network or ’slipnet’ and it makesno use of a methods are implemented and limited annotating of the case memory. problem diagram works. The case memoryconsists of only a few test cases with relatively little similarity between Case-Based Reasoning with Diagrams them. Weexpect that there are a numberof problemsthat will POLYA(McDougal & Hammond, 1995) is a CBRsystem have to be addressed if our approach is to work. First is that constructs proofs for high school geometry problems. howto choose initially whichcases to apply to the problem The diagram of the problem POLYAis to solve is used to diagram. The problem here is that all football play provide features which are used as indices to plans in case diagrams share substantial superficial similarity, and the memory.POLYA interleaves the execution of two types of main reason behind using cases for recognition was to plans: plans that extract more features from the diagram avoid the difficulty of trying to analyze the deeper structure (search plans) and plans that actually write the proof. This of the problem diagram without the focus cases provide. is a very interesting piece of work and shares a lot of

12 This problem will be exacerbated with a large case base. assignmentof a player in the case does not transfer well to One approach is to use abstractions of the top level its corresponding player in the problem diagram, which structures in the cases and retrieve cases based on what maymean that the rating of that correspondence should types of similar structures are found in the problem suffer. diagram. Another approach might be that it maynot matter The evaluation of the effectiveness of our approach will that muchwhich cases are initially applied if cases are require that we also test other domains. One interesting indexed by the specific structures they contain. For extension that would broaden the range of applicable example, a group may be found in response to some case domains would be to consider sequences of diagrams be activated; as the internal description of that group is representing movementas the problem situation. Cases fleshed out, a case or cases with a similarly described would also be sequences where each case represents an group maybe activated. episode. The problemwould be to use the cases to predict It will still be difficult to determineexactly howto prefer subsequent diagrams in the problem sequence and modify certain cases over others. There are competingfactors such predictions dynamically. as the number of structures mapped, the quality of the individual mappings,and inter-structure relationships (e.g. symmetry). Howwe do this will depend on actual results, Conclusions giving us a better idea of whichset of metrics are useful This paper describes an approach to diagrammatic and howthey should be weighted. reasoning that uses high level perception to recognize Another difficulty is knowing when to stop looking. Howdo you know when the best set of correspondences concepts and relationships in diagrams and make inferences by recognizing correspondences with similar you have at somepoint is as good as it is going to get, or diagrams. This work is largely unimplementedat present. that further recognizing and mapping will not yield any Evaluation of the effectiveness of our approachwill follow significant improvements. One way to handle this maybe an analysis of the performance of the working system. An to think of the approach as an anytime algorithm and the assessment of its utility will require application of our answer is the case-to-problem diagram mappings in order approachto a variety of domains. of ranking. This of course is inadequate if the goal is something specific like ’identify the primary receiver’ since the primary receiver elements in the various cases References being considered mayyet to mappedto any element in the problem diagram. The approach we have in mind is to have Anderson, M. & McCartney, R. (1996). Diagrammatic the system to quit whenactivity falls below somethreshold Reasoning and Cases. 13th National Conference on (e.g. when the frequency of finding new structures drops Artificial Intelligence, August, 1996, Portland, Oregon. off) and the rankings have stabilized. Whenwe get some AAAI. results we will be able to judge howeffective this approach is. Anderson, M. & McCartney, R. (1995). Inter- There is also the problemof context. Each case that is Diagrammatic Reasoning. 14th International Joint considered will involve different concepts or similar Conference on Artificial Intelligence, August, 1995, concepts to different degrees. For example,considering one Montreal, Canada. Morgan Kaufmann Publishers, Inc. case may prompt activation of the DIAGONAL- 878-884. SYMMETRYconcept while another may prompt activation of the MIRROR-SYMMETRYconcept. We would like to be A. F. C Association. (1995). Football CoachingStrategies, able to switch contexts without any adverse influence. One HumanKinetics. Champaign,Illinois approach might be for each case to have its own private concept activation levels. However,it seems that it would Chandrasekaran, B., Glasgow, J. & Narayanan, N. H. eds. still be desirable to be able to have cases respond in some (1995). Diagrammatic Reasoning: Cognitive and degree to global activation levels so that successes maybe Computational Perspectives, AAAI Press. Menlo Park, repeated. This is a very interesting problem. Note for California. examplethat this context switching can occur with a single case: the elements of a group in the problem diagram may Chandrasekaran, B., Narayanan, N. H. & Iwasaki, Y. mapdiagonally to the elements of the corresponding group (1993). Reasoning With Diagrammatic Representations-- in the case, while the rest of the diagrammaps straight on. A Report on the Spring Symposium.In AI Magazine, Vol. An interesting extension to our work would be to 14, pp. 49-56. consider the dynamic aspects represented in the case diagrams. Since real play diagrams(e.g. Figure 1) represent Collins, G. C. (1989). Plan Creation. In Inside Case-Based plans and imply movementof the elements, it would be Reasoning, eds. C. K. Riesbeck & R. C. Schank, Lawrence interesting to consider this movementin matchingcases to ErlbaumAssociates, Publishers. Hillsdale, N.J., pp. 249- the problem diagram. For example, differences in the 290. alignment of the players may mean that the route

13 Dreayer, B. (1994). Teach Me Sports--Football. General Theorem-Proving. In Diagrammatic Reasoning: Cognitive Publishing Group,Inc., Santa Monica,California. and Computational Perspectives, eds. B. Chandrasekaran, J. Glasgow & N. H. Narayanan, AAAIPress. Menlo Park, Estes, W. K. (1994). Classification and Cognition. Oxford California, pp. 691-709. University Press, NewYork, NewYork. Millikan, R. G. (1984). Language, Thought, and Other French, R. M. (1995). The Subtlety of Sameness--ATheory Biological Catggories. MIT Press, Cambridge, and Computer Model of Analogy-Making. MIT Press, Massachusetts. Cambridge, Massachusetts. Mitchell, M. (1993). Analogy-Making as Perception: A Gentnerl D. & Stevens, A. L. eds. (1983). Mental Models. ComputerModel. MITPress, Cambridge, Massachusetts. LawrenceErlbaum Associates, Inc. Hillsdale, NewJersey. Nahinsky, I. D. (1992). Episodic Componentsof Concept Goel, V. (1996). Sketches of Thought. MIT Press, Learning and Representation. In Percepts, Concepts and Cambridge, Massachusetts. Categories: The Representation and Processing of Information, ed. B. Burns, Elsevier Science Publishers B. Hammond,K. J. (1986). CHEF: A Model of Case-Based V. Amsterdam,The Netherlands, pp. 381-410. Planning. Proceedings of AAAI-86. AAAI Press/MIT Press. Novick, L. R. (1988). Analogical transfer: Processes and Individual Differences. In Analogical Reasoning: Hawkes,D. D. (1995). Football’s Best Offensive Playbook, Perspectives of Artificial Intelligence, Cognitive Science, HumanKinetics. Champaign,Illinois. and Philosophy, ed. D. H. Helman, Kluwer Academic Publishers. Dordrecht, The Netherlands, pp. 125-145. Hofstadter, D. (1995). Fluid Concepts & Creative Analogies--Computer Models of the Fundamental Mechanisms of Thought. BasicBooks, New York, New York.

Kolodner, J. (1993). Case-Based Reasoning. Morgan KaufmannPublishers, Inc., San Mateo, California.

Kolodner, J. L. (1984). Retrieval and Organizational Strategies in Conceptual Memory: A Computer Model. Lawrence Erlbaum Associates, Limited, Hillsdale, New Jersey.

Kolodner, J. L. & Riesbeck, C. K. eds. (1986). Experience, Memory, and Reasoning. Lawrence Erlbaum Associates, Limited. Hillsdale, NewJersey.

Leake, D. B., Kinley, A. & Wilson, D. (1997). Case-Based Similarity Assessment: Estimating Adaptability from Experience. Proceedings of the Fourteenth National Conference on Artificial Intelligence, July 1997, Providence, R.I. AAAPress/MIT Press. 674---679.

Lockhead, G. R. (1992). On Identifying Things: A Case for Context. In Percepts, Concepts and Categories: The Representation and Processing of Information, ed. B. Burns, Elsevier Science Publishers B. V. Amsterdam,The Netherlands, pp. 381-410.

McCartney, R. (1993). Episodic Cases and Real-time Performance in a Case-Based Planning System. Expert Systems with Applications, 6, pp. 9-22.

McDougal, T. F. & Hammond, K. J. (1995). Using Diagrammatic Features to Index Plans for Geometry