Deriving Depictive from Descriptive Knowledge Representations Using Constraint Satisfaction Techniques

Deriving Depictive from Descriptive Knowledge Representations using Constraint Satisfaction Techniques

Cynthia A. Yendt-Lunn

A thesis submitted to the Depaflment of Computing und Information Science in conformity with the requirements for the degree of Muster of Science Queen's University, Kingston, Ontario. Canada May. 1997

Copyright O 1997, Cynthia A. Yendt-Lunn National Library Bibliothèque nationale I*(of Canada du Canada Acquisitions and Acquisitions et Bibliographic Services services bibliographiques 395 Wellington Street 395. rue Wellington Ottawa ON K1A ON4 ûüawa ON K1A ON4 Canada Canada Ywr t& Vme mfemme

Our Norie njfBrenCû

The author has granted a non- L'auteur a accordé une licence non exclusive licence allowing the exclusive permettant à la National Library of Canada to Bibliothèque nationale du Canada de reproduce, loan, distribute or sell reproduire, prêter, distribuer ou copies of this thesis in rnicrofom, vendre des copies de cette thèse sous paper or electronic formats. la forme de microfiche/nlm, de reproduction sur papier ou sur format électronique.

The author retains ownership of the L'auteur conserve la propriété du copyright in this thesis. Neither the droit d'auteur qui protège cette thèse. thesis nor substantid extracts fkom it Ni la thèse ni des extraits substantiels may be printed or othenivise de celle-ci ne doivent être imprimés reproduced without the author's ou autrement reproduits sans son permission. autorisation. Abstract

Evidence suggests the use of both descriptive and depictive representations in the human mind. Cognitive studies indicate the use of depictive representations in activities such as recognition, communication and problern solving. It is suggested that tasks such as spatial reasoning involve depictive representations and that in such tasks a depictive representation is more efficient, informative and intuitive to use than a description. Traditional computational knowledge representation schemes are Iinguistically based. The work herein describes a scheme for computationally deriving a depictive representation of knowledge from a description. In particular it proposes and demonstrates the appropriateness of using wnstraint satisfaction techniques to accomplish this task- The implementation focuses on the spatial domain. In the spatial domain, knowledge is descnbed using propositions involving relational information. For example, "Vancouver is west of Toronton. Such propositions represent constraints between objects in space. Constraint propagation techniques are used to propagate the information in the descriptive representation and wmpute a depictive representation by mapping the description to a corresponding depiction. The implemented depictive representation is model-based and facilitates model-based reasoning [Glasgow, 941. Acknowledg ments

A special thank you to Mn. C. Jenkins, who so many years ago, as a high school guidance wunsellor, saw potential and set me on this road.

Thank you to al1 the professors who taught and guided me, especially Janice Glasgow, Mike Jenkins, Pat Martin and Brian Butler - a special thank you for your comments and for being on my cornmittee. A gracious thank you to Janice Glasgow for being my supervisor - your guidance and patience are very much appreciated. And to lrene LaFleche for al1 your assistance and for ahnrays making us smile - you are very special and 1 know you've touched many students lives and made their stay a pleasant one. You will ahvays be fondly remembered by so many. And thank you to Ann Grbavec, my office mate who became a dear friend. Thank you al1 for your kindness and support.

Thank you to many others in and around the Department who made this a fun place to be. There are many friends who l'II never forget.

Thank you to my farnily, especially Phillip, Greg, Morn & Dad, for your inspira- tion, love, support, patience and understanding. Thank you also to the rest of our farnily and friends who were negleded many tintes over the years while I was studying. Thank you for understanding and still being there - now it's done and I'm coming to visit !

II Table of Contents

Abstract ...... i

Acknowiedgments ...... ii ... Table OF Contents...... ,...... -.- ...... III

.* List of Tables ...... vil

.*. List of Figures...... viii

Chapter 1

Introduction...... 1

1.1 Thesis Approach and Organization ...... 3

Chapter 2

Contributions from Psychology ...... 5

2.1 Cognitive Theories of Image Representation ...... 5

2.1 -1 Propositional Theories ...... 6

2.1.2 Propositional Representations...... 7

2.2 Depictive Theories...... 9

2.3 Multiple Representations ...... 10

2.4 Interaction between Descriptive & Depictive lmage Representations 1 1

2.5 Summary of Contributions From Psychology ...... 17

Chapter 3

Knowledge Representation ...... 20

iii 3.1 Descriptive Representations ...... 2 1

3.1 .1 Logic-Based Representations...... 21

3.1 -2Structural Representations ...... 23

3.1 .2.1 Semantic Nets ...... 23

3.1 .2.2 Frames & Scripts ...... 24

3.1.3 Descriptive Conclusion ...... 26

3.2 Depictive Representations ...... 26

3.2.1 Occupancy and Sym bolic Arrays ...... 27

3.2.2 Image Representation System: S-percepts & A-percepts ...... 28

3.2.3 Funt's Retina ...... 29

3.2.4 RTONs (Relative Topological and Orientation Node) ...... 30

3.2.5 Depictive Conclusion ...... 30

3.3 Constraint Networks ...... 31

3.4 Summary and Conclusions ...... 32

Chapter 4 Constraint Satisfaction Techniques...... 35

Representing Constraint Knowiedge...... 37

Constraint Propagation...... 38

Consistency Checking ...... 39

BacMracking ...... 40

Constraint Relaxation ...... 40

Surnmary ...... 41 Chapter 5 Contributions from Temporal and Spatial Reasoning Research ...... 42

5.1 Representation of Spatial and Temporal Relations...... 43

5.1 -1 Temporal Relations ...... 43

5.1.2 Spatial Relations ...... 43

5.1.3 Relational Knowledge Represented As Constraint Networks ...... 45

5.2 Reasoning and Problem-Solving...... 46

5.2.1 A Constraint Propagation and Reasoning Tool: The composition Table ...... 46

5.2.2 Constraint Propagation...... 48

5.2.2.1 Guesgen and HertzbergasTwo Constraint Propagation Approaches for Building a Constraint Network ...... 49

5.3 Summary. Conclusions and Related Work ...... 49

Chapter 6

Descriptions to Depictions: A Derivation Scheme ...... 51

6.1 The Objective ...... 52

6.2 The Descriptive Representation w ...... 53

6.3 The Depictive Representation W...... 54

6.4 The Derivation Process .F: w + W ...... 64

6.4.1 The Intemediate Constraint Network Representation .CN ..... 65

6.4.2.1 The lmplementation of Fw+ =,,, . w + CN ...... 72 6.4.3 Fm,*: CN j W ...... 78

6.5 Summary ...... 82

Chapter 7

Conclusions...... 84

7.1 Main Contributions ...... 64

7.1 -1 Sample Applications ...... 85

7.2 Summary and Future Work ...... 87

7.3 Conclusion...... 89

References ...... 9 1

Appendix A - lmplernentation of the Derivation Scheme (in Nial) ...... 106 List of Tables

Table 5.1 : Allen's Thirteen Possible Temporal Relationships ...... 44

Table 5.2: Guesgen and Hertzberg's Eight Possible Spatial Relationships .. -44

Table 5.3: Composition (Transitivity) Table ...... 47

Table 6.1 : Number of Sym bols and Corresponding Number of Scenarios.. ... 76

vii List of Figures

Figure 3.1 : Monotonicity and First Order Logic ...... 22

Figure 3.2. Semantic Net Representation ...... 24

Figure 3.3. Frarne Representation...... -25

Figure 3.4. Sym bolic Anay Representation...... -28

Figure 3.5: lnterpretation of an Image Symbol Structure in the Image Representation System ...... 29

Figure 3.6. Relative Topological and Orientation Node (RTON) ...... 30

Figure 3.7. Constraint Network Representation ...... 31

Figure 3.8a. One-Dimension Model-Based Representation...... 32

Figure 3.8b. One-Dimension Constraint Network Representation...... 32

Figure 3.9: Example of Corresponding Descriptive. Constraint Network and Depictive Representations ...... -34

Figure 4.1 : Constraint Satisfaction Problem (CSP) Component Definitions .... -35

Figure 4.2. Sample Constraint Satisfaction Problem Definition and Solution ... 36

Figure 4.3. A Sample Constraint Network Representation...... -38

Figure 4.4. Example of Constraint Propagation ...... -39

Figure 5.1 a: Constraint Network Representing Spatial Relationships...... 45

Figure 5.1 b: Alternative Constraint Network Representation ...... 45

Figure 6.1 : Overview of Objective for Deriving a Depictive Representation W frorn a Descriptive Representation w ...... 53

Figure 6.2. Components of a Depictive Representation...... 54

Figure 6.3. Descriptive Representation of a Simple Scene ...... 54 . . Figure 6.4. Definition of W ...... 55

viii Figure 6.5: Depictive Representation of a Simple Scene ...... -...... -. .-55

Figure 6.6: Propositions and Corresponding Depictive (Symbolic Anay) Representations ...... 56

Figure 6.7: Example Depictive (Symbolic Amy) Representations Corresponding to Guesgen and Hertzberg's Eight Possible Spatial Relationships ...... 57

Figure 6.8a: Example of Equivalence Classes...... -58 Figure 6.8b: Definition of Array Scenario Equivalence 4 - 5...... -58

Figure 6.9: Example of a Complete Set of Array Scenarios, Aw, representing three objects and one relation ...... 59

Figure 6.10: Example Descriptive and wrresponding Depictive Representation ...... -60

Figure 6.1 1: Blocks World Example #2: Determinate Example ...... 61

Figure 6.1 2: Blocks World Example #3: Indeteminate Example ...... 62

Figure 6.13: Definition of Aw given inwnsistent, determinate or indeteminate information...... -63

Figure 6.14: Summary of Rules Governing W representing w ...... -64

Figure 6.1 6: CN - w t/ pi E w, pi is represented in CN...... 65

Figure 6.1 7: Blocks World Example #4 ...... -66

Figure 6.18: Constraint Network Representation of Blocks World Example #4 ...... -66

Figure 6.19: Constraint Subnehnrork Representation of Blocks Worîd Example #4 ...... -67

Figure 6.20: Constraint Subnetworks Represented as Lists ...... 67

Figure 6.21 : CN = {cn, ... cn, ), where R = {ri ... r,) ...... *-68

Figure 6.22: Constraint Network representing Necessary Knowledge only ...... 69 Figure 6.23: Set of Constraint Networks representing Necessary Plus Possible Knowledge ...... 71

Figure 6.24: Relation-based sublists of propositions...... - - .. -...... -.. -73

Figure 6.25: Constraint Sublists and Generated Constraint Subnetworks ...... -73

Figure 6.26: Constraint Sublists and Generated Constraint Subnetworks ...... 74

Figure 6.27: Generating ALL scenarios and Removing lnconsistencies through Constraint Propagation ...... - ...... -75

Figure 6.28: Trace of Generate-One-CS given the symbols (A,B,C) ...... 75

Figure 6.29: Trace of Generate-One-CS given the sym bols (A, B, C) and the constraint (left,A, B)...... 76

Figure 6.30: Constraint Subnetworks Produced by Generate-Constraint-Subnetworks ...... -...... 77

Figure 6.31 : 1-Dimensional Array Representation ...... -. .. -...... 79

Figure 6.32: 2-Dim ensional Array Representation ...... -79

Figure 6.33: 3-Dimensional Array Representation ...... 80

Figure 6.34: Constraint Subnetworks and Derived Array Indices...... 81

Figure 6.35: Example Constraint Network Representation and Generated Depictive (Amy Scenarios) Representation ...... -83 Chapter 1 Introduction

The ability to derive depictive information from a description and vice versa is a beneficial hurnan attribute which provides alternative mediums for presenting and manipulating knowledge. Such variety enhances communication, understanding and problem solving. To enable such an ability computationally would result in a powerful computing tool which could be used to improve humanam- puter communication and problem solving.

Psychological research has provided evidence and suggestions of how humans store, manipulate, and translate between descriptive and depictive representations and supplies models from which computational implementations can be based. As well as yielding a powerful computing tool, computational implementations designed using evidence from psychological research in human imagery could be useful in discovering finer details of the human system. The work in this thesis is more wncemed with providing a computational tool for artficially intelligent problem solving than a mode1 for psychological research.

Evidence exists which suggests that the human mind possesses and uses multiple foms of knowiedge representations including descriptive (based on words) and depictive (based on images or abstract diagrarns) representations. Further experïments have indicated the existence and use of distinctive visual and spatial depictive representations [Farah et al. 19881. This variety of representations allows humans to effÏciently process relevant information. For example, tasks requiring recognition may require the generation of a detailed visual representation. Navigation and spatial planning tasks may involve the generation of a spatial representation. The spatial representation allows for more efficient processing by not including irrelevant visual information.

Mether humans posses multiple representational formats independently or whether there exists one underlying representation from which others are generated has been a topic of ongoing debate [Glasgow 1993, Anderson 1978, Block 198la, Finke 1989, Kosslyn and Pomerantz 1977, Pylyshyn 1981, Tye 19911. However, evidence does support the clairn that humans have an ability to derive differing representations from another [Glenberg & McDaniei 19921. For example, consider the act of reading - a goal of the novelist is to create images (a depictive representation) in the readers mind from a description. Conversely, hurnans also manage to eficiently derive linguistic or descriptive information from depictive representations. For example, if asked to describe a familiar object, such as your home or garden, relevant descriptive information is often elicited from a depictive imege. The translation between descriptive and depictive representations is a natural, and in several cases. unconscious occurrence involved in rnany everyday human activities.

Activities involving reasoning about time (such as a sequence of events) or space (such as the arrangement of objects in a scene) provide example domains which use and benefit from multiple representational formats. For example, in a trial a jury may benefit from diagrams demonstrating the possibility or impossibility of testimony describing a sequence of events or arrangement of objects. Reason- ing in the spatial and temporal domains can be described as reasoning about relations. In the temporal domain the relations are between events in time. In the spatial domain the relations are between objects. Temporal and spatial knowiedge is comrnonly communicated using relational knowledge, such as event A occurred before event BI or object C is to the left of object D. These relations represent constraints on the possible sequence of events or arrangement of objects in the respective domains. Hence, relational descriptions are constraint-based descriptions and constraint satisfaction techniques offer an obvious approach to computationally representing and reasoning about relational knowledge.

Constraint satisfaction is a terni used to represent a variety of programming techniques used in Artficial Intelligence (Al) and related disciplines. Constraint satisfaction techniques can be used in computationally deriving depidive from descriptive representations; this is the thesis of this paper.

1.1 Thesis Approach and Organization

To begin to explore techniques used in deriving depictive representations from descriptions, it is necessary to understand the difierent types of representations and their relevance. Chapter 2 presents cognitive theories of imagery including issues such as the underlying representations used to store and manipulate descriptive and depidive knowledge and processes involved with these representations. Experimental evidence supporting these theories is referenced. Of particular interest to this thesis are experiments demonstrating the usefulness and eficiency provided through different types of representations and the derivation of information from one form to another. Evidence suggesting the existence and relevance of different underlying representational formats (descriptive and depictive; spatial and visual) has been provided, suggesting that humans have the ability to communicate between different foms of representation and can derive one from another allowing for a variety of reasoning modes and providing efficient and intuitive problem solving. The scheme proposed and described in this thesis provides an implementation of such a derivation.

Having introduced relevant and influential evidence from psychology in chapter 2, chapter 3 presents knowledge representation techniques apropos to the derivation of alternative representations (using constraint satisfaction techniques) and to the implementation of a derivation scheme involving such ability.

3 Chapter 4 introduces constraint satisfaction techniques. Such techniques are used in the derivation scheme which computationally derives a depictive representation from a description. Constraint satisfaction techniques can also be used in reasoning and problem solving using these representations.

Chapter 5 presents closely related and infiuential works used in the development of the scheme presented in Chapter 6. These works are from research in the temporal and spatial domains. The derivation scherne proposed focuses on the spatial domain. The temporal and spatial domains are closely related. The spatial works ckdwere infiuenced by research in the temporal domain; such is the case with the derivation scheme deswibed in Chapter 6.

Chapter 6 describes a computational derivation scheme which generates a depictive representation from a description using constraint satisfaction techniques. The scheme conforms with a semantics for model-based reasoning as proposed by Glasgow and Malton (1994). In this semantics, the depictive representation is model-based and consists of a world containing a set of models each representing a different possible scenario consistent w*th the descriptive representation; the proposed derivation scheme produces such a world. In this world, the models are represented depictively using a symbolic array representation [Glasgow 1994aI. The derivation scheme also contributes to, and borrows from, the computational imagery scheme of Glasgow and Papadias (1992), which makes use of both descriptive and depictive representations. The development and implementation of the derivation scheme focuses on problems in the spatial domain; that is to Say, it focuses on a subset of problems which involve representations and techniques appropriate for spatial reasoning. This scheme can be extended and modified to accommodate other areas, such as the temporal domain.

Chapter 7 provides a summary and suggestions for future work. Chapter 2 Contributions from Psychology

This chapter presents psychological theories of the underlying image representations used by humans and experimental evidence supporting these theories. Of particular interest is the use of, and interaction between, descriptive and depictive representations of image information and their importance and usefulness in problem-solving; experimental evidence of such activities in the human imagery system are presented in this chapter.

2.1 Cognitive Theories of Image Representation There is an ongoing imagery debate' which is concemed with the underlying representations used in human imagery - whether they are propositional or analogical (descriptive or depictive). Propositionalists (descriptive theorists) argue that there is a single underlying representation which is propositional. In the descriptive view images are non-functional epiphenomena which are generated as biproducts of imagery from the underlying propositional representation [Pyly- shyn 19731. Depictive theofists argue that images are stored in an underlying format analogous to the depiction they represent; they are stored as depictive or semi-depictive entities. Depictionists argue that such representations exist independently from other possible representations, and that they play a functional role; they do not disclaim the existence of a propositional representation but rather daim the existence of an analogical representation with its own importance and

1. For detailed discussions regarding the imagery debate see Anderson (1978), BIock (1981a), Finice

(1989), Kossvn and Pomerantz (1 977), Pylyshyn (198 1), Tye (199 1) and Comprctational Intelligence. Vol- ume 9. Numk4. Novanber 1993. usefulness [Kosslyn 19801.

2.1.1 Propositional Theories One of the most notable propositionalists is Zenon Pylyshyn. Pylyshyn describes propositions as conceptual entities which are neither verbal nor visual; they are abstract entities and are not available to cunsciousness [Pylyshyn 19731. Propositions provide an interlingua for cornmunicating between mental words and mental pictures, and hence provide a basis from which either visual or verbal representations can be generated.

Pylyshyn supports the interlingua claim with evidence provided by Sperling [Sperling 19671. Sperling perfonned an experiment in which subjects were given a letter as a visual (depictive) stimuli and were asked to verbally identify it This experiment provides evidence that there is a lapse in time between the stages of visual recognition and verbal identification. Sperling purposes a model in which the information between these two stages is stored in a "recognition buffer memory." Pylyshyn implies that the propositional representation is what is used in the "recognition buffer rnernory" to comrnunicate between these two medium. Since both verbal and depictive information can be deflved from the propositional representations, Pylyshyn suggests that this must be the cornmon underlying representation from wliich others can be deflved.

Another point, which Pylyshyn uses to support propositional theories as opposed to depidive theories, is that propositions are interpreted entities; this supports the ability of humans to manage the multitudes of potential translations between verbal (words) and depictive representations and allows only relevant information to be translated and retrieved. This point was based on a misunder- standing that depictive theones claim a representation completely analogous to pictures or raw uninterpreted visual inputs. This is not the case; depictionists argue for picture-like representations, which capture depictive infomation but are not stored in a raw, uninterpreted format.

Pylyshyn also suggests that a propositional representation contains more information than a corresponding depictive representation. As an example Pyly- shyn describes the task of leaming paired associates. Given the pair boy-play, one rnight fom an image of a boy throwing a ball. Later Menthe stimulus word boy is presented, the same image is retrieved and the appropriate response, play, is retrieved despite alternative interpretations of the depictive representation, such as throw, ball, or catch. The propositional process involved between the recognition of the verbal stimuli and producing the correct response not only retrieves the depidion but also correctly interprets the infomation [Pylyshyn 19731.

The fact that humans can sometimes recall the existence of an object but not its location within a scene or its visual attributes is also provided as evidence of a non-depictive representation [Pylyshyn 19731. Kintch also provides support for a conceptual representation by supplying evidence that people tend to remernber the rneaning of sentences but not their verbatim structure [Kintch 19761.

2.1.2 Propositional Representations As Pylyshyn describes propositions, they are not a string of words but rather what is meant or asserted by words. A proposition is either true or false and may be asserted by any number of strings of words [Pylyshyn 19731. Kosslyn States that propositional representations consist of language-like entities which represent knowledge by describing or asserting facts (propositions) [Kosslyn 19801. Propositions represent facts (or beliefs) about the world which are believed to be true or false. Propositional knowledge representation schemes are usually gov- emed by a predicate calculus', although other logics have been proposed to expand and explore their potential as a more complete or accurate scherne for representing and functioning with human-like ability [Kosslyn 1980; Ringland and Duce 19881.

There are three cornmon characteristics of propositions: 1) Propositions are abstract, meaning that they are language independent and can represent class or instance knowiedge, 2) Propositions have a syntax (rules of formation), and 3) Each proposition has a truth value. Kosslyn further classifies propositional representations with two additional properties: each proposition wnsists of a relation and at least one argument. Propositions are not Iinguistic structures. As previously mentioned, they are often expressed using a predicate calculus' [Kosslyn, 19801. Propositional logics exist but they la& the ability for quantification. It is the quantifying ability of predicate calculus which makes it more wnvenient for knowledge representation. Using predicate calw lus, propositions can be concatenated using connectives such as AND and OR, and quantified using 3 (there exists) and V (for all). It is the syntax of the propositions which allows for logical inferences to be made [Anderson 19781.

Examples of propositional representations schemes include Quillian's (1968) associative network, Schank's (1972) conceptual dependency, and WinogradJs (1972) SHRDLU data base.

Propositional knowiedge representation schemes have proven to be very powerful in explaining human-like behaviour but they are not without problems. A cru- cial problem with propositional representations (at least from a computational point of view, and not to discredit the human brain) is that they can be quite cum- bersorne. To represent a simple problem or state in the world may require an undesirable, if not unmanageable, set of propositions. One of the main goals of

1. Aiso refwred to as predicate logic, frrst order logic (FOL), classical logic, lower PC, FO functional cafcuius. and general logic [Rmgland and Duce 19881. knowledge representation, at least frorn a computational point of view, is to find adequate, compact ways to conveniently represent knowledge. This is the main advantage provided in analogue (depictive) knowledge representation schemes.

2.2 Depictive Theories Depictive theonsts argue that depictive or semidepictive representations do exist in the human imagery system independently of an underlying propositional representation; descriptive representations are not rnerely epipkrncnmô and do play an important functional role. Descriptive representations wntain symbok, interpreted information whereas depictive representations contain geometric, uninterpreted information [Finke, Pinker, and Farah 19891.

Studies have demonstrated that there exist separate depidive visual and Spa- tial components used in imagery [Farah et al. 19881. Mishkin, Ungerleider, and Macko (1983) showed that there are two distinct cortical visual systems - the temporal cortex, involved in recognizing objects, and the parietal cortex, involved in detemining where objects are located. Supporting evidence has kenprovided by patients demonstrating difficulties in activities involving either the temporal or parietal mile not demonstrating diffiwlties in the activities of the other cortex [De Remi 1982; Bauer and Rubens 19851. The spatial component used in imagery preserves information about the relative object locations within an image; it is concemed with where objects are [Kosslyn 1980; Landau and Jackendoff 19931. The concern of the spatial component is to preserve spatial and topological relationships among objects within an image; it involves symbolic representations and ignores visual details. The visual component preserves infomation about what the scene, or meaningful parts of a scene, look like (e.g. size, shape, color, etc.). Farah (1988) suggests that hurnans typically use mental imagery for spatial reasoning. Different tasks require the use of the diFferent cornponents. For example, problems such as motion or route planning make use of the spatial component. This component provides an efficient means to reason about spatial relationships without regard for visual accuracy or detail.

2.3 Multiple Representations There have been many psychological experiments that suggest the existence of multiple underlying image representations including both desctiptive and depictive representations, each individually or jointly accessible and appropriate to different types of processing.

Paivio's dual-mde theory is a wellknown and frequently-referenced multiple representation theory [Paivio 19751. The dualcode theory proposes independent verbal (descriptive) and imaginal (depictive) representations involved in imagery. These verbal and imagina1 representations operate in parallel and are intercon- nected, allowing information to be translated from one representation into another as necessary [Paivio 19751. Evidence supporting such a theory and independent access to the individual representations has been provided by clinical evidence in which the imaginal (visual) or verbal memory of patients has been independently damaged hindenng the performance of the damaged memory mile leaving activities of the undamaged memory intact [Cohen 19831.

Intuitively, mental images can be broken into, or created from, subimages. The notion of a hierarchical organization of subimages has been proposed by many researchers including Pylyshyn [Pylyshyn 19731 and Marr and Nishihara [Mar and Nishihara 19781. Reed provides experimental evidence supporting a hierarchical organization in which complex images are hierarchimlly cornposed of subimages [Reed 19741. Such a concept also intuitively supports the notion of creative synthesis in which mental images are composed to make creative discoveries [Finke and Slayton 19881.

Kosslyn proposed a computational theory of imagery, which states that images are stored in a long-terni memory as propositions and that depictive (or quasi- depictive) representations are generated in working memory as needed [Kosslyn 19801. The knanrledge in long-terni memory is stored as subimages which are organized hierarchically according to a "part-of " hierarchy. Kosslyn also proposes three classes of imagery processes: those to generate, transform, and inspect images. images can be composed of, or broken into, subimages. Individual imagery processes and activities are invoked as necessary, depending to the demands of the task at hand. Johnson-Laird (1983) proposes a propositional, descriptive representation, and two depictive representations: a mental model, which is an abstract structural (spatial) representation, and a visual image, corresponding to a perceptual viewer-centered representation [Johnson-Laird 19831.

2.4 Interaction between Descriptive and Depictive Image Representations In the past few yean, interest in the interplay between linguistic (descriptive) and pictoriaVspatial (depictive) information has expanded in the field of cognitive science and publication of such research has been increasing [Glenberg and McDaniel 19921. This section references a fraction of these papers. Evidence which is consistent dualcode theories and supporting separate depictive visual and spatial wmponents is reported. Also reported are suggestions of how depictive representations (in humans) are created from, and probed by, linguistic and pictorial information. Related issues are presented such as: the limitations of linguistics wi-th regard to transfoming depictive information into descriptive, the structure of the mental organization of descriptive and depictive information, the importance of depictive representations in comprehension, the effect of information from perspectives on the creation of mental models (depictive representations), as well as other factors suggested to influence the creation of depidive representations, such as order of presentation of linguistic statements. Denis and Zimmer (1992) use results uf spatial pnming, distance cornparison, and mental scanning experiments as evidence that people are able to convert verbal descriptions into mental representations similar to those derived from visual experience. These experiments focus on comparable spatial information king derived from either a descriptive or depictive source; they provide evidence that such conversions are possible. These experiments are not concemed with the efficiency or techniques used in deriving the mental models. Talmy (1983) points out the limitations of language (English as well as other languages) and emphasizes the problem of accurately (and efficiently) communi- cating al1 the information found in a depiction through linguistically based representations. Talmy's paper focuses on a subset of language, the closedclass elements that convey spatial relations - the prepositions. In reference to this work, Glenberg and McDaniel (1992) summarize: 'the structure of language provides but a small set of prepositions to enwde the vast number of spatial relations that we can perceive." Glenberg and McDaniel further suggest that, due to the limitations of language. the integration of linguistic (descriptive) and spatial information (depictive) is not only a possibility but a requirement for effective communication." [Glenberg and McDaniel 1992, p. 4581. Such work suggests the importance of depictions; depictions can be used to efficiently and intuitively represent information which is dficult, if not impossible, to represent linguistically. Conversely, language also poses a problem when using a linguistic representation to construct a depictive representation - different interpretations of linguistic representations can lead to multiple potential depictive representations [Johnson- Laird 19831. A partial solution to the complexity imposed by the limitations of linguistics is suggested by Glenberg and McDaniel, who use an example provided by Landau and Jackendoff [Landau and Jackendoff 19911. The idea is to extend the language by providing new words to more efficiently express mat usually requires complex descriptions. A simple example is provided by Glenberg and McDaniel conceming the new word œnbove to describe something that is both centered and above something else. Such extensions could ease implementation. There is much written about the limitations of language and expressability. The interested reader is referred to [Jackendoff and Landau 1991, 19931 and [Lang, Cartensen, and Simmons 19911. Glenberg and Kruley (1992) point out that pictures facilitate comprehension and explain that "the effect of a picture may be to expand the functional capacity of working memory thereby facilitating comprehension"; this statement suggests that depictive representations in the mind are more efficient, taking up less space than a comparable descriptive representation and containing more information in an equ ivalent space. Waddill and McDaniel(1992) examine the effects of detail pictures (similar to a visual image) versus relational depictions (abstract spatial representations, e.g. maps). Their results suggest that, depending on the subject's comprehension ability, relational queues enhance recall of pictured (depictive) relational information and, regardless of comprehension skill, detail pictures enhance recall of targeted details. This information supports separate detail (visual) and relational (spatial) mental depictions of whicb access to the relational representation may require more processing than that of the detail representation. Basically this work irnplies that the relational representation is more complex than the detail representation. Previous evidence suggesting a more wmplex relational system than object (visual) system is reported in Pezdek, Roman, and Sobolik (1986).Their results indicate separate item and location mernories for words (relational representations involving labels) but not for objects. Previous research has found a linear relation between distance and scanning times for spatial configurations in depictive representations; this is known as the time-distance relation. Several researchers have used the time-distance relation to examine the connection between descriptive and depictive representations. It has been suggested that both depidive and descriptive representations are created synonymously, regardless of the format of the initia! stimulus. The time-distance relation is used in experiments by Denis and Cocude (1992). These experiments involve the creation of mental maps (depictive representations) of environrnents from descriptive information. Denis and Cowde use the tirnedistance relation to measure the quality of the mental models created from poorly- and well-struckired descriptive representations. These experiments demonstrate the importance of presenting well-stnictured descriptions as opposed to poorly stnictured or randomly presented descriptive prepositions. Although multiple presentation of poorly presented information improved results, providing well-stnictured descriptions is more efficient and yields a more correct mental model. Denis and Cocude write: "the structure of descriptions can effect the intrinsic stmdure of images of described objects and hence the mental operations performed subsequently on these images. Denis and Cocude also report that "image wherence and resolution improves even after the verbal description is accurately recalled." [Denis and Cocude 1992, p. 4971. Many papers report the importance of the presentation order of descriptions used as a basis for creating depictive representations. Foos (1980) Wtes that the construction of cognitive mapst from descriptive information is best when there is a complete rather than a partial ordering - when new information 'fits' into the existing model and can be immediately integrated. Ehrlich and Johnson-Laird (1982) describe the importance of presenting referentially continuous descriptions, ordered such that each sentence or proposition includes previously presented referents - "Subjects try to integrate each inwming sentence into a single coherent mental modelu [Ehrlich and Johnson-Laird 1982, p. 2961. They report

1. Co-@ive maps - another sponym for depictive representation / spatial representation / mental mode1 that providing such a presentation yields the creation of 'more correct diagrams'. Ehrlich and Johnson-Laird further wfite that "those sentences which cannot be immediately integrated are represented in a propositional fom [Ehrlich and Johnson-Laird 1982, p. 2961. Other experiments suggesting the creation of both propositional and depictive representations from spatial descriptions are reported by Mani and Johnson-Laird (1982). Mani and Johnson-Laird describe the creation of "propositional representations that are relatively hard to remember but correspond closely to the sentences in the description, and mental models that are relatively hard to remember but analogous to spatial arrays and accordingly poor in linguistic detail.^ [Mani and Johnson-Laird, 1982, p. 1811. Another series of experiments explored the effects of presenting linguistic information in the form of route andor survey descriptions. Route and survey descriptions provide infomation from two different perspectives. Survey descriptions provide information in geographic ternis, objects can be described in reference to landmarks, and North, South, West, East directions for example. Information from survey descriptions can be discontinuous. Route descriptions provide continuous infomation in terms of point-to-point directions, from landmark to landmark for example. Pemg and Kintsch (1985) provide subjects with a route or survey description. They wnclude that 'free recall' (recalling verbatirn detail) depends on the construction of a propositional textbase during comprehension, while inferences required a 'situational model', either in the form of a mental map or procedural representation. It wuld be shown that the fom of situation model depended both on the representation invited by the text", (route or survey), "and by the subject biases." [Perrig and Kintsch 1985, p. 5031. Taylor and Tversky (1990) provided maps as initial information from which subjects created mental models of fictitious environments. Subjects were later required to write descriptions or draw maps of these environments. Taylor and Tvenky reported that the order of mentioning and drawing landmarks indimted a similar organization of the descriptive and depictive representations in memory and that "about half the descriptions took either a route or survey perspective, but the other half mixed perspectives, indicating that perspective is not inherent in the representation but is instead imposed at the time of descn'ption." [Taylor and Tversky 19901. Another paper by Taylor and Tvenky (1992a) fmses on the mental organization of corresponding depictive and descriptive representations. Given information in the fom of maps, subjects were later to draw or describe the map environment. Results indicate that organization is quite similar between descriptive and depictive representations, and revealed a hierarchical structure based on spatial and functional features of the environments and on conventions for sequencing landmarks raylor and Tversky 1992a, p. 4831. Taylor and Tversky (1992b) provided either a map of the environment, or a route or survey description as information from which readers created mental models and reported that 'Readers apparently form the same spatial mental models capturing the spatial relations between landmarks from both survey and route descriptions, and from maps." paylor and Tversky 1992b, p. 2611. In testing both verbatim and inference questions, Taylor and Tversky reported that subjects were faster and more accurate to verbatim than to inference questions and provided further support for the creation of both descriptive and depictive mental representations from either depictive or descriptive information. Other experiments, involving different notions of perspective, were performed by Franklin, Tversky, and Coon (1992) and Bryant, Tversky, and Franklin (1992). Franklin, Tversky, and Coon (1 992) conclude that subjects form mental models corresponding to a single perspective when given infonation from the perspective of one character but when given the same infonation form the perspective of multiple characters the subjects seemed to adopt a neutral perspective rather than approaching eacb perspective separately. Subjects appeared to make separate mental models for separate places. Bryant, Tversky, and Franklin (1992) presented environments described from one of two perspectives; an intemal perspective of an observer within a scene and having surrounding objects, and an extemal perspective of an observer outside a scene al1 objects king in front. Reaction times to identw objects in the inter- na1 example refleded perceptual and biological asymmetry biases which were demonstrated by faster responses to abovelbelow questions, then fronthack, and then rÏght1leR The extemal example did not yield such time differences, as al1 objects were in front of the viewer. Mile it might be beneficial for a computational model to provide both intemal and extemal perspectives, unlike in the hurnan system, biological asyrnmetry biases should not affect cornputer processing tirne.

2.5 Summary of Contributions From Psychology Although the imagery debate is ongoing, techniques and investigations in neuropsychology are helping to resolve the confiict [Kosslyn 1993; Farah 19881. Strong evidenœ that both descriptive and depictive representations are involved in imagery is increasing. The evidence also supports distinct visual and spatial components involved in imagery. It is also apparent that different tasks involve the individual representations in varying degrees. Hence, it seems wise that a computational imagery scheme incorporate descriptive representations and both visual and spatial (relational) depictive representations or some method to derive these basic representations. Such a variety of representations would be consistent with evidence from the human model and would provide alternative representations for efficiently processing information as a partiwlar task demands.

Section 2.4 describes work from psychology conceming the creation, interaction, and use of multiple mental representations (descriptive and depictive). It cm be surnmarized as follows: The works reported in this section support dual code theories including separate visual and spatial cornponents [Mani and Johnson- Laird 1982; Pemg and Kintsch 1985; Taylor and Tvewky 1990, 1992~1,1992b; Waddill and McDaniel 1992; Pezdek, Roman, and Sobolik 19861. Experirnents designed to interrogate the individual representations suggest that both descriptive and depictive representations are mstructed regardless of the presentation format of the stimulating information [Taylor and Tversky 1992a, 1992b; Penig and Kintsch 1985; Mani and Johnson-Laird 19821. Taylor and Tversky report a similar organization within descriptive and depictive representations which is hierarchical in structure and based on spatial and functional features paylor and Tversky 1990, 1992al. Evidence that abstract depictive representations (spatial representations) are more cornplex than visual representations is provided by Waddill and McDanieI (1992) and Pezdek, Roman, and Sobolik (1986), who suggest separate item and location memones for spatial representations. Evidence that access to propositional representations is faster and less cornplex than accessing de pictive representations (especially spatial) is provided as wel l raylor and Tversky i990; Perrig and Kintsch 19851.

Taylor and Tversky (1990) also report that information presented from either a route or survey description, or from a map, result in the creation of the same Spa- tial mental models. Conceming single perspective versus multiple perspective information, subjects appear to create models adopting either the single perspective or a neutral perspective if information is presented from multiple perspectives [Franklin, Tversky, and Coon 19921. Bryant, Tversky, and Franklin (1992) report biological asymmetry biases when querying subjects about information presented from an intemal perspective (of an observer within a scene) but not when the information was presented from an extemal perspective. Such biases need not be inherent in computational models.

Issues conceming the limitations of linguistics are presented in Talmy (1983), Glenberg and McDaniel (1992), Johnson-Laird (1983), Landau and Jackendoff (1993), and Lang, Cartensen, and Simmons (1991 ). Suggestions regarding the presentation of linguistics are included in Denis and Cocude (1992), Foos (1980), and Ehrlich and Johnson-Laird (1982). In summary, these authors report that the creation of mental representations is facilitated by presenting well-stnictured descriptive (linguistic) information; that is when successive statements contain previous referents, and when new information 'fits' into the existing model.

The evidence from psychology has given a guideline as to what a cornputational model of imagery should incorporate and how to mode1 it. Given evidence supporting the existence of both descri-ptive and depidive representations, and separate depictive visual and spatial components, it is desirable to devise a means of computationally providing such representations.

Psychological evidence has demonstrated the importance and relevance of different representations and their combined or separate uses in various tasks. Providing a variety of representations for computational implementations wuid enhance the development of intuitive problem-solving techniques and improve effectiveness and eficiency. Knowiedge representation techniques for cornputer implementations are dis- wssed in the ensuing chapter. Chapter 3 Knowledge Representation

Psychological evidence has suggested the existence of a variety of underlying knowledge representations used by humans, including propositional (descriptive, linguistically-based) and depictive (or semidepictive, analogue) representations with separate visual and spatial components. A number of computational kncb-4edge representation techniques have been devised. Of particular interest to this thesis are descriptive, propositionally-based and depictive representations. Propositional representations include logic-based approaches. Semantic nets, frames and scripts are well-known and popular knowledge representation techniques which offer stnictured methods for representing propositions. Section 3.1 introduces descriptive representations including logics and structural representations. Section 3.2 introduces depictive representations. Depictive representations include detailed visual (depictonal) and abstract symbolic (diagrammatic) representations. Chandrasekaran and Narayanan's Image Representation System, Papadias and Glasgow's Occupancy Array, and Funt's 'Retina' are designed to represent knowledge depictorially. Papadias and Glasgow's Symbolic Amy and Hemandez1s RTONS (Relative Topological and Orientation Node) provide methods for representing knowledge abstractly. The Image Representation System can also be used to represent knowledge in an abstract diagrammatic manner. Diagrammatic representations offer a more efficient and intuitive means for representing visual knowledge to be used in certain tasks, such as spatial reasoning, where visual details are not necessary. Constraint networks are a representation of particular interest to this thesis and are described in Section 3.3. The task of deriving one representation from another is viewed as a constraint satisfaction problem. By representing knowledge in a wnstraint network, constraint satisfaction techniques can easily be applied. In the problem of deriving depictive representations from descriptive representations, constraint networks provide a natural and intuitive representation and are applied as an intemediate representation. 3.1 Descriptive Representations Descriptive representations involve representing knowledge through linguistically-based descriptions. Such is the case of logical representations. Logic-based representations are dedarative and represent static knowledge. Reasoning with logic representations nomally involves formal logical proof proœdures. Although suitable for some applications, in others, logic-based representations can be wmbersome and can lead to non-intuitive representations and reasoning strategies. Semantic nets and frames are examples of structural representations which offer a more compact method for representing large amounts of knowledge. Knowledge is hierarchically organized in groups and subgroups of relevant information. Such representations allaw reasoning through inheritance. The representation of dynamic information is provided through frames and scripts.

3.1.1 Logic-Based Representations

There exists a vatiety of logics which are of interest in representing knowledge. Most of these logics have foundations in First Order Logic (FOL) - also known as Classical logic, Predicate Calwlus (PC), Lower PC, First Order functional calculus, and general logic [Ringland and Duce 19881. In logic representations, knowledge is represented descriptively through statements. The statements in FOL are prepositions which are either true or false.

FOL includes connectives such as A (and), v (or), (not), and + (implies), and the quantifiers V (for all) and 3 (there exists). FOL provides a precisely- defined and wellunderstood method for representing and reasoning about knowledge. It is particularly suited to problems involving fomal domains; that is, problems in which knowledge can be easily expressed in true or false statements. Reasoning in logic is deductive. However, many real world problems involve non- fomal domains and the extent and eficiency of deductive methods for real world problem solving is questionable [McDemott 19871. Monotonicity is a characteristic of traditional logic which has inspired much work in the development of alternative logics for dealing with real world knowledge representation. Monotonicity assumes cornplete representative information: given a set of statements, the addition of a statement must be consistent with the initial set. The notion of updating information (changing and retracting statements) does not exist in monotonic systems.

Given:V x bird(x) + flies(x) bird(ostrich) the new information, fiies(ostrich) cannot be added.

Figure 3.1 : Monotonocity and First Order Logic

Clearly, real world knowledge is not complete and the property of monotonicity is not good for representing real world knowledge. Another unrealistic characteristic of traditional logics pertaining to representing real world knowledge is that of absolute tnith. Many modal logics have been devised to incorporate varying degrees of truth or certainty. Non-monotonie logics attempt to deal with incomplete or uncertain information, common sense, default values, changing conditions, and other staternents or assertions subject to retraction or revision [Thro 19911. Cirwmscription, default logic, autoepisternic logic, and closed-world systems are examples of nonmonotonic log ics. Modal logics are concemed wi-th the notions of necessity and possibility. Statements can be classified as true or impossible, or they can have a probability association. In modal logics, the operators O and O are used to denote necessity and possibility respectively. Modal logics also include the notion of strict implication denoted by =. In traditional logic the notion of causal information does not exist. For example, the implication (pigs lack wings) + (the earth is round) is valid even though the hostatements are not related. In modal logic, A => B is valid only if it is impossible that A be true and B false. Temporal logics, dynamic logics, and the logics of knowiedge and belief are related to modal logic [Shapiro 19901. Temporal logic deals with reasoning about time. Approaches to temporal logic include situation calculus, event calculus, and modal temporal logicl. Reasoning about time is important in many problems. For example, problems where reasoning about actions and effects is necessary, and natural language understanding problems, where time has an influence on interpretation [Bundy 1WO]. Approaches to temporal reasoning include linear programming, space-time maps, point algebra, end point representation, and interval algebra pro19911. Logic-based representations are particularly suited to formal, welldefined dornains and provide for well-understood pmof-theoretic reasoning strategies. However, many real-world problems do not provide a welldefined domain and representing knowledge using fomal logic can be cumbersome and awkward. Furthermore, many real-world problems are not intuitively solved through logi- cally-based proof-theoretic reasoning. Adding structure, for example through the use of hierarchies, semantic nets, and frames can provide a more efficient and comprehendible way of storing world knowledge descriptively, allowing for default values and inherent information, and providing an alternative reasoning strategy.

3.1.2 Structural Representations

Structural representations attempt to represent knowiedge in some type of structure usually based on what is believed to be used in humans. Well known examples are semantic nets, frames, and scripts. As well as adding structure, frarnes and scripts provide for the addition of procedural information. Procedural attachment provides a means for representing dynamic information.

3.1.2.1 Semantic Nets lnitially proposed by Quillian as a model of human associative memory, sernantic nets support a large scale organization of knowiedge emphasizing the multiple associations of individual concepts. Knowledge is represented as a net-

1. For information about Artif~cialIntelligence Techniques (specific howledge representation and constraint satisfaction techniques) and example implementations see Bundy (1990). work of linked nodes where each node represents a concept, object, or entity, and labelled, direded links represent the relationships between the adjoining nodes.

parts / I \

right of

Figure 3.2: Semantic Net Representation

Nodes and arcs may be added at any time. Semantic networks have a hierarchical structure providing reasoning through inhei-itance. Subclasses and instances of classes typically inherit attributes. Complex entities can be easily decomposed into their parts so that attention can be focused on relevant information.

3.1.2.2 Frames 8 Scripts Frames are data structures composed of dots containing values. These values are commonly stereotypical features or default values. Each frame represents an object, entity, concept, or parts thereof. Frames can be organized hierarchically, providing a wnvenient method for sharing data through inheritanœ and reasoning through inheritance. A frame system can be viewed as a semantic net containing classes and instances of objects with slot values linked hierarchically by associationsfrelations such as 'has parts' and 'is a'. Frames provide a less wmplicated knowledge structure than semantic nets. Complex concepts can be represented by a single frame as opposed to a subnetwork of simple concepts in a semantic net. Frames focus more on knowiedge structure than relationships. dot value

parts: lamp has part 1 1 table I 1

object:

right of bal1

Figure 3.3: Frame Representation

Procedur.attachment allows slots to contain program text which updates the frame base as desired when executed. For example, frame slots can contain procedures such as what to do should something be deleted or added (Le. if the table is removed, the location of the lamp and bal1 should change to reflect that they are no longer supported by the table). Scripts are similar to frames except that they are used to represent stereotypical information of events or actions rather than objects. Scripts contain information regarding a typical sequence of events involved in an activity or action; for example, the those which take place during a restaurant visit, or what happens to objects when their supporting object is taken away. Like frames, scripts may contain default values and procedural attachments. Scripts are used to manage dynam ic information. In case-based reasoning, scripts can be used to hold previous cases of experience to be used for reasoning about similar situations in the future. It has been suggested that such a technique is used by humans. 3.1.3 Descriptive Conclusion

Descrieptiverepresentations are particularly appropriate for tasks involving linguistically-indexed recall. For example, natural language understanding. How- ever, there are many tasks for which this is not the main wncem, such as spatial planning and visual recognition. Knowledge for such tasks is more appropriately presented using depictive representations. As previously stated, both formats are more or less appropriate for differing tasks. Some tasks, such as requesting directions, or describing an object or place, may require both types of representation. For example, requesting directions may first require a depictive representation of a map to be inspected, then the appropriate information to be communicated via a description. Being able to communicate computationally between descriptions and depictions is a useful, and desirable, human-like ability.

3.2 Depictive Representations

Depictive representations are stnicturally analogous to the reality they represent. They include detailed visual (depictorial) and abstract symbolic (diagrammatic) representations. Reasoning with depictions is achieved through the inspection and transformation of the representation in ways that are analogous to visually inspecting and physically transfoming objeds in the real world.' Depic- tive representations allow inferences to be made through inspection and manipulation as opposed to formal logic [Denis 19911. There is a growing body of evidence supporting the suggestion that much of human problem-solving is achieved through this inspect and manipulate type of reasoning. Application areas that have been investigated include qualitative physics [Forbus 19831, mechanistic devices [de Kleer and Brown 1988; Weld and de Kleer 19901, and spatial and temporal reasoning [Glasgow and Malton 19941. Many approaches have been proposed which model, andior provide a computational basis for, a human-like imagery system wnsisting of both depictorial and abstract diagrammatic representations. These approaches include the occupancy

1. See .\lental Images and tlieir Transfinnations [Shepard and Cooper 1 982 1. and symbolic array representations proposed by Papadias and Glasgow (1991) and the Image Representation System [Chandrasekaran and Narayanan 19901. Funt (1980) describes a depictiveldiagrammatic reasoning systern which makes use of a retinal image representation. Many other specialized depictiveldiagrammatic representations have been proposed for specifically representing spatial knowiedge, induding RTONs (Relative Topological and Orientation Node) [Herniindez 19941.

3.2.1 Occupancy and Symbolic Arrays

Papadias and Glasgow (1991) propose a method for computational imagery which includes separate visual and spatial representational components. Both the visual and spatial components are depictive in nature. Occupancy arrays are used to represent the visual components in a bit-map like manner. The visual representation includes information such as shape, relative distance, and relative size. Symbolic arrays are used to represent spatial information abstractly. Syrnbolic arrays provide discrete sym bolic representations which depict entities in a scene and the spatial relationships between them. Analog (dynamic) information can be represented by a sequential series of symbolic array representations. Knowledge is stored hierarchically in symbolic arrays where low level representations can be very detailed and picture-like. Higher level representations ignore detail and provide abstract representations which can be more efficient and appropriate as detennined by the task at hand. Symbolic arrays can be constnicted using descriptive information, for example in the fom of propositions such as on(ball, table), on(lamp, table), leR(bal1, lamp). Objects can be represented using more than one cell if necessary, for example, the table in Figure 3.4. An array or set of arrays representing a particular scene is referred to as a mode1 of that scene. Indeteminacy in a description leads to an increase of potential models. Symbolic arrays provide for intuitive problem-solving in domains such as spatial reasoning. Figure 3.4: Symbolic Array Representation

representing: on(bal1, table), on(lamp, table), lefi(bal1, hmp)

3.2.2 lmage Representatian System:

Chandrasekaran and Narayanan (1990) propose an lmage Recognition Sys- tem (IRS) which wnsists of visual representations and visual modality-specific operations for interpreting these representations. The visual representations are referred to as lmage Symbol Structures (ISSs). The interpretation of lSSs gives rise to mental irnagery. The ISS represents objects only in ternis of visual attributes (for example shape, color and texture). Non-visual conceptual knowledge (such as weight) associated with the objects is available in the IRS but is not part of the ISS. In naming objects, the ISS invokes the conceptual knowledge associated with the objects. Chandrasekaran and Narayanan (1991) describe the visual representation as consisting of imaginal and symbolic components. The imaginal component is represented as a two dimensional pixel array and is implemented as a bitmap. Detailed spatial information is included in the imaginal component. The symbolic component contains abstract symbolic descriptions (S-percepts) based on shape primitives such as circles, rectangles, etc. (see Figure 3.5). An ISS representing a scene is composed of S-percepts in a hierarchiml manner where the higher levels represent the scene abstractly and the lower levels becorne more and more refined and picture-like. S-percepts contain only visual aspects of an object or its parts and are represented as parameterized primitives, for example representing an objects shape and color. S-percepts also contain descriptions of the spatial relations among the primitives. Each S-percept has a corresponding analogic percept (A-percept) which has pictorial proparties and represents the image that results from the interpretation of the corresponding S-percept.

1. bal1 (circle, -5,(on 3, left 2))

Figure 3.5: Interpretation of an Image Symbol Structure in the Image Representation Systern Chandrasekaran and Narayanan [IWO; 19911

3.2.3 Funt's Retina

Funt (1980) describes a computational depictonal representation he calls the retina. Funt's retina is actually a mechanism for detecting depidorial or diagrammatic features. The retina is similar to the human retina in that it is able to fixate on objeds or locations with resolution decreasing from its centre to its periphery. Funt's retina is composed of receptors, each covering a separate segment of the depiction or diagram. The receptors recognize the color of a region. The receptors are arranged in wncentric circles or rays emanating from a central point. Each receptor can communicate with its immediate neighbors and the retinal supervisor. Funt suggests that parallel processing techniques can be used to implement the retina where each receptor is associated with a single proces- sor. Unlike arrays and image symbol structures, the retinal interpretation is not hierarchical in nature. The abstractness of the representation is determined by the depiction being processeâ. Different abstractions require different depictions for the retina to inspect. 3.2.4 RTONs (Relative Topological and Orientation Node)

Hemandez (1994) describes a diagrammatic representation for spatial knowledge. It combines topological and relational information and is wmposed of RTONs (Relative Topological and Orientation Node). Each object in a scene is represented by a separate RTON. Orientation information is represented using a uniform cirwlar neighboring structure. Topological information is stored by divid- ing the sections of the orientation representation into subsections, each representing a particular relation (Figure 3.6).

back

front

Figure 3.6: Relative Topolog ical and Orientation Node (RTON)

Hemandez (1994) also cites many alternative depictiveldiagrammatic representations that have been proposed for the spatial domain including Adomi (1983), Habel (1989). Khenkhar (1989), Haugeland (1987), Lindsay (1988), Mohnhaupt (1987), Sander (1991 ), and Waltz and Boggess (1979).

3.2.5 Depictive Conclusion Depictive representations provide an efficient and intuitive alternative to descriptively representing depictive knowiedge. However, it may not be desirable to store knowledge in both formats, or it may be more efficient to store depictive information desaiptively and derive depictive representations as necessary. The fact that constraint satisfaction techniques cm be used to derive depictive from descriptive representations is the thesis of this paper. The ensuing chapter presents constraint satisfaction techniques, but before this, one final knowledge representation section describes a specialized knowiedge representation for representing, reasoning, and problem-solving using constraint knowledge.

3.3 Constraint Networks Constraint networks offer a diagrammatical representation for constraint satisfaction problems (see ensuing chapter for more details). Constraint networks are a sernantic network type representation but focus only on constraint-based knowledge. Such a representation offers a simple means for representing constraint knowledge and provides a basis for the implementation and design of constraint satisfaction algorithms for reasoning and problem-solving. Similar to semantic networks, a constraint network can be represented as a direded graph. In the constraint graph, nodes represent entities or objects, and arcs represent the con-

Figure 3.7: 1 Constraint Network Representation 1 Represents the mnstrainfs iefi(ball,larnp), on(bal1, table), and on(1arnp. table).

Constraint networks can be used to eficiently represent the potential multiple models (symbolic arrays) resulting from incomplete or indeterminate information. For example, given the information leff(bal1, lamp) and leff(ba1l. box), at least two possible models exist:

[ bal1 1 larnp 1 box 1 1 bal1 1 box 1 lamp

1 Figure 3.8a: One Dimension Model-Based Representation

The same information can be represented by the wnstraint network:

Figure 3.8b: One Dimension Constraint Network Representation

It is easy to see how the number of possible models can quickly multiply in the presence of indeterminate and incomplete infomation. Constraint networks can be used to represent the sarne information in a more efficient fashion. Array models can be extracted from the constraint network representation when required. Constraint networks can also be used for reasoning. One might ask, why do we need models? The answer is that we want a model which is structurally analogous to the reality it represents. Constraint networks do not offer such an analogous representation. However, they do offer a wnvenient and efficient means for implementing a wnstraint satisfaction solution to the problem of deriving a depictive (symbolic array) representation from a descriptive (atom ic formula) representation. Constraint satisfaction techniques are more thoroughly presented in the following chapter.

3.4 Summary and Conclusions This Chapter introduces historical approaches to knowledge representation focusing on approaches of particular relevance to this thesis. Section 3.1 introduced descriptive representations including many branches of logic and structural representations. Fint order logic (FOL) provides a foundation for many of the alternative deswiptively (propositionally) based formalisms. For this reason the derivation scheme developed and purposed in chapter 6 uses a descriptive representation based on FOL wtiich involves simple atom ic fomulas. Section 3.2 presented approaches to representing knowledge depictively. Depictive representations include detailed visual (depictorial) and abstract symbolic (diagrammatic) representations. Abstract diagrammatic representations can be more efficient and intuitive to use than detailed depictonal representations for tasks such as spatial and temporal reasoning. The goal of the derivation scheme which is proposed in Chapter 6 is to derive a dep ictive representation from a description. The im plementation of this scheme focuses on a spatial domain. The description used is propositionally-based and describes the spatial configuration of objects within a scene. The depiction derived is a diagrammatic representation in the fom of symbolic arrays. The scheme derives a world of models representing al1 possible scenarios consistent with the input description. The process of deriving a depictive representation from a descriptive representation cornputationally is not straight-forward. The meaning of the description must be interpreted and a corresponding depidive representation computed. In creating a depictive representation, the descriptive representation constrains the potential depictive representation. The problem is a constraint satisfaction problem and, accordingly, constraint satisfaction techniques can be used in finding a solution. Constraint-networks provide an appropriate and necessary intemediate representation for this task. The derivation scheme proposed in Chapter 6 involves representing the input descriptive representation as a constraint network, using a constraint network and constraint satisfaction techniques to solve the derivation problem, and finally producing a depictive modal-based world representation consisting of symbolic amy models. DESCRIPTIVE REPRESENTATION 111111111~~~~~~11~- Left(bal1, lamp) Left(bal1, box) (simple atomic formulas)

REPRESENTATION (modei-based worid representation consisting of syrnbolic amy models conesponding to atternative valid 1 bal1 1 box 1 lamp 1 scenarios)

Figure 3.9: Example of corresponding descriptive, constraint network and depicitve representations Chapter 4 Constraint Satisfaction Techniques

This chapter presents a bnef overview of constraint satisfaction and constraint satisfaction techniques relevant to works cited in ensuing chapters and the derivation scheme proposed in chapter 6. Constraint satisfaction is a terni used to represent a variety of programming techniques used in Artficial Intelligence (Al) and related disciplines. A major use of constraint satisfaction is for solving the class of discrete combinatorial problems: for example, scene labelling and scene matching in Al and vision research, spatial planning problems, data-base consistency maintenance, circuit analysis, planning experiments in molewlar biology, temporal reasoning, diagnosis, query- answering, Graph lsomorphism detection, the Graph Colouring problem, many puzzles such as crypto-arithmetic, scheduling and cutting-stock problems in operations research, VLSI (Very Large Scala Integration) routing and layout in hard- ware design, and representation derivations in Al. Discrete combinatorial problems are said to be NPcomplete, which means that there exists no general and efficient algorithm for solving them WanHenten- ryck 19891. Constraint satisfaction methods are used to improve efficiency and solvability of such problems. A constraint satisfaction problem (CSP) is fomially defined as having a set of variables, a set of domains from which the variables take values, and a set of constraints over relations between the variables WanHentenryck 1989; Liu and Ku 19921. For a more fonal definition of CSP components see Figure 4.1 .

The standard Constraint Satisfaction Problem (CSP) has three cornponents:

1) a finite set of n constraint variables (entities or objects) v = (~1,V2, ..., v,) 2) a set of rn dornains from whidi the variables take values D = {dvl, dv2, ...( dvm) 3) a set of p constraints over relations between the variables c = {CI, a..... 5) Figure 4.1 : Constraint Satisfaction Problem (CSP) Component Definitions The objective is to find an assignment of values to variables such that all constraints are satisfied. This is achieved through the use of constraint satisfaction techniques. For example, in a spatial domain, the constraint variables represent objects, and their domain sets wnsist of al1 possible positions they could occupy in a spatial configuration. The set of constraints over relations between the variables would be spatial and could include information such as le# of, infront of, behind, etc. The objective is to determine a poslion vector for al1 objects such that al1 constraints are satisfied. Each position vector could contain multiple components, one for each dimension or relation. Each of the vector components could contain a set of possible position values for the object in a specific relation or dimension. For example, given two constraint variables A and BI and the constraint set A is /et? of B and A is above B, the set of possible domains can be represented by numerical position vedors wrresponding to each relationldimension; i.e. the domain for A (DA) consists of {{possible positions for the le# of relation}, {possible positions for the above relation}}. The solution for each relation can be represented by a linear ordering and thus the domain for A (DA) and B (De) could initially be represented by {{O, 1}, {O, 1)). The wnstraint A is leR of 8 would result in the leffof component of domain A an6 S being affected and reduced to DA= {{O}, {0,1}} and DB = {{1}, (0.1)). Similarly, the A is above 8 constraint would result in the above component of the respective domains being effeded, and a resulting solution being produced in which DA= ({O}, {O}} and DB = {{l), (1)).

Given: The Solution is:

Figure 4.2: Sarnple Constraint Satisfaction Problern Definition and Solution Each dimension and relation is mutually exclusive and can be considered independently. For simplicity, in following examples only one relation will be explained. For the spatial domain, the solution can be described as set of objects and corresponding position vedors, where each component of the position vector consists of indices representing a position in a linear ordering in each dimension andlor each relation. Finding this solution is achieved through constraint satisfaction techniques mich are described more thoroughly in the ensuing sections. CSPs can be thought of as search problems. Using constraint satisfaction methods to solve CSPs involves finding a solution, in a discrete finite space, satis- fying a set of constraints [VanHentenryck 19891. More precisely, the goal is to find one or al1 possible assignments of values to the set of constraint variables, from the possible values provided in the domain set, such that al1 (or required subsets) of the constraints are satisfied. As compared to exhaustive searches, constraint satisfaction techniques can greatly reduce the amount of search required in discovering solutions. The basic idea behind constraint satisfaction is to manage the complexity of the problem by constraining it and enabling manageable problem- solving. Among the most popular çonstraint satisfaction techniques are constraint propagation, consistency checking, backtracking, and constraint relaxation. These techniques are described in the following sections. A typical constraint satisfaction program involves checking and maintaining wnsistency while propagating the constraints. Typically, constraints are propagated as far as possible, and if a solution has not already presented itself, backtracking andlor constraint relaxation may be required. The cycle of propagating, consistency checking, backtracking, andlor relaxing continues until a satisfactory solution has been found.

4.1 Representing Constraint Knowledge In order to implement and describe constraint satisfaction techniques, it is con- venient to adopt an appropriate knowiedge representation scheme. Constraint networks provide an appropriate representation. As described in the previous chapter, constraint networks are depicted as graphs. The nodes can be used to represent objects and relations can be represented by directed edges. For example, the constraints (iefl;,A,B),(left, B, C) and (leR,D,B ) are represented in the constraint graph in Figure 4.3.

Figure 4.3: A Sample Constraint Network Representation (representing the constraints (leff,A,B), (lefi,B, C) and (leff,D,B)).

4.2 Constraint Propagation

Constraint propagation is the activity in wtiich constraints are processed. Typi- cally, input constraints are considered and evaluated sequentially. Grouping dis- joint subsets of constraints would permit parallel processing of such groups. Processing a constraint means, first, checking it for consistency with previously processed constraints. The next step may involve backtracking, constraint relaxation, or termination of the program. The following subsections will present these processes. Constraint propagation attempts to solve subproblems of the overall problem in a way that will improve and accelerate the overall solution. That is, constraint propagation attempts to instantiate the constraint variables in an intelligent way by considering influential factors (constraints) in a global fashion. The followïngexample provides a simple demonstration of wnstraint propagation. Assume the set of constraints to be propagated are represented by simple atomic formulas and consist of: leR(A,B), leR(B, C) and leR(D,8), meaning A is to the left of BI B is to the left of Cland D is to the lett of B. If propagating the constraints means their incorporation into a constraint network, then constraint propagation would proceed as in Figure 4.4. Constraint being Propaaated Constraint Network

Sep 1 leR(A, B)

Step 2

Step 3

Figure 4.4: Example of Constraint Propagation

4.3 Consistency Checking

Consistency checking is a technique used to detemine the consistency within a set of constraints. consistency checking is applied to the initial set of input constraints and, as well, when adding constraints in reasoning, hypothesizing, and problem-solving. The importance of consistency may Vary depending on the problem domain and the task In some cases, inconsistencies rnay lead to backtracking or constraint relaxation (described in sections 4.4 and 4.5 respectively). There are several approaches to consistency chding including fonvard checking and look ahead, as well as many other specialized versions [Liu & Ku 1 992; Shapiro 19901. Mackworth (1977) provides algorithms yielding different levels of consistency and ranging correspondingly in wmplexity (up to exponen- tial). In the previous example (see Figure 4.4), propagating the constraint leit(B,A) would yield an inconsistency (as the constraint leR(A,B) has already been propagated) and would trigger appropriate actions, whether backtracking or relaxation

1. For a more thorough description please see the Encyclopedia of Artificial intelligence -- Shapiro 1990. (for example, discarding the inconsistent constraint or allowing the inconsistent information for the time being and evaluating the conflicting alternatives at a later time.) The specific task dictates the appropriate actions to take place upon finding inconsistencies.

4.4 Backtracking

Backtracking is a standard technique used in solving CSPs [Nadel 19881. Backtracking involves systematically exploring the potential solution space of the CSP. The potential values for the variables are tested and evaluated for validity in an orderiy fashion. Such testing and evaluation involves use of the constraint set. Mena potential assignment value fails (for a variable) the remainder of potential solutions involving such an instantiation are eliminated from the search. If al1 other remaining potential values also fail, backtracking occurs. In this case backtracking retums to the last instantiated variable with unassigned values remaining in its domain and processing continues by testing and evaluating the next possible assignment. The backtracking procesç continues until a solution is found or until al1 possible solutions have been eliminated. In problems where an exhaustive search is not necessary, backtracking can provide a more efficient solution process. Adding consistency checking can enhance backtracking by reducing the potential value domains before searching begins.

4.5 Constraint Relaxation Constraint relaxation is the process of relaxing constraints to aid in finding a solution. It may require ignoring and thrmving out inconsistent constraints, or it may involve some form of evaluation and inclusion for the time being. For exam- pie, given the constraints (left,A,B), (left,B,C), (leR,D,B), and (left,B,A), consistency checking could result in simply reporting the (M,B,A) wnstraint as inconsistent (as it is in direct conflict with (left,A,B)) and continue as if it did not exist, or, the consistency could be acknowiedged and result in the solution path being divided into two possible paths; the division being made by finding the inconsistent information and having one path including it and not including the new information, and the other path excluding the conflicting information yet including the new information (Le. one path considering the constraints (lefi,A,B), (leR,B,C), and (lefi,D, B) and the other considering the constraints (lefi,B, C), (leR,D,B), and (leR,B,A) ). The decision conceming which path to follow could be made at a later time. Obviously, the latter approach greatly increases the cornplexity of finding a solution but rnay be necessary as dictated by the problem definition.

4.6 Sumrnary Research in constraint satisfaction offers a simple knowledge representation scheme for spatial and temporal problems (Section 4.1 ). It also provides well- researched algorithms for solving constraint satisfaction problems. This chapter has introduced cornmon constraint satisfaction techniques including wnstraint propagation, mnsistency checking, backtracking, and constraint relaxation. Section 4.1 provided a simple example of a constraint network. Con- straint networks are used to represent knowledge and implement constraint- based problem-solving and reasoning strategies. Such strategies involve constraint propagation (described in Section 4.2) and consistency checking (Section 4.3) and may involve constraint relaxation (Section 4.4) andlor backtracking (Sec- tion 4.5). Examples from a spatial domain have been incorporated to illustrate the concepts as this domain is the focus of concem for the derivation scheme purposed in Chapter 6. Temporal and spatial reasoning research involving constraint satisfaction techniques is the topic of the next chapter. The temporal and spatial domains are closely related and temporal research has assisted work in the spatial domain. Chapter 5

Contributions From Temporal and Spatial Reasoning Research

A scheme for deriving a depictive knowledge representation from a description is proposed in Chapter 6. This work involves a spatial domain (it is concemed with representing and reasoning about spatial relationships between objects). Research which has influenced the development and implernentation of the scheme is presented in this chapter. These works include research from both Spa- tial and temporal domains. The temporal and spatial domains are closely related. The temporal domain involves representing and reasoning about relations between events in time. The spatial domain involves representing and reasoning about spatial relations between objects. Section 5.1 presents knowledge representation strategies developed for temporal and spatial domains. An important work, frequently referenced in both temporal and spatial research, is that of Allen (1983). Allen proposes a method for representing and reasoning about temporal knowledge. Section 5.1 -1 presents Allen's set of temporal relations which are used to express temporal knowiedge. Guesgen and Hertzberg (1992) provide an adaptation of this work for the spatial domain; this is described in Section 5.1 -2.In temporal and spatial reasoning, relational knowledge constrains the possible sequence of events or potential arrangements of objects in space. Section 5.1.3 illustrates the use of constraint networks as a simple and efficient means to represent such relational knowledge. Constraint propagation is the process of propagating constraint knowledge and deriving the effect's of constraints. Allen (1983) describes a Transitivity Table which is used to derive implicit knowiedge and can be used as a constraint propa- gationlreasoning tool. Guesgen and Hertzberg (1992) describe a rnodified version of this, referred to as a Composition Table, for the spatial domain. Section 5.2 describes the Composition Table and its use in constraint propagation. Having provided the foundational background for the implementation proposed in Chapter 6, Section 5.3 discusses additional research and alternative approaches to be considered.

5.1 Representation of Temporal and Spatial Relations In the temporal and spatial domains, knowiedge is commonly expressed by propositions involving relational information. For example, "Jane ate breakfast before she read the newspapef, and, "The bal1 is leR of the book". Such propositions represent constraints between events or objects and are used to represent temporal and spatial knowledge respectively. Allen (1983) proposed a set of temporal relations for representing and reasoning in the temporal dornain. Section 5.1.1 presents these relations. Guesgen and Hertzberg (1992) borrowed frorn this work and suggested a set of relations for the spatial domain. Their research is presented in section 5.1 -2. These works suggest temporal and spatial relations which should be wnsidered and incorporated into prograrns for representing and reasoning about temporal and spatial problems respectively.

5.1.1 Temporal Relations

Allen's intewaî algebra [Allen 19831 is based on the possible relationships that hold between intervals of tirne. Allen deduced that there are only thirteen possible ways in which an ordered pair of time intervals wuld be related, and proposed a set of temporal relations to describe these interrelations. Each relation is mutually exclusive. This set of primitive relations fom the basis from which more cornplete relations can be defined or composeci. Table 5.1 illustrates these relations. A major attraction of Allen's work is that it provides a representation scheme which is simple and easy to implement using constraint satisfaction algorithms [Vilain, Kautz and van Beek 19901.

5.1.2 Spatial Relations

lnspired by Allen's set of temporal relations, Guesgen and Hertzberg (1 992) defined a set of relations for the spatial domain. Similar to Allen's set of temporal relations, each relation is mutually exclusive and together they provide a primitive

43 set of spatial relations from which more cornplex relations can later be defined or cornposed. For example, adjacent could be defÏned as either left, right, in front, behind, above, below, or any combination thereof. Table 5.2 illustrates Guesgen and Hertzberg's spatial relations.

Relationship Symbol Sym bol for Pictonal Inverse Example

X before Y < > XXXW X equal Y - - XXX YYY X meets Y m mi xxxnni. X overlaps Y O O i XXX YYY X during Y d di XXX ri"rr(Y X starts Y s si XXX YYYYY X finishes Y f fi XXX YYYYY Table 5.1 Allen's Thirteen Possible Temporal Relationships

Relationship Sym bol Symbol for P ictorial Converse Exam ple

0,aftached to O2 I

O1 overlapping O2 t

Table 5.2 Guesgen and Hertzberg's Eight Possible Spatial Relationships 5.1.3 Relational Knowledge Represented As Constraint Networks

The propositions commonly used to descn-be temporal and spatial knowledge represent constraints on the sequence of events in the temporal domain and on the relative positions of objects in the spatial domain. This knowledge mn be easily represented by wnstraint networks. Constraint network representations are used by many researchers investigating temporal and spatial knowledge representation and reasoning. Guesgen and Hertzberg illustrate a constraint network for the spatial domain. Figure 5.1a shows their graph of a constraint network representing the chair is leff of both the desk and the lamp, and fhe desk is le8 of the lamp [Guesgen and Hertzberg 19921. The circles are used to represent objects (constraint variables) and rectangles represent the constraints.

Figure 5. l a: Constraint network representing spatial relationships. [Guesgen and Herîzberg 19921

Van Beek (1990) provides a simplification of Guesgen and Hertzberg's constraint network representation in which nodes represent objects and edges are directed and labeled with the constraints (see Figure 5.1 b). This representation is used in the derivation scheme desm'bed in Chapter 6.

($!?yJChair

Figure 5.1 b: Alternative constraint network representation. [derived from van Beek 19901 5.2 Reasoning and Problem Solving Temporal and spatial problem solving involves reasoning about relations. Once knowledge is represented as a constraint network, reasoning may simply involve deriving information from the network or it may involve the addition of new knowledge into the network andlor the deletion of inconsistencies. In adding new constraints it is desirable to represent implicit knowledge. For exarnple, given the chair is leffof the desk and the desk is lefi of the lamp it is implicit that the chair is le# of the iamp. Section 5.2.1 describes Guesgen and Hertzberg's Composition Table which provides a tool for deriving implicit knowledge and for detenining inconsistencies. Constraint propagation is the method used to build and reason with wnstraint networks. Section 5.2.2 describes constraint propagation using relational constraints.

5.2.1 A Constraint Propagation and Reasoning Tool: The Composition Table

As a propagation and reasoning tool, Allen proposed a Transifivity Table (for temporal relations) [Allen 19831. The transitivity table is used when propagating the effects of new knowledge and incorporating the new information into an existing constraint network. The idea is to unveil implicit knowledge and to rernove inconsistencies by computing the transitive closure on the existing network. Guesgen and Hertzberg modified Allen's transitivity table for the spatial domain and renamed it the Composition Table (Table 5.3). The use of the composition and transitivity tables are similar. Sinœ the derivation scheme proposed in Chapter 6 is wncemed with a spatial domain, the following explanation will focus on Guesgen and Hertzberg's Composition Table. The entries in Guesgen and Hertzberg's Composition Table define the relationships that are possible between two objects based on knowledge of their relationship with a common third object. For exarnple, assume three objects represented by 01,02, and 03. The relationship between O1 and O3 can be cornputed based on the re~ationshi~s'between O1 and O2 as well as O2 and 03. Table 5.2 wntains Guesgen and Hertzberg's Composition Table. In this table, the entry '?" means that the relationship between 0,and O3 may be any of the eight possible primitive relations [Guesgen and Hertzberg 19921.

Relation between O2 and O3 I

Table 5.3:

Composition (Transitivity) Table for deterrnining the possible spatial relations between 0,and O3 based on the relations between 0,and O*, and, O2 and 03-

* c means leff oc 2 and I mean attached fo,t and= rnean overlapping, C means inside. ? means any of the possible relations. Similar work, done by Freska (1990), points out that not al1 combinations of spatial relations are plausible in the real world and that the mmplexity of the constraint satisfaction algorithms can be significantly reduced by not considering these. Allen (1983), and Guesgen and Hertzberg (1992) incorporate their respective tables into standard constraint satisfaction algorithms for propagating wnstraints. Such techniques enable representing and reasoning about both explicit and implicit information.

5.2.2 Constraint Propagation Constraint propagation algorithms are used to build constraint networks from propositional knowledge and for reasoning and problem solving. Using constraint knowledge for reasoning and problem solving requires propagating constraints. This process involves computing the effects of each wnstraint. In reasoning about temporal relations, the constraint propagation process can be achieved by comparing each new wnstraint with al1 other known (previously processed) constraints and denving the effects. Allen achieves this through the use of the Transifivity Table [Allen 19831. The Transitivity Table allows one to look up eacb constraint relation and retums the combined result (the goal being to corn- pute the transitive closure). Guesgen and Hertzberg achieve this (for spatial relations) using their Composition Table. The effects of propagating constraints may involve the removal of derived impossibilities or the addition of implicit information (constraints) which in tum need to be propagated. For example, assume the relation set between O1 and O2 consisted of 5,t, and 3.Adding and propagating the constraints 01 < O3and O2 s O3 results in the relation a being removed from the possible relation set for O1 and O2 (Le. look up the result of combining O1 c O2 and 022 O3 hmthe Compo- sition Table; the new possible relation set equals the intersection of the set from the Composition Table and the old possible relation set.) An example of deriving and adding a constraint is given in the following: Given 0, c O2 and O2 e Cl3 the constraint 0,< O3wouid be derived and propagated. 5.2.2.1 Guesgen and Hertzberg's Two Constraint Propagation Approaches for Buiiding a Constraint Network Guesgen and Hertzberg (1992) describe two appmaches for building a constraint network representation from a set of constraints. Guesgen and Hertzberg's first approach to building a constraint network involves two steps: 1) lnserting new constraints into the network and using the Composition Table to unveil any implicit knowledge resulting from the addition of the new constraint (this may involve the generation of new constraints to be propagated.); and 2) using the Composition Table to detemine any inconsistencies resulting from the addition of the new constraint and to remove them. The second approach allows the first step to be sub- sumed by the second by beginning with a complete network and removing inconsistencies (by using the Composition Table to detemine the inconsistencies Mile propagating each constraint.) Guesgen and Hertzberg represent a complete constraint network simply by having each rectangle initialized to contain al1 possible relations.

5.3 Summary, Conclusions and Related Work

Allen (1983) proposed an interval-based algebra for temporal reasoning which has influenced research in both the temporal and spatial domains. In his 1983 paper, Allen derived a set of primitive temporal relations for representing and reasoning about time. Guesgen and Hertzberg (1992) borrowed from this work and presented a primitive set of spatial relations which should be implemented in a spatial representation and reasoning system. Furthemore, Guesgen and Hertzberg point out how simple it would be to extend such a system to include more relations and involve more dimensions. Guesgen and Hertzberg suggest that each spatial dimension can be thought of independently and that each analo- gously maps to a dimensional linear spatial representation similar to a time-line. The location (relational information) of an object in multidimensional space is represented separately for each dimension. Hence, the addition of a dimension only requires generating the spatial configuration of objects in that dimension. Mile such a representation scheme is easily extendable, Hemandez (1994) points out that it is not cognitively plausible and proposes an alternative representation, which is implied to be more intuitive, based on topological and orientation knowledge. Both Allen's and Guesgen and Hertzberg's works use constraint satisfaction techniques for representing and reasoning in the respective domains (temporal and spatial). Knowiedge is represented using constraint networks where the nodes represent events or objeds respectively and labelled edges are used to delineate the relationships among them. The Composition Table (or transitivity table as Allen (1983) refers to it) is used as a propagation tool. It provides information wnceming the relations between Wo objects (spatial domain) or events (temporal domain) given their relation with a cornmon object or event (Le. given left(A,B) and left(B,C), looking up the entry in the table under the 'left' wlumn and the 'left' row shows that A must also be left of C). Freksa (1992a) enhances the simplicity of the table by introducing conceptual neighbohoods which are based on physical constraints imposed by the real world and eliminate some of the physically impossible relations given in the original tables. These neighborhoods increase inferencing efficiency and significantly reduce the complexity of the constraint algorithms [Hernandez 1994; Guesgen and Hertzberg 19921. This chapter has presented background information which has inspired, and been incorporated into, the derivation scheme proposed in Chapter 6. The purpose of this scheme is to demonstrate the practical use of constraint satisfaction techniques in deriving a depictive representation from a description. This scheme is concemed with a spatial domain; that is, representing, descriptively and depictively, the spatial relations between objects. The concepts presented in this chapter have provided a foundation from which the proposed derivation scheme has been developed and implemented. Chapter 6 Descriptions to Depictions: A Derivation Scheme

This chapter presents a scheme for deriving a depictive representation of knowledge from a description. The scheme involves a spatial domain; that is, it takes a description of a spatial scene and produces a depictive representation that preserves the relevant (relational) information given in the description. The proposed derivation scheme draws on research from a variety of areas including cognitive psychology (1 inguistic analysis and cognitive modellhg), knowledge representation, constraint satisfaction, and temporal and spatial reasoning. Evidence from psychology pertaining to the use and manipulation of descriptive and depictive representations in humans was presented in Chapter 2. Computational knowledge representation schemes were overviewed in Chapter 3. Both Chapters 2 and 3 pointed out the appropriateness of using depictive representations for solving some problems as opposed to formal logical reasoning with descriptions. Evidence supports the daim that in a variety of activities humans transfomi descriptive knowledge into depictive representations and vice versa. It has been suggested that depictive representations are a natural and intuitive tool used in problem solving by humans. Such suggestions are supported by the wide- spread use of depictive representations such as graphs, maps, and diagrams. Providing a comparable computational tool, capable of deriving a depictive fom for representing and reasoning with descriptive knowledge provides for ease of understanding, implernentation and use. The derivation scheme proposed in this thesis provides a computational tool which can be used to generate depictive representations of knowledge from descriptions. Such a tool is motivated by tasks, such as spatial reasoning, in which a depictive representation is more efficient, intuitive, informative and useful than a description. Previous chapters have explained the appropriateness of using constraint satisfaction to solve the description to depiction derivation problern; hence, the proposed scheme uses constraint satisfaction techniques to achieve its goal. Chapter 4 introduced constraint satisfaction techniques. Chapter 5 presented particularly relevant research which uses constraint satisfaction techniques and has inspired the proposed derivation scheme. The scheme involves two fundamental steps: 1) Generating a constraint network representation from the initial description; and 2) Generating a depictive representation from the constraint network. The background works described in Chapter 5 provide particularly useful information and techniques for step 1). An approach for generating a depictive representation (in the form of symbolic arrays - suitable for spatial reasoning) from the constraint network (step 2) is introduced in the proposed scheme. (Syrnbolic arrays were introduced and described in Chapter 3 Section 2.1 .) For ease of implementation, an alternative method of representing constraint networks (through the use of subnetworks) is used, and is also presented in this chapter. The remainder of this chapter describes the proposed scheme and an exam- pie implementation (Appendix A) for deriving a depictive representation, in the fom of symbolic arrays, from a description wnsisting of spatial propositions. A Blocks World environment has been chosen to elucidate the concepts. Section 6.1 provides an overview of the strategy used to derive a depictive representation from a description of a spatial scene. Sections 6.2 through 6.4 present and explain the proposed theory.

6.1 The Objective

The goal of the proposed scheme is to take a descriptive representation of a spatial scene and derive a depictive representation which preserves the pertinent information given in the description. For a spatial domain, this pertinent information consists of knowledge that represents the relations between objects within the scene. In the scheme, a partiwlar scene is referred to as a worid [Glasgow 1994b3. The initial descriptive representation descnebinga world is denoted by w and the depictive representation by W. The objective is to derive Wfrom w such that W preserves the spatial relations specified in the world description, W. A set of functions, F, has been defined and is used in the derivation of Wfrorn W. w, Wand F are described in detail in the following sections, 6.2 through 6.4 respec- tivel y. Figure 6.1 highlights the basic components and general goal of the derivation scheme.

1 Given: W - initial descriptive representation I- w - final depidive representation F - a fbnction. or functions, that when applied to w yields W

Objective: - to define F such that I F maps w to W. ' Figure 6.1 : Overview of Objective for Deriving a Depictive Representation W from a Descriptive Representation w

6.2 The Descriptive Representation w The initial descriptive representation, w, consists of a set of propositions, {p,...p,}, which describe the world. Each proposition is defined as representing a relational constraint between two objects in the world. Such a definition is justified by the assumption that propositions involving more than two objects, or more than one relation, can be broken into an informationally equivalent set of propositions where each proposition in the set involves only one relation and two objects. By informationally equivalent it is meant that each relationship between each pair of objects described in the proposition having greater than two objects, or greater than one relation, is also deswibed in the set of propositions where each proposition describes a single relation between a pair of objects. It is assumed that the initial cornplex propositions are such that no information is lost or gained in the translation into single relation, object pair propositions. The set of objects of interest in w are represented by S,,,,. Each object of interest is uniquely represented in S, The set of spatial relations which may hold between the obje~tsin w are represented by R,,,,. Rw may contain relations such as those suggested by Guesgen and Hertzberg (left of, attached to, overlapping, inside, and their converses - see Section 5.1.2) as well as relations in other dimensions such as above and below, and in front and behind. The propositions given in w are represented as tuples, p = (r, &, sy) where r is an element of Rw and s, and sy are elements of S,,,,. Figure 6.2 provides a summary of, and highiights, the components of a depictive representation. Figure 6.3 provides a simple Blocks World example illustrating

& - a finite set of symbois that uniquely denote entities of interest in the world. Rw - a finite set of relations that hoid between entities in the world.

p = (r. sX, s,,) is a proposition such that r E R,,,,, s, and sy E S,,,, w = {PI Pz1-.- PJ is a world description for propositions pl ,...p,

Figure 6.2: Components of a Depictive Representation a descriptive representation of a scene. The scene involves the two relations left and above (Rw = {left, above}) and two symbols representing the two objects of interest, blocks A and B (S,,,, = {A, B}). w contains two propositions (w=(pl, p2)) describing the spatial relationships for al1 relations r E Rw and al1 objects s E S, block A is to the left of block B (pl = (left,A,B)) and block A is above block B (pz = (above,A, B)).

Blocks World Example #1

S, ={A, BI R,,,, = {left, above} w = {pl, p2) where pl = (left,A,B) and p;! = (above,A,B).

Figure 6.3:Descriptive Representation of a Simple Scene

6.3 The Depictive Representation W W consists of a set of anays Aw = {Al .. . An) which are used to depictively represent the information containeci in the descriptive representation w, and a set of functions FW = (fl ... fy which are used to interpret the information in the arrays. (This set of functions is different from the set of functions used to map w to w- 1 Figure 6.4: Definition of W Amy representations were introduced in Section 3.2.1. Men deriving a depictive representation Wfrom a descriptive representation w, each anay Ai E Wcorresponds to a different possible scenario representing the information in W. Each array ceIl can contain a symbol Si E SW, where SW is the finite set of symbols which denote the objects of interest in RWrepresents a finite set of relations of interest in W. Menderiving a depidive representation of w, Sw and SW represent the same set of entities, and, Rw and RW represent the same set of relations. Fwcontains a boolean inspection function, (fi), for al1 relations E RW. These functions retum tnie or false depending on whether a proposition is represented in the given array representation. Given a proposition, p = {rn,%,~},p is represented in 4 E Wif and only if fn(g,%,Ai) = true. Figure 6.5 provides a simple example which highlights the components used to describe and define the depictive representation W. The array scenarios (Al and A*) depicted in Figure 6.5 represent A to the left of B (left,A, B); applying fieft(A,B,A1 ), and fie fi(^, B.A~), retums true.

w =(Aw.FW) Figure 6.5: Depictive Representation of a Simple Scene

Figure 6.5 and other examples provided assume a level of representation and reasoning in whicb it suffices to represent objects using one anay ceIl. In essence, the notion of relative size and shape are not represented depictively in the array representation. Such an assumption is justified by the fact that many levels of reasoning do not require such detail (for example, consider a chess board and pieces; it suffices to ignore the shape and size of the pieces. Also consider route planning via landmarks where the important information is directional - from City Hall go west to City Park, then south to the Yacht Club (distance, shape and size are not required)). Using only one array cell to represent an object keeps the storage requirements for the representation to a minimum and simplifies the processing requirements involved with the representation. Figures 6.6 and 6.7 illustrate how several spatial relations can be represented within array representations. Figure 6.6 illustrates how the relations left, above, and in front are represented in arrays.

Figure 6.6: Propositions and Corresponding Depictive (Symbolic Array) Representations

Figure 6.7 provides an example of how Guesgen and Hertzberg's spatial relations (see Section 5.1 -2)could be represented using arrays. lntroducing the attached relation increases the compiexity of the depictive array representation by introducing the notion of distance. Attached implies that there is no spaœ or distance between objects. Not attached implies there is a space (actual distance length wuld be ignored) and requires a distinction to be made between objects that are attached or not attached. Such a distinction is intuitively implemented by representing a space (empty array cell) between the objeds, thereby increasing the size of the anay representation. For example, (left,A,B) now requires a space in the representation (in the absence of (attached,A,B)) whereas in a domain where attached is irrelevant, no spaœ needs to be represented (compare depictive representations of leffin Figures 6.6 and 6.7). Syrn bolic Array Symbolic Array Representation Proposition Representation of Converse

inside, A, 8

Figure 6.7: Example Depictive (Symbolic Array) Representations Corresponding to Guesgen and HertzbergJs Eight Possible Spatial Relationships

The inclusion of ovedapping, as shown in Figure 6.7, requires objects to be represented using more than one array ceIl. Using one or more cells to represent an object produces an equivalent or larger storage requirement than a scheme requiring only one cell to represent an object. The inclusion of relations requiring a greater number of array celis to represent an object increases the time and space requirements needed to generate, store and manipulate the depictive representations. Devising methods based on array representations requiring fewer array cells to represent an object may jeopardize and reduce the intuitive quality of the representation scheme. Another issue relating to the required space needed to depictively represent spatial relations in arrays is the unbounded number of representations con- sistently representing the same relational information (see Figure 6.8a). Equivalence classes are defined for the array scenarios in Wto constrain the infinite number of possible sœnarios corresponding to descriptive information given in W. Figure 6.8a provides a simple one dimension, one relation example to illustrate how an equivalence class can be used to manage the concept of distance.

representativef equivalence class representation wiai minimal space requirement

Figure 6.8a: Example of Equivalence Classes

The example in Figure 6.8a is only concemed the relation left; each array in the equivalence class represents (left,A,B), and fieft(AIB,Ai} retums true for i = 1 to infinity. Equivalence classes are assumed for each applicable relation in w being represented in W. Each member of an equivalence class represents the same relevant relational information. For any two array scenarios of the same equivalence class, Ai and Ajl fr(~,çyIAi)= fr(~xl~ylAj)for al1 functions fr E Fwand for al1 sx+ E SW. It is assumecl that only one array scenario representative of an equivalence class is necessary in each Aw to depict the wrresponding relational information described in W. This enables an infinite set of possibilities to be represented by a finite set. The notion of equality and equivalence between array scenarios is defined as follows: two array scenarios are equal (=) if the contents of each cell in each array are the same (Ai[x]= Aj[xl for x = minimum to maximum amy ceIl index). Figure 6.8b defines anay scenario equivalence; two array scenarios are equivalent (-) if they retum the same value for al1 f, E FW and for al1 pairs of symbols s, and sy E SW(in the case of equivalence Ai[xl may or may not be equal to Aj[xl). In Figure 6.8a1Al - A2 but Al t A2.

Ai - Aj iff fr(~,syIAi)= fr(%,sylAj) for al1 functions fr E F and al1 %,% E Sw Figure 6.8b: Definition of Array Scenario Equivalence Ai - Ai Each array Ai E AW represents a unique possible scenario representing a possible arrangement of the objeds {sl ... s,) = Swaçcording to the relations {rl ... r,) = RW- By unique it is meant not only that no two array scenarios are equal but also that no two arrays are equivalent Expressed another way, uniqueness can also be defined as ~OIIOVVS:V A, and Ai E AW (A, + Ail NOT(Ai - Ai),and (% # s+), there exists at least one fr(~,sy,Ai) = true, fr(+,5y1 Aj) = false. The set of amys AW = (A, ... A,) is complete if there exists one array scenario representing each possible arrangement of al1 objects in S according to al1 relations in RW. (Note that it is assumed that representing only one member of each equivalence class suffices, and allows a cornplete set of array scenarios to be represented effkiently.) Figure 6.9 illustrates an exarnple of a wmplete set of array scenarios; A, contains an array scenafio for each possible arrangement of al1 objects in Sw (A,B, and C), according to al1 relations in RW, (left).

Rw = {left}

Figure 6.9: Example of a Complete Set of Array Scenarios, Aw representing three objects and one relation.

Wis defined as a cornplete representation of w if and only if both Wand w represent the same set of objects and relations (S w= Sw, R W= Rw), A wconsists of al1 possible array scenarios consistent with the propositions given in w, and there exists interpretation functions (F which allow al1 the relational knowledge represented by the propositions in w to be derived from W. The array scenanos are considered consistent if each scenan'o (Ai E AW) represents ail the propositions pi E w, and no scenario represents a çontradictory proposition NOT(h E w) for al1 pi E W. Figure 6.10 provides a simple one dimension, one relation example of a descriptive representation w and derived depictive representation W. In Figure 6.10,AW is a complete set of array scenarios, where each corresponds to a different scenario consistent with the propositions in w, and W is a complete representation of W.

w = (pi , p2) where pl = (leff,A,B), pz = (left,A,C).

Figure 6.1O: Example Descriptive and Corresponding Depidive Representation

Each array Ai E AW corresponds to a unique scenario consistent with the information in W. The actual number of array scenarios in a complete representation Wof w depends on whether the information in w is consistent, complete, or incomplete. Consistent information means that for al1 pi E w, where pi is of the form (~,s~,çy),a contradictory relation, not(ri,%,sy), does not exist in w and is not derivable frorn the infomation in W. How to handle inconsistent information is influenced by the domain being represented. Non-monotonic logics could be used to assist in the management of inconsistencies. For example, given a real world domain and the propositions "the cat is on the chaiP (on,cat,chair) and "the chair is on the catn (on,chair,cat), wmmon sense eliminates the latter, inconsistent proposition. Alternative methods include assuming inconsistent infomation is invalid and ignoring inconsistencies as they are encountered and continuing on, or assuming the input set is invalid and halting processing. In the implementation of the derivation scheme, inconsistent infomation in w is not allowed and inconsistencies invalidate the input set and result in no corresponding array scenarios being produced, A = 0. Allowing inwnsistencies could be accommodated by irnplementing a non-rnonotonic rnethod. One approach could be to implement a preprocessing rnethod which eliminates inconsistencies resulting in a valid input set in W. The descriptive representation, w, contains complete information if the relation between a symbol q E S, and al1 the other symbols in Sw are described for each relation ri E Rw, or are deducible from the given description W. If a descriptive representation contains corn plete and consistent infomation then there is one corresponding depictive scenario and AW consists of only one array, AW = {Al). Such a representation is referred to as deteminate. Figure 6.11 is a determinate Blocks World example.

Blocks World Example #2: Deteminate Exarnple

Sw=& ={A,B.C)

Rw=R,,,, =(lefi}

W = PZ) where pl = (left,A,B), p;! = (left,B,C).

F~ = fied

The example in Figure 6.1 1 involves three blocks, A, B and Cl and two propositions (left,A.B) and (left,B,C). In the anay scenano A, B and C are uniquely identified and ocwpy only one array cell. The propositional information is complete in that the relation between al1 objects in SW is provided (or transitively deducible) for al1 relations in Rw The resulting world W consists of only one scenario, Al, representing A to the left of B and C, and B to the left of C. No other possible scenarios exist which are consistent W. if the descriptive representation, w, contains incornplete (but consistent) information, then more than one possible corresponding scenario exists and W consists of a finite set of array representations, AW = {Al ... A,}- Assuming consistency and representing only one member of an equivalence class, the number of possible array scenarios in Aw is finite with an upper bound of (R x S!), where R and S represent the nurnber of relations in RW and symbols in Sw. Together, this set of arrays represent ali possible scenarios consistent wïththe

information in W. The knowledge preserved in each array, Ai E Aw, is consistent with the knowledge contained in w (in terms of relational information) and each array represents a consistent yet different possible scenano corresponding to the given descriptive representation w (V pi E w, pi is represented in every Ai E A*). Such a representation is referred to as incornpiete or indeterminate and yields multiple possible, yet unique, array representations (AW= {A, .. . A,}; v Ai and Ai E A (Ai + Ajl NOT(Ai - Ai), and (% # s~)),there exists at least one f,(%,ç~, Ai) = true, fr(%,%, Aj) = false). Figure 6.12 illustrates an exarnple of an indeterminate representation. As in Figure 6.1 1, each block in the example in Figure 6.12 is

B locks Worid Example #3: Indetemiinate Exam ple

Sw=S, ={A.B.C}

RW=R, ={lefi}

W = {pi , p2) where pl = (ief€,A,B), p;! = (left,A,C).

FW = Ged

Figure 6.12 uniquely identified and only occupies one array position. The example involves three blocks, A, B and C, and two propositions (leftlAIB) and (left,A,C). The propo sitional information is incomplete with respect to whether B is Ieft of C or vice versa. The resulting world AW mnsists of multiple (Mo) possible scenarios. Ai, containing the possibility of B being left of Cl and A2, containing the possibility that C is left of B. Either scenario is possible, and only cne could actually exist in the real world, but until more inforrnation is gained, neither scenario is mled out The different array scenarios resulting from indeteminate infomation in w can be described by defining necessary and possible knowledge. Assuming the relations pi E w are consistent, they are defined as representing necessary knowledge; necessary knowledge is represented in each array scenario Ai E Aw and no array scenafio represents knowledge that conflicts with necessary knowledge. Possible knowledge is defined as knowledge which is consistent with the necessary relations pj E w, but has not been represented in w (explicitly or derivable by transitive deduction). Possible relations are represented as Q = {ql ... q,) where for ail qi E QIqi E w, qi is consistent with w and qi is not deducible from W. In the case of indeterrninacy, each Ai E AW represents a unique possible scenario (V

Ai and Aj E Aw ((Ai#Aj) and NOT(Ai-Ai)) and q, E Ai 1 9, E A,) and (3 qy E Ai 1 qy e Ai); where q, and q, are possible and q, # qy. Figure 6.1 3 summarizes the number of array scenarios in AW based on consistent. deteminate (complete), or indeterminate (incomplete) inforrnation given in

W. Given indeteminate information, the maximum nurnber of array scenarios is R x S! where R and S represent the number of relations in Rwand S wrespectively.

If w contains inconsistent information. AW = 0 If w contains &teminate information. A = (A1)

If w contains Nideteminate information. A = {A1 .. . A,} Figure 6.13

Figure 6.14 provides a summary of how Wcorresponds to and represents W. The followi*ng section (6.4) describes how a complete representation, W, is derived from W. Wis a complete representation of w ifi

SW= Sw - both sets of syrnbois represent the same set of objecb of interest

R W= R,,,, - both sets of relations represent the same set of relations of interest

VGE RWt3fi€Fw - an interpretation function exists for al1 relations

V Pj = (rk,%,sy) E w 3 fk E Fw( fk(~x,~y,Ai)= true V Ai E AW - al1 propositions are represented in ail array scenan'as

A is c0mpIefe - al! possible scenanos are represented (see definition page 59-60)

V Ai and Aj E Aw *Ai - Aj) - ail scenarios are unique

Figure 6.14: Summary of Rules Goveming Wrepfesenthg W.

6.4 The Derivation Process - F: w + W The goal of the proposed scheme is to derive a depictive representation, W, from a descriptive representation, w, through the application of some function(s), F where F maps w to W(F: w + W). Blocks World Examples #2 (Figure 6.1 1) and #3 (Figure 6.1 2) provide examples of correspondhg descriptive (w) and depictive representations (WI. In the proposed scheme, constraint satisfaction techniques are employed to achieve the denvation goal. Such techniques dictate the use of an intermediate constraint network representation. Constraint networks provide a useful representation and reasoning tool and are used to propagate the input descriptive propositions before the depidive representation is generated. Therefore, F is defined as having two components F, , ,,: w -, CN and FCN, *: CN + Fw+ cN: w -t CN maps the descriptive representation, w, to a constraint network representation, CN. FcN, , : CN -+ W maps the constraint network representation CN to the depictive representation W.

: Definition:

Figure 6.15

6.4.1 The Intemediate Constraint Netwark Representation - CN

As the descriptive representation w consists of propositions (pi E w) where each proposition represents a wnstraint, a wnstraint network is appropnately used to represent W. The quivalence of a constraint network representation to a representation w is defined in Figure 6.16. A proposition pi is represented in a

Definition: CN - W iff pi E w, pi is represented in CN. I Figure 6.16 I depictive representation when, in the descriptive interpretation of the depictive representation, pi is derived or derivable. A wnstraint network can be represented as a directed graph where the nodes represent objects (s E S) and the edges represent the relations (r E R) that hold between the two object nodes that the edge connects. Consider the three dimensional example in Figure 6.17 which depicts w, where w = {(left,A,B), (lefi,A,C), (left, B,A), (above, C,A), (above,C, B), (above, B,A), (front,A, B), (front,A, C), (front,B,C)}. The depiction includes the three blocks, A, B and C. A is left of B and C, B is left of C; C is above A and B, B is above A; A is in front of B and C, and B is in front of C. A constraint network representation corresponding to Figure 6.17 is provided in Figure 6.1 8. In this figure, the nodes of the constraint network represent the blocks A, 6,C and are labeled accordingly. The directed edges are labeled to indicate the relation(s) that hold between the two nodes they conneci. 7"r left

above Y\c)

Figure 6.18 Figure 6.1 7 Constraint Network Representation Blocks Workl Example #4 of Blocks World Example #4

The relational knowledge represented in the constraint network in Figure 6.18 is equivalent to the relational knowledge represented in Figure 6.17; As in Figure 6.1 7, Figure 6.18 represents three blocks, A, B and C where A is left of B and Cl B is left of C; B is above A and C, B is above C; A is in front of B and C, and B is in front of C. By the definition in Figure 6.16, the constraint network in Figure 6.1 8 is equivalent to the descriptive representation w given in Figure 6.1 7; each proposition in Figure 6.1 7 is represented and derivable from the constraint network in Fig- ure 6.18. The constraint network can be represented using named subnetworks (cm), where each subnetwork corresponds to one particular relation, ri E R. Figure 6.19 provides an example subnetwork representation consisting of q,~,Cn,bove and c%,,,. This subnetwork example corresponds to Blocks World Example #4 (Fig- ure 6.1 7) where R = {left, above, front} and w = {(left,A,B), (left,A,C), (left,B.C), (above, C,A), (above, C, B), (above, B,A), (front,A, B), (front,A,C), (front,B,C)}. Using named subnetworks, where the name is used to indicate the relation the particular subnet is representing, eliminates the need to label edges. By arranging the objects in the subnetworks in a left to right order (where the directed edges al1 point from left to right) the need to represent edges is eliminated as well. Named subnetworks can then be easily implemented as an ordered list of objects (Figure 6.20). Figure 6.20 illustrates a subnetwork representation corresponding to Fig- ures 6.1 8 and 6.1 9. The relational knowledae represented in the constraint sub-

Figure 6.19 Figure 6.20 Constraint Subnetwork Representation Constraint Subnetworks of Blocks World Example # Represented as Lists networks shown in Figure 6.20 is equivalent to the relational knowledge represented in Figures 6.18 and 6.1 9; Again, three blocks are represented, A, B and C. The subnetwork indicates that A is left of B and Cl and B is left of C; cn,bov,, indicates that B is above A and Cl and B is above C; and cnkOntindicates that A is in front of B and C, and B is in front of C. Figure 6.20 shows the simple list representation used to represent the constraint network (CN) as subnetwoks (cns). The relation these lists represent is indicated by the name of the particular list. There is one list for each relation (ri E R) represented in W. The lists are interpreted in a left to right manner and the resulting relational knowledge is equivalent to the relational knowledge represented in Figures 6.17, 6.1 8 and 6.1 9. In the example implernentation (Appendix A) the constraint network (CN) is represented as lists of subnetworks (cm).The following sections use CN to refer to the collection of constraint subnetworks used to represent CN. Figure 6.21 describes the relationship between the constraint network CN and corresponding subnetworks. 1 Definition:

Figure 6.21

When mapping one representation of knowledge to another the goal is to preserve al1 relevant information. Menmapping w to CN the goal is to preserve and represent al1 relational knowledge. Two approaches for generating a constraint network (CN) from a set of constraints (w) are introduced in Section 5.2.2.1 [Guesgen and Hertzberg 19921. The fundamental difference between the hnro approaches can be explained using three types of knowiedge: necessary, impossible, and possible. One approach results in a constraint network which represents only the necessary information described in W. The other approach results in a constraint network which represents both the necessary and possible information described by W. Necessary knowledge has already been defined (Section 6.3) as the knowledge contained in w (or derivable therefrom). It is assumed this knowiedge contains no contradictions. It is further assumed that this knowledge is true and fadual knowledge. Impossible knowfedge is that which is in direct contradiction to necessary knowiedge. In Section 6.3, possible knowiedge is defined as knowledge which is consistent with necessary knowledge but has not been represented in, and is not derivable from, the knowledge explicitly stated in W. Altematively, possible knowiedge can be described as that which is neither necessary nor impossible. The necessary only approach for building a constraint network representation generates a constraint network which explicitly represents only the necessary knowledge described by W. The necessary knowfedge in the descriptive representation w is represented by the propositions, p, in W. Each pi E w describes a relationship, r, between two objects, s, and % Explicitly representing this knowledge in a constraint network means that for al1 pi E w (where pi = (rk,~,sy))there exist nodes in the constraint network representation, correspondhg to s, and %, with a directed edge leading from s, to sy and labeled rk. Figure 6.22 depicts an example of a constraint network generated using the necessary only approach given w<(A, B, left)}, R={left} and SqA,B, C}. 1

Figure 6.22 1 Constraint Network representing Necessaty Kno wledge only

Impossible and possible knowledge are not explicitly represented in the constraint network constnicted using the necessary knowledge only approach. Impossible knowledge is derivable through inspection of the constraint network or by manipulating the propositions in the descriptive representation W. By inspection, it can be observed that a relationship holds between two objects (i.e. there exist nodes representing the two objects and a labeled directed edge connecting them, representing the relation between them); it can then be deduced that a contradictory relation between these two objects is impossible. For example, in Figure 6.22, A is depicted as being to the left of B. It can then be deduced that it is impossible for B to be left of A. Similarly, impossible knowledge cm be derived by revening the order of the objects in each

proposition described in W. For example, given the proposition (left,A,B) in w, (left,B,A) is deemed impossible. The concept of impossible and possible knowledge relies on the laws goveming the represented domain. Given these laws, appropriate rules can be defined which enable the detenination of impossible and possible knowledge. The sample domain in this thesis is a Blocks World where the blocks are equivalent in shape and size. Such a domain dictates that if block A is left of block 8, it is impossible for B to be left of A (at the same time). In the necessary knowledge only approach, possible knowledge is any non-explicit knowledge which is not impossible. In Figure 6.22, it is unknown whether, and therefore possible that, A is left of C or C is left of A, and 6 left of C or C left of B. Possible knowledge can be detenined by inspection of the constraint network. Given a proposition (rk,çxt%), if the NOnodes, s, and sy, do not have an edge labeled rk wnnecting them then (rkt~,sy)is possible (time cornplexity O(s)). To compute the wmplete set of possible knowledge through inspection of the constraint network requires that all pairs of objects (s,and sy E S 1 s+,,) are inspected to see if an edge for each rk E R exists between them. If no edge labeled rk exists between s, and sy then (rk,sxl%) and (T~,s~,s~)are added to the list of possible knowledge (time complexity 0(rs2) where s is the number of objects in S and r is the number of relations represented in R.) Another approach is to build a constraint network which explicitly represents both neceçsary and possible knowledge. In the necessary plus possible approach, necessary and impossible knowledge are represented and detennined as in the necessary only approach. Unlike in the necessary only approach, possible knowledge is represented explicitly. Figure 6.23 depicts an example constraint network representation generated using the necessary plus possible approach given ww={(left,A,B)], R={left} and S={A, B, C}.

Figure 6.23 Set of Constraint Networks representing Necessary Plus Possible Knowledge

This approach generates a set of constraint network representations where each member of the set uniquely represents one possible scenario corresponding to the knowledge in W. Each constraint network scenario generated represents ail objects s in S and relations r in R. Together the set of constraint networks generated represent al1 the possible scenarios described by

W. If w represents corn plete knowledge (sedion 6.3) both the necessary only and the necessary plus possible approaches lead to the generation of an identical constraint network representation. The space requirement for this constraint network is O(rs) where r is the number of relations in R and s is the number of symbols representing objects in S. R and S respectively represent the sets of relations and objects involved in the descriptive representation. In the case of incomplete knowledge, the necessary knowledge only approach results in a disconnected constraint network representation (Figure 6.22). The space required for this representation is at maximum O@). The approach that represents both necessary and possible knowledge (given incornplete knowledge in w) generates a set of constraint networks where each network uniquely represents one of the possible scenarios consistent with the information in W. Together, the set of constraint networks generated by the necessary plus possible approach, represent al1 of the possible scenarios consistent with w (Figure 6.23). The maximum space requirement for the representation produced by the necessary plus possible approach is maximum 0(rs2). The maximum space requirement would be necessary if no wnstraints existed (Le. w=0, R#0, Sg0). However, given just one constraint (w={pl}) reduces the maximum space requirement by one half. Essentially both of the constraint network representatiwis produced by the different approaches (necessary only and necessary plus possible) contain the same knowledge and reasoning processes can be designed for either. It can be argued that the necessary plus possible approach generates a more intuitive constraint network representation, explicitly representing both necessary and possible information. Such a scheme subsumes representing default knowledge - a common approach taken by humans in the absence of information. Computing and representing all possible scenarios may be viewed as going overboard. Simi- larly, representing only necessary knowledge may be viewed as not sufficient. The desired use of the representation plays a fundamental role in deciding wtiich approach to take. Providing an intuitive depictive representation is a main goal of the work provided in this thesis. Hence, the implementation incorporates the necessary plus possible approach to generating the constraint network representation. Further investigation into the pros and cons of one approach versus the other is left as future work. The necessary only approach has been introduced as a possible alternative which may irnprove the time and space complexity for applications in which the intuitive quality may not be a fundamental concem.

6.4.2.1 The lmplementation of F,,,,: w -+ CN The derivation scheme has been implemented using the array-based programming language NlAL (Nested Interactive Array Language)[Jenkins 19931. As previously mentioned, the constraint network generated in the implementation represents necessary plus possible knowiedge as described in section 6.4.2. The constraint network generated is represented as subnetworks as described in section 6.4.1 (Figure 6.20). The implementation generates the constraint network representation CN from the given propositions, W. As described in section 6.2, each proposition represents a relational constraint between two objects. In the implementation, the user enters the set of symbols representing objects of interest (S,). The relations left, above, and in front have been implemented. These relations provide a basis for representing 3-Dimensional scenes. ldeas for the implementation of additional relations as well as potential implications are presented in section 6.3. The constraint network CN is generated in the fonn of relation-based subnetworks by first sorting the propositions into subsets according to the relation involved in the proposition (Figure 6.24).

w = {(left,A, B), (left,A, C), (left, B, C), (above,CA), (above,Cl B), (above, B,A), (front,A, B), (front,A, C), (front, B, C)}.

kft 1 kt = {(left,A, B), (left,A, C), (left, B, C)) above list= {(above,C,A), (above,C,B), (above,B,A)) front [kt = {(front,A,B), (front,A, C), (front, B,C)}

Figure 6.24: Relation-based sublists of propositions

The constraint subnetworks are generated independently using a corresponding sublist of propositions. For example, the lefi subnetwork is generated using the sublist consisting of propositions involving the relation lefi. The subnetworks are represented in NIAL as ordered Iists of symbols representing the arrangement of al1 objects (S,,,,) with respect to a specific relation (Figure 6.25).

--- Sublists of Constraints: lefi kt = {(lefilAl B), (left,A,C), (left,BI C)) above kt= {(above,C,A), (above,C,B), (above,B,A)) front kt = {(front,A, B), (front,A,C), (front,B,C)}

Generated Constraint Subnetworks:

Cnleft = (ABC)

Cnabove = (CBA) %ont = (ABC)

Figure 6.25 Constraint Sublists and Generated Constraint Subnetworks

If the propositions provided in w do not represent complefe information, more than one scenario exists, and a constraint subnetwork representation of each possible scenario is generated. Each subnetwork represents each possible scenario as a separate list (Figure 6.26).

.- Sublists of Constraints: left list = {(left,A,B)) above kt= {(above,A,C)) front list = {(front,B,C))

Generateci Constraint Subnetworks:

Cnleft = (ABC, ACB, CAB)

Cnabove = (ABC, ACB, BAC)

Cnftont = (ABC, BAC, BCA)

Figure 6.26 Constraint Sublists and Generated Constraint Subnetworks To generate al1 possible scenarios consistent with the given propositions, al1 possible arrangements of objects (Sw) could first be generated and then the constraints couid be propagated to remove any scenarios that are inconsistent with the constraints (Figure 6.27). Such an approach is suggested by Guesgen and Hertzberg (see Chapter 5 section 5.2.2.1). The time and space required by this approach can be improved upon by propagating the constraints at the sarne time as generating al1 possible scenarios. The implementation (Appendix A) uses the propagate as generate approach.

------Sublists of Constraints:

left list = {( lefi,A, B1) above list = {(above,A,C))

front list = {(frant,B,C)}

Constraint Subnetworks Representing ALL Possible Scenarios:

Cnt eft = {ABC, ACB, BAC, BCA, CAB, CBA}

Cnabove = {ABC, ACB, BAC, BCA, CAB, CBA}

Cnfront = {ABC,ACB,BAC,BCA,CAB,CBA}

Figure 6.27 Generating ALL Scenarios and Removing lnconsistencies through Constraint Propagation

The implernentation developed uses a recursive operation, Generate-One-CS (see Appendix A), to propagate constraints and generate the individual scenarios (scenario lists) for each subnetwork, from right to left as shown in Figure 6.28. The algorithm generates al1 possible scenarios by taking a list of syrnbols (Le. A, B, C) and shuffling the symbols in the following manner: from left to right, each symbol, Level 1 ABC BAC CAB L TL tS Level2 BC-CCB AC+CA AB+BA \1 Result ABC +- ACB t BAC c- BCA t CAB t CBA Figure 6.28 Trace of Generate-One-CS given the symbols (A,B,C) in turn, is placed in the first (left most) position in the list, with the remaining symbols positioned from left to right as in the original list (Le. ABC, BAC, CAB). This procedure recurses, remembering the first (left most symbol) and passing the symbol list less the first symbol until the passed list wntains only one symbol. Menonly one symbol is left, that symbol is retumed and the recursion unravels to construct each possible configuration from right to left (i.e. ABC t ACB t BAC c BCA t CAB t CBA). The scenarios are constructed using the maintained first symbol at each lsvel of recursion and concatenating any combinations generated successively. For exarnple, at the top level A, B, and C represent the first (left most) symbols successively, recursively calling the same operation passing the remaining lists BC, AC, and AB respectively. Given the fist AB, the operation retums BA and AB which are concatenated to C to produce the lists CBA and CAB. The lists BCA, BCA, ACB and ABC are wnstnicted in the same manner. By propagating the constraints as the scenarios are being generated, the generation of inconsistent scenarios is avoided (Figure 6.29). The algorithm propagates the

Level 1 ABC CAB I + 1

4 Result ABC t ACB + CAB

Figure 6.29 Trace of Generate-One-CS given the symbols (A,B,C) and the constraint (left,A,B) constraints before it recursively calls itself by detemining if the resulting combinations will be consistent with the constraints. For example, given the constraint (left, A. B), any combinations having 6 left of A need not be generated; when B is the first (left most) symbol in the symbol list, if A is in the remainder of the Iist, the rewrsion is teminated as al1 successive combinations would eventually be concatenated with B representing B to the left of A. Propagating while generating eliminates inconsistent scenarios before they are generated. Given N symbols, the number of possible scenarios (unique arrangements of objects) is N!. The propagation of just one constraint results in the number of scenarios generated being halved (NY2). For example, given three symbols, there are six possible scenarios (unique arrangements of objects). Propagating just one constraint results in three less scenarios (three scenarios would be inconsistent with the constraint). Table 6.1 shows the number of scenarios reduced by propagating just one constraint. The number of scenarios eliminated by propagating further constraints can Vary depending on the nature of the constraints; Mat relation and what sym bols they involve.

1 Number of Scenarios 1 Total Number of Total Possible After Propagating Syrnbols Scenanos Only One Constraint (number of possbb scsnarios (pc~~ibleunique arnngements d objtcts) Representing O bjects ~onsiïbtntwiththe consûaint)

1 Table 6.1 : Num ber of Sym bols and Corresponding Num ber of Scenarios 1

Figure 6.30 illustrates the resulting constraint lists produced by Generate-One-CS given the set of symbols {A, B,C) and the constraints (1eft.A. B), (above,A,C) and (front,B,C). The operation Generate-Constraint-Subnetworkç (Appendix A) is the main function that maps w to CN (Fw, ,,: w -+ CN). Generate-Constraint Subnetworks breaks the constraint list into sublists according to the relations (r E R,,,). Generate-Constraint-Sublists then calls Generate-One-CS to generate each constraint subnetwork list independently, passing the appropriate wnstraint sublist (according to the subnetwork being generated). The irnplemented algorithm generates the individual subnetworks sequentially, however, these subnetworks could be generated in parallel.

Given: g = {A, B1 Cl Rw = {left, above,front}

w= {PI, Pz1 ~3) where pl =(left,A,B), pz = (above,A,C), and p3 =(front,B,C)

Cnleft = {ABC, ACB, CAB)

Figure 6.30: Constraint Subnetworks Produced by Generate-Constraint-Subnetworks

The subnetworks produced by Generate-Constraint-Sublists, and hence Generate-One-CS, can be used foi reasoning and problern solving in their Iist fom. However, the goal of the derivation scheme is to produce a more intuitive (more picture-like) depictive representation. Mapping the wnstraint network representation into a depictive representation is discussed in the following section.

The goal when mapping CN to W is to produce a depictive representation (W) that is equivalent to the constraint network representation (CN) in terrns of relevant knowledge represented. CN represents al1 possible scenarios, or arrangement of symbols, consistent with the set of constraints given in the descriptive representation W. W contains a set of array scenarios each corresponding to a unique scenario represented in CN. Creating anay scenarios from the constraint network representation can be achieved simply as follows: 1) detenine the array dimensions required to represent each scenario; 2) create an empty array of appropriate dimensions; 3) detemine the locations to place symbols to intuitively depict the location of symbols such that al1 relationships are represented correctly; 4) place the symbols in the array representation at the determined location. Steps 2) through 4) are repeated for al1 scenarios represented in CN. The array dimensions are determined from the relations and symbols to be represented. A onedimensional array is al1 that is necessary to represent one relation. Each dimension requires enough cells to represent the symbols and relationships pertaining to that dimension. For example, given the relation leff and the symbols A, B and C, a 1 x 3 array can be used to represent a possible scenario as in Figure 6.31,where A is depicted to the left of B and B is to the left

+ left -

Figure 6.31 : 1-Dimensional Array Representation

Given two relations, a two-dimensional array may be necessary. For exarnple, given the relations left and above, a twodimensional array can be used to intuitively depict both the leR and above relations as in figure 6.32. As in Figure

+ left - 012

Figure 6.32: ZDirnensional Array Representation I

6-31, the numbers in Figure 6.32 represent array indices. A is shown at location (2,0), B is at location (1, l)and C is at location (0.2).The first position in the index represents the position of the symbols with respect to the above relation, the second position represents the position of the symbols with respect to the relation le#. Given three relations, such as leR, above and in front dictates the need for a t hreedimensional representation. Figure 6.33 shows how a threedimensional array could be depicted. A is shown at location (0,2,0),B is at location (Ill,1) and C is at location (2,0,2).The first position in the index represents the position of the symbols with respect to the in front relation, and the followi-ng positions represent the position of the symbols with respect to the relations above and le# respectively. Calculating the array indices for the sym bols can be straight foiward using the

front

Figure 6.33: 3-Dimensional Array Representation constraint subnetwork representations. For example, given the three relations lefi, above and in front, a three dimensional array is required to represent al1 of these relationships between symbols. A three-dimensional array index contains three components, each corresponding to a position in each dimension. The relations le@, above and in fmnt each correspond to a different dimension in the three- dimensional representation. The index for a symbol in each dimension is it's position in the corresponding constraint subnehivork list. The rnultidimensional index is simply the conjunction of the individual dimension indices. As show in Figure 6.34, the three-dimensional index derived for the symbol A is (0,2,0).The number of components in an index coincides with the number of dimensions represented. The lehost position in the index corresponds to the highest dimension represented. In this example, the relations correspond to the dimensions as follows: in front corresponds to the third dimension, above corresponds to the second, and leff to the first dimension. The three dimensional index can thus be described as of the fom (front index, above index, leff index). The individual indexes for a symbol correspond to it's position in the corresponding wnstraint subnetwork representation, which in the case of the symbol A is O for the front or third dimension, 2 for the above or second dimension, and O for the leff or first dimension. Similarly, the three-dimensional indices for B and C are determined to be (1 ,l,1) and (2,0,2) respectively.

Constraint Subnetworks: position in Iists 77

Derived Array Indices: A = (0,2,0) B = (qtltl) c = (2,0,2) The components of the index are in the order (front index, above index, left index)

Figure 6.34 Constraint SubneWorks and Derived Array Indices

The implemented scheme for mapping CN to W derives one-, two-, or three- dimensional array scenafios according to the relations to be represented. The implementation involves the use of predefined functions (or operations) from a NlAL Library and previous related (unpublished) works. These operations have been combined with additional new code to provide a new function which maps CN to W. The algorithm and operations are described in detail in the following section.

6.4.3.1 The Implementation of F,, ,,CN+W The implementation (Appendix A) denves the depictive representation Wfrom CN as described in Section 6.4.3 and as depicted in Figure 6.35. The NlAL operation Generate-Scenados is the controlling function used to map the scenarios in CN to the array scenarios in W. Generate-Scenarios determines how many array dimensions are required to represent the information wntained in the wnstraint network representation and calls the appropriate operation Generate-Scenarios- 1 D, Generat-Scenanos-ZD, or Generate-Scenarios-3D. The difference between these operations is the number of constraint subnetworks involved in each; only one constraint network is manipulated by Generate-Scenarios-1 D, two and three subnetworks are manipulated by Generate-Scenarios-2D and 30 respedively. As described in Section 6.4.2.1 and depided in Figure 6.34, the constraint subnetworks are represented in NlAL as ordered lists of symbols. In the implementation the constraint network lists are referred to as le#-lists, above_lsts, and infronuists accordingly. The NlAL operations lists2arr and find- pafhs use the constraint subnetwork lists to constmct the array scenarios. Generate-Scenarios passes the constraint subnetwork lists corresponding to each scenario sequentially to lists2arr. Lists2arr detemines the array dimensions necessary for each scenario and creates the anay. Findpaths is used by lists2arr to derive the multidimensional array index for each symbol from the constraint sublists. ListsZan uses the indexes from findpaths to place each symbol in the array scenario at the appropriate location. Figure 6.35 provides a trace of the algorithms and illustrates how the depictive scenarios are derived from the constraint subnetworks. The constraint network lists are passed to Generate-Scenarios. Generate-Scenarios constnicts a scenario sublist by uniquely wmbining one element from each constraint network list. AI1 possible scenarios are generated by combining one element from each list until al1 combinations of single elernents is exhausted. For example, using the lists le-lists = (ABC, ACB, CAB), above-lists = (CBA) and infront-lists = (ABC), three possible scenaflos are cornputed. The first scenario sublist includes the elements (ABC) from left-lists, (CBA) frorn above-lists and (ABC) frorn infront-lists. A second sœnario is constructeci using the second element of left-lists, (ACB), corn- bined with the elements from above-lists (CBA) and infront-lists (ABC). The last possible scenarÏo is generated using the final element of left-lists, (CAB), corn- bined with the elements from above-lists (CBA) and infront-lists (ABC). Findpaths generates the array indices for each symbol in each scenario using the position information of each symbol in each element of the scenario sublist. For example, in Scenario 1, (ABC), (CBA), (ABC), the index for A is detemined to be (0,2,0) as A is in position O in the first element, position 2 in the second elernent and position O in the third element. The first element in the index refers to the position of A with respect to the left relation. The second and third index positions refer to the position of A with respect to the relations above and in front respectively. Similarly the index for B and C are determined to be (1,1,1) and (2,0,2) respectively. The indices for the symbols in Scenarios 2 and 3 are computed in the same manner. Using the cornputeci indices, Generate-Scenarios construds the array rnodels representing each possible scenario and places the symbols in each model at the mmputed locations.

6.5 Summary

This chapter has described a scheme for deriving a depictive representation of knowledge from a description through the use of constraint satisfaction techniques. The scheme was inspired by the works of Allan and Guesgen and Hertzberg, as described in the previous chapter. It's efforts are encouraged and supported by the appearance of more and more papers inspired by the same works (see Hemandez 1994).The follow-ng chapter provides a summary of what the derivation scheme accomplishes and suggestions for further development. CN: Constraint Network lists passed to Generate-ScenaMs:

cnwt = ~eft-kts = (ABC, ACB, CAB)

C~IOW= above-lisb = (CBA) c%nt = infront-Iists = (ABC)

Sublists corresponding to each possible scenario passed to lists2arr

Scenan'o i: (ABC), (CBA), (ABC)

Scenario 2: (ACB), (CBA), (ABC)

Scenario 3: (CAB), (CBA), (ABC)

For a& smmrb the sub6sk am in the order blt-list, abave-riat. mfront_hst

Array indices generated by findpaths:

Scenario 1: A= (0,2.0), B = (l,l,l),C = (2,0,2)

scenario 2: A = (0,2,0), B = (2,1,1), C = (1,0,2)

Scenario 3: A= (1,2.0), 6 = (2,1,1), C=(0,0,2) The components of the index are in the order (front index. above index. left index)

W Array scenarios generated by Generate-Scenanos:

Scenario 1 :

Scenario 2:

Scenario 3:

Figure 6.35 Example Constraint Network Representation and Generated Depictive (Array Scenarios) Representation Chapter 7 usions

This chapter highlights the main contributions of this thesis and provides a summary and suggestions for future work. Research for this thesis has involved a variety of areas. Section 7.1 describes the main contributions. Section 7.2 provides a summary and suggestions for future work. It is intended to provide a brief recount of the work in this thesis and to lead the interested reader to similar areas of research for interest and future work.

7.1 Main Contributions

Theoretical contributions are made in the areas of artificial intelligence, spatial and temporal reasoning and cognitive psychology.

The fundamental contributions to artificial intelligence are in the areas of knowledge representation and automated reasoning. A contribution is made to the development of an intuitive knowledge representation and reasoning scheme, which can aid, encourage, and simplify the development of automated problem- solving systems.

The implernentation demonstrates a computational scheme for representing spatial knowledge depictively. Such a representation is intuitive, efficient, and easy to reason with. As the implementation example focuses on the spatial domain, the contribution to temporal reasoning may not be as evident as the contribution to spatial reasoning. The implernentation can be easily adapted to a temporal domain. The spatial domain involves representing and reasoning about relations in space. The temporal domain involves representing and reasoning about relations between events in time. By assimilating time-related concepts to spatial concepts, spatial reasoning strategies can be adapted for temporal reasoning. A simple example is using a time-line and assimilating the spatial relation leR with the temporal relation befom and nght with a#er. More information con- cerning the temporal domain can be found in Allen (1983). Guesgen and Hertzberg (1992) relate Allen's temporal work to the spatial domain.

In the realm of cognitive psychology such research helps to shed light on human representation schemes (be they descriptive, depictive, or both) and the enhinement of language. Computer implementations of such schemes allow an alternative to human test bed to implernent and test hypothesis regarding how the human mind functions. Studies using computer implementations may help to mle out or support different theones conceming how the human mind stores and uses knowledge. It should be noted that the main contribution lies in providing a computational tool, not in providing an accurate human model.

7.1.1 Sample Applications

The contributions to practical applications are numerous. In theory, such a scheme could be appropriately applied to any problem in which a human might use imagery (depictive representations) in finding a solution. Some common examples of application areas involving a spatial dornain include navigation (route planning, wayfinding, robot navigation), architectural design (geographic layout, building structure, plumbing, electrical, telephone and computer integration), strategic arrangement (interior design, graphic design, warehouse layout, pattern layout), game playing (board games (Chess), adventure games (Zelda, Link), etœtera). Applications could involve fully-automated or interactive problem solving.

The implemented scheme assumes an initial descriptive representation. The descriptive representation rnay come from an existing database, or it rnay origi- nate from an interactive language recognition or text-based system processing a description on-the-fly for example. The origin of the description is not a main concem of this work; it is assumed to exist. The concern is to provide an alternative. depictive representation that is more intuitive to reason and problem solve with.

The important information that allows a descriptive representation of a spatial scene to be mapped to a depictive representation is the relative location of objects. In the case of navigation, the relative location information rnay involve countries, provinces, states, cities, etcetera and/or it rnay involve the relative location of objects such as landmarks and streets within a ci@. The relational information rnay be described in ternis of north, south, west and east as opposed to above, below, left and right, or it rnay involve both. The depictive representation could be likened to a map. Techniques could then be implemented to process the representation in a way similar to a person manually inspecting a map. These techniques could be used for route planning and wayfinding. Sirnilarly, robot navigation rnay involve representing objects in a warehouse. The depictive representation could be used in planning paths for the robot to take. For more route planning examples see [Ramaswamy 19961.

Architectural design rnay involve the relative locations of buildings within a campus for example. The placement of rooms within a building, of booths within a room, of plumbing and wiring for electricity, phones or cornputer networks rnay be desired. A depictive representation similar to a blueprint could be produced and techniques could be incorporated to assist in the design process. The strategic arrangement of fumiture could also be automated and used to evaluate the alternatives and alleviate the brute force method of actually moving the fumiture amund to determine the best arrangement. Similar procedures could be used to detemine and evaluate the placement of stock items within a warehouse. Such procedures might also be ernployed in solving best-fit problems where space is limited and detemining the most efficient use of space is desired, such as in pattern layout. Automated strategic arrangement processes could also be employed in graphic design tasks such as page layout and other layout tasks.

In games such as Chess, the relative locations of game pieces is important. Given a depictive representation of a Chessboard and game pieces, the representation can be manipulated according to the game niles. Displaying the game board and locations of game pieces after each move simulates the playing of the garne. Playing could be completely interactive (two players), serni-autornated (player versus corn puter) or fully-automated (cornputer versus corn puter).

Many TV video and cornputer games are visually based adventure games that involve the inspection and manipulation of objeds within a variety of scenes. The derivation scheme could be used to create a depictive representation from a description of such scenes. Exploring the scene could then be achieved by manipulating objects within the scene according to implemented niles goveming inspection and manipulation.

7.2 Summary and Future Work

The main focus of the work in this thesis is to derive a depictive representation from a description. Such a representation could be described as more appropriate, more efficient, andlor more intuitive than the original representation, and can lend itseif more appropriately to problem solving, especially in the spatial domain. This derivation task is achieved effectively and efkiently through the use of constraint satisfaction techniques. Such is the thesis of this paper. The next step in providing an automated problem solving system is to incorporate a reasoning engine (see Kalin 1996). The implernentation of the derivation scheme focuses on the spatial domain. The scheme takes a description and provides a model-based depictive representation consisting of a set of models each representing a depic tive scenario. The set contains a model wrresponding to each possible scenario, or possible configurations of objects, consistent with the initial descriptive representation. Each element of the set uniquely represents one of the possible configurations. In the case of complete information given in the original descriptive representation, the resulting set of depidive representations consists of only one model representing only one possible scenario. Multiple scenario models are the result of incomplete knowledge given in the initial descriptive representation. A model-based reasoning system can be attached to provide automated or interactive reasoning and problem solving. Glasgow and Malton (1994), Glasgow (1 994b) and Kalin (1996) provide references for model-based reasoning. Incorpo- rating the scheme into an automated reasoning system is beyond the goal of this thesis and is left as future work.

Depending on the application, different reasoning strategies may be required. The proposed scheme is intended as a generic wmponent used to provide a depictive representation. The application specific reasoning strategies can then be incorporated accordingly. Menimplementing intuitive reasoning strategies, it may be desirable to provide strategies for reasoning with a descriptive representation or a depictive representation, or both. Whatever the case, the derivation scheme can be used before, after, or during the reasoning processes.

Problem solving in ihe spatial dornain involves reasoning about relations. Guesgen and Hertzberg (1992) provide a set of primitive spatial relations which should be implemented in any general purpose spatial reasoning system. The implementation of the proposed derivation scheme involves a subset of these relations. Future work could extend the implementation to include al1 these relations. Previous chapters have described to some extent how this can easily be achieved.

The implementation provides a sample set of relations (le#, above and in front) that are used to represent a threedimensional scene. The implementation of additional relations has been left as future work. The relations right, below and behind can be easily implemented using the existing relations as templates or, alternatively, by converthg al1 corresponding relations to their opposites; for exarnple (right,A,B) could be wnsidered equivalent to (leR,B,A). The implementation of directional relations such as nom, south, west and east can also be achieved simply by mapping thern to the existing relations. Mapping these relations should be as commonly expected, with north mapped to above (upwards), south to below (downwards), west to the left, and east to the right. Relations such as left and right could be used as well as north, south, west and east in a geo- graphically based representation, but care should be taken to make sure that such relations are mapped accordingly within the knowledge base (i.e. left means west, nght rneans east). The irnplementation of additional relations, such as attached and overlapping is discussed in Chapter 6, Section 3. These relations are not as straightfoward to implernent as the previously mentioned relations, and can increase the wmplexity of the representation and reasoning strategies. Chapter 6 suggests a rnethod for representing these relations but expanding the implementation to include them has been left for future work.

Unlike quantitative knowledge, qualitative knowledge ignores precise numeric details such as size and distance. lgnoring quantitative detail and focusing only on essential infonation results in a less wmbersome and more efficient representation and reasoning scheme. Quantitative details are often not required in problem solving. The proposed scheme is qualitative. For certain classes of problems it is desirable or necessary to involve quantitative knowledge. The existing scheme can be extended to include additional infonation by including quantitative knowledge in object frames. Quantitative knowledge would then be available and could be used as necessary. The suggestion of adding quantitative information is also made by Hemandez (1994) and somewhat described by Giunchiglia and others (1992).

The curent implementation of the derivation scherne provides a depictive representation in the form of symbolic arrays. For some applications an alternative depictive representation may be more intuitive. A logical extension of a general- ized scheme would be to provide alternative depictive representations. Similarly, the addition of alternative descriptive representations could be explored or implemented as necessary. Chapter 3 provides a brief overview of alternative descriptive and depictive representations. Other references include Lohse, Biolsi, 'Nalker and Rueler (1994) and Giunchiglia, Ferrari, Traverso and Tnicco (1992). Lohse, Biolsi, Walker and Rueler (1994) contains many examples of commonly used depictive representations used by humans in a variety of application areas. Giunchiglia. Ferrari, Traverso and Trucco (1992) describes NALlG (Natural-Lan- guage driven Image Generator) which provides an example of a system incorporating different depictive/geometric representations.

7.3 Conclusion

The denvation scheme described in this thesis provides a component of an automated spatial reasoning system. The design is intended to promote human interaction by providing an intuitive tool for automated or interactive problem solving. That is to Say, it is intended to provide solutions and partial solutions to problems and present them in an easy to undentand, human-like way. The implemented scheme supports intuition by providing an alternative means to represent spatial knowledge; it takes a descriptive representation and provides a corresponding depictive representation, preserving al1 relevant knowledge. This is achieved effectively and efkiently through the use of constraint satisfaction techniques. REFERENCES

Allen, James F. (1983). Maintaining Knowledge about Temporal Intervals. Com- munications of the ACM. 26,832-843.

Anderson, J.R. (1978). Arguments conceming representations for mental imagery. Psvcholoaical Review. 85, 239-247.

Atwood, GE. (1971). An expenrnental study of visual imagination and memory. Capnitive Psvchology. 2.

Baddeley, A.D. and Lieberman, K. (1978). Spatial working memory. In Raymond S. Nickerson (Ed.), Attention and Performance VIII. Hillsdale, NJ: Erlbaum.

Bauer, R.M. and Rubens, A.B. (1985). Agnosia. In Kenneth M. Heilrnan and Edward Valenstein (Eds. ), Clinical Neuroosvcholoqy, (2nd Ed.), (pp. 187- 241). New York: Oxford University Press.

Block, N. (Ed.) (1981 a). Imaaerv. Cambridge, MA: MIT Press.

Block, N. (Ed.) (1981 b). Readinas in Philosophv of Psvcholoav, (Vol. 2). Cam- bridge, MA: Harvard University Press.

Bryant, D.J.; Tversky, B.; & Franklin, N. (1992). lntemal and extemal spatial frameworks for representing desci-ibed scenes. Journal of Memory & Lan-

auaoe. (31 ), 74-98. 92 Bundy, Alan (Ed.). (1990). Cataloaue of Artificial Intelliaence Techniaues, (Third, Revised Edition). New York: Springer-Verlag.

Cercone, N. and McCalla, G. (Eds.) (1993, November). Taking Issue/Forum: The lmagery Debate Revisited. Computational Intelliaence, (Vol. 9, No. 4).

Cohen, G. (1983). The psvcholoav of manition, (2nd Edition). Academic Press.

de Kleer, J. and Brown, J.S. (1988). Assumptions and ambiguities in mechanistic mental models. In A. Collins and E.E. Smith (Eds.), Readinas in Coanitive Science: A Pers~ectivefrom Psvcholoav and Artificial Intelliaence, (pp. 270-287).San Mateo, CA: Morgan-Kaufmann.

Dean, Thomas L. and McDemott, Drew V. (1987). Temporal data base management. Artificial InteIlicience. 32,l-55.

Dechter, Rina; Meiri, Itay; and Pearl, Judea. (1991). Temporal constraint networks. Artificial Inteilicrence. 49, 61-95.

Denis, M. (1 991). lmaae and Coanition. London: Hawester Wheatsheaf.

Denis, Michel and Cocude, Marguerite. (1992). Structural properties of visual images constructed from poorly or well-structured verbal descriptions. Memorv & Coanition. 20(5),497-506.

Denis, Michel and Zimmer, Hubert D. (1992). Analog properties of cognitive maps

93 constmded from verbal descriptions. Psvcholoclical Research. 54, 286- 298.

De Renzi, E. (1982). Disorders of space exploration and coanition. New York: Wiley.

Ehrlich, K & Johnson-Laird, P.N. (1982). Spatial descriptions and referential con- tinuity. Journal of Verbal Leamina & Verbal Behavior. 21, 296306.

Farah, M. J . (1 988). The neuropsychology of mental imagery: Converging evidence from braindamaged and normal subjects. In Spatial Coqnition - Brain bases and develo~ment,(pp. 35-56). Hillsdale, NJ: Erlbaum.

Farah, M.K; Hammond, K.; Levine, D.; and Calvanio, R. (1988). Visual and Spa- tial Mental Imagery: Dissociable Systems of Representation. cognitive Psvcholoay. 20,439462.

Finke, R.A. (1989). Principles of Mental Imaaerv. Cambridge, MA: MIT Press.

Finke, R.A. (1990). Creative Irnaoew: Discoveries and Inventions in Visualization. Hillsdale, NJ: Lawrence Erlbaum Associates.

Finke, R.A.; Pinker, S.; 8 Farah, M.J. (1989). Reinterpreting visual patterns in mental imagery. Coanitive Science. 13, 51-78.

Finke, R.A. and Slayton, K. (1988). Explorations of creative and visual synthesis

94 in mental irnagery. Memorv & Coanition. 16, 252-257.

Foos, Paul W. (1980). Constmding cognitive maps from sentences. Journal of Ex~enmentalPsvcholoav: Human Leamina & Memorv. 161 25-38.

Forbus, K.D. (1983). Qualitative Reasoning about Space and Motion. In D. Gent- nef and A.L. Stevens (Eds.), Mental Models, (pp. 53-74). Hillsdale, NJ: Lawrence Erlbaum Assoc.

Fortier, S.; Casteldon, 1.; Glasgow, J.; Conklin, D.; Walmsley, C.; Leherte, L.; and Allen, F. (1993). Molecular Scene Analysis: The integration of direct methods and artificial intelligence strategies for solving protein crystal structures. Acta Crvstallo~ra~hica,Section D, 1.

Franklin, Nancy; Tversky, Barbara; and Coon, Vicky. (1992). Switching points of view in spatial mental models. Memory & Coanition. 20(5),507-518.

Freksa, C. (1991 ). Qualitative spatial reasoning. In Mark and Frank (Eds. ), Coani- tive and Linauistic Aspects of Geoara~hicS~ace, (361 -372).

Freksa, C. (1992a). Temporal reasoning based on semi-intervals. Artficial Intelli- cience. 54, 199-277.

Freksa, C. (199Zb). Using orientation information for qualitative spatial reasoning. In Volume 639 of Lecture Notes in Com~uterScience (Frank et al.. Theo- ries and Methods of S~atio-TernooralReasonina in Geoara~hicSpace.

9s Intl. Conf. GIS - From S~aceto Tenitorv. Pisa), (pp. 162-178). Berlin: Springer.

Freksa, C. and Zimmerman, K. (1992). On the utilkation of spatial stmctures for wgnitively plausible and efficient reasoning. Proceedinas of the 1992 lEEE International Conference on Svstems. Man. and Cvbernetics. Chicago.

Freuder, E.C. and Mackworth, AK (Eds.) (1994). Constraint-Based Reasoninq. Cambridge, MA: The MIT Press.

Giunchiglia, Faisto; Ferrari, Carlo; Traverso, Paolo and Trucco, Emanuele. (1992). Understanding scene descriptions by integrating difFerent sources of knowledge. International Journal of ManMachine Studies. 37, 4781.

Glasgow, J. 1. (1994a). Array Representations for Model-Based Spatial Reason- ing. Proceedinas of the Sixteenth Annual Conference of the Coanitive Sci- ence Society. Atlanta.

Glasgow, J. 1. (1994b). A Unified Array Model for SpatialKemporal Reasoning. Proceedings on Spatial and Temporal Reasonina. AAAl Worksho~.

Glasgow, J. 1. (1993). The lmagery Debate Revisited: A Cornputational Perspec- tive. Computational Intelli

Glasgow, J. I . (1990). Art ficial Intelligence and Imagery. Proceedinas of Tools for Artificial Intelliaence. Washinuton.

96 Glasgow, J.I.; Fortier, S.; and Allen, F.H. (1 991 ). Crystal Structure Recognition Through Irnagery. Proceedinas of the Seventh lEEE Conference on Artfi- cial lntelliaence Amlications. Miami. FL, 98-1 05.

Glasgow, J. I.; Fortier, S.; and Allen, F.H. (1992). Molecular Scene Analysis: Crys- ta1 Structure Detemination Through Imagery. In L. Hunter (Ed.), Artificial lntelliaence and Molecular Bioloay. AAAl Press.

Glasgow, J.I. and Malton, A. (1994). A Semantics for Model-Based Reasoninq, (Technical Report 94-360). Department of Computing and Information Sci- ence, Queen's University, Kingston, Ontario, Canada.

Glasgow, J. I. and Papadias, D. (1992). Computational Imagery. Counitive Sci- ence. 16(3), 355-394.

Glenberg A.M. and Kmley P. (1992). Pictures 8 Anaphora: Evidence for independent processes. Memorv & Coanition. 20(5), 461 -471.

Glenberg, Arthur M. and McDaniel, Mark A (1992). Mental Models, Pictures, and Text: lntegration of spatial and verbal information. Memorv & Coanition, -20(5), 458460.

Guesgen, H. W. and Hertzberg, J. (1992). A Perspective of Constraint-Based Rea- soning, An lntroductory Tutonal. In J. Siekrnann (Ed.), Lecture Notes in Artificial Intelliaence Subseries of Lecture Notes in Cornouter Science. 597. New York: Springer-Verleg. Hemandez, Daniel. (1994). Qualitative Representation of Spatial Knowledge. In Lecture Notes in Artficial intelliaence. 804. New York: Springer-Verlag.

Holton, G. (1972). Some demonstrations of the effects of structural descriptions in mental imagery. Coanitive Science. 3, 231-250.

Jackendoff, Ray & Landau, B. (1993). What and Where in Spatial Language and Spatial Cognition. The Behavioral and Brain Sciences. Jun 01 1993. Vol. 16. No. 2, 217.

Jackendoff, Ray & Landau, B. (1991). Spatial Language and Spatial Cognition. In D.J. Napoli & J.A Kegl (Eds.), Bridaes between osvcholoav and linauistics: A Swathmore festschrîft for Lila Gleitman, (pp. 145-169). Hillsdale, NJ:

Erlbaum .

Jannsen, W.H. (1976). Selective interferenœ during the retrieval of visual images. Quarterlv Journal of Exoerimental Psvcholoav. 28, 535-539.

Jenkins M.A. (1993). QJNIAL User's Guide and Reference Manual Version 61, (Second Edition). Ottawa: Nial Systems Lirn ited.

Johnson-Laird, P. N. (1983). Mental rnodels: Towards a coqnitive science of lan- auaoe. inference. and consciousness. Cam bridge: Cam bridge University Press.

Kalin, Paul. (1996). Reasonable Models for S~atialReasoninq. MSc Thesis,

98 QueenJsUniversity, Kingston, Ontario. Canada.

Kosslyn, S.M. (1994). lmaae and brain: the resolution of the imaaerv debate. Cambridge, MA: MIT Press.

Kosslyn, S.M. (1993). Images in the cornputer and images in the brain. Com~uta- tional InteIlicience. Vol. 9. No. 4, 340-342.

Kosslyn, S.M. (1980). lmaae and Mind. Cambridge, MA: Harvard University Press.

Kosslyn, S.M. (1987). Seeing and imagining in the cerebral hernispheres: A cornputational approach. Psvcholo~icalReview. 94, 148-175.

Kosslyn, S.M. and Pomerantz. J.P. (1977).Imagery, propositionç, and the form of intemal representations. Coanitive Science. 9, 52-76.

Kumar, V. (1992, Spring). Algorithms for Constraint-Satisfaction Problems: A Sur- vey. Al Maaazine. Vol. 13, No. 1, 32-44.

Landau, B.; Jackendoff, R. (1993). What and Where in Spatial Language and Spatial Cognition. Behavioural and Brain Sciences. Vol. l6(2), 21 7-265.

Lang, Ewald; Carstensen, Kai-Uwe; & Simmons, Geoffrey. (1991). Modelling Spa- tial Knowledge on a Linguistic Basis: Theory, Prototype, Integration. In Lec- ture Notes in Arîificial Intelliqence. 481. New York: Springer-Verlag.

99 Liu, Bing and Ku, Yuen-Wah. (1992, November). ConstraintLisp: An Object-Ori- ented Constraint Programming Language. ACM SIGPLAN Notices. Vol. 27. NO. 1 1, 17-26.

Logie, R.H. and Denis, M. (Eds.) (1991). Mental lmaoes in Human Co~nition. Amsterdam, The Netherlands: Elsevier Science Publishers.

Lohse, G.L.; Biolsi, K.; Walker, N and Rueler, H.H. (1994, December). A Classifi- cation of Visual Representations. Communications of the ACM. Vo1.37, No. 12.

Mackworth, A.K. (1977). Consistency in Networks of Relations. Journal of Artificial Intelliaence. 8, 99-1 18.

Mani K. & Johnson-Laird, P.N. (1982). The mental re~resentationof spatial descridions. Memorv 8 Coqnition. (1O), 181 -1 87.

Man; D. and Nishihara, H.K. (1 978). Representation and recognition of the spatial organization of three dimensional shapes. Proceedin~sof the Roval Soci- etv. 8200, 269-294.

McCarthy, J. (1986). Applications of Circumscription to Formalizing Common- sense Reasoning. Artificial Intelliaence 28(1), 89.

McDemott, D. (1987). A Critique of Pure Reason. Journal of Com~utationalIntel- ligence.

1O0 McNamara, Timothy P.; Halpin, John A; Hardy, James K (1992). The representation and integration in memory of spatial and nonspatial information. Memow 8 Coanition. 20(5),51 9-532.

Mercadal, Dennis. (1990). Dictionary of Artificial Intelliaence. New York: Van Nos- trand Reinhold.

Mishkin, M.; Ungerleider, L.G.; Macko, KA (1983). Object vision and spatial vision: Two cortical pathways. Trends in Neuroscience. 6, 414-417.

Moore, R.C. (1985). Semantical Considerations on Nonmonotonic Logic. Artificial Intelliaence. 25, 75.

Nadel, B. (1988). Tree Search and Arc Consistency in Constraint Satisfaction Algorithm. In L. Kanal and V. Kumer (Eds.), Search in Artificial Intelli- aence, (pp. 287-342). New York: Springer-Verlag.

Narayanan, N.H. (1993). Imagery: Computational and Cognitive Perspectives. Com~utationalIntellicrence. Vol. 9. No. 4, 303-308.

Nosek, J.T. and Roth, 1. (1990). A cornparison of fomal knowledge representation schemes as communication tools: Predicate logic vs semantic network. International Journal of Man-Machine Studies. 33, 227-239

O'Neill, John Joseph. (j 944). Prodiaal Genius: The Life of Nikola Tesla. New York: 1. Washburn Inc. Paivio, A. (1975). Perceptual comparisons through the mind's eye. Memory & Coanition. 3, 635-647.

Papadias, D. (1990). A Knowied~eRepresentation Scheme for Imaaerv. MSc Thesis, Queen's University, Kingston, Ontario, Canada.

Papadias, D. and Glasgow, J.I. (1991). A Knowledge Representation Scheme for Computational lmagery. Proceedinas of the Thirteenth Annual Meetina of

the Coanitive Science Societv, Chicago. Hillsdale NJ: Lawrence Erlbaum Associates.

Perrig W. & Kintsch, W. (1985). Propositional and situational representations of text. Journal of Memorv & Lanauane. (241, 503-51 8.

Pezdek, K; Roman, 2.;& Sobolik, K-G. (1986). Spatial memory for objects and words. Journal of Ex~erimentaiPsvcholoav: Leamina. Mernorv. & Coani- tion. (121, 530-537.

Pylyshyn, LW. (1973). Mat the mind's eye tells the mind's brain: A critique of mental imagery. Psvcholoaical Bulletin. 80, 1-24.

Pylyshyn, ZW. (1978). lmagery and Artificial Intelligence. In C.W. Savage (Ed.), Minnesota studies in the ~hiioso~hvof science. Vol. 9, (pp. 19-55). Minne- apolis, MN: University of Minnesota Press.

Pylyshyn, Z.W. (1981 ). The lmagery Debate: Analogue Media versus Tacit Knowl-

102 edge. Psvcholoaical Review. 88,1645.

Quillian, M.R. (1968). Semantic Memory. In M. Minsky (Ed.), Semantic Informa- tion Processinq. Cambridge, MA: MIT Press.

Ramaswamy, A. (1996). A Frarnework for Model-Based Route Planninq. MSc Thesis, Queen's University, Kingston, Ontario, Canada.

Reed, S.K. (1974). Structural descriptions and the limitations of visual images. Memorv & Coanition. 2, 329-336

Ringland, G.A. and Duce, D.A. (1988). A~~roachesto Knowledae Representa- tion: An Introduction. New York: Wiley.

Rohrig, R. and Freksa, C. (1994). A Framework for Comparing Theories for Qual- itative Spatial Reasoning. Proceedings on Spatial and Tem~oralReason- ina. AAAI-94 Worksho~.Seattle. Washington, 157-159.

Shapiro, Stuart C. (1990). Encvclo~ediaof Artificial Intellioence, (Vols. 1 & 2). New York: John Wiley & Sons.

Shepard, R.N. and Cooper, LA. (1982). Mental lmaaes and their Transforma- -- -- tions Cambridge, MA: MIT Press.

Sloman, A. (1985). Artificial intelligence - We are here. Artificial Intelli~ence~ -25(1), 386. 1 O3 Talmy, Leonard. (1983). How Language Structures Space. In H. Pick & L. Acred- olo (Eds.), Soatial orientation: Theorv. research. and a~~lication,(pp. 225- 282). New York: Plenum.

Taylor, Holly A. and Tversky, Barbara. (1990). Spatial Descri~tionsand De~ic- -tions. Paper presented at the meetings of the Psychonornic Society, New Orleans, November, 1990.

Taylor, Holly A. and Tversky, Barbara. (1992a).Descriptions and depictions of environrnents. Memorv & Cognition. 20(5),483496.

Taylor, Holly A. and Tversky, Barbara. (1992b). Spatial Mental Models Derived from Suwey & Route Descriptions. Journal of Memorv and Lanauage. 31, 261 -292.

Thro, E. (1 991). The Artificial Intelliaence Dictionary. San Marcos CA: Microtrend Books, Slawson Communications, Inc.

Tye, M. (1991).The lmaaerv Debate. Cambridge, MA: MIT Press.

van Beek, P. (1990). Reasoning about Qualitative Temporal Information. Pro- ceedings AAAl-90. Boston. MA, 728-734.

van Beek, P. (1 992). Reasoning about Qualitative Temporal Information. Artficial IntelIiaence. 58, 297-326. VanHentenryck, Pascal. (1989). Constraint Satisfaction in Loaic Proaramminq. Cambridge, MA: The MIT Press.

Vilain, M. and Kautz, H. (1986). Constraint Propogation Algorithms for Temporal Reasoning. Proceedinas of the 5th National Conference on Al, 377-382.

Vilain, M.; Kautz, H.; and van Beek, P. (1989).Constraint Propogation Algorithms for Temporal Reasoning: A Revised Report. In D.S. Weld and J. de Kleer (Eds.), Readings in Qualitative Reasonina about Phvsical Svstems, (pp.

373-381 ). Morgan-Kaufman.

Waddill, Paula J. and McDaniel, Mark A. (1992). Pictorial enhancement of text memory: Limitations imposed by picture type and comprehension skill. Memorv & Coanition. 20(5), 472482.

Watson, J.D. (1968). The double helix. New York: Wiley.

Weld, D.S. and de Kleer, J. (Eds.). (1990). Readinas in Qualitative Reasoninq About Phvsical Svstems. Palo Alto, CA: Morgan Kaufmann Publishers Inc.

Zadeh, LA., (1974). Fuzzy Logic and its application to approximate reasoning. In Infornation Processinq, (pp. 591-594). Amsterdam: North-Holland. Appendix A

lmplementation of the Derivation Scheme (in Nial)

%%%%%%%%Oh%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Thesis work - Cindy Yendt-Lunn - November 18,1995 % % DERIVATION ALGORITHM ------% - FOR GENERATING ALTERNATIVE FORMS OF KNOWLEDGE % REPRESENTATION. o/o======% % * more precisely... for generating a depidive representation in the fom % of a symbolic array, from a descriptive representation, in the fom of % predicate tuples involving one relation and two objects or places.

Input: a descriptive representation using predicate tuples wnsisting of a spatial relation and two objects, (r, O,, O*),where 0,has the relation r with O*.

e.g. " left "a "b " left "b "c "above "a "b "above "b "c "infront "a "b "infront "b "c

Output: a depictive representation in the fom of a symbolic array. (Stored in the NONLOCAL variable depictive-scenarios. ) % % Brief Description:

% This implementation for deriving a depictive representation from a % descriptive representation uses symbolic arrays and binary predicates % consisting of only two objects or places and a relation which holds between % them. The predicates are the constraints which are propagated to reduce the % set of possible depictive scenarios represented by the description. % There are two approaches which wuld be taken to achieve the goal, % using the constraints to build a representation, or using the constraints % to weed out impossible depictive scenarios from the set of al1 possible % scenarios (or arrangements of al1 objects). This implementation is based % on the latter. % For instructions on how to use this program see the section lntroduction % and Instruction Display following. % % - January 11, 1995 - Cindy Yendt-Lunn % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Yo%%%%%%

%- % Utility convenience features %

Iibrary "../aitools/frames/frames 1 ; writescreen ' If the input descriptive knowiedge represents a determinate world, ' ; writescreen ' then only one depictive scenario will be generated. If there is ' ; writescreen ' any indeterrninacy, al1 differing possible scenarios will be generated.

6. writescreen ' ' ; writescreen ' This implementation is designed to work in 1D using only the left' ; vmitescreen ' relation, or 2D using the left and above relations, or 3D using the ' ; writescreen ' left, above, and infront relations. ' ; write~creen ' ' ; writescreen ' An example of node and constraint initialization and derivation ' ; writescreen ' startup is: ' ; writescreen ' ' ; writescreen ' nodes GETS "a "b "c ' ; writescreen ' constraints GETS ("1 "a "b) ("1 'b "c) ("a "a 'b) ("a "b "c)' ; witescreen ' ("i "a "b) ("i "b "c) ' ; writescreen ' generate-depidive-rep' ; Wtescreen ' ' ; witescreen ' An indeteminate 1D example is: ' ; wn'tescreen ' ' ; wn'tescreen ' nodes GETS "a "b "c ' ; writescreen ' constraints GETS ("1 "a 'b) ("1 "a 'c) ' ; writescreen ' generate-depictive-rep ' ; writescreen ' ' ; wn'tescreen ' The generation of the depictive representation involves the ' ; writescreen ' propagation of the constraints. More constraints can be added ' ; wn-tescreen ' and propagated by adding the new constraints to the constraints ' ; Wtescreen ' variable and calling generate-depidive-rep again. For example: ' ; writescreen ' ' ; writescreen ' constraints gets constraints link ("a "a 'b) ("a 'b 'c) ' ; writescreen ' generate-depidive-rep ' ; vvritescreen ' ' ; writescreen ' Sirnilarly, more objects can be added to nodes variable. For exam- wn'tescreen ' ' ; Mtescreen '------1. wn'tescreen ' ' ; writescreen '- The implemented relations thus far are: ' ; writescreen ' ' ; writescreen ' "1 for left, "a for above, and "i for infront. ' ; writescreen ' ' ; writescreen '- An example of the proper format for a wnstraint is: ("1 "a "b) ' ; writescreen 'which means a is to the left of b. (Waming - do not forget ' ; writescreen 'the brackets around each constraint.) ' ; writescreen ' ' ; vnitescreen '- To generate a depictive representation from a description ' ; writescreen 'initialize the constraints and nodes variables accordingly and ' ; wn'tescreen 'cal1 the generate-depictive-rep operation (gdr for short). ' ; writescreen ' ' ; writescreen 'For example: ' ; writescreen ' ' ; writescreen ' nodes GETS "a "b "c ' ; writescreen ' constraints GETS ("1 "a "b) ("1 "b "c) ("a "a "b) ("a 'b "c)' ; writescreen ' gdr ' ; Mtescreen ' ' ; wfitescreen '- To propagate more constraints add the new constraints to the ' ; writescreen 'constraints variable and call generate-depictive-rep ' ; wn'tescreen ' ' ; witescreen 'For example: constraints gets constraints link ("a "a "b) ("i "b "c) ' ; writescreen ' gdr ' ; writescreen ' ' ; writescreen '- To add more objects, update the nodes variable accordingly ' ; writescreen ' and cal1 generate-depictive-rep. ' ; wn'tescreen ' ' ; writescreen 'For example: nodes gets nodes link "e "f "g ' ; writescreen ' gdr ' ; writescreen " ; vwitescreen '- The variable depictive-scenarios is used to hold the curent ' ; ~~tescreen'depictive representation. ' ; vwitescreen ' ' ; writescreen '- The variables left-lists, above-lists and infront-lists represent ' ; wn-tescreen The wnstraint subnetworks generated for the left, above and infront ' ; vmitescreen 'relations respectively. ' ; writescreen ' ' ; writescreen '- Hint: Shortcuts provided: ' ; writescreen ‘ generate-constrain~subnetworks= gcs ' ; writescreen ' generate-depictiv-ep = gdr ' ; wrifescreen ' ' ; Mfescreen '------'. writescreen ' ' ; 1

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Operations for Step 1) Generating a constraint network representation % from the initial descriptive representation (F, ,,,: W -+ CN). %

%- % generate-one-~ % % From Mike Jenkins: October 1995 % Modified by Cindy and OK'd by Mike November 1995 % Modified (made more efficient) and tested: November 14, 1996 % by Cindy Yendt-Lunn % % USES: generate-one-- % USED BY: generate-connstraint-subnetworks % % % Builds individual constraint networks for individual relations, propagating % constraints in the process to improve efficiency by reducing space and tirne % usage. % % INPUT: constraint-sublist - a Iist of constraints pertaining to one relation % (either left, above, or infront) % % OUTPUT: a list representing the constraint subnetwork generated for the % relation relating to the input list constraint-sublist. % generate-one-= IS OP csl nodes { perrnl IS OP A B { C := A EACHRIGHT -= B SUBLIST B; % propagate constraints here .. . ; % look only at relevant constraints i.e. given (r,x,y) ; % look only at the constraints where y = A ; tempy gets sublist (EACH third match A csl) csl; % determine what objects must have the relation r with A ; tempx gets each second tempy; % Only continue if the permutations abide by the constraints ; % Le. if there are no inwnsistencies then continue ; IF NOT (tme in match tempx c) THEN % remove already propagated constraints from cd ; newcsl gets sublist (not (csl eachleft in tempy)) cd ; (A EACHRIGHT HlTCH (newcsl generate-one-cs C)) ELSE nuil ENDIF ); 1 F TALLY nodes = 1 THEN nodes ELSE D gets link (nodes EACHLEFT penl nodes) ENDIF

%- % generate-wnstraint-subnehn/orks % % USES: generate-one-- - to generate one constraint subnetwork at a time. % USED BY: generate-depictive-rep % % INPUT: constraints - the complete list of constraints to be propagated. % % OUTPUT: NONLOCAL Left-lists, Above-lists, Infront-lists % - Each Iist represents the constraint subnetwork generated for the % indicated spatial relation.

% - generates individual constraint lists for each relation and calls % generate-one-= to generate one constraint subnehivork at a time (one for % each relation.) % - generate-mnstraint-subnetworks assigns the result of generate-one-- % to the appropriate relation list representing that relations constraint % subnetwork. % - each subnetwork represents the possible arrangement of objects AFTER al1 % constraints have been propagated. % - if new constraints are added, generate-mnstraint-subnetworks needs to be % called again to update the relational lists representing the constraint % subnetworks BEFORE generate-depictive-representations is called. % generate-constraint-subnetworks IS OP constraints { NONLOCAL Left-lists Above-iists Infront-Iist Nodes ; relns gets (each first constraints) ; invalid gets sublist (not(eachal1 or (("1 "a "i)eachleft match relns))) relns ; If (tally invalid > O) then writescreen 'ERROR: Cannot handle the following relation(s) yet - ; write invalid ; ENDIF ; Left-Lists gets nuIl ; % initialize ALL possible constraint ; Above-Lists gets nuIl ; % subnetworks to NULL ; Infront-Lists gets nuIl ; IF "i in reins THEN % if infront constraints exist then 30 ; left-constraints gets sublist((each first constraints) match "1) constraints ; above-constraints gets sublist((each fint constraints) match 'a) constraints; infront-constraints gets sublist ((each first constraints) match "i) constraints ; Left-lists gets generate-one-CS left-constraints Nodes ; Above-Jists gets generate-one-= above-constraints Nodes ; Infront-lists gets generate-one-= infron-nstraints Nodes ; ELSEIF "a in relns THEN % if above constraints exist then 20 ; left-constraints gets sublist((each first constraints) match '1) constraints ; above-constraints gets sublist((each first constraints) match "a) constraints; Left-lists gets generate-one-= left-constraints Nodes ; Above-lists gets generate-one-CS above-constraints Nodes ; ELSEIF "1 in relns THEN % if only left constraints exist then 1D ; left-constraints gets sublist((each first constraints) match "1) constraints ; Left-lists gets generate-one-= leftconstraints Nodes ; ENDIF ; writescreen ' ' ; writescreen 'Constraint Subnetworks Generated. ' ; writescreen ' ' 1 gcs IS generate-constraint-subnetworks ; % convenience feature ; %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Operations for Step 2) Building a depictive representation in the form of % symbolic arrays from the individual relational lists representing the % relational constraint subnetworks (FCN-, ,,,,. CN+ W). %

%- % findpaths - This operation retums a list of the paths to an % item in the given anay. % % Nial library - 1994 % % USED BY: lists2arr % findpaths IS OPERATION ltem Array { findpath2 IS OPERATION ltem Amy Pathsofar { IF Array = ltem THEN solitary Pathsofar ELSEIF atomic Array THEN Null ELSE link EACHALL findpath2 (single Item) Array(Pathsofar EACHRlGHT append grid Array) ENDIF ) ; findpath2 Item Array Null } % lists2arr :: L* -> A % % Takes the symbolic lists representation and chums out the equivalent % symbolic array. If the symbolic lists represent more than one object in % a ceIl, then the retum array is NULL. This is an important feature % required by compose. % WARNING: Currently, this function does not work when a symbol exists % more than once. % % ===== % uses : findpaths % used by : generate-scenario ops.

O)(, ===== % % Paul Kalin - 1994 % lists2arr IS OP lists { dimensions := each tally Iists; % get dimensions of the array; array := dimensions reshape [null]; % an array of empties of correct shape; elements := link first lists; % the contents of the cells ; pos-listl := elements eachleft eachright findpaths lists; pos-list2 := each each each first pos-listl ; pos-list := each link each link pos-list2; % found positions of these elements; for i with (tell tally elements) do pos := pos-list@i; if (tally array@pos) = O then array@pos := elem entsai; % place the elements; else array := null; exit null; endif; endfor; array

%- % generate-scenarios-1 D, 2D,3D operations. % % USES - lists2arr % USED BY - generate-scenarios % % INPUT: dimension-lists - the lists created by the propagate operations. % (i. e. dimension-lists := Left-lists Above-lists Infront-lists ) % % OUTPUT: the depictive representation in the form of sym bolic array % scenarios. %

%- % generate-scenarios-1 D % % - generates 1D symbolic array representation. % generate-scenariosl D IS OP dimension-lists { sa-scenarios := nuIl ; wn'tescreen ' ' ; wrîtescreen'The possible scenarios are: ' ; first dimension-lists 1 % % - generates 2D syrnbolic array representation. %

generate-scenariosJD IS OP dimension-lists { sa-scenarios := nuIl ; diml gets dimension-lists@O ; dim2 gets dimension-lists@l ; FOWl VVlTH diml DO FOR d2 WTH dim2 DO scenario gets (each solitary d2) (each solitary dl) ; s := Iists2arr scenario ; sa-scenarios gets sa-scenarios link solitary s ; ENDFOR ; ENDFOR ; writescreen 'The possible scenarios are: ' ; sa-scenarios 1

%- % generate-scenarios-3D % % - generates 3D sym bolic array representation. % generate-scenario-D IS OP dimension-lists ( sa-scenarios := nul1 ; diml gets dimension-lists@O ; dim2 gets dimension-lists@l ; dim3 gets dimension-listsa2 ; FOR dl WlTH diml DO FOR d2 WiTH dim2 DO FOR ci3 WTH dim3 DO scenario gets (each solitary d3) (each solitary d2) (each solitary dl) ; s := lists2at-r scenario ; sa-scenarios gets sa-scenarios Iink solitary s ; ENDFOR ; ENDFOR ; ENDFOR ; wntescreen 'The possible scenafios are: ' ; sa-scenarios 1

%- % generate-scenarios % % General generate-scenario operation which calls the appropriate % generate-scenario operation according to what the dimensions variable % indicates (1 for 1D, 2 for 20, 3 for 3D). % % USED BY - generate-depictive-rep % % INPUT: dimension-lists - the lists created by the propagate operations. % (i.e. dimension-lists := Left-lists Above-lists Infront-lists ) % % OUTPUT: the depictive representation in the fom of symbolic array % scenarios. % % Note: There are slightly different generate-scenario ops for 1D, 20, and 3D. % generate-scenarios IS OP dimension-lists { NONLOCAL Left-lists Above-lists Infront-lists ; writescreen ' ' ; writescreen 'Generating Scenarios ... ' ; writescreen ' ' ; dimensions := 3 ; % default set to 3 ; If tally Infront-lists = O THEN dimensions gets dimensions - 1 ; % update dimensions to 2D ; Endif ; % if no "i (infront) constraints ; If tally Above-lists = O THEN dimensions gets dimensions - 1 ; % update dimensions to 1D ; Endif ; % if no "a (above) wnstraints ; sa-scenarios := nuil ; IF dimensions = 1 THEN % ieft rein only ; generate-scenarios-1 D dimension-lists ELSEIF dimensions = 2 THEN % ieft and above relns only ; generate-scenarios-2D dimension-lists ELSEIF dimensions = 3 THEN % lefi, above, and infront relns ; generate-scenarios-3D dimension-lists ELSE writescreen 'Do not handle greater than three dimensions.'; ENDlF 1

%- % generate-depictive-rep % % - main controlling operation which initializes variables and calls % appropriate operations for deriving a depictive representation (in the % fom of symbolic arrays) from a description (in the fom of a list of % constraint tuples.) % % USES: generate-scenarios % % OUTPUT: depictive-scenanos - a NONLOCAL variable containing the %- resulting depictive scenarios in the fom of symbolic arrays. % % % Input: NONLOCAL nodes constraints - assumed correctly initialized %- by user (see instructions in % Introduction and Instruction % Display section above.) % % OUTPUT: depictive-scenarios - a NONLOCAL variable containing the %- resulting depictive scenarios in the fom of symbolic arrays. % generate-depiciive-rep IS { NONLOCAL nodes constraints Left-lists Above-lists Infront-lists dimension-lists depictive-scenarios ; IF nodes = null THEN writescreen 'WARNING: nodes = null; nodes should be given an initial ' ; witescreen ' value before calling generate-depictive-rep. ' ; writescreen ' ' ; ENDIF ; IF constraints = nuIl THEN writescreen 'WARNING: constraints = null; ALL possible scenarios will ' ; witescreen ' be generated. ' ; witescreen ' ' ; ENDlF ; IF isphrase constraints@O THEN % To accommodate a single ; constraints GETS [constraints] ; % constraint ; ENDiF ; generate~constraint~subnetworksconstraints ; % Step 1 ) generating wnstraint % subnetworks (represented as % individual relation lists. ; dimension-lists GETS Le-lists Above-lists Infront-lists ; depictive-scenarios GETS generate-scenarios d imension-lists }

% Step 2) constructing symbolic % arrays from individual relation % lists. ; gdr IS generate-depictive-rep ; % convenience feature ;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %- % UTlLllY OPERATIONS % % The following operations were created or used during the development of the % implementation of the derivation scheme. They are no longer % mandatory for the implementation, but are useful functions on their % own and have been included here for future use/reference. %- % prop-one % % Written By: Cindy - November 1995 % % USED BY: wnstprop % % Retums tme if a constraint holds in a scenario, false othennn'se. prop-one IS OPERATION constraint scenario { % For each scenario, each constraint is tested. ; X gets second wnstraint ; Y gets third constraint ; IF ((X IN scenario) and (Y IN scenario)) THEN first ((X FlND scenario) c (Y FlND scenario)) ELSE % constraint does not apply to the curent scenario. ; True ENDlF }

%- % constprop % % Written By: Cindy - November 1995 % % USES: prop-one % % Tests validity of ALL constraints in a possible ordering. % Retums true if the possibility does not conflict with any constraints; % returns false otherwise. constprop IS OP constraint-sublist scenario { % Each constraint is tested in the possible scenario ; IF False IN (wnstraint-sublist eachleft prop-one scenario) THEN False ELSE True ENDlF }

%- % attach % % - Like append, but allows any number of parameters % ')(, ===== % uses % used by : arr2lists Oh ===== % % Original by Paul Kalin - 1994 % Simplified by Mike Jenkins - 1996 % attach IS OP A { (A EACHLEFT -= null) SUBLIST A 1

%- % arr2lists :: A -> L* % % Takes a symbolic array of any dimension and retums the equivalent symbolic % lists representation. % For each axis of a symbolic array, an "ais list" represents the ordering of % al1 the elements along the axis in question. A symbolic list structure is a % list of al1 the "mis lists". %

0' ===== % uses : attach % used by : % ===== % % Paul KaIin - 1994 % arr2lists IS OP A { allsplits := axes A eachleft split A ; % equivalent to getting rows & columns; each eachall attach allsplits 1