Open University Computing Department Research Report 95/16

Spatial Expressions in Idea Capture Languages

Martin Stacey

Computing Department, The Open University, Milton Keynes, UK. [email protected]

Paper for the AID'96 Workshop on Visual Representation, Reasoning and Interaction in Design Convened by J.C.B. Damski and N.H. Narayanan.

Abstract. Intelligent support systems for in have so far failed to support spatial thinking in conceptual design. But multimedia interface technology including techniques for recognising speech and drawn gestures offers solutions to the HCI problems involved in computer support for spatial conceptual design. Effective computer support could be made possible by the use of a design idea capture language for expressing and changing shapes and qualitative spatial and functional relationships, fluently in a machine-understandable way. The design idea capture language would serve to constrain the expression of design ideas sufficiently to enable the successful use of AI techniques for generating coherent spatial representations from sets of spatial expressions in the language. This paper discusses the design meanings required for spatial conceptual design with reference to linguistic studies of spatial expressions in natural language, which show that geometric approaches are insufficient for representing spatial relationships important in design.

1. Introduction

So far spatial aspects of conceptual design have been neglected in the development of intelligent support systems for conceptual design in engineering. This paper takes the view that the goal of research on support systems for engineering should be developing systems that provide workspaces in which express their ideas as they create them, so that creating computer representations of to support AI reasoning is coextensive with designing. Creating useful intelligent workspaces for conceptual design involves enabling designers to use visual representations and design operations that match human designers' early conceptions of their designs, which are fluid, qualitative and cut across conceptual categories, and are often essentially spatial. This paper argues that including spatial thinking in conceptual design within the scope of intelligent support systems involves solving significant human computer interaction problems as well as artificial intelligence problems, but multimedia interface technology provides the means to solve these problems. The key challenge is finding a good compromise between the human designers' need to express fluid and conceptually messy design ideas easily and fluently, and the machine's need for input that is unambiguous and well-structured enough to enable it to construct coherent internal representations of the design. This conflict can be tackled with design idea capture languages, in which designers can express qualitative ideas in linguistic or quasi-linguistic spatial expressions with a precise syntax and semantics that the machine can interpret in terms of its model of the design. Building an intelligent support system for spatial conceptual design is partly a human computer interaction problem, defining a spatial design language that is easy and natural for designers to use, and partly an AI problem, constructing mechanisms for interpreting expressions in that language to construct and modify coherent spatial representations of designs. 1.1. Spatial Thinking in Conceptual Design 2 M.K. Stacey

Conceptual design is the stage at the beginning of the design process when the major decisions are made about what the product is: how it works, how big it is, what its major components are, and so on. The of possible designs is explored at an abstract , before a small subset of that space is selected for further exploration at more concrete and detailed levels. The output of conceptual design is a skeletal qualitative description of the product that is refined in the course of detail design. The concepts designers reason about at the conceptual design stage are characterised by informality, provisionality, incompleteness, approximate and qualitative relationships, and rapid switches between conceptual categories; this makes representing them and reasoning about them a hard challenge for artificial intelligence. Whether spatial design activities are essential to the conceptual design of a product depends on the nature of the design problem. Some design domains are clearly spatial, notably and clothing design. Software development and most electronic engineering is non-spatial. Mechatronics - multi-technology engineering - is harder to characterise. It involves both spatial and non-spatial design processes. Rzevski et al (1995) argue that for large classes of engineering products, including many mechatronic systems, the critical decisions are about technologies and physical principles, and about the choice of types of components and how they are connected; and that these decisions precede thinking about spatial layout. However for another large class of products, involving complex or innovative mechanical systems, spatial thinking is central to the conceptual design process. Moreover many engineers think spatially even when spatial representations are not essential to the task. Spatial thinking in conceptual design in engineering involves the rapid creation and modification of qualitative spatial relationships between objects, and between objects and non-object concepts such as movement paths and magnetic fields. The creation and modification of spatial relationships is combined with the creation and modification of functional relationships; some important relationships are both spatial and functional, so making a clear separation between layout and behaviour is not always straightforward. Many such relationships in mechanical engineering concern movement and physical causality. Changing spatial relationships is often coextensive with adding or removing components such as gears or universal joints. Spatial conceptual design also often includes rapid qualitative changes in the shapes of individual objects.

1.2. Requirements for Tools for Creating Spatial Conceptual Designs

Two objectives of research on computer support for conceptual design are to create tools that (1) only restrict the design process in planned and intended ways, and that (2) provide a seamless integration of conceptual design and detail design, by enabling the construction and progressive refinement of coherent internal product models. These objectives conflict. Any computer embodies an inevitable tradeoff in its choices of primitive operations and design elements, between allowing designers freedom to express whatever human-understandable ideas they wish, and enabling the machine to interpret designs by restricting designers to machine-understandable actions. Paintboxes and three dimensional modelling systems for computer aided manufacture constitute opposite choices to meet different needs. Any design tool provides a limited range of primitive operations and design elements. The closeness of the mapping between the designers' conceptual structures and actions and the tool's primitive elements and actions, and simple combinations of them, has a profound influence on the usability of the tool (Green, 1989, 1991; Green and Petre, 1996). Different tools make different designs easy or difficult to create, and this can have a powerful biasing effect on the design process (Stacey, 1995). For instance, pencils are effective tools because primitive operations incur very low cost, but they make it very hard to create precise angles and projections of three dimensional shapes, as well as perform duplication, mirroring or rotation. Three dimensional modelling tools make it easy to create designs comprising regular geometric shapes, but not with subtleties of shading or the appearance of roughness and provisionality, for which inaccurate or seemingly inaccurate pencil lines are ideal. Design tools can also influence design by favouring mental representations that match the tools, especially in complex design fields like engineering, in which designers can have many different mental representations of the design, none encompassing every aspect of it. Stacey et al (1996b) argue that design tools should be designed to both exploit and mitigate these biasing effects, by supporting visual representations and operations that Spatial Expressions in Design Idea Capture Languages 3

encourage the use of effective mental representations, and by enabling designers to escape the biasing effects of any one formalism; the FACADE system is designed with this in mind. In order to be worth having, a tool for creating conceptual designs must be easy and natural to use, both as an external memory and communication tool for rough ideas, and as a workspace for designers who create by sketching and reacting to their own sketches (see for instance, Schön and Wiggins, 1992). (If this is an unattainable goal, the alternative objective of design tool development is a tool that enables the efficient encoding of a design worked out on paper or in the head of the .) Thus an effective tool for creating conceptual designs in engineering must provide easy ways to create design elements and relations between them that correspond to engineers' mental representations of designs; the operations for manipulating the design elements must correspond to the transformations of design ideas that engineers perform in their minds. Mental representations can be influenced, but strict formalisms and their attendant mental representations are only used for specialised tasks. If there is a serious mismatch with mental representations that designers find essential, the tool will only be used to encode designs created without it. An intelligent design support system that provides a workspace for spatial conceptual design requires a means to express and transform shapes, and spatial and functional relationships, in a machine- understandable way that is not much more difficult than drawing a few lines on paper or describing the relationship in a spoken sentence. Most of the technology required to create such a tool is now available. What remains is to put the pieces together, and work out how to make effective use of them. As Neilson and Lee (1994) point out, this requires an understanding of both task structures and human communicational codes.

2. Design Idea Capture Languages

An intelligent design support system for conceptual design must allow designers to express ideas as freely as possible, while constructing coherent internal representations of designs capable of supporting AI techniques for making inferences about them. Allowing designers to draw anything makes the machine interpretation problem intractable, and requiring designers to work directly with the elements of a coherent internal product model is inappropriate for the rapid and fluid thinking of conceptual design. The conflict between freedom for the designer and understandability for the machine is resolved by employing a design idea capture language, in which design meanings can be expressed in words or symbols, and combined using a syntax. Creating the non-spatial aspects of conceptual designs involves expressing design meanings about the functions and characteristics of components, and about the functional and configurational relationships between components. Spatial conceptual design involves the combination of functional and configurational design meanings with the expression of ideas about shapes and geometric and topological relations. 2.1. Degrees of Formality

There are two fundamentally different approaches to creating design idea capture languages for intelligent support systems: one is to provide formal languages that correspond closely to the structure of the system’s internal product model; the other is to provide a less formal language designed to correspond to the users’ mental representations, which requires reasoning to interpret it in terms of the system’s product model. This paper argues that the second approach is essential for languages for capturing spatial ideas in conceptual design. The extreme version of this approach is to use a natural language like English. Dyer et al (for instance, 1987) have developed parsing techniques and representations for interpreting natural language descriptions of mechanical systems in terms of primitive functional elements and scenarios for use (episodic qualitative representations); their approach depends on mapping words to prestored models of a limited range of design elements, as well as relationships. The EDISON project (Dyer et al, 1986, 1987) has useful lessons about interpreting languages and the information required to describe engineering systems, although it does not deal directly with spatial design or with the reasoning involved in the conceptual design of complex systems. While a language for design requires much of the expressive power of natural language to describe spatial relationships, interpretation is made tractable by the use of a limited set of words and symbols and a simple and unambiguous syntax. Where necessary the language can be augmented by the 4 M.K. Stacey use of numbers to indicate exact values. The use of spatial symbols and gestures can increase the expressive power of the language, while decreasing the complexity of the artificial intelligence techniques needed to interpret expressions in the language in the context of the design representation displayed on screen. There is now an extensive body of psycholinguistic research on how people express spatial relationships in language (for instance Landau and Jackendoff, 1993; Herskovits, 1985, 1986; Talmy, 1988; Coventry 1995; Briffault 1995), which can be used to guide the design of an intuitive and effective language. A range of AI approaches to understanding, representing and communicating qualitative spatial relationships might be employed to relate context dependent statements in a design idea capture language to a coherent internal product model (for instance Heidorn 1995; Ó Nualláin et al, 1994).

2.2. Design Languages in Intelligent Support Systems for Conceptual Design

A number of researchers and theorists have recognised the importance of languages for expressing designs (for instance, Cross, 1989; Dasgupta, 1991; Ullman, 1992). The most sophisticated existing support systems for conceptual design use languages for creating abstract functional descriptions of machines, which can be progressively refined by breaking up each formally defined function into a sequence of subfunctions. The design languages these systems supply to the user are formal and directly related to the systems’ underlying product models. They are intended to push engineers into using particular design processes fitting structured design methodologies based on the theory of mechatronics (for instance, Kersten, 1995; Hildre and Aaslund, 1995). (See Buur (1990) for a review of mechatronic theory.) MAX (de Vries, 1994) explicitly adopts a language based approach to computer support for conceptual design, in which users construct conceptual designs in functional terms using iconic diagrams and bond graphs. Schemebuilder is essentially a conceptual design system based on a graphical symbolic language for describing machine functions, bond graphs (Sharpe and Bracewell, 1993; Bracewell et al, 1993, 1995). The design system developed at the Cambridge EDC supports automatic functional synthesis of mechanisms from functional requirements (Chakrabarti and Tang, 1996; Tang, 1995). These systems only support non-spatial conceptual design, though Schemebuilder supports spatial layout design after the functional form is decided in detail, by calling out to a CAD system. They are irrelevant to problems requiring creative spatial thinking. The approach we have adopted in the FACADE project (Rzevski, 1995; Stacey et al, 1996, 1996b) is rather different. We regard the formalist analyses of the structures of mechatronic systems that dominate the theory of mechatronics, such as Andreasen's (1980) Theory of Domains, as too complex and too neat for designers to use (Rzevski et al, 1995). Thinking in bond graphs is difficult, and only appropriate for design tasks suited to top-down functional synthesis. Real engineers use concepts that cut across the theorists’ categories. Instead the FACADE project aims to provide visual representations and operations in a suite of design environments for conceptual design in engineering, that enable designers to express conceptual design ideas more freely using relatively informal concepts. The representations designers create directly in its design environments cannot be mapped directly into coherent product models; instead translation modules construct product models using AI techniques to fill in the details left implicit in the designers' actions. However the scope of the project is limited to non-spatial aspects of the conceptual design of mechatronic systems: deciding on the major components of a system and how they are connected functionally in flows of matter, energy and information. Nor do the existing design environments require the use of sophisticated design idea capture languages. Artificial intelligence has an important role to play in many design tasks, by performing algorithmic and backtracking subtasks, managing information for decision support, checking integrity and compliance with requirements, propagating constraints, and so on. AI systems have performed these functions on spatial aspects of designs, and qualitative spatial representations have been used in AI design support systems (for instance Wong and Sriram, 1993). But these systems have focused on later stages of the engineering design process. Their operations require at least a skeletal representation of the design.

3. Multimedia Interfaces for Design Idea Capture Spatial Expressions in Design Idea Capture Languages 5

Conventional interfaces requiring the designer to use screen buttons and menus to select operations are too clumsy for expressing rapid and fluid changes of design ideas. This is especially true when designers want to make rapid mode changes to swap between different categories of operations. The desire to make interfaces less cumbersome has driven the development of a variety of pen-based interfaces, which allow simpler and more natural interaction (cf Buxton, 1994).

3.1. Words and Gestures

Supporting spatial conceptual design requires a multi-media interface making use of voice input as well as graphical input to communicate mode changes and design meanings to the system. Speaking can be done in parallel with drawing, and it communicates a wide range of alternative words much faster than writing them. While the power of speech recognition systems is not sufficient to handle unrestricted natural language, a design support system only requires a limited set of words and legal combinations of words. Sufficiently powerful speech recognition systems are now available. Pattern recognition techniques can identify a wide range of drawn symbols (Yang et al, 1994). The Handwriting Recognition Group at Nottingham Trent University has developed techniques for recognising symbols drawn with a digitiser pen on a digitiser or electronic paper (Welbourn and Whitrow, 1988; Evett et al, in preparation; Powalka et al, 1994, 1995), which enable the communication of design meanings whose referents can easily be determined by their position in the visual representation of the design on screen. The Electronic Cocktail Napkin for architectural design recognises drawn glyphs and shapes, and automatically detects qualitative geometric relationships between shapes (for instance, Gross, 1996). We envisage design idea capture languages comprising drawn symbols that derive part of their meaning - their referents - from their location on the screen, with spoken input for major mode changes; or languages in which drawn symbols are juxtaposed with spoken words contributing part of the meaning of the whole statement. The syntax of such languages can encode meaning in the order in which symbols are drawn, but not in when the words are spoken. Three dimensional hand gestures could supplement or replace pen input of spatial symbols and gestures (for instance, Bordegoni, 1994; Sagawa et al, 1996; Tijerino et al, 1994).

3.2. Graphical and Verbal Input for Spatial Design

The most natural way to express and modify ideas about shapes and spatial relationships is graphically, by sketching, and drawing spatial symbols. But existing computer tools for drawing designs are inadequate as interfaces to intelligent support tools for conceptual design. Most such tools for drawing designs fall into two categories. Paintboxes and sketching tools allow the user to produce sketches without restrictions, but the elements of the sketches don't have any semantics: they do not constitute elements of a coherent representation of a design, nor can they easily be interpreted by an AI system. In an intelligent design support system, the most uninterpreted drawings can do is provide memory aids to human designers; whether free drawing included or excluded is an issue of how best to bias the design process. CAD systems can produce drawings whose elements constitute a coherent machine-understandable representation of a design, but are relatively tedious to use and require or imply a degree of precision that is alien to conceptual design. They are used to redraw and refine designs created in the designers' heads and on paper. The Electronic Cocktail Napkin (Gross, 1996) is based on the view that architectural drawings can be interpreted automatically to a useful degree. Simple geometric shapes and geometric relationships between them play a dominant role in architecture; interpreting engineering sketches is much less tractable because engineering involves a much wider range of shapes and relationships, so we think the automatic interpretation of sketches can be at most a supplement to linguistic expressions of design ideas. There are a variety of programs for enabling the rapid creation of three dimensional shapes. These methods have a role in conceptual design when appearance or the details of three dimensional shape are important, but the systems do not address the need to integrate the representation of shape, movement and spatial and functional relationships. Probably the best approach is to provide methods like van Dijk's (1992) shape network sketching as a supplement to the qualitative methods proposed here, though defining an appropriate semantics for qualitative shape operators applied to precisely specified irregular shapes presents problems. 6 M.K. Stacey

A number of systems have used speech input for creating and changing shapes and geometric relationships, for instance VerbalImage (Heidorn, 1995), SpokenImage (Ó Nualláin et al, 1994) and VIENA (Cao et al, 1995; Wachsmuth and Cao, 1995). Tijerino et al (1994) relate verbal input to three dimensional hand gestures for defining shapes. They provide shape that are useful in spatial conceptual design, and one approach to a language for spatial relationships. But using input very similar to English or another natural language may not make the constraints on design input imposed by the system obvious enough to the user. As Neilson and Lee (1994) show, the interaction between graphic and verbal descriptions of spatial situations is complex and subtle, so a more obviously artificial language has advantages in highlighting the precision and the limits of the statements it makes possible.

3.3. Design Choices for a Design Idea Capture Language

The cost of adopting the design idea capture language approach is in the time needed to learn the language. While immediate usability and ease of learning are overrated virtues for CAD tools used every day by professionals (Gilmore, 1995; Eckert, 1995), conceptual design is an infrequent activity for most engineering designers, so any design idea capture language needs to be small enough and simple enough to be learnt quickly. But keeping a language small limits its expressive power; the language must be expressive enough for designers to design with it without finding themselves unable to express essential forms and relationships. The most obvious approach is to support the most important design meanings; and enable designers to create generic attributes and relationships of each major type, labelled with uninterpreted phrases, as notes about design features beyond the scope of the system. The use of speech input inevitably means the use of words taken from a natural language, some of which have a range of meanings, when the design system only supports one precise meaning. As we see in section 4.1, natural language spatial expressions can have complex and subtle meanings when design idea capture language spatial expressions require simple and precise meanings. While a number of systems use pure speech input for geometric information, we hypothesise that a design idea capture language using symbols and isolated words, and so more obviously artificial, will be more appropriate for engineering. But it will be harder to learn initially. The range of natural language words that can be used in a design idea capture language is limited by the power of the system to recognise them. If the words are written with a keyboard, recognition is only a problem of what to do with typos, but the available speech and handwriting recognition systems can only cope with relatively limited vocabularies in the difficult circumstances likely when a design tool is used for designing. So designing an artificial language is a four way tradeoff between the expressive power of the language, the potential for inappropriate use of words overloaded with everyday meanings, the potential for errors caused by overstretching the technology for recognising speech and handwriting, and the ease with which the system can be learnt. Resolving these issues requires experimentation with different options, and evaluation by practising designers.

4. Spatial Expressions in a Design Idea Capture Language

In order to determine what spatial expressions are required in a design idea capture language for spatial conceptual design, we need to study designers in action, both by observing them and by analysing the meanings they express in sketches and schematic diagrams. So far no observational study has concentrated sufficiently closely on this aspect of design. But we can base a preliminary language specification on a consideration of what spatial and functional relationships are important in the structure of machines. In this paper we are concerned with properties and relationships defining spatial structure; an implemented language would need to include characteristics, actions and relationships describing other aspects of how a machine functions. A design idea capture language needs to include three types of spatial expression, which we discuss in sections 4.3, 4.4. and 4.5: changes to objects, principally shape changes; statements of the features of objects (principally movements, which may be shown on screen by action symbols); and relationships between objects (which may be shown on screen by relationship symbols). Non-spatial design actions fall into the same categories. Changes to objects differ from other types of expression in that their consequences must be worked out immediately, and where necessary displayed on screen; mutually inconsistent changes to objects require an immediate resolution of the conflict. Of course, what Spatial Expressions in Design Idea Capture Languages 7

is a change to an object and what is a relationship between objects can be a matter of perspective, if an object at one level of detail is decomposed into a set of objects at the next level of detail. But this matter of perspective is important for the behaviour expected of a design support system and the ways in which is used.

4.1. Some Insights from Cognitive Linguistics

We can gain important insights into the types of spatial meanings a language for describing design ideas is likely to need from linguistic studies of the use of spatial expressions in natural language. Making a clear division between spatial relationships and functional relationships is a tricky business. As Garrod and Sanford (1989) and Coventry (1995) point out, studies of the meaning of prepositions like in and on show that some relationships between objects that appear at first sight to be geometric are in fact functional. As a rough characterisation, in denotes a containment relationship; and on a supporting relationship. (Giving a fully satisfactory characterisation of the meaning of spatial prepositions is a horrendously complex undertaking. For instance Brugman (1988), Lakoff (1987) and Brugman and Lakoff (1988) outline nearly 100 different kinds of uses of over, grouped around three prototypical senses, which Brugman and Lakoff (1988) call the above and no contact schema, the by way of above schema, and the covering schema. The meaning of words like over and how they are represented in the mental lexicon remains a controversial issue in cognitive linguistics (for instance, Coventry and Mather (in preparation).) What this work tells us is that attempts to relate spatial language to the spatial world that rely on specifying purely geometric relationships between objects in visual scenes are unlikely to be successful (Coventry, 1995). If we are designing artificial languages, we can avoid some of these murky issues by defining a clear semantics for the terms we include. But the functional relationships denoted by words like in and on are vitally important to how we understand and reason about the three dimensional world, so languages for describing the spatial characteristics of machines will need to include them. We have to go beyond quantitative or qualitative geometry, even for describing static structures. Some natural language expressions are topological in character, and can only be interpreted in the context of the surrounding spatial situation, such as near, beside, behind and next to (Edwards and Moulin, 1995). For example, what near means depends on what other objects are relevant to the comparison. Is Stanford University near the hotel? Is Stanford University near Berkeley ? Many of the important relationships between the components of machines concern how they affect each other's behaviour. In mechanical engineering, the actions of machine parts to cause or prevent the movement of other parts are often vitally important. Talmy (1988) points out that there are striking similarities in the words and grammatical forms people use to describe how things interact with respect to force, in physical and psychosocial domains and also intrapsychic domains like psychoanalysis. He argues that these common linguistic forms reflect important commonalities in the mental models people have of the actions of things on each other; and argues further that force dynamics is a neglected fundamental semantic category in grammar. It subsumes causality as a linguistic category. Talmy (1988) analyses linguistic forms according to whether the agent's tendency is to move or be stationary, whether the agent or the antagonist is stronger, and whether the antagonist's influence is constant, begins or ends, and so on. Identifying a set of basic force dynamic relationships provides a useful starting point for representing the relationships between the actions of different components in a machine, and for defining a grammar for such relationships in a design idea capture language. How many force dynamic relationships are required by engineers is an open research question. Talmy's work provides one set of hypotheses to explore. Finding satisfactory spatial or functional definitions for important natural language relationships is not always straightforward. For instance, the complementary action of the spatial preposition along is very similar to guides considered as a force dynamic relationship - an issue for the cognitive linguists. The word through is an especially tricky case, and one relevant to engineering. One important lesson the designers of artificial languages need to learn from the experience of their predecessors is the need to keep the important things simple. In the seventeenth century, philosophers including Leibniz among others sought to invent universal languages, comprising terms representing the elementary concepts and relationships; so that all knowledge could be expressed clearly and unambiguously (cf Slaughter, 1982). They found that far from clarifying things, statements in their languages soon became impossibly verbose. Simple relationship words in natural languages like English 8 M.K. Stacey denote important types of relationship that turn out to be quite complex when decomposed into conceptually pure primitive elements. Artificial intelligence researchers learned the same lesson by describing the content of statements in English in the Conceptual Dependency notation developed by Schank (1975). The argument put forward in this paper rests on the hypothesis that the set of spatial and functional relationships required by engineers is small enough to make tractable the development of a design idea capture language with clearly defined semantics. But this language will need to include macros: words or symbols that denote complex relationships composed of a larger number of primitive relationships.

4.2. Objects in Spatial Conceptual Design

A design idea capture language must deal with the major classes of spatial entities that engineers reason with in spatial conceptual design. The most obvious class are physical components of machines, which are ordinarily three dimensional objects that have particular shapes, though the shape may be undefined. Most of the functional relationships used in conceptual design relate components to each other. In many knowledge representation schemes for product modelling all entities are physical components; for instance DROOL (Stacey et al, 1996b), and the EDC Product Model (Murdoch and Ball, 1994; Murdoch, 1995; Ball and Murdoch, 1995). But when engineers perform spatial design they also need to think about how these components move, and sometimes explicitly about regions of space. We hypothesise that a design tool supporting spatial conceptual design will need to support the expression of ideas about at least the first four types of spatial entity listed here. · Components. Physical components of machines have every type of attribute and relationship. These include shape, though the shape of a component may be undefined. Shapes of components are usually three dimensional, though some objects like sheets of paper are best treated as surfaces. · Axes. In mechanical systems with moving parts, the parts usually move around particular axes of rotation or in particular directions. A design idea capture language for mechanical systems requires ways to define and change the axes of linear and rotational movement. Spatial conceptual design in engineering requires the ability to describe curved axes. A preliminary examination of the problems involved in defining axes leads us to the provisional conclusion that manipulating axes would be made much simpler and easier if all spatial entities are required to have axes. · Movement Spaces. Spatial entities moving along axes, or irregularly, sweep out volumes of space that they can occupy, which we call movement spaces. It is sometimes necessary to think explicitly about movement spaces. Movement paths can be defined as the product of a shape, an axis and a distance; or can be constructed as shapes. · Exclusion Spaces. Regions of space that are conceptually significant but are not physical objects can have their shape defined in the same way as components. · Bounding Surfaces. Design is often constrained by the space into which the designed system must fit. The simplest and most pervasive case is the requirement that an object should sit on a flat surface like the ground. Two dimensional surfaces or the boundaries of three dimensional regions can be used to represent boundaries. · Fields. Some spatial conceptual design involves reasoning with electric or magnetic fields, which are spatial entities with functional and spatial relationships with components, but are not physical objects. What would be an appropriate representation for electric and magnetic fields in conceptual design is an open question. It is unlikely to be sufficiently important to justify the development of special representations or language elements, except in a special purpose tool.

4.3. Shapes

Objects have shapes, which are usually three dimensional, though some physical objects can be treated as surfaces, and axes are one dimensional. In addition to mechanisms for defining spatial and functional relationships, a design idea capture language requires mechanisms for defining and changing shapes. A lot of work has been done on linguistic descriptions of shape changes (see section 3.2), so we won't discuss shape change statements here. In conceptual design, components can often have their shapes left undefined, or be treated as if they had simple shapes. But in some situations, components with complex shapes are best thought of as Spatial Expressions in Design Idea Capture Languages 9

comprising connected regions or features that have shape. In conceptual design these regions are likely to be subcomponents; in detail design, when more complex geometric representations are needed, this may not be true. For instance, The EDC product model decomposes its assembly objects into features, which are geometry-owning regions; a decomposition in terms of geometrical features does not necessarily map onto a decomposition in terms of subcomponents (Ball and Murdoch, 1995). If spatial objects are represented as sets of connected features (permanently or for the purposes of one operation), new features can be inserted into the connections between features, as well as added to the boundaries of the objects. When adding features to object (perhaps a subcomponent such as a hinge or a bend), we might want to indicate that the new elements should be inserted between two existing parts of the object. Similarly we might want to make a change to an object, such as stretching it or bending it, and want the change to apply to one part of the object but not another part. We can divide screen objects into regions by drawing partition lines across them, and apply change operators at the partition, or to the region on one side of the partition line. This requires a representation of the shape or spatial structure of objects as comprising elements with simple connections, such that the existing elements can be connected to the new element rather than each other. This means rerepresenting geometric objects cut by partition lines as connections between their subparts or surfaces. For example, geometric objects like cuboids or cylinders can be rerepresented as connections (with rectangular or circular cross-sections) between their ends.

4.4. Actions

The components of machines perform actions. Many are non-spatial, though they may have significant indirect implications for the layout of the machine. The spatially significant actions are movements and shape changes. The visual representations and design idea capture language of a design support tool must describe movements and (where necessary) shape changes in the same way as the other unary actions of components, so that spatial and functional aspects of design can be handled in a uniform manner. The type of movement (linear, rotary, spiral, irregular) is logically separate from the fact of movement, but the simplest approach to language design is probably to define separate terms for movements of each important type. The distinguishing feature of spatial actions is that they have direction. The syntax for expressing movement and shape change in a design idea capture language must provide for the specification of direction and extent. The obvious directions are along an axis, and along an axis in the reverse direction, or in both directions; or in a different direction shown by drawing a line. (Both direction and extent are optional elements of movement statements, if all objects include an axis that can be used as a default direction. Symbols are required to signal reverse movement or an explicit axis. The extent can be left undefined in conceptual design.) A first collection of terms for a language is move-along-axis, rotate-around-axis, spiral-around-axis, move-irregularly.

4.5. Relationships

Spatial conceptual design involves creating and reasoning about different kinds of relationships besides geometric relationships. Coherent product models require relationships with clear semantics and implications, which may not correspond to the concepts naturally employed by designers. Designing a design idea capture language involves using natural language statements as a source for identifying design meanings that are both important and conceptually clear. Qualitative geometry and force dynamics provide useful structuring principles, as do functional analyses of spatial prepositions. · Spatial Function Relationships. Some functional relationships important for engineering are indicated with spatial prepositions and other standard forms in natural languages. These include contains (in), supports (on), anchors, holds (anchors + supports), surrounds (within), guides (along). · Qualitative Geometric Relationships. A large number of qualitative geometric relations can be defined. Which ones are important enough to include in a design idea capture language is an open question. Obvious candidates are parallel to, perpendicular to, collinear with, concentric with, coplanar with, aligned with. · Topological Relationships. Some topological relationships such as next to are important in engineering. It is unclear whether the topological design meanings required by engineers can be 10 M.K. Stacey

handled simply, as qualitative geometric relationships, or whether they have to be interpreted in terms of topological models of the entire design (as by Edwards and Moulin, 1995). · Force Dynamic Relationships. Some relationship essential to spatial conceptual design concern how the actions of objects affect the actions of other objects. While force dynamic relationships connect actions, the action of an object might simply be being in a particular place: "The fence [being there] stopped the ball from rolling down the hill". So the syntax for expressing force dynamic relations must allow the connection of both the actions of objects and objects themselves; in the latter case the referent of the relationship is the object's primary function, or its being where it is, or is an unspecified action by the object. Some force dynamic relationships particularly important in engineering are causes/makes, blocks/prevents, bounds/limits, powers, controls, guides.

4.6. The Semantics of Lines

Lines can have a number of meanings, depending on what type of expression they are part of, and where they are drawn. While drawing lines is easy, interpreting two dimensional lines in three dimensions requires a choice of convention, perhaps that lines are drawn in the plane of the view and have depth corresponding to the objects they refer to; plus ways to manipulate existing lines. · Relationship Connections. A line drawn from one object shown in the display to another; that is, its startpoint is in the area belonging to one object, and its endpoint is in the area belonging to the other object. The direction of the line indicates the direction of the relationship; a double line indicates a reflexive relationship. Ternary relationships are indicated by two lines, connecting the three objects in the relationship. One-to-many relationships are indicated by drawing lines from the first object to each of the objects it is connected to by the same relationship. · Axes. A line drawn with a unary movement action indicates an axis, as does a line drawn with a special axis creation operation. It is an open question how much curve smoothing one should do to a drawn line; probably in conceptual design it does not matter. For some movements along axes, orientation at start and finish is significant. This can be encoded by the end tangent vectors of drawn lines, but attaching significance to precise values of end tangent vectors is only worth doing if the lines are drawn accurately enough; and if the significance can be encoded symbolically so that it is preserved through qualitative changes. Using end tangent vectors may prove more trouble than it is worth. · Partitions. A partition line is a line through an object, cutting it in half. It passes through the area belonging to one screen object, and has a startpoint and an endpoint outside it.

5. Building an Intelligent Support System for Spatial Conceptual Design

The research strategy we propose is to build a multimedia interface based on a drawing package, capable of accepting spoken input and drawn gesture input, and to use it as a testbed for design idea capture languages. This requires that the interface is connected to a product modelling system capable of representing both functional and physical connections between the components of machines, and spatial relationships between those components. A number of knowledge representation schemes for product models in engineering have been developed, for example the EDC Product Model (Murdoch and Ball, 1994; Murdoch, 1995; Ball and Murdoch, 1995) and SHARED (Wong and Sriram, 1993), besides mechanical modelling schemes such as those of Chakrabarti and Tang (1996) and Dyer et al (for instance, 1987). An obvious choice for us is DROOL, a representation scheme for conceptual designs in mechatronics developed by us at the Open University as part of the FACADE Project, which distinguishes between functional and physical connections, but does not yet include spatial relationships (Stacey et al, 1996b). In this plan the development of design idea capture languages will be guided primarily by studies of how engineers design, including observations of spatial design episodes, analyses of diagrams and sketches, and interviews. The objective of these studies will be to discover what spatial design meanings designers might want to express in single actions, construct a taxonomy and evaluate the parallels to other sets of design meanings employed in non-spatial conceptual design. This study of spatial design meanings will necessarily include a study of the meanings designers employ to create and explain the operation of their machines; spatial and functional terms will probably prove impossible Spatial Expressions in Design Idea Capture Languages 11

to separate, and even an early prototype language will need some terms for describing the functional interactions of the components of machines. The languages are then designed to provide syntax and a set of terms for expressing those meanings as simply and unambiguously as possible. Each language constitutes a set of hypotheses about what meanings are useful to a designer performing spatial conceptual design.

6. Open Research Questions

The mental processes involved in spatial design activities have not yet been sufficiently studied, so further psychological work is needed to extend to design the research done on mental imagery and mental spatial representations and their relation to the spatial meanings people express in language. This is a subject of current research at the Open University (Petre et al, 1995). Beyond this, the development of a prototype intelligent design tool for spatial conceptual design involves the exploration of a number of open issues: · An appropriate selection of ways to create and manipulate shapes in a multimedia interface, ideally combining symbolic descriptions with drawing or direct manipulation techniques. · The set of design meanings essential for the conceptual design of machines. · An analysis of the meanings of important natural language terms like above, through, beside, next to, sufficient to guide the selection of useful precise design meanings. · An appropriate syntax for the design idea capture language. · An appropriate symbol set for the design idea capture language. · The qualitative geometry most appropriate to describing conceptual designs of machines in a coherent product model. Should it include topological relationships ? · The translation mechanisms most appropriate to relating the users' expressions of design meanings to the spatial representations used in the product model.

Acknowledgements

This work has been supported by EPSRC Grant GR/J48689 to George Rzevski, Helen Sharp and Marian Petre at the Open University, for the FACADE Project. It has benefited from discussions about design with Claudia Eckert and the members of the FACADE Project team, and about drawn gesture recognition with Lindsay Evett and Nasser Sherkat at Nottingham Trent University. Marian Petre and Claudia Eckert commented helpfully on earlier drafts, as did the referees. Patrick Olivier at the University of Wales, Aberystwyth, helped us find some significant literature.

References

Andreasen, M.M. (1980). Syntesemetoder på systemgrundlag. PhD Thesis, Technical University of Lund, Lund. Ball, N.R. & Murdoch, T.N.S. (1995). A Layered Framework for Sharing Design Data. Proceedings of the 10th International Conference on Engineering Design. Heurista, Prague, pp. 1471-1476. Bordegoni, M. (1994). An Environment for the Specification and Recognition of Dynamic Gestures. Journal of Visual Languages and Computing, 5, 205-225. Bracewell, R.H., Bradley, D.A., Chaplin, R.V., Langdon, P.M. & Sharpe, J.E.E. (1993). Schemebuilder: A design aid for the conceptual stages of . Proceedings of the 9th International Conference on Engineering Design. Heurista, Den Haag, pp 1311-1318. Bracewell, R.H., Chaplin, R.V., Langdon, P.M., Li, M., Oh, V.K., Sharpe, J.E.E., & Yan, X.T. (1995). Integrated Platform for AI Support of Complex Design (Part I): Rapid Development of Schemes from First Principles. In J.E.E. Sharpe (ed.) AI System Support for Conceptual Design, Springer-Verlag, Berlin, pp 170-188. Briffault, X. (1995). On the use of mental imagery in spatial dialogues. Proceedings of the First Workshop on the Representation and Processing of Spatial Expressions, Montreal. Brugman, C. (1988). The story of 'over': Polysemy, semantics and the structure of the lexicon. Garland Press. Brugman, C. & Lakoff, G. (1988). Cognitive topology and lexical networks. In G.W. Cottrell, S. Small & M.K.Tannenhause (eds.) Lexical ambiguity resolution: perspectives from psycholinguistics, neuropsychology and artificial intelligence. Morgan Kaufmann, San Mateo. Buur, J. (1990). A theoretical approach to mechatronics design. PhD Thesis, Institute for Engineering Design, Technical University of Denmark, Lyngby. Buxton, W. (1994). The Pragmatics of Haptic Input. Cambridge University Press, Cambridge. 12 M.K. Stacey

Cao, Y., Jung, B. & Wachsmuth, I. (1995) Situated Verbal Interactin in Virtual Design and Assembly. Proceedings of IJCAI-95. Morgan Kaufmann, Montreal, pp 2061-2062. Chakrabarti, A. & Tang, M. (1996). Generating conceptual solutions on FuncSION: Evolution of a functional synthesiser. In J.S. Gero & F. Sudweeks (eds.) Artificial Intelligence in Design '96. Kluwer Academic Publishers, Dordrecht. Coventry, K.R. (1995). Spatial Prepositions, Functional Relations and Lexical Specification, in Proceedings of the Workshop on the Representation and Processing of Spatial Expressions, Montreal. Coventry, K.R. & Mather, G. (in preparation) The Real Story of Over. Cross, N.G. (1989). Engineering Design Methods. Wiley, Chichester. Dasgupta, S. (1991). and Computer Science. Cambridge University Press, Cambridge. Dijk, C.G.C. van (1992). Fast surface design basd on sketched networks. SPIE Vol 1830. Curves and Surfaces in Computer Vision and Graphics III. Dyer, M.G., Flowers, M. & Hodges, J. (1986). Edison: An Engineering Design Invention System Operating Naively. International Journal of Artificial Intelligence in Engineering, 1. Dyer, M.G., Hodges, J. & Flowers, M. (1987). Computer Comprehension of Mechanical Device Descriptions. Technical Report UCLA-AI-87-7. Computer Science Department, University of California, Los Angeles. Eckert, C.M. (1995). Intervention Strategies for Critiquing Professional Designers. Open University Centre for the Design of Intelligent Systems Research Report 9504. Edwards, G. & Moulin, B. (1995). Towards the Simulation of Spatial Mental Images Using the Voronoi Model. Proceedings of the First Workshop on the Representation and Processing of Spatial Expressions. Montreal. Evett, L.J., Wilson, M.S., Goddard, T.A. & Whitrow, R.J. (in preparation). An Interface for Pen-Based Applications: Gesture Recognition and Interface Design. Gilmore, D.J. (1995). Interface Design: Have we got it wrong? In K. Nordby, P.H. Helmersen, D.J. Gilmore & S.A. Arnesen (eds.) Human Computer Interaction: Interact '95. Chapman & Hall, London, pp 173-178. Green, T.R.G. (1989). Cognitive dimensions of notations. In A. Sutcliffe & L. Macaulay (eds.) People and Computers V. Cambridge University Press, Cambridge. Green, T.R.G. (1991). Describing information artefacts with cognitive dimensions and structure maps. In D. Diaper and N.V. Hammond (eds.) Proceedings of "HCI'91: Usability Now", Annual Conference of the BCS Human-Computer Interaction Group. Cambridge University Press, Cambridge. Green, T.R.G. & Petre, M. (1996). Usability Analysis of Visual Programming Environments: a 'cognitive dimensions' framework. To appear in Journal of Visual Languages and Computing. Open University Computing Department Research Report 96/01. Gross, M.D. (1996). The Electronic Cocktail Napkin - a computational environment for working with design diagrams. , 17, 53-69. Heidorn, P.B. (1995). Shapes from Natural Language in VerbalImage, Proceedings of the First Workshop on the Representation and Processing of Spatial Expressions, Montreal. Herskovits, A. (1985). Semantics and pragmatics of locative expressions. Cognitive Science, 9, 341-378. Herskovits, A. (1986). Language and spatial cognition: an interdisciplinary study of the preposition in English. Cambridge University Press, New York. Hildre, H.R. & Aaslund, K. (1995). Conceptual Design for Mechatronics. In J.E.E. Sharpe (ed.) AI System Support for Conceptual Design. Springer-Verlag, Berlin, pp 390-404. Kersten, T. (1995). 'Modessa' A Computer Based Conceptual Design Support System. In J.E.E. Sharpe (ed.) AI System Support for Conceptual Design. Springer-Verlag, Berlin, pp 241-259. Lakoff, G. (1987). Women, fire and dangerous things. Chicago University Press, Chicago. Landau, B. & Jackendoff, R. (1993). "What" and "where" in spatial language and spatial cognition. Behavioural and Brain Sciences. 16, 217-265. Murdoch, T.N.S. (1995). Sharing Design Data. Technical Report CUED/C-EDC/TR-28. Cambridge University Engineering Department, Cambridge. Murdoch, T.N.S. & Ball, N.R. (1994). The Development of an EDC Product Data Model. Technical Report CUED/C- EDC/TR-21, Cambridge University Engineering Department, Cambridge. Neilson, I. & Lee, J. (1994). Conversations with graphics: implications for the design of natural language/graphics interfaces. International Journal of Human Computer Studies, 40, 509-541. Ó Nualláin, S., Farley, B. & Smith, A.G. (1994). The Spoken Image System: On the visual interpretation of verbal scene descriptions. In P. McKevitt (ed.) Proceedings of the Workshop on integration of natural language and vision processing. Twelfth American National Conference on Artificial Intelligence, Seattle. Petre, M., Sharp, H.C. & Eckert, C.M. (1995). MIND Mechanisms of Inspiration in Novel Design. Successful grant proposal to the ESRC Cognitive Engineering Initiative, Open University Computing Department, Milton Keynes. Powalka, R.K., Sherkat, N., Evett, L.J. & Whitrow, R.J. (1994). Dynamic cursive script recognition. A hybrid approach. In C. Faure, P. Keuss, G. Lorette & A. Vinter (eds.) Advances in Handwriting and Drawing: a multidisciplinary approach. Powalka, R.K. Sherkat, N. & Whitrow, R.J. (1995). Zoning invariant wholistic recognizer for hybrid recognition of handwriting. Proceedings of the Third International Conference on Document Analysis and Recognition, Montreal, pp 64-67. Rzevski, G. (1995). FACADE: Concurrent Engineering Applied to Multi-Technology Products. Proceedings of the International Workshop on Concurrent/Simultaneous Engineering Frameworks and Applications. Lisbon. Rzevski, G., Buckland, R.A., Petre, M., Stacey, M.K., & Sharp, H.S. (1995). Conceptual Design of Mechatronic Systems. The Open University, Centre for the Design of Intelligent Systems Report 9503, Milton Keynes. Sagawa, H., Ohki, M., Sakiyama, T., Oohira, E., Ikeda, H. & Fujisawa, H. (1996). Pattern Recognition and Synthesis for a Sign Language Translation System. Journal of Visual Languages and Computing, 7, 109-127. Schank, R.C. (1975). Conceptual Information Processing. North-Holland, Amsterdam. Spatial Expressions in Design Idea Capture Languages 13

Schön, D.A. & Wiggins, G. (1992). Kinds of seeing and their functions in designing. Design Studies, 13, 135-156. Sharpe, J.E.E. & Bracewell, R.H. (1993). Application of Bond Graph Methodology to Concurrent Conceptual Design of Interdisciplinary Systems. International Conference on Systems, Man and Cybernetics 1, Le Touquet, IEEE Press, pp 7-13. Slaughter, M.M. (1982). Universal Languages and Scientific Taxonomy in the Seventeenth Century. Cambridge University Press, Cambridge. Stacey, M.K. (1995). Distorting Design: Unevenness as a Cognitive Dimension of Design Tools. In G. Allen, J. Wilkinson & P. Wright (eds.) Adjunct Proceedings of HCI'95. University of Huddersfield School of Computing and Mathematics, Huddersfield. Stacey, M.K., Petre, M., Rzevski, G., Sharp, H.C. & Buckland, R.A. (1996). Beyond Engineering Bias: designing a tool to liberate conceptual design. Open University Computing Department Research Report 96/04. Stacey, M.K., Sharp, H.C., Petre, M., Rzevski, G. & Buckland, R.A. (1996b). A Representation Scheme to Support Conceptual Design of Mechatronic Systems. In J.S. Gero & F. Sudweeks (eds.) Artificial Intelligence in Design '96. Kluwer Academic Publishers, Dordrecht. Talmy, L. (1988). Force dynamics in language and cognition. Cognitive Science, 12, 49-100. Tang, M. (1995). Development of an Integrated AI System for Conceptual Design Support. In J.E.E. Sharpe (ed.) AI System Support for Conceptual Design. Springer-Verlag, Berlin, pp 153-169. Tijerino, Y.A., Miyasato, T. & Kishino, F. (1994). From Verbal and Gestural Input to 3-D Visual Feedback. AAAI-94 Workshop on Integrating Natural Language and Vision Processing, Seattle. Ullman, D.G. (1992). The Mechanical Design Process. McGraw-Hill, New York. Vries, T.J.A. de (1994). Conceptual design of controlled electro-mechanical systems, a modeling perspective. PhD Thesis, Universiteit Twente, Enschede. Wachsmuth, I. & Cao, Y. (1995). Interactive graphics design with situated agents. In W. Straßer & F. Wahl (eds.) Graphics and Robotics, Springer Verlag, Berlin, pp 73-85. Welbourn, L.K. & Whitrow, R.J. (1988). A gesture-based text editor. In D. Jones and R. Winder (eds.) People and Computers IV. Cambridge University Press, Cambridge, pp 363-371. Wong, A. & Sriram, D. (1993). SHARED: An Information Model for Cooperative Product Development. Research in Engineering Design, 5(1), 21-39. Yang, D., Garret, J.H., Shaw, D.S. & Rendell, L.A. (1994). An Intelligent Symbol Usage Assistant for CAD Systems. IEEE Expert, 2, 32-41.