Semantic Interpretation of Architectural Drawings

Section 3 Shape Grammars

Olubi Babalola Georgia Institute of Technology, USA

Charles Eastman Georgia Institute of Technology, USA

Abstract The paper reviews the needs and issues of automatically interpreting architectural drawings into building model representations. It distinguishes between recognition and semantic interpretation and reviews the steps involved in de- veloping such a conversion capability, referring to the relevant literature and concepts. It identifies two potentially useful components, neither of which has received attention. One is the development of a syntactically defined drafting language. The other is a strategy for interpreting the semantic content of architectural drawings, based on the analogy of natural language interpretation Keywords Semantic Interpretation, Drawing Understanding

166 2001: ACADIA Olubi Babalola Charles Eastman Semantic Interpretation of Architectural Drawings Part I: An Overview and Survey and building operation. Yet, given the amount of data archived in paper as well as non-intelligent 1 Introduction CAD formats, it is clear that some means of auto- Semantic interpretation of architectural drawings matically migrating drawings from these formats has the potential for simplifying the process of into intelligent model representations would save conversion of archived drawing information in millions of man hours. paper or old CADD formats into the newer and 1.2 Current view of the drawing recognition intelligent model-based CADD representations problem which posses vastly greater capabilities. The fol- Semantic recognition of architectural drawings lowing section addresses some of the pertinent from paper-based input is a potentially vast re- questions, such as the difference between archi- search area. Small but relevant segments of the tectural and other recognition. process have been separately tackled from the perspective of different drawing recognition do- 1.1 Justification for Research mains. The most relevant effort, tackled from a Interest in extending the functionality of CAD directly architectural viewpoint, is the Knowledge beyond simple drafting and editing through the Based Interpretation of Architectural Drawing addition of model-based design support and system, KBIAD [CJ90] and has a focus somewhat analysis intelligence is almost as old as CAD re- similar to that of this paper. The KBIAD research search itself. Progress in these extensions hinge sought to develop a generalized method for draw- on the evolution of CAD systems from drawing- ing interpretation, and demonstrated this using oriented representations to model-oriented rep- an architectural floor-plan example. A number of resentations, which identify each element in the ongoing efforts have also been directed towards design, associating its geometry and other prop- semantic reconstruction of drawing information erties and relations. A growing number of archi- using layer-separated CAD input. tectural CAD products of this sort exist. While each uses its own internal representation, most of Outside of architectural drawing recognition, these intelligent CAD systems interface with a there is a large body of drawing recognition re- standard building model, for exchange and inter- search from other domains, such as office docu- facing with special applications. A standard build- ments, schematic drawings, and mechanical drawing model is a public-domain open standard for ings. Many issues are common across these do- representing the useful information about a build- mains, especially in the early stages of their re- ing, defined as a composition of objects at differ- spective processes, providing a useful body of rel- ent levels of aggregation. evant research sources, techniques and occasionally ready solutions to borrow. Non-architectural Considerable progress has been made in the de- domains also overlap similarly, and it is conceiv- velopment of a standard building model. The able that discoveries in architectural recognition Industry Alliance for Interoperability group (IAI) may in turn influence approaches in other do- is about to release the fifth version of its Industry mains. Foundation Classes (IFC 2.X), an industry-led effort to develop a standard building model. Simi- The objectives of drawing recognition vary widely, larly, The Standard for the Exchange of Product ranging from the conversion of raster-scanned model data under the International Standards drawings into layered CADD documents, to gen- Organization (ISO-STEP) has developed ex- erating the input for a specific analysis applica- change technologies that have been used to de- tion, to the identification of the represented ob- velop a range of standard exchange models in jects, along with their various properties. Our in- building, steel structures, precast concrete, and terest in this paper focuses upon interpreting non- other industry domains [CE99]. Eventually, re- layered CAD drawings for input to building prod- searchers in the building industry expect that such uct models, which offer a common base for all a standard building model will become the basis other uses. for most work in design, contracting, fabrication

2001: ACADIA 167 Section 3 Shape Grammars

1.3 An overview of issues addressed in the paper tive is to arrive at the beginnings of a theory of This paper has two parts. The first defines the architectural drawing understanding, in the be- various aspects of the drawing understanding lief that such formalization will help in bridging problem and the multiple issues that must be ad- some of the existing gaps in the process. dressed for product model interpretation. A rough 2 Drawing Recognition Input and Process process model is provided both as a means of dem- Model Outline onstrating relationships and a possible sequence 2.1 Detail Description of Architectural Drawings of the various activities, as well as providing some A variety of drawing-types are employed in ar- structure for discussion of our survey of related chitecture at different stages in the design pro- research. The purpose of the survey is to identify cess, ranging from unstructured sketches, to sche- work and concepts specifically relevant to the field matic designs, through to a richer and more for- of architectural drawing recognition, including malized representation used in communicating techniques, principles and/or algorithms. construction information to a contractor [GN97]. The second part of the paper explores how rich The latter, called construction drawings, is the semantics may eventually be extracted from ar- focus of this paper. chitectural drawings, using natural language pro- Architectural construction drawings are graphi- cessing as an analogy. We identify the difference cally complex, depicting an assemblage of sub- between drawing recognition and drawing under- assemblages and parts. A typical architectural floor standing, pointing out the importance of seman- plan for a 3 bedroom residence may run in excess tics in understanding. We discuss some of the of 3500 graphical entities, while a drawing for a major problems involved, drawing parallels with machine part may consist of less than 300. We the resolution of similar problems in natural lan- define complexity here in terms of the number of guage understanding, and in the process identi- geometric entities involved in depicting the ob- fying several useful areas of research. The objec-

Figure 1. Architectural Drawing Samples

168 2001: ACADIA Olubi Babalola Charles Eastman Semantic Interpretation of Architectural Drawings ject, as well as the nature of the spatial relation- · Views are orthographic 2D, 3D parallel projections ship between these entities, such as connections, (isometric or oblique), and occasionally 3D perspective views. overlap, containment and nesting. Figure 1 illus- · Multiple views must be interpreted to derive all 3D trates this complexity with sample architectural shape and detail aspects of the design; Views required floor plans from a construction drawing. for complete object-description typically reside on multiple drawing sheets. We summarize some general properties of draw- · Views represent building assemblages. Distinct isolated ing relevant for the task of interpretation below building parts are seldom represented separately. (some of these properties are graphically presented · A view within a drawing often depicts multiple sys- in Figure 2.) tems (architectural, electrical, structural, MEP) to in- dicate their placement or relationships. As a result, a · Architectural drawings are typically a mixture of pre- view often is a complex overlay of different types of sentation format and semantic content, with a bordered systems. sheet, with a panel detailing general drawing content, version and ownership, individually labeled and scaled · Floor plans are the central organizing view in contract views in each of the sheets. drawings, with sections and other views keyed on the plan. · Drawings consist of a variety of symbol types, ranging from text through more restricted text notation types. Abstract symbols consisting of text, graphic symbols or a combination are used in conjunction with scaled geometry representing physical object parts.

These properties of architectural construction drawings are unique and not exactly matched by drawings in other domains. 2.2 The Drawing Recognition Process A block diagram of the recognition process is illustrated in fig 3. It will be used as a backdrop for discussion of the architectural drawing recognition process, with raster or CADD produced drawings as input, and ending in an interpreted building model representation. Figure 2. Component-level Schematic of Typical Architectural The stages in the drawing recognition process are Drawing broadly separated into low-level and high-level

Figure 3. Drawing Recognition Process

2001: ACADIA 169 Section 3 Shape Grammars recognition respectively. Low-level refers to any lems, that must be controlled and eliminated be- process whose input and output do not carry any fore later stages. Raster pre-processing is applied understanding of what real-world object the to clean up the scanned raster image and applies graphic depicts [DT95]. An alternative taxonomy thresholding to convert the image to a binary-val- describes recognition in terms of a three level hi- ued (black-white) bitmap [KT97]. Other routines erarchy (each level in turn consisting of multiple attempt to eliminate small isolated pixels in the stages). These are lexical, syntactic, and semantic drawing, smooth lines and edges, and sharpen line recognition [KH94, DT95, and PP99]. Lexical intersections [TK94]. Raster pre-processing re- recognition is generally considered as the pro- mains an area of active research. cess through which the basic drawing primitives Vectorization & post processing are derived. In drawing recognition, these typi- The position of the vectorization stage in fig.3 is cally take the form of lines, arcs, circles and text. variable, dependent upon the approach being Dashed lines and hatch lines are additional primi- employed. The output from this process is a sub- tives worth identifying as a single object rather set of the lexical (graphical) entities referred to in than as multiple lines, and could thus serve as an the similarly named recognition phase. They are extension to a basic graphical lexicon or graphlex. basic domain-independent geometric primitives Syntactic recognition processes the graphlex to such as lines, arcs, circles and text and are valid extract complete symbols, through the applica- primitives across most drawing domains. The tion of grammar-based rules that check for local- mechanical drawing recognition domain has also ized spatial relationships between graphical enti- dealt with hatch lines and arrowheads, and we ties. Dimension lines and certain symbols can be propose a few extras for the architectural domain. partially assembled in this manner. Semantics The task is to convert graphical entities from their mainly address process control and the introduc- raster representation into vectors. A variety of low- tion of external contextual knowledge about the level algorithms have been developed in response recognition process. The objective is to incorpo- to this task, with qualified success. In one ap- rate drawing understanding knowledge residing proach, lines and arcs are detected using a 2-stage in the interpreter’s expertise into the recognition process, first involving algorithms that attempt process. The term “high-level recognition” is used to create a centerline through skeletonization, or in reference to approaches utilizing an object by contour tracing algorithms that generate a model to guide the understanding process. In fig. centerline by tracing the outlines of black-pixel 3, the stages from “initial drawing” through to areas, and computing the midpoint of the out- “feature extraction (graphlex)” will be classified lines [AF97, AB97]. The vector representations as low-level processes, while all processes beyond returned vary by implementation, and range from this will be termed high-level recognition. collections of short vectors (x, y) to actual full- 2.2.1 Review of the Interpretation Stages length vector line representations [AF97]. Post- Traditionally, architectural drawings are produced processing is then applied combining segments and archived in a variety of paper- based media. based upon an analysis of relationships. Algorith- The production medium of drawings has a direct mic design decisions arise in post-processing, such impact on the ability to automatically recognize them. The implication is that some production media may always prove unsuitable for computer interpretation [KH94]. Rasterization Scanning is used to convert paper-based architectural representations into a digital raster format. It is usually undertaken in 8-bit grayscale, at 200 dpi or higher. The scanning process can in- troduce skewing, poor contrast and other prob- Figure 4. Post & Pre section symbol

170 2001: ACADIA Olubi Babalola Charles Eastman Semantic Interpretation of Architectural Drawings as whether to return maximal lines or break lines shapes incorporating multiple graphical elements at each intersection. These have potential impli- e.g. drawing symbols as primitives [BR82, BT88]. cations in the later recognition process. Global separation More complex basic entity shapes, including poly- This process, like the next, could precede the gons, arrowheads and hatch patterns, are recog- vectorization process. Its purpose is to separate nized with further post-processing. They are typi- views from each other and from non-view infor- cally recognized using grammar-based techniques mation such as drawing panels and tables. Ap- [KV95]. The problem with using such approaches proaches to separation have been developed in is that it is sometimes difficult to fully predict the the engineering drawing domain [IW83] and in range of malformations that may arise from office document recognition. There has been no vectorization. This makes it difficult to establish architectural application of global separation. The a grammar-based description that is guaranteed challenges in separation of architectural drawings to capture all instances of the sought pattern. A are that multiple views are sometimes so closely variation is a grammar-based genetic algorithm placed in a drawing that the process may join approach, which is somewhat akin to a vector- multiple views together as one, creating confu- based template matching technique, with invari- sion in the subsequent stage. ance to scale, rotation and translation [EC96]. View Type identification and relations among views A third alternative approach maps raster patterns Each segmented view needs to be identified as to to be recognized into an alternative metric space. its projection or view type and what part of the The most common method is through the use of whole project it represents. Plan, elevation or sec- generalized Hough Transforms. Hough trans- tion views depict different information and use forms are generalized template matching systems different symbols and drawing conventions. with invariance to scale, rotation and translation. While in the mechanical drawing domain, efforts Some allow the recognition of simple geometric have been made to automatically associate different mechanical drawing views [KH94], their approach, based mainly on the comparison of dimensional relationships, will not work for architectural drawings. The problems include: · Often, on a given page, different scales may be employed, such as for details and sections · Architectural drawings are not arranged according to standard orthographic convention · Architectural drawings are typically a lot more com- plicated than individual machine parts.

Automatic determination of correspondence between different views thus becomes a different and substantial problem in the architectural domain. There has been little or no work in view identification or segmentation for architectural or related domains. It appears that the plan view plays a narrative role in a recognition process involving multiple views, similar to the role it plays in human drawing interpretation and it seems a logical starting point for machine recognition as well. Plans usually include symbols designating the relation of sections and some details. Figure 5. Segmented Views

2001: ACADIA 171 Section 3 Shape Grammars

Feature (graphlex) extraction: Semantic recognition Feature selection is a major challenge in any rec- Semantics in a formal linguistic sense is defined ognition problem, regardless of approach. The as language meaning independent of the specific domain independent features defined in the ras- wording. In architectural drawings we shall deter preprocessing stage are combined in vector fine semantics as real-object associated meaning, processing into higher-level domain independent hence we consider a building in terms of its func- entities, including hatch patterns, dotted lines, tional properties and structural components, and dashed-lines, polygons and arrowheads. The chal- these alone or in combination will be referred to lenge is to identify lower-level graphical proper- as building semantics. In architectural construc- ties that characterize a domain without semantic tion drawings, the semantics of drawings should knowledge of what the depicting geometry rep- capture associations between elements of the de- resents. In architectural drawing recognition, par- picted entity and the graphical entities depicting allel lines, collinear lines, door and toilets sym- them. For example, window-wall relationships bols may be useful lexical entities. While the ba- should be captured, along with the window rep- sic graphlex entities are typically derived from low- resenting geometry in different views. Also, the level routines, many of the more complex ones relationship of these elements with other associ- are derived from vector entities using context-free ated non-depictive symbols, such as reference grammars or knowledge-based routines in post- symbols and annotation such as window symbols processing operations [AF97, KH94]. An issue must be associated the wall geometry. is the high degree of variability of many symbols, The use of context-free grammars in segmenta- such as toilet, often requiring higher-level pro- tion (typically categorized as bottom-up recogni- cesses to recognize them. tion) is useful in restricting the search space, and Syntactic segmentation may actually be used to find simple symbols ei- We earlier referred to the graphical complexity ther partially or in their entirety (ref.). For ex- that characterizes the architectural drawing rec- ample, a dimension line consists of numerical text ognition domain (Figure 1). In the view separa- coexisting in certain proximal relationships with tion and identification stage, the focus is on lim- line geometry. As far as verifying actual drawing ited regions of a drawing sheet. However, infor- elements in a context-sensitive grammar however, mation relevant to the recognition of an individual some meta-knowledge that is usually difficult to view also must be partitioned, to distinguish the express in grammars is required. In the dimen- type of information it is. This is referred to as sion line example stated above, drawing complex- segmentation. ity makes it possible to associate the wrong set of In an architectural floor plan, even after the more text and lines using simple bottom-up or local- complex basic entities are recognized, a large num- ized context-free rules. It is possible however to ber of simple line and arc geometry remain, such eliminate some of the wrong selections after sub- as dimension lines, grid geometry, symbols, etc. jecting the set to some higher level check. These could just as easily represent a part of wall At an implementation level, the top-down high- geometry or dimension lines. The object here is level knowledge controlling recognition and veri- to separate the possible dimension lines from the fication tends to be distributed. Some implemen- possible wall lines etc. in a segmentation process. tations embody this knowledge in a Knowledge In many ways this segmentation stage involves the Based System (KBS), others in the algorithmic same effect as the layering process used in CADD. flow of the program. Still other object-oriented Without segmentation, the number of lexical en- implementations have incorporated some of the tities is very large and the likelihood of errors is knowledge in object methods (member overwhelming. We are aware of no work in this functions)[CJ90]. area. Our notion of semantics in architectural drawing understanding requires that we read the drawing into a representation that seeks to duplicate a

172 2001: ACADIA Olubi Babalola Charles Eastman Semantic Interpretation of Architectural Drawings human reader’s understanding. As in many AI domains. While considerable work has been car- models of human understanding, this may take ried out in some of the stages, others remain un- on the form of a semantic network [AL95]. The touched, perhaps as a consequence of the nature representation of drawing understanding includes of the domains and focus of the related research. not only knowledge of buildings and the compo- Amongst the drawing domains, Mechanical draw- nent parts and their respective relationships, but ing recognition offers the greatest amount of rel- also knowledge of what the different graphical evant precedence, since both represent 3D ob- entities in the drawing language represent rela- jects with multiple 2D views, using somewhat tive to each other as well as to a real world build- similar drawing conventions. ing. These two aspects must be distinguished, Similarities notwithstanding, there are consider- even though the differences are subtle. Our un- able differences between architectural drawing derstanding of buildings allows us to design our understanding and most of the exiting research high-level checks or top-down processes accord- in mechanical drawing recognition, where most ing to what we know about them in the real world. efforts (save for 1 or 2) have focused on geomet- For example, toilets always exist inside rooms, ric descriptions of individual parts, rather than doors exist within walls, occupiable spaces must the interpretation of assemblages or semantic in- be accessible through openings or doors. terpretation of the depicted objects. Given the These examples of context sensitivity present a input representation (2D drawings) we need an- considerable challenge in parsing. Architectural swers to questions such as: drawings are representations of assemblages, just • How do we deal with the graphical complexity that as sentences are collections of smaller meaning- characterizes architectural drawings? ful parts such as words and clauses. It is the exist- • Is it possible to achieve (partial) semantic understand- ence of certain context-free relationships between ing of a view, or does the entire object need to be si- elements at these lower levels of meaning that multaneously understood? provides some useful insight into tackling the in- • If partial semantic understanding is possible, what are the intermediate representations that will carry the se- terpretation problem. mantic view information? Multi-view integration • Can these partial semantic views be defined such that The process described so far has focused largely they can be integrated in a subsequent process? on a single view. Integration of several views is required to define much of the represented ge- These questions reveal the fact that in spite of ometry. A complete understanding of a floor plan widespread familiarity in the use of architectural will not result in a full understanding of the build- drafting conventions, there is a lack of formal ing, even for a human interpreter. While multi- understanding of the elements in terms of func- view integration is an integral aspect of 3D-re- tional and structural relationships composing and construction in mechanical drawing understand- connecting these. The semantics of the drafting ing, it proves to be considerably more daunting language is not the same as the semantics of the for architectural drawing understanding. Indepen- represented object, and we need an understand- dent understanding of the individual views may ing of the former in order to be able to answer have to precede multi-view integration. There has the questions listed earlier. Simply put, in order been no work in architectural multi-view integra- to read drawings into a semantic model, we need tion to our knowledge. some formal understanding of the architectural drawing language. Summary Architectural drawing understanding consist of This leads us to an examination of issues relating multiple stages and activities. The preceding sur- to the definition of an architectural drafting lan- vey attempted to frame the architectural drawing guage, and searches for analogy from the better- understanding problem, collating some of the wide and disparate body of related research work in this and other related drawing understanding

2001: ACADIA 173 Section 3 Shape Grammars researched field of natural language understanding models. At a more specific level, both domains ing. share several common structural and functional Notes properties. In both cases, understanding exists at 1) A graphical entity in this case is a line, arc, box , text or multiple levels. In NLU, an understanding sys- any other basic visible constituent of the drawing tem can be queried at a somewhat low-level about 2) The stage could alternatively be located after the view actors, actor interactions, events and event chro- identification and segmentation stage. nologies. At a higher level, the same system can 3) In several implementations, both the raster to vector be queried about the moral of a story, which re- and post processing operations appear integrated quires inferences at a more abstract level. Simi- 4) A distinction must be made here between graphical primitives and components of a represented object larly with drawings, at a low-level a drawing un- 5) Annotation symbols, dimension objects, reference sym- derstanding system may be queried about the spa- bols and cross-reference symbols must be associated with tial relationships of physical building elements, real world-object denotations in the drawing, such as or at a higher level about spatial “parti” or “style”. the walls to which they refer Both Natural Language as well as Architectural Part II: Natural Language and Drawings Drafting are context-sensitive. This implies that a small subset of the information can have different possible meanings depending upon the na- 3 Theoretical background ture and meaning of the other information that 3.1 The language/drawing analogy surrounds it, or in other words its context. Both Interpretation of architectural drawings implies systems involve a variety of kinds of knowledge at the very least an ability to recognize building for successful implementation. Parallels can be elements, properties and relationships. Technical drawn between the different types of knowledge drawings are the primary means for communi- needed [AL95]. cating product information between members of the construction team. In this regard technical While the domains are by no means identical, an drawings can be considered as a language of con- analysis of some approaches to text understand- struction communication. Nelson Goodman in ing, including the design of knowledge structures ‘Languages of Art’ [GN97] attempts to identify and process control mechanisms should help in the properties of notational and non-notational establishing a starting point for an architectural symbol systems. He lists the properties of nota- drawing theory. The implementations of most tional symbol systems as consisting of a combi- knowledge representation structures in cognitive nation of specific syntactic and semantic proper- psychology and Artificial Intelligence take on the ties: syntactic disjointness, syntactic infinite dif- form of inter-linked data structures or semantic ferentiability, semantic disjointness, semantic networks. At a functional level, the final repre- dissambiguity and semantic infinite differentia- sentation should be capable of creating new asso- bility. Satisfaction of these properties by archi- ciative links. Understanding in such systems could tectural construction drawings supports the no- be described as different patterns of association tion that architectural construction drawings are links [QM68]. indeed a notational symbol system [VG96]. Schank and Abelson, in their work on text under- 3.1.1 Architectural Drawings Understanding and standing [SA97], propose a hierarchy of knowl- Natural Language Understanding edge structures that attempt to simulate a com- Architectural Drawing Understanding (ADU) is puter representation of the complexities of hu- faced with many problems similar to those in man understanding, including its dynamic and Natural Language Understanding (NLU). The incremental nature. The cognitive emphasis of goal in both domains is the parsing of input to- this work had direct implications in the knowl- kens into understanding structures, specifically edge structures designed, as well as process de- words into semantic structures representing sign implications. meaning and construction drawings into build-

174 2001: ACADIA Olubi Babalola Charles Eastman Semantic Interpretation of Architectural Drawings

Figure 6. Dimension & Dotted Segmentation (Subsets of figure. 1)

3.1.2 Elements in Understanding Systems and sorted according to some practical spatial cri- The input tokens we are interested in here are at teria associated with each element. This resolves word-level. Preprocessing operations may be re- the problem of trying to sort circles and squares. quired in order to arrive at such input. In the case This also provides crucial spatial information to of speech, this presents a problem similar to that guide searches of the concept list, just as linear encountered in drawing understanding, where the order is essential for NLU concept lists. raw input requires considerable preprocessing. A dictionary or lexicon of all words that the recog- 3.1.3 Process control mechanisms in an architectural drafting interpreter: nizer “understands” is an integral aspect of the The semantic interrelationship of building ele- recognition process. It contains instances of word ments suggests a possible paradox in the process types (e.g. nouns, verbs, adjectives, etc.), defined of semantic interpretation. Which object forms as data-structures, with fields for the instance the starting process for drawing recognition, given name and interrelationships with other word that its definition is almost certainly tied to that types. The recognition process consists of plac- of other objects? Text interpretation involves the ing input words matched in the lexicon on a con- parsing of semantic input tokens, and the gram- cept list, instantiating the object methods which matical rules determine when and how these to- attempt to fill the empty fields/slots with some kens should be inter-linked. The differences be- other word on the concept list, guided by seman- tween single-stage semantic interpretation and tic syntax rules. Since the word definitions, like hierarchical interpretation are best illustrated in the design of a data structure, largely determine Figures. 7a and 7b. what is possible in the program, particular attention and their definition is required. 3.2 NLU and ADU: Differences and Similarities: Drawing knowledge representation and recogni- The concept list for drawings will likely take on a tion process control have not enjoyed the level of different form in order to deal with the differ- formalization and development that natural lan- ences in the nature of the input. The concept list guage text understanding systems have. Beyond will carry the various semantic drawing elements, classification of architectural drawings as a lan-

2001: ACADIA 175 Section 3 Shape Grammars

Table 1: Architectural Notational Systems Table 2: NLU / ADU Equivalent Elements

Architectural Drawing Notational Systems Characteristics Sample Members NLU / ADU Equivalent elements Natural language Architectural drawings 1 Textual Phonemes Raw primitives (text and symbols have building object • lines association. either by leader line or by • arcs containment) • circles • natural language text Straight text in natural • room labels • text language. There are • annotation Morphemes Graphlex actually some • Polygons restricted subsets of • Hatch lines this, including room • Isolated symbols areas i.e. ‘2000mm x • Collinear lines 2000mm’ • Loops 2 Symbolic Graphics • etc. • • a) combined text & graphics Objects that do not annotation with Words drawing semantic elements exist in the physical leader lines • drawing symbols world. Graphically • View reference • drawing building elements composed of text and symbols Syntactic constructs (NP, VP, PP) Segmented views geometry in some • Dimensions • possible walls simple relationship. • grid • possible dimension lines b) • non object graphics Real world building • door symbol • opening object geometry elements represented • toilet symbol Predicates (active) Predicates (stative) abstractly atrans • containment 3 Literal graphics ptrans • connectivity • scaled building geometry Directly scaled parts of • walls move • overlap objet geometry. • columns ingest • proximity • slab edges Conceptualizations/Clause Conceptualizations/Clause 4 Inferred symbols implemented as data structure • implemented as data structure • spaces Not explicitly • rooms represented in the • zones drawing, but tational systems that comprise the architectural guage, little effort has gone into defining the spe- drawing language. cific elements of the language and the syntactic 3.3 Syntactic and semantic constructs or other rules guiding the composition of the In seeking appropriate decompositions of archi- component parts into meaningful architectural tectural drawing parts, we must also consider them drawings. What are the structural and functional according to the syntactic and semantic elements relationships between the two languages, and what that compose them. are the equivalent elements in the two domains? Some initial responses to the questions are of- 3.3.1 Syntactic constructs At some level, there is a limit to the kind of syn- fered below. tactic spatial relationships that any two symbols 3.2.1 Formalizing architectural drawing descrip- share. There are connectivity, proximity, containment tions: and possibly other predicates that can define syn- Architectural construction drawings are used to tactic relationships, again irrespective of domain. communicate descriptive information about an Different drawing domains are distinguishable by artifact, existing or proposed, and are composed the reoccurrence of a small group of such pat- of a variety of sub-symbol systems, both depictive terns, which make it easy for a human being to (the “object” graphics and convention symbols) trivially assign a given drawing to its appropriate and descriptive (the various natural language sym- domain. These constitute potential candidate fea- bols). Table 1 identifies some of the types of no- tures.

Figure 7. Single-Stage and hierarchical semantic interpretations

176 2001: ACADIA Olubi Babalola Charles Eastman Semantic Interpretation of Architectural Drawings

3.3.2 Semantic constructs: the system can recognize the physical elements Semantic building constructs share both a physi- of the building, and their spatial relationships. cal part-of relationship, and a hierarchical sequen- Most routine queries of a drawing understanding tial relationship. Cherneff describes this as an in- system occur at this level of reasoning. heritance hierarchy, and represents it in a tree- Table 2 proposes some of comparable structures like structure [CJ90]. Graphical building elements across both domains based upon the preceding in the architectural drawing language such as criteria. doors and toilets, and drawings symbols such as section and datum symbols can be considered as We summarize some of the main differences in the equivalent of words in natural language, since textual and drawing input as follows: they carry semantic information about the de- · Text is a set of 0 dimensional objects in a 1 dimen- picted/denoted object. The dependencies of the sional string. There is a notion of precedence in the semantic parts upon each other need to be deter- input sequence. Drawings on the other hand are laid mined, and may be embedded in the building out in 2 dimensions, and there is no meaningful notion of input precedence. Control structures will differ model representation. From this, appropriate considerably as a result. grammars can be determined and a verifiable un- · Some drawing elements in their initial form are non- derstanding of recognition dependencies can be semantic (lines, circles) and others are (door and win- derived. dow symbols), while textual input strings are words, which carry semantic meaning. Some structural parallels relations between NLU · Textual conceptualizations generally focus on (human) and ADU entities are proposed in Table 2. The actions, while drawings require stative semantic building elements, which are compared conceptualizations, focusing on objects and descriptions to words, are the ones defined in the drawing rec- of states. · Some of the problems of ambiguity in word sense defi- ognition lexicon. nitions (may be) less of a problem in the drawing do- 3.3.3 Challenges in semantic structure design main. The problem of the various levels of understanding and the possible open-ended nature of defin- 3.4 Using the drafting representation model ing these levels becomes apparent as different Having identified possible elements of a drafting notions of semantics come into play. This raises language, we need to integrate the semantics of the question as to whether there is a generalized the language with the object semantics. One way notion of understanding that can be defined a- of representing this would be through the use of priori, either in drawing or language understand- a drafting representation model. This is best de- ing. It can be argued that for drawings, such a veloped as partial models with well-defined level exists, and is the physical-object description boundaries, such as a door model, a window model level of understanding. This is the level at which and a wall model. The partial models can then be integrated into a comprehensive mode, with all the related semantics. This model is useful in several ways. First, grammars for the different semantic elements or parts can be derived from these models, and this can be expressed in formal logic or other means. The modularity of the model makes it easier to focus on the subset of the description that is necessary in the formalization. Secondly, the model if properly developed provides a ready interface to the building model, which is the final representation into which we seek to read drawings. Figure 9 represents a possible module, a part representation of a wall in plan view. Finally, the model can help clarify pro- Figure 8. Partial Wall Model

2001: ACADIA 177 Section 3 Shape Grammars cess issues such as recognition dependencies and process control and knowledge representation semantic orders of precedence. design. 4 Conclusion Finally, it is important to establish a strong frame- Drawing recognition is a broad area, with a lot of work for drawing interpretation research, so that cross-domain interrelationships. Architectural problems are understood and can be addressed drawing recognition is one of the sub domains, with a clear focus as to where a particular prob- and is perhaps most closely related to mechanical lem falls within the larger recognition framework. drawing recognition. Some of the important challenges of this have Architectural drawing understanding should be been discussed in the preceding sections, though seen not only as a geometric recognition prob- further study is necessary before a suitably robust lem, which it is at the lower levels, but also as a framework is arrived at. language understanding problem, hence we wish Bibliography to take a look at research in natural language text [AF97] Ablameyko, S. Bereishik V. Frantskevitch O. understanding. This is an area where a lot of work (1997) Algorithms For Recognition of The Main Engineer- ing Drawing Entities IEEE PP 776-779?? has been done, a similar level of which needs to be done for the drawing language. [AB97] Ablameyko, S. Bereishik V. (1997) Recognizing Engineering Drawing Entities: Technology And Results. IEEE There are obvious differences in the form of the Conference Publication No. 443 PP 15-17 data, hence it is necessary to establishing a greater [AL95] Allen J. (1995) Natural Language Understanding. level of formalization in our understanding of the Benjamin/Cummins Book Publishers Inc. Redwood Cali- drafting language. This may proceed along the fornia. following lines: [BB87] Barton, G. E. Berrywick, R.C. Ristad, E.S. (1987) Computational Complexity and Natural Language. 1. Clear functional identification of all the component sym- MIT Press, Cambridge MA bol systems in the language, partially outlined in table [BR82] Ballard .D and Brown. C (1982) Computer Vision, 1: Prentice-Hall, 2. Semantic modeling of the relationships of these to each other: determination of the grammars for the different [BG98] Bechtel, W. and Graham, G. (1998) A Companion annotation types: to Cognitive Science 3. Syntactic and semantic modeling of the relationship [BS74] Bijl A. Shawcross G (1974) Housing Site Layout between views, including association between graphi- System : Report on System Developed For Scottish Spe- cal representations across views. cial Housing Association & Department of the Environ- 4. Enumeration of semantic building elements such as ment doors and walls represented in construction drawings. [BT88] R. Boyle and R. Thomas (1988) Computer Vision: 5. Partial modeling of each building element. Complete A First Course, Blackwell Scientific Publications, structural (syntactic) descriptions are not possible for [EC99] Eastman, C (1999) Building Product Models (Com- many symbols because of the variability in their repre- puter Environments Supporting Design) CRC Press sentation. Functionally (semantically), there is largely [CJ90] Cherneff, J. (1990) Knowledge Based Interpretation consistence. Also, in spite of variability, there are often of Architectural Drawings : Ph.D. Thesis, Department of some consistent structural characteristics. Syntactic and Civil Engineering, Massachusetts Institute of Technol- semantic descriptions of the parts at multiple aggrega- ogy, Cambridge, MA tion levels are needed. 6. Integration of the partial models into a drafting lan- [CL75] Collins, A.M. and Loftus, E.F. (1975). A spread- guage model, which itself integrates readily into the IFC ing-activation theory of semantic processing. Psychological or step building model. Review, 82, 407-429. 7. Development of a practical process model for the rec- [GK86] Gessima E. Kamal L. (1986) Computer Aided ognition process that takes into account the graphical Analysis of Schematic Drawings Pattern Recognition In complexity of the domain, and simplifies this in some Practice II Elsevier Science Publications. sequence. [VG96] Goel V. (1996) Sketches of Thought, MIT Press, Cambridge, MA. A clearer understanding of the limit of the draw- [GR78] Gonzalez R. Thomason M. ( 1978) Syntactic Pat- ing language analogy is needed, as a guide to both tern Recognition : An Introduction. Addison Wesley Pub- lishing

178 2001: ACADIA Olubi Babalola Charles Eastman Semantic Interpretation of Architectural Drawings

[GN97] Goodman Nelson (1997) Languages of Art: An [SG83] Sakurai, H., Gossard, D. (1983) Solid Model Input Approach to a Theory of Symbols Hackett Publishing Co, Through Orthographic Views. Computer Graphics, Vol 17, Inc No3, July [GV86] Groen, F.C.A.., Van Munster, R.J (1986.) Com- [SA77]Schank, R. Abelson, R. (1997) Scripts, Plans, Goals, puter Aided Analysis of Schematic Diagrams : in Pattern and Understanding: An Inquiry into Human Knowledge Recognition in Practice ll, E.S Gelseman Structures Lawrence Erlbaum Associates and L.N. Kanals eds. Elsevier Science Publishers B.V. , [TK94] Terzides Kostadinos (1994) Computer Aided Ex- North Holland, traction of Morphological Information from Architectural [HQ82] Haralick, R.M. Queeny, D. (1982) Understand- Drawings Ph.D. Dissertation, Michigan ing Engineering Drawings Computer graphics and image [TS93] Tsai, D.H. Shaw, D. (1993) Syntactic Pattern Rec- Processing Vol 20, pp. 244-258 ognition and Aided System for Mechanical Drawing Diagno- [HG86] Hirschtick, J. Gossard, D. (July 1986) Geometric sis. American Society of Mechanical Engineers Winter Reasoning for Design Advisory Systems: Proceedings of Annual Meeting, New Orleans, LA Pub. ASME, New the1996 American Society of Mechanical Engineers York, NY, Vol. 64, pp. 273-280 Computers in Engineering Conference and Exhibition, [WD75] Waltz, D. (1975) Understanding Line drawings of Chicago, IL, pp. 263-270 Scenes with Shadows. In The Psychology of Computer Vi- [HO87] Hofer-Alfeis, J., (1987) Automated Conversion of sion, P. Winston, ed., McGraw-Hill Book Company, New Existing Mechanical Drawings to CAD Data Structures : State York. of the Art, in K. BØ, L. Estensen, P.Falster, and E.A. Warman (eds) Computer Applications in Production and Engineering, CAPE’86, North-Holland Amsterdam, pp. 259-267 [HU79] Hopcroft, J.E. ,Ullman, J.D. , (1979) Introduc- tion to Automata Theory, Languages and Computation, Addisson Wessly [IW86] Iwaki O (1986) Document Recognition System for Office Automation Proceedings 8th ICP [JO89] Joseph, S.H., (1989) Processing of Engineering Draw- ings for Automatic input to CAD. Pattern Recognition, Vol. 22 pp. 1-11 [KH94] Kanugo T. Haralick R. Dori D. (1994) Under- standing Engineering Drawings: A Survey. Intelligent Sys- tems Laboratory Department of Electrical Engineering University of Washington. [KV95] Kiyko V. (1995) Recognition of Objects In Images of Paper Based Line Drawings. IEEE xxxx PP 970-9736 [LR82] Lawson, B.R., Riley J.P.,(1982) A Technique for the Automatic Interpretation of Spaces from Draw Building Floor Plans. In CAD 82: 5th International Conference and Ex- hibition on Computers in Design Engineering, Butterworths, Guildford pp. 663-666 [ML86] Miclet, L., (1986) Structural Methods in Pattern Recognition North Oxford Academic, London [MW87] Mitchell, W., (1987) The Logic of Architecture MIT Press, Cambridge, MA [MS76] Mitchell W. Steadman J. Ligett R. (1976) Syn- thesis And Optimization of Small Rectangular Floor Plans: Environment And Planning B, Vol. 3 PP 37-70 [PP99] Prabhu B. Pande S. (1999) Intelligent Interpreta- tion of CADD Drawings Computer and Graphics Vol 23 PP 25-44 [QM68] Quillian, M. R. (1968). Semantic Memory. In Minsky, M. (ed.). Semantic Information Processing. Cam- bridge, Mass.: MIT Press.

2001: ACADIA 179