Conjoining Gestalt Rules for Abstraction of Architectural Drawings

Liangliang Nan 1 Andrei Sharf Ke Xie1 Tien-Tsin Wong3 Oliver Deus sen" Daniel Cohen-Or5 Baoquan Chen1

1 SlAT 2 Ben Gurion Univ. 3 CUHK 4 Konstanz Univ. 5 Tel Aviv Univ.

Figure 1: Simplification a/a cOlllplex cityscape line-drawing obtained [(sing o/./ /" Gestalt-based abstraction.

Abstract those of architectural models. Our approach to abstracting shape directly aims to clarify shape and preserve meaningful structures We present a method for structural summari zation and abstraction using Gestalt principles. of complex spatial arrangements found in architectural draw in gs. Thc wdl-knllwn Gcstalt principles hy Wcrthciml:r rI 92:1 ], rdkct The method is based on the well -known Gestalt rules, which sum­ strategies of the human vi sual system to group objects into forms mari ze how form s, pattern s, and semantics are perceived by humans and create internal representations for them. Wheneve r groups of I"rOI11 hits and pi ccl:s 0 1" gl:o l11 l: tri c inl"ormati on. Although defining visual element have one or several characteristics in common, they a computational model for each rul e alone has been extensively s­ get grouped and form a new larger vi sual object - a gestalt. Psychol­ tudied, modeling a conjoint of Gestalt rules remains a challenge. ogists have tried to simulate and model these principles, by find in g In thi s work, we develop a computational framework which mod­ computational means to predict wh at human perceive as gestalts in els Gestalt rules and more importantly, their complex interaction­ images. s. We apply conjoining ru.les to line drawings, to detect groups of objects and repetitions that conform to Gestalt principles. We sum­ The notion of Gestalt is very well -k nown and widely used in var­ mari ze and abstract such groups in ways that maintain structural ious fi elds. In particular, it ex pl ains the tendency of the human semantics by displaying only a reduced number of repeated ele­ vi sual recognition to form whole shapes and forms just from bits ments, or by replacing them with si mpler shapes. We show an ap­ and pieces of geometric information. Naturall y, Gestalt principles plication of our method to line drawings of architectural models of have been used in computer vision, primarily in context with object various styl es, and the potenti al of extending the technique to other recognition and scene understanding. In computer graphics, Gestalt computer-generated illustrations, and three-dimensional models. principles have been applied to a variety of applications, like scene completion [Orori et a!. 2003], image and scene abstraction [Wang et a!. 2004; Mehra et a!. 2009], stroke synthesis [Barla el a!. 2006; Ijiri et a!. 2008] and emerging images generation [Mitra et a!. 2009]. Tn general, these works rely on discrete Gestalt principles, but none addresses the complex interactions emerging from the multitude of Gestalt principles operating simultaneously. A diffi cult problem while dealin g wit.h gestalts is th e conj oin ed ef­ fect of two or more Gestalt principles operating at the same time 1 Introduction on the same site. Modeling gestalts in such cases is especiall y challenging due to the complexity and ambiguity of the scene. Re­ Artistic im agery, architectural renderings, cartography and games cently, attempts to di scover how grouping principles interact were often ex pl oit abstracti on to clari fy, exaggerate, simplify or empha­ made in psychology and computer vision [Oesolneux et a!. 2002; size the visual content. Abstraction is a strategy for communicating Feldman 2003; Cao et a!. 2007; Kubovy and van den Berg 2008]. in fo rmat ion effectively. It all ows artists to hi ghli ght specifi c visu­ These works model limited gestalt interactions, by finding com­ al inform ation and thereby direct the viewer to important aspects putationalmeans which are physiologically pl ausible. Kubovyand of the structure and organization of the scene. In thi s paper, we van den Berg [2008] explore the quantification of perceptual group­ present. an approach to the abstracti on of 20 shapes, in particul ar ings formed conjointly by two grouping principles: similarity and prox imity. Nevertheless, providing general computational means for modeling the interaction of multiple Gestalt principles remains a diffi cult chall enge. In thi s paper, we take a first step in developing a computati onal model fo r conjoining Gestalt rul es. We model a subset of Gestalt rules and their mutual interaction for abstracting architectural line drawings. We choose to focus on architectural drawings since typ­ ically their visual elements are of rather low complexity and their spatial arrangement is strongly biased to the main axes (due to en­ gineering considerati ons). Hence, architectural drawings consist of prevalent similarities, proximities and regul arities among their el- 2

• • • • • • • • • • 0 0 • • • • • 0 0 0 0 0 0 0 0 0 0 • • • • • 0 0 • • • • • (a) (b) (c)

Figure 2: Conjoining gestalts (frOIl! KClIlizsa [1 980]). Overlapping (a): white dots are elemellts oj the grid (regularity) and sim ultaneously belong to a curve (continuity). Conflicting (b): colltinllit)' principle OJ IlliO closed curves (b-Ieft) cO Ilf/iets wilh the symlllell y princip le (b-righl). Masking (c): the basis oj the triangle becomes invisible as it is embedded ill a grollp oj regularly paraliellines.

ements, forming complex grouping conligurati ons that can be de­ ies focused main ly on qualitative and empirical studies, more recent scribed by Gestalt principles and their interaction. These rules de­ psychological works study quantitative aspects of Gesta lt principles fin e vari ous perceptual scene groupings, naturall y lending them­ and their conjoining. Desolneux et al. [2002] and Cao et al. [2007] selves to simplilicati on and ahstraction of architectural drawings. formulate probabilistic quantities for di stinctive Gesta lt rules and utilize them for detecting collinearity, regul arity and p roximity in Interacting gestalts rules cast a difli cult problem sin ce they oper­ images. In hi s work, Feldman [2003] suggests a hi era rchical rep­ ate simultaneously on common sites. They can compete, confl ict, resentati on for modeling proximity and collinearity grouping prin­ overlap, and mask with each other yielding complex vi sual phe­ ciples. He applies a minimal model theory for selecting the best nomena. Up to now, psychologists are stili studying how exactl y grouping interpretation, pre ferring a maximally explained group. multiple Gestalt principles interact. It is not in our scope to study Kubovy and van den Berg [2008] present a probabilistic model for the human visual system under these complex phenomena. Our measuring proximity and similarity grouping quantities. Similar to goal is to quantitatively model and apply conjoining rules with re­ us, they analyze and model the conjoint effect o f grouping princi­ spect to a specific type of data. Specifically, we propose a co m­ ples operati ng simultaneously. Nevertheless, their work focuses on putational f ramcwork to fa cilitate the integrati on of fi ve principal interacti on of only two principles performing on simple dot lattice Gestalt rul es, namely similarity, proximity, contintlity, closure, and confi gurations, while our goal is to provide a generic fra mework fo r regularity, and then apply it to the abstracti on of architectural line resolving conjoining gestalts in the context of architectural abstrac­ drawings. Due to the grouping nature of Gestalt rules, we fo rmulate tion. Claessens and Wagemans [2008] present a B ayesian model abstraction as a grouping optimizati on problem. The additive nature for contour detecti on using proximity and collinearity grouping. In of Gestalt principles interactions (i.e., gestalts combine additively contrast to ours, their method handles simplified inte racti ons be­ as showed in [Elder and Goldberg 2002; Feldman 2003; Kubovy tween two gestalts that perform as independent variables on dot and van den Berg 2008)) is modeled by a multi-label normali zed lattices. In recent work by Cole et al. [2008; 2009], the authors graph cut formulation. study the correlation between shape conveying hand d rawn lines and speei fi e 3D shape reatures. Their work quanti fi cs the strength We apply conjoining Gestalt grouping rules, inspired by the com­ of different line drawings at depi cting shape. putati onal model of Kubovy and van den Berg [2008]. We devel­ op a global energy function that relates grouping strength , number Abstraction and S implification of 20 Content. Gestalt effect is of di sjoint groups and inter-part characteri sti cs to Gestalt princi­ hi ghl y re lated to abstracti on and simplifi cati on. By understanding ples. The minimal energy de fin es a global segmentati on into di s­ how a collecti on of detail s aggregates, we can replace this collec­ joint groups while accounting for metrics that measure grouping in tion with a simpler, coarser or abstract representati on, while pre­ terms of Gestalt rules. The set of e lements grouped is then abstract­ serving the o riginal semantics. In computer graphics and vi sion, ed by means of repl acin g them with simplilied or abstract represen­ Gestalt prin ciples have been studi ed in the contex t of simpli fica ti on tatives. Our method computes a progressive series of abstracti ons and abstraction in several works. We focus here on works which from the fi nest, most detailed drawin g, up to the coarsest most ab­ address simpli fication usin g perceptual principles. stracted one (see Fi gure I).

Our framework naturall y lends itselr to effi cient abstracti on of ar­ In an earl y work, DeCarl o and Santell a [2002] compute image ab­ chitectural line drawings confo rming to perceptual grouping con­ stracti on by preserving meaningful structures using an eye tracker cepts. We demonstrate th al our method succeeds in generating ab­ to assist their im age analysis. In their work, M i et al. [2009] com­ stractions of architectural drawings which mimic the work of skill ­ pute a decomposition o f 2D shapes into parts using shape symme­ fu l arti sts. We also extend our method and show some basic results tri es. Abstraction is achi eved by removing parts according to their on mosaics and 3D bui ldings which are other fa milies of objects size. that consist of di screte elements. The e ffectiveness of our method is evaluated by representing drawings in a range of scales, compar­ A large amount o f work exists on abstracti on o f line draw ings. In isons with previous methods and a user study. the context of perceptual abstraction, Grabli et al. [2004] simpli­ fy line drawings using a complexity measure which accounts for stroke density and regul arity vari ations. Nevertheless, the interre­ 2 Related Work lations between density and regul arity is not addressed . Barla et al. [2005; 2006], present algorithms fo r line drawing simplifica­ Perceptual Gestalt Grouping. Gestalt psychology is a theory on tion and synthesis based on perceptual line grouping, accounting how humans perceive forms ( fi gures or o bj ects) in stead of a co ll ec­ fo r proximity, color and continuation principles. Nevertheless, line tion of simple li nes and curves. Wertheimer [1 923] establishes the stroke pairs are clustered greedily by selecting line pairs that satisfy well -known Gestalt princ iples that dcscribe strategies how hum an one or more principles. Shcsh and Chen [2008] present an efficient vision groups objects into forms. While earl y psychological stud- proximity measure for dynami c linc groupin g and simplifi cation. 3

Perceptual grouping for synthesis purposes has been investigated recently. Ijiri et ai. [2008] study element arrangement patterns by DB 0 0 DB 0 0 analyzing the relations between neighboring elements for the pur­ 000000 0000000 pose of texture synthesis. Hurtut et ai. [2009] perform analysis of element appearance for synthesizing 2D arrangements of stroke­ DOo 0 0 ODDo 0 based vectors. In their work, they measure the appearance of ele­ ...-----:::::::=::=::::::::..:.-::::::::::::::::::::::..-:;:::----- ment qualitatively usin g a statistical model for findin g meaningful appearance features. DB 0 0 Besides abstraction of 2D image content, one can also perform ab­ stracti on and simplification in other domains. Wang et al. [2004] ••••••• perform abstraction of video sequences by semi-automatic segmen­ ODD 0 tation of semantical contiguous volumes. In a recent work MehIa et o ai. [2009] create envelope shapes for complex 3D objects to guide their simplification. Figure 3: COllflicling case I: vertical (left) vs. horiZO lltal (right) regularity gestalls. III top-left slIbjigllre, \'ertical regularity is mask­ Although th ese works address perceptual principl es for simplifica­ ing th e horizol1tal ol1e due to higher density. 111 top-right sulijigure, tion, they do not address the complex interrelations emerging from horizontal regularity overrules the vertical due to an equal density conjoining Gestalt principles using a computational model. although a larger group. Bottom row shows our gmuping result­ s(blue). Abstraction of Architectural Drawings. Our motivation of uti­ li zing conjoining Gestalt rules is for simplification of architectural drawings. Several works deal with abstraction and simplification of DB 0 0 buildings and urban scenes for improving, clarifying and emphasiz­ E§gg ing the urban data. They play an important role in many location­ 000000 0 0 based services, navigation and map generation applications (e.g., DO 0 0 ~ODD tourist maps). 0 0 Sidiropoulos and Vasilakos [2006] explore visualization method­ s for digital city representations. In their work they discuss var­ ious symbolic and realistic representations for urban visualiza­ 0 = 0 0 tion. GrableI' et al. [2008] simplifies building appearance to de­ 0 . 0000 $ g 0 0 emphasize less important buildings and reduce the complexity of 0 . ~ODD tourist maps. Building complexity is measured using rectangular­ 00 0 ity and normals variation while simplification is performed usin g • proximity of facets. Adabala et al. [2007; 2009] compute stylized Figure 4: Conflicting case 2: vertical regu.larity (left) V.I'. pmximity maps and abstractions by straightening edges and modeling a fa­ (right) gestalts. In top-left subjigure, vertical regularity is masking cade detail variation (i.e., windows) using a combination of peri­ proximity due to its stmng regularity. In top-right, pmximity over­ odic "facade waveforms". Similarly, Loya et al. [2008] compute rules vertical regularity due to the stronger proximity. Bollom mw periodic features of building facades using Fourier series and ren­ shows our grouping results (blue). der a reduced pattern to obtain simplification. Tn our work we uti­ lize similar periodic waveform for modeling regular groups in the input. Finally, to improve perception of complex city areas, Glan­ der et al. [2008; 2009] present a hierarchical abstraction in which ~O buildings and streets are merged and removed by their proximity E8ag 0 0 =0 0 0 and size. DODD DoDD Although the above works consider implicitly, perceptual princi­ 0 0 ples for architecture abstraction, our work is the first attempt to explicitly employ conjoining Gestalt ru les for architecture abstrac­ c:flp:O tion. In the next section we describe our method in detail. Section 3 defines the Gestalt principl es and their conjoining interactions. Tn $ g 0 0 -- I I I section 4, we defi ne quantitative measures for grouping elements DODD Do l O to one or more gestalts. Section 5 describes our graph-cut fonnu­ 0 I lation for modeling conjoining gestalts and Section 6 presents our simplifi cation procedures for abstraction of Gestalt groups. Final­ Figure 5: Confliclillg case 3: pl'Oxilllity (lefi) V.I'. shape sirllilarity ly, in Section 7, we present our results, perform a user study and (right) gestalts. Top-right, shape similarity is masking proximity conclude. due to two groups of very similar shapes. 80110111 row shows our grouping results (blue). 3 Gestalt Basics and Interrelations

Gestalt principles describe how humans recogni ze a group of fine • Similarity - parts which share visual characteristics such as elements as a larger aggregate entity. This suggests that we can for­ shape, size or orientation can form a perceptive group. mulate the Gesta lt phenomenon as a grouping problem and solve • Proximity - parts which are closer together can be regarded as for an optimal grouping. We choose to model in this work a subset one group. of Gestalt principles common in architectural drawings: similari­ ty, proximity, continuity, closure, and regularity. These grouping • Regularity - parts which are regularly spaced are seen as be­ principles are defin ed qualitatively in psychology as follows: longing together. 4

Figure 6: Progressive abstractioll of a cOlllplex facade based on conjoining gestaLts. Zoom-im of two different regions demonstrate our preservation of meaningfuL structures .

• Continuity - pre ference for continuous shapes, thus seeing perceived by human. In Figure 7 we show a similar scenario of aligned disjoint elements as one group. cunlli ctin g gestalts in a detailed winduw drawin g .

• Closure - if enough shape is indicated, the whole is obtained by li lling mi ssin g data, thus closin g simple li gures. 4 Quantifying Gestalt Principles

Gestalt principles are stated as independent grouping rul es as they Our input consists of 20 vector drawings of architectural scenes. start from the same building elements. When interacti ons between We de fin e shape-eLements as closed connected poly lines (open grouping rules occur (denoted as conj oining gestalts), the same polylines fo r mosaics examples) which we automatically detect in scene might have different in terpretati ons, which can lead to per­ the drawing. We fi rst compute the spatial relationshi p among ele­ cepti on of someti mes in compati ble groups in a give n fi gure. The ments, so as to construct an associated graph. An ele ment corre­ challenging phenomena of conj oining Gestalt pri nciples were s­ sponds to a node in the graph while edges correspond to the spatial tudied in the seminal work of Kani zsa [1 980] , where conj oining relati onship among elements. We formulate Gestalt principles as gestalts are described as gestalt principl es in an equilibrium, strug­ probability functi ons and compute the probabilities of each element gling tu give the lin al rigure its urgani zati on (see Figure 2). The belonging to one or more gestalts. three cases psychology menti ons are: The gestalts (hence groups) are the labels that are assigned to the nodes. We employ a multi-[abel graph cut (in Secti on 5) to parti ­ I. Ove rLapping: Two grouping principles act simultaneously on ti on thi s hi ghly connected graph structure into optimal and consis­ the same elements and give ri se to two overl apping groups. tent gestalts (groups of elements). Once the set of Gestalt-based 2. Conflictillg: Both grouping principles are potenti all y active, groups is computed, the input detail drawin g is simpli fied by ab­ but groups cannot ex ist simultaneously. Therefore, none of stracting/simplifying elements in the same gestalt and render them the grouping principl es wins clearl y leading to ambiguity as in various styles (Section 6). viewers can see both groupings. Proximity Graph Structure. Our input is a vector drawing P 3. Masking: Two con ni cting groupin g pricni ples compete and consisting of polylines denoted p. Although they can be di sjoin­ one of them wins. The other one is inhibited. t, intersect, or in clude each other (see Figure 6), we assume that a po lylin e represents a unique shape eleillent. We first cOlllpute a There are many empirical studies on Gestalt behavior, but not many prox imity graph G , with each element corresponding to a node in quantitative ones exist. Quantitative study on the interacti on among p the graph. For each element P'i, we fi nd its k-closest neighbors in multiple Gestalt principl es is even scarce [Kubovy and van den the drawing and connect the corresponding nodes in the graph wi th Berg 2008]. Note th at different rules may act at di ffe rent levels edges eii , associated with the Hausdorff-distance between P'i and and may interfere with each other. In our work , we quantify the Pi, defin ed as: affi nity of an clcmcnt to a gestalt accordi ng to quantitati vc mea­ sures. The conj oin ing gestalts interactions are modeled by formu­ lating the spatial relati onshi p among elements as a graph. Our com­ put at ionalmodel accounts for conl1i ctin g and masking in teracti ons and resolves them by rind ing an optimal consistent grouping using and a weighted energy minimi zati on scheme. We currently do not han­ dlc ovcrl apping sin cc computati onall y it is diffi cu lt to diffc rcntiate I'ru m maskin g and cunlli etin g phenumena.

Figures 3 - 5 demonstrate six scenari os in which one Gestalt rul e masks anothe r. Readers are encouraged to hide the bottom rows, and look at the top rows fo r a while to observe the forming of gestalts. [n each input conrig urati on (two for each case), there are at least two potential gestalts th at can be observed, but at the same time, they compete with each other and the gestalt with stronger affinity masks the othe r. We pai r in to three cases, to accentuate turning-points, where a small change in the conligurati on results in (a) (b) (c) (d) a completely diffe rent gestalt. The ill ustrati ons aim to emphasize that th e decisions taken can be quite complex, invo lvin g proximi ty, Figure 7: III a willdow (a) from Figure 6 cOlljoining gestaLts com­ regul arity and simil ari ty. Our graph-cu t soluti on resolves conl1 ict­ pete by: similarity (b), vertical regularity (c ) and horizolltal regu­ s (colored bottom rows in Figures 3-5) and mimics the gestalts as larity (d) which is the winlling gestalt. 5 D

Figurc 9: An example of closure and continuity gestalts. RANSAC de tects a circle that is filled to til e fish eye and defines a strong clo­ sure gestalt. Similarly, coutinuily gestalls occur al Ih e tail defin ing continuous lines. Figu rc 8: A simple illustration of our graph construction. An element P is connected to its neighbor q by edges defining th e s­ moot/mess term Vp ,q. We compute potential gestalts denoted here Regularity gestalts are computed by detecting regular structures in by I f, If, I f where I !.i denotes regularity type gestalt and I :, the scene. We defi ne the regul arity as a group of elements (larger proximity. An element is assigned to potential gestalts using a data th an 2) which are positioned at regul ar intervals a long a certain di ­ cost D( q, j), measuring the penalty of assigning label I to element rection. Although sophisticated symmetry analysis [Liu et al. 2004] q. can be employed, we reduce our search space to 10 regul arities along X and Y axes following the Manhattan-world assumption, commonly applied to architectural models. We perform a 20 fre­ where Vi and Vj are verti ces in P i and Pj, respectively. This distance quency decomposition of the input scene into axis aligned verti­ is assigned as the weight of the edge. Figure 8 shows a simple cal and horizontal dominant frequencies similar to [Adabala 2009], example of our graph construction. Specifica ll y, we subdivide the 20 image into a set of hori zontal and verti cal non-overlapping tiles and compute the I D F FT for Quantifying Affinity. Initially, we loosely de fin e potential graup­ each tile. We lil ter low magnitude frequencies thus obtaining a set ings based on Gestalt rules, which will serve as graph labels in our of dominant verti cal and hori zontal regul ar candidates correspond­ graph-cut formulation. For each potential group, we quantify the in g to hi gh magnitude frequencies. We defin e regul arity potenti al a ffinity of an element Pi to it by computing a data term that mea­ gestalts as: sures the probability of Pi to belong to a gestalt (i.e. being assigned with that label). Additionally, we prioritize labels and assign each label a cost by its grouping strength measuring the al'finity in met­ ri cs we de fin e below. Next, we de fin e the potenti al groups. Proximity gestalts are computed by detecting groups of connected where ~( p i) is a detected dominant horizontal or vertical frequency elements in Gl' with edge lengths below a threshold t p • Thus, a proximity group is defin ed as: magnitude of element P ·i . Closure gestalts refers to a group of elements that forms a simple shape. Continuity is a special case in which elements lie on a line or curve. We compute closure potential groups by fittin g simple ge­ ometri c primitives 0, such as straight lines (for continuity), circles We observe that elements may intersect or enclose each other (for and squares (for closure) to the scene elements. For each primi­ example, a small window inside a door). Since proximity relations ti ve type, we fit it to drawing element s using R ANSAC and group arc undefin ed in such cases, we avoid con sidering such elements together elements with a hi gh fitting score as: in one proximity gestalt. I. e. , for enclosed elements, we do not consider proximity outside their enclosing element. We enforce thi s by automatically detecting and storing intersecti on and inclusion Lc = U{p;} I fi t(pi , f)) > te , relati ons between shape elements in the drawing. Similarity gestalts are computed by detecting groups which share where fitO measures the fi lting quality of Pi to () by counting the a high shape similarity. Since we mainl y focus on architectural number of points (i.e. polyline vertices) which are within an c: dis­ e lements, it was suff'i cient to measure similarity by comparing the tance from e. See Figure 9 for an example of closure gestalts, by aspect rati o of bounding boxes of elements. required, complex If fittin g a circle (top zoom) and straight li nes (bottom zoom) to poly­ shape similarity metrics such as the transformation-invari ant shape line verti ces. The resulting fitted verti ces are grouped together and context [Belongie et al. 2002] can be employed. A similarity group simplified as w ill be described 'in section 6. is defin ed as : With the above de finiti ons, we form many potential groups (gestalt­ s) from the input drawing, each corresponding to a di stinct label. - U{· .} I R (H i, H j ) + R(Wi , Wj) L 5 - P'"PJ 2 > t" , Obviously, an element may belong to several gestalts (labels) even i by the same Gestalt rule (in Figure 8 labels I{l, Id ). Thus, the scene is over-segmented into groups which possibly interact. We where W i and Hi are width and heights of element P'i; and use graph-cut to resolve interactions and achieve a consistent seg­ mentati on of the scene into groups with minimal energy. During our alb if a < b, graph cut optimizati on, potential groupings may break into separate R(a,b) = { bla oth erwise. parts due to contli cts. 6

Figure 10: A sequence oj abstraction steps. We color-code corre.ljJonding element groupings to visualize the computed gestalts. Two abstraction operatiollS are peljormed, (a) sUllllllarization by reducing railings number injences, and (b) embracing by replacing window elements with enclosing object. Althollgh railings alld doors overlap, their illteraction is solved as railings are grollped together by reglliarity gestalt.

5 Conjoining Gestalts via Graph Cut • Continuity and closure label w st is measured by the fillin g quality, defin ed as the distance varian ce of group members to Resolvin g conjoining gestalts in a scene is equi va lent to findin g a fitted geometric primitive: hi = pE L o var (fit(pi ' 0)) consistent segmentation of elements into groups which comply with Gestalt rules. We formu late the problem as a multi-label normalized Smoothness Cost Smoothness term measures the spatial cor­ graph cut minimization. As an element p can potentially belong to relation of neighboring elements. Elements with a smaller di s­ many gestalts, it gets assigned data cost for different gestalt labels tance have a much hi gher probability to belong to the same gestalt (colored nodes in Figure 8). Given n elements, k labels and n . k lh an those di stal1l ones. This is defint:d in our energy minimiza­ costs, finding the minimal assignment is a combinatorial problem tion scheme as the smoothness term. Between two neighboring and typically NP-hard. In stead, we follow Delong [2010] and et al. elements p and q, the smoothn ess energy term is defined by the use an approximate multi-label graph-cut energy minimization. inverse Euclidean Hausdorff-distance between p and q (normalized We compute an ass ignment of labels fp to elements p EP such by mapping min-max to 0 - 1 respectively): that th e joint labeling f minimizes an objective function E (f). Our function consists of three terms: data, smoothness, and label costs. V;", = d(p,q) - l Kubovy and van den Berg [2008] showed th at proximity group­ in g strength decays exponentiall y with Euclidean distance. We ex­ plored severalmetrics for quantifying gestalts and have found them Data Cost. Data cost D(p, fl' ) measures how well an element p to behave very similar in presence of conjoining gestalts. We select fit s to a gestalt fl" (normalized by mapping min-max to 0 - 1). Al­ simplified gestalt metri cs which improve our energy minimizati on though theoreti cally each element and potenti al gestalt defin e a data convergence. Without loss of generality, we assume all label costs cost, we take only elements within a threshold from each gestalt. to be normalized and bounded. This does not affect the solution since elements too distant from a gestalt will not be grouped together. We de fin e the data cost for each Gestalt type as follows : Label Cost Label cost penalizes overly-complex models and fa­ vors the explanation of the input scene with fewest and cheapest Proximity data cost is simply defi ned as the closest di stance of the labe ls. The label cost fun ction is defined as: element p from the proximity group Lp de fin ed as:

F;;OHt = L hi . 61 (f) D(p, f p) = min d(p ,q) q E Lp I E L with L being the set of labels, hi a non-negative label cost of label Similarity data cost is defin ed as the average shape similarity di s­ 1 and 610 an indicator fun ction: tance of p to elements in the simil arity group L s,

3p : fl' = I, D(p, f,,) = I;sl L {R(H" , Hq) + R(WI" W'/ )} otherwise (JE L S

"In the roll owin g, we define hi as the label cost measuring the gestalt Regularity data cost is defin ed as the distance from the regul ar pat­ affinity for each specifi c Gestalt rul e (normali zed by mapping min­ tern defined by Ln. Given a regular pattern, we compute the di s­ max to 0 - 1 respectively): tance d(p, p') of p from the ideal element p' that perfectly ali gns with the regular pattern. • For proximity gestalts, label cost is measured by the inverse density defi ned as the area difference between the uni on of shapes and their convex hull: hi =l,ELp C H(p) - U (p) D(p, f l') = d(p,p') • For simi larity gestalts, label cost is de fin ed by shape simi ­ larity variance against an arbitrary shape within the gestalt: COllfinuity (closure) data cost is measured as the closest distance or an element to the fitted geometri c primitive. Given a continuity hi =(l'"l)j)ELS var(R(Hi , Hj ) + R(Wi' Wj )) group L e which defines a fill ed geomctric primitive, d' (p , *) is the • Regularity label cost is measured as the inverse density mul­ closest distance of p to the primitive defined by L e. tiplied by the elements di stance variance from the perfect fre­ quency pattern and inverse number of elements: hi = I'ELR (CH(p) - (p)) x var(p, 0 x II Lnl1 I 7

Optimization. Hence, the overall energy fun ction is:

E(1) = L D(p, f) + L V;"q + L hi ·61 (1) p E P l E L (a) (b) (c) Cd) (e) Fi nding a solution to thi s labeling problem is optimized using a mul ti-label normali zed graph-cut algorithm as proposed by Delong Figure 11: Gestalts are abstracted by creating a simple represen­ et al. (2010]. Theoretically, we can compute data costs D(p, J) for tation for the overall form. Here we show progressive grouping each p and I, usi ng the complete set of poss ible groupings. Howev­ (from left to right) by regularity, proximity and similarity. Since ab­ er, thi s would be too large and in stead use the thresholding of the straction i.l· govern ed by an LOD t/ireslwld, wefirst simplify smaller gestalts, alld later larger ones. formed gestalts (tl , , t." t,,) to limit data cost computation to only group members.

with di ffe rent abstracti on styles. We defin e abstrac ti on styles in a Iterative Conflict Resolution. As menti oned in Section 3, con­ content dependent manner. flicts in a scene occur when partial gestalts are hidden by other par­ tial gestalts and give equivalent or better explanation of the scene. Abstracting Architectural Drawing. We propose two types of The natural outcome of our graph-cut minimization is a segmen­ operators, embracing and sllmmarization, for abstracting architec­ tation into minimal cost gestalt assignment. Thus, when gestalts tural drawing. Embracing replaces elements in a gestalt by a si m­ compete due to conflicts, naturall y the strongest gestalt in terms of plified enclosing shape. Summarization represents the repeated ele­ its energy terms wins. Thus, the model clearly complies with the ments in a gestalt with a small er number of repeated elements. Our conflicting andmGsking phenomena. choice of abstraction operator depends on the number of repeated Still, our graph-cut assigns labels to overruled groups resulting in elements in a gestalt. partial gestalts. We detect such assignments and remove them. To If a gestalt contai ns more than tk repeated elements (in our im­ do so, we evalu ate the graph-cut assignment by measuring the total plementation tk = 20), we apply a summarization operator which data cost term of each group before and after graph-cut. If there is gradually reduces the number of elements. Otherwise gestalts are a large drop in data cost, it means the group has been overruled abstracted using an embracing operation. We use the convex hull of by another gestalt and we denote it as invalid and mark its ele­ the elements in a gestalt for its e mbracing. We observe that archi­ ments as unassigned. Note that overruled gestalts may still create tectural drawings typically consist of dominant horizontal and ver­ new (smaller) gestalts by themselves and therefore we repeat graph­ tical directions denoted as Manhattan World alignment. Therefore, cut computation iteratively. Tn each iteration we compute potential if an enclosing convex hull edge is within a threshold of J0 degrees groups, perform graph cut and detect valid gestalts. Next we re­ with vertical/horizontal direction, we align it with that direction. move the valid groups from the graph and repeat the whole process on the remaining elements. We stop when no new gestalts can be Figures 6, and 10 demonstrate progressive simplifi cation sequences formed. usin g the two different ways of abstracti on. The fin e de tail s on the door and windows are abstracted using embracing, while the fences Tn conf1i ct cases, where two gestalts equally compete on common are abstracted with summarizati on operation. elements, graph cut will choose one rule arbitrarily. The phenome­ na of both groups breaking together seldom occurs si nce graph-cut Figure II shows a window from Figure 10 being progressively sim­ minimizes the number of di stinct labels and data cost, thus prefer­ plified, where convex hulls and ax is ali gned boxes are used for em­ ring large low cost groups. bracing. This interesting example demonstrates the effects of dif­ ferent Gestalt principles during abstrac tion. From (a) to (b), the Figures 3- 5 shows confli ctin g element confi gurations (upper row) rules of proximity and similarity take part in the upper part of the and our optimization results (lower row). Tn Figure 3(top), two can­ window, while the rules of regularity and similarity take effect in didate vertical and horizontal regularity gestalts compete with each the lower part. The rule of regul arity continues to take effect from other as they share a common element. In the left example, the ver­ (b) to (c). From (c) to (d), the rules of similarity and proximity tical gestalt wins as its regularity is denser thus hav ing lower label take part, and fin all y in (e) the rule of proximity groups the half cost. On the right, the horizontal gestalt wins since both vertical disk with the box. For abstraction, the embracing operator applies and hori zontal gestalts have equal density but the hori zontal gestalt convex hulls (upper window) and boxes (lower window) for the el­ has a higher element number thus a lower label cost. The colored ements. elements in the low rows of the fi gure show our computed gestalt. Tn Figure 4(right), elements congregate together, leading to com­ petition between rul es of (vertical) regularity and proximity. Our Level-of-Detail To gem:rate progressively simplified results. we method selects the proximity gestalt (lower right), due to the su­ can repeatedly apply the gestalt computation in a progressive man­ periori ty of proximity. If we change the shapes of elements as in ne r. That is, we take the current computed gestalt simplification Figure 5(ri ght), the previously computed gestalt in Figure 5(lower result as input elements, and apply the gestalt simplificati on a­ left) is split due to the superiority of simil arity. gain. This process repeats until reaching the desired level or no new gestalts arise.

6 Visual Abstraction Gestalts computation does not expli citly account for progressive level-of-details although locall y, using our label cost formul ation, From the computed gestalts, we can apply different abstraction denser groups are superior to sparser ones. Hence, we can obtain methods to achieve different styles. Unlike previous abstraction gestalts that are arbitrari ly large. In order to achieve a level-of-detai l techniques in non-photoreali stic rendering, our gestalt-based ap­ hie rarchy we defi ne a threshold parameter 0 :::; tLOD :::; 1 and limit proach decouples the iden ti fica ti on of abstrac ting regions from ab­ forming gestalts th at contain neighbor elements p, q with di stance straction styles. In other words, the same gestalt can be presented d(p,w) ~ iLOD . Dbbo,", where D/,bo:c is the drawing's bounding B

box. fin e buil din g detail s with curved structures and thus is not limited to grid-based rectangul ar architecture. We can synchroni ze the level-oF-detail with screen resolution by measuring the D b//O,,; in screen space coordinates denoted by D~"o x In order to measure the effectiveness of our method we have run a and definin g koo = ..,;L-. Thus, our coarsest resolution is thorough user study. In this study we have asked a group of 200 Dbbo:l: users to view simplifi ed drawings and choose the one th at best rep­ achieved when D ~box = 2 pi xels. As tLOO increases, larger resents the ori ginal input. For that, we have built a test case contain ­ gestalts are permitted to form and hence coarser level-of-detail s. ing twel1ly arehitectural urawin gs simplified by our me thou, hand­ Figures I , 13 and 15 show progressively simpli fie d examples in draw n by an artist and a straightforward geometri c simpl ifica ti on reducing scale. Note that the core stru cture is preserved and stil ­ technique (in Figures 12 (b), (d) and (e» . We compare equiva­ I clearly apparent even when the building is signi fica ntl y scaled lent simpli fica ti ons. by measuring the amount of geometri c detail down. and choosing drawings with similar amount of detail. As expected, users preferred our simplifi cati on over the straight-forward geomet­ Abstracting Mosaics In architectural drawings, the rul es of ri c approach with 74% vo tings for us against 26% for the geometric proximity, similarity, and regularity dominate, while the rul es of approach. We were surprised to di scover that our method has done continuity and closure seldom apply. To demonstrate the effect of better even against the artist with 66% votings for us again st 34% these rul es, we ex tend our applicati on to non-architectural content. fo r the artist. In particul ar, we focus on mosaic art since arti sts cleverly positi on Sin ce our method simplifies architectural drawin gs while preserv­ and orient the di screte tiles to exhibit the structure through con­ ing meaningful structures, we demonstrate in Figure 14 an immedi­ tinuation and cl osure effec ts. By computing gestalts, we can ex­ ate application of our method for computing thumbnail directori es tract the structure, mostl y due to rules of continuity and closure. with increased vi sual perception. F igure 15 shows a building being Starting from a set of colored tiles given as polygonal elements, we zoomed out fromlefl to ri ght. Note how the scaled-down simplified cluster them by color similarity. In a clu ster, we connect between versions preserve the major features and structures, and hence, can centers of neighbor tiles yieldin g a set of disjoint poly lines. Fig­ serve as representative icons of the original buildings. In Figures ure 9(left) shows our in put polylin es ex trac ted fro m a fi sh mosaic I, 6, II , 13 and 15 we render results using an NPR sketchy styl e (in Figure 16(left». for vi sual emphasis of abstracti on. We include all results usin g a We show the continuity and closure gestalts obtained at di fferent clean rendering style in the supplemental materi al. level-of-detail s on the fis h mosai c. Polylines representin g di fferent Fin ally, in the mosaic example (Figure 16), the continuity/closure til e clusters get grouped together mostly by continuity and closure gestalts pl aya dominant ro le in the simplificati on sequence (see gestalts. Primitive shapes as eirek s and straight lines are filled to also Figure 9). Here, a circle is fi tted to the fis h eye during the the polyline elements yielding closure and continuity gestalts re­ closure gestalt computati on. The strength of this gestalt preserves spectively. For abstraction purposes, we compute for each gestalt a the fi tted circle even after several iterati ons of simplificati on. representing polyline by averaging vertex positions and a width that encloses the gestalt elements. We draw the new lines with average color of elements inside the gestalt. 8 Concluding Remarks

In this paper, we have proposed a framework that models sever­ 7 Results al prominent Gestalt principles and competition/conflicts among them. The framework is formed in a computational model that is We have applied our gestalt-based abstracti on on a variety of archi ­ reali zed with graph cuts. The effecti veness of thi s framework has tectural drawings, exploring the behav ior of our model in presence been demonstrated in summarizin g and abstracting architectural el­ of complex conjoining gestalts. In all our experiments we have ements, with extended applicability to other objects, illustrated on used the foll owing thresholds of potential gestalts (see 4): prox­ mosaics arts. imity t l ) = 10 pixels, similarity ts = 0.8, regularity frequency The presented work represents still a first attempt to modelin g the magnitude t,. = 0.7 and closure fi lling t" = 5 with t: = 2 pixel­ s. The average computa ti onal time for a coarse-to-fin e abstrac ti on complex interaction among multiple Gestalt principles. We will sequence is 3 minutes and max imal time is 5 minutes. continue to henefi t fi ndings fro m the psychological domai n. The proposed computational framework can be easily adapted to con­ In Figure 12, we evaluate our method by comparing our fi rm to psychological findin gs and ex tended to model other Gestalt result again st several manual and automatic techniques on behav iors. We believe that tools have be built for this framework the Taj Mahal drawing. To normali ze the compari son, we and can be used to facilitate psychological experiments and leading bounded all simplificati ons by thc same amount of geomet­ to new fin di ngs. ric detai l as measured by th e amount of present line geom­ While in this paper we demonstrate how our Gestalt analysis ben­ etry. In Figure 12(a) is the result of a professional hand­ efi ts abstracti on fo r the computer graphi cs purpose, naturally the drawn abstracti on from a website th at teaches drawing ab­ proposed method can also be effective for scene understanding in stract buildings (http://lI c.howstuffworks .com/family/how-to-draw­ general , and further for bettering visual communication. buildings6. htm). In Figure 12(b), we asked an amateur artist to draw a simpli fied version whil e maintaining important structures. Figure 12(c) is the result by the technique of Shesh and Chen [2008] Acknowledgments We thank the anonymous reviewers for their and (d) shows the result or appl ying proximity based simplificati on valuabl e suggesti ons. This work was supported in part by NSFC, of geometric elements. Fin all y our result is in (e). Both ours (e) and 863 Program, CAS One Hundred Scho lar Program, CAS Vi siting the expert artist si mplificati on (a) preserve the maj or structures an d Professorship for Seni or Internati onal Scientists, CAS Fellowship are comparable in term s of the visual quality. Nevertheless, Shesh fo r Young Intern ati onal Scientists, Shenzhen Science and Technol­ and Chen [2008] (c) and geometri c simplificati on (d) use low-level ogy Foundati on, China Postdoctoral Science Foundati on, Israel Sci­ di stance metri cs for abstracti on and fa il to preserve the importan­ ence Fo undati on, European IRG FP7, Lynn and William Frankel t high-level structures as in ours. Figure 13 shows a detailed se­ Center for Computer Sciences, Tuman Fund, Hong Kong RGC quence of our abstracti on of the Taj Mahal. Our method can handle General Research Fund and CUHK SHIAE Project Fund ing. 9

Figure 12: Comparing various abstraction techniques on the Taj Mahal (see Figure 13). (a) shows a professional hand-drawn abstraction, (b) is a hand-drawn abstraction. by an amatellr artist, (c) shows stale-of -tll. e-art simplification usillg Shesh and Chen [20081, (d) is geome/l )' simplificatioll by proximity and (e) sholl's 011 1' gestalt-based abstraClioli result.

Figure 13: A sequence of gestalt based abstractions on the highly-detailed Taj Mahal.

BARLA, P. , BRESLAV, S. , THOLLOT, J ., SILLlON, F. X., AND .. .•mr...... • :..r:sM.:::::.~=. .... h ..-.U ...... ~ •• (J ~ MARKOSIAN, L. 2006. Stroke pattern analysis and sy nthesi s. .-'­­ Compo Graph. Forum 25, 3, 663- 671. 11~ ...... - 110< .. BELONG IE, S., MALIK, J. , AND PUZICHA, 1. 2002. Shape match­ ing and object recognition using shape contexts. IEEE 11'tlllsac­ tions on Pattern Analysis and Machine Intelligence 24, 509- 522.

CAO, E , DELON, J., D ESOLNE UX, A., MUSE, P., AND SUR, E 2007. A unified framework for detecting groups and app lication to shape recognition. f Math IlIlaging Vis 27, 91 - 119.

CLAESSENS, P., AND WAG EMANS, J. 2008. A bayesian frame­ work for cue integration in multi stable grouping: Proximity, collinearity, and orientation priors in zigzag lattices. foumalof Vision 8, 4, 33: 1- 23. Figure 14: An architecture directol)' is illustrated using thumbnails a/simplilied drawings ji'om our method. COLE, E, GOLOVINSKIY, A., LiMPAEC HER, A., BARROS , H., FINKELSTEIN, A., FUNKHOUSER, T., AND RUSINK IEWtCZ , S. 2008. Where do people draw lines? In ACM SIGGRAPH 2008 papers , ACM, I- II. References COLE, E, SANIK, K., DECARLO, D., FINKELSTEIN, A., FUNKHOUSER, T., RUS INK IEWICZ, S. , AND SINGH , M . 2009. ADABALA, N., VARMA, M., TOYAMA, K ., AND BAN GALORE , How well do line drawings depict shape? In ACM Transactions S. N. 2007. Computer aided generation of stylized maps. Compo Oil Graphics (TOG), vol. 28, ACM, 28. Anim. Virtual Worlds 18, 133- 140. DECARLO, D., AND SANTELLA, A. 2002. Stylization and ab­ ADA BALA, N. 2009. Building represe ntation in oblique-view maps straction of photographs. In ACM Transactiolls all Graphics of modern urban areas . Cartographic fournal46, 2, 104-114. (TOG), vol. 21, ACM, 769- 776.

BARLA , P. , THOLLOT, J., AND SILLION , E X. 2005. Geometric DELONG , A. , OSOKIN, A. , ISACK, H. N., AND BOYKOV, Y. clustering for line draw ing simpli fication. In III Proceedings of 2010. Fast approximate energy minimization with label costs. the Eurographics Symposium on Rendering. In CVPR, 2173- 2 /80. 10

Figure 15: Our method models the conjoining gestalts and correctly groups horizontal and vertical window structures andformationsfrom a complex building facade.

Figure 16: Progressive gestalt abstraction oj a fish mosaic. The major curve structures are identified and preserved through abstraction.

DESOLNEUX, A., MOISAN , L., AND MICHEL MOREL , J. 2002. KANI ZSA, G. 19S0. Grammatica del Vedere. Gestalt theOlY alld Computer Vision . KUBOVY, M. , AND VAN DEN BERG, M. 200S. The whole is DRORI , I., COH EN- OR , D., AND YESHURUN, H. 2003. equal to the sum of its parts: A probabilistic model of grouping Fragment-based image completion . In Proc. ACM SIGGRAPH by proximity and similarity in regular patterns. Psychological 2003, 303- 312. Review 1.

ELDER , J. H., AND GOLDBERG , R. M. 2002. Ecological statis­ LIu, Y., COLLINS , R . T., AND TSIN , Y. 2004. A computational tics of gestalt laws for the perceptual organization of contours. model for periodic pattern perception based on frieze and wall­ Journal of Vision, 4. paper groups. IEEE Trans. Pattern Anal. Mach. Intell. 26, 3, 354- 371. FELDMAN, J. 2003. Perceptual grouping by selection of a logically minimal model. International JOllrnal of Computer Vision 55, I, LOYA, A., ADABALA , N., DAS , A. , AND MISHRA , P. 200S. A 5- 25. practical approach to image-guided building facade abstraction. In Computer Graphics International 2008. GLANDER , T. , AND DLLNER , J. 2009. Abstract representations for interactive visualization of virtual 3d city models. Comput­ MEHR A, R. , ZHOU , Q. , LONG, 1. , SH EFFER , A., GOO CH, A. , ers, Environment and Urban Systems 33,5,375 - 3S7. AND MITRA , N. J. 2009. Abstraction of man-made shapes. ACM Transactions on Graphics 28, 5, 137: 1- 137: 10. GLANDER , T. , AND DOLLN ER, J. 200S. Techniques for general ­ izing building geometry of complex virtual 3d city models. In MI, X., DECA RLO , D. , AND STONE, M. 2009. Abstraction of2D Advances in 3D Geoil!formation Systems, Springer, P. van Oost­ shapes in terms of parts. In Proceedings of the 7th Internation­ erom, S. Zlatanova, F. Penninga, and E. M. Fendel, Eds., Lecture al Symposium on Non-Photorealistic Animation and Rendering, Notes in Geoinfonnation and Cartography, 3S1-400. ACM , 15- 24. GRABLER , F, AGRAWALA , M. , SUMN ER, R. W., AND PAULY, MITRA , N. J. , CHU, H.-K., LEE , T.-Y. , WOLF, L., YESHURUN , M. 200S. Automalic generation of tourist maps. ACM Trans. H. , AND COHEN-OR, D. 2009. Emerging images. ACM Trans­ Graph. 27 (August) , 100: 1- 100: II. actions on Graphics 28, 5 (December), 163: 1- 163:S. SHESH, A., AND CHEN , B. 2008. Effi cient and dyn amic simplifi­ GRABLI , S., DURAND , F. , AND SILLlON , F 2004. Density mea­ cation of line drawings. Comput. Graph. Forum 27, 2, 537- 545. sure for line-drawing simplification. In Pro ceedings oj Pa cific Graphics. SIDIROPOULOS , G. , AND VASILAKOS , A. 2006. Ultra-real or symboli c visualization? the case of the ci ty through time. Com­ HURTUT, T. , LAND ES, P., THOLLOT, J. , GOUSS EA U, Y. , puters & Graphics 30, 2, 299 - 3 10. DROUILLHET, R., AND COEURJOLLY, J. 2009. Appearance­ guided synthesis of element arrangements by example. In WANG , J. , XU , Y. , SHUM , H.-Y. , AND COHEN , M . F 2004. Proceedings of the 7th Intel'llationa/ Symposium on Non­ Video looning. ACM Trans. Graph. 23 (August) , 574-5S3. Photorealistic Animation and Rendering, ACM, 51 - 60. WERTH EIM ER, M. 1923. Untersuchun gen zur lehre der gestalt ii ,. IJlRI , T. , MECH, R., IGARASHI , T. , AND MILLER , G. 200S. An Psycho!. Forsch., 4, 30 1- 350. Translation published as Laws Example-based Procedural System for Element Arrangement. In of Organi zation in Perceptual Forms, in: W. Elli s, A Source Computer Graphics Forum, vo l. 27, Wiley Online Library, 429- Book of Gestalt Psychology, Routledge and Kegan Paul , Lon­ 436. don, 1935,pp. 71 - 88.