Visual and linguistic predictors for the definiteness of referring expressions
Manjuan Duan, Micha Elsner and Marie-Catherine de Marneffe Linguistics Department The Ohio State University Columbus, OH, 43210, USA {duan,melsner,mcdm}@ling.osu.edu
Abstract features like area and low-level visual salience are positively associated with definite referring This study explores the impact of visual expressions as a whole, suggesting that visually context on the conceptual salience of a dis- prominent objects are treated as more conceptu- course entity, using descriptions of how ally salient when we describe them. to find specific targets in cartoon scenes. These results are important for two reasons. Significant positive correlation is observed First, they draw a firm connection between linguis- between larger and more salient objects tic theories of reference which appeal to salience, and definite expressions, whereas more and the low-level perceptual mechanisms from cluttered images are positively related to which salience arises. By doing so, they help to indefinite expressions. Incorporating these situate these theories in the wider context of cogni- findings with other linguistic factors, we tive science. Secondly, there is comparatively lit- build a mixed-effects logistic regression tle research investigating the effect of visual prop- model for predicting referring forms. The erties on referring forms. Most previous research model reaches 62% accuracy. This study on text generation for REG has focused on con- helps us to understand better how physi- tent selection (Krahmer and van Deemter, 2012) cal context, like an image, determines the where several studies have found effects of visual linguistic properties of a discourse. salience. These results suggest that human speak- ers also take vision into account during sentence 1 Introduction planning and realization. When presented with a picture, how do you start We use mixed-effects regression models to an- your description? How will visual factors affect alyze the importance of several visual and lin- the expressions you use? And how do these factors guistic factors in a corpus of visual-scene refer- interact with contextual and discourse features? ring expressions (Clarke et al., 2013). These mod- Answering these questions will help to build a els can be used to predict referring forms on new connection between the visual clues we perceive data. Our classifier achieves 62% accuracy, which from a picture and the particular linguistic expres- is 30% better than the majority baseline, and 6% sions we choose to describe it. It will also facilitate better than a classifier without visual features, referring expression generation (REG) (Krahmer demonstrating its usability for generating contex- and van Deemter, 2012): the task of generating tually appropriate referring expressions for visual natural and contextually proper referring expres- scenes. sions. In this study, we examine the roles played by 2 Background visual features of an object and its visual con- 2.1 Linguistic theory text in determining whether, in a description, it will be mentioned by a simple definite NP, a long, Linguists have proposed many different theo- descriptive definite expression, an indefinite, a ries to account for the relationship between the demonstrative or a pronoun. We find that visual salience of discourse entities and the kinds of re- ferring expressions that can be used to describe cessibility Marking Scale in which she differenti- them. Although different terms, such as topical- ates nominal descriptions with modifiers and those ity, givenness, accessibility, prominence, familiar- without modifiers. In general, expressions with ity or salience, are used, they all converge on one modifiers refer to entities with lower accessibil- point: referring expressions reflect the cognitive ity. For example, short definite expressions denote status of discourse entities they refer to (Prince, discourse referents that are more accessible than 1999; Chafe, 1976; Givon, 1983; Gundel et al., long ones; the descriptive content of long definites 1993; Ariel, 1988; Roberts, 2003). helps to further single out their discourse referents. Several of these theories match the cognitive or These theories are attractive to us because they attentional states associated with a discourse entity make efforts to capture the correlation between and specific linguistic forms of reference. Gundel cognitive status on the one side and linguistic et al. (1993) use the term givenness to illustrate forms of referring expression on the other side. how salient a discourse entity is. They specify a What is generally missing is a fully grounded scale of attentional states corresponding to differ- theory which explains how low-level percepts af- ent forms of referring expressions. The Givenness fect the cognitive status ranking. While it is uni- Hierarchy they suggest is: in focus > activated versally acknowledged that non-linguistic factors > familiar > uniquely identifiable > referential play a role, most research has focused on linguistic > type identifiable. Related to these six cognitive features which can create or indicate a high cog- statuses are the forms of referring expressions that nitive status for an entity: for instance, Grosz et these statuses license: it > that, this, this N > that al. (1995) proposes a ranking scale of grammati- N > the N> indefinite this N > a N. Each status cal roles played by the discourse entities, subject on the hierarchy is a necessary and sufficient con- > object > others, see also (Kameyama, 1986; dition for the appropriate use of a linguistic form. Hudson et al., 1986). Other factors like the dis- For example, a discourse entity has to be in focus tance between the entity and its previous men- to be referred by a pronoun, or a discourse entity tion, the competition from other discourse entities has to be uniquely identifiable to license the defi- and the (in)animacy of the discourse entities have nite expression the N. also been studied as cues to determine the cog- Along the same line, Roberts (2003) proposes nitive status of a discourse entity (Hobbs, 1976; that the use of a definite NP presupposes that that Mitkov, 1998; Haghighi and Klein, 2010). When the NP is familiar (i.e., that there is a correspond- present, these linguistic features are highly influ- ing discourse referent already in the discourse con- ential, often overriding non-linguistic perceptual text), and that this discourse referent is unique factors (Viethen et al., 2011a). But when they are among the discourse referents in the context. She not, less is known about which perceptual features also further differentiates familiarity into strong matter in selecting appropriate referring forms. familiarity and weak familiarity. Strong familiar- ity is reserved for the more commonly assumed 2.2 Referring expression generation notion of familiarity, where it usually involves ex- Both psycholinguists and text generation special- plicit previous mention of the entity in question, ists have examined precisely the case in which vi- while an entity is weakly familiar when its exis- sual information has the greatest influence: one- tence is entailed by the local context. Hence, weak shot referring tasks (i.e., without discourse con- familiarity subsumes strong familiarity but is more text) involving an object in a visual scene. inclusive, including discourse referents introduced Viethen et al. (2011b) analyze a corpus of map- non-linguistically, on the basis of contextual en- task dialogues and find that visual context is not tailments (including perceptually accessed infor- an important factor in deciding content of a refer- mation) alone. ring expression, even for first mentions. However, Ariel (1988; 1991) proposes a similar theory in other studies have found effects for visual features. which the complexity of referring forms reflects Kelleher et al. (2005) claim that salience—both their accessible status in our mind. Basically, more visual and linguistic—is an important overarch- reduced linguistic forms suggest more accessible ing semantic category structuring visually situated or more salient status in the discourse. Based on discourse. They describe a system which uses sim- her empirical study, she proposes a graded Ac- ple measurements of visual salience—bounding box area and distance to screen center—for both different types of NP. Moreover, our one-shot re- language understanding and REG content selec- ferring task provides no linguistic context to begin tion, and find these features are helpful. Duck- with, while the story completion task of Vogels et ham et al. (2010) use a variety of visual and per- al. (2013) provides previous referring expressions ceptual features to select landmarks for computer- for the entities in all experimental conditions. generated navigation instructions. All the research introduced above shows that Clarke et al. (2013) also find a role for visual salient landmarks are more likely to be chosen in features in content selection. (They argue that route description or scene descriptions than less the discrepancy with Viethen et al. might be ac- salient ones and salient objects are more likely to counted for by the stimuli—the images Clarke et be chosen as subject referent, which establishes al. use are more complex.) They find that vi- the important role that visual salience plays in sual properties (salience, clutter, area, and dis- content selection. Both Montag and MacDonald tance) influence referring expression generation (2011) and Vogels et al. (2013) study how visual for targets embedded in images from “Where’s salience affect our choice of concrete linguistics Wally?” books. Referring expressions for large forms, but these studies involve highly controlled target objects are shorter than those for small tar- experimental environments in which perceptual gets, and expressions about targets in highly clut- variables are manipulated in a fairly coarse way, tered scenes use more words. Also, people are so that visual salience can be considered as a cat- more likely to choose large, salient objects which egorical variable rather than a continuum. More- are close to the target as landmarks in relational over, although Vogels et al. (2013) considers the descriptions. choice of pronouns vs NPs , they leave open the Comparatively fewer studies have investigated issue of definiteness: what kind of NP to produce. how low-level visual features affect linguistic In this paper, we reanalyze Clarke et al. (2013)’s forms. Montag and MacDonald (2011) examined data, investigating which visual features of an ob- how visual salience affects the linguistic structure ject in an image or visual properties of the image choice in terms of passive or active voice in rela- as a whole affect people’s choice of concrete lin- tive clauses. guistic referring forms. This study not only re- Closer to our work, Vogels et al. (2013) study veals the effects of various perceptual factors but how visual salience affects the choice of referent also quantifies their relative importance. We show and the choice of referring forms when interact- that both visual characteristics of the referent (vi- ing with linguistic context in two story-completion sual salience and size) and a characteristic of the experiments. They find that visual salience in- image as a whole (clutter) correlate with increased fluences the choice of referent and does so in- use of definite expressions. Furthermore, since vi- dependently of linguistic salience. But visual sual factors have measurable effects on people’s salience does not affect the choice of referring choice of referring forms, then consideration of forms, which are strongly affected by linguistic these factors in referring expression generation salience. They conclude that visual salience has an tasks should be beneficial. influence on the global interpretation of the scene, but does not directly affect the accessibility status 2.3 Visual salience of individual entities— that is, people use differ- The visual salience (Salience) of an object (Toet, ent types of information in choosing a referent and 2011) is a description of how much the ob- choosing a referring expression. ject stands out from the background. Percep- In contrast, we do find effects from visual infor- tual psychologists have developed models of vi- mation on referring form, but nonetheless, we be- sual salience, which typically aggregate low-level lieve our study accords with Vogels et al. (2013). features such as color and contrast, and compare In their study, the two possible linguistic forms the features around each point to those in the im- considered are pronouns and full noun phrases. age in general in order to predict how different the Pronouns are a referring form which is highly sen- point will look from its surroundings. The size sitive to linguistic context, and our results also and central location of an object are also important show they are relatively insensitive to visual ef- (Tatler, 2007). Such models can predict fixations fects; our strongest effects are in distinguishing during scene viewing (Itti and Koch, 2000). Re- Figure 1: An image from our corpus and the cor- responding visual salience map produced by the bottom-up component of Torralba et al. (2006); red indicates high salience scores, blue low salience scores. Under
Table 2: Coefficients learned by the one-vs-all mixed-effects models for predicting referring forms. Significance codes: p-value < 0.001, ∗ ∗ ∗; p-value < 0.01, ∗∗; p-value < 0.05, ∗; p-value < 0.1, ·. The model includes all pairwise interactions, but only significant interactions are shown. The “Def” column shows coefficients for a merged class containing both long and short definites. vor definites (Ward and Birner, 1995). positions like subject position to achieve better Syntactic position: We checked whether the tar- discourse coherence (Grosz and Sidner, 1986). get is directly preceded by any form of to be (Be); As for visual features, we find main effects of whether it is directly preceded by a preposition Area in favor of short definite NPs, against long (Prep) or whether it appears sentence-initially, a definites and strongly against indefinites. This re- proxy for the subject grammatical role (First). sult accords with the Accessibility Marking Scale proposed by Ariel (1988), which uses short defi- 4 Results and analysis nites for more accessible objects, then long defi- nites and finally indefinites. The coefficients from our one-vs-all mixed ef- The results for Salience are smaller, but appear fects logistic regression analysis are shown in Ta- to be similar. Visual salience has non-significant 5 ble 2. The linguistic features generally behave positive associations with both short and long def- as the existing literature leads us to expect. A inites; if both classes of definite are analyzed to- previous coreferent mention has the expected im- gether, the effect reaches significance. We suspect pact on referring forms (Roberts, 2003): pronouns the failure to find it with either subgroup is due and demonstratives are favored as indicated by the to reduced power because of the relatively smaller positive estimate for Coref, whereas indefinites are datasets. Overall the results confirm our hypoth- disfavored (negative coefficient). Indefinite NPs esis that larger and more visually salient objects are positively associated with There and Be. Def- are also perceived as more prominent and tend to inite NPs are positively related to Prep, indicat- be referred to by definite expressions, especially ing that uniquely identifiable discourse entities are short definites. more likely to be the complements of prepositions. Overlap is positively related to indefinite ex- First is positively related to pronouns, which sup- pressions and Distance is positively related to def- ports the hypothesis that back-looking centers like inite expressions. A closer look will show that pronouns tend to appear at linguistically salient these two measures are inversely related; usually, 5We also considered the distance of an object to the center when two objects are overlapped, the centroid dis- of the image, but its effect was not significant. tance between them is short. In other words, Features Accuracy Sig vs. Features Accuracy Sig vs. Baseline (majority) 32.01 Baseline (majority) 51.7 Task features 38.92 ∗∗∗ baseline Task features 55.32 ∗∗∗ baseline Linguistic features 54.68 ∗∗∗ baseline Linguistic features 72.44 ∗∗∗ baseline Visual features 42.19 ∗∗∗ baseline Visual features 55.94 ∗∗∗ baseline Task + visual features 43.30 ∗∗∗ task Task + visual features 56.78 ∗∗∗ task Task + ling features 56.11· ling Task + ling features 73.27 ling Ling + visual features 58.08 ∗∗∗ ling Ling + visual features 74.15 ling Task + ling + visual 62.06 ∗∗∗ ling + visual Task + ling + visual 74.74 ling
Table 3: Prediction results for the different fea- Table 4: Prediction results for the different fea- ture types, with distinction between short and long ture types with short and long definite expressions definite referring expressions. The last column in- combined. The last column indicates whether re- dicates whether results significantly differ (Mann- sults significantly differ (Mann-Whitney U test). Whitney U test). Gold ↓ Proposed → Pron Demo SDef LDef Indef speakers use more definite expressions to refer to objects far from the target of the description, while Pron 454 17 32 62 10 using more indefinites to refer to objects close by. Demo 49 26 28 108 2 Landmarks that are close by can be helpful even if SDef 44 11 398 463 97 they are hard to see (by helping the listener con- LDef 63 3 157 1180 181 firm that they have found the target). But distant Indef 31 0 31 488 1044 landmarks must be easy to find in their own right, and this makes them better candidates for definite Table 5: Confusion matrix for predicted referring mentions. forms. Converging with the findings discussed above, the estimates for Clutter suggest that indefinite ex- curacy from 32% to 55%. Adding visual features pressions are more likely to be used in a more also leads to significant improvement of predicting crowded image. Area also interacts with Clut- results on the top of baseline, linguistic features ter: large objects are more likely to be definite and task-based features, which gives stronger sup- and less likely to be indefinite when the image is port for our hypothesis that low-level visual fea- more cluttered overall. This supports the results tures play an important role in predicting linguis- from linguistic research that indefinites need to be tic forms for referring expressions. Our strongest type identifiable (Gundel et al., 1993) while def- model, using all feature sets together, scores 62%. inites need to have uniquely identifiable referents Table 4 shows the classification accuracies (Gundel et al., 1993; Roberts, 2003). In an im- when short and long definite expressions are com- age where a lot of similar objects crowd together, bined. All three types of features are still signif- many objects, especially smaller ones, will be hard icantly more effective than the baseline majority, to uniquely identify, so speakers may avoid using now definites. However, adding visual features definite references for them. Alternatively, speak- does not lead to a significant improvement on top ers might not be able to easily verify that the object of linguistic and task-based features. This means is in fact unique in the image. combining short and long definite expressions re- Using the predictions obtained from the five duces the prediction of visual features, which sug- one-vs-all logistic regressions, we classify 479 gest visual features are most effective in differen- randomly chosen NPs held out as test data, us- tiating short and long definite expressions. ing the standard highest score strategy. Table 3 Table 5 shows per-category prediction results shows the classification accuracies. We find that for each of the referring forms, cross-validated all three types of features are significantly more over the entire dataset. Most pronouns are pre- effective than a majority baseline (always “indefi- dicted to be pronouns, despite their low percent- nites”). Linguistic features are very robust in pre- age in our data (11%); 16% are labeled as def- dicting referring forms as widely recognized by inites, and less than 2% as indefinites. Very prior research, which itself improve the overall ac- few demonstratives (12%) are correctly predicted, since they are extremely under-represented in our data (4%). However, most of them are predicted as definites (64%) and pronouns (23%). 11% of def- inites are labeled as indefinite, showing that pro- nouns, demonstratives and definite expressions, as a group, share some common features, and our model draws a relatively sharp distinction between this group and the indefinites. Although different cognitive states are proposed Figure 3: Logistic regression lines for proportion in linguistic research as necessary conditions for of definiteness as predicted by area for each of the definite expressions, such as uniquely identifiable 151 speakers in our data (data items shown as col- by Gundel et al. (1993) and weak familiar by ored points). In general, larger area leads to more Roberts (2003), all these theories claim that dis- definite descriptions, but the effect varies across course entities which have higher cognitive status speakers and describing tasks. in the givenness scale, like in focus or activated can be referred to by either definite NPs or pro- forms like pronouns or demonstratives. We ob- sifier achieves 62% overall accuracy, 30% higher serve this predicted overlap in the usage of these than the majority baseline. This study helps us three referring forms in the confusion results. to better grasp the interaction between linguistic Of the remaining errors, we believe many are properties of the discourse and the physical con- due to individual differences between speakers text in which utterances are grounded. In future in terms of visual perception or describing style. work, we hope to incorporate these features into a Inspection of the random intercepts reveals that full-scale surface realization system. speakers vary in the overall proportions of differ- ent referring forms they use. In some cases this Acknowledgements seems to be a matter of style: some people phrase We thank Jefferson Barlew, Alasdair Clarke, Gre- their referring expressions as instructions (“Look gory Kierstead, Hannah Rohde, Craige Roberts for the man standing aside the red truck”), oth- and the reviewers for their useful comments on ers describe (“A man standing...”) and some use earlier drafts of this paper. a telegraphic style (“man, in blue jeans, stand- ing...”). Figure 3 also suggests that visual properties like References “area” have different effects on people’s choice of Mira Ariel. 1988. Referring and accessibility. Journal whether to use definite or indefinite expressions. of Linguistics, 24:65–87. Most subjects (lines curving sharply to the upper Mira Ariel. 1991. The function of accessibility in a left) follow the general trend of using definite ex- theory of grammar. Journal of Pragmatics, 16:443– pressions for larger objects, but a few show weaker 464. trends, or no trend at all. Whether the variance Matthew F. Asher, David J. Tolhurst, Tom Troscianko, is caused by speakers perceiving the image differ- and Iain D. Gilchrist. 2013. Regional effects of clut- ently, or reacting differently to visual factors, de- ter on human target detection performance. Journal serves future study. of Vision, 13(5):1–15. Doug Bates, Martin Maechler, and Ben Bolker. 2011. 5 Conclusion lme4: Linear mixed-effects models using S4 classes. Comprehensive R Archive Network. In this study, we have revealed the correlation be- tween the visual features of discourse entities and Wallace Chafe. 1976. Givenness, contrastiveness, def- initeness, subjects, topics and point of view. In their referring forms. We find visual features like C. Li, editor, Subject and Topic. Academic Press, area and salience are positively related to definite New York. expressions and indefinite expressions are more Alasdair D. F. Clarke, Micha Elsner, and Hannah Ro- likely to be used in crowded images. Based on hde. 2013. Where’s Wally: The influence of these findings, we train a classifier to predict the visual salience on referring expression generation. referring forms for these visual objects. Our clas- Frontiers in Psychology (Perception Science), Issue on Scene Understanding: Behavioral and computa- Megumi Kameyama. 1986. A property-sharing con- tional perspectives. straint in centering. In Proceedings of the 24th an- nual meeting on Association for Computational Lin- Robert Dale and Nicholas J. Haddock. 1991. Gen- guistics. erating referring expressions involving relations. In EACL, pages 161–166. J. Kelleher, F. Costello, and J. van Genabith. 2005. Dynamically structuring, updating and interrelating Matt Duckham, Stephan Winter, and Michelle Robin- representations of visual and linguistic discourse son. 2010. Including landmarks in routing instruc- context. Artificial Intelligence, 167:62–102. Con- tions. Journal of Location Based Services, 4(1):28– necting Language to the World. 52. Emiel Krahmer and Kees van Deemter. 2012. Com- Talmy Givon. 1983. Topic continuity in discourse: putational generation of referring expressions: A an introduction. In T. Givon, editor, Topic conti- survey. Computational Linguistics, 38(1):173–218, nuity in discourse: a quantitative cross-language March. study. John Benjamins Publishing Company, Ams- terdam/Philadelphia. Ruslan Mitkov. 1998. Robust pronoun resolution with limited knowledge. In Proceedings of the 36th An- Dave Golland, Percy Liang, and Dan Klein. 2010. nual Meeting of the Association for Computational A game-theoretic approach to generating spatial de- Linguistics and 17th International Conference on scriptions. In Proceedings of the 2010 Conference Computational Linguistics, Volume 2, pages 869– on Empirical Methods in Natural Language Pro- 875, Montreal, Quebec, Canada, August. Associa- cessing, pages 410–419, Cambridge, MA, October. tion for Computational Linguistics. Association for Computational Linguistics. Jessica Montag and Maryellen MacDonald. 2011. Barbara J. Grosz and Candace L. Sidner. 1986. Atten- How visual salience affects structure choice: Impli- tion, intentions, and the structure of discourse. Com- cations for audience design. In Poster presented at putational Linguistics, 12(3):175–204. the 24th Annual CUNY Conference on Human Sen- tence Processing, Stanford, CA. Barbara J. Grosz, Aravind K. Joshi, and Scott Wein- stein. 1995. Centering: A framework for model- Malvina Nissim. 2006. Learning information status ing the local coherence of discourse. Computational of discourse entities. In Proceedings of EMNLP, Linguistics, 21(2):203–225. pages 94–102, Morristown, NJ, USA. Association for Computational Linguistics. Jeanette K. Gundel, Nancy Hedberg, and Ron Zacharski. 1993. Cognitive status and the form Ellen Prince. 1999. How not to mark topics: ‘topical- of referring expressions in discourse. Language, ization’ in English and Yiddish. In Texas Linguistics 69(2):274–307, June. Forum. University of Texas, Austin. Craige Roberts. 2003. Uniqueness in definite noun Aria Haghighi and Dan Klein. 2010. Coreference phrases. Language and Philosophy, 26:287–350. resolution in a modular, entity-centered model. In Human Language Technologies: The 2010 Annual Ruth Rosenholtz, Yuanzhen Li, and Lisa Nakano. Conference of the North American Chapter of the 2007. Measuring visual clutter. Journal of Vision, Association for Computational Linguistics, pages 7:1–21. 385–393, Los Angeles, California, June. Associa- tion for Computational Linguistics. Ben W. Tatler. 2007. The central fixation bias in scene viewing: Selecting an optimal viewing position in- John M. Henderson, Myriam Chanceaux, and Tim J. dependently of motor biases and image feature dis- Smith. 2009. The influence of clutter on real-world tributions. Journal of Vision, 7(14):1–17. scene search: Evidence from search efficiency and eye movements. Journal of vision, 9(1):1–8. Alexander Toet. 2011. Computational versus psy- chophysical bottom-up image saliency: A compara- Jerry R. Hobbs. 1976. Pronoun resolution. Technical tive evaluation study. Pattern Analysis and Machine Report 76-1, City College New York. Intelligence, IEEE Transactions on, 33(11):2131 – 2146. Susan B. Hudson, Michael K. Tanenhaus, and Gary S. Dell. 1986. The effects of the discourse center on Antonio Torralba, Aude Oliva, Monica S. Castelhano, the local coherence of a discourse. In Program of the and John M. Henderson. 2006. Contextual guidance Eighth Annual Conference of the Cognitive Science of attention in natural scenes: The role of global Society. features on object search. Psychological Review, 113:766–786. Laurent Itti and Christof Koch. 2000. A saliency- based search mechanism for overt and covert Jette Viethen, Robert Dale, and Markus Guhe. 2011a. shifts of visual attention. Vision research, 40(10- Generating subsequent reference in shared visual 12):1489–1506. scenes: Computation vs re-use. In Proceedings of the 2011 Conference on Empirical Methods in Nat- ural Language Processing, pages 1158–1167, Edin- burgh, Scotland, UK., July. Association for Compu- tational Linguistics. Jette Viethen, Robert Dale, and Markus Guhe. 2011b. The impact of visual context on the content of re- ferring expressions. In Proceedings of the 13th Eu- ropean Workshop on Natural Language Generation, pages 44–52, Nancy, France. Association for Com- putational Linguistics. Jorrig Vogels, Emiel Krahmer, and Alfons Maes. 2013. Who is where referred to how, and why? the influ- ence of visual saliency on referent accessibility in spoken language production. Language and Cogni- tive Processes, 28(9):1323–1349. Gregory Ward and Betty Birner. 1995. Definiteness and the English existential. Language, 71(4):722– 742, December. Jeremy M. Wolfe. 1994. Guided search 2.0: A revised model of visual search. Psychonomic Bulletin & Re- view, 1:202–238.