Generating and Resolving Vague Color References Timothy Meo1 and Brian McMahan2 and Matthew Stone2;3 1Linguistics, 2Computer Science, 3Center for Cognitive Science Rutgers University 1New Brunswick, NJ 08901-1184, 2Piscataway, NJ 08854-8019 [email protected] Abstract us to use empirical methods to induce semantic We describe a method for distinguish- representations on a wide scale from multimodal ing colors in context using English color corpus data (Roy and Reiter, 2005; Steels and Bel- terms. Our approach uses linguistic the- paeme, 2005; McMahan and Stone, 2014). ories of vagueness to build a cognitive We present our ideas through a case study of model via Bayesian rational analysis. In the color vocabulary of English. In particular, particular, we formalize the likelihood that we study the problem solving involved in us- a speaker would use a color term to de- ing color descriptors creatively to distinguish one scribe one color but not another as a func- color swatch from another, similar color. In our tion of the background frequency of the model, these descriptions inevitably refine the in- color term, along with the likelihood of terpretation of language in context. We assume selecting standards in context that fit one that speakers make choices to fulfill their commu- color and not the other. Our approach ex- nicative goals while reproducing common patterns hibits the qualitative flexibility of human of description. Using corpus data, we are able color judgments and reaches ceiling per- to quantify how representative of typical English formance on a small evaluation corpus. speakers’ behavior a particular context-dependent semantic interpretation is. 1 Introduction Our model naturally exhibits many of the pref- A range of research across cognitive science, sum- erences of previous work on vague descriptions. marized in Section 2, suggests that people negoti- For example, the system avoids placing thresh- ate meanings interactively to draw useful distinc- olds in small gaps (van Deemter, 2006), that is, tions in context. This ability depends on using in regions of conceptual space that account for lit- words creatively, interpreting them flexibly, and tle of the probability mass of possible interpreta- tracking the effects of utterances on the evolving tions. In such circumstances, the system prefers context of the conversation. We adopt a computa- more specific vocabulary, where interlocutors are tional approach to these fundamental skills. Our more likely to draw fine distinctions (Baumgaert- goal is to quantify them, scale them up, and eval- ner et al., 2012). Our approach realizes these ef- uate their possible contribution to coordination of fects by simple and uniform decision making that meaning in practical dialogue systems. extends to multidimensional spaces and arbitrary Our work extends three traditions in computa- collections of vocabulary. tional linguistics. Our approach to semantic repre- We begin the paper by describing the semantic sentation builds on previous research that empha- representation of vagueness in dialogue. Vague- sizes the context dependence and interactive dy- ness, we assume, is uncertainty about where to namics of meaning (Barker, 2002; Larsson, 2013; set the threshold in context for the concept evoked Ludlow, 2014). Our approach to pragmatic rea- by a term. Speakers have the option to triangu- soning builds on work on referring expressions late more precise thresholds by interactive strate- and its characterization of the problem solving in- gies such as accommodation, and this helps ex- volved in using vague language to identify entities plain how vague descriptions can be used to refer uniquely in context (Kyburg and Morreau, 2000; to objects precisely (van Deemter, 2006). van Deemter, 2006). Finally, we take a perceptu- In Section 3, we describe our model of speak- ally grounded approach to meaning, which allows ers’ decisions in conversation. We focus on speak- ers that aim to distinguish one thing from another; in these cases, we assume speakers aim to choose a term that’s interpreted so that it fits the target and excludes the distractor, while matching broader patterns of language use. We show how to combine the ideas in Section 4. We formalize the likelihood that a speaker would use a color term to describe one color but not an- other as a function of the likelihood of selecting standards to justify its application in this context, along with the background frequency of the color term. We describe an implementation of the for- malism and report its the qualitative and quan- Figure 1: A brown dog and a tan one—or a tan titative behavior in Section 5. It works with a dog and a white one (Young et al., 2014). generic lexicon of more than 800 color terms and reaches ceiling performance in interpreting user self (Ludlow, 2014). And we won’t consider the color descriptions in the data set of Baumgaertner way multiple descriptions constrain one another, et al. (2012). While substantial additional research as in Figure 1, although we expect to explain it is required to explore the dynamics of vagueness as a side-effect of holistic interpretive processing in conversation, our results already suggest new (Stone and Webber, 1998). We see our work as a ways to apply generic models of the use of vague prerequisite for the model building and data col- language in support of sophisticated, open-ended lection required to address such issues. construction of meaning in situated dialogue. In our view, the users of Young et al. (2014) 2 The Linguistics of Vagueness are using tan to name color categories. Colors are visual sensations that vary continuously across a Figure 1 shows an image from a public data set space of possibilities. Color categories are clas- developed to study how people label images with sifiers that group regions in color space together captions (Young et al., 2014). One user chose to (Gardenfors,¨ 2000; Larsson, 2013). Color terms distinguish the dogs by calling one brown and the in English also have another sense, not at issue in other tan. Another distinguished the dogs by call- this paper, where they refer to an underlying prop- ing one tan and the other white. Each used the tan erty that correlates with color, as in red pen (writes dog to refer to a different dog—yet the way each in red ink) (Kennedy and McNally, 2010). described the other dog left no doubt about the Empirically, color categories seem to be con- correct interpretation. This variability and context vex regions (Gardenfors,¨ 2000; Jager,¨ 2010)—in dependence is characteristic of vagueness in lan- fact, we model them as rectangular box-shaped re- guage. The dogs in Figure 1 are borderline cases; gions in hue–saturation–value (HSV) space. Thus, there’s no clear answer about whether they are tan color categories involve boundaries, thresholds or or not, and speakers are free to talk of either, both, standards that delimit the regions in color space or neither of them as tan, depending on their pur- where they apply; context sensitivity can be mod- poses in the conversation. eled as variability in the location of these bound- In this paper, we explore the descriptive vari- aries (Kennedy, 2007). For example, when we cat- ability seen in Figure 1. How is it that speak- egorize the lighter dog of Figure 1 as being distinc- ers can settle borderline cases in useful ways to tive in its color, we must have a color category that move a dialogue forward, and how can hearers rec- fits this dog but not the darker one. This category ognize those decisions? We won’t consider the will group together colors with a suitable interval interactive strategies that interlocutors can use to of yellow hues, suitable low levels of saturation, confirm, negotiate or contest potentially problem- and suitably high values on the white–black con- atic descriptions, although that’s obviously crucial tinuum. We can think of this category as one pos- for successful reference (Clark and Wilkes-Gibbs, sible interpretation for the word tan. By contrast, 1986), for coordinated meaning (Steels and Bel- categorizing the darker dog of Figure 1 as distinc- paeme, 2005), and perhaps even for meaning it- tively tan involves choosing a category with dif- ferent thresholds for hue, saturation and value— dog. The next section explains how to formalize thresholds that fit the color of the darker dog but the reasoning involved in assessing these proba- exclude that of the lighter one. bilities, reviews one instantiation of this reasoning When interlocutors use vague terms in conver- for learning semantics, and develops another in- sation, they constrain the way others can use those stantiation for distinguishing colors in context. terms in the future (Lewis, 1979; Kyburg and Mor- reau, 2000; Barker, 2002). For example, if we hear 3 Rational Analysis of Descriptions one or the other dog of Figure 1 described as tan, Speakers can use language for a variety of pur- it constrains how we will interpret subsequent uses poses. Their decisions of what to say thus depend of the word tan. Concretely, we might update the on knowledge of language, their communicative perceptual classifier we associate with tan in this situation, and their communicative goals. Follow- context so that it fits the target dog and excludes its ing Anderson (1991), rational analysis invites us alternative (Larsson, 2013). We see this as a case to explain an agent’s action as a good way to ad- of accommodation, in the sense of (Lewis, 1979). vance the agent’s goals given the agent’s informa- As speakers, we often count on our interlocu- tion. When applied to communication, this ap- tors to accommodate us (Thomason, 1990). We proach allows us not only to derive utterances for can use vague terms confidently as long as the dis- systems but also to infer linguistic representations tinction we aim to draw with them is clear in con- from utterances when we know the agent’s com- text and as long as our choice is sufficiently in line municative situation and communicative goals.