A Web-Based Game for Collecting Commonsense Goals
Total Page：16
File Type：pdf, Size：1020Kb
Common Consensus: a web-based game for collecting commonsense goals Henry Lieberman Dustin A Smith Alea Teeters MIT Media Lab MIT Media Lab MIT Media Lab 20 Ames St. E15-384A 20 Ames St. E15-320F 20 Ames St. E15-X Cambridge, MA 02139, USA Cambridge, MA 02139, USA Cambridge, MA 02139, USA +1-617-253-0315 +1-617-253-0315 +1-617-253-0315 [email protected] [email protected] [email protected] ABSTRACT everyday goals to a large knowledge base. A knowledge base In our research on Commonsense reasoning, we have found of goals and associated information will serve as a resource that an especially important kind of knowledge is knowl- for intelligent interfaces to model the motivations and actions edge about human goals. Especially when applying Com- of their users. We seek to make the game entertaining enough monsense reasoning to interface agents, we need to recog- to motivate players to contribute, while making sure to get nize goals from user actions (plan recognition), and generate knowledge that answers the questions that interest us and that sequences of actions that implement goals (planning). We represents consensus knowledge according to our users. also often need to answer more general questions about the situations in which goals occur, such as when and where a Collecting Commonsense Knowledge from Volunteers particular goal might be likely, or how long it is likely to take to achieve. A major disparity between computers and humans is that computers do not have the vast resource of everyday knowl- In past work on Commonsense knowledge acquisition, users edge that we humans rely on to solve-problems and com- have been directly asked for such information. Recently, municate. Information such as “people sleep at night” and however, another approach has emerged—to entice users into “doors can be opened” is trivial and implicit to people, but playing games where supplying the knowledge is the means is absent from computer software. The problem of acquir- to scoring well in the game, thus motivating the players. This ing this enormous body of knowledge is known to the arti- approach has been pioneered by Luis von Ahn and his col- ficial intelligence community as the knowledge acquisition leagues, who refer to it as Human Computation. bottleneck. Offering an innovative solution to this problem, the Open Mind [19] and CYC [9] projects demonstrated that Common Consensus is a fun, self-sustaining web-based game, the Internet can be used for distributed knowledge collec- that both collects and validates Commonsense knowledge tion, particularly commonsense knowledge, which, by def- about everyday goals. It is based on the structure of the TV 1 inition, is non-expert and possessed by everyone. Subse- game show Family Feud . A small user study showed that quently, many similar projects have been developed [3] [8] users find the game fun, knowledge quality is very good, and that collect knowledge from volunteers and store them in var- the rate of knowledge collection is rapid. ious formats. The OpenMind project, for example, maintains ACM Classification: H.3.3 [INFORMATION STORAGE the knowledge in basic English statements. AND RETRIEVAL]: Information Search and Retrieval; I.2.6 [ARTIFICIAL INTELLIGENCE]: Learning Motivating Volunteers to Contribute Keywords: knowledge acquisition, commonsense, games, If we expect to continue collecting knowledge from volun- goals, contexts, commonsense knowledge retrieval teers, we must focus on ways to motivate them to contribute high-quality knowledge. Although we have collected a lot of INTRODUCTION knowledge from projects like OpenMind, we are far from the Common Consensus is an on-line game, designed to motivate hundred-of-millions to billions of “pieces of knowledge” that users to contribute Commonsense knowledge about people’s are estimated to be involved with human intelligence [15]. This challenge is exacerbated by the fact that the number of 1 Family Feud is a trademark of FremantleMediaOperations BV. volunteer contributors drops over the life of the project. Permission to make digital or hard copies of all or part of this work for In 2004, von Ahn and colleagues started building web-based personal or classroom use is granted without fee provided that copies are games which serve the dual purposes of acquiring knowledge not made or distributed for profit or commercial advantage and that copies and providing entertainment to their users. Notable such ef- bear this notice and the full citation on the first page. To copy otherwise, to forts include the ESP Game for annotating images; Peek-a- republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. boom, a game designed for segmenting objects in images; IUI'07, January 28–31, 2007, Hawaii, USA.. and Verbosity, a game for collecting commonsense knowl- Copyright 2007 ACM X-XXXXX-XXX-X/XX/XXXX ...$5.00. edge [20], [22] and [21]. Goal-Oriented Commonsense Knowledge To motivate users to contribute, we developed a competitive Having a large amount of knowledge is just one part of the multi-player game where users race against each other to commonsense reasoning problem: we also need good ways contribute commonsense knowledge. Users have found the to retrieve, represent and reason with this knowledge [14]. scoring and user interaction elements of the game enjoyable, Particularly important is knowledge about human goals. Start- and a small user study suggested that most subjects wanted ing from Maslow’s Hierarchy of Human Needs [12], people to continue playing. have desires that motivate their behavior. These goals are broken down into subgoals, and the goal tree terminates in Automatic knowledge validation by consensus Common- concrete actions. Goals often answer the why questions about sense knowledge bases developed by volunteers must antic- human behavior, and provide good clues as to the when, how, ipate noisy data, which comes in the form of misspellings, and other considerations. They are therefore fundamental to incorrect information, and, most commonly, knowledge at explanation. varying levels of detail. This granularity problem leads to knowledge that is difficult to reason with reliably. The game One of our long-term goals is to index Commonsense knowl- structure of Common Consensus inherently provides a data- edge via goals and knowledge associated with goals. To validation mechanism: the scores for players’ answers are our knowledge, this has not yet been done in a comprehen- computed by counting the number of other people who con- sive way for large-scale, everday Commonsense knowledge. tributed the same answer. The more people agree on a spe- Many applications of Commonsense knowledge rely on rel- cific answer, the more confidence we have that this answer is atively simple matching techniques. The simplest of these valid. fall back on Information Extraction technologies such as key- word matching and statistics of word co-occurrence such as Four subjects evaluated the data obtained during the user test- Latent Semantic Analysis to match up goals with statements ing and all of the answers that one third of all users had en- of their methods and results. More complex structural tech- tered were consistently marked as excellent answers for the niques perform limited kinds of reasoning over semantic net- question. The consensus mechanism can serve as a way to works, such as the spreading activation reasoning of Con- screen data. For example, when users were presented with ceptNet [11], Case-Based Reasoning, and structure-mapping the question: What are some things you would use to: cook analogy. Plan recognizers also have a limited capability to dinner? their aggregate answers gravitated toward the super- recognize when a sequence of actions is consistent with the ordinate and basic categories [13]. The most common an- desire to accomplish a certain goal. swers (by the number of unique users) were: food (7), pots (3), pans (3), meat (3), knife (2), oven (2), microwave (2)... But in order to do plan recognition, generation, monitoring, We also collected specific and atypical answers, like garlic and debugging over a wide spectrum of everyday situations, press and obscure answers, like cans but they had a low count it is desirable to collect enough explicit knowledge about (in this case, 1). It should be noted that there is a trade-off goals to reduce the burden of inferring every detail about involved with only using the popular answers: many good goal-oriented behavior from first principles. People are of- uncommon answers are neglected. ten (but not always!) quite articulate about why they are do- ing something, when asked, even though they leave out this Goals as questions and answers: a continuous supply of knowledge as already understood in normal discourse. Thus questions We are collecting first-person goals, which we an application that explicitly asks users for goal-oriented define as a verb and at least one object (e.g., “to write an explanations can quickly amass a large collection of goal email”). Goals can be represented in a hierarchy, where each knowledge. goal may have parent and children goals. A plan is a specific Design Objectives for the Game sequence of sub-goals, and each parent goal can have many We developed a game, Common Consensus, that has the fol- particular sequences of sub-goals (plans). In other words, lowing characteristics: 1) provides entertainment to the users there may be many ways to satisfy a goal (with the email ex- and thus motivation to contribute; 2) defines the quality of an ample: “open Gmail”, “use your Blackberry” are both valid answer by the number of consenting answers; and 3) avoids sub-goals), and each of those ways can be expressed as a se- convergence by replenishing seed questions with common ries of sub-goals.