Common Sense Data Acquisition for Indoor Mobile Robots Mykel J. Kochenderfer Rakesh Gupta Honda Research Institute USA, Inc. Honda Research Institute USA, Inc. 800 California St., Suite 300 800 California St., Suite 300 Mountain View, CA 94041 Mountain View, CA 94041 [email protected] [email protected] ABSTRACT as those associated with the Open Mind Initiative[7], The objective of this research is to enhance the intel- have been quite popular. The Open Mind Common ligence of mobile robots that work in home and o±ce Sense project, led by Push Singh at the MIT Media Lab, environments with common sense. Since indoor com- has accumulated a corpus of 600,000 pieces of knowl- mon sense knowledge is extremely vast, it may be col- edge from 11,500 users (as of August 2003) over the lected in a similar fashion to the Open Mind family of past three years1. Other projects such as Open Mind distributed knowledge capture projects over the Inter- Word Expert2 and Open Mind 1001 Questions3, have net. This paper describes the collection of data through also been successful. the Open Mind Indoor Common Sense (OMICS) web- site. The knowledge was collected through sentence This paper describes the Open Mind Indoor Common templates that were dynamically generated based on Sense (OMICS) project that became publicly available user input. This was converted into relations and saved at the beginning of August 20034. We report on our into a database. We discuss the results of this online results and experiences with working with online con- collaborative e®ort and mention various applications of tributors. The ¯nal section of the paper discusses how the collected data. we will use the data we collected. Categories and Subject Descriptors 2. METHODS I.2.4 [Arti¯cial Intelligence]: Knowledge Represen- In this section we ¯rst describe the framework of the tation Formalisms and Methods|Frames and scripts; knowledge base and the relations necessary to capture I.2.6 [Arti¯cial Intelligence]: Learning|Knowledge those types of common sense most useful to an indoor acquisition robot. We then describe how we built a website to cap- ture this data in a way that is friendly to non-expert users and how we converted the data into machine- 1. INTRODUCTION understandable relations. To accomplish more than extremely narrow tasks in a home or o±ce environment, a mobile robot must posses some amount of common sense. Such common sense 2.1 Knowledge Base Framework includes knowledge about human desires, objects and The framework of this project is object-centric. It is their locations, and causality. Although our domain is assumed that the desires of the users and the actions restricted to indoor home and o±ce environments, the taken by the robot are grounded in the properties of amount of knowledge required for a robot to function objects in the world. The robot is useful to humans is still quite vast. Since common sense does not require because it can perform operations on objects in such a expert knowledge, the data may be collected as part of way that a desired e®ect is achieved. a public online collaborative e®ort over the Internet. The robot can observe properties of objects in its vicin- Distributed online knowledge acquisition projects, such ity and it can perform actions that change the proper- ties of objects. In this system, a statement is a pair Á = (o; p) where o is some object and p is an ad- jective describing the property. Statements may be thought of as assertions about the property of an ob- ject in the world or actions to be taken on an object to 1http://commonsense.media.mit.edu 2http://www.teach-computers.org/word-expert.html 3http://www.teach-computers.org/learner.html 4http://openmind.hri-us.com Image labelling can link multiple labels to the same ob- ject (e.g. the labels \sofa" and \couch" might be both associated with \h0141.gif"). The images themselves can be used for training the object recognition system of the robot. When the website became public, the database initially contained a set of over 400 images of indoor objects selected by hand from a collection of stock photography. Figure 1: A template sentence. Statements In the `statements' activity, the user is prompted with a question such as, \You often want a fan to be blank." achieve a particular e®ect. For example, the statement This activity pairs objects with properties in the knowl- (cup-of-co®ee; hot) can mean \a cup of co®ee is hot" or edge base. The objects that appear in these sentence represent the action \make a cup of co®ee hot." templates come from the objects entered by users in the other activities, such as the `objects' activity. Since we want our robot to understand the consequences of its actions, we wish to capture such common sense knowledge as: (o1; p1) causes (o2; p2). For example, the Uses statement (fan; on) causes (room; cool). We also wish This activity associates objects with their uses. For to capture knowledge such as (o1; p1), which would in- example, the user might be prompted with the form, dicate that you would want (o2; p2). For example, the \A hanger is used to ." Again the objects come statement (cup-of-co®ee; cold) would make you want the from user input. statement (cup-of-co®ee; hot) ful¯lled. Causes At any given point in time, the robot observes a set This activity captures causality. For example, a form of statements that are true and the robot is able to might ask, \A computer is o® when a is ." execute a set of statements. The general problem is to If the user enters a new object or a new object-property decide which statements to execute in order to achieve pair, it will be entered into the object or statement ta- perceived goals. ble. The object and property that makes up the ¯rst part of the sentence is formed dynamically by selecting It is important that the robot knows where objects are a random object from the knowledge base. generally located (e.g. a pillow may be found in the bedroom). The robot should also know what various objects are used for (e.g. a microwave is used to heat Desires food). This activity helps the robot determine what needs to be done in various situations. A template form might ask, \You might want a fan to be blowing if you notice 2.2 Sentence Templates that your has become ." We need some way to convert the common sense in the minds of non-expert users into relations in a knowledge base. Following the style of the Open Mind Common Locations Sense website, we decided to use sentence templates. This activity associates objects with the rooms where Users are prompted to ¯ll in the blanks of sentences they are typically found. For example, the user might with words, as shown in Fig. 1. be prompted with, \A room where you generally ¯nd a dinner table is the ." Di®erent activities capture di®erent types of knowledge. Below is a summary of some of the activities: Proximity This activity associates objects with each other based Objects on proximity. A sample form would be, \You generally ¯nd a frying pan in the same room as a ." In this activity, the user is asked to identify the types of objects commonly found in a home or o±ce. The user may be prompted to type in an object name that Senses comes to mind or simply label an image of an object. This activity disambiguates the intended sense of var- It is important to allow a user to simply type in any ious objects entered into the database by other users. indoor object that comes to mind because we want to Fig. 2 shows a sample form. The objects to disam- include all relevant objects in the database even if we biguate are selected from previous user entries and the do not have their picture. senses are from WordNet [3]. Figure 2: A word sense disambiguation form. Freeform This activity allows users to type in any form of com- mon sense knowledge that might not be captured by any of the other activities. Although it is quite di±- cult to convert freeform sentences into useful relations, it provides us with a sense of the types of knowledge the general public would like an indoor robot to under- stand. Analysis of freeform sentences will later lead to the creation of new activities. 2.3 Data Collection Figure 3: The review form used for administra- Once a user logs on with their account, they are pre- tors to approve of user entries. Administrators sented with a random activity. After a random number may either commit, uncommit, or reject entries. of entries for a particular activity, the system prompts them with a new activity. Users may also manually 17,000 of them accepted. We had a weekly contest (last- switch between the activities. ing four weeks) where the top contributor was awarded an Open Mind t-shirt. Other Open Mind projects have 2.4 Data Quality Review used similar contests to help motivate submissions. Win- The completed sentence templates are stored in the ners had their names and hometowns listed on the title database as raw sentences. These sentences are not page of the site. parsed into relations until after an administrator \ap- proves" of them (see Fig. 3). It is important that there 3.1 Observations be some way of ensuring data quality since the data The OMICS site was publicly announced on August 5th (such as names of objects and their properties) are used and the ¯rst t-shirt prize was awarded August 6th.