A Multimodal Home Entertainment Interface via a Mobile Device Alexander Gruenstein Bo-June (Paul) Hsu James Glass Stephanie Seneff Lee Hetherington Scott Cyphers Ibrahim Badr Chao Wang Sean Liu MIT Computer Science and Artificial Intelligence Laboratory 32 Vassar St, Cambridge, MA 02139 USA http://www.sls.csail.mit.edu/ Abstract portant role in managing digital media libraries. For instance, a web-enabled mobile phone can be used to We describe a multimodal dialogue system for remotely schedule TV recordings through a web site interacting with a home entertainment center or via a custom application. Such management tasks via a mobile device. In our working proto- type, users may utilize both a graphical and often prove cumbersome, however, as it is challeng- speech user interface to search TV listings, ing to browse through listings for hundreds of TV record and play television programs, and listen channels on a small display. Indeed, even on a large to music. The developed framework is quite screen in the living room, browsing alphabetically, generic, potentially supporting a wide variety or by time and channel, for a particular show using of applications, as we demonstrate by integrat- the remote control quickly becomes unwieldy. ing a weather forecast application. In the pro- totype, the mobile device serves as the locus Speech and multimodal interfaces provide a nat- of interaction, providing both a small touch- ural means of addressing many of these challenges. screen display, and speech input and output; It is effortless for people to say the name of a pro- while the TV screen features a larger, richer gram, for instance, in order to search for existing GUI. The system architecture is agnostic to recordings. Moreover, such a speech browsing ca- the location of the natural language process- pability is useful both in the living room and away ing components: a consistent user experience from home. Thus, a natural way to provide speech- is maintained regardless of whether they run based control of a media library is through the user’s on a remote server or on the device itself. mobile device itself. In this paper we describe just such a prototype 1 Introduction system. A mobile phone plays a central role in pro- People have access to large libraries of digital con- viding a multimodal, natural language interface to tent both in their living rooms and on their mobile both a digital video recorder and a music library. devices. Digital video recorders (DVRs) allow peo- Users can interact with the system—presented as a ple to record TV programs from hundreds of chan- dynamic web page on the mobile browser—using nels for subsequent viewing at home—or, increas- the navigation keys, the stylus, or spoken natural ingly, on their mobile devices. Similarly, having language. In front of the TV, a much richer GUI is accumulated vast libraries of digital music, people also available, along with support for playing video yearn for an easy way to sift through them from the recordings and music. comfort of their couches, in their cars, and on the go. In the prototype described herein, the mobile de- Mobile devices are already central to accessing vice serves as the locus of natural language in- digital media libraries while users are away from teraction, whether a user is in the living room or home: people listen to music or watch video record- walking down the street. Since these environments ings. Mobile devices also play an increasingly im- may be very different in terms of computational re- 1 Proceedings of the ACL-08: HLT Workshop on Mobile Language Processing, pages 1–9, Columbus, Ohio, USA, June 2008. c 2008 Association for Computational Linguistics sources and network bandwidth, it is important that remote control. They explore a Speech-In List-Out the architecture allows for multiple configurations in interface to searching for episodes of television pro- terms of the location of the natural language pro- grams. cessing components. For instance, when a device (Portele et al., 2003) depart from the model of is connected to a Wi-Fi network at home, recogni- adding a speech interface component to an exist- tion latency may be reduced by performing speech ing on-screen menu. Instead, they a create a tablet and natural language processing on the home me- PC interface to an electronic program guide, though dia server. Moreover, a powerful server may enable they do not use the television display as well. Users more sophisticated processing techniques, such as may search an electronic program guide using con- multipass speech recognition (Hetherington, 2005; straints such as date, time, and genre; however, they Chung et al., 2004), for improved accuracy. In sit- can’t search by title. Users can also perform typi- uations with reduced network connectivity, latency cal remote-control tasks like turning the television may be improved by performing speech recognition on and off, and changing the channel. (Johnston et and natural language processing tasks on the mobile al., 2007) also use a tablet PC to provide an inter- device itself. Given resource constraints, however, face to television content—in this case a database of less detailed acoustic and language models may be movies. The search can be constrained by attributes required. We have developed just such a flexible ar- such as title, director, or starring actors. The tablet chitecture, with many of the natural language pro- PC pen can be used to handwrite queries and to point cessing components able to run on either a server or at items (such as actor names) while the user speaks. the mobile device itself. Regardless of the configu- We were also inspired by previous prototypes in ration, a consistent user experience is maintained. which mobile devices have been used in conjunc- tion with larger, shared displays. For instance, (Paek 2 Related Work et al., 2004) demonstrate a framework for building such applications. The prototype we demonstrate Various academic researchers and commercial busi- here fits into their “Jukebox” model of interaction. nesses have demonstrated speech-enabled interfaces Interactive workspaces, such as the one described in to entertainment centers. A good deal of the work (Johanson et al., 2002), also demonstrate the utility focuses on adding a microphone to a remote con- of integrating mobile and large screen displays. Our trol, so that speech input may be used in addition prototype is a departure from these systems, how- to a traditional remote control. Much commercial ever, in that it provides for spoken interactions. work, for example (Fujita et al., 2003), tends to fo- cus on constrained grammar systems, where speech Finally, there is related work in the use of mobile input is limited to a small set of templates corre- devices for various kinds of search. For instance, of- 1 sponding to menu choices. (Berglund and Johans- ferings from Microsoft (Acero et al., 2008), Vlingo, 2 son, 2004) present a remote-control based speech and Promptu allow users to search for items like interface for navigating an existing interactive tele- businesses and songs using their mobile phones. vision on-screen menu, though experimenters man- These applications differ from ours in that speech ually transcribed user utterances as they spoke in- is used only for search, without any accompanying stead of using a speech recognizer. (Oh et al., 2007) command and control capabilities. Also, these ser- present a dialogue system for TV control that makes vices do not allow interaction with your own de- use of concept spotting and statistical dialogue man- vices at home. Efforts have been made to use mo- agement to understand queries. A version of their bile devices for control of devices in the home, such system can run independently on low-resource de- as in (Nichols and Myers, 2006), however these ef- vices such as PDAs; however, it has a smaller vo- forts have not involved the use of speech as an input cabulary and supports a limited set of user utterance modality. templates. Finally, (Wittenburg et al., 2006) look mainly at the problem of searching for television 1http://www.vlingo.com programs using speech, an on-screen display, and a 2http://www.promptu.com 2 Status icon indicates connection to server, Media playback Tabs allow easy switching microphone icon when listening controls among domains Hold- to-talk button Opera web browser Firefox web browser Navigation pad or stylus can Synchronized selection on mobile and be used to navigate GUI television displays Figure 1: User interface overview. U: What’s on tonight at seven o’clock? S: I found seventy entries. [Displays program guide.] U: Are there any action movies on HBO tomorrow? S: I found two entries. [Displays results.] U: When is American Idol on? S: I found three entries. [Displays results.] U: Record the first one. S: I have added American Idol to the list of programs to be recorded. [Displays recording settings.] (a) Program Guide U: Please record Nova on Wednesday night. S: I have added Nova to the list of programs to be recorded. [Displays recording settings.] U: Show all my scheduled recordings. S: There are seven scheduled recordings. [Shows list.] U: Delete BBC World News. S: I have deleted the scheduled recording: BBC World News.[Updates List.] U: Show my videos. S: [Displays list of videos.] U: Delete Frasier. (b) Scheduled Recordings S: I have deleted the recording Frasier.[Updates list.] U: Play Grey’s Anatomy. S: Playing Grey’s Anatomy.[Plays recording.] U: Stop Playing. S: Stopping. [Stops playback.] U: [Clicks on Music tab.] S: [Displays artist list.] U: Show albums by Billie Holiday. S: I found three albums by Billie Holiday.[Shows albums.] U: Please play A Hard Rain’s A-Gonna Fall by Bob Dylan.