SoundPark: Exploring Ubiquitous Computing through a Mixed Reality Multi-player Game Experiment Romain Pellerin Nicolas Bouillot Tatiana Pietkiewicz Conservatoire National des Centre for Interdisciplinary Conservatoire National des Arts et Metiers´ Research in Music Media and Arts et Metiers´ Paris, France Technology Paris, France [email protected] McGill University tatiana.pietkiewicz Montreal, QC, Canada @ubidreams.com [email protected] ABSTRACT support rich interaction between multiple users and their We describe an ubiquitous computing architecture through environment. Traditionally, this has been done entirely in a multi-player game application based on the objective of the space of a virtual world, with no obvious relationship collecting audio clips and depositing them in a staging area. to the user's surroundings. However, ubiquitous computing Central to the game are the themes of highly coupled in- lends itself well to supporting mixed reality applications, teraction and communication between players with different thus leveraging the richness of the physical environment for roles and an engaging blend of interaction with both the more engaging interaction. physical and virtual worlds. To this end, numerous technolo- The research described here is intended to explore dis- gies including locative sensing, miniature computing, and tributed system features for coupled interaction in the ubiq- portable displays had to be integrated with a game middle- uitous computing context, where real-time audio communi- ware and audio scene rendering engine. The result provides cations and frequent state updates are imperative to support the game activity. Our proof of concept application, Sound- a compelling example of future distributed systems that this 1 paper describes. Park , is designed to let users navigate in a reasonably sized area, communicating with each other through high-fidelity, low-latency audio, and interacting with a rich, mixed-reality Categories and Subject Descriptors environment in which spatialized virtual audio sources are D.4.7 [Operating Systems]: [Real-time systems and em- distributed throughout the physical space. bedded systems]; H. 5.3 [Multimedia Information Sys- Section 2 describes different ubiquitous games and their tems ]: [Artificial, augmented, and virtual realities] main features. Section 3 is dedicated to SoundPark game design. Section 4 provides a description of the full game General Terms architecture while Section 5 deals with implementation is- sues. Section 6 includes a discussion of a fully distributed Design, Experimentation architecture and finally, Section 7 concludes the paper. Keywords Ubiquitous computing, mixed reality game, mobility, em- 2. RELATED WORK bedded systems Technologies commonly found in large-scale pervasive com- puting games include mobile phones, PDAs, and portable 1. INTRODUCTION game systems, in conjunction with a web server to publish information related to player missions or object collections. Ubiquitous computing architectures enable a wealth of ap- GPS technology is used extensively for geo-localization, in plications that go far beyond the traditional realm of Dis- particular those based on mixed reality, either to provide tributed Systems. Taking advantage of the distinguishing players' locations or to collect virtual objects in the environ- features of ubiquitous computing architectures, namely, a ment [14]. Examples include Botfighter [21], CanYouSeeMeNow distributed computational infrastructure, support for com- [9], Mogi [13], and CatchBob [11]. The remainder of this sec- munication, location- and context-awareness, and critically, tion surveys some of the relevant efforts involving locative mobility of users, allows for the emergence of games that technology, interaction with a virtual audio environment, and multi-player pervasive games involving networked com- munication between the participants. Several applications of locative media technology focus on Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are virtual audio overlays in large-scale spaces. In the sim- not made or distributed for profit or commercial advantage and that copies plest case, geographical locations of interest are tagged with bear this notice and the full citation on the first page. To copy otherwise, to sound. One of the earliest such efforts was the [murmur] republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 1 NOTERE 2009 June 23-27, 2009, Montreal,´ Canada A sample video is available from http://www.audioscape. Copyright 2009 ACM X-XXXXX-XX-X/XX/XX ...$5.00. org/twiki/bin/view/Audioscape/MobileAudioscape. project2, in which the tags take the form of telephone num- them in a staging area or home base, using walkie-talkie-like bers that can be dialed by other users on their mobile phones, communication to coordinate their activities. The loops are in order to listen to audio recordings related to the loca- placed initially at pre-determined locations, which are dis- tions. Another example, supporting continuous exploration closed to one type of player, the scout, in the form of clues of location-based audio, was the Hear&There project [19], read from RFID-tagged bollards. These sound loops may which allowed users to record audio at given GPS coordi- only be discovered (via directed listening) and collected by nates, while hearing a spatial rendering of other recordings another type of player, the hunter, when he is close to the as they navigate. This was limited to a single-person experi- corresponding location, once the associated clue has been ence, with the scene state maintained by a single computer. discovered. Sound manipulation functions, required by the These efforts established the basic concept, which has since third player, the D.J., are represented as virtual objects, been adopted by many other groups [20, 22], of selecting discovered by the scout in a similar manner. The D.J. is re- audio samples based on the user's physical location. The sponsible for organizing the collected sounds, using acquired SoundPark project uses the same principles but address a capabilities of sound sequencing or mixing. multi-player experience. Consistent with the theme of a mixed reality environment, In the area of multi-player mobile games, a seminal exam- game objects typically exist as both real (physical) and vir- ple is Cheok's Human Pacman [6], which inherits its design tual entities: Clues are represented by their associated bol- from the eponymous arcade game of the early 1980's, but lards and RFID tags, but reveal their content graphically has human players, wearing head-mounted displays (HMD), on the scout's mobile phone display. Although sound loops who take on the role of pacman or ghosts. Although Human are only perceived in the virtual domain, i.e., heard through Pacman is an engaging, media-rich, multi-player game, the the players' headsets, they exist in association with specific equipment users required was bulky and, in particular with physical locations in the real environment and can thus be regard to the HMD, awkward to wear on a long-term basis. moved from place to place. To increase the challenge of the Human Pacman has been extended by different projects, in- game, clues indicate the location of individual sound loops, cluding Pac-Lan RFID [18] where both game objects (pills, but not their musical properties, which can only be discov- super-pills, and ghosts) and human players have associated ered by the hunters as they draw near. Furthermore, not RFID tags. Such tags, increasingly popular in entertain- all sound loops will fit with the theme of a target musical ment applications, are used for the purposes of location, arrangement. For example, a baroque guitar loop cannot identification and content delivery [2, 12]. Our game is less be part of a jazz arrangement. Attempts to deposit such equipment-intensive than Human Pacman, requiring only a sound loop in the staging area fail with an appropriate small form factor devices, such as gumstix and near-field feedback message. Inclusion of a modest number of these communications (NFC)-enabled Nokia 5140 mobile phone, distractors throughout the environment adds a layer of com- allowing for more user mobility in the gameplay. In addition plexity, fun, and opportunity for musical pedagogy to the to the common use of geo-localization, certain recently pro- game activity. posed games, such as MeetYourHeartTwin [8], propose the A team is composed of one scout, multiple hunters and one additional integration of biosensors to establish social rela- D.J. Activity is initiated by the scout, who discovers clues tionships between players. In this example, players are able by reading the RFID tags distributed throughout the game to see others, on their PDA displays, sharing their heartbeat area and observing the location information displayed on the characteristics, e.g., in a similar frequency range. mobile phone screen. At that point, the associated sound SoundPark shares many characteristics of these earlier loop is unlocked, and can be heard by a hunter who passes works. However, rather than a focus on 3D graphics (e.g., within proximity of its physical location, typically guided by as in Human Pacman), our emphasis is on audio interaction, the scout using a wireless, low-latency audio communication for communication between players, for rendering of game protocol. The location of the hunter is updated in real time state, and for exploration of the virtual world. SoundPark on the scout's mobile phone map display. also uses RFID tags, but only for content delivery of clues. With guidance from the scout, the hunter must retrieve This bias toward the auditory modality is motivated in part sound loops from their locations by pressing a button on by observing that the physical world offers an environment a Wiimote, which communicates to the mobile computer that is enormously rich visually, and from which, we do not worn by the player. Audio spatialization is used to guide wish to isolate users.