Exploring 3D Gestural Interfaces for Music Creation in Video Games Jared N. Bott James G. Crowley Joseph J. LaViola Jr. University of Central Florida University of Central Florida University of Central Florida School of EECS School of EECS School of EECS Orlando, Florida 32816 USA Orlando, Florida 32816 USA Orlando, Florida 32816 USA [email protected] [email protected] [email protected] ABSTRACT interest. These games offer the ability to simulate playing In recent years the popularity of music and rhythm-based music by using custom input devices designed to emulate the games has experienced tremendous growth. However almost instrument being played. Although it has been shown that all of these games require custom hardware to be used as in- such props can lead to a more immersive experience [7], these put devices, and these devices control only one or two similar input devices represent an additional investment required of instruments. In this paper we describe One Man Band, a the player. Furthermore while these games may give the prototype video game for musical expression that uses novel player the feeling of actually playing an instrument, there 3D spatial interaction techniques using accelerometer-based is almost no opportunity for individual musical expression. motion controllers. One Man Band provides users with 3D Instead, the user is relegated to using the input device to gestural interfaces to control both the timing and sound of trigger a predefined sequence of notes. Thus, most games of the music played, with both single and collaborative player the genre are more about matching rhythms than actually modes. We further investigate the ability to detect different creating music. musical gestures without explicit selection of mode, giving the user the ability to seamlessly transition between instru- ment types with a single input device. A formal user study is then presented comparing the musical interface of One Man Band to that of Nintendo’s Wii Music. Our results indicate that users generally preferred the interface of One Man Band over that of Wii Music. We also found that users desire to express their own ideas and have explicit control of the melodies created in music-based video games. Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation]: User Interfaces—Auditory (non-speech) feedback ; K.8.0 [Personal Computing]: General—Games; I.5.5 [Pattern Recogni- tion]: Implementation—Interactive Systems Figure 1: Four people playing One Man Band in General Terms guided mode. 3D gestural interfaces, video games, music, Wii Remote, user evaluation We posit that it is possible to create a music game that gives the user more creative control while preserving those 1. INTRODUCTION characteristics that have made rhythm games popular. To Recently in the gaming industry there has been growing explore this idea we have created One Man Band, a proto- interest in music and rhythm games. Franchises such as type video game that uses 3D gestural interfaces to simulate Guitar Hero and Rock Band have sold over a million units, a number of different musical instruments. Our prototype with newly released titles continuing to generate significant uses Nintendo Wii controllers, that utilize accelerometers, to identify gestures made by the players and translate them into musical notes or sounds. An important design goal in One Man Band was to enable musical novices to collaborate Permission to make digital or hard copies of all or part of this work for and create pleasing compositions. personal or classroom use is granted without fee provided that copies are Our prototype contains several different gameplay options, not made or distributed for profit or commercial advantage and that copies including a guided mode, a free play mode, and a mecha- bear this notice and the full citation on the first page. To copy otherwise, to nism for a single player to create complex songs on their republish, to post on servers or to redistribute to lists, requires prior specific own. In guided mode, players can choose a song and an permission and/or a fee. ICFDG 2009 April 26–30, 2009, Orlando, FL, USA. instrument and are presented with graphical cues indicat- Copyright 2009 ACM 978-1-60558-437-9 ...$5.00. ing which notes to play. Free play mode allows players to explore their own musical ideas and offers a visual represen- tation of any notes played. A looping mechanism is provided to each player so that they can create complex rhythms and songs on their own or as part of an ensemble. Finally, One Man Band provides a special gestural interface that allows a single player to use different instruments without explicit selection of mode. We evaluate the One Man Band interface by presenting both a short informal study of the effectiveness of playing instruments without explicit selection of mode, and a for- mal user study comparing the One Man Band interface to that of Wii Music, a commercially available game from Nin- tendo that also uses a Wii Remote as an input device. We conducted our experiment by having subjects play instru- ments from both interfaces and gathering feedback about what aspects of the interfaces they liked and disliked. Sub- jects ranked their preferences on a seven point Likert scale and we used standard statistical techniques to analyze the data. Figure 2: Musical instruments are played through several gestural interfaces. Clockwise from top-left: 2. RELATED WORK drums, guitar, violin, theremin Interactive music games have been developed and stud- ied by both the academic community and the entertainment a drum pad. Additional projects have also explored the con- industry. Guitar Hero [1] and Rock Band [2] are just two re- cept of interactive or collaborative musical systems [9, 12, cent examples of music-based games that have experienced 14]. Our work builds upon these ideas by using off the shelf tremendous success. Both of these games provide the user hardware and the PC platform to implement a collabora- with specialized input devices designed to mimic the instru- tive gesture-based music game that does not require custom ment they represent. Players are asked to press buttons on a interfaces. guitar controller or strike drum pads to match symbols that appear on screen. If the player performs the correct action with proper timing the in-game music continues to play; if 3. IMPLEMENTATION the player instead fails to press the correct button or does One Man Band uses a game controller in the form of Nin- so with incorrect timing, a non-musical sound is triggered tendo Wii remotes and nunchuks, a common accessory for instead. Guitar Hero has also recently introduced a studio the Wii remote. Wii remotes contain a three-axis digital mode that allows players to develop their own musical ideas accelerometer, an infrared camera, a directional pad, and using the supplied input devices. For an excellent discussion a number of buttons. The nunchuk, which plugs into the of additional games and interfaces in this space see [4]. Al- Wii remote, contains the same accelerometer as well as a though each of these games provide some collaborative and thumbstick and two buttons. creative abilities, our work also gives the player a simple and Our prototype provides six primary single-instrument ges- powerful gestural interface for many instruments using only tural interfaces: drums, guitar, violin, bass, trombone, and a single input device. theremin. These interfaces allow players to produce sounds Wii Music [3] is a recent title that focuses on using a sim- by mimicking the way each instrument is played (Figure 2). ple interface to allow musical expression. This software uses The interfaces are designed to give the user an intuitive feel an accelerometer-equipped input device to recognize musi- for how each instrument type should be used. This scheme cal gestures. However, Wii Music is limited with respect to allows for an interface that is easy, realistic, powerful, and melodic expression. Players are not able to compose music. expressive. We carefully chose the notes and sounds of the Instead, players choose from a list of songs, and use ges- instruments so players can effortlessly play and combine tures to define rhythms that are applied to notes selected by sounds together in a way that is both musical and pleas- the software itself. This is a significant omission, as users ing to the ear, and we provide several sets of sounds for are provided no mechanism to actually create music. Our each instrument type so that the player can create songs in work also uses simple gestures to implement musical inter- a variety of genres. faces, but additionally gives users creative control over ex- actly what melodies and rhythms they want to play. 3.1 Interfaces The topic of interactive music has also been explored by Drums. The drums are implemented as virtual drum- researchers. World Beat [6] provides players with infrared sticks, with a gesture simulating a drumming motion along batons that can be used to conduct a virtual orchestra or the Z-axis of the device triggering a drum sound. At first control several instruments. Researchers in [13] also exam- we simply played a drum sound whenever we saw a suffi- ine digital batons as gestural interfaces. CoJIVE [8] explores ciently large acceleration peak, but this allowed players to musical collaboration by allowing novices to use digital ba- trigger drum sounds with an upward motion as well as a tons to collaborate on jazz compositions. Jam-O-Drum [5] downward motion. Furthermore, anytime the Wii remote also focuses on the collaborative aspects of music making, changes direction or stops, two distinct peaks will be seen, placing users at a tabletop installation where they can par- one with a positive acceleration and another with a negative ticipate in a number of interactive music games by striking acceleration.