AUDIOVERDRIVE: EXPLORING BIDIRECTIONAL COMMUNICATION game engine for the iPhone and iPad platforms, with gen- erative spaces of music composed in Ableton Live, a pop- BETWEEN MUSIC AND GAMEPLAY ular piece of digital audio workstation software aimed at live performance, via a composer/designer-configurable Nils Iver Holtar, Mark J. Nelson, Julian Togelius set of mappings. Audioverdrive’s aesthetics are loosely based on the first author’s experiences—previously sepa- Center for Computer Games Research rate ones—as composer/keyboardist for the game-music- IT University of Copenhagen influenced synth band Redacted For Blind Review, and [email protected], mjas, juto @itu.dk designer of procedurally varying arcade games. One of { } the goals in that regard is to produce an actually playable version of the coupled gameplay–music experiences one often finds imagined in music videos in this genre. ABSTRACT The core is complex and closely coupled, but its interface with the music system is cleaner and one-way. The core We describe a system for bidirectionally coupling the mu- gameplay system sends signals to the music system as dis- 2. BACKGROUND sic and gameplay of digital games, so gameplay proce- crete state-transition cues: The music transitions to “boss Figure 1. Otocky, in which notes and instruments are tied durally varies the music, and music procedurally varies Our goal of architecting the gameplay–music coupling as music” when the player enters a boss battle, or to “hurry to weapons. the game. Our goal is to inject game music more closely up” music when a timer runs low (the states may of course a bidirectional feedback loop will no doubt sound famil- into the core of gameplay, rather than having the music be more complex, and there are often substates as well).1 iar to audio installation artists. In contrast to games’ lim- serve as an aesthetic layer on top; traditionally, music re- More recently, dynamic game music aims to make this ited experimentation with such feedback loops, feedback to it, making the soundtrack directly player-influenced. sponds to game state, but not vice versa. Such a coupling interaction more complex—but still one-way. Instead of between audiences and generative music systems (often Otocky’s playable notes always map to a note belonging has compositional and design implications: composing pre-composed scores that are cued by a finite set of state conceptualized as a form of cybernetic coupling) is a com- to the mode and harmony in the current accompanying game music becomes a kind of composition of gameplay, transitions, gameplay events and changing game state feed mon strategy deployed and explored by interactive sound layer, so that player-created melodies will never contain and furthermore, game-design decisions feed back into into parameters of a generative-music system [2]. Intrigu- installations. Therefore, in a sense our work can be seen as notes that sound “wrong” or “off” from the perspective of the music-composition process. We discuss an arcade- ingly, from the perspective of our current interests, in mu- part of a recent trend in experimental digital games, which traditional harmonic structures. Furthermore, shooting is style 2d side-scrolling game, Audioverdrive, demonstrat- sic games such as Guitar Hero the influence is the other canvasses electronic art for ideas and design elements to quantized to the beat so that all notes played will fit the ing this integrated music/game composition approach. way around. There, gameplay takes input from the music: selectively borrow [8]. rhythmic structure. the music is in effect the “level design”, specifying what Despite intriguing connections to audio installations The mapping here is still mostly one-directional: mu- 1. INTRODUCTION the player will have to do to pass a level. and interactive electronic art more broadly, we see our- sic is dynamically generated from player actions, but does selves as situated primarily in game design as a starting not then feed back into the gameplay. However, musical Music is a key part of the culture and aesthetics of digital 1.2. Bidirectional game–music communication point. The distinction is admittedly not a clean one [10], considerations implicitly feed back into gameplay design games, so much so that it often spills out of games proper, but we find it productive here to start with existing game- through the constraints that were added to make “playing” Why not throw music right into that vortex of multidi- and into popular music culture. Games’ soundtracks form design and game-music composition practices, and exper- produce the desired effect. This is seen most clearly in the rectional close coupling and feedback that makes up the a large part of their overall feel, and the cultural attach- iment with adding bidirectional coupling between the two. shooting quantization. Although implemented straightfor- heart of gameplay? That’s our long-term goal: game mu- ment they engender is such that fans flock to see sym- It’s possible the result may converge nearer to electronic wardly in the gameplay domain as a quantization of shots, sic drilled into in the core of a game’s dynamics. phonies perform the soundtracks [6]. Meanwhile, the par- art, especially in particular designs using feedback loops which in turn results in the notes produced by the shots be- In this paper, we ask something closely related yet ar- ticularized aesthetic sound signatures of the sound chips aesthetically modeled on those common in cybernetic art. ing quantized, clearly the purpose of the constraint is the chitecturally simpler. We do maintain the nicely sealed in systems like the Atari VCS have inspired modern-day But so far, in our own use of these experimental tools (see audio-domain quantization. It is therefore best thought of computational boundary between “the game system” and chiptune and bitpop musicians to repurpose the hardware Section 4), the result still feels much like game design and conceptually as a constraint in the audio domain, which “the music system”, in part so we can reuse existing tech- for music-making outside the gameplay context [4]. game-music composition, albeit in a weirdly coupled way. travels “backwards” through the shot-to-note mapping to nology on each side. However, we aim to break the com- Despite this rich cultural spillover outside of digital produce a constraint in the gameplay domain. The con- positional boundary: the two systems communicate in a games into a ferment of influences and cross-influences, 2.1. The composition-instrument straint here is fairly simple to hand-code in either domain, pervasively bidirectional manner, with neither layer treated when we examine what digital game music does back at but with more complex musical constraints it is easy to as subsidiary. This produces a closely coupled system The existing style of game design closest to our goal of home, in its original habitat of games, it usually plays a see how less obvious interplay may arise. strangely cautious role, with only selective involvement. with complex interaction patterns and feedback between bidirectional coupling is probably the one theorized by In games, we can locate the center of action, where the gameplay and music. Our particular interest is in treating Herber [7] as a composition-instrument. In a composition- 2.2. Bidirectional procedural content generation ferment of influences and cross-influences happens in this this closely coupled system as a unified compositional sit- instrument, the player can “play” or “compose” music in uation. In a quite direct sense the composer of game music real-time while playing the game. In contrast to music- medium, in the closely-coupled mess made up of game- In addition to the experiments with dynamic and genera- becomes a composer of gameplay—and in the other direc- matching games such as Guitar Hero, the player generates play, interaction, and system dynamics, full of feedback, tive game audio already discussed, there has also, since tion, gameplay design becomes a kind of music design. (part of the) music through gameplay rather than matching emergence, and interlocking effects. But game music typ- the early 1980s, been work on procedurally generating Our contributions to enable this integrated composi- gameplay to pre-defined music. ically only tiptoes around the edges of that nexus, adding game levels and other content. For example, classic games tion process are a framework for bidirectional gameplay– A particularly intriguing example from 1987 stands an aesthetic layer to it but not getting caught in any dan- like Civilization, Rogue and Elite feature content that is music communication, and Audioverdrive, an arcade-style out, Otocky.2 A sidescrolling arcade shooter for the Nin- gerous feedback. automatically generated rather than created by a human game designed and composed using the framework. The tendo Famicom Disk System, it places players in con- designers. In some cases, levels are randomly generated, framework connects games programmed in Cocos2D, a trol of a flying avatar that fires a short-range ball pro- 1.1. Game-to-music communication while in other cases a human player or designer is given jectile in order to deal with enemies. Each of the eight 1This style of gameplay-driven, state-based music transition was pi- some kind of control over the generated content [9]. In possible shooting directions has a musical note attached The traditional way game music is coupled with gameplay oneered by the LucasArts iMuse system in the early 1990s [12], and other cases, content is generated based on external data. is by receiving information from the core game system. remains the dominant mode of game-to-music coupling [1, 3]. 2Toshio Iwai, SEDIC/ASCII, 1987. We discuss only Otocky here, as In one example, Monopoly boards are generated based on a pioneer of the style; additional games are discussed elsewhere [7, 11].