Game Sound Technology and Player Interaction: Concepts and Developments

Mark Grimshaw University of Bolton, UK

InformatIon scIence reference Hershey • New York Director of Editorial Content: Kristin Klinger Director of Book Publications: Julia Mosemann Acquisitions Editor: Lindsay Johnston Development Editor: Joel Gamon Publishing Assistant: Milan Vracarich Jr. Typesetter: Natalie Pronio Production Editor: Jamie Snavely Cover Design: Lisa Tosheff

Published in the United States of America by Information Science Reference (an imprint of IGI Global) 701 E. Chocolate Avenue Hershey PA 17033 Tel: 717-533-8845 Fax: 717-533-8661 E-mail: [email protected] Web site: http://www.igi-global.com

Copyright © 2011 by IGI Global. All rights reserved. No part of this publication may be reproduced, stored or distributed in any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher. Product or company names used in this set are for identification purposes only. Inclusion of the names of the products or com- panies does not indicate a claim of ownership by IGI Global of the trademark or registered trademark.

Library of Congress Cataloging-in-Publication Data

Game sound technology and player interaction : concepts and development / Mark Grimshaw, editor. p. cm. Summary: "This book researches both how game sound affects a player psychologically, emotionally, and physiologically, and how this relationship itself impacts the design of computer game sound and the development of technology"-- Provided by publisher. Includes bibliographical references and index. ISBN 978-1-61692-828-5 (hardcover) -- ISBN 978-1-61692-830-8 (ebook) 1. Computer games--Design. 2. Sound--Psychological aspects. 3. Sound--Physiological effect. 4. Human-computer interaction. I. Grimshaw, Mark, 1963- QA76.76.C672G366 2011 794.8'1536--dc22 2010035721

British Cataloguing in Publication Data A Cataloguing in Publication record for this book is available from the British Library.

All work contributed to this book is new, previously-unpublished material. The views expressed in this book are those of the authors, but not necessarily of the publisher. 22

Chapter 2 Sound for Fantasy and Freedom

Mats Liljedahl Interactive Institute, Sonic Studio, Sweden

AbstrAct Sound is an integral part of our everyday lives. Sound tells us about physical events in the environment, and we use our voices to share ideas and emotions through sound. When navigating the world on a day-to-day basis, most of us use a balanced mix of stimuli from our eyes, ears and other senses to get along. We do this totally naturally and without effort. In the design of computer game experiences, traditionally, most attention has been given to vision rather than the balanced mix of stimuli from our eyes, ears and other senses most of us use to navigate the world on a day to day basis. The risk is that this emphasis neglects types of interaction with the game needed to create an immersive experience. This chapter summarizes the relationship between sound properties, GameFlow and immersive experience and discusses two projects in which Interactive Institute, Sonic Studio has balanced perceptual stimuli and game mechanics to inspire and create new game concepts that liberate users and their imagination.

INtrODUctION put to play. In our work we use perspectives and methods from art, science, and technology and At the Interactive Institute, Sonic Studio in Piteå, we utilize digital technology as a vehicle for our Sweden, we do research on sound and auditory ideas and experiments. perception in order to find new ways to use sound, In a series of projects we have explored in- new contexts where sound can be utilized, and tuitive, emotional, imaginative, and liberating new applications for sound in general. Of special properties of sound. These projects have resulted interest to us is how sound resembles and differs in new insights and knowledge as well as in new from other sensory stimuli and how this can be and innovative applications for sound, audio, and technology. In this chapter I will describe our per- DOI: 10.4018/978-1-61692-828-5.ch002 spective on a number of sound properties and how

we have put these to work in various ways. The disbelief. To achieve this, the computer game projects are based on and inspired by an ecologic industry must build broader and broader bridges and everyday-listening approach to sound, like over the reality gap to make the virtual game reality the ones proposed by R. Murray Shafer, William more immersive. The traditional way to increase Gaver, and their followers. immersion and suspension of disbelief has primar- As human beings, we are good at interpreting ily been to increase graphics capability and, today the soundscape constantly surrounding us. When we can enjoy near photo-realistic, 3D-graphics in we hear a sound we can make relatively accurate real time. But there might be alternative ways to judgments about the objects involved in generating tackle the problem. Potentially, computer games the sound, their weight, the materials they are made could be more engaging and immersive without of, the type of event or series of events that caused having to build long and broad bridges over the the sound, the distance and direction to the sound reality gap. What about narrowing the gap instead source, and the environment surrounding the sound of building broader bridges over it? source and the listener, for example. Much of the existing research on sound and auditory perception is about how to convey clear and unambiguous bAcKGrOUND information through sound. In computer games, however, the aim is also to create other effects, Sound and light work in different ways and reach effects that have as much to do with emotions, the us on complementary channels. Our corresponding subconscious, intuition, and immersion as they do input devices, the visual and auditory perceptions, with clear and unambiguous messages. show both similarities and differences and we This article describes a couple of projects in have an innate ability to experience the world which we have worked with the balance between around us by combining the visual, auditory, touch eye and ear, between ambiguity and un-ambiguity, and olfactory perceptions into one, multimodal between cognition and intuition and between body whole. We are built for and used to handling and mind. The aim has been to create experiences the world through a balanced mix of perceptual built on a multitude of human abilities and af- input from many senses simultaneously. This fordances, mediated by new media technology. can be exemplified in different ways. One is by In a traditional computer game setting, the TV crossmodal illusions, for example, the McGurk screen or computer monitor is the center of atten- effect (Avanzini, 2008, p. 366) which shows how tion. The screen depicts the virtual game world and our auditory perception is influenced by what we the player uses some kind of input device, such as see. Another example is the ventriloquist illusion a game pad, a mouse, a keyboard, or a Wiimote, to in which the perceived location of a sound shifts remotely control the virtual gameworld and objects depending on what we see (O’Callaghan, 2009, and creatures in it. The action takes place in the section 4.3.1). If the signal on one sensory channel virtual world and the player is naturally detached is weak, we more or less automatically fill in the from the game action by the gap between the gaps with information from other channels and, player’s physical world and the virtual world of in this way, we are able to interpret the sum of the game. Much work has to be done and complex sensory input and make something meaningful technology used in order to bridge that gap and of that sum. Watching lip movements in order to to have the player experience a sense of presence hear what your friend is saying at a noisy party is in the virtual gameworld. The aim is to make the just one everyday example of this phenomenon. player feel as immersed as possible in the game A third example is Stoffregen and Bardy’s con- experience and to make her suspend her natural cept of “global array” (Avanzini, 2008, p. 350).

23 Sound for Fantasy and Freedom

According to this concept, observers are not its shape. Even if the individual has a previous separately sensitive to structures in the optic and experience and memory that connects the sound acoustic flows, but are rather sensitive to patterns of broken glass to the event of a broken bottle, that extend across these flows: the global array. the ear, not knowing the exact cause of the sound, Another way to describe this is that we do not might hesitate. Was it a bottle that crashed or was “see and hear” but rather “see-hear”, what we it perhaps a large drinking glass that broke? The perceive is the sum of sensations reaching our eye can give the correct answer, whereas the ear different modalities. is left to interpret and to guess in various degrees. What we really hear, what a sound is, where Tuuri, Mustonen, and Pirhonen (2007) have a sound is located and so forth are questions that continued along this path and propose a hierarchi- philosophers have been arguing over for several cal scheme of listening modes. Two of these are hundreds of years. O’Callaghan (2009) gives a preconscious, two are source-oriented, three are broad summary of the history and current state context-oriented and one is quality-oriented. In of the field. What most philosophers and sound the two preconscious-oriented listening modes, the researchers agree on is that sounds are the result focus is on what reflexive, emotive and associative of events in the physical world. Sound holds responses a sound evokes in the listener. In the information about these events and the objects two source-oriented modes, the focus is on how involved in them. This means that to our percep- the listener perceives the source of a sound and tion, sounds are strongly linked to the physical what event caused it. In the three context-oriented world and we are “hard wired” to treat sounds as modes, the focus is on whether the sound had a tokens of physical activity, matter in motion and specific purpose, if it represents any symbolic or matter in interaction. conventional meaning, and if the sound in that case In this context the pioneering work of William was suitable and understandable in the context. Gaver (1993) on sound classification and listening In the last, quality-oriented listening mode, the modes is still often cited and relevant for game focus is on the acoustic properties of the sound, sound design. Gaver makes a distinction between its pitch, loudness, duration and so forth. To use musical listening and everyday listening. In musi- these or other, complementary, identified listen- cal listening, you listen to the acoustic properties ing modes is a powerful way to inform the sound of the sound, for example, its pitch, loudness, and design process of not only computer games, but timbre. In everyday listening, on the other hand, sound design processes in general. The important you listen to events rather than sounds. When you thing to notice here is that research on listening hear a car passing by or you hear a bottle breaking modes in general shows that sound can indeed you do not pay much attention to pitch or loud- be used to evoke emotions and associations, to ness but more to the event as such. In everyday communicate properties of physical objects and listening, the interpretation and the mapping of events and to convey meaning and purpose. sounds to the individual’s previous experiences Already from the time before we are born our and memories are crucial. When a bottle crashes auditory perception starts giving us information against the floor, loses its original shape and turns about the world around us (Lecanuet, 1996). From into a number of smaller and larger pieces, this day one we start building our library of associations is immediately obvious to the eye. But, in order to individual sounds and to whole soundscapes. to be able to pinpoint the event that caused the Gradually, we learn what they mean and we train sound of the broken bottle, the ear has to learn and our ability to interpret them. Furthermore, some form a memory that connects the sound of broken researchers argue that we experience sounds “as glass to the event of a bottle crashing and losing of” a bigger whole. O’Callaghan (2009) argues

24 Sound for Fantasy and Freedom

that the sound of hooves of a galloping horse Gaver (1997). This is true also when it comes to is not identical with the galloping. Instead, it is sound in computer games but, in this context, the part of the particular event of galloping: “Audi- need to interpret and disambiguate the computer tory perceptual awareness as of the whole [sic] game system is not the only aspect of the issue. On occurs in virtue of experiencing the part” (part the contrary, some authors argue that ambiguity 2.3.2). This strong linkage between the sounds and the need to interpret a system instead can be we hear and the physical world we inhabit can used as an asset (Sengers & Gaver, 2006; Sengers, be brought into play in computer games through Boehner, Mateas, & Gay, 2007). Here, we argue rich soundscapes in order to convey information that this is certainly the case. When the ideas of about objects, environments, and events in the ambiguity and interpretation are combined with game world. Try this simple experiment. Pick the concepts of flow and GameFlow described an environment with a reasonable number of ac- below, the sum can be used to inform the game tivities, people, birds, machines or whatever you design process in new ways. can find that makes everyday sounds. Close your Development of computer games has so far eyes and try not to interpret, make associations mostly been geared towards vision. When it comes and create mental pictures from what you hear. to sound in games, much of the work is inspiring It is very hard to put the auditory interpreter to case studies but less research. Sweetser and Wyeth rest and this is true for sounds from all types of list three aspects of usability in games that have sources, including the headphones playing sounds previously been in focus for research (Sweetser from your iPod. This interpretation, mapping, or & Wyeth, 2005). These are interface (controls and disambiguation of individual sounds and whole display), mechanics (interacting with the game- soundscapes involves high-level mental processes world), and gameplay (problems and challenges). related to our conscious and subconscious, cogni- Lately, also other dimensions of the design and use tive and emotional layers. As such, these processes of computer games have started to gain interest have the potential to invoke a myriad of physical among game researchers, dimensions that incor- and mental responses: fear, flight, well-being, porate new and more complex aspects and ideas happiness, anger, understanding and so on. In of player enjoyment and computer game design. computer game design, this means huge potential Several research groups have, for example, made to both convey cognitive meaning and to create connections between interactivity in general and, moods and affect. more specifically, player enjoyment in games on Auditory perception can be understood as be- the one hand and the concept of flow developed coming aware of the whole by virtue of the parts. by Mihaly Csíkszentmihályi on the other. In the Sounds can also be said to be more ambiguous and 1970s and 1980s Csíkszentmihályi conducted leave wider space for interpretation than visual extensive research into what makes experiences stimuli do, at least when it comes to interpreting enjoyable. He found that optimal experiences are where and what we have heard. In Human Com- the same all over the world and can be described puter Iinterface (HCI) contexts, ambiguity has in the same terms regardless of who is enjoying often been thought of in terms of disadvantage the experience. He called these optimal experi- and problem (Sengers & Gaver, 2006) and much, ences flow. A flow experience is defined by perhaps even a majority, of the research done in Csíkszentmihályi (1990) as being “so gratifying the field has tried to overcome this and find ways that people are willing to do it for its own sake, to create clear and unambiguous systems and in- with little concern for what they will get out of terfaces. Research on sound interaction design is it, even when it is difficult, or dangerous” (p. 71). no exception to this, as described by, for example,

25 Sound for Fantasy and Freedom

Judging from the volume and type of work be used in sound design for games is covered in built on and derived from Csíkszentmihályi’s more detail below. flow principle, it can be argued that the concept A number of research projects report on sound is relevant in the context of computer games. and audio’s ability to create rich, strong and im- Andrew Polaine (2005) has written about The mersive experiences using mobile platforms that Flow Principle in Interactivity. This work does give physical freedom to the users. These projects not relate to computer games per se, but is also support the general idea that sound and audio closely related to the subject in that it connects are well suited for use in the design of computer flow with both “willing suspension of disbelief” game experiences based on the GameFlow model. (a term borrowed from narratives in theater and Reid, Geelhoed, Hull, Cater, and Clayton report on film) and the experience of play. The GameFlow a public, location-based audio drama called Riot model developed by Sweetser and Wyeth builds 1831. The evaluation of the project showed that a directly on the concept of flow and is a model for majority of the users had rich and immersive expe- evaluating computer games from an enjoyment riences created from the sounds of an audio-based perspective. Another example is Kalle Jegers’ narrative. Based on the results from this project, (2009) “Pervasive GameFlow” model that takes the authors argue that “immersion is a positive Sweetser’s and Wyeth’s GameFlow concept to the determinant for enjoyment (and vice versa)” (Reid, pervasive game arena. A final example is Cowley, Geelhoed, Hull, Cater, & Clayton, 2005). It should Charles, Black, and Hickey’s (2008) USE model be noted that the drama took place in a square in (User, System, Experience) that looks at games, Bristol, UK, which gives this project similarities player interaction, and flow from an information to pervasive and location-based games where system perspective. the virtual gameworld and the physical world of Built on the Flow concept, Sweetser and Wy- the player are blended. Friberg and Gärdenfors eth’s GameFlow model consists of eight elements (2004) report on a project in which three audio- for achieving enjoyment in games. The model based games (what the authors term TiM games) can be used both when designing new games and were developed. Based on audio communication when evaluating existing game concepts. In sum- with the users, the authors report that these games mary, according to Sweetser and Wyeth, games give the users spatial freedom, encourage physical must keep the player concentrated through a high activity and open up possibilities to create new workload. At the same time, the game tasks must types of interfaces for input to the game. Ekman be sufficiently challenging and match the skill et al. report on the development of a game for a level of the player. The game tasks must have clear mobile platform (Ekman et al., 2005). They point goals and the player must be given clear feedback out that sound and audio can indeed be used to on progression towards these goals. Enabling create immersion, but also that the use of sound deep yet effortless involvement in the game can does not automatically create immersion. Great potentially create immersion in the game. Accord- care must be taken when designing the game ing to Sweetser and Wyeth, experiences can be sounds and the developers must also carefully immersive if they let us concentrate on the task select the best technology and equipment to play of the game without effort. “Effortlessly” can, in back the game audio to get the desired effects. this context, be interpreted in several ways: one In two projects, the Interactive Institute’s Sonic way to think about it is in terms of how true to Studio has investigated how sound in games can real life a gaming experience is and how trans- be used to bring the user’s own fantasy into play parent the interaction with the game creating the to create new gaming experiences (Liljedahl, experience it is. How the GameFlow model can Lindberg, & Berg, 2005; Liljedahl, Papworth, &

26 Sound for Fantasy and Freedom

Lindberg, 2007). In both these projects the bal- MIND tHE GAP—sOUND FOr ance point between visible and audible stimuli FEEDbAcK AND IMMErsION from the game has been moved away from the visual and towards the audible. In both cases the Pictures are not the real world; they are merely users of the computer games are given only a the shadows of it. René Magritte’s provoking pipe minimum of visual information and are, instead, is a painting about exactly this: the picture of a given rich and varied soundscapes. The projects pipe and beneath it the text “Ceci n’est pas une have shown that the users have had rich and im- pipe” (This is not a pipe). We are surrounded by mersive gaming experiences and are given other still and moving images and we are used to treat- types and amounts of freedom compared to more ing pictures as pictures and not the real, physical traditional computer games. These projects will world. Even the most violent computer games and be described in more detail later in this chapter. Hollywood film productions are assumed to be Humankind has, in recent centuries, invested physically and mentally non-hazardous to us just considerable energy and creativity in creating because we are supposed to be able to discriminate complex technology. We have a long tradition between reality and the fictive picture of it. Sound, in replacing human capability with machinery. on the other hand, seems to work slightly differ- In the early days it was mostly muscle power ently. When striving for engagement, immersion, that was mimicked, replaced, and superseded by and suspension of disbelief in computer games steam, combustion, and, later, electricity. It can and films, sound, plays a very prominent role be argued that research into artificial intelligence and, according to Parker and Heerema (2007), is striving to do the same with human cognitive “sound is a key aspect of a modern video game”. and emotional capabilities. Following this long Natural sounds in the physical world are the result tradition, it seems that we often neglect human of events in that world and we become aware of capabilities, affordances, gifts, and needs when physical events to a large degree through sound. designing computer games and other systems. It can thus be argued that sound is a strong link Much of the focus has been on creating photore- to the physical world. In fact, Gilkey and Weisen- alistic 3D-environments in real time and less on berger argue that “…an inadequate, incomplete how the players’ internal, fantasy-driven, “sound or nonexistent representation of the auditory interpreter and mapper” can be put into play to background in a VE [Virtual Environment] may create complementary, mental images. In the compromise the sense of presence experienced by following I will describe how we at Interactive users” (quoted in Larsson, Västfjäll, & Kleiner, Institute, Sonic Studio work with finding ways 2002). It is this mechanism that is utilized when to increase user satisfaction and involvement creating the sound tracks to films and games. in gaming situations by using existing technol- Just seeing Donald Duck smash into a wall is not ogy in slightly new ways. Often, this has meant enough. It is not until the sound effect is added that moving complexity from technology to the user, the nature and the full consequence of the smash decreasing the demands on technology used, and are made evident to the audience. When we hear increasing the demands on the user to invest and the sound of the smash, all of us have our own, spend energy physically and mentally in a game slightly unique, experiences of and relationship experience. to the sound. The sound has the power to immediately trigger our interpretation machinery and evoke memories and fantasies. In a fraction of a second the sound makes us re-live our own experiences and we can feel what Donald feels:

27 Sound for Fantasy and Freedom

pain, anger, and humiliation. In this way it can be • Social Interaction. Games should support argued that the sound is playing us. Like a guitarist and create opportunities for social interac- plucking a string that generates sound, sound is tion (Sweetser & Wyeth, 2005). plucking our interpretation, spawning memories, understanding, and emotions. The string cannot As can be seen from the list, these criteria are stop the guitarist from plucking it and we cannot very general and could be applied to many aspects stop sound from triggering our understanding, our of life, from children’s play to high school educa- memories, associations, and emotions. tion, working life, and leisure. When it comes to For a computer game to be successful it is sound design for computer games, some of these crucial that the players can immerse themselves in criteria are more relevant than others. When look- the gaming experience and that they are invited to ing at Tuuri, Mustonen, and Pirhonen’s (2007) a gameworld and game experience in which they hierarchical listening modes, a clear link to the are willing to suspend their natural disbelief. After GameFlow concepts Feedback and Immersion all, World of Warcraft is not the real world. In their criteria can be found. Sweetser and Wyeth divide GameFlow concept, Sweetser and Wyeth (2005) the Feedback criterion into the following parts: set up a number of criteria that game designers and game researchers can use when designing and • Players should receive feedback on their evaluating games with respect to immersion and progress towards their goals suspension of disbelief. Some of these criteria • Players should receive immediate feed- are general, overarching principles that relate to back on their actions many human activities, while other criteria relate • Players should always know their status or more closely to gaming and the media used to score (Sweetser & Wyeth, 2005). convey the game’s metaphor and narrative. The GameFlow model lists the following criteria for The Immersion criterion is similarly divided player enjoyment in games: into the following parts:

• Concentration. Games should require • Players should be less aware of their concentration and the player should be surroundings able to concentrate on the game • Players should be less self-aware and less • Challenge. Games should be sufficiently worried about everyday life or self challenging and match the player’s skill • Players should experience an altered sense level of time • Player Skills. Games must support player • Players should feel emotionally involved skill development and mastery in the game • Control. Players should feel a sense of • Players should feel viscerally involved in control over their actions in the game the game (Sweetser & Wyeth, 2005). • Clear Goals. Games should provide the player with clear goals at appropriate times Given our ability to listen on several cogni- • Feedback. Players must receive appropri- tive abstraction levels, as indicated by Tuuri, ate feedback at appropriate times Mustonen, and Pirhonen’s hierarchical listening • Immersion. Players should experience modes, it can be argued that sound is well suited deep but effortless involvement in the to communicate feedback to the user and to sub- game stantially add to the game’s ability to immerse the player in the gaming experience. In the following

28 Sound for Fantasy and Freedom

we will look at how sound can be used and what behind, and other dangers. Our ears are under a sound properties could be brought into play in constant bombardment of auditory input from all order to give immediate and continuous feedback directions and we cannot simply turn away from to users, to help them become less aware of their a sound. To be able to handle all this information surroundings and themselves, and to help them and to avoid fatigue and sensory overload, we get involved in the game. handle most of the input subconsciously. Luckily, we also have the ability to focus on specific parts in the soundscape. We can, for ex- sOUND PrOPErtIEs At ample, isolate a conversation with a friend in a YOUr DIsPOsAL noisy restaurant from a dozen nearby, unrelated conversations. This is often referred to as “the There are a number of properties of sound as a cocktail party problem” (Bregman, 1990, p. 529). physical, acoustic phenomenon that, in conjunc- In GameFlow terms, the omni-directional tion with the inherent workings of our auditory qualities of sound relate to both feedback and perception and our ability to use different listening immersion. Sound for feedback from a game does modes, are at our disposal to use, explore, and not force the user to look at a special location on exploit when designing computer game experi- a screen: in fact, it does not require a screen at all. ences. Most of these properties are well known Sound is a strong carrier of emotions, events, and in everyday contexts and most people will im- objects, as discussed above. In our everyday lives, mediately be able to connect to the descriptions we are also used to being surrounded by sound. of them, have own experiences of them and to Mimicking this in a computer game scenario can understand the implications of them. These prop- make profound contributions to the immersive erties can, of course, be described in physical and qualities of the game. acoustic terms of frequency, amplitude, overtone spectrum, envelopes and so forth. Unfortunately Uninterruptible these terms say very little about our human experiences of sounds, sound sources, and soundscapes. Along the same line is the fact that we do not have It is therefore important to also describe sound “earlids” and cannot just shut our ears to get rid of properties in relation to how our hearing works. the sounds around us or choose to hear just one of The following is a summary of what we have the sounds of the total mass that reaches our ears. discussed above and an attempt to start making From an evolutionary point of view, it has been the discussion more concrete and applicable to an advantage to get early warnings and hear all sound design for computer games. dangers, not only the dangers you choose to listen to, but all. It also means that our eyes and our ears Omni-Directionality are designed differently and that the streams of sensory input from those senses complement and Sound is omni-directional and reaches our ears interact with each other. Again, this means that a from all directions (almost) simultaneously. Ac- constant stream of input data must be handled. The tively and consciously, as well as automatically and way to cope with this is to do it subconsciously. pre-consciously, we use this omni-directionality In our everyday lives we are submerged in to navigate in our everyday lives. Even though the ever-present stream of sounds from the world we do not have to look out for saber-toothed ti- surrounding us. By supplying a relevant and well- gers anymore, we are constantly warned for cars designed stream of sounds from a computer game, and buses from left and right, falling trees from the users can get constant and natural feedback

29 Sound for Fantasy and Freedom

on their actions, very much like in real life. This us, but we are also forced to process most of what in turn adds in a natural way to the sought-after we hear subconsciously. effortless immersion. Often, we do not know exactly what the source of a sound is or from what direction and distance sound connects to the it comes. We can hear a vehicle approaching from Physical World behind but have to guess what type of vehicle it is and how fast it is approaching. We can roughly tell Sound connects you to the physical world by tell- if it is a truck or a car and make educated guesses ing about physical objects and events that involve about when it will pass us, but usually not more physical objects. We can be described as hardwired than that. Sound leaves a relatively large space to perceive and automatically interpret sounds as within which we can (or are forced to) fill out results of events occurring in the physical world. the details ourselves and make assumptions and This is true even if the sound is mediated through interpretations based on our individual memories, a loudspeaker: our internal interpreter does not experiences and associations. make much difference between sounds from a When telling stories, making films or design- physical coffee cup being placed on a table and ing computer games, this ambiguity can be of the recorded sound of the same event played back great value. By planting a well-designed sound through a pair of headphones as long as the tech- at the right moment, you can trigger a person’s nical quality is sufficient. It is still a coffee cup imaginative and emotive mechanisms by forcing being placed on a table. As with the real-world her to consciously or subconsciously interpret and example you were asked to listen to above, try disambiguate the sound. Leaving the user space listening to a film with your eyes shut. It is vir- open to her own interpretation, inviting her and tually impossible to turn off the flow of images, giving her the freedom to use her own imagination feelings and associations flowing through you as can potentially help the user to be emotionally and you listen. You have to concentrate very hard on viscerally involved in the game. something else not to be affected by the sounds that reach your ears. The sound of a dentist’s drill sound reaches us on gives a direct bodily sensation and you can almost subconscious channels feel the drill in your own mouth. The picture of the drill alone, without the sound, does not have Our ears are constantly capturing the soundscape the same power over our imagination, emotions, around us. If all that data were to be processed by and physiology. the cognitive and conscious layers in our brains, Again, sound can be used to immerse the user we would either suffer from mental overload in the gameworld in a way that strongly resembles or have another brain constitution. But thanks the way we handle and work in everyday life. to the limited bandwidth of our consciousness, our subconscious, emotional and intuitive layers sound can be Ambiguous process most of the sounds we hear. This does not mean that we are not affected We constantly hear sounds from all directions and, by what our ears pick up and what our brains are to some degree, we can decide the direction and processing. What it does mean is that the effect the distance to the sound source. At will, we can is not totally controllable by us and that we are, consciously filter out discrete sounds of special to a large degree, victims of the sonic world. interest to us from the whole soundscape around Often this is useful, sometimes it is stressful and sometimes it is fun. We are more or less forced

30 Sound for Fantasy and Freedom

to interpret and react to what we hear. A sound and musicals, music and ambience, and music heard spawns meaning and interpretations based and sound effects in games and films. on our previous experiences. In games this can be extremely useful as a way to invite the players speech and Dialog to invest and get deeply involved in the game. This relates strongly to the GameFlow criterion When you want to convey a clear and unambiguous “immersion” described above. message, the human voice is a natural choice. The same is true if you want to tell a riddle or recite a poem or just want to be vague and ambiguous. sOUND tYPEs At YOUr DIsPOsAL Human language is so rich and there are a myriad of ways to use this in computer game contexts. There are a number of ways to categorize and clas- Speech and dialog can be used to address several sify sounds. In this context it makes sense to use of the criteria for player enjoyment included in the the three categories traditionally used for sound in GameFlow concept. They can be used to promote films and computer games (Sonnenschein, 2001; concentration on the game by providing a comple- see also Hug, 2011; Jørgensen, 2011 for more mentary source of stimuli, getting the player’s involved taxonomies of computer game sound): attention without disrupting the player’s visual focus, or spreading the total game workload on • Speech and dialog. Human language complementary channels, for instance. Sometimes brought to sound, the sounding counterpart it is necessary to give instructions to the player to the visual text. The most cognitive and on what to do next, or what is expected from the unambiguous of the three types often used game. If you do not want to exclude the player to convey clear messages with least pos- from an ongoing game sequence or if you have sible risk of misunderstanding problems with limited screen size, using speech • Sound effects and the subcategory ambient as a complement to text is one solution. Today, sounds. The result of events in the physical more and more computing and gaming platforms world. A falling stone hitting the ground; have built-in support for voice recognition, which air fluttering in the feathers of a bird; a means that the player can control the game by mechanical clock ticking; a heavy piece issuing voice commands. Since this is totally in of frozen wood dragged over a horizontal, line with what we do in our everyday lives, it dry concrete floor; the ever-present, ever- also supports a very natural way to co-create the changing sounds of the atmosphere game world and to get a desired sense of impact • Music. Sometimes referred to as “the lan- upon it. Speech is a natural way to get feedback guage of emotion”. An integral part of from a game on player progress and distance to human cultures since the dawn of Homo game goals without having to force the player to sapiens. shift visual focus to get the necessary feedback. Speech and human voices are totally natural parts Note that these categories are only for clarity of human society and of everyday lives. The hu- and discussion. It is important to point out the fact man voice is therefore very well suited to making that, in reality, the possible borders between them the players forget that they are participating in the are floating. he borders between music particularly game through a medium and it helps to make the and the other two categories have been blurred for game interface less visible and less obtrusive to centuries: for example, music and dialog in opera the player. Voices can therefore be integrated into

31 Sound for Fantasy and Freedom

the background soundscape of the game to give as freed and part of the physical world through a sense of human presence. the added sound. Apart from the above-mentioned rather objec- Friberg and Gärdenfors use a number of tive and technical uses of speech and dialog, all categories for the sounds in the TiM games variations of subjective, expressive and dramatic mentioned above. Most of their categories can qualities of the human voice are also available. be seen as subcategories to the traditional sound A bad result uttered with an offensive voice will effect category. The categories listed by Friberg be something radically different from the same and Gärdenfors (2004) are: result uttered with a friendly and supportive voice. Here, the thin border between computer games, • Avatar sounds refer to the effects of avatar film, theater and other narrative media is clear. activity, such as footstep sounds, shooting or bumping into objects sound Effects Make it real • Object sounds indicate the presence of objects. They can be brief, recurring sounds Events in the physical world generate sounds. It or long, continuous sounds, depending on is actually very hard to live and be active in this the chosen object presentation world without giving rise to sounds. Sounds heard • Character sounds are sounds generated by in the physical world are the results of events non-player characters involving physical objects. Explosions in a com- • Ornamental sounds are sounds that are not bustion engine, oscillations of the vocal cords in necessary for conveying gameplay infor- your throat, putting down your cappuccino cup mation, such as ambient music, although on the saucer. Sounds are the proofs that you are they enrich the atmosphere and add to the still firmly attached to the physical world of your complexity of the game. senses. The absence of sound, on the other hand, could be the sign that what you are experiencing In GameFlow terms this means that sound is not real, that it is a dream or virtual reality. effects and ambient, background sounds can add A green rectangle silently moving over a to several of the criteria for player enjoyment. computer screen is probably perceived as just Presenting a lot of stimuli to the player on vari- a green rectangle on the screen. But if you add ous channels is crucial for the ability of the player the sound of a heavy stone dragged over asphalt to concentrate on the game. We are also used to to this simple animation, the green rectangle au- constantly interpreting the soundscape surround- tomagically turns into a heavy stone. Sound and ing us, and a well designed game soundscape will computer game audio is a bridge on which the have great potential to grab the player’s attention virtual visual worlds can travel out and become and help them focus on the game. Sound effects part of the real, physical world. are today absolutely necessary for feedback to Ambient or background sounds are the sound- the players of computer games. Everything from ing counterparts to the graphic background. Hav- game control commands issued by the player to ing no ambient sounds is like having a pitch-black virtual events caused by non-player characters can visual background and can be perceived as an be signaled and embodied using sounds. almost physical pressure on the ears. Adding just Sound effects and ambient sounds are very a virtually inaudible ambient sound to the virtual important for player immersion and to involve world of a computer game can create an immedi- the player emotionally and viscerally in the game. ate experience of presence and reality. The silent Many of the sound stimuli that reach our ears are virtual world that was locked in can be perceived processed subconsciously and handling sound

32 Sound for Fantasy and Freedom

on this level of perception is totally natural to municate the same thing to two persons. But if the us. This fact also supports the idea that sound is music is paired with something else, for example, a very well suited to adding to the total experience film or a game, something happens. People that are of immersion in the game world. said to hate classical music, and would never put on a recording of classical music, can spend hours Music Makes You Feel watching films with music tracks firmly grounded in western classical music tradition, sounding like Sound in general and music in particular have a something composed in the late 19th Century by very strong ability to touch our feelings. Music Richard Wagner or Gustav Mahler. When musical works emotionally in two significant ways. Firstly, sounds meet other sensory inputs, for example, it tells us stories about feelings that we do not music in an animated film, the individual stimuli necessarily feel ourselves: the music works like tend to blend together and become a new whole. sounding pictures of emotions (Gabrielsson & The “film + music” object is perceived as being Lindström, 2001, p. 230). Secondly, music can radically different from the film alone and the have the power to induce feelings in us, that is, music alone. The music becomes more universal to actually make us feel (Juslin & Västfjäll, 2008, and has the ability to communicate relatively p. 562). Today, the borders between music, sound universal values, emotions, and moods. effects, ambient background sounds and voices Music is normally a very linear phenomenon: become more and more blurred and music is used a song starts at A and ends at B, and the journey as sound effects and sound effects can be used as between the two is always the same and takes the music. It can therefore be hypothesized that the same amount of time to travel each time. This is emotional qualities of music are also, to some especially true of recorded, mediated music. In extent, true for other types of sounds. a non-linear and interactive context, this linear Research has shown that music alone, in the music concept does not necessarily apply. Most absence of supporting pictures or other sensory often, music has a form that creates successions of input, can in many cases and for a majority of tension and relief, which in turn creates expecta- people induce feelings of happiness and sadness. tions on how the music will continue: the music Most people can also accurately tell if a piece of can therefore not be altered as quickly and easily music is composed and intended to express sad- as other media. To function and be perceived as ness or happiness. Other, more complex emotions music, it has to follow at least some basic musical like jealousy or homesickness are harder to dis- rules of form and continuity. tinguish: Music, alone, has less power to induce A number of techniques and systems have such feelings and to actually make us feel them. been developed to cope with the gap between However, if you add pictures and other media linear music and non-linear environments. Many to the musical expression, the musical power of these are proprietary systems developed by the increases exponentially. commercial game developers and are not available Auditory perception tends to dominate judg- to the general public. What most of the systems ment in the temporal dimension (Avanzini,, 2008, seem to agree on is a division between a vertical p. 390). Music is a special case of this, since it is and horizontal dimension. The vertical dimension sound that is highly structured in time. By syn- controls aspects of musical intensity and emotion chronizing sound and visual movements, very and the horizontal dimension controls aspects of strong effects can be created. time and form. The vertical dimension is often Some of the music we hear affects us very implemented using a layered approach whereby individually: it is not universal and does not com- a number of musical tracks play in parallel. Each

33 Sound for Fantasy and Freedom

track plays music with a certain content represent- and awareness about vision, graphic design and so ing a level of intensity or emotion and the game forth is also remarkably higher, more general and engine cross-fades between the tracks to create more common than their sounding counterparts, as the correct blend of intensity and emotion. The are the creative tools available. In the Association horizontal dimension is often implemented using for Computing Machinery’s Computing Classifi- short phrases of 1, 2 or 4 bars linked together. cation System (2010), sound and audio are added When a transition from one musical segment to late compared to, for example, computer graphics. another is motivated by the state of the game, the Sound and audio are also mentioned on a lower current phrase is played to the end and the chain level (level three) in the classification system, of linked phrases takes another route than if the whereas computer graphics is a level two item. game state had not changed. balance the senses 3D-Positioned Audio Our eyes play a dominant role in our everyday lives Since sounds are the results of physical events and computer game development has tradition- in three-dimensional space, it is often vital to ally put most emphasis on graphics and vision. At be able to give the impression of game sound the same time, other modalities and media types as emanating from a certain point in a 3D space such as sound and hearing can be described as (see Murphy, 2011). 3D-positioned audio is a underused. This suggests that new computer game powerful technique to bridge the gap between concepts could potentially be found by chang- the virtual game reality and the physical world ing the balance between modalities and media of the player’s senses. This is especially true for types. What happens for example if we reduce sound effects but is also very useful for speech graphics and visual stimuli and instead build the and dialog. Music and ambient sounds are most gaming experience more on sound and audition? often not 3D-positioned. What would the effect be if you had a computer game with only an absolute minimum of graphics and instead a rich, varied and gameplay-driving sOUND FOr FANtAsY soundscape? Potentially such a game would be AND FrEEDOM immersive in other ways and give different types of game experiences compared to more traditional, We cannot hear away from a sound like we can graphics-based games. A couple of things im- look away from an object, and we have no “earlids” mediately become obvious. First of all, the game to shut as we can our eyelids. These simple facts designer must let other qualities than computer makes sound ideal to use if you are looking for new graphics build and drive gameplay. Secondly the game concepts to contrast the traditional screen player is liberated from the need to keep her eyes on and eye-based computer games. Western societies a 20-something-inch rectangle (in mobile applica- are often said to be vision-based or eye-centric. tions only a few inches). Instead, all of a sudden, This suggests that we rely mostly on our eyes and she becomes free to move over much larger areas use our other senses and abilities more or less just or even volumes. Both these open up possibilities as support for what we see. In language this is to create radically new types of computer games reflected in that we “watch” things. We “watch” for radically new computer game experiences. TV and films despite silent movies being history They also represent new challenges for both game since the 1930s. We even “watch” music concerts designers and computer game players (see Hug, (at least this is true in Swedish). Our knowledge 2011 for an expansion of such ideas).

34 Sound for Fantasy and Freedom

Our auditory perception is good at interpret- based on questions related to the ideas outlined ing sounds as tokens of events. When we hear a above. These projects have shown that by shifting sound we know something has happened, matter the balance between graphics and other media has interacted with matter. The sounds of broken types and between eyes and other modalities, glass, of cars colliding, of footsteps, our own games with new qualities can indeed be created: breathing, and combustion engines all contain games that attract new user groups and games information about materials, weights, speeds, that can be used in new contexts, in new ways surface roughness and so on. In our everyday lives and for new purposes. we are constantly immersed in a soundscape that In this context it is also relevant to make a we receive through two streams, one in the left distinction between gameplay or game mechanics ear and one in the right. From day one we start and metaphor. Gameplay can be defined as the set training our perception in order to be able to make of rules and the mechanics that drive the game, priorities and pick out the relevant information the game’s fundamental natural laws. Metaphor from these two streams. Since sound reaches us on the other hand defines the world in which these from all directions, it can be hypothesized that most abstract laws work. Gameplay can, for example, of the events we hear, we do not see. In the light define that you are able to navigate in 4 directions of the above, it becomes natural to use sounds as called north, south, east and west, that you will means to convey feedback on both player actions be presented with challenges you can either win and other events occurring in the virtual world or lose, and that you win the game by winning of a computer game. Since sound tells us in a a defined number of these challenges. Metaphor totally natural way about things we do not see, defines the world in which the navigation takes sound can be used to expand the game world far place and the nature of the challenges. When beyond what is displayed on a screen. Sound is gameplay defines an abstract challenge, metaphor very well suited for delivering the feedback and can, for example, show an enemy soldier that creating the immersion necessary for successful must be eliminated or it can present the player game concepts, as described by the GameFlow with a falling egg that must be caught before it concept above. hits the floor. A good game must have both a well- The use of sound to convey information about designed gameplay and a metaphor that supports events, creatures and things that are not visible adds that gameplay: both are equally important. Often, yet another dimension to the game experience: the sound designer works with the metaphor side imagination, a word originally meaning “picture of a game. The metaphor chosen dictates the to oneself”. When we hear a sound without seeing possibilities available to the sound designer. A the sound source we make an interpretation of metaphor with a large number of natural sounds what we have heard. The interpretation is based that the players are likely to be able to relate to is on previous experiences of memories of and as- potentially more immersive than a metaphor with sociations to sounds with similar properties. The few and/or unknown sounds. interpretation is often subconscious and made without effort. To invite the players to use their two case studies imagination, fantasy, and associations to fill out the gaps in this way and complement what they In two projects, alternative ways to balance visual see on the screen is one way to make the players and audible stimuli in computer games have been emotionally and viscerally immersed in the game. explored by the Interactive Institute, Sonic Studio. In a series of research and development projects In the first project, called Beowulf, a game for de- we have conducted investigations and experiments vices with limited screen size, such as cell phones,

35 Sound for Fantasy and Freedom

Figure 1. The Beowulf game window

was developed (Liljedahl, Papworth, & Lindberg, and differences just as in real life. Feedback on 2007). In this project, the hypothesis was set up player actions and progress is given by footstep that a game with most of the graphics removed, sounds, breathing sounds, the sound of a swinging having, instead, a rich, varied and challenging sword, and other sounds natural in the context of soundscape, can create a new type of immersive the game’s world metaphor. Immersion is created game experience. The hypothesis also included through the natural and effortless interaction with the idea that a game built mostly on audio stimuli the sounding dimension of the gameworld. will be more ambiguous and open for interpre- In the second project, called DigiWall, the tation than a game built on visuals and that the computer monitor was removed totally (Liljedahl, need for the users to interpret and disambiguate Lindberg, Berg, 2005). Instead, a computer game the soundscape will create a rich and immersive interface in the form of a climbing wall was de- game experience with new qualities compared to veloped (see Figure 2). The 144 climbing grips traditional computer games. The game uses both a are equipped with sensors reacting to the touch well-known gameplay and a traditional metaphor of hands, feet, knees, and other body parts. The to keep as many parameters as possible constant. grips are also equipped with red LEDs and can Although the gameplay is very simple, the game’s be lit, turning the wall’s climbing area into an sound-based metaphor makes it a both challenging irregular and very low-resolution visual display. and rewarding game to play. A number of games were then developed based The Beowulf game world is graphically repre- on a balanced mix of sounds, physical activity, sented by a revealing map, a map showing only and the sparse visuals of the climbing grips. The the parts of the game world you have visited so absence of traditional computer game graphics far as a red track (see Figure 1). Your position and the shift in balance between modalities and in the game world is indicated by a blue triangle media types gives another effect: the games be- pointing in your current direction. The player uses come open for the players to adapt to their own headphones to listen to the gameworld, which is level of physical ability, their familiarity with the described in much greater detail audially than games, how they chose to team up, to create visually. The player navigates this gameworld by variation and so on. In this sense, the new balance listening to sound sources positioned in a 3D space. between modalities and media types means new Navigating includes localizing sound sources freedom for the players. by turning and moving to experience changes

36 Sound for Fantasy and Freedom

Figure 2. DigiWall climbing wall computer game interface

Both projects explored questions related to gameworld’s environments, materials, tempera- how computer game players could be offered new ture, atmosphere, inhabitants and so forth none of and unique gaming experiences in terms of free- which had any visible cues. In the DigiWall case, dom and fantasy. In Beowulf the hypothesis was positive user investment ranked highly both in that a shift in balance between eye and ear would player satisfaction and the subsequent publicity invite the players to co-create the game experience and commercial success of the project. and to bring their imagination into play in new In these projects, audio is used in a number ways, compared to traditional, graphics-based of ways to create a sense of presence and to link, games. The studies performed on the game concept as closely as possible, the virtual reality of the showed that, to a majority of players, this was game to the physical reality of the player. Sound also the case. The DigiWall concept is based on was also used to communicate instructions, cues, the players’ freedom to use their whole bodies clues, feedback, and results from the game to the and to play the games by moving over the whole, player. The aim was to create new balances be- 15m2 game interface. The absence of a tradi- tween sound and graphics compared to traditional tional computer monitor also opens up the rules computer game applications and to explore if and of play in such a way that the users are invited to how sound could be used to drive gameplay and co-create and adapt the basic gameplays offered to create fun, challenging, rewarding and im- to their own needs and desires. mersive gaming experiences. The aim was also to In this context it is also important to mention the use sound to blur the borders between the virtual term “user investment”. Both projects eventually reality of the game and the physical reality of the showed that the need to interpret and disambigu- player. In both cases, game metaphors were chosen ate the soundscape of the games was in fact an to match the gameplays and to present as many asset. Both games more or less forced the players possibilities and large design spaces as possible to use their own imagination and experiences to for the sound designers. flesh out the sounding skeletons supplied by the Here follows a brief description of how sound game’s metaphors. In the Beowulf case, the user was implemented in the two projects. investment was expressed as high-ranking in game satisfaction as well as in vivid descriptions of the

37 Sound for Fantasy and Freedom

Ambience and background players and, for example, “whisper” that speed to bridge the reality Gap is increasing or that time is running out and you must hurry. Physical environments are (almost) never silent. Air, water, objects, creatures and machines around sound Effects and Music us all more or less make sounds. The absence of for cues and clues sound is unnatural and scary; it is an auditory counterpart to pitch black. Sounds are the signs of Often game designers want to encourage the play- presence, life and function. By adding just a very ers to go in certain directions or to take certain soft sound of moving air, an otherwise dead and actions. By carefully planting sound effects and/ detached game environment can come alive. If or music, the player can be guided, inspired or the sound is well designed, it is possible to create even intentionally misled. Beowulf uses a large an experience where the game-generated sounds number of natural sounds to warn the player of blend with the sounds from the gamer’s physical potential dangers such as predators, bottomless environment, creating an inseparable whole. The holes or boiling lava. The DigiWall games use gap between the realities closes. music and sound effects with musical properties Ambient sounds can be strong carriers of to guide attention in certain directions on the emotion and mood. They share this ability with wall. One example is the game Catch The Grip, music and the fact is that the border between the in which the direction from the last grip caught two is more and more often blurred by film and to the next to catch is represented by a series of game sound designers (Dane Davis, cited in Son- notes. The length of the series tells the physical nenschein, 2001, p. 44). Carefully “composing” distance on the wall. The panning of the notes in an ambient or background sound can serve several the loudspeaker system signals the direction left/ purposes at the same time. It can create a sense of right. In the game Scrambled Eggs, sound effects physical presence, it can set the basic mood and with a falling pitch denote the movement of “eggs” it can communicate emotion and arousal. falling from the top of the wall towards the floor. In the Beowulf game, the ambient sounds were the sound of air softly flowing through the game- speech, Music and sound Effects world’s system of caves and tunnels. The sounds for Information and Feedback had a slight amount of reverb added to create a sense of volume in the caves and the reverb was Many sounds are emotional and meant to create and removed for tunnels. The ambient sounds were communicate mood and presence. Other sounds also deliberately freed as much as possible from are meant to convey cognitive information about musical components such as pitch and rhythm: rules, scores, results and so forth. Speech is, of We wanted to give the players as much freedom course, very versatile and useful in this case. It as possible to use their own imagination, not is very effective to have a voice read the initial influencing them in any direction defined by us instructions for a game, especially if it is a game more than necessary. with relatively simple gameplay and few rules. Most of the DigiWall games use music tracks The same is true for scores and results. Who won, as ambient and background sounds. In this case the left or right team? How many points did you the purpose is the opposite. Music is used to set score? To have a voice read these results creates the basic mood of the games and to encourage a strong feeling of presence and makes the game physical activity in the players. The music is de- come alive. One drawback with speech is, of signed to communicate subconsciously with the course, language. For example, Swedish voice-

38 Sound for Fantasy and Freedom

overs in a game do not make very much sense sounds that the players can immediately relate to in the UK. As with text, it is necessary to have can greatly enhance the gameplay aspect of the localized versions and this quickly starts add- same game as well as create the sought-after sense ing cost in terms of computer memory, coding, of presence and immersion. The DigiWall’s game development time, and other resources. But then Scrambled Eggs uses the sound of broken eggs to again, sometimes it is worth it. In the DigiWall signal points lost and the sound of an egg rescued games serving as an example here, speech is in the palm of your hand to signal points gained. used as introduction to all games. A majority of In Beowulf, if the player enters a forbidden game the games also present scores and results using tile, the sound of a scream receding down a hole speech. The DigiWall game interface is equipped together with the sound of falling rocks signals with two buttons, so the players can select one of life lost. When this is followed by the funeral two available languages. march, failure and the end of the game are obvious A danger with speech is the risk of wearing to anyone, without the need for speech or text. out often-repeated phrases. It is therefore useful to give the players the option to skip, for example, instructions when they are no longer needed. FUtUrE rEsEArcH DIrEctIONs Music and sound effects can also serve as carriers of information, albeit not as clear and It is often said that sound is still underused and unambiguous as speech. This is not an innate that audio is a media type with potential yet to be disability though, but rather an effect of the way unleashed. In order to free this unused potential, we use music and sound effects. Rhythm, for research and development efforts must be carried example, can be used to convey semantics just as out on several parallel fronts. We need to develop well as any speech: what is required is simply to more in-depth knowledge about auditory percep- learn the system (Morse code, for example). One tion and how heard experiences affect users of of the advantages with sound effects and music is computer games and other interactive systems. that they are not limited by language, but are more This also implies richer taxonomies and more universal. This can of course be used in many ways. developed languages for writing about, talking In the Beowulf game, each new round starts with about and reflecting over this new knowledge and a short, horn melody, as if it were announcing the making it useful in wider contexts. Furthermore, approach of the king’s ambassador. The players a number of current ideas and traditions in the learn very quickly what this signal means and, field must be challenged and a set of updated since it is very short, the risk of becoming bored ideas must be developed. Ambiguity and wider with it is minimal. Beowulf also uses pure music interpretation spaces treated as design assets to signal success and failure. Success is signaled rather than problems in the design of interactive by a short triumphant brass fanfare and failure is systems is one example. Another example is when signaled by a short funeral march. simple efficiency metrics for player enjoyment By carefully selecting the metaphor aspect are replaced with more complex systems for the of a game’s design, tremendous opportunities to design and evaluation of computer game experi- create sound effects for feedback and information ences, such as the GameFlow concept. Finally, can be opened up. By placing the game in an en- new technology that can carry and realize the new vironment (metaphor) that the players are likely knowledge and ideas must be developed. This to have some kind of relation to, the designer can includes technologies for procedural audio (see choose sounds for feedback and information that Farnell, 2011; Mullan, 2011 for further descrip- are natural in that environment. Using natural tions of this technology)) and systems for dynamic

39 Sound for Fantasy and Freedom

simulation of room acoustics and acoustic occlu- How we interpret a sound depends on, and sion and obstruction, just to name a few. draws from, our previous personal experiences. Well-known sounds will spawn a myriad of pictures in our inner, mental movie theaters. Unknown cONcLUsION sounds can create both confusion and excitement. Working in parallel with the gameplay and the Sound is a complex stimulus and it is only in metaphor aspects of computer game design, and recent years that science has started to under- making sure that the two match and support each stand auditory perception in any depth. Much other, is a powerful way to find and design the of the knowledge and practice in sound design sounds that build the total soundscape of the game. for computer games and other interactive ap- By working in parallel with and carefully balanc- plications is based on experience and anecdotal ing the graphics and the sounds of a computer evidence. But the awareness of sound’s potential game the users’ bodies and fantasies can be set and scientifically-based knowledge in sound de- free, creating unique, immersive, and rewarding sign is slowly increasing. This is not only true in gaming experiences. the computer game industry, but in industry and society in general. The implications of the fact that our ears and our eyes complement each other rEFErENcEs are slowly beginning to have an effect. Graphics alone gives one type of experience: sound alone Association for Computing Machinery. (2010). gives another type of experience, and graphics ACM computing classification system. New York: plus sounds gives new and unique experiences. ACM. Retrieved February 4, 2010, from http:// By working with the balance of ears, eyes and www.acm.org/about/class/. other senses and human abilities, new opportuni- Avanzini, F. (2008). Interactive sound . In Polotti, ties emerge for the computer game designer. The P., & Rocchesso, D. (Eds.), Sound to sense, sense Wii, Dance Dance Revolution and DigiWall are to sound – A state of the art in sound and music just a couple of examples of this. computing (pp. 345–396). Berlin: Logos Verlag. Sounds in the physical reality of our bodies are the results of physical events in that same Bregman, A. S. (1990). Auditory scene analysis: reality. Our hearing is designed and “hardwired” The perceptual organization of sound. London: to constantly scan and analyze the soundscape MIT Press. surrounding us and react rationally to the sounds Cowley, B., Charles, D., Black, M., & Hickey, R. heard. Most of the time this is done subconsciously (2008). Toward an understanding of flow in video and our hearing can therefore be described as, to a games. ACM Computers in Entertainment, 6(2). large degree, intuitive, emotional, or pre-cognitive. The soundscape reaching our ears demands inter- Csíkszentmihályi, M. (1990). Flow: The psychol- pretation and disambiguation in other ways than ogy of optimal experience. New York: Harper the visual stimuli reaching our eyes. This need Collins. to interpret and disambiguate can be turned into Dance dance revolution [Computer game]. (2010). a great asset in computer game design. A game Tokyo: Konami. with a well-designed, rich, and varied soundscape will play on the user’s intuition and emotions: the DigiWall [Computer game]. (2010). Piteå, Swe- game will be immersive and give fun and reward- den: Digiwall Technology. Retrieved February ing gaming experiences. 10, 2010, from http://www.digiwall.se/.

40 Sound for Fantasy and Freedom

Ekman, I., Ermi, L., Lahti, J., Nummela, J., Jegers, K. (2009). Elaborating eight elements of Lankoski, P., & Mäyrä, F. (2005). Designing sound fun: Supporting design of pervasive player enjoy- for a pervasive mobile game. In Proceedings of ment. ACM Computers in Entertainment, 7(2). the ACM SIGCHI International Conference on Jørgensen, K. (2011). Time for new terminology? Advances in Computer Entertainment Technol- Diegetic and non-diegetic sounds in computer ogy,2005. games revisited . In Grimshaw, M. (Ed.), Game Farnell, A. (2011). Behaviour, structure and causal- sound technology and player interaction: Con- ity in procedural audio . In Grimshaw, M. (Ed.), cepts and developments. Hershey, PA: IGI Global. Game sound technology and player interaction: Juslin, P. N., & Västfjäll, D. (2008). Emotional Concepts and developments. Hershey, PA: IGI responses to music: The need to consider under- Global. lying mechanisms. The Behavioral and Brain Friberg, J., & Gärdenfors, D. (2004). Audio games: Sciences, 31, 559–621. New perspectives on game audio. In Proceedings Larsson, P., Västfjäll, D., & Kleiner, M. (2002). of the ACM SIGCHI International Conference on Better presence and performance in virtual envi- Advances in Computer Entertainment Technol- ronments by improved binaural sound rendering. ogy2004, 148-154. In AES 22nd International Conference on Virtual, Gabrielsson, A., & Lindström, E. (2001). The Synthetic and Entertainment Audio. influence of musical structure on emotional ex- Lecanuet, J. P. (1996). Prenatal auditory experi- pression . In Juslin, P., & Sloboda, J. A. (Eds.), ence . In Deliège, I., & Sloboda, J. (Eds.), Musical Music and emotion: Theory and research. Oxford, beginnings: Origins and development of musical UK: Oxford University Press. competence (pp. 3–36). Oxford, UK: Oxford Gaver, W. (1993). What in the world do we University Press. hear? An ecological approach to auditory event Liljedahl, M., Lindberg, S., & Berg, J. (2005). perception. Ecological Psychology, 5(1), 1–29. Digiwall: An interactive climbing wall. Proceed- doi:10.1207/s15326969eco0501_1 ings of theACM SIGCHI International Conference Gaver, W. (1997). Auditory interfaces . In Hel- on Advances in Computer Entertainment Technol- ander, M. G., Landauer, T. K., & Prabhu, P. (Eds.), ogy, 2005, 225-228. Handbook of human-computer interaction (2nd Liljedahl, M., Papworth, N., & Lindberg, S. (2007). ed.). Amsterdam: Elsevier Science. doi:10.1016/ Beowulf: An audio mostly game. Proceedings of B978-044481862-1/50108-4 the International Conference on Advances in Com- Gaver, W. W., Beaver, J., & Benford, S. (2003). puter Entertainment Technology, 2007, 200–203. Ambiguity as a resource for design. Proceedings Mullan, E. (2011). Physical modelling for sound of the ACM CHI Conference on Human Factors synthesis . In Grimshaw, M. (Ed.), Game sound in Computing Systems, 2003, 233-240. technology and player interaction: Concepts and Hug, D. (2011). New wine in new skins: Sketching developments. Hershey, PA: IGI Global. the future of game sound design . In Grimshaw, M. (Ed.), Game sound technology and player interaction: Concepts and developments. Hershey, PA: IGI Global.

41 Sound for Fantasy and Freedom

Murphy, D., & Neff, F. (2011). Spatial sound for Tuuri, K., Mustonen, M. S., & Pirhonen, A. (2007). computer games and virtual reality . In Grimshaw, Same sound – different meanings: A novel scheme M. (Ed.), Game sound technology and player infor modes of listening. In Proceedings of Audio teraction: Concepts and developments. Hershey, Mostly 2007 – 2nd Conference on Interaction PA: IGI Global. with Sound, 13-18. O’Callaghan, C. (2009 Summer). Auditory (2010). World of warcraft [Computer game]. Reno, perception. In E. N. Zalta (Ed.), The Stanford NV: Blizzard Entertainment. encyclopedia of philosophy, Retrieved January 24, 2010, from http://plato.stanford.edu/archives/ sum2009/entries/perception-auditory/. ADDItIONAL rEADING Parker, J. R., & Heerema, J. (2008). Audio Interac- tion in Computer Mediated Games. International Altman, R. (Ed.). (1992). Sound theory sound Journal of Computer Games Technology, 2008, practice. New York: Routledge. 1–8. .doi:10.1155/2008/178923 Boehner, K., DePaula, R., Dourish, P., & Sengers, Polaine, A. (2005). The flow principle in interac- P. (2005). Affect: From information to interaction. tivity. In Proceedings of the Second Australasian In Proceedings of the 4th Decennial Conference Conference on interactive Entertainment. on Critical Computing: Between Sense and Sen- sibility. Reid, J., Geelhoed, E., Hull, R., Cater, K., & Clayton, B. (2005). Parallel worlds: Immersion Brown, E., & Cairns, P. (2004). A grounded in location-based experiences. In CHI ‘05 Ex- investigation of game immersion. In CHI ‘04 tended Abstracts on Human Factors in Comput- Extended Abstracts on Human Factors in Com- ing Systems. puting Systems. Sengers, P., Boehner, K., Mateas, M., & Gay, G. Juslin, P., & Sloboda, J. A. (Eds.). (2001). Music (2008). The disenchantment of affect. Personal and emotion: Theory and research. Oxford, UK: and Ubiquitous Computing, 12(5), 347–358. Oxford University Press. doi:10.1007/s00779-007-0161-4 Kaptelinin, V., & Nardi, B. A. (2009). Acting with Sengers, P., & Gaver, B. (2006). Staying open technology: Activity theory and interaction design. to interpretation: Engaging multiple meanings Cambridge, MA: MIT Press. in design and evaluation. Proceedings of the 6th Norman, D. A. (1988). The design of everyday Conference on Designing Interactive Systems, things. New York: Basic Books. 2006, 99-108. Polotti, P., & Rocchesso, D. (Eds.). (2008). Sound Sonnenschein, D. (2001). Sound design: The to sense, sense to sound: A state of the art in sound expressive power of music, voice and sound ef- and music computing. Berlin: Logos Verlag. fects in cinema. Studio City, CA: Michael Wiese Productions. Schafer, R. M. (1977). The soundscape: Our sonic environment and the tuning of the world. Sweetser, P., & Wyeth, P. (2005). GameFlow: A Rochester, VT: Destiny Books. model for evaluating player enjoyment in games. ACM Computers in Entertainment, 3(3). Sider, L., Freeman, D., & Sider, J. (Eds.). (2003). Soundscape: The School of Sound lectures 1998 – 2001. London: Wallflower Press.

42 Sound for Fantasy and Freedom

KEY tErMs AND DEFINItIONs Gameplay: The rules and mechanics defining the functionality of a computer game. Auditory Perception: The process of attaining Game Metaphor: The embodiment of the awareness or understanding of auditory informa- virtual environment comprising the game world. tion or stimulus. Immersion: Deep mental involvement. Avatar: A controllable representation of a Pervasive Game: A computer game tightly person or creature in a virtual reality environment. interwoven with our everyday lives through the Feedback: Output from a computer game to objects, devices and people that surround us and inform the user of various changes in game state. the places we inhabit. Flow: The mental state of operation in which a Suspension of Disbelief: A silent agreement person is fully immersed in what he or she is doing between an audience and an entertainment pro- by a feeling of energized focus, full involvement, ducer in which the audience agrees to provision- and success in the process of the activity. ally suspend their judgment in exchange for the promise of entertainment.