CAPTIVATING SOUND THE ROLE OF AUDIO FOR IMMERSION IN COMPUTER GAMES by Sander Huiberts Thesis submitted in fulfilment of the requirements for the degree of PhD at the Utrecht School of the Arts (HKU) Utrecht, The Netherlands and the University of Portsmouth Portsmouth, United Kingdom November 2010 Captivating Sound The role of audio for immersion in computer games © 2002‐2010 S.C. Huiberts Supervisor: Jan IJzermans Director of Studies: Tony Kalus Examiners: Dick Rijken, Dan Pinchbeck 2 Whilst registered as a candidate for the above degree, I have not been registered for any other research award. The results and conclusions embodied in this thesis are the work of the named candidate and have not been submitted for any other academic award. 3 Contents Abstract__________________________________________________________________________________________ 6 Preface___________________________________________________________________________________________ 7 1. Introduction __________________________________________________________________________________ 8 1.1 Motivation and background_____________________________________________________________ 8 1.2 Definition of research area and methodology _______________________________________ 11 Approach_________________________________________________________________________________ 11 Survey methods _________________________________________________________________________ 12 2. Game audio: the IEZA model ______________________________________________________________ 14 2.1 Understanding the structure of game audio _________________________________________ 14 2.2 Game audio typologies ________________________________________________________________ 15 2.2 The IEZA model for audio in games __________________________________________________ 20 2.2.1 The diegetic dimension of game audio __________________________________________ 22 2.2.2 The second dimension: interdependence of game audio ______________________ 24 2.2.3 The domains of the IEZA model__________________________________________________ 24 2.3 Two functionalities of game audio and IEZA ________________________________________ 29 2.4 Design properties and IEZA ___________________________________________________________ 32 2.5 Summary _______________________________________________________________________________ 33 3. Immersion __________________________________________________________________________________ 36 3.1 Introduction ____________________________________________________________________________ 36 3.2 The definition of immersion __________________________________________________________ 37 3.3 Classifications of immersion __________________________________________________________ 41 3.4 Description of the three dimensions of immersion _________________________________ 45 3.5 The process of immersion in time ____________________________________________________ 47 3.6 Conclusion______________________________________________________________________________ 49 4. Audio design and immersion______________________________________________________________ 50 4.1 The positive or negative influence of audio on immersion _________________________ 50 4.2 Audio enhancing immersion __________________________________________________________ 54 4.2.1 Audio enhancing sensory immersion____________________________________________ 54 4.2.2 Audio enhancing challenge‐based immersion __________________________________ 68 4.2.3 Audio enhancing imaginative immersion _______________________________________ 82 4.2.4 Summary and Conclusion _______________________________________________________ 101 4.3 Audio disrupting immersion _________________________________________________________ 106 4 4.3.1 The negative effect of audio on immersion ____________________________________ 106 4.3.1 Audio disrupting sensory immersion___________________________________________ 109 4.3.2 Audio disrupting challenge‐based immersion _________________________________ 110 4.3.3 Audio disrupting imaginative immersion ______________________________________ 111 4.3.4 Absence of audio disrupting immersion _______________________________________ 112 4.3.5 Summary and Conclusion _______________________________________________________ 112 Conclusion____________________________________________________________________________________ 117 Summary___________________________________________________________________________________ 117 Discussion _________________________________________________________________________________ 119 List of Figures ________________________________________________________________________________ 122 List of Tables _________________________________________________________________________________ 124 Bibliography _________________________________________________________________________________ 125 References ______________________________________________________________________________ 125 Game References _______________________________________________________________________ 131 Other References and Sources of Inspiration ________________________________________ 132 APPENDIX A__________________________________________________________________________________ 134 5 ABSTRACT This thesis explores the relation between game audio and (computer) game immersion. The main contribution in this research is a framework for the conceptual design of game audio in relation to immersion. A model for game audio is presented defining the communication by means of game audio during active gameplay. This model, named IEZA (Huiberts & van Tol, 2008), defines four conceptual domains of communication of game audio. IEZA was developed to enable a more detailed and consistent investigation into game audio and has been evaluated as a design resource in educational, academic and practical settings. Immersion is considered to be one of the key factors making games worthy to play. The unity that seems to exist in the fact that many parties value this phenomenon is more apparent than real: there is a lot of dispute on the scope of immersion. Several classifications of immersion are compared for a better understanding of the nature of an immersive experience. The three‐dimensional SCI‐model by Ermi & Mäyrä (2005) is considered as valuable representation of the multi‐dimensional character of immersion. Audio is an important constituent of most games and its role for immersion in games has hardly been investigated. In this thesis, audio is studied using the IEZA‐model and the SCI‐model, and several design issues are described. This yields a conceptual framework that describes various audio design issues that can be used to reflect upon conceptual decisions relevant for the design of audio in relation to immersion. This framework can be used for a conceptual analysis of design and is relevant for audio designers as well as game designers. PREFACE It all started with a quick email reply in the year 2000 to a tempting request to participate as an audio designer in a project that focused on developing games for visually disabled children. Not only did this introduce me to the field of development of applications and games for the blind, it also made me aware of my primary fascination: the experiential value of sound and music. In an interactive process, sound and music can provide a captivating experience: we forget ourselves, stop noticing time and the experience is all that counts: the experience becomes alive. In my personal opinion, computer games excel in this process and were therefore chosen as the second field of study of this thesis. A pervasive experience in game play, whether caused by audio or by another medium, is usually described by the term immersion. Audio plays an important role for immersion, and in this thesis, useful concepts and points of view in order to better understand this concept will be examined. Before continuing, two general matters need to be explained. In the text, a player is always referred to as 'he', although 'she' could apply as well. This is only done to maintain consistency and because the vast majority of respondents were male. A note on the term game audio: this is the current and most frequently found term to define sound and music in games. Whenever a paragraph refers to music or sound only, these words will be used. I would like to express my sincere gratitude to all who supported the writing of this thesis: Jan IJzermans, Tony Kalus, Adriaan Braat, Chris Creed, Han Onno ter Hoor, Micah Hrehovcsik, Rens Machielse, Paul Newland, Dan Pinchbeck, Dick Rijken, Hans Timmermans, Gerard van Wolferen, the other members of the Music Design Research Group, the gamers and moderators from the forums (Tweakers Forum, the 3Dgamers Forum, the Insidegamer Forum, the Gamers‐Forum, AudioGames Forum), the participants of PUGS, Espressofabriek and my parents Hans and Riet Huiberts. I would like to thank Richard van Tol for the very pleasant and fruitful cooperation and his important role in our game audio research activities. And last but not least, I am very grateful to Barbara Lotti for giving me her endless support, love and motivation. Sander Huiberts 7 1. INTRODUCTION 1.1 MOTIVATION AND BACKGROUND The field of computer games has developed enormously over the forty years of its existence.1 In the very beginning, game designers were only able to draw simple pixels on a screen and synthesise very basic sounds, but currently, most games feature highly detailed game worlds.2 While playing games, players often become immersed, which is an important aspect for the game