Open Research Online The Open University’s repository of research publications and other research outputs Automatic Generation of Dynamic Musical Transitions in Computer Games Thesis How to cite: Cutajar, Simon (2020). Automatic Generation of Dynamic Musical Transitions in Computer Games. PhD thesis The Open University. For guidance on citations see FAQs. c 2018 Simon Cutajar https://creativecommons.org/licenses/by-nc-nd/4.0/ Version: Version of Record Link(s) to article on publisher’s website: http://dx.doi.org/doi:10.21954/ou.ro.00010e48 Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyright owners. For more information on Open Research Online’s data policy on reuse of materials please consult the policies page. oro.open.ac.uk Typeset in LATEX using TEX Gyre Pagella and PT Serif. Declaration of Authorship I, Simon Cutajar, confirm that the research in this dissertation is my own work or that where it has been carried out in collaboration with, or supported by others, that this is duly acknowledged below and my contribution indicated. I attest that I have exercised reasonable care to ensure that the work is original, and to the best of my knowledge does not break any UK law, infringe any third party’s copyright or other Intellectual Property Right, or contain any confidential material. I confirm that this dissertation has not been previously submitted for the award of a degree by this or any other university. I understand that my dissertation may be made available to the public, electronically or otherwise. Details of collaboration The musical corpus used in my second study (detailed in Chapter 6, and transcribed in the appendix in Section E.3) was commissioned. The entire corpus was composed by Jan Flessel, who was compensated accordingly for his work. To my mother Isabelle Acknowledgements First of all, I’d like to thank my supervisors Dr. Robin Laney and Dr. Alistair Willis. Their help and direction has been invaluable to the shaping of my PhD journey. Similarly, I’d like to thank Prof. Marian Petre for running the PG Forum, a forum aimed at postgraduate students in the computing department that served to not only bring together students from different disciplines, but also for providing guidance on how exactly one should go about their PhD. Other facilities at the Open University, such as the natural language processing research group, also helped to better focus my research. I’d also like to thank my third party monitor Dr. Soraya Kouadri for her advice, as well as Dr. Allan Jones, Dr. Robert Samuels, and Dr. Alex Kolassa for interesting comments and discussions about my research. Finally, I would also like to thank my viva examiners Dr. Colin Johnson and Dr. Ben Winters. My friends, both old and new, have all helped to support me during my PhD by providing feedback, suggestions, and being there to listen to my venting when things went wrong. I’d like to thank Aaron, Allan, Amel, Anders, Andrea F., Andrea M., Angelo, Antoniette, Barbara, Bernard, Christina, Eiman, Emilie, Enrico, Germán, Giorgio, Giulia, Maria, Kris, Kristín, Jaisha, Julian G., Julian Z., Matteo, Matthew, Paco, Patrizia, Pinelopi, Sarah, Suraj, Theodoros, Tina, Tracie, William, Yannick, and many more for all the encouragement and support. I’d also like to thank all my study participants for their interest and for taking the time to get involved in my research. I’d also like to thank my Dungeons & Dragons group for all the adventures we had together, and for being such wonderful players in the world I had built. I would also like to thank the house band at the KMi department at the Open University: the Knowledge Media Instruments, for taking me on as a singer and percussionist. Both groups served to make sure that I had hobbies that took my mind off the PhD. My amazing girlfriend, Stef, has been a stable element in an often wildly unpredictable PhD. I am ever grateful for her constant love, patience, and support. Last but not least, I’m grateful to my family for encouraging me to pursue this PhD, for providing useful advice and feedback, and for being supportive in every way. I’d like to thank my mother Isabelle, my brother Stephen, and my first cousins once removed Godwin and Adrian. This page intentionally left blank. xi Abstract In video games, music must often change quickly from one piece to another due to player interaction, such as when moving between different areas. This quick change in music can often sound jarring if the two pieces are very different from each other. Several transition techniques have been used in industry such as the abrupt cut transition, crossfading, horizontal resequencing and vertical reorchestration among others. However, while several claims are made about their effectiveness (or lack thereof), none of these have been experimentally tested. To investigate how effective each transition technique is, this dissertation empirically evaluates each technique in a study informed by music psychology. This is done based xii ABSTRACT on several features identified as being important for successful transitions. The obtained results led to a novel approach to musical transitions in video games by investigating the use of a multiple viewpoint system, with viewpoints being modelled using Markov models. This algorithm allowed the seamless generation of music that could serve as a transition between two composed pieces of music. While transitions in games normally occur over a zone boundary, the algorithm presented in this dissertation takes place over a transition region, giving the generated music enough time to transition. This novel approach was evaluated in a bespoke video game environment, where participants navigated through several pairs of different game environments and rated the resulting musical transitions. The results indicate that the generated transitions perform as well as crossfading, a technique commonly used in the industry. Since crossfading is not always appropriate, being able to use generated transitions gives composers another tool in their toolbox. Furthermore, the principled approach taken opens up avenues for further research. CONTENTS xiii Contents List of Tables xix List of Figures xxiii List of Musical Examples xxix List of Acronyms xxxii 1 Introduction 1 1.1 Problem Area . .2 1.2 Research Question and Objectives . .3 xiv CONTENTS 1.3 Clarification of Scope . .4 1.4 Video Games as a Primary Source . .4 1.5 Main Contributions . .6 1.6 Dissertation Overview . .8 2 Literature Review 11 2.1 Music in Video Games . 12 2.1.1 Definitions . 13 2.1.2 Similarities Between Film Music and Game Music . 18 2.1.3 Music’s Role in Video Games . 20 2.1.4 History of Music in Games . 27 2.2 Algorithmic Composition . 29 2.2.1 Algorithmic Composition . 29 2.2.2 Algorithmic Techniques . 33 2.2.3 Procedural Content Generation . 38 2.2.4 Algorithmic Music in Games . 42 2.3 Transitions . 46 2.3.1 Definition . 46 2.3.2 Morphing . 59 2.3.3 Transitions in Games . 62 2.3.4 The Ideal Transition . 73 3 Investigating the Detection of Musical Transitions 79 3.1 Aims and Objectives . 80 3.2 Methodology . 81 3.2.1 Iterative Approach . 82 3.2.2 Third Iteration . 86 3.3 Dataset . 89 CONTENTS xv 3.3.1 Example Pieces . 89 3.3.2 Study Pieces . 91 3.3.3 Musical Representation . 93 3.3.4 Dataset Configuration for the Study . 95 3.4 System Design . 102 3.5 Conclusions . 103 4 Evaluating the Detection of Musical Transitions 105 4.1 Participants . 106 4.2 Non-Parametric Tests . 108 4.2.1 Using Discrete Visual Analogue Scales . 108 4.2.2 Issues and biases with self-reporting and scales . 109 4.2.3 Techniques . 110 4.3 Observations . 115 4.3.1 Transition Detections . 115 4.3.2 Differences Between Transition Techniques . 118 4.3.3 Comparison of Source and Target Pieces . 123 4.4 Conclusions . 125 5 Towards a New Transition Algorithm 127 5.1 Rethinking Transitions . 128 5.2 Building A New Transition Algorithm . 130 5.2.1 Markov models . 131 5.2.2 Viewpoints . 133 5.2.3 Viewpoint Classes . 135 5.2.4 Contextual Use of Viewpoints . 147 5.3 Generating Transitions . 155 5.3.1 Weighted Averaging . 155 xvi CONTENTS 5.3.2 Model Selection . 157 5.3.3 Weighted Viewpoint Selection . 161 5.4 Conclusions . 165 6 Generative Transitions in a Game Environment 167 6.1 Aims and Objectives . 168 6.2 System Design . 169 6.2.1 Game Environment Implementation . 169 6.2.2 Integration of the Transition Algorithm . 179 6.2.3 Study Session Generation . 185 6.3 Dataset . 185 6.4 Creating a Lookup Table . 187 6.5 Methodology . 191 6.5.1 Evaluating Transitions . 193 6.5.2 Immersion . 194 6.6 Conclusions . 195 7 Evaluating Generative Transitions in Games 197 7.1 Participants . 198 7.2 Evaluation of Transitions . 200 7.2.1 Success . 201 7.2.2 Rate of Change . 203 7.2.3 Perception . 206 7.2.4 Degree of Fitting . 208 7.3 Immersion . 210 7.4 Conclusions . 212 CONTENTS xvii 8 Conclusions 213 8.1 Research Insights . 213 8.1.1 Investigating the Detection of Musical Transitions . 214 8.1.2 Generative Transitions in a Game Environment . 216 8.2 Future Work . 218 8.2.1 Musical Transitions with Harmony . 218 8.2.2 Improvements to the Multiple Viewpoint System . 219 8.2.3 Integration into a Game Engine . 220 8.2.4 Evaluating Music Transitions . 221 8.3 Closing Remarks . 224 9.