VORPAL: a Middleware for Real-Time Soundtracks in Digital Games
Total Page:16
File Type:pdf, Size:1020Kb
VORPAL: A Middleware for Real-Time Soundtracks in Digital Games Wilson Kazuo Mizutani Thesis Submitted to the Institute of Mathematics and Statistics of the University of São Paulo for the Masters Degree in Computer Science Program: Computer Science Advisor: Prof. Fabio Kon This research was supported by CNPq, Brazil São Paulo, January, 2017 VORPAL: A Middleware for Real-Time Soundtracks in Digital Games This version of the thesis contains the corrections and modifications suggested by the Examining Committee during the defense of the original work, performed on January 24th, 2017. A copy of the original version is available at the Instituto de Matemática e Estatística from the University of São Paulo. Examining Committee: • Prof. Fabio Kon - IME-USP • Prof. Ricardo Nakamura - POLI-USP • Prof. Jônatas Manzolli - UNICAMP Acknowledgements I would like to thank all sound designers and musicians who helped me better understand their work and how my research could help them. Among them, my greatest thanks goes to Dino Vicente, who showed immense interest in my work and tagged along for a whole year, always believing in the potential contributions it could bring to soundtracks in games and in general. If not for him, the game we developed to validate the developed technology would be that much less rich and actually representative of what a real sound designer needs. I am also grateful for everyone at both the Systems and Computer Music research groups at the Institute of Mathematics and Statistics of the University of São Paulo who helped me in the many obstacles towards my Masters title. Special thanks go to Antonio Goulart, who showed me the works of Farnell; to Pedro Bruel, who introduced me to libpd; and to Lucas Dario, who chose to participate in my research for his final course monograph. Also, many thanks to my advisor, Fabio Kon, and to professors Marcelo Queiroz and Alfredo Goldman, for their support and insight. i ii Abstract Although soundtracks play an essential role in the experience delivered by digital games, there are a number of design restrictions it suffers from due to technology limitations. This is spe- cially true for real-time effects, a natural demand in the interactive media of games. Developers may either implement their own solutions each time, rely on proprietary software, or neglect the soundtrack altogether. Besides, even the best proprietary tools support only sample-based audio, which is one of the main causes for the aforementioned design restrictions. Thus, this thesis proposes VORPAL, a free software game audio middleware implementation that focuses on procedural audio instead – while maintaining the possibility of sample-based audio – as a more accessible and adequate tool for composing real-time soundtracks for digital games. The middleware, inspired by its proprietary predecessors, is divided in two main pieces of software: an audio engine and a soundtrack creation kit. The audio engine comprises a native C++ program- ming library, which games and game engines can be linked to to play and control in real-time soundtrack pieces created using the soundtrack creation kit, which consists of building blocks provided as Pure Data abstractions. We have interviewed and partnered with professional sound designers to validate our technology, and came to develop a proof of concept game called Sound Wanderer, which showcases the possibilities and limitations of the VORPAL middleware. Keywords: digital games, dynamic audio, adaptive music, real-time soundtrack, audio middle- ware, game soundtrack. iii iv Resumo Muito embora trilhas sonoras desempenhem um papel essencial na experiência criada por jo- gos digitais, elas sofrem de diversas restrições de projeto causadas por limitações tecnológicas. Isso afeta principalmente efeitos em tempo real, que são uma demanda natural na mídia inter- ativa dos jogos. Desenvolvedores precisam optar entre implementar uma solução própria caso a caso, investir em software proprietário, ou simplesmente negligenciar a trilha sonora por falta de opção melhor. Além disso, mesmo as melhores ferramentas comerciais trabalham apenas com áudio baseado em amostras, o que é uma das principais causas das ditas restrições de projeto. Portanto, esta dissertação propõe VORPAL, um middleware livre para áudio em jogos que foca em áudio procedural – mas mantém compatibilidade com áudio baseado em amostras – como uma ferramenta mais acessível e adequada para a composição de trilhas sonoras em tempo real para jogos digitais. O middleware, inspirado em seus antecessores comerciais, é dividido em duas principais componentes de software: um motor de áudio e um kit de criação de trilhas sonoras. O primeiro é constituído por uma biblioteca de programação nativa em C++, com a qual jogos e mo- tores de jogos podem se ligar para reproduzir e controlar, em tempo real, peças da trilha sonora criadas usando a outro componente, que é um kit de blocos de construção providos como ab- strações de Pure Data. Projetistas de som profissionais foram entrevistados e depois trabalharam em parceria com os autores para validar a tecnologia proposta, o que levou ao desenvolvimento de um jogo de prova conceitual chamado Sound Wanderer, que demonstra as possibilidades e limitações do middleware VORPAL. Palavras-chave: jogos digitais, áudio dinâmico, música adaptativa, trilhas sonoras em tempo real, middleware de áudio, trilhas sonoras para jogos, games. v vi Contents 1 Introduction 1 1.1 Motivation........................................2 1.1.1 Related problems................................5 1.1.2 Available Solutions...............................7 1.1.3 Challenges....................................8 1.2 Objective........................................ 10 1.2.1 Intermediate Goals............................... 11 1.2.2 Contributions.................................. 12 1.2.3 Validation.................................... 13 1.3 Text Organization................................... 13 2 Related Work 15 2.1 Game Audio in General................................ 15 2.2 Music Automation................................... 18 2.3 Physically Based Real-Time Synthesis........................ 21 2.4 Other Works...................................... 22 3 Concepts and tools 23 3.1 Digital Audio...................................... 23 3.1.1 Digital Signal Processing (DSP)........................ 25 3.1.2 Symbolic Representation............................ 27 3.2 Soundtracks....................................... 28 3.2.1 Traditional Production Process........................ 30 3.2.2 Real-Time Soundtracks............................ 31 3.3 Digital Games...................................... 32 3.3.1 Development Process.............................. 33 3.3.2 Software Architecture............................. 35 3.3.3 Tool and Technologies............................. 38 3.3.4 Algorithms and Data Structures for Real-Time Audio in Games...... 42 3.4 Related Technologies.................................. 47 3.4.1 iMuse...................................... 47 3.4.2 Wwise...................................... 50 3.4.3 FMOD Studio................................. 55 3.4.4 Elias....................................... 60 3.4.5 Pure Data .................................... 62 3.4.6 Comparison................................... 66 vii viii CONTENTS 4 Proposed solution 67 4.1 Methodology...................................... 67 4.2 System Requirements.................................. 68 4.2.1 Rodolfo Santana................................ 69 4.2.2 Kaue Lemos................................... 70 4.2.3 Dino Vicente De Lucca............................. 71 4.2.4 Final List of System Requirements...................... 72 4.3 Digital Representation of Real-Time Soundtracks.................. 74 4.3.1 Considered Formats and Comparison..................... 75 4.3.2 Chosen Format................................. 76 4.4 Architecture....................................... 76 4.4.1 Audio Engine.................................. 77 4.4.2 Soundtrack Creation Kit............................ 78 4.4.3 Components Integration............................ 79 5 Implementation 81 5.1 Prototype........................................ 82 5.2 Audio Engine...................................... 84 5.2.1 High-Level API................................. 84 5.2.2 Pure Data Patch Management........................ 88 5.2.3 Sound Playback................................. 91 5.2.4 Real-Time Soundtrack Processing....................... 93 5.3 Soundtrack Creation Kit................................ 94 5.3.1 Output Bus................................... 95 5.3.2 Commands................................... 96 5.3.3 Music Sequencing................................ 97 5.3.4 Samples..................................... 97 5.3.5 Sound Synthesis................................ 98 5.4 Middleware Usage................................... 100 5.4.1 Distribution................................... 101 5.4.2 Programmer’s Workflow............................ 101 5.4.3 Sound Designer’s Workflow.......................... 104 5.4.4 Game Engine Integration........................... 105 5.4.5 Examples.................................... 107 6 Results 113 6.1 Sound Wanderer.................................... 113 6.2 Advanced Features................................... 117 6.3 Usage Feedback..................................... 121 6.4 Middleware Limitations................................ 122 7 Conclusion 125 7.1 Final Considerations.................................. 125 7.2 Future Work.....................................