Physics-Based Real-Time Sound Synthesis for Virtual Reality Musical Instruments (VRMI) State of the Art, Design and Implementation of VRMIs Master Thesis Lui Albæk Thomsen Aalborg University Copenhagen Department of Architecture, Design & Media Technology A. C. Meyers Vænge 15 2450 Copenhagen SV Copyright c Lui Albæk Thomsen, Aalborg University 2017 Department of Architecture, Design & Media Technology A. C. Meyers Vænge 15 2450 Copenhagen SV Title: Abstract: Physics-Based Real-Time Sound Synthe- sis for Virtual Reality Musical Instru- With the recent wave of development ments in Virtual Reality (VR) technology, new purposes have been found for the Theme: medium. The complex motion track- Master Thesis ing and limitless environments allows for interesting new ways of interacting Project Period: with musical instruments. Recently, re- Spring Semester 2017 searchers have examined the concept of musical interaction in VR, which has Participant(s): led us to a new category in expres- Lui Albæk Thomsen sive musical interfaces; Virtual Real- ity Musical Instruments (VRMI). The Supervisor(s): thesis project presents interactive pro- Stefania Serafin totypes with integrated physics-based sound synthesis through an iterative de- sign process of usability and crossmodal Copies: 1 association evaluation. The main con- clusion of the usability evaluation in- Page Numbers: 60 cludes further improvements to the ro- bustness between gesture mapping and Date of Completion: collision detection system. From the August 15, 2017 crossmodal association experiments, a slight tendency towards successful iden- tification of size and material of the sound producing object was found, how- ever, given the small sample size and potential flaws in the experimental de- sign, future work and evaluation re- mains a requirement for raising the va- lidity of the indicative tendencies. The content of this report is freely available, but publication (with reference) may only be pursued due to agreement with the author. Contents Preface vii 1 Introduction1 1.0.1 Background.............................1 1.0.2 Summary of Research.......................4 2 Analysis5 2.1 Mixed Reality................................5 2.2 Music in Mixed Reality..........................7 2.2.1 Research...............................7 2.3 Sound Synthesis..............................9 2.3.1 FAUST (Functional Audio Stream)................9 2.3.2 FAUST-STK............................ 10 2.3.3 Physical Modeling Synthesis.................... 10 2.3.4 Digital Waveguide Synthesis.................... 11 2.3.5 Finite Element Analysis (FEA).................. 11 3 Methodology 13 3.0.1 Virtual Reality........................... 13 3.0.2 Crossmodal Associations...................... 14 4 Design 15 4.0.1 VRMI Interaction......................... 16 4.0.2 Non-Immersive Interaction.................... 17 4.0.3 Summary of Requirements..................... 17 5 Implementation 19 5.0.1 Plugin Workflow.......................... 19 5.0.2 Physical Model Parameters.................... 20 5.0.3 Virtual Reality........................... 21 5.0.4 Parametric Mapping using C# Marshalling........... 24 5.0.5 Revisited Version for Crossmodal Experiment.......... 25 v vi Contents 6 Evaluation 31 6.0.1 Usability Testing.......................... 31 6.0.2 Crossmodal Association...................... 32 7 Discussion 35 7.0.1 Interaction Prototype (VR).................... 35 7.0.2 Crossmodal Experiment...................... 37 7.0.3 Future Perspective......................... 37 8 Conclusion 39 Bibliography 41 A Acronyms 43 B Scripts 45 C Virtual Reality Technology 51 D Musical VR Applications 53 E Physical Model Parameters 59 Preface Thank you to Stefania Serafin for all the guidance, support and supervision of my thesis project and for most of my projects during my time as a student on the master’s degree of Sound & Music Computing at Aalborg University Copenhagen. Another thank you to Smilen Dimitrov, who has been an excellent help as technical supervisor for many projects during my studying. I would also like to take the oppotunity to reach out with a special thank you to the developers of FAUST for the guidance and support during the development of the thesis project. Therefore, a thank you to Romain Michon from CCRMA at Stanford University for introducing me to the programming language and for getting me in contact with some of the very friendly and helpful developers of FAUST; thank you to Yann Orlarey and Stéphane Letz from GRAME (Centre National de Création Musicale). Aalborg University, August 15, 2017 Lui Albæk Thomsen <[email protected]> vii viii Preface Chapter 1 Introduction With the resurgence and commercialization of Virtual Reality (VR) technology, a new wave of interactive and creatively expressive musical interfaces is surfacing the community and world of VR. The interfaces have been investigated and coined as Virtual Reality Musical Instruments (VRMIs) [1]. Following the definition, a set of conceptual design principles has been established [1] with respect to prior research in the field of New Interfaces for Musical Expression (NIME). By default, VR is dependent on the same development platforms used in traditional game design for 3D experiences (i.e., game engines). In terms of the audio, the wave of VR has pushed the abilities of game engines to spatialise audio in real-time using various newly developed libraries and plugins. However, it seems that the common approach (given e.g., Unity’s features in terms of playback of sounds) for sound design is to use third-party software like Digital Audio Workstations (DAW) for synthesizing, processing and recording the intended sound design of a game. In regards to musical experiences in VR, and the sophisticated and low-cost motion tracking that most common VR systems include, the idea of using motion data for interacting with musical tones generated by mathematics should be more accessible than ever. The following thesis project will pursue the task of investigating options for real-time embedment of sound synthesis in game engines intended for musical interaction in Virtual Environments (VE), while establish and discussing the technicalities in the development of interactive audio plugins for use in a VE. The thesis project will also document several implementations of real-time sound synthesis related to interaction with and realism of musical instruments. 1.0.1 Background The following sections will briefly describe the core areas of research in the thesis project. 1 2 Chapter 1. Introduction Virtual Reality The current state of the art VR technology allows for multi- ple tracking points (e.g., head and hands obtained from the Head-Mounted Display (HMD) and hand-held controllers) with positional and rotational information, which allows for complex gestures in controlling behaviour within games and VEs. The hand-held controllers enable the interfaces to be partly tangible with its physical form and interactive buttons, however, the actual virtual environment is intangi- ble and absent of true haptic feedback – unless information about user actions is conveyed through the actuators of the hand-held controller. The author of the thesis has previously co-authored a paper considering the use of VR for musical instruments [2]. Part of the paper concerned an implementation of virtual percussive instruments designed for a commercial VR system containing a HMD, hand-held controllers, and room-scale tracking (6DoF). The main conclusion from the evaluation of the design and implementation was that a lack of dynamic audio, related to the gestures being generated in real-time, decreased the playability of the VRMIs. The term of playability has been discussed in [3], however, throughout the following thesis project the term will be simplified to merely describe interaction as feasible for simple musical performances. A larger scale study on playability might be performed in the future covering all aspects of playability. Synthesis Engines Various development platforms and programming languages like SuperCollider 1, JUCE 2, Max/PD 3, CSound 4 and FAUST 5 allows for im- plementation and control of real-time audio synthesis models and more. A strong motivator for the thesis project has been the intrusiveness of providing high-quality sound synthesis in VR experiences. In previous personal experiences (as well as other research projects), the audio engine has been interoperable through an Open Sound Control (OSC) 6 network protocol connection with the aforementioned third-party software. However, such an approach requires additional software to run on the side, as well as an implemented library for setting up the server and listener of the protocol. As VR is usually highly dependent on visual content, interactive virtual environments are designed similarly to traditional video games played on a screen. Therefore, the embedment of the sound synthesis engine into the executable VR experience is interesting for the prevalence of musical instruments in VR that are convenient and easy to use. Apart from avoiding intrusive software setups, the thesis project will also be developed under the same principle in relation to the involved hardware. As described previously, the current state of the art in VR technology has certain restrictions on the types of inputs that can be provided to the system, and is mostly based on tracking of motions. 1SuperCollider: http://supercollider.github.io/ 2JUCE: https://www.juce.com/ 3Max: https://cycling74.com/products/max 4CSound: http://csound.github.io/