Aalborg Universitet Resurrecting the Tromba Marina: a Bowed Virtual
Total Page:16
File Type:pdf, Size:1020Kb
Aalborg Universitet Resurrecting the Tromba Marina: A Bowed Virtual Reality Instrument using Haptic Feedback and Accurate Physical Modelling Willemsen, Silvin; Paisa, Razvan; Serafin, Stefania Published in: Proceedings of the 17th Sound and Music Computing Conference Creative Commons License CC BY 3.0 Publication date: 2020 Document Version Publisher's PDF, also known as Version of record Link to publication from Aalborg University Citation for published version (APA): Willemsen, S., Paisa, R., & Serafin, S. (2020). Resurrecting the Tromba Marina: A Bowed Virtual Reality Instrument using Haptic Feedback and Accurate Physical Modelling. In S. Spagnol, & A. Valle (Eds.), Proceedings of the 17th Sound and Music Computing Conference (pp. 300-307) General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. ? Users may download and print one copy of any publication from the public portal for the purpose of private study or research. ? You may not further distribute the material or use it for any profit-making activity or commercial gain ? You may freely distribute the URL identifying the publication in the public portal ? Take down policy If you believe that this document breaches copyright please contact us at [email protected] providing details, and we will remove access to the work immediately and investigate your claim. Proceedings of the 17th Sound and Music Computing Conference, Torino, June 24th – 26th 2020 RESURRECTING THE TROMBA MARINA: A BOWED VIRTUAL REALITY INSTRUMENT USING HAPTIC FEEDBACK AND ACCURATE PHYSICAL MODELLING Silvin Willemsen, Razvan Paisa and Stefania Serafin Multisensory Experience Lab, CREATE Aalborg University Copenhagen fsil, rpa, [email protected] ABSTRACT This paper proposes a multisensory simulation of a tromba marina – a bowed string instrument in virtual reality. The auditory feedback is generated by an accurate physical model, Figure 1. A tromba marina owned by Nationalmuseet in the haptic feedback is provided by the PHANTOM Omni, Copenhagen, Denmark. and the visual feedback is rendered through an Oculus Rift CV1 head-mounted display (HMD). Moreover, a user study exploring the experience of interacting with a virtual bowed own specific challenges worth investigating. Sile O’Modhrain string instrument is presented, as well as evaluating the is one of the pioneers that noticed the tight connection be- playability of the system. The study comprises of both tween auditory and haptic feedback and investigated how qualitative (observations, think aloud and interviews) and haptic feedback can improve the playability of virtual in- quantitative (survey) data collection methods. The results struments [4]. At the same time, Charles Nichols devel- indicate that the implementation was successful, offering oped the vBow, a haptic human computer interface for bow- participants realistic feedback, as well as a satisfactory mul- ing [5]. For several years, researchers from ACROE in tisensory experience, allowing them to use the system as a Grenoble have developed multisensory instruments based musical instrument. on the mass-spring-system paradigm, with custom-made bowing interfaces [6,7]. Such multisensory simulations 1. INTRODUCTION have recently been made open source [8]. Haptic feedback has also been combined with digital waveguide models for The tromba marina is a bowed monochord from medieval simulating bowed string interactions [9]. Europe [1] (see Figure1). The string rests on a loose bridge Simulating the feeling of string-instrument vibrations is that rattles against the body. This rattling mechanism cre- particularly important since it has been shown how vibra- ates a sound with brass- or trumpet-like qualities. Unlike tions’ level can be strongly perceived [10]. We use de- other bowed string instruments, different frequencies are mocratized VR technologies controlled by a commercial created by slightly damping the string with a finger of the device called the PHANTOM Omni (or simply Omni) by non-bowing hand as opposed to pressing the string fully SenseAble Technologies (now 3D Systems) [11]. The Omni against the neck. This interaction at different locations is a six-degrees-of-freedom system providing the tracking along the string triggers the different harmonics of the open and haptic feedback (up to 3.3 N) in our application. Us- string. Furthermore, the tromba marina is bowed closer to ing the same device, Avanzini and Crosato tested the in- the nut, and the finger determining the frequency is closer fluence of haptic and auditory cues on perception of ma- to the bridge (below the bow). As the tromba marina is a terial stiffness [12]. Auditory stimuli were obtained using rare instrument which can be merely found in museums, a physically-based audio model of impact, in which the very few have the opportunity to play it and discover its colliding objects are described as modal resonators that in- interesting timbral possibilities. We wish to recreate the teract through a non-linear impact force [13]. Auditory feeling of playing this instrument by using physics based stiffness was varied while haptic stiffness was kept con- multisensory simulations [2]. stant. Results show a significant interaction between au- In the context of musical applications, physics based mul- ditory stiffness and haptic stiffness, the first affecting the tisensory simulations have shown some interest in the sound perception of the second. Passalenti et. al’s also used and music computing community. As stated in [3], the the Omni to simulate the act of plucking a virtual guitar combination of haptics and audio visual content has its string [14–16]. The goal of this project is to explore the experience of Copyright: © 2020 Silvin Willemsen et al. This is an open-access article distributed interacting with virtual bowed instrument by using physics under the terms of the Creative Commons Attribution 3.0 Unported License, which based simulations and haptic feedback, together with a vi- permits unrestricted use, distribution, and reproduction in any medium, provided sual virtual reality (VR) experience. This effectively makes the original author and source are credited. the implementation a virtual reality musical instrument (VRMI) 300 Proceedings of the 17th Sound and Music Computing Conference, Torino, June 24th – 26th 2020 [17]. The tromba marina is used solely as inspiration be- cause it affords itself to being a solid starting point by hav- ing only one string. Besides that, the rarity of the instru- ment ensures that the participants do not have prior expe- rience playing a tromba marina, nullifying possible com- parisons between a real instrument and the virtual one. At no point the system was evaluated as an alternative to the real tromba marina. The system (and its evaluation) is targeted towards musicians in order to avoid discourage- ment frequently encountered when non-musicians interact with musical instruments. It is assumed that musicians ac- knowledge that mastering any instrument require extended study, therefore it is expected that they will not evaluate this system exclusively based on its difficulty to play. We start by describing the implementation of the system, Figure 2. Diagram showing the system layout of the appli- both from the hardware and software perspective in Sec- cation. The user interacts with the system using the Omni tion2, followed by presenting a study that evaluates the – which in turn provides haptic feedback – and the Ocu- setup in Section3. Section4 shows the results of the eval- lus Touch controller. These trigger the physical model of uation and Section5 discusses these. Finally, concluding the tromba marina. Auditory feedback then comes from remarks appear in6. speakers and visual feedback from the Oculus Rift head- set. A detailed explanation can be found in Section 2.5. 2. IMPLEMENTATION The virtual tromba marina consists of three main compo- nents: auditory, visual and haptic feedback, all of which will be elaborated on in this section. For visuals, the Ocu- lus Rift CV1 setup was used [18]. The setup consists of a head-mounted display (HMD) and a pair of of wireless controllers that provide tracking information and user input through several buttons and a joystick. A diagram showing the full setup of the system can be found in Figure2. The controls, their mapping to the system and the final setup of the system will also be presented. A video showing the implementation can be found in [19]. 2.1 Auditory Feedback The audio is generated by a physical model of the tromba Figure 3. The view from the head-mounted display marina presented in a companion paper [20]. Some pa- (HMD). The damping finger is highlighted and shown as rameters of the model are exposed and can be controlled a transparent white sphere. by the user. These are the velocity, force and position of the bow and the position of the finger inducing the harmon- ics. The algorithm will not be discussed in detail here, but 2.3 Haptic Feedback the mapping to the various parameters of the model will be The PHANTOM Omni (or simply Omni) is a six-degrees- described in Section 2.4. of-freedom tracking and haptic system developed by Sens- Able Technologies (see Figure4). The device has a pen- 2.2 Visual Feedback shaped arm that a user interacts with. The application was built using the cross-platform game The raw data provided by the Omni are 1) the absolute engine Unity3D (or simply Unity) [21] which can be used position of pivot point B2 (three degrees of freedom), 2) to build VR applications. Even though the visual feed- the rotation (three degrees of freedom), and 3) the pressure back is not the focus of the implementation and eventual (touching depth). The latter is calculated from the absolute evaluation, it was used to guide the users’ movements and euclidean distance between the virtual collision point of give them a sense of where the virtual instrument was lo- the object (in our case the bow) and the virtual position of cated.