Stefania Serafin,∗ Cumhur Erkut,∗ Juraj Kojs,† Niels C. Nilsson,∗ Musical and Rolf Nordahl∗ Instruments: State of the ∗Multisensory Experience Lab Aalborg University Copenhagen Art, Design Principles, A.C. Meyers Vænge 15 2450 Copenhagen, Denmark and Future Directions {sts, cer, ncn, m}@create.aau.dk †Frost School of Music University of Miami P.O. Box 248165 Coral Gables, FL 33146, USA [email protected]

Abstract: The rapid development and availability of low-cost technologies have created a wide interest in virtual reality. In the field of computer music, the term “virtual musical instruments” has been used for a long time to describe software simulations, extensions of existing musical instruments, and ways to control them with new interfaces for musical expression. Virtual reality musical instruments (VRMIs) that include a simulated visual component delivered via a head-mounted display or other forms of immersive visualization have not yet received much attention. In this article, we present a field overview of VRMIs from the viewpoint of the performer. We propose nine design guidelines, describe evaluation methods, analyze case studies, and consider future challenges.

In 1965, Ivan Sutherland described his idea of an between novel interfaces and sound synthesis algo- ultimate display, conceived as a multisensorial rithms have been at the forefront of development. virtual experience able to recreate the Wonderland Progress in interactive auditory feedback and novel where Alice once walked (Sutherland 1965). More control mechanisms has made great strides. On than 50 years later, such ultimate display is nearly the other hand, the new interfaces for musical ex- at hand. In the past couple of years, the rapid devel- pression (NIME) community has not shown a wide opment of low-cost virtual reality displays, such as interest in the visual aspects of VMIs. One of the the Rift (www.oculus.com/rift), HTC’s Vive reasons why virtual reality musical instruments (www.htcvive.com), and open-source virtual reality (VRMIs) have not drawn more attention in this (www.razerzone.com/osvr), has boosted interest in community might be the fact that musicians mostly immersive virtual reality technologies. Hardware rely on their auditory and tactile feedback and on the devices once exclusive to high-end laboratories are interaction with the audience. Another reason could now available to consumers and media technol- be that low cost and portable visualization devices ogy developers. We define virtual reality (VR) as have only become available in the last five years. an immersive artificial environment experienced Researchers working with interactive audio through technologically simulated sensory stimuli. and haptic feedback have called their instruments Furthermore, one’s actions have the potential to VRMIs (see, e.g., Leonard et al. 2013); however, shape such an environment to various degrees. visual feedback was absent. In this article we dis- In the fields of new interfaces for musical ex- tinguish, on the one hand, between virtual musical pression and sound and music computing, virtual instruments (VMIs), defined as software simulations musical instruments (VMIs) have been developed or extensions of existing musical instruments with and refined in recent decades (see, e.g., Cook 2002; a focus on sonic emulation, for example, by physical Smith 2010). Physical interfaces for musical ex- modeling synthesis (Valim¨ aki¨ and Takala 1996) and, pression, gestural control, and mapping systems on the other hand, virtual reality musical instru- ments (VRMIs), where the instruments also include a simulated visual component delivered using either Computer Music Journal, 40:3, pp. 22–40, Fall 2016 doi:10.1162/COMJ a 00372 a head-mounted display (HMD) or other forms of c 2016 Massachusetts Institute of Technology. immersive visualization systems such as a CAVE

22 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 (Cruz-Neira, Sandin, and De Fanti 1993). We do not the audience to the mechanisms of digital musical consider those instruments in which 3-D visual instruments using 3-D visualization. The topic of objects are projected using 2-D projection systems, audience participation in virtual reality experiences such as the environment for designing virtual mu- lies, however, beyond the scope of this article. sical instruments with 3-D geometry proposed by Virtual reality musical instruments have the Axel Mulder (1998) and the tabletop based systems potential to bridge the fields of NIME and VMIs. proposed by Sergi Jorda` (2003). We focus instead on Adding meaningful immersive visualization to a the examination of 3-D immersive visualizations digital musical instrument can result in a new way from the viewpoint of the performer. of experiencing music. In the following sections, We believe that virtual reality shows the greatest we present an overview and the current status of potential when facilitating experiences that cannot research and practice of VRMIs. We propose nine be encountered in the real world. As Jaron Lanier design guidelines, provide case studies, and discuss envisioned in Zhai (1998): potential future challenges. I have considered being a piano. I’m interested in being musical instruments quite a lot. Also, Background Work you have musical instruments that play reality in all kinds of ways aside from making sound in Although research and development in VMIs has a Virtual Reality. That’s another way of describing long history and many examples can be found in the arbitrary physics. With a saxophone you’ll be literature, the same cannot be said for VRMIs. In able to play cities and dancing lights, and you’ll the following we propose an overview of the most be able to play the herding of buffaloes made of significant VRMI research efforts. crystal, and you’ ll be able to play your own body Cadoz and colleagues have been developing their and change yourself as you play the saxophone CORDIS-ANIMA system for many years. It is a (Heilbrun 1989, p. 110). modeling and simulation programming language for In line with Lanier’s thoughts, immersive tech- audio and image synthesis based on physical models nologies enable new musical experiences that extend (Cadoz, Luciani, and Florens 1993). Although to beyond those offered by traditional musical instru- our knowledge the system has never been used in ments. Immersive visualizations thus stimulate immersive virtual environments, it includes all additional levels of abstraction, immersion, and the elements of immersive multimodal musical imagination. instruments, covering auditory, haptic, and visual On the other hand, virtual reality research has feedback. focused mainly on the exploration of visual feedback, Maki-Patola¨ and coworkers reported on a software with other senses playing a secondary role. In some system designed to create virtual reality musical cases, the sole focus on visual domain can provide instruments (Karjalainen and Maki-Patola¨ 2004; a powerful visual experience away from the daily Maki-Patola¨ et al. 2005). The team presented four life. Ben Shneiderman has stated that virtual reality case studies: a virtual xylophone, a membrane, a offers a lively new alternative for those who seek theremin, and an air guitar. The authors discuss the immersion experiences via blocking out the real quality, efficiency, and learning curve as outlined by world with goggles on their eyes (Preece et al. 1994). Jorda` (2004). For example, low spatial and temporal It is important to note that virtual reality can pose resolution and latency decrease efficiency (Maki-¨ some challenges for the audience that witnesses a Patola et al. 2005). They suggest that existing performer wearing head-mounted display for im- technologies can improve these deficiencies and mersive visualization as stated by Berthaut, Zappi, allow for more responsive instruments, expanding and Mazzanti (2014). This is mainly because the both design and performance possibilities. audience is unable to share performer’s experience. One of the most important features of VR in- Berthaut et al. (2013) presented an attempt to expose terfaces is their potential for visualization. Virtual

Serafin et al. 23

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Figure 1. Virtual FM Synthesizer (a) and Virtual Air Guitar (b). Images are from http://mcpatola.com /almaproject/HUTTMLIndex.html, used with kind permission of Teemu Maki-Patola.¨

reality musical instruments can stimulate intel- visualization. Gelineck and colleagues further de- ligent visual feedback that may aid the player in tailed possibilities that could not be achieved in the performing. This expands the interaction with physical counterpart, such as the ability to change the physical instruments, where visual feedback the dimensions of the simulated instruments while is directly connected to the mechanism of sound playing them. production, for example, vibrating strings. Gelineck Similarly, the FM synthesizer and the virtual air et al. (2005) proposed several physics-based VRMIs, guitar that were developed within the Algorithms such as a flute and drum. The instruments used for the Modeling of Acoustic Interactions (ALMA) a combination of physical models, 3-D visualiza- project and presented by Maki-Patola¨ and coworkers tions, and alternative sensing techniques. Blowing sonically expand the theremin and guitar, respec- into a plastic tube would trigger a dynamo motor, tively (Karjalainen and Maki-Patola¨ 2004; Maki-¨ allowing a natural performance experience. The Patola et al. 2005) (see Figure 1). Their visual rep- performer could see the virtual flute through a 3-D resentations, however, expand beyond the expected

24 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 originals. The instruments are further described in connecting individual keys into lines and sequences. the case studies section of the present article. Visual feedback thus assists in the conversion of Examples of interactive multimodal VR envi- performing discrete events to continuous piano ronments were described by Rodet et al. (2005); playing. these were a part of the project Plateforme haptique d’application sonore pour l’eveil musical (PHASE). The goal of the PHASE project was to provide Design Principles for VRMIs musical game installations with auditory, haptic, and visual feedback. Multisensory interactions en- Ge Wang (2014) presented a set of principles for abled a general audience to explore new musical visual design of computer music, inspired by possibilities. principles for designing computer music controllers Rob Hamilton (2009) proposed an understanding put forward by Perry Cook (2001, revised and of virtual environments as enactive instruments in expanded 2009). In particular, Cook’s principle their own right, able to react to gesture and action. that “copying an instrument is dumb, leveraging Hamilton suggested that just as the laws of physics expert technique is smart” (i.e., advice not to can be set aside within a virtual space, so could replace or copy existing instruments, but to consider the traditional methodologies of coupling gesture controllers inspired by virtuosity: violins, trumpets, to sound within space. In this way, traditional microphone stands, turntables, beat-boxers, ethnic relationships between gesture, sound, and space instruments, and so forth) is relevant also to VRMIs. could be recontextualized and up-ended. Hamilton Wang proposed a total of 13 principles, cate- also stated that VR and VRMIs allow possibilities gorized either as user-oriented or aesthetic, with of extensions to the real world, which become some additional observations. For the user-oriented particularly interesting in an artistic domain. Such principles, Wang advocates the need for real time, possibilities have been only modestly explored, for the ability to design sound and visuals in tandem however. and seek salient mappings, for hiding the technology Most VMIs have been developed as single-process and making the viewer experience substance, for instruments, i.e., instruments that allow control adding constraints, and for the use of graphics to over only one synthesis or effect process or musical reinforce physical interaction. For the aesthetic navigation tool. In the case of VRMIs, a principal principles, Wang suggests that the designer simplify, advantage of graphical musical interfaces is the animate, be organic, and have not only a function ability to handle multiprocess instruments. Based on but a personal aesthetic sense. Other principles are visual feedback from the selected sound processes, iteration, visualization, and the possibility of being Berthaut, Desainte-Catherine, and Hachet (2011) inspired by movies and computer games. Many of suggested relying on the concept of reactive widgets. these principles, if extended as explained in the The reactive widgets can be defined as complex 3-D following, are also relevant for VRMIs. objects associated with particular sound processes Design principles for new interfaces for musical (Levin 2000). expression were also proposed by Garth Paine (2009). The addition of visual feedback can aid the player Paine in particular stresses the importance of the in performance and can enhance the aesthetic fact that relationships to sound are in part physical: experience. As an example, Xiao, Tome, and Ishii musical instruments generally require the performer (2014) proposed to visualize music as animated to blow, pluck, strum, squeeze, stroke, hit, or bow. characters walking on a physical keyboard, to For VRMIs, some principles from new interfaces for promote an understanding of music rooted in musical expression and visual design are still valid. the body, taking advantage of walking as one of When designing VMIs based, for example, on the most fundamental human rhythms. Although physical models, a common assumption is that not immersive, the visualization is an interesting there is no need to simulate existing instruments, way to support music learning and thinking through because the real counterparts are of higher acoustic

Serafin et al. 25

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Figure 2. A general structure of the different input and output modalities required to design a virtual reality musical instrument.

quality and and can be more easily played. On the other hand, there are several reasons to build virtual musical instruments: for example, to provide software tools to musicians who do not have access to physical players of those instruments, or to understand the acoustics of such instruments though simulations. Another interesting aspect of VMIs is the fact that, because the instruments are virtual, they allow the creation of possibilities that could not be achieved in the real world. To cite a just few historical creative examples from established composers, there is Silicon Valley Breakdown by David Jaffe, written in 1983, where the simulation of plucked string instruments, achieved by implementing extensions of the Karplus-Strong algorithm (Jaffe and Smith 1983), allowed the sonic reproduction of strings as long as the Golden Gate Bridge. Another example is Oxygen Flute (Chafe 2005), an installation in a gallery chamber made with bamboo and containing four continuously performing virtual flutes, where For virtual reality musical instruments, it is not the visitors create the sounds by breathing. Patterns only relevant to design sound and visual in tandem, in levels of carbon dioxide measured inside the and to consider the mapping; touch and motion chamber created the music. should also enter into the equation, as well as full Kojs and colleagues have described more examples body interaction. From the perspective of auditory of augmentation (e.g., unfeasibly rapid performance feedback, virtual reality musical instruments have on a stretched marimba), hybridization (e.g., pluck- been presented through speakers or headphone- ing tubes and plates), and abstraction (e.g., bowing based systems. For virtual reality, 3-D sound is an the conductor and belling the composition) of virtual especially relevant issue, because the location and musical instruments created with physical model- motion of the visual virtual objects need to match ing techniques and their modes of interactions (Kojs, the location and motion of the auditory objects Serafin, and Chafe 2007; Kojs 2013). (Begault 1994). Figure 2 shows a general schematic representation From the perspective of tactile and haptic feed- of the components involved when designing virtual back, several studies (e.g., O’Modhrain et al. 2000a,b; reality musical instruments. The player performs Marshall and Wanderley 2011), suggest that the de- actions that are tracked and mapped to visual, velopment of musical skills strongly relies on tactile auditory, and haptic feedback. The feedback is and kinesthetic cues. While performing on acous- interpreted by the player, who performs the resulting tic or electro-acoustic musical instruments, an actions in a feedback loop. intense haptic experience is not only unavoidable, We now present our design principles in detail. it is highly relevant to the musical performance. Moreover, tactile feedback enhances playability of virtual musical instruments (O’Modhrain 2001). Principle 1: Design for Feedback and Mapping Haptic feedback can be delivered in free air, as in the case of the AIRREAL (Sodhi et al. 2013) or by We design for sound, visual, touch, and proprio- using a tangible interface. One of the purposes of ception in tandem, and we consider the mappings augmented visual feedback can be to help to create between these modalities. more intuitive, learnable instruments, as is the case

26 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 in Jorda’s´ FMOL and reacTable* (Jorda` 2003) for flicting information from the visual and vestibular table-based interfaces. Although the field of NIME senses; e.g., a user driving a virtual vehicle will has seen the proposal of several novel tangible be physically stationary while the visual stimuli interfaces and input devices, this is not the case for suggests that movement is occurring. System factors virtual reality. Most hand-based input devices for believed to influence cybersickness include latency, virtual reality are, as a matter of fact, still in the display flicker, calibration, and ergonomics (Davis, form of gloves, remote controllers, or joysticks. It Nesbitt, and Nalivaiko 2014). Aside from optimizing might be beneficial for VRMI researchers to consider factors such as efficient tracking and high update tangible interfaces developed in the NIME commu- and frame rates, developers of VRMIs should be nity to extend possibilities provided by traditional mindful of how the user’s movement around the VR input devices. Lately, the Leap virtual environment is made possible. Particularly, (www.leapmotion.com) has become a popular op- it is advisable to use a one-to-one mapping between tion in the consumer VR community, mainly for virtual and real translations and rotations, and if the its ability to track and represent both hands in the user has to move virtually while being physically 3-D world without the need for the user to wear any stationary, accelerations and decelerations should additional interface, such as gloves or joysticks. be minimized as the vestibular system is sensitive to such motion.

Principle 2: Reduce Latency Principle 4: Make Use of Existing Skills All interactions should be smooth and have min- imum latency. Because VRMIs are inherently This principle is related to Cook’s notion that it multisensory, reduction of latency with respect to is less interesting to copy existing musical instru- both sound and visuals is paramount. Particularly, ments; what is more interesting is to be inspired synchronization between the arrival of stimuli in by existing musical instruments and extend their different modalities is known to influence the per- possibilities. Virtual reality musical instruments ceptual binding that occurs in response to an event have the potential to become interesting when they producing multimodal stimulation (Kohlrausch and provide immersive experiences that are not possible van de Par 1999). Thus, it is crucial that VRMIs in the real world. The role of virtual reality, e.g., represent timely and synchronized audiovisual the role of the immersive visualization hardware feedback in response to the user’s actions. More gen- and software, is to enhance the overall multimodal erally, it is crucial to reduce system latency, since it experience. Virtual reality is a different medium is believed to increase cybersickness (LaViola 2000), than the physical world. Replicating interfaces of the topic of the next principle. traditional instruments in VR may not bring about useful results unless we are extending existing in- struments with additional control. Familiarity with Principle 3: Prevent Cybersickness a real-world counterpart helps to grasp the concept and supports playing in the beginning. But it may Cybersickness, or VR sickness (Fernandes and not be easy to find new ways of playing, which is Feiner 2016), may involve a range of different symp- needed as the instruments are different. toms including, but not limited to, disorientation, Users of traditional instruments may not even headaches, sweating, eye strain, and nausea (Davis, be interested in using virtual replicas, since the Nesbitt, and Nalivaiko 2014). Although alternative original ones work well. Instead, we should discover explanations exist (Riccio and Stoffregen 1991; the kinds of interfaces that are best suited for the Bouchard et al. 2011), the most popular is known VR medium. Using metaphors derived from inter- as sensory conflict theory. This theory stipulates actions existing in the real world offer interesting that cybersickness arises as a consequence of con- possibilities.

Serafin et al. 27

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Principle 5: Consider Both Natural Principle 7: Create a Sense of Presence and “Magical” Interaction The experience of “being there” in a computer- In a continuation of the previous principle, and in generated environment is often referred to as the line with general recommendations by Bowman sensation of presence or “place illusion” (Slater et al. (2004) for 3-D user interfaces, creators of 2009). The illusion is believed to be present in VRMIs should consider the use of both natural proportion to the degree of technological immersion and “magical” interactions and instruments. In offered by the system. The degree of technologi- relation to VRMIs, either an interaction or an cal immersion offered by a given system can be instrument will qualify as magical if it is not characterized by the range of normal sensorimotor limited by real-world constraints, such as the ones contingences supported by the system. Particularly, imposed by the laws of physics, human anatomy, Slater (2009) describes sensory motor contingencies or the current state of technological development. as the set of actions a user will know how to perform On the other hand, interactions and instruments in order to perceive something (e.g., moving one’s qualify as natural if they conform to real-world head and eyes to change gaze direction, kneeling to constraints. Notably, natural interactions can be get a closer look at something on the ground, or turn- combined with magical instruments and vice versa ing one’s head to localize the position of an invisible (e.g., the user may manipulate a physically plausible sound source). Thus, it is advisable for developers instrument located at a great distance using a of VRMIs to construct virtual instruments with the nonisomorphic approach (e.g., the Go-Go interaction limitations of the system in mind, to discourage technique, Poupyrev et al. 1996), or the user may users from relying on sensorimotor contingencies use natural gestures to manipulate an instrument not fully supported by the system. Beside depending that defies the laws of physics. The advantage on the range of normal sensorimotor contingencies, of natural techniques is that the familiarity of presence is also believed to be influenced by the such approaches may increase usability, whereas illusion of body ownership, discussed next. In other magical techniques allow the player to overcome words, seeing a body in the virtual environment that the real-world limitations normally imposed on the one feels ownership of contributes to the sensation musician and the instrument (Bowman et al. 2004). that one is inside that environment. As an example, because the virtual body can be scaled, musicians could play instruments located outside of their normal reach. Principle 8: Represent the Player’s Body

One of the challenges of VR is the fact that people Principle 6: Consider Display Ergonomics cannot see their own body represented in the , unless the real body is tracked and mapped to All the devices necessary to create virtual reality a virtual representation. The resulting sensation is experiences introduce the additional challenge called virtual body ownership, and several studies of how to hide the technology to focus on the have shown that it is possible in VR to generate experience. The recently introduced HMD devices perceptual illusions of ownership over a virtual are improving from an ergonomic perspective. While body seen from the first-person perspective, in wearing HMDs such as the HTC Vive or Oculus other words, of a body that visually substitutes the Rift, the user remains tethered and the weight person’s real body (Gallagher 2000). of the two displays is noticeable, however. Thus, Kilteni, Bergstrom, and Slater (2013) show how when creating VRMIs, developers should be mindful differences between the real and virtual body have of the potential strain and discomfort introduced temporary consequences for participants’ attitudes by wearing an HMD and of the issues currently and behavior under the illusory experience of introduced by wires (e.g., a 360-degree turn can body ownership. Particularly, differences in body leave the user entangled). movement patterns while drumming were found

28 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Figure 3. A schematic sound engine for a flute representation of a virtual and that animates a 3-D reality flute. The player visualization presented on blows on a controller that the screen. drives a physics-based

a virtual representation of the user is important, it may also be important that the appearance of this representation is appropriate for the precision of possible interactions.

Principle 9: Make the Experience Social

One of the fundamental aspects of music is its ability to create shared social experiences. On the other hand, as of this writing, virtual reality has merely been an individual activity where one person is immersed in one virtual world. This is mostly due to the occlusive properties of HMDs, blocking any visual communication with the outside world. Recent developments in VR technologies, however, are pointing towards the desire to create shared VR experiences. It is interesting to use other channels when the visual modality is blocked; in this case depending on whether the perceived body repre- the auditory and tactile channel can create some sentation fulfilled expectations of what appearance interesting possibilities that have not been explored. was and was not appropriate for the context. This This can create novel social musical experiences in study has interesting implications because it shows virtual reality. An alternative to facilitating social that virtual reality can help to train and adapt interaction between individuals in the same physi- movements and gestures of musicians. cal space is for users to share experiences virtually The presence of a virtual body representation and potentially experience social presence or “co- is also important to provide the necessary visual presence” (Nowak and Biocca 2003). Examples of feedback to the player. Consider, for example, a social applications of VRMIs include, among others: virtual reality flute, as displayed in Figure 3: The (1) using VRMIs for collaborative performances; for instrument consists of a controller (a cylinder) with example, musicians transported together while play- a breath-pressure sensor driving a flute physical ing the same or different instruments, and (2) using model, together with 3-D glasses to visualize the VRMIs for virtual performances; for example, the flute. The flute controller is tracked, so the virtual performance of the musician playing a VRMI is visual flute matches the position of the physical transmitted to an audience that shares the virtual controller. It is important for the player to receive environment with the performer. Finally, previous visual feedback of the position of the hands; this work has documented that when individuals are is achieved by tracking the player’s hands and asked to present in front of an audience of virtual displaying them in the virtual world. The lack of characters, they tend to respond similarly to how visual feedback breaks the sense of presence as they would in a similar real-world situation (Slater, described in relation to principle 7. Pertaub, and Steed 1999). Thus, it seems possible Notably, a study by Argelaguet et al. (2016) that virtual reality can provide performers with a explored the effects of varying degrees of realism in surrogate for actually being on stage while playing visualizing players’ hands and found that abstract VRMIs or traditional instruments. and iconic hand representations produced a greater sense of agency than realistic representations. One potential explanation is that the realistic Evaluation of VRMIs representation made the participants expect a more natural interaction than was offered by the Leap Evaluation of digital musical instruments involves Motion device used for the study. Thus, although several different parties, including instrument

Serafin et al. 29

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 makers, novice and skilled performers, and the property. It is clear that advances in VR display and audience. In this article we focus on performers. tracking technologies keep reducing the latency. Even then, several aspects have to be assessed, from Note, however, that there is a perceptual and motor mere task performance, as carried out by Wanderley tolerance to latency that can positively impact the and Orio (2002), to the playability, user experience, evaluation (Maki-Patola and Ham¨ al¨ ainen¨ 2004). and emotions, as outlined by O’Modhrain (2011). The concept of presence (DP 7) needs also special Andrew Johnston proposed a practice-based attention in this layer. Previously defined as the design for the evaluation of new interfaces for sensation of being in the simulated space, or the musical expression (Johnston, Candy, and Edmonds perceptual illusion of nonmediation (Lombard and 2008; Johnston 2011). The idea is a broader study Ditton 1997), presence has recently been redefined into performers and their creative practice in the as the combination of two orthogonal components: context of their use of new instruments. As an place illusion and plausibility (Slater 2009). Place experience-oriented framework, MINUET considers illusion corresponds to “being there,” the quality evaluation as a central activity in designing musical of having a sensation of being in a real place. interfaces (Morreale, De Angeli, and O’Modhrain Plausibility illusion refers to the illusion that 2014). Similar evaluation frameworks would also the scenario being depicted is actually occurring. be useful for VRMIs. Fortunately, the nine design In the music domain, Charles Ford (2010) has principles (DPs) outlined in the previous section given presence a different definition, referring in can be converted to an evaluation framework, with general to those qualities that make a performance three distinct layers. outstanding, and are hard to quantify scientifically. The first layer concerns the modalities of in- From the player’s point of view, it is probably teraction (Erkut, Jylha,¨ and Discioglu 2011). The more relevant for VRMIs to use engagement as an alignment between each input and output modality evaluation method, referred to as the ability of the (cf. Figure 1) would provide a good start in this layer player to keep interest in playing the instrument (DP 1). After considering each modality in tandem, for a significant amount of time (Wessel and Wright we can then proceed with perceptual integration 2002). A relevant element for VMIs is the concept of and mapping (DP 1), based on perceptual, senso- agency, e.g., the ability to take meaningful actions rimotor, and cognitive abilities and capacities of and see the results of one’s decisions and choices users. Such a structured approach has been previ- (Herrera, Jordan, and Vera 2006). ously used by Erkut, Jylha,¨ and Discioglu (2011) The final layer focuses on the quality and goals to evaluate a rhythmic interaction system with of interaction and on the specific mechanisms in- a virtual tutor (Jylha¨ et al. 2011) and to redesign volved, such as natural and magical interactions the auditory and visual displays of the interactive (DP 5), and leveraging expert techniques (DP 4) for system. Besides specific issues in each modal- musical expression. This layer focuses on higher ity, the approach identified the effects of latency design goals, embracing experience and “meaning- (DP 2), social factors (DP 9), and to some degree, making” (Bødker 2006). The experience can evolve the ergonomics of the display, in terms of fatigue over time, particularly in social contexts (DP 9, (DP 6). cf. Bødker 2015). In this layer, evaluators should The second layer is VR-specfic, and considers avoid breaks in presence; therefore, methods to evaluation of cybersickness (DP 3), virtual body assess the user experience without interruptions representation and ownership (DP 8), and presence are desirable. One widely adopted way to assess (DP 7). The VR-specific evaluation methods should user experience in virtual reality is by using pos- be followed in this layer; see the related work in texperimental questionnaires, sometimes coupled corresponding design principles. Latency (DP 2), for with physiological measurements captured during instance, which was considered in the previous layer the experiment (Meehan 2001). Extensions towards for each interaction modality, should be revisited practice- and experience-based frameworks are also in the VR-specfic level as an integrated system possible at this layer.

30 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 To summarize, an evaluation of VRMIs from teraction and subsequent prolongation of the user’s the performer’s point of view presents challenges experience. Maki-Patola¨ and coworkers comment, similar to those with the evaluation of VMIs in however, that the lack of tactile feedback diminishes terms of usability, playability, and engagement. But the speed and accuracy of the performance, and they it also presents additional challenges introduced by suggest using a physical controller such as a MIDI the immersive visualization component, such as drum set to compensate. Extending the interaction a sense of presence and the need of a virtual body to feet controlling attacks would free the hands and representation to create agency. These challenges thus deepen the connection between the player and can be structured with a three-layer evaluation instrument. This virtual instrument presents a 3-D scheme: interaction modalities (the lowest layer), a audiovisual scenario focused on the reproduction VR-specific middle layer, and the highest layer aim- of naturally observed interactions (a mallet hitting ing to evaluate the goals, practices, and experiences a membrane) and the resulting sonorities (sounds of VRMI players. of an excited membrane). The ability to dynami- cally alter parameters such as dimensions, tension, damping, and material in real time also enables Case Studies expansion of the existing instrument’s sonic palette beyond what is possible in the physical world, In this section we analyze VRMIs and examine se- however. The resulting sounds were described as lected instruments through the lens of our proposed potentially useful, for example, as special effects for design principles. We summarize this discussion in film media. Latency was reported as a major prob- Table 1. lem (observed above a threshold of about 60 msec). Although cybersickness was not addressed in the analysis, the fact that the player’s gestures would Virtual Membrane, Xylophone, and Air Guitar pass through the visualized membrane suggests an upsetting experience. The visual display consists Maki-Patola¨ and coworkers describe a number of waveform animations traveling along a virtual of VRMIs such as a virtual membrane, a vir- plate. The player’s body is not represented in the tual xylophone, and a virtual air guitar. We have VR. The lack of the player’s immersion in the VR described the virtual air guitar as an abstract in- environment suggests a weakened sense of presence. strument (see Figure 1) earlier in this article; here Although one can imagine a potential for shared we focus on the virtual membrane and the vir- experience, any social interaction is absent in the tual xylophone (Maki-Patola¨ et al. 2005; see also current iteration of the virtual membrane, since a mcpatola.com/almaproject/TMLvideos.html). The single player operates the instrument. instruments are depicted in Figure 4. The virtual xylophone is an extended reproduc- The virtual membrane is an instrument based tion of the traditional xylophone, and, like the on a physical model of a rectangular membrane. virtual membrane, it is played with mallets. The The player operates two virtual mallets, hitting first version of the instrument had a traditional, the virtual membrane simulations at various loca- linear key layout (see Figure 4b), whereas the plates tions. Such control and interactions were reported can be moved in 3-D space in the second version (see as intuitive and natural, as they are derived from Figure 4c). The performer can customize the plates the real-life situation. The location hit determines using an interface designed for this purpose. In this which of the particular modes of the membrane way, the instrument follows DP 4, which prefers would be excited, thus enabling timbral differen- leveraging expert techniques to copying existing tiation associated with location. The pairing is instruments. well aligned with the behavior of a physical mem- The virtual air guitar is a digital reproduction brane. It is noteworthy that sufficient flexibility and of a traditional electric guitar using an extended complexity of the sound model allows for rich in- Karplus-Strong model for the sound production

Serafin et al. 31

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 32

Table 1. Summary of VRMIs Described in Case Studies VRMI DP 1: Feed- DP 2: DP 3: DP 4: Do Not DP 5: Interac- DP 6: Er- DP 7: Sense of DP 8: Body DP 9: Social back and Latency Cyber- Copy tion gonomic Presence Represen- Experience Mapping sickness Design tation Virtual AV in tandem Ca. 60 Not Copy More natu- Yes Weakened by No No: single membrane no haptic msec addressed ral, some the lack player but tactile magical of tactile feedback feedback Virtual AV in tandem Ca. 60 Not Not a copy More natural Yes Weakened by No No: single xylophone no haptic msec addressed the lack player but tactile of haptic feedback feedback Virtual air AV in tandem Ca. 60 Not Not a copy, but More natural Yes Weakened by No No: single guitar msec addressed an extension the lack player of tactile feedback Gestural FM AV in tandem Not Not Extension More natural Yes Weakened by No No: single optrMscJournal Music Computer synthesizer no haptic noticeable addressed the lack player no tactile of tactile feedback feedback Crosscale AV in tandem Not Potential Extension Natural trig- Standard VR Through inter- No No: single noticeable gering Oculus plus action player Hydra ChromaChord AV in tandem Not Potential Not a copy or Not natural Standard VR Through Hands No: single noticeable extension Oculus plus hands player Leap Motion Wedge AV in tandem Not Not Not a copy or Mostly natural Standard VR Through Hands No: single noticeable addressed extension Oculus plus hands player Leap Motion Cirque des AV in tandem Not Not Copy Not natural Standard VR Through Hands No: single Bouteilles noticeable addressed Not magical Oculus plus hands player Leap Motion AV = audiovisual. Figure 4. The virtual membrane (a) and the virtual xylophone in two versions (b and c). Images are from http://mcpatola .com/almaproject/HUTTMLIndex.html, with permission of Teemu Maki-Patola.¨

engine (Karjalainen et al. 2006) but without tactile pected on an acoustic instrument, however. The feedback. Maki-Patola¨ and colleagues reported it to absence of both a sustain pedal and any resonance be the most successful among the VR instruments, of the simulated instrument’s body further lim- principally because of the popularity of its physical its the expressivity of performance. Furthermore, counterpart, the electric guitar. the digital display resembles a game environment more than an instrument. Although latency is not observed, the speed of interaction is limited to Gestural FM the performer’s arm movement. This is because distances between target points on the screen The gestural FM synthesizer, also developed by are larger than those observed on a physical key- Maki-Patola¨ et al. (2005) and described earlier, board, and the arm movement replaces the nuanced is an extension of the theremin instrument (see finger control. Cybersickness may become a prob- www.theremin.info for more information). A com- lem with extending an arm forward and retracting plex FM synthesizer replaces the simple frequency it, since the sense of depth is not stimulated by spectrum of the original theremin. The visual any sense of vibrotactile feedback. Moreover, in- feedback was reported to increase playability as creased speed of gestures may cause a sense of compared with the original theremin instrument, nausea and physical fatigue, as the parallelism where the only feedback is aural. between arm movements and traveled distances between the actual points on the display are dispro- portionate. Natural interaction between reaching Crosscale and triggering is successful, although, this is pre- cisely where the limitation of the magical appears. Crosscale (Cabral et al. 2015; see also youtu.be Enlarging the sound bank and using the dynamic /9ecZIPQnDXQ) uses an HMD and a properties of the arm motion would enable further Razer Hydra, a joystick-like interaction device to expansion of expressive potential. Oculus Rift and play a set of notes displayed as colored spheres on Razer Hydra provide a typical level of ergonomic the screen. The intention is to perform notes in a comfort by today’s standards. The body image is 3-D space (chords can be controlled on the z-axis), not present in the system. Finger triggering and instead of the linear left-to-right convention of a arm movement are the principal vehicles that en- traditional keyboard. In contrast, the use of both able a sense of internal presence. Although this hands enhances the parallel between piano playing project was designed for a single user, one can and playing the virtual Crosscale. The performer’s imagine a participatory experience with multi- gestures trigger discrete events (single sounds); ple users performing complex sequences on the these do not evolve dynamically as would be ex- instrument.

Serafin et al. 33

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 ChromaChord That is, players are aware of their body parts, yet the hands are detached from the rest, undermining ChromaChord (Fillwalk 2015; see also youtu.be the sense of realism. Attaching the Leap Motion to /5C1O1MvouXc) is a system that combines the Oculus Rift is an elegant solution, yet it adds a small Oculus Rift headset with a Leap Motion controller amount of weight and awkwardness to the interface. attached to it in order to create an immersive 3-D The project was designed for a single user, although environment. The Oculus Rift allows a performer it would be reasonable to consider a version in which to view a three-panel visual interface with a black multiple players share the proposed virtual musical background that includes a pane with two rows of space. coloured rectangles, a white oval-centered display with particle-flocking around the central oval, and a white-and-gray table consisting of three rows. Wedge Performers can navigate between the three panels by turning their head. A performer uses both hands The Wedge musical interface, designed by Moore and fingers to interact with the interface. The Leap et al. (2015; see also youtu.be/tPxdVSrEC1U), also Motion Controller supports hand tracking, and ges- utilizes bimodal interaction and 3-D visualization tures such as point, pinch, swipe, grab, etc., result using the Leap Motion controller and an Oculus in specific interactions. Musical effects generated Rift DK2 headset. Inside–out tracking allows for and processed in Max/MSP complete the interaction 360-degree control and bimanual visualization and loop. Pointing at the colored pane enables perfor- interaction. Wedge is conceived as a customizable mance of single notes and gray arrows facilitate environment for composition and performance. octave transposition. Melodic sequences can be con- Utilizing the 3-D surroundings, a performer is able trolled with a single hand. Grabbing and pointing to move and arrange virtual rectangular blocks with gestures enable the performer to expand and reduce assigned pitches and MIDI note numbers. A virtual the amount of modulation on the synthesized sound image of a robotic hand operates in two modes, via the white oval-centered panel. The three-row called “build” and “play.” In build mode, palm and table display enables key and chord selection. Al- index finger are oriented sideways, allowing for though interaction with the three-pane interface organization of notes into sequences and snapping will appear obvious to those performers familiar them into chords laid out on the z-axis. The palm with today’s touchscreen interfaces, players with and index finger point upwards in play mode. For less computer experience may find the interaction the most part, the tip of the index finger is used for more difficult to learn; the gestures for triggering alignment, arrangement, and triggering of sounds. chords, in particular, may seem unclear. Fast and The FluidSynth synthesis engine affords a variety precise movement may be difficult to track with the of instrumental sounds that can be selected from Leap Motion, resulting in involuntary note trigger- an additional virtual scroll-down menu overlaid ing. Fast movements may result in slight latency over the virtual blocks. This solution is effective, and a sense of cybersickness. The display is both although, it can cloud the performance space. panel- and keyboard-oriented, somewhat similar to Perhaps moving it to a side would be more efficient. that of CrossScale. Given the “vintage” nature of Because occlusion of the blocks and vibrotactile synthesized sounds, it is plausible that the authors feedback are missing, the hands pass through intended a display corresponding to early synthe- the objects without resistance. This may cause sizer computer keyboards. Hands (fingers, palms, cybersickness and problems with sense of presence. and wrists) are visualized in three dimensions, using Single-finger interaction is effective, yet limiting to white solid color on black background. The sense of the speed and precision of the gestures. The VRMI hands detached from the body and suspended in the is suspended in a planetary environment, which black space is intriguing. It simultaneously enables gives it a sense of depth. The hand is modeled with and denies a sense of presence in different ways. 3-D joints, giving it a sense of robotic veracity and

34 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 scale superimposed onto the quasi-science-fiction are displayed on the label, creating a direct motion– 3-D environment. The blocks emerge from the visual-feedback loop. Pointing to the opening of layer of atmospheric space, further enhancing the the bottle and virtually touching it with the right sense of presence. Interaction modes rely heavily hand produces the sound of a blown bottle. While on natural gestures, and the resulting sounds do the bottle is excited, an accompanying visualization not push beyond the boundaries of the expected. of the bottle turning a gray color with a gust of The Oculus Rift and Leap Motion constellation air coming out of it enhances the veracity of the supports standard ergonomic practices. Similarly situation. to the previous projects, Wedge was designed for a Thanks to a certain playfulness of Cirque des single performer, although, it is clear that a social Bouteilles, the project encourages the engagement of interaction with multiple users can be implemented. a variety of performance styles. Grabbing, pointing, and sliding bimanual gestures enable a sense of partial body presence. Using the natural interaction Cirque des Bouteilles of blowing would certainly increase the sense of presence, aligning the natural and virtual actions. In Cirque des Bouteilles (described by its authors as Perhaps this was not an objective of this project, "The Art of Blowing"), the player performs on bottles, however. The project’s character is primarily hu- creating an audiovisual feedback loop (Zielasko et al. morous, and thus touching the bottle instead of 2015; see also youtu.be/l-4TJsU5Gb8). The system blowing it is, in this case, deliberately unnatural. is realized with the aid of an Oculus Rift headest As a result of this misalignment (continuous blow- for visualization and a Leap Motion controller ing excitation replaced by pointed triggering), the for gesture tracking. A virtual space is set up as attack and release of the sounding samples seem too a 3-D visualization of a recording studio room, abrupt, producing unnatural fade-ins and cut-offs. complete with door, glass window separation from Latency is not noticeable; the precision in initiating the recording booth, some audio equipment, a floor the sound can be troublesome, however. This is rug, and a mixing console whose size and color due to the fact that excitation point (the opening of match the physical desk at which the performer the bottle) is rather small. Performance calibration sits. Virtual ambient lighting and shadows are added while playing on multiple bottles can be extremely for enhanced realism. Simulations of seven bottles challenging, even at a moderate tempo, and it can are located on the desk in place of a mixer, showing induce cybersickness. The desingers’ crafting of the the playful spirit of the designers. Each bottle has 3-D environment, hands, and bottles enhances the a label indicating the frequency it plays, and the sense of presence. Although the current iteration is percentage of liquid in it, together presenting a designed for a single user, it is clear that a shared per- collection of a musical scale. Seated at the desk, formance could lead to lively musical explorations performers can look around to situate themselves and social interactions. in the environment. In performance, both hands are in action and tracked by a Leap Motion unit placed on the desk. Simulations of palms and fingers are Summary designed with focus on an animation of the bone structure. Additionally, hands are color-coded: the Crosscale, ChromChord, The Wedge, and Cirque left hand in solid red and the right in solid green. des Bouteilles were among the instruments en- The left hand can grab the bottles, one by one, tered in the 2015 3DUI contest (http://3dui.org and place them closer in front of the player. Upward /2015/contest awards.htm, accessed 6 July 2016), and downward movements of the right hand with which was part of the International Symposium on the index and middle fingers extended forward 3-D User Interfaces (3DUI) held in Arles, France, are used to tune the bottle. Immediate changes of 23–25 March 2015. The state-of-the-art symposium frequency, pitch, and liquid percentage parameters focused on providing an opportunity for industrial

Serafin et al. 35

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 and academic researchers in the field of 3DUI to tency did not hinder the performance experience. present their products and exchange ideas. The Positioning the instrument in a crafted 3-D envi- sixth contest, named 3DUIIdol, was conceived to ronment did enhance the sense of presence. The highlight innovative and creative solutions to chal- predominant combination of Oculus Rift and Leap lenging 3DUI problems. The requirements included: Motion units showed the reliance on commercially expansion beyond the mouse and keyboard inter- available technologies, which raise some precision action with VRMI; playing a sequence of notes, and mapping problems as well as being somewhat scales, and chords; the ability to learn and operate bulky interfaces. the system by those who can read music with ease; Developing a virtual object is typically a process and the versatility to perform other scores. Addi- rather than an abrupt change (Levy 1998). In the end, tionally, the contestants were asked to prepare a challenges arise from enabling the imagination to performance of “Frere` Jacques” in a live demo. (See guide the musical exploration. Establishing perfor- http://3dui.org/2015/contest.htm for the complete mance practice, repertoire, and lasting technologies guidelines.) will complement the process. Perhaps additional de- Out of six finalists, a single winner was cho- velopment of shared experiences in VR can become sen. Although the competition did not specifically a playground for future explorations. require the use of immersive visualization, out As a final remark, it is worth considering why of those six finalists, five included an immersive the winner of the competition was, as previously visualization via head-mounted displays. Surpris- mentioned, the Digital Intonarumori (Serafin et al. ingly, the winner of the competition was the digital 2015). This is a successful anomaly given that Intonarumori, a physical interface lacking any im- the instrument lacked any augmented visual feed- mersive visualization and developed by the authors back. The instrument captured attention, however, of this article. The interface is a reproduction of because of the use of a physical controller with the original Intonarumori instruments developed enhanced tangible presence and an elaborate synthe- by Luigi Russolo and coworkers ([1913] 1967). An sis engine that enabled complex sound production. earlier prototype of such instruments was presented The performer’s gestures stimulate a continuous at the New Interfaces for Musical Expression con- flow of change, thus providing auditory feedback ference (Serafin et al. 2006). At the end of the article, not based on discrete points. Perhaps the most we will discuss why this interface was judged as the successful features of the instrument were map- most successful. Crosscale was the runner-up in the ping, natural playing technique, and intuitive contest. control. Based on these case studies, it is safe to state As for its construction, the instrument consists that the VRMI design is still in its infancy. It is not of a lever and crank. The lever movement controls surprising, however, as future VRMI designs will frequency of the resonator and rotational speed of spring forward precisely from the tension between the crank provides continuous excitation for the imitating the physical world while attempting to instrument. We have been working on implementing detach oneself from it. Historically, the theremin an option that would allow the performer to select and the Intonarumori have already paved a path. different classes of instruments directly from For the moment, the difficulty of completing the the tangible interface. Further addition of visual basic musical task of performing a melody with feedback is also being considered. The physical conviction on most of the described instruments interface enabled us to minimize the latency, reflects such tension. Because most of the instru- making the audio–tactile feedback loop smooth ments lacked a vibrotactile feedback, performance and believable. While measuring cybersickness, was imprecise and cybersickness was introduced. body representation, sense of presence, and VR Relying on crude arm and finger movements in medium inclusion did not apply to this iteration situations that required precision and speed posed of the project, visualization of the different hand further challenges in accomplishing the task. La- positions while operating the lever and crank as well

36 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 as an animated resultant frequency display are being be initiated and maintained by both designers and crafted for a future version of the instrument. performers. Then transformation from musical toys In this case the visual augmentation would and gadgets to fully developed instruments will work as an additional aid to the player, to better become joyfully realistic. understand where to position the lever to achieve a desired sonority. References Potential versions for and virtual reality systems, in which the the performer Argelaguet, F., et al. 2016. “The Role of Interac- would operate a 3-D lever and crank, are being tion in Virtual Embodiment: Effects of the Virtual considered. The solid physical design defines the Hand Representation.” In IEEE Virtual Reality. set of performance gestures; however, the flexibility doi:10.1109/VR.2016.7504682. of switching classes of instruments in the software Begault, D. 1994. 3-D Sound for Virtual Reality and enables surprising magical interactions. The same Multimedia. Cambridge, Massachusetts: Academic physical gestures could thus initiate diversely con- Press. trasting sonic processes. Expanding on Russolo’s Berthaut, F., M. Desainte-Catherine, and M. Hachet. ingenious design, the Digital Intonarumori physical 2011. “Interacting with 3D Reactive Widgets for interface manifests ergonomic features that enable Musical Performance.” Journal of New Music Research easy access and operation of the instrument. Al- 40(3):253–263. though the instrument was designed for a single Berthaut, F., V. Zappi, and D. Mazzanti. 2014. “Scenog- raphy of Immersive Virtual Musical Instruments.” performer, a social experience could be stimulated In IEEE VR Workshop: Sonic Interaction in Virtual through an ensemble of instruments. Environments, pp. 19–24. Berthaut, F., et al. 2013. “Rouages: Revealing the Mecha- nisms of Digital Musical Instruments to the Audience.” Conclusion In Proceedings of the International Conference on New Interfaces for Musical Expression. Available online Virtual reality technologies have provided new at hal.archives-ouvertes.fr/hal-00807049v1/document. possibilities for exploration of musical interaction. Accessed August 2016. Development of new techniques, graphic displays, Bødker, S. 2006. “When Second Wave HCI Meets immersion methods, and navigation systems has Third Wave Challenges.” In Proceedings of the been crucial in advancing research and applications. Nordic Conference on Human-Computer In- teraction: Changing Roles. Available online at We believe that the main challenge is to develop dl.acm.org/citation.cfm?id=1182476. Accessed August sophisticated VRMIs that expand on the currently 2016, subscription required. available VR musical installations and musical toys. Bødker, S. 2015. “Third-Wave HCI, 10 Years Later: In order to achieve this goal, continuous physical Participation and Sharing.” Interactions 22(5):24– interactions and multimodal feedback with the in- 31. strument will need to be established. The challenges Bouchard, S., et al. 2011. “Exploring New Dimensions in include understanding the unique potential that VR the Assessment of Virtual Reality Induced Side Effects.” and any other immersive technologies can bring to Journal of Computer and Information Technology the design strategies. 1(3):20–32. Learnability and training are yet another area Bowman, D., et al. 2004. 3D User Interfaces: Theory and for investigation and improvement. As we know, Practice. Boston, Massachusetts: Addison-Wesley. Cabral, M., et al. 2015. “Crosscale: A 3D Virtual Musical it takes years of practice to master a traditional Instrument Interface.” In IEEE Symposium on 3D User instrument. We believe that such practice could Interfaces, pp. 199–200. also be beneficial in the case of NIME and VRMIs. Cadoz, C., A. Luciani, and J. L. Florens. 1993. “CORDIS- Substantial time dedicated to practice and iterative ANIMA: A Modeling and Simulation System for design would be greatly beneficial to sustaining Sound and Image Synthesis; The General Formalism.” any VRMI. Such commitment, however, needs to Computer Music Journal 17(1):19–29.

Serafin et al. 37

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Chafe, C. 2005. “Oxygen Flute: A Computer Music Hamilton, R. 2009. “Building Interactive Networked Instrument That Grows.” Journal of New Music Musical Environments Using q3osc.” In Proceedings Research 34(3):219–226. of the AES 35th International Conference: Audio for Cook, P. 2001. “Principles for Designing Computer Music Games, pp. 1–6. Controllers.” In Proceedings of the International Heilbrun, A. 1989. “An Interview with Jaron Lanier.” Conference on New Interfaces for Musical Expression, Whole Earth Review (64):108–119. pp. 3–6. Herrera, G., R. Jordan, and L. Vera. 2006. “Agency and Cook, P. R. 2002. Real Sound Synthesis for Interactive Presence: A Common Dependence on Subjectivity?” Applications. Boca Raton, Florida: CRC. Presence 15(5):539–552. Cook, P. R. 2009. “Re-Designing Principles for Computer Jaffe, D. A., and J. O. Smith. 1983. “Extensions of the Music Controllers: A Case Study of SqueezeVox Mag- Karplus-Strong Plucked-String Algorithm.” Computer gie.” In Proceedings of the International Conference on Music Journal 7(2):56–69. New Interfaces for Musical Expression, volume 9, pp. Johnston, A. 2011. “Beyond Evaluation: Linking Practice 218–221. and Theory in New Musical Interface Design.” In Cruz-Neira, C., D. J. Sandin, and T. A. De Fanti. 1993. Proceedings of the International Conference on New “Surround-Screen Projection-Based Virtual Reality: Interfaces for Musical Expression, pp. 280–283. The Design and Implementation of the CAVE.” In Johnston, A., L. Candy, and E. Edmonds. 2008. “Designing Proceedings of the Annual Conference on Com- and Evaluating Virtual Musical Instruments: Facilitat- puter Graphics and Interactive Techniques, pp. 135– ing Conversational User Interaction.” Design Studies 142. 29(6):556–571. Davis, S., K. Nesbitt, and E. Nalivaiko. 2014. Jorda,` S. 2003. “Interactive Music Systems for Everyone: “A Systematic Review of Cybersickness.” In Exploring Visual Feedback as a Way for Creating Proceedings of the Conference on Interactive More Intuitive, Efficient, and Learnable Instruments.” Entertainment, pp. 1–9. Available online at In Proceedings of the Stockholm Music Acoustics dl.acm.org/citation.cfm?id=2677780. Accessed August Conference, pp. 681–684. 2016, subscription required. Jorda,` S. 2004. “Instruments and Players: Some Thoughts Erkut, C., A. Jylha,¨ and R. Discioglu. 2011. “A Structured on Digital Lutherie.” Journal of New Music Research Design and Evaluation Model with Application to 33(3):321–341. Rhythmic Interaction Displays.” In Proceedings of Jylha,¨ A., et al. 2011. “Design and Evaluation of Rhythmic the International Conference on New Interfaces for Interaction with an Interactive Tutoring System.” Musical Expression, pp. 477–480. Computer Music Journal 35(2):36–48. Fernandes, A. S., and S. K. Feiner. 2016. “Combating VR Karjalainen, M., and T. Maki-Patola.¨ 2004. “Physics- Sickness through Subtle Dynamic Field-of-View Modi- Based Modeling of Musical Instruments for Interactive fication.” In IEEE Symposium on 3D User Interfaces, Virtual Reality.” In IEEE Workshop on Multimedia pp. 201–210. Signal Processing, pp. 223–226. Fillwalk, J. 2015. “ChromaChord: A Virtual Musical In- Karjalainen, M., et al. 2006. “Virtual Air Guitar.” Journal strument.” In IEEE Symposium on 3D User Interfaces, of the Audio Engineering Society 54(10):964–980. pp. 201–202. Kilteni, Konstantina, Ilias Bergstrom, and Mel Slater. Ford, C. 2010. “Musical Presence: Towards a New Philos- 2013. “Drumming in Immersive Virtual Reality: The ophy of Music.” Contemporary Aesthetics 8. Available Body Shapes the Way We Play.” IEEE Transactions on online at hdl.handle.net/2027/spo.7523862.0008.020. Visualization and Computer Graphics 19(4):597–605. Accessed August 2016. Kohlrausch, A., and S. van de Par. 1999. “Auditory-Visual Gallagher, S. 2000. “Philosophical Conceptions of the Interaction: From Fundamental Research in Cognitive Self: Implications for Cognitive Science.” Trends in Psychology to (Possible) Applications.” In International Cognitive Sciences 4(1):14–21. Society for Optics and Photonics (SPIE) Proceedings Gelineck, S., et al. 2005. “Virtual Reality Instru- on Human Vision and Electronic Imaging IV, pp. 34– ments Capable of Changing Dimensions in Real- 44. Available online at dx.doi.org/10.1117/12.348440. Time.” In Proceedings of the International Con- Accessed August 2016, subscription required. ference on Enactive Interfaces. Available online at Kojs, J. 2013. “Virtual Reality and Sonic Cyberspaces: www.researchgate.net/publication/255634646. Ac- Augmentation, Hybridisation and Abstraction.” Inter- cessed August 2016. national Journal of Arts and Technology 6(2):167–180.

38 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Kojs, J., S. Serafin, and C. Chafe. 2007. “Cyberinstruments Simon Fraser University, School of Kinesiology, Van- via Physical Modeling Synthesis: Compositional couver, British Columbia. Applications.” Leonardo Music Journal 17:61–66. Nowak, K. L., and F. Biocca. 2003. “The Effect of the LaViola, J. J. 2000. “A Discussion of Cybersickness Agency and Anthropomorphism on Users’ Sense of in Virtual Environments.” ACM SIGCHI Bulletin , Copresence, and Social Presence in 32(1):47–56. Virtual Environments.” Presence 12(5):481–494. Leonard, J., et al. 2013. “A Virtual Reality Platform for O’Modhrain, S. 2001. “Playing by Feel: Incorporating Musical Creation: GENESIS-RT.” In Proceedings of the Haptic Feedback into Computer-Based Musical In- 2013 International Symposium on Computer Music struments.” PhD dissertation, Stanford University, Multidisciplinary Research, pp. 346–371. Department of Music. Levin, G. 2000. “Painterly Interfaces for Audiovisual Per- O’Modhrain, S. 2011. “A Framework for the Evaluation of formance.” PhD dissertation, Massachusetts Institute Digital Musical Instruments.” Computer Music Journal of Technology, Media Arts and Sciences, Cambridge, 35(1):28–42. Massachusetts. O’Modhrain, S., et al. 2000a. “Incorporating Haptic Levy, P. 1998. Becoming Virtual: Reality in the Digital Feedback into Interfaces for Music Applications.” In Age. Cambridge, Massachusetts: Da Capo. Proceedings of the World Automation Congress, (pages Lombard, M., and T. Ditton. 1997. “At the Heart unnumbered). of It All: The Concept of Presence.” Journal of O’Modhrain, S., et al. 2000b. “Qualitative and Quantitive Computer-Mediated Communication 3(2). Available Assessment of a Virtual Bowed String Instrument.” online at onlinelibrary.wiley.com/doi/10.1111/j.1083 In Proceedings of the International Computer Music -6101.1997.tb00072.x/full. Accessed August 2016. Conference, pp. 66–69. Maki-Patola, T., and P. Ham¨ al¨ ainen.¨ 2004. “Effect of La- Paine, G. 2009. “Towards Unified Design Guidelines for tency on Playing Accuracy of Two Gesture Controlled New Interfaces for Musical Expression.” Organised Continuous Sound Instruments without Tactile Feed- Sound 14(02):142–155. back.” In Proceedings of the International Conference Poupyrev, I., et al. 1996. “The Go-Go Interaction Tech- on Digital Audio Effects, pp. 11–16. nique: Non-Linear Mapping for Direct Manipulation Maki-Patola,¨ T., et al. 2005. “Experiments with Virtual Re- in VR.” In Proceedings of the ACM Symposium on ality Instruments.” In Proceedings of the International User Interface Software and Technology, pp. 79– Conference on New Interfaces for Musical Expression, 80. pp. 11–16. Preece, J., et al. 1994. “Interview with Ben Shnei- Marshall, M. T., and M. M. Wanderley. 2011. “Examining derman.” Available online at www.id-book.com the Effects of Embedded Vibrotactile Feedback on the /downloads/BenShneiderman.pdf. Accessed Septem- Feel of a Digital Musical Instrument.” In Proceedings ber 2016. of the International Conference on New Interfaces for Riccio, G. E., and T. A. Stoffregen. 1991. “An Ecological Musical Expression, pp. 399–404. Theory of Motion Sickness and Postural Instability.” Meehan, M. 2001. “Physiological Reaction as an Objective Ecological Psychology 3(3):195–240. Measure of Presence in Virtual Environments.” PhD Rodet, X., et al. 2005. “Study of Haptic and Visual dissertation, University of North Carolina at Chapel Interaction for Sound and Music Control in the Phase Hill, Department of Computer Science. Project.” In Proceedings of the International Conference Moore, A. G., et al. 2015. “Wedge: A Musical Interface on New Interfaces for Musical Expression, pp. 109– for Building and Playing Composition-Appropriate 114. Immersive Environments.” In Proceedings of the Russolo, L. [1913] 1967. The Art of Noise: Futurist IEEE Symposium on 3D User Interfaces, pp. 205– Manifesto, 1913. R. Filliou, trans. New York: Something 206. Else. Morreale, F., A. De Angeli, and S. O’Modhrain. 2014. Serafin, S., et al. 2006. “Synthesis and Control of Everyday “Musical Interface Design: An Experience-Oriented Sounds Reconstructing Russolo’s Intonarumori.” In Framework.” In Proceedings of the International Proceedings of the International Conference on New Conference on New Interfaces for Musical Expression, Interfaces for Musical Expression, pp. 240–245. pp. 467–472. Serafin, S., et al. 2015. “The Digital Intonarumori.” In Mulder, A. G. 1998. “Design of Virtual Three-Dimensional IEEE Symposium on 3D User Interfaces, pp. 207– Instruments for Sound Control.” PhD dissertation, 208.

Serafin et al. 39

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021 Slater, M. 2009. “Place Illusion and Plausibility Can Wanderley, M. M., and N. Orio. 2002. “Evaluation of Input Lead to Realistic Behaviour in Immersive Virtual Devices for Musical Expression: Borrowing Tools from Environments.” Philosophical Transactions of the HCI.” Computer Music Journal 26(3):62–76. Royal Society B: Biological Sciences 364(1535):3549– Wang, G. 2014. “Principles of Visual Design for Computer 3557. Music.” In Proceedings of the Joint International Slater, M., D.-P. Pertaub, and A. Steed. 1999. “Public Computer Music Conference and the Sound and Music Speaking in Virtual Reality: Facing an Audience of Computing Conference, pp. 391–396. Avatars.” IEEE Computer Graphics and Applications Wessel, D., and M. Wright. 2002. “Problems and Prospects 19(2):6–9. for Intimate Musical Control of Computers.” Computer Smith, J. O. 2010. Physical Audio Signal Processing for Music Journal 26(3):11–22. Virtual Musical Instruments and Audio Effects.Palo Xiao, X., B. Tome, and H. Ishii. 2014. “Andante: Walking Alto, California: W3K Publishing. Figures on the Piano Keyboard to Visualize Musical Mo- Sodhi, R., et al. 2013. “AIREAL: Interactive Tactile tion.” In Proceedings of the International Conference Experiences in Free Air.” ACM Transactions on on New Interfaces for Musical Expression, pp. 629– Graphics 32(4):134. 632. Sutherland, I. E. 1965. “The Ultimate Display.” In Infor- Zhai, P. 1998. Get Real: A Philosophical Adventure mation Processing: Proceedings of the IFIP Congress, in Virtual Reality. Lanham, Maryland: Rowman and pp. 506–508. Littlefield. Valim¨ aki,¨ V., and T. Takala. 1996. “Virtual Musical Zielasko, D., et al. 2015. “Cirque des Bouteilles: The Art Instruments: Natural Sound Using Physical Models.” of Blowing on Bottles.” In IEEE Symposium on 3D User Organised Sound 1(02):75–86. Interfaces, pp. 209–210.

40 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00372 by guest on 02 October 2021