Advancements in Actuated Musical Instruments DAN OVERHOLT*, EDGAR BERDAHLy andROBERT HAMILTONz *Aalborg University, Department of Architecture, Design and Media Technology, Niels Jernes Vej 14, DK-9220 Aalborg, Denmark E-mail: [email protected] y,zStanford University, Department of Music, Center for Computer Research in Music and Acoustics, Stanford, California 94305, USA E-mail: [email protected]; [email protected] This article presents recent developments in actuated musical electronic (transducer) actuation, we feel that these instruments created by the authors, who also describe an instruments all share the need for an ecosystemic ecosystemic model of actuated performance activities that approach to performance technique and interaction. blur traditional boundaries between the physical and virtual Actuated instruments invite and even require the elements of musical interfaces. Actuated musical instruments performer to engage with programmed models of are physical instruments that have been endowed with virtual musical interaction that involve one or more layers of qualities controlled by a computer in real-time but which are nevertheless tangible. These instruments provide intuitive interactive feedback systems. We seek to incorporate and engaging new forms of interaction. They are different actuation into new musical instrument designs in from traditional (acoustic) and fully automated (robotic) order to explore alternative methods of leveraging a instruments in that they produce sound via vibrating performer’s (slower) consciously controlled gestures element(s) that are co-manipulated by humans and and (faster) pre-learned gestures, by sharing some of electromechanical systems. We examine the possibilities the control with the machine while still providing that arise when such instruments are played in different proactive and intimate human control of the resulting performative environments and music-making scenarios, and musical sounds. In our experience, effectively employed we postulate that such designs may give rise to new methods actuation can of musical performance. The Haptic Drum, the Feedback Resonance Guitar, the Electromagnetically Prepared Piano, > free up some human cognitive bandwidth in order to the Overtone Fiddle and Teleoperation with Robothands are promote concentration on other facets of musical described, along with musical examples and reflections on the endeavour; emergent properties of the performance ecologies that these > promote combined human and technological cap- instruments enable. We look at some of the conceptual and abilities that can enable and even compel the per- perceptual issues introduced by actuated musical instruments, former to interact in new ways via systemic feedback; and finally we propose some directions in which such research for example, an actuated musical instrument can may be headed in the future. enable a performer to make gestures that would otherwise be difficult or impossible; > allow the performer to physically attend to other 1. INTRODUCTION aspects of playing; instead of always having to In this article we discuss recent developments of actu- inject energy into an instrument, the performer ated musical instruments and the implications they can steer external sources of (usually) electrical may hold for the performative ecosystem surrounding energy by controlling when and how they are the instruments themselves, the performer and the applied; > environment in which they are used. Figure 1 repre- combine musical gestures from multiple musical sents some of the authors’ developments, in which performers into a networked ecosystem of co- electronic signal processing techniques are applied to controlled instruments, allowing performers to actuators physically embedded into the instruments directly control or affect the behaviours and themselves. For example, instead of using audio com- capabilities of other performers; and > pression algorithms to simulate the prolonged sustain endow a physical instrument with virtual qualities of a plucked string, actuation of the strings and/or that are adjustable by computer in real-time but body of the instrument enables physically palpable which are nevertheless tangible, facilitating more real-world behaviours (e.g., with strings that are intuitive and possibly even intimate interactions. magnetically actuated even infinite sustain is possi- Moreover, there exists a continuum of possible ble). All of these instruments are unique – each is instrument designs, between real (non-augmented, imbued with its own form of actuation. However, due always requiring human energy input) and virtual to their tight integration of human (gestural) and (augmented with external energy sources, possibly Organised Sound 16(2): 154–165 & Cambridge University Press, 2011. doi:10.1017/S1355771811000100 Advancements in Actuated Musical Instruments 155 Visual BRAIN Auditory Position, Velocity, or Related Musical Instrument Haptic Figure 1. Clockwise from top left: the Electromagnetically Figure 2. Musician interacting with a musical instrument Prepared Piano, the Haptic Drum, the Feedback Reso- providing auditory, visual and haptic force feedback. nance Guitar, a teleoperated acoustic shaker held by the Falcon force-feedback device, and the Overtone Fiddle. instrument. For this reason, we believe that haptic even fully automated) instruments. We define actu- feedback is essential for a performer who wishes to ated musical instruments as those which produce sound quickly and intimately interact with an ecosystemic via vibrating element(s) that are co-manipulated by musical instrument. humans and electromechanical systems. These instru- Without yet considering environmental or audi- ments include the capability for control to be exerted ence intervention, Figure 2 attempts to represent all by a simple or complex system of external agency the necessary parts to describe a musician’s perfor- in addition to existent methods of control through mance ecosystem (Berdahl 2009: 2). We include the traditional instrumental performance technique with brain as the primary link in the composite feedback the instrument. system where the visual, auditory and haptic feed- back modalities are combined. The brain and the hand are internally interconnected by the human 1.1. Model: performance ecosystems motor-system control loop. In order to understand Musicians have been playing musical instruments for the complex behaviours enabled by the entire system, thousands of years. In most live musical performance all of the parts must be considered. In this sense, the paradigms, a musician employs the natural feedback model is an ecological model. In a well-designed provided by an instrument to help him or her control system, it can be shown that the whole can become a performance. For example, consider the model more than the sum of its parts, leading to emergent shown in Figure 2 describing a musician playing a behaviours. However, computer musicians are well musical instrument. The model is adapted from Bill aware that achieving this wholeness can be challen- Verplank’s ‘Interaction Design Sketchbook’ (Ver- ging (Waters 2007: 1–20). Nevertheless, based on our plank 2003), and we assume here that the musician experience we believe that actuated musical systems uses his or her hand to provide mechanical excitation can exhibit emergence if they leverage the concepts of to the instrument. We acknowledge that a wide range tangibility, energy steering, enabling new kinds of of other excitations and/or modifications could of performance interactions and freeing up cognitive course be provided using the mouth, the remainder of bandwidth as outlined in Section 1. In addition to the body, and so on, but we do not explicitly depict these discoveries, we hypothesise that composers can them here for the sake of brevity. obtain a whole that is larger than the sum of the parts While it is clear that musicians rely heavily on either by trial and error, or by considering which auditory feedback for ecosystemic performance, as elements of the system the performer controls con- discussed by Di Scipio (Di Scipio 2003), visual and sciously or unconsciously. haptic feedback also play an important role. For Moving beyond traditional musical performance example, musicians can employ visual and haptic ecologies, wherein performers interact with one feedback to help orient their hands in relation to an another through anticipated or improvised musical interface. Performers may wish to use their sense of gesture, there exists a novel ecosystemic model for vision for looking at a score or looking at the audi- musical performance with actuated instruments in ence; in contrast, haptic feedback provides localised which physical control of musical instruments can be information at any point on the human body. Fur- shared between musical performers, based upon thermore, in comparison with other feedback mod- intentional or derived musical gesture or analysis. alities, haptic feedback enables the performer to Actuated instruments capable of external physical respond the fastest to the behaviour of the musical excitation allow musicians to interact at an entirely 156 Dan Overholt, Edgar Berdahl and Robert Hamilton new level, both with one another as well as with rule- virtual masses (Besnainou 1999). More recent work based or reactive computer systems. By linking musical has been carried out in Paris on controlling a xylo- instruments to one another, routing musical output phone bar