vltn trntd brd Prfrn n, r l Computer Music Journal, Volume 38, Number 4, Winter 2014, pp. 68-79 (Article) Pblhd b Th T Pr For additional information about this article http://muse.jhu.edu/journals/cmj/summary/v038/38.4.yang.html Access provided by University of Michigan @ Ann Arbor (20 Jan 2015 06:39 GMT) Qi Yang and Georg Essl Electrical Engineering and Computer Science Evaluating University of Michigan 2260 Hayward Ave Gesture-Augmented Ann Arbor, Michigan 48109-2121, USA yangqi, gessl @umich.edu Keyboard Performance { } Abstract: The technology of depth cameras has made designing gesture-based augmentation for existing instruments inexpensive. We explored the use of this technology to augment keyboard performance with 3-D continuous gesture controls. In a user study, we compared the control of one or two continuous parameters using gestures versus the traditional control using pitch and modulation wheels. We found that the choice of mapping depends on the choice of synthesis parameter in use, and that the gesture control under suitable mappings can outperform pitch-wheel performance when two parameters are controlled simultaneously. In this article we discuss the evaluation of a the number of strings and by the geometry of the musical-keyboard interface augmented with free- hand that provides the fingering. Also, the initial hand gestures. Keyboards are musically expressive activation and reactivation of notes on a keyboard and are well suited for performance of discrete notes. does not require preparation like stopping strings or Smooth adjustments of performance parameters that activating multiple valves on a wind instrument. are important for digital synthesizers or samplers Despite the ease of keyboard playing, it does are difficult to achieved, however. Since the 1970s, come with drawbacks. After the onset of each note, such adjustments have often been achieved using the player has limited control of the quality of pitch and modulation wheels at the left side of the sound. This is in contrast to bowed or wind the keyboard. Contemporary sensor technology instruments, which have a range of expressive now makes it increasingly easy to offer alternative timbre controls after the onset of each note. In the means to track continuous input. We augmented case of the traditional piano, limited timbre controls the musical keyboard with a 3-D gesture space are provided by pedals to control the damping of the using the Microsoft Kinect, an infrared-based depth strings and, therefore, the amount of sympathetic camera for sensing and top–down projection for resonance between strings. visual feedback. This interface provides 3-D gesture The pipe organ does offer means of timbre control controls to enable continuous adjustments to through knobs or tabs, commonly referred to as multiple acoustic parameters, such as those found organ stops. The player pushes or pulls on the stops on typical digital synthesizers. Using this system, to activate or mute different sets of pipes, changing we conducted a user study to establish the relative the timbre of the sound produced by actuating the merits of free-hand gesture motion versus traditional keys. Pipe organs have developed a wide range of continuous controls. timbres that are enabled by different combinations of pipes, but the physical interface has seen little change, as the the stops are not designed for timbre Keyboard as Interface changes while keys are being held down (more recent pipe organs allow configurations to be saved It is easy to understand the popularity of the piano- in advance and loaded during the performance), style musical keyboard. A keyboard enables the while the crescendo and swell foot pedals provide player to address multiple discrete pitches concur- limited continuous timbre controls. Continuous rently and directly. In contrast, wind instruments control via a pedal is an interesting possibility, but produce a single pitch at a time and require complex we will not be considering it here. chorded fingering. Further, in string instruments Digital synthesizers, sampler instruments, and such as violin or guitar, polyphony is limited by MIDI controllers usually feature a keyboard for pitch selection and note activation. For parameter adjust- Computer Music Journal, 38:4, pp. 68–79, Winter 2014 ment during live performance, they traditionally doi:10.1162/COMJ a 00277 feature one or two wheels (or in some cases, joy- c ⃝ 2014 Massachusetts Institute of Technology. sticks) next to the keyboard to control modulation 68 Computer Music Journal and/or pitch bend. We wanted to see if open-air hand digital synthesizers. Still, continuous control in gestures provide better means of adjustment during keyboard performance remained an important topic. live performance. Moog, later with collaborators Tom Rhea and John It is easy to perform continuous gestures using Eaton, experimented for decades with prototypes to hand motions in space, hence they make a good add continuous control to the surface of the keys candidate for continuous timbre control in real time, themselves (Moog 1982; Moog and Rhea 1990; Eaton especially in improvised music. When performing and Moog 2005). This idea has also been explored by on the keyboard, the player can quickly lift their others (Haken and Tellman 1998; McPherson and hand from the keyboard and move into and out of the Kim 2010; Lamb and Robertson 2011; McPherson gesture space. Recent advances in sensor technology 2012). make gesture sensing easy and affordable. Our Another idea that has been proposed is the prototype system uses an off-the-shelf depth camera augmentation of the action of the key itself. The to track a range of hand motions, positions, and classic aftertouch, where extra levels of control gestures in real time, making it suitable for live are available once the keys are fully depressed, is an performance and the goals of our project. The early example of this (Paradiso 1997). Precise sensing sensing of position and hand width creates a space of key position can be achieved through various with multiple, continuous degrees of freedom, means, such as optical interruption sensing (Freed allowing multiple parameters to be controlled and Avizienis 2000). More recently, McPherson simultaneously. The gesture space also allows either and Kim (2011) described the augmentation of hand to be used for hand-gesture controls, in contrast traditional piano keys through a light-emitting diode to the fixed location of pitch and modulation wheels (LED) sensing mechanism that is capable of inferring on the far left side of a standard MIDI keyboard. performance parameters from the key action. This, in turn, can be used to augment performance. More narrowly, our work augments the musical- Related Work keyboard interface with continuous hand gesture control in open space. Perhaps the most famous This article brings together two important strands previous example of open gesture control is the in the design of new musical instruments: the Theremin, which uses capacitive sensing. Open- augmentation of established, traditional musical space gestures can be tracked using different tech- instruments, and the use of gestures for continuous nologies. Our prototype uses visual sensing via control of musical instruments. The prior art in both depth cameras. Visual tracking of hands has been these fields is extensive, and we refer the reader to explored previously (Gorodnichy and Yogeswaran comprehensive reviews (Paradiso 1997; Miranda and 2006; Takegawa, Terada, and Tsukamoto 2011). Wanderley 2006). Concurrently to our work, Aristotelis Hadjakos How best to support continuous control in con- (2012) used the Kinect for hand, arm, and posture junction with the keyboard interface is a longstand- detection in piano performance. The key differences ing problem and has seen many proposals. When between his work and ours is that we consider the designing the first hard-wired commercial analog visual tracking for generic gesture interactions that synthesizers, Bill Hemsath, in collaboration with augment piano performance, whereas Hadjakos is Bob Moog and Don Pakkala, invented the pitch and interested in sensing for medical and pedagogical modulation wheels (Pinch and Trocco 2004), which purposes. Hence the system does not include visual became the canonical forms of continuous control feedback. Visual feedback did appear in work by on electronic keyboard interfaces ever since. Early Takegawa, Terada, and Tsukamoto (2011), who pro- analog synthesizers had many continuous controls jected score and fingering information to guide early via rotary potentiometers and sliders, but in many piano pedagogy. William Brent (2012) presented a canonical cases the pitch and modulation wheels visual tracking system based on infrared blob detec- were the only ones that survived the transition to tion. In that work, an ordinary camera is suspended Yang and Essl 69 Figure 1. Configuration of the augmented keyboard. above a piano together with an array of infrared lights. The depth information is then inferred from the size of the blob. The purpose of that work was to detect central parts of the performer to allow extra control parameters to be derived from the po- sition of the hand center relative to the lower arm. The author reports problems with independence of the control parameters thus detected. Our system avoids this problem by directly sensing position, in a 3-D volume, of the field of view using a depth camera. In addition, literature exists on evaluation methodologies for