Proceedings of the 2008 Conference on New Interfaces for Musical Expression (NIME08), Genova, Italy Enhancing the visualization of percussion gestures by virtual character animation Alexandre Bouenard¨ Sylvie Gibet∗ Marcelo M. Wanderley∗ Samsara / VALORIA Bunraku / IRISA IDMIL / CIRMMT Univ. Europeenne´ Bretagne Univ. Europeenne´ Bretagne McGill University Vannes, France Rennes, France Montreal, Qc, Canada [email protected] [email protected] [email protected] ABSTRACT is composed of different views of both the virtual character A new interface for visualizing and analyzing percussion ges- and the instrument. It is finally enhanced with interactions tures is presented, proposing enhancements of existing mo- between graphics modeling, physics synthesis of gesture and tion capture analysis tools. This is achieved by offering a sound replay. percussion gesture analysis protocol using motion capture. The paper is organized as follows. In section 2, previous A virtual character dynamic model is then designed in or- work and motivations are discussed. The analysis process der to take advantage of gesture characteristics, yielding to of percussion (timpani) gestures is detailed in section 3. Vi- improve gesture analysis with visualization and interaction sualization and interaction concerns are discussed in section cues of different types. 4. Finally, we conclude with further perspectives. Keywords 2. RELATED WORK Gesture and sound, interface, percussion gesture, virtual Previous works concern both percussion-related models character, interaction. and interfaces, and works combining virtual character ani- mation and music. Most of the work about percussion gesture and sound 1. INTRODUCTION deals with the design of new electronic percussion devices, Designing new musical interfaces is one of the most impor- thus creating either new interfaces (controllers) and/or new tant trends of the past decades. Efforts have constantly been sound synthesis models and algorithms. made to elaborate more and more efficient devices in order On the one hand, new interfaces are based on increasingly to capture instrumental gestures. These technical advances efficient devices that are able to track gestures. Electronic have given rise to novel interaction opportunities between percussions such as Radio Baton [1], Buchla Lightning [3], digital instruments and performers, and the creation of new Korg Wavedrum [20] and ETabla [14] are digital musical sound, image or tactile synthesis processes. Our main guide- instruments that are improving or emulating acoustic phe- line aims at providing a set of pedagogical tools for helping nomena, by taking into account gesture cues such as posi- the study of percussion gestures. Among these, rendering tion, touch and pressure. More recent work take advantage real instrumental situations (interaction between performers of various techniques, such as magnetic gesture tracking [17], and instruments) and exploring the gestural space (and its computer vision [16] or the physical modeling of the drum corresponding visual, gestural and sounding effects) are of skin [13]. great interest. Eventually, our final goal is to build new vir- On the other hand, it is also achieved by designing sound tual instrumental situations, especially with gesture-sound synthesis models and algorithms, ranging from purely signal- interactions controlled by virtual characters. This paper of- based to physically-based methods [9]. These works rarely fers a new tool for visualizing percussion gestures, which include the study of the instrumental gesture as a whole, exploits both the analysis and synthesis of percussion ges- especially regarding to its dynamical aspects or its playing tures. The analysis process is achieved by capturing the techniques, even if some take into account real measure- movements of performers, while a physical model of virtual ments [2] and physical parameters mapping with percussion character is designed for the synthesis. The visualization gesture [5]. Playing techniques can be qualitatively observed and used ([14] [12] [8]) for a better understanding of percus- ∗Also with Samsara / VALORIA, Universit´e Europ´eenne de Bretagne (UEB), Vannes, France sive gestures. They can also be quantified thanks to capturing tech- niques [24], among which the most used is motion capture by camera tracking. But whichever method is used to re- Permission to make digital or hard copies of all or part of this work for produce the quality of the instrumental gesture, it generally personal or classroom use is granted without fee provided that copies are fails to convey its dynamic aspect. That is why we explore not made or distributed for profit or commercial advantage and that copies in this paper the possibility to physically animate a virtual bear this notice and the full citation on the first page. To copy otherwise, to character performing percussive gestures so that its intrinsic republish, to post on servers or to redistribute to lists, requires prior specific features are available to our interface. permission and/or a fee. NIME08 Genova, Italy As for previous work combining virtual character anima- Copyright 2008 Copyright remains with the author(s). tion and music, very few studies are available, especially 38 Proceedings of the 2008 Conference on New Interfaces for Musical Expression (NIME08), Genova, Italy in a mean of taking advantage of virtual character anima- tion for helping the visualization of gestures. The DIVA project1 used virtual character animation for audiovisual performances output driven by MIDI events [11]. Hints about motion capture characteristics towards the quality of re-synthesis of the movement [15] have been proposed. The influence of music performance on virtual character’s behavior [23] has also been emphasized. Some work aims at extracting expressive parameters from video data [4] for enhancing video analysis. Eventually, a solution consists in directly animating virtual models from the design of sound2. These studies are nevertheless out of the scope of virtual character animation as a gestural controller for enhancing the visualization and the analysis of instrumental situations. Figure 2: Left: French (top) and German (bottom) 3. TIMPANI PERFORMANCE grips; Right: Impact locations on the drumhead. There are many classifications of percussion instruments, one of the most established typologies is based on physical Players commonly use three distinct locations of impacts characteristics of instruments and the way by which they (Figure 2, right side). The most used is definitely the one- produce sound. According to this classification, timpani are third location, while the rim appears rather rarely. considered as membranophones, ”producing sound when the A database of timpani gestures has been created and is membrane or head is put into motion” [6]. composed of five gestures: legato, tenuto, accent, vertical accent and staccato. Each gesture is presented on Figure 3.1 Timpani Basics 3, showing the space occupation (Y-Z projection) of each Timpani related equipment is mainly composed of a bowl, drumstick’s trajectory, and highlighting the richness of tim- a head and drumsticks (Figure 1). In general, timpanists pani playing pattern variations. have to cope with several timpani (usually four) with bowls varying in size [19]. As for timpani drumsticks, they consist of a shaft and a head. They are designed in a wide range of lengths, weights, thicknesses and materials [6] and their choice is of great importance [18]. Figure 3: Timpani playing variations - Tip of the drumstick trajectories (Y-Z projection). Legato is the standard up-and-down timpani gesture. Tenuto Figure 1: Timpani player’s toolbox: bowl, head and and accent timpani variations show an increase in drumsticks. velocity and a decrease in space occupation (in the Y direction). Vertical accent and staccato timpani Timpani playing is characterized by a wide range of play- variations also show an increase in velocity, and are ing techniques. First, there are two main strategies for hold- characterized by an increase of space occupation (in ing drumsticks (Figure 2, left side): the ”French” grip (also the Y direction) for a more powerful attack and called ”thumbs-up”) and the ”German” grip (or ”matched” loudness. grip). Taking into account these various features, timpani ges- 1DIVA project : www.tml.tkk.fi/Research/DIVA tures are thus characterized by a wide variability. Next ses- 2Animusic : www.animusic.com sion will concern the quantitative capture of these variations. 39 Proceedings of the 2008 Conference on New Interfaces for Musical Expression (NIME08), Genova, Italy 3.2 Motion capture protocol and database of these gestures, the performer has been asked to change We propose to quantitatively characterize timpani ges- the location of the beat impact according to Figure 2 (right tures by capturing the motion of several timpani perform- side). Finally, our database is composed of fifteen examples ers. We use a camera tracking Vicon 460 system3 and a of timpani playing variations for each subject, and to each standard DV camera that allow both the retrieval of ges- example corresponds five beats per hand. This database ture and sound. will be used when studying in detail the variations of the The main difficulty using such hardware solutions is then timpani gesture. the choice of the sampling frequency for the analysis of per- The use of widespread analysis tools integrated in Vicon cussive gestures (because of the short duration of the