Bowing the violin A case study for auditory-motor pattern modelling in the context of music performance Quim Llimona Torras Sonologia Enric Guaus Térmens 2013/2014 ABSTRACT This project addresses methodological and technological challenges in the develop- ment of multi-modal data acquisition and analysis methods for the representation of instrumental playing technique in music performance through auditory-motor patterning models. The case study is violin playing: a multi-modal database of violin performances has been constructed by recording different musicians while playing short exercises on different violins. The exercise set and recording protocol have been designed to sample the space defined by dynamics (from piano to forte) and tone (from sul tasto to sul ponticello), for each bow stroke type being played on each of the four strings (three different pitches per string) at two different tempi. The data, containing audio, video, and motion capture streams, has been processed and segmented to facilitate upcoming analyses. From the acquired motion data, the positions of the instrument string ends and the bow hair ribbon ends are tracked and processed to obtain a number of bowing descriptors suited for a detailed de- scription and analysis of the bow motion patterns taking place during performance. Likewise, a number of sound perceptual attributes are computed from the audio streams. Besides the methodology and the implementation of a number of data acquisition tools, this project introduces preliminary results from analyzing bowing technique on a multi-modal violin performance database that is unique in its class. A further contribution of this project is the data itself, which will be made available to the scientific community through the repovizz platform. iii ABSTRACT (CATALA)` Aquest projecte adrec¸ a reptes metodologics` i tecnics` en el desenvolupamnt de metodes` d’adquisio´ i analisi` de dades multi-models per la representacio´ de tecniques` instrumentals en la interpretacio´ musical mitjanc¸ ant models de patrons auditori- motors. El cas d’estudi es´ la interpretacio´ del viol´ı: s’ha constru¨ıt una base de dades multi-modal amb musics´ tocant diferents exercicis al viol´ı enregistrant-los amb diferents instruments. El conjunt d’exercicis i el protocol d’enregistrament s’han dissenyat per mostrejar l’espai definit per la dinamica` (de piano a forte) i to (de sul tasto a sul ponticello), per cada tipus d’arcada tocada en cadascuna de les quatre corda (i tres notes diferents per corda) a dos tempi diferents. Les dades, que contenen fonts d’audio,` v´ıdeo i captura de moviment, s’han processat i segmentat per facilitar analisis` posteriors. A partir de les dades de moviment adquirides, la posicio´ dels l´ımits de les cordes de l’instrument i de les cerdes de l’arc es segueixen i processen per tal d’obtenir un nombre de descriptors de l’arc, que permeten fer una descripcio´ i analisis` detallada dels patrons de moviment de l’arc durant l’acte interpretatiu. De manera similar, es calculen uns quants atributs perceptuals a partir de les fonts d’audio.` A part de la metodologia i implementacio´ d’un conjunt d’eines per a la adquisicio´ de dades, aquest projecte introdueix resultats preliminars fruit de l’analisi` de la tecnica` de l’arc en una base de dades multi-modal amb interpretacions de viol´ı unica´ en la seva especie.` Una altra contribucio´ del projecte son´ les dades en si mateixes, que es posaran a la disponibilitat de la comunitat cient´ıfica mitjanc¸ant la plataforma repovizz. v ABSTRACT (ESPANOL)˜ En este proyecto se tratan retos metodologicos´ y tecnicos´ en el desarrollo de metodos´ de adquisicion´ y analisis´ de datos multi-modales para la representacion´ de tecnicas´ instrumentales en la interpretacion´ musical mediante modelos de patrones auditorio-motoros. El caso de estudio es la interpretacion´ del viol´ın: se ha constru- ido una base de datos multi-modal con musicos´ tocando diferentes ejercicios al viol´ın grabados con distintos instrumentos. El conjunto de ejercicios y el protocolo de grabacion´ han sido disenados˜ para muestrear el espacio definido por la dinamica´ (de piano a forte) y tono (de sul tasto a sul pointicello), por cada tipo de arcada tocada en cada una de las cuatro cuerdas (y con tres notas distintas por cuerda) a dos tempi distintos. Los datos, que contienen fuentes de audio, v´ıdeo y captura de movimiento, se han procesado y segmentado para facilitar posteriores analisis.´ A partir de los datos de movimiento adquiridos, se siguen la posicion´ de los l´ımites de las cuerdas del instrumento y de las cerdas del arco para procesarlas y obtener unos descriptores del arco, que permiten hacer una descripcion´ y analisis´ detallado de los patrones de movimiento del arco durante la interpretacion.´ De forma similar, se calculan atributos perceptuales a partir de las fuentes de audio. A parte de la metodolog´ıa e implementacion´ de un conjunto de herramientas para la acquisicion´ de datos, este proyecto introduce resutlados preliminares fruto del analisis´ de la tecnica´ de arco en una base de datos multi-modal sobre la interpretacion´ del viol´ın unica´ en su especie. Otra contribucion´ del proyecto son los datos en si mismos, que se publicaran para su uso en la comunidad cient´ıfica mediante la plataforma repovizz. vii Contents ABSTRACT iii ABSTRACT (CATALA)` v ABSTRACT (ESPANOL)˜ vii Index of figures xi Index of tables xiii 1 INTRODUCTION 1 1.1 Context . 1 1.1.1 The author . 1 1.1.2 The MUSMAP project . 2 1.2 Motivation . 3 1.3 Background: the violin . 6 1.3.1 Basics of the the bowed string motion . 8 1.3.2 Bowing control parameters . 9 1.3.3 Early studies on bowing control in performance . 11 1.3.4 Acquisition of bowing parameters in violin performance . 11 1.4 Objectives . 13 1.5 Structure of the thesis . 14 2 EXPERIMENTAL DESIGN 17 2.1 Sampling space . 18 2.1.1 Performer . 19 2.1.2 Instrument . 19 ix 2.1.3 Articulation . 22 2.1.4 Sounding point . 22 2.1.5 Dynamics . 23 2.1.6 Pitch (string and string length) . 23 2.1.7 Bow direction . 23 2.2 Sample permutations (score) . 24 3 DATA ACQUISITION 27 3.1 Overview . 27 3.2 Motion Capture . 27 3.2.1 Body markers . 28 3.2.2 Violin markers . 29 3.2.3 Bow markers . 31 3.2.4 Load cell markers . 33 3.3 Audio . 34 3.4 Video . 35 3.5 Load cell . 36 3.6 Synchronization . 37 3.7 Recording protocol . 38 4 FEATURE EXTRACTION 39 4.1 Overview . 40 4.1.1 High-level features . 40 4.1.2 Low-level features . 41 4.2 Computation of low-level descriptors . 41 4.2.1 Vector basis . 41 4.2.2 Bow to string distances . 44 4.2.3 Bow deformation . 45 4.3 Computation of high-level features . 45 4.3.1 Noise reduction . 47 4.4 Force estimation . 47 4.4.1 Load cell calibration . 48 4.4.2 Regression model . 49 4.4.3 Evaluation of the force estimation process . 53 4.5 Audio features . 54 x 4.5.1 Pitch . 55 4.5.2 Aperiodicity . 55 4.5.3 Energy . 55 5 DATABASE 57 5.1 Score-performance alignment . 57 5.1.1 Zero-crossing finder . 58 5.1.2 Graphical User Interface and program flow . 58 5.2 Annotations . 60 5.2.1 Look-up tables . 61 5.3 The repovizz platform . 62 6 PRELIMINARY DATA ANALYSIS 65 6.1 Introduction . 65 6.1.1 Player selection . 67 6.2 Bowing technique . 67 6.3 Dynamics . 70 6.4 Tone . 73 6.5 Duration . 76 6.6 Player . 78 7 CONCLUSION 81 7.1 Achievements . 81 7.2 Future work . 82 7.3 Acknowledgments . 82 Bibliography 85 Appendices 87 A MUSICAL SCORES 91 B QUESTIONNAIRE 109 xi List of Figures 1.1 From an instrumental gesture perspective, musical score, instru- mental gestures, and produced sound represent the three most accessible entities for providing valuable information on the music performance process. 5 1.2 Music versus dance notation . 6 1.3 Parts of the violin . 6 2.1 Admittance and radiation plots . 20 2.2 Excerpt of the score that was given to the musicians . 25 3.1 Violins used during the recordings, ordered from left to right. 30 3.2 Bow that was used for the recordings. 32 3.3 Pickup and close-up microphone mounted on the violin. 35 3.4 Load cell mounted on a support with motion tracking markers. 36 3.5 Calibrated weights that were used for characterizing the load cell. 37 4.1 Block diagram of the feature extraction process . 39 4.2 Illustration of the violin and bow planes, the sounding points and some of the basis vectors . 42 4.3 Illustration of the hair ribbon and string deflection, together with the shortest segment joining the sounding points . 42 4.4 Plot of the force measured during the load cell characterization. 48 4.5 Fitted loadcell polynomial together with the measured samples . 49 4.6 Bow force regression example . 50 4.7 Histograms of the loadcell recordings . 52 5.1 Output of the automatic segmentation software. 59 5.2 Screenshot of the repovizz visualizer displaying a datapack with one of the takes from MUSMAP I . 63 xiii 6.1 Bow-bridge distance vs bow force for Players 1, 2 and 3 . 67 6.2 Scatter plots for legato and martele bow strokes . 68 6.3 Bow velocity and force temporal profiles for legato and martele bow strokes . 69 6.4 Audio energy and aperiodicity temporal profiles for legato and martele bow strokes .