Studies on Customisation-Driven Digital Music Instruments with Strong Gesture to Sound Relationships
Total Page：16
File Type：pdf, Size：1020Kb
Studies on customisation-driven digital music instruments Bruno Zamborlin October 2014 Thesis jointly submitted to Goldsmiths, University of London and Universit´ePierre et Marie Curie - Paris 6, Ecole´ Doctorale EDITE for the degree of Doctor of Philosophy. Supervised by: Marco Gillies, Fr´ed´ericBevilacqua, Mark d'Inverno, G´erardAssayag Dedicated to Camila. Declaration This thesis is a presentation of my original research work. Whatever contri- butions of others are involved, every effort is made to indicate this clearly, with due reference to the literature, and acknowledgement of collaborative research and discussion. In my capacity as supervisor of the candidate's thesis, I certify that the above statements are true to the best of my knowledge. Date: ii Abstract From John Cage's Prepared Piano to the turntable, the history of mu- sical instruments is scattered with examples of musicians who deeply customised their instruments to fit personal artistic objectives, objec- tives that differed from the ones the instruments have been designed for. In their digital counterpart however, musical instruments are of- ten presented in the form of closed, finalised systems with a-priori symbolic rules set by their designer that leave very little room for the artists to customise the technologies for their unique art practices; in these cases the only possibility to change the mode of interaction with digital instrument is to reprogram them, a possibility available to programmers but not to musicians. This thesis presents two digital music instruments designed with the explicit goal of being highly customisable by musicians and to provide different modes of interactions, whist keeping simplicity and immedi- ateness of use. The first one leverages real-time gesture recognition to provide continuous feedback to users as guidance in defining the be- haviour of the system and the gestures it recognises. The second one is a novel tangible user interface which allows to transform everyday objects into expressive digital music instruments and whose sound gen- erated strongly depends by the particular nature of the physical object selected. R´esum´e Cette th`eseexplore un nouveau paradigme d'interaction pour le design d'instruments de musique num´erique,en considrant comme fonda- mentaux les processus de customisation. Plut^otque de demander aux musiciens d'apprendre des r`eglessp´ecifiquesimpos´eespar les instru- ments, j'explore des techniques d'adaptations pour faciliter la matrise des instruments par les musiciens, refl´etantleur style et devenant une partie int´egrante de leur production artistique. Deux nouveaux syst`emesnum´eriquessont r´ealis´esavec un objectif commun: faciliter le ph´enom`enede customisation dans le processus d'apprentissage. Le premier travail pr´esent´eest un syst`emede reconnaissance de geste qui guide les utilisateurs dans la d´efinitionde leurs propres vocabu- laires gestuels. Ce syst`emeleur permet de les utiliser pour contr^oler les m´ediasnum´eriquesen s'adaptant ´aleurs pratiques artistiques. Le deuxi`emetravail est un instrument mobile bas´esur des transducteurs pi´ezo´electriques qui permettent aux utilisateurs de transformer facile- ment des objets physiques quotidiens en des instruments de musique originaux. Ce syst`emeconvertit les vibrations cr´e´eeslorsque les util- isateurs touchent ou frappent les objets en son musical. Plus pr´ecis´ement, le syst`ememet en oeuvre des techniques de synth`eseinspir´esaux mod`elsphysiques qui d´ependent des propri´et´esacoustiques de l'objet lui-m^emeet de la fa¸condans la quelle il est touch´epar l'utilisateur. Cette approche ´ala musique `reality-based' est ´evalu´eepar une s´erie d'´etudesd'utilisation dans des situations concr`etesdu monde r´eel. `My topic is the shift from `architect' to `gardener', where `architect' stands for `someone who carries a full picture of the work before it is made', to `gardener' standing for `someone who plants seeds and waits to see exactly what will come up" - Brian Eno v Contents Contents vi List of Figures xii 1 Introduction1 1.1 Context . .2 1.2 Proposed methodology . .5 1.2.1 Case studies . .6 1.2.2 Structure of the work . .7 2 Literature Review 10 2.1 On embodiment and customisation of technologies . 11 2.1.1 Embodied interaction . 11 2.1.2 The art of transforming musical instruments . 14 2.1.3 Designing for appropriation . 18 2.1.4 Conclusions . 22 2.2 Gestural interfaces . 24 2.2.1 Gestural interfaces and gesture classification . 24 2.2.2 Interactive machine learning . 25 2.2.3 Continuous gesture classification . 26 2.2.4 Conclusions . 28 2.3 Tangible Acoustic Interfaces . 30 2.3.1 Tangible interaction . 30 2.3.2 Musical Tangible User Interfaces . 31 2.3.3 Musical Tangible Acoustic Interfaces . 34 2.3.4 Technologies . 38 vi CONTENTS 2.3.5 Conclusions . 44 3 Applications of continuous recognition for fluid gesture interac- tion design 46 3.1 Introduction . 48 3.2 GIDE: Gesture Interaction Designer application . 50 3.2.1 Implementation . 51 3.2.2 Desiderata . 52 3.2.3 Background: the Gesture Follower algorithm . 54 3.2.4 Algorithm evaluation . 59 3.2.5 Implications for the proposed desiderata . 61 3.3 Workflow . 63 3.3.1 Phase 1: recording a gesture . 63 3.3.2 Phase 2: "Follow" mode and real-time feedback . 64 3.3.2.1 Batch testing . 67 3.3.3 Phase 3: parameters tuning . 67 3.3.4 Three DMIs based on GIDE . 69 3.4 User evaluation . 72 3.4.1 Participants . 72 3.4.2 Workshop procedure . 73 3.4.3 Measures . 75 3.4.4 Results . 76 3.4.4.1 Questionnaire . 76 3.4.4.2 Achievement of the properties . 77 3.4.4.3 Comparison between the different types of feed- back proposed . 78 3.4.4.4 Parameter tuning . 79 3.4.4.5 Observed users strategies . 80 3.5 Discussion . 83 3.5.1 Contributions . 83 3.5.2 Limitations . 86 4 Reality-driven music interaction: using one piezo sensor to aug- ment everyday objects 89 4.1 Introduction . 91 vii CONTENTS 4.1.1 Sonically-augmented physical objects . 92 4.1.2 Capturing a sound . 92 4.1.3 Gestures and musical notes . 93 4.1.4 Reality-driven design . 93 4.1.5 Outline of the chapter . 94 4.2 Features of the system . 95 4.2.1 Background: physical modelling synthesis . 95 4.2.1.1 Mechanical vibrations . 97 4.2.1.2 Modal synthesis and digital resonators . 98 4.2.2 Physically-inspired, audio-driven sound synthesis . 99 4.2.2.1 Audio-driven . 99 4.2.2.2 Physically-inspired . 100 4.2.3 Capturing sounds . 102 4.2.3.1 Interaction workflow . 102 4.2.3.2 Implementation . 102 4.2.4 Playing musical notes . 103 4.3 Realisation . 107 4.4 Credits of this work . 111 4.5 Analysis evaluation . 113 4.5.1 Features . 113 4.5.2 Constraints . 113 4.5.3 Database . 115 4.5.4 Methodology . 115 4.5.5 Results and discussion . 116 4.6 User study . 120 4.6.1 Participants . 120 4.6.2 Procedure . 120 4.6.3 Results . 121 4.6.3.1 What our subjects think about the sounds they created? . 123 4.6.3.2 Playing the notes . 124 4.6.3.3 Why exploring? . 126 4.7 Conclusions . 129 viii CONTENTS 5 Evaluating Mogees for music education in primary schools 133 5.1 Introduction . 134 5.1.1 Context and participants . 135 5.1.2 Aims and objectives of the study . 135 5.2 Methodology . 137 5.2.1 Technical setup . 137 5.2.2 Participants . 137 5.2.3 Tasks . 137 5.3 Discussions . 140 5.3.1 Immediate reward, need for practice, room for virtuosity . 140 5.3.2 A rule-free environment . 141 5.3.3 Finding the notes . 142 5.3.4 Developing generic music skills . 143 5.3.5 Instrument-specific skills . 144 5.4 Contributions . 146 6 The technology in use: an 18-month collaboration with a music producer 148 6.1 Introduction . 150 6.2 The working process . 152 6.2.1 Exploration . 152 6.2.2 Software development . 152 6.2.3 Fitting the system to the existing working environment . 153 6.2.4 The `sequencing' mode . 154 6.2.5 Studio production . 154 6.3 Strategies of play: a subjective perspective . 156 6.3.1 Space is not enough . 156 6.3.2 Selecting the exciters . 156 6.3.3 Objects complexity and illusion . 156 6.3.4 A trial-and-error approach . 157 6.4 Produced works . 159 6.4.1 Live performances . 159 6.4.2 The 'Elementary Excitations' video clip . 161 6.5 Discussion . 164 ix CONTENTS 6.5.1 Exploration is the point of departure . 164 6.5.2 Master what you need, forget the rest . 165 6.5.3 Delegated performance . 166 6.5.4 DIY virtuosism . 166 6.6 Conclusions . 168 7 Conclusions 169 7.1 Developmental narrative . 169 7.2 Potential academic impact . 171 7.3.