Two-dimensional gesture mapping for interactive real-time electronic music instruments Thomas Grill [email protected] October 2008 Eidesstattliche Erkl¨arung: Ich habe die vorliegende Arbeit selbst verfasst. Außer den angefuhrten¨ wur- den keine weiteren Hilfsmittel oder Quellen verwendet. Stellen, welche w¨ortlich oder sinngem¨aß aus anderen Arbeiten ubernommen¨ wurden, sind als solche gekennzeichnet. Thomas Grill Abstract Electronic music instruments have become widespread complements to the variety of traditional acoustic instruments, opening up multifaceted pos- sibilities for the creation of sound and structure in the domains of both composition and live performance. Yet, the prevalent dependence on com- plex technologies and computer-based hardware tends to hide the interesting artistic potentials behind abstract interface and control concepts that are hardly compatible to musical thinking and acting. This thesis presents fundamental thoughts and technical developments in order to make electronic instruments as intuitive and expressive as their acoustic ancestors and explores a novel approach to the transformation of a player's musical gestures into actual sound. We can build this transformation on two principles: First, the idea of en- ergy propagation and transformation through the instrument from gestural actions to sonic developments, and second, the foundation of the trans- formation process on a representation which is based on perceptually and musically meaningful structure elements. As gestural and sonic developments are complex by their nature, the presented concept constricts the focus of musical action to user-compilable repertories of gesture and sound representation. This allows to break down the high-dimensional complexity to two-dimensional map-like collections of structure elements. By basing the entire transformation process of gestures to sound on such a two-dimensional projection, a more immediate control of transfor- mation strategies and more intuitive visual feedback is possible, since the two-dimensional structures can be easily laid out and controlled on screen. This thesis researches both the theoretical background as well as the technological prerequisites. As a proof of concept an actual implementation of a new software-based real-time instrument using such two-dimensional gesture mapping is presented. Zusammenfassung Elektronische Musikinstrumente bilden eine mittlerweile weit verbreitete Er- g¨anzung zur Vielfalt an traditionellen akustischen Instrumenten. Sie er¨off- nen vielf¨altige M¨oglichkeiten zur Gestaltung von Klang und Struktur in- nerhalb der Bereiche von sowohl Komposition als auch Live-Performance. Die vorherrschende Abh¨angigkeit von komplexen Technologien und compu- terbasierter Hardware fuhrt¨ jedoch zu einer Abschottung der interessanten kunstlerischen¨ Potentiale hinter abstrakten Interfaces und Steuerungskon- zepten, die kaum mit kunstlerischem¨ Denken und Agieren vereinbar sind. Diese Arbeit stellt grundlegende Uberlegungen¨ und technische Entwick- lungen vor, um elektronische Instrumente genau so intuitiv und expressiv wie ihre akustischen Vorfahren zu machen und untersucht einen neuartigen Zugang zur Transformation von musikalischen Gesten in erzeugten Klang. Eine solche Transformation kann auf zwei Prinzipien basieren: Erstens, der Idee der Energiefortpflanzung und -transformation durch ein Instrument { von gestischen Abl¨aufen zu klanglichen Entwicklungen { und zweitens, der Verankerung des Transformationsprozesses in einer Repr¨asentation be- stehend aus perzeptiv und musikalisch sinnvollen Strukturelementen. Weil gestische und klangliche Entwicklungen von Natur aus komplex sind, schr¨ankt das vorgestellte Konzept den Fokus musikalischer Aktion auf durch den Benutzer zusammenstellbare Repertoires ein. Dies erlaubt, die hochdimensionale Komplexit¨at auf zwei-dimensionale landkartenartige Best¨ande von Strukturelementen herunterzubrechen. Die Fundierung des gesamten Transformationsprozesses von Gesten zu Klang auf einer solchen zweidimensionalen Projektion erm¨oglicht unmittel- barere Kontrolle von Transformationsstrategien und intuitiveres visuelles Feedback, da zweidimensionale Strukturen einfach auf dem Bildschirm zu veranschaulichen sind. Diese Arbeit untersucht sowohl die theoretischen Grundlagen wie auch technologische Voraussetzungen. Als `proof-of-concept' wird die Implemen- tierung eines neuen Software-basierten Echtzeit-Instruments unter Verwen- dung eines solchen zweidimensionalen Gesten-Mappings pr¨asentiert. Acknowledgements My heartfelt thanks go out { for everything related to perceiving music { to Wolfgang Musil, Gunther¨ Rabl, Don van Vliet, et al. for everything related to performing music { to Ang´elicaCastell´o,Katharina Klement, Low Frequency Orchestra, et al. for everything related to researching music { to Volkmar Klien, Martin Gasser, et al. for everything, really { to Marion P¨otz 1 Contents Abstract 1 Acknowledgements 1 1 Introduction 4 1.1 Electronic instruments . .4 1.2 Personal motivation for this project . .5 1.3 Anticipated instrument architecture . .8 2 Background 12 2.1 Representation of sound and music . 14 2.1.1 Time-domain representation . 16 2.1.2 Time-frequency representation . 16 2.1.3 Sinusoidals and residuum . 20 2.1.4 Acoustic quanta . 21 2.1.5 Sound objects and spectromorphologies . 24 2.1.6 Representation based on perceptive similarity . 26 2.2 Sound synthesis . 28 2.2.1 Sample-based synthesis . 28 2.2.2 Spectral modeling synthesis . 30 2.2.3 Physical modeling . 32 2.3 Playing an (electronic) music instrument . 33 2.3.1 Man-machine interaction . 35 2.3.2 Sensoric input . 38 2.3.3 Gesture mapping . 43 2.3.4 Instrument feedback . 46 2.3.5 Software development systems . 52 2 CONTENTS 3 3 Instrument design 54 3.1 Basic considerations . 54 3.2 Sensoric input . 55 3.3 Input analysis . 58 3.4 Gesture repertory . 60 3.5 Sound repertory . 63 3.6 Gesture transformation . 66 3.7 Sound synthesis . 67 3.7.1 Granular synthesis . 68 3.7.2 Spectral modeling synthesis . 69 3.8 Instrument feedback . 70 4 Implementation and assessment 71 4.1 Repertory preprocessing . 73 4.2 Real-time operation . 75 4.3 Hardware considerations . 79 4.4 Early practical experiences . 82 5 Conclusions and future perspectives 83 A Pure Data patches 85 List of figures 93 Bibliography 96 Chapter 1 Introduction Which is more musical, a truck passing by a factory or a truck passing by a music school? (John Cage [Cag73]) 1.1 Electronic instruments Traditional acoustic instruments have been with us for a long time. We have become accustomed to the variety of different sounds and to the physical materials they are made of, just as we have become familiar to the (occasion- ally somewhat peculiar) human gestures performed in order to make these instruments play what we call music. And, at the same time, we are embedded in a universe of non-musical sound { a gigantic sea of noises effortlessly absorbing any conception of what be musical sound within a wealth of possibilities in structure and timbre. It is owing to electronic instruments that music is not bound to the acoustics of vibrating sound-boxes any longer { whatever one's imagination provides, it can be cast into sound. There are no physical fundamentals or material properties to be considered, there are no bodily restraints to be respected, there is no canonical school of techniques to be followed. One of the things I see as a possibility with these [electronic] instruments is [...] that we could take some of the focus off of physical virtuosity and the kind of athleticism of learning to 4 CHAPTER 1. INTRODUCTION 5 play an instrument, and put as much focus as we could on the mental and emotional activities of music, whether it's being a better listener or imagining things and making them happen or interpreting things to your liking. (Tod Machover [Mac99]) Yet, how can we keep the complexity and broad spectrum of all these possibilities manageable? How can we make our powerful new instruments instantly playable? Present-day makers of music instruments can choose from two traditions to follow, \a millennial one, as old and rich as the history of music making, and a half-century old one, that of computer mu- sic" [Jor01]. It is an important aspect of this thesis to consider experiences from both branches: To make use of our physical intimacy and technical familiarity with traditional acoustic instruments and look into all those new methods of digital music technology in order to allow for previously unheard sounds and structures. 1.2 Personal motivation for this project By the year 2001 I was gradually becoming busier playing concerts as a computer musician. Up to this point I followed the habit of modifying my software instruments for every single performance in the belief that the instrument design should in some way reflect the concept of the musical piece to perform and vice versa. Gradually I discovered that the increased complexity of the respective instruments and their perpetual modification overburdened my skills to properly handle them while performing. It became necessary to spend more time on practice rather than on developing new instruments. As a resort I decided to design a more general and long-lasting instrument that should serve all my basic needs without major modifications for each individual project. After some weeks of intense development I was able to perform on my new instrument (figure 1.1): A simple 4-track sample player with access to a sound library, live-recording and mix-in possibility, looping and