Sound to Sense, Sense to Sound: A State-of-the-Art *** Draft Version 0.09 *** The first S2S2 Summer School Genova — July 2005 Ed. Marc Leman, Damien Cirotteau DRAFT Contents Introduction 14 1 Sound, sense and music mediation 17 1.1 Introduction ....................................... 17 1.2 Statingtheproblem.................................. 18 1.3 Frommusicphilosophytomusicscience. ....... 19 1.4 Thecognitiveapproach. ...... 21 1.4.1 Pioneers ...................................... 21 1.4.2 Gestalt psychology and systematic musicology . ....... 21 1.4.3 Information theoretical accounts of sense . ...... 22 1.4.4 Phenomenology and new media technology . ..... 23 1.4.5 Computational modelling of music cognition . ..... 23 1.5 Beyondcognition .................................. 24 1.5.1 Subjectivism and postmodern musicology . 25 1.5.2 Embodiedmusiccognition . 25 1.5.3 Musicandemotions ................................ 28 1.5.4 GestureModelling ................................ 28 1 CONTENTS 2 Physicalmodelling:................................. 29 Motortheoryofperception: . 30 1.6 Conclusion....................................... 31 2 Learning music 40 2.1 Implicit processing of musical structures . ......... 41 2.1.1 How do non-musician listeners acquire implicit knowledgeofmusic? . 41 2.1.2 Implicit learning of Western pitch regularities . ......... 41 2.1.3 Connectionist model of musical knowledge representation .......... 43 2.1.4 Studying implicit learning processes with artificial materials......... 45 2.1.5 Implicit learning of new musical systems . ....... 46 2.2 Perspectives in musical learning: using multimedia technologies .......... 51 2.2.1 How optimize learning of Western tonal music . ..... 51 2.2.2 Creating learning multimedia tools for music . ...... 55 Reduction of information and optimisation of presentation forms . 55 Synthesis of knowledge and implementation of continuity . 57 3 From Sound to “Sense” via Feature Extraction and Machine Learning 68 3.1 Introduction ....................................... 68 3.2 Bottom-up Extraction of Descriptors from Audio . ...... 70 3.2.1 Simple Audio Descriptors for MusicClassification . ........ 71 Time-DomainDescriptors . 71 Frequency-DomainDescriptors. 72 3.2.2 Extracting Higher-level Musical Patterns . ...... 73 3.3 Closing the Gap: Prediction of High-level Descriptors via Machine Learning . 75 3.3.1 ClassificationviaMachineLearning . ....... 75 3.3.2 Learning Algorithms Commonly Used in Music Classification........ 77 CONTENTS 3 3.3.3 Genre Classification:Typical Experimental Results . .......... 78 3.3.4 Trying to Predict Labels Other Than Genre . ...... 79 3.4 A New Direction: Inferring High-level Descriptors from Extra-Musical Information 80 3.4.1 Assigning Artists to Genres via Web Mining . ..... 81 3.4.2 Learning Textual Characterisations . ...... 83 3.5 Research and Application Perspectives . .......... 87 4 “Sense” in Expressive Music Performance 95 4.1 Introduction ....................................... 96 4.2 DataAcquisitionandPreparation. ...... 97 4.2.1 Using Specially Equipped Instruments . 98 HistoricalMeasurementDevices . 98 Mechanical and Electro-Mechanical Setups . 98 PianoRollsasDataSource. 99 TheIowaPianoCamera . 100 ContemporaryMeasurementDevices . 100 Henry Shaffer’s Photocell Bechstein . 100 Studies with Synthesiser Keyboards or Digital Pianos . 101 TheYamahaDisklavierSystem . 101 Bosendorfer’sSESystem¨ . 103 4.2.2 MeasuringAudioByHand ............................ 103 4.2.3 Computational Extraction of Expression from Audio . 106 4.2.4 Extracting Expression from Performers Movements . ........108 4.2.5 Extraction of Emotional Content from MIDI and Audio . 109 4.3 Computational Models of Music Performance . 110 4.3.1 ModelingStrategies .............................. 111 CONTENTS 4 AnalysisByMeasurements . 112 AnalysisBySynthesis ...............................113 MachineLearning .................................113 Case-BasedReasoning ...............................114 MathematicalTheoryApproach . 114 4.3.2 Perspectives ................................... 115 ComparingPerformances . 115 Modeling Different Expressive Intentions . 115 ExpressionRecognitionModels. 116 4.4 OpenProblemsandFuturePaths . 116 5 Controlling Sound with Senses and Influencing Senses with Sound 136 5.1 Introduction ....................................... 136 5.2 A conceptual framework for gestural control of interactive systems .........139 5.2.1 SyntacticLayer .................................. 140 5.2.2 SemanticLayer .................................. 143 5.2.3 Connecting Syntax and Semantics: Maps and Spaces . 143 5.3 Methodologies, perspectives, and tools for gesture analysis ..............145 5.3.1 Bottom-upapproach ................................ 145 5.3.2 Subtractiveapproach.............................. 146 5.3.3 Spaceviews .................................... 146 5.3.4 Timeviews ..................................... 147 5.3.5 Examplesofmotiondescriptors . 148 5.3.6 Tools: the EyesWeb open platform . 151 5.3.7 Tools: the EyesWeb Expressive Gesture Processing Library..........154 5.4 Controlofmusicperformance. 156 CONTENTS 5 5.4.1 Introduction..................................... 156 A fuzzy analyzer of emotional expression in music and gestures . 158 Applications using the fuzzy analyser . 162 Summaryanddiscussion . .165 5.4.2 The KTH rule system for music performance . 165 pDM - Real time control of the KTH rule system . 174 Ruleapplication ..............................174 pDMplayer .................................174 pDM Expression mappers . 175 5.4.3 Ahomeconductingsystem ... .... .... .... .... .... .... 176 Gesture cue extraction . 178 Mapping gesture cues to rule parameters . 178 5.5 Controllingsoundproduction. 179 5.5.1 Introduction..................................... 180 Soundandmotion .................................182 Soundandinteraction ...............................183 Examples ......................................184 Controlofmusicalinstruments . 184 Controlofsoundingobjects . 185 5.5.2 DJscratchingwithSkipproof . 185 Overviewandbackground . 186 Stateoftheartinscratchtools . 186 Skipproof’sfeatures ................................187 Implementation of synthesized techniques . 189 Controllingthepatch................................190 Skipproofusedinconcerts . 191 CONTENTS 6 Feedback from the DJ and test persons . 191 Conclusions.....................................192 5.5.3 Virtualairguitar .................................. 193 Introduction.....................................193 Synthesizing the electric guitar . 194 VirtualStratocaster.............................195 Simulation of tube amplifier and loudspeaker . 195 Controllersanduserinterfacing . 196 Datagloves .................................197 Controlsticks ................................197 Hand-trackingbyawebcamcamera . 199 Musicalintelligence . .200 Summaryandconclusion . .201 5.5.4 ThereacTable*................................. 201 Antecedents.....................................202 FMOLandVisualfeedback . 202 TangibleUserInterfaces . 203 Conceptionanddesign...............................204 Everythingispossible . 204 Modular synthesis and visual programming . 205 Objects, connections and visual feedback . 205 ThereacTable*Architecture . 207 Vision ....................................208 Connection manager: dynamic patching . 209 Audiosynthesizer .............................209 Visualsynthesizer .............................211 CONTENTS 7 reacTable*hardware . 211 PerformingwiththereacTable* . 212 Novel and occasional users: discovering the reacTable* . 212 Advanced reactablists: performing and mastering the instrument . 213 Towards the luthier-improviser continuum . 213 The bongosero-karateka model . 214 The caresser-masseur-violinist model . 214 Thepaintermodel .............................214 The reacTable* as a collaborative multi-user instrument . 214 ThereacTable*:Conclusion . 215 5.5.5 Theinteractivebook .............................. 215 Introduction.....................................216 Thehistoryofinteractivebooks. 216 Futureperspectives.................................219 ScenariosDesign ..................................220 Steps .....................................220 Conclusions.....................................221 5.6 Multimodal and cross-modal control of interactive systems . ...........221 5.6.1 Cross-modal processing: visual analysis of acoustic patterns.........222 5.6.2 Cross-modal processing: auditory-based algorithms for motion analysis . 224 5.6.3 Multimodal processing for analysis of touch gestures . .......226 5.6.4 Future perspectives for cross-modal analysis . ........228 5.7 Acknowledgements................................. 229 6 Physics-based Sound Synthesis 241 6.1 Introduction ....................................... 241 CONTENTS 8 6.2 GeneralConcepts .................................. 242 6.2.1 DifferentflavorsofmodelingTasks. 242 6.2.2 Physical domains, systems, variables, and parameters . ............243 6.2.3 Dichotomies, problem definition, and schemes . ........244 6.2.4 Important concepts explained . 247 Physical structure and interaction . 247 Signals, signal processing, and discrete-time modeling . ........247 Linearityandtimeinvariance. 248 Energeticbehaviorandstability . 249 Modularity and locality of computation . 250 Types of complexity in physics-based modeling . 251 6.3 State-of-the-Art ................................... 251 6.3.1 K-models ...................................... 252 Finite differencemodels ..............................252 Mass-springnetworks