Embodied Cognitive Science of Music Modeling Experience and Behavior in Musical Contexts Inaugural-Dissertation zur Erlangung der Doktorwurde¨ der Philosophischen Fakult¨at der Universit¨at zu K¨oln Luder¨ Schmidt K¨oln 2010 ii Contents 1 Introduction1 2 Cognitive Science of Music (CSM)5 2.1 State of CSM............................. 10 2.2 Criticism of “Music Cognition”................... 11 3 Embodied Cognitive Science of Music 13 3.1 Role of the Body........................... 15 3.1.1 Spatial Hearing: Shape of the Body, Active Motion, Moda- lity Interaction, and Neural Plasticity............ 15 3.1.2 Formation of Conceptual Structure: Evidence from Cogni- tive Linguistics........................ 20 3.1.3 Expressive Movement..................... 27 3.1.4 Interactive Technology: New Interfaces........... 30 3.1.5 Remarks............................ 31 3.2 Notions of Embodiment....................... 32 3.3 Agents as a Modeling Framework.................. 35 3.3.1 Tropistic Agent........................ 36 3.3.2 Hysteretic Agent....................... 38 3.3.3 Knowledge-Level Agent.................... 39 3.3.4 Complete Agent........................ 41 3.4 Conclusion............................... 44 4 Theory of Dynamic Systems 45 4.1 Spring-Mass Pendulum........................ 48 4.1.1 No Damping.......................... 53 iii iv CONTENTS 4.1.2 Weak Damping........................ 56 4.1.3 Critical / Strong Damping.................. 59 4.1.4 Negative Damping: β < 0.................. 63 4.1.5 Intermediate Summary – Harmonic Oscillator....... 65 4.2 Resonance Model Analysis...................... 66 4.2.1 Quality Management..................... 74 4.3 van-der-Pol Oscillators........................ 77 4.4 Gesture Analysis........................... 87 5 Dynamic Systems: A Framework for CSM? 91 5.1 Agents as Dynamical Systems.................... 91 5.2 Dynamical Turing Machine...................... 96 5.3 Benefits of Dynamic Systems..................... 99 6 Musical Robotics 101 6.1 Animated Sound Installations.................... 104 6.2 Interaction in Musical Contexts................... 105 6.3 Human Music Performing Capabilities............... 109 6.4 Social Interaction and Synchronization in Musical Contexts.... 111 6.5 Communication of Musical Expression............... 112 6.6 Conclusion............................... 113 7 Khepera III 115 7.1 Khepera III: Technical Description................. 117 7.2 Khepera III: Pd Interaction..................... 122 7.3 Low Level C Programming...................... 127 7.3.1 Decoding Messages: Command Table and Command Parser 127 7.3.2 Retrieving IR Data: getproxir → proxIR ......... 130 7.3.3 Integrating the Communication Protocol: kh3 proximity ir ...................... 131 7.3.4 Interacting with the dsPIC: kh3 sendcommand and kh3 getcommand ........... 132 CONTENTS v 8 Synchronization 137 8.1 Synchronization: Technical Notion.................. 139 8.2 Synchronization without Periodicity................. 142 8.3 Observing Synchronization...................... 145 8.4 Entrainment and Interaction..................... 150 8.4.1 Text of the Proposal..................... 150 A “Robots can’t . ”? 155 B Mathematical Supplements 157 B.1 Critical Damping, Initial Conditions................. 157 B.1.1 Position 1, Velocity 1..................... 157 B.1.2 Position 1, Velocity -1.................... 158 B.1.3 Position -1, Velocity -1.................... 158 B.1.4 Position -1, Velocity 1.................... 158 B.2 Strong Damping, Initial Conditions................. 159 B.2.1 Position 1, Velocity 0..................... 159 B.2.2 Position 1, Velocity 1..................... 160 B.2.3 Position 1, Velocity -1.................... 160 B.2.4 Position -1, Velocity -1.................... 160 B.2.5 Position -1, Velocity 1.................... 161 C Khepera III: Driving Circles 163 C.1 Code Listing k3 circle test.c ................... 165 D Khepera III: Motion Tracking 171 E Khepera III: C-Source for Pd 173 Bibliography 187 vi CONTENTS List of Figures 2.1 Cognitive Science...........................6 3.1 Spatial Hearing............................ 18 3.2 Notions of Embodiment....................... 33 3.3 Cart in Maze............................. 36 4.1 No Damping.............................. 56 4.2 Weak Damping............................ 59 4.3 Critical Damping........................... 62 4.4 Strong Damping............................ 63 4.5 Negative Damping.......................... 64 4.6 Phase Shift Forced Oscillations................... 70 4.7 Standard Resonance Curves..................... 71 4.8 Effective Resonance Curve...................... 73 4.9 Curious Resonance Curve...................... 75 4.10 Two trajectories of the van der Pol oscillator for γ = 0.3...... 80 4.11 Oscillations of van der Pol oscillator with γ = 0.3.......... 81 4.12 Two trajectories for γ = 5....................... 82 4.13 Oscillations for γ = 5......................... 83 4.14 Oscillations for γ = 10 and γ = 20.................. 83 4.15 Attraction to Limit Cycle...................... 84 4.16 Schematic Face............................ 88 5.1 Agent-Enviroment Coupling..................... 92 5.2 Coupling Nervous System – Body – Environment......... 93 5.3 Mutual coupling of within-agent sub-systems............ 94 vii viii LIST OF FIGURES 6.1 Haile.................................. 106 6.2 Spherical Robot............................ 108 6.3 Waseda Flutist............................ 110 6.4 M[ε]X, taken from Burger / Bresin 2007 [46]............ 112 7.1 Khepera II Extended......................... 116 7.2 Khepera III.............................. 118 7.3 K3 Control by Pd........................... 123 7.4 Khepera III OSC Control...................... 125 8.1 Phase................................. 140 8.2 Synchronous Movement........................ 144 8.3 Degrees of freedom for Keepon’s movements. Taken from Micha- lowski / Sabanovic / Kozima 2007 [187], page 91.......... 146 8.4 Interaction(s) between one speaker and N listeners......... 147 8.5 Facial display of a child interacting with Keepon, illustrating the importance of sequential analysis. Screen shots from Kozima and Michalowski’s movie Keepon dancing to Spoon’s “Don’t You Evah”; for a link see Footnote4....................... 148 8.6 Entrainment.............................. 151 C.1 Driving a circle with a Khepera III robot.............. 164 D.1 MaxMSP / softVNS Patch...................... 172 Chapter 1 Introduction The considerations put down in the following chapters have grown out of previous work on modeling perceptual auditory processes, aimed at understanding (certain aspects of) music perception and music cognition. This work draws on the anatomy and physiology of the auditory system, fin- dings from psychoacoustics, and signal processing procedures (see Schmidt 2000 [241] for an introduction to basic concepts). Auditory anatomy and physiology provide hints for the architecture of a system intended to model the function of the peripheral auditory system, i.e. to produce a comparable output given the same input, for instance concerning different processing stages to be taken into account. A combination of physiological data (single and multiple cell recor- dings) and psychoacoustic measurements (e.g. discrimination of tones, masking patterns) is commonly used to obtain a detailed specification of the response cha- racteristics of the peripheral auditory system. Formalisms provided by the signal processing literature allow to design appropriate procedures generating the de- sired output. Such a system, frequently called auditory model (e.g. Leman 1995 [171]), is typically implemented in the form of a filterbank. The outputs of the individual filters essentially constitute a time-varying spectral representation of the acoustic input and are sometimes interpreted as representing neural activity of the auditory nerve (as a so-called neural activation pattern, see e.g. Patterson / Allerhand / Giguere 1995 [213]). Research on the perceptual organization of auditory input, dubbed auditory scene analysis (ASA) by Bregman 1990 [43], attempts to utilize rules inspired by Gestalt psychology in connection with informal applications of concepts from artificial intelligence to give an account of the way a listener arrives at a description of objects or events in the environment solely based on auditory information (for applications to music perception see Bregman 1990 [43], Chapter 5). The rules set forth by are ASA research are assumed to operate on some form of time- frequency representation of acoustical input, inviting a combination with research 1 2 CHAPTER 1. INTRODUCTION on auditory models as described above. Accordingly, within computational auditory scene analysis (CASA) (e.g. Schmidt / Seifert / Eichert 1997 [245]; for a recent overview see Wang / Brown 2006 [300]), the attempt is undertaken to integrate rules provided by ASA with auditory models in the specification of systems actually performing tasks such as separating speech or music from noisy backgrounds or segregating different musical voices; for a system description addressing music listening see Scheirer 2000 [240]. Some problems, however, seem to arise in this approach: • The rules of auditory scene analysis are formulated rather vaguely (e.g. Ei- chert / Schmidt / Seifert 1997 [76]), leaving room for situations in which competing rules may apply; little is known about the resolution of such conflicts (see van Valkenburg / Kubovy 2004 [293]). Thus, more investiga- tion of the processes underlying phenomena