A Model of Inter-musician Communication for Artificial Musical Intelligence Oscar Alfonso Puerto Melendez Thesis of 60 ECTS credits Master of Science (M.Sc.) in Computer Science June 2017 ii A Model of Inter-musician Communication for Artificial Musical Intelligence by Oscar Alfonso Puerto Melendez Thesis of 60 ECTS credits submitted to the School of Computer Science at Reykjavík University in partial fulfillment of the requirements for the degree of Master of Science (M.Sc.) in Computer Science June 2017 Supervisor: David Thue, Assistant Professor Reykjavík University, Iceland, Examiner: Hannes Högni Vilhjálmsson, Examiner Associate Professor, Reykjavík University, Iceland Thor Magnusson, Examiner Senior Lecturer in Music, University of Sussex, UK Copyright Oscar Alfonso Puerto Melendez June 2017 iv A Model of Inter-musician Communication for Artificial Musical Intelligence Oscar Alfonso Puerto Melendez June 2017 Abstract Artificial Musical Intelligence is a subject that spans a broad array of disciplines related to human cognition, social interaction, cultural understanding, and music generation. Although significant progress has been made on particular areas within this subject, the combination of these areas remains largely unexplored. In this dissertation, we propose an architecture that facilitates the integration of prior work on Artificial Intelligence and music, with a fo- cus on enabling computational creativity. Specifically, our architecture represents the verbal and non-verbal communication used by human musicians using a novel multi-agent inter- action model, inspired by the interactions that a jazz quartet exhibits when it performs. In addition to supporting direct communication between autonomous musicians, our architec- ture presents a useful step toward integrating the different subareas of Artificial Musical Intelligence. Titll verkefnis Oscar Alfonso Puerto Melendez júní 2017 Útdráttur Tónlistargervigreind er grein sem spannar fjölbreytt svið tengd mannlegri þekkingu, félags- legum samskiptum, skilningi á menningu og gerð tónlistar. Þrátt fyrir umtalsverða fram- þróun innan greinarinnar, hefur samsetning einstakra sviða hennar lítið verið rannsökuð. Í þessari ritgerð eru lögð drög að leið sem stuðlar að samþættingu eldri verka á sviði gervi- greindar og tónlistar, með áherslu á getu tölva til sköpunar. Nánar tiltekið lýtur ritgerðin að samskiptum sem notuð eru af tónlistarmönnum við nýju samskiptalíkani fyrir fjölþætt samskipti, sem er innblásið af samskiptum jazz kvarteta á sviði. Til viðbótar við stuðning við bein samskipti sjálfvirkra "tónlistarmanna, felst í rannsókninni framþróun samþættingar ólíkra undirsviða tónlistargervigreindar. vi A Model of Inter-musician Communication for Artificial Musical Intelligence Oscar Alfonso Puerto Melendez Thesis of 60 ECTS credits submitted to the School of Computer Science at Reykjavík University in partial fulfillment of the requirements for the degree of Master of Science (M.Sc.) in Computer Science June 2017 Student: Oscar Alfonso Puerto Melendez Supervisor: David Thue Examiner: Hannes Högni Vilhjálmsson Thor Magnusson viii The undersigned hereby grants permission to the Reykjavík University Library to reproduce single copies of this Thesis entitled A Model of Inter-musician Communication for Arti- ficial Musical Intelligence and to lend or sell such copies for private, scholarly or scientific research purposes only. The author reserves all other publication and other rights in association with the copyright in the Thesis, and except as herein before provided, neither the Thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author’s prior written permission. date Oscar Alfonso Puerto Melendez Master of Science x To my parents José Jacobo Puerto and Iris Melendez. xii Acknowledgements I would like to thank my supervisor David Thue for his remarkable guidance throughout my research and to my family for all the support and encouragement. xiv xv Contents Acknowledgements xiii Contents xv List of Figures xvii List of Tables xix 1 Introduction 1 1.1 General Background . 2 1.1.1 Artificial Music Intelligence . 2 1.1.2 Computational models of music creativity . 3 1.1.3 Common algorithms applied to musical composition . 3 1.1.3.1 Cellular automata . 4 1.1.3.2 Grammar based music composition . 4 1.1.3.3 Genetic Algorithms . 5 1.1.4 Interactive musical systems for live music . 6 1.1.5 Communication protocols in a musical multi-agent system . 7 1.1.5.1 The FIPA Protocols . 7 1.2 Musebot Ensemble . 8 1.2.1 Limited real-time capabilities . 8 1.2.2 Changing musical roles . 9 1.2.3 Collaboration between agents . 9 1.3 Summary . 9 2 Problem Formulation 11 2.1 Our vision: an ideal AMI system . 11 2.1.1 Autonomy . 12 2.1.2 Musical communication . 12 2.1.3 Performance of alternative musical styles . 12 2.1.4 Taking on different roles within a musical ensemble . 12 2.2 Our focus: enabling communication between musical agents . 13 2.3 Jazz as a good domain for AMI research . 13 2.4 Expected challenges . 14 2.5 Summary . 14 3 Related Work 15 3.1 Algorithmic Composition . 15 3.1.1 Generative grammars . 15 3.1.2 Evolutionary methods . 16 xvi 3.1.3 Artificial neural networks . 17 3.2 Live Algorithms . 17 3.3 Musical multi-agent systems . 18 3.3.1 Multi-agent system for collaborative music . 19 3.3.2 Multi-agent system for simulating musicians’ behaviors . 19 3.3.3 Multi-agent system towards the representation of musical structures. 20 3.4 Efforts toward integration . 20 3.5 Commercial software for music . 21 3.6 Summary . 21 4 Proposed Approach 23 4.1 General Architecture . 24 4.1.1 The CADIA Musebot . 24 4.1.1.1 Musician Agent . 26 4.1.1.2 Composer Agent . 28 4.1.1.3 Synchronizer Agent . 30 4.1.1.4 Ensemble Assistant Agent . 31 4.2 Interaction Between Agents . 31 4.2.1 Interaction Protocols . 31 4.3 Development Process . 33 4.3.1 Initial Considerations . 33 4.3.2 Development Challenges and Implementations . 33 4.4 Summary . 34 5 Evaluation 35 5.1 Testing and Simulations . 35 5.1.1 Bastien and Hostager’s case study . 35 5.1.2 Song 1 . 36 5.1.2.1 Results . 37 5.1.2.2 Analysis . 38 5.1.3 Song 2 . 38 5.1.3.1 Results . 39 5.1.3.2 Analysis . 41 5.1.4 Song 3 . 41 5.1.4.1 Results . 42 5.1.4.2 Analysis . 43 5.2 Summary . 43 6 Discussion and future work 45 6.1 Benefits and limitations . 45 6.2 Future Work . 49 6.2.1 Towards Algorithmic Composition . 49 6.2.2 Towards Live Algorithms . 50 6.2.3 Towards a Standard Taxonomy in AMI . 50 6.2.4 Further Considerations . 50 7 Conclusion 51 Bibliography 53 xvii List of Figures 1.1 An example of a linear single Cellular Automaton. 4 1.2 Example of the binary and notational representations of the cellular automata rhythm extracted from Brown’s work [36] . 4 4.1 An example of our Musebot agent architecture. Each Musebot represents an autonomous musician and is a multiagent system composed of four agents: mu- sician, composer, synchronizer, and ensemble assistant. 24 4.2 The musician agent’s (MA’s) Finite State Machine (FSM). 26 4.3 Example of the musician agent’s behaviours hierarchy. 27 4.4 Cycle process in which the MAs exchange roles. 28 4.5 The Intro FSM. 29 4.6 The Accompaniment FSM. 29 4.7 The Solo FSM. 30 4.8 One of the agent interaction protocols that we implemented to support commu- nication in our agent architecture (the FIPA Contract Net Interaction Protocol). 32 6.1 Synchronization process of two agents in the Musebot Ensemble. The Musebot Bass comes from being a leader and is ready to support the Musebot Piano, but it will need to request its Synchronizer Agent to obtain the current section being played by the Musebot Piano. 47 6.2 Interaction between three internal agents through custom protocols. 49 xviii xix List of Tables 1.1 Set of notes in an input melody . 5 1.2 Transition probability matrix from Brown’s markov process example [42]. 5 1.3 FIPA Interaction Protocols . 8 4.1 This table represents the properties of the song’s structure object. The tempo and time signature’s numerator and denominator are stored as integer values. The form stores a string of characters, and the sections of the form store a list of musical chords. 27 5.1 This table enumerates each agent’s possible actions during a song’s progression. The actions are classified into communicative behaviours and musical behaviours. 36 5.2 Introduction of Song 1. The leader waits while the piano plays an introduction, and all three accompanists compose their parts. 37 5.3 First Chorus (AAB) of Song 1. The leader (sax) waits until the end of the first section, because it needs to give a time to the CA to find out the current section (First A) being played by the accompanists. The leader then starts to play the solo at the beginning of the second section A. Finally at section B, the "sax" passes the leadership to the "bass". All three accompanists play their parts dur- ing this chorus. 37 5.4 Second Chorus (AAB) of Song 1. The new leader (piano), plays a solo in each section of this chorus, and passes the leadership to the "bass" at the last section. The rest of the members play the accompaniments. 37 5.5 Third Chorus (AAB) of Song 1. The "bass" takes its turn to be the leader and play a solo while the accompanists play their parts. At section B of this chorus, the "drums" accept to be the new leader of the ensemble. 37 5.6 Fourth Chorus (AAB) of Song 1.