Computational Modelling of Expressive Music Performance in Jazz Guitar: A Machine Learning Approach Sergio Iv´anGiraldo TESI DOCTORAL UPF / 2016 Director de la tesi Dr. Rafael Ram´ırez Department of Information and Communication Technologies By Sergio Giraldo (2016) and licensed under Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported Please see licence conditions at http://creativecommons.org To Camilo... Acknowledgements Four years have passed since I joined the Music Technology Group to begin my PHD. During this period, many people have come across (inside and outside the MTG) giving important contributions to this work. For con- tributions I want to refer not only to the theoretical aspects of the investi- gation, but also to other types of helping support. Received contributions have taken several forms, from babysitting my kid for free in times of work overload, to informal discussions about the investigation, performing tests, doing recordings, or giving very specific theoretical advise. The list is large, and the contributors are many: Firstly i would like to express my gratitude to my advisor, Rafael Ram´ırez, who has been the main contributor to this work. Not only for his excellent advise and guidance, but also for his patience, support, and friendship. Also for considering me to actively participate in the research projects that have been and are being developed right now. I would like to thank to Xavier Serra, Perfecto Herrera, Emilia G´omezand Enric Guaus for their advise and help on several concrete topics needed for this dissertation. I would like also to acknowledge the blind reviewers of the different publications done during this research for their useful and constructive comments. Ad- ditionally I would acknowledge the financial support received by the TIMUL and TELMI projects. Secondly, I would like to express my gratitude to Camilo, for his patience, support, and love (and for always being so enthusiast about coming to the lab with dad!). To my family who has been supporting me from de distance with love and interest. To Yeliz for her unconditional support and love during these last months. v vi Thirdly I would like to thank people and colleagues inside the MTG, spe- cially to Zacharias Vamvakuosis, Nadine Kroner, Sankalp Gulati, Ajay Srini- vasamurthy, Helena Bantul`a,Jose Zapata, and also outside the MTG to Vanessa Picone, Danni Pucha, Anna Herrero, Joan Vi~nals,Alejandro Gall´on, Paula Betancur, Flavia Becerra, Vicktoria Triebner and all the friends and colleagues who have also been direct and indirect contributors to this disser- tation by helping with recordings, giving very useful musical feedback, par- ticipating on the events, doing annotations, correcting/translating to cata- lan, reading and correcting English typos, playing music together, babysit- ting Camilo... etc. To all of them I want to deeply acknowledge their help and support. Abstract Computational modelling of expressive music performance deals with the analysis and characterization of performance deviations from the score that a musician may introduce when playing a piece in order to add expression. Most of the work in expressive performance analysis has focused on expres- sive duration and energy transformations, and has been mainly conducted in the context of classical piano music. However, relatively little work has been dedicated to study expression in popular music where expressive per- formance involves other kinds of transformations. For instance in jazz music, ornamentation is an important part of expressive performance but is seldom indicated in the score, i.e. it is up to the interpreter to decide how to orna- ment a piece based on the melodic, harmonic and rhythmic contexts, as well as on his/her musical background. In this dissertation we present an inves- tigation in the computational modelling of expressive music performance in jazz music, using the as a case study. High-level features are extracted from the scores, and performance data is obtained from the corresponding audio recordings from which a set of performance actions are obtained semi auto- matically (including timing/energy deviations, and ornamentations). After each note is characterized by its musical context description, several ma- chine learning techniques are explored to, on one hand, induce regression models for timing, onset and dynamics transformations, and classification models for ornamentation to render expressive performances of new pieces, and, on the other hand, learn expressive performance rules to analyse its musical meaning. Finally. we report on the relative importance of the con- sidered features, quantitatively evaluate the accuracy of the induced models, and discuss some of the learnt expressive performance rules. Moreover, we present different approaches for semi-automatic data extraction-analysis, as well as, some applications in other research fields. The findings, methods, data extracted, and libraries developed for this work are a contribution to vii viii resumen expressive music performance field, as well to other related fields. Resumen El modelado computacional de la expresividad en la interpretaci´onmusical trata sobre el an´alisisy la caracterizaci´onde las desviaciones que, con re- specto a la partitura, los m´usicosintroducen cuando interpretan una pieza musical para a~nadirexpresividad. La mayor´ıadel trabajo en an´alisisde la expresividad musical hace ´enfasis en la manipulaci´onde la duraci´ony el volumen de las notas, y ha sido principalmente estudiada en en el contexto de piano cl´asico.Sin embargo, muy poco esfuerzo ha sido dedicado al estu- dio de la expresividad en m´usicapopular. Concretamente, en m´usicajazz acciones expresivas como los ornamentos son una parte importante de la expresividad musical ya que estos no est´anindicados en la partitura y es tarea del m´usicohacer uso de los mismos a~nadiendoo substituyendo notas en la partitura. Los m´usicosa~nadenornamentos teniendo en cuenta el con- texto mel´odico,arm´onicoo r´ıtmicodel tema, o bien seg´unsu experiencia en el lenguaje jazz´ıstico. En este trabajo, presentamos una investigaci´on en el modelado computacional de la expresividad musical en m´usicajazz, tomando la guitarra el´ectricacomo caso de estudio. En primer lugar, ex- traemos descriptores de alto nivel de las partituras y obtenemos datos de la ejecuci´ona partir de las correspondientes grabaciones de audio, de donde obtenemos semiautom´aticamente la desviaciones temporales y de energ´ıa de cada nota, as´ıcomo la detecci´onde ornamentos. Despu´esde que cada nota ha sido caracterizada por su contexto musical, varios algoritmos de aprendizaje autom´aticoson explorados para, de un lado, inducir modelos de regresi´onpara duraci´on,comienzo de nota y volumen, y modelos de clasi- ficaci´onpara ornamentos para, finalmente, renderizar ejecuciones musicales expresivas. Por otra parte, aplicamos t´ecnicasde inducci´onautom´aticade reglas al conjunto de descriptores obtenidos para obtener reglas de ejecuci´on musical analizando su sentido musical. Por ultimo, analizamos la importan- cia relativa de los descriptores considerados, cuantitativamente evaluamos la exactitud de los modelos y discutimos acerca de las reglas obtenidas. Igual- mente, reportamos m´etodos para la extracci´on-an´alisissemi-autom´aticode datos, asi como aplicaciones en otros campos de investigaci´on. Los resul- tados, los m´etodos presentados, as´ıcomo los datos extra´ıdosy las librer´ıas de c´odigogeneradas para llevar a cabo esta investigaci´onconstituyen un aporte relevante en el campo de estudio computacional de la expresividad musical, as´ıcomo en otras ´areasde investigaci´onrelacionadas. Resum El modelatge computacional de l'expressivitat en la interpretaci´omusical, tracta sobre l’an`alisii la caracteritzaci´ode les desviacions que els m´usics introdueixen quan interpreten una pe¸camusical, per afegir expressivitat, respecte la partitura. La major part del treball en an`aliside l'expressivitat musical fa `emfasien la manipulaci´ode la durada i el volum de les notes. La majoria dels estudis s'han fet en el context de piano cl`assici molt poc esfor¸cha estat dedicat a la m´usica popular. Concretament, en m´usica jazz, accions expressives com els ornaments, s´onuna part important de l'expressivitat musical; Tot i no estar indicats en la partitura, ´estasca del m´usicfer ´usdls ornaments, afegir o substituir notes en la partitura, tot tenint en compte el context mel`odic,harm`onico r´ıtmic del tema, o b´e segons la seva experi`enciaen el llenguatge jazz´ıstic.En aquest treball, pre- sentem una recerca en el modelatge computacional de l'expressivitat musical en m´usicajazz, prenent la guitarra el`ectricacom a cas d'estudi. En primer lloc, extraiem descriptors d'alt nivell de les partitures i obtenim dades de l’execuci´oa partir dels corresponents enregistraments d’`audio, d'on tamb´e obtenim semiautom´aticament les desviacions temporals i d'energia de cada nota aix´ıcom la detecci´od'ornaments. Despr´esque cada nota hagi sigut caracteritzada pel seu context musical, diversos algoritmes d'aprenentatge autom`atics´onexplorats per a diferents fins. D'un costat, induir models de regressi´oper a la durada, el comen¸cament de nota i el volum, i models de ix x resum classificaci´oper a ornaments per, finalment, renderitzar execucions musi- cals expressives. D'altra banda, apliquem t`ecniques d'inducci´oautom`atica de regles al conjunt de descriptors obtinguts, per obtenir regles d’execuci´o musical analitzant les seves implicacions musicals.