Engineering Systematic Musicology Methods and Services for Computational and Empirical Music Research
Total Page:16
File Type:pdf, Size:1020Kb
Universiteit Gent Faculteit Letteren en Wijsbegeerte Vakgroep Kunst-, Muziek- en Theaterwetenschappen Engineering systematic musicology Methods and services for computational and empirical music research Joren Six Proefschrift voorgelegd tot het behalen van de graad van Doctor in de Kunstwetenschappen Academisch jaar 2017-2018 Universiteit Gent Faculteit Letteren en Wijsbegeerte Vakgroep Kunst-, Muziek- en Theaterwetenschappen Promotor: Prof. dr. Marc Leman Doctoraatsbegeleidingscommissie: Dr. Olmo Cornelis Dr. Frans Wiering Examencommissie: Dr. Federica Bressan Dr. Olmo Cornelis Prof. dr. ir. Tijl De Bie Prof. dr. Pieter-Jan Maes Dr. Frans Wiering Dr. Micheline Lesaffre Dr. Luc Nijs Proefschrift voorgelegd tot het behalen van de graad van Doctor in de Kunstwetenschappen Academisch jaar 2017-2018 Universiteit Gent Faculteit Letteren en Wijsbegeerte Vakgroep Kunst-, Muziek- en Theaterwetenschappen, IPEM De Krook, Miriam Makebaplein 1, B-9000 Gent, België Kaft: Mer Du Nord, Étude n◦1, Thierry De Cordier, 2011 v Acknowledgement This doctoral dissertation is the culmination of my research carried out at both IPEM, Ghent University and the School of Arts, Ghent.I have been lucky enough to pursue and combine my interests for both music and computer science in my research. As a trained computer scientist I have been applying my engineering background to problems in systematic musicology. The output of this work has been described in various articles some of which are bundled in this dissertation. Admittedly, my research trajectory does not follow the straightest path but meanders around several fields. This meandering has enabled me to enjoy various vistas and led me to a plethora of places - physi- cally and intellectually - not easily reached without taking a turn now and again. I think this multi-disciplinary approach prepared me bet- ter for a hopefully somewhat stable career in research. I also had the time required to cover a lot of ground. At the School of Arts, Ghent I was employed for four years as a scientific employee. At IPEM, Ghent University I was given the opportunity to continue my work as a doc- toral student again for four years. This allowed me to not only have a broad perspective but also reach the depth required to contribute new knowledge and propose innovative methods. It is safe to say that without Olmo Cornelis I would not have started this PhD project. Olmo wrote the project proposal which eventually allowed me to start my research career at the School of Arts. The concept of having an engineer next to a humanities scholar was defi- nitely enriching to me and I do hope that the opposite is also somewhat true. His guidance during those first four (and following) years was in- dispensable. His pointers on music theory, his support with academic writing and ideas on computer aided (ethno)musicology are just a few examples. Actually, I want to profoundly thank the whole group of colleague researchers at the School of Arts, or what was then known as the Royal Conservatory of Ghent. Undeniably, they had a defining influence on my research and personal life. I fondly remember many discussions in the cellar and at my desk at the Wijnaert. I struggle to think of examples where gossiping, scheming and collusions between a colleague and partner of a colleague could have had a more positive outcome. Indeed, I mean you: Ruth and Clara. Later on, Marc Leman gave me the opportunity to continue my research at IPEM. I am very grateful to have been offered this privilege. I am quite aware that being able to pursue one’s interests by doing research is exactly that: a privilege. IPEM provided fertile ground to further my research. Marc’s hands-off approach displays a great amount of trust in his research staff. This freedom worked especially well for me since it made me self-critical on priorities and planning. I would also like to acknowledge the IPEM bunch: a diverse col- lective of great individuals each in their own way. I especially would like to thank Ivan for the many hardware builds and creative ideas for practical solutions. Katrien for taking care of the administrative side of things. Esther for having the patience to listen to me whining about my kids. Jeska and Guy for proofreading this work, all remaining spelling errors are surely introduced while reworking the text afterwards. I would also like to thank all the others for the many discussions at the kitchen table during lunch and generally for being great colleagues. Furthermore, I am very grateful to the RMCA (Royal Museum for Central Africa), Tervuren, Belgium for providing access to its unique archive of Central African music. Thanks to my friends and family for the support over the years. I would especially want to thank Will for proofraeding parts of this work and Emilie for pointing me to VEWA: next year will be good. Of course, it is hardly an understatement to claim that this work would not be here without my parents. Thanks for kindling an interest in music, letting me attend piano lessons and for keeping me properly fed and relatively clean, especially in my first years. Thanks also to Bobon for caring for Oscar and Marit, often on short notice in those many small emergency situations. On the topic: I would like to leave Oscar and Marit out of this acknowledgment since they only sabotaged this work, often quite successfully. But they are a constant reminder on the relativity of things and I love them quite unconditionally. Finally, I would like to thank the light of my life, the daydream that accompanies me at night, the mother of my children: Barbara. Contents Acknowledgement v Nederlandstalige samenvatting xi Summary xiii Outline xv 1 Problem definition and methodology 1 1.1 Problem definition . .1 1.1.1 Services versus methods . .4 1.1.2 MIR-techniques vs. techniques for empirical re- search . .9 1.1.3 My solutions in the humanities-engineering plane 20 1.1.4 Digital humanities . 25 1.2 Methodology . 28 1.2.1 Intellectual property rights and sharing research code . 28 1.2.2 Copyrights on music and sharing research data . 30 1.2.3 Publication culture and providing incentives . 31 1.2.4 Research software versus end-user software . 33 1.3 Summary . 35 2 Methods 37 2.1 Introduction . 37 2.2 Tarsos - a platform for pitch analysis . 39 2.2.1 Introduction . 40 2.2.2 Exploring the capabilities of Tarsos through case studies . 54 2.2.3 Musicological aspects of Tarsos . 67 2.2.4 Conclusion . 71 2.2.5 Appendix A - pitch representation . 72 2.2.6 Appendix B - audio material . 75 2.3 A case for reproduciblity in MIR . 77 2.3.1 Introduction . 78 2.3.2 Method . 84 2.3.3 Evaluation . 88 2.3.4 Discussion . 96 2.3.5 Conclusion . 97 2.4 Applications of duplicate detection . 101 vii viii Contents 2.4.1 Introduction . 102 2.4.2 Duplicate detection . 102 2.4.3 Acoustic fingerprinting . 106 2.4.4 Archive of the Royal Museum for Central-Africa 108 2.4.5 De-duplication in practice . 112 2.4.6 Conclusions . 114 3 Services 115 3.1 Introduction . 115 3.2 TarsosDSP - audio processing in Java . 117 3.2.1 Introduction . 118 3.2.2 Design decisions . 119 3.2.3 Example applications . 123 3.2.4 Availability and license . 123 3.2.5 Conclusion . 125 3.3 Panako - scalable acoustic fingerprinting . 127 3.3.1 Introduction . 128 3.3.2 Method . 129 3.3.3 Results . 134 3.3.4 Conclusion . 139 3.4 Audio alignment for synchronization . 141 3.4.1 Introduction . 142 3.4.2 Audio-to-audio alignment . 143 3.4.3 Synchronizing data streams . 146 3.4.4 Results . 148 3.4.5 Discussion . 152 3.4.6 Conclusion . 153 3.5 A framework for active listening . 155 3.5.1 Introduction . 156 3.5.2 System overview . 157 3.5.3 Evaluation . 159 3.5.4 Results . 161 3.5.5 Discussion . 162 3.5.6 Conclusion . 164 4 Discussion and conclusion 167 4.1 Contributions . 167 4.2 Digital towards augmented humanities . 168 4.3 Conclusion . 170 Contents ix A List of output 173 A.1 Journal articles . 173 A.2 Conference contributions . 174 A.3 Lectures and discussions . 177 A.4 Software systems . 178 A.4.1 Panako: an acoustic fingerprinting framework . 178 A.4.2 Tarsos: software for pitch analysis . 179 A.4.3 TarsosDSP: a Java library for audio processing . 180 A.4.4 SyncSink: synchronize multimodal data . 181 A.4.5 TeensyDAQ: data acquisition with teensy . 182 B List of figures, tables and acronyms 185 B.1 Figures . 185 B.2 Tables . 187 B.3 Acronyms . 188 References 191 xi Nederlandstalige samenvatting Een van de grote onderzoeksvragen in systematische muziekwetenschap- pen is hoe mensen met muziek omgaan. Deze wetenschap onderzoekt bijvoorbeeld hoe effecten van muziek in relatie staan tot specifieke muzi- kale structuren. Deze fundamentele relatie kan op verschillende manie- ren benaderd worden. Bijvoorbeeld een perspectief vertrekkende vanuit traditie waarbij muziek aanzien wordt als een fenomeen van menselijke expressie. Een cognitief perspectief is een andere benadering, daarbij wordt muziek gezien als een akoestische informatiestroom gemoduleerd door perceptie, categorisatie, blootstelling en allerhande leereffecten. Een even geldig perspectief is dat van de uitvoerder waarbij muziek voortkomt uit gecoördineerde menselijke interactie. Om een muzikaal fenomeen te begrijpen is een combinatie van (deel)perspectieven vaak een meerwaarde. Elk perspectief brengt methodes met zich mee die onderzoeksvra- gen naar concrete musicologische onderzoeksprojecten kan omvormen. Digitale data en software vormen tegenwoordig bijna altijd de kern van deze methodes. Enkele van die algemene methodes zijn: extractie van akoestische kenmerken, classificatie, statistiek, machine learning.