Software-Based Extraction of Objective Parameters from Music Performances vorgelegt von Diplom-Ingenieur Alexander Lerch aus Erlangen Von der Fakultät I – Geisteswissenschaften der Technischen Universität Berlin zur Erlangung des akademischen Grades eines Dr. phil. genehmigte Dissertation Promotionsausschuss: Vorsitzender: Prof. Dr. Eberhard Knobloch Berichter: Prof. Dr. Stefan Weinzierl Berichter: Prof. Dr. Roger B. Dannenberg Tag der wissenschaftlichen Aussprache: 27.10.2008 Berlin 2008 D83 Abstract Different music performances of the same score may significantly differ from each other. It is obvious that not only the composer’s work, the score, defines the listener’s music experience, but that the music performance itself is an integral part of this experience. Music performers use the information contained in the score, but interpret, transform or add to this information. Four parameter classes can be used to describe a performance objectively: tempo and timing, loudness, timbre and pitch. Each class contains a multitude of individual parameters that are at the performers’ disposal to generate a unique physical rendition of musical ideas. The extraction of such objective parameters is one of the difficulties in music performance research. This work presents an approach to the software-based extraction of tempo and timing, loudness and timbre parameters from audio files to provide a tool for the automatic parameter extraction from music performances. The system is applied to extract data from 21 string quartet performances and a detailed analysis of the extracted data is presented. The main contributions of this thesis are the adaptation and development of signal processing approaches to performance parameter extraction and the presentation and discussion of string quartet performances of a movement of Beethoven’s late String Quartet op. 130. music performance, music performance analysis, automatic tempo extraction, loudness analysis, timbre analysis, string quartet performance, audio content analysis, audio-to-score-matching Zusammenfassung Verschiedene Aufführungen des gleichen musikalischen Werkes unterscheiden sich deutlich voneinander. Es ist offensichtlich, daß das Musikerlebnis des Hörers nicht nur durch die zugrundeliegende Partitur bestimmt wird, sondern auch maßgeblich von der Interpretation dieser Partitur durch die aufführenden Musi- ker. Diese deuten, modifizieren oder erweitern die im Notenbild enthaltenen Informationen im Zuge ihrer Darbietung. Eine solche Musikaufführung läßt sich mit Parametern der Parameterkategorien Tempo, Lautheit, Klangfarbe und Tonhöhe objektiv beschreiben. Jede der vier Kategorien stellt eine Vielzahl von Parametern bereit, die es den Musikern ermöglicht, musikalische Ideen auf eine einmalige physikalische Art umzusetzen. Die Extraktion solcher Parameter ist eine der typischen Problemstellungen der Aufführungsanalyse. Diese Arbeit präsentiert ein Softwaresystem, das als Werkzeug zur automatischen Extraktion von Tempo-, Lautheits- und Timbre- merkmalen angewendet werden kann. Dieses System wurde für eine systematische Analyse von 21 Streichquartettauf- nahmen eingesetzt. Die Arbeit widmet sich hauptsächlich zwei Thematiken, der Entwicklung und Optimierung von Algorithmen der Audiosignalverarbeitung zur Parameterex- traktion aus Audioaufnahmen musikalischer Aufführungen sowie der Analyse und Diskussion von Streichquartrettaufführungen eines Satzes aus Beethovens spätem Streichquartett op. 130. musikalische Interpretation, Aufführungsanalyse, Tempoerkennung, Lautheits- analyse, Klangfarbenanalyse, Streichquartettanalyse, Musikanalyse Acknowledgments First and foremost, I would like to express my gratitude to Stefan Weinzierl without whom this work never would have started. He piqued my interest in music performance analysis and made this work possible. My sincere thanks go to Roger B. Dannenberg for his commitment, his insightful comments and the interesting discussions. I have been very lucky to be able to count on Tim Flohrer and Martin Schwerdt- feger, my friends and colleagues at zplane.development, who not only encouraged my plans to write a doctorate thesis but actively supported me during all this time. I am very grateful to all the people who have reviewed the thesis or parts of it and helped me to improve it. Particularly, I would like to express my gratitude to Alexander Vorwerk and Odoch Hawkins for their detailed and invaluable feedback. I am obliged to my family for their continuous encouragement and support. This research has been partially funded by the City of Berlin by means of a NaFöG PhD scholarship. Contents 1 Introduction1 2 Music Performance and its Analysis5 2.1 Music Performance......................... 5 2.2 Music Performance Analysis.................... 10 2.3 Analysis Data............................ 11 2.3.1 Data Acquisition ...................... 11 2.3.2 Instrumentation & Genre.................. 15 2.3.3 Variety & Significance of Input Data ........... 15 2.3.4 Extracted Parameters ................... 20 2.4 Research Results .......................... 20 2.4.1 Performance......................... 20 2.4.2 Performer.......................... 23 2.4.3 Recipient .......................... 23 2.5 Software Systems for Performance Analysis............ 25 3 Tempo Extraction 27 3.1 Performance to Score Matching .................. 29 3.1.1 Score Following....................... 29 3.1.2 Audio to Score Alignment ................. 30 3.2 Proposed Algorithm ........................ 32 3.2.1 Definitions.......................... 33 3.2.2 Pre-Processing ....................... 34 3.2.3 Processing.......................... 42 3.2.4 Similarity Measure ..................... 46 i 3.2.5 Tempo Curve Extraction.................. 53 3.2.6 Evaluation.......................... 55 4 Dynamics Feature Extraction 63 4.1 Implemented Features ....................... 65 4.1.1 Peak Meter ......................... 65 4.1.2 VU Meter.......................... 66 4.1.3 Root Mean Square Based Features ............ 66 4.1.4 Zwicker Loudness Features................. 67 4.2 Example Results .......................... 69 5 Timbre Feature Extraction 73 5.1 Implemented Features ....................... 76 5.1.1 Spectral Rolloff....................... 76 5.1.2 Spectral Flux........................ 77 5.1.3 Spectral Centroid...................... 77 5.1.4 Spectral Spread....................... 78 5.1.5 Mel Frequency Cepstral Coefficients............ 78 5.2 Example Results .......................... 79 6 Software Implementation 81 6.1 Data Extraction........................... 82 6.1.1 FEAPI............................ 83 6.1.2 Performance Optimizations ................ 88 6.2 Performance Player......................... 88 6.2.1 Smoothing Filter ...................... 90 6.2.2 Overall Results for each Feature.............. 91 6.2.3 Graphical User Interface.................. 94 7 String Quartet Performance Analysis 97 7.1 Musical Score............................ 97 7.2 Recordings.............................. 99 7.3 Procedure..............................100 7.3.1 Audio Treatment......................100 7.3.2 Analysis Data........................100 7.3.3 Feature Space Dimensionality Reduction . 101 ii 7.4 Overall Performance Profiles....................104 7.4.1 Tempo............................104 7.4.2 Timing............................107 7.4.3 Loudness...........................108 7.4.4 Timbre............................110 7.5 Performance Similarity.......................111 7.5.1 Repetition Similarity....................111 7.5.2 Overall Similarity......................115 7.6 Overall Observations ........................115 7.6.1 Dimensionality of Overall Observations . 119 7.6.2 Relationships between Overall Observations . 120 7.7 Summary ..............................122 8 Conclusion 125 8.1 Summary ..............................125 8.2 Potential Algorithmic Improvements . 128 8.3 Future Directions..........................129 List of Figures 131 List of Tables 133 A Standard Transformations 135 A.1 Discrete Fourier Transformation..................135 A.2 Principal Component Analysis...................136 B Software Documentation 137 B.1 Parameter Extraction........................137 B.1.1 Command Line .......................138 B.1.2 Input and Output Files...................138 B.2 Performance Player.........................139 B.2.1 Loading Performances ...................140 B.2.2 Visualize Parameters....................140 B.2.3 Play Performances .....................140 C Result Tables - String Quartet Analysis 141 Bibliography 153 iii List of Symbols and Abbreviations ACA Audio Content Analysis ADSR technical description of the volume envelope phases of single sounds, frequently used in synthesizers: attack, decay, sustain and release ANOVA Analysis of Variance AOT Acoustic Onset Time API Application Programmer’s Interface AU Audio Unit (Apple) B similarity matrix b(m; n) similarity matrix entry C number of audio channels CD Compact Disc CPU Central Processing Unit (processor) CQT Constant Q Transform δ evaluation criterion DFT Discrete Fourier Transformation DIN Deutsches Institut für Normung DMA Deutsches Musikarchiv DTW Dynamic Time Warping f frequency in Hz fS sample rate in Hz v FEAPI Feature Extraction Application Programmer’s Interface FFT Fast Fourier Transformation FLTK Fast Light Toolkit GPL Gnu General Public License GUI Graphical User Interface H hop-size between STFT blocks HMM Hidden Markov Model IBI Inter-Bar-Interval IEC International Electrotechnical Commission IOI Inter-Onset-Interval