Automatic Music Classification with jMIR Cory McKay Music Technology Area Department of Music Research Schulich School of Music McGill University, Montreal Submitted January 2010 A thesis submitted to McGill University in partial fulfillment of the requirements of the degree of Doctor of Philosophy. © Cory McKay 2010 2 Abstract Automatic music classification is a wide-ranging and multidisciplinary area of inquiry that offers significant benefits from both academic and commercial perspectives. This dissertation focuses on the development of jMIR, a suite of powerful, flexible, accessible and original software tools that can be used to design, share and apply a wide range of automatic music classification technologies. jMIR permits users to extract meaningful information from audio recordings, symbolic musical representations and cultural information available on the Internet; to use machine learning technologies to automatically build classification models; to automatically collect profiling statistics and detect metadata errors in musical collections; to perform experiments on large, stylistically diverse and well-labelled collections of music in both audio and symbolic formats; and to store and distribute information that is essential to automatic music classification in expressive and flexible standardised file formats. In order to have as diverse a range of applications as possible, care was taken to avoid tying jMIR to any particular types of music classification. Rather, it is designed to be a general-purpose toolkit that can be applied to arbitrary types of music classification. Each of the jMIR components is also designed to be accessible not only by users with a high degree of expertise in computer-based research technologies, but also by researchers with valuable musical expertise, but perhaps less of a background in computational research. Moreover, although the jMIR software can certainly be used as a set of ready-to-use tools for solving music classification problems directly, it is also designed to serve as an open- source platform for developing and testing original algorithms. This dissertation also describes several experiments that were performed with jMIR. These experiments were intended not only to verify the effectiveness of the software, but also to investigate the utility of combining information from different types of musical data, an approach with the potential to significantly advance the performance of automatic music classification in general. 3 Sommaire La classification automatique de la musique est un vaste domaine de recherche, multidisciplinaire par nature, et qui donne lieu à des avancées significatives tant du point de vue scientifique que du point de vue des applications commerciales. La présente dissertation s’articule autour d’un thème central qui est la conception et le développement de jMIR, une suite originale de logiciels qui sont à la fois puissants, flexibles et accessibles. Ces outils peuvent être utilisés pour concevoir, partager et appliquer une grande variété de technologies de classification automatique de la musique. jMIR permet à l’utilisateur d’extraire de l’information significative des enregistrements audio, des représentations musicales symboliques et des informations culturelles disponibles sur l’Internet; de se servir des technologies d’apprentissage automatiques afin de construire des modèles de classification, de compiler automatiquement des profils statistiques, de détecter les erreurs de métadonnées dans les collections de pièces musicales; d’effectuer des expériences sur de larges corpus de musique; et enfin de répertorier et distribuer de l’information essentielle à la classification automatique de la musique sous des formats expressifs, standardisés et flexibles. jMIR est plutôt conçu comme une boîte à outils d’usage général pouvant être appliquée à n’importe quel type de classification de la musique. Chaque élément de jMIR est aussi conçu pour être accessible autant à des utilisateurs experts en technologies de l’information qu’à des, chercheurs dont l’expertise musicale précieuse ne serait pas doublée d’une formation en technologies de l’information. Bien que jMIR intègre un jeu d’outils prêts à résoudre directement des problèmes de classification de la musique, il permet également de s’en servir comme d’une plate-forme ouverte de développement logiciel ainsi que de validation de nouveaux algorithmes. Enfin, la présente dissertation décrit plusieurs expériences réalisées au moyen de jMIR. Ces expériences avaient non seulement pour but de vérifier la pertinence de cet environnement logiciel, mais également d’investiguer les bénéfices qu’apportent l’utilisation conjointe d’informations musicales de nature diverse, cette dernière approche ayant le potentiel de faire avancer de façon significative la classification automatique de la musique en tant que discipline de recherche. 4 Table of Contents Abstract _______________________________________________________________ 3 Sommaire ______________________________________________________________ 4 Table of Contents _______________________________________________________ 5 List of Figures _________________________________________________________ 11 List of Tables __________________________________________________________ 15 Acknowledgements _____________________________________________________ 16 1. Introduction and background __________________________________________ 17 1.1 General overview ________________________________________________________ 17 1.1.1 Introduction ________________________________________________________ 17 1.1.2 Essential concepts in automatic music classification _________________________ 21 1.1.3 Overview of the jMIR components ______________________________________ 24 1.2 Structure of this dissertation _______________________________________________ 30 1.3 Context and background information ________________________________________ 32 1.3.1 Music information retrieval (MIR) _______________________________________ 32 1.3.2 Advantages of automatic music classification ______________________________ 36 1.3.3 Applications of automatic music classification _____________________________ 40 1.3.4 Applying automatic music classification technologies to musicology and music theory _______________________________________________________________________ 44 1.3.5 Comparing musical classification and similarity measurement _________________ 50 1.3.6 Existing non-academic systems _________________________________________ 55 1.3.7 Existing academic systems _____________________________________________ 64 1.4 jMIR’s core objectives and characteristics ____________________________________ 67 1.4.1 Bridging the gaps between the MIR-related disciplines _______________________ 67 1.4.2 Facilitating the effective combination of different types of musical data _________ 69 1.4.3 Accessibility and ease of use ___________________________________________ 71 1.4.4 Longevity and extensibility ____________________________________________ 74 1.4.5 Providing a framework for developing new approaches ______________________ 76 1.4.6 Facilitating and promoting inter-institution collaboration _____________________ 79 1.5 Highlights of research contributions _________________________________________ 83 2. Related research from psychology and the humanities ______________________ 87 2.1 Chapter overview ________________________________________________________ 87 2.2 General psychological classification models and research ________________________ 88 2.2.1 Classical classification theory __________________________________________ 88 2.2.2 Exemplar-based classification theory _____________________________________ 90 2.2.3 General recognition theory _____________________________________________ 93 2.2.4 Assumptions and evaluation of classification models ________________________ 94 2.2.5 Additional issues in classification theory __________________________________ 94 2.3 Insights from music psychology ____________________________________________ 96 2.3.1 Music classification __________________________________________________ 96 2.3.2 Musical similarity ___________________________________________________ 101 5 2.4 Musicological and music theoretical insights __________________________________ 106 3. jAudio: Extracting features from audio ________________________________ 113 3.1 Overview of audio feature extraction and jAudio _______________________________ 113 3.1.1 Introduction ________________________________________________________ 113 3.1.2 Iterative feature development __________________________________________ 114 3.1.3 Contributions to jAudio _______________________________________________ 116 3.1.4 Downloading jAudio _________________________________________________ 117 3.2 Background information __________________________________________________ 117 3.2.1 Digital audio _______________________________________________________ 118 3.2.2 Fourier analysis _____________________________________________________ 120 3.2.3 Windowing functions_________________________________________________ 124 3.2.4 Pre-processing ______________________________________________________ 127 3.2.5 File formats and associated problems ____________________________________ 129 3.2.6 Common low-level features extracted using the DFT ________________________ 133 3.2.7 Common low-level features extracted in the time domain ____________________ 136 3.2.8 High-level information that can be usefully extracted from audio signals ________ 138 3.3 Previous music