Music Genre Classification Systems - A Computational Approach Peter Ahrendt Kongens Lyngby 2006 IMM-PHD-2006-164 Technical University of Denmark Informatics and Mathematical Modelling Building 321, DK-2800 Kongens Lyngby, Denmark Phone +45 45253351, Fax +45 45882673 [email protected] www.imm.dtu.dk IMM-PHD: ISSN 0909-3192 Summary Automatic music genre classification is the classification of a piece of music into its corresponding genre (such as jazz or rock) by a computer. It is considered to be a cornerstone of the research area Music Information Retrieval (MIR) and closely linked to the other areas in MIR. It is thought that MIR will be a key element in the processing, searching and retrieval of digital music in the near future. This dissertation is concerned with music genre classification systems and in particular systems which use the raw audio signal as input to estimate the corresponding genre. This is in contrast to systems which use e.g. a symbolic representation or textual information about the music. The approach to music genre classification systems has here been system-oriented. In other words, all the different aspects of the systems have been considered and it is emphasized that the systems should be applicable to ordinary real-world music collections. The considered music genre classification systems can basically be seen as a feature representation of the song followed by a classification system which predicts the genre. The feature representation is here split into a Short-time feature extraction part followed by Temporal feature integration which combines the (multivariate) time-series of short-time feature vectors into feature vectors on a larger time scale. Several different short-time features with 10-40 ms frame sizes have been exam- ined and ranked according to their significance in music genre classification. A Consensus sensitivity analysis method was proposed for feature ranking. This method has the advantage of being able to combine the sensitivities over several ii resamplings into a single ranking. The main efforts have been in temporal feature integration. Two general frame- works have been proposed; the Dynamic Principal Component Analysis model as well as the Multivariate Autoregressive Model for temporal feature integra- tion. Especially the Multivariate Autoregressive Model was found to be success- ful and outperformed a selection of state-of-the-art temporal feature integration methods. For instance, an accuracy of 48% was achieved in comparison to 57% for the human performance on an 11-genre problem. A selection of classifiers were examined and compared. We introduced Co- occurrence models for music genre classification. These models include the whole song within a probabilistic framework which is often an advantage compared to many traditional classifiers which only model the individual feature vectors in a song. Resum´e Automatisk musik genre klassifikation er et forskningsomr˚ade, som fokuserer p˚a, at klassificere musik i genrer s˚asom jazz og rock ved hjælp af en computer. Det betragtes som en af de vigtigste omr˚ader indenfor Music Information Retrieval (MIR). Det forventes, at MIR vil spille en afgørende rolle for f.eks. behandling og søgning i digitale musik samlinger i den nærmeste fremtid. Denne afhandling omhandler automatisk musik genre klassifikation og specielt systemer, som kan prædiktere genre udfra det r˚a digitale audio signal. Mod- sætningen er systemer, som repræsenterer musikken i form af f.eks. symboler, som det bruges i almindelig node-notation, eller tekst-information. Tilgangen til problemet har generelt været system-orienteret, s˚aledes at alle komponenter i systemet tages i betragtning. Udgangspunktet har været, at systemet skulle kunne fungere p˚a almindelige folks musik samlinger med blandet musik. Standard musik genre klassifikations-systemer kan generelt deles op i en fea- ture repræsentation af musikken, som efterfølges af et klassifikationssystem til mønster genkendelse i feature rummet. I denne afhandling er feature repræsen- tationen delt op i hhv. Kort-tids feature ekstraktion og Tidslig feature integra- tion, som kombinerer den (multivariate) tidsserie af kort-tids features i en enkelt feature vektor p˚a en højere tidsskala. I afhandlingen undersøges adskillige kort-tids features, som lever p˚a 10-40 ms tidsskala, og de sorteres efter hvor godt de hver især kan bruges til musik genre klassifikation. Der foresl˚as en ny metode til dette, Konsensus Sensitivitets Anal- yse, som kombinerer sensitivitet fra adskillige resamplings til en samlet vurder- ing. iv Hovedvægten er lagt p˚aomr˚adet tidslig feature integration. Der foresl˚as to nye metoder, som er Dynamisk Principal Komponent Analyse og en Multivariat Au- toregressiv Model til tidslig integration. Den multivariate autoregressive model var mest lovende og den gav generelt bedre resultater end en række state-of-the- art metoder. For eksempel gav denne model en klassifikationsfejl p˚a 48% mod 57% for mennesker i et 11-genre forsøg. Der blev ogs˚a undersøgt og sammenlignet et udvalg af klassifikationssystemer. Der foresl˚as desuden Co-occurrence modeller til musik genre klassifikation. Disse modeller har den fordel, at de er i stand til at modellere hele sangen i en prob- abilistisk model. Dette er i modsætning til traditionelle systemer, som kun modellerer hver feature vektor i sangen individuelt. Preface This dissertation was prepared at the Institute of Informatics and Mathemat- ical Modelling, Technical University of Denmark in partial fulfillment of the requirements for acquiring the Ph.D. degree in engineering. The dissertation investigates automatic music genre classification which is the classification of music into its corresponding music genre by a computer. The ap- proach has been system-oriented. Still, the main efforts have been in Temporal feature integration which is the process of combining a time-series of short-time feature vectors into a single feature vector on a larger time scale. The dissertation consists of an extensive summary report and a collection of six research papers written during the period 2003–2006. Lyngby, February 2006 Peter Ahrendt vi Papers included in the thesis [Paper B] Ahrendt P., Meng A., and Larsen J., Decision Time Horizon for Mu- sic Genre Classification using Short Time Features, Proceedings of European Signal Processing Conference (EUSIPCO), Vienna, Austria, September 2004. [Paper C] Meng A., Ahrendt P., and Larsen J., Improving Music Genre Classifi- cation by Short-Time Feature Integration, Proceedings of IEEE In- ternational Conference on Acoustics, Speech, and Signal Processing (ICASSP), Philadelphia, USA, March 2005. [Paper D] Ahrendt P., Goutte C., and Larsen J., Co-occurrence Models in Music Genre Classification, Proceedings of IEEE International Workshop on Machine Learning for Signal Processing (MLSP), Mystic, Connecticut, USA, September 2005. [Paper E] Ahrendt P. and Meng A., Music Genre Classification using the Multivariate AR Feature Integration Model, Audio Genre Classi- fication contest at the Music Information Retrieval Evaluation eXchange (MIREX) (in connection with the annual ISMIR conference) [53], London, UK, September 2005. [Paper F] Hansen L. K., Ahrendt P., and Larsen J., Towards Cognitive Compo- nent Analysis, Proceedings of International and Interdisciplinary Con- ference on Adaptive Knowledge Representation and Reasoning (AKRR), Espoo, Finland, June 2005. [Paper G] Meng A., Ahrendt P., Larsen J., and Hansen L. K., Feature Integration for Music Genre Classification, yet unpublished article, 2006. viii Acknowledgements I would like to take this opportunity to first and foremost thank my colleague Anders Meng. We have had a very close collaboration which is a fact that the publication list certainly illustrates. However, most importantly, the collabora- tion has been very enjoyable and interesting. Simply having a counterpart to tear down your idea or (most often) help you build it up, has been priceless. I would also like to thank all of the people at the Intelligent Signal Processing group for talks and chats about anything and everything. It has been interesting, funny and entertaining and contributed a lot to the good times that I’ve had during last three years of studies. Last, but not least, I would like to thank my wife for help with careful proof- reading of the dissertation as well as generally being very supportive. x Contents Summary i Resum´e iii Preface v Papers included in the thesis vii Acknowledgements ix 1 Introduction 1 1.1Scientificcontributions........................ 3 1.2Overviewofthedissertation..................... 3 2 Music Genre Classification Systems 5 2.1Humanmusicgenreclassification.................. 7 2.2 Automatic music genre classification . 11 xii CONTENTS 2.3Assumptionsandchoices....................... 14 3 Music features 17 3.1Short-timefeatureextraction.................... 18 3.2Featurerankingandselection.................... 29 4 Temporal feature integration 31 4.1GaussianModel............................ 36 4.2MultivariateAutoregressiveModel................. 38 4.3DynamicPrincipalComponentAnalysis.............. 47 4.4FrequencyCoefficients........................ 48 4.5LowShort-TimeEnergyRatio................... 48 4.6HighZero-CrossingRateRatio................... 49 4.7BeatHistogram............................ 49 4.8BeatSpectrum............................ 50 5 Classifiers andPostprocessing 51 5.1GaussianClassifier.......................... 54 5.2GaussianMixtureModel....................... 56 5.3LinearRegressionclassifier....................