12th International Society for Music Information Retrieval Conference (ISMIR 2011)

MUSICLEF: A BENCHMARK ACTIVITY IN MULTIMODAL MUSIC INFORMATION RETRIEVAL

Nicola Orio David Rizo Riccardo Miotto, Nicola Montecchio Markus Schedl Olivier Lartillot University of Padova University of Alicante University of Padova Johannes Kepler University Academy of Finland

[email protected] [email protected] {miottori,montecc2}@dei.unipd.it [email protected] [email protected]

ABSTRACT The same need has been perceived in MIR, motivating the development of an important evaluation campaign, the Mu- This work presents the rationale, tasks and procedures of sic Information Retrieval Evaluation eXchange (MIREX). MusiCLEF, a novel benchmarking activity that has been de- MIREX campaigns 1 are organized since 2005 [4] by the In- veloped along with the Cross-Language Evaluation Forum ternational Music Information Retrieval Systems Evaluation (CLEF). The main goal of MusiCLEF is to promote the de- Laboratory (IMIRSEL) at the Graduate School of Library velopment of new methodologies for music access and re- and Information Science, University of Illinois at Urbana- trieval on real public music collections, which can combine Champaign. Due to the many limitations posed by the mu- content-based information, automatically extracted from mu- sic industry, the organizers of the MIREX chose to distribute sic files, with contextual information, provided by users via only publicly available test collections. Participants are in tags, comments, or reviews. Moreover, MusiCLEF aims at charge to create their own collection and after local experi- maintaining a tight connection with real application scenar- mentation submit their software that is run by the organizers. ios, focusing on issues on music access and retrieval that are This approach has two drawbacks, which have already been faced by professional users. To this end, this year’s evalua- debated by the MIR research community: the results of pre- tion campaign focused on two main tasks: automatic catego- vious campaigns cannot be easily replicated and the results rization of music to be used as soundtrack of TV shows and depend on the individual training sets and not only on the automatic identification of the digitized material of a music submitted algorithms. digital library. A recent relevant initiative, that aims at overcoming the limitations imposed by not sharing the datasets between re- 1. INTRODUCTION searchers, is the Million Songs Dataset (MSD). Thanks to MSD 2 , researchers can access a number of features from a The increasing availability of digital music accessible by very large collection of songs [2]. Unfortunately, the algo- end users is boosting the development of Music Informa- rithms used to extract these features are not public, limiting tion Retrieval (MIR), a research area devoted to the study the possibility to carry out research on content description of methodologies for content- and context-based music ac- techniques. Another ongoing initiative related to the eval- cess. As it appears from the scientific production of the last uation of MIR approaches is the Networked Environment decades, research on MIR encompasses a wide variety of for Music Analysis (NEMA), that aims at providing a web- different subjects that go beyond pure retrieval: the defini- based architecture for the integration of music data and an- tion of novel content descriptors and multidimensional sim- alytic/evaluative tools 3 . NEMA builds upon the achieve- ilarity measures to generate playlists; the extraction of high ments of MIREX campaigns regarding the evaluation of MIR level descriptors – e.g. melody, harmony, rhythm, struc- approaches, with the additional goal of providing tools for ture – from audio; the automatic identification of artist and resource discovery and sharing. genre. As it is well known, the possibility to evaluate the Within this scenario, MusiCLEF is an additional bench- different research results using a shared dataset has always marking initiative, that has been proposed in 2011 as part played a central role in the development of information re- of the activities of the Cross-Language Evaluation Forum trieval methodologies, as it is witnessed by the success of (CLEF). CLEF focuses on multilingual and multimodal re- initiatives such as TREC and CLEF, which focus on textual trieval 4 and gathers researchers in different aspect of in- documents. formation retrieval, ranging from plagiarism and intellectual property rights to image retrieval. Permission to make digital or hard copies of all or part of this work for The goal of MusiCLEF is to promote the development of personal or classroom use is granted without fee provided that copies are 1 not made or distributed for profit or commercial advantage and that copies http://www.music-ir.org/mirex 2 http://labrosa.ee.columbia.edu/millionsong/ bear this notice and the full citation on the first page. 3 http://www.music-ir.org/?q=nema/overview c 2011 International Society for Music Information Retrieval. 4 http://clef-campaign.org/

603 Oral Session 6: Databases and Evaluation novel methodologies for music access and retrieval, which the music descriptors are public – and in particular are based can combine content-based information, automatically ex- on the set of tools provided by the MIRToolbox – but also tracted from music files, with contextual information, pro- that participants can submit their own original algorithm for vided by users through tags, comments, or reviews. The feature extraction, that will be run locally. Therefore, Musi- combination of these two sources of information is still under- CLEF goals are to fill the gap between the other important investigated in MIR, although it is well known that content- initiatives in MIR evaluation: researchers can test and com- based information alone is not able to capture all the relevant pare their approaches using a shared number of tasks, as in features of a given music piece (for instance, its usage as a MIREX, while accessing a shared collection of content de- soundtrack or the year of release), while contextual informa- scriptors, as in MSD. tion suffers from the typical limitations for new items and new users (also known as cold start). 2. APPLICATION SCENARIOS Aiming at investigating and promoting research on the combination of textual and music information, MusiCLEF As mentioned in the previous section, a major goal of Musi- has a strong focus on multimodality that, together with mul- CLEF is to maintain a tight connection with real application tilingualism, is the main objective of the CLEF evaluation scenarios, in order to promote the development of techniques forum. Moreover, the tasks proposed for MusiCLEF 2011 that can be applied to solve issues in music accessing and re- are motivated by real scenarios, discussed with private and trieval that are faced by professional users. The choice of fo- public bodies involved in music access and dissemination. In cusing on professional users is motivated by the fact that they particular, MIR techniques can be exploited for helping mu- need to address a number of real-life issues that are usually sic professionals to describe music collections and for man- not taken into account by music accessing systems aimed at aging a music digital library of digitized analogue record- the general public. At the same time, the evaluation of the ings. To this end, the organizers of MusiCLEF exploited effectiveness of the proposed automatic solution is easier to the ongoing collaborations with both a company for music assess, because professional users have a clear idea of what broadcasting services (LaCosa s.r.l.) and a public music li- are their information needs. brary (University of Alicante’s Fonoteca). In the following we present the two professional partners Two tasks are proposed within MusiCLEF 2011, and both of MusiCLEF, and we also describe the motivations that in- are based on a test collection of thousands of songs in MP3 duced us to organize the two tasks mentioned in the previous format. To completely overcome copyright issues, only low- section. level descriptors will be distributed to participants. Figure 1 depicts the tasks workflow of MusiCLEF, which is described 2.1 LaCosa s.r.l. in more detail in the following sections. LaCosa was founded as a service provider of the major TV broadcasting – public and private – companies in Italy with MusicCLEF public web site MusicCLEF not public the goal of managing and describing a large music collec- if(conn SELEC WHERE if(conn 1.read print tion of songs to be used for TV programs, including jingles, SELEC 2. produce WHERE print features MP3 background and incidental music, and music themes for TV extractor Library last.fm data 7. publish Low level shows. LaCosa has a strong cooperation with RTI, a com- features Campaign Results 1. http request pany that, apart from buying and storing songs issued by

3. download Participant evaluation Results the major record companies, produces its own music cata-

Participant participant pc . webservice logue. At present, RTI library contains about 320,000 songs 6. submit 6. last.fm of pop-rock, jazz, and classical music. Besides playing the role of music consultant, being one of the biggest private

4. read music repositories in Italy, RTI offers a number of services Low level to external companies of music consultants, who can browse features