Music Information Retrieval & Visualization Tim Langer Abstract—The growing possibilities for digital storage has led to large personal music collections that require new technologies to use and maintain them. There has been a lot of research done related to visualizing music collections in order to improve browsing, exploring and searching of music. While almost every publication in this subject has its own purpose and approach to achieve it there still exist common problems, ideas and methods. In the following I will identify major causes that led to such development, based on state of the art technology, at first. I will then further specify a couple of commonly used ideas to solve those problems as well as techniques to realise them. At last a couple of chosen examples are presented to demonstrate approaches to the prior specification. The ambition of this paper is to identify the development within Music Information Retrieval & Visualization and present a survey of recent research. Index Terms—MIR, Music, Information Visualization, Music Collections, Music Visualization, Music Similarity 1 INTRODUCTION With the growing impact of technology on everyday life the research 2.1.1 Playlists field of Information Visualization has developed the new and impor- As stated above the basic systems used nowadays rely on playlists to tant topic of Personal or Casual Information Visualization [24] which visualize music collections. But as pictures are easier and faster to puts the focus on technologies that try to include visualization into the recognise for a human, it is quite intuitive that such would be a better common daily life cycle. The focus of this paper will be on the do- choice than text based methods. [31] states this as following: ”How- main of music collections by clarifying the current problems within ever this [text] approach by itself is inadequate for effective search and wide-spread music players and giving an overview about recent re- retrieval”. Also it is quite hard to put a lot of information into a short search that tries to solve those problems and add new functionality to text whilst a graphical visualization could present different informa- support the user. The next section will summarize existing problems tion by varying itself in terms of size, colour, form or other parame- while the third and fourth section will feature new concepts to adopt ters. On going with a growing music collection and a lot of cross-links to the growing demand for development in terms of strategy, ideas and information (such as different artists on the same song or one song on visualization. The fifth section lists a couple of exemplary research different albums) a large amount of playlists is needed to keep up with and at the end conclusions will be drawn. all this and thereby the clear view gets lost. As mentioned before, some of those music players already try to create automatic playlists. This 2 STATUS QUO is done by either studying the users listening behaviour, and grouping In today’s world digital music collections are usually organized based favourite tracks together, or by analyzing the metadata. The thereby on a selfmade system by the owner of the collection. Such organisation extracted information is then used to create playlists for different cate- systems usually vary a lot in their structure - one might sort his collec- gories, usually distinguished by the genre (see 2.1.2). L As listening to tion by artist, by album or by the date of release. Influenced by those music is correlated to emotions the choice of music tends to depend on heterogeneous structures the current methods for displaying and or- our current mood [27]. So searching music that fits this mood would ganzing music in state of the art digital music players [19] are playlists be a quite intuitive feature! But with playlists this is only possible if created by the user. People tend to fill such playlists with similar mu- the user knows what kind of music is stored in a playlists and/or if he sic to create playlists for specific emotional states (for example slow already created a playlists that fits this mood. So the system hardly music that helps to relax). This human necessity is one of the main aids the user with his choice. reason why new researches in MIR topic usually rely on similarity- measures to arrange music collections and it has been proved to be a 2.1.2 Tags well working concept ([19] [31] [26]). The adding of meta information with ID3-Tags brings a whole lot of new problems with it. Firstly, tags provide information appended to 2.1 Problems and not derived from the music which therefore can obviously con- As stated above the tools to listen to music are typically commercial tain false information. Secondly, as such tags are added by commu- products with a large spread. With the growing size of a music col- nities of voluntary users stored in large online databases faults (such lection it gets harder to find the music you are looking for or simply as typing errors) are inevitable. Thirdly, the process of assigning the browse your collection. As the only possibility to search the music, metadata to a musical track is a subjective process ([31] [19]). One through the file system of the operating system, is based on text, the person might classify a song simply as ”Rock” while another person user has to know at least some part of the file’s title to actually find it might go into more detail and categorise it as ”Hard-Rock”. Hilliges through the search. When thinking about use cases as described in sec- et. al.[19] provide a good example for this when stating that the well- tion 3.1 this gives no opportunity at all to get results related to music known iTunes music store puts both punk-rockers Anti-Flag and mu- a user already knows which is a basic demand. To solve this problem sician James Blunt into the category of Rock. But not only the sub- meta data is appended to the digital music to provide further infor- jective assignment of Genres (and other tags) is a problem, also the mation. The thereby developed problems will be explained by using number of possibilities that can be used is problematic field. On the genres as an example in section 2.1.2 one hand when specifying too many different choices (such as classi- fying dozens of different subcategories for Rock) the large amount of details makes it almost impossible to still maintain an informational • Tim Langer is studying Media Informatics at the University of Munich, overview. On the other hand tagging is almost futile when putting all Germany, E-mail: [email protected] your data into the same categories as it provides no further informa- • This research paper was written for the Media Informatics Advanced tion. And last but not least most musicians develop different styles Seminar on Information Visualization, 2008/2009 throughout their musical career. It might usually be possible to cat- egorize one album into a matching genre, but rarely the whole artist based on all his or her releases. And sometimes it even impossible to sum up the different songs in one album into one matching genre as 2.2.2 Amarok the style of the tracks might vary because of different influences, fea- Amarok2 is a Linux based music player. Just like the Windows Me- tured artists or for other reasons. In fact ”few artists are truly a single dia Player it uses the album covers to visualize a collection of albums ’point’ in any imaginable stylistic space but undergo changes through- (see figure 2). The playlists view (see figure 3) is a bit more advanced out their careers and may consciously span multiple styles within a though. It does not only list the current tracks of the playlist (or all the single album, or even a single song” [2] playlists available when using another view) but also appends some beneficial information to the whole view like other albums from the same artist or similar artists. It also provides automatical playlists by searching through the whole music collection (as defined by the user) and merging it using the tags chosen by the user. Track-wise visual- izations are available by installing a plugin (see figure 4) Fig. 1. Streaming Media Players - Unique Users by www.websiteoptimization.com Fig. 3. Amarok Playlist 2.2 Examples 2.2.3 iTunes 3 The following section will introduce three state of the art music and With the spread of the iPod , a portable music (and nowadays also media players. They were chosen as a representative of the three media) player developed by Apple, their associated software called biggest operating systems: Microsoft’s Windows, Apple’s Mac and iTunes has experienced an increasing distribution as well. Similar to the Linux system. The Windows Media Player and iTunes also posses the Windows Media Player it offers a grid-based overview of music al- a large market share within digital media players. bums included to the iTunes library (see figure 2) but also a so-called coverflow view that reminds of a jukebox (see figure 5 bottom). Again, just as the Windows Media Player, they use animated art (see figure 4) to visualize tracks on their own, the only outstanding difference is the fact that they use 3D. With the integration of the newly developed fea- ture called ”Genius” (more information at section 3.2.3) they approach the research done in the MIR field. Fig. 2. Visualization using Album Covers (example from Amarok) 2.2.1 Windows Media Player (WMP) Due to the enormous market share of the Miccosoft operating system Windows 1, the Windows Media Player enjoys a market share of about 50% [18] (and more than 70 million unique streaming users (see fig- ure 1)).