Day 5: Music Recommendation Douglas Eck Research Scientist, Google, Mountain View ([email protected])

Day 5: Music Recommendation Douglas Eck Research Scientist, Google, Mountain View (Deck@Google.Com)

1 Music Music I Like You Like Music I Used To L i ke

Get this t-shirt at dieselsweeties.com

Thursday, July 7, 2011 Overview

• Overview of music recommendation. • Content-based method: autotagging. • Side issues of interest

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Three Approaches to Recommendation • Collaborative ﬁltering (Amazon) “Many people who bought A also bought B. You bought A, you’ll probably like B.” Cannot recommend items no one has bought. Suffers from popularity bias

• Social recommendation (Last.FM) Community members tag music. Tag clouds used as basis for similarity measure. Cannot recommend items no one has tagged. Popularity bias (all roads lead to Radiohead)

• Expert recommendation (Pandora) Trained experts annotate music based on ~=400 parameters Not scalable (thousands of new songs online daily)

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Music Recommendation Point - Counterpoint:

What’s the best way to help users ﬁnd music they like?

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Point: Use content analysis for music recommendation.

• Paul Lamere (EchoNest) • Audio helps us know more about music in the long tail. • Evidence: Examples, observations.

Douglas Eck ([email protected]) / Google November 2009

Thursday, July 7, 2011 Help! My iPod thinks I’m emo.

SXSW Interactive March 17, 2009 #sxswemo

Paul Lamere Anthony Volodkin Photo (CC) by Jason Rogers

Thursday, July 7, 2011 Music recommendation is broken A recommendation that no human would make

If you like Britney Spears ...

You might like the Report on Pre-War Intelligence

Thursday, July 7, 2011 Why do we care? Compulsory Long Tail slide

Thursday, July 7, 2011 State of music discovery We can’t seem to ﬁnd the long tail

Sales data for 2007 - 4 million unique tracks sold But ... - 1% of tracks account for 80% of sales - 13% of sales are from American Idol or Disney artists

State of the Industry 2007 - Nielsen Soundscan

Thursday, July 7, 2011 State of music discovery We can’t seem to ﬁnd the long tail

Sales data for 2007 - 4 million unique tracks sold But ... - 1% of tracks account for 80% of sales - 13% of sales are from American Idol or Disney artists

State of the Industry 2007 - Nielsen Soundscan

Thursday, July 7, 2011 State of music discovery We can’t seem to ﬁnd the long tail

Sales data for 2007 - 4 million unique tracks sold But ... - 1% of tracks account for 80% of sales - 13% of sales are from American Idol or Disney artists

State of the Industry 2007 - Nielsen Soundscan

Make everything available

Thursday, July 7, 2011 State of music discovery We can’t seem to ﬁnd the long tail

Sales data for 2007 - 4 million unique tracks sold But ... - 1% of tracks account for 80% of sales - 13% of sales are from American Idol or Disney artists

State of the Industry 2007 - Nielsen Soundscan

Make everything available Help me find it

Thursday, July 7, 2011 Help! I’m stuck in the head The limited reach of music recommendation

Popularity

83 Artists 6,659 Artists 239,798 Artists Sales Rank Study by Dr. Oscar Celma - MTG UPF

Thursday, July 7, 2011 Help! I’m stuck in the head The limited reach of music recommendation

48% of recommendations

Popularity

83 Artists 6,659 Artists 239,798 Artists Sales Rank Study by Dr. Oscar Celma - MTG UPF

Thursday, July 7, 2011 Help! I’m stuck in the head The limited reach of music recommendation

48% of recommendations

52% of recommendations

Popularity

83 Artists 6,659 Artists 239,798 Artists Sales Rank Study by Dr. Oscar Celma - MTG UPF

Thursday, July 7, 2011 Help! I’m stuck in the head The limited reach of music recommendation

48% of recommendations

0% of recommendations

52% of recommendations

Popularity

83 Artists 6,659 Artists 239,798 Artists Sales Rank Study by Dr. Oscar Celma - MTG UPF

Thursday, July 7, 2011 Help! My iPod thinks I’m emo

Why is music recommendation broken?

Thursday, July 7, 2011 The Wisdom of Crowds How does collaborative ﬁltering work?

35% 4% 62% 8% 60%

Overlap Data based on listening behavior of 12,000 Last.fm Listeners

Thursday, July 7, 2011 The stupidity of solitude The Cold Start problem

If you like Blondie, you might like the DeBretts ...

But the recommender will never tell you that.

Thursday, July 7, 2011 The stupidity of solitude The Cold Start problem

If you like Blondie, you might like the DeBretts ...

But the recommender will never tell you that.

Thursday, July 7, 2011 The stupidity of solitude The Cold Start problem

If you like Blondie, you might like the DeBretts ...

But the recommender will never tell you that.

Thursday, July 7, 2011 The Harry Potter Problem If you like X you might like Harry Potter

Thursday, July 7, 2011 Popularity Bias Rich get richer - diversity is the biggest loser

Results of popularity bias: - Rich get richer - Loss of diversity - No long tail recommendations

Thursday, July 7, 2011 Popularity Bias Rich get richer - diversity is the biggest loser

Results of popularity bias: - Rich get richer - Loss of diversity - No long tail recommendations

Thursday, July 7, 2011 The Novelty Problem If you like The Beatles you might like ...

Thursday, July 7, 2011 The Napoleon Dynamite Problem Some items are not easy to categorize

Thursday, July 7, 2011 Help! My iPod thinks I’m emo

Fixing music recommendation

Thursday, July 7, 2011 Fixing music recommendation Eliminating popularity bias and feedback loops

Adam Paul Tim Eric Jim Brian Aaron Peter Chris Liz Yury To d d Bethe Kirk Erik Jason Tristan Sue Jen To d d

Cari Scotty Erik Jason

Thursday, July 7, 2011 Fixing music recommendation Semantic-based recommendation

pop legend dance diva sexy american guilty pleasure

90s teen pop 00s rnb pop rock rock singer-songwriter dance-pop soul

emo hot alternative 90s

Thursday, July 7, 2011 Fixing music recommendation Where does this information come from?

Playlists The web Artist Bios Reviews

Blogs

Lyrics

pop legend dance diva Forums sexy american guilty Events pleasure 90s teen pop 00s rnb pop rock rock singer- songwriter dance-pop soul emo Social sites

hot alternative 90s

female fronted metal dark rock alternative goth metal metal goth rock emo gothic dark gothic rock heavy metal gothic metal hard rock melodic metal symphonic metal rock metal Crawler pop nu

Thursday, July 7, 2011 Fixing music recommendation Content-based recommendation

4 x 10 2

! -1

! -2 0 5 10 15 20 25 sec. BPM 190

143 114 96

72 60 0 5 10 15 20 25 sec. 1

0.8

0.6

0.4

0.2

0 60 72 96 114 143 BPM 190 240

Thursday, July 7, 2011 Content-based recommendation Using machines to listen to music

Perceptual features audio: Pattern layer linear ratio - time signature / tempo square ratio

- key/ mode Beat layer 1:4 - timbre 1:16

- pitch Segment layer ~1:3.5 - loudness ~1:12.25

- structure ~1:40 Frame layer ~1:1600

1:512 1:262144 Audio layer

4 x 10 2

1 m r o f

e 0 v a w ! -1

! -2 0 0.5 1 1.5 2 2.5 3 sec. 25 m a r 20 g o r t

c 15 e p s

d 10 n a b

- 5 5 2 1 0 0.5 1 1.5 2 2.5 3 sec. 1

0.8 s

Thursday, s July0.6 7, 2011 e n d

u 0.4 o l 0.2

0 0 0.5 1 1.5 2 2.5 3 sec. 1 n o i t

c 0.8 n u f 0.6 n o i t c

e 0.4 t e d 0.2 w a r 0 0 0.5 1 1.5 2 2.5 3 sec.

n 1 o i t c

n 0.8 u f

o 0.6 i t c e t

e 0.4 d

h t 0.2 o o m

s 0 0 0.5 1 1.5 2 2.5 3 sec. Hybrid Recommendation The best of all worlds

listener data Hybrid Recommender

user history preference editorial 35% 4% 62% 8% 60%

12% 18% 17% 34% 21% recommendations audio data 4% 49% 5% 48% 7% collaborative ﬁlterer artist audio analysis track web data content-based

random guessing is: adj Term K-L bits np Term K-L bits aggressive 0.0034 reverb 0.0064 softer 0.0030 the noise 0.0051 user a N a synthetic 0.0029 new wave 0.0039 KL = log punk 0.0024 elvis costello 0.0036 N (a + b) (a + c) sleepy 0.0022 the mud 0.0032 b N b funky 0.0020 his guitar 0.0029 + log noisy 0.0020 guitar bass and drums 0.0027 Ncultural(a + b) (b + d) angular 0.0016 instrumentals 0.0021 c N c acoustic 0.0015 melancholy 0.0020 + analysislog N (a + c) (c + d) romantic 0.0014 three chords 0.0019 d N d + log Table 2. Selected top-performing models of adjective and N (b + d) (c + d) noun phrase terms used to predict new reviews of music (3) with their corresemantic-basedsponding bits of information from the K-L distance measure. This measures the distance of the classifier away from a degenerate distribution; we note that it is also the mu- Thursday, July 7, 2011 tual information (in bits, if the logs are taken in base 2) 7.4. Review Regularization between the classifier outputs and the ground truth labels Many problems of non-musical text and opinion or per- they attempt to predict. sonal terms get in the way of full review understanding. A Table 2 gives a selected list of well-performing term similarity measure trained on the frequencies of terms in a models. Given the difficulty of the task we are encour- user-submitted review would likely be tripped up by obvi- aged by the results. Not only do the results give us term ously biased statements like “This record is awful” or “My models for audio, they also give us insight into which mother loves this album.” We look to the success of our terms and description work better for music understand- grounded term models for insights into the musicality of ing. These terms give us high semantic leverage without description and develop a ‘review trimming’ system that experimenter bias: the terms and performance were cho- summarizes reviews and retains only the most descriptive sen automatically instead of from a list of genres. content. The trimmed reviews can then be fed into fur- ther textual understanding systems or read directly by the listener. To trim a review we create a grounding sum term oper- 7.3. Automatic review generation ated on a sentence s of word length n,

n The multiplication of the term model c against the testing P (ai) g(s) = i=0 (4) gram matrix returns a single value indicating that term’s n relevance to each time frame. This can be used in re- where a perfectly grounded sentence (in which the predic- view generation as a confidence metric, perhaps setting a tive qualities of each term on new music has 100% preci- threshold to only allow high confidence terms. The vector sion) is 100%. This upper bound is virtually impossible in of term and confidence values for a piece of audio can also a grammatically correct sentence, and we usually see g(s) be fed into other similarity and learning tasks, or even a of 0.1% .. 10% . The user sets a threshold and the sys- natural language generation system: one unexplored pos- tem{ simply remo}ves sentences under the threshold. See sibility for review generation is to borrow fully-formed Table 3 for example sentences and their g(s). We see that sentences from actual reviews that use some amount of the rate of sentence recall (how much of the review is kept) terms predicted by the term models and form coherent varies widely between the two review sources; AMG’s re- paragraphs of reviews from this generic source data. Work views have naturally more musical content. See Figure 4 in language generation and summarization is outside the for recall rates at different thresholds of g(s). scope of this article but the results for the term prediction task and the below review trimming task are promising for these future directions. 7.5. Rating Regression One major caveat of our review learning model is its Lastly we consider the explicit rating categories provided time insensitivity. Although the feature space embeds time in the review to see if they can be related directly to the at different levels, there is no model of intra-song changes audio, or indeed to each other. Our first intuition is that of term description (a loud song getting soft, for example) learning a numerical rating from audio is a fruitless task as and each frame in an album is labeled the same during the ratings frequently reflect more information from out- training. We are currently working on better models of side the signal than anything observable in the waveforms. time representation in the learning task. Unfortunately, The public’s perception of music will change, and as a re- the ground truth in the task is only at the album level and sult reviews of a record made only a few months apart we are also considering techniques to learn finer-grained might wildly differ. In Figure 1 we see that correlation models from a large set of broad ones. of ratings between AMG and Pitchfork is generally low Counterpoint: Ignore content. Look at users instead

• Malcolm Slaney (Yahoo) • Using content hurts performance. • Evidence: Netﬂix competition.

Douglas Eck ([email protected]) / Google November 2009

Thursday, July 7, 2011 An email exchange on Music-IR

Douglas Eck ([email protected]) / Google November 2009

Thursday, July 7, 2011 Douglas Eck ([email protected]) / Google November 2009

Thursday, July 7, 2011 Anatomy of an Autotagger

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Our approach: content-based music recommendation

Acoustic feature extraction

Machine Learning

“I hear 1970s glam rock. It’s David Bowie, but with a harder punk edge, like the Clash, but wearing platform shoes and silk jumpsuits.”

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Our approach: content-based music recommendation

Acoustic feature extraction

Machine Learning

0.74 80s, 0.68 classic_rock 0.65 proto-punk A more realizable goal: generate tag clouds useful 0.71 glam 0.67 england 0.64 new_wave for annotation and 0.69 70s 0.65 english 0.64 glam_rock retrieval

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Recommendation from tags • Annotate all tracks using Autotagger model. • Use TF-IDF normalization to downweight overused words. • Cosine distance over word vectors for simliarity. • Combine autotag signal with other signals: • Social tags, • Explicit user preferences, • Implicit user preferences (skips, long plays) • Similarity among users, etc.

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 ML challenges and previous approaches

• Challenges • What features to use? • What machine learning algorithm to use? • How to scale to huge datasets?

• ML approaches (tag, genre and artist prediction): • SVM (Ellis & Mandel 2006) • Decision Trees (West, 2005) • Nearest Neighbors (Palmpalk, 2005) • Hierarchical Mixture Models (Turnbull et al, 2009) • AdaBoost / FilterBoost (our work)

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 One autotagging pipeline

1. Extract features 2. Train on labeled data 3. Predict unseen data

Training Training Waveform Features Tags Unseen Features

Feature Extractor Classiﬁer Predicted Tags Trained Model Feature (e.g. MFCC)

Thursday, July 7, 2011 Audio Feature Demos

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Curse of dimensionality

• A 3min stereo CD-quality audio sequence contains 254,016,000 bits (44100 * 2 * 60 * 3 * 16) • Number of possible unique bit conﬁgurations for 3min songs : 2254,016,000 • We need to process >100K audio ﬁles for lab work >1M for commercial work

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Representing different musical attributes

MFCC Timbre / instrumentation

Autocorrelation Temporal structure (rhythm, meter)

Spectrogram Pitch, melody Pink Floyd "Money"

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Aggregate Features Pink Floyd "Money" MFCCs

• Aggregate chunks of feature 1-second segments frames into longer-timescale segments

• Vote over these larger segments.

• Question: What is the best 4-second segments segment size?

• One answer: 3-5 seconds (Bergstra et.al.)

8-second segments

16-second segments

Thursday, July 7, 2011 Sparse coding techniques

• Example: K-Means Analysis. • Simpler than (but similar to) a Gaussian Mixture Model

Illustration of K-means from wikipedia

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Sparse coding techniques.

• Performed k-means on MFCCs • K=3000 / 20,000 30sec audio ﬁles • Used to build sparse representation of audio (Bengio et al; Google) • Song represented as a sparse histogram of frame centroids. Extremely sparse. • Motivation: sparse document similarity approaches. Can a single MFCC frame function as a concept? Song is a histogram of concepts.

Representation: [19=2, 722=2, 1387=1]

722 722 19 19 1387

Thursday, July 7, 2011 A more complete (and complex) example

/ , / , ,------7! -- : 1 I I I I I I I i I -----______1

:r------3 I I I I I I !..I ______J ______...... - ::::----- ______I .,;

I - - '" I·· ·101010111010101010101· · ·1 //

.. ·101010111010101010101· .. .. ·101010101110101010101· .. .. ·101010111010101010101· .. L . . ·101010101010101110101· .. .. ·101010121110101110101··· :: :w :. ::: :1 ,...,..... ,.....,-,.....,.....,-

[FIG4] Generating sparse codes from an "audio document," in four steps: 1) cochlea simulation, 2) stabilized auditory image creation, 3) sparse coding by vector quantization of multiscale patches, and 4) aggregation into a "bag of features" representation of the entire audio document. Steps 3 and 4 here correspond to the feature extraction module in the four- module system structure. To the fourth module, a PAMIR-based learning and retrieval system, this entire diagram represents a front end providing abstract sparse features for audio document characterization.

product of features times matrix times to another front-end representation vec- and applications. Each one can provide query. The matrix is trained to optimize tor quanitized mel-frequencyFrom cepstral “Machine ideas Hearing:and inspiration An for Emergingmachine hearing Field ” a ranking criterion, such that it attempts coefficients (MFCCs). TheyRichard all worked F. Lyon.techniques IEEE and Sig.applications. Proc. Most Mag. applica- Sept. 2010. to rank "relevant" documents higher, by fairly well, but the PAMIR technique was tions are trainable, based on "machine giving them a higher score, than "non- learning." The game is mostly about how Thursday, July 7, 2011 much faster to train, and the auditory relevant" ones, in the training set, for a image features gave the best perfor- to extract features, from images or sounds, large number of training queries that mance, if we increased the dimensional- that work well with machine learning sys- include multiword queries formed from ity by going to larger codebooks [12] . tems, and then train these systems to the tag vocabulary. We are presently doing experiments meet the needs of an application. The attractiveness of this approach with more challenging, but still con- Some learning systems work best with was that we could use PAMIR for our trolled, sound mixtures for which we fairly low feature dimensionality. ASR Stage 4, since it didn't contain anything have known text tags, constructed for systems typically use a 39-dimensional specific to images, and we could use a example by adding pairs of sound files MFCC-based feature vector, and learn dis- simple abstract VQ-based feature extrac- together, and finding that the auditory tributions in feature space as mixtures of tion for Stage 3, not tied to any particu- sparse-coding approach shows an advan- Gaussians. Other techniques, such as lar sound classes or ideas of where in the tage in interference. PAMIR from the vision field, deal best auditory image the important distin- with very high feature dimensionality and guishing infor.mation might be. We com- LEVERAGING MACHINE VISION don't try to model the distribution in fea- pared the PAMIR approach to other AND MACHINE LEARNING ture space. By paying attention to what trainable classifiers, support vector We have dozens of books with "machine machines and mixture of Gaussians, and vision" in the title, exploring techniques (continued on page 139)

IEEE SIGNAL PROCESSING MAGAZINE [135] SEPTEMBER 2010 Beat-based aggregation

Cheap to compute and popular (e.g. Dan Ellis cover song detector).

Thursday, July 7, 2011 Training data

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Data source: Last.FM

• Social tags obtained via data mining (Last.fm AudioScrobbler API) • Identiﬁed 350 most popular tags • Mined tags and tag frequencies for nearly 100,000s artist from Last.FM

• Genre, mood, instrumentation account for 77% of tags

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Thursday, July 7, 2011 Constructing datasets

• Built list of 350 most popular tags • Generate classiﬁcation targets for each tag: • All songs by top 10 artists for a tag used as positive examples • All songs by next 200 artists ignored (uncertain) • All remaining songs treated as negative examples • Matched songs to audio collection and extracted features from audio.

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Learning details and results

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Voting over blocks of features

• MFCCs calculated over timescale where audio should be steady-state (~100ms) • MFCCs aggregated into 3 to 5sec blocks (mean, std, covariance) • Train segments (columns) individually; all on same song-level label • Integrate predictions over song (vote) to choose winner

Target glam? glam? glam? ... Prediction -.397 .92 .74 Vote (average score for song) “Yeah! Glam.”

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Classiﬁer

• Used AdaBoost ensemble learner (Freund & Schapire 1995) • Basic idea: 1) Search for best weak learner in set of learners 2) Add it to list of active learners (store its weight and confidence) 3) Reweight data to avoid wasting resources on points already classified • Builds smart classifier from weighted linear combination of relatively- stupid “weak learners” • Feature selection based on minimization of empirical error

Douglas Eck ([email protected]) CCRMA MIR Workshop Day 5

Thursday, July 7, 2011 Principle of Adaboost

 Three cobblers with their wits combined equal Zhuge Liang the master mind.  Failure is the mother of success

Strong Weak classifier classifier Weight Features vector

From rii.ricoh.com/~liu/homepage/adaboost.ppt (Xu and Arun)

Thursday, July 7, 2011 Toy Example – taken from Antonio Torralba @MIT

Each data point has a class label: +1 ( ) yt = -1 ( )

and a weight:

wt =1

Weak learners from the family of lines

h => p(error) = 0.5 it is at chance From rii.ricoh.com/~liu/homepage/adaboost.ppt (Xu and Arun)