Music, Language and Computation

Aline Honingh LoLaCo Guestlecture 2012 Outline

• Music at the ILLC • Music and Language

• Music and Computation – Overview of the field – Research examples

Music at the ILLC

• Henkjan Honing – music cognition – Fleur Bouwer – Gabor Haden – Ashley Burgoyne – Berit Janssen – Carlos Vaquero • Rens Bod – computational linguistics/musicology • Aline Honingh – computational musicology – Bruno Rocha

• Monthly seminar/discussion group on music cognition and computation (email Fleur Bouwer for participation) Music & Language

Commonalities between language and music processing

• Perceived incremental • Alphabet – A to Z – A to G, #, b

• Syntax – Language: strong relation between syntax and meaning – Music: three layers • Scale degrees • Chord structure • Key structure

• Generative – Both use rule-governed combinations of notes/syllables to generate an unlimited number of signals

• Culturally transmitted – Learned by experience

• Transposable – Melody or sentence is the ``same’’ when song/spoken higher or lower

W.T. Fitch. The biology and evolution of music: A comparative Perspective. Cognition 100 (2006) 173-215 Hierarchical grouping in language and music? Groups in Language form a tree-structure (Wundt 1880) Grouping structure in music

• Grouping structure represents how parts combine compositionally and recursively into a whole Common representation

Claim • There exists one model that predicts the perceived structure in language, music, vision and other modalities… (cf. Newell 1999) Rhythm in Language and Music

• Does linguistic rhythm influences the rhythm of instrumental music? – English: stress timed – French: syllable timed

• nPVI: normalized pairwise variability index (Durational difference between successive elements)

A High nPVI

B Low nPVI nPVI of Britisch English and French sentences (Patel and Daniele 2003) nPVI of Britisch English and French musical themes (Patel and Daniele 2003)

Speech rhythm is reflected in musical rhythm

Melody in Language and Music

Music and Computation

The field of musicology

Traditionally, it consists of:

• Historical musicology – To understand musical works in their historical context • Ethnological musicology – Study of music in its cultural context • Music Theory and analysis – Study of how music works

1960s 1980s 2000

• Computational projects with an academic aim in music research started

– computers reduce the unmanageable mass of material

“Sweeping plans for an information revolution were made”

W. B. Hewlett and E. Selfridge-Field, Computing in Musicology, 1966-91. Computers and the Humanities, Vol. 25, No. 6. 1991 1960s 1980s 2000

This denouement of the promises of the Sixties had led by the early Eighties to widespread skepticism about computing in music scholarship.

W. B. Hewlett and E. Selfridge-Field, Computing in Musicology, 1966-91. Computers and the Humanities, Vol. 25, No. 6. 1991

… we seem to be without a sufficiently well-defined "theory" of music that could provide that logically consistent set of relationships between the elements which is necessary in order to program…

Vercoe, Review of „The Computer and Music“ by Harry B. Lincoln, Perspectives of New Music, 1971. 1960s 1980s 2000

• Rapid increase in digitization of music

Success story?

… has led to a greater visibility of musicology, especially outside the humanities.

Henkjan Honing

H. Honing, On the Growing Role of Observation, Formalization and Experimental Method in Musicology, Empirical Musicological Review, 1:1, 2006b,

However, scientific research about music has often happened outside of university music and musicology departments … Richard Parncutt

R. Parncutt, Systematic Musicology and the History and Future of Western Musical Scholarship, Journal of Interdisciplinary Music Studies, 1:1, 2007 Overview of musicology nowadays • Historical musicology • Ethnomusicology • Music theory and analysis

• Cognitive musicology – Studies musical habits of the mind • Music information retrieval – To design methods to retrieve musical information from large databases • Mathematical musicology – To formalize musical concepts using mathematics

All fields contain methods from computational musicology

Computational musicology

Useful for: – Musicology • Automatic analysis • Testing implicit knowledge

Computational musicology

Useful for: – Musicology • Automatic analysis • Testing implicit knowledge – Cognitive science • Understanding of cognitive processes through modelling

Computational musicology

Useful for: – Musicology • Automatic analysis • Testing implicit knowledge – Cognitive science • Understanding of cognitive processes through modelling – Commercial application • Search machines for music • Music recommendations (Apple’s Genius) • Music notation software (Finale etc.) • Genre classification • Song recognition (audio fingerprinting: Shazam)

Computational musicology

Testing implicit (musicology) knowledge through data-rich approach

Two examples Arch-shape of melodies (Huron 1996)

• Musicologists have noted arch shape of melodies • This hypothesis can be tested on large database – Essen Folk song collection is used; phrase boundaries are indicated – Humdrum software has been used

David Huron 1996. The melodic arch in Western folksongs. Computing in Musicology, Vol. 10, pp. 3-23. Arch-shape of melodies (Huron 1996)

Analysis 1: • Use pitch numbers: C4=0, C#4=1, … • All phrases of a certain note-length are averaged together • All averaged phrases are plotted

David Huron 1996. The melodic arch in Western folksongs. Computing in Musicology, Vol. 10, pp. 3-23.

Arch-shape of melodies (Huron 1996) Analysis 2: • For each phrase: – The first and the last pitch are determined – The middle pitches are averaged together • These three values make up nine different contours categories

contour type number of phrases percent ascending 6,983 19.4% descending 10,376 28.8% concave 3,496 9.7% convex 13,926 38.6% horizontal-ascending 181 0.5% horizontal-descending 439 1.2% ascending-horizontal 307 0.9% descending-horizontal 174 0.5% horizontal 193 0.5% TOTAL: 36,075 100%

David Huron 1996. The melodic arch in Western folksongs. Computing in Musicology, Vol. 10, pp. 3-23. Gapp fill

• Registral direction/ gapp fill: listeners expect a leap to be followed by a change in direction

Von Hippel, P. & Huron (2000). "Why do skips precede reversals? The effect of tessitura on melodic structure." Music Perception, Vol. 18, No.1, pp. 59-85. Gapp fill

• Test on data collection

Hypotheses: • Gapp fill • Regression towards the mean

• To test, four types of skips: – Median departing skips – Median crossing skips – Median landing skips – Median approaching skips

Von Hippel, P. & Huron (2000). "Why do skips precede reversals? The effect of tessitura on melodic structure." Music Perception, Vol. 18, No.1, pp. 59-85.

• Huron and Von Hippel (2000) showed that ‘regression to the mean’ fits data better • Gapp fill still used as explaining principle (Levitin 2006; Hodges & Sebald 2011 ) • Lessons: – Good model can be overruled by better model – Be carefull not to stick to the old (simple) model

Von Hippel, P. & Huron (2000). "Why do skips precede reversals? The effect of tessitura on melodic structure." Music Perception, Vol. 18, No.1, pp. 59-85. Computational musicology

Music recommendation

• User playlists – Others who liked this, also liked …. • Metadata – Other music by the same artist, genre, .. • Audio – Find a similar melody, rhythm, … Digital humanities project

Collaboration between university and company • Elephantcandy: Company specialized in audio applications for mobile devices

• People involved: – Aline Honingh – Bruno Rocha – Victor Bergen Henegouwen (Elephantcandy) – Niels Bogaards (Elephantcandy) Find similar music

Similar tempo

Similar timbre Electronic dance music APP Similar rhythm

…. Find similar music

Innovative character

• Similarity split into subsimilarities • Audio-based (no meta-data) • Focus on segments of songs Find similar music

Similar segments w.r.t. timbre