Workshop HfM Karlsruhe Music Information Retrieval

Harmony Analysis

Christof Weiß, Frank Zalkow, Meinard Müller International Audio Laboratories Erlangen

[email protected] [email protected] [email protected] Book: Fundamentals of Music Processing

Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: 978-3-319-21944-8 Springer, 2015

Accompanying website: www.music-processing.de Book: Fundamentals of Music Processing

Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: 978-3-319-21944-8 Springer, 2015

Accompanying website: www.music-processing.de Chapter 5: Chord Recognition

5.1 Basic Theory of Harmony 5.2 Template-Based Chord Recognition 5.3 HMM-Based Chord Recognition 5.4 Further Notes

In Chapter 5, we consider the problem of analyzing harmonic properties of a piece of music by determining a descriptive progression of chords from a given audio recording. We take this opportunity to first discuss some basic theory of harmony including concepts such as intervals, chords, and scales. Then, motivated by the automated chord recognition scenario, we introduce template-based matching procedures and hidden Markov models—a concept of central importance for the analysis of temporal patterns in time-dependent data streams including speech, gestures, and music. Dissertation: Tonality-Based Style Analysis

Christof Weiß Computational Methods for Tonality-Based Style Analysis of Classical Music Audio Recordings Dissertation, Ilmenau University of Technology, 2017

Chapter 5: Analysis Methods for Key and Scale Structures Chapter 6: Design of Tonal Features Recall: Chroma Features

. Human perception of pitch is periodic . Two components: tone height (octave) and chroma (pitch class)

Chromatic circle Shepard’s helix of pitch Recall: Chroma Features Chroma Salience / Likelihood

Time (seconds) Recall: Chroma Representations Salience / Likelihood

L. van Beethoven, Fidelio, Overture, Slovak Philharmonic Recall: Chroma Representations

. Orchestra

L. van Beethoven, Fidelio, Overture, Slovak Philharmonic

. Piano

Fidelio, Overture, arr. Alexander Zemlinsky M. Namekawa, D.R. Davies, piano four hands Tonal Structures

. Western music (and most other music): Different aspects of harmony . Referring to different time scales

Movement level Global key C major Global key detection

Segment level C majorLocal key G major C major Local key detection

Chord level CM GM7 Am Chords Chord recognition

Melody Note level Middle voices Music transcription Bass line Tonal Structures

. Western music (and most other music): Different aspects of harmony . Referring to different time scales

Movement level Global key C major Global key detection

Segment level C majorLocal key G major C major Local key detection

Chord level CM GM7 Am Chords Chord recognition

Melody Note level Middle voices Music transcription Bass line Chord Recognition

Source: www.ultimate-guitar.com Chord Recognition

Time (seconds) Chord Recognition Chord Recognition

Prefiltering Postfiltering ▪ Compression ▪ Smoothing ▪ Overtones ▪ Transition ▪ Smoothing ▪HMM

Audio Chroma Pattern Recognition representation representation matching result

Major triads CG Minor triads Chord Recognition: Basics

. Musical chord: Group of three or more notes . Combination of three or more tones which sound simultaneously . Types: triads (major, minor, diminished, augmented), seventh chords… . Here: focus on major and minor triads

→ C Major (C)

C Chord Recognition: Basics

. Musical chord: Group of three or more notes . Combination of three or more tones which sound simultaneously . Types: triads (major, minor, diminished, augmented), seventh chords… . Here: focus on major and minor triads

C Major (C)

C Minor (Cm)

. Enharmonic equivalence: 12 different root notes possible → 24 chords Chord Recognition: Basics

Chords appear in different forms: . Inversions

. Different voicings

. Harmonic figuration: Broken chords (arpeggio)

. Melodic figuration: Different melody note (suspension, passing tone, …) . Further: Additional notes, incomplete chords Chord Recognition: Basics

. Templates: Major Triads

CD ♭ DE ♭ EF G ♭ GA ♭ AB ♭ B

B A♯/B♭ A G♯/A♭ G F♯/G♭ F E D♯/E♭ D C♯/D♭ C Chord Recognition: Basics

. Templates: Major Triads

CD ♭ DE ♭ EF G ♭ GA ♭ AB ♭ B

B A♯/B♭ A G♯/A♭ G F♯/G♭ F E D♯/E♭ D C♯/D♭ C Chord Recognition: Basics

. Templates: Minor Triads

Cm C♯mDmE ♭mEmFmF♯mGmG♯mAmB ♭mBm

B A♯/B♭ A G♯/A♭ G F♯/G♭ F E D♯/E♭ D C♯/D♭ C Chord Recognition: Template Matching

CC♯ D…CmC♯mDm … Chroma vector 24 chord templates B 000…000… for each audio frame (12 major, 12 minor) A♯ 000…000… A 001…001… G♯ 010…010… Compute for each frame the G 100…100… similarity of the chroma vector F♯ 001…000… to the 24 templates F 010…001… E 100…010… D♯ 000…100… D 001…001… C♯ 010…010… C 100…100… Chord Recognition: Template Matching

. Similarity measure: Cosine similarity (inner product of normalized vectors)

Chord template:

Chroma vector:

Similarity measure: Chord Recognition: TemplateMatching

Chord Chroma Time (seconds) Chord Recognition: Label Assignment

CC♯ D…CmC♯mDm … Chroma vector 24 chord templates B 000…000… for each audio frame (12 major, 12 minor) A♯ 000…000… A 001…001… G♯ 010…010… Compute for each frame the G 100…100… similarity of the chroma vector F♯ 001…000… to the 24 templates F 010…001… E 100…010… D♯ 000…100… Assign to each frame the chord label D 001…001… of the template that maximizes the C♯ similarity to the chroma vector 010…010… C 100…100… Chord Recognition: Label Assignment Chord Chord

Time (seconds) Chord Recognition: Evaluation C G Am F C G F C

Time (seconds) Chord Recognition: Evaluation

. “No-Chord” annotations: not every frame labeled

. Different evaluation measures: . Precision: #TP 𝑃 #TP #FP . Recall: #TP 𝑅 #TP #FN

. F-Measure (balances precision and recall):

2⋅𝑃⋅𝑅 𝐹 𝑃𝑅

. Without “No-Chord” label: 𝑃𝑅𝐹 Chord Recognition: Smoothing

. Apply average filter of length 𝐿∈ℕ:

Time (seconds) Chord Recognition: Smoothing

. Apply average filter of length 𝐿∈ℕ:

Time (seconds) Chord Recognition: Smoothing

. Evaluation on all Beatles songs

0.8

0.75

0.7

0.65

0.6

0.55

F-measure 0.5

0.45 Binary templates 0.4

0.35 15913172125 Smoothing length Markov Chains

. Probabilistic model for sequential data . Markov property: Next state only depends on current state (no “memory”) . Consist of: . Set of states (hidden)

. State transition probabilities 0.8 . Initial state probabilities C

0.2 0.1

0.1 0.1 0.1 G F 0.7 0.3 0.6 Markov Chains Notation:

States 𝛼 for 𝑖∈1:𝐼

State transition probabilities 𝑎

A α1 α2 α3

α1 𝑎 𝑎 𝑎

α2 𝑎 𝑎 𝑎 α3 𝑎 𝑎 𝑎 0.8 Initial state probabilities 𝑐

C α1 α2 α3 C 𝑐 𝑐 𝑐 0.2 0.1 𝛽 for 𝑘∈1:𝐾 0.1 0.1 0.1 G F 0.7 0.3 0.6 Markov Chains

. Application examples: . Compute probability of a sequence using given a model (evaluation) . Compare two sequences using a given model . Evaluate a sequence with two different models (classification)

0.8

C

0.2 0.1

0.1 0.1 0.1 G F 0.7 0.3 0.6 Hidden Markov Models

. States as hidden variables

. Consist of: . Set of states (hidden)

. State transition probabilities

. Initial state probabilities 0.8

C

0.2 0.1

0.1 0.1 0.1 G F 0.7 0.3 0.6 Hidden Markov Models

. States as hidden variables

. Consist of: . Set of states (hidden)

. State transition probabilities

. Initial state probabilities 0.8

. Observations (visible) C

0.2 0.1

0.1 0.1 0.1 G F 0.7 0.3 0.6 Hidden Markov Models

. States as hidden variables

. Consist of: . Set of states (hidden) 0.9 0.2 0 . State transition probabilities 0.7 0.3 0 0 0.1 0.8 . Initial state probabilities 0.8

. Observations (visible) C

. Emission probabilities 0.2 0.1 0.1 0.1 0.1 G F 0.7 0.3 0.6 Hidden Markov Models Notation:

States 𝛼 for 𝑖∈1:𝐼

State transition probabilities 𝑎

A α1 α2 α3 0.9 0.2 α 𝑎 𝑎 𝑎 0 1 0.7 0.3 α 𝑎 𝑎 𝑎 2 0 0 0.1 0.8 α3 𝑎 𝑎 𝑎 0.8 Initial state probabilities 𝑐

C α1 α2 α3 C 𝑐 𝑐 𝑐 0.2 0.1 Observation symbols 𝛽 for 𝑘∈1:𝐾 Emission probabilities 𝑏 0.1 0.1 0.1

B β1 β2 β3 G F 0.7 0.3 0.6 α1 𝑏 𝑏 𝑏

α2 𝑏 𝑏 𝑏

α3 𝑏 𝑏 𝑏 Markov Chains

. Analogon: the student‘s life . Consists of: . Set of states (hidden)

. State transition probabilities 0.7 0.5 . Initial state probabilities

0.1 sleep coding 0.2 0.1 0.2 0.3 0 0.2 0.1 0.1 0.4 0.2 social eating 0.4 activity 0.2 0.3 Hidden Markov Models

. Analogon: the student‘s life Noise . Consists of: Light . Set of states (hidden) 0.5 0.4 0.3 . State transition probabilities 0.9 0.7 0.5 . Initial state probabilities

0.1 . Observations (visible) sleep coding 0.2 . Emission probabilities 0.1 0.2 0.3 0 0.2 0.1 0.1 0.4 0.2 social eating 0.3 0.4 activity 0.2 0.3 Smell 0.7 Hidden Markov Models

. Only observation sequence is visible! Different algorithmic problems: . Evaluation problem . Given: observation sequence and model . Calculate how well the model matches the sequence . Uncovering problem: . Given: observation sequence and model . Find: optimal hidden state sequence . Estimation problem („training“ the HMM): . Given: observation sequence . Find: model parameters . Baum-Welch algorithm (Expectation-Maximization) Uncovering problem

. Given: observation sequence 𝑂 𝑜,…,𝑜 of length 𝑁∈ℕand HMM 𝛩 (model parameters) ∗ ∗ ∗ . Find: optimal hidden state sequence 𝑆 𝑠,…,𝑠 . Corresponds to chord estimation task!

Observation sequence 𝑂 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜

β1 β3 β1 β3 β3 β2 Uncovering problem

. Given: observation sequence 𝑂 𝑜,…,𝑜 of length 𝑁∈ℕand HMM 𝛩 (model parameters) . Find: optimal hidden state sequence . Corresponds to chord estimation task!

Observation sequence 𝑂 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜

β1 β3 β1 β3 β3 β2

α1 α1 α1 α3 α3 α1

∗ ∗ ∗ ∗ ∗ ∗ ∗ Hidden state sequence 𝑆 𝑠 , 𝑠 , 𝑠 , 𝑠 , 𝑠 , 𝑠 Uncovering problem

. Given: observation sequence 𝑂 𝑜,…,𝑜 of length 𝑁∈ℕand HMM 𝛩 (model parameters) . Find: optimal hidden state sequence . Corresponds to chord estimation task!

Observation sequence 𝑂 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜

β1 β3 β1 β3 β3 β2

CCCGGC

∗ ∗ ∗ ∗ ∗ ∗ ∗ Hidden state sequence 𝑆 𝑠 , 𝑠 , 𝑠 , 𝑠 , 𝑠 , 𝑠 Uncovering problem

. Optimal hidden state sequence? . “Best explains” given observation sequence 𝑂 . Maximizes probability 𝑃 𝑂, 𝑆 Θ

Prob∗ max 𝑃 𝑂, 𝑆 Θ 𝑆∗ argmax 𝑃 𝑂, 𝑆 Θ

. Straight-forward computation (naive approach): . Compute probability for each possible sequence 𝑆 . Number of possible sequences of length 𝑁 (𝐼number of states):

𝐼·𝐼·… ·𝐼 𝐼 computationally infeasible! 𝑁 factors Viterbi Algorithm

. Based on dynamic programming (similar to DTW) . Idea: Recursive computation from subproblems . Use truncated versions of observation sequence

𝑂 1: 𝑛 ≔ 𝑜,…,𝑜 , length 𝑛∈1:𝑁

. Define 𝐃 𝑖, 𝑛 as the highest probability along a single state sequence 𝑠,…,𝑠 that ends in state 𝑠 𝛼

𝐃 𝑖, 𝑛 max 𝑃 𝑂1: 𝑛, 𝑠 ,…,𝑠,𝑠 𝛼 Θ ,… ,

. Then, our solution is the state sequence yielding

Prob∗ max𝐃 𝑖, 𝑁 ∈: Viterbi Algorithm

. 𝐃: matrix of size 𝐼𝑁 . Recursive computation of 𝐃 𝑖, 𝑛 along the column index 𝑛 . Initialization: . 𝑛1

. Truncated observation sequence: 𝑂 1𝑜

. Current observation: 𝑜 𝛽

𝐃 𝑖, 1 𝑐 ⋅𝑏 for some 𝑖∈1:𝐼 Viterbi Algorithm

. 𝐃: matrix of size 𝐼𝑁 . Recursive computation of 𝐃 𝑖, 𝑛 along the column index 𝑛 . Recursion: . 𝑛∈2:𝑁

. Truncated observation sequence: 𝑂 1: 𝑛 𝑜,…,𝑜

. Last observation: 𝑜 𝛽

∗ ∗ 𝐃 𝑖, 𝑛 𝑏 ⋅𝑎 ⋅𝑃 𝑂1: 𝑛 1, 𝑠 ,…,𝑠 𝛼 Θ for 𝑖∈1:𝐼 must be maximal!

∗ ∗ 𝐃 𝑖, 𝑛 𝑏 ⋅𝑎 ⋅ 𝐃 𝑗 ,𝑛1 Viterbi Algorithm

. 𝐃: matrix of size 𝐼𝑁 . Recursive computation of 𝐃 𝑖, 𝑛 along the column index 𝑛 . Recursion: . 𝑛∈2:𝑁

. Truncated observation sequence: 𝑂 1: 𝑛 𝑜,…,𝑜

. Last observation: 𝑜 𝛽

∗ ∗ 𝐃 𝑖, 𝑛 𝑏 ⋅𝑎 ⋅𝑃 𝑂1: 𝑛 1, 𝑠 ,…,𝑠 𝛼 Θ for 𝑖∈1:𝐼 must be maximal!

∗ ∗ 𝐃 𝑖, 𝑛 𝑏 ⋅𝑎 ⋅𝐃 𝑗 ,𝑛1

must be maximal (best index 𝑗∗)

𝐃 𝑖, 𝑛 𝑏 ⋅ max 𝑎 ⋅𝐃 𝑗,𝑛1 ∈: Viterbi Algorithm

∗ ∗ ∗ . 𝐃 given – find optimal state sequence 𝑆 𝑠,…,𝑠 ≔ 𝛼,…,𝛼 . Backtracking procedure (reverse order) . Last element: . 𝑛𝑁

. Optimal state: 𝛼

𝑖 argmax𝐃 𝑗, 𝑁 ∈: Viterbi Algorithm

∗ ∗ ∗ . 𝐃 given – find optimal state sequence 𝑆 𝑠,…,𝑠 ≔ 𝛼,…,𝛼 . Backtracking procedure (reverse order) . Further elements: . 𝑛𝑁1,𝑁2,…,1

. Optimal state: 𝛼

𝑖 argmax 𝑎 ⋅𝐃 𝑖, 𝑛 ∈: Viterbi Algorithm

∗ ∗ ∗ . 𝐃 given – find optimal state sequence 𝑆 𝑠,…,𝑠 ≔ 𝛼,…,𝛼 . Backtracking procedure (reverse order) . Further elements: . 𝑛𝑁1,𝑁2,…,1

. Optimal state: 𝛼

𝑖 argmax 𝑎 ⋅𝐃 𝑖, 𝑛 ∈:

. Simplification of backtracking: Keep track of maximizing index 𝑗 in

𝐃 𝑖, 𝑛 𝑏 ⋅max 𝑎 ⋅𝐃 𝑗, 𝑛 1 ∈: . Define 𝐼𝑁1matrix 𝐄:

𝐄 𝑖,𝑛 1 argmax 𝑎 ⋅𝐃 𝑗,𝑛1 ∈: Viterbi Algorithm Summary

Initialization

8

7

6 States 5 𝑖∈1:𝐼 4 𝐷 𝑖,1 𝑐𝑏 𝑖 𝑖𝑘1 3 …

2

1

1

Sequence index 𝑛 ∈ 1: 𝑁 Viterbi Algorithm Summary

Initialization Recursion

8 …

7 … … … …

6 States 5 … 𝑖∈1:𝐼 4 𝐷 𝑖,1 … …

𝑐𝑏 … … 𝑖 𝑖𝑘1 … 3 …

2 … 1

1 n – 1 n

Sequence index 𝑛 ∈ 1: 𝑁 Viterbi Algorithm Summary

Initialization Recursion

8 …

7 … … … …

6 States … 5 𝑏 𝑖∈1:𝐼 𝑖𝑘𝑛 4 𝐷 𝑖,1 … … 𝐷𝑖, 𝑛

𝑐𝑏 … … 𝑖 𝑖𝑘1 … 3 …

2 … 1

1 n – 1 n

Sequence index 𝑛 ∈ 1: 𝑁 Viterbi Algorithm Summary

Initialization Recursion

8 …

7 … … … …

6 States 𝐷𝑗, 𝑛 1 5 … 𝑖∈1:𝐼 𝑏𝑖𝑘 𝑎𝑗𝑖 𝑛 4 𝐷 𝑖,1 … … 𝐷𝑖, 𝑛

𝑐𝑏 … … 𝑖 𝑖𝑘1 … 3 …

2 … 1

1 n – 1 n

Sequence index 𝑛 ∈ 1: 𝑁 𝐷𝑖,𝑁

… … Termination … … … … … 𝐷𝑖, 𝑛 nN 𝑛 𝑘 𝑖

… 𝑏 … 𝑗𝑖 𝑎

Recursion … … 1 n –

… … indexSequence 𝑛 ∈ 1: 𝑁 𝐷𝑗, 1 𝑛 … … … 1 𝑘 … …𝑖 … 𝑏

𝑖 𝑖,1 𝑐 𝐷 1 8 7 6 5 4 3 2 1 Initialization Summary Viterbi Algorithm States 𝑖∈1:𝐼 𝐄 matrix Backtracking 𝐷𝑖,𝑁

… … Termination … … … … … 𝐷𝑖, 𝑛 nN 𝑛 𝑘 𝑖

… 𝑏 … 𝑗𝑖 𝑎

Recursion … … 1 n –

… … indexSequence 𝑛 ∈ 1: 𝑁 𝐷𝑗, 1 𝑛 … … … 1 𝑘 … …𝑖 … 𝑏

𝑖 𝑖,1 𝑐 𝐷 1 8 7 6 5 4 3 2 1 Initialization Summary Viterbi Algorithm States 𝑖∈1:𝐼 ·𝑁 𝐼·𝐼 𝐼 Total recursion: Total Per recursion step: recursion Per 𝐷𝑖, 𝑛 n 𝑛 𝑘 𝑖

… 𝑏 … 𝑗𝑖 𝑎

Recursion … … 1 n –

… … indexSequence 𝑛 ∈ 1: 𝑁 𝐼states 𝐼states 𝐷𝑗, 1 𝑛 Computational Complexity Viterbi Algorithm States 𝑖∈1:𝐼 Viterbi Algorithm Summary Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

α1 𝑎 𝑎 𝑎 α1 𝑏 𝑏 𝑏 𝑐 𝑐 𝑐

α2 𝑎 𝑎 𝑎 α2 𝑏 𝑏 𝑏

α3 𝑎 𝑎 𝑎 α3 𝑏 𝑏 𝑏 Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0

α3 0.1 0.3 0.6 α3 00.20.8 Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0

α3 0.1 0.3 0.6 α3 00.20.8

Input

Observation sequence

O = (o1,o2,o3,o4,o5,o6)

β1 β3 β1 β3 β3 β2 Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0

α3 0.1 0.3 0.6 α3 00.20.8

Input Viterbi algorithm

o = β o = β o = β o = β o = β o = β Observation sequence D 1 1 2 3 3 1 4 3 5 3 6 2 α O = (o ,o ,o ,o ,o ,o ) 1 1 2 3 4 5 6 α2

α3

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1

α2 β1 β3 β1 β3 β3 β2 α3 Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

Input α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0 o1 o2 o3 o4 o5 o6 α3 0.1 0.3 0.6 α3 00.20.8 β1 β3 β1 β3 β3 β2

Viterbi algorithm Initialization

D o = β o = β o = β o = β o = β o = β 1 1 2 3 3 1 4 3 5 3 6 2 𝐃 𝑖, 1 𝑐 ⋅𝑏 α1

α2

α3

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1

α2

α3 Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

Input α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0 o1 o2 o3 o4 o5 o6 α3 0.1 0.3 0.6 α3 00.20.8 β1 β3 β1 β3 β3 β2

Viterbi algorithm Initialization

D o = β o = β o = β o = β o = β o = β 1 1 2 3 3 1 4 3 5 3 6 2 𝐃 𝑖, 1 𝑐 ⋅𝑏 α1 0.4200

α2 0.0200

α3 0 Recursion

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 α 𝐃 𝑖, 𝑛 𝑏 ⋅max 𝑎 ⋅𝐃 𝑗, 𝑛 1 1 ∈: α2 α 3 𝐄 𝑖, 𝑛 1 argmax 𝑎 ⋅𝐃 𝑗,𝑛1 ∈: Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

Input α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0 o1 o2 o3 o4 o5 o6 α3 0.1 0.3 0.6 α3 00.20.8 β1 β3 β1 β3 β3 β2

Viterbi algorithm Initialization

D o = β o = β o = β o = β o = β o = β 1 1 2 3 3 1 4 3 5 3 6 2 𝐃 𝑖, 1 𝑐 ⋅𝑏 α1 0.4200 0.1008

α2 0.0200 0

α3 0 0.0336 Recursion

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 α 1 𝐃 𝑖, 𝑛 𝑏 ⋅max 𝑎 ⋅𝐃 𝑗, 𝑛 1 1 ∈: α2 1 α 1 3 𝐄 𝑖, 𝑛 1 argmax 𝑎 ⋅𝐃 𝑗,𝑛1 ∈: Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

Input α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0 o1 o2 o3 o4 o5 o6 α3 0.1 0.3 0.6 α3 00.20.8 β1 β3 β1 β3 β3 β2

Viterbi algorithm Backtracking

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2 α 0.4200 0.1008 0.0564 0.0135 0.0033 0 𝑖 argmax𝐃 𝑖, 𝑛 1 ∈: α2 0.0200 0 0.0010 0 0 0.0006

α3 0 0.0336 0 0.0045 0.0022 0.0003 𝑖 𝐄 𝑖,𝑛 E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 11111

α2 11113

α3 13133 Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

Input α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0 o1 o2 o3 o4 o5 o6 α3 0.1 0.3 0.6 α3 00.20.8 β1 β3 β1 β3 β3 β2

Viterbi algorithm Backtracking

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2 α 0.4200 0.1008 0.0564 0.0135 0.0033 0 𝑖 argmax𝐃 𝑖, 𝑛 1 ∈: α2 0.0200 0 0.0010 0 0 0.0006

α3 0 0.0336 0 0.0045 0.0022 0.0003 𝑖 𝐄 𝑖,𝑛 E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 11111 α2 11113i6 = 2 α3 13133 Viterbi Algorithm: Example

HMM: States Observation symbols 𝛼 for 𝑖∈1:𝐼 𝛽 for 𝑘∈1:𝐾

State transition probabilities Emission probabilities Initial state probabilities 𝑎 𝑏 𝑐

A α1 α2 α3 B β1 β2 β3 C α1 α2 α3

α1 0.8 0.1 0.1 α1 0.7 0 0.3 0.6 0.2 0.2

α2 0.2 0.7 0.1 α2 0.1 0.9 0

α3 0.1 0.3 0.6 α3 00.20.8

Input Viterbi algorithm Output

o = β o = β o = β o = β o = β o = β Observation sequence D 1 1 2 3 3 1 4 3 5 3 6 2 Optimal state sequence α 0.4200 0.1008 0.0564 0.0135 0.0033 0 O = (o ,o ,o ,o ,o ,o ) 1 S* = (α ,α ,α ,α ,α ,α ) 1 2 3 4 5 6 α2 0.0200 0 0.0010 0 0 0.0006 1 1 1 3 3 2

α3 0 0.0336 0 0.0045 0.0022 0.0003

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 11111 α2 11113i6 = 2 β1 β3 β1 β3 β3 β2 α3 13133 HMM: Application to Chord Recognition

. Effect of HMM-based chord estimation and smoothing:

C Dm G C

(a) Template Matching (frame-wise) (b) HMM

Time (seconds) Time (seconds) HMM: Application to Chord Recognition

. Parameters: Transition probabilities . Estimated from data i α State Log probability

State αj HMM: Application to Chord Recognition

. Parameters: Transition probabilities . Estimated from data Log probability Major chords Minor chords

Major chords Minor chords HMM: Application to Chord Recognition

. Parameters: Transition probabilities . Transposition-invariant Log probability Major chords Minor chords

Major chords Minor chords HMM: Application to Chord Recognition

. Parameters: Transition probabilities . Uniform transition matrix (only smoothing) i α State Log probability

State αj HMM: Application to Chord Recognition

. Evaluation on all Beatles songs

0.8

0.75

0.7

0.65

0.6

0.55

F-measure 0.5

0.45 binary 0.4 HMM 0.35 1 5 9 13172125 Smoothing length Chord Recognition: Further Challenges

. Chord ambiguities

Cmaj7 Am Cm C C A E♭ C C EG EG

B B Em Em

. Acoustic ambiguities (overtones) . Use advanced templates (model overtones, learned templates) . Enhanced chroma (logarithmic compression, overtone reduction) . Tuning inconsistency Chord Recognition: Public System

. Chord recognition . Typically: Feature extraction, pattern matching, filtering (HMM) . „Out-of-the-box“ solutions (Sonic Visualizer, Chordino plugin)

Sonic Visualizer, Chordino Vamp Plugin (Queen Mary University of London) Tonal Structures

Movement level Global key C major Global key detection

Segment level C majorLocal key G major C major Local key detection

Chord level CM GM7 Am Chords Chord recognition

Melody Note level Middle voices Music transcription Bass line Tonal Structures

Movement level Global key C major Global key detection

Segment level C majorLocal key G major C major Local key detection

Chord level CM GM7 Am Chords Chord recognition

Melody Note level Middle voices Music transcription Bass line Local Key Detection

. Key as an important musical concept (“Symphony in C major”) . Modulations → Local approach . Key relations: Circle of fifth (keys)

C F G Am Dm Em B♭ D Gm Bm

E♭ Cm ♭ ♯ F♯m A

Fm C♯m A♭ E B♭m E ♭m G♯m D♯m D♭ B G♭ F♯ Local Key Detection

. Key as an important musical concept (“Symphony in C major”) . Modulations → Local approach . Diatonic Scales . Simplification of keys . Perfect-fifth relation

Circle of fifths (pitches) →

2b diatonic0 diatonic1# diatonic Local Key Detection

. Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) . Score – Piano reduction Local Key Detection

. Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) . Audio – Waveform (Scholars Baroque Ensemble, Naxos 1994)

Time (seconds) Tonal Structures: Local Diatonic Scales

. Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) . Audio – Spectrogram (Scholars Baroque Ensemble, Naxos 1994) Local Key Detection: Chroma Features

. Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) . Audio – Chroma features (Scholars Baroque Ensemble, Naxos 1994) Local Key Detection: Chroma Smoothing

. Summarize pitch classes over a certain time

. Chroma smoothing . Parameters: blocksize b and hopsize h

b h b h b Local Key Detection: Chroma Smoothing

. Choral (Bach) Local Key Detection: Chroma Smoothing

. Choral (Bach) — smoothed with b = 42 seconds and h = 15 seconds Local Key Detection: Diatonic Scales

. Choral (Bach) — Re-ordering to perfect fifth series Local Key Detection: Diatonic Scales

. Choral (Bach) — Re-ordering to perfect fifth series Local Key Detection: Diatonic Scales

. Choral (Bach) — Diatonic Scale Estimation (7 fifths)

4# Local Key Detection: Diatonic Scales

. Choral (Bach) — Diatonic Scale Estimation (7 fifths)

5# Local Key Detection: Diatonic Scales

. Choral (Bach) — Diatonic Scale Estimation: Multiply chroma values* Local Key Detection: Diatonic Scales

. Choral (Bach) — Diatonic Scale Estimation: Multiply chroma values Local Key Detection: Diatonic Scales

. Choral (Bach) — Diatonic Scale Estimation Local Key Detection: Diatonic Scales

. Choral (Bach) — Diatonic Scale Estimation

4 # (E major) Local Key Detection: Diatonic Scales

. Choral (Bach) — Diatonic Scale Estimation: Shift to global key

4 # (E major) Local Key Detection: Diatonic Scales

. Choral (Bach) — 0 ≙ 4# Weiss / Habryka, Chroma-Based Scale Matching for Audio Tonality Analysis, CIM 2014 Local Key Detection: Examples

. L. v. Beethoven – Sonata No. 10 op. 14 Nr. 2, 1. Allegro — 0 ≙ 1 (Barenboim, EMI 1998) Local Key Detection: Examples

. R. Wagner, Die Meistersinger von Nürnberg, Vorspiel — 0 ≙ 0 (Polish National Radio Symphony Orchestra, J. Wildner, Naxos 1993) DFG-funded Project: Computational Analysis of Harmonic Structures

. With Prof. Rainer Kleinertz, Musicology, Uni Saarland

. Richard Wagner, Der Ring des Nibelungen – Four operas, up to 15 hours of music – How is harmony organized at the large scale? – Analyses by A. Lorenz 1924 – Hypothesis of „Poetico-musical periods“

Source: Wikipedia Cross-Version Analysis

. Up to 18 versions . 3 versions manually annotated

No. Conductor Recording hh:mm:ss 1 Barenboim 1991–92 14:54:55 2 Boulez 1980–81 13:44:38 1 2 3 4 3 Böhm 1967–71 13:39:28 4 Furtwängler 1953 15:04:22 5 Haitink 1988–91 14:27:10 6 Janowski 1980–83 14:08:34 7 Karajan 1967–70 14:58:08 8 Keilberth/Furtwängler 1952–54 14:19:56 9 Krauss 1953 14:12:27 10 Levine 1987–89 15:21:52 11 Neuhold 1993–95 14:04:35 12 Sawallisch 1989 14:06:50 13 Solti 1958–65 14:36:58 14 Swarowsky 1968 14:56:34 15 Thielemann 2011 14:31:13 16 Weigle 2010–12 14:48:46 Cross-Version Analysis

. Idea: Use analysis results based on different interpretations (versions) . Tonal characteristics should not depend on interpretation → Test reliability of the method

. Visualize consistency with gray scheme Die Walküre WWV 86 B

Act 1

Act 2

Act 3 Die Walküre WWV 86 B

Act 1 Die Walküre WWV 86 B

. Act 1, measures 955–1012 . Sieglinde´s narration Die Walküre WWV 86 B

. Act 1, measures 955–1012 . Sieglinde´s narration Die Walküre WWV 86 B

Act 1

Act 2

Act 3 Die Walküre WWV 86 B

. Act 3, measures 724–789 . Wotan´s punishment Die Walküre WWV 86 B

. Act 3, measures 724–789 . Wotan´s punishment Die Walküre WWV 86 B Exploring Tonal-Dramatic Relationships

. Histograms of Analysis over time

Das Rheingold Die Walküre Siegfried Götterdämmerung WWV 86 A WWV 86 B WWV 86 C WWV 86 D 3897 measures 5322 measures 6682 measures 6040 measures Diatonic Scales

Time (measures) Exploring Tonal-Dramatic Relationships

Sword motif – Die Walküre

Diatonic Scales Exploring Tonal-Dramatic Relationships

Sword motif – Siegfried

Diatonic Scales Exploring Tonal-Dramatic Relationships

Valhalla motif – Das Rheingold

Chords Exploring Tonal-Dramatic Relationships

Valhalla motif – Die Walküre

Chords Tonal Structures: Complexity

. Global chroma statistics (audio) . 1567 – G. da Palestrina, Missa de Beata Virgine, Credo

1

0.8

0.6

0.4 Salience

0.2

0 Eb Bb F C G D A E B F# C# G# Pitch class Circle of fifths → Tonal Structures: Complexity

. Global chroma statistics (audio) . 1725 – J. S. Bach, Orchestral Suite No. 4 BWV 1069, 1. Ouverture (D major)

1

0.8

0.6

0.4 Salience

0.2

0 F C G D A E B F# C# G# D# A# Pitch class Circle of fifths → Tonal Structures: Complexity

. Global chroma statistics (audio) . 1783 – W. A. Mozart, „Linz“ symphony KV 425, 1. Adagio / Allegro (C major)

1

0.8

0.6

0.4 Salience

0.2

0 Eb Bb F C G D A E B F# C# G# Pitch class Circle of fifths → Tonal Structures: Complexity

. Global chroma statistics (audio) . 1883 – J. Brahms, Symphony No. 3, 1. Allegro con brio (F major)

1

0.8

0.6

0.4 Salience

0.2

0 Ab Eb Bb F C G D A E B F# C# Pitch class Circle of fifths → Tonal Structures: Complexity

. Global chroma statistics (audio) . 1940 – A. Webern, Variations for Orchestra op. 30

1

0.8

0.6

0.4 Salience

0.2

0 Ab Eb Bb F C G D A E B F# C# Pitch class Circle of fifths → Tonal Structures: Complexity

. Realization of complexity measure Γ . Entropy / Flatness measures . Distribution over Circle of Fifths

Γ0 Γ1 0Γ1

length 𝑟 Γ 1𝑟 . Relating to different time scales! Tonal Structures: Complexity Γ Complexity

Weiss / Müller, Quantifying and Visualizing Tonal Complexity, CIM 2014 Tonal Structures: Complexity

L. van Beethoven Sonata Op. 2, No. 3 1st movement Tonal Structures: Complexity

Op. 2, No. 3 Op. 57, No. 1 Op. 106, No. 1 „Appassionata“ „Hammerklavier“