Christof Weiß

Ringvorlesung TechTalk . Physics Diploma Philosophische Fakultät, FAU, WS 2019/20 Universität Würzburg . Composition Neue Wege für die Musikforschung mittels HfM Würzburg Digitaler Signalverarbeitung . Ph. D. in Media Technology Fraunhofer IDMT, Ilmenau Christof Weiß, Meinard Müller . Postdoc in Music Processing & Composer International Audio Laboratories Erlangen [email protected], [email protected] AudioLabs / Erlangen-Nürnberg University . 2018: KlarText award for science communication

2

Harmony Analysis Harmony Analysis

. Different concepts . Different concepts . Concepts relate to different temporal granularity . Concepts relate to different temporal granularity

Movement level Global key C major Global key detection Movement level Global key C major Global key detection

Segment level C majorLocal key G major C major Local key detection Segment level C majorLocal key G major C major Local key detection

Chord level CM GM7 Am Chords Chord recognition Chord level CM GM7 Am Chords Chord recognition

Melody Melody Note level Middle voices Music transcription Note level Middle voices Music transcription Bass line Bass line

3 4

Harmony Analysis Harmony Analysis

. The Beatles, Let it be – Chords . Different concepts . Concepts relate to different temporal granularity

Movement level Global key C major Global key detection

Segment level C majorLocal key G major C major Local key detection

Chord level CM GM7 Am Chords Chord recognition

Melody Note level Middle voices Music transcription Source: www.ultimate-guitar.com Bass line

5 6 Harmony Analysis: Local Keys Harmony Analysis: Local Keys

. Johann Sebastian Bach, Choral “Durch Dein Gefängnis” . Johann Sebastian Bach, Choral “Durch Dein Gefängnis” (St. John’s Passion) – Local keys (St. John’s Passion) – Local keys

E maj B maj

E maj B maj E maj B maj E maj

Modulation E maj . Musical form:

Stollen Stollen Abgesang

„Bar form“

7 8

Harmony Analysis: Local Keys Harmony Analysis: Local Keys

. Johann Sebastian Bach, Choral “Durch Dein Gefängnis” . Johann Sebastian Bach, Choral “Durch Dein Gefängnis” (St. John’s Passion) – Local keys (St. John’s Passion) – Local keys

Circle of fifths Circle of fifths

C C E maj B maj F G E maj B maj F G Am Am Dm Em Dm Em B♭ D B♭ D Gm Bm Gm Bm

Series of fifths E♭ Cm ♭ ♯ F#m A Series of fifths E♭ Cm ♭ ♯ F#m A

Fm C#m Fm C#m A ♭ E A ♭ E B ♭ m E ♭m G#m B ♭ m E ♭m G#m D#m D#m D♭ B 0 diatonic D♭ B G ♭ G ♭ F# F#

9 10

Harmony Analysis: Local Keys Harmony Analysis: Local Keys

. Johann Sebastian Bach, Choral “Durch Dein Gefängnis” . Johann Sebastian Bach, Choral “Durch Dein Gefängnis” (St. John’s Passion) – Local keys (St. John’s Passion) – Local keys

Circle of fifths Circle of fifths

C C E maj B maj F G E maj B maj F G Am Am Dm Em Dm Em B♭ D B♭ D Gm Bm Gm Bm

Series of fifths E♭ Cm ♭ ♯ F#m A Series of fifths E♭ Cm ♭ ♯ F#m A

Fm C#m Fm C#m A ♭ E A ♭ E B ♭ m E ♭m G#m B ♭ m E ♭m G#m D#m D#m 1# diatonic D♭ B 2♭ diatonic D♭ B G ♭ G ♭ F# F#

11 12 Harmony Analysis: Local Keys Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Johann Sebastian Bach, Choral “Durch Dein Gefängnis” . Score – Piano reduction (St. John’s Passion) – Local keys

Circle of fifths

C F G 4# 5# Am Dm Em B♭ D Gm Bm

Series of fifths E♭ Cm ♭ ♯ F#m A

Fm C#m A ♭ E B ♭ m E ♭m G#m D#m D♭ B G ♭ F#

13 14

Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Audio – Waveform (Scholars Baroque Ensemble, Naxos 1994) . Audio – Spectrogram (Scholars Baroque Ensemble, Naxos 1994)

Time (seconds)

15 16

Visualization of Diatonic Scales Chroma Features

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Human perception of pitch is periodic . Audio – Chroma features (Scholars Baroque Ensemble, Naxos 1994) . Two components: tone height (octave) and chroma (pitch class)

Shepard’s helix of pitch Chromatic circle

17 18 Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Audio – Chroma features (Scholars Baroque Ensemble, Naxos 1994) . Chroma features – smoothing

19 20

Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Chroma features – smoothing . Chroma features – smoothing

21 22

Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Chroma features – smoothing . Re-ordering to perfect fifth series

23 24 Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Re-ordering to perfect fifth series . Diatonic Scales (7 fifths)

4#

25 26

Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Diatonic Scales (7 fifths) . Diatonic Scales – multiplication

5#

27 28

Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Diatonic Scales – multiplication . Diatonic Scales – multiplication

29 30 Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Diatonic Scales – multiplication . Diatonic Scales – shift to global key

4 # (E major)

4# 5# 4# 5# 4#

31 32

Visualization of Diatonic Scales Visualization of Diatonic Scales

. Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Example: J.S. Bach, Choral “Durch Dein Gefängnis” . Diatonic Scales – shift to global key . Diatonic Scales – relative [1] C. Weiß, J. Habryka, “Chroma-Based Scale Matching for Audio Tonality Analysis” Proc. Conference on Interdisciplinary Musicology, 2014.

Modulation

4 # (E major)

33 34

Visualization of Diatonic Scales Visualization of Diatonic Scales

. L. v. Beethoven – Sonata No. 10 op. 14 Nr. 2, 1. Allegro — 0 ≙ 1 . R. Wagner, Die Meistersinger von Nürnberg, Vorspiel — 0 ≙ 0 (Barenboim, EMI 1998) (Polish National Radio Symphony Orchestra, J. Wildner, Naxos 1993) Exposition Exposition Durchführung Reprise

sonata form

35 36 Cooperation: Musicology Cross-Version Analysis

. DFG-funded project: “Computer-Assisted Analysis of . 16 different performances (versions) Harmonic Structures” . Manual measure annotations for 3 versions . Harmony analysis and visualization No. Conductor Recording hh:mm:ss . 1st phase: 2014–2018, 2nd phase: 2019–2023 1 Barenboim 1991–92 14:54:55 2 Boulez 1980–81 13:44:38 3 Böhm 1967–71 13:39:28 . Partners: 1 2 3 4 4 Furtwängler 1953 15:04:22 . Rainer Kleinertz, Musicology Univ. Saarland 5 Haitink 1988–91 14:27:10 6 Janowski 1980–83 14:08:34 . Stephanie Klauk, Musicology Univ. Saarland 7 Karajan 1967–70 14:58:08 . Meinard Müller, AudioLabs FAU 8 Keilberth/Furtwängler 1952–54 14:19:56 . Christof Weiß, AudioLabs FAU 9 Krauss 1953 14:12:27 10 Levine 1987–89 15:21:52 11 Neuhold 1993–95 14:04:35 . Central work: Richard Wagner, Der Ring des Nibelungen 12 Sawallisch 1989 14:06:50 13 Solti 1958–65 14:36:58 14 Swarowsky 1968 14:56:34 . How is harmony organized at the large scale? 15 Thielemann 2011 14:31:13 16 Weigle 2010–12 14:48:46

37 38

Cross-Version Analysis Die Walküre WWV 86 B

Act 1 . 16 different performances (versions) . Manual measure annotations for 3 versions

Time (measures)

Act 2 Amplitude

Time (seconds)

. Visualize cross-version consistency with gray scale Act 3

39 40

Die Walküre WWV 86 B Die Walküre WWV 86 B

. Act 1, measures 955–1012 . Act 1, measures 955–1012 . Sieglinde’s narration . Sieglinde’s narration

41 42 Die Walküre WWV 86 B Die Walküre WWV 86 B

Act 1 . Act 3, measures 724–789 . Wotan’s punishment

Act 2

Act 3

43 44

Die Walküre WWV 86 B Die Walküre WWV 86 B A. Lorenz, 1924 . Act 3, measures 724–789 . Wotan’s punishment

45 46

Exploring Tonal-Dramatic Relationships Exploring Tonal-Dramatic Relationships

. Histograms of Analysis over time Die Walküre – Sword motif

Das Rheingold Die Walküre Siegfried Götterdämmerung WWV 86 A WWV 86 B WWV 86 C WWV 86 D 3897 measures 5322 measures 6682 measures 6040 measures Diatonic Scales

Diatonic Scales

Time (measures)

47 48 Exploring Tonal-Dramatic Relationships Exploring Tonal-Dramatic Relationships

Siegfried – Sword motif Das Rheingold – Valhalla motif

Diatonic Scales Chords

49 50

Exploring Tonal-Dramatic Relationships

Die Walküre – Valhalla motif Corpus Analysis: Composer Styles

Chords

51 52

Tonal Complexity Tonal Complexity

. Global chroma statistics (audio) . Global chroma statistics (audio) . 1783 – W. A. Mozart, „Linz“ symphony KV 425, 1. Adagio / Allegro . 1883 – J. Brahms, Symphony No. 3, 1. Allegro con brio (F major)

1 1

0.8 0.8

0.6 0.6

0.4 0.4 Salience Salience

0.2 0.2

0 0 Eb Bb F C G D A E B F# C# G# Ab Eb Bb F C G D A E B F# C# Pitch class Pitch class Circle of fifths → Circle of fifths →

53 54 Tonal Complexity Tonal Complexity

. Global chroma statistics (audio) . Realization of complexity measure Γ . 1940 – A. Webern, Variations for Orchestra op. 30 . Distribution over Circle of Fifths

1 𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 0 𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 1 𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 0.6

0.8

0.6

0.4 Salience

0.2

0 Ab Eb Bb F C G D A E B F# C# Pitch class Circle of fifths → 𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 1 𝐿𝑒𝑛𝑔𝑡ℎ . Relating to different time scales!

55 56

Tonal Complexity – Chords Analyzing Composer Styles

. 2000 pieces . Piano and Orchestra Γ music Complexity

1650 1700 1750 1800 1850 1900 1950 2000

57 58

Analyzing Composer Styles Analyzing Composer Styles

Haydn, Joseph 1732 – 1809 100 works in dataset

1 Γ 0.9 Complexity Global

Complexity Mid-scale 0.8 Complexity Local Complexity

0.7

1700 1750 1800 1850 1900 1950

1650 1700 1750 1800 1850 1900 1950 2000 1650 1700 1750 1800 1850 1900 1950 2000

59 60 Clustering Composition Years Clustering Composers

R. Schumann, Sonata No. 2 C. P. E. Bach, Sonata Wq. 49 op. 22, II. Andantino No. 3 in E minor, II. Adagio B. Glemser, Piano C. Colombo, Piano

1650 1700 1750 1800 1850 1900 1950 2000

61 62

Conclusions Conclusions

Interval types Chord Interval types Chord Chroma-based transitions Chroma-based transitions features Modulations features Modulations Visualization Visualization Chromagram Local keys Harmony analysis Chromagram Local keys Harmony analysis

Signal Processing & Composer clustering Music Analysis & Signal Processing & Composer clustering Music Analysis & Machine Learning Style analysis Music Theory Machine Learning Style analysis Music Theory

Spectrogram Diatonic scale Spectrogram Diatonic scale Evolution of relations Evolution of relations tonal features Musical form tonal features Musical form Tonal complexity Tonal complexity Engineering Musicology Engineering Musicology Computer Science Humanities Computer Science Humanities

„Computational“ phenomena!

63 64

Lecture: Summer Term 2020 Tonal Complexity: Jazz Solos

. Symbolic transcription

Digitale Musikanalyse: Wie gut können Computer hören? Dr. Christof Weiß Complexity . BA Digital Humanities and Social Sciences . Lecture + Exercises (5 SWS) Year . Summer term 2020 . Audio recording (harmonic part) . Time and place to be announced . No musical prerequisites! Complexity

Year 65 66 Tonal Complexity: Jazz Solos Clustering Composers

[7] Weiss / Balke / Abesser / Müller, Computational Corpus Analysis: A Case Study . Symbolic transcription on Jazz Solos, Proc. Int. Conference on Music Information Retrieval 2018 . Hierarchical clustering . From Bioinformatics Complexity

Year . Audio recording (harmonic part) Complexity

Year 67 68

Clustering Individual Pieces Tonal Complexity – Beethoven’s Sonatas

69 70

Signal Processing: Chroma Features Signal Processing: Chroma Features

. Example: C major scale (piano) . Example: C major scale (piano)

. Score . Audio – Chromagram

. Audio – Waveform Pitch class

. Audio - Spectrogram Time (seconds) . Audio – Chromagram (normalized) Frequency (Hz)

Time (seconds) Pitch class

71 Time (seconds)72 Music Genre Classification Music Genre Classification

Subgenre Categories:

Period / Era

J. S. Bach, L. van Beethoven, R. Schumann, A. Webern, Sub-era Brandenburg Concerto Fidelio, Overture, Sonata No. 2 op. 22, Variations for Orchestra op. 30 No. 2 in F major, I. Allegro, Slovak Philharm. II. Andantino Ulster Orchestra Cologne Chamber Orch. B. Glemser, Piano Composer

73 74

Music Genre Classification Music Genre Classification

. Typical approach: Supervised machine learning . Experimental design: Evaluation with Cross Validation (CV) . Separate data into different parts (folds)

Feature Dimensionality Classifier Training set extraction reduction training

Dataset

Feature Dimensionality . Distribution of classes balanced for all folds Test set Classification extraction reduction training test

Baroque Classical Romantic Modern

75 76

Classification Scenario Classification Scenario

. Dataset: CrossEraDB (Historical Periods) . Balanced Piano (p) – Orchestra (o) . Each 200 pieces → 1600 in total

16501700 1750 1800 1850 1900 1950 2000

77 1809 78 Classification Features Dimensionality Reduction

Standard Dim. Tonal Dim. . Reduce feature space to few dimensions (prevent curse of dimensionality) MFCC 16 Interval cat. 6 x 4 . Maximize separation of classes with Linear Discriminant Analysis (LDA) OSC 14 Triad types 4 x 4 . Using standard features (MFCC, spectral envelope, …) ZCR 1 Complexity 7 x 4 ASE 16 Chord progr. 11 x 5 SFM 16 SCF 16 SC 16 LogLoud 12 NormLoud 12

Sum 119 Sum 123 Mean & Std x 2 Mean & Std x 2 Total 238 Total 246

79 80

Dimensionality Reduction Dimensionality Reduction

. Reduce feature space to few dimensions . Reduce feature space to few dimensions . Maximize separation of classes with Linear Discriminant Analysis (LDA) . Maximize separation of classes with Linear Discriminant Analysis (LDA) . Using tonal features (interval, triad types, tonal complexity, … 4 time scales) . Using tonal & standard features

81 82

Classification methods Classification methods

. Gaussian Mixture Models (GMM) . Gaussian Mixture Models (GMM)

Singing Voice Singing Voice Accompaniment Accompaniment Unknown data Unknown data

𝑓 𝐱

𝑤 · 𝑝 𝐱𝜇Σ

Gauss components

Slides: Slides: Christian Dittmar Christian Dittmar 83 84 Classification Results Classification Results

. Gaussian Mixture Model (GMM) classifier, LDA reduction, 3-fold cross validation . Gaussian Mixture Model (GMM) classifier, LDA reduction, 3-fold cross validation

Full Dataset Piano Orchestra Full Dataset Piano Orchestra Standard features 87 % 88 % 85 % Standard features 87 % 88 % 85 % Tonal features 84 % 84 % 86 % Tonal features 84 % 84 % 86 % Combined 92 % 86 % 80 % Combined 92 % 86 % 80 %

Overfitting???

Weiss / Mauch / Dixon, Timbre-Invariant Audio Features Weiss / Mauch / Dixon, Timbre-Invariant Audio Features for Style Analysis of Classical Music, ICMC / SMC 2014 for Style Analysis of Classical Music, ICMC / SMC 2014

85 86

Classification Results Classification Results

. GMM classifier, LDA reduction, 3-fold cross validation . GMM classifier, LDA reduction, 3-fold cross validation . No composer filter Full Dataset Piano Orchestra Full Dataset Piano Orchestra Standard features 87 % 88 % 85 % Standard features 87 % 88 % 85 % Tonal features 84 % 84 % 86 % Tonal features 84 % 84 % 86 % Combined 92 % 86 % 80 % Combined 92 % 86 % 80 %

. Using composer filter training test Full Dataset Piano Orchestra Baroque Standard features 54 % 36 % 70 % Classical Romantic Tonal features 73 % 70 % 78 % Modern Combined 68 % 44 % 68 %

Flexer, A Closer Look on Artist Filters for Weiss / Müller, Tonal Complexity Features for Style Musical Genre Classification, ISMIR 2007 Classification of Classical Music, ICASSP 2015

87 88

Classification Results Classification Results

. GMM classifier, LDA reduction, 3-fold cross validation . What is actually learned? . No composer filter . Pay attention to: Full Dataset Piano Orchestra . Overfitting Standard features 87 % 88 % 85 % . „Curse of dimensionality“ – use dimensionality reduction techniques . Artist / album effects Tonal features 84 % 84 % 86 % . Evaluation: „Figures of merit“: Combined 92 % 86 % 80 % . Confusion matrix . Error examples: Consistently misclassified items . Using composer filter Full Dataset Piano Orchestra . Listening tests . Evaluation on unseen data (no cross validation) Standard features 54 % 36 % 70 % Tonal features 73 % 70 % 78 % Bob Sturm, Classification Accuracy is not enough, Combined 68 % 44 % 68 % Journal of Intelligent Information Systems, 2013

Weiss / Müller, Tonal Complexity Features for Style Classification of Classical Music, ICASSP 2015

89 90 Classification Results – Confusion Matrix Classification Results: Error Examples

. 80 tonal features, GMM with 1 Gaussian, LDA, composer filtering . 80 tonal features, GMM with 1 Gaussian, LDA . Full dataset . Look at consistently and persistently misclassified items . Mean accuracy: 75 % . Inter-class standard deviation: 6.7 %

91 92