Computational Methods for Tonality-Based Style Analysis of Classical Music Audio Recordings
Dipl.-Phys. Diplommusiker Christof Weiß
Promotion an der Fakultät für Elektrotechnik und Informationstechnik der TU Ilmenau Öffentlicher Teil der wissenschaftlichen Aussprache
Ilmenau, 03.04.2017
Slide 1 Acknowledgements
Slide 2 Music Genre Classification
Subgenre Categories:
Period / Era
Sub-era
Composer
Slide 3 Music Genre Classification
¢ Standard approach ( content-based ) ¢ Supervised machine learning ¢ Based on spectral / timbral features
¢ In classical music → Instrumentation
¢ Better categories? ¢ Musical style ¢ Independent from instrumentation ¢ → Tonality / Harmony
Slide 4 Outline
1. Motivation 2. Tonal Audio Features 3. Subgenre Classification 4. Musical Style Analysis 5. Conclusions
Slide 5 Tonal Structures
Movement level Global key C major Key detection Segment level C majorLocal key G major C major
Chord level CM GM 7 Am Chords Chord recognition
Melody Note level Middle voices Music transcription Bass line
Slide 6 Chroma Representations Salience / Likelihood
L. van Beethoven, Fidelio , Overture, Slovak Philharmonic
Slide 7 Chroma Representations
¢ Orchestra
L. van Beethoven, Fidelio , Overture, Slovak Philharmonic
¢ Piano
Fidelio , Overture, arr. Alexander Zemlinsky M. Namekawa, D.R. Davies, piano four hands
Slide 8 Chroma Representations
¢ Orchestra
L. van Beethoven, Fidelio , Overture, Slovak Philharmonic
[1] Gómez, Tonal Description of Polyphonic Audio , PhD thesis, Barcelona 2006
[2] Müller / Ewert, Towards Timbre-Invariant Audio Features for Harmony-Based Music , IEEE TASLP, 2010
[3] Mauch / Dixon, Approximate Note Transcription for the Improved Identification of Difficult Chords , ISMIR 2010
Slide 9 Tonal Structures: Chords
¢ Chord recognition ¢ Typically: Feature extraction, pattern matching, filtering (HMM) ¢ „Out-of-the-box“ solutions
Sonic Visualizer, Chordino Vamp Plugin (Queen Mary University of London)
Slide 10 Tonal Structures: Chord & Interval Categories
¢ Chromagram Salience Pitchclass
Time (seconds) ¢ Chord type Salience Chord type
Time (seconds) ¢ Interval categories Salience Interval cateogry
Time (seconds)
Slide 11 Tonal Structures: Final Chord
¢ Global key detection – typical approach: ¢ Full piece chroma statistics ¢ Template matching
¢ Idea: Use particular role of final chord
¢ Full piece chroma statistics → Diatonic scale [4] Weiss, Global Key Extraction Based on the Final Chord , SMC 2013 [5] Weiss / Schaab, On the Impact of Key Detection Performance , ISMIR 2015
Slide 12 Tonal Structures: Final Chord
¢ State-of-the-art ¢ Learning of pitch class profiles ¢ Weighting of beginning and ending sections (15 seconds)
[6] Van de Par et al., Musical Key Extraction from Audio Using Profile Training , ISMIR 2006
¢ Evaluation with optimized parameters ¢ Dataset: symphonies, piano, chamber music, 478 pieces ¢ Final chord (own): 94 % ¢ Profile learning (Van de Par ): 92 %
¢ Evaluation on unseen data ¢ Dataset: orchestra, piano, 1200 pieces ¢ Final chord (own): 85 % ¢ Profile learning (Van de Par ): 87 %
Slide 13 Tonal Structures: Local Diatonic Scales
¢ Modulations → Local approach
¢ Diatonic Scales ¢ Simplification of keys ¢ Perfect-fifth relation
Circle of fifths →
+1 diatonic level
0 diatonic level
-2 diatonic level
Slide 14 Tonal Structures: Local Diatonic Scales
¢ J. S. Bach, Choral „Durch Dein Gefängnis“ ( Johannespassion ), E major
¢ Relative diatonic level The Scholars Baroque Ensemble, NAXOS 1993
E major (4#)
[7] Weiss / Habryka, Chroma-Based Scale Matching for Audio Tonality Analysis , CIM 2014
Slide 15 Tonal Structures: Complexity
¢ Global chroma statistics (audio) ¢ 1783 – W. A. Mozart, „Linz“ symphony KV 425, 1. Adagio / Allegro (C major)
1
0.8
0.6
0.4 Salience
0.2
0 Eb Bb F C G D A E B F# C# G# Pitch class Circle of fifths →
Slide 16 Tonal Structures: Complexity
¢ Global chroma statistics (audio) ¢ 1883 – J. Brahms, Symphony No. 3, 1. Allegro con brio (F major)
1
0.8
0.6
0.4 Salience
0.2
0 Ab Eb Bb F C G D A E B F# C# Pitch class Circle of fifths →
Slide 17 Tonal Structures: Complexity
¢ Global chroma statistics (audio) ¢ 1940 – A. Webern, Variations for Orchestra op. 30
1
0.8
0.6
0.4 Salience
0.2
0 Ab Eb Bb F C G D A E B F# C# Pitch class Circle of fifths →
Slide 18 Tonal Structures: Complexity
¢ Realization of complexity measure Γ ¢ Entropy / Flatness measures ¢ Distribution over Circle of Fifths Γ 0 Γ 1 0 Γ 1
length Γ 1 ¢ Relating to different time scales!
Slide 19 Tonal Structures: Complexity Γ Complexity
[8] Weiss / Müller, Quantifying and Visualizing Tonal Complexity , CIM 2014
Slide 20 Outline
1. Motivation 2. Tonal Audio Features 3. Subgenre Classification 4. Musical Style Analysis 5. Conclusions
Slide 21 Classification Scenario
¢ Approach: Supervised Machine Learning
Feature Dimensionality Classifier Training set extraction reduction training
Dataset
Feature Dimensionality Test set Classification extraction reduction
Slide 22 Classification Scenario
¢ Dataset: CrossEraDB (Historical Periods) ¢ Balanced Piano (p) – Orchestra (o) ¢ Each 200 pieces → 1600 in total
Slide 23 Classification Scenario
16501700 1750 1800 1850 1900 1950 2000
Slide 24 Classification Scenario
16501700 1750 1800 1850 1900 1950 2000 1809
Slide 25 Dimensionality Reduction
¢ Reduce feature space to few dimensions ¢ Maximize separation of classes with Linear Discriminant Analysis (LDA) ¢ Using standard features (MFCC, spectral envelope, …)
Slide 26 Dimensionality Reduction
¢ Reduce feature space to few dimensions ¢ Maximize separation of classes with Linear Discriminant Analysis (LDA) ¢ Using tonal features (interval, triad types, tonal complexity, … 4 time scales)
Slide 27 Dimensionality Reduction
¢ Reduce feature space to few dimensions ¢ Maximize separation of classes with Linear Discriminant Analysis (LDA) ¢ Using tonal & standard features
Slide 28 Classification Results
¢ GMM classifier, LDA reduction, 3-fold cross validation
Full Piano Orchestra Dataset Standard features 87 % 88 % 85 % Tonal features 84 % 84 % 86 % Combined 92 % 86 % 80 %
[9] Weiss / Mauch / Dixon, Timbre-Invariant Audio Features for Style Analysis of Classical Music , ICMC / SMC 2014
Slide 29 Classification Results
¢ GMM classifier, LDA reduction, 3-fold cross validation
Full Piano Orchestra Dataset Standard features 87 % 88 % 85 % Tonal features 84 % 84 % 86 % Combined 92 % 86 % 80 %
Baroque Classical Romantic Modern
training test
Slide 30 Classification Results
¢ GMM classifier, LDA reduction, 3-fold cross validation ¢ No composer filter Full Piano Orchestra Dataset Standard features 87 % 88 % 85 % Tonal features 84 % 84 % 86 % Combined 92 % 86 % 80 %
¢ Using composer filter Full Piano Orchestra Dataset Standard features 54 % 36 % 70 % Tonal features 73 % 70 % 78 % Combined 68 % 44 % 68 %
[10] Weiss / Müller, Tonal Complexity Features for Style Classification of Classical Music , ICASSP 2015
Slide 31 Classification Results – Summary
¢ Different types of tonal features
¢ Combination of time scales
¢ Classifiers (SVM, Random Forest)
¢ State-of-the-art ¢ Few studies on audio ¢ Good separation of tonal-vs.-atonal ( 91 %):
[11] Izmirli, Tonal-Atonal Classification of Music Audio Using Diffusion Maps , ISMIR 2009
¢ Composer Identification ¢ Up to 78 % for 11 composers
[12] Hamel, Pooled Features Classification , MIREX 2011
Slide 32 Outline
1. Motivation 2. Tonal Audio Features 3. Subgenre Classification 4. Musical Style Analysis 5. Conclusions
Slide 33 Musical Style Analysis
1650 1700 1750 1800 1850 1900 1950 2000
Slide 34 Musical Style Analysis
1 Γ 0.9 Complexity Global
Complexity Mid- 0.8 scale
Complexity Complexity Local 0.7
1700 1750 1800 1850 1900 1950
1650 1700 1750 1800 1850 1900 1950 2000
Slide 35 Clustering Composers
1650 1700 1750 1800 1850 1900 1950 2000
Slide 36 Clustering Composers
1650 1700 1750 1800 1850 1900 1950 2000
Slide 37 Clustering Composers
1650 1700 1750 1800 1850 1900 1950 2000
Slide 38 Clustering Composers
1650 1700 1750 1800 1850 1900 1950 2000
Slide 39 Outline
1. Motivation 2. Tonal Audio Features 3. Subgenre Classification 4. Musical Style Analysis 5. Conclusions
Slide 40 Conclusions
Machine Learning
Subgenre classification Classification Signal Processing & Clustering Composer identification Chroma-based features
Feature Extraction Musical style Key detection analysis Interval categories Modulations Tonal complexity Music Analysis Diatonic scale Chord relations progressions
Music Theory
Slide 41