about music technology mir applications research summary & credits
Music Information Retrieval Introduction, Technology, and Applications
alexander lerch
March 21, 2018
1 / 16 about music technology mir applications research summary & credits introduction about alexander lerch
education Electrical Engineering (Technical University Berlin) Tonmeister (University of Arts Berlin) professional Assistant Professor at the Georgia Tech Center for Music Technology previous: CEO at zplane.development research focus music information retrieval, audio content analysis intelligent music software
2 / 16 about music technology mir applications research summary & credits introduction music technology
what is music technology technology for creating, assessing, and consuming music
technology has influenced music for a long time new styles and genres crooning (microphone & amplification) rock (electric guitar) hip-hop (sampling) music production technology recording & reproduction quality new audio effects
3 / 16 about music technology mir applications research summary & credits introduction music technology
what is music technology technology for creating, assessing, and consuming music
technology has influenced music for a long time new styles and genres crooning (microphone & amplification) rock (electric guitar) hip-hop (sampling) music production technology recording & reproduction quality new audio effects
3 / 16 about music technology mir applications research summary & credits introduction music technology
what is music technology technology for creating, assessing, and consuming music
technology has influenced music for a long time new styles and genres crooning (microphone & amplification) rock (electric guitar) hip-hop (sampling) music production technology recording & reproduction quality new audio effects
3 / 16 about music technology mir applications research summary & credits introduction music technology
what is music technology technology for creating, assessing, and consuming music
technology has influenced music for a long time new styles and genres crooning (microphone & amplification) rock (electric guitar) hip-hop (sampling) music production technology recording & reproduction quality new audio effects
3 / 16 about music technology mir applications research summary & credits introduction music technology
what is music technology technology for creating, assessing, and consuming music
technology has influenced music for a long time new styles and genres crooning (microphone & amplification) rock (electric guitar) hip-hop (sampling) music production technology recording & reproduction quality new audio effects
3 / 16 about music technology mir applications research summary & credits music information retrieval overview
MIR: analysis and automatic extraction of any kind of information from the audio signal score info: melody, structure, instruments, chords, composer, . . . performance info: tempo, tuning, artist, . . . production info: sound engineer, equipment, . . . complex high-level info: style, mood, . . .
4 / 16 about music technology mir applications research summary & credits music information retrieval overview
MIR: analysis and automatic extraction of any kind of information from the audio signal score info: melody, structure, instruments, chords, composer, . . . performance info: tempo, tuning, artist, . . . production info: sound engineer, equipment, . . . complex high-level info: style, mood, . . .
4 / 16 about music technology mir applications research summary & credits music information retrieval overview
MIR: analysis and automatic extraction of any kind of information from the audio signal score info: melody, structure, instruments, chords, composer, . . . performance info: tempo, tuning, artist, . . . production info: sound engineer, equipment, . . . complex high-level info: style, mood, . . .
4 / 16 about music technology mir applications research summary & credits music information retrieval overview
MIR: analysis and automatic extraction of any kind of information from the audio signal score info: melody, structure, instruments, chords, composer, . . . performance info: tempo, tuning, artist, . . . production info: sound engineer, equipment, . . . complex high-level info: style, mood, . . .
4 / 16 about music technology mir applications research summary & credits music information retrieval overview
MIR: analysis and automatic extraction of any kind of information from the audio signal score info: melody, structure, instruments, chords, composer, . . . performance info: tempo, tuning, artist, . . . production info: sound engineer, equipment, . . . complex high-level info: style, mood, . . .
4 / 16 about music technology mir applications research summary & credits music information retrieval interdisciplinarity
machine learning and artificial intelligence data-driven approaches: classification, regression, clustering signal processing rule-based approaches: transforms, pattern recognition perception and cognition psycho-acoustics, perception of mood or similarity music theory rules and expectations: structure and repetition, chord progressions
5 / 16 about music technology mir applications research summary & credits music information retrieval related tasks & fields
audio forensics: crime scene reconstruction, crime detection context/activity detection: indoor/outdoor, meeting/working out environmental monitoring: noise pollution analysis acoustic fault detection: machine diagnosis through sound
speech & natural language processing: time-domain signal, high level meaning with structural properties 6 / 16 about music technology mir applications research summary & credits music information retrieval related tasks & fields
audio forensics: crime scene reconstruction, crime detection context/activity detection: indoor/outdoor, meeting/working out environmental monitoring: noise pollution analysis acoustic fault detection: machine diagnosis through sound
speech & natural language processing: time-domain signal, high level meaning with structural properties 6 / 16 about music technology mir applications research summary & credits music information retrieval differences to other fields analysis of music data in some ways unique: image & video: no/low res time domain 2-dimensional data objects don’t superpose but mask each other environmental audio: many non-pitched sources no structure, sources uncorrelated speech: low-complexity signal language models and meaning comparably clearly defined
7 / 16 about music technology mir applications research summary & credits music information retrieval history & growth
ISMIR: International Society for Music Information Retrieval (est. 2008)
academia increasing number of publications increasing number of society members rising impact factor of conference industry increasing number of high profile conference sponsorships sponsoring research growing number of start-ups aggressive hiring
8 / 16 about music technology mir applications research summary & credits music information retrieval history & growth
ISMIR: International Society for Music Information Retrieval (est. 2008)
academia increasing number of publications increasing number of society members rising impact factor of conference industry increasing number of high profile conference sponsorships sponsoring research growing number of start-ups aggressive hiring
8 / 16 about music technology mir applications research summary & credits music information retrieval history & growth
ISMIR: International Society for Music Information Retrieval (est. 2008)
academia increasing number of publications increasing number of society members rising impact factor of conference industry increasing number of high profile conference sponsorships sponsoring research growing number of start-ups aggressive hiring
8 / 16 about music technology mir applications research summary & credits music information retrieval general uses
learn: academic gain of knowledge listen: allow browsing and discovery produce: enhance music production intelligently educate: support music students interactively create: generate new content
9 / 16 about music technology mir applications research summary & credits music information retrieval application examples
fingerprinting: identify songs playing recommendation systems: playlist generation, music similarity automatic music coach: computer-assisted practice performance generator: render scores like a human music generation: generate sound tracks, elevator music
10 / 16 about music technology mir applications research summary & credits music information retrieval application examples
fingerprinting: identify songs playing recommendation systems: playlist generation, music similarity automatic music coach: computer-assisted practice performance generator: render scores like a human music generation: generate sound tracks, elevator music
10 / 16 about music technology mir applications research summary & credits music information retrieval application examples
fingerprinting: identify songs playing recommendation systems: playlist generation, music similarity automatic music coach: computer-assisted practice performance generator: render scores like a human music generation: generate sound tracks, elevator music
10 / 16 about music technology mir applications research summary & credits music information retrieval application examples
fingerprinting: identify songs playing recommendation systems: playlist generation, music similarity automatic music coach: computer-assisted practice performance generator: render scores like a human music generation: generate sound tracks, elevator music
10 / 16 about music technology mir applications research summary & credits music information retrieval application examples
fingerprinting: identify songs playing recommendation systems: playlist generation, music similarity automatic music coach: computer-assisted practice performance generator: render scores like a human music generation: generate sound tracks, elevator music
10 / 16 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: drum transcription
goal: detect all drum events
(CC) (HT) usage: (MT) (RC) rhythm analysis/modification (HH) (KD)
approach: (SD) (LT) Non-Negative Matrix Factorization advantages: low number of training samples computationally efficient competitive results
11 / 16 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: drum transcription
goal: (HH) (a) (KD) detect all drum events (SD) (b)
usage: (HH) (c) (SD) rhythm analysis/modification (KD)
Time in seconds (d)
approach: (HH) (SD) Non-Negative Matrix Factorization (KD) X X B G advantages: ≈ = ∙ low number of training samples computationally efficient 퐾 × 푀 퐾 × 푀 퐾 × 푅 푅 × 푀 competitive results B휏 G
= ∗
퐾 × 푅 × 푇 푅 × 푀 11 / 16
2 2.2 2.4 2.6 2.8 3 3.2 3.4 3.6 3.8 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: drum transcription
goal: (HH) (a) (KD) detect all drum events (SD) (b)
usage: (HH) (c) (SD) rhythm analysis/modification (KD)
Time in seconds (d)
approach: (HH) (SD) Non-Negative Matrix Factorization (KD) X X B G advantages: ≈ = ∙ low number of training samples computationally efficient 퐾 × 푀 퐾 × 푀 퐾 × 푅 푅 × 푀 competitive results B휏 G
= ∗
퐾 × 푅 × 푇 푅 × 푀 11 / 16
2 2.2 2.4 2.6 2.8 3 3.2 3.4 3.6 3.8 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: music performance assessment
goal: ??? assess and rate student music performances
L1 L2 L3
approach: feature design/learning & regression challenges: subjective human assessments different musical scores technical vs. artistic proficiency
12 / 16 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: music performance assessment
goal: Training Regression Model Feature Extraction (SVR) Audio 1. Baseline assess and rate student music performances Training Files 2. Score-independent 3. Score-based 4. Combination Outlier Removal
Audio Predicted approach: Feature Extraction Testing Files Assessments Testing (SVR) feature design/learning & regression challenges: subjective human assessments different musical scores technical vs. artistic proficiency
12 / 16 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: music performance assessment
goal: Training Regression Model Feature Extraction (SVR) Audio 1. Baseline assess and rate student music performances Training Files 2. Score-independent 3. Score-based 4. Combination Outlier Removal
Audio Predicted approach: Feature Extraction Testing Files Assessments Testing (SVR) feature design/learning & regression challenges: subjective human assessments different musical scores technical vs. artistic proficiency
12 / 16 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: music generation
goal: robot that can compose, improvise, and interact approach:
symbolic generation with DNNs youtu.be/l9OUbqWHOSk challenges: very style dependent learned model hard to understand/interpret evaluation nearly impossible
13 / 16 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: music generation
goal: robot that can compose, improvise, and interact approach:
symbolic generation with DNNs youtu.be/l9OUbqWHOSk challenges: very style dependent learned model hard to understand/interpret evaluation nearly impossible
13 / 16 about music technology mir applications research summary & credits music information retrieval Georgia Tech research examples: music generation
goal: robot that can compose, improvise, and interact approach:
symbolic generation with DNNs youtu.be/l9OUbqWHOSk challenges: very style dependent learned model hard to understand/interpret evaluation nearly impossible
13 / 16 about music technology mir applications research summary & credits music information retrieval summary MIR: young field with many interesting research questions unsolved tasks, e.g. music transcription source separation music generation current approaches only work to a certain degree feature learning DNN architectures (CNN, RNN, etc.) user-adaptivity, online learning challenges meaning in music: understanding high level concepts domain knowledge: how to enhance data-driven approaches data: how to deal with unlabeled/weakly labeled data potential commercialization still immature 14 / 16 about music technology mir applications research summary & credits music information retrieval summary MIR: young field with many interesting research questions unsolved tasks, e.g. music transcription source separation music generation current approaches only work to a certain degree feature learning DNN architectures (CNN, RNN, etc.) user-adaptivity, online learning challenges meaning in music: understanding high level concepts domain knowledge: how to enhance data-driven approaches data: how to deal with unlabeled/weakly labeled data potential commercialization still immature 14 / 16 about music technology mir applications research summary & credits music information retrieval summary MIR: young field with many interesting research questions unsolved tasks, e.g. music transcription source separation music generation current approaches only work to a certain degree feature learning DNN architectures (CNN, RNN, etc.) user-adaptivity, online learning challenges meaning in music: understanding high level concepts domain knowledge: how to enhance data-driven approaches data: how to deal with unlabeled/weakly labeled data potential commercialization still immature 14 / 16 about music technology mir applications research summary & credits music information retrieval summary MIR: young field with many interesting research questions unsolved tasks, e.g. music transcription source separation music generation current approaches only work to a certain degree feature learning DNN architectures (CNN, RNN, etc.) user-adaptivity, online learning challenges meaning in music: understanding high level concepts domain knowledge: how to enhance data-driven approaches data: how to deal with unlabeled/weakly labeled data potential commercialization still immature 14 / 16 about music technology mir applications research summary & credits music information retrieval summary MIR: young field with many interesting research questions unsolved tasks, e.g. music transcription source separation music generation current approaches only work to a certain degree feature learning DNN architectures (CNN, RNN, etc.) user-adaptivity, online learning challenges meaning in music: understanding high level concepts domain knowledge: how to enhance data-driven approaches data: how to deal with unlabeled/weakly labeled data potential commercialization still immature 14 / 16 about music technology mir applications research summary & credits GT Center for Music Technology music informatics group
other project examples: sample detection instrument recognition style transfer concert stitching ... musicinformatics.gatech.edu
Georgia Tech Center for Music Technology: BS, MS, & PhD programs research intensive some funded student GRA positions industry partners, good job placement
15 / 16 about music technology mir applications research summary & credits thank you!
contact Alexander Lerch: [email protected]
www.AudioContentAnalysis.org www.alexanderlerch.com
Georgia Tech Center for Music Technology gtcmt.gatech.edu
Music Informatics Group musicinformatics.gatech.edu 16 / 16