State of the Art and Future Directions in Musical Sound Synthesis
Xavier Serra Music Technology Group (http://mtg.upf.edu) Universitat Pompeu Fabra, Barcelona [email protected]
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Index
Research context Brief history Current trends or Open issues Conclusions: Beyond sound synthesis
MMSP 2007 Invited talk Chania, Oct. 1st 2007
1 Context: Sound and Music Computing Research
SMC research approaches the whole sound and music communication chain from a multidisciplinary point of view. By combining scientific, technological and artistic methodologies it aims at understanding, modeling and generating sound and music through computational approaches. http://www.soundandmusiccomputing.org
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Context: SMC Research areas
from the SMC roadmap
MMSP 2007 Invited talk Chania, Oct. 1st 2007
2 Context: SMC Research - Sound
Sound Description/Understanding Perceptual versus motor-based models Sound source recognition and classification Sound search and retrieval based on content Sound Synthesis/Processing Interaction-centered sound modeling Physical modeling based on data Harmonic analysis from MTG-UPF analysis Audio content processing
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Context: SMC Research - Interaction
Music Interfaces Multimodal music interfaces Integration of control and sound generation Feedback systems Effective interaction metaphors Mobile music Performance Modeling and Control Understanding music performance Models for artistic music performance Multi-modal processing of expression Interaction in multimedia applications Sound Interaction Design Evaluation of for sound design Reactable by MTG-UPF Everyday listening systems Soundscape Design
MMSP 2007 Invited talk Chania, Oct. 1st 2007
3 Context: SMC Research - Music
Music Description and Understanding `Narrow' SMC vs. multidisciplinarity research Reductionist vs. multi-dimensional models Bottom-up vs. top-down modeling Understanding the music signal vs. understanding MUSIC
Music Generation Modeling Computational models Computer-assisted composition tools MusicSurfer by MTG-UPF Notation and multiple interfaces
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Brief History: The origins
Max Mathews in Science 1963: “.. generating sounds from numbers is a completely general way to synthesize sound because the bandwidth and dynamic range of hearing are bounded and therefore any sound we perceive may be generated in this way.”
Max Mathews in Technology of Computer Music 1969: “The two fundamental problems in sound synthesis are (1) the vast amount of data needed to specify a The GROOVE System pressure function - hence the necessity of a very at the Bell Telephone Labs, fast program - and (2) the need for a simple and c1970 powerful language in which to describe a complex sequence of sounds”.
MMSP 2007 Invited talk Chania, Oct. 1st 2007
4 Brief History: First algorithms
Sinusoidal wavetable
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Brief History: FM synthesis - DX7
DX7 by Yamaha, from 1983
MMSP 2007 Invited talk Chania, Oct. 1st 2007
5 Brief History: Granular synthesis
“All sound is an integration of grains, of elementary sonic particles, of sonic quanta” Xenakis (1971).
Helmuth’s example
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Brief History: Sampling - Fairlight
Fairlight, from 1980 Pioneered two innovations that transformed music making, namely sampling and sequencing.
MMSP 2007 Invited talk Chania, Oct. 1st 2007
6 Brief History: Physical modeling - VL1
VL1 by Yamaha, from 1994
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Brief History: Spectral modeling - Vocaloid
Vocaloid, released in 2005
MMSP 2007 Invited talk Chania, Oct. 1st 2007
7 Current Trends: Hybrid systems
Commuted waveguide synthesis
Spectral-based excitation plus resonance model
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Current Trends: Concatenative synthesis
A possible sounds produced by the instrument B sounds produced by the Spectral Shape performer playing Interpolation Factor SPP Spectrum n-i
recorded audio samples Frame n-k Frame n-1 Frame n Spectral Interpolation zone Shape k frames Interpolation Factor Instrument sonic space (SSIntp)
Stretch Stretch
1 stretched Left EpR Right EpR differential spectrum envelope Timbre f Envelope
F3 n-1 f
F2 n-1 F3 n-1
F1 n-1 F2 n-1
F1 n-1 F0 n-1 1-SSIntp f F0 F1 F2 F3 n-1 n-1 n-1 n-1 F0 n-1 Sample concatenation EpR frequency mapping f F0n-1 F1 n-1 F2 n-1 F3 n-1 EpR frequency mapping
MMSP 2007 Invited talk Chania, Oct. 1st 2007
8 Current Trends: Corpus-based synthesis
Freesound database freesound.iua.upf.edu
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Open Issues: Interaction centered sound modeling
Measuring/modeling sound plus instrumental gesture
MMSP 2007 Invited talk Chania, Oct. 1st 2007
9 Open Issues: Physical modeling from data analysis
Various sensors on a violin
Measuring impulse responses from a violin
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Open Issues: Control/sound integration
Mathews with the Radio Drum
Waisvisz with the Hands and Anderson with an electric violin
MMSP 2007 Invited talk Chania, Oct. 1st 2007
10 Open Issues: Feedback systems
vBow by Charles Nichols
MMSP 2007 Invited talk Chania, Oct. 1st 2007
Beyond sound synthesis
MMSP 2007 Invited talk Chania, Oct. 1st 2007
11 Thanks !!!
MMSP 2007 Invited talk Chania, Oct. 1st 2007
12