History and Practice of Digital Sound Synthesis

Home , Additive synthesis, FM (No Static at All), IBM 704, Wavetable synthesis

Julius Smith CCRMA, Stanford University

AES-2006 Heyser Lecture

October 6, 2006

Julius Smith AES-2006 Heyser Lecture – 1 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Overview

Julius Smith AES-2006 Heyser Lecture – 2 / 84 Outline

Overview Digital sound synthesis approaches in approximate historical order: • Outline • CCRMA Perspective • Wavetable (one period) Early Digital Synthesis • Subtractive Spectral Modeling • Additive Physical Modeling • Frequency Modulation (FM) Summary • Sampling • Spectral Modeling • Physical Modeling Some connections with audio coding will be noted

Emphasis: • Sound examples • Block diagrams • Historical notes

Julius Smith AES-2006 Heyser Lecture – 3 / 84 CCRMA Perspective

Overview • Outline • CCRMA Perspective

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

The Knoll, Stanford University

Julius Smith AES-2006 Heyser Lecture – 4 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Early Digital Sound Synthesis

Julius Smith AES-2006 Heyser Lecture – 5 / 84 Wavetable Synthesis in Music I-V (1957-1969)

Overview

Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 6 / 84 Music V Scripting Language (“Note Cards”)

Overview

Spectral Modeling

Physical Modeling

Summary • Essentially Supported in MPEG-4 Structured Audio Orchestra Language (SAOL) (Music V → csound → SAOL) • “Encoding sounds” as “instruments” is hard, in general

Julius Smith AES-2006 Heyser Lecture – 7 / 84 Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis • Music V … y(n) • KL Music e(n) • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis Glottal Pulse • FM Formula Train or Noise 1 + k 1 1 + k M • FM Patch − − Speech e(n) z 1 2 … z 1 2 • FM Spectra Output • FM Examples y(n) k 1 − k 1 R 1 k M − k M R M • FM Voice • Sampling Synthesis (Unused − … − • Modern Example Allpass z 1 2 z 1 2 Output) 1 − k 1 1 − k M Spectral Modeling Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical) Physical Modeling

Summary John L. Kelly and Carol Lochbaum (1962)

Julius Smith AES-2006 Heyser Lecture – 8 / 84 Sound Example

Overview “Bicycle Built for Two”: (WAV) (MP3) Early Digital Synthesis • Music V • KL Music • “Daisy” • Vocal part by Kelly and Lochbaum (1961) • Additive Analysis • Musical accompaniment by Max Mathews • Additive Synthesis • FM Synthesis • Computed on an IBM 704 • FM Formula • Based on Russian speech-vowel data from Gunnar Fant’s book • FM Patch • FM Spectra • Probably the ﬁrst digital physical-modeling synthesis sound • FM Examples • FM Voice example by any method • Sampling Synthesis • Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey” • Modern Example — the computer’s “ﬁrst song” Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 9 / 84 Classic Additive-Synthesis Analysis (Heterodyne Comb)

Overview

Spectral Modeling

Physical Modeling

Summary

John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online)

Julius Smith AES-2006 Heyser Lecture – 10 / 84 Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

Overview

Spectral Modeling

Physical Modeling

Summary

John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online)

Julius Smith AES-2006 Heyser Lecture – 11 / 84 Classic Additive Synthesis Diagram

Overview A1(t) f1(t) A2(t) f2(t) A3(t) f3(t) A4(t) f4(t)

Spectral Modeling noise FIR Physical Modeling Σ

Summary

4 t y(t)= Ai(t) sin ωi(t)dt + φi(0) Z0 Xi=1

Julius Smith AES-2006 Heyser Lecture – 12 / 84 Classic Additive-Synthesis Examples

Overview • Bb Clarinet Early Digital Synthesis • Eb Clarinet • Music V • KL Music • Oboe • “Daisy” • • Additive Analysis Bassoon • Additive Synthesis • Tenor Saxophone • FM Synthesis • FM Formula • Trumpet • FM Patch • English Horn • FM Spectra • FM Examples • French Horn • FM Voice • Flute • Sampling Synthesis • Modern Example Spectral Modeling • All of the above Physical Modeling • Independently synthesized set Summary (Synthesized from original John Grey data)

Julius Smith AES-2006 Heyser Lecture – 13 / 84 Frequency Modulation (FM) Synthesis

Overview FM synthesis is normally used as a spectral modeling technique. Early Digital Synthesis • Music V • Discovered and developed (1970s) by John M. Chowning • KL Music • “Daisy” (CCRMA Founding Director) • Additive Analysis • Key paper: JAES 1973 (vol. 21, no. 7) • Additive Synthesis • FM Synthesis • Commercialized by Yamaha Corporation: • FM Formula • FM Patch ◦ DX-7 synthesizer (1983) • FM Spectra ◦ OPL chipset (SoundBlaster PC sound card) • FM Examples • FM Voice ◦ Cell phone ring tones • Sampling Synthesis • Modern Example • On the physical modeling front, synthesis of vibrating-string Spectral Modeling waveforms using ﬁnite differences started around this time: Physical Modeling Hiller & Ruiz, JAES 1971 (vol. 19, no. 6) Summary

Julius Smith AES-2006 Heyser Lecture – 14 / 84 FM Formula

Overview

Early Digital Synthesis x(t)= Ac sin[ωct + φc + Am sin(ωmt + φm)] • Music V • KL Music where • “Daisy” • Additive Analysis (Ac, ωc, φc) specify the carrier sinusoid • Additive Synthesis • FM Synthesis (Am, ωm, φm) specify the modulator sinusoid • FM Formula • FM Patch Can also be called phase modulation • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 15 / 84 Simple FM “Brass” Patch (1970–)

Jean-Claude Risset observation (1964–1969): Overview Brass bandwidth ∝ amplitude Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis 0 • Additive Synthesis fm = f • FM Synthesis • FM Formula g • FM Patch A F • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example fc = f0

Spectral Modeling

Physical Modeling A F Summary

Out

Julius Smith AES-2006 Heyser Lecture – 16 / 84 FM Harmonic Amplitudes (Bessel Function of First Kind)

Overview Harmonic number k, FM index β:

Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula 1 • FM Patch • FM Spectra • FM Examples • FM Voice 0.5

• Sampling Synthesis ) β ( k

• Modern Example J 0 Spectral Modeling 0

Physical Modeling 2 −0.5 Summary 4 0 5 10 6 15 20 8 25 Order k 30 10 Argument β

Julius Smith AES-2006 Heyser Lecture – 17 / 84 Frequency Modulation (FM) Examples

Overview All examples by John Chowning unless otherwise noted: Early Digital Synthesis • Music V • FM brass synthesis • KL Music • “Daisy” ◦ Low Brass example • Additive Analysis • Additive Synthesis ◦ Dexter Morril’s FM Trumpet • FM Synthesis • FM Formula • FM singing voice (1978) • FM Patch Each formant synthesized using an FM operator pair • FM Spectra • FM Examples (two sinusoidal oscillators) • FM Voice • Sampling Synthesis ◦ Chorus • Modern Example ◦ Voices Spectral Modeling ◦ Basso Profundo Physical Modeling Summary • Other early FM synthesis ◦ Clicks and Drums ◦ Big Bell ◦ String Canon

Julius Smith AES-2006 Heyser Lecture – 18 / 84 FM Voice

Overview FM voice synthesis can be viewed as compressed modeling of Early Digital Synthesis spectral formants • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis Carrier 2 • FM Synthesis Carrier 1 • FM Formula Carrier 3 • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis Magnitude • Modern Example

Spectral Modeling

Physical Modeling

Summary 0 Frequency f Modulation Frequency (all three)

Julius Smith AES-2006 Heyser Lecture – 19 / 84 Sampling Synthesis History

Overview • 1979 - Fairlight Computer Music Instrument - 8-bit Early Digital Synthesis • Music V ◦ First commercial sampler • KL Music ◦ • “Daisy” Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono) • Additive Analysis ◦ Editing, looping, mixing • Additive Synthesis • FM Synthesis ◦ One could draw waveforms and additive-synthesis amplitude • FM Formula envelopes (for each harmonic) with a light pen • FM Patch • FM Spectra ◦ $25,000–$36,000! • FM Examples • FM Voice • 1981 - E-mu Systems Emulator • Sampling Synthesis • Modern Example ◦ First “affordable” sampler ($10,000) Spectral Modeling ◦ Eight voices, 128 K RAM, 8-bit, 80 lb. Physical Modeling

Summary • 1986 - Ensoniq Mirage ◦ Breakthrough price-point ($1695) ◦ Eight voices, 144 K RAM, 8-bit

Julius Smith AES-2006 Heyser Lecture – 20 / 84 Modern Sampled Piano

Overview Example:1 Early Digital Synthesis • Music V • 40 Gigabytes on ten DVDs (three sampled pianos) • KL Music • “Daisy” • Every key sampled • Additive Analysis • 4–10 “velocity layers” • Additive Synthesis • FM Synthesis • Separate recordings with soft pedal down • FM Formula • • FM Patch Separate “release” recordings, for multiple striking velocities • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

Spectral Modeling

Physical Modeling

Summary

1Synthogy Ivory, $349 (Electronic Musician, October 2006)

Julius Smith AES-2006 Heyser Lecture – 21 / 84 Fundamental Problem with Sampling Synthesis

Overview Piano timbre is determined by Early Digital Synthesis • Music V • key number (1 byte) • KL Music • • “Daisy” key velocity (2 bytes more than enough) • Additive Analysis • pedal state (1 bit [or byte] per pedal) • Additive Synthesis • FM Synthesis Piano control is relatively low-dimensional: • FM Formula • FM Patch • Less than six bytes of information per note played • FM Spectra • FM Examples • No continuous controls (typically) • FM Voice • Sampling Synthesis • Ratio of total sampled data to one note of control data • Modern Example ≈ one billion Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 22 / 84 Now consider bowed strings

Overview Control parameters: Early Digital Synthesis • Music V • Left-hand ﬁnger position(s) • KL Music • • “Daisy” Left-hand vibrato • Additive Analysis • Bow velocity • Additive Synthesis • FM Synthesis • Bow force • FM Formula • Bow position • FM Patch • FM Spectra • Bow angle • FM Examples • Shoulder damping • FM Voice • Sampling Synthesis • Instrument orientation • Modern Example • Player motion (within a room) Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 23 / 84 Difﬁculty of sampling bowed strings

Overview • Bowed-string control is inﬁnite-dimensional in principle Early Digital Synthesis • Many time-varying functions — “gestures” • Music V • KL Music (we counted more than 10) • “Daisy” • • Additive Analysis Complete sampling of bowed strings on the level of pianos has • Additive Synthesis apparently never been done • FM Synthesis • FM Formula • Rule-driven navigation of the most useful recorded playing • FM Patch regimes has worked well (e.g., Synful Orchestra) • FM Spectra • FM Examples • Model-based approaches greatly reduce data requirements: • FM Voice • Sampling Synthesis ◦ Spectral models (inspired by sound perception) • Modern Example ◦ Physical models (model the sound source) Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 24 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Spectral Modeling Synthesis (Historical Summary)

Julius Smith AES-2006 Heyser Lecture – 25 / 84 Classic Vocoder Analysis & Resynthesis (Dudley 1939)

magnitude, or Overview magnitude and phase extraction Early Digital Synthesis

Spectral Modeling • A Vocoder x0(t) xˆ0(t) • Vocoder Examples • Sinusoidal Modeling f • Spectral Trajectories Data Compression, • Sines + Noise A Transmission, xˆ(t) • S+N Examples x(t) x1(t) Storage, xˆ1(t) Manipulation, • S+N FX f Noise reduction, ... • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows A N−1 • HF Noise Modeling x xˆN−1 • HF Noise Band f • S+N+T Examples • SM Summary • Spec Future

Physical Modeling Analysis Processing Synthesis

Summary

Julius Smith AES-2006 Heyser Lecture – 26 / 84 Phase Vocoder Channel Model

Overview Analysis Model Synthesis Model Early Digital Synthesis a a (t) ω +∆ω (t) Spectral Modeling Channel Filter k k k k • Vocoder Response • Vocoder Examples A F ∆ω Sine Osc • Sinusoidal Modeling k • Spectral Trajectories • 0 ω ω Sines + Noise k • S+N Examples Out • S+N FX • S+N XSynth • Early “channel vocoder” implementations (hardware) only • Sines + Transients • S + N + Transients measured amplitude ak(t) (Dudley 1939) • S+N+T TSM • The “phase vocoder” (Flanagan and Golden 1966) added phase • S+N+T Freq Map • S+N+T Windows tracking in each channel • HF Noise Modeling • Portnoff (1976) developed the FFT phase vocoder, • HF Noise Band • S+N+T Examples which replaced the heterodyne comb in computer-music • SM Summary additive-synthesis analysis (James A. Moorer) • Spec Future

Physical Modeling • Inverse FFT synthesis (Rodet and Depalle 1992) gave faster

Summary sinusoidal oscillator banks

Julius Smith AES-2006 Heyser Lecture – 27 / 84 Amplitude and Frequency Envelopes

a (t) Overview k

Early Digital Synthesis

Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX t • S+N XSynth ∆ωk(t)= φ˙k(t) • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows 0 • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future t Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 28 / 84 Channel Vocoder Sound Examples

Overview • Original Early Digital Synthesis Spectral Modeling • • Vocoder 10 channels, sine carriers • Vocoder Examples • 10 channels, narrowband-noise carriers • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • 26 channels, sine carriers • S+N Examples • S+N FX • 26 channels, narrowband-noise carriers • S+N XSynth • 26 channels, narrowband-noise carriers, channels reversed • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • Phase Vocoder: Identity system in absence of modiﬁcations • S+N+T Windows • HF Noise Modeling • HF Noise Band • The FFT Phase Vocoder next transitioned to the Short-Time • S+N+T Examples Fourier Transform (STFT) (Allen and Rabiner 1977) • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 29 / 84 Tracking Spectral Peaks in the Short-Time Fourier Transform

Overview Phases Early Digital Synthesis atan

Spectral Modeling • Vocoder s(t) • FFT Vocoder Examples Frequencies • Sinusoidal Modeling Quadratic • Peak Spectral Trajectories Peak dB mag tracking • Sines + Noise Interpolation • S+N Examples window w(n) Amplitudes • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • STFT peak tracking at CCRMA: mid-1980s (PARSHL program) • S+N+T Freq Map • Motivated by vocoder analysis of piano tones • S+N+T Windows • HF Noise Modeling • Inﬂuences: STFT (Allen and Rabiner 1977), • HF Noise Band ADEC (1977), MAPLE (1979) • S+N+T Examples • SM Summary • Independently developed for speech coding by McAulay and • Spec Future Quatieri at Lincoln Labs Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 30 / 84 Example Spectral Trajectories

Overview f Early Digital Synthesis

Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band t • S+N+T Examples • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 31 / 84 Sines + Noise Synthesis (1989) PSfrag 1 1 2 2 3 3 4 4 Overview A (t) f (t) A (t) f (t) A (t) f (t) A (t) f (t)

Early Digital Synthesis

Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows white noise • HF Noise Modeling u(t) ﬁlter(t) • HF Noise Band h (τ) • S+N+T Examples t • SM Summary P • Spec Future 4 Physical Modeling t y(t)= Ai(t) cos 0 ωi(t)dt + φi(0) +(ht ∗ u)(t) =1 Summary iP hR i

Julius Smith AES-2006 Heyser Lecture – 32 / 84 Sines + Noise Sound Examples

Overview Xavier Serra 1989 thesis demos (Sines + Noise signal modeling) Early Digital Synthesis • Spectral Modeling Piano • Vocoder • Vocoder Examples ◦ Original • Sinusoidal Modeling ◦ Sinusoids alone • Spectral Trajectories • Sines + Noise ◦ Residual after sinusoids removed • S+N Examples ◦ Sines + noise model • S+N FX • S+N XSynth • Sines + Transients • Voice • S + N + Transients • S+N+T TSM ◦ Original • S+N+T Freq Map ◦ Sinusoids • S+N+T Windows • HF Noise Modeling ◦ Residual • HF Noise Band ◦ Synthesis • S+N+T Examples • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 33 / 84 Musical Effects with Sines+Noise Models (Serra 1989)

Overview • Piano Effects Early Digital Synthesis ◦ Spectral Modeling Pitch downshift one octave • Vocoder ◦ Pitch ﬂattened • Vocoder Examples • Sinusoidal Modeling ◦ Varying partial stretching • Spectral Trajectories • Sines + Noise • Voice Effects • S+N Examples • S+N FX ◦ Frequency-scale by 0.6 • S+N XSynth • Sines + Transients ◦ Frequency-scale by 0.4 and stretch partials • S + N + Transients ◦ Variable time-scaling, deterministic to stochastic • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 34 / 84 Cross-Synthesis with Sines+Noise Models (Serra 1989)

Overview • Voice “modulator” Early Digital Synthesis Spectral Modeling • • Vocoder Creaking ship’s mast “carrier” • Vocoder Examples • Voice-modulated creaking mast • Sinusoidal Modeling • Spectral Trajectories • Same with modiﬁed spectral envelopes • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 35 / 84 Sines + Transients Sound Examples (Serra 1989)

Overview In this technique, the sinusoidal sum is phase-matched at the Early Digital Synthesis cross-over point only (with no cross-fade). Spectral Modeling • Vocoder • Marimba • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories ◦ Original • Sines + Noise ◦ Sinusoidal model • S+N Examples • S+N FX ◦ Original attack, followed by sinusoidal model • S+N XSynth • Sines + Transients • Piano • S + N + Transients • S+N+T TSM ◦ Original • S+N+T Freq Map • S+N+T Windows ◦ Sinusoidal model • HF Noise Modeling ◦ Original attack, followed by sinusoidal model • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 36 / 84 Multiresolution Sines + Noise + Transients (Levine 1998)

Overview Why Model Transients Separately? Early Digital Synthesis • Spectral Modeling Sinusoids efficiently model spectral peaks over time • Vocoder • Filtered noise efficiently models spectral residual vs. t • Vocoder Examples • Sinusoidal Modeling • Neither is good for abrupt transients in the waveform • Spectral Trajectories • Phase-matched oscillators are expensive • Sines + Noise • S+N Examples • More efficient to switch to a transient model during transients • S+N FX • Need sinusoidal phase matching at the switching times • S+N XSynth • Sines + Transients • S + N + Transients Transient models: • S+N+T TSM • S+N+T Freq Map • Original waveform slice (1988) • S+N+T Windows • Wavelet expansion (Ali 1996) • HF Noise Modeling • HF Noise Band • MPEG-2 AAC (with short window) (Levine 1998) • S+N+T Examples • Frequency-domain LPC • SM Summary • Spec Future (time-domain amplitude envelope) (Verma 2000) Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 37 / 84 Time Scale Modiﬁcation of Sines + Noise + Transients Models

Overview transients transients sines+ sines+ sines+ Early Digital Synthesis originalsignal noise noise noise Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling •

Spectral Trajectories transients transients • Sines + Noise time-scaled sines+ sines+ sines+ • S+N Examples signal noise noise noise • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients time • S+N+T TSM • S+N+T Freq Map • S+N+T Windows Time-Scale Modification (TSM) becomes well defined: • HF Noise Modeling • HF Noise Band • Transients are translated in time • S+N+T Examples • SM Summary • Sinusoidal envelopes are scaled in time • Spec Future • Noise-filter envelopes also scaled in time Physical Modeling • Dual of TSM is frequency scaling Summary

Julius Smith AES-2006 Heyser Lecture – 38 / 84 Sines + Noise + Transients Time-Frequency Map

16 Overview 14 Early Digital Synthesis 12 Spectral Modeling • Vocoder 10 • Vocoder Examples • Sinusoidal Modeling 8 • Spectral Trajectories 6

• Sines + Noise frequency [kHz] • S+N Examples 4 • S+N FX 2 • S+N XSynth • Sines + Transients 0 0 50 100 150 200 250 • S + N + Transients 1 • S+N+T TSM • S+N+T Freq Map 0.5 • S+N+T Windows 0 • HF Noise Modeling

• HF Noise Band amplitude −0.5 • S+N+T Examples −1 • SM Summary 0 50 100 150 200 250 • Spec Future time [milliseconds] Physical Modeling (Levine 1998) Summary

Julius Smith AES-2006 Heyser Lecture – 39 / 84 Corresponding Analysis Windows

Overview

Early Digital Synthesis transient Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling

• Spectral Trajectories high octave • Sines + Noise • S+N Examples • S+N FX •

S+N XSynth middle octave • Sines + Transients • S + N + Transients • S+N+T TSM

• S+N+T Freq Map low octave • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary

• Spec Future amplitude

Physical Modeling 0 50 100 150 200 250 Summary time [milliseconds]

Julius Smith AES-2006 Heyser Lecture – 40 / 84 Quasi-Constant-Q (Wavelet) Time-Frequency Map

16 Overview 14 Early Digital Synthesis

Spectral Modeling 12 • Vocoder 10 • Vocoder Examples • Sinusoidal Modeling 8 • Spectral Trajectories 6 •

Sines + Noise frequency [kHz] • S+N Examples 4 • S+N FX 2 • S+N XSynth • Sines + Transients 0 0 4 50 100 150 200 250 • S + N + Transients x 10 • S+N+T TSM 2 • S+N+T Freq Map 1 • S+N+T Windows 0 • HF Noise Modeling

• HF Noise Band amplitude −1 • S+N+T Examples −2 • SM Summary 0 50 100 150 200 250 • Spec Future time [milliseconds] Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 41 / 84 Bark-Band Noise Modeling at High Frequencies (Levine 1998)

Overview 85

Early Digital Synthesis 80 Spectral Modeling • Vocoder 75 • Vocoder Examples • Sinusoidal Modeling 70 • Spectral Trajectories • Sines + Noise 65 • S+N Examples • S+N FX 60 • S+N XSynth • Sines + Transients 55 • S + N + Transients • magnitude [dB] S+N+T TSM 50 • S+N+T Freq Map • S+N+T Windows 45 • HF Noise Modeling • HF Noise Band 40 • S+N+T Examples • SM Summary 35 • Spec Future 30 Physical Modeling 0 500 1000 1500 2000 2500 3000 3500 4000 4500 frequency [Hz] Summary

Julius Smith AES-2006 Heyser Lecture – 42 / 84 Amplitude Envelope for One Noise Band

Overview 80

Early Digital Synthesis 70 Spectral Modeling • Vocoder • Vocoder Examples 60 • Sinusoidal Modeling • Spectral Trajectories 50 Original Mag. [dB] • Sines + Noise • S+N Examples 40 • S+N FX 0 50 100 150 200 250 300 350 • S+N XSynth • Sines + Transients • S + N + Transients 80 • S+N+T TSM 75 • S+N+T Freq Map 70 • S+N+T Windows 65 • HF Noise Modeling • HF Noise Band 60 • S+N+T Examples 55 LSA Mag. [dB] • SM Summary 50 • Spec Future 45 Physical Modeling 0 50 100 150 200 250 300 350 time [milliseconds] Summary

For more information, see Scott Levine’s thesis.2 Julius Smith 2http://ccrma.stanford.edu/~scottl/thesis.htmlAES-2006 Heyser Lecture – 43 / 84 Sines + Noise + Transients Sound Examples

Overview Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps) Early Digital Synthesis (http://ccrma.stanford.edu/~scottl/thesis.html) Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise Mozart’s Le Nozze di Figaro • S+N Examples • S+N FX • Original • S+N XSynth • Compressed using MPEG-AAC at 32 kbps • Sines + Transients • S + N + Transients • Compressed using sines+transients+noise at 32 kbps • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • Multiresolution sinusoids alone • HF Noise Modeling • HF Noise Band • Residual Bark-band noise • S+N+T Examples • Transform-coded transients (AAC) • SM Summary • Spec Future • Bark-band noise above 5 kHz

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 44 / 84 Rock Example

Overview Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps) Early Digital Synthesis (http://ccrma.stanford.edu/~scottl/thesis.html) Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories “It Takes Two” by Rob Base & DJ E-Z Rock • Sines + Noise • S+N Examples • S+N FX • Original • S+N XSynth • MPEG-AAC at 32 kbps • Sines + Transients • S + N + Transients • Sines+transients+noise at 32 kbps • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • Multiresolution sinusoids • HF Noise Modeling • • HF Noise Band Residual Bark-band noise • S+N+T Examples • Transform-coded transients (AAC) • SM Summary • Spec Future • Bark-band noise above 5 kHz

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 45 / 84 Time Scale Modiﬁcation using Sines + Noise + Transients

Overview Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps) Early Digital Synthesis (http://ccrma.stanford.edu/~scottl/thesis.html) Spectral Modeling • Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories Time-Scale Modiﬁcation (pitch unchanged) • Sines + Noise • S+N Examples • S+N FX • S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5] • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM S+N+T Pitch Shifting (timing unchanged) • S+N+T Freq Map • S+N+T Windows • Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12] • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 46 / 84 Spectral Modeling History Highlights

• Fourier’s theory (1822) Perceptual audio coding: • Teleharmonium (1906) • Princen-Bradley ﬁlterbank (1986) • Hammond organ (1930s) • K. Brandenburg thesis (1989) • Channel Vocoder (1939) • Auditory masking usage • Phase Vocoder (1966) • Dolby AC2 • “Additive Synthesis” (1969) • Musicam • FFT Phase Vocoder (1976) • ASPEC • Sinusoidal Modeling • MPEG-I,II,IV (1977,1979,1985) (incl. S+N+T “parametric sounds”) • Sines+Noise (1989) • Sines+Transients (1989) • Sines+Noise+Transients (1998)

Julius Smith AES-2006 Heyser Lecture – 47 / 84 Future Prospects

Overview Observations: Early Digital Synthesis • Spectral Modeling Sinusoidal modeling of sound is “Unreasonably Effective” • Vocoder • Basic “auditory masking” discards ≈ 90% information • Vocoder Examples • Sinusoidal Modeling • Interesting neuroscience observation: • Spectral Trajectories • Sines + Noise “... most neurons in the primary auditory cortex A1 • S+N Examples are silent most of the time ...” • S+N FX • S+N XSynth (from “Sparse Time-Frequency Representations”, Gardner and • Sines + Transients • S + N + Transients Magnesco, PNAS:103(16), April 2006) • S+N+T TSM • S+N+T Freq Map • What is a true and correct “psychospectral model” for sound? • S+N+T Windows • HF Noise Modeling ◦ The cochlea of the ear is a real-time spectrum analyzer • HF Noise Band ◦ How is the “ear’s spectrogram” represented at higher levels? • S+N+T Examples • SM Summary • Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 48 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Physical Modeling Synthesis (Historical Summary)

Julius Smith AES-2006 Heyser Lecture – 49 / 84 Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis

Spectral Modeling e(n) … y(n) Physical Modeling • KL Music • Digital Waveguide • Signal Scattering • Plucked String Glottal Pulse + + • Train or Noise 1 k 1 1 k M Struck String − − Speech e(n) z 1 2 … z 1 2 • Karplus Strong Output • EKS Algorithm y(n) k − k R k − k R • Clarinet 1 1 1 M M M • Wind Examples (Unused − − • Bowed Strings Allpass z 1 2 … z 1 2 • Distortion Guitar Output) 1 − k 1 1 − k M • Acoustic Strings Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical) • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis John L. Kelly and Carol Lochbaum (1962) • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 50 / 84 Digital Waveguide Models (1985)

∆ Overview Lossless digital waveguide = bidirectional delay line Early Digital Synthesis at some wave impedance R Spectral Modeling

Physical Modeling • KL Music • Digital Waveguide − N • Signal Scattering z • Plucked String • Struck String • Karplus Strong R • EKS Algorithm • Clarinet − N • Wind Examples z • Bowed Strings • Distortion Guitar • Acoustic Strings Useful for efﬁcient models of • Sound Examples • Linearized Violin • strings • Commuted Piano • Pulse Synthesis • bores • Complete Piano • plane waves • Sound Examples • Phy Audio Coding • conical waves • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 51 / 84 Signal Scattering

Overview Signal scattering is caused by a change in wave impedance R: Early Digital Synthesis

Spectral Modeling R2 − R1 k 1 = Physical Modeling R2 + R1 • KL Music • Digital Waveguide + • Signal Scattering 1 k 1 • − − Plucked String z N z N • Struck String • Karplus Strong • EKS Algorithm k 1 − k 1 • Clarinet R 1 R 2 • Wind Examples • Bowed Strings − N − N • Distortion Guitar z z • Acoustic Strings 1 − k 1 • Sound Examples • Linearized Violin • Commuted Piano If the wave impedance changes every spatial sample, the • Pulse Synthesis • Complete Piano Kelly-Lochbaum vocal-tract model results. • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 52 / 84 Ideal Plucked String (Displacement Waves)

Overview

Early Digital Synthesis

Spectral Modeling + + y (n) y (n-N/2) Physical Modeling • KL Music • Digital Waveguide ªBridgeº -1 (x = Pluck Position) -1 ªNutº • Signal Scattering • Plucked String • Struck String y- (n) y- (n+N/2) • Karplus Strong • EKS Algorithm • Clarinet (x = 0) (x = L) • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Load each delay line with half of initial string displacement • Linearized Violin • Sum of upper and lower delay lines = string displacement • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 53 / 84 Ideal Struck String (Velocity Waves)

Overview

Early Digital Synthesis c Spectral Modeling + + v (n) v (n-N/2) Physical Modeling • KL Music • Digital Waveguide ªBridgeº -1 (x = Hammer Position) -1 ªNutº • Signal Scattering • Plucked String c • Struck String v- (n) v- (n+N/2) • Karplus Strong • EKS Algorithm (x = 0) (x = L) • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings Hammer strike = momentum transfer = velocity step: • Sound Examples • Linearized Violin • Commuted Piano mhvh(0−)=(mh + ms)vs(0+) • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 54 / 84 Karplus-Strong (KS) Algorithm (1983)

Overview Early Digital Synthesis + + Output y (n) y (n-N) Spectral Modeling N samples delay

Physical Modeling • KL Music 1/2 • Digital Waveguide • Signal Scattering • 1/2 Plucked String z -1 • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Discovered (1978) as “self-modifying wavetable synthesis” • Distortion Guitar • • Acoustic Strings Wavetable is preferably initialized with random numbers • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 55 / 84 Karplus-Strong Sound Examples

Overview • “Vintage” 8-bit sound examples: Early Digital Synthesis • Original Plucked String: (WAV) (MP3) Spectral Modeling

Physical Modeling • Drum: (WAV) (MP3) • KL Music • Stretched Drum: (WAV) (MP3) • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 56 / 84 EKS Algorithm (Jaffe-Smith 1983)

Overview −N Hp(z) Hβ (z) z HL(z) Early Digital Synthesis

Spectral Modeling

Physical Modeling Hρ(z) Hs(z) Hd(z) • KL Music • Digital Waveguide • Signal Scattering N = pitch period (2× string length) in samples • Plucked String • Struck String 1 − p • Karplus Strong Hp(z) = −1 = pick-direction lowpass filter • EKS Algorithm 1 − p z • Clarinet −βN • Wind Examples Hβ(z) = 1 − z = pick-position comb filter, β ∈ (0, 1) • Bowed Strings • Distortion Guitar Hd(z) = string-damping filter (one/two poles/zeros typical) • Acoustic Strings • Sound Examples Hs(z) = string-stiffness allpass filter (several poles and zeros) • Linearized Violin −1 • ρ(N) − z Commuted Piano H (z) = = first-order string-tuning allpass filter • Pulse Synthesis ρ 1 − ρ(N) z−1 • Complete Piano • Sound Examples 1 − RL • Phy Audio Coding HL(z) = −1 = dynamic-level lowpass filter • Phy Audio Coding? 1 − RL z Summary Julius Smith AES-2006 Heyser Lecture – 57 / 84 STK EKS Sound Examples

Overview • Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and Early Digital Synthesis others — distributed by CCRMA: Spectral Modeling Google search: STK ToolKit Physical Modeling • KL Music • Digital Waveguide STK Plucked String: (WAV) (MP3) • Signal Scattering • Plucked String • Plucked String 1: (WAV) (MP3) • Struck String • Karplus Strong • Plucked String 2: (WAV) (MP3) • EKS Algorithm • Plucked String 3: (WAV) (MP3) • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 58 / 84 EKS Sound Example (1988)

Overview Bach A-Minor Concerto—Orchestra Part: (WAV) (MP3) Early Digital Synthesis

Spectral Modeling

Physical Modeling • Executed in real time on one Motorola DSP56001 • KL Music (20 MHz clock, 128K SRAM) • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Developed for the NeXT Computer introduction at Davies • Karplus Strong Symphony Hall, San Francisco, 1988 • EKS Algorithm • Clarinet • Wind Examples • Solo violin part was played live by Dan Kobialka of the San • Bowed Strings • Distortion Guitar Francisco Symphony • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 59 / 84 Digital Waveguide Single Reed, Cylindrical Bore Model (1986)

Mouth Overview Pressure − ( ) p b n Early Digital Synthesis p ( n) h m Output m Reed to Bell Delay 2 Filter Spectral Modeling Reed Table - ρ Physical Modeling Ã Reflection • KL Music * Filter • Digital Waveguide + h ∆ • Signal Scattering - • Plucked String Bell to Reed Delay - + • Struck String Embouchure ( ) p b n • Karplus Strong Offset • EKS Algorithm Reed Bore Bell • Clarinet • Wind Examples Digital waveguide clarinet • Bowed Strings • Distortion Guitar • Acoustic Strings • Control variable = mouth half-pressure • Sound Examples • Linearized Violin • Total reed cost = two subtractions, one multiply, and one table • Commuted Piano lookup per sample • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 60 / 84 Digital Waveguide Wind Instrument Sound Examples

Overview • STK Clarinet: (WAV) (MP3) Early Digital Synthesis Google search: STK clarinet Spectral Modeling

Physical Modeling ◦ Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and • KL Music others — distributed by CCRMA: • Digital Waveguide • Signal Scattering Google search: STK ToolKit • Plucked String • Struck String • Staccato Systems Slide Flute • Karplus Strong • EKS Algorithm (based on STK ﬂute, ca. 1995): (WAV) (MP3) • Clarinet • Yamaha VL1 “Virtual Lead” synthesizer demos (1994): • Wind Examples • Bowed Strings • Shakuhachi: (WAV) (MP3) • Distortion Guitar • Acoustic Strings • Oboe and Bassoon: (WAV) (MP3) • Sound Examples • Tenor Saxophone: (WAV) (MP3) • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 61 / 84 Digital Waveguide Bowed Strings (1986)

Bow Force Overview Bow Velocity + − v , ls v s , r Early Digital Synthesis Body Nut to Bow Delay Bow to Bridge Delay Filter Spectral Modeling - + ρ v b v ∆ Ã Reflection Physical Modeling -1 Filter • - * KL Music Bow Table • Digital Waveguide + v s , r • Signal Scattering Bow to Nut Delay Bridge to Bow Delay − • Plucked String v , ls • Struck String Bridge- Nut String Bow String Air • Karplus Strong Body • EKS Algorithm • Clarinet • Wind Examples • Reflection filter summarizes all losses per period • Bowed Strings (due to bridge, bow, finger, etc.) • Distortion Guitar • Acoustic Strings • Bow-string junction = memoryless lookup table • Sound Examples (or segmented polynomial) • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 62 / 84 “Electric Cello” Sound Examples (Peder Larson)

Overview • Staccato Notes: (WAV) (MP3) Early Digital Synthesis (short strokes of high bow pressure, as from a bouncing bow) Spectral Modeling • Bach’s First Suite for Unaccompanied Cello: (WAV) (MP3) Physical Modeling • KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 63 / 84 Soft Clipper

Overview 2 − 3 , x ≤−1 Early Digital Synthesis  x3 Spectral Modeling f(x)= x − , −1 ≤ x ≤ 1  3 Physical Modeling  2 • KL Music 3 , x ≥ 1 • Digital Waveguide  • Signal Scattering  • Plucked String x=−1:0.01:1; plot([−(2/3)*ones(1,100), x−x. 3 /3, (2/3)*ones(1,100)]) • Struck String 0.8 • Karplus Strong 0.6 • EKS Algorithm • Clarinet 0.4 • Wind Examples • Bowed Strings 0.2 • Distortion Guitar 0 • Acoustic Strings f(x(n)) • Sound Examples −0.2 • Linearized Violin −0.4 • Commuted Piano • Pulse Synthesis −0.6 • Complete Piano • −0.8 Sound Examples −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 • Phy Audio Coding x(n) • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 64 / 84 Ampliﬁer Distortion + Ampliﬁer Feedback

Overview Sullivan 1990 Early Digital Synthesis

Spectral Modeling

Physical Modeling • KL Music Pre-distortion output level • Digital Waveguide String 1 • Signal Scattering Nonlinear Distortion • Plucked String • . Struck String . Output Signal • Karplus Strong • EKS Algorithm Pre-distortion gain Distortion output level • Clarinet String N Amplifier • Wind Examples Feedback • Bowed Strings Gain • Distortion Guitar • Acoustic Strings Amplifier Feedback Delay • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano Distortion output signal often further filtered by an amplifier cabinet • Sound Examples • Phy Audio Coding filter, representing speaker cabinet, driver responses, etc. • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 65 / 84 Distortion Guitar Sound Examples

Overview (Stanford Sondius Project, ca. 1995) Early Digital Synthesis • Spectral Modeling Distortion Guitar: (WAV) (MP3)

Physical Modeling • Ampliﬁer Feedback 1: (WAV) (MP3) • KL Music • Ampliﬁer Feedback 2: (WAV) (MP3) • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 66 / 84 Commuted Synthesis of Acoustic Strings (1993)

Overview e(t) s(t) y(t) Early Digital Synthesis Trigger Excitation String Resonator Output

Spectral Modeling

Physical Modeling • KL Music Schematic diagram of a stringed musical instrument. • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong Trigger Excitation Resonator String Output • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings Equivalent diagram in the linear, time-invariant case. • Distortion Guitar • Acoustic Strings • Sound Examples a(t) x(t) Aggregate • Trigger String Output Linearized Violin Excitation • Commuted Piano • Pulse Synthesis • Complete Piano Use of an aggregate excitation given by the convolution of original • Sound Examples excitation with the resonator impulse response. • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 67 / 84 Commuted Components

Overview

Early Digital Synthesis a(t) x(t) Aggregate Trigger String Output Spectral Modeling Excitation Physical Modeling • KL Music “Plucked Resonator” driving a String. • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet s(t) Bridge Guitar Air Room y(t) • Wind Examples Coupling Body Absorption Response Output • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples Possible components of a guitar resonator. • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 68 / 84 Sound Examples

Overview Electric Guitar (Pick-Ups and/or Body-Model Added) (Stanford Early Digital Synthesis Sondius Project → Staccato Systems, Inc. → ADI, ca. 1995) Spectral Modeling Physical Modeling • Example 1: (WAV) (MP3) • KL Music • Digital Waveguide • Example 2: (WAV) (MP3) • Signal Scattering • Example 3: (WAV) (MP3) • Plucked String • Struck String • Virtual “wah-wah pedal”: (WAV) (MP3) • Karplus Strong • EKS Algorithm • Clarinet STK Mandolin • Wind Examples • Bowed Strings • STK Mandolin 1: (WAV) (MP3) • Distortion Guitar • Acoustic Strings • STK Mandolin 2: (WAV) (MP3) • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 69 / 84 Sound Examples

Overview More Recent Acoustic Guitar Early Digital Synthesis • Spectral Modeling Bach Prelude in E Major: (WAV) (MP3)

Physical Modeling • soundexamplewavBach silenceLoure in E Major: (WAV) (MP3) • KL Music • Digital Waveguide • Signal Scattering Virtual performance by Dr. Mikael Laurson, Sibelius Institute • Plucked String • Struck String 3 • Karplus Strong Virtual guitar by Helsinki Univ. of Tech., Acoustics Lab • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding 3 • Phy Audio Coding? http://www.acoustics.hut.fi/

Summary Julius Smith AES-2006 Heyser Lecture – 70 / 84 Commuted Synthesis of Linearized Violin

a) Output Overview e(n) s(n) Amplitude(n) Impulse x(n) String Resonator Early Digital Synthesis Frequency(n) Train

Spectral Modeling b) Output Physical Modeling e(n) Amplitude(n) Impulse a(n) x(n) • KL Music Resonator String Frequency(n) Train • Digital Waveguide • Signal Scattering • Plucked String c) Output a(n) • Struck String Amplitude(n) Impulse-Response x(n) String • Karplus Strong Frequency(n) Train • EKS Algorithm • Clarinet • Wind Examples • Assumes ideal Helmholtz motion of string • Bowed Strings • Distortion Guitar • Sound Examples (Stanford Sondius project, ca. 1995): • Acoustic Strings ◦ Bass: (WAV) (MP3) ◦ Violin 1: (WAV) (MP3) • Sound Examples • Linearized Violin ◦ Cello: (WAV) (MP3) ◦ Violin 2: (WAV) (MP3) • Commuted Piano ◦ Viola 1: (WAV) (MP3) ◦ Ensemble: (WAV) (MP3) • Pulse Synthesis • Complete Piano ◦ Viola 2: (WAV) (MP3) • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 71 / 84 Commuted Piano Synthesis (1995)

Overview Hammer-string interaction pulses (force): Early Digital Synthesis

Spectral Modeling Force 0.5 Physical Modeling 0.4 • KL Music 0.3 • Digital Waveguide • Signal Scattering 0.2 • Plucked String 0.1 • Struck String 5 10 15 20 Time • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 72 / 84 Synthesis of Hammer-String Interaction Pulse

Overview Impulse Impulse Response Early Digital Synthesis

Spectral Modeling Lowpass Physical Modeling Filter • KL Music • Digital Waveguide Time Time • Signal Scattering • Plucked String • Struck String • Karplus Strong • Faster collisions correspond to narrower pulses • EKS Algorithm • Clarinet (nonlinear ﬁlter) • Wind Examples • Bowed Strings • Distortion Guitar • For a given velocity, ﬁlter is linear time-invariant • Acoustic Strings • Sound Examples • Linearized Violin • Piano is “linearized” for each hammer velocity • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 73 / 84 Multiple Hammer-String Interaction Pulses

Overview Superimpose several individual pulses: Early Digital Synthesis

Spectral Modeling

Physical Modeling Force • KL Music • Impulse 1 Digital Waveguide δ LPF1 • Signal Scattering 1 String Impulse 2 • Plucked String δ LPF2 + Input • Struck String 2 Impulse 3 LPF3 • Karplus Strong δ 3 • EKS Algorithm 0 Time • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 74 / 84 Multiple Hammer-String Interaction Pulses

Overview Superimpose several individual pulses: Early Digital Synthesis

Spectral Modeling

Physical Modeling Force • KL Music • Impulse 1 Digital Waveguide δ LPF1 • Signal Scattering 1 String Impulse 2 • Plucked String δ LPF2 + Input • Struck String 2 Impulse 3 LPF3 • Karplus Strong δ 3 • EKS Algorithm 0 Time • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar As impulse amplitude grows (faster hammer strike), output pulses • Acoustic Strings • Sound Examples become taller and thinner, showing less overlap. • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 74 / 84 Complete Piano Model

Natural Ordering: Overview

Early Digital Synthesis v c Spectral Modeling LPF1 Impulse δ Tapped δ Physical Modeling 1 1 Sound Board • Gener- Delay LPF2 String KL Music Trigger δ + & Enclosure • ator Line 2 Output Digital Waveguide LPF3 • δ Signal Scattering 3 • Plucked String • Struck String • Karplus Strong Commuted Ordering: • EKS Algorithm • Clarinet v c • Wind Examples • Bowed Strings LPF1 Sound Board Tapped • Distortion Guitar & Enclosure Delay LPF2 String Output • Acoustic Strings + Trigger Impulse Response Line • Sound Examples LPF3 • Linearized Violin • Commuted Piano • Pulse Synthesis • Soundboard and enclosure are commuted • Complete Piano • Sound Examples • Only need a stored recording of their impulse response • Phy Audio Coding • An enormous digital ﬁlter is otherwise required • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 75 / 84 Piano and Harpsichord Sound Examples

Overview (Stanford Sondius Project, ca. 1995) Early Digital Synthesis

Spectral Modeling • Piano: (WAV) (MP3) Physical Modeling • Harpsichord 1: (WAV) (MP3) • KL Music • Digital Waveguide • Harpsichord 2: (WAV) (MP3) • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 76 / 84 More Recent Harpsichord Example

Overview • Harpsichord Soundboard Hammer-Response: (WAV) (MP3) Early Digital Synthesis • Musical Commuted Harpsichord Example: (WAV) (MP3) Spectral Modeling

Physical Modeling Reference: • KL Music • Digital Waveguide • Signal Scattering “Sound Synthesis of the Harpsichord Using a Computationally • Plucked String Efﬁcient Physical Model”, • Struck String • Karplus Strong • EKS Algorithm by Vesa Valim¨ aki,¨ Henri Penttinen, Jonte Knif, Mikael Laurson, • Clarinet • Wind Examples and Cumhur Erkut • Bowed Strings • Distortion Guitar • Acoustic Strings JASP-2004 • Sound Examples • Linearized Violin • Commuted Piano Google search: Harpsichord Sound Synthesis • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 77 / 84 Physical Modeling in Audio Coding

Spectral modeling synthesis is ﬁnding application in audio coding. Overview Can physical modeling synthesis be used as well? Early Digital Synthesis

Spectral Modeling • MPEG-4/SAOL already supports essentially all sound synthesis Physical Modeling methods • KL Music • Digital Waveguide • Ability to encode sounds automatically is limited • Signal Scattering • Plucked String ◦ Codebook-Excited Linear Prediction (CELP) is a successful • Struck String source-ﬁlter model (not quite physical) • Karplus Strong • EKS Algorithm ◦ There are many isolated examples of model-ﬁtting to • Clarinet • Wind Examples recorded data • Bowed Strings ◦ Good model-based denoising results have been obtained • Distortion Guitar • Acoustic Strings ◦ Coder problem much harder when many sources are mixed • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 78 / 84 Best Known Model-Based Audio Coders

Overview A “cover band” can put together a very convincing facsimile of Early Digital Synthesis popular music performance Spectral Modeling

Physical Modeling • KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding JOS high-school band “Bittersweet” • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 79 / 84 Future Physical Modeling in Audio Coding?

Overview A “Cover Band” Approach to Model-Based Audio Coding: Early Digital Synthesis

Spectral Modeling 1. Recognize individual “audio streams” in a mix (CASA)

Physical Modeling (“I hear a trap set, electric bass, Fender Rhodes, and a strat”) • KL Music 2. For each stream, calibrate its model heuristically • Digital Waveguide • Signal Scattering (“Here is what I hear the bass part doing: ...”) • Plucked String 3. Fine-tune the synthetic mix to the real mix • Struck String • Karplus Strong (joint “maximum likelihood estimation”) • EKS Algorithm • Clarinet Features of “Cover-Band Coding” (CBC): • Wind Examples • Bowed Strings • The “playing experience” of each “virtual performer” prevents • Distortion Guitar • Acoustic Strings artifacts — “musically unreasonable” parameters are made • Sound Examples unlikely (“Bayesian priors”) • Linearized Violin • Commuted Piano • An incorrect instrument must “imitate” its assigned stream • Pulse Synthesis • Complete Piano • New arrangements can be synthesized by deliberately choosing • Sound Examples a new ensemble! • Phy Audio Coding • Phy Audio Coding?

Summary Julius Smith AES-2006 Heyser Lecture – 80 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Summary

Julius Smith AES-2006 Heyser Lecture – 81 / 84 Summary

Overview We have reviewed a “CCRMA-centric slice” of the history of digital Early Digital Synthesis sound synthesis (usually starting with results from Bell Labs): Spectral Modeling

Physical Modeling • Wavetable (one period)

Summary • Subtractive • Additive • FM • Sampling • Spectral Modeling • Physical Modeling (more in tomorrow’s 4:30 PM masterclass) • Connections to audio coding

Julius Smith AES-2006 Heyser Lecture – 82 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary Sound Acknowledgment

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 83 / 84 Sound Acknowledgment

Overview Thanks to Emu / Creative Labs for providing a superb-quality Early Digital Synthesis external D/A converter for this talk (an E-Mu 0404/USB) Spectral Modeling

Physical Modeling

Summary

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 84 / 84