History and Practice of Digital Sound Synthesis Julius Smith CCRMA, Stanford University AES-2006 Heyser Lecture October 6, 2006 Julius Smith AES-2006 Heyser Lecture – 1 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary Overview Julius Smith AES-2006 Heyser Lecture – 2 / 84 Outline Overview Digital sound synthesis approaches in approximate historical order: • Outline • CCRMA Perspective • Wavetable (one period) Early Digital Synthesis • Subtractive Spectral Modeling • Additive Physical Modeling • Frequency Modulation (FM) Summary • Sampling • Spectral Modeling • Physical Modeling Some connections with audio coding will be noted Emphasis: • Sound examples • Block diagrams • Historical notes Julius Smith AES-2006 Heyser Lecture – 3 / 84 CCRMA Perspective Overview • Outline • CCRMA Perspective Early Digital Synthesis Spectral Modeling Physical Modeling Summary The Knoll, Stanford University Julius Smith AES-2006 Heyser Lecture – 4 / 84 Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary Early Digital Sound Synthesis Julius Smith AES-2006 Heyser Lecture – 5 / 84 Wavetable Synthesis in Music I-V (1957-1969) Overview Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary Julius Smith AES-2006 Heyser Lecture – 6 / 84 Music V Scripting Language (“Note Cards”) Overview Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary • Essentially Supported in MPEG-4 Structured Audio Orchestra Language (SAOL) (Music V → csound → SAOL) • “Encoding sounds” as “instruments” is hard, in general Julius Smith AES-2006 Heyser Lecture – 7 / 84 Kelly-Lochbaum Vocal Tract Model Overview Early Digital Synthesis • Music V … y(n) • KL Music e(n) • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis Glottal Pulse • FM Formula Train or Noise 1 + k 1 1 + k M • FM Patch − − Speech e(n) z 1 2 … z 1 2 • FM Spectra Output • FM Examples y(n) k 1 − k 1 R 1 k M − k M R M • FM Voice • Sampling Synthesis (Unused − … − • Modern Example Allpass z 1 2 z 1 2 Output) 1 − k 1 1 − k M Spectral Modeling Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical) Physical Modeling Summary John L. Kelly and Carol Lochbaum (1962) Julius Smith AES-2006 Heyser Lecture – 8 / 84 Sound Example Overview “Bicycle Built for Two”: (WAV) (MP3) Early Digital Synthesis • Music V • KL Music • “Daisy” • Vocal part by Kelly and Lochbaum (1961) • Additive Analysis • Musical accompaniment by Max Mathews • Additive Synthesis • FM Synthesis • Computed on an IBM 704 • FM Formula • Based on Russian speech-vowel data from Gunnar Fant’s book • FM Patch • FM Spectra • Probably the first digital physical-modeling synthesis sound • FM Examples • FM Voice example by any method • Sampling Synthesis • Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey” • Modern Example — the computer’s “first song” Spectral Modeling Physical Modeling Summary Julius Smith AES-2006 Heyser Lecture – 9 / 84 Classic Additive-Synthesis Analysis (Heterodyne Comb) Overview Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online) Julius Smith AES-2006 Heyser Lecture – 10 / 84 Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes) Overview Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online) Julius Smith AES-2006 Heyser Lecture – 11 / 84 Classic Additive Synthesis Diagram Overview A1(t) f1(t) A2(t) f2(t) A3(t) f3(t) A4(t) f4(t) Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling noise FIR Physical Modeling Σ Summary 4 t y(t)= Ai(t) sin ωi(t)dt + φi(0) Z0 Xi=1 Julius Smith AES-2006 Heyser Lecture – 12 / 84 Classic Additive-Synthesis Examples Overview • Bb Clarinet Early Digital Synthesis • Eb Clarinet • Music V • KL Music • Oboe • “Daisy” • • Additive Analysis Bassoon • Additive Synthesis • Tenor Saxophone • FM Synthesis • FM Formula • Trumpet • FM Patch • English Horn • FM Spectra • FM Examples • French Horn • FM Voice • Flute • Sampling Synthesis • Modern Example Spectral Modeling • All of the above Physical Modeling • Independently synthesized set Summary (Synthesized from original John Grey data) Julius Smith AES-2006 Heyser Lecture – 13 / 84 Frequency Modulation (FM) Synthesis Overview FM synthesis is normally used as a spectral modeling technique. Early Digital Synthesis • Music V • Discovered and developed (1970s) by John M. Chowning • KL Music • “Daisy” (CCRMA Founding Director) • Additive Analysis • Key paper: JAES 1973 (vol. 21, no. 7) • Additive Synthesis • FM Synthesis • Commercialized by Yamaha Corporation: • FM Formula • FM Patch ◦ DX-7 synthesizer (1983) • FM Spectra ◦ OPL chipset (SoundBlaster PC sound card) • FM Examples • FM Voice ◦ Cell phone ring tones • Sampling Synthesis • Modern Example • On the physical modeling front, synthesis of vibrating-string Spectral Modeling waveforms using finite differences started around this time: Physical Modeling Hiller & Ruiz, JAES 1971 (vol. 19, no. 6) Summary Julius Smith AES-2006 Heyser Lecture – 14 / 84 FM Formula Overview Early Digital Synthesis x(t)= Ac sin[ωct + φc + Am sin(ωmt + φm)] • Music V • KL Music where • “Daisy” • Additive Analysis (Ac, ωc, φc) specify the carrier sinusoid • Additive Synthesis • FM Synthesis (Am, ωm, φm) specify the modulator sinusoid • FM Formula • FM Patch Can also be called phase modulation • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary Julius Smith AES-2006 Heyser Lecture – 15 / 84 Simple FM “Brass” Patch (1970–) Jean-Claude Risset observation (1964–1969): Overview Brass bandwidth ∝ amplitude Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis 0 • Additive Synthesis fm = f • FM Synthesis • FM Formula g • FM Patch A F • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example fc = f0 Spectral Modeling Physical Modeling A F Summary Out Julius Smith AES-2006 Heyser Lecture – 16 / 84 FM Harmonic Amplitudes (Bessel Function of First Kind) Overview Harmonic number k, FM index β: Early Digital Synthesis • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula 1 • FM Patch • FM Spectra • FM Examples • FM Voice 0.5 • Sampling Synthesis ) β ( k • Modern Example J 0 Spectral Modeling 0 Physical Modeling 2 −0.5 Summary 4 0 5 10 6 15 20 8 25 Order k 30 10 Argument β Julius Smith AES-2006 Heyser Lecture – 17 / 84 Frequency Modulation (FM) Examples Overview All examples by John Chowning unless otherwise noted: Early Digital Synthesis • Music V • FM brass synthesis • KL Music • “Daisy” ◦ Low Brass example • Additive Analysis • Additive Synthesis ◦ Dexter Morril’s FM Trumpet • FM Synthesis • FM Formula • FM singing voice (1978) • FM Patch Each formant synthesized using an FM operator pair • FM Spectra • FM Examples (two sinusoidal oscillators) • FM Voice • Sampling Synthesis ◦ Chorus • Modern Example ◦ Voices Spectral Modeling ◦ Basso Profundo Physical Modeling Summary • Other early FM synthesis ◦ Clicks and Drums ◦ Big Bell ◦ String Canon Julius Smith AES-2006 Heyser Lecture – 18 / 84 FM Voice Overview FM voice synthesis can be viewed as compressed modeling of Early Digital Synthesis spectral formants • Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis Carrier 2 • FM Synthesis Carrier 1 • FM Formula Carrier 3 • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis Magnitude • Modern Example Spectral Modeling Physical Modeling Summary 0 Frequency f Modulation Frequency (all three) Julius Smith AES-2006 Heyser Lecture – 19 / 84 Sampling Synthesis History Overview • 1979 - Fairlight Computer Music Instrument - 8-bit Early Digital Synthesis • Music V ◦ First commercial sampler • KL Music ◦ • “Daisy” Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono) • Additive Analysis ◦ Editing, looping, mixing • Additive Synthesis • FM Synthesis ◦ One could draw waveforms and additive-synthesis amplitude • FM Formula envelopes (for each harmonic) with a light pen • FM Patch • FM Spectra ◦ $25,000–$36,000! • FM Examples • FM Voice • 1981 - E-mu Systems Emulator • Sampling Synthesis • Modern Example ◦ First “affordable” sampler ($10,000) Spectral Modeling ◦ Eight voices, 128 K RAM, 8-bit, 80 lb. Physical Modeling Summary • 1986 - Ensoniq Mirage ◦ Breakthrough price-point ($1695) ◦ Eight voices, 144 K RAM, 8-bit Julius Smith AES-2006 Heyser Lecture – 20 / 84 Modern Sampled Piano Overview Example:1 Early Digital Synthesis • Music V • 40 Gigabytes on ten DVDs (three sampled pianos)