Historical Overview of Audio Spectral Modeling Julius Smith CCRMA, Stanford University Music 421 Applications Lecture April 2, 2018 Julius Smith Music 421 Applications Lecture – 1 / 59 Milestones in Audio Spectral Modeling • Fourier’s theorem (1822) Outline • Telharmonium (1898) Telharmonium • Voder (1920s) Voder • Channel Vocoder Vocoder (1920s) Phase Vocoder • Hammond Organ (1930s) Additive Synthesis • Phase Vocoder (1966) FM Synthesis • Digital Organ (1968) Sinusoidal Modeling • Additive Synthesis (1969) Spectrogram Synth • FM Brass Synthesis (1970) DDSP • Synclavier 8-bit FM/Additive synthesizer (1975) Future • FM singing voice (1978) • Sinusoidal Modeling (1985) • Sines + Noise (1988) • Sines + Noise + Transients (1988,1996,1998,2000) • Inverse FFT synthesis (1992) • Spectrogram Synthesis (2017) • Future Directions Julius Smith Music 421 Applications Lecture – 2 / 59 Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Future Telharmonium (1898) Julius Smith Music 421 Applications Lecture – 3 / 59 Telharmonium (Cahill 1898) U.S. patent 580,035: “Art of and Apparatus for Generating and Distributing Music Electrically” Julius Smith Music 421 Applications Lecture – 4 / 59 Telharmonium Rheotomes Forerunner of the Hammond Organ Tone Wheels Julius Smith Music 421 Applications Lecture – 5 / 59 Telharmonium Rotor (early “Tonewheel”) Hammond influenced: https://en.wikipedia.org/wiki/Tonewheel Julius Smith Music 421 Applications Lecture – 6 / 59 Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Future The Voder (1939) Julius Smith Music 421 Applications Lecture – 7 / 59 The Voder (Homer Dudley — 1939 Worlds Fair) > http://davidszondy.com/future/robot/voder.htm Julius Smith Music 421 Applications Lecture – 8 / 59 Voder Keyboard http://www.acoustics.hut.fi/publications/files/theses/ lemmetty mst/chap2.html — (from Klatt 1987) Julius Smith Music 421 Applications Lecture – 9 / 59 Voder Schematic http://ptolemy.eecs.berkeley.edu/~eal/audio/voder.html Julius Smith Music 421 Applications Lecture – 10 / 59 Voder Demos • Video Outline • Audio Telharmonium Voder • Voder Keyboard • Voder Schematic • Voder Demos Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Future Julius Smith Music 421 Applications Lecture – 11 / 59 Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP The Channel Vocoder (1928) Future (“Voice Coder”) Julius Smith Music 421 Applications Lecture – 12 / 59 Vocoder Analysis & Resynthesis (Dudley 1928) Analysis: Outline Telharmonium • Ten analog bandpass filters between 250 and 3000 Hz: Voder Bandpass → rectifier → lowpass filter → amplitude envelope Channel Vocoder • Voiced/Unvoiced decision made • Vocoder Examples • Fundamental frequency F0 measured for voiced case Phase Vocoder Additive Synthesis Synthesis: FM Synthesis • Ten matching bandpass filters driven by a Sinusoidal Modeling Spectrogram Synth ◦ “buzz source” (voiced), or DDSP ◦ “hiss source” (unvoiced) Future • Bands were scaled by amplitude envelopes and summed • Said to have an “unpleasant electrical accent” Related Speech Models: • The Vocoder is an early source-filter model for speech • Linear Predictive Coding (LPC) of speech is another Julius Smith Music 421 Applications Lecture – 13 / 59 Vocoder Filter Bank Analysis/Resynthesis magnitude, or Outline magnitude and phase extraction Telharmonium Voder A Channel Vocoder x0(t) xˆ0(t) • Vocoder Examples f Phase Vocoder Data Compression, A Transmission, Additive Synthesis xˆ(t) x(t) x1(t) Storage, xˆ1(t) Manipulation, FM Synthesis f Noise reduction, ... Sinusoidal Modeling Spectrogram Synth DDSP Future A xN−1 xˆN−1 f Analysis Processing Synthesis Julius Smith Music 421 Applications Lecture – 14 / 59 Channel Vocoder Sound Examples Outline • Original Telharmonium Voder • 10 channels, sine carriers Channel Vocoder • • Vocoder Examples 10 channels, narrowband-noise carriers Phase Vocoder Additive Synthesis • 26 channels, sine carriers FM Synthesis • 26 channels, narrowband-noise carriers Sinusoidal Modeling • 26 channels, narrowband-noise carriers, channels reversed Spectrogram Synth DDSP • Phase Vocoder: Identity system in absence of modifications Future • The FFT Phase Vocoder next transitioned to the Short-Time Fourier Transform (STFT) (Allen and Rabiner 1977) Julius Smith Music 421 Applications Lecture – 15 / 59 Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Future The Phase Vocoder (1966) Julius Smith Music 421 Applications Lecture – 16 / 59 Phase Vocoder Analysis for Additive Synthesis (1976) Outline Analysis Model Synthesis Model ω ∆ω t Telharmonium Channel Filter ak ak(t) k+ k( ) Voder Response A F ∆ω Sine Osc Channel Vocoder k ω Phase Vocoder 0 ωk Out Additive Synthesis • Early “channel vocoder” implementations (hardware) only FM Synthesis Sinusoidal Modeling measured amplitude ak(t) (Dudley 1939) • Spectrogram Synth The “phase vocoder” (Flanagan and Golden 1966) added phase DDSP tracking in each channel Future • Portnoff (1976) developed the FFT phase vocoder, which replaced the heterodyne comb in computer-music additive-synthesis analysis (James A. Moorer) • Inverse FFT synthesis (Rodet and Depalle 1992) gave faster sinusoidal oscillator banks Julius Smith Music 421 Applications Lecture – 17 / 59 Amplitude and Frequency Envelopes ak(t) Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling t ωk t φ˙k t Spectrogram Synth ∆ ( )= ( ) DDSP Future 0 t Julius Smith Music 421 Applications Lecture – 18 / 59 Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Future Additive Synthesis (1969) Julius Smith Music 421 Applications Lecture – 19 / 59 Classic Additive-Synthesis Analysis (Heterodyne Comb) Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis • Additive Analysis • Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Future John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online) Julius Smith Music 421 Applications Lecture – 20 / 59 Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes) Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis • Additive Analysis • Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Future John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online) Julius Smith Music 421 Applications Lecture – 21 / 59 Classic Additive Synthesis Diagram (Computer Music, 1960s) Outline A1(t) f1(t) A2(t) f2(t) A3(t) f3(t) A4(t) f4(t) Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis • Additive Analysis • Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP noise Future FIR Σ 4 t y(t)= Ai(t)sin ωi(t)dt + φi(0) Xi=1 Z0 Julius Smith Music 421 Applications Lecture – 22 / 59 Classic Additive-Synthesis Examples Outline • Bb Clarinet Telharmonium • Eb Clarinet Voder • Oboe Channel Vocoder • Bassoon Phase Vocoder • Tenor Saxophone Additive Synthesis • Additive Analysis • Trumpet • Additive Synthesis • English Horn FM Synthesis • French Horn Sinusoidal Modeling • Flute Spectrogram Synth DDSP • Future All of the above • Independently synthesized set (Synthesized from original John Grey data) Julius Smith Music 421 Applications Lecture – 23 / 59 Outline Telharmonium Voder Channel Vocoder Phase Vocoder Additive Synthesis FM Synthesis Sinusoidal Modeling Spectrogram Synth DDSP Frequency Modulation Synthesis Future (1973) Julius Smith Music 421 Applications Lecture – 24 / 59 Frequency Modulation (FM) Synthesis Outline FM synthesis is normally used as a spectral modeling technique Telharmonium • Voder Discovered and developed (1970s) by John M. Chowning Channel Vocoder (CCRMA Founding Director) Phase Vocoder • Key paper: JAES 1973 (vol. 21, no. 7) Additive Synthesis • Commercialized by Yamaha Corporation: FM Synthesis ◦ • FM Synthesis DX-7 synthesizer (1983) • FM Formula ◦ OPL chipset (SoundBlaster PC sound card) • FM Patch • FM Spectra ◦ Cell phone ring tones • FM Examples • FM Voice • On the physical modeling front, synthesis of vibrating-string Sinusoidal Modeling waveforms using finite differences started around this time: Spectrogram Synth Hiller & Ruiz, JAES 1971 (vol. 19, no. 6) DDSP Future Julius Smith Music 421 Applications Lecture – 25 / 59 FM Formula Outline Telharmonium x(t)= Ac sin[ωct + φc + Am sin(ωmt + φm)] Voder where Channel Vocoder Phase Vocoder (Ac,ωc,φc) specify the carrier sinusoid Additive Synthesis (Am,ωm,φm) specify the modulator sinusoid FM Synthesis • FM Synthesis Can also be called phase modulation • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice Sinusoidal Modeling Spectrogram Synth DDSP Future Julius Smith Music 421 Applications Lecture – 26 / 59 Simple FM “Brass” Patch (Chowning 1970–) Jean-Claude Risset observation (1964–1969): Outline Brass bandwidth ∝ amplitude Telharmonium Voder Channel Vocoder Phase Vocoder fm = f0 Additive Synthesis g FM Synthesis • FM Synthesis A F • FM Formula • FM Patch • FM Spectra • FM Examples fc = f0 • FM Voice Sinusoidal Modeling Spectrogram Synth A F DDSP Future Out Julius Smith Music 421 Applications Lecture – 27 / 59 FM Harmonic Amplitudes (Bessel