History of Virtual Musical Instruments and Effects Based on Physical Modeling Principles

Julius Smith CCRMA, Stanford University

Digital Audio Effects (DAFx-17) Conference Keynote 1

September 6, 2017

Julius Smith DAFx-17 – 1 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Overview

Julius Smith DAFx-17 – 2 / 50˜ Outline

Overview Virtual Instruments and Effects • Outline • • Virtual Instruments Physical Modeling Overview • Physical Models History of Virtualization • Finite Differences Virtual Voice • Early History Virtual Strings • Voice Models Selected Recent Developments String Models •

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research

Outro

Julius Smith DAFx-17 – 3 / 50˜ Virtualization

Merriam-Webster deﬁnes virtual as • ... being on or simulated on a computer or computer network — print or virtual books — of, relating to, or existing within a virtual reality ... According to Elon Musk: • “There’s a billion to one chance we’re living in base reality” Therefore, it could be “virtual all the way down” •

Let’s say virtualization involves one level of simulation •

Julius Smith DAFx-17 – 4 / 50˜ Example: Virtual Duck Call

A duck call is a virtual quacking device • A duck-call synthesizer is technically a virtual virtual (V 2) duck •

Duck Call (Virtual Quacker)

More typical unit

Julius Smith DAFx-17 – 5 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Physical Models

Julius Smith DAFx-17 – 6 / 50˜ Physical Models

We generally follow Isaac Newton’s System of the World:

No relativistic effects (yet) • No quantum effects (almost) • Newtonian mechanics sufﬁcient • ⇒ Newton’s three laws of motion (1686) can be summarized by one classic equation: f = ma (Force = Mass times Acceleration)

∆ Expresses conservation of momentum ( f =p ˙, p = mv ) • Models based on Newton’s laws can quickly become complex • Need many simpliﬁcations that preserve both sound quality and • expressivity of control

Julius Smith DAFx-17 – 7 / 50˜ Physical Modeling Formulations

Our physical modeling tool box: Ordinary Differential Equations (ODE) [f(t)=p ˙(t)= m v˙(t)= ma(t)] • ′′ Partial Differential Equations (PDE) [Ky (x, t)= ǫ y¨(x, t)] • Difference Equations (DE) [p(n +1) = p(n)+ Tf(n)] • Finite Difference Schemes (FDS) [p(n +1) = p(n)+ Tf(n + 1/2)] • (Physical) State Space Models (Vector First-Order ODE) p˙ = Ap + Bu • Transfer Functions (between physical signals) [H(s)= P (s)/F (s)] • Modal Representations (Parallel Biquads) [H(s)= Hk(s)] • Pk Equivalent Circuits and their various Solvers (Node Analysis, ...) • Impedance Networks [ Lumped Models] • → Wave Digital Filters (WDF) [Masses/Inductors, Springs/Capacitors, ...] • Digital Waveguide (DW) Networks [Strings, Acoustic Tubes, ...] •

Julius Smith DAFx-17 – 8 / 50˜ Recent History

Numerical Sound Synthesis, Stefan Bilbao, Wiley 2009

Julius Smith DAFx-17 – 9 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Finite Differences

Julius Smith DAFx-17 – 10 / 50˜ Finite-Difference Approaches (2009)

Overview

Physical Models

Finite Differences • Wave Digital Filters (WDF), Recent Results • Bassman Example

Early History

Voice Models

String Models

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research

Outro

Julius Smith DAFx-17 – 11 / 50˜ Wave Digital Filters (WDF), Recent Results

Overview See 2015 Keynote talk (and the papers mentioned) for details:

Physical Models WDFs can now model arbitrary circuit topologies Finite Differences • • Wave Digital Filters (not just parallel/series connections) (WDF), Recent Results • Bassman Example Any number of nonlinear elements can be included • Early History See recent PhD thesis by Kurt Werner (and recent DAFx papers) • Voice Models Nonlinear Newton solvers remain an active area of research String Models •

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research

Outro

Julius Smith DAFx-17 – 12 / 50˜ Fender Bassman Example

Overview “The Fender Bassman 5F6-A Family of Preampliﬁer Circuits—A Physical Models Wave Digital Filter Case Study,” Finite Differences • Wave Digital Filters Ross Dunkel, Maximilian Rest, Kurt James Werner, Michael Jørgen (WDF), Recent Results • Bassman Example Olsen and Julius O. Smith

Early History

Voice Models T1B T1N T2 T3 CB2

String Models VPK VGK VPK VGK VPK VGK VPK VGK RGB2 COB CB1 Bowed Strings RIB RVBP Distortion Guitar RGN1A iOS Guitars R Recent Research PB RGN1B

Outro RPN RIN

RGB1A

CON RVNN RVBN RGN2 RK2 RL RP2 VP CK1 RGB1B VIN

RVNP RK1 Ross Dunkel’s Fender Bassman WDF (Note the 25-port R-Node)

Julius Smith DAFx-17 – 13 / 50˜ Back to History

Numerical Sound Synthesis, Stefan Bilbao, Wiley 2009

Julius Smith DAFx-17 – 14 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Early Virtualization of Strings

Julius Smith DAFx-17 – 15 / 50˜ Early Virtualization of Strings

Overview The vibrating string was a critically important object of study in the Physical Models middle-18th century, leading to Finite Differences The ﬁrst wave equation (ﬁrst PDE) Early History • • Virtualization Traveling-wave solution of the string wave equation • Darrigol • • Mersenne Additive synthesis solution for the terminated string • • Sauveur First glimpses of Fourier-series expansions • Taylor • • Rameau (before Fourier was born in 1768) • J Bernoulli The concept of superposition in linear systems • D Bernoulli • • Harmonics • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 16 / 50 Main Reference

Overview Excellent account of the period discussed here: Physical Models Finite Differences Olivier Darrigol Early History “The Acoustic Origins of Harmonic Analysis” • Virtualization • Darrigol Archive for History of the Exact Sciences, 2007 • Mersenne • Sauveur • Taylor • Rameau • J Bernoulli • D Bernoulli • Harmonics • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 17 / 50 Marin Mersenne (1636) - On Audible Overtones

“[Since the vibrating string] produces ﬁve or six tones..., it seems that it is entirely necessary that it beat the air ﬁve, four, three, and two times at the same time, which is impossible to imagine unless one says that half of the string beats the air twice, one third beats it three times, etc. while the whole strings beats it once. This picture runs against experience, which clearly shows that all parts of the string make the same number of returns in the same time, because the continuous string has a single motion, even though parts near the bridge move more slowly.”

Plucked string video: • https://www.youtube.com/watch?v=Qr rxqwc1jE Since there was no notion of spectrum at this time, the fundamental • frequency of a sound was the periodic repetition rate of pulses in the time domain

Julius Smith DAFx-17 – 18 / 50˜ Joseph Sauveur (1701)

Overview “While meditating on the phenomena of sound, I was Physical Models made to observe that especially at night one may hear Finite Differences

Early History from long strings not only the principal sound, but also • Virtualization other small sounds, a twelfth and a seventeenth above.... • Darrigol • Mersenne I concluded that the string in addition to the undulations • Sauveur it makes in its entire length so as to form the • Taylor • Rameau fundamental sound may divide itself in two, three, four, • J Bernoulli • D Bernoulli etc. undulations which form the octave, the twelfth, the • Harmonics ﬁfteenth of this sound.” • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 19 / 50 Harmonic Overtones Discovered

Overview

Physical Models

Finite Differences

Early History • Virtualization • Darrigol • Mersenne • Sauveur • Taylor • Rameau • J Bernoulli • D Bernoulli • Harmonics • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution Sauveur plucked a monochord having a light obstacle mounted to • Fourier • • Animations create “nodes” • Composers and Geometers He was surprised that the string did not move at the nodal points • Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 20 / 50 Sauveur, Continued

String-playing musicians surely discovered how to play harmonics by lightly • damping a node (The Lyre dates back to perhaps 3000 BC) Sauveur coined the term node, inspired by nodes in the lunar orbit: • Lunar nodes are points where the orbit of the Moon crosses the ecliptic ◦ The ecliptic is the apparent path of the sun around the celestial sphere ◦ An eclipse can only happen at one of the two lunar node points ◦ Sauveur also coined the term harmonic, so named because they are • “harmonious” with the fundamental

Julius Smith DAFx-17 – 21 / 50˜ Brook Taylor (1713)

Best known for Taylor Series Expansions • First to derive the string fundamental frequency f = K/ǫ/2L, • p where K = string tension, ǫ = mass density, L = string length Derived that the string restoring force is proportional to string curvature • Approximated this by the distance of the string from its rest axis (oops) • (but this works for a sinusoidal displacement) Concluded that a sinusoidal shape was the only possibility, • despite Wallis having described higher-order modes previously Although many initial shapes were clearly possible, he assumed that the • vibration would quickly assume a sinusoidal shape (only one mode of vibration supported by the string—the fundamental mode) In other words, he assumed the string vibrated like a mass-spring system •

Julius Smith DAFx-17 – 22 / 50˜ Jean-Philippe Rameau (1726)

Rameau was a composer interested in studying harmonic overtones toward • establishing a more scientific foundation for music theory His theory of harmony was based on the first three audible harmonics • Cited Sauveur and Mersenne • Collaborated extensively with d’Alembert • Anticipated a kind of spectrum analyzer theory of hearing: • “What has been said of [the separate vibrating modes of] sonorous bodies should be applied equally to the fibers which carpet the bottom of the ears cochlea [le fond de la conque de loreille]; these fibers are so many sonorous bodies, to which the air transmits its vibrations, and from which the perception of sounds and harmony is carried to the soul.” Thus, Rameau regarded the hair cells as a bank of little sympathetic resonators. Crude forms of the “place theory” of hearing (aka “resonance theory”) are said to have begun in 1605, and Helmholtz published his treatment in 1857.

Julius Smith DAFx-17 – 23 / 50˜ Johann Bernoulli (1727)

Overview Johann Bernoulli studied the mass-loaded ideal massless string, Physical Models also called the beaded string, thereby avoiding the need for a PDE Finite Differences

Early History Made the same mistake as Taylor by assuming the restoring • Virtualization • • Darrigol force was proportional to distance from the rest axis, instead of • Mersenne the force component in that direction applied by adjacent string • Sauveur • Taylor segments • Rameau Also overlooked higher modes of vibration, considering only the • J Bernoulli • • D Bernoulli fundamental • Harmonics • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 24 / 50 Daniel Bernoulli (1733)

Overview Daniel Bernoulli extended his father’s work: Physical Models Studied the vertically suspended chain Finite Differences • Observed multiple modes of vibration (as many as there were Early History • • Virtualization masses), and their inharmonicity • Darrigol • Mersenne Found the limiting mode shapes (now called Bessel functions) • • Sauveur Realized that the inﬁnitely long chain was equivalent to a musical • Taylor • • Rameau string • J Bernoulli Investigated higher-order modes experimentally • D Bernoulli • • Harmonics Went on to study elastic bands: • D’Alembert • • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 25 / 50 Daniel Bernoulli (1742)

Daniel Bernoulli studied vibrating elastic bands with inharmonic modes, hearing them out:

“Both sounds exist at once and are very distinctly perceived.... This is no wonder, since neither oscillation helps or hinder the other; indeed, when the band is curved by reason of one oscillation, it may always be considered as straight in respect to another oscillation, since the oscillations are virtually inﬁnitely small. Therefore oscillations of any kind may occur, whether the band be destitute of all other oscillation or executing others at the same time. In free bands, whose oscillations we shall now examine,I have often perceived three or four sounds at the same time.”

Superposition of small vibrations at different frequencies clearly conceived • Likely suggested by the perceptual superposition of overtones • Did the spectrum analyzer nature of hearing give us the concept of • superposition?

Julius Smith DAFx-17 – 26 / 50˜ String Harmonic Overtones

Overview

Physical Models

Finite Differences

Early History • Virtualization • Darrigol • Mersenne • Sauveur • Taylor • Rameau • J Bernoulli • D Bernoulli • Harmonics “Obvious” to string-playing musicians • D’Alembert • • Euler Observed by Mersenne, Sauveur, Bernoulli, and others • Bernoulli Replies • • Not obvious to everybody! Math Puzzle • • Paradoxes (Taylor, J. Bernoulli, d’Alembert, Euler, . . . ) • Resolution • Fourier Pushed as “reality” most strenuously by Daniel Bernoulli • Animations • • D’Alembert and Euler to Bernoulli: Composers and • Geometers How can sinusoids add up to a propagating pulse?!? Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 27 / 50 Jean La Rond D’Alembert (1746)

Overview Invented the PDE by plugging Taylor’s restoring force f into • Physical Models Newton’s f = ma written as a differential form Finite Differences (as developed by Euler) Early History • Showed that any solution was a traveling wave to left and/or right Virtualization • • Darrigol Showed that Taylor’s sinusoidal fundamental mode was a special • Mersenne • • Sauveur case solution • Taylor Disagreed with Taylor that all initial conditions lead to a sinusoid • Rameau • • J Bernoulli Did not allow an initial triangular shape (not twice differentiable), • D Bernoulli • • Harmonics and suggested using a beaded string for this case • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 28 / 50 Leonard Euler (1749)

Overview Euler quickly generalized d’Alembert’s results to any initial string • Physical Models shape Finite Differences He showed that the solution space included Bernoulli sums of Early History • • Virtualization “Taylorian sines”: • Darrigol • Mersenne y(x, t)= A sin(kπx/L)cos(kπf t) • Sauveur X k 0 • Taylor • Rameau • J Bernoulli • D Bernoulli He did not consider this a general solution • Harmonics • • D’Alembert (one “obviously” could not make a triangular initial shape out of • Euler sines, for example) • Bernoulli Replies • We of course know that it is quite general, but Joseph Fourier Math Puzzle • • Paradoxes was not yet born (to happen 19 years later in 1768) • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 29 / 50 Daniel Bernoulli (1753)

Bernoulli was annoyed with d’Alembert and Euler: Overview

Physical Models He had published the superposition-of-sinusoids solution long • Finite Differences ago Early History The supposedly new solutions of d’Alembert and Euler were • Virtualization • • Darrigol simply a mixture of simple modes • Mersenne He didn’t like fusing the simple pure oscillations into a single • Sauveur • • Taylor formula • Rameau He considered his “physical” analysis preferable to their abstract • J Bernoulli • • D Bernoulli mathematical treatment: • Harmonics • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 30 / 50 Daniel Bernoulli (1753)

Overview “I saw at once that one could admit this multitude of

Physical Models [Taylorian sine] curves only in a sense altogether

Finite Differences improper. I do not less admire the calculations of Early History Messrs. dAlembert and Euler, which certainly include • Virtualization • Darrigol what is most profound and most advanced in all of • Mersenne analysis, but which show at the same time that an • Sauveur • Taylor abstract analysis, if heeded without any synthetic • Rameau [physical] examination of the question proposed, is more • J Bernoulli • D Bernoulli likely to surprise than enlighten. It seems to me that • Harmonics • D’Alembert giving attention to the nature of the vibrations or strings • Euler sufﬁces to foresee without any calculation all that these • Bernoulli Replies • Math Puzzle great geometers have found by the most difﬁcult and • Paradoxes abstract calculations that the analytic mind has yet • Resolution • Fourier conceived.” • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 31 / 50 The Mathematical Puzzle of the Vibrating String

Daniel Bernoulli (1733): Physical vibrations can be understood Overview • as a superposition of “simple modes” (pure sinusoidal vibrations) Physical Models

Finite Differences

Early History In Euler’s formulation: • • Virtualization ∞ • Darrigol • Mersenne y(t, x)= Ak sin(kπx/L)cos(kπνt) • Sauveur X • Taylor k=0 • Rameau • J Bernoulli (displacement of length L vibrating string at time t, position x) • D Bernoulli • Harmonics • D’Alembert D’Alembert (1747): String vibration can be understood as a pair • Euler • • Bernoulli Replies of traveling-waves going in opposite directions at speed c: • Math Puzzle • Paradoxes + x − x • Resolution y(t, x)= y t + y t + • Fourier − c c • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 32 / 50 Mathematical Paradoxes

Overview Reasonable question of the day: Physical Models

Finite Differences How can a superposition of standing waves give you a Early History propagating pulse? • Virtualization • Darrigol • Mersenne ∞ • Sauveur • Taylor y(t, x) = Ak sin(kπx/L)cos(kπνt) • Rameau X k=0 • J Bernoulli • x − x D Bernoulli =? y+ t + y t + • Harmonics c c • D’Alembert − • Euler • Bernoulli Replies Another reasonable question of the day: • Math Puzzle • Paradoxes How can a sum of sinusoids give an arbitrary (e.g., • Resolution • Fourier non-smooth) function? • Animations • Composers and Geometers Life without Fourier theory was difﬁcult indeed

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 33 / 50 Paradox Resolved

Overview Thanks to Fourier theory, we now know that the • Physical Models sum-of-standing-waves and traveling-waves are interchangeable Finite Differences and essentially complete descriptions: Early History • Virtualization Standing-wave = sum of opposite-going traveling waves • Darrigol ◦ • Mersenne Any initial state can be projected onto standing-wave “basis • ◦ Sauveur functions” and reconstructed • Taylor • Rameau For the ideal vibrating string, the basis functions are the • J Bernoulli ◦ • D Bernoulli sinusoidal harmonics • Harmonics For more general systems, the basis functions are • D’Alembert ◦ • Euler eigenfunctions of Hermitian linear operators beyond the basic • Bernoulli Replies wave equation (Lagrange, Sturm-Liouville) • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 34 / 50 Fourier Expansion

First 4 Sinusoidal Components of a Leaning-Triangle String Shape 1

0.5

0 Displacement

-0.5

-1 0 0.5 1 1.5 2 Position

First four modes of plucked string, showing both right-going and left-going string images (two string copies).

Julius Smith DAFx-17 – 35 / 50˜ Animations by Dan Russel

Overview From http://www.acs.psu.edu/drussell/demos.html Physical Models

Finite Differences 1. [Modes of a hanging chain]

Early History 2. [Standing waves on a string] • Virtualization 3. [Standing wave as two traveling waves] • Darrigol • Mersenne • Sauveur • Taylor • Rameau • J Bernoulli • D Bernoulli • Harmonics • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 36 / 50 Notable Collaborations between Composers and Geometers

Overview Composers and scientists have frequently banded together to Physical Models pursue joint goals and understanding. Finite Differences The long-time collaboration between Rameau and d’Alembert can be Early History • Virtualization regarded as an early instance of “computer music:” • Darrigol • Mersenne 1700s - Jean-Philippe Rameau and Jean La Rond D’Alembert • Sauveur • • Taylor 1960s - Herb Deutsche and Bob Moog • Rameau • • J Bernoulli ADSR envelope was suggested by Deutsche • D Bernoulli ◦ It was prototyped immediately by Moog using a doorbell • Harmonics ◦ • D’Alembert Originally the notes were only gated (on/off) • Euler ◦ • Deutsche said articulation was needed—”consider the ’ta’ of Bernoulli Replies ◦ • Math Puzzle the trumpet” • Paradoxes • Resolution Moog immediately thought of one-pole ﬁltering of the gate • ◦ Fourier with variable Attack and Release • Animations • Composers and (Probably Decay to Sustain Level as well, but not mentioned Geometers

Voice Models explicitly)

String Models ˜ BowedJulius Strings Smith DAFx-17 – 37 / 50 Collaborations, Continued

1960s - Lejaren Hiller and Pierre Ruiz Overview • 1966 - John Lennon and Ken Townsend: Flanging effect: Physical Models • Finite Differences Ken Townsend was an engineer at EMI’s Abbey Road Studio ◦ Early History John Lennon suggested there should be an automatic way to • Virtualization ◦ • Darrigol get the sound of double-tracked vocals • Mersenne Townsend developed Artificial Double Tracking (ADT) • Sauveur ◦ • Taylor George Martin jokingly explained it to Lennon as a “double • ◦ Rameau vibrocated sploshing flange with double negative feedback” • J Bernoulli • D Bernoulli Lennon later called it “flanging” and this may have been the • Harmonics ◦ • D’Alembert origin of later usage • Euler • Bernoulli Replies (Source: Wikipedia) • Math Puzzle See Also “Double vibrocated sploshing flange” • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 38 / 50 Collaborations, Continued

1970s - John Chowning and Dave Poole, Andy Moorer, Peter Overview • Samson, and many others: Physical Models

Finite Differences PDP-1 DAC (Poole) • Early History FM Bessel functions (AI Lab engineer) • Virtualization • • Darrigol 12-bit D/A (Moorer) • • Mersenne Samson Box (Samson and Moorer) • Sauveur • • Taylor CCRMA (Moorer, Grey, et al.) • Rameau • • J Bernoulli 1970s - Prof. Barry Vercoe (MIT, CSound) and • D Bernoulli • • Harmonics Miller Puckett (FTS on Analogic AP500 array processor, • D’Alembert • • Euler score following) • Bernoulli Replies Roger Dannenberg • Math Puzzle • • Paradoxes Joe Paradiso • • Resolution ... • Fourier • • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 39 / 50 Collaborations, Continued

1980s - Pierre Boulez and Andrew Gerzso: IRCAM, Concerts Overview • 1983 - David Jaffe and JOS (Music DMA & EE PhD students): Physical Models • Extended Karplus Strong (“make it play in tune” etc.) Finite Differences 1990 - Prof. Paul Lansky (Princeton) and Charles Sullivan: Early History • • Virtualization Distortion-feedback guitar • Darrigol • Mersenne • Sauveur • Taylor • Rameau • J Bernoulli • D Bernoulli • Harmonics • D’Alembert • Euler • Bernoulli Replies • Math Puzzle • Paradoxes • Resolution • Fourier • Animations • Composers and Geometers

Voice Models

String Models ˜ BowedJulius Strings Smith DAFx-17 – 40 / 50 Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Voice Models

Julius Smith DAFx-17 – 41 / 50˜ Early Talking Machines (Virtual Talking Heads)

Joseph Faber’s Euphonia

Julius Smith DAFx-17 – 42 / 50˜ Early Brazen Heads (Wikipedia)

1125: First talking head description: • ≈ William of Malmesbury’s History of the English Kings: Pope Sylvester II said to have traveled to al-Andalus and stolen a tome ◦ of secret knowledge Only able to escape through demonic assistance ◦ Cast the head of a statue using his knowledge of astrology ◦ It would not speak until spoken to, but then answered any yes/no ◦ question put to it Early devices were deemed heretical by the Church and often destroyed • 1599: Albertus Magnus had a head with a human voice and breath: • ”A certain reasoning process” bestowed by a cacodemon (evil demon) ◦ Thomas Aquinas destroyed it for continually interrupting his ruminations ◦ (not everyone wants a yacking cacodemon around all the time) By the 18th century, talking machines became acceptable as • “scientiﬁc pursuit”

Julius Smith DAFx-17 – 43 / 50˜ Wolfgang Von Kempelin’s Speaking Machine (1791)

Overview

Physical Models

Finite Differences

Early History

Voice Models • Virtual Heads • Von Kempelin • Euphonia • Voder Keyboard • Voder Schematic • Voder Demos • KL Voice • “Daisy” • “Shiela” • Linear Prediction • Glottal Model • Source Estimation • LF Glottal Model • Phonation Variation • Lu Sounds • Tak Replica of Wolfgang Von Kempelin’s Speaking Machine • 2D Vowels • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 44 / 50˜ Distortion Guitar Joseph Faber’s Euphonia (1846)

Overview

Physical Models

Finite Differences

Early History

Voice Models • Virtual Heads • Von Kempelin • Euphonia • Voder Keyboard • Voder Schematic • Voder Demos • KL Voice • “Daisy” • “Shiela” • Linear Prediction • Glottal Model • Source Estimation • LF Glottal Model • Phonation Variation • Lu Sounds • Tak • 2D Vowels Joseph Faber’s Euphonia • Pink Trombone 17 levers, a bellows, and a telegraphic line — sang “God Save the Queen” String Models

JuliusBowed Strings Smith DAFx-17 – 45 / 50˜ Distortion Guitar Joseph Faber’s Euphonia (1846)

Overview

Physical Models

Finite Differences

Early History

String Models

JuliusBowed Strings Smith DAFx-17 – 46 / 50˜ Distortion Guitar Joseph Faber’s Euphonia The Voder (Homer Dudley — 1939 Worlds Fair)

> http://davidszondy.com/future/robot/voder.htm

Julius Smith DAFx-17 – 47 / 50˜ Voder Keyboard

http://www.acoustics.hut.fi/publications/files/theses/ lemmetty mst/chap2.html — (from Klatt 1987)

Julius Smith DAFx-17 – 48 / 50˜ Voder Schematic

http://ptolemy.eecs.berkeley.edu/~eal/audio/voder.html

Julius Smith DAFx-17 – 49 / 50˜ Voder Demos

Video Overview • Audio Physical Models • Finite Differences

Early History

String Models

JuliusBowed Strings Smith DAFx-17 – 50 / 50˜ Distortion Guitar Kelly-Lochbaum Vocal Tract Model (Discrete-Time Transmission-Line Model)

Overview

Physical Models

Finite Differences

Early History

Voice Models • Virtual Heads • Von Kempelin • Euphonia • Voder Keyboard • Voder Schematic • Voder Demos • KL Voice • “Daisy” • “Shiela” • Linear Prediction • Glottal Model • Source Estimation • LF Glottal Model • Phonation Variation John L. Kelly and Carol Lochbaum (1962) • Lu Sounds • Tak • 2D Vowels • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 51 / 50˜ Distortion Guitar Sound Example

Overview “Bicycle Built for Two”: (WAV) (MP3) Physical Models

Finite Differences

Early History Vocal part by Kelly and Lochbaum (1961) • Voice Models Musical accompaniment by Max Mathews • Virtual Heads • Computed on an IBM 704 • Von Kempelin • • Euphonia Based on Russian speech-vowel data from Gunnar Fant’s book • Voder Keyboard • • Probably the ﬁrst digital physical-modeling synthesis sound Voder Schematic • • Voder Demos example by any method • KL Voice • “Daisy” Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey” • • “Shiela” — the computer’s “ﬁrst song” • Linear Prediction • Glottal Model • Source Estimation • LF Glottal Model • Phonation Variation • Lu Sounds • Tak • 2D Vowels • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 52 / 50˜ Distortion Guitar “Shiela” Sound Examples by Perry Cook (1990)

Overview

Physical Models

Finite Differences

Early History Voice Models Diphones: (WAV) (MP3) • Virtual Heads • • Von Kempelin Nasals: (WAV) (MP3) • • Euphonia Scales: (WAV) (MP3) • Voder Keyboard • • Voder Schematic “Shiela”: (WAV) (MP3) • Voder Demos • • KL Voice • “Daisy” • “Shiela” • Linear Prediction • Glottal Model • Source Estimation • LF Glottal Model • Phonation Variation • Lu Sounds • Tak • 2D Vowels • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 53 / 50˜ Distortion Guitar Linear Prediction (LP) Vocal Tract Model

Can be interpreted as a modified Kelly-Lochbaum model Overview • In linear prediction, the glottal excitation must be an Physical Models • Finite Differences impulse, or • Early History white noise Voice Models • • Virtual Heads This prevents LP from finding a physical vocal-tract model • Von Kempelin A more realistic glottal waveform e(n) is needed before the vocal • Euphonia • • Voder Keyboard tract filter can have the “right shape” • Voder Schematic How to augment LPC in this direction without going to a • Voder Demos • • KL Voice full-blown articulatory synthesis model? • “Daisy” • “Shiela” • Linear Prediction • Glottal Model Jointly estimate glottal waveform e(n) so that the vocal- • Source Estimation • LF Glottal Model tract filter converges to the “right shape” • Phonation Variation • Lu Sounds • Tak • 2D Vowels • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 54 / 50˜ Distortion Guitar Klatt Derivative Glottal Wave

Two periods of the basic voicing waveform Overview

Physical Models 0.05 Finite Differences 0 Early History

Voice Models −0.05 • Virtual Heads • Von Kempelin Amplitude −0.1 • Euphonia • Voder Keyboard −0.15 • Voder Schematic • Voder Demos −0.2 • KL Voice • “Daisy” −0.25 • “Shiela” 0 25 50 75 100 125 150 175 200 225 250 Time (samples) • Linear Prediction • Glottal Model Good for estimation: • Source Estimation • LF Glottal Model • Phonation Variation Truncated parabola each period • Lu Sounds • • Tak Coefficients easily fit to phase-aligned inverse-filter output • 2D Vowels • • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 55 / 50˜ Distortion Guitar Sequential Unconstrained Minimization

(Hui Ling Lu, 2002) Overview

Physical Models Klatt glottal (parabola) parameters are estimated jointly with vocal Finite Differences tract filter coefficients Early History Voice Models Formulation resembles that of the equation error method for • Virtual Heads • • Von Kempelin system identification (also used in invfreqz in matlab) • Euphonia For phase alignment, we estimate • Voder Keyboard • • Voder Schematic pitch (time varying) • Voder Demos • • KL Voice glottal closure instant each period • “Daisy” • • “Shiela” Optimization is convex in all but the phase-alignment dimension • Linear Prediction • • Glottal Model one potentially nonlinear line search • Source Estimation ⇒ • LF Glottal Model • Phonation Variation • Lu Sounds • Tak • 2D Vowels • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 56 / 50˜ Distortion Guitar Liljencrantz-Fant Derivative Glottal Wave Model LF glottal wave and LF derivative glottal wave 40 Uo glottal wave Overview 30 Physical Models 20 Finite Differences Amplitude Early History 10

Voice Models 0 Tp Tc To • Virtual Heads 0 0.005 0.01 0.015 Time (sec) • Von Kempelin • Euphonia Ta • Voder Keyboard derivative glottal wave • Voder Schematic 0 Tp Te Tc To • Voder Demos

• KL Voice −0.5 •

“Daisy” Amplitude • “Shiela” −1 • Linear Prediction −Ee • Glottal Model 0 0.005 0.01 0.015 Time (sec) • Source Estimation • LF Glottal Model • Phonation Variation Better for intuitively parametrized expressive synthesis • Lu Sounds • LF model parameters are fit to inverse filter output • Tak • • 2D Vowels Use of Klatt model in forming filter estimate yields a “more • Pink Trombone • physical” filter than LP String Models

JuliusBowed Strings Smith DAFx-17 – 57 / 50˜ Distortion Guitar Parametrized Phonation Types

Overview pressed Physical Models 1

Finite Differences 0

Early History −1 Voice Models 100 200 300 400 500 600 700 800 900 1000 • Virtual Heads • Von Kempelin 1 normal • Euphonia • Voder Keyboard 0 • Voder Schematic • Voder Demos −1 • KL Voice 100 200 300 400 500 600 700 800 900 1000 • “Daisy” • “Shiela” 1 breathy • Linear Prediction • Glottal Model 0 • Source Estimation

• LF Glottal Model −1 • Phonation Variation 100 200 300 400 500 600 700 800 900 1000 • Lu Sounds • Tak • 2D Vowels • Pink Trombone

String Models

JuliusBowed Strings Smith DAFx-17 – 58 / 50˜ Distortion Guitar Sound Examples by Hui Ling Lu

Original: (WAV) (MP3) Overview • Synthesized: Physical Models • Finite Differences Pressed Phonation: (WAV) (MP3) • Early History Normal Phonation: (WAV) (MP3) • Voice Models Breathy Phonation: (WAV) (MP3) • Virtual Heads • • Von Kempelin Original: (WAV) (MP3) • Euphonia • • Voder Keyboard Synthesis 1: (WAV) (MP3) • Voder Schematic • Synthesis 2: (WAV) (MP3) • Voder Demos • • KL Voice • “Daisy” where • “Shiela” • Linear Prediction • Glottal Model Synthesis 1 = Estimated Vocal Tract driven by estimated • Source Estimation • • LF Glottal Model KLGLOT88 Derivative Glottal Wave (Pressed) • Phonation Variation Synthesis 2 = Estimated Vocal Tract driven by the ﬁtted LF • Lu Sounds • • Tak Derivative Glottal Wave (Pressed) • 2D Vowels • Pink Trombone Google search: singing synthesis Lu String Models

JuliusBowed Strings Smith DAFx-17 – 59 / 50˜ Distortion Guitar Voice Model Estimation

Overview (Pamornpol (Tak) Jinachitra 2006) Physical Models

Finite Differences Noise/Error Noise Early History v(n)Vocal tract w(n) Voice Models • Virtual Heads 1 x(n) • Von Kempelin g(n) A(z) y(n) • Clean Euphonia Derivative Noisy • Voder Keyboard speech speech • Voder Schematic glottal waveform • Voder Demos • KL Voice • “Daisy” Parametric source-ﬁlter model of voice + noise • “Shiela” • • Linear Prediction State-space framework with derivative glottal waveform as • • Glottal Model input and A model for dynamics • Source Estimation • LF Glottal Model Jointly estimate AR parameters and glottal source parameters • Phonation Variation • • Lu Sounds using EM algorithm with Kalman smoothing • Tak Reconstruct a clean voice using Kelly-Lochbaum and • 2D Vowels • • Pink Trombone estimated parameters

String Models

JuliusBowed Strings Smith DAFx-17 – 60 / 50˜ Distortion Guitar Online 2D Vowel Demo (2014)

Jan Schnupp, Eli Nelken, and Andrew King http://auditoryneuroscience.com/topics/two-formant-artificial-vowels

Julius Smith DAFx-17 – 61 / 50˜ Pink Trombone Voical Synthesis (March 2017) Neil Thapen Overview Institute of Mathematics of the Academy of Sciences Physical Models Czech Republic Finite Differences http://dood.al/pinktrombone/ Early History

String Models

JuliusBowed Strings Smith DAFx-17 – 62 / 50˜ Distortion Guitar Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Digital String Models

Julius Smith DAFx-17 – 63 / 50˜ Karplus-Strong (KS) Algorithm (1983)

Overview Physical Models + + Output y (n) y (n-N) Finite Differences N samples delay

Early History 1/2 Voice Models

String Models • 1/2 Karplus Strong -1 • EKS Algorithm z • Physical Excitation • Pick Position FFCF • Digital Waveguides • Waveguide Reverb • Allpass Networks Discovered (1978) as “self-modifying wavetable synthesis” • Digital Waveguide • • Wavetable is preferably initialized with random numbers Signal Scattering • • Moving Termination No physical interpretation until much later • Waveguide Model • • Plucked String • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 64 / 50˜ Outro EKS Algorithm (Jaffe-Smith 1983)

−N Overview Hp(z) Hβ (z) z HL(z)

Physical Models

Finite Differences Hρ(z) Hs(z) Hd(z) Early History Voice Models N = pitch period (2 string length) in samples String Models × • Karplus Strong 1 p Hp(z) = − − = pick-direction lowpass filter • EKS Algorithm 1 p z 1 • Physical Excitation − • −βN Pick Position FFCF Hβ(z) = 1 z = pick-position comb filter, β (0, 1) • Digital Waveguides − ∈ • Waveguide Reverb Hd(z) = string-damping filter (one/two poles/zeros typical) • Allpass Networks • Digital Waveguide Hs(z) = string-stiffness allpass filter (several poles and zeros) • Signal Scattering − • Moving Termination ρ(N) z 1 • Waveguide Model first-order string-tuning allpass filter Hρ(z) = − −1 = • Plucked String 1 ρ(N) z • Struck String − 1 RL Bowed Strings dynamic-level lowpass filter HL(z) = − −1 = 1 RL z Distortion Guitar − iOS Guitars

Recent Research Julius Smith DAFx-17 – 65 / 50˜ Outro Karplus-Strong Sound Examples

Overview “Vintage” 8-bit sound examples: Physical Models • Original Plucked String: (WAV) (MP3) Finite Differences • Drum: (WAV) (MP3) Early History • Stretched Drum: (WAV) (MP3) Voice Models • String Models • Karplus Strong • EKS Algorithm • Physical Excitation • Pick Position FFCF • Digital Waveguides • Waveguide Reverb • Allpass Networks • Digital Waveguide • Signal Scattering • Moving Termination • Waveguide Model • Plucked String • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 66 / 50˜ Outro EKS Sound Examples

Overview Plucked String: (WAV) (MP3) Physical Models

Finite Differences Plucked String 1: (WAV) (MP3) • Early History Plucked String 2: (WAV) (MP3) • Voice Models Plucked String 3: (WAV) (MP3) String Models • • Karplus Strong (Computed using in the C++ Synthesis Tool Kit • EKS Algorithm Plucked.cpp • Physical Excitation (STK) by Perry Cook and Gary Scavone) • Pick Position FFCF • Digital Waveguides • Waveguide Reverb • Allpass Networks • Digital Waveguide • Signal Scattering • Moving Termination • Waveguide Model • Plucked String • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 67 / 50˜ Outro EKS Sound Example (1988)

Overview Bach A-Minor Concerto—Orchestra Part: (WAV) (MP3) Physical Models

Finite Differences

Early History Executed in real time on one Motorola DSP56001 • Voice Models (20 MHz clock, 128K SRAM)

String Models • Karplus Strong • Developed for the NeXT Computer introduction at Davies EKS Algorithm • • Physical Excitation Symphony Hall, San Francisco, 1988 • Pick Position FFCF • Digital Waveguides • Waveguide Reverb Solo violin part was played live by Dan Kobialka of the San • Allpass Networks • • Digital Waveguide Francisco Symphony • Signal Scattering • Moving Termination • Waveguide Model • Plucked String • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 68 / 50˜ Outro Example EKS Extension

Overview

Physical Models

Finite Differences

Early History Voice Models Several of the Karplus-Strong algorithm extensions String Models • Karplus Strong were based on its physical interpretation. • EKS Algorithm • Physical Excitation • Pick Position FFCF • Digital Waveguides • Waveguide Reverb • Allpass Networks • Digital Waveguide • Signal Scattering Originally, transfer-function methods were used (1982) • Moving Termination • • Waveguide Model Below is a digital waveguide derivation • Plucked String • • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 69 / 50˜ Outro String Excited Externally at One Point

Example Output Overview f +(n) Del M Delay N Physical Models ªAgraffeº ªBridgeº Finite Differences Rigid Hammer Strike f(t) Filter Yielding Termination Termination Early History f -(n) Del M Delay N Voice Models

String Models (x = 0) (x = striking position) (x = L) • Karplus Strong • EKS Algorithm “Waveguide Canonical Form (1986)” • Physical Excitation • Pick Position FFCF • Digital Waveguides Equivalent System by Delay Consolidation: • Waveguide Reverb • Allpass Networks String Output • Digital Waveguide • Signal Scattering Hammer • Del 2M Delay 2N Moving Termination Strike f(t) • Waveguide Model • Plucked String Filter • Struck String

Bowed Strings

Distortion Guitar Finally, we “pull out” the comb-ﬁlter component: iOS Guitars

Recent Research Julius Smith DAFx-17 – 70 / 50˜ Outro EKS “Pick Position” Extension

Overview Equivalent System: Comb Filter Factored Out Physical Models

Finite Differences Out (from Del N)

Early History g(t) Hammer Voice Models Delay 2M+2N Strike f(t) String Models • Karplus Strong Del 2M Filter • EKS Algorithm • Physical Excitation − − • 2M N Pick Position FFCF −N 1+ z −2M z • Digital Waveguides H(z)= z = 1+ z −(2M+2N) −(2M+2N) • Waveguide Reverb 1 z 1 z • Allpass Networks − − • Digital Waveguide Excitation Position controlled by left delay-line length • Signal Scattering • • Moving Termination Fundamental Frequency controlled by right delay-line length • • Waveguide Model “Transfer function modeling” based on a physical model (1982) • Plucked String • • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 71 / 50˜ Outro Digital Waveguide Models (1985)

Overview Historically, it would make the most sense to say that digital Physical Models waveguide synthesis arose as follows: Finite Differences D’Alembert derived the string vibration as a superposition of left- Early History • Voice Models and right-going traveling-waves

String Models Acoustic tubes became known to obey the same form of wave • Karplus Strong • • EKS Algorithm equation as vibrating strings • Physical Excitation Signal scattering at impedance discontinuities was known from • Pick Position FFCF • • Digital Waveguides physics (elastic particle scattering theory) • Waveguide Reverb and was also incorporated in transmission-line theory • Allpass Networks • Digital Waveguide Kelly and Lochbaum conceived of the piecewise cylindrical tube • • Signal Scattering model of the vocal tract, and digitized it via sampling • Moving Termination • Waveguide Model Digital waveguide synthesis followed as sparsiﬁcation of • Plucked String • • Struck String Kelly-Lochbaum vocal synthesis, applied to strings Bowed Strings However, that’s not the actual story ... Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 72 / 50˜ Outro Digital Waveguide Reverberation (1985)

Overview How it really happened: Physical Models In a shuttle bus to ICMC-85, Gary Kendall (Northwestern U of IL) Finite Differences • Early History commented on how hard it was to safely modify large digital

Voice Models reverberators by adding feedback here and there

String Models Instability highly likely as a result of any “random” change • Karplus Strong • As a ﬁlter guy, I accepted the challenge: • EKS Algorithm • • Physical Excitation • Pick Position FFCF How do we connect any signal in a network to any • Digital Waveguides other point in the network without causing instability? • Waveguide Reverb • Allpass Networks • Digital Waveguide • Signal Scattering • Moving Termination • Waveguide Model • Plucked String • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 73 / 50˜ Outro Allpass Networks

Overview Lossless networks were clearly a good angle to pursue • Physical Models Reverberators are typically nearly allpass from point to point Finite Differences • How do we preserve the allpass property of a large network Early History • when editing it? Voice Models The main resource I studied was Belevitch: String Models • • Karplus Strong Classical Network Theory • EKS Algorithm Belevitch discussed allpass networks extensively • Physical Excitation • • Pick Position FFCF Belevitch also introduced the scattering-theory formulation of • Digital Waveguides • • Waveguide Reverb circuit theory (the basis for WDFs) • Allpass Networks The idea of closed waveguide networks for reverberation • Digital Waveguide • • Signal Scattering occurred while studying Belevitch (also using basic knowledge of • Moving Termination transmission-line theory) • Waveguide Model • Plucked String Waveguide synthesis was a later afterthought: • Struck String • Any waveguide branch could be treated as a vibrating string or Bowed Strings woodwind bore Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 74 / 50˜ Outro Digital Waveguide Models (1985)

∆ Overview Lossless digital waveguide = bidirectional delay line Physical Models at some wave impedance R Finite Differences Early History z−N Voice Models

String Models • Karplus Strong R • EKS Algorithm • Physical Excitation z−N • Pick Position FFCF • Digital Waveguides • Waveguide Reverb Useful for efﬁcient models of • Allpass Networks • Digital Waveguide strings • Signal Scattering • • Moving Termination bores • Waveguide Model • plane waves • Plucked String • • Struck String conical waves Bowed Strings •

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 75 / 50˜ Outro Signal Scattering

Overview Signal scattering is caused by a change in wave impedance R: Physical Models

Finite Differences

Early History

Voice Models

String Models • Karplus Strong • EKS Algorithm • Physical Excitation • Pick Position FFCF • Digital Waveguides • Waveguide Reverb • Allpass Networks • Digital Waveguide • Signal Scattering • Moving Termination • Waveguide Model • Plucked String • Struck String If the wave impedance changes every spatial sample, the

Bowed Strings Kelly-Lochbaum vocal-tract model results (also need reﬂecting Distortion Guitar terminations) iOS Guitars

Recent Research Julius Smith DAFx-17 – 76 / 50˜ Outro Moving Termination: Ideal String

Overview

Physical Models

Finite Differences

Early History

Voice Models

String Models • Karplus Strong • EKS Algorithm Moving rigid termination for an ideal string • Physical Excitation • Pick Position FFCF • Digital Waveguides Left endpoint moved at velocity v0 • Waveguide Reverb • External force f0 = Rv0 • Allpass Networks • • Digital Waveguide R = √Kǫ is the wave impedance (for transverse waves) • • Signal Scattering Relevant to bowed strings (when bow pulls string) • Moving Termination • • Waveguide Model String moves with speed v0 or 0 only • Plucked String • • String is always one or two straight segments Struck String • “Helmholtz corner” (slope discontinuity) shuttles back and forth at Bowed Strings • Distortion Guitar speed c = K/ǫ p iOS Guitars

Recent Research Julius Smith DAFx-17 – 77 / 50˜ Outro Digital Waveguide “Equivalent Circuits”

Overview v0 Physical Models

Finite Differences a) -1 -1 Early History

Voice Models

String Models (x = 0) (x = L) • Karplus Strong • EKS Algorithm • Physical Excitation • Pick Position FFCF = • Digital Waveguides f 0 R v0 f(n) • Waveguide Reverb • Allpass Networks • Digital Waveguide b) • Signal Scattering • Moving Termination • Waveguide Model (x = 0) (x = L) • Plucked String • Struck String

Bowed Strings a) Velocity waves. b) Force waves. Distortion Guitar (Animation) iOS Guitars (Interactive Animation) Recent Research Julius Smith DAFx-17 – 78 / 50˜ Outro Ideal Plucked String (Displacement Waves)

Overview

Physical Models

Finite Differences + + y (n) y (n-N/2) Early History

Voice Models ªBridgeº -1 (x = Pluck Position) -1 ªNutº String Models • Karplus Strong • EKS Algorithm y- (n) y- (n+N/2) • Physical Excitation • Pick Position FFCF • Digital Waveguides (x = 0) (x = L) • Waveguide Reverb • Allpass Networks • Digital Waveguide • Signal Scattering • Moving Termination Load each delay line with half of initial string displacement • Waveguide Model • Sum of upper and lower delay lines = string displacement • Plucked String • • Struck String

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 79 / 50˜ Outro Ideal Struck String (Velocity Waves)

Overview

Physical Models c Finite Differences + + v (n) v (n-N/2) Early History

Voice Models ªBridgeº -1 (x = Hammer Position) -1 ªNutº String Models • Karplus Strong c • EKS Algorithm v- (n) v- (n+N/2) • Physical Excitation • Pick Position FFCF (x = 0) (x = L) • Digital Waveguides • Waveguide Reverb • Allpass Networks • Digital Waveguide • Signal Scattering Hammer strike = momentum transfer = velocity step: • Moving Termination • Waveguide Model • Plucked String mhvh(0 )=(mh + ms)vs(0+) • Struck String −

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research Julius Smith DAFx-17 – 80 / 50˜ Outro Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Bowed Strings (1986)

Julius Smith DAFx-17 – 81 / 50˜ Digital Waveguide Bowed Strings (1986)

Overview

Physical Models

Finite Differences

Early History

Voice Models

String Models

Bowed Strings • Bowed Strings

Distortion Guitar iOS Guitars Reflection filter summarizes all losses per period Recent Research • (due to bridge, bow, finger, etc.) Outro Bow-string junction = memoryless lookup table • (or segmented polynomial)

Julius Smith DAFx-17 – 82 / 50˜ Bowed and Plucked Sound Examples by Esteban Maestre

Overview Synthetically Bowing STK’s Bowed.cpp • Physical Models Finite-Width Thermal Friction Model for Bowed String Finite Differences • Waveguide Strings Coupled to Modal-Synthesis Bridge Early History •

Voice Models

String Models

Bowed Strings • Bowed Strings

Distortion Guitar iOS Guitars

Recent Research

Outro

Julius Smith DAFx-17 – 83 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Electric Guitar with Overdrive Outro and Feedback

Julius Smith DAFx-17 – 84 / 50˜ Ampliﬁer Distortion + Ampliﬁer Feedback

Overview Sullivan 1990 Physical Models

Finite Differences Direct-signal gain

Early History String 1 Nonlinear Distortion Voice Models . Output String Models .

Bowed Strings Pre-distortion gain Post-distortion gain Distortion Guitar String N Ampliﬁer iOS Guitars Feedback Gain Recent Research Outro Ampliﬁer Feedback Delay

Distortion output signal often further filtered by an amplifier cabinet filter, representing speaker cabinet, driver responses, etc.

Julius Smith DAFx-17 – 85 / 50˜ Distortion Guitar Sound Examples

Overview (Stanford Sondius Project, ca. 1995) Physical Models Distortion Guitar: (WAV) (MP3) Finite Differences • Ampliﬁer Feedback 1: (WAV) (MP3) Early History • Voice Models Ampliﬁer Feedback 2: (WAV) (MP3) • String Models

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research

Outro

Julius Smith DAFx-17 – 86 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Virtual Electric Guitars Now

Julius Smith DAFx-17 – 87 / 50˜ moForte Guitar for iOS and (later) Android

Real-time on iPhone 4S and iPad 2 (and later) Guitar and effects written in the FAUST language:

Full physically modeled electric-guitar + effects: • Six vibrating strings — general excitations Distortion Feedback Compression Wah pedal or Autowah Phaser Flanger Five-band parametric equalizer Reverb Responds to • accelerometer, gyros, touches (plucks), swipes (strumming), . . . Hard to fully utilize ﬁve points of multitouch on iPhone and ten on iPad! • Android audio latency has gotten much better • The Android scheduler remains an issue • (need real-time protection for audio callbacks) JUCE + Faust looking good for Android version •

Julius Smith DAFx-17 – 88 / 50˜ These Effects Plus Six Feedback-Distortion Guitar Strings Together Require 115% of an iPhone 4S or iPad 2 CPU

Overview

Physical Models

Finite Differences

Early History

Voice Models

String Models

Bowed Strings

Distortion Guitar iOS Guitars • moForte Guitar • CPU Performance • CPU & GPU • GeoShred • GeoShred Demos

Recent Research

Outro

Julius Smith DAFx-17 – 89 / 50˜ ARM CPU Performance

Overview

Physical Models

Finite Differences

Early History

Voice Models

String Models

Bowed Strings

Distortion Guitar iOS Guitars • moForte Guitar • CPU Performance • CPU & GPU • GeoShred • GeoShred Demos

Recent Research

Outro

Julius Smith DAFx-17 – 90 / 50˜ Update: ARM CPU and GPU Performance as of iPhone 6s

Overview

Physical Models

Finite Differences

Early History

Voice Models

String Models

Bowed Strings

Distortion Guitar iOS Guitars • moForte Guitar • CPU Performance • CPU & GPU • GeoShred • GeoShred Demos

Recent Research

Outro

Julius Smith DAFx-17 – 91 / 50˜ GeoShred: Virtual Distortion Electric Guitar (iOS) GeoShred merges moForte Guitar with Geo Synthesizer for iOS

GeoShred for iOS The iPad turns out to be a kick-ass virtual musical instrument!

Julius Smith DAFx-17 – 92 / 50˜ GeoShred Demos

Overview GeoShred Demos Physical Models

Finite Differences Jordan on GeoShred (YouTube): • Early History GeoShred Pro 2.5 (Sep 2017) Voice Models ◦ NAMM 2015 String Models ◦ Bowed Strings JOS Under the Moon (YouTube) • Distortion Guitar iOS Guitars • moForte Guitar • CPU Performance • CPU & GPU • GeoShred • GeoShred Demos

Recent Research

Outro

Julius Smith DAFx-17 – 93 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro Selected Recent Research

Julius Smith DAFx-17 – 94 / 50˜ Scattering Delay Network (SDN), Four Walls

Overview

Physical Models

Finite Differences

Early History

Voice Models

String Models

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research • SDN • TASLP • 2D Bridge

Outro

Four-Wall Scattering Delay Network (SDN)

Julius Smith DAFx-17 – 95 / 50˜ Scattering Delay (Digital Waveguide) Networks (SDN)

Overview

Physical Models

Finite Differences

Early History f1,s1 f2,s2

Voice Models A A s1,s2 A String Models A d A A Bowed Strings

Distortion Guitar f1,s2 f2,s1 iOS Guitars

Recent Research • SDN Image Method Analysis of a Two-Wall SDN • TASLP • 2D Bridge All paths are present Outro • Direct and first-order reflections can be exact • Higher-order reflections are lengthened • Can combine with exact early reflections to obtain perceptual • equivalence

Julius Smith DAFx-17 – 96 / 50˜ IEEE Tr. Speech & Language Processing

“Joint Modeling of Bridge Admittance and Body Radiativity for Overview Efﬁcient Synthesis of String Instrument Sound by Digital Physical Models Waveguides” (IEEE-TASLP, March 2017; Finite Differences

Early History see also IEEE-SPL, Nov. 2016)

Voice Models Esteban Maestre, Gary Scavone, and Julius Smith

String Models −10 wProny Bowed Strings −20 −30 Distortion Guitar −40 iOS Guitars dB −50

Recent Research −60 • SDN −70

• TASLP 3 −10 10 wStmcb • 2D Bridge −20

−30 Outro −40

dB −50

−60

−70

−80 3 10 Hz Violin Radiativity Model

Julius Smith DAFx-17 – 97 / 50˜ 2D Bridge Modeling for Bowed Strings

Overview

Physical Models

Finite Differences

Early History

Voice Models

String Models

Bowed Strings

Distortion Guitar iOS Guitars

Recent Research • SDN • TASLP • 2D Bridge

Outro Two-dimensional Bridge Model

Julius Smith DAFx-17 – 98 / 50˜ Overview Physical Models Finite Differences Early History Voice Models String Models Bowed Strings Distortion Guitar iOS Guitars Recent Research Outro In Conclusion

Julius Smith DAFx-17 – 99 / 50˜ Conclusion

It all begins and ends with

string theory!

(Physics Joke)

Julius Smith DAFx-17 – 100 / 50˜ Summary

Overview Physical Modeling Approaches • Physical Models Update on Wave Digital Filters Finite Differences • Recent and Early History of Virtual Strings and Voice Early History • Update on Voice Models Voice Models • Update on String Models and Effects String Models • Selected Research Updates Bowed Strings • Distortion Guitar iOS Guitars

Recent Research

Outro • Conclusion • Summary

Julius Smith DAFx-17 – 101 / 50˜