SOUND SYNTHESIS FABIÁN ESQUEDA NATIVE INSTRUMENTS GMBH 1.3.2019 © 2003 – 2019 VESA VÄLIMÄKI AND FABIÁN ESQUEDA SOUND SYNTHESIS 1.3.2019 OUTLINE ‣ Introduction ‣ Introduction to Synthesis ‣ Additive Synthesis ‣ Subtractive Synthesis ‣ Wavetable Synthesis ‣ FM and Phase Distortion Synthesis ‣ Synthesis, Synthesis, Synthesis SOUND SYNTHESIS 1.3.2019 Introduction ‣ BEng in Electronic Engineering with Music Technology Systems (2012) from University of York. ‣ MSc in Acoustics and Music Technology in (2013) from University of Edinburgh. ‣ DSc in Acoustics and Audio Signal Processing in (2017) from Aalto University. ‣ Thesis topic: Aliasing reduction in nonlinear processing. ‣ Published on a variety of topics, including audio effects, circuit modeling and sound synthesis. ‣ My current role is at Native Instruments where I work as a Software Developer for our Synths & FX team. ABOUT NI SOUND SYNTHESIS 1.3.2019 About Native Instruments ‣ One of largest music technology companies in Europe. ‣ Founded in 1996. ‣ Headquarters in Berlin, offices in Los Angeles, London, Tokyo, Shenzhen and Paris. ‣ Team of ~600 people (~400 in Berlin), including ~100 developers. SOUND SYNTHESIS 1.3.2019 About Native Instruments - History ‣ First product was Generator – a software modular synthesizer. ‣ Generator became Reaktor, NI’s modular synthesis/processing environment and one of its core products to this day. ‣ The Pro-Five and B4 were NI’s first analog modeling synthesizers. SOUND SYNTHESIS 1.3.2019 About Native Instruments ‣ Pioneered software instruments and digital DJing. ‣ Combining software development, hardware engineering, sound design. ‣ Strong focus on integrated software/hardware systems (e.g. Maschine). SOUND SYNTHESIS 1.3.2019 Team Structure ‣ Cross-functional teams working using the Agile methodology ‣ Usual roles: ‣ Application developers ‣ DSP developers ‣ User experience designers (UX) ‣ User interface designers (UI) ‣ Sound designers ‣ Quality and assurance engineers (QA) ‣ Team direction set by Product Owner (PO) ‣ Day to day organization and people management handled by Agile Coach. SOUND SYNTHESIS 1.3.2019 The Agile Process ‣ Work using fast two week iterations called sprints. ‣ Tasks specified from a user’s perspective: ‣ As a … [user] … ‣ I want … [desire] … ‣ So that … [benefit] … ‣ Divide larger tasks into smaller ones that can be implemented and tested within one sprint. ‣ Goal is to always have a potentially “releasable” product at the end of a sprint. SOUND SYNTHESIS 1.3.2019 Audio DSP Development at NI ‣ Low-level research on DSP algorithms and technique. ‣ Turn algorithms into engaging devices for users (w/ input from UX designers and sound designers) ‣ Efficient C++ implementation and integration into larger products (working closely with application developers). ‣ Participate actively in workshops, “hackathons” and trainings. INTRODUCTION TO SOUND SYNTHESIS SOUND SYNTHESIS 1.3.2019 Applications of Sound Synthesis MAKING MUSIC COMMUNICATION COMPUTER GAMING AND VIRTUAL REALITY TEST SIGNALS CAN YOU GUESS THE SOUND? SOUND SYNTHESIS 1.3.2019 SOUND SYNTHESIS 1.3.2019 SOUND SYNTHESIS 1.3.2019 Cool reading: 'Pop 'n Pour': This Electronic Music Pioneer Created the Sound of Coke's Beloved Bubbles: https://www.coca-colacompany.com/stories/meet-suzanne-ciani-the-legendary-creator-of-cokes-pop-n-pour SOUND SYNTHESIS 1.3.2019 THE PROMISE OF DIGITAL SOUND SYNTHESIS ▸ “I have indicated how almost any sound can be produced by treating the numbers generated by a computer as samples of the sound pressure wave.” — Max Mathews, Science, 1963 ▸ If sound can be stored by numbers then it must be possible to compute those numbers… RIGHT? SOUND SYNTHESIS 1.3.2019 Problems in Sound Synthesis ▸ How to compute sound efficiently? ▸ Realistic synthesis requires a complicated system. ▸ If simplified too much, sounds artificial. ▸ Computers are getting faster. ▸ How to play synthetic sound? ▸ Control data must be obtained from user. ▸ Solved partly by real-time control interfaces and music software (e.g., MIDI devices, sequencers). SOUND SYNTHESIS 1.3.2019 (Linear vs Nonlinear) ADDITIVE SYNTHESIS SOUND SYNTHESIS 1.3.2019 ADDITIVE SYNTHESIS ▸ Also called Fourier synthesis or sinusoidal modeling. ▸ Each partial is generated separately! ▸ Accurate control but lots of data! ▸ Extensions to the method: ▸ Sines + Noise Modeling (Serra & Smith, 1990) ▸ Sines + Noise + Transients (Verma & Meng, 2000) ▸ FFT–1 synthesis (Rodet and Depalle, 1992) Source: http://www.cs.princeton.edu/~prc ADDITIVE SYNTHESIS DEMO: RAZOR SUBTRACTIVE SYNTHESIS – PART I SOUND SYNTHESIS 1.3.2019 SUBTRACTIVE SYNTHESIS – PART I ▸ Started by Bob Moog (1934 – 2015) in the 1960s ▸ (Most likely) started with publication of Moog’s seminal paper A Voltage- Controlled Low-Pass High-Pass Filter for Audio Signal Processing at the 1965 AES Convention. ▸ Moog’s instruments were designed around the traditional keyboard interface. SOUND SYNTHESIS 1.3.2019 FUNDAMENTALS OF SUBTRACTIVE SYNTHESIS ▸ Based on the source–filter model ▸ Start with a signal that has a rich frequency spectrum ▸ Process it using a filter Source Filter Sound ▸ Perhaps it should be called “source–filter synthesis”? Nothing is subtracted, really… IS IT REALLY THAT SIMPLE? NO SOUND SYNTHESIS 1.3.2019 AN ACTUAL ANALOG SUBTRACTIVE SYNTHESIZER SOUND SYNTHESIS 1.3.2019 AN ACTUAL ANALOG SUBTRACTIVE SYNTHESIZER SOUND SYNTHESIS 1.3.2019 IMPLEMENTING SUBTRACTIVE SYNTHESIS ▸ Example architecture of a subtractive synthesizer: Envelope Envelope Oscillator 1 Mixer Lowpass Filter Amplifier Out Oscillator 2 ▸ One or more oscillators typically used as the source ▸ Second- or fourth-order resonant lowpass filters typically used ▸ Envelope generators (ADSR) used for modulation SOUND SYNTHESIS 1.3.2019 OSCILLATORS IN SUBTRACTIVE SYNTHESIS ▸ Periodic geometric waveforms are at the heart of subtractive synthesis ▸ These waveforms are harmonically- rich ▸ Might contain all or only even harmonics ▸ Digital emulation of these waveforms must suppress aliasing (Figure from: T. D. Rossing: The Science of Sound. Second Edition. Addison-Wesley, 1990.) SOUND SYNTHESIS 1.3.2019 IMPLEMENTING SUBTRACTIVE SYNTHESIS Envelope Envelope Oscillator 1 Mixer Lowpass Filter Amplifier Out Oscillator 2 SOUND SYNTHESIS 1.3.2019 ENVELOPE GENERATORS IN SUBTRACTIVE SYNTHESIS SOUND SYNTHESIS 1.3.2019 A SECOND LOOK AT THE KORG MS-20 SOUND SYNTHESIS 1.3.2019 VIRTUAL ANALOG (VA) Emulation of analog audio circuits in the digital domain SUBTRACTIVE SYNTHESIS DEMO REAKTOR SOUND SYNTHESIS 1.3.2019 ALIASING IN VA OSCILLATOR WAVEFORMS ▸ Rationale: ▸ Periodic geometric waveforms contain infinite harmonics. ▸ Sampling theorem tells us sampling rate must be at least twice the highest frequency component. ▸ Highest frequency component is infinite. ▸ No sample rate will ever be high enough! ▸ Synthesizing geometric waveforms trivially is equivalent to sampling a waveform with infinite frequency content. SOUND SYNTHESIS 1.3.2019 ALIASING: THE MOVIE ▸ Example: trivial sawtooth waveform (e.g. using a wrapping ramp function) ▸ Aliasing causes distinctive artifacts: ▸ Inharmonicity ▸ Beating ▸ Completely useless! Video by Andreas Franck, 2012. SOUND SYNTHESIS 1.3.2019 ALIASING: THE MOVIE ▸ Example: Sawtooth waveform generated using additive synthesis. ▸ Generate only components below Nyquist limit. ▸ Method is computationally expensive! Video by Andreas Franck, 2012. SOUND SYNTHESIS 1.3.2019 VA OSCILLATOR ALGORITHMS ▸ Bandlimited Synthesis Methods ▸ Additive synthesis and it’s variations. ▸ Quasi-Bandlimited Synthesis Methods ▸ BLIT: Bandlimited impulse train + filtering (Stilson & Smith, ICMC’96) ▸ MinBLEP: Minimum-phase bandlimited step (Brandt, ICMC’01) ▸ PolyBLEP: Polynomial bandlimited step (Välimäki & Huovilainen, 2007) ▸ Alias-Suppressing Synthesis Methods ▸ Oversampling ▸ DPW: Differentiated parabolic waveform (Välimäki, 2005) ▸ Post-Processing Synthesis Suppression Methods ▸ Aliasing suppression via filtering (Pekonen & Välimäki, 2008) SOUND SYNTHESIS 1.3.2019 DPW ALGORITHM ▸ An innovative way to generate sawtooth waveforms with reduced aliasing. Published by V Välimäki in IEEE Signal Processing Letters, March, 2005. ▸ Motivation: If aliasing is attenuated sufficiently, its effects can be neglected. ▸ Algorithm is extremely simple to implement and requires two input parameters only: fundamental frequency and sampling rate. f Modulo Counter 2 FIR Differentiator Out (Trivial Sawtooth) ( . ) fs SOUND SYNTHESIS 1.3.2019 DWP ALGORITHM EXAMPLE 1 ▸ Trivial Sawtooth x[n] 0 -1 0 10 20 30 40 50 1 ▸ Squared signal (Parabolic Waveform) 0.5 x2[n] 0 0 10 20 30 40 50 1 ▸ Differentiated Signal 0 c(x2[n] − x2[n − 1]) -1 0 10 20 30 40 50 Discrete time SOUND SYNTHESIS 1.3.2019 DPW ALGORITHM EXAMPLE (CONT’D) 0 ▸ Trivial Sawtooth Spectrum -20 -40 Level(dB) -60 0 5 10 15 20 0 ▸ Parabolic Waveform Spectrum -20 -40 Level(dB) -60 0 5 10 15 20 0 ▸ Differentiated Signal -20 Spectrum -40 Level(dB) -60 0 5 10 15 20 Frequency (kHz) SOUND SYNTHESIS 1.3.2019 COMPARISON OF ALGORITHMS ▸ Test signal: musical scale at high fundamental frequencies at audio rate (44.1 kHz) ▸ Trivial sawtooth ▸ DPW sawtooth ▸ Ideal sawtooth (additive synthesis) SOUND SYNTHESIS 1.3.2019 GENERATING RECTANGULAR WAVEFORMS USING DPW ▸ One possible approach (shown in diagram): ▸ Subtract two sawtooth waveforms. ▸ Second sawtooth has a 50% phase offset. ▸ Modulate phase offset for easy pulse width modulation (PWM). ▸ Second option (not shown): ▸ Use a single sawtooth and delay a copy of it using variable delay line. f Sawtooth Waveform −+ Out f Sawtooth Waveform