Audio Effects Processing

Home , Audio filter, Double tracking, Flanging, Loudness war, Phaser (effect), Pumping (audio)

15.02.2019

Vesa Välimäki, Fabián Esqueda & Benoit Alary

DEMO: Pink Noise Generators Eero Lehtimäki & Uljas Pulkkis

1 Course Schedule in 2019 (Periods III, IV)

0. General issues (Vesa & Benoit) 11.1.2019 1. History and future of audio DSP (Vesa) 18.1.2019 2. Digital filters in audio (Vesa) 25.1.2019 3. Audio filter design (Vesa) 1.2.2019 4. Analysis of audio signals (Vesa) 8.2.2019 5. Audio effects processing (Vesa) 15.2.2019 * No lecture (Evaluation week for Period III) 22.2.2019 6. Synthesis of audio signals (Fabian) 1.3.2019 7. Reverberation and 3-D sound (Benoit) 8.3.2019 8. Physics-based sound synthesis (Vesa) 15.3.2019 9. Sampling rate conversion (Vesa) 22.3.2019 10. Audio coding (Vesa) 29.3.2019

Välimäki, Parker, Esqueda & Alary 15.2.2019 3

Outline

• Echo / Delay • Flanging and Phasing • Chorus • Pitch shifting / Time stretching • Dynamic processing (compression & expansion) • Other effects Demo

• Beat-Aligning Looper • Pink Noise Generators

Välimäki, Parker, Esqueda & Alary 15.2.2019 4

2 What is an audio effect? . Any kind of audio signal processing applied to a captured or synthesized sound for creative purposes

. Possible purposes: . Impression of space (echo, reverb) . Increasing perceived size of a sound (chorus) . Introducing movement into a static sound (flanging, phasing) . Altering timbre (distortion) . Altering dynamics (compression, limiting)

Echo/Delay . One of the simplest and earliest audio effects . Initially they were made using tape loops . Digital version very simple . Delay line with feedback . Filtering or distortion can be added to the feedback loop . Extra taps can be added for more complex pattern . Real-time implementation using “circular buffer”.*

Sound example taken from: http://en.wikipedia.org/wiki/Delay_%28audio_effect%29 *Good reference: The Audio Programming Book by R. Boulanger & V. Lazzarini

Välimäki, Parker, Esqueda & Alary 15.2.2019 6

3 Bucket-Brigade Devices (BBDs)

• Tape machines are expensive!  • BBDs are discrete-time analog delay lines • Invented by F. Sangster and K. Teer at the Phillips Research Labs in 1968. • Input signal is sampled in time and passed through a series of capacitors and switches. • Charge in each capacitor is passed to subsequent stage at a rate determined by clock.

http://www.electrosmash.com/mn3007-bucket-brigade-devices

Välimäki, Parker, Esqueda & Alary 15.2.2019

BBDs (cont’d)

• CLK 1 and 2 in anti-phase configuration. • BBD flangers typically have a single 1024-stage unit. • Number of stages fixed. Delay length determined by clock’s rate.

Välimäki, Parker, Esqueda & Alary 15.2.2019 8

4 BBDs (cont’d)

• Although analog, BBD delay samples input signal and we must adhere to Sampling Theorem • Appropriate anti-aliasing and anti-imaging filters required at input and output, respectively. • Signal-to-noise ratio (SNR) is typically poor. • To ameliorate this, BBD is preceded by compressor and succeeded by expander (compander). • Very smart but not so intuitive design!

BBD Flanger

http://ant-s4.unibw-hamburg.de/dafx/paper-archive/2005/P_155.pdf

Välimäki, Parker, Esqueda & Alary 15.2.2019

Flanging . Invented by Les Paul (1915-2009) in 1945, but the name came from John Lennon in 1966 (http://en.wikipedia.org/wiki/Flanging) . Original analog method for flanging . Copy the same sound on two open-reel tapes . Play the 2 tapes on 2 synchronized tape machines . Touch the flange of one tape reel to slow it down . Get a nice “wooshing” phase-cancellation effect

Välimäki, Parker, Esqueda & Alary 15.2.2019

5 Flanging: Analog-Era Realization

http://www.audiotechnology.com/tape-flanging-in-the-new-world/

Välimäki, Parker, Esqueda & Alary 15.2.2019

Flanging Sounds Familiar

. Many everyday cases . C. Huygens (1693): the sound of a fountain has a pitch when it reflects from a staircase . Moving and hissing sound source (or listener moving) . Jet airplane flying over a city . Direct sound and its echo . Time-varying delay

Välimäki, Parker, Esqueda & Alary 15.2.2019

6 Digital Flanger – Naive Version

. A copy of the signal is fed through a variable digital delay line and added to the original . Produces a time-varying comb filter . Magnitude response contains many uniformly spaced, moving notches

Välimäki, Parker, Esqueda & Alary 15.2.2019

Digital Flanger – Naive Version with LFO

. Delay-line length is modulated with a Low Frequency Oscillator (LFO) . Slow modulation frequency, approx. 0.1 Hz – 10 Hz

Pink noise Pink noise

E-Guitar E-Guitar

Drums Drums E-gtr examples by Timo Hiekkanen and Tuukka Lyly, TKK, 2007

Välimäki, Parker, Esqueda & Alary 15.2.2019

7 Digital Flanger – Thru Zero

. Problem with naive implementation . Dry and delayed signal never coincide exactly . Solution: Add a static delay l to the ‘undelayed’ path, which is about half of the max value of m

Välimäki, Parker, Esqueda & Alary 15.2.2019

Interpolated Variable Delay Line

. In flanging, the delay-line length must vary smoothly to avoid discontinuities and clicks . Otherwise “zipper noise” is produced . A fractional delay is needed . Usually an FIR interpolation filter

x(n) z-1 z-1 z-1

h(0) h(1) h(2) ... h(N)

y(n)

Välimäki, Parker, Esqueda & Alary 15.2.2019

8 Delay Line with Linear Interpolation

• For digital audio effects, linear interpolation may be sufficiently good – The two-tap FIR is a mild lowpass filter, which varies with d x(n) z  M z 1

1  d d

y(n)  xˆ(n  M  d )

Välimäki, Parker, Esqueda & Alary 15.2.2019

Flanging – The Movie

Flanging - No Interpolation Flanging - Linear Interpolation (notches move in steps) (smooth sliding with filtering)

Välimäki, Parker, Esqueda & Alary 15.2.2019

9 Phasing . Allpass-filtered signal is added to the original signal . Originally an analog electronic version of flanging (“a poor man’s flanger”) . Usually a series of allpass filters . Each allpass filter is of low order, e.g., first or second order . Phase shift of each allpass is modulated with LFO

Välimäki, Parker, Esqueda & Alary 15.2.2019

Phasing - Notches . Notches are generated at frequencies where the phase response of filter chain is multiple of –π (or –180˚) . For example, four 2nd-order allpass filters in cascade → 4 notches . Change of coefficients moves the notches

Phase response of the allpass chain Magnitude response of the overall system

Frequency Frequency

Välimäki, Parker, Esqueda & Alary 15.2.2019

10 The Allpass Filter RevisitedFilters

• The z-domain transfer function of a digital allpass filter is given by

• Parameter a1 determines break frequency. • The phase response of a single allpass and several cascaded units is then: One allpass filter Several allpass filters

Välimäki, Parker, Esqueda & Alary 15.2.2019

Phaser with 10 Allpass Filters

• Topology similar to that of MXR Phase 100 Pedal (5 notches, i.e. 10 filters). • DC Blocker at the input. • Naive approach; parameter a1 fully modulated. Not so useful!

a1 = 0.1 a1 = 0.9

10 10

5 5 0 0 -5 -5 -10 -10 -15 -15 -20

Magnitude (dB) Magnitude -20 Magnitude (dB) Magnitude -25 -25 -30 -30 -35 -35 -40 100 1k 10k 20k -40 Frequency (Hz) 100 1k 10k 20k Frequency (Hz)

Välimäki, Parker, Esqueda & Alary 15.2.2019

11 Phaser Measurements

• Time-varying behavior of pedal can be measured using an allpass chirp train • Modulation patterns and notch locations can be extracted from measurements • Case Study: Fame Sweet Tone Phaser (MXR 100 clone)

R. Kiiski, F. Esqueda and V. Välimäki, “Time-variant gray-box modeling of a phaser pedal”, DAFx-16.

Välimäki, Parker, Esqueda & Alary 15.2.2019

Gray-Box Phaser Model

• Same topology as previous example. • Measurement-based, modulation of a1 restricted to frequencies of interest. • Measurement also exhibited LFO waveform.

a1 = –0.8 a1 = –0.4 10 10 5 5 0 0 -5 -5 -10 -10 -15 -15 -20

Magnitude (dB) -20 Magnitude (dB) Magnitude -25 -25 -30 -30 -35 -35 Original -40 100 1k 10k 20k -40 Frequency (Hz) 100 1k 10k 20k Frequency (Hz) Sound examples by Ricardo Falcón and Aleksi Myöhänen, ASP 2017

Välimäki, Parker, Esqueda & Alary 15.2.2019

12 Gray-Box Phaser Model

R. Kiiski, F. Esqueda and V. Välimäki, “Time-variant gray-box modeling of a phaser pedal”, DAFx-16.

Välimäki, Parker, Esqueda & Alary 15.2.2019

Flanging vs. Phasing

. Flanging . Phasing . Variable time-delay . Variable phase shift . Short delay (< 10ms) . Very short delay . Hundreds of notches . Few notches (1-10) . Notches harmonically related . Notches not harmonic . Number of notches is time- . Notches can be individually varying modulated . Number of notches is fixed

Välimäki, Parker, Esqueda & Alary 15.2.2019

13 Flanging or Phasing?

Flanger Phaser Phaser

Flanger Phaser Flanger

Välimäki, Parker, Esqueda & Alary 15.2.2019

DEMO: Beat-Aligning Looper Jon & Petteri

14 Chorus . The goal: make one source sound like many sources . Useful for vocals and electrical instrument sounds . Very similar structure to flanger and echo effects



Välimäki, Parker, Esqueda & Alary 15.2.2019

Famous Chorus Examples

Chorus Unit: Boss Chorus CE-2

Chorus Unit: EHX Small Clone

Välimäki, Parker, Esqueda & Alary 15.2.2019

15 Chorus Implementations (1) . One choice: multiple feedforward paths with modulated delay-lines (Orfanidis, 1996) . Modulation waveforms may be sinewaves or lowpass-filtered noise (“random walk” signal)

Välimäki, Parker, Esqueda & Alary 15.2.2019

Chorus Implementations (2) . “Industry standard” (Dattorro, 1997)

Välimäki, Parker, Esqueda & Alary 15.2.2019

16 Chorus Implementations (3) . The “industry standard” structure is cheap to implement . Use one for each stereo channel, or more . Generalized allpass-comb filter . Becomes an allpass filter, when delays and coefficients are equal . Negative feedback is used for flattening the spectrum (“white chorus”) . For clean effect, allpass fractional delay filter must be used for the variable delay, not linear interpolation

Välimäki, Parker, Esqueda & Alary 15.2.2019

Chorus Implementations (4) . The industry standard structure can produce many effects . Vibrato: blend = 0, feedback gain = 0, large modulation depth . Flanger: small delay (< 10 ms) . Doubling (double tracking) when blend = feedforward gain, feedback gain = 0, large delay (> 10 ms), with random modulation . Echo: feedback or feedforward gain is zero; a lowpass filter is inserted in the non-zero path; delay is large (> 50 ms) . Stereo effects: modulating sine waves out of phase or in quadrature for the 2 channels

Crazy cartoon- Great reference for this: like effects! Udo Zölzer’s DAFX Book.

Välimäki, Parker, Esqueda & Alary 15.2.2019

17 Chorus vs. Flanging . Flanging . Chorus . Small time delays (<10ms) . Larger time delays (>5ms) . Signals not separable by ear . Separate signals perceived (integration time of ear ≈ 2ms) . Min. delay approx 5 ms . Min. delay = 1 or 0 sample(s) . Notches usually . Deep notches wanted for undesirable strong effect

Välimäki, Parker, Esqueda & Alary 15.2.2019

Pitch Shifting

• Multiply all the frequencies with a constant value • Can be done with resampling (reading a buffer faster), but it changes the playback speed as well –1.5 x

18 Time Stretching using Phase Vocoder

• Use the short-time Fourier transform • Reconstruct the signal with a larger hop size • Makes a good pitch shifter – Pitch up and time stretch to preserve time • What happens to the phase?

https://cycling74.com/tutorials/the-phase-vocoder-%E2%80%93-part-i

Time Stretching

• Stretching the duration of a signal while retaining the pitch

19 Sound microscope

Damskägg, E.-P.; Välimäki, V. Audio Time Stretching Using Fuzzy Classification of Spectral Bins. Appl. Sci.2017, 7, 1293.

• Decomposes sound into three classes: 1. Tonal

2. Transients

3. Noise

Sound Microscope

• Tonal layer: – Must restore phase continuity

20 Sound Microscope

• Transient layer: – Must avoid repeating or smearing

Sound Microscope

• Noise layer: – Must avoid windowing artefacts

21 Dynamic range processing

Dynamics Processing . Compressor reduces the dynamic range of an audio signal . Pop music, radio and TV broadcasting, live PA systems . Special case: Limiter, which saturates at a certain max. level . Expander increases the dynamic range . For example to reduce background noise in silent passages . Special case: Gate, which mutes the signal below a threshold y(dB) y(dB) Compressor Expander Limiter y0 y0

Gate

x0 x(dB) x0 x(dB)

Välimäki, Parker, Esqueda & Alary 15.2.2019

22 Time-Domain View of Compression . When signal level gets high, the gain is reduced automatically

. 2 parameters: attack time TA and release time TR

Välimäki, Parker, Esqueda & Alary 15.2.2019

Digital Feed-forward Compressor . Signal level detector . Temporal envelope: average power over a short time interval . Gain must not be changed instantaneously (aliasing can occur) . Gain computer . Gain G is adjusted based on signal level (power)

Level detector

Välimäki, Parker, Esqueda & Alary 15.2.2019

23 Level Detector . Full-wave rectification (abs) and temporal averaging . For example, a leaky integrator

y(n) = (1 – a1) |x(n)| + a1 y(n –1)

where a1 = 1 – ε (such as a1 = 0.99)

Välimäki, Parker, Esqueda & Alary 15.2.2019

Compression of Musical Signals . After compressing the signal, the overall gain is increased by applying make-up gain . Many potential uses . Maximizing loudness, like in the recent “loudness war” . Controlling transients/emphasizing decay

Välimäki, Parker, Esqueda & Alary 15.2.2019

24 Multiband Compression • Split audio signal into sub-bands for improved performance. • Avoid “pumping” effect.

comp

Demo by Tae Ho Kim and Elias Raninen, ASP-2016

Välimäki, Parker, Esqueda & Alary 15.2.2019

Gating and Limiting of Musical Signals . Limiting is an extreme version of a compressor . Signal value is not allowed to exceed a certain level . Mainly used to maximize loudness . Gating mutes audio when below a certain threshold amplitude . Used to e.g. remove background noise between notes . Can also be used creatively (e.g. gate chord sound based on amplitude of high-hats)

Välimäki, Parker, Esqueda & Alary 15.2.2019

25 Gating and Limiting Examples

. The outcome can sound very loud but lacking natural dynamics . Original music file . Gated (< -20 dB) . Hard limited (> -10 dB) . Gated (< -40 dB) & limited (> -20dB)

Audio examples by Mika Luukkanen, TKK, 2004

Välimäki, Parker, Esqueda & Alary 15.2.2019

Audio Antiquing

• Render a new recording to sound aged  For example, imitate the lo-fi sound of LP, gramophone, or phonograph recordings • Simulate degradations with signal processing techniques (González, thesis 2007; Välimäki et al., JAES 2008)  Local degradations: clicks and thumps (low-frequency pulses)  Global degradations: hiss, wow, distortion, limited dynamic range, frequency band limitations, resonances

26 Audio Antiquing Example

1. CD (original) 2. Phonograph cylinder (new – best quality) 3. Phonograph cylinder (worn)

Phonograph and Gramophone Simulation

• Insert local and global disturbances  Local disturbances: clicks, thumps, tracking errors (editing)  Global (all samples affected): background noise, pitch variation, distortion, coloration (e.g. bandlimiting, acoustic horn modeling), dynamic range limitation

27 Other Effects (not exhaustive)

. Vocoder . Technique from speech coding misused for effect. . Uses large banks of bandpass filters to analyse the spectral envelope of a sound and apply it to another sound . Sounds like robot voice . Frequency shifting (do not confuse with pitch-shifting) . Shifts all frequencies by a fixed amount additively, hence ruining harmonic relationships . Can also be used to produce very rich chorus sounds . Autotune (1998-) . Highly popular pitch corrector with quantization (“Cher”) . Uses LPC and interpolation, or spectral techniques

Välimäki, Parker, Esqueda & Alary 15.2.2019

Other Effects (not exhaustive)

. Wah-wah . Center frequency of a resonator is modulated with envelope follower, control pedal, or LFO . Filtering (telephone sound, resonances etc.) . Enhancer . Harmonic distortion of only high frequencies to increase brightness . Distortion and tube-amp modeling effects . Reverberations effects . Spatial audio effects . Stereo expansion, 3-D sound etc. . More?

Välimäki, Parker, Esqueda & Alary 15.2.2019

28 Resources

Välimäki, Parker, Esqueda & Alary 15.2.2019

Literature – Effects

. E.-P.Damskägg and V. Välimäki, ”Audio time stretching using fuzzy classification of spectral bins,” Applied Sciences, 2017, 7, 1293. . J. Dattorro, “Effect design—part 2: Delay line modulation and chorus,” J. Audio Eng. Soc., vol. 45, no. 10, pp. 764–788, Oct. 1997. Available online at: http://www.stanford.edu/~dattorro/research.html . R. Dobson, A Dictionary of Electronic & Computer Music Technology.Oxford University Press, 1992. . W. M. Hartmann, “Flanging and phasing,” J. Audio Eng. Soc., vol. 26, no. 6, pp. 439–443, June 1978. . S. J. Orfanidis, Introduction to Signal Processing. Prentice-Hall, 1996. Section 8.2 (“Digital Audio Effects”), pp. 355-388. . G. D. White, The Audio Dictionary (2nd ed.). University of Washington Press, 1991. . D. Giannoulis, M. Massberg and J. D. Reiss, ”Digital dynamic range compressor design – A tutorial and analysis”, J. Audio Eng. Soc., vol. 60, no. 6, June 2012, pp. 399–408

Välimäki, Parker, Esqueda & Alary 15.2.2019

29 Literature – Effects (2)

• R. Kiiski, F. Esqueda & V. Välimäki, “Time-variant gray-box modeling of a phaser pedal,” in Proc. 19th Int. Conf. Digital Audio Effects (DAFx-16), pp. 31–38, Brno, Czech Republic, Sept. 2016. • V. Välimäki, S. González, J. Parviainen, and O. Kimmelma, ”Digital audio antiquing – Signal processing methods for imitating the sound quality of historical recordings,” Journal of the Audio Engineering Society, vol. 56, no. 3, pp. 115–139, March 2008.

Välimäki, Parker, Esqueda & Alary 15.2.2019