Grafting Synthesis Patches Onto Live Musical Instruments

GRAFTING SYNTHESIS PATCHES ONTO LIVE MUSICAL INSTRUMENTS Miller Puckette University of California, San Diego frequency IN ABSTRACT phasor Hilbert cos sin A new class of techniques is explored for controlling MU- SIC N or Max/MSP/pd instruments directly from the sound Arctan Mag- of a monophonic instrument (or separately acquired inputs wavetable nitude from a polyphonic instrument). The instantaneous phase [n] m[n] and magnitude of the input signal are used in place of pha- envelope sors and envelope generators. Several such instruments are generator described; this technique is used in SMECK, the author’s open-source guitar processing patch. OUT 1. INTRODUCTION OUT (a) (b) One of several possible approaches to real-time expressive Figure 1. A MUSIC N instrument: (a) classically imple- control of computer music algorithms is to use the acousti- mented; (b) adapted to use an incoming sound’s phase and cal sound of a live instrument directly as a source of control. amplitude This has the advantage of tapping into the skill of a trained musician (rather than requiring the player to learn entirely new skills). Further, using the sound of the instrument is ness and more expressivity (because they can avoid the la- sometimes preferable to using physical captors; the captor tency of analysis). These have used the incoming audio sig- approach is more expensive, harder to migrate from one in- nal simultaneously as a processing input and to compute the strument to another, and downright unavailable for some in- parameters of the audio effect such as a delay line [2], or an struments such as the human voice. Also, arguably at least, all-pass filter or frequency shifter [3]. In these approaches, the sound of the instrument itself is what the musician is fo- audio signal processing (at a low latency) is parametrized cussed on producing, so it is in a sense more direct to use by (higher-latency) analysis results: for instance, filtering or the instrumental sound than data taken from captors. modulating instrumental sound depending on its measured We will be concerned with the direct manipulation, at pitch [5]. The approach adopted here [7] is similar to, but the sample level, of almost-periodic signals. These may generalizes, the frequency shifting approach. come from a monophonic pitched instrument or from a polyphonic string instrument with a separated pickup. Rather 3. GENERAL TECHNIQUE than analyzing the signal to derive pitch and amplitude envelopes for controlling a synthesizer, we will work directly Our strategy is to extract time-dependent phase and am- from the signal itself, extracting its instantaneous phase and plitude signals from a quasi-periodic waveform, which can magnitude, for direct use in a signal processing chain. then be plugged into a wide variety of classical or novel MUSIC-style instruments. For example, the simple instru- 2. PRIOR WORK ment shown in Figure 1 (part a), an amplitude-controlled wavetable oscillator, is transformed into the instrument of Much instrument-derived synthesis takes an analysis/synthesis part (b). Whereas the instrument of part (a) has frequency, approach, in which time-windowed segments of incoming start time, and duration as parameters, that of part (b) has an audio are analyzed for their pitch (such as in guitar syn- audio signal as input coming from an external instrumen- thesizers) and/or spectrum (as in Charles Dodge’s Speech tal source. The phase and amplitude are computed from the Songs). incoming audio signal. Algorithms that can act directly on the signals without To extract the phase and magnitude, the input signal is the intermediary of an analysis step can attain higher robust- first converted into a pair of signals in phase quadrature us- m[n] [n] s k OUT Figure 3. Phase modulation. The parameter k is a fixed Figure 2. Extracted “magnitude” and “phase” of a sum of modulation index, and s is the sensitivity of modulation in- two sinusoids. dex to input amplitude. ing an appropriately designed pair of all-pass filters, labeled the output follows the amplitude of the input, since the table “Hilbert” in the figure. Denoting these by x[n] and y[n], the lookup portion depends only on phase and the amplitude of magnitude and phase are computed as the input is used linearly to control the amplitude of the output. Second, scaling the input by a constant factor does not q m[n] = x2[n] + y2[n] affect the waveform but only scales the output correspond- ingly. Both of these properties are sometimes desirable but other behaviors are possible and often desirable. f[n] = arctan(y[n]=x[n]) As a special case, if the wavetable in Figure 1(b) is cho- If the incoming signal is a sinusoid of constant magnitude sen to be a single cosine cycle, the instrument recovers an m and frequency f , then the calculation of m[n] retrieves the all-pass-filtered version of the incoming sound: value m and f is a phase signal (a sawtooth) of frequency f . u[n] = m(n)cos(f[n]) = x[n]: If the input signal contains more than one sinusoidal component, the result is more complicated, but if one sinusoidal It is thus possible to move continuously and coherently be- component is dominant then the effect of the others is simul- tween the nearly unaltered sound of the original instrumen- taneously to modulate the signals m[n] and f[n]; i.e., they tal signal and highly modified ones. are heard as amplitude and phase modulation [7]. This is illustrated in Figure 2, which shows the phase 4. PHASE MODULATION and magnitude measured from a synthetic input consisting of first and ninth partials with amplitudes in the ratio four to As a first example of a useful type of audio modification, the one. At top the two phase-quadrature outputs of the “Hilbert” instrument of Figure 3 adds phase modulation to the sound filters are graphed on the horizontal and vertical axes. As of the incoming instrument. There are two parameters, a shown underneath, the measured magnitude and phase are, fixed index k, and a sensitivity s to allow the index to vary to first order, those of the fundamental, modulated sinu- with amplitude. To analyze this we first put s = 0 so that the soidally at eight times the fundamental. (Although the phase index is fixed at k. If the incoming sound is a sinusoid, then signal appears to have jumps, it is continuous provided phases the amplitude signal is constant and the phase signal is a reg- zero and 2p are taken as equivalent). ular sawtooth, so the result is classic 2-operator phase mod- The particular way a patch responds to changes in am- ulation [1] with index k. If we now allow s to be nonzero, plitude in the input signal can be specified at will. For ex- the modulation index becomes amplitude-dependent. ample, the patch of Figure 1(b) respects the amplitude of If the input is a near but not exact sinusoid, deviations the incoming signal in two ways. First, the amplitude of from sinusoidality will appear as phase modulation on both tiple of the frequency of the incoming signal. The wave packet formulation of formant synthesis also a allows for the use of more complex waveforms in place of the sinusoids, permitting us to generate periodic sounds with arbitrary, time-varying spectral envelopes [6]. To do this, we replace the wraparound tables in the block diagram of b Figure 6 with a two-dimensional table, whose rows each act as waveforms to packetize. Collectively, the rows describe a time-varying waveform, whose variation is controlled by a separate control parameter (shown as an envelope generator c in the figure). An interesting application is computer-determined scat singing, in which instrumental notes are shaped into scat syllables, chosen to reflect the timing and melodic contour of the instrumental melody. Figure 4. Formant generation: (a) a format as an overlapped sum of wave packets; (b, c) decomposition of the sum into realizable parts. 6. RESULTS The testing ground for these techniques has been a guitar the carrier and modulation oscillators, and also as amplitude with a separated pickup (the Roland GK-3). Most of the modulation on the output and index of modulation, and, if available Roland gear only provides MIDI or synthesized s 6= 0, on the index of modulation of the 2-operator system. outputs, but for this work the six raw string signals are needed; In any case, the amplitude of the output still follows that of they may be obtained via the KeithMcmillin Stringport [4]. the input faithfully. The six guitar strings are each treated separately according to the algorithms described here. The system has been used 5. WAVE PACKETS by the author in many performances starting at Off-ICMC 2005. A comprehensive patch with many presets is avail- One powerful approach to synthesizing pitched sounds is able both in Max/MSP through the KeithMcmillin website, formant synthesis as in VOSIM [10], CHANT [9], PAF [8, and in Pd from crca.ucsd.edu/˜msp. Ch. 6], and so on. Our approach here is to use wave packet synthesis (although the PAF also works well in this context). Figure 4 shows a wave packet series with an overlap factor 7. CONCLUSION of two. Each packet is the product of a Hann window with a This paper has described an exceedingly general technique. sinusoid; each sinusoid’s phase is zero at the center of its Almost any classical MUSIC N-style computer music in- packet. The fundamental frequency is controlled by the spac- strument may be adapted so that one or more of its phase ing of the packets, the formant center frequency is the fre- and amplitude controls are replaced by ones obtained from quency of the individual “packetted” sinusoids, and the band- an incoming quasiperiodic instrumental input.

Load more