Implementing a Polyphonic MIDI Software Synthesizer using Coroutines, Realtime Garbage Collection, Closures, Auto-Allocated Variables, Dynamic Scoping, and Continuation Passing Style Programming Kjetil Matheussen Norwegian Center for Technology in Music and the Arts. (NOTAM) [email protected] Abstract however the programming only concerns han- This paper demonstrates a few programming tech- dling blocks of samples where we only control a niques for low-latency sample-by-sample audio pro- signal graph, there are several high level alter- gramming. Some of them may not have been used natives which makes it relatively easy to do a for this purpose before. The demonstrated tech- straightforward implementation of a MIDI soft niques are: Realtime memory allocation, realtime synth. Examples of such high level music pro- garbage collector, storing instrument data implicitly gramming systems are SuperCollider [McCart- in closures, auto-allocated variables, handling signal ney, 2002], Pd [Puckette, 2002], CSound4 and buses using dynamic scoping, and continuation pass- many others. ing style programming. But this paper does not describe use of block processing. In this paper, all samples are in- Keywords dividually calculated. The paper also explores Audio programming, realtime garbage collection, possible advantages of doing everything, alloca- coroutines, dynamic scoping, Continuation Passing tion, initialization, scheduling, etc., from inside Style. the realtime audio thread. At least it looks like everything is performed 1 Introduction inside the realtime audio thread. The under- lying implementation is free to reorganize the This paper demonstrates how to implement a code any way it wants, although such reorga- MIDI software synthesizer (MIDI soft synth) nizing is not performed in Snd-RT yet. using some unusual audio programming tech- Future work is making code using these tech- niques. The examples are written for Snd-RT niques perform equally, or perhaps even better, [Matheussen, 2008], an experimental audio pro- than code where allocation and initialization of gramming system supporting these techniques. data is explicitly written not to run in the real- The techniques firstly emphasize convenience time audio thread. (i.e. few lines of code, and easy to read and modify), and not performance. Snd-RT1 runs on top of Snd2 which again runs on top of the 2 MIDI software synthesizer Scheme interpreter Guile.3 Guile helps gluing The reason for demonstrating a MIDI soft synth all parts together. instead of other types of music programs such It is common in music programming only to as a granular synthesis generator or a reverb, is compute the sounds themselves in a realtime that the behavior of a MIDI soft synth is well priority thread. Scheduling new notes, alloca- known, plus that a MIDI soft synth contains tion of data, data initialization, etc. are usually many common challenges in audio and music performed in a thread which has a lower prior- programming: ity than the audio thread. Doing it this way 1. Generating samples. To hear sound, we helps to ensure constant and predictable CPU need to generate samples at the Audio usage for the audio thread. But writing code Rate. that way is also more complicated. At least, when all samples are calculated one by one. If 2. Handling Events. MIDI data are read at a rate lower than the audio rate. This rate is 1 http://archive.notam02.no/arkiv/doc/snd-rt/ commonly called the Control Rate. 2http://ccrma.stanford.edu/software/snd/ 3http://www.gnu.org/software/guile/guile.html 4http://www.csound.com 3. Variable polyphony. Sometimes no notes coroutine to make it run. The spawned are playing, sometimes maybe 30 notes are coroutine will run automatically as soon6 playing. as the current coroutine yields (by calling yield or wait), or the current coroutine 4. Data allocation. Each playing note re- ends. quires some data to keep track of frequency, phase, envelope position, volume, etc. The Coroutines are convenient in music pro- challenges are; How do we allocate memory gramming since it often turns out practi- for the data? When do we allocate memory cal to let one dedicated coroutine handle for the data? How do we store the memory only one voice, instead of mixing the voices holding the data? When do we initialize manually. Furthermore, arbitrarily placed the data? pauses and breaks are relatively easy to im- plement when using coroutines, and there- 5. Bus routing. The sound coming from the fore, supporting dynamic control rate simi- tone generators is commonly routed both lar to ChucK [Wang and Cook, 2003] comes through an envelope and a reverb. In ad- for free. dition, the tones may be autopanned, i.e. panned differently between two loudspeak- (wait n) waits n number of frames before con- ers depending on the note height (similar tinuing the execution of the current corou- to the direction of the sound coming from tine. a piano or a pipe organ). (sound ...) spawns a special kind of coroutine where the code inside sound is called one time per sample. (sound coroutines are 3 Common Syntax for the Examples stored in a tree and not in a priority queue The examples are written for a variant of since the order of execution for sound the programming language Scheme [Steele and coroutines depends on the bus system and Sussman, 1978]. Scheme is a functional lan- not when they are scheduled to wake up.) guage with imperative operators and static A simple version of the sound macro, scoping. called my-sound, can be implemented like A number of additional macros and special this: operators have been added to the language, and some of them are documented here because of (define-stalin-macro (my-sound . body) the examples later in the paper. ‘(spawn (while #t ,@body (<rt-stalin>...) is a macro which first trans- (wait 1)))) forms the code inside the block into clean R4RS code [Clinger and Rees, 1991] un- However, my-sound is inefficient compared 5 derstood by the Stalin Scheme compiler. to sound since my-sound is likely to do (Stalin Scheme is an R4RS compiler). Af- a coroutine context switch at every call ter Stalin is finished compiling the code, the to wait.7 sound doesn’t suffer from this produced object file is dynamically linked problem since it is run in a special mode. into Snd-RT and scheduled to immediately This mode makes it possible to run tight run inside the realtime audio thread. loops which does not cause a context switch (define-stalin signature ...) defines variables until the next scheduled event. and functions which are automatically in- (out <channel> sample) sends out data to serted into the generated Stalin scheme the current bus at the current time. (the code if needed. The syntax is similar to current bus and the current time can be define. thought of as global variables which are im- (spawn ...) spawns a new coroutine [Conway, plicitly read from and written to by many 1963; Dahl and Nygaard, 1966]. Corou- 6Unless other coroutines are placed earlier in the tines are stored in a priority queue and it is queue. not necessary to explicitly call the spawned 7I.e. if two or more my-sound blocks or sound blocks run simultaneously, and at least one of them is a my- 5Stalin - a STAtic Language ImplementatioN, sound block, there will be at least two coroutine context http://cobweb.ecn.purdue.edu/ qobi/software.html. switches at every sound frame. operators in the system)8 By default, the :where is just another way to declare local current bus is connected to the sound card, variables. For example, but this can be overridden by using the (+ 2 b in macro which is explained in more de- :where b 50) tail later. If the channel argument is omitted, the is another way of writing sample is written both to channel 0 and 1. (let ((b 50)) It makes sense only to use out inside a (+ 2 b)) sound block. The following example plays a 400Hz sine sound to the sound card: There are three reason for using :where instead of let. The first reason is that (<rt-stalin> the use of :where requires less parenthe- (let ((phase 0.0)) sis. The second reason is that reading the (sound code sometimes sounds more natural this (out (sin phase)) (inc! phase (hz->radians 400))))) way. (I.e “add 2 and b, where b is 50” in- stead of “let b be 50, add 2 and b”.) The (range varname start end ...) is a simple lo- third reason is that it’s sometimes easier op iterator macro which can be imple- to understand the code if you know what mented like this:9 you want to do with a variable, before it is defined. (define-macro (range varname start end . body) (define loop (gensym)) 4 Basic MIDI Soft Synth ‘(let ,loop ((,varname ,start)) (cond ((<,var ,end) We start by showing what is probably the sim- ,@body plest way to implement a MIDI soft synth: (,loop (+ ,varname 1)))))) (range note-num 0 128 (wait-midi :options ...) waits until MIDI data (<rt-stalin> is received, either from an external inter- (define phase 0.0) face, or from another program. (define volume 0.0) (sound wait-midi has a few options to specify the (out (* volume (sin phase)))) kind of MIDI data it is waiting for. In the (inc! phase (midi->radians note-num))) examples in this paper, the following op- (while #t tions for wait-midi are used: (wait-midi :command note-on :note note-num (set! volume (midi-vol))) :command note-on (wait-midi :command note-off :note note-num Only wait for a note on MIDI message. (set! volume 0.0)))) :command note-off This program runs 128 instruments simul- Only wait for a note off MIDI mes- taneously.