Teaching Sound Synthesis in C/C++ on the Raspberry Pi

Henrik von Coler and David Runge Audio Communication Group, TU Berlin [email protected] [email protected]

Abstract added, they are not considered for exploring ap- For a sound synthesis programming class in C/C++, plied sound synthesis algorithms in this class. a Raspberry Pi 3 was used as runtime and develop- In a more application-based context, graphi- ment system. The embedded system was equipped cal programming environments like Pure Data with an Arch Linux ARM, a collection of libraries (Pd), MAX MSP or others would be the first for sound processing and interfacing, as well as with choice. They allow rapid progress and interme- basic examples. All material used in and created diate results. However, they are not the best for the class is freely available in public git reposito- platform to enhance the knowledge of sound ries. After a unit of theory on sound synthesis and Linux system basics, students worked on projects synthesis algorithms on a sample-wise level by in groups. This paper is a progress report, point- nature. ing out benefits and drawbacks after the first run of the seminar. The concept delivered a system with C/C++ delivers a reasonable compromise be- acceptable capabilities and latencies at a low price. tween low level access and the comfort of using Usability and robustness, however, need to be im- available libraries for easy interfacing with hard- proved in future attempts. ware. For using C/C++ in the development of sound synthesis software, a software devel- Keywords opment kit (SDK) or application programming Jack, Raspberry Pi, C/C++ Programming, Educa- interface (API) is needed in order to offer access tion, Sound Synthesis, MIDI, OSC, Arch Linux to the audio hardware. Digital audio workstations (DAW) use plug- 1 Introduction ins programmed with SDKs and APIs like Stein- The goal of the seminar outlined in this pa- berg’s VST, Apple’s Audio Units, Digidesign’s per was to enable students with different back- RTAS, the Linux Audio Developer’s Simple grounds and programming skills the develop- Plugin API (LADSPA) and its successor LV2, ment of standalone real-time sound synthesis which offer quick access to developing audio projects. Such a class on the programming synthesis and processing units. A plugin-host of sound synthesis algorithms is, among other is necessary to run the resulting programs and things, defined by the desired level of depth in the structure is predefined by the chosen plat- signal processing. It may convey an application- form. The JUCE framework [34] offers the oriented overview or a closer look at algorithms possibility of developing cross platform applica- on a sample-wise signal processing level, as in tions with many builtin features. Build targets this case. Based on this decision, the choice can be audio-plugins for different systems and of tools, respectively the programming environ- standalone applications, including Jack clients, ment should be made. which would present an alternative to the cho- sen approach. Script languages like Matlab or Python are Another possibility is the programming of widely used among students and offer a comfort- Pd externals in the C programming language able environment, especially for students with- with the advantage of quick merging of the self- out a background in computer science. They programmed components with existing Pd in- are well suited for teaching fundamentals and ternals. FAUST [6] also provides means for cre- theoretical aspects of signal processing due to ating various types of audio plugins and stan- advanced possibilities of debugging and visual- dalone applications. Due to a lack in functional isation. Although real-time capabilities can be programming background it was not chosen. For various reasons we settled for the Jack Unfortunately the on-board audio interface API [18] to develop command line programs on of the Raspberry Pi could not be configured a Linux system. for real-time audio applications. It does not feature an input and the Jack server could • Jack clients are used in the research on bin- only be started with high latencies. After try- aural synthesis and sound field synthesis, ing several interfaces, a Renkforce USB-Audio- for example by the WONDER interface [15] Adapter was chosen, since it delivered an ac- or the SoundScape Renderer [14]. Results ceptable performance at a price of 10 e. The of the projects might thus be potentially Jack server did perform better with other inter- integrated into existing contexts. faces, yet at a higher price and with a larger • Jack clients offer quick connection to other housing. clients, making them as modular as audio- Students were equipped with MIDI interfaces plugins in a DAW. from the stock of the research group and private • The Jack API is easy to use, even for be- devices. The complete cost for one system, in- ginners. Once the main processing func- cluding the Raspberry Pi 3 with housing, SD tion is understood, students can immedi- card, power adapter and the audio interface, ately start inserting their own code. were thus kept at about 70 e(vs. the integrated low-latency platform bela[5], that still ranks at • The omission of a graphical user interface around 120 eper unit). for the application leaves more space for focusing on the audio-related problems. 2.2 Operating System • The proposed environment is independent of proprietary components. First tests on the embedded system involved Raspbian [12], as it is highly integrated and has a graphical environment preinstalled, that A main objective of the seminar was to equip eases the use for beginners. Integration with the students with completely identical systems the libraries used for the course proved to be for development. This avoids troubles in han- more complicated however, as they needed to dling different operating systems and hardware be added to the software repository for the ex- configurations. Since no suitable computer pool amples. was available, we aimed at providing a set of machines as cheap as possible for developing, A more holistic approach, integrating the op- compiling and running the applications. The erating system, was aimed at, in order to leave students were thus provided with a Raspberry as few dependencies on the students’ side as pos- Pi 3 in groups of two. Besides being one of the sible and not having to deal with the hosting of cheapest development systems, it offers the ad- a custom repository of packages for Raspbian. vantage of resulting in a highly portable, quasi Due to previous experience with low latency se- embedded synthesizer for the actual use in live tups using Arch Linux [7], Arch Linux ARM applications. [2] was chosen. With its package manager pac- man [9] and the Arch Build System [1] an easy The remainder of this paper is organized as system-wide integration of all used libraries and follows: Section 2 introduces the used hard- and software was achieved by providing them as pre- software, as well as the infrastructure. Section 3 installed packages (with them either being avail- presents the concept of the seminar. Section 4 able in the standard repositories, or the Arch briefly summarizes the experiences and evalu- User Repository[3]). Using Arch Linux ARM, ates the concept. it was also possible to guarantee a systemd [13] 2 Technical Outline based startup of the needed components, which is further described in Section 2.5. At the time 2.1 Hardware of preparation for the course, the 64bit vari- A Raspberry Pi 3 was used as development ant (AArch64) - using the mainline kernel - was and runtime system. The most recent version not available yet. At the time of writing it is at that time was equipped with 1.2GHz 64-bit still considered experimental, as some vendor quad-core ARMv8 CPU, 802.11n Wireless LAN, libraries are not yet available for it. Instead the a Bluetooth adapter, 1GB RAM, 4 USB ports ARMv7 variant of the installation image was 40 GPIO pins and various other features [11]. used, which is the default for Raspberry Pi 2. 2.3 Libraries As these settings were not realizable with the All libraries installed on the system are listed internal audio card, the snd-bcm2835 module - in Tab. 1. For communicating with the au- the driver in use for it - was blacklisted using dio hardware, the Jack API was installed. The /etc/modprobe.d/*, to not use the sound device jackcpp framework adds a C++ interface to at all. Jack. Additional libraries allowed the handling For automatic start of the low-latency au- of audio file formats, ALSA-MIDI interfacing, dio server and its clients, systemd user ser- Open Sound Control, configuration files and vices were introduced, that follow a user session Fast Fourier Transforms. based setup. The session of the main system user is started automatically as per systemd’s [email protected]. This is achieved by enabling the Table 1: Libraries installed on the development linger status of said user with the help of loginctl system [8], which activates its session during boot and Library Ref. Purpose starts its enabled services. jack2 [18] Jack audio API A specialized systemd user service [30] allows jackcpp [26] C++ wrapper for jack2 for Jack’s startup with an elevated CPU sched- uler without the use of dbus [21]. The stu- sndfile [19] Read and write audio files dents’ projects could be automatically started rtmidi [32] Connect to MIDI devices as user services, that rely on the audio server liblo [23] OSC support being started by enabling services such as this example: yaml [20] Configuration files [ Unit ] fftw3 [22] Fourier transform Description=Example project boost [4] Various applications After=jack@rpi-usb-44100. service [ Service ] ExecStart=/path/to/executable \ parameter1 \ 2.4 Image paramter2 The image for the system is available for down- Restart=on-failure 1 [ Install ] load and can be asked for by mailing to the WantedBy=default.target authors in case of future unavailability. Installa- tion of the image follows the standard procedure 2.6 Infrastructure of an Arch Linux ARM installation for Rasp- berry Pi 3, using the blockwise copy tool dd, For allowing all students in the class the access which is documented in the course’s git reposi- via SSH, a WIFI was set up, providing fixed tory [31]. IP addresses for all Raspberry Pis. Network performance showed to be insufficient to handle 2.5 System Settings all participants simultaneously, though. Thus, The most important goal was to achieve a additional parallel networks were installed. round-trip latency below 10 ms. This would not For home use, students were instructed to be sufficient for real-time audio applications in provide a local network with their laptops, or general, but for teaching purposes. With the using their home network over WiFi or cable. hardware described in Section 2.1, stable Jack Depending on their operating system, this was server command line options were evaluated, more or less complicated. leading to a round-trip latency of 2.9 ms: 3 The Seminar /usr/bin/jackd -R \K The seminar was addressed to graduate stu- -p 512 \ dents with different backgrounds, such as com- -d alsa \ puter science, electrical engineering, acoustics -d hw:Device \ -n 2 \ and others. One teacher and one teaching assis- -p 64 \ tant were involved in the planning, preparation -r 44100 and execution of the classes. The course was divided into a theoretical and 1 https://www2.ak.tu-berlin.de/~drunge/ a practical part. In the beginning, theory and klangsynthese basics where taught in mixed sessions. The fi- nal third of the semester was dedicated to su- at the architecture and programming of the pervised project work. Yamaha DX7 synthesizer. 3.1 System Introduction 3.3 Projects Students were introduced to the system by giv- Out of the 35 students who appeared to the first ing an overview over the tools needed and pre- meetings, 18 worked on projects throughout the senting the libraries with their interfaces. Users whole semester. It should be noted that (with of Linux, Mac and Windows operating systems one exception) only Linux and MAC users con- took part in the class. Since many students tinued. lacked basic Linux skills, first steps included us- No restrictions were made regarding the ing Secure Shell (SSH) to access the devices, choice of the topic, exept that it should result which proved to be hard for attendees without in an executable program on the Raspberry PI. any knowledge on the use of the command-line The student projects included: interface. Windows users were aided to install and use PuTTY [10] for the purpose of connect- • A vector synthesis engine, allowing the ing, as there is no native SSH client. After two mixture of different waveforms with a suc- sessions, each group was able to reach the Rasp- ceeding filter section berry Pi in class, as well as at home. • A subtractive modeling synth with modu- 3.2 Sound Synthesis Theory lar capabilities Sound synthesis approaches were introduced • A physical string model, based on the from an algorithmic point of view, showing ex- Karplus-Strong Extended with dispersion amples of commercially available implementa- filter tions, also regarding their impact on music pro- • A sine-wave Speech Synthesis [28] effect, duction and popular culture. Students were which includes a real-time FFT provided with ready-to-run examples from the course repository for some synthesis approaches, • A guitar-controlled subtractive synthesizer, as well as with tasks for extending these. using zero-crossing rate for pitch detection Important fundamental literature was pro- • A wave-digital-filter implementation with vided by Z¨olzer[35], Pirkle [27] and Roads [29]. sensor input from the GPIOs The taxonomy of synthesis algorithms proposed by Smith [33] was used to structure the outline In order to provide a more suitable platform of the class, as follows. for running embedded audio applications, one A section on Processed Recording dealt with group used buildroot2 to create a custom oper- sampling and sample-based approaches, like ating system. wave-table synthesis, granular synthesis, vector synthesis and concatenative synthesis. 4 Conclusions Subtractive synthesis and analog modeling The use of the Raspberry Pi 3 for the program- were treated as the combination of the basic ming of Jack audio applications showed to be a units oscillators, filters and envelopes. Fil- promising approach. A system with acceptable ters were studied more closely, considering IIR capabilities and latencies could be provided at and FIR filters and design methods like bilin- a low price. Accessibility and stability, how- ear transform. A ready to run biquad example ever, need to be improved in future versions: A was included and a first order low-pass was pro- considerable amount of time was spent work- grammed in class. ing on these issues in class and the progress in Additive Synthesis and Spectral Modeling the projects was therefore delayed considerably. were introduced by an analysis-resynthesis pro- The overhead in handling Linux showed to be a cess of a violin tone in the SMS model [25], con- major problem for some students and probably sidering only the harmonic part. caused some people to drop out. A possible step Physical Modeling was treated for plucked would be to provide a set with monitor, key- strings, starting with the Karplus-Strong algo- board and mouse, as this would increase acces- rithm [17], advancing to bidirectional [24]. sibility. The stability of the Jack server needs to FM Synthesis [16] was treated as a repre- be worked on, as sometimes the hardware would sentative of abstract algorithms in the class. The concept was mainly taught by a closer look 2https://buildroot.org not be started properly, leading to crashing Jack [18] Paul Davies. JACK API. http://www. clients. jackaudio.org/. In future seminars, which are likely to be con- [19] Erik de Castro Lopo. Libsndfile. http://www. ducted after these principally positive experi- mega-nerd.com/libsndfile/. ences, most of the issues should be worked out [20] Clark C. Evans. YAML: YAML Ain’t Markup and the image, as well as the repository will Language. http://yaml.org/. have been improved. [21] Free Desktop Foundation. dbus home- page. https://wiki.freedesktop.org/www/ References Software/dbus/. [1] Arch Build System - ArchWiki. https: [22] Matteo Frigo and Steven G. Johnson. FFTW //wiki.archlinux.org/index.php/Arch_ Fastest Fourier Transform in the West. http: Build_System. //www.fftw.org/. [2] Arch Linux ARM homepage. https://www. [23] Steve Harris and Stephen Sinclair. liblo Home- archlinuxarm.org/. page: Lightweight OSC implementation. http: [3] Arch User Repository. https://aur. //liblo.sourceforge.net/. . archlinux.org/ [24] Matti Karjalainen, Vesa V¨alim¨aki, and Tero [4] Boost C++ Libraries - Homepage. http:// Tolonen. Plucked-string models: From the www.boost.org. Karplus-Strong algorithm to digital wave- [5] buildroot homepage. https://bela.io. guides and beyond. Computer Music Journal, 22(3):17–32, 1998. [6] FAUST - Homepage. http://faust.grame. fr/. [25] Scott N. Levine and Julius O. Smith. A Sines+Transients+Noise Audio Representation [7] Linux Audio Conference 2015 - Workshop: for Data Compression and Time/Pitch Scale Arch Linux as a lightweight audio platform Modi cations. Proceedings of the 105th Audio - Slides. http://lac.linuxaudio.org/2015/ Engineering Society Convention, 1998. download/lac2015_arch_slides.pdf. [26] Alex Norman. JackCpp. http://www.x37v. [8] loginctl man page. https://www. info/projects/jackcpp/. freedesktop.org/software/systemd/man/ loginctl. [27] Will Pirkle. Designing Software Synthesizer Plug-Ins in C++. Focal Press, 2014. [9] Pacman homepage. https://www.archlinux. org/pacman/. [28] Robert E Remez, Philip E Rubin, David B Pisoni, Thomas D Carrell, et al. Speech percep- [10] PuTTY Homepage. http://www.putty.org/. tion without traditional speech cues. Science, [11] Raspberry PI Homepage. https: 212(4497):947–949, 1981. //www.raspberrypi.org/products/ [29] Curtis Roads. The computer music tutorial. raspberry-pi-3-model-b/. MIT press, 1996. [12] Raspbian homepage. https://www.raspbian. [30] David Runge. uenv homepage. https://git. org/. sleepmap.de/software/uenv.git/about/. [13] systemd homepage. https://www. [31] David Runge and Henrik von Coler. AK- freedesktop.org/wiki/Software/systemd/. Klangsynthese Repository. https://gitlab. [14] Jens Ahrens, Matthias Geier, and Sascha Spors. tubit.tu-berlin.de/henrikvoncoler/ The soundscape renderer: A unified spatial au- Klangsynthese_PI. dio reproduction framework for arbitrary ren- [32] Gary P. Scavone. RtMidi. http://www.music. dering methods. In Audio Engineering Soci- mcgill.ca/~gary/rtmidi/. ety Convention 124. Audio Engineering Society, 2008. [33] Julius O. Smith. Viewpoints on the History of Digital Synthesis. In Proceedings of the Inter- [15] Marije AJ Baalman. Updates of the wonder national Computer Music Conference, pages 1– software interface for using wave field synthe- 10, 1991. sis. LAC2005 Proceedings, page 69, 2005. [34] Jules Storer. JUCE. https://www.juce.com/. [16] John M Chowning. The synthesis of complex audio spectra by means of frequency modula- [35] Udo Zoelzer, editor. DAFX: Digital Audio Ef- tion. Journal of the audio engineering society, fects . John Wiley & Sons, Inc., New York, NY, 21(7):526–534, 1973. USA, 2 edition, 2011. [17] Julius O. Smith David A. Jaffe. Extensions of the Karplus-Strong Plucked-String Algorithm. Computer Music Journal, 7(2):56–69, 1983.