Proceedings of the 17th Linux Audio Conference (LAC-19), CCRMA, Stanford University, USA, March 23–26, 2019 SC-HACKS: A LIVE CODING FRAMEWORK FOR GESTURAL PERFORMANCE AND ELECTRONIC MUSIC Iannis Zannos Department of Audiovisual Arts Ionian University, Corfu, Greece [email protected] ABSTRACT demanding task which is subject of research in various gestural in- terface applications. The work related in the present paper repre- This paper presents a library for SuperCollider that enables live cod- sents an indirect and bottom-up approach to the topic, based on DIY ing adapted to two domains of performance: telematic dance with and open source components and emphasizing transparency and self- wireless sensors and electroacoustic music performance. The library sufficiency at each step. It does not address the task of gesture recog- solves some fundamental issues of usability in SuperCollider which nition, but rather it aims at supporting live coding in conjunction with have been also addressed by the established live-coding framework dancers and instrumental performers. Ongoing experiments together JITLib, such as modifying synth and pattern processes while they with such performers, are helping to identify low-level tasks and fea- are working, linking control and audio i/o between synths, and gen- tures which are essential for practical work. This type of work is eration of GUIs. It offers new implementations, which are more purely empirical, and tries to identify useability criteria purely from compact and easy to use while emphasizing transparency and scal- practice, rather than to develop features that are inferred from known ability of code. It introduces binary operators which when coupled interaction paradigms in other related domains. At this stage of the to polymorphism facilitate live coding. Several foundation classes project it is still too early to formulate conclusions from these ex- are introduced whose purpose is to support programming patterns or periments. Instead, this paper concentrates on the fundamentals of commonly used practices such as the observer pattern, function call- the implemenation framework on which this work is based. These backs and system-wide object messaging between language, server are readily identifiable and their potential impact on further develop- processes and GUI. ment work as well as experiments are visible. This paper therefore The use of the library is demonstrated in two contexts: a telem- describes the basic principles and design strategy of the sc-hacks li- atic dance project with custom low-cost movement sensors, and dig- brary, and discusses its perceived impact on performances. Finally, ital implementations of early electroacoustic music scores by J. Har- it outlines some future perspectives for work involving data analysis vey and K. Stockhausen. The latter involves coding of a complex and machine learning. score and generation of a GUI representation with time tracking and live control. 1.2. Live Coding Frameworks in SuperCollider 1.2.1. Types of Live Coding Frameworks 1. BACKGROUND Live Coding libraries can be divided into two main categories de- 1.1. Bridging Live Coding and Gestural Interaction pending on the level of generality of their implementation and their application scope. First, there are libraries which extend Super- The performance practice known as live coding emerged from the Collider usage in order to simplify the coding of very behaviors ability of software to modify state and behavior through the inter- or features which are very common in performance, but are other- active evaluation of code fragments and to synthesize audio at run- wise inconvenient to code in SuperCollider. To this category be- time. As a result, several programming environments and technolo- longs the JITLib framework. JITLib (Just-In-Time programming Li- gies supporting live coding have been developed in the past 20 years, brary) has been around since at least August 2006, with an early such as SuperCollider[1], Impromptu[2], ChucK[3] , Extempore[4], version since ca 2000 1 and is very widely used in the commu- Gibber[5], and others. It has been noted, however, that such envi- nity, being the de-facto go-to tool for live coding in SuperCollider. ronments and practices suffer from a lack of immediacy and those The second category consists of libraries that concentrate on spe- visible gestural elements that are traditionally associated with live cialized usage scenarios and attempt to create domain-specific mini- performance [6]. Recent research projects attempt to re-introduce languages for those scenarios on top of SuperCollider. Such are: gestural aspects or to otherwise support social and interactive ele- IXI-Lang (a sequencer / sample playing mini-language by Thor Mag- ments in musical performance using technologies associated with nusson [12]), SuperSampler (a polyphonic concatenative sampler live coding ([7], [8], [9], [10]). Amongst various types of gestural with automatic arrangement of sounds on a 2-dimensional plane, interaction, dance is arguably the one least related to textual coding. by Shu-Cheng Allen Wu [13]), and Colliding (An "environment for Few recent studies exist which prepare the field for bridging dance synthesis-oientd live coding", simplifying the coding of Unit Gen- with coding ([11]). The challenges in this domain can be summa- erator graphs, by Gerard Roma [14]). Finally, TidalCycles by Alex rized as the problem of bridging the symbolic domains of dance and McLean [15] should be mentioned, which develops its own live cod- music notation and the subsymbolic numerical domain of control ing language based on Haskell and focussing on the coding of com- data streams input from sensors. This also implies translating be- tween continuous streams of data and individual timed events, pos- 1See https://swiki.hfbk-hamburg.de/ sibly tagged with symbolic values. This is a technologically higly MusicTechnology/566 (accessed 20-December-2018) Proceedings of the 17th Linux Audio Conference (LAC-19), CCRMA, Stanford University, USA, March 23–26, 2019 plex layers of synchronized beat cycles with sample playback and a Class which holds a Dictionary and ensures that a predefined cus- synthesis, and uses the SuperCollider synthesis server as audio en- tom function is executed each time that a value is stored in one of gine. the keys of the Dictionary. Sc-hacks defines a subclass of Environ- mentRedirect similar to ProxySpace, but defines a custom function 1.2.2. sc-hacks Objectives and Approach that provides extra flexibility in setting values which is useful during performance in accessing control parameters. This enables keeping sc-hacks belongs to the first category of frameworks, and its initial track of which parameter refers to which process, storing parameter motivation was partly to implement some of the solutions of JITLib values between subsequent starts of a process belonging to a player, in more robust, simple, and general ways. In parallel, inspiration and updating GUI elements to display values as these change. Ad- from ChucK’s => operator led to the development of a minimal ex- ditionally, sc-hacks makes the environment of the player current af- tension of the language based on 4 binary operators (+>, <+, *> ter certain operations, in order to make the current context the one *<), which, coupled with polymorphism, permit simplified and com- normally expected by the performer. This however is not always a pact coding of several common sound-structure coding patterns. Fur- secure solution. For this reason, the target environment can be pro- thermore, the implementation of some basic programming patterns 2 vided as adjective argument in binary operators involving players, opened new possibilities for the creation of GUI elements which up- which ensures that code will work as expected even when changing date their state. This led to a proliferation of GUI building and man- the order of execution of code in irregular manner. agement facilities and resulted in several interfaces for live coding tasks, such as a code browser based on the concept of code snippets, 2.1.3. Modifying event generating processes on the fly a browser for editing and controlling the behavior of named players holding synth or pattern items, and shortcuts for building GUI wid- Event generating algorihm processes are implemented in SuperCol- gets displaying values of parameters controlled by OSC, MIDI or al- lider through class Pbind. Pbind takes an array of keys and associ- gorithmic processes. Finally, ongoing experiments with dancers and ated streams as argument and creates a Routine that calculates pa- instrumentalists are giving rise to new interface and notation ideas. rameters and event types for each set of keys and values obtained The current focus is on building tools for recording, visualising and from their associated streams, and schedules them according to the playback of data received from wireless sensors via OSC, in order duration obtained from the stream stored under the key dur. The im- to experiment with the data in performance, and to apply machine- plementation of Pbind allows no access to the values of each event, learning algorithms on them. i.e. it is not possible to read or to modify the value of a key at any moment. Furthermore, it is not possible to modify the structure of the dictionary of keys and streams while its event-generating pro- 2. APPROACH cess is playing. This means that Pbind processes cannot be modified 2.1. Players and Player Environments interactively while they are playing. In order to circumvent this lim- itation, a number of techniques have been devised which require to JITLib addresses four fundamental problems in coding for concur- add code for any key that one wishes to read or to modify. JITLib rent sound processes: (a) Use of named placeholders for sound gen- uses such techniques and also provides a way to substitute a Pbind erating processes, (b) managing the control parameters of processes process while it is running with a new one, thereby indirectly al- in separate namespaces, (c) modifying event-generating algorithmic lowing modification of that process.