Systems Level Liveness with Extempore Andrew Sorensen Henry Gardner Research School of Computer Science, CECS, Research School of Computer Science, CECS, The Australian National University The Australian National University Canberra, ACT 0200, Australia Canberra, ACT 0200, Australia [email protected] [email protected] Abstract real time: with physical systems such as loud speakers Live programs can be modified while the program is and digital musical instruments, with other Live Coders, executing in order to provide a more reactive experi- with conventional musicians and with streaming media. ence for the programmer. In demanding applications, Systems support for Live Coding has achieved a high such programs traditionally utilise pre-defined function level of sophistication in order to meet the computa- calls to compiled libraries. We present a system that tional demands of professional-level audio production enables demanding live programs to be built where the and equipment. The concerns of these systems are at supporting stack of libraries is, itself, live. In such sit- the vanguard of live programming. uations, the top level code might be thought of as a This paper describes an extension of Live Coding into simple \live environment" that can be created live and the realms of live systems programming. This enhance- that encapsulates code that has \bubbled-up" from the ment has been motivated by several desires: to improve supporting libraries. Our system enables this bubbling the performance and flexibility of Live Coding languages, up to be achieved in an ad-hoc way and with minimal to extend their capability to encompass supporting li- performance penalty. The deep, systems-level liveness braries and the operating system itself, and thence to that it exhibits is described and compared with other explore their real-world applications { both in artistic approaches to Live Coding and liveness generally. Live Coding and in the wider world of live programming The work described here has its origins in artistic Live systems. Coding of computer music and multimedia. As well Starting from the Impromptu system described by [30], as this application area we will also discuss its wider the present work has resulted in a new programming uses including the development of interactive multimedia language, a new support compiler and substantial library installations and the harnessing of scientific simulation. development { in total encompassing some 180,000 lines of source code. Its challenge has been to bring some CCS Concepts Applied computing ! Sound and of the run-time tooling, small compilation units and music computing; Software and its engineering late binding popular in dynamic languages, into a new ! General programming languages; Frameworks; system-level direct-to-native language and run-time en- Human-centered computing ! Interaction paradigms; vironment suitable for Live Coding. The new system, Keywords Live Coding, Live Programming, Liveness, called Extempore, has been running in production since Computer Music 2014 and has been used extensively for artistic perfor- mance, for research, and for commercial development. ACM Reference format: This is the first paper that describes the design and Andrew Sorensen and Henry Gardner. 2017. Systems Level architecture of Extempore and that justifies its claim to Liveness with Extempore. In Proceedings of Onward! 2017, Vancouver, BC, Canada, October 22-27, 16 pages. having systems level liveness. DOI: 10.1145/nnnnnnn.nnnnnnn After a review of Live Coding in the next section, we then describe the design and architecture of Extempore 1 Introduction in Section 3. We particularly discuss the new XTLang language which, together with Scheme, provides Extem- Over almost two decades, an international community of pore with its bilingual programming interface. In Section artists and computer scientists has developed an artistic 4, we then describe three examples of the application of performance practice known as \Live Coding" where Extempore: computer music, scientific simulation and computer programs are written and modified in front interactive multi-media. Each of these examples might of a live audience to generate, manipulate and respond be thought of as being characterized by an approach to music and visual media in real time. \Live Coders" where a \live language" has been \developed live" on write and modify code that interfaces with the world in top of a systems substrate with minimal penalty to either Onward! 2017, Vancouver, BC, Canada 2017. 978-x-xxxx-xxxx-x/YY/MM. $15.00 DOI: 10.1145/nnnnnnn.nnnnnnn Onward! 2017, October 22-27, Vancouver, BC, Canada Andrew Sorensen and Henry Gardner performance or low level flexibility. We discuss this sys- A similar situation exists in scientific computing, where tems level liveness in Sections 5 and 6 and we conclude higher level languages such as R, Python and Mathemat- with Section 7. ica are coupled to high-performance numerical frame- works written in C/C++/Fortran. The half-stack design provides scientists with a live \interactive programming experience", with the heavy lifting managed by coupled numerical libraries written in a lower level systems lan- 2 Live Coding and the Systems Stack guage. While an argument can be advanced that such a 2.1 Half-Stack Environments half-stack design represents a reasonable separation of A standard approach to developing musical Live Coding concerns, real-world problems have a tendency to bub- systems has been to take a managed higher-level pro- ble up through the software stack. When analyzing a gramming language, such as Ruby(Sonic Pi[2]), corpus of some 3.5 million lines of R code, Morandat et JavaScript(Gibber[25]), Scheme(Impromptu[29], al. [20] noted that the average run-time spent in calls Fluxus[10]), Clojure(Overtone[2]), Lua(LuaAV[37]), and to foreign functions was only 22%. This is despite the to couple that managed language to a native audiovisual expectation that R programs (and R programmers!) are framework (typically implemented in C/C++). Such expected to push heavy computational work down to Live Coding environments could be called \half-stack" the C FFI. Mordandat et al. showed that this is perhaps because their ability to \live modify" the behavior of the not happening in practice. With R's run-time perfor- underlying audiovisual framework is generally restricted mance trailing C by a factor of 500, the performance for reasons of performance, accessibility, or both. By implications of the 68% of the execution time spent in way of example, it is generally not possible to write non- the R layer is significant. Our purpose here is not to trivial real-time Digital Signal Processing (DSP) code in denigrate R (indeed we are fans of R and the interactive these higher-level languages. This is due to the lack of programming experience that it provides), but rather to sufficiently low-level access to the CPU instruction set highlight that programmers are still attempting to do required to write real-time DSP code as well as to perfor- significant computational work above the C abstraction mance limitations on these languages (even if low-level layer despite R's known performance limitations. In access were possible). practice, it seems that such programming problems are Despite these limitations, the half-stack approach has bubbling up through the abstraction stack. also been the dominant approach to building live pro- Ideally we would like the live programmer's program- gramming environments outside of Live Coding (outside ming environment to provide the performance and ac- of the audiovisual domain). Computer game engines cessibility necessary to solve any domain problems that (Unity3D, Unreal Engine, CryEngine, OGRE, Torque, might reasonably arise. Half-stack live programming etc.) commonly use managed languages (particularly environments, such as R, often fall short of this ideal JavaScript and C#) coupled to low-level, real-time game because they either (a) do not support the necessary engines (in C/C++), to provide a degree of \live in- low-level primitives required (such as explicit Advanced teractive scripting". The \scripting engine" supports Vector Extensions) or (b) do not provide the necessary programmer productivity while the underlying game en- run-time performance required (such as that needed for gine maintains highly specialized, and highly optimized, heavy numerical computation). run-time performance. Games programmers (the end- For systems-level programming, what constitutes \suf- users of game engines) are able to proceduralize many ficient performance", or “sufficient access", can be collo- aspects of their game at the scripting layer. However, quially benchmarked by the level of access and perfor- whenever high numerical performance, real-time I/O, or mance provided by the C language { a language com- hardware interfacing are required, games programmers monly described, in antipathy as often as reverence, as a are expected to drop down to the underlying C/C++ \portable assembly language". Ultimately, full-stack live substrate. Significantly, dropping down the game engine programming attempts to provide on-the-fly interactive abstraction tree from scripting language to C++ also real-time programming down to the machine (accessibil- means sacrificing the \live" interactive programming ity) with optimal hardware utilization (performance) as experience provided by the half-stack environment. To benchmarked against C. be explicit, there are limitations in the scripting layer that limit the domain of problems to which the scripting 2.2 The \Full-Stack" Experience layer can provide a reasonable solution. Beyond this The general ideal of \Full-Stack" live programming can \half-stack" the programmer is forced to change their be traced back at least as far as Smalltalk. Alan Kay operating environment, and in so doing, lose the ability envisaged a future where computer users would be com- to live program the system. puter programmers - computer programmers empowered Systems Level Liveness with Extempore Onward! 2017, October 22-27, Vancouver, BC, Canada from the elimination of conventional loading and system generation cycles"[15, p.22].