A graphical user interface for live music coding Trabajo de Fin de Grado en Desarrollo de Videojuegos Curso 2020–2021 Autores Mario Tabasco Vargas Gonzalo Cidoncha Pérez Director Jaime Sánchez Hernández Facultad de Informática Universidad Complutense de Madrid Una interfaz gráfica para programación de música en vivo A graphical user interface for live music coding Trabajo de Fin de Grado en Desarrollo de Videojuegos Departamento de Sistemas Informáticos y Computación Autores Mario Tabasco Vargas Gonzalo Cidoncha Pérez Director Jaime Sánchez Hernández Convocatoria: Junio 2021 Facultad de Informática Universidad Complutense de Madrid 15 de junio de 2021 Abstract A graphical user interface for live music coding In recent years, live music coding environments have attracted many programmers with musical roots and interests. However, current live music coding environments require a certain level of technical knowledge that might result inaccessible for amateurs that lack a solid programming background. The objective of this project is to develop an application that will allow the user to generate improvised music, but through a graphical interface that abstracts all the programming. This front-end desktop application, developed in the Unity engine, will provide a user interface greatly inspired by the visual programming language Scratch, which will allow the user to configure sequences of actions by fitting together blocks that resemble puzzle pieces. The sounds will be generated by the Sonic Pi live coding environment, as it runs a program that translates command messages coming from the front-end application into actual commands that can be performed by Sonic Pi. Keywords desktop application, graphical user interface, live music coding, Open Sound Control, visual programming, Scratch, Sonic Pi, Unity. v Resumen Una interfaz gráfica para programación de música en vivo Los entornos de programación de música en vivo (live music coding) han despertado mucho interés en los últimos años entre los programadores con inquietudes musicales. Sin embargo, los entornos actuales requieren habilidades no triviales de programación y pueden resultar poco accesibles para los aficionados sin esa formación. El objetivo de este trabajo será desarrollar una aplicación que permita al usuario generar música improvisada, pero a través de una interfaz gráfica que abstraiga la programación como tal. Esta aplicación de escritorio front end, desarrollada en el motor Unity, ofrecerá al usuario una interfaz visual inspirada en gran medida en el lenguaje de programación visual Scratch, y permitirá al usuario configurar secuencias de acciones encajando bloques similares a piezas de puzle. Los sonidos se generarán en el entorno de live coding Sonic Pi, donde se ejecutará un programa que traduce los mensajes con comandos provenientes de la aplicación front end en comandos que Sonic Pi pueda llevar a cabo. Palabras clave aplicación de escritorio, interfaz gráfica de usuario, live music coding, Open Sound Control, programación visual, Scratch, Sonic Pi, Unity. vii Contents 1. Introduction 1 1.1. Background . .1 1.1.1. Live coding and music . .1 1.1.2. Visual programming languages . .2 1.2. Motivation and objectives . .3 1.3. Work structure . .4 2. Application User Manual 5 2.1. Installation . .5 2.2. Understanding the interface . .6 2.3. Creation of a basic loop . .8 2.4. Blocks . .9 2.4.1. Sleep Block . .9 2.4.2. Synth Block . .9 2.4.3. Sample Block . 10 2.5. Recording a session . 11 3. Technologies 13 3.1. Open Sound Control protocol . 13 3.2. The Unity engine . 14 ix 3.2.1. Unity User Interface framework . 15 3.2.2. UnityOSC: A package for handling OSC messages in Unity . 16 3.2.3. Working with JSON files in Unity . 17 3.3. Sonic Pi . 18 3.3.1. Deterministic execution in Sonic Pi . 19 3.3.2. Time handling in Sonic Pi . 20 3.3.3. Live loops . 21 3.3.4. OSC messages in Sonic Pi . 24 4. Implementation 25 4.1. Architecture of the program in Sonic Pi . 25 4.1.1. Command and Attributes structures . 25 4.1.2. Functioning of the listening thread . 27 4.1.3. Functioning of the processing thread . 28 4.1.4. Initialization of the Sonic Pi program . 29 4.2. Development of the main application . 30 4.2.1. Architecture and main components . 30 4.2.2. Communication with Sonic Pi . 32 4.2.3. Addition, modification and removal of blocks . 33 4.2.4. Addition, modification and removal of loops . 35 4.2.5. Design and implementation of the graphical user interface . 36 4.3. Application deployment . 39 5. Contributions 41 5.1. Mario Tabasco Vargas . 41 5.2. Gonzalo Cidoncha Pérez . 43 6. Conclusions and future work 45 6.1. Conclusiones . 45 6.2. Future work . 46 7. Conclusiones y trabajo futuro 49 7.1. Conclusiones . 49 7.2. Trabajo futuro . 50 Bibliography 53 List of figures 1.1. An example of code using the Scratch web application. .2 2.1. A screenshot from Sonic Pi Controller’s installer. .5 2.2. A screenshot showing how to load the Launcher file inside Sonic Pi. .6 2.3. A screenshot of the Launcher file loaded in the Sonic Pi editor. .6 2.4. Screenshot of the first elements the user can see in the application. .7 2.5. Screenshot of the application with the element menu displayed. .7 2.6. An example of a basic loop. .8 2.7. Basic Synth Block in a loop. .9 2.8. Note Selection panel with the default octave and the twelve notes on an octave........................................ 10 3.1. Screenshot showing the Unity Editor and some of its sub-windows, taken from the official Unity documentation [1]. 15 3.2. Screenshot of the Unity Editor showing the Rect Transform component of an interface object, as it is displayed in the inspector window, taken from the official Unity documentation [2]. 16 3.3. A screenshot showing Sonic Pi’s text editor interface. 18 4.1. An outline of the structure of the OSC messages that can be handled by the program. 27 4.2. An screenshot of the main application where a Synth Block is being added to a loop next to a Sleep Block. 34 xiii 4.3. An screenshot that shows the effects of the block addition taking place in figure 4.2. 34 4.4. A screenshot of the Unity inspector window showing the configuration of the Canvas Scaler for the application. 37 4.5. A visual representation of the parts that make up a block in the application. 37 4.6. A screenshot of the Unity hierarchy window showing the Sleep Block prefab. 38 4.7. A screenshot of the Unity inspector window with the files used in the appli- cation attached to the Sonic Pi Manager component. 39 4.8. A screenshot of the building settings used for the application deployment in Unity. ....................................... 39 Chapter 1 Introduction 1.1. Background 1.1.1. Live coding and music Live coding is the concept of programming on-the-fly, improvising the code that is being written and making changes in real time. It is usually associated with the control of light systems or the creation of graphics and music. The beauty of live coding lies in the potential to create improvised artworks through code, typically in front of an audience. Improvised musical performances are one of the most known uses for live coding. The relationship between live coding and music comes in a natural way, as music has been related to improvisation and spontaneity all throughout history (in [3], several examples of improvisation in music are mentioned, including jazz solos, improvisation in Baroque music and Native American shamanic chants). At the time this paper is being written, a lot of live coding environments focused in the creation of musical pieces and performances (or used for these purposes) exist. Some of the most well-known are: Super Collider, a "real time audio synthesis" environment [4]. Super Collider was not initially conceived as a live coding environment, but it provides a platform for audio synthesis and algorithmic composition that can be used by performers or as a basis for other live coding languages. TidalCycles, an open source tool that allows to generate musical patterns with code [5]. It is one of the most common languages used at algoraves (algorithm and rave), events where people can dance to live coding performances and music generated from algorithms in general. FoxDot, a Python-based language and editor for live music coding that communi- cates with the SuperCollider synthesis engine to make music [6]. 1 2 Chapter 1. Introduction Sonic Pi, a live coding environment developed by Samuel Aaron in collaboration with the Raspberry Pi Foundation with educational purposes [7]. Pure Data, an open source visual programming language for multimedia works [8]. It is commonly used as a sound synthesis and sequencing tool for creating electronic music. 1.1.2. Visual programming languages In recent years, a trend has appeared that consists on the development of programming languages that can be used to code by interacting with different elements in an editor’s user interface instead of writing actual code. This visual programming languages are often used for tasks such as the implementation of graphic shaders in applications with real- time graphics or scripting specific behaviours. The use of a visual environment for these tasks allows less technical professionals, like designers and artists, to implement specific components of an application without the intervention of a programmer. The two main approaches that are being used today are: node based systems (e.g., Unreal Blueprints for visual scripting [9], Blender nodes for creating materials [10], etc.) and puzzle-like applications (e.g., Scratch [11], App Inventor [12], etc.) Figure 1.1: An example of code using the Scratch web application.