A Web-Based Framework for Distributed Music System Research and Creation Benjamin Matuszewski To cite this version: Benjamin Matuszewski. A Web-Based Framework for Distributed Music System Research and Cre- ation. AES - Journal of the Audio Engineering Society Audio-Accoustics-Application, Audio Engi- neering Society Inc, 2020. hal-03033143 HAL Id: hal-03033143 https://hal.archives-ouvertes.fr/hal-03033143 Submitted on 1 Dec 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. PAPERS A Web-Based Framework for Distributed Music System Research and Creation Benjamin Matuszewski, ([email protected]) CICM/musidance EA1572, Universite´ Paris 8 STMS Ircam-CNRS-Sorbonne Universite´ Paris, France This paper presents soundworks, a framework dedicated to prototyping and developing distributed multimedia applications using Web technologies. Since its first release in 2015, the framework has been used in numerous artistic and research projects such as concerts, installations, workshops, teaching or experimental setups. We first present how this diversity of contexts and objectives permitted to identify a set of patterns able to support recurring needs of expert users in exploratory tasks. We then detail new developments that have been achieved to provide better support to these patterns. More particularly, we describe the novel distributed state management system dedicated at simplifying the implementation of remote control and monitoring interfaces and, the plug-in system implemented to improve the extensibility of the framework and foster composition of dedicated functionalities. We believe that these new developments can provide a solid ground to further research and artistic practices in the area of distributed music systems. The soundworks framework is open-source and released under BSD-3-Clause license. 0 INTRODUCTION sciences and humanities—and thus gather a great diversity of goals, skills and methodologies (e.g. experimental stu- The specification and development of the WebAudio dies, practices-based research). It appears that a common API [1, 2]—alongside Application Programming Inter- ground for the support of this diversity can be found in faces (API) such as WebSockets [3] or WebGL [4] and the concept of experimental systems—as systems compo- the possibilities offered by a full-featured scripting lan- sed of epistemic things and technical objects in constant guage such as JavaScript [5]—has permitted to envision evolution and reconfiguration—developed by Rheinberger the Web platform [6] as a viable technical platform for ar- [16, 17] and pursued by Schwab in the context of artis- tistic creation and more precisely for computer music prac- tic research [18]. We postulate that such epistemological tices [7]. Furthermore, the recent developments of ubiqui- ground can lead to the implementation of particular pat- tous and pervasive computing [8], with the democratiza- terns [19] in order to support this diversity of research tion of smartphones and large spread of nanocomputers, practices effectively. led to consider Web technologies as a possible solution for recurring integration and interoperability issues [9]. These soundworks [20]—initiated by S. Robaszkiewicz and two complementary aspects therefore authorize to consi- N. Schnell [21] in 2015—is a framework dedicated to der the Web as an interesting environment in the deve- the development of distributed multimedia applications lopment of Networked Music Systems [10, ?, 11]. Moreo- on the web. It has known two major revisions (in 2016 ver, this novel approach could unfold novel possibilities in and 2017) and has been used in numerous artistic and related areas such as multi-source electro-acoustic music research projects (e.g. concerts, installations, workshops, [12, 13] or interfaces for musical expression [14, 15]. In pedagogical or experimental setups) [22, 23, 24]. While this context, the development of a dedicated framework, these achievements tended to validate the efficacy of the designed to support both the specificities of the web plat- framework considered as an experimental platform, they form and of computer music research and practices seems also permitted to highlight some inherent and recurring essential. difficulties. The third version of soundworks—initiated Indeed, computer music is a field that spans across mul- in 2019 [25]—presented in this paper aims to address some tiple disciplines—from scientific to artistic through social of these difficulties, as well as to provide solid foundations J. Audio Eng. Sco., Vol. 68, No. 10, author’s prepint, 2020 October 1 Matuszewski PAPERS upon which environments facilitating the inclusion and 2.1 Concerts and Performances agency of non-expert developers users can be built. Playground is an application that allows a composer / performer to remotely distribute and control audio mate- After a short review of the related works (cf. Section 1), rials rendered on the smartphones of the audience. we describe in Section 2 different contexts in which our framework has been successfully used in last two years, and which informs us about recurring and important needs our framework must support. In Section 3, we present an overview of the framework architecture, philosophy and basic functionalities. Finally, in Sections 4 and 5, we de- tail new features dedicated at supporting patterns—namely remote monitoring and control, composability and extensi- bility—that we consider of primary importance to provide an effective experimental platform supporting distributed music systems research and creation. FIGURE 1. On the left, Garth Paine performing Future Perfect. On the right, screenshots of two of the four control interfaces of 1 RELATED WORKS the Playground application. Max/MSP or Pure Data [26, 27] are well established The application expose several dynamic control inter- environments used since many years by artists and resear- faces, optimized for touch interfaces such as tablets (see chers in a wide range of contexts. The success of these Figure 1), that can be jointly used : visual programming environments lies in part in their suc- — The first one allows for triggering sound files on a cessful implementation of certain patterns that permitted given smartphone represented on the screen as a co- users to create and compose their own application in a lored square. very interactive fashion [24]. However, the environments — The second one allows for controlling granular syn- also come with their drawbacks in our context. First, there thesis among subsets of the audience’s smartphones. are not primarily oriented toward distributed applications — The third one is dedicated at controlling the spa- and are difficult to operate in large and dynamic networks tial rendering of audio files synchronized among all of computers. Second, they necessitate the installation of smartphones. a software, making applications difficult to distribute and — Finally, the fourth one is dedicated at managing all thus to deploy in large collective settings, precluding new presets and configuration variables as well as at as- forms of public and collective participation. signing particular sound banks to the other control interfaces. On the Web platform, attempts have been made to im- plement equivalent environments [28]. However, while in- In this application, a number of strategies are implemen- teresting, these tools are far for from being as mature as ted to provide the composer and performer a dynamic envi- the original ones. Also, they tends to neglect one of the ronment in which they can test sonic material and configure most interesting aspects of using the Web platform, na- many aspect of the synthesis (e.g. dynamic update of sound mely the network. Finally, some frameworks dedicated to files, creation of presets) in the studio, but also have use- network music systems, such as Rhizome [29] or Nexus, ful feedback on the state of audience’s smartphones (e.g. [30] have been proposed. While similar to soundworks loading states, position in concert hall) during the perfor- in their scope, these tools do not seem to be maintained or mance. in active development. Playground has been designed together with the compo- ser Garth Paine and implemented for the creation of Fu- 1 2 CONTEXTS ture Perfect, an immersive 3D audio visual performance . Since then, the application has been used for the crea- In this section we review different contexts in which tion of several pieces—by the composer himself or other we deployed web-based distributed systems implemented composers—, as well as in workshops and pedagogical si- using our framework. Each of these contexts will be illus- tuations. trated with a particular project that have been developed in the last years. Note that while these 3 applications have 2.2 Installations been designed and developed with the previous version of The context of an installation comes with different soundworks, two of them (i.e. Playground and CoMo) constraints and requirements than the ones of perfor- have already been ported to the novel version. These rewri- tings permitted to simplify the code base, and moreover, enabled new artistic and research possibilities, assessing 1. Future Perfect has been composed and realized during a re- thus the concepts and design decisions presented in this search / creation residency that took place in 2018 between Ircam paper. and ZKM. 2 J. Audio Eng. Sco., Vol. 68, No. 10, author’s prepint, 2020 October PAPERS Soundworks mances. In such contexts, the usage of so-called nano- CoMo has also been used in different settings such as computers is interesting for several reasons [24], the most music, design and dance researches, artworks [36] and important one being the simplicity they offer in term of workshops.