COMPOSING WITH SWARM ALGORITHMS – CREATING INTERACTIVE AUDIO-VISUAL PIECES USING FLOCKING BEHAVIOUR Jan C. Schacher Daniel Bisig Martin Neukom Zurich University of the Arts Institute for Computer Music and Sound Technology Baslerstrasse 30 8048 Zürich, Switzerland {jan.schacher, daniel.bisig, martin.neukom}@zhdk.ch ABSTRACT 2. BACKGROUND In this paper, methods and concepts for algorithmic The transposition of an abstract model, such as the musical composition with flocking algorithms are flocking algorithms, into musical forms poses a set of discussed. Taking three pieces from an exhibition as interesting challenges. Any computational model of examples, the strategies for building the specific natural phenomena such as flocking behaviours relationships between each swarm-system and its represents a mathematical abstraction of reality. The musical results are shown. General considerations about result of this abstraction involves several aspects that are the structures of self-organising systems and their relevant for artistic applications of the model. The potential for shaping musical dynamics as well as about properties and behaviours of the natural system have to adaptive and interactive behaviours in an installation- be stripped down to a level of generality, that allows context follow. The central aspects of a low-level their creative reconfiguration into an artificial unified algorithmic thinking in artistic processes are phenomenon. This could for example embody a musical made evident and further scenarios for the organisation idea. Yet, the process of abstraction also strips the of flocking systems and the application to algorithmic natural phenomenon of its naturally perceivable musical composition are developed. Finally, some characteristics. As a consequence, the computational details are given about the technical tools that were used model loses many of the perceivable qualities present in in the implementations. the original system and therefore is less constraining and facilitates the application of a much wider variety of 1. INTRODUCTION musical strategies than the natural phenomenon. The The transformation of algorithms and simulation almost unlimited number of possibilities for model techniques from the fields of theoretical biology and customisation and transformation into music constitutes artificial life into generative mechanisms for musical some of the main challenges of simulation-based creation provides a vast and fruitful terrain for artistic computer music. The two research projects entitled ISO experimentation. [8][9][10] The adaptation of flocking (Interactive Swarm Orchestra) and ISS (Interactive simulations for the creation and control of algorithmic Swarm Space), which have been running for several music constitutes a paradigmatic example of such a years at the Institute for Computer Music and Sound transfer. [2][3][4] Even in their simplest form, dating Technology (ICST) of the Zurich University of the Arts back to the seminal work by Craig Reynolds [12], these provide an ideal context to study these issues. This self-organized multi-agent systems can give rise to a research combines conceptual considerations, range of emergent phenomena, which can be exploited engineering developments and public disseminations in for musical creation. Consequently, flocking simulations artistic venues. As part of the conceptual aspects of this prove to possess a great appeal for musicians and research, we tried to define a set of fundamental numerous examples of swarm-based music have been correspondences between swarms simulations and created. [5][6][7][16][17] In most of these examples, the musical algorithms that can help inform artistic flocking simulation employed is (almost) identical to strategies. As part of our engineering activities, we have Craig Reynolds' original "Boids" system. Since that developed an open-source programming library entitled flocking simulation was not designed with any musical “ISO Flock” that facilitates the creation of highly application in mind, a significant part of the effort in the customized multi-agent simulations. Since then, this creation of the swarm based piece needs to be invested library has progressed into a flexible environment for into establishing a meaningful mapping between the real-time creation, configuration and modification of simulation’s properties and the musical results. We multi-agent simulations that can control a wide range of believe that by shifting the focus of the musical creation audio engines through OSC. [18] The main focus of the away from the mapping issue towards the design of the ISO Flock library lies on the generic definition of agent flocking simulation itself, the mapping challenge is and environment properties and behaviours. Due to this diminished and the diversity of the musical possibilities generality, the library permits not only the is increased. In this publication, we try to elucidate this implementation of ordinary flocking simulations, but claim by describing three implementations of swarm can also be adapted to deal with a much broader range based music pieces. of phenomena, that may for example be used to implement physics simulations or even entirely fictitious and unique correspondence between the properties of systems. The flexibility of the ISO Flock simulation each swarm simulation and its musical manifestation. At library is intended to encourage a creative methodology the core of each piece is a musical concept that guides that situates the customisation of a multi-agent the design of the flocking simulation, the musical simulation directly within the musical composition composition and their mutual relationships. Similarly, process. With this approach, the characteristics of the each visual composition transforms an important aspect multi-agent simulation are as much the reflection of a of the swarm simulation into a key element in the musical idea as would be the configuration of the audio graphic language. engine and the definition of the mapping relationships. 3.1. Impacts 3. THREE SWARM PIECES Starting from a strongly interactive premise, the In the context of an exhibition entitled “Milieux algorithm in this piece explores the possibility of a Sonores” which was curated by Marcus Maeder and hierarchical field of relationships within several flocks, shown in 2009 in the "Walcheturm" exhibition space in representative, in some ways of the interdependence Zürich and in 2010 in the “Gray Area Foundation for the within an ecosystem or food chain. On a second level Arts” in San Francisco, a set of three interactive swarm the perceptually most significant events are extracted based audiovisual compositions were created by the from the continuous flow of data of all the agent's authors. [1] Each of the three compositions is based on a motions and used as the fundamental impulses for the specific swarm simulation and a musical and visual music. (See Figure 1.) composition, which are both controlled by that swarm There are three types of entities present in the "Impacts" simulation. The musical compositions are spatialised via model: the first type of agent is the attractor. Its a twenty speakers three-dimensional Ambisonic behaviour is fully dependant on the visitor’s action since projection array. [13] Interactivity is provided via a it can't move by itself but is displaced by the visitor's touch-sensitive surface, which allows visitors to directly touch. The attractors' positions are constrained to a plane manipulate some of the simulated agents and thereby within the simulation's three-dimensional space in order affect the swarm-based generative mechanisms that to be coherent with the gestural interaction space. The underlie the audio-visual compositions. The visual agents of the secondary swarm react to the attraction compositions are projected onto two of the pentagonal forces of the first swarm and also influence their own surfaces of the structure and fill the visitor's field of kind. They themselves serve as attractors to the agents view. Each composition is structured into several stages within the third swarm, which exhibits the same that are characterized by distinct simulation behaviours behaviours as the secondary agents but on a lower and audio-visual aesthetics. They transition from one hierarchical level. The behaviours of the agents within stage to the next according to a simple state machine the second and third swarms are based on the classic mechanism that takes both the duration of a particular attraction, evasion, alignment paradigm [12], but are state and the current activity level in the visitor’s parameterized in such a manner as to create visually and interaction into account. musically interesting dynamic motion patterns. The main emphasis in the creation and exhibition of the three pieces is placed on the development of a specific Figure 1. Schematics of the swarm music piece “Impacts”. The hierarchy of the swarm simulations is depicted on the left side. The intermediate mapping stage performs the analysis of swarm behavior and generates the control events and data for the musical engine, which is shown on the right. being located in certain sectors of the surround field. The granular shadow notes create a cloud on the trails of the agents after each impact. Since they get repeatedly triggered during the escape starting from an impact until they reach the largest spatial displacement, a noticeable perceptual widening of the pitch-space occurs. Finally the spatial positions of the