About This Issue

A map can be viewed as a projection Feller and Margaret Cahill, respec- George Tzanetakis from CMJ.James of the points in one space onto the tively. Mr. Feller analyzes a disc by Harley has been a major asset to the points of another space that does not computer music composer Matthew Journal for the better part of two necessarily have the same number Burtner, and Louis Bigo reviews a decades, serving at various times as of dimensions. Mapping is a general conference at McGill University on reviews editor, products editor, and concept in mathematics. In computer mathematics and computation in CD or DVD producer. In addition to music, the term is most often used in music. The concluding product an- his excellent editing and production the specific context of digital musical nouncements report on a wide variety work, he wrote many insightful instruments, typically to refer to the of audio equipment and controllers, reviews himself. He also guest-edited correspondence between the parame- as well as a more unusual item: con- a special issue of CMJ in memory ter values of a controller and those of ductive paint with which one can of Iannis Xenakis (which included asoundsynthesizer.Wearepleased draw circuits, repair electronics, and his own article surveying Xenakis’s to present here a special issue on so on. electronic music) and served as a mapping in computer music, guest- The past year has seen some curator for the CMJ CD. He passed edited by two authorities in the field: changes in the Journal’s editorial staff. the Products of Interest torch to Marcelo Wanderley and Joseph Mal- With this issue, Keeril Makan turns Margaret Cahill years ago, and the loch, who are both affiliated with the over the reins as managing editor to Reviews torch to Ross Feller more Centre for Interdisciplinary Research Peter Castine, who will also continue recently, while continuing to produce in Music Media and Technology in his previous role as manuscripts the DVD and edit the DVD program (CIRMMT) at McGill University in editor. Keeril Makan has served notes. His work as producer and Montreal, Canada. After issuing a call over ten years as managing editor, editor of the CMJ DVD was taken for manuscripts and managing the skillfully discharging his duties in over in early 2014 by Doug Van Nort. peer review of the submitted papers, overseeing the editors’ delivery of George Tzanetakis, a well-known they selected five significant repre- manuscripts and in communicating authority on music information re- sentatives of the current state of the with authors and editors about change trieval, has stepped down as editor field. Please read their Editors’ Notes requests. During the same decade he in charge of manuscript peer review, for an overview of their thinking and has also established a successful after over eleven years in this posi- a summary of each of the articles they career for himself as a composer and tion. His technical expertise has been chose. We are indebted to the guest as a professor at the Massachusetts invaluable in assessing the suitability editors for their contribution. Institute of Technology, leaving him of submitted manuscripts before peer The Reviews and Products of less time for the Journal. review and after authors’ subsequent Interest sections in this issue were Other recent staff changes include revisions. His broad knowledge of assembled and edited as usual by Ross the retirement of James Harley and computer music has also stood him in doi:10.1162/COMJ e 00251 Front cover.Thethemeofmapping Back cover.Threefiguresfrom is illustrated by three figures from the the article by Robert Tubb and Simon issue’s first article. That article of- Dixon. Top left: A line segment is fers an in-depth theoretical analysis mapped to a continuous path along of mapping in computer music (re- the edges of a cube, defining a se- flected in part by the top two figures) quence of three-dimensional binary and also describes some concrete in- coordinates illustrated by the pattern stantiations (see the bottom figure, for of black or white squares immedi- example). ately to the right. Bottom left: Simi- lar sequences of coordinates from the corners of a five-dimensional hyper- cube. Right: The user interface of the authors’ Sonic Zoom software, which uses the principles behind the left-hand figures to map a two- dimensional control space to a ten- dimensional sonic parameter space.

About This Issue 1 good stead in identifying prospective Van Nort. We are sincerely grateful and we are confident that the roles peer reviewers. The responsibilities of to Keeril Makan, James Harley, and they filled remain in good hands. managing peer review now rest on the George Tzanetakis for their dedica- shoulders of Lonce Wyse and Doug tion to the Journal for all these years, —Douglas Keislar

2 Computer Music Journal Editors’ Notes

When we use digital tools for making authors’ framework Library of Maps manner, the sensor data generated by music, the properties and parame- (LoM) for performing interpolation of the biomusician exhibits complex in- ters of both sound synthesizers and parameters between two spaces. terrelationships due to the dynamics human interfaces have an abstract In “A Zoomable Mapping of a Mu- of the underlying human body. representation. One consequence of sical Parameter Space Using Hilbert Although the insights and discus- the digital nature of these signals Curves,” Tubb and Dixon concentrate sion of these authors is of course and states is that gesture and ac- on the design of interfaces for easily invaluable, many of the software tion are completely separable from navigating large parameter spaces tools discussed are also publicly sound production, and the system during the ideation or inspiration available for experimentation and use designer must artificially associate phase of musical creation. This fo- by readers. Specifically, the high-level them with sound production in a cus leads them to prioritize ease of tools ISO Flock (Schacher, Bisig, and process commonly called mapping. use and access to the entire output Kocher) and Sonic Zoom (Tubb) are The importance of mapping in parameter space over repeatability available both in application form digital musical instruments has been of control: Users of their system can and as source code, and Caramiaux studied since the 1990s, when several mark and return to points of inter- et al.’s gesture variation follower li- works discussed the role of mapping est in a ten-dimensional parameter brary is also freely downloadable for and many related concepts. Roughly space using only two dimensions of use in C projects. ++ since the mid 2000s, several tools input. Finally, while discussion of map- have been offered that facilitate the Caramiaux, Franc¸oise, Schnell, ping concepts and approaches is some- implementation of mappings, drasti- and Bevilacqua address the use of times necessarily quite technical, in cally reducing the necessary technical machine-learning techniques for the these contributions the discussion knowledge and allowing a large com- semi-automatic creation of mappings is firmly grounded in musical prac- munity to easily implement their between performer gesture and ex- tice. The mathematical formulation ideas. This fact, coupled with the isting sonic material. Their article, presented by Van Nort, Wanderley, availability of inexpensive sensors “Mapping through Listening,” advo- and Depalle is used to describe— and hardware and the emergence of cates a careful consideration of the from a performer’s perspective—the astrongdo-it-yourselfcommunity, perceptual salience of the features of mapping approaches they have used made the time seem right to dis- both musical material and physical for controlling live granulation ef- cuss the main directions for research gestures, grounded in a review and fects using a Wacom tablet. Schacher on mapping in digital musical in- unification of taxonomies by Scha- and co-workers discuss the use of struments. This special issue brings effer, Gaver, Huron, and others. An their tools and approaches for de- together contributions showing the appendix describes technical details signing interactive installations and variety of research on mapping and of the work. artistic work for human dancers and exposing new applications of the field In “The Map and the Flock: Emer- simulated agents; and artistic and per- to musical practice. gence in Mapping with Swarm Algo- formance concerns are the primary The first article, “Mapping Con- rithms,” Schacher, Bisig, and Kocher focus of the contribution from Lyon, trol Structures to Sound Synthesis: take a different approach, using sim- Knapp, and Ouzounian, even while it Functional and Topological Perspec- ulations of agent-based flocking and discusses the treatment of particular tives,” by Van Nort, Wanderley, and other dynamic systems to provide sensors. Depalle, addresses mapping from two added interest and complexity to the We hope that readers will find this perspectives: as parameter mapping control of media output for their material to be interesting and infor- (e.g., associating one type of sensor work. mative. The “mapping problem” and parameter to one type of synthesis Lastly, the contribution by Lyon, its various approaches and solutions parameter) or as continuous naviga- Knapp, and Ouzounian discusses are equally applicable to the design of tion between parameter spaces, and their approach to mapping between any digital system intended for “free” it proposes a unified approach that the members of a trio ensemble; interaction—interaction in which takes both perspectives into account. their concept of “digital musical the precise use cases cannot be well It furthermore shows example control instrument” includes sonic mate- defined in advance. As computers structures that are applications of the rial contributed by the violinist, a and real-time sensor data become “biomusician” wearing various sen- ubiquitous (try counting the sensors sors, and a laptop performer. Even in your mobile phone) these issues doi:10.1162/COMJ e 00252 if mapped in a simple one-to-one will arise more and more, and the

4 Computer Music Journal solutions will define our experiences lowing reviewers who generously con- Hunt, Alexander Refsum Jensenius, interacting with a variety of systems tributed to assessments: Fred´ erique´ Sergi Jorda,` Matthew Paradis, D. An- and environments of both artistic and Bevilacqua, Baptiste Caramiaux, In- drew Stewart, Kojiro Umezaki, and conventional nature. sook Choi, Christopher Dobrian, Paul Doug Van Nort. We would like to thank Doug Keis- Doornbusch, Shlomo Dubnov, Paulo lar for agreeing with our proposal for a Ferreira-Lopes, Sergio Freire, Rolf Inge —Marcelo M. Wanderley and special issue on mapping, and the fol- Godøy, Camille Goudeseune, Andy Joseph Malloch, Guest Editors

Editors’ Notes 5 † Doug Van Nort,∗ Marcelo M. Wanderley, Mapping Control Structures and Philippe Depalle† ∗School of Arts, Media, Performance, for Sound Synthesis: and Design York University Functional and Topological 4700 Keele St. Toronto, Ontario M3J 1P3, Canada [email protected] Perspectives †Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT) McGill University 527 Sherbrooke St. West Montreal, Quebec H3A 1E3, Canada [email protected] [email protected]

Abstract: This article contributes a holistic conceptual framework for the notion of “mapping” that extends the classical view of mapping as parameter association. In presenting this holistic approach to mapping techniques, we apply the framework to existing works from the literature as well as to new implementations that consider this approach in their construction. As any mapping control structure for a given digital instrument is determined by the musical context in which it is used, we present musical examples that relate the relatively abstract realm of mapping design to the physically and perceptually grounded notions of control and sonic gesture. Making this connection allows mapping to be more clearly seen as a linkage between a physical action and a sonic result. In this sense, the purpose of this work is to translate the discussion on mapping so that it links an abstract and formalized approach—intended for representation and conceptualization—with a viewpoint that considers mapping in its role as a perceived correspondence between physical materials (i.e., those that act on controllers and transducers) and sonic events. This correspondence is, at its heart, driven by our cognitive and embodied understanding of the acoustic world.

Mapping is generally considered as the correspon- rather to develop a language and conceptual appara- dence one creates between the control parameters of tus by which one may define a personal instrument an input device and the parameters of a given sound- design space, taking into account the mutual influ- processing technique (Hunt and Wanderley 2002; ence between functional properties of a mapping and Miranda and Wanderley 2006). Moving beyond this, the gestural response of the instrumental system as the precise meaning of mapping changes depending first articulated by Van Nort (2010a). upon one’s intent, and its role can similarly change depending on one’s relationship to the notion of “gesture” (both human input and resulting sonic Perspectives on Mapping gestures) as well as the on choice of controller and sound-synthesis algorithms. In this article we In the context of instrument design there are several examine the important, but understudied, topic of different ways that one might conceptualize the the properties of mappings and how they help to mapping component, which in turn will affect define the feel of an instrument, particularly func- design strategies for associating control and sound- tional properties related to a mapping’s geometric processing parameters. Chadabe (2002) differentiates and topological structure in the case of continuous, a hierarchical point of view, in which a few high- many-to-many mappings. Our goal is not to propose level parameters unidirectionally control many the definitive taxonomy of musical mappings, but low-level parameters, from a network point of view, in which all parameters can equally send and Computer Music Journal, 38:3, pp. 6–22, Fall 2014 receive control information in a non-deterministic doi:10.1162/COMJ a 00253 manner. This viewpoint assumes the perspective c ⃝ 2014 Massachusetts Institute of Technology. that mapping is a series of correspondences, or the

6 Computer Music Journal Figure 1. A generalized Figure 2. A generalized systems-oriented view of functional view of mapping from control to mapping from control to sound parameters. sound parameters.

out-of-time snapshot of input/output (I/O) control each association may be modified, conditioned, or potential. We refer to this as a systems point of view otherwise warped in order to tune the response of on mapping, which is the liaison or correspondence agivenparameter,asisillustratedbythetransfer- between control and sound parameters. This view function-like representation of the intermediate is represented by the classical “flowchart” paradigm stage. that is ubiquitous in engineering. We differentiate Meanwhile, a functional view is concerned with this from a functional point of view, defined by the dimensionality and structural properties of the operations that associate different sets of the set of I/O parameters that a mapping acts on, variables, such as between control and sound- as well as the properties of the mapping function synthesis parameters. These sets can, themselves, itself. This is illustrated in Figure 2, which shows a be endowed with properties that may be inherited, composition of mapping functions f and g, mapping constrained, or altered by the operations that map between control set X and sound-synthesis set between them. There is further a perceptual point Y.Ratherthanindividualparameterconditioning of view, which is the sensation of human intention or interconnectedness, as in a systems view, the leading to some representative sonic result. This focus here is on the properties of X, Y, f , g,etc.Of view is integrally tied to the notion of gesture, particular interest is the mapping topology—that particularly in the broadly defined sense presented is, the nature of the continuity, connectedness, and by Godøy and Leman (2009). Unlike a perceptual boundary definition in the mapping association sound feature (e.g., pitch) a perceptual view on (or associations) between control and sound sets. mapping can only be identified after an action Taking the mapping topology as a main point of has taken place, and so there is no quantifiable concern, the functional nature of mappings will be description a priori. the primary subject of our discussion, although it We discuss the perceptual view in more detail will become clear that a systems viewpoint must elsewhere (Van Nort 2010a); here, we focus on the be adopted at times in order to fully explain an systems and functional points of view. These two instrument design, and a perceptual point of view viewpoints act as duals of one another. Shifting arises whenever sonic or musical intent is described between perspectives helps one to have a complete or reflected upon. picture of a performance system. In some instances, Looking again to Figure 1, the direction arrows the views are vastly different and give unique signify the given control-to-sound parameter as- insights into design possibilities. sociation or “what to map where.” Meanwhile, The representation of a systems view, depicted in the transfer and warping functions of the middle Figure 1, considers the number of degrees of freedom layer describe the (potentially) continuous response afforded by a physical controller or synthesizer, extended to any given gestural input. This distin- and how each of these degrees or parameters guishes the “what” aspect of mapping in the former can be associated. Also represented is the level case from the “how” aspect in the latter, or what of complexity in terms of interconnectivity of Tanaka (2010) would refer to as a “basic” versus an parameters, often discussed in the literature in “expressive” mapping. From a functional viewpoint, terms of one-to-one, one-to-many, or many-to-one the “what” aspect of mapping manifests itself as the mappings (Hunt and Wanderley 2002). Further, point-wise association between an n-dimensional

Van Nort et al. 7 controller space and an m-dimensional space of apoint(orstate)throughthisgivenparameterspace. sound-synthesis parameters. If we associate such a point in “control space” with At the same time, the “how” aspect can be seen aparticularpointinacorresponding“soundspace” as the rules governing the association of entire (i.e., a set of sound-synthesis variables), and we subregions of the respective parameter spaces. The define a mapping function between spaces, then “what” aspect dominates discussion on mapping our action will in turn continuously drive the given and clearly is of great importance, as it pertains to point that exists in the sound variable space, thereby the important issue of perceived “feel” and control affecting the sonic output. complexity (Hunt and Kirk 2000). The “how” Now, in practice one will encounter situations element, however, also influences perceived feel, in which control- and sound-parameter spaces have particularly in the case of mappings concerned with different dimensions (i.e., their parameter sets will continuous shaping of sound (Van Nort 2010a), have different sizes). In this case, and in a functional such as the subsequent examples that focus on mapping context, the degree of freedom of motion continuous gestural control of timbral and textural within any given parameter space is bounded by the elements of sound. lowest dimensional space through which we map. That is, the smallest parameter space will be the constraint on the number of possible directions in Dimensionality and Mapping as Embedding which we might drive the system. In mapping a control space into a space of sound variables, we can In order to assess the benefits of a functional view conceptualize this as placing the control space itself on mappings, we need to establish a conceptual “inside” the sound-synthesis space. The particular understanding that draws on the worlds of geometry manner in which a control surface “occupies” and topology. We develop such an understanding in the space of sound-synthesis parameters will be this section, reserving more rigorous mathematical aproductofhowitscontroldataisembedded. formalisms and analysis for the Appendix. To This is an important determinant of the perceived that end, a musical instrument can be viewed as quality of the larger musical control structures that acollectionofdiscreteandcontinuouscontrol result. variables and their sonic effect. Although it is In the world of mathematics, the notion of an certainly true that gestures acting on discrete embedding suggests that the essential shape of the controls can be quite a complex system in and source (or domain) space is preserved under a given of themselves, the underlying control gesture mapping—which is to say that the mapping is a embedded within the control stream is often directed continuous deformation of the space, but that it towards the end of exciting a dynamic sound model, does not break or “tear” it. That said, we expand the which then contains most of the complexity of the term “embedding” here to include any parameter system. Consider, for example, the excitation of a space continuously mapped into another space of piano by striking a key, after which the instrument the same or larger dimension. These mappings resonates in a rich and complex manner with no can also have inverses. In this context, an inverse direct influence from a human hand. In this article corresponds to the unique function that “undoes” our goal is to include the most complex cases of agivenmapping,associatingeachsoundstate control structuring in general, and so from this point back into the corresponding state in the control forth we assume an underlying set of continuous domain. In the context of this discussion, the control parameters n.Fromamathematicalpoint mapping examples that we will present will all be of view, this set can be seen as a subspace of the n- continuous, though whether or not their inverses are dimensional Euclidean space Rn,witheachpossible continuous will depend on the point-wise mapping, combination of these n variables being a single point which represents the “what” element of a functional in this space. Following with this representation, mapping in the sense that it arises from a direct acting on the given controller can be seen as moving correspondence between input and output points. A

8 Computer Music Journal point-wise mapping differs from a set of parameter many intermediate timbres, greatly affecting the correspondences in that the functional and spatially dynamics of the sound parameters and therefore the oriented design approach arises from a holistic resultant sonic gesture. From a functional point of manner of conceptualizing an instrument (Hunt and view, we also cannot guarantee that an input control Kirk 2000). On the other hand, a systems-oriented space is smoothly and uniformly deformed into its view presents a more analytic cognitive model— sound output counterpart (i.e., that they are what a being focused on the correspondence of individual mathematician would call homeomorphic spaces). parameters rather than overall states. Meanwhile, Rather, the newly mapped space may “collapse” in the “how” aspect of a functional-mapping approach on itself so that two different inputs to a controller corresponds to the geometry of a given mapping could potentially result in the same sound. technique and its influence on the “interdynamics” As a general rule this is not musically undesirable of all parameters, rather than conditioning of single- behavior, and in fact it could be a design goal to parameter dynamics. have the same sound outputs occurring in several From a practical standpoint, the mapping geom- different control states. By contrast, this would be etry is determined by the manner in which a given aprobleminsystemsbasedoninversemodeling, technique performs interpolation or extrapolation where an intended output becomes the domain on the known control or sound states. Further, for an inverse mapping to the given control input. the nature of the embedding for a given mapping Examples can be found in the control of semi- influences the perceived feel of control. In particular autonomous robots (Hulsen¨ 2008) or in interactive we distinguish between what we will call a direct system design projects that define mappings based on embedding and a mixture embedding. sound similarity measures such as those discussed In the case of direct embedding,theusablecontrol by Eigenfeldt and Pasquier (2010). In these cases, space—with boundary and interior defined by a given alearningalgorithmisrequiredtoguaranteethe mapping technique—is “placed” directly within a preservation of the I/O topology, so that the two target sound space (of the same or higher dimension) are properly conditioned and the mapping geometry through a straightforward linear mapping. This can does not cross through itself, tear, etc. A direct be of interest if, for example, one would like to approach is an embedding in the strict sense of “hear the geometry” of a shape that is embedded mathematics, so generally speaking it is musically in sound-parameter space. An example would be less interesting, in that it retains its shape at the a mapping used for the sonification of data (Axen expense of not considering perceptual metrics or 1998), where one might want to hear the actual musical interrelations within control-to-sound nature of, say, the curvature, volume, or number space. This type of embedding may therefore be of holes of a given geometric shape via a direct more useful for very high-level control spaces, for embedding into the space of sound parameters. the aforementioned world of data sonification, or The difference between a mixture embedding as visual feedback. In general we find a mixture and a direct embedding lies in the point-wise embedding to be more powerful for designing useful mapping. Instead of preserving the geometry of control structures, particularly for timbral control. an input space, a mixture embedding creates the structure by user-defined associations between input and output states. In doing this, there is Relevant Properties an implicit perceptual distance imposed on the two spaces: Something that is considered by the In addition to the nature of the embedding, certain designer as “close” and grouped accordingly may other functional properties of the mapping can help actually be quite far in the Euclidean space of one to understand the local geometric and topolog- sound parameters. The result is that movement ical structure, as well as how this influences the of a controller may induce a trajectory in sound overall control structure. The continuity of a map- space that covers a large distance and potentially ping is clearly important for control contexts such

Van Nort et al. 9 Figure 3. A projection from determines the ultimate regions of a topology when mapped two-dimensional control into a three-dimensional space onto a sound-synthesis parameter one-dimensional space (with grey “shadow” intermediate space, which on the x–y plane).

as sound morphing. In past work we have discussed other relevant properties including differentiability, smoothness, and linearity (Van Nort, Wanderley, and Depalle 2004), and the importance of distinguishing between explicit and implicit mappings is discussed by Hunt and Wanderley (2002). Also important is exactness: A mapping is exact if the underlying, point-wise mapping is preserved within the larger of the entire control structure. Whereas one may mapping geometry. This is important if a certain often encounter a topology determined by a single sonic response is required at a particular state of control-to-sound point-wise association and a single the control input. Also, the global versus local structure of functional mapping, in a multi-layered definition is key. A mapping is globally defined if mapping context the dimension of “topology space” all states from the underlying point-wise map exert need not be the same as the underlying control or influence on the entire geometric structure of the sound spaces. As an example, which will be further mapping. If the influence at a given point is isolated elaborated in the Musical Examples section, con- to certain nearby neighbor points, then the mapping sider a standard two-dimensional trackpad. Suppose is locally defined. Finally, a mapping is parametric that we wish to drive output sound parameters in if it depends on input control variables that can such a way that the two degrees of control freedom influence it, which can be important for adapting a strictly influence a localized area of sound space, mapping to achieve a certain gestural response. whose topology is equivalent to a one-dimensional line. In this case we may map entire regions of input space onto an intermediate one-dimensional Multiple Layers, Effective Topology, space by continuously projecting onto it. This, in and Space versus Time turn, maps into sound-synthesis space, as illustrated in Figure 3. In short, just as available degrees of In practice, a given musical context will often require freedom determine the underlying dimensionality a modular combination of different mapping struc- of parameter spaces, the number of intermediate tures, which may mean the use of multiple mapping mapping layers and their topology will likely in- layers. This is discussed by Wanderley, Schnell, fluence the perception of a system’s immediacy of and Rovan (1998), Hunt and Wanderley (2002), and control. later by Arfib and colleagues (2002), where the This perception of sonic control is inherently authors introduce the notion of “abstracting” a temporal, and so a spatial representation of mapping mapping between high-level gesture-perceptual and must be somehow reconciled with a temporal one. sound-perceptual spaces. Approaches such as these There is a “hidden temporality” of a spatial view seek to build up from many low-level parameters on mapping which arises during the construction to the few most relevant and salient parameters by of a point-wise mapping. One is not just defining introducing intermediate layers. These layers may asteady-stateresponseoftheinstrumentbut, have varying degrees of relationship to either the act rather, the nature of the embedding will define of signal conditioning on the one hand, or the core acertaingesturalresponsefortheinstrument. structure of the mapping proper on the other (as we Goudeseune (2002) presents a mapping approach will discuss further in later sections). If the desired that is spatial, wherein he constructs point-wise perceptual mapping consists of holistic control over mappings automatically, thus giving maximal some abstracted parameters that have d degrees of timbral variety within the sonic space. This requires freedom (where d n, m,withn the dimensionality one to adjust the I/O topology manually, however, to of the controller space̸= and m the dimensionality of tune the overall gestural and temporal quality after the space of sound-synthesis parameters), the multi- the fact. Additionally, the mapping can vary over layered mapping must address the effective topology time in order to influence the perceived “feel,” such

10 Computer Music Journal as the dynamic mappings presented by Momeni and of simplicial interpolation (SI) within Max/MSP. Henry (2006). The technique begins with a point-wise mapping between control and sound spaces, afterwards creating a triangulation of the points in control Mapping Functions space. This, in turn, induces a triangulation in the higher-dimensional sound space, so that localized In order to ground these notions of multiple mapping simplices (i.e., n-dimensional triangles) in the layers, the influence of mapping topology, and the control space are associated with similar regions interdependence of spatial and temporal qualities, in the sound space. The distance to each of the we will present four mapping examples that have vertices (i.e., control states) of a given simplex arisen in the context of the computer music perfor- creates a continuous weighting, which in turn mance practice that was first described by Van Nort interpolates an output based on the associated area (2010b). To that end, we must briefly describe three in the sound space. This piecewise linear technique mapping techniques that we utilize as modules is a mixture embedding that preserves its local within the example mapping control structures. topology, although, depending on the user-defined Then we will formally compare and contrast these point-wise map, the output surface may be stretched three along with other existing techniques that take and “crinkled” in sound space. afunctionalapproachtomappingintheAppendix, resulting in a “mapping design space.” RST: Drawing from Terrain Mapping

Colorblobs: Gaussian Kernels We have drawn on work from geographic infor- mation systems (GIS) through application of the Momeni and Wessel (2003) present a technique in completely regularized spline with tension (RST, which a mixture of Gaussian kernels (called “blobs,” cf. Mitas and Mitasova 1988). We first presented due to their visual depiction) map a two-dimensional the RST method as a musically inspired mapping control space into an m-dimensional sound space. technique (Van Nort, Wanderley, and Depalle 2004). Users may determine the spatial layout and position The RST technique complements the other methods for each Gaussian blob. The blobs are associated in that it has smoothness and exactness that can be with sound space via point-wise mapping in order parametrically tuned (in a trade-off), and in that it to create a mixture embedding. This continuous is globally defined. The tension parameter controls mapping is a linear combination of the normalized the exactness of the surface, and has been optimized Gaussian kernels and is global, as these kernels are to move between a model of a rigid surface (a “thin defined everywhere, thereby exerting an influence plate”) and a malleable one (a “flexible membrane”), on the output at every point in the input control while the smoothing parameter is used to deter- space. The mapping follows what we refer to as mine the global smoothness of the surface through a gravitational system model, in that each blob minimization of a function which models “bending exerts its mass over objects that hover around it. In energy.” These two parameters interact and must be addition to moving and placing blobs, this approach tuned manually, with guidelines given by Mitasova can be tuned by changing the kernel variances to and Mitas (1993). exert more or less influence in the x or y direction.

Examples: Granular Scrubbing SI: Scattered Simplicial Interpolation In 2007 we presented control structures that focused We build upon work from Goudeseune (2002) and on modular combinations of mapping functions, Bowler et al. (1990) through creation of a method with parametric control used to define gestural

Van Nort et al. 11 Figure 4. Subsections of layer). Conditioning of the tablet surface are these through mapped to playback for overlap-and-minimum of sections of a sound the parallel control spaces recording (top layer) while creates a one-dimensional also being associated with topology in sound space distinct two-dimensional (bottom layer). control spaces (middle dynamics (Van Nort and Wanderley 2007). We extend that work here by focusing on the gestural act of “scrubbing” through waveforms, using granular resynthesis and multiple layers of these three mapping techniques, via examples taken from our granular-feedback expanded instrument system (GREIS) for performance (Van Nort, Oliveros, and Braasch 2013), which has been used in many performances internationally over the past ten years. In this scrubbing interaction, there are in fact three gestures that must be considered in the design process: the control gesture, the sonic gestural output, and the sonic gesture inherent in the material to be scrubbed.

Designing Topology Space

In scrubbing through sound samples, mapping the sample playback position to one of the positional dimensions of a Wacom tablet allows for the ma- terial to be played at variable speeds and starting points—a more complex and flexible version of turntable scratching. This type of control becomes most interesting when other sound parameters are modulated continuously as the material is scrubbed. such a continuous one-dimensional topology in the Doing this in a holistic way is difficult, because source sound space, the chosen tablet axis that is the mapping from linear position to sample time mapped to sample time is segmented, with each dominates the control possibilities. Musical source resulting 2-D “slice” being mapped into a different signals (e.g., vocal samples) are, however, generally two-dimensional control space. States in this space highly structured, and different combinations of are created such that overlapping areas between sound parameters are more appropriate at different segments on the tablet share the same mapping to points in the file, as are different transitions be- granular synthesis parameters—thus the mapping is tween these sound states (e.g., between vowels and duplicated and in the same spatial location in each consonants). control subspace (see the middle layer of Figure 4). Both of these facts can be considered in the map- Using this structure, one can define different sec- ping design. An effective approach is to utilize a con- tions of the sample to have unique sound-processing tinuous global scheme, such as the aforementioned characteristics due to the layout of each control Gaussian kernel method. Recall that this behaves in subspace. a“gravitationalsystem”fashion,inthatinsertion In regards to global scrubbing across frames: of states with suitable mass (i.e., sufficiently large Taking the minimum for each parameter of two x and y variances) can block the influence of more given contiguous states (from neighboring control distant states. In this way, the two-dimensional subspaces) results in the correct number of point- control surface of a tablet can be projected onto the wise mappings, and ensures that the final output is one-dimensional topology defined by the left-to- continuous. Further, the variance of each Gaussian right states of a given sample (a simplified version of kernel can be changed individually so that the which was depicted in Figure 3). In order to produce transition between two states is different when

12 Computer Music Journal Figure 5. Control structure is dynamically altered as input velocity influences the intermediate topology space, as well as the shape of the mapping-to-sound space. approached from the left or right sides of the tablet. This structure thus allows one to define the modulation potential based on particular regions of an audio sample, with continuous control of the sound arising as one scrubs across it. For example, if scrubbing is mapped to x position, then every left-to-right trajectory with the same x speed will give rise to the same scrubbed source material. A unique overall sound will, however, be produced by virtue of the y-axis variations arising from a particular path taken across the tablet surface, as this in turn creates a unique path through sound space. Thus, one can design an intuitive modulatory control that is based on the inherent sonic gestural quality of the source material by tailoring the transitions between appropriately selected sonic states to fit this material.

Controlling Topology Space

If the sonic source material used for scrubbing is changing dynamically, then a system must likely adapt to the changing sonic gestures found in this material. Allowing for influence from nonadjacent SI strategy for this layer as it provides continuous states can make the control gesture adapt to the control while allowing for constant mapping in changing material by dynamically mapping to localized sections of the “velocity space.” different trajectories in sound space and allowing for To further condition the dynamics, an expo- “alternate paths” between sound states. Therefore, nentially windowed moving-average (MA) filter (or we have designed a version of the Gaussian overlap- leaky integrator) is applied to the output of the and-mininimum mapping structure that moves SI velocity space. The addition of this particular continuously between a one- and a two-dimensional dynamics conditioning becomes most interesting by topology, while adapting the dynamics of a given gating the input so that this mapping layer is active control gesture, as depicted in Figure 5. The result is only when the stylus is in contact with the tablet. that the path, as well as the speed in sound space, is By doing this, when a scrubbing control gesture is parametrically varied. looped the initial processing will occupy a localized In order to account for the contextual effects of two-dimensional region of sound space. As the leaky scrubbing speed, we utilize the velocity in both MA filter dies down, however, the granular process- x and y directions to influence the size of the ing will fade back to a particular one-dimensional segments. This indirectly controls the topology of path as in the previous mapping control structure, the space, in that larger segments allow for influence which may be tailored for a particular sound sample. from non-adjacent states. Also in the design of this Therefore, in this mapping control structure, the mapping layer, the x and y variances of the Gaussian initial excitation energy is mapped to a unique states are driven by x and y velocities, which serves continuous transformation, while a looped control the dual purpose of influencing the 1-D versus 2-D gesture can traverse another part of sound space topology, as well as the between-state transitional that is designed to work well for particular sound nature of the resulting control dynamics. We use the material.

Van Nort et al. 13 Figure 6. Example mapping (SI) mapping. The response structure in which and dynamics of the scrubbing is based on control gestures are temporality of gesture. conditioned by another Speed of input action filtering layer that is, itself, determines granular gesturally controlled by processing through position or tilt. simplicial interpolation

Control Gesture Design Through Control of Mapping Layer

When improvising with acoustic musicians, the tablet-based player may want to modulate sound streams as they happen, with the scrubbing rate being controlled indirectly as a product of the input gestural energy. In these instances, a musically relevant approach is to use input gesture time itself as the temporal position variable, and to utilize the speed of input gestures to define the resultant control gestures. One such design is presented in Figure 6, wherein we map the time of contact with the tablet into the scrubbing playback time of the sample buffer. In this way, the player can move across the tablet in any direction and this will scrub through the material in the same manner. A physically intuitive mapping layer results if we map x and y velocities (taking the order to compensate for this and to accentuate the maximum of these two) into the speed of scrubbing. resonance of certain scrubbing speeds, we added This allows the entire area of the tablet to be mapped an RST-based mapping from the x and y velocities into sound parameters for the underlying granular to the responsiveness of the MA filter (defined by synthesis. Although one could simply map the window size and exponential damping coefficient). tablet surface area—using a mixture embedding— In adding this response-conditioning layer, a given into sound parameters, the variable-speed nature of scrubbing speed will result in a trajectory through this control gesture suggests that the type of granular sound space that varies in a nonlinear way, such that processing similarly be tied to the control dynamics. certain speeds can be tuned to move more slowly or With granular scrubbing, certain parameter values quickly as needed. (such as window size and grain density) are more In the language we have developed in this coherent and produce fewer artifacts at particular article, this response mapping is therefore a two- speeds (e.g., during slow scrubbing). And so we map to-one direct embedding from velocity space into from the x/y velocity space into granular parameters the responsiveness of the MA control filter. This using the SI technique to create a mapping that velocity-to-response mapping creates a complex is exact and local, and that may be defined by a control gesture that varies based on speed and scattered set of points. overall gesture time. Through initial explorations with this instrument In order to make the response itself more control- it became clear that some latency and feeling of lable, the velocity-to-response mapping is varied by mass were needed in the case of a fast gesture and its changing the smoothness and tension of the RST ultimate granular modulation. So, an exponential layer, by mapping either from x/y position or by MA filter was, once again, used to condition the tilt. Therefore it is the action-to-control gesture mapping from x/y velocity—greatly improving the that changes, and this indirectly influences the overall feel of the scrubbing. This feel was still overall sonic gestural output. This resultant control somewhat static, however, in that certain speeds structure thus illustrates how a highly dynamic were perceived to have a smoother decay than mapping can give a coherent and repeatable gestu- others, owing to the perceptual distance between ral response through a modular construction that parameter sets in the granular-parameter space acts primarily on the control-side parameters. The (i.e., due to the SI-based mixture embedding). In overall mapping structure is depicted in Figure 6;

14 Computer Music Journal Figure 7. Velocity-to- tension (RST). This causes Figure 8. Feedback- circle at the bottom of the response mapping. The an interpolation between adaptive mapping diagram labeled TFS actual path in this space is these possible response structure: changing represents temporal fine determined by controlling curves. The set of “x” mapping layer based on structure. the smoothness and symbols represents the sonic gestural output. The tension parameters of the point-wise mapping from regularized spline with velocity to response.

the velocity-to-response mapping is illustrated in Figure 7.

Sonic Gesture Design Through Feedback Control of Mapping Layer in order to link control and sound gestures in a The previous example was inspired by the need perceptually more meaningful way. for a control gesture that varied with the temporal We have chosen to end the section with this behavior of the scrubbing, so as to link control and particular structure, depicted in Figure 8, because it sonic gestures in a contextual way (changing with arises not only from a consideration of the gestural sonic source material). By focusing on sonic gestural aspect of control design, but also takes a synthetic output, this process can be extended by designing view of mapping from functional, systems, and acontrolstructureinwhichsoundfeaturesdrive perceptual points of view. The functional point of this adaptation in a feedback control loop. One view considers the structure of the parameter space, effective mapping control structure resulted from the systems point of view considers the complexity the desire to focus more on textural qualities during of parameter associations, and the perceptual point slow scrubbing, eliciting different control “feel” of view takes the ultimate action-to-sound response depending on the spectral profile of the source into account. material. In order to create such a design we have drawn upon the “morphological” sound features described Discussion in prior work (Van Nort 2009). In particular, for this structure we extract the temporal fine structure We began the article by establishing the notion of a (TFS) as a measure of transient grain and use this to functional view on mapping, and then we examined adapt the scrubbing speed. Further, the RST-based various relevant properties of a mapping function. control–response mapping (which was driven by Such discussion is necessarily abstract until it gestural input in the last example) is adapted, first is applied to an actual musical context, which by extracting a short-time window of amplitude we have just done through the given examples. and matter profile features (e.g., spectral centroid, Indeed, in the design of any mapping structures deviation, and spread), then mapping these back one should consider the musical context within into RST smoothness and tension. This results in a which an instrument will be used, as well as the mapping control structure that takes into account expressive goals of performer and composer. This the sonic gestural response of the instrument will determine the level of control—whether it

Van Nort et al. 15 be guiding some higher-level musical processes or temporal window over which control-sound events controlling the moment-by-moment production of take place. sound in response to a performer’s gestural input. In contrast, we did not discuss mapping as a cre- The former often necessitates the consideration ation of relationships between musical gestures and of completely new metaphors for interaction (cf. larger compositional forms (Doornbusch 2010). This Wessel and Wright 2002), whereas the latter can notion of “mapping as composition” exists out-of- more directly find inspiration in the paradigm time, in the sense of acting in the space of musical of instrumental music performance or that of ideas, and in actuality it falls well outside the physi- ecologically based object interaction. For example, cal realm of human action and response. In terms of the tablet-based instrument we have described is functional mapping and higher levels of control of inspired not only by turntablism, but also by bowing musical structure, however, a mapping may act on a gestures and similar continuous friction interactions parameter set’s algebraic structure if it is concerned (Serafin et al. 1999). In more instrument-oriented with transformations of the symbolic interrelations cases such as this, the performer has direct control within the set. For example, Milne et al. (2007) over the smallest details of performance (Schnell presented an “isomorphic controller” for dynami- and Battier 2002), and the system in question is the cally altering the pitch mappings of a specialized short-term dynamic human input as well as the keyboard. That said, we focused here on a functional energy-dependent sound processing that is activated mapping’s geometric and topological structure, and by this. the definition of mappings that act dynamically on Where along this spectrum of immediacy the values themselves, resulting in a mapping approach musical interaction context lies will determine, focused on signals rather than symbols. from a designer’s perspective, the role that mapping We have articulated several qualities that are plays. Chadabe (2002) suggests that the mapping musically relevant for consideration in the design concept becomes limiting in digital “instrument” process. And we feel it a worthwhile endeavor to design, in that more novel and contemporary designs present an overview of existing mapping techniques tend to have intermediary and indeterministic that take a holistic and geometrically oriented ap- functions that lack a clear mapping of human proach, comparing and contrasting these properties input to sonic output. Although this may often be to inform choices for others who might be engaged the case, even highly complex interactive systems in the design of mapping control structures. Owing often consist of global networks of local control- to the mathematical nature of this discussion, we to-sound mappings that behave, from a perceptual present this in the Appendix. control point of view, “nearly” deterministically Regarding the “what” and “how” aspects of a or causally. A direct response to physical actions mapping strategy, we must remember that mapping might have a correlated sonic response, which per se is ultimately tied to perception and, more may then evolve independently of the performer’s directly, to intentionality. In the former case this actions or intentions. This is the case with the means building a mapping around certain key GREIS system, wherein scrubbing of sound is action–sound associations through the design of an immediate excitation that is then propagated appropriate correspondences of system states. through the system’s memory and subjected to In the latter case, this means conditioning this machine decisions (Van Nort, Oliveros, and Braasch association towards the continuous gestures that 2013). The examples we have presented focused on will ultimately be experienced. In this process, this scrubbing excitation, wherein the definition the continual transformations of input parameters of the mapping layer was a central concept. Its to intermediate or output parameters may be an relevance was a product of the immediate level element of the mapping, if they have a direct of control over the sound-production mechanism effect on the perception of performer intention, or (influencing lowest-level signal parameters rather if they establish a link between input and sonic than symbolic musical parameters) and the short gestures. If this is not the case, however, such

16 Computer Music Journal transformations can be considered as relating to that this strictly describes parameter association. signal conditioning—that is, focusing on the input The relevancy of mapping was seen to be a function compatibility requirements of a system related to, of musical control context—ranging from interactive for example, linearity, boundaries, or bandwidth. In systems that control high-level musical events to the design process, this manifests itself either as the immediacy of digital instruments with a close the conditioning of input control signals or as the coupling of action to sound. Assuming the latter creation of transformations that ultimately affect context, three different perspectives on mapping the sound parameters. were identified: a systems view that is focused on Thus, although the boundary between operations interparameter association, a functional view that is on signals that can be considered mapping and concerned with a holistic approach to constructing ones that we consider as signal conditioning is parameter spaces, and a perceptual view that is blurred, one separating factor is whether such tied to perceived intentionality and expression. operations act on data that are considered part of We delved more deeply into a functional approach the control data or part of the sound parameter to mapping, formalizing this with a geometric data. Given the unidirectional control-to-sound interpretation and suggesting a framework for nature of most music performance systems, it is analysis. We have applied that framework to existing common that signal operations on the “control mappings from the literature as well as our own side” manifest themselves as signal conditioning, extensions, leading to a design space through which while similar operations on the “sound side” tend to understand their deeper structure. to be more properly a part of the mapping, which Informed by this groundwork and this set of itself is precisely the “in-between” that considers techniques, we presented examples of mapping both control and sound equally. Naturally, these control structures that were designed for natural feel things are a product of design context, and in our and subtle articulation possibilities, using a fairly examples we presented control structures whose limited input control device (specifically, a Wacom layers had differing roles related to the conditioning, tablet). These examples focused on adaptive control mapping, and design of control-to-sound gestural of mapping parameters themselves, altering the response. In these examples, a key aspect was to geometric and topological structure of the mapping define mapping functions that were parametric and in order to dynamically change the control-to-sound could be dynamically altered to achieve a desired coupling. Such an approach illustrates the way that gestural response. This presented one solution to the desire to create certain control or sonic gestures the difficult and intrinsic problem of space versus is built into the mapping control structure itself, time in defining mapping structures. Another subject to the given musical control context, and approach would be to begin by parametrically includes consideration of the dual representations describing the actual dynamics of control or of of space and time in mapping design. sound (e.g., Metois´ 1996; Schoner et al. 1999), Our hope is to further open up the dialogue on and to work towards a spatial structure in order mapping, to move beyond a purely systems-oriented to understand “near” versus “far” control states. view, extending it to a functional and perceptual Regardless of the “direction,” there exist these two view of the subject. dual representations of a control structure, and one must explore external relationships, such as human perception, in order to realize a coherent Acknowledgments link between the two. The first author wishes to acknowledge the McGill Tomlinson fellowship program, the National Science Conclusion Foundation CreativeIT program, and the Banting fellowship program for supporting various phases This article began with a discussion of what precisely of this research. The second author would like constitutes “mapping,” expanding upon the notion to thank the Natural Sciences and Engineering

Van Nort et al. 17 Research Council of Canada (NSERC) Discovery Hunt, A., and M. Wanderley. 2002. “Mapping Performance Grant program for their support. Parameters to Synthesis Engines.” Organised Sound 7(2):97–108. McAulay, R., and T. Quatieri. 1986. “Speech Analysis/ References Synthesis Based on a Sinusoidal Representation.” IEEE Transactions on Acoustics, Speech, and Signal Arfib, D., et al. 2002. “Strategies of Mapping between Processing 34(4):744–754. Gesture Data and Synthesis Model Parameters Using Metois,´ E. 1996. “Musical Sound Information.” PhD Perceptual Spaces.” Organised Sound 7(2):127–144. Dissertation, Massachusetts Institute of Technology, Axen, U. 1998. “Topological Analysis Using Morse Theory Cambridge, Massachusetts. and Auditory Display.” PhD Dissertation, University Milne, A., W. Sethares, and J. Plamondon. 2007. “Iso- of Illinois at Urbana-Champaign. morphic Controllers and Dynamic Tuning: Invariant Bencina, R. 2005. “The Metasurface: Applying Natural Fingering over a Tuning Continuum.” Computer Music Neighbor Interpolation to Two-to-Many Mappings.” In Journal 31(4):15–32. Proceedings of the Conference on New Interfaces for Miranda, E. R., and M. Wanderley. 2006. New Digital Musical Expression,pp.101–104. Musical Instruments: Control and Interaction beyond Bevilaqua, F., R. Muller, and N. Schnell. 2005. “MnM: the Keyboard.Middleton,Wisconsin:A-REditions. AMax/MSPMappingToolbox.”InProceedings of the Mitas, L., and H. Mitasova. 1988. “General Variational Conference on New Interfaces for Musical Expression, Approach to the Interpolation Problem.” Computers pp. 85–88. and Mathematics with Applications 16:983–992. Bowler, I., et al. 1990. “On Mapping N Articulation onto Mitasova, H., and L. Mitas. 1993. “Interpolation by M Synthesiser-Control Parameters.” In Proceedings Regularized Spline with Tension: I. Theory and Imple- of the International Computer Music Conference, mentation.” Mathematical Geology 25(6):641–655. pp. 181–184. Momeni, A., and C. Henry. 2006. “Dynamic Independent Chadabe, J. 2002. “The Limitations of Mapping as Mapping Layers for Concurrent Control of Audio and aStructuralDescriptiveinElectronicMusic.”In Video Synthesis.” Computer Music Journal 30(1):49– Proceedings of the Conference on New Interfaces for 66. Musical Expression,pp.38–42. Momeni, A., and D. Wessel. 2003. “Characterizing Doornbusch, P. 2010. “Mapping in Algorithmic Com- and Controlling Musical Material Intuitively with position and Related Practices.” PhD Dissertation, Geometric Models.” In Proceedings of the Conference Royal Melbourne Institute of Technology, Melbourne, on New Interfaces for Musical Expression,pp.54–62. Victoria, Australia. Rovan, J., et al. 1997. “Instrumental Gestural Mapping Eigenfeldt, A., and P. Pasquier. 2010. “Real-Time Timbral Strategies as Expressive Determinants in Computer Organisation: Selecting Samples Based upon Similar- Music Performance.” In Proceedings of Kansei: The ity.” Organised Sound 15(02):159–166. Technology of Emotion Workshop,pp.68–73. Godøy, R. I., and M. Leman, eds. 2009. Musical Gestures: Schnell, N., and M. Battier. 2002. “Introducing Composed Sound, Movement, and Meaning.Abingdon,UK: Instruments, Technical and Musicological Implica- Routledge. tions.” In Proceedings of the Conference on New Goudeseune, C. 2002. “Interpolated Mappings for Musical Interfaces for Musical Expression,pp.138–142. Instruments.” Organised Sound 7(2):85–96. Schoner, B., et al. 1999. “Data-Driven Modeling of Acous- Hulsen,¨ H. 2008. “Learning Controller for Microrobots.” tical Instruments.” Journal of New Music Research In S. Fatikow, ed. Automated Nanohandling by 28(2):81–89. Microrobots.Berlin:Springer,pp.57–101. Serafin, S., et al. 1999. “Gestural Control of a Real-Time Hunt, A., and R. Kirk. 2000. “Mapping Strategies for Physical Model of a Bowed String Instrument.” In Musical Performance.” In M. M. Wanderley and Proceedings of the International Computer Music M. Battier, eds. Trends in Gestural Control of Mu- Conference,pp.375–378. sic.Paris:InstitutdeRechercheetdeCoordination Sibson, R. 1981. “A Brief Description of Natural Neighbor Acoustique/Musique, pp. 231–258. Available online at Interpolation.” In Interpreting Multivariate Data. www.music.mcgill.ca/ mwanderley/Trends/Trends Hoboken, New Jersey: Wiley, pp. 21–36. ∼ in Gestural Control of Music/DOS/HunKir.pdf. Tanaka, A. 2010. “Mapping Out Instruments, Affordances, Accessed April 2014. and Mobiles.” In Proceedings of the International

18 Computer Music Journal Conference on New Interfaces for Musical Expression, interpretation is a hyperplane for a control space pp. 88–93. that is directly embedded in an m-dimensional sound Van Nort, D. 2009. “Instrumental Listening: Sonic Gesture space. The exactness of the mapping is bound to the as Design Principle.” Organised Sound 14(2):177–187. number of mapped points: If this is less than or equal Van Nort, D. 2010a. “Modular and Adaptive Control to n,theplanewillbeanexactmapping;ifgreater of Sound Processing.” PhD Dissertation, McGill than n,theplanewillpassbetweenthepresetsound University, Montreal, Quebec. Van Nort, D. 2010b. “Multidimensional Scratching, values. Sound Shaping and Triple Point.” Leonardo Music Journal 20:17–18. Metasurface: Natural Neighbors Van Nort, D., P. Oliveros, and J. Braasch. 2013. “Elec- tro/Acoustic Improvisation and Deeply Listening Ross Bencina (2005) presents a technique for map- Machines.” Journal of New Music Research 42(4):303– ping that was created for the AudioMulch software. 324. The approach is based on natural neighbor (NN) Van Nort, D., and M. Wanderley. 2006. “The LoM interpolation (Sibson 1981), and a point-wise map- Mapping Toolbox for Max/MSP/Jitter.” In Proceedings ping defines a mixture embedding. This technique of the International Computer Music Conference, creates “patches” that define a neighborhood as all pp. 397–400. space that is closer to a given state than to any other. Van Nort, D., and M. Wanderley. 2007. “Control Strategies for Navigation of Complex Sonic Spaces.” In Proceed- The implementation is constrained to two input ings of the International Conference on New Interfaces space dimensions. for Musical Expression,pp.379–382. Van Nort, D., M. Wanderley, and P. Depalle. 2004. “On the Colorblobs: Gaussian Kernels Choice of Mappings Based on Geometric Properties.” In Proceedings of the Conference on New Interfaces for We presented this technique in the Mapping Func- Musical Expression,pp.87–91. tions section. Wanderley, M., N. Schnell, and J. Rovan. 1998. “Escher: Modeling and Performing Composed Instruments in Real Time.” In Proceedings of the IEEE Interna- Cycling74’s Nodes tional Conference on Systems, Man, and Cybernetics, The nodes method is visually similar to Colorblobs pp. 1040–1044. Wessel, D., and M. Wright. 2002. “Problems and Prospects in that they both depict circular points of attraction for Intimate Musical Control of Computers.” Computer on a 2-D surface. Here, however, the influence for Music Journal 26(3):11–22. agivenstateisisolatedtotheareainsideagiven circle, within which the weighting for a given node is linearly and inversely proportional to the distance Appendix A between the input position and the center of the node. Therefore, in order to mix between a number In this section we compare and contrast existing of states, one needs to define nodes that overlap mapping strategies that take a functional approach. within a given area.

Overview of Existing Mapping Techniques Grid-Based Simplicial Interpolation This section presents an overview of existing Bowler and colleagues (1990) present a mapping functional mapping techniques from the literature f : Rn Rm that interpolates points spaced in a as we know it. grid. The→ presets in control space need to conform to a shape that is topologically equivalent to an MnM: Multiple Linear Regression n-dimensional lattice. The point-wise mapping Bevilaqua, Muller, and Schnell (2005) consider between control and sound spaces provides an mapping as a linear operator whose geometric exactness that is preserved by a mixture embedding

Van Nort et al. 19 that first partitions the grid into simplices, in order construct larger mapping control structures to express an input point in terms of barycentric more flexibly. They are available online at coordinates with respect to its containing simplex www.music.mcgill.ca/ doug/lom.zip. (Van Nort, Wanderley, and Depalle 2004). ∼ SI: Updated Simplicial Interpolation Scattered Simplicial Interpolation As discussed in the Mapping Functions section, we Building upon the simplex-based technique devel- have created an updated Max implementation of the oped by Bowler and co-workers, Goudeseune (2002) scattered simplicial interpolation technique. developed his own method of simplicial interpola- tion. Although also based on a point-wise mapping Multilinear between x Rn and y Rm,thisapproachallows the set of preset⊂ states⊂ within control space to be We have extended the aforementioned bilinear scattered by creating a triangulation of the points in interpolation into a multilinear mapping for higher- x,ratherthanfittingthemtoagrid.Thefactthat dimensional use. This is a mapping f : Rn Rm → this technique allows for scattered data points and which, for a given input state x (x1, ..., xn), may be can be edited locally (i.e., inserting new states only expressed as = changes the neighboring point-wise mapping) makes n it more flexible than Bowler et al.’s. f (x) ω 1 x xi (A.1) = i − j − j i H(x) j 1 ∈! "= # $ $% Bilinear Interpolation $ $ where H(x)isthesetofcontrolstatesofthe Both Rovan et al. (1997) and Wanderley, Schnell, hypercube H that contains x,whichweindexas and Rovan (1998) use a bilinear mapping to morph xi (xi , ..., xi ), and the ω R are the weights = 1 n { i}⊂ between additive-synthesis models aligned in a two- for each of these respective states. This expression dimensional grid structure. The bilinear mapping is used for a direct embedding into a higher- scheme makes sense in the context of these works, dimensional sound space, or for controlling a single because it is defined relative to a lattice structure, scalar value such as loudness within a high-level it is locally defined (by the four corners of given space, as with the aforementioned bilinear map area of the space), and it is both continuous as well into a space of additive sound models. We have as differentiable. This last point is important for built an extension to this technique in order to moving between additive models. It is not smooth, define mixture embeddings. The expression of however, in that it is not continuously differentiable Equation A.1 is modified to become at the boundaries between cells, which could be a n limiting factor for additive data that may require i i f (x) ωi y 1 xj x (A.2) smoother interpolation (McAulay and Quatieri = − − j i H(x) j 1 ∈! "= # $ $% 1986). The mapping geometry is hyperbolic, so a $ $ given mapping surface will vary greatly when close where the set of yi Rm are sound states that to these states, and little when moving far from are defined through{ } a⊂ point-wise map with the these states. respective xi . This mapping retains its hyperbolic nature, despite{ } being warped by virtue of its mixture embedding. Extensions and New Developments RST Building on the work by Van Nort and Wan- derley (2006), we have implemented the fol- This technique was described in the Mapping Func- lowing techniques in Max/MSP in order to tions section. We have implemented an RST based

20 Computer Music Journal Table A.1. Mapping Strategies and Certain Relevant Properties

Regularized Grid-Based Scattered Spline Simplicial Simplicial with MnM Natural Multilinear Interpolation Interpolation Tension Colorblobs Hyperplane Neighbor Nodes

Continuous !! !!!!!! Differentiable !!!!!! Locally Editable !! Parametric Smoothing ! Bounded Control !! !! Dimension Local versus Global L L L G G G G L Definition Ck k 0 k 0 k 0 k k k 1 k 0 k 1 = = = =∞ =∞ = = = Grid versus Scattered G G S S S S S S Exact versus E E E A A A E E Approximate Geometry HB PL PL NC GM L NC L Complexity 6 4 5 8 3 2 7 1 The geometric nature of each mapping varies from hyperbolic (HB) to piecewise linear (PL), a general nonlinear curve (NC), a Gaussian mixture (GM), and a simple linear (L) surface. on code from the Geographic Resources Analysis scattered SI and nodes methods are, however, the Support System (GRASS) for GIS modeling, for either only ones whose topologies may be edited locally. two or three control dimensions. The smoothness Only the RST technique allows for variable control and tension parameters can be dynamically tuned in of smoothness and degree of exactness, but it will real time, in order to adapt the surface geometry for never be purely exact per se. So it is considered as agivenmappingcontext. an approximation technique, as are the Colorblobs (see the discussion of the gravitational model) and MnM (assuming the number of points is greater Mapping Design Space: Comparisons and Strategies than control dimension n). The geometric nature of each mapping varies We now compare and contrast these mapping from hyperbolic (HB) to piecewise linear (PL), a strategies in order to move towards a design space general nonlinear curve (NC), a Gaussian mixture of multi-parametric, spatial mapping strategies— (GM), and a simple linear (L) surface. not to create an objective taxonomy, but to inform Finally, the computational complexity is rated, designers of potential tradeoffs that exist in choosing with the run-time computation requirement of the different techniques available. To that end, consider nodes and MnM matrix mappings being fastest Table A.1. Here we see that all strategies are (speed rank 1, 2) and the radial-basis computation continuous, and all but the simplicial techniques of the RST= method being the most costly (rank are differentiable. Further, most are not smooth: the 8). The computational complexity of RST is MnM is, in the sense that it is a simple hyperplane, actually= bound to the number of points rather than but this geometry is, in and of itself, limiting. The to the control or sound-space dimension, and so this Colorblobs and RST techniques are highly smooth. technique can actually be quite efficient. Most of the eight mapping techniques can be To conclude, a reflection that arises from our own defined by scattered control-to-sound states, except experience with these techniques: In light of these for the multilinear and grid SI techniques. The qualities, one can span the mapping design space

Van Nort et al. 21 quite well by choosing to focus on the multilinear, efficient implementations of nodes, CB, and MnM scattered SI scheme, and RST techniques. Given exist already in the Max/MSP environment and these three, one may define curves having rapid may be used in conjunction with these techniques. variation (multilinear), the ability to parametrically The NN technique, although smoother than SI, is control smoothing to a high degree (RST), and the not straightforward to implement in higher control possibility of working with scattered data that dimensions, and its other traits are covered well by can be edited locally and on the fly (SI). Further, the other three mappings.

22 Computer Music Journal Robert Tubb and Simon Dixon Centre for Digital Music AZoomableMappingofa Queen Mary University of London Mile End Road Musical Parameter Space London, UK E1 4NS Using Hilbert Curves r.h.tubb, s.e.dixon @qmul.ac.uk { }

Abstract: This article presents an interface for navigating a musical parameter space. The entire combinatorial space of a ten-parameter synthesizer is laid out as a two-dimensional surface on a multi-touch screen. The surface can be scrolled and zoomed using touch-screen swipe and pinch gestures, reminiscent of a map application. The user can place markers on the surface to flag favorites and to explore regions of different sizes around these points. The mapping from the two-dimensional surface to the higher-dimensional parameter space uses a space-filling curve. Hilbert curves constructed from Gray codes with long bit runs can be used to preserve locality as much as is possible, while still maintaining access to all possibilities. A user study was performed to compare this parameter mapping with a more traditional one-slider-per-parameter interface. Questionnaire responses indicate that different mapping strategies suit different stages of the creative process. The combination of the two interfaces was deemed more useful than either individually, reinforcing the notion that a combination of divergent and convergent processes is important for creative tasks.

Creativity is often characterized as a combination of (Xenakis 1971; Jones 1981). One of the earliest idea creation and idea selection. While this is a very exhibitions of computer art and music was entitled simplistic model, these two processes usually form “Cybernetic Serendipity” (Reichardt 1968). Gener- the core of more sophisticated multi-stage process ating large amounts of widely scattered data quickly models (Sawyer 2012, pp. 88 and 132). We shall refer is certainly not difficult, but computers are rather to these contrasting processes as “divergent” and bad at evaluating those data artistically: Listening “convergent,” after Guilford (1967). to candidate sounds becomes the bottleneck. To express this idea in terms of search strategies in Figure 1a shows one view of the creative process. parameter space, idea creation would be a scattered The artists do the creative thinking and arrive at exploration of a large neighborhood creating as the technology with a clear idea in mind. They then many candidate points as possible. Idea selection proceed to realize this idea within the computer would be evaluating and choosing the best of those via the interface. This conforms to a traditional candidates. This strategy resembles optimization view of the recording studio as a tool for recording metaheuristics that are used to search large, complex precomposed, prerehearsed material. The interface solution spaces, such as Monte Carlo methods, is therefore judged by how faithfully and painlessly particle swarms, and genetic algorithms. artists can realize their vision. Increasingly though, Musicians, particularly those working in the the computer studio is used to experiment and electronic domain, say that a large proportion of the generate ideas (Duignan, Noble, and Biddle 2010). raw material they use comes from unplanned, emer- Figure 1b illustrates an alternative view of the gent phenomena. Many claim that in the process creative process, where the computer can be used of making music, accidents will occur and sounds to produce many divergent ideas, and indeed diver- will be discovered that were never intended—but gences may occur at any point within the interaction prove invaluable. This would imply that some of loop of brain, interface, and software. In this picture the divergent process has been outsourced to the the technology is judged not on ease of brain-to- computer. Use of computers to induce “happy ac- computer input, but on how many good ideas emerge cidents” via stochastic processes has a long history from the hybrid human–computer system. Our main hypothesis is that standard computer music interfaces do not support multi-stage cre- Computer Music Journal, 38:3, pp. 23–33, Fall 2014 ativity as well as they could. The majority of inter- doi:10.1162/COMJ a 00254 actions consist of adjusting individual parameters, c ⃝ 2014 Massachusetts Institute of Technology. leading to a navigational “Etch-A-Sketch” style: one

Tubb and Dixon 23 Figure 1. Two views of the creative process: the “studio” view (a) and a view involving a feedback loop with technology (b).

l

l

l

(a) (b) dimension at a time. This means that moving to The presets can then be “morphed” by navigating anewpointalwaysinvolvestravellingatleastthe the subspace using a gestural controller. Given Manhattan distance (i.e., the distance calculated as a D-dimensional controller, and a P-dimensional the absolute difference of each coordinate, as when parameter space, D 1presetscanbeusedtoforma + navigating streets laid out as a grid), and requires D-dimensional subspace within RP . In-depth treat- discrete analytical decisions to switch parameters. ments of the geometry of these mappings have been This method of navigation would seem to be best investigated by Goudeseune (2002) and by Van Nort, suited to the convergent stage of the design process. Wanderley, and Depalle (2004). Applications that Divergent ideation, however, would seem to require have implemented these ideas include Syter (Allouis the ability to travel in any direction quickly, with and Bernier 1982), Bencina’s (2005) metasurface , varying levels of predictability. and the nodes object in Max/MSP. Researchers have tried to create dimensions that correspond to high-level perceptual descriptors of Prior Work the character of the sound, using techniques such as multi-dimensional scaling to create a “timbre space” Much research has been carried out in how to (Grey 1977; Wessel 1979; Arfib et al. 2002). This reduce the difficulty of navigating timbre parameter approach certainly fits many users’ expectations, but space. Indeed, almost all music technology must the nature of timbre is extremely hard to quantify address this problem in one way or another, as it is (Aucouturier and Pachet 2004). Useful dimensions impossibly laborious to specify an entire musical may vary widely between musical styles and signal by hand. different users. This timbre-space approach has The simplest way to make a space quickly navi- been used in zoomable interfaces that predate the gable is to save the coordinates of preferred points, one we describe in this article. Examples include these are referred to here as “presets.” Once a set of SoundExplorer by Yee-King (2011), using a timbre presets has been created, a lower-dimensional sub- similarity metric based on Mel-frequency cepstral space can be created from them, the simplest being coefficients and multi-dimensional scaling to create alinethatinterpolatesbetweentwopresetpoints. a 2-D zoomable timbre map, and the ISEE (Intuitive

24 Computer Music Journal Sound Editing Environment) by Vertegaal and Bonis Linearity — When a gesture such as scrolling (1994), where zooming into a region of timbre space occurs, it will have a certain effect on that would take the user further down a hierarchy of sound, more extreme versions of this gesture instrument categories. should produce more of the same effect. Most techniques throw away a large proportion of This property is hard to achieve with any the parameter space. Even with the best subspace- dimensionality reduction method, although finding algorithm, many interesting settings will smoothness implies some linearity in the become inaccessible. immediate neighborhood. In the mapping literature there tends to be a focus on the problem of controlling synthesis algorithms It is difficult to perfectly satisfy all of these via physical controllers. But for this study, rather requirements with any mapping method. We can than facilitating expressive musical performances, nonetheless experiment with mappings with differ- we apply mapping concepts to a different stage of the ent points of failure and ask if they can assist the creative process, namely, the initial idea generation creative process in different ways. Because little phase. Predictability is deemed less important here, work has been done using mappings that can access and access to the full space of sonic possibilities the entire combinatorial space, we shall use this more so. We are attempting to investigate, and space-filling property at the expense of linearity and speed up, the transitions between the exploration, smoothness, and see how users fare. evaluation, and refinement stages of creativity.

The Hilbert Curve Desirable Properties for Mappings Aspace-fillingcurveisacontinuousparameterized The following properties are desirable for controller function that maps a line segment to a continuous mappings: path in an N-dimensional unit hypercube. The curve can approach any point in the space arbitrarily Low dimensionality —Controldevicesoftenhave closely as the iteration parameter is increased. fewer parameters than synthesis engines. Given The curve is constructed recursively, and is self- the brain’s limited conscious multi-tasking similar. These mappings have proved to be useful abilities and working-memory capacity, simple in all kinds of applications such as clustering, controllers are preferable. data indexing, parallel computing, and even a Locality,ordistance preservation —Having computationally cheap solution to the travelling travelled a certain distance in control space, we salesman problem (Bartholdi and Platzman 1982), want that to be reflected in the distance travelled due to their properties of locality preservation. A in parameter space, and ideally perceptual 3-D color space can be distributed onto a 2-D chart distance too. (Jaffer 2005), a good example of dimension reduction Revisitability —Ifwereturntothesamepoint, increasing usability. Hilbert curves in particular we wish it to sound the same. The location of have good locality (Gotsman and Lindenbaum 1996). preset points should be stable. This property relates very well to the zooming idea: Continuity —Ifapointisadjacenttoanother The further the user zooms in, the smaller the point on the lower-dimensional surface, both accessible sonic neighborhood will become. should be adjacent in the higher-dimensional Figure 2 shows four iterations of a Hilbert curve space. in R2.Initsfirstiterationthecurvesimplyvisits Smoothness —Continuoushigherderivativesare each corner of the unit square. For the next iteration, desirable to eliminate sudden changes in direc- smaller squares are formed at these four corners, and tion. This has relevance to the predictability of the corners of each of these are visited in a similar a control. manner, but the bottom left and right sub-squares

Tubb and Dixon 25 Figure 2. Four iterations of Figure 3. The Hamiltonian Figure 4. Five-digit binary which all the digits Hilbert curve construction path visiting the eight codes: (a) standard binary transition either six or in two dimensions. Arrow corners of a cube. The numbers; (b) Gray code, in seven times; (d) “High a demonstrates a locality sequence of coordinates is which a single bit flips MRL” Gray code, in which violation: a small aGraycode(right). from row to row; (c) the minimum distance high-dimension movement “Balanced” Gray code, in between flips is four steps. results in a large low-dimension distance.

Figure 3

l l

Figure 2

are rotated to ensure the continuity of the line. The process is iterated until the plane is filled to some desired resolution. There are always upper bounds for distance in RN given a certain distance in R1.Unfortunately,distancepreservationinthe opposite direction can be worse, as can be seen if the x direction is traversed at the bottom of Figure 2d. In (a) (b) (c) (d) the application this corresponds to moving a slider asmallamount,butyetjumpingtoadistantpoint Figure 4 on the zoom surface. For a higher-dimensional curve, the basic unit is an N-dimensional hypercube. Each hypercube, at For the purposes of parameter mapping, there is each level of iteration, must have all of its corners still a problem with this Gray code: The rightmost visited exactly once, and the path must only run bit flips 16 times whereas the leftmost only flips along the edges of the cube. Such paths are called once. This would be rather frustrating for the user Hamiltonian paths, and if the corners of the cube are as one parameter will flip very fast and another will labelled with binary coordinates the sequence forms hardly ever change. We wish to achieve some kind of aGraycode(seeFigure3).Graycodesarebinary smoothness by distributing these transitions more numeral systems where only one digit changes at evenly. atime(Gray1953).Figure4ashowsthebinary Gray codes with two particular properties can numerals for five digits, and Figure 4b shows the mitigate this issue. The first is a “balanced” Gray standard Gray code. code (Bhat et al. 1996). Here the transitions are

26 Computer Music Journal Figure 5. Sonic Zoom app screenshot showing both interfaces. A preset is being hovered over in the center of the screen. For the timed session, the sliders or the Zoomer are hidden. spread as evenly as possible between dimensions (see Figure 4c). The second property is “minimum run length” (MRL, cf. Goddyn et al. 2003). If the MRL is maximized, the bit swaps never occur within acertaindistanceofoneanother(seeFigure4d). This is also desirable: When one parameter goes high we would not want it to immediately flip low again. Beyond three dimensions there are rapidly in- creasing numbers of alternative Gray codes and Hilbert curves. It is beyond the scope of this article to investigate these in detail, as their construction seems to follow no simple method. For practical purposes we assume that the long MRL code shown in Figure 4d will be indistinguishable from better balanced codes.

The Algorithm

The 2-D surface is displayed as a grid, and the ten individual parameters as 1-D sliders (see Figure 5 later in this article). Consideration of the 2-D curves in Figure 2 may help to imagine the result. In this case, the path would be traversed by a single 1-D control, and cause two parameters to change, altering each according to the horizontal and vertical positions the path visits. In the case of Sonic Zoom, each dimension of the zoomable where the individual base 2P digits are calculated as surface uses a separate 5-D Hilbert curve: Moving in follows the x direction will change the first five sliders, and moving in the y direction will change the other five. x P an mod 2 (2) When fully zoomed in, the smallest subdivisions of = 2Pn the grid become visible. These correspond to sliders ! " Each of the a are then converted to P-digit binary changing by a single unit. Zooming out fades these n numbers (b , ..., b , b )usingtheGraycode low-level grid lines, revealing lines corresponding n,P 1 n,1 n,0 G(), via a look-up− table such as the code in Figure 4d. to the entry and exit points of larger and larger hypercubes. When zoomed out fully, the grid lines b G(a )(3) correspond to hypercubes of 64 units. n,p = n The algorithm to convert an x or y coordinate to The parameter control values cp can be then built up PN-bit parameters, giving 2P parameters for each by treating the N different scales as standard binary point on the surface, is as follows: digits. Each coordinate is first expressed as a base-2P N-tuple N 1 − c b 2n (4) p = n,p a (aN 1, ..., a1, ..., a0)(1) n 0 = − #=

Tubb and Dixon 27 The number of points along one coordinate necessary The Lock Sequencer and Lock Synth buttons for full resolution is (2P )N.Inourimplementation constrain the surface to move in only the x or y we require ten 7-bit MIDI control parameters (five direction, respectively. In this case only the five per axis), so P 5and0 x < 327. sliders exclusively relating to the sequencer or So far we have= yet to≤ ensure correct rotation of synthesizer will change. Once a preset is saved, the sub-cubes such that a true Hilbert curve can be it appears as a colored dot on the surface. If the constructed from a long bit-run Gray code, as the listen point moves near to a preset, it will snap standard method assumes certain symmetry prop- to the preset coordinates. Without snapping it is erties of the original reflected Gray code (Hamilton impossible to line the preset up precisely. and Rau-Chaplin 2008). Therefore the jump at tran- sitions between sub-cubes is sometimes audible; this will be addressed in future work. Interface Evaluation Experiment

The experiment consisted of three 5-min intervals, where the sliders, the Zoomer, and a combination of Implementation both were presented. The order was randomized for each session. All user interactions were recorded and Ascrollable,zoomable2-Dsurfaceiswellsuitedto sent over a Wi-Fi Internet connection to a logging implementation on a multi-touch screen. The Sonic server. Because many concurrent users were antic- Zoom app is an implementation of the described ipated for the public release, the event data were mapping for the Apple iPad. Although we envisaged thinned to 15 Hz before being sent, but care was this mapping being used for timbre exploration, it taken to preserve the start and end points of each was initially felt that a single tone would become scroll, zoom, or slider change. Although assigning fatiguing for users. So five of the interface’s ten specific search tasks may make for a more controlled parameters were used for a simple melodic pattern experiment, this behavior is only relevant for “con- generator. This creates a 16-step sequence based vergent” interaction modes. Therefore, users were on five sine waves of integer frequencies; this instructed to explore the space however they wished, “Frequencer” is detailed in earlier work (Tubb 2012). but to intentionally search for sounds they liked and The other five sliders control a simple synthesizer save them as presets. At the end of the session, users based on frequency modulation and subtractive were presented with a short questionnaire. synthesis. Figure 5 shows the interface. The Likert-style questions (questions with re- For the “standard” interface, ten sliders (or faders) sponses on a discrete scale, usually from “strongly were used. These appeared as an overlay on the left agree” to “strongly disagree”) on the questionnaire of the screen. When both the sliders and the Zoomer fell into two categories. Statements that partici- are on screen together, movements in one space pants rated on a 5-point agreement scale are shown are immediately reflected in the other. The “listen in Table 1. Questions regarding which interface point” location is represented as a cross-hair in was best for a certain task, with a 5-point scale for the middle of the screen. The absolute locations of interface preference, are shown in Table 2. touch points have no bearing on the sound. A single For the pilot study, twenty PhD students partic- finger drags the surface, changing the coordinates ipated. Most were familiar with computer music, of the listen point and, hence, the positions of the but unaware of the focus of the experiment. sliders. A two finger pinch-out gesture will zoom into an area around the listen point, while keeping the listen point stationary. As the user zooms, Results smaller subdivisions of the grid become visible. The subdivisions are colored according to the iteration In Figure 6 the strongest agreement was for state- level of their Hilbert curves. ments AD 4 and AD 7. This indicates that spatial

28 Computer Music Journal Table 1. Questions Requiring a 5-Point quickly” and “generating new ideas” (questions SZ 9 Agree-or-Disagree Answer and SZ 13), related to divergent thinking, were slightly in favor of the Zoomer. The question about AD 1 I am familiar with music software and sound which felt more creative (SZ 12) narrowly came synthesis. down in favor of the Zoomer, but with very large AD 2 The correspondence between the sliders and the grid was understandable. variance. Questions SZ8 and SZ15 generated neutral AD 3 Scrolling greater distances on the grid made results. larger differences in the sound. Surprising neutral results include the fact that AD 4 The ability to see other presets laid out on the performing live (question SZ 14) came out as evenly grid was useful. balanced between the two interfaces. One would AD 5 The sounds were too limited or of too poor expect the more predictable sliders to be preferred. quality to be able to judge the interface. The understandability of the mapping (AD 2) was AD 6 The Zoomer was an improvement on just using a neutral, despite the expectation that this statement randomizer. would be strongly disagreed with, due to the complex AD 7 The combination of Zoomer and Sliders was nature of the Hilbert curve. better than either individually.

Table 2. Questions Requiring a 5-Point Navigational Data Analysis Sliders-versus-Zoomer Answer

SZ 8 The best interface to get a feel for the Analysis of users’ navigational data is in an initial possibilities of the synthesizer was... phase; the release to the public will enable a much SZ 9 The best interface for discovering interesting larger data set to be analyzed. Some trends have sounds quickly was... already emerged, however. Figure 7 shows a plot SZ 10 The best interface for fine tuning a sound was... of the total time spent at various zoom scales for SZ 11 The interface that I felt more in control using all users. The scale unit corresponds to the seven was... iteration scales of the Hilbert curve (with higher SZ 12 The interface that felt more creative was... values meaning the user had zoomed out further). SZ 13 The interface better for generating new ideas There is an obvious peak at the larger scales: was... The smaller the scale, the less time people spent SZ 14 The interface better for performing live would exploring it. Levels 1, 2, and 3 are largely avoided, be... SZ 15 Overall, the interface I preferred using was... presumably because the sound does not change rapidly enough. An interesting question is whether the clear peak visualization of preset points on the surface is a good at large scales corresponds to a real optimum scale feature and that people definitely felt that the com- for divergent searches. If this is so, the results seem bination of the two navigation methods was more surprising in that the preferred scale is smaller powerful that either individually. The improvement (about Level 4) for the combination of the two over a randomizer (AD 6) was also a good result interfaces than for the Zoomer only (Level 6). for the Zoomer. Most people got an impression of Figure 8 shows the distribution of distances the distance preservation property (AD 3), though travelled between listen points. Any point that was this was not conclusive, and two users disagreed hovered over for more than 0.2 seconds is assumed strongly. to have been assessed, at least for timbre, and is SZ 10 and SZ 11 were questions specifically categorized as a “evaluation point.” A histogram was relating to convergent processes. Unsurprisingly, constructed for the distances between evaluation people felt much more in control using the sliders, points. As one might expect, this distance is related and would prefer them to fine tune a sound. to the time spent at the scale factor, and the two The statements about “finding interesting sounds curves show a more or less similar distribution.

Tubb and Dixon 29 Figure 6. Questionnaire Figure 7. Plot of the time Figure 8. Frequency of responses to Likert items spent at different zoom journeys between (left) and interface prefer- scales, totaled across all evaluation points by ences (right). The median is participants. log 2-D distance. marked by a dotted circle, thick bars indicate the first quartile, and thin bars indicate the third quartile.

Strongly agree Definitely Zoomer

Maybe Agree Zoomer

Neither/ Neutral both equal

Disagree Maybe sliders

Definitely Strongly disagree sliders AD1: Knowledge AD3: Distance AD4: See presets AD5: Limited AD6: > Random AD7: Combination SZ8: Possibilities SZ9: Speed SZ10: Fine tune SZ11: Control SZ12: Creative SZ13: New ideas SZ14: Live SZ15: Overall AD2: Understand

Figure 6

30

Both interfaces Zoomer only 㼘 25

20

15

Total time spent (minutes) 10

5

0 1 2 3 4 5 6 7 Zoom level

Figure 7 Figure 8

30 Computer Music Journal Table 3. Total Number of Presets Saved for the parameters are added. People also felt it took too Three Interface Views, and by Timed Stage long to zoom between the highest and lowest scales. Many users suggested smoother transitions, both Interface Sliders Both Zoomer Stage 1 Stage 2 Stage 3 in a general sense, and between specific presets. Totalsaves 64 61 47 66 56 50 Future implementations will make it possible to interpolate between various levels of complexity: Detail will only reveal itself as one zooms in. Again, the sharp drop off below a distance of 324 There were frequent requests for an easier way implies that fine resolutions can be discarded. This to return to where one had visited before, but had would, however, abandon the one-to-one mapping not saved as a preset. It is too easy to accidentally between the sliders and the surface, and render overshoot something that caught one’s ear. Two presets obtained with the sliders inaccessible and users came up with the intriguing suggestion of a invisible on the Zoomer. visual, scrubbable undo trail for these situations. The other quantity of interest was the rate of Session histories can promote new ways of thinking preset discovery. The save rate could be taken by providing an overview of one’s own creative as a measure of how effective an interface is for process (Shneiderman 2000). discovery. The total number of presets saved in each Afeaturelessgridwasnotenoughtomakeuseof different interface mode are shown in Table 3. visual spatial memory. Color-coded lines were not The Zoomer on its own seems to fare worse sufficient for people to instantly grasp what scale by this measure. Despite the positive results from they were at, though this can be learned with use. the questionnaire, the actual sound discovery rate One user suggested using audio feature-extraction seems highest with the sliders. It could be that techniques to create a texture that would convey people were too busy exploring the interface itself the nature of the sounds underneath. Alternatively, to fully concentrate on sound design: The Zoomer patterns or shapes could be associated with presets is far less familiar than the sliders. The rate of and then morphed, as suggested by Van Wijk and preset saving actually went down slightly over the Van Overveld (2004). Real geographical data could duration of the experiment, however, suggesting be displayed from a database such as OpenStreetMap that unfamiliarity does not have a negative effect (Haklay and Weber 2008). on the save rate. The other possibility is that, given Most users expressed some confusion over the more different options, people’s expectations rise sequencer parameters, and another suggested that and they become more selective about what they asetlistofMIDIsequencesofvariousgenres save. should be provided, and the surface left just for An average of 15 sec elapsed between users timbre exploration. One user was most interested switching interfaces when both were on screen. in the sequencer, however, and spent some time Although not a rapid process, it indicates that both constructing a selection of points that enabled navigational methods were alternated many times sequences to be navigated to generate a chord before new sounds were located. sequence.

User Comments and Suggestions Conclusion Some users felt that it was very hard to find the ideal scale at which to scroll around. To feel one The study presented in this article indicates that is “making progress” a large scale is needed, but different ways of navigating a parameter space are the sounds change too suddenly and unpredictably. suited to different stages of the creative process. The On the other hand, at smaller scales it is easy to Zoomer was preferred for divergent exploration, the get lost in a sea of minor variations. This dilemma sliders were preferred for convergent fine tuning. can be expected to get considerably worse as more Users responded very positively to the assertion

Tubb and Dixon 31 that the combination of the two interfaces was References better than either individually. This could mean that being able to alternate navigation styles is Allouis, J., and J. Bernier. 1982. “The SYTER Project: valuable, although, preset save rates do not support Sound Processor Design and Software Overview.” In this assertion. Proceedings of the International Computer Music While no single mapping can be perfect, the Conference,pp.232–240. Hilbert curve is far from an ideal instrumental Amabile, T. M. 1996. Creativity and Innovation in mapping due to its lack of smoothness and linearity. Organizations.Boston:HarvardBusinessSchool. Arfib, D., et al. 2002. “Strategies of Mapping between It can, however, be very useful in cases where a Gesture Data and Synthesis Model Parameters Using representation of the complete parameter space Perceptual Spaces.” Organised Sound 71(2):127–144. is desired, and was shown to be preferred to a Aucouturier, J.-J., and F. Pachet. 2004. “Improving Timbre randomizer. Similarity: How High Is the Sky?” Journal of Negative The fact that people tended to use very high Results in Speech and Audio Sciences 1(1):1–13. zoom levels (i.e., they zoomed out) indicates that an Bartholdi, J., and L. Platzman. 1982. “An O(N log N)Planar overview of the entire space is a useful feature, but Travelling Salesman Heuristic Based on Spacefilling access to every level of detail is not. Given that users Curves.” Operations Research Letters 1(4):121–125. prefer to fine tune with the sliders, we can eliminate Bencina, R. 2005. “The Metasurface: Applying Natural some of the parameter space with no noticeable ill Neighbour Interpolation to Two-to-Many Mapping.” effect. Further testing should be done to establish In Proceedings of the International Conference on New Interfaces for Musical Expression,pp.101– what users feel is the ideal scale, but certainly the 104. lowest two levels seem disposable, leading to at least Bhat, G. S., et al. 1996. “Balanced Gray Codes.” The a millionfold reduction in the area of the surface. Electronic Journal of Combinatorics 3(1):25. Evaluating creativity is a hard task, and this Duignan, M., J. Noble, and R. Biddle. 2010. “Abstraction study has not addressed many issues, for instance, and Activity in Computer-Mediated Music Produc- the quality of the saved points. Further work tion.” Computer Music Journal 34(4):22–33. could utilize a “consensual assessment technique” Goddyn, L., et al. 2003. “Binary Gray Codes with Long (Amabile 1996), by enabling other users’ presets Bit Runs.” The Electronic Journal of Combinatorics to appear on the grid and to be rated. Once the 10(27):1. application is made available to the general public, Gotsman, C., and M. Lindenbaum. 1996. “On the Metric analysis may reveal clearer statistical relationships, Properties of Discrete Space-Filling Curves.” IEEE Transactions on Image Processing 5(5):794–797. such as how experts perform compared with non- Goudeseune, C. 2002. “Interpolated Mappings for Musical experts and whether distinctive behavior occurs Instruments.” Organised Sound 7(2):85–96. immediately prior to saving a preset. Gray, F. 1953. “Pulse Code Communication.” US Patent Many electronic musicians rely heavily on 2,632,058, filed 13 November 1947; and granted 17 technology as an extension of their artistic thought March 1953. processes. If creativity does indeed involve rapid Grey, J. M. 1977. “Multidimensional Perceptual Scaling alternation between idea creation and idea selection, of Musical Timbres.” Journal of the Acoustical Society systems should be designed with this fact in mind. of America 61(5):1270–1277. Divergent or convergent features on their own may Guilford, J. P. 1967. The Nature of Human Intelligence. be less effective than a well integrated combination New York: McGraw-Hill. of the two. Haklay, M., and P. Weber. 2008. “Openstreetmap: User- generated Street Maps.” IEEE Pervasive Computing AvideoofSonicZoominusecanbefoundonline 7(4):12–18. at http://youtube.com/watch?v=11AAinjrTbI, and Hamilton, C. H., and A. Rau-Chaplin. 2008. “Compact the app is distributed through the Apple App Hilbert Indices: Space-Filling Curves for Domains with Store (https://itunes.apple.com/us/app/sonic-zoom/ Unequal Side Lengths.” Information Processing Letters id643750094). 105(5):155–163.

32 Computer Music Journal Jaffer, A. 2005. “Color-Space Dimension Reduction.” Geometric Properties.” In Proceedings of the Con- Available online at http//people.csail.mit.edu/jaffer/ ference on New Interfaces for Musical Expression, Color/CSDR. Accessed 12 February 2014. pp. 87–91. Jones, K. 1981. “Compositional Applications of Stochastic Van Wijk, J. J., and C. Van Overveld. 2004. “Preset Based Processes.” Computer Music Journal 5(2):45–61. Interaction with High Dimensional Parameter Spaces.” Reichardt, J., ed. 1968. Cybernetic Serendipity: The In F. H. Post, G. Nielson, and G.-P. Bonneau, eds. Data Computer and the Arts.London:StudioInternational. Visualisation: The State of the Art.Berlin:Springer,pp. Sawyer, R. K. 2012. Explaining Creativity: The Science 391–406. of Human Innovation.NewYork:OxfordUniversity Vertegaal, R., and E. Bonis. 1994. “ISEE: An Intuitive Press. Sound Editing Environment.” Computer Music Journal Shneiderman, B. 2000. “Creating Creativity: User Inter- 18(2):21–29. faces for Supporting Innovation.” ACM Transactions Wessel, D. L. 1979. “Timbre Space as a Musical Con- on Computer-Human Interaction 7(1):114–138. trol Structure.” Computer Music Journal 3(2):45–52. Tubb, R. 2012. “Frequency Domain Patterns for Live Reprinted in C. Roads and J. Strawn, eds. 1985. Founda- Improvisatory Performance.” In Proceedings of the tions of Computer Music. Cambridge, Massachusetts: International Conference on Live Interfaces.Avail- MIT Press, pp. 640–657. able online at lipam.lurk.org/index.php?conference= Xenakis, I. 1971. Formalized Music: Thought and Mathe- LIPAM&schedConf=lipam2012&page=paper&op= matics in Composition.Bloomington:IndianaUniver- view&path%5B%5D=94&path%5B%5D=10. sity Press. Accessed 12 February 2014. Yee-King, M. J. 2011. “Automatic Sound Synthesizer Van Nort, D., M. M. Wanderley, and P. Depalle. Programming: Techniques and Applications.” PhD 2004. “On the Choice of Mappings Based on Dissertation, University of Sussex.

Tubb and Dixon 33 † Baptiste Caramiaux,∗ Jules Franc¸oise, Mapping Through Listening Norbert Schnell,† and Fred´ eric´ Bevilacqua† ∗Department of Computing Goldsmiths College University of London New Cross London, SE16 6NW, UK [email protected] †STMS Lab IRCAM-CNRS-UPMC Institut de Recherche et Coordination Acoustique/Musique 1placeIgorStravinsky 75004 Paris, France jules.francoise, norbert.schnell, frederic.bevilacqua{ @ircam.fr }

Abstract: Gesture-to-sound mapping is generally defined as the association between gestural and sound parameters. This article describes an approach that brings forward the perception–action loop as a fundamental design principle for gesture–sound mapping in digital music instrument. Our approach considers the processes of listening as the foundation—and the first step—in the design of action–sound relationships. In this design process, the relationship between action and sound is derived from actions that can be perceived in the sound. Building on previous work on listening modes and gestural descriptions, we propose to distinguish between three mapping strategies: instantaneous, temporal, and metaphorical. Our approach makes use of machine-learning techniques for building prototypes, from digital music instruments to interactive installations. Four different examples of scenarios and prototypes are described and discussed.

In digital musical instruments, gestural inputs action (Hunt and Kirk 2000; Wanderley and Orio obtained from motion-sensing systems, image 2002). analysis, or sound analysis are commonly used to It has often been discussed that for digital music control or to interact with sound processing or instruments, unlike most acoustic instruments sound synthesis (Miranda and Wanderley 2006). (Cadoz 1988; Wanderley and Depalle 2004), there is This has led artists, technologists, and scientists to no direct coupling between the gesture energy and investigate strategies for mapping between gestural the acoustic energy. More precisely, as the mapping inputs and output sound processes. is programmed in the digital realm, the relationship Considered as an important vector of expres- between the input and output digital data streams sion in computer music performance (Rovan et al. can be set arbitrarily. This offers unprecedented 1997), the exploration of mapping approaches has opportunities to create various types of mapping led to a flourishing of research work dealing with: that can be seen as part of the creative endeavor to taxonomy (Wanderley 2002); the study of various build novel digital instruments. strategies based, for example, on perceptual spaces After several years of experimentation, we (Arfib et al. 2002), mathematical formalization (Van have developed an approach that brings back the Nort, Wanderley, and Depalle 2004), or dynamical perception–action loop as a fundamental design systems (Momeni and Henry 2006); and evaluation principle. As a complement to approaches that focus procedures based on user studies and other tools on building active haptic feedback to enhance the borrowed from the field of human–computer inter- action–perception loop (Castagne et al. 2004), we propose a methodology rooted in the concept of embodied music cognition. This methodology con- siders listening as a process from which gestures and Computer Music Journal, 38:3, pp. 34–48, Fall 2014 doi:10.1162/COMJ a 00255 interactions, defining key elements for the design of c ⃝ 2014 Massachusetts Institute of Technology. mappings, emerge.

34 Computer Music Journal Our approach is anchored in advances in cognitive tions that make use of machine-learning techniques sciences and rooted in embodied cognition (Varela, to implement these scenarios. Thompson, and Rosch 1991). The enactive point The article is structured as follows. In the follow- of view on perception and the idea of embodied ing section, we review previous work characterizing cognition cover aspects of cognition as shaped different listening modes and how they relate to by the body, which consitute the perceptual and gestural descriptions of sounds. We then describe motor systems (Varela, Thompson, and Rosch 1991; our approach for the design of mappings inspired Noe2005).Fromthispointofview,theaction¨ by these different modes of listening. The proposed of listening—as is the case with perception in mappings are presented as real-world applications general—is intrinsically linked to the process of and stem from our past and current research in this acquiring knowledge and applying this knowledge field. In the final section, we discuss the different when interacting with our environment (Merleau- scenarios and mapping strategies. Ponty 1945). In music-making—as well as in speech and many other everyday activities—listening plays aparticularroleintheidentification,evaluation, Describing Sound Gesturally and execution of actions. The intrinsic relationship between action and listening in human cognition As mentioned previously, we are interested in has been confirmed by many studies (Liberman and examining mapping strategies through the theory of Mattingly 1985; Fadiga et al. 2002; Zatorre, Chen, embodied music cognition. In particular, we focus and Penhune 2007). By extension, embodied music on listening processes that might induce gestural cognition, developed by Marc Leman (2007) and Rolf representations in order to conceptually invert the Inge Godøy (2006), tends to see music perception as process, going from gesture to sound, to create based on actions. Many situations involve people the mapping. In this section we first review work moving while listening to music. In the framework describing different listening modes that can be of embodied music cognition, these movements are related to specific sound properties. Then we show seen as conveying information about the perceived that these listening modes can be related to different sonic moving forms (Leman et al. 2009). action strategies. Although embodied music cognition provides us with a theoretical framework for the study of listening in a musical context and for the study of the link between music perception and human actions, Listening Modes digital music performance requires computational tools to implement experimental breakthroughs. Sound, as considered here, refers to recorded audio Recent tools coming from the field of machine material. Recorded sound can be played back learning research allow for building scenarios and and processed using various techniques, which, prototypes implementing concepts borrowed from importantly, leads to different listening experiences. embodied music cognition. Such scenarios are, A vast body of work is devoted to the mechanisms of indeed, usually best defined from high-level gesture listening, gathering together various research fields and acoustic descriptions, which cannot generally such as psychoacoustics, neurosciences, auditory be easily programmed with other techniques. For scene analysis, and musicology. In this section, we example, the use of machine-learning techniques focus on conceptual approaches of listening that allows one to set the gesture–sound relationships principally originated from music theory and the from examples or from a database. theory of ecological perception. Our goal is to create In this article we propose a new approach of acomprehensiveoverviewoflisteningmodesand gesture-to-sound mapping that relies on the concept their functions, which will eventually be linked, in of embodied sound cognition, and we report applica- the next section, to gestural representations.

Caramiaux et al. 35 First, in the context of musique concrete` ,Pierre John Cage that aim at hearing the everyday world as Schaeffer (1966) defined four functions of listening. music. (Note that the translation of Schaeffer’s terms is far Recent studies have proposed to enrich these from trivial. For this reason, in this article we use previous taxonomies by adding an emotional di- both our translation and the original French term.) mension, evoked by the auditory stimulus. David These functions are: (1) listening (ecouter´ ), which Huron (2002) proposed an analytic framework sup- focuses on the indexical value of the sound (i.e., porting the idea that emotional experiences may the sound source); (2) perceiving (ouır¨ ), the most be usefully characterized according to a six-part primitive mode, consisting of receiving the sound classification, categorized as follows: (1) reflexive, through the auditory system; (3) hearing (entendre), referring to fast, automatic physiological responses; referring to the selective process between auditory (2) denotative, allowing the listener to identify signals, the attention to inherent characteristics sound sources; (3) connotative,allowingthelistener of the perceived sound; and (4) comprehending to infer various physical properties about sound (comprendre), which brings semantics into sounds, sources such as size, proximity, energy, material, treating them as signs. These different functions and mode of excitation; (4) associative, referring of listening are not mutually exclusive and operate to arbitrary learned associations; (5) empathetic, competitively. referring to auditory empathy that allows the lis- Based on Schaeffer’s theoretical taxonomy, and tener to detect emotion from the sound (such as motivated by new concepts from auditory display, fear in a voice) coming from an animate agent Michel Chion (1983) proposed a taxonomy com- (be it human or animal); and, finally, (6) critical, prising three categories, called modes of listening: referring to conscious cognitive processes by which (1) causal listening, consisting of listening to a the intentions of a sound-producing agent are sound in order to gather information about its evaluated. cause (or source); (2) semantic listening, refering Recently, Kai Tuuri and colleagues proposed to a code or a language to interpret a message; an extended taxonomy of listening modes (Tuuri and (3) reduced listening, focusing on the quali- and Eerola 2012). The taxonomy is hierarchical ties of the sound itself, independent of its cause with three levels: experiential, denotative, and and of its meaning. (Note that reduced listening reflective. The experiential level encompasses is a concept that was first introduced by Schaeffer Huron’s reflexive and connotative modes. The to motivate the concept of the “sound object” in connotative mode more precisely focuses on the musique concrete` .) Hence, Chion does not consider relation between the action and the external world the low-level aspect of perception called perceiving (i.e., object, people, and cultural context). In this (ouır¨ ). taxonomy the experiential mode also induces a Modes of listening have also been of interest in kinaesthetic mode that refers to the inherent the ecological approach to auditory perception. One movement qualities in the sound (for example, important application has been the design of sounds characterizing a sound as “wavy”). The second in human–computer interaction. In this context, level in the hierarchy is the denotative mode. This William Gaver (1993a, b) considered environmental mode was first defined by Huron and extended by sounds and proposed a differentiation between two Tuuri in order to separate between modes focusing types of listening defined as everyday listening, on sound sources and those focusing on sound in which the perception focuses on events rather contexts. Finally, the top level is the reflective than sounds, and musical listening, in which mode, encompassing Chion’s reduced mode as well perception is centered on the sound characteristics. as Huron’s critical mode. As noted by Gaver (1993b, p. 1), musical listening to The important point here is to realize that several environmental sounds can be achieved by listening of the listening modes make reference, explicitly “to the world as we do music.” Gaver used, as or implicitly, to motor imagery or action. Both examples, compositions by the American composer Chion’s causal listening mode and the denotative

36 Computer Music Journal Figure 1. A simplified related to acoustic taxonomy of listening qualities of the sound. modes. Causal listening Semantic listening refers to an explicit integrates higher level association between sound notions of meaning and and its producing action. interpretation. Acoustic listening is

Causal Acoustic Semantic listening listening listening

Listening (opposed to hearing, Hearing Comprehending comprehending, perceiving) (Schaeffer 1966) (Schaeffer 1966) (Schaeffer 1966) Reduced listening Semantic listening Causal listening (Schaeffer 1966; Chion 1983) (Chion 1983) (Chion 1983) Musical listening Associative Everyday listening (Gaver 1993) (Huron 2002) (Gaver 1993) Connotative Denotative (functional, semantic) Denotative (Huron 2002) (Tuuri and Eerola 2012) (Huron 2002) Reduced listening Denotative (causal) (Tuuri and Eerola 2012) Connotative ( Kinaesthetic listening (Tuuri and Eerola 2012)

listening mode, used by both Huron and Tuuri, refer we will consider in this article, with the goal of to associating a sound to the action that created associating them with gestural representations. the sound. Such actions are generally linked to clear interactions and motion between objects (e.g., astickhittingacymbal).Wewillkeeptheterm Linking Gesture and Listening causal listening throughout this article to denote such an association between the action and the In the previous section we reviewed the listening sound. modes as introduced by various authors in the The reduced listening mode of Schaeffer and literature. These were summarized as an approach Chion, Huron’s connotative mode, and Tuuri’s using three modes, accounting for causal, acoustic, kinaesthetic mode refer to acoustic properties of and semantic listening. In this section, we posit the sound. We will use the term acoustic listening that these modes of listening can be linked to throughout this article for such a type listening. specific gestural strategies. We base this statement These acoustic aspects could be quantified using on a review of important work within the field of asetofsounddescriptorsfromthesoundsignal. behavioral approaches in embodied music cognition Acrucialpoint,however,istoacknowledgethat that reported on gestural sound description. defining the reduced listening mode is also linked Interactions between sound perception and to sound descriptions such as the Schaeffer’s typo- motion have been studied either through a neuro- morphology of sonic objects (Schaeffer 1966), or scientific perspective or a behavioral perspective later to temporal semiotic units (unites´ semiotiques´ (Zatorre, Chen, and Penhune 2007). Generally, the temporelles,cf.Freyetal.2009).Aselucidatedby motor–auditory interaction has been recognized as Godøy (2006), these descriptions can, in many cases, important for describing sound perception. Neuro- be linked to notions of motions and actions. science studies have shown how listeners activate The last mode of listening encompasses se- cognitive action representations while listening to mantic aspects of sound perception and is named music performances, whether they are expert musi- accordingly. Figure 1 summarizes the three modes cians or novices (Haueisen and Knosche¨ 2001; Lahav of listening—causal, acoustic, and semantic—that et al. 2005; Zatorre, Chen, and Penhune 2007).

Caramiaux et al. 37 In a behavioral approach, a common experimental mentally how participants can associate different methodology consists of asking participants to types of motion in the acoustic and causal listening perform movements along with music while it modes. We observed two related strategies: mimick- is played. The movement analysis can reveal ing the action related to the sound source (causal important insights into the underlying embodied listening mode) or tracing the shape of the sound cognitive processes related to music perception. A parameter (acoustic listening mode). In particular, wide range of work concerns controlled tasking, for we showed that the identification of sound sources instance, the task of tapping on beats (Large 2000; (i.e., the mode of listening) has a direct consequence Large and Palmer 2002). on the gestural strategies. If the participants can In systematic musicology, exploratory procedure identify the sound source as an action, they tend to is more commonly used. Examples include asking mimic the action. On the other hand, a sound that participants to spontaneously gesticulate while cannot be identified leads participants to trace the listening to a sound stimulus or music. For instance, profile of perceived sound features. Leman and co-workers (2009) studied participants’ This experimental study showed a link between movements made with a joystick while listening to acoustic (or causal) listening modes and analog (or a performance of guqin music. Also, Mats Kussner¨ mimicking) motion strategies. This study provides (2013) considered free tracing movements on a arationaleforestablishingmappingstrategiesbased tablet while two of Fred´ eric´ Chopin’s preludes were on listening modes and associated motion strategies. played. Other works concern specifically designed Mapping strategies can stem from the reviewed stimuli with well characterized musical parameters. experimental findings, and they can evoke particular Consequently, it is possible to investigate how the links between listening modes and motion through chosen musical parameters affected the resulting ascenarioanddesignofinteraction.Inthenext movements. section we describe specific examples illustrating Godøy is one of the pioneers of this type of re- the link between causal, acoustic, and metaphorical search. He proposed using the morphology of sound listening modes and gestural strategies. stimuli based on Schaeffer’s typology (impulsive, iterative, and sustained; cf. Godøy et al. 2006). This methodology was then used by other authors From Listening to Controlling such as Adrien Merer (2011) and Kristian Nymoen et al. (2011). Recently Kussner¨ (2013) proposed the In this section we describe concrete examples use of sequences of pure tones while changing the that we developed and that were used in different parameters pitch, loudness, and tempo. settings, from experiments and demonstrations to This previous work provides us with a promising interactive installations and performances. All these methodology for the study of gestural description examples are based on modeling the target sound of sounds. Most of these studies rely on exploring from a gestural perspective: a prior listening to (or analog relationships between gestural and sound evocation of) the sound provides performers with parameters. We will refer to such an approach as insights into possible strategies for gesture control. tracing (or analog) experiments, where the motion These strategies are then made possible using trajectories are associated with acoustic parameters. machine-learning techniques. Similar approaches In addition, in the following we will refer to sound have been described by Godøy (2006), Doug Van morphology to designate the temporal profile of the Nort (2009), Rebecca Fiebrink (2011), and Pieter-Jan acoustic characteristics of sound (e.g., amplitude, Maes (2012). pitch, and timbral aspects). Our general methodology is as follows. The In prior work (Caramiaux et al. 2014), we con- first step corresponds to listening to recorded ducted experiments to give evidence regarding the sounds from different perceptual perspectives, as link between gestural description and both acoustic described in the previous section. This leads one to and causal listening modes. We examined experi- consider scenarios and metaphors where the motion

38 Computer Music Journal Figure 2. The shaking scenario. A recorded rhythmic sound is analyzed and segmented. An incoming gesture is analyzed and its energy is computed and drives the selection of the segment to be played.

in interaction is linked to the targeted sounds. nonmusical actions and sounds. Consequently, the Mapping strategies are then designed to implement mapping designed for this scenario can be applied the interaction scenarios. In most cases, the mapping to any sound that is composed of percussive events is built using machine-learning techniques from of varying intensity, and it can be applied to any examples gathered during a “learning” phase, before movement that resembles shaking or waving (i.e., the final “playing” phase. movements that are periodic and modulated in intensity). This mapping is designed to be a direct rela- Interaction Scenarios and Mapping Strategies tionship between the movement energy and the energy of the sound played. The sound can, how- Four interactions have been created that implement ever, be chosen to be any percussive recorded sound. distinct mapping strategies illustrating the approach. The mapping relies on a first phase called learn- These four scenarios are shaking, shaping, fishing, ing. During this phase, an offline analysis of a and shuffling. sound database segments the recorded materials into percussive events and describes each seg- ment by its perceived intensity. Each segment is Shaking consequently structured according to its inten- The action–sound mapping of this scenario emerges sity level. During the second phase, playing, the from the action metaphor of shaking, associating the performer’s motion is analyzed in real time by performer’s shaking movement to the generation computing its energy. Sounds are then selected of percussive sounds. This scenario is meant to be from the database according to the motion’s level related to the causal mode of listening, since the of energy. The intensity of shaking has a direct performer mimics the gesture of shaking. Although relationship to the intensity of the synthesized per- this metaphor may refer, in music performance, cussive sound event whereas the performer does not to percussion instruments such as a shaker or control the rhythmic pattern. Figure 2 illustrates the maracas, it can also be associated with various scenario.

Caramiaux et al. 39 We use accelerometers to sense the performer’s performer (e.g., using buttons on the interface). A motion. Concrete implementations were featured sound is selected as soon as the gesture starts, using in different performances using the musical object areal-timeshape-matchingalgorithmthatfinds, interfaces (e.g., performances at the 2011 Margaret at each time step, the audio-feature morphology Guthman Musical Instrument Competition or the closest to the gesture morphology and aligns the two 2013 International Conference on Tangible, Embed- morphologies temporally. Note that the algorithm ded and Embodied Interaction, cf. Rasamimanana can be configured to allow transitions between et al. 2011). The shaking intensity can be obtained by gestures, which enables the algorithm to switch integrating the variations of the measured acceler- between sounds during the execution of a gesture. ation magnitude. Audio segmentation is performed Figure 3 illustrates the scenario. by onset detection. A mean loudness measure is The implementation, called the gesture follower, computed for each segment. Both feature spaces, is based on a machine-learning technique using motion and sound, are normalized, so that each hidden Markov models (HMMs) and is presented in sound segment can be associated with a correspond- the Appendix. Because the sound is aligned to the ing shaking intensity lying between the lowest and gesture in real time, it translates the variations in the highest possible values. The system used a k-nearest gesture morphology, such as the speed of execution, neighbor (k-NN) search algorithm based on a k- to variation in the playback, reinterpreting the dimensional (k-D) tree to select a sound event of a recorded sound. In a demonstration presented at given intensity from among the available segments the 2010 Sound and Music Computing Conference (Schwarz, Schnell, and Gulluni 2009). (Caramiaux, Bevilacqua, and Schnell 2010a), gesture and sound were represented by a unidimensional time series, the energy of a gesture controlling the Shaping loudness. The energy of a gesture was computed Shaping refers to scenarios where performers control as its absolute speed (an infrared camera motion sound morphologies by “tracing” in the air those capture system was used to capture the gesture). salient sound features they desire to control. It is Being of different physical dimensions, the time thus related to acoustic listening as we defined series were scaled beforehand into the same range of previously, where the performer pays attention to values. acoustic qualities of the sound and, in particular, to its temporal evolution. Fishing The interaction scenario leads the performer to design gestures related to specific recorded sound The fishing scenario relies on a metaphor where the morphologies. Rather than using a metaphor, the performer mimics an action in order to select and link between gestures and sounds is built by analogy, play a specific sound. In other words, the performer as the design of gestures needs to tightly reflect the virtually “fishes” for the sound by mimicking the aspects of the sound the performer perceives and associated action that supposedly caused the sound. intends to affect. Again, the mapping relies on two Therefore, the fishing scenario is meant to be related distinct phases: learning and playing. The learning to the causal aspect of listening where a performer phase consists of a prior construction and analysis of focuses on the event that has produced the sound adatabaseofsounds.Eachsoundisanalyzedoffline and tries to mimic it. to compute the feature representation. The playing The application is based on the recognition of the phase starts with a gesture executed by a performer. performed action and requires a learning phase: A The performer gesturally draws the morphology of database of actions is built by recording one example aparticularsoundandreplaysthesoundinreal of each action to be recognized. An action is a single time, translating the time variations of the input unit represented as a multidimensional continuous gestures to sound variations. The beginning and time series of its parameters. In addition, each the end of the gesture must be marked by the action has an associated sound meant to illustrate

40 Computer Music Journal Figure 3. The shaping performance is extracted scenario. Multiple sounds and used to select and are analyzed by computing control the sound whose feature shapes. On the feature shape is the closest other hand, the motion to the gesture. shape of a live gesture

the possible sound produced by the action. During mimicked. The algorithm was set to play the sound the playing phase, the user performs a gesture that, associated with an action as soon as this action is if recognized as an action from the database, will recognized. In addition, the algorithm was set to trigger playback of theassociated sound. Because output the time progression in the executed action. the system relies on action recognition, both the When the user reached 90% of the recognized action, performed and the predefined actions must have the sound was set to be fished. The user has to do the aconsistentrepresentation,whichcouldimply same with the second action. Once both sounds are they were performed with the same device and, successfully fished, another set of pairs is presented. consequently, with the same set of parameters taking their values into the same range. Figure 4 Shuffling illustrates the scenario. The system uses the same algorithm (the gesture The shuffling scenario consists in gesturally recom- follower) as the shaping scenario presented in the posing and reinterpreting complex sound sequences. Appendix. In the installation version of the system, This is achieved by processing short pieces of presented during a meeting of Sound and Music for recorded sounds put in relationships with gesture Everyone Everyday Everywhere Everyway project segments. The scenario does not involve pre- (SAME, www.sameproject.eu/), the actions were established metaphors as in the previous examples, captured through the use of mobile phones with but defers the design choices to the performers, embedded accelerometers. The training process is allowing them to interactively implement their own part of the design and not seen by the performer. metaphors and control strategies. The playing phase was implemented with a gaming The mapping is designed by demonstration: The scenario. A set of two action–sound pairs from the gestures performed by the performer in conjunc- database was presented to the user in order to be tion with particular sounds are used to train a

Caramiaux et al. 41 Figure 4. The fishing performance tries to “fish" scenario. A set of recorded asoundbymimickingthe sounds is loaded together associated action. If with associated actions successful, the sound is that represent the sound. played. The incoming live gesture

machine-learning model that encodes their rela- and aligned to their reference in real time to dynam- tionships. When the performers perform a new ically select and replay the appropriate sequence of gesture sequence, sounds are resynthesized and sound segments along with the gesture performance. aligned in real time, using phase vocoding. In some Figure 5 illustrates the shuffling scenario. aspects, the present scenario generalizes some of The mapping is based on a hierarchical model for the previous examples by allowing the performer to continuous gesture recognition and segmentation, mimic sound-producing actions (cf. fishing), to trace called a hierarchical HMM (see the Appendix for sound features (cf. shaping), or to combine these details). The model has two levels. The lower level approaches sequentially. precisely encodes the time structure of the segments, Designing the mapping by demonstration in- and the higher level governs their sequencing, defin- volves an interaction loop divided into two distinct ing the possible transitions between various points phases: learning and playing. During the learning within the gesture. The model can be built from phase, the performer begins by selecting sounds asinglesegmentedexample.Therecognitionis and manually defining their segmentation using performed in real time and the model estimates a graphical editor. Then the performer records the alignment of the new gesture compared with one or multiple gestures associated with each the reference, allowing for the reinterpretation sound, for example, by recording a template gesture of the sound with a fine time precision. Thus, synchronously while listening to a given sound. the temporal variations of the live gestural perfor- Additionally, one can specify authorized transitions mance are translated to sound variations using a between each gesture and sound segment. During phase vocoder (superVP in Max/MSP). the playing phase, the performer recomposes the Aspecificimplementationwasintroducedby original sounds by performing arbitrary sequences Franc¸oise, Caramiaux, and Bevilacqua (2012). Each of gestural segments. The gestures are recognized gesture and each sound morphology is segmented

42 Computer Music Journal Figure 5. The shuffling while listening to a given scenario. A learning phase sound. A playing phase allows the performer to allows the performer to select a segmented sound recompose the original and to record one gesture sound by performing associated to it, for arbitrary sequences of example by recording gestural segments.

into attack, sustain, and release segments, possibly attack can be related to mimicking (e.g., using complemented by a preparation phase anticipating a metaphor such as hitting an object) while the the attack of the sound. sustain and release phases can implement a tracing Two aspects of this decomposition are particularly gestural description. interesting. First, the consistency of the relationships between gesture and sound can be guaranteed by specifying Discussion and Conclusion constraints for the sound synthesis on particular segments (e.g., silence during preparation or tran- We presented four mapping examples illustrating sient conservation on attack phases). In addition, our approach, based on a perceptual analysis of the the features extracted from the performer’s gesture target sound. All examples use synthesis techniques in one action segment can be mapped to sound to gesturally “reinterpret” the recorded sounds. features of the following segments. In this way, the Each scenario and mapping strategy can be described silent trajectory of a preparation gesture can define by a top–down approach. In particular, each can be the features at the beginning of the sound that, linked to particular listening modes and gesture in the following segments, can be shaped by the strategies presented in the section “Describing performer’s gesture. (In the design of traditional in- Sound Gesturally.” struments, similar possibilities are obtained through Figure 6 summarizes how the examples are the instrument’s geometry, allowing the performer related to the different listening modes and gestural to interact—or not—with different parts of the strategies we have discussed. In addition, we require instrument responding to action in different ways.) the different strategies of mapping that are used Second, this decomposition allows for designing in the different examples. We distinguish between strategies that involve multiple gestural descriptions instantaneous, temporal, and metaphorical aspects related to listening: For example, preparation and that define the relationship between gesture and

Caramiaux et al. 43 Figure 6. Classification of strategy, which describes the scenarios along three how gestures derive from dimensions: the listening listening; and the mapping mode, related to listening strategies implementing processes; the gestural each gestural strategy.

Listening Mode Gestural Description Mode Mapping Strategies

Causal Acoustic Mimicking Tracing Instantaneous Temporal Metaphoric (sound source) (sound features) Iconic Analogic Shaking XX X X X X Shaping X XX Fishing X X X X X X X X X

sound. Instantaneous mapping strategies refer to the scenario can be seen as the closest mapping example translation of magnitudes between instantaneous to previous ideas developed by Godøy (2006) or Van gesture and sound features or parameters. Temporal Nort (2009). The difference with shaking resides mapping strategies refer to the translation and in the precise control over the sound’s temporal adaptation of temporal morphologies (i.e., profiles, evolution, supporting a listening mode focussed timing, and event sequences) between the gesture on acoustic sound features. Our experiments with and sound data streams. Metaphorical mapping this scenario showed that a sonic profile must be strategies refer to relationships determined by memorized beforehand in order to consciously target metaphorical or even semantic aspects, which do it and, eventually, to reproduce it with temporal not necessarily rely on morphological congruences variations. between gesture and sound. The shaping scenario makes use of a temporal The shaking scenario makes use, principally, of mapping between gesture parameters and sound fea- an instantaneous mapping strategy between ges- tures. This mapping allows the performer to reshape ture and sound: The shaking intensity is directly asoundbasedonthetemporalmorphologyofhisor related to the intensity of each percussive sound her gesture. The general concept of temporal map- event. Interestingly, we have observed how per- ping was previously introduced by Bevilacqua et al. formers spontaneously synchronize their shaking (2011) for the cases where temporal relationships movements to the tempo generated by the system. between gesture and sound parameter profiles are This creates a direct action–perception loop: the established. sound “feedback” produced is similar to a shaker The fishing scenario makes use of a mapping sound and encourages the player to pursue a shaking that can be considered as metaphorical: Unlike the movement. The listening mode is causal and there is shaking and shaping scenarios, the morphologies of a metaphorical association between the action and gesture and sound in this example can be incongru- sound. Owing to the strong action metaphor, the ent in some cases. The action–sound relationship scenario can also supply completely unconventional is, nevertheless, clear from the perspective of causal sounds for the performer to shake. listening. As mentioned previously, this scenario has In the shaping scenario performers mainly focus been shown at an installation during the EU Project on “acoustic” properties of the sound. They must SAME. Feedback from users showed that such a “trace” the temporal profile of a sound feature mapping was highly appreciated and characterized to be able to select and modify a sound whose as ludic. Indeed, the sounds chosen were easily iden- morphology matches the motion shape. Relying tified and the action easily reproducible. Although on temporal morphologies, the mapping of this the scenario focuses on a causal mode of listening,

44 Computer Music Journal an extended version comprising a metaphorical wireless, well-understood signal characteristics, and mode of listening can be envisaged and can enrich sufficient precision for most musical applications. the scenario. The scenarios discussed in this article make Finally, the shuffling scenario makes use of a extensive use of methods based on machine learning mapping strategy that can be characterized as both (k-NN, HMM, hierarchical HMM). The role of temporal and metaphorical. The temporal charac- machine learning is to implement the top–down teristic of the mapping is similar to the shaping approach of our scenarios based on perceptual or scenario, and the metaphorical characteristic is metaphorical action and sound description. Indeed, enabled by the implementation of a general al- all scenarios imply implicit relationships between gorithm for the recognition of actions and action sound and gestural features. As discussed by Tom sequences. The combined mapping consequently Mitchell (2006), machine-learning techniques are offers additional control opportunities and action– effective for modeling such implicit relationships. perception loop feedback. It drives performers in Moreover, such an approach has started to be both causal and acoustic listening modes, making implemented and evaluated in different cases them conscious of both the sound morphologies in computer music performance (Fiebrink 2011; (as in shaping) and the control of sound segments Gillian 2011; Caramiaux and Tanaka 2013). The through iconic gesture segments (as in fishing). The ongoing research in this area examines the use shuffling example can be seen as an unified approach of machine learning for automatically selecting in the sense that it can be configured to activate gesture and sound features (Caramiaux, Bevilacqua, several modes of listening and several modes of and Schnell 2010b), for jointly modeling their gestural description (and it can also easily include interactions over time to implicitly capture their the shaking scenario). correlations and the expressive variations emerging The temporal aspects of mapping are particularly in different interpretations (Franc¸oise, Schnell, and important when designing action–sound relation- Bevilacqua 2013), or the use of machine learning as ships based on the transformation of recorded adesigntool(Fiebrink,Cook,andTrueman2011). sounds. In this case, temporal mapping strategies The possibilities arising from the introduction allow for adapting the temporal morphologies ini- of machine-learning techniques into the interac- tially present in the recorded sounds to the actions tion loop are twofold. First of all, they allow the of the performer. We believe, nevertheless, that instrument to integrate notions of recognition and temporal mapping strategies are equally powerful prediction that support the implementation of in- when considering other synthesis methods. They teractions based on the performer’s listening. As the allow one to segment the performers’ actions and performer always adapts his or her actions to the to define different action–sound relationships for behavior of the instrument—either spontaneously different segments. There is, for example, a need for or by strenuous learning—these new instruments, adistinctionbetweenactionsegmentsthatactually for their part, adapt themselves to the performer’s produce sound or induce sound changes, and those behavior, preferences, and playing style. It is worth that do not. noticing that machine-learning techniques are prone One design choice in the examples presented to errors or may require time to converge to an accu- here concerns the motion-sensing technology. Any rate estimate. Latency is inherently involved, and it sensing system provides a partial gesture description, may be an issue for specific types of control. On the which might impact sound controllability. In the other hand, latency can be handled by design. For four scenarios presented, we used accelerometers. instance, in the fishing scenario we chose to use the Although these sensors have inherent limitations recognition latency, namely the fact that the user (e.g., they are unable to sense spatial information), has executed 90 percent of the action, as a visual they are sensitive to small changes in orientation and progress bar for the user. Interestingly, with latency dynamics. The choice, moreover, has been motivated represented in this manner, it challenged the user by other advantages of this technology: low cost, during the interaction, enhancing the game play.

Caramiaux et al. 45 In conclusion, we propose a design approach for Caramiaux, B., F. Bevilacqua, and N. Schnell. 2010b. mapping based on the concept of embodied listening. “Towards a Gesture–Sound Cross-Modal Analysis.” In Building on previous work on listening modes and S. Kopp and I. Wachsmuth, eds. Gesture in Embod- gestural descriptions we propose to distinguish three ied Communication and Human-Computer.Berlin: mapping strategies: instantaneous, temporal, and Springer Verlag, pp. 158–170. Caramiaux, B., and A. Tanaka. 2013. “Machine Learn- metaphorical. Our approach considers the processes ing of Musical Gestures.” In Proceedings of the of listening as the foundation—and the first step—in Conference on New Interfaces for Musical Expres- the design of action–sound relationships. In this sion.Availableonlineatbaptistecaramiaux.com/blog/ design process, the relationship between action and wp-content/uploads/2013/05/nime2013 mlrev.pdf. sound is derived from actions that can be perceived Accessed March 2014. in the sound. We believe that the described examples Caramiaux, B., et al. 2014. “The Role of Sound Source only scratch the surface of the possibilities arising Perception in Gestural Sound Description.” ACM from this approach. Transactions on Applied Perception 11(1):1–19. Caramiaux, B., et al. In press. “Adaptive Gesture Recogni- tion with Variation Estimation for Interactive Systems.” Acknowledgments ACM Transactions on Iterative Intelligent Systems. Castagne, N., et al. 2004. “Haptics in Computer Music: This work is supported by the mixed research lab AParadigmShift.”InProceedings of the Eurohaptics Meeting,pp.174–181. Sciences and Technologies for Music and Sound Chion, M. 1983. Guide des objets sonores : Pierre Schaeffer (STMS), the Institut de Recherche et Coordination et la recherche musicale.Paris:Buchet/Chastel. Acoustique/Musique (IRCAM), the Centre National Fadiga, L., et al. 2002. “Speech Listening Specifically de la Recherche Scientifique (CNRS), the Universite´ Modulates the Excitability of Tongue Muscles: A TMS Pierre et Marie Curie (UPMC), and the Legos project Study.” European Journal of Neuroscience 15(2):399– (ANR 11 BS02 012). 402. Fiebrink, R. A. 2011. “Real-Time Human Interaction with Supervised Learning Algorithms for Music Composition References and Performance.” PhD Thesis, Princeton University, Department of Computer Science. Arfib, D., et al. 2002. “Strategies of Mapping Between Fiebrink, R. A., P. R. Cook, and D. Trueman. 2011. Gesture Data and Synthesis Model Parameters Using “Human Model Evaluation in Interactive Supervised Perceptual Spaces.” Organised Sound 7(02):127–144. Learning.” In Proceedings of the SIGCHI Conference Bevilacqua, F., et al. 2010. “Continuous Realtime Ges- on Human Factors in Computing Systems,pp.147–156. ture Following and Recognition.” In S. Kopp and Franc¸oise, J., B. Caramiaux, and F. Bevilacqua. 2011. I. Wachsmuth, eds. Gesture in Embodied Commu- “Realtime Segmentation and Recognition of Gestures nication and Human–Computer Interaction.Berlin: Using Hierarchical Markov Models.” Master’s Thesis, Springer, pp. 73–84. Universite´ Pierre et Marie Curie (Paris VI). Bevilacqua, F., et al. 2011. “Online Gesture Analysis and Franc¸oise, J., B. Caramiaux, and F. Bevilacqua. 2012. Control of Audio Processing.” In J. Solis and K. Ng, eds. “A Hierarchical Approach for the Design of Gesture- Musical Robots and Interactive Multimodal Systems. to-Sound Mappings.” In Proceedings of the Sound Berlin: Springer, pp. 127–142. and Music Computing Conference.Availableonline Cadoz, C. 1988. “Instrumental Gesture and Musical at smcnetwork.org/system/files/smc2012-203.pdf. Composition.” In Proceedings of the International Accessed March 2014. Computer Music Conference,pp.1–12. Franc¸oise, J., N. Schnell, and F. Bevilacqua. 2013. “A Caramiaux, B., F. Bevilacqua, and N. Schnell. 2010a. Multimodal Probabilistic Model for Gesture-Based “Analysing Gesture and Sound Similarities with a Control of Sound Synthesis.” In Proceedings of the ACM HMM-Based Divergence Measure.” In Proceedings of International Conference on Multimedia,pp.705–708. the Sound and Music Computing Conference.Available Frey, A., et al. 2009. “Temporal Semiotic Units as Minimal online at smcnetwork.org/files/proceedings/2010/9.pdf. Meaningful Units in Music? An Electrophysiological Accessed March 2014. Approach.” Music Perception 26(3):247–256.

46 Computer Music Journal Gaver, W. W. 1993a. “How Do We Hear in the World? Merer, A. 2011. “Caracterisation´ acoustique et perceptive Explorations in Ecological Acoustics.” Ecological du mouvement evoqu´ elessonspourlecontr´ oleˆ de la Psychology 5(4):285–313. synthese.”` PhD Dissertation, UniversitedeProvence´ Gaver, W. W. 1993b. “What in the World Do We Hear? An Aix-Marseille 1. Ecological Approach to Auditory Event Perception.” Merleau-Ponty, M. 1945. La Phenomenologie´ de la Ecological Psychology 5(1):1–29. Perception.Paris:Gallimard. Gillian, N. 2011. “Gesture Recognition for Musician Miranda, E., and M. Wanderley. 2006. New Digital Computer Interaction.” PhD Thesis, Queen’s Univer- Musical Instruments: Control and Interaction beyond sity Belfast, School of Music and Sonic Arts. the Keyboard.Middleton,Wisconsin:A-REditions. Godøy, R. I. 2006. “Gestural–Sonorous Objects: Embodied Mitchell, T. M. 2006. “The Discipline of Machine Learn- Extensions of Schaeffer’s Conceptual Apparatus.” ing.” Technical Report CMU-ML-06-108. Pittsburgh, Organised Sound 11(2):149–157. Pennsylvania: Carnegie Mellon University, School of Godøy, R. I., E. Haga, and A. R. Jensenius. 2006. “Ex- Computer Science, Machine Learning Department. ploring Music-Related Gestures by Sound-Tracing: A Momeni, A., and C. Henry. 2006. “Dynamic Independent Preliminary Study.” In Proceedings of the International Mapping Layers for Concurrent Control of Audio and Symposium on Gesture Interfaces for Multimedia Video Synthesis.” Computer Music Journal 30(1):49–66. Systems,pp.27–33. Noe,¨ A. 2005. Action in Perception.Cambridge,Mas- Haueisen, J., and T. R. Knosche.¨ 2001. “Involuntary Motor sachusetts: MIT Press. Activity in Pianists Evoked by Music Perception.” Nymoen, K., et al. 2011. “Analyzing Sound Tracings: A Journal of Cognitive Neuroscience 13(6):786–792. Multimodal Approach to Music Information Retrieval.” Hunt, A., and R. Kirk. 2000. “Mapping Strategies for Mu- In Proceedings of the International ACM Workshop on sical Performance.” In M. M. Wanderley and M. Battier, Music Information Retrieval with User-centered and eds. Trends in Gestural Control of Music.Paris:Institut Multimodal Strategies,pp.39–44. de Recherche et Coordination Acoustique/Musique, Rasamimanana, N., et al. 2011. “Modular Musical pp. 231–258. Objects Towards Embodied Control of Digital Music.” Huron, D. 2002. “A Six-Component Theory of Auditory- In Proceedings of the International Conference on Evoked Emotion.” In Proceedings of the International Tangible, Embedded, and Embodied Interaction, Conference on Music Perception and Cognition, pp. 9–12. pp. 673–676. Rovan, J., et al. 1997. “Instrumental Gestural Mapping Kussner,¨ M. 2013. “Music and Shape.” Literary and Strategies as Expressivity Determinants in Computer Linguistic Computing 28(3):1–8. Music Performance.” In Proceedings of Kansei: The Lahav, A., et al. 2005. “The Power of Listening: Auditory- Technology of Emotion Workshop,pp.68–73. Motor Interactions in Musical Training.” Annals of the Schaeffer, P. 1966. TraitedesObjetsMusicaux´ .Paris: New York Academy of Sciences 1060(1):189–194. Editions´ du Seuil. Large, E. W. 2000. “On Synchronizing Movements to Schwarz, D., N. Schnell, and S. Gulluni. 2009. “Scalability Music.” Human Movement Science 19(4):527–566. in Content-Based Navigation of Sound Databases.” In Large, E. W., and C. Palmer. 2002. “Perceiving Temporal Proceedings of the International Computer Music Regularity in Music.” Cognitive Science 26(1):1–37. Conference,pp.13–16. Leman, M. 2007. Embodied Music Cognition and Me- Tuuri, K., and T. Eerola. 2012. “Formulating a Revised diation Technology. Cambridge, Massachusetts: MIT Taxonomy for Modes of Listening.” Journal of New Press. Music Research 41(2):137–152. Leman, M., et al. 2009. “Sharing Musical Expression Van Nort, D. 2009. “Instrumental Listening: Sonic Gesture Through Embodied Listening: A Case Study Based on as Design Principle.” Oganised Sound 14(02):177– Chinese Guqin Music.” Music Perception 26(3):263– 187. 278. Van Nort, D., M. M. Wanderley, and P. Depalle. 2004. Liberman, A. M., and I. G. Mattingly. 1985. “The Motor “On the Choice of Mappings Based on Geometric Theory of Speech Perception Revised.” Cognition Properties.” In Proceedings of the Conference on New 21(1):1–36. Interfaces for Musical Expression,pp.87–91. Maes, P.-J. 2012. “An Empirical Study of Embodied Varela, F., E. Thompson, and E. Rosch. 1991. The Embod- Music Listening and Its Applications in Mediation ied Mind: Cognitive Science and Human Experience. Technology.” PhD Dissertation, Ghent University. Cambridge, Massachusetts: MIT Press.

Caramiaux et al. 47 Wanderley, M. M. 2002. “Mapping Strategies in Real-Time Monte Carlo inference on the parameters of a non- Computer Music.” Organised Sound 7(2):83–84. linear dynamic system. It allows for the continuous Wanderley, M. M., and P. Depalle. 2004. “Gestural adaptation to variations of gesture characteristics Control of Sound Synthesis.” Proceedings of the IEEE (Caramiaux et al. in press). Indeed, once the ges- 92(4):632–644. ture template is recorded, a similar live gesture Wanderley, M. M., and N. Orio. 2002. “Evaluation of Input can be performed with variations in speed, scale, Devices for Musical Expression: Borrowing Tools from HCI.” Computer Music Journal 26(3):62–76. rotation, etc. These characteristics can be explicitly Zatorre, R. J., J. L. Chen, and V. B. Penhune. 2007. “When taken into account by the method as invariant for the Brain Plays Music: Auditory–Motor Interactions the recognition. To that extent, the method con- in Music Perception and Production.” Nature Reviews tinuously estimates the relative characteristics of Neuroscience 8(7):547–558. the gestural variations, which can then be used in continuous interaction scenarios.

Appendix: Algorithms Hierarchical Extension In this appendix we describe the algorithm used in the interaction scenarios. Rather than a full The gesture follower has been extended to com- technical specification, we outline the model used prehend time structures that are more complex, and how the model has been adapted to the context. allowing for the representation of gestures as or- dered sequences of segments. The method is based on hierarchical HMMs with a two-level structure Gesture Follower (Franc¸oise, Caramiaux, and Bevilacqua 2011). The lower level models the fine time structure of a The gesture follower (GF, cf. Bevilacqua et al. 2010) segment using a template-based approach identical is a template-based gesture-recognition method to the GF. The higher level governs how segments based on HMMs. The model is learned from a single can be sequenced by a high-level transition struc- example gesture, using its whole time series as a ture, whose probabilities constrain the possible template. The model is built by assigning a state transitions between segments. Thus, the model can to each frame, similarly to dynamic time warping. be built from a single demonstration of the gesture The time structure is modeled by a left-to-right complemented by prior annotation defining the transition structure. A causal forward inference segmentation. The recognition is based on a for- allows for decoding in real time and returns the ward algorithm allowing for the causal estimation currently recognized template, as well as the of the performed segment (informed by the high- time progression in the template, performing an level transition structure) and the time position alignment of the live gesture to the template. within this segment (as detailed in the the previous section). This representation provides both fine- Adaptive Extension grained and high-level control possibilities, allowing one to reinterpret gestures through a segment- The model has been recently extended to quantify level decomposition that can be authored by the and adapt to gesture variations by using sequential performer.

48 Computer Music Journal Jan C. Schacher, Daniel Bisig, and Philippe Kocher The Map and the Flock: Institute for Computer Music Emergence in Mapping and Sound Technology Zurich University of the Arts Baslerstrasse 30 with Swarm Algorithms 8048 Zurich,¨ Switzerland jan.schacher, daniel.bisig, philippe.kocher{ @zhdk.ch }

Abstract: In mapping for computer music and interactive media, flocking algorithms represent a special case, offering dynamic, self-organized domain translations. In this article we attempt a classification of fundamental mapping relationships that can be established with the help of swarm simulations. By regarding flocks as systems of abstract entities, a number of models arise that deal with the reassignment of perceptual and semantic qualities to the simulated entities. These models represent basic mapping processes, but become domain-specific when used for music and interactive art. To illustrate these concepts, we outline a number of strategies that relate to musical practice, fostering an understanding of the role of swarm simulations in mapping. We show two artistic use cases where these concepts are applied in an exemplary manner. In the first artwork, swarms play a central role in the compositions presented in an audiovisual installation, and serve as an intermediate translation space between audience and artwork. In the second realization, swarms interact with dancers and together they control the visual and musical aspects of the piece. Both examples show how the emergent behavior of flocks can be mapped conceptually and can evoke natural phenomena, thus making the mapping relationships less predictable and more organic.

Flocking algorithms (Reynolds 1987) are an impor- transforms input from a control domain to a target tant class of rule-based emergent systems (Johnson domain and hence functions as a mediation layer 2004) that enable the exploration of life-like be- in the mapping system. Swarms offer the ability to haviors and structures. The algorithms represent decouple low-dimensional parametric inputs from abstract formalisms that need to be translated into higher-dimensional target domains such as synthesis aperceivableoutputinordertobecomeaccessible processes, sound spatialization systems, or symbolic to a human audience. This translation, be it visual, structure generators. Finally, flocking algorithms musical, or physical, forms an important part of the may also be regarded as a means of translation on a artistic process. In the case of flocking algorithms, conceptual or perceptual level, where the intrinsic this task needs to be approached by posing the classic qualities of the emergent behavior of a swarm evoke mapping questions of how to generate coherent out- or allude to phenomena that normally occur in the put appropriate for the emergent behavior displayed domain of natural experiences. by these multi-agent systems. In many applications, In this article, we discuss the potential of these an additional layer of mapping is necessary, when algorithms and their application from both a con- the flocking algorithm is exposed to some kind ceptual and practical point of view. We attempt to of control, either from a direct human–computer explore the concepts systematically, based on the interaction or some other form of significant data insights obtained as the result of artistic realiza- originating from outside the simulation. tions and developments that took place within two Conceptually, a flocking algorithm can fulfil research projects. In order to ground these concepts different roles in a generative system. It can either in real artistic applications, the experiences gained be the source of all activity as well as generating by applying flocking algorithms are discussed with the main constitutive shapes and behaviors of a the aid of two exemplary use cases. work, or it may represent an intermediate step in asystemthatisgovernedbyotherprinciplesof control. In the latter case, the flocking algorithm Background

Computer Music Journal, 38:3, pp. 49–63, Fall 2014 A vast field for artistic experimentation is provided doi:10.1162/COMJ a 00256 by adaptating scientific simulation techniques as c ⃝ 2014 Massachusetts Institute of Technology. generative mechanisms that control the creation

Schacher et al. 49 of music and image (Dorin 2001; Galanter 2003; approach imposes the scientific context, within McCormack 2003). Of particular interest are sim- which the simulation was originally developed, ulations that model complex natural phenomena. onto an artistic work and thereby leads to a naive Such phenomena are characterized by a structural confusion of scientific and artistic motivations. The organization, which emerges from processes of main assertion of this article is that the planning self-organization and combines regular and chaotic and realization of mapping relationships between properties. These characteristics establish on a a simulation and a musical or visual form are an metaphorical level a proximity to the aesthetic prin- important part of the translation of concepts and ciple of balancing order and disorder. It is through a mechanisms that were originally situated outside of simulation-based abstraction that a similar proxim- the arts into integral elements of artistic creation. ity can be established on a formal and operational level. In this context, biological phenomena are of special interest, as they exhibit a particularly high Mapping Concepts level of complexity. Algorithms that model these phenomena are often derived from research in the Any simulation represents a computer-based op- field of artificial life, where their development forms erationalization of functional relationships among an important part of the scientific methodology. highly abstract syntactic entities. Although a sim- Flocking simulations (Eberhart, Shi, and Kennedy ulation is meant to maintain the structural and 2001) are a classic example of such algorithms. behavioral aspects of a natural phenomenon on a These simulations were originally invented in order formal level, it is semantically and perceptually to model the coordinated movements of individuals entirely disconnected from said phenomenon. In within large groups, such as flocks of birds or schools abroadersense,thedesignofmappingstrategies of fish. Such simulations operate by formalizing bi- using flocks deals with the reassignment of outside ological organisms as abstract entities, so-called perceptual and semantic qualities to the simulated agents, whose properties and behaviors are imple- entities. Accordingly, the realization of a genera- mented as vector quantities and distance-based tive artwork that provides a meaningful experience mathematical transformations. It is the character- involves the central task of establishing mapping istics and flexibility of these formalizations that mechanisms that exploit the syntactically powerful render flocking simulations useful for musical and but perceptually indeterminate characteristics of a artistic applications. Accordingly, swarm simula- simulation. tions enjoy a high level of popularity in algorithmic music and generative art, and their application has been explored by numerous artists such as Black- well and Bentley (2002), Blackwell and Young (2004), Fundamental Relationships Boyd, Hushlak, and Jacob (2004), Shiffman (2004), Ramos (2004), Uozumi (2007), Uozumi, Takahashi, An arbitrary number of mapping strategies exist and Kobayashi (2008), and Davis and Karamanlis that permit one to relate aesthetic qualities of (2007). an artwork to simulation-based processes. The It may be tempting to use swarm simulations following section attempts to organize this space of or any other type of scientific simulation as a possibilities into a limited number of fundamental generative “ready-made” whose numerical output categories. These categories are specified in an is post-processed in some manner to meet the extremely abstract and generic form in order to make particular requirements of a musical and visual them applicable to a wide range of simulation-based method. Such an approach falls short, however, music and art forms. In addition, the categories are of transforming a simulation into an artistic tool not mutually exclusive. The practical examples of that informs the conception and realization of a swarm music relationships that follow in the latter work on a more fundamental level. Rather, this part of this article demonstrate several of these

50 Computer Music Journal fundamental mapping relationships at the same considered to represent entities that exist at the same time. time in the simulation and rendering environment. Formal relationships deal with the establish- Asimpleexampleofsuchamappingwouldassign ment of correspondences between simulation-based to each simulated object an equivalent object in the processes and computational mechanisms that rendering system. As a result, formally discernible underpin the creation of music and image. The sim- objects in a simulation can be transformed into per- plest form of a formal relationship deals with the ceptual objects. A similar form of mapping applies selection and application of the numerical output such equivalence to a group of elements. With this of a simulation in order to control specific param- kind of a mapping, group-specific qualities of homo- eters of a rendering process. This numerical data geneity or diversity can be translated into analogous is usually mathematically transformed in order to qualities of a perceptual mass or cloud. The strongest match the number, range, and dimensionality of the form of ontogenetic relationship is established by target parameters. Simple mathematical transfor- matching the modeling characteristics between mations such as scaling or the application of transfer the simulation and rendering system. In this case, functions preserve the continuity and dynamics of the simulation and rendering system merge into a the simulation’s numerical output. In this case, the single system. As a result, the simulated elements continuity can be used as an aesthetic principle and processes become inherent parts of the render- that causes an analogous continuity and dynamics ing system and thus contribute to the perceptual in the behavior of the rendering mechanism. If, on result. the other hand, the simulation produces numerical The establishment of a conceptual relationship output at intermittent intervals or if this interval between a simulation and an artwork helps to is created through mathematical discretization—for situate the simulation as an integral part of an example, by applying thresholds—the numerical artistic idea. Such a relationship emerges, for input to the rendering algorithm gains the charac- instance, by abstracting an artistic creation process teristics of an event. These events can subsequently to such a degree that it becomes comparable to the serve as triggers to create discrete perceptual events. formal characteristics of a simulation. As a result, A different and more sophisticated category of asimulationisappliedasasyntheticmechanism formal relationships deals with the establishment of that embodies aspects of an artist’s compositional an organizational similarity between a simulation strategy. Alternatively, a conceptual relationship and a rendering mechanism. Such a mapping could, may be established through the opposite approach. for instance, be based on the assignment of spatial Such an approach assigns metaphorical connotations properties to elements in both the simulation and to the simulation and transfers it to the domain of rendering mechanisms. This assignment transforms affinity that is connected to a familiar artistic the elements into components of a shared spatial experience (Lakoff and Johnson 1980). organization that relate to each other based on Interaction relationships provide a simulation proximity criteria. An alternative form of mapping with the means of sensing and responding to a arises from the transfer of hierarchical organizations physical gesture in such a way that a subsequent from the level of the simulation to that of the transformation through the rendering system gives rendering system. Such an approach is possible if rise to a perceivable feedback. This type of relation- both the simulation and the rendering system use ship complements the traditional mapping between modular components that are organized in tree-like simulation and rendering with an additional step graph structures or other forms of hierarchical that relates the output of a sensory system to proper- groupings. ties and processes that are intrinsic to the simulation Ontogenetic relationships are based on the as- (see Figure 1). The dynamics of gestural activities signment of identical properties and behaviors to manifest themselves across the first mapping step as the elements in both the simulation and the ren- aperturbationofthesimulation’sinternaldynam- dering system. Accordingly, these elements can be ics. This perturbation emerges eventually through

Schacher et al. 51 Figure 1. Schema for the to generate discrete events swarm simulation of the in the audio domain. The piece Impacts.Herethe simulation-domain three swarms are in a decouples the interaction hierarchical relationship. from the visual and sound Their continuous state output of the piece. change is analyzed in order

the second mapping step as a form that is part of the dences have to mediate between the algorithm’s generated aesthetic result. intrinsic properties, the perceptual peculiarities of Finally, ecological relationships serve to establish the chosen feedback modalities, and any stylis- asharedspacewithinwhichtheperceptualand tic constraints or requirements imposed from the behavioral properties of simulated entities and outside. humans overlap and interrelate. Such a situation Direct parameter mapping is the simplest and resembles an ecosystem in the sense that the most commonly applied correspondence between activities of its various inhabitants become part swarm behaviors and sound processes. It translates of dense network of causal relationships. The agent properties directly to musical parameters. The flexibility and openness of this mutual engagement agent position in a Euclidean space, for example, is needs to be defined through the design of specific directly converted into a position in the parameter ecological relationships (Gibson 1986). space. The relationship established in this way is particularly compelling when it connects to familiar metaphors of spatial geometry. Other common Mapping Swarms to Music metaphors applied in these mappings are those that describe acoustic or musical properties. Examples Generative art emphasizes the utilization of pro- of these metaphors are “high–low” for pitches or cesses for the automated creation of artworks “left–right” for stereo panning. (Galanter 2008). These processes are often the result In the case of a one-to-one mapping, every of formalisms or data collections that were not single agent is associated with an individual sound- originally created with an artistic intention. There- generating unit. Even if these individual units are fore, every artistic realization involves the task of simple from a technical point of view, the emergent establishing a meaningful correspondence between characteristics of the swarm as a whole may produce the characteristics of the underlying process and the arichsoundoutput.Inordertomatchthenumber aesthetic properties of the piece. These correspon- of musical parameters that are controlled by a

52 Computer Music Journal swarm, the dimensionality of individual agent or musical control parameters changes throughout properties needs to be adapted. For this purpose, the course of a piece. Examples of this spatial several agent properties such as position, velocity, correspondence concept would be the mapping and acceleration are combined in a many-to-one of agent positions to playback positions of sound mapping. As a last step before being applied, the files—using one dimension of the swarm’s Euclidean values of these agent properties need to be scaled space mapped to the time axis of a sound file—or the and possibly discretized in order to limit them to triggering of sound playback by detecting an agent’s what the sound control parameters require. proximity. Regardless of the specifics of the mapping, Procedural mapping represents the next higher it is always the emergent characteristics of the level of relationship between flocking algorithms swarm behavior that are rendered audible in the and sound processes. In all the mapping methods musical result. The acoustic output, for example, presented so far, the role of the swarm was limited to exhibits movements whose velocity and diversity merely modulating prespecified control parameters. match those of the agents: they cluster or disperse Procedural mapping, however, assigns an entirely depending on the strength of attractive and repulsive different and less constrained role to a swarm. forces among agents, and they reveal textural Here, the audio signal flow is constructed on the qualities depending on the number and movement fly according to rules whose execution depends on patterns of the agents. Thus, the overall sonic gestalt the properties of the swarm. While the swarm sim- is formed by a multitude of entities that relate to ulation progresses, the audio signal flow continues each other. Other examples of this perceptual fusion to evolve as the rules are applied to its structure in traditional computer music processes are additive over and over again. Although this approach offers or granular synthesis techniques. great flexibility, finding satisfying technical and Applying swarm properties to spatialized audio aesthetic solutions poses major challenges. On a represents a straightforward connection between technical level, the sound synthesis engine needs agent parameters and acoustic properties. Rather to support on-the-fly reconfiguration, the dynamic than assigning the agent’s position to the parameter range of the resulting audio signal has to be kept space of a sound property, the agent position is within permissible limits, and the computational directly mapped to the position of a source in demands of the signal processing chain must not a spatialized audio scene. Rendered on a multi- exceed the available processing power. On an aes- speaker surround system, the sound objects become thetic level, the challenges stem from the fact that entities in the space and the swarm’s movements the introduction of construction rules for the au- and spatial behaviors are reproduced in an evidently dio signal flow adds a second generative layer and perceivable manner. As a consequence, virtual and an additional level of complexity. Accordingly, it physical spaces fuse into one. becomes difficult to make aesthetically informed The distinct spatial properties of swarms may design decisions for such a system. In addition, it also be exploited by establishing a proximity- becomes difficult to preserve the gestalt perception based mapping between agents and musical control of a swarm’s behavior. Despite these challenges, parameters. For this purpose, the agents’ properties procedural mapping may arguably be regarded as are projected as points into a parameter space. The an aesthetically rewarding form of swarm-based Euclidean distance between these points may then computer music. be applied to change sound synthesis parameters or Conceptual mapping occurs when formal ele- trigger musical events. This approach is attractive ments that are shared between flocking algorithms because it takes into account the inherently spatial and music or image processes constitute the core nature of swarms. Because the number of agent link. Working with these conceptual analogies or properties does not need to match the number of resemblances becomes part of the creative process control parameters, this approach also offers an and informs the chosen materials and processes interesting flexibility when the number of agents before a mapping task as such is approached.

Schacher et al. 53 Conceptual mapping between swarms and music case, applying the concept of forces concurrently represents a cross-domain mapping and generates a in both domains generates an inherent mapping blended space. The two spaces of swarm simulation relationship. and music or image are partially projected onto athirdspace,calledtheblend,whichunitesthe common characteristics of both domains. Thus “the Practical Approaches to Mapping with Flocks blend has emergent structure not provided by the inputs” (Fauconnier 1997, p. 150). The model of The fundamental categories and mapping concepts this intermediate, intersecting domain serves as described here only make sense when viewed in an abstract template for many mapping concepts. concrete implementations. The leap from concept The proposition of perceptual spaces as defined by to application requires a clear strategy for dealing Arfib and colleagues (2002) in their strategies for with the different domains involved (Schacher 2010). mapping between gesture and sound synthesis is an This is true not only when composing autonomous exemplary application of this model. artworks that use swarms in conjunction with Afundamentalchallengeingenerativeartand electronic media, but also for situations where a composition relates to the establishment of mean- performer interacts with a generative system that ingful and traceable mapping relationships between uses this class of algorithms. the underlying algorithmic processes and the re- For composers, on the one hand, it is important sulting aesthetic output (Bisig and Neukom 2008). to be aware of the nonlinearity of a performer- Viewed from this perspective, the customization of driven musical form. The sonic materials need to be the generative algorithms themselves as a means organized in ways that render their recombination of matching a particular artistic goal does, indeed, possible in many permutations. In addition, a offer a valid mapping method (Schacher, Bisig, and number of compositional constraints need to be Neukom 2011). addressed by such an interactive algorithmic system: Any computational model of natural phenomena, the structural organization and layering of media such as flocking behaviors, represents a mathemati- material, the establishment of correspondences cal abstraction of reality. In order for the properties among media both within a modality and across and behaviors of the natural system to be applicable different modalities, the arrangement of media into to music, they have to be reduced to a level of gener- coherent paratactic (i.e., parallel structures), and ality that enables their creative reconfiguration into the temporal development of these structures—all an artificial phenomenon. The process of abstrac- these have to be clarified. At the beginning of the tion, however, also strips the natural phenomenon compositional process lies the task of recognizing of its naturally perceivable characteristics. As a the structural similarities shared by a particular consequence, the computational model loses many swarm simulation and the intended musical process. of the essential qualities that were present in the The awareness of such similarities should inform original system. It thereby becomes less constrain- the design of the behavioral characteristics of ing and facilitates the application of a wider variety the swarm and the composition of the musical of musical strategies than the original phenomenon. algorithms. Ideally, they will be implemented Connecting physical properties of a swarm sim- simultaneously and reflect each other conceptually. ulation directly to physical-model based synthesis The implementation of the mapping relationship may serve as an exemplary use case for this type between the flocking algorithm and the musical of conceptual mapping (see also the subsequent processes then follows as a consequence from the description of the piece Membranes). The physical previous choices. attributes constituting the forces that determine For choreographers, on the other hand, a great the behavior of a flock can be derived directly from deal of similarity between flocking algorithms and the functions of the physics-based system with dance can be found and applied. The interrelation- which the flock is designed to interact. In this ships and feedback loops between dancers, swarm

54 Computer Music Journal simulations, music, and image offer the opportunity (ISS). These projects were aimed at establishing a to shape the basic elements of choreography. Swarm systematic framework for the integration of swarm simulations constitute multi-agent systems that simulations as generative processes in computer- model group formation and spatial movement. In based music and art. The scope of these research this, the simulations deal directly with some of activities encompassed the identification of creative the fundamental constituents of dance. Both swarm strategies, the development of technical tools, and simulations and choreography deal with relation- the realization of prototypical artworks. ships between local and global patterns, with the As part of the technical activities, the develop- occupation of space through clustering and disper- ment of a flocking simulation environment, called sion, and with the synchronization of behaviors. ISO-Flock, was of particular importance. ISO-Flock The distinction between individuals and groups is is an open-source C++ library that is geared towards blurred in the sense that individuality appears both the development of a wide range of flocking behav- on the level of the single dancer or agent and on the iors (Bisig, Neukom, and Flury 2008). The library is level of the entire group. By relating to these analo- highly generic and it abstracts the characteristics of gies, the choreography can integrate more closely agent properties and behaviors to a level of generality with swarm behaviors and create a clear conceptual that dispenses with any semantic relationships with mapping. physical and biological aspects. This abstraction For the performer, finally, one of the main chal- abandons the model character of the simulation in lenges when interacting with emergent structures favor of an operationalization of the fundamental is the unpredictable nature of the output produced principles of spatial self-organization and coher- by the swarm. In order to be able to deal with this ence. This enables the conception and realization uncertainty, the learning process needs to include of artistic and musical works based on principles the memorization of the guiding principles of the in- other than swarm behaviors (Schacher, Bisig, and teraction between flock and output. Such a learning Neukom 2011). As a result, the simulation itself can process helps to establish the necessary familiarity become the embodiment of an artistic idea and can with the musical materials, the mappings, and the thus form an integral part of the musical and visual combinations afforded by the interactive system. To composition process. achieve this, it is important to gain an understanding ISO-Flock provides a communication interface of the rules of play, and to be able to enter into the based on Open Sound Control (OSC), both for re- performance without having to assert total control ceiving configuration commands and for sending over the entire interaction process (Schacher 2012). simulation data, thus permitting the modification Lastly, a common technique for working with a and reconfiguration of the simulation at run time. swarm is to visualize the swarm simulation in a This messaging system enables the simulation to graphically reduced manner. This “scientific visual- run as a standalone server application that com- ization” preserves the spatial and temporal patterns municates with any OSC-capable client software of a simulation through a symmetrical “one-to-one” in the artistic tool chain. Thanks to this, artists mapping onto visual properties. Thanks to this un- and musicians can work with various program- ambiguous visualization it becomes easier to assess ming languages and environments for the creation the aesthetic potential of a simulation. and modification of swarm simulations, as well as software for sensor-data acquisition and camera tracking. This makes the simulation responsive to Research Projects and Software Tools extended forms of interaction. The swarm simulation software has subsequently The studies and artistic realizations that form the been extended with a graphical user interface (see basis for this article have been conducted in the Figure 2). The interface was initially intended as context of two research projects, Interactive Swarm a tool for non-expert users to facilitate the manip- Orchestra (ISO) and Interactive Swarm Spaces ulation of the simulation, but it has proven to be

Schacher et al. 55 Figure 2. Software tools: the swarm visualization of ISO-Flock together with the graphical user interface.

equally important as a means for the rapid creation audiovisual installation Flowspace that was devel- and experimentation with different simulation oped by the authors between 2008 and 2010. It was types. Bisig and Kocher (2012) discuss these tools in shown twice to the public, in Zurich¨ in 2009 and San greater depth. Francisco in 2010, in the context of a thematic ex- hibition about sound, space, and virtuality entitled Milieux Sonores (Maeder 2010). Example Pieces The architectural structure of the installation takes the shape of a dodecahedron measuring about The following two use cases serve as an illustration four meters (approximately 13 feet) in height. The of the application of flocking algorithms in varying frame holds in its vertices a 20-channel speaker- roles to interactive and performance situations. array pointing inwards and represents a perfectly Different mapping concepts are applied in these symmetrical, spherical setup for surround audio. pieces. In one of the cases the swarm serves as All but one of the pentagonal faces are covered in a mapping algorithm itself, and in the other case fabric and some are used as rear-projection surfaces. mappings are used to translate into or out of the The frame offers enough standing space for four to simulation domain. five visitors and presents them with an immersive experience. Flowspace The shape of the installation informs the char- acteristics of the installation’s generative contents. The first use case for the application of flocking As a result, the architecture of the installation simulations to music is the interactive, immersive, supports the blending of physical and virtual space.

56 Computer Music Journal Figure 3. A partial view of rear-projected images the Flowspace installation. above represent an artistic The touch-based interpretation of the interaction surface swarm. The visitor is displays a “scientific surrounded by a visualization” of the 20-channel audio system. simulation. The

properties exist for swarm simulation and audiovi- sual processes and are organized as discrete states in a finite state machine. The selection of states is controlled by the visitor’s long-term level of activity. The installation’s characteristic as a hybrid ecosystem results from the interrelations among the activities of its natural and virtual inhabitants, which occur on several temporal, spatial, and causal levels. The simplicity and immediacy of the in- terface’s physical manipulation and its subsequent effect on the installation’s responses provides a natural form of interaction that helps to balance the visitors’ intuition, familiarity, curiosity, and surprise. The simulation space overlaps with the installation Strong perceptual relationships form an impor- space that surrounds the visitors. In addition, the tant aspect of the experience. In Flowspace,the simulation space is mapped onto a two-dimensional audiovisual compositions and the visual and tactile segment of the dodecahedron surface and forms feedback of the interface are linked by mapping part of the installation’s interface. This enables the to the same swarm simulation. The installation visitors to experience a spatial immersion within provides feedback through the modalities of touch, the virtual flock and to simultaneously assume hearing, and vision. The correlation among these an observer position outside the simulation (see modes shapes the aesthetic experience, directs the Figure 3). visitor’s attention, and enables the perception of the The installation creates an interactive, immer- installation’s emergent behaviors. The translations sive, and generative environment for audiovisual and mappings between the different elements are compositions that are controlled through simula- what effectively constitute this correlation. In an tions of flocking behavior. Flowspace uses these abstract sense, the different states of the pieces and generative algorithms not only for the creation their expressive characteristics correspond to affect of aesthetic feedback, but also to establish coher- spaces, where valence and arousal are organised as ence among spatial, perceptual, behavioral, and orthogonal dimensions, as described in concepts social phenomena that manifest themselves in this that formalize emotions in autonomous systems installation in an ecological way (Schacher 2009). (Masuch, Hartman, and Schuster 2006). Multiple layers of behavioral relationships are In the exhibitions, the installation presented simultaneously present. Mapping connections are three pieces by different authors. These pieces established at the intersections of the different were all centered around a swarm simulation and layers: between interaction, simulation, spatialized interactively generated sound and image. sound, and visualization. The behaviors of the vis- The algorithm in the first piece, Impacts,explores itors and the swarm agents affect each other on the possibility of a hierarchical field of relationships multiple levels, differing in immediacy and spatial within several flocks. On a secondary level the extension. By touching the surface of the interface, perceptually salient events are extracted from the the visitor’s touch is mapped to the spatial posi- continuous flow of data and used as basic impulses tions of a particular class of agents. Other agents for the music: The impacts of (near-) collisions subsequently respond to these changes. These in- between agents are used to generate the discrete terrelating agent-behaviors transform the scope notes that form the musical gestalt. This swarm- of the visitor’s action from an initially local and music realization represents a formal relationship of immediate effect to an element that contributes direct mapping based on proximity (see Figure 1). to the emergent dynamics of the installation’s au- In the second piece, Flow,themusicalideacen- diovisual compositions. Different combinations of ters on periodically changing agent neighborhoods,

Schacher et al. 57 Figure 4. Schema for the mapping space and is swarm simulation of the bordered by connection piece Flow.Herethe layers between the user’s swarm simulation fulfils interaction and the the role of an intermediate audio-producing processes.

rendered audible through rhythmical musical struc- connected to their neighbors by elastic springs. As tures. The agents of the secondary flock tend to the secondary agents evade the primary agents, settle into cyclical trajectories. That causes them some of the connected neighbors move beyond the to encounter primary agents in a periodical fashion, breaking distance of the springs. Consequently, which forms the basis for the creation of the rhyth- these springs are removed from the simulation. New mical structures. Sounds are triggered whenever a springs are formed whenever two secondary agents secondary agent gets sufficiently close to a primary get sufficiently close to one another. Whenever a one. The duration, amplitude, and frequency spec- spring is created, a corresponding acoustic spring is trum of the sounds are obtained from the position of instantiated and becomes audible in a strong but the primary agent, the distance between the agents, brief impulse excitation. The musical result of this and the secondary agent’s velocity, acceleration, mapping is produced through the combination of a and jerk (the first, second, and third derivatives, diffuse musical background, which is punctuated respectively, of position changes over time). Here by bright sound events that occur in correlation again, the swarm-music realization creates a formal with the events inside the swarm simulation. relationship of proximity-based direct mapping (see In this final realization we find an ontogenetic Figure 4). relationship of mapping, where both flock and sound In the third piece, Membranes,theswarm synthesis share the same physical characteristics simulation and sound-synthesis technique are both and behaviors. based on the same physical model: the behavior All three pieces create a conceptual relationship of interconnected springs. The flocking simulation where the shared elements between flock and music consists of two types of flocks: a primary, mostly or image processes bridge the domains thereby stationary flock and a secondary, mobile flock unifying them. An ecological relationship exists in that implements a mass–spring system. In this the synergistic space that is shared by visitors and model the agents represent point masses that are the artificial flocks, and the interaction relationship

58 Computer Music Journal Figure 5. Schema for the implements a mass–spring Figure 6. A scene from swarm simulation of the system, which is directly Stocos.Theswarm piece Membranes.The connected to the physical becomes visible through swarm simulation consists model that produces the traces of light and is of two types of flocks. The sound. attracted to the dancer’s secondary flock movements.

Figure 5 arises from the generation of synthetic sound and images in reaction to the visitors’ behavior (see Figure 5).

Stocos

The second use case extends the scope of the application of flocking algorithms to the domain of interactive dance, music, and image in a stage setting. The piece Stocos was developed in close collaboration between the second author, the choreographer Muriel Romero, and the composer Figure 6 Pablo Palacio (Bisig and Palacio 2012). Stocos focuses on the notion of gesture as a means of connecting bodily movement, simulation-based models the Brownian motion of microscopic parti- movement, sound synthesis, and video rendering cles. Video rendering displays the agent’s movement (see Figure 6). This abstraction establishes a formal as graphical line segments that interconnect pre- compatibility and consistency among the elements vious agent positions. Several choreographical se- of the performance. The simulation is based on the quences are realized as random movements through coherent spatial movement of flocking agents. The sequences of previous dance gestures. A conceptual sound-generation mechanism uses the method of compatibility is established through the notion of dynamic stochastic synthesis (Xenakis 1992), which gesture as an element of expressivity. This notion

Schacher et al. 59 provides the means to metaphorically relate the them to move autonomously, whereas others allow simulation-based processes to performance activi- them to respond to elements in the simulation ties. An additional emphasis of the piece lies in the space. These elements either represent other agents treatment of the stage as a synergistic environment. or are the result of a geometrical mapping of The term “synergy” refers to the cooperative activ- tracking data. The characteristics and diversity of ities of several elements of a system, which give these behaviors change throughout the performance. rise to a property or behavior that is unachievable If the agents respond predominantly to tracking- by each component alone (Fuller 1979). The term based spatial objects, then their behavior resembles “synergistic space” emphasizes the fact that the that of a physical system reacting passively to appearance and behavior within the performance the movements of the dancers (see Figure 6). If, space is not dominated by individual activities but on the other hand, the behavioral repertoire of is rather the result of the relationship and feedback the agents is larger and encompasses behaviors mechanisms among the activities of the dancers, the that respond to interagent relationships, then the simulation-based entities, and the generative-music simulation exhibits a higher degree of behavioral and image processes. In this sense, the presence of complexity and independence. This gives rise to the dancers becomes an indispensable element of an improvisational form of interaction between the space and adds the human involvement into the dancers and simulation. interactive environment (Spagnolli and Gamberini The appearance of the performance is character- 2005). ized by an acoustic and visual merging of the physical In Stocos,thestageisinhabitedbyhumandancers and the simulation space. The stage setup consists and simulated entities, both of which possess a be- of two white projection surfaces, one horizontal and havioral repertoire and the capability to perceive and one vertical, that delineate the performance space, respond to each other. The perceptual capabilities of and an octophonic speaker array that surrounds the agents rely on computer vision in order to detect the stage and audience. The stage is divided into the dancers’ positions, contours, and movements. distinct regions, each of which is associated with The video-tracking information is subsequently aparticularsetofsound-synthesiscontrolparame- transformed into spatial data structures that mani- ters. By relating the dancers’ positions to this space, fest themselves within the simulation, such as force their movements become perceivable as acoustic fields or polygonal chains. The agents are able to trajectories. The projection of the generative video perceive these structures through distance-limited image covers the stage floor, the stage background, neighborhood calculations between agent positions and the dancers’ bodies. Through this projection and geometrical objects. The dancers are able to the spatial relationships are translated into visual perceive the agents’ activities through the simu- elements. As a projection on the entire stage, the lation’s influence on the generation of music and images create a visual environment that supersedes visuals. the appearance of the physical space and the dancers. The piece uses two different levels of behavioral As a projection within the vicinity of the dancers, relationships among dancers and simulated agents. the images coalesce into clearly confined shapes The first level creates a gestural identity between that appear as visual counterparts to the dancers. choreography and simulation, which is based on a By aligning simulation space and corporeal space, direct mapping of the dancers’ movements to the the video image can be projected solely onto the agents’ positions and velocities. The mapping of dancers’ bodies. In this configuration, video images this remotely controlled swarm onto the rendering and the dancers’ physical bodies merge into a single mechanisms enables the dancers to gain direct entity whose appearance possesses both natural and control over the video and sound generation. The artificial properties. second level is based on an indirect behavioral A variety of formal mapping categories can relationship between dancers and agents: The agents be identified in the realization of Stocos.The possess a variety of behaviors, some of which cause identification of a gestural connection between

60 Computer Music Journal dancers, simulation, music, and image constitutes a flocking algorithm and its simulation space as a conceptual relationship. The notion of a synergistic blended space, control inputs with few elements can space that is shared by dancers and simulated easily be converted to complex effects in the target entities represents an ecological relationship. The process. Inversely, a swarm can translate a complex connection between swarm simulation and dynamic control situation into a few salient parameters in stochastic synthesis implements a formal numerical the output medium. Some of the common strategies relationship. The swarm-based control of the visual for the application of swarms, such as spatialization, rendering mechanism represents a simple form of an are obvious. But swarms also offer simple control ontogenetic relationship based on the fact that both over highly parallel polyphonic processes such as agents and visual elements share the same position. granular or additive synthesis or as simulation The activities of the dancers are related to the equivalence to physical models, with abstract enti- generation of synthetic sound and images through ties that directly reflect the synthesis algorithm. The several interaction relationships. The assignment of methods and toolset we present here are intended the dancers’ positions to spatial regions of sound- for the exploration of swarm algorithms in many synthesis control parameters implements a formal different scenarios. The swarm and its “scientific relationship based on proximity. The camera-based visualization” can be regarded as an extended graph- analysis of the dancers’ contours and movements ical user interface, and the flock may be regarded as as forms in physical space and their subsequent acomplexgeneratorforstructuredevents.Flocking transformation into forms within the simulation algorithms can also be considered from a systemic or space constitutes an ontogenetic relationship. even a metaphorical perspective. In particular, when Finally, the response of the agents to the presence of translating musical ideas through such conceptual these forms adds an additional mapping layer, which relationships, the mapping task is brought into the represents a formal proximity relationship. artistic domain and may occur at a moment in the creation process that precedes the establishment of parametric connections. The pieces that we have Conclusion shown in this article demonstrate only a fraction of the potential of using swarms as mapping methods, The usage of flocking algorithms as mechanisms yet they hopefully manage to convey the appeal that for mapping tasks in computer music presents a such behavior-driven forms can have. In conclu- number of advantages. Thanks to the emergent sion, it becomes apparent that lifelike behaviors of nature of swarm behaviors, relationships can be simulated flocks represent an attractive addition to established that offer rich and varied results that are the mapping strategies commonly used in computer less predetermined than traditional “connectivistic” music. Through their emergent qualities, they can mapping strategies. Based on a reflection of the dif- make mapping relationships less predictable and ferent roles of flocking algorithms in mapping tasks give the pieces a more autonomous evolution and a and the variety of situations for the application of more organic feeling. these mappings, we have presented what we believe to be fundamental categories of relationships that occur when translating between these domains. The References application of flocks in generative and performance- oriented composition situations is reflected in these Arfib, D., et al. 2002. “Strategies of Mapping Between categories. But these relationships may also ex- Gesture Data and Synthesis Model Parameters Using tend beyond the domain of pure mapping or the Perceptual Spaces.” Organised Sound 7(2):127–144. application of flocking algorithms. They describe Bisig, D., and P. Kocher. 2012. “Tools and Abstractions underlying principles that are useful for organizing for Swarm Based Music and Art.” In Proceedings heterogeneous materials and processes in computer of the International Computer Music Conference, music and generative art in general. By using a pp. 297–300.

Schacher et al. 61 Bisig, D., and M. Neukom. 2008. “Swarm Based Com- Masuch, M., K. Hartman, and G. Schuster. 2006. “Emo- puter Music—Towards a Repertory of Strategies.” In tional Agents for Interactive Environments.” In Pro- Proceedings of the Generative Art Conference,pp.101– ceedings of the International Conference on Creating, 111. Connecting and Collaborating through Computing, Bisig, D., M. Neukom, and J. Flury. 2008. “Interactive pp. 92–102. Swarm Orchestra: A Generic Programming Environ- McCormack, J. 2003. “Art and the Mirror of Nature.” ment for Swarm Based Computer Music.” In Proceed- Digital Creativity 14(1):3–22. ings of the International Computer Music Conference, Ramos, V. 2004. “Self-organizing the Abstract: Canvas as a pp. 101–111. Swarm Habitat for Collective Memory, Perception and Bisig, D., and P. Palacio. 2012. “STOCOS: Dance in Cooperative Distributed Creativity.” In Proceedings aSynergisticEnvironment.”InProceedings of the of the First Art and Science Symposium: Models Generative Art Conference,pp.96–109. to Know Reality,pp.59–60.Availableonlineat Blackwell, T., and M. Young. 2004. “Swarm Granula- arxiv.org/pdf/cs/0412073v1.pdf. Accessed April 2014. tor.” In A. I. Esparcia-Alcazar,´ ed. Applications of Reynolds, C. W. 1987. “Flocks, Herds, and Schools: A Evolutionary Computing.Berlin:Springer,pp.399–408. Distributed Behavioral Model.” Proceedings of the Blackwell, T. M., and P. Bentley. 2002. “Improvised Music Conference on Computer Graphics and Interactive with Swarms.” In Proceedings of the Congress on Techniques 21(4):25–34. Evolutionary Computation,vol.2,pp.1462–1467. Schacher, J. C. 2009. “Action and Perception in Interactive Boyd, J. E., G. Hushlak, and C. J. Jacob. 2004. “SwarmArt: Sound Installations: An Ecological Approach.” In Interactive Art from Swarm Intelligence.” In Proceed- Proceedings of the Conference on New Interfaces for ings of the Annual ACM International Conference on Musical Expression,pp.286–289. Multimedia,pp.628–635. Schacher, J. C. 2010. “Motion to Gesture to Sound: Davis, T., and O. Karamanlis. 2007. “Gestural Control Mapping for Interactive Dance.” In Proceedings of the of Sonic Swarms: Composing with Grouped Sound Conference on New Interfaces for Musical Expression, Objects.” In Proceedings of the Sound and Music pp. 250–254. Computing Sound and Computing Music Conference, Schacher, J. C. 2012. “The Body in Electronic Music pp. 192–195. Performance.” In Proceedings of the Sound and Music Dorin, A. 2001. “Generative Processes and the Electronic Computing Conference,pp.194–200. Arts.” Organised Sound 6(1):47–53. Schacher, J. C., D. Bisig, and M. Neukom. 2011. “Com- Eberhart, R., Y. Shi, and J. Kennedy. 2001. Swarm Intelli- posing with Swarm Algorithms—Creating Interactive gence.Burlington,Massachusetts:MorganKaufmann. Audio-Visual Pieces Using Flocking Behavior.” In Fauconnier, G. 1997. Mappings in Thought and Language. Proceedings of the International Computer Music Cambridge: Cambridge University Press. Conference,pp.100–107. Fuller, R. B. 1979. Synergetics: Explorations in the Shiffman, D. 2004. “Swarm.” Demonstration presented Geometry of Thinking.London:Macmillan. at the SIGGRAPH Emerging Technologies Exhibition, Galanter, P. 2003. “What is Generative Art? Complexity 8–12 August 2004, Los Angeles, California. Theory as a Context for Art Theory.” In Proceedings Spagnolli, A., and L. Gamberini. 2005. “A Place for of the Generative Art Conference.Availableonlineat Presence: Understanding the Human Involvement in www.philipgalanter.com/downloads/ga2003 paper.pdf. Mediated Interactive Environments.” PsychNology Accessed March 2014. 2(1):6–15. Galanter, P. 2008. “Complexism and the Role of Evolu- Uozumi, Y. 2007. “GISMO2: An Application for Agent- tionary Art.” In J. Romero and P. Machado, eds. The Art Based Composition.” In M. Giacobini, et al., eds. of Artificial Evolution.Berlin:Springer,pp.311–332. Applications of Evolutionary Computing.Berlin: Gibson, J. J. 1986. The Ecological Approach to Visual Springer, pp. 609–616. Perception.Hillsdale,NewJersey:LawrenceErlbaum. Uozumi, Y., M. Takahashi, and R. Kobayashi. 2008. Johnson, S. B. 2004. Emergence: The Connected Lives of “A Musical Framework with Swarming Robots.” In Ants, Brains, Cities, and Software.NewYork:Scribner. R. Kronland-Martinet, S. Ystad, and K. Jensen, eds. Lakoff, G., and M. Johnson. 1980. Metaphors We Live By. Computer Music Modeling and Retrieval: Sense of Chicago, Illinois: University of Chicago Press. Sounds.Berlin:Springer,pp.360–367. Maeder, M., ed. 2010. Milieux Sonores/Klangliche Milieus: Xenakis, I. 1992. Formalized Music: Thought and Mathe- Klang, Raum und Virtualitat¨ .Bielefeld,Germany: matics in Composition.Hillsdale,NewYork:Pendragon transcript Verlag. Press.

62 Computer Music Journal Figure 7. Three images Figure 8. Three images from Flowspace. from Stocos. (Correspon- (Corresponding URLs are ding URLs are given in the given in the Appendix.) Appendix.)

Figure 7

Figure 8

Appendix: Online Resources http://vimeo.com/20158277 http://vimeo.com/14816324 The Web site dedicated to the artistic projects and Stocos videos (see Figure 8): software described in this article is available online at http://swarms.cc. Videos depicting the art works http://vimeo.com/39332848 are available online to complement the descriptions http://vimeo.com/39769944 in this article; see the following URLs. http://vimeo.com/39332770 Flowspace videos (see Figure 7): All URLs were verified in August 2014. http://vimeo.com/15294032

Schacher et al. 63 † Eric Lyon,∗ R. Benjamin Knapp, Compositional and and Gascia Ouzounian∗∗ ∗School of Performing Arts Performance Mapping in Virginia Polytechnic Institute and State University Computer Chamber Music: 195 Alumni Mall Blacksburg, Virginia 24061, USA †Institute for Creativity, Arts, A Case Study and Technology Virginia Polytechnic Institute and State University 190 Alumni Mall Blacksburg, Virginia 24060, USA ericlyon, benknapp @vt.edu { } ∗∗Sonic Arts Research Centre Queen’s University Belfast University Road Belfast, BT7 1NN Northern Ireland, UK [email protected]

Abstract: The mapping problem is inherent to digital musical instruments (DMIs), which require, at the very least, an association between physical gestures and digital synthesis algorithms to transform human bodily performance into sound. This article considers the DMI mapping problem in the context of the creation and performance of a heterogeneous computer chamber music piece, a trio for violin, biosensors, and computer. Our discussion situates the DMI mapping problem within the broader set of interdependent musical interaction issues that surfaced during the composition and rehearsal of the trio. Through descriptions of the development of the piece, development of the hardware and software interfaces, lessons learned through rehearsal, and self-reporting by the participants, the rich musical possibilities and technical challenges of the integration of digital musical instruments into computer chamber music are demonstrated.

The Biomuse Trio was formed in 2008 to create practices in addition to the hardware and software of and perform computer chamber music integrating the DMI. Certain aspects of instruments come into biosignals (Ouzounian 2012). The first composition high relief when tested in the crucible of musical resulting from this work was the Trio for Violin, creativity, and it is those aspects that we hope to Biosensors, and Computer by Eric Lyon, which illuminate in this article. has been performed at the 2009 Conference on New Interfaces for Musical Expression (NIME), along with performances in London, Dublin, Atlanta, and New York City, at venues such as ISSUE Project Room Digital Musical Instruments and Diapason Gallery. At the outset, the challenge of composing for digital musical instruments (DMIs), In New Digital Musical Instruments: Control and with their inherent mapping problems, was a central Interaction Beyond the Keyboard, Miranda and focus in our rehearsals and discussions. As the Wanderley (2006, p. 3) define a digital musical in- Biomuse Trio began its work, it became clear strument as “an instrument that contains a control that the anticipated DMI mapping problems were surface (also referred to as a gestural or perfor- embedded within broader questions that required a mance controller, an input device, or a hardware consideration of our compositional and performance interface) and a sound generation unit. Both units are independent modules related to each other by Computer Music Journal, 38:3, pp. 64–75, Fall 2014 mapping strategies.” The history of DMIs goes back doi:10.1162/COMJ a 00257 at least as far as the GROOVE system (Mathews c ⃝ 2014 Massachusetts Institute of Technology. and Moore 1970), with pre-digital precursors such

64 Computer Music Journal as the Theremin of 1919 already displaying the key kinetic control. The Biomuse has been used since property of separating the control interface from the 1989 in numerous musical performances and com- sound generation unit. Ever since the 1980s, with positions, such as Tibet by Atau Tanaka and Ben the introduction of the MIDI protocol, commercial Knapp (2002). DMIs such as the Yamaha DX7 proliferated and were disseminated widely. Most commercial DMIs have been conservative in design, but the occa- Introducing the Biomuse Trio sional experimental interface may be found, such as Yamaha’s Miburi, a wearable MIDI controller. The The Biomuse Trio consists of Ben Knapp, Biomuse laptop computer and related digital devices, such as designer and biosignal performer; Eric Lyon, com- tablets and smart phones, are also quite prevalent as poser and audio programmer; and Gascia Ouzounian, DMIs. violinist. Besides having these core competencies Whereas commercial DMIs tend to be limited by required for performance in the trio, Knapp is an the conservative outlook of their design, the laptop accomplished keyboard player and Ouzounian is a as DMI, despite full flexibility of sound generation, musicologist and composer. Lyon received early in- is characterized by severe limitations in perfor- strumental training as a violinist. Thus, in addition mance intimacy, a criterion for computer music to participation as creators and performers, each of performance discussed by F. R. Moore (1988). A third us was well situated to participate in, and evaluate, stream of DMIs, developed since the 1990s, involves the project from a distinctly different interpretive innovative focus on the performance interface and perspective. A DMI is characterized by its separation has centered around the NIME community since of gestural interface and sound production engine the early 21st century. Such DMIs can afford both (Malloch et al. 2006). This characteristic divide was performance intimacy and new potentials for mu- certainly present for our trio, requiring ongoing sical interaction. The Biomuse (Knapp and Lusted mapping processes. Additional mapping processes 1990) falls into this category. were required between composer and performers, between software interfaces to both the Biomuse sensors and audio sound production, and between Biosignals and Musical Interaction the performance gestures of the Biomuse performer and those of the violinist. The use of biosignals for musical interaction may be found in a subset of DMIs. Miguel Ortiz (2014) has compiled a detailed breakdown of various biosignals The Compositional Framework used for musical performance. Ortiz provides a common-practice definition of biosignals as “signals The ensemble of Biomuse, laptop, and violin pre- that are bioelectric in nature, and that manifest sented an ideal laboratory for gestural DMI-based as the change in electrical potential across a spe- composition. The pairing of violin and Biomuse cialized tissue or organ in a living organism.” An provided opportunities to test the expressive poten- important precursor to biosignal-based DMIs is tial of a DMI in a chamber music context with a Alvin Lucier’s 1965 composition Music for Solo highly expressive classical music instrument that Performer,inwhichelectroencephalographicsig- has a centuries-old performance practice. The laptop nals from the performer drive loudspeakers at might have been more fully integrated into the very low frequencies, which then excite acoustic performance arena, although in our rehearsals, we percussion instruments (Lucier 1976). The Biomuse found that potentials of laptop gesture and expres- is a biosignal-based DMI, affording control through sivity were relatively limited compared with that of biosignals using electromyography, electroen- the violin and Biomuse. cephalography, electrocardiography, and electroocu- An important compositional constraint was that lography, as well as through accelerometers for all sounds produced by the DMI should articulate

Lyon et al. 65 sounds sampled from the violin during performance. digital signal processing (DSP), such as recording Earlier chamber music compositions by Lyon, and editing violin samples, initiating changes to the such as Introduction and Allegro and Trio for current DSP configuration, and initiating sample Flute, Clarinet, and Computer,demonstratedthe playback. The role of the violinist was to perform richness and sonic variability possible from using notated music, some of which would provide live-sampled acoustic instruments (Lyon 2008). In samples for the Biomuse to later articulate. The only the present work, additionally using the Biomuse sensor deployed on the violin was a DPA clip-on to articulate violin sounds would bring the two microphone. instruments into greater timbral proximity than if synthetic or prerecorded sounds were used instead. Finally, it was our intention that the composition The Biomuse be fixed and notated to the greatest extent possible, as the ability to accurately recreate our performances The Biomuse consists of a set of on-body sensor would serve as a test of the reliability and stability bands, such as armbands and waistbands (Knapp of our system. Essentially, we would build into the and Lusted 1990). Each individual sensor band is music audience-observable indicators of musical placed on a part of the body, and it remains there success or failure, ones that were just as clear as, say, for the duration of the performance. The integration an elegantly executed piano flourish or a flubbed of multiple sensor bands onto the body of a single brass note. performer constitutes one of the great advantages of the Biomuse. At least ten degrees of freedom may be continuously controlled, providing a rich data The DMI: An Overview stream for controlling a DMI. For the Trio,weused the following combination of accelerometer (ACC) We begin with mapping aspects of the DMI and and electromyogram (EMG) sensors: work our way outward to relationships that emerged in the interface across the DMI, in compositional Left Forearm Front EMG management, and, finally, in performance of the r Left Forearm Back EMG composition with the DMI. The DMI comprises a r Left Forearm Pitch ACC variable configuration integrating the violin, the r Left Forearm Roll ACC Biomuse, software synthesis on the laptop, and, r Right Forearm Front EMG to a much lesser extent, the laptop itself as a r Right Forearm Back EMG performance surface. We do not subscribe to the r Right Forearm Pitch ACC notion that the laptop is an inherently uninteresting r Right Forearm Roll ACC performance interface. An earlier composition by r Lyon, Introduction and Allegro,pairedalaptop EMG sensors record electrical activity during soloist with an instrumental ensemble, utilizing muscle contractions, and accelerometers measure laptop writing that was highly virtuosic, a virtuosity acceleration. that was easily grasped by the audience (Lyon The initial instrumental strategy in composing 2008). It was therefore our intention at the outset for the Biomuse was not to determine specific that the laptop be an equal partner in the gestural sensors or gestures in advance, but rather to specify performance paradigm of the Trio. In our initial in the score the desired musical result of a Biomuse rehearsals of compositional sketches, however, performance gesture. The mapping of Biomuse the gestural interactions between Biomuse and gestures to musical results was determined through violin proved by far the most compelling, so the experimentation in rehearsal. Each gesture involved laptop was shortly dropped as a primary gesture two stages of mapping: first, the selection of one performance instrument. The role of the laptop or more sensors to accomplish a musical task, and performer became primarily management of the second, the choreography of the bodily motion

66 Computer Music Journal Figure 1. An excerpt from Trio for Violin, Biosensors, and Computer,showing Biomuse gestural notations.

that activates these sensors. The first stage was (www.infomus.org/eyesweb ita.php), with simple concerned with reaching a particular value, or filtering to stabilize the output. EyesWeb sends motion between values, for one or more sensors. Open Sound Control (OSC) messages over Ethernet The second stage involved deciding on the specific to a laptop running Max/MSP. The OSC messages gesture to properly activate the sensor, considering received by Max/MSP are mapped in two stages. The the expressive content of the gesture as well as its first is a linear calibration stage. The full range of physical efficacy. Finally, choreography was designed a sensor band might not be comfortably accessible to connect successive gestures into musical phrases. to the performer, so in the calibration stage, the The design of the gesture would often include its comfortable minimum and maximum values are preparation—for example, the position of a wrist measured and stored. Once both these values have prior to turning it, or the relaxation of a muscle, prior been acquired, the range between comfortable to tensing it. Once a gesture, or series of gestures, minimum and maximum is linearly mapped from proved successful in rehearsal, it was notated into zero to one. Once calibrated, the sensors can be the score (see Figure 1). mapped in any number of ways to sound-production and sound-processing algorithms. The sensors are mapped either singly or in combination with other Interfacing to Max/MSP sensors, and the mappings are sometimes nonlinear. This great flexibility in mapping sensors to a large The DMI consists primarily of an interface between variety of sound-producing strategies constitutes the Biomuse and the laptop. Output from the various one of the great advantages of DMIs over traditional Biomuse sensors is linearly mapped in EyesWeb acoustic instruments. Because all sound produced by

Lyon et al. 67 Figure 2. The opening bars of Trio for Violin, Biosensors, and Computer.

the DMI is based on recordings of the violin played There are two aspects to the mapping. First is the by Ouzounian that were made during performance, choice of sensor, in this case the forearm EMG band, the violinist herself must be considered as part of which triggers the chord on exceeding a threshold the DMI. This is particularly the case because, in of muscle tension hard-coded into the Max patch certain passages, the DMI does not directly produce that forms the audio software side of the DMI. The sound but rather modifies the sound of the violinist second aspect of the mapping is the choreography in real time. of the gesture, which was devised by the biosensor performer (Knapp, from here on described as the “biomusician”). The gesture is a “throw” away Mapping Gesture to Sound from the biomusician’s body toward the violinist in which his hand opens at the end of the gesture, as In this section we will discuss two gestures from if releasing the sound to her. There are many ways the Trio in detail, and then describe several other in which the biomusician could produce the desired gestures that involved striking mapping strategies. spike in EMG muscle tension, including gestures that would be largely invisible to the audience. Thus the articulation of the gesture as an open hand A Gesture from the First Movement release is an aesthetic decision to convey affect to the audience, much like the gestures of a conductor The first gesture performed on the DMI is heard in (who, given a properly rehearsed orchestra, could measure 3 of the first movement (see Figure 2). It is produce a similar sonic effect from mechanical notated as a “left arm throw (forearm front EMG),” beating as from flamboyant gesticulation). Although where the actual sound is an aggregation of several the combination of sensor choice, threshold setting, transpositions of the sampled chord. and gesture choreography accurately describes

68 Computer Music Journal this particular gestural moment, the full gestural The Max subpatch that routes the biosignal phraseology is more accurately described starting controls is shown in Figure 4. The score excerpt from the opening of the piece: The violinist first puts notating this gesture is shown in Figure 5. A system sound into the space. The laptop performer then of gating is used such that control signals are captures the sound to computer, and hands it off to only routed to a given synthesis algorithm when the biomusician. Finally the biomusician gives the that algorithm has been activated in the piece. sound back to the violinist in a multiplied fashion. Given the complex parameter space, it was deemed This integral analysis of the gesture underscores unrealistic to notate simultaneous motion of limbs our view that an instrument, including a DMI, is within the control space. Instead, the biomusician not truly a musical instrument until one takes into is instructed to improvise a passage, giving him account not just the affordances of the instrument, the freedom to explore the parameter space at but the actual ways that performers transform those will. This passage was uniquely enabled by the affordances into music. integral control of multiple sensor bands operated simultaneously by a single human body. The continuous, coherent, simultaneous manipulation of this parameter space would not be possible by A Gesture from the Second Movement trying to manipulate a bank of sliders; this was a truly idiomatic use of the Biomuse. It is notable that In the first movement there was a compositional our principle of compositional control broke down focus on relatively basic uses of the Biomuse, which in favor of improvisation at precisely the moment were not always the most idiomatic, as it turned out. where we discovered this maximally idiomatic For example, the use of EMG as a triggering sensor passage. Experimentation with the DMI overrode proved reliable for single triggers, but unreliable our preconceptions of how the piece should work. when articulated multiple times in sequence. In order to strenuously test our systems, all biosensor motions were fully notated, and there was no Further Gesture Mappings room for improvisation. In the second movement, we began to introduce mappings in which the Many more gestures were mapped onto the DMI. biomusician controlled a relatively large number We briefly mention here a few that we considered of parameters simultaneously. In a digital signal of particular interest. The entire score can be processing configuration affectionately referred to downloaded at this article’s Supplementary Content as the “pleasure ball” (the basic hand positions of page, (http://www.mitpressjournals.org/doi/suppl/ the biomusician are reminiscent of Woody Allen’s 10.1162/COMJ a 00257) and the two movements character holding the “orgasmic orb” in the 1973 of the Trio can be viewed online at youtube.com/ film Sleeper), the biomusician simultaneously watch?v N1dBo7V4dzA and youtube.com/watch? controls six parameters that modify a live-captured v nlyJK80lYuE.= In measure 23 (1:28 into the first of sample of a violin crunch tone. the= two videos), the biomusician initiates a granular The synthesis patch is shown in Figure 3. The texture by tensing both forearms. Part of the gesture parameter controls are as follows: the pitch (in the requires that the hands be spread apart from each sense of motion) of the left forearm accelerometer other. The next part of the gesture involves gradually controls filter speed; the pitch of the right forearm bringing the hands together, changing the density of controls frequency deviation, the roll of the right the granular passage. Another hand clench causes forearm accelerometer controls granular transpo- the granular sample to be played backwards. A sition; the roll of the left forearm controls tuned final opening of the hands gradually changes the resonance; the right forearm EMG combined con- spectral “peakiness” (i.e., the amount of spectral trols the low-pass filter; and the left forearm EMG variance) of the granular texture. The accordion-like controls the amount of filter feedback. two-handed nature of the biomusician’s gesture

Lyon et al. 69 Figure 3. The multi- parametric synthesis subpatch affords correlated control for the biomusician.

70 Computer Music Journal Figure 4. The controller subpatch displays mappings from on-body sensors to synthesis parameters.

seemed particularly apropos to the “stretching and biomusician. The actual gesture that the audience shrinking” sound of the musical passage. sees is a light lift of the hand, which is symbolic of In measure 55 (3:47 into the first video), a right- the general lightness and upper-midrange tessitura hand EMG grab is used to lock an audio-buffer of the passage. playback munger into a fixed loop (the munger In measure 89 (0:08 into the second video), the is an algorithmic reordering of the contents of a start of the second movement, the violin is sampled sample buffer). The gesture of grabbing seemed quite into a comb filter with a right-hand clench. We appropriate to the musical idea of locking a sound found it particularly interesting to use a grabbing loop into place. motion to symbolize capturing a sound. Starting in measure 59 (4:05 into the first video), a As these brief examples demonstrate, we were sequence of chords is traversed by either right or left as much concerned with the expressive potential forearm EMG, with the choice of arm given to the of the gesture as we were with the effective use

Lyon et al. 71 Figure 5. Notation of the “pleasure ball” passage.

of a sensor to produce specified musical effects. It experience from both primary gestural performers was crucial that the biomusician be as comfortable of the Biomuse Trio. For Knapp, there were two as possible with these mappings, so these creative main areas of reflection: first, approaching the decisions on gesture choreography were largely design of the DMI from an engineering standpoint, made by the biomusician in consultation with the and second, the actual experience of performing composer, and also often in consultation with the the DMI in a chamber music context. The design violinist about how the expressivity of the gesture began on a whiteboard with a list of sensors, and might complement her corresponding musical abstract gestural ideas. At this stage, the question passage. was, “What can we use for control, and how do we sense these controllers?” We would start with ataxonomyofrudimentarygesturesandthen The Performance Experience build more complex gestures on top of them. For example, in considering the rotation of an arm as Acrucialelementintheevaluationofaninstrument acontrollinggesture,weaskedwhatsensorwould comes from its use in performance. In this section, be most appropriate, what was the range of values we summarize observations about the rehearsal we could extract from the sensor, and what was the

72 Computer Music Journal consistency of the calibration. From that point we At the performance level, Knapp observed the could build more complex gestures, such as sampling deep nature of the collaboration, which extended the current position of the arm, redefining that point directly from our work methods to the nature of the as zero, and generating a new mapping from the new DMI performance. In our rehearsals, despite clear zero to the previously calibrated maximum. That divisions of labor regarding hardware management allowed us to use arm rotation without requiring and performance, software management and per- the arm to first be unnaturally moved all the way formance, and violin performance, each member to one side before performing the gesture. After of the trio was constantly questioning aspects of coding these chains of calculations, we would test the ensemble beyond his or her own role. Constant them to see how well they performed. If there discussion was a central feature of all our rehearsals. was sufficient reliability in sensing the gesture, we And in the DMI, there was a seamless line from added it to our vocabulary of useful gestures for the muscle to skin, to sensor, to EyesWeb, to Max, to DMI. digital audio, to speakers, and in a converging line Another aspect of the design concerned the from violin to microphone, to analog-to-digital con- mappings on both hardware and software. Every verters, to Max, and then over to the biomusician’s stage of mapping adds a layer of complexity, and body again. Knapp reports feeling this connection even before being tested in performance, our DMI profoundly in performance, with Ouzounian tossing involved mapping from a sensor to a control surface, sounds to Knapp, which he would then sculpt as if mapping from a control surface to one or more reaching in and pulling sounds from the violin. parameters of sound, and finally mapping from the Ouzounian, reporting from the perspective of parameters to the sound itself. The mappings of an instrumental performer, found the interaction sensors on the EyesWeb side are linear, with the fascinating in different ways. Having not previ- possible addition of simple stabilization filtering. On ously worked with physiological interfaces in a the Max side, the mappings might be either linear performance context, she found it wonderful to be or non-linear, such as in the case of live calibration involved in conversations around gesture, expres- discussed earlier. And prior to this mapping chain are sion, movement, and calibration when the piece questions of the Biomuse interface itself: reaction was in development. In contrast to the violin, she time, latency, and stability, such as how steadily the found the biomusician quite limited in the range of performer could hold a particular level of muscle gestures he could perform in the notated sections. tension. At the same time, she and the biomusician had Knapp further observed that the kinds of complex to find ways of interacting in a chamber music multi-sensor mappings we were creating at the context, so the main question was how to play with software level raised a hardware design question someone, respond, and initiate different musical of the trade-off between modularity and possible ideas when the person’s instrument is, essentially, integration of multiple sensors into a single band. A his or her physiology. Ouzounian found that playing key motivator here was the perceived coherence of with a biomusician required a different kind of the sonic space. It is not just how many controllers sensitivity because of the unpredictability of human are used, but how they are mapped. A pianist physiology. There was a much higher risk of fail- playing six different notes is not a single coherent ure compared with other forms of computer-based sonic space, but a biomusician simultaneously interactive music that she had been involved with. manipulating six interacting parameters could well At the same time, in the most successful moments, be coherent. (Consider that a pianist can trigger each Ouzounian found a degree of musical intimacy that note independently of the others, whereas most of a was quite unique. Because the DMI was not a fixed biomusician’s physical gestures will simultaneously instrument, but a constantly changing environment, affect multiple sound synthesis parameters in a she needed to learn what kinds of violin sounds, continuous manner, constrained and integrated by articulations, or timbres might be most fruitful in a the sensor-on-human-body system.) particular section of the piece. Above all, she found

Lyon et al. 73 it exciting to perform with someone with an expert One audience member observed being struck understanding of physiology and of physiological by the “naturalness” of the movements of the response in performance contexts. biomusician, and correspondingly, she did not notice Commenting on the rehearsal process, Ouzounian any particular difficulty or virtuosity. In fact the observed that developing music with biosensors is Biomuse part is quite difficult to perform, requiring much more laborious compared with chamber constant intense concentration and precise control music situations involving traditional instruments. of body motions. It is a virtuoso performance. The The group felt truly experimental in the sense that opaqueness of virtuosity to the observer might be every new gesture or idea had to be investigated taken as a flaw in the DMI; we do not view it as such, through multiple lenses and would succeed or fail however. The virtuosity level of a composition tends in multiple, unpredictable ways. There was a high to degrade over time. For example, the virtuosity degree of learning that came with being involved in embedded in NiccoloPaganini’s` Caprices (Paganini the group, which was found to be exciting, and also 1973 [1819]) was truly extraordinary when they were sometimes terrifying. first composed in the early 19th century. Since the It should be clear from this self-reporting by 20th century, though, thousands of young violinists the performers that the biosensor-based DMI we trained in conservatories have routinely been able developed afforded strikingly different musical to manage the techniques required by the Caprices. opportunities than would be possible using acoustic Virtuosity is not that important to us. We would instrumentation, or even computer-augmented prefer to move the audience with expressivity, rather instruments. than dazzle it with the show of technique.

Audience Reactions Comparison to Improvisational In presenting the work to an audience, we needed Performance Strategies to decide whether the biomusician should wear the sensors visibly or hide them under a suit jacket. We Akeyelementofourapproachhasbeentheuseofa chose the latter, as we wished to present the human fixed score that notates not only traditional elements body, rather than the technology, as the primary such as notes and rhythm, but also DMI-specific instrument. As a result, audience members did not controls. Improvisational passages are incorporated see the full extent of the DMI, though in any case into the compositional structure, but in every case they would not have seen the Max patch, since are constrained by both the kind of violin sounds showing the patch would have been an unacceptable specified as input, and the audio DSP and DMI con- distraction from the human musical interactions trol configuration specified for the passage. Above onstage. all, the relatively fixed nature of the composition Audience members generally understood quickly required repetition and testing of gestures. Gestures that the body of the biomusician was directly that initially did not work were either reassigned performing and controlling sound. This was largely new sensors, practiced until they did work, or re- due to early, directly observable analogies between moved from the composition. The score provides a performed gesture and resulting sound. Once this benchmark of accuracy for evaluating musical ges- link was established, even the performance of tures in both rehearsal and performance that is not complex sounds driven with multiple sensors was available for fully improvised performances. We be- accepted as plausible, despite the audience’s not lieve that this approach is highly complementary to necessarily understanding the exact relationships improvisational uses of DMIs that focus on expres- between aspects of the performer motion in its sivity, as in Atau Tanaka’s Biomuse performances entirety, on the one hand, and a complex synthesis (Tanaka 1993), and the expansion of sonic variety algorithm, on the other. for established instrumental performers, such as

74 Computer Music Journal the collaboration by Marsh and Paradis (2006) with Marsh, R., and M. Paradis. 2006. “Interactive Live improvisers Barry Guy and Jos Zwaanenburg. Composition Using Advanced Real-Time Digital Transformation Techniques.” In Proceedings of the COST287-ConGAS Second International Symposium Conclusion on Gesture Interfaces for Multimedia Systems,pp.48– 51. In this case study we discussed mapping issues Mathews, M. V., and F. R. Moore. 1970. “GROOVE: that arose in the composition and rehearsal of Eric AProgramtoCompose,Store,andEditFunctions of Time.” Communications of the ACM 13(12):715– Lyon’s Trio for Violin, Biosensors, and Computer. 721. We argued that mapping for DMIs is not just a Miranda, E. R., and M. M. Wanderley. 2006. New question of assignment of controllers to parameters. Digital Musical Instruments: Control and Interac- The performance context and specific musical tion Beyond the Keyboard.Middleton,Wisconsin: uses dictate a much broader understanding of A-R Editions. the mapping question. It is only through musical Moore, F. R. 1988. “The Dysfunctions of MIDI.” Computer exercise of a DMI that we can discover unique Music Journal 12(1):19–28. affordances that might significantly broaden the Ortiz, M. 2014. “A Brief History of Biosignal-Driven expressive potential of computer chamber music. Art: From Biofeedback to Biophysical Perfor- mance.” Available online at cec.sonus.ca/econtact/ 14 2/ortiz biofeedback.html. Accessed 23 February References 2014. Ouzounian, G. 2012. “The Biomuse Trio in Conversation: Knapp, R. B., and H. Lusted. 1990. “A Bioelectric Con- An Interview with R. Benjamin Knapp and Eric troller for Computer Music Applications.” Computer Lyon.” Available online at http://cec.sonus.ca/econtact/ Music Journal 14(1):42–47. 14 2/ouzounian biomuse.html. Accessed 15 March Lucier, A. 1976. “Statement on Music for a Solo Per- 2013. former.” In D. Rosenboom, ed. Biofeedback and the Paganini, N. 1973 (1819). 24 Caprices for Solo Violin.I. Arts, Results of Early Experiments.Vancouver,British Galamian, ed. New York: IMC. Columbia: Aesthetic Research Centre, pp. 60–61. Tanaka, A. 1993. “Musical Technical Issues in Using Lyon, E. 2008. “A Computer Music Retrospective.” Interactive Instrument Technology with Application Organised Sound 13:209–216. to the Biomuse.” In Proceedings of the International Malloch, J., et al. 2006. “Towards a New Conceptual Computer Music Conference,pp.124–126. Framework for Digital Musical Instruments.” In Tanaka, A. and R. B. Knapp. 2002. “Multimodal Interaction Proceedings of the International Conference on Digital in Music Using the Electromyogram and Relative Audio Effects,pp.49–52. Position Sensing.” In Proceedings of the Conference on New Interfaces for Musical Expression,pp.1–6.

Lyon et al. 75 Reviews

[Editor’s note: Selected reviews of mathematics and computation, are posted on the Web at applied to musical problems, were computermusicjournal.org (click on presented along with paper and poster the Reviews tab). In some cases, they sessions. The 20 paper presentations are either unpublished in the Journal were grouped into nine sessions: itself or published in an abbreviated Analytical Algorithms, Generalized form in the Journal.] Tonnetze, Tone Systems and Interval Content, Musical Performance: The- ory and Analysis, Harmonic Spaces, Corpus Studies of Harmony, Stur- Events mian Words, Group Actions on Or- dered Sets and Style, and Creativity. The nine posters were grouped into table was organized by Guerino Maz- MCM 2013: The Fourth Inter- two sessions. Unfortunately, only the zola (University of Minnesota), David national Conference on Math- abstracts for the poster sessions ap- Clampitt (Ohio State University), pear in the conference’s proceedings. ematics and Computation in Thomas Noll (Escola Superior de Overall, both the paper and poster Musica´ de Catalunya), Thomas Fiore Music sessions were characterized by an (University of Michigan-Dearborn), The Fourth International Conference increased use of computational meth- Emmanuel Amiot (CPGE Perpignan), on Mathematics and Computation ods to analyze large musical corpora. and Anja Volk (Utrecht University). in Music, 12–14 June 2013, McGill This approach raises questions about The participants agreed on the point University, Montreal, Canada. Infor- the relationships between research that the intersection between math- mation on the conference is available methods in mathematics and music, ematics and music is still, today, an at www.music.mcgill.ca/mcm2013/. and other approaches that are more uncommon idea for a majority of stu- common in the field of music infor- dents specialized in either of the two Reviewed by Louis Bigo mation retrieval, such as machine fields. Nevertheless, they find it to be Paris, France learning and statistical analyses of an exciting prospect, full of promise. data sets. The variety of topics in the Some participants lamented the lack The fourth International Conference MCM conferences since 2007 clearly of books accessible to nonspecialists on Mathematics and Computation shows a plethora of approaches within in the mathematical theory of music, in Music (MCM 2013) took place the math and music communities, as with the notable exception of a few last summer at McGill University in well as the evolution of certain topics. nonspecialist texts such as Music: Montreal. Since its beginning in 2007, For example, there seems to be an AMathematicalOfferingby Dave the biennial conference, organized by increasing number of works related Benson, or The Geometry of Musical the Society of Mathematics and to word theory since 2011. The 2013 Rhythm by Godfried Toussaint. At Music (SMCM) has taken place on edition dedicated a whole session to present there is no comprehensive the two sides of the Atlantic in Sturmian words (an infinitely long textbook on mathematical models in alternation. Previous conferences sequence of words), investigating music theory, analysis, and composi- were held in Berlin (2007), New both scale and rhythmic applications. tion, from the study of temperament Haven, Connecticut (2009), and Paris Additionally, Gilles Baroin organized to discrete Fourier transforms and (2011). Thanks to the low registration an exceptional multimedia session chordal and rhythmic classifications. fee, the conference is easily accessible about inherent pedagogical problems The Springer Computational Music to everyone, especially students. in math and musical research. Science series, co-edited by Guerino The conference was hosted jointly Pedagogy was, indeed, one of the Mazzola and Moreno Andreatta, is by the Schulich School of Music at major themes of this edition. A panel planning to soon fill this gap with McGill University, and by the Cen- discussion entitled “Mathematical atextbook,andwehopetolearn tre for Interdisciplinary Research in Music Theory in Academia: Its Pres- more about it at the next MCM Music Media and Technology (CIR- ence, Role and Objectives in Depart- conference. MMT). Contributions in the field ments of Mathematics, Music, and Aconcerttookplaceattheend Computer Science” took place at the of the second day at the CIRMMT in end of the first day and was chaired by the New Music Building. Composi- doi:10.1162/COMJ r 00259 Mariana Montiel. A discussion round tions by Elliott Carter, Pierre Boulez,

76 Computer Music Journal Preston Beebe, Luciano Berio, and and the variety of topics related to Luis Naon´ were played by McGill mathematical and musical research. University students. The audience was particularly impressed by vio- linist Marjolaine Lambert, who per- formed Antheme` II by Pierre Boulez. Recordings Anewinterfaceforpercussion(the SpectraSurface) was used by Zachary Hale for the piece Unsounding Ob- Matthew Burtner: Noise Plays jects, composed by Preston Beebe. Burtner The performance, both inventive and impressive, was very well received. Compact disc, 2013, Innova 871; Following the highly successful Innova Recordings, ACF, 332 Min- MCM 2011 conference in Paris was nesota Street #E-145, St. Paul, adelicatechallenge.Organizedby Minnesota 55101, USA; tele- phone: ( 1-651) 251-2823; electronic l’Institut de Recherche et Coordina- + (UPIC) system at the Centre d’Etudes tion Acoustique/Musique (IRCAM), mail [email protected]; de Mathematique´ et Automatique the 2011 conference included a num- http://www.innova.mu/. Musicales (CEMAMu). His work at ber of exceptional side events. Of these two institutions profoundly particular note was the public dis- Reviewed by Ross Feller impacted his approach to computer cussion between two well-known Gambier, Ohio, USA music composition. He was able to French figures—mathematician (and use technological tools that were Alaskan-born composer Matthew Fields Medalist) Alain Connes and unavailable at most other institu- Burtner specializes in chamber music composer Pierre Boulez—about cre- tions. This was especially true with and interactive new media. He is ativity in music and mathematics. the UPIC system, which provided the inventor of the Metasaxophone, Furthermore, the Paris conference him with a real-time composition the Mobile Interactive Computer was paired with an important exhibi- environment to transform graphics Ensemble (MICE), and the Network- tion on the topic “Mathematics and into sounds. It also affected the way Operational Mobile Applied Digital Arts” that took place at the Palais de he composed his scores for acoustic System (NOMADS). For over a decade la Decouverte´ in Paris, with concerts instruments, providing them with an he has worked with the San Diego– at the end of each day. Although iconic sense of time. based ensemble Noise and with the 2013 concert at CIRMMT was According to the liner notes for Innova Recordings. The present com- strongly appreciated, it was the only this compact disc, “Burtner’s interest pact disc harnesses these two forces, one of the conference, and the orga- in the whole world of sound origi- joined together in three electro- nizers abandoned the idea of having nated from his childhood experience acoustic and electroacoustic-inspired keynote lecturers and hosting special growing up in Alaska where the works in which noise features promi- events. The 2011 conference was thus snow, wind, and sea create a ceaseless nently. The Noise ensemble is an unquestionably richer in density. The soundscape.” These elements served instrumentally mixed sextet with professionalism and cheerfulness of as the background for an ecoacoustic flute, violin, cello, guitar, piano, and the 2013 organization team, however, work entitled Snowprints (2001), the percussion. Here they are joined with made this event a complete success. second piece on this disc. Burtner the composer on the saxophone and The conference ended with a clos- creates a sense of place by mixing with an additional member on the ing reception on the last day. Most of recordings of snow and acoustic in- computer. the participants attended the closing struments. The recordings of snow, Burtner worked in Paris at event, which enabled a number of taken during different conditions and l’Institut de Recherche et Coordina- exchanges and projects for future in- at different times of day, were uti- tion Acoustique/Musique (IRCAM) ternational collaborations. The next lized in the creation of a fluctuating and with Iannis Xenakis’s UnitePoly-´ edition of the conference, planned noise bed that acoustically supports agogique Informatique CEMAMu for 2015 at Queen Mary University and surrounds the instruments. Each of London, will surely confirm the instrument has a computer-generated richness of this interdisciplinary field doi:10.1162/COMJ r 00258 counterpart, produced with physical

Recordings 77 modeling and granular synthesis The first piece, Polyrhythmicana laid onto this, one hears a sustained, techniques. (2002), lays out Burtner’s composi- fan-like whooshing noise, or like hiss Snowprints begins in a spec- tional terrain, especially with respect from a very slow tape deck without tral manner with a low-frequency to his approach to rhythm, in a five- noise reduction. Melody Triangles, fundamental that gives way to high- movement, 15-min composition. For the final movement, initially uti- frequency overtones. When he first this piece, the composer designed lizes unison rhythms and pitch lines. began working with computer music, acomputerinstrumentcalledthe Gradually each part separates as the Burtner saw it as a way to create Polyrhythmicon, based on the Rhyth- texture thickens, giving the music different sound resources, produced micon, an instrument made by Henry apronouncedsenseofurgency.The at the spectral level, which could Cowell and Leon Theremin in the final slow tremolo gesture provides blend with instrumental timbres. 1920s. The instrumentalists in this closure, yet given the previous 14 Snowprints amply demonstrates this piece perform with click tracks that minutes, seems like an arbitrary close principle. The spectral treatment follow various polytemporal tra- to a piece that could easily have had is occasionally disrupted with dis- jectories. The listener encounters many more sections or movements. sonant breaks in the texture, and noise in two respects. First, Burt- Each movement in Polyrhythmi- embellished with breath tones and ner produces metaphorical noise as cana maps out a specific timbral microtones performed by the en- he introduces small perturbations and/or rhythmic territory through semble. Both the electronic and the between the constantly changing fluctuating accent and stress pat- acoustic sounds are shaped by synthe- tempi. And second, the instruments terns, as well as through timbral sis and filtering techniques. One gets are wrapped in tinfoil in order to modifications such as sul ponticello. the sense that the constant hiss/noise create sympathetic resonance. This The composition clearly owes a debt drone serves as a frame from which kind of sonic noise is a common el- to Henry Cowell but also resembles the music emerges. It also colors the ement in African drumming and the the important work done by Con- music in the sense that it conjures the mbira. lon Nancarrow, except that Burtner experience of listening to recorded Metal XY,thefirstmovementof composes his expanding and con- music with a low dynamic range (LPs, Polyrhythmicana, can be heard as a tracting pulse streams onto a variety cassettes, etc.). The intentional use of shock to the system, or a wake-up of instruments with noise added for noise in this case presents the listener call to listen. One encounters loud seasoning. Burtner’s use of noise does with an intriguing contradiction, es- bursts of noise along with an not merely encompass the sonic, but pecially if the listener is a composer instrumental ostinati, each evolv- also includes semiotic noise in which or sound engineer and routinely filters ing at a different rate of speed. In various associations (like the passing out such noises. Ultimately, I would the second movement, Split/Joined of time represented by a ticking clock say that Burtner conjures a sense of Diamonds (in Wood),woodblocks sound) or meanings are suggested, nostalgia without hinting at loss or and other percussion instruments are subverted, and masked. melancholy, two concepts that are set against sustaining instruments The final piece on this disc, normally associated with it. At one such as the flute. The separate tempi (dis)Sensus (2008), is an explosive, point in the piece Burtner uses a sim- are clearly in evidence, thanks to the multi-movement piece that explores ple repetitive pattern that increases composer’s consistent use of attacks principles of formal contrast, dissent, in speed until it begins to sound as if and pulses. CAccelerationPhase and consensus. According to the it were produced with granular syn- sounds like several clocks ticking liner notes this piece was inspired by thesis techniques. Toward the middle that slowly become out of sync with the political philosophy of Jacques of the piece the composer introduces each other. Burtner composes out Ranciere.` Ranciere` is known for his slow hocketing rhythms between these pulse streams using flute, cello, ideas about disagreement and visual the flute and cello, playing pitches and percussion blended with har- aesthetics. The percussionist plays that are rendered tonally unstable monics and their computer-generated the role of a provocateur, at times because of glissandi and microtonal counterparts. The fourth movement, mocking the saxophone soloist by inflections. This piece is a tour de Slow 2:3 (in Noise),presentsdra- playing a saxophone mouthpiece, and force featuring all of the composer’s matic percussive strikes followed by doing the same to the piano soloist techniques found elsewhere on the single-pitch sustains and glissandi. by playing on a toy piano. In the disc, but here they become solidified The result sounds strangely like the first movement, called Dissensus,the into a unified whole. music for Japanese Noh dramas. Over- percussionist injects loud bursts of

78 Computer Music Journal snare drum noise in order to initiate by-now familiar pulse fields, but in Sensus, the last movement, begins sectional changes, and writes a quo- more urgent and distorted forms. with a motoric, tutti pulse pattern tation from Ranciere’s` work with a One timbre sounds suspiciously like not unlike the octave Cs (C7 and C8) pencil on paper that is amplified and it came from an early video game in Terry Riley’s In C. Surprisingly, 20 used by Burtner’s computer program soundtrack (i.e., Pac-Man). It accel- seconds into this 95-sec movement, to generate chaotic pulse patterns erates right up to the end of this the ensemble performs a crazed, that sound as if they are triggered movement, accompanied with fast, circus-like series of asymmetrical every time the pencil is replaced on spiccato glissandi in the violin. In rhythms that would have made Frank the paper after first being lifted up, video game music, this technique Zappa proud. As if to underscore presumably between words, or at the usually accompanies ever more dif- the circus quality, midway through ends of sentences. All of this happens ficult challenges faced by the gamer the piece we hear a whistle, loud in just 32 seconds. as time is running out. To include airstream sounds, humorous bird Sxape immediately follows the this sonic trope in a piece of concert sounds, and a clave-like sound (all of end of Dissensus.Thismovementis music is highly suggestive. which can also be found in Zappa’s ushered in with the attack of a small Modification 2 begins with a bar- compositions). bell, followed by sustained resonance rage of snare drum attacks sounding Given the variety of styles and produced by the computer and the like fireworks or gunshots. Following techniques found in the pieces and ensemble playing a micropolyphony this the violin performs a series of movements on this disc, pinpointing of attack points. Underneath this is high-frequency glissandi suggestive Burtner’s compositional style seems an ominous, low-frequency rumble of certain insects or birds. This, in near impossible. Nevertheless, Burt- sound. The saxophone, featured in turn, is followed by a dense, atonal ner has consistently applied the tools this movement and performed by piano barrage straight out of the Ce- of his trade, even in cases where the the composer, oscillates between cil Taylor songbook, punctuated by sonic surfaces are markedly disparate. raucous multiphonics and more- the percussion and scratchy violin Noise Plays Burtner offers us some refined materials. sounds. All this transpires within the intriguing approaches to electro- Modification 1,thethirdmove- space of 37 seconds. acoustic composition in which live ment, is a short, 24-sec exercise The sixth movement, ianopianop, instruments engage with interactive in atonal polyphony, initiated, and focuses on the live processing of technologies. The Noise perform- brought to a close, by rim shots sound. The piano and vibraphone ers are seasoned professionals, as is on the snare drum. The movement sharply attack notes that are then clearly evident from their fine per- (vio)Lens offers a potpourri of styles processed through filtering, reversal, formances. The computer-generated and musical inflections mixed into and spatialization techniques. There parts are timbrally integrated with the atastefulmelange.´ The sounds are is also a pronounced emphasis on live performers so well that it is diffi- so well integrated that it is diffi- harmonic resonance. This movement cult at times to tell them apart. One cult to tell the difference between could have benefitted from a longer gets the impression after hearing this the extended acoustic sounds and duration than its 4-min length. A disc that the compositional ideas con- their electroacoustic counterparts longer duration would have allowed tained therein required the technolo- or modifications. Along with these, for changes of textural density and for gies and instruments that were used, the composer presents us with the instrumental variation. and this is no trivial accomplishment.

Recordings 79 Products of Interest

Line 6 Amplifi Guitar Amplifier tone-matching function that allows feedback to the user. Thirty-two and Bluetooth Speaker System guitarists to match the tone of the wave and wavetable-based patches amplifier to tracks found in their are available to choose from and the Line 6’s Amplifi is a guitar amplifier music libraries. Almost 200 clas- user can set the scale and root note, and Bluetooth speaker system com- sic amplifiers, cabinet models, and the pitch range, and stereo delay. A bined into one. The amplifier has five effects are built in and up to eight speaker is built in to the unit and a speakers: two tweeters, two mid-bass effects can be used simultaneously. headphone output is also available. drivers, and a custom-designed Ce- The user can save four presets on the Two line level audio outputs, a pitch lestion guitar speaker (see Figure 1). It amplifier and an unlimited number CV output, and a mini USB jack are is available in two models, a 150-watt on the app. provided on the rear panel. The CV version and a 75-watt portable ver- The Amplifi 150 is listed for US$ output can be set from 0–5V or 0–10V. 699.99 and the 75-watt model for The front panel features a 128 sion. The larger amplifier has a 12-in × speaker and the smaller 75-watt US$ 599.99. Contact: Line 6, 26580 64 pixel backlit LCD screen. Con- model has an 8-in speaker. A gui- Agoura Road Calabasas, California trol knobs are provided for volume, 91302-1921, USA; telephone ( 1-818) pitch correction/quantization, delay tar input, mini-jack stereo auxiliary + 575-3600; fax ( 1-818) 575-3601; Web amount, and presets. The user con- input, full size headphone output, + and USB connection are provided. A line6.com. trols also include buttons for scale, tap tempo/tuner and wet/dry effects root note, and delay length. The pitch control are also built in. antenna is removable for transport Amplifi is compatible with stream- Moog Theremini and storage. The Theremini has a ing Bluetooth from Android, iOS, built-in 3.8-in microphone, a camera Macintosh, and Windows devices. Moog has released a new electronic stand adaptor, and rubberized feet. When used as a Bluetooth speaker music instrument that is based on the It weighs 3 lb and measures 22.75 Theremin and features the sound en- 6.5 15 in with the antenna in system, the custom guitar speaker × × acts as a subwoofer. An iOS app is gine from their Animoog synthesizer place. available for remote control of the (see Figure 2). A pitch quantization The Theremini is listed for $319. amplifier. The app also features a function allows the Theremini to Contact: Moog Music, 160 Broadway be played as a traditional Theremin St. Asheville, North Carolina 28801, or with a discrete pitched scale. A USA; telephone ( 1-828) 251-0090; + doi:10.1162/COMJ r 00260 built-in tuner provides visual pitch Web www.moogmusic.com.

Focusrite iTrack Dock

Focusrite’s iTrack Dock is a recording interface for the iPad with built-in Focusrite microphone pre-amplifiers,

Figure 1. The Amplifi 150 Guitar Amplifier and Bluetooth speaker. Figure 2. Moog’s Theremini.

80 Computer Music Journal Figure 3. The iTrack Dock from Figure 4. The M-Audio two-channel Focusrite. M-Track interface.

1 /4-in jack insert input is provided on each channel. MIDI input/output, 1 balanced main outputs on /4-in jacks, and a dedicated headphone output with level control are also included. two line inputs, a direct input for The iTrack Dock works with The interface supports sample rates guitar instruments, as well as stereo any Core Audio iPad app, includ- up to 48 kHz at 24-bit resolution. outputs, headphone outputs, and a ing Garage Band and Cubasis. It is The unit is powered and connected USB port (see Figure 3). It supports bundled with Focusrite’s Tape, a to a computer through the USB sampling rates up to 96 kHz, 24-bit two-track recording application. It port. This interface weighs 0.85 lbs resolution, and a dynamic range of measures 64 280 168 mm and and measures 6.1 4.9 2in.It × × × × 105 dB. weighs 0.7 kg. includes AIR’s Ignite music software The iPad slots into place on the top The iTrack Dock is listed for US$ application and Ableton Live Lite. A panel of the iTrack Dock, allowing 199.99. Contact: Focusrite Audio Plus version of the M-Track interface the user easy access to both the Engineering, Windsor House, Turn- includes digital input/output on a hardware and iPad controls. The pike Road, High Wycombe, Bucks coaxial S/PDIF connection and is dock is compatible with any recent HP12 3FX, UK; telephone ( 44-1494) shipped with a copy of Avid Pro Tools + iOS devices that have a Lightning 462246; fax ( 44-1494) 459920; Express. + connector. The connection point on electronic mail [email protected]. The next model up is the M-Track the iTrack can be moved to match Quad, a four-channel interface that the position of any iPad socket. The supports sample rates up to 96 kHz iPad is powered and charged when and 24-bit sample depth. It has four M-Audio M-Track Audio 1 connected to the dock. combination XLR/ /4-in inputs, four 1 Standard XLR ports are provided Interfaces /4-in inserts, MIDI input/output, for microphone inputs, jack connec- two balanced outputs with dedicated tions for guitar inputs, and TRS ports M-Audio’s latest M-Track range of level controls and two with fixed for line inputs and monitor outputs. USB audio/MIDI interfaces includes level controls, a stereo/mono monitor AUSBportsupportsMIDIinputand two-, four-, and eight-channel models. switch, and a headphone output with output. The microphone preampli- They feature phantom power, guitar- dedicated level control. It is bundled fiers include 48V phantom power level switching, an input jack per with Avid’s Pro Tools Express and for use with condenser microphones. channel, and each is bundled with a AIR’s Ignite. It weighs 4.2 lbs and Hardware controls are provided for selection of music software programs. measures 13.1 3.5 6.1 in. × × monitor, headphone, and input levels. The M-Track is the basic two- The newest addition to the series The input gain control features a ring channel interface (see Figure 4). It is the largest M-Track Eight, which of light that gives visual signal feed- has two XLR microphone inputs offers eight combination XLR/jack back to the user, lighting up green or with a signal-to-noise ratio of 97 dB, inputs with individual metering, 1 1 red as appropriate. Direct monitoring and two balanced /4-in inputs with eight balanced /4-in outputs, dual is available to avoid latency delays. asignal-to-noiseratioof96dB.A headphone outputs, and a dedicated

Products of Interest 81 Figure 5. The Behringer X-Touch Figure 6. The Behringer X-Touch control surface. Mini control surface.

Figure 7. Zoom’s iQ5 condenser microphone for iPad and iPhone.

Figure 6 control room output in a rack- tions. A large job/shuttle wheel is mountable unit. included in the transport section. A The M-Track is listed for US$ USB port can be used for connec- 149.95, the Plus model for US$199.95, tion to a computer and two further the M-Track QUAD for US$ 349.95, powered USB ports are available for and the M-Track Eight for US$ 499.95. connecting other devices. MIDI in- Contact: M-Audio, 200 Scenic View put/output is provided, along with Drive, Cumberland, Rhode Island two dedicated footswitch controllers 02864, USA; telephone ( 1-401) 658- and a remote control connection. The + 5765; fax ( 1-401) 658-3640; Web Ethernet port allows the user to con- + www.m-audio.com. nect to a wireless network for control of wireless compatible mixers. A switching power supply is used. The controller is shipped with Tracktion Figure 7 Behringer X-Touch Control 4, a DAW with an unlimited number Surface of tracks, MIDI support, and VST and AU plug-in compatibility. Zoom iQ5 Mid-Side Condenser The X-Touch from Behringer is a con- AcompactversionoftheX-Touch trol surface with nine touch-sensitive with 9 faders, 16 rotary controls, Microphone for Mobile iOS motorized faders and Ethernet, USB, and 39 buttons is also available. A Devices and MIDI connectivity (see Figure 5). further X-Touch Mini model offers It is designed for use as a remote a single 60-mm master fader, eight Japanese company Zoom has pro- controller for DAW software in studio rotary controls with LED rings, and duced a number of handheld audio and live situations, and it supports 18 illuminated buttons, including recorders. Among their latest releases HUI and Mackie Control. A 100-mm transport controls (see Figure 6). It is is a mid-side condenser microphone motorized fader is provided for each USB powered. for use with iPhone, iPad, and iPod channel, along with a dynamic LCD The X-Touch is listed for US$ 599, Touch devices, allowing the user to scribble strip; an eight-segment LED the Compact model for US$ 399, record high-quality audio directly meter; a rotary control with LED and the Mini for US$ 99. Contact: to a mobile device without the ring; and record, solo, mute, and se- Behringer, 18912 North Creek Park- need for a dedicated recorder (see lect buttons. A total of 92 separate way #200, Bothell, Washington 98011, Figure 7). color-coded illuminated buttons are USA; telephone ( 1-425) 939-3200; The microphone can rotate so + provided for direct access to func- Web www.behringer.com. that the mobile device can be used

82 Computer Music Journal Figure 8. Bare Conductive Electric Figure 9. Akai Professional’s EWI Paint pen. 5000 wireless wind controller.

An iOS device that supports the Lightning connection and iOS 6 or The company also manufacturers in a vertical position or horizontally newer is required. simple flashing gift-card sets that to record video with audio. The mi- The iQ5 is listed for US$ 124.99. include a gift card, electric paint pen, crophone contains both a directional Contact: Zoom, 4-4-3 Kanda- LEDs, and batteries. Their Touch microphone to record sounds directly surugadai, Chiyoda-ku, Tokyo 101- Board is an Arduino compatible 0062, Japan; telephone ( 81)352- circuit board with twelve electrodes, in front and a bidirectional micro- + 971-040; fax ( 81) 352-971-009; atouchsensorinterface,anon-board phone to record the ambient sound + from the side. The user can adjust the electronic mail [email protected]; MP3 player/MIDI device, a MicroSD stereo width from 90 to 120 degrees Web www.zoom.co.jp. card socket, audio jack, and a Lithium or can record the audio signal as polymer battery. It can be powered raw data and adjust the stereo field and programmed by USB. afterwards from 30 to 150 degrees The Electric Paint retails for using a software app. A gain control Bare Conductive Electric approximately US$ 18, the Elec- is built into the body of the micro- Paint tric Paint Pens for approximately phone or one of three automatic gain US$ 6, and the Touch Board for presets can be used. A limiter and Electric Paint, from British company approximately US$ 50. Contact: headphone jack connection are also Bare Conductive, is a conductive The Bare Conductive Studio, included. paint that can be used to draw cir- First Floor, 98 Commercial St., The Hand Recorder app has been cuits and cold solder. It is non-toxic, London, E1 6LZ, UK; telephone designed specifically for the iQ5 and free of solvents, and water soluble. ( 44-20) 7650-7977; electronic mail + is available as a free download from It is available in pots in liquid form [email protected]; Web the App Store. It allows the user to or as paint pens (see Figure 8). The www.bareconductive.com. record in linear PCM or AAC formats paint can be applied with a brush, at a sampling rate of 44.1 kHz and bit roller, stencil, or printmaking equip- depth of 16. Compression, equalizer, ment. The user can paint circuits, and reverb effects are available and the use the paint as a conductive adhe- Akai Professional EWI 5000 user can normalize and divide sound sive, screen-print capacitive circuits, Wireless Wind Controller files. The app also has an option or simply use it to carry out elec- to change the signal to a phase- tronic repairs. The paint dries in The EWI 5000 is the latest in Akai coherent monophonic signal for 5–10 minutes at room temperature. Professional’s range of wind con- treatment of voice recordings such as Once dry, Electric Paint can be trollers (see Figure 9). This newest interviews. Audio files can be directly painted over using standard acrylic model transmits audio wirelessly uploaded to SoundCloud from the or water-based paints, and it can (2.4 GHz) and has a rechargeable bat- app. be made waterproof using varnish, tery with operation time of more than The maximum sound pressure paint, or waterproofing spray. It can 12 hours, giving musicians greater level of the microphone is 120 dB. It be used on paper, plastics, textiles, as flexibility on the performance stage. uses a Lightning connector and also well as on regular electronics compo- The controller has twelve touch- features a mini USB port for charging. nents such as PCBs, microcontrollers, sensitive metal keys on the front The microphone measures 58.6 and newer conductive thread, and panel and a roller on the back for × 62.45 38.5 mm and weighs 30.5 g. e-textiles. changing octaves with the thumb. ×

Products of Interest 83 Further control knobs give the user (snap to zero, center, or no change). A the features of the audio signal. access to filter, reverb, chorus, and transpose button is also provided. Among the new features in this latest LFO processing. Physical controls The Wind mode of the app is aimed release are a single-click function to are also provided for tweaking the at wind musicians and is based on create a doubler track, an intelligent air-pressure level and bite amount the Boehm system. The user changes automatic setting for audio source coming from the mouthpiece. Semi- octave using the left thumb. The key and output track selection, more tone and fine tuning controls are also and thumb pads can be resized to quick key functions, the ability to located on the instrument body. fit and moved on the screen. This monitor Revoice through a DAW, The user can choose from flute, mode also features the xy pad and it and to slave it to the DAW playback. oboe, and saxophone fingering modes. is positioned on screen so it can be Improved support for larger Revoice More than 3 GB of programmed controlled using the right thumb. sessions has now been added. Larger sounds from SONiVOX are built in The app can also be used as a audio files display faster, the user to the controller and the user can edit standalone breath controller. In can choose to only display the tracks these sounds using a Sound Editor Breath-only mode, MIDI control being processed, the energy and pitch application and save them to the messages are sent as the user blows displays of tracks not being processed controller’s library. MIDI software into the microphone. The sensitivity can be switched off, and session files synthesizers and modules can be can be controlled, the incoming are now written in a new compact controlled via the USB port or 5-pin breath noise smoothed, and a format that loads more quickly. MIDI connection. A mini headphone threshold set to avoid triggering from Windows Vista (with Service Pack 2), 1 output and /4-in jack for connection environmental noise. MIDI messages or Windows 7 or 8 is required. to amplifiers is also built in. The can be sent to any instruments on the Revoice Pro is listed for US$ controller weighs just under 2 lbs iPad through Core MIDI and MIDI 599.95. A 3-month rental option is and measures 2.4 26.6 1.7 in and can be sent to other devices over also available. Contact: Synchro × × the wireless receiver is provided with Wi-Fi networks. User settings are Arts Limited, 13 Links Road, purchase. stored when the app is closed. Epsom, Surrey, KT17 3PP, UK; The EWI 5000 is listed for US$ Blowfinger is available on the App telephone ( 44-1372) 811934; + 999. Contact: Akai Professional, Store (https://itunes.apple.com/us/ fax ( 44-1372) 817976; electronic + 200 Scenic View Drive, Suite 201, app/blowfinger/id767728385) for mail [email protected]; Web Cumberland, Rhode Island 02864, $17.99. Contact: batZode; Web www.synchroarts.com. USA; telephone ( 1-401) 658-4032; www.batzode.com. + Web www.akaipro.com. AudioGaming AudioWind and Revoice Pro Version 2.2 for AudioRain Plug-Ins Blowfinger Breath-Controller Windows App for iPad AudioWind and AudioRain are audio Revoice Pro was first released in 2012 plug-ins for creating wind and rain The Blowfinger app from BatZode as a standalone application for time sounds. They can be used to edit allows the user to play a keyboard alignment and pitch correction. It parameters in real time and can be or wind instrument shown on-screen was designed to be used on double used with up to 5.1-channel audio. using breath control input from an tracks and for automated dialogue The plug-ins use the AudioGaming iPhone microphone or other external replacement. The software works Dynamics Objects Sound Engine microphone. In keyboard mode, the by analyzing two audio files and (DOSE) to dynamically model audio interface features a double-manual applying the timing of one file to the components in real time rather than velocity sensitive keyboard, eight other. use wavetables or sampling. In this pads, and an xy pad. The eight pads are Version 2.2 of Revoice Pro now way, the audio can be flexible and uses velocity-sensitive and can be custom- supports the Windows operating less memory than storing audio files. labeled by the user and used to send systems as well as Mac OS X. It can AudioWind uses five models of program changes or key switches. be used to automatically match the wind sound: wind behavior, back- The user can set a MIDI continuous pitch, loudness, vibrato, intonation, ground noise, whistling, gusts, and controller for each axis of the xy pad, and timing from one track to another. squalls. Each has a number of detailed as well as the behavior on release The user can also manually adjust controls ranging from wind speed to

84 Computer Music Journal Figure 10. The Rob Papen Blue II virtual synthesizer.

whistle and gust shapes and frequen- cies. The preset are organized into Hot Wind, Cold Wind, Room Tone, Outside, and FX, and range from desert to blizzard, snow storm, un- derground, empty space, toilet, forest, mountain, meadow, and seaside. AudioRain models the sound of drops hitting a hard surface in the Rumble module, the background sound in Shower, and drops hitting puddles and other soft surfaces. Presets include rain sounds like umbrella, car, jungle, lake, city, monsoon, storm, rain shower, under tree. AudioRain can be set up as a slave of AudioWind. A randomize button is available in both plug-ins for generating new, fresh sounds. More than 100 presets are available to the user in each plug-in. The plug- ins are compatible with Macintosh and Windows platforms and are available in VST, AU, RTAS, and AAX formats. The latest version of the software adds 64-bit compatibility, anewinterfacedesign,andOSC AAX on Windows Vista 7 or 8; and The effects can be controlled using control using an iPad. in AU, VST, and AAX formats for the modulation matrix. An Easy Edit The plug-ins are listed for ap- Mac OS X 10.6. It has six oscillators feature is provided for quick changes proximately US$ 400 each and are and two stereo filters modeled on to sound parameters. Almost 3,000 available in a bundle for approxi- analog filters, with 27 different types presets are built in to the synthesizer mately US$ 615. Contact: Audio- available. Processing, modulation and can be navigated using a Bank Gaming, 42 Avenue du Gen´ eral´ de options, a sequencer and arpeggiator Manager. Croutte, 31100 Toulouse, France; are built in. A new set of waveforms Blue II is listed for US$ telephone ( 33-9) 53 27 31 67; + based on high-quality sampled sounds 179. Contact: Rob Papen; Web fax ( 33-9) 55 42 48 20; Web + has been added. www.robpapen.com. www.audiogaming.net. The xy pad from the developer’s Blade synthesizer has been included and it can be used to mix between Eisenberg Einklang Synthesizer Rob Papen Blue II Virtual the four oscillators. It can record Synthesizer up to 128 separate points or can be The Einklang synthesizer, from Ger- quantized to 4, 8, 16, 32, or 64 points. man software company Eisenberg, Blue II is a new release of Rob Pa- Asetoffourvirtualknobsisprovided uses instrument models based on pen’s Blue virtual synthesizer (see on the vertical and horizontal axis their artificial intelligence studio Figure 10). The original software of the xy pad for further control of technology. The user can alter the featured a combination of FM and each oscillator, allowing for sound parameters of the instrument models subtractive synthesis, phase distor- exploration as the oscillators are and morph between tones to create tion, and wave-shaping synthesis that combined. Four effects processors are new sounds. Each set of controls is the developer called cross-fusion syn- available and each offer the user a clearly marked on the user interface thesis. This new version is available choice of 35 different types of effects, and a distinct triangular section used in 32- and 64-bit formats for VST and including a new high-quality reverb. for the morphing controls.

Products of Interest 85 Figure 11. Aratechlabs’ Arapolarmic Figure 12. The m.phase multitouch microphone directivity app. gesture control app.

On the left of the interface are the main instrument and synthesis controls. These are arranged into three groups: loudness, timbre, and modulation. The user can change the harmonicity and dissonance of a sound using the timbre controls, and can change the attack, release, and percussiveness using the loudness controls. The rate and intensity of modulation can also be altered, and aportamentocontrolisavailable. The synthesizer parameters can be controlled by MIDI. The morph triangle on the right of the user interface has slots at each of its three corners for tone colors. The user can choose how much of each tone is used to contribute to a new sound. A set of 150 tones is provided to get the user started. The Einklang synthesizer runs on Macintosh and Windows platforms. coming from the device’s camera. It can be used as a standalone applica- The polar patterns can be scaled and tion or as a VST, AU, RTAS, or AAX rotated and the user has a choice of plug-in. A range of tone packs are color schemes to make sure the data available for purchase. These include is easily distinguishable from the synthesizer, acoustic, and drone/pad background image. packs. The developer, Aratechlabs, hopes Prices for the Einklang synthesizer the software will be useful for sound start at US$ 39 and rise to US$ 239 engineers, musicians recording them- for the synthesizer with full tone selves, and instructors. As well as pack bundle. Contact: Eisenberg; Web using the app to optimize the mi- www.eisenberg-audio.de. crophone direction before recording, users can browse the included micro- phone specification sheets, and can Aratechlabs Arapolarmic visually compare the technical char- Microphone Directivity App acteristics of different microphones models and types. The details of Arapolarmic is a software application around 80 microphone models are in- multitouch hand gestures instead that is designed to help the user cluded in the initial version of the app. of traditional control knobs and position microphones. It is available Contact: Aratechlabs; electronic faders (see Figure 12). The app allows for iOS and Android mobile devices. mail [email protected]; Web for wireless control from any loca- It visualizes the microphone, its polar www.aratechlabs.com. tion with the room. The user can pattern, and the environment in real change multiple parameters on two time, allowing the user to make channels simultaneously with the informed choices about microphone m.phase Multitouch Gesture app. placement relative to the sound Control App for DAWs The most obvious feature of the source (see Figure 11). The polar interface is the presence of two large pattern for the specific microphone The m.phase app for iPad is a con- spheres, which are used for volume being used is laid on top of the image troller for DAW software that uses and pan control of the two active

86 Computer Music Journal Figure 13. Korg’s iKaossilator app with X-Y touchpad.

channels. The spheres are designed as his or her own instrument lines on the main controls so that the app can top. Individual parts can be muted be used without looking at the iPad. and soloed. The view can be changed to a single Audio can be recorded in real sphere if needed. A list of channels time, exported as a WAV file, or being used with the current DAW directly uploaded to SoundCloud. project are listed across the top of the Audio copy and paste to other audio screen. The user drags a channel to applications is also supported. The one of the two large spheres on the app supports WIST, which allows interface to make it an active channel audio to be synchronized over a for gesture control. A simple up and wireless connection. Macintosh iOS down finger movement on a sphere 5.1 on iPad and iPod touch, or iPhone controls the volume, while left and 5operatingsystems,arerequired. right is used for pan. Fine-tuning of The iKaossilator App is listed on the volume control is possible using a the Apple App Store for US$ 19.99. gesture that involves a second finger. Contact: Korg; telephone ( 1-631) + Afastthree-fingeredswipeupor 390-6800; Web www.korg.com. down on the sphere sets the channel to mute or solo. A red circle around the spheres indicates clipping and a clip-hold feature is provided. The user Bitwig Studio DAW for Windows, can set the color of the sphere and Korg iKaossilator App Mac OS X, and Linux channel. Arangeoftriangular-shapedgraph- The iKaossilator from Korg is an iOS Bitwig Studio is a new cross-platform ical buttons around the edge of the App version of their well-known DAW for Windows, Mac OS X, and screen act as button controls for Kaossilator synthesizer. The xy Linux. The default view shown functions like channel settings and touchpad from the original hard- on launching Bitwig Studio is the switching between banks. These but- ware synthesizer has been transferred Arrange View, where tracks and clips tons don’t become active until they to this software application, allowing are arranged. An unlimited number are selected and dragged towards the the user to easily create melodies of audio, note, and effects tracks are center of the screen, thus ensuring the and sounds using finger gestures (see available. Mix and Edit views are user does not change parameters inad- Figure 13). Pitch is controlled on the also provided and the interface can vertently. Gestures for play/pause and y axis and timbre on the x axis, by be displayed over three monitors. A start/stop are available on all screens use of filters, low-frequency oscillator Browser panel, located to the right and a Transport Mode switches the modulation, and delay and reverb of the main Arrange panel, gives the main display and touch controls to a effects. The scale and key can be set user access to samples, audio files, dedicated transport mode. In Settings for the xy pad and finger movements effects, and instruments. view, the user can choose a DAW, are quantized. The Detail Editor allows the user enable/disable clip holding, flash Colorful visual effects based on the to view and edit multiple tracks feedback, assign colors to tracks, and audio are displayed automatically. A with stretch, split, pan, pitch, copy, adjust the touch sensitivity. The user set of 150 instrument preset sounds reverse, and rearrange functions pro- can also create new gesture and touch are built in. A loop sequencer with vided. Editing is non-destructive. An controls. up to five parts is included and each inspector allows multiple notes or The m.phase app is listed on the can be assigned its own instrument. events to be selected and the statis- App Store for US$ 19.99. Contact: Loops can be controlled in real time. tics of their properties viewed and MPhase Studio, Steinhublstraße¨ 1, Fifty preset loops are provided to get edited using an interactive histogram. 4800 Attnang-Puchheim, Austria; the user started and new loops can In Bitwig Studio, audio and note telephone ( 43-664) 220-2965; elec- be imported. The user can delete and events have expressions associated + tronic mail keep-in-touch@mphase record over parts of these built-in with them, and these can be set by -studio.com; Web www.mphase loops and keep others—selecting the the user and automated across the -studio.com. drum part, for example, and adding duration of the event to create a

Products of Interest 87 more musically expressive playing Cockos REAPER DAW controlled pitch shifter, graphic style. Audio expressions include gain, FFT-based equalizer, convolution pan, pitch, stretch, and onsets. The REAPER is a low-cost DAW applica- reverb, side-chainable noise gate, MIDI expressions that can be edited tion with multitrack audio and MIDI and vocoder. Effects can be applied include gain, pan, micro-pitch, and recording, editing, processing, mixing, in real time or rendering with the timbre. and mastering functionality. The user output. Networked effects processing The DAW comes with a range of 25 can record from multiple audio and is also possible using other computers effects including filters, equalizers, re- MIDI inputs simultaneously. Tracks as slaves, with minimal latency on verb, chorus, delays, flanger, limiter, are not restricted to media types and wired networks. ReWire, ReaRoute, gate, ring modulator, and transient can be a mix of audio, MIDI, and ReaInsert, MIDI sync, and Open control. Nine instruments are in- video. The user can switch inputs and Sound Control are supported. cluded: Polysynth, Organ, Sampler, punch in and out of recording while Most of the interface elements FM-4, E-Tom, E-Snare, E-Kick, E-Hat, playing back existing tracks. They can be customized by the user, using and E-Clap. A further nine “contain- can act as a bus with a simple drag his or her own images or using a ers” can be used to design new sounds and drop used for sends. All editing set of built-in themes. Image files and these are: Drum Machine, FX and effects are nondestructive, with can be used as visual identifiers Chain, FX Layer, Instrument Layer, amultilevelundosystem.Theuser on tracks. An extensive set of user Mid-Side Split, Multiband FX, Re- has immediate access to split, glue, preferences extends to the meter placer, XY Effect, and XY Instrument. resize, trim, loop, time stretch, pitch ranges, behavior of loops, priority Any parameter of these instruments, shift, fade, crossfade, slip, and snap of audio over graphics rendering, containers, or VST parameters can be to grid functions. REAPER can inte- behavior of audio items on a track modulated. Both 32- and 64-bit plug- grate with external audio editors from when overlapping, and more. Control ins can be used and mixed within a within its arrange view. surfaces can be easily mapped to project. A crash-protection system AMIDIeditorisalsoincludedand faders, plug-in knobs, actions, and ensures that a plug-in failure will not the color-coding, the displayed and custom action lists, and can be saved crash the application itself. hidden data, behavior, and naming for automatic recall as the default Multiple projects can be opened of pitches can be customized. A step settings. The program is small in size, simultaneously and the user can sequencer, pattern copy and paste, can be installed in less than a minute, drag and drop files between them. quantization, and a humanization loads quickly, and can be run from a Smart control integration is included function are provided. MIDI and USB key. and a number of MIDI controllers audio are handled in the same way Reaper is listed for US$ 60 for an in- are supported directly. The Open in terms of routing and processing. dividual license. Contact: Cockos In- Controller API is available for editing MIDI processing can be applied at any corporated, 319 Lafayette Street #255, and creating control maps, allowing point in the signal chain, MIDI tracks New York, New York 10012, USA; the user to customize the application can be rendered as audio, and MIDI electronic mail [email protected]; for any MIDI controller and use and audio can be mixed. Web www.cockos.com. scripts for almost every feature. Plug-ins in VST, VSTi, DX, DXi, Audio files in WAV, MP3, AAC, AU, and JS scripting formats are WMA, FLAC, and Ogg Vorbis formats supported. JS is a scripting language SONiVOX Eighty Eight Ensemble can be imported. The application that is compiled in real time. The requires 5 GB of free disk space, a software features automatic plug- Eighty Eight Ensemble is a virtual in- minimum screen resolution of 1280 in delay compensation and uses a strument based on a 9-foot Steinway 768 pixels, and at least 2 GB firewall to protect plug-in crashes × grand piano. A total of 1,500 samples RAM. from crashing the whole project. Any were recorded for the instrument, Bitwig Studio is listed for US$ 399. plug-in can be side-chained, even if it with 16 levels of dynamics used, giv- Contact: Bitwig GmbH, Schwedter is not natively supported. A set of 64- ing 11 GB of content in total. There Str. 13, 10119 Berlin, Germany; bit audio plug-ins include a multiband are 52 solo piano presets available as telephone ( 49-30) 6093-9430; fax equalizer, configurable compressor, + well as 35 combination patches that ( 49-30) 6093-943-323; electronic unlimited multiband compressor, + add a second instrument on top of the mail [email protected]; Web unlimited multitap delay, multivoice piano. A further 16 split-combination www.bitwig.com. pitch shifter, pitch corrector, MIDI- patches are also included and these

88 Computer Music Journal Figure 14. The Altiverb 7 convolution reverb plug-in.

allow the user to play two instru- ment separately, using the piano as an accompaniment and adding a solo instrument, for example. Afour-bandequalizerallowsthe user to adjust the timbre of the preset patches. Dynamics can be controlled with a limiter. The user can also change the action of the sustain pedal and hammer release, and can alter the high-frequency damping and parameters of the room size. A MIDI Learn Mode allows a hard- ware controller to be easily mapped to instrument parameters. Eighty Eight Ensemble is currently at release 2.1. It can be used with Mac OS X and Windows and is compatible spaces. The reverb gate can be set 1000 digital reverb from the 1980s, + with VST, RTAS, AAX, and AU 32- according to note values and linked asetofspringreverbs,Chapman and 64-bit host applications. 25 GB of to the tempo of a track. A brightness recording EMT plates, the EMT 140 hard disk space is needed and 4 GB setting adds a classic synthetic reverb plate reverb, and the EMT 250 digital RAM is recommended. layer to brighten the sound and a reverb. A set of spaces and industrial Eighty Eight Ensemble is listed modulation function adds an element building for use in postproduction for US$ 149.99. Contact: SONiVOX, of chaos so that the reverb does not include large gas tanks, Alcatraz 561 Windsor Street, Suite A402, sound exactly the same every time. prison, the cooling tower of a power Somerville, Massachusetts 02143, Altiverb is fully automatable. A news station, castles, warehouses, streets, USA; Web sonivoxmi.com. tab in the plug-in gives the user access alleys, train stations, caves, trains, to new IRs, which can be downloaded planes, and cars. and installed without restarting the Altiverb supports AAX, RTAS, Audio Ease Altiverb 7 Reverb software. Any audio can be turned VST, AU, and MAS formats. An XL Plug-In into an IR by simply dragging it into version of the plug-in supports TDM the plug-in window. on Macintosh OS X. Sampling rates Altiverb is a convolution reverb plug- Altiverb’s extensive list of IRs is up to 96 kHz are supported. in for 64-bit Mac OS X and Windows 3.4 GB in size and is provided on Altiverb is available as a free (see Figure 14). It includes a range of the product Web site, along with download for any current owners spaces, plate and digital reverbs, and descriptions and information on the who purchased the plug-in since 2010 spaces for post-production use. This space or effect. The large concert halls and at a discounted price for older release of the plug-in features a new include the Sydney Opera House, the users. The full price is 499. Con- visual browser with photos of the Walt Disney Concert Hall in Los tact: Audio Ease, Vlampijpstraat 57, rooms and spaces. The impulse re- Angeles, the Vienna Concert Hall, 3534 AR Utrecht, The Netherlands; sponses (IRs) are organized by size and and the Royal Opera in Sweden. A telephone ( 31-30) 243-3606; elec- + the user can save selected responses range of churches and cathedrals tronic mail [email protected]; Web to their favorites. A keyword search is includes Notre Dame in Paris. A www.audioease.com. included and a Similar button locates number of recording studios and the and lists similar sounding IRs. The Fox and Paramount scoring stages user can alter the equalization set- in Los Angeles are also available tings using an interactive equalizer and a range of tombs, mausoleums, New Releases curve that allows them to set the and Wembley Stadium in London band widths. are among other interesting spaces. Publications AStagePositionerallowstheuser Some well-known vintage reverb IRs to position the sound on concert hall are also available, including Roland’s Alessandro Cipriani and Maurizio stages and within other room Space Echo tape delay, Sony’s SDR Giri: Electronic Music and Sound

Products of Interest 89 Design: Theory and Practice with Pauline Minevich and Ellen Wa- Ultimate Electronic Music In- Max and MSP, Volume 2 (softcover, terman (eds.): Art of Immersive strument (softcover, 2014, ISBN 2014, ISBN: 978-88-905484-4-4, Soundscapes (softcover, 2013, 978-0-19-539481-8, New York Rome, Italy: ConTempoNet, with a DVD edited by James City, New York: Oxford University www.contemponet.com). Harley, ISBN: 978-0-88977-258-8, Press, www.global.oup.com). Mark Grimshaw (ed.): The Ox- Regina, Saskatchewan, Canada: ford Handbook of Virtuality University of Regina Press, Recordings (hardcover, 2014, ISBN: 978-0- www.uofrpress.ca). 19-982616-2, New York City, New Winifred Phillips: A Composer’s Joseph Klein: Improbable Encounters York: Oxford University Press, Guide to Game Music (hard- (CD, 2014, innova 873, Innova www.global.oup.com). cover, 2014, ISBN: 978-0-262- Recordings, www.innova.mu). Elizabeth Hellmuth Margulis: On 02664-2, Cambridge, Mas- Peter Van Zandt Kane: Hackpolitik Repeat: How Music Plays the Mind sachusetts: The MIT Press, (CD, 2014, innova 892, Innova (hardcover, 2014, ISBN: 978-0-19- www.mitpress.mit.edu). Recordings, www.innova.mu). 999082-5, New York City, New Mark Vail: The Synthesizer: A Com- Guy Barash: Facts About Water York: Oxford University Press, prehensive Guide to Understand- (CD, 2014, innova 893, Innova www.global.oup.com). ing, Playing, and Recording the Recordings, www.innova.mu).

90 Computer Music Journal