Sound Theatre.” Organised Sound, 15(3), [18] James, S. “From Autonomous to Performative FORMAL SEMANTICS FOR MUSIC NOTATION CONTROL FLOW 2010. Control of Timbral Spatialisation.” Proceedings of the Australasian Computer [4] Smalley, D. “Space-Form and the Acousmatic Music Conference, Brisbane. 2012. Zeyu Jin Roger Dannenberg Image.” Organised Sound, 12(1), 2007. [19] Oram, D. “Oramics,” Carnegie Mellon University Carnegie Mellon University [5] Wishart, T. On Sonic Art. Harwood Academic, http://daphneoram.org/oramarchive/oramics/ Amsterdam, 1996. School of Music, Pittsburgh, PA Computer Science Department, Pittsburgh, PA [20] Penrose, C. “HyperUpic” (1991, no longer [email protected] [email protected] [6] Sazdov, R., Paine, G., Stevens, K. “Perceptual publicly available) Investigation into envelopment, spatial clarity, http://www.music.princeton.edu/~penrose/soft/ and engulfment in reproduced multi-channel audio”, AES 31st International Conference, [21] UISoftware. MetaSynth. ABSTRACT 2007. http://www.uisoftware.com/MetaSynth Music notation includes a specification of control flow, [7] Chowning, John. “The Simulation of Moving [22] Marino, G., Serra, MH., and Raczinski, JM, which governs the order in which the score is read us- Sound Sources.” Journal of the Audio “The UPIC System: Origins and Innovations,” ing constructs such as repeats and endings. Music theory Engineering Society 19, 1971. Perspectives of New Music 31(1): 258-269. provides only an informal description of control flow no- 1993. [8] Smalley, D. “Spectromorphology: Explaining tation and its interpretation, but interactive music systems Sound Shapes.” Organised Sound: Vol. 2(2), [23] Lyon, E. “Image-Based Spatialization,” need unambiguous models of the relationships between 1997. Proceedings of the International Computer the static score and its performance. A framework is in- Music Conference, Slovenia. 2012. troduced to describe music control flow semantics using [9] Ando, Y. Architectural Acoustics: Blending theories of formal languages and compilers. A formaliza- Figure 1: Control flow definition in Read’s book Sound Sources, Sound Fields, and Listeners. [24] Barreiro, D. L. 2010. “Considerations on the tion of control flow answers several critical questions: Are Springer- Verlag, New York, 1998. Handling of Space in Multichannel the control flow indications in a score valid? What do the Electroacoustic Works”, Organised Sound, [10] Griesinger, D. “The psychoacoustics of control flow indications mean? What is the mapping from a gap between often simplified theoretical ideals and ac- 15(03). 2010. apparent source width, spaciousness and performance location to static score location? Conven- tual practice, especially in modern works. In practice, we envelopment in performance spaces,’’ [25] Interview with Davies, 15 October 2006, tional notation is extended to handle practical problems, find nested repeats, exceptions and special cases indicated Acustica, 83, 1997. Reverie, southern Quebec. and an implementation, Live Score Display, is offered as by textual annotations, multiple endings, and symbols for rearrangement. [11] Rumsey, F. “Spatial Quality Evaluation for [26] James, S. Particle I. Perf. Stuart James. Stasis a component for interactive music display. reproduced Sound: Terminology, Meaning, Ecstatic, Heartless Robot, 2012 We encountered this gap between theory and practice and a Scene-based Paradigm." Journal of the in the implementation of music notation display software. [27] Hope, C. and Vickery, L. The Talking Board. 1. INTRODUCTION AES, 50(9), 2002. We needed a formal (computable) way to relate notation 2012. to its performance, and we found conventional notions too [12] Kendall, G. and A. Cabrera. “Why Things Music notation has been evolving for centuries, creating [28] Francis, M. When Traffic Rises. 2012. limiting to express what we found in actual printed scores. Don’t Work: What you need to know about a symbolic system to convey music information. Early music notation contained only lines and notes, which are To address this problem, we developed new theoretical spatial audio”, Proceedings of the 2011 [29] James, S. Veden Ja Tulen Elementit. 2012. foundations based upon models of formal language and International Computer Music Conference, sufficient for communicating pitches and durations. It was [30] Hofmann, B. “Spatial Aspects in Xenakis’ compilation, and we applied these developments to the Huddersfield, UK. 2011. later that bar lines and time signatures emerged, grouping Instrumental Works,” Definitive Proceedings music into measures and introducing the idea of beats.1 implementation of a flexible music display system. [13] Ligeti, G. “Metamorphoses of Musical Form,” of the International Symposium Iannis The notation for music control flow, like repeats and co- Music control flow is the reading order of measures Die Reihe, (ed.) K. Stockhausen and H. Xenakis, Athens, 2005. das, came even later. Control flow helps to identify repeat- affected by control symbols including the time signature, Eimert, (trans.) by C. Cardew, Bryn Mawr: [31] Schaeffer, P. Traité des objets musicaux. 1977. ing structures of music and eliminates duplication in the measures, repeats, endings, etc. It can also be viewed for- Theodore Presser, 1965. printed score. In the Classical period, control flow nota- mally as a function f that maps the performed beat k to a [32] James, S. “Developing a Flexible and [14] Torchia R, and Lippe, C. “Techniques for tion is closely tied to music forms such as binary, ternary location of a score, <m,b>, a measure and beat pair. f (k) Expressive Realtime Polyphonic Wave Terrain Multi-Channel Real-Time Spatial Distribution and sonata and is more of a musical architecture than a describes the reading order of the score. In principle, we Synthesis Instrument based on a Visual and Using Frequency-Domain Processing.” means of saving space.2 Conventional practice for control can rewrite the score in the order f (1), f (2), . to cre- multidimensional methodology.” Masters Proceedings of the International Computer flow notation is well established. The literature [6, 15] has ate an equivalent score with no control flow (other than Diss., Edith Cowan University, 2005. Music Conference, Singapore. 2003. formalized the notation in all kinds of ways and there is reading sequentially). We call this the “flattened score” little conflict among definitions. However, traditional mu- or “performance score.” Audio recordings and MIDI se- [15] Kim-Boyle, D. “Spectral Spatialization - An quences are both in the order of the flattened representa- Overview”, Proceedings of the 2008 sic theory has not explored the possibilities of expanded tion of the corresponding score. International Computer Music Conference, or enriched representations for control flow, and there is Existing music theory devotes little attention to con- Belfast. 2008. 1 Far beyond formalizing the notion of beats, music notation led to trol flow, and in fact, there does not seem to be even a [16] Kim-Boyle, D. “Spectral and Granular the “discovery” of time as an independent dimension that did not de- pend upon physical actions. In particular the musical rest is the first standard term for the concept of control flow. To define Spatialization with Boids”, Proceedings of the direct representation of “nothingness” existing over time, or of time it- the meaning of control flow symbols, the conventional 2006 International Computer Music self. Composers developed this concept centuries before the scientific practice is to use words and visuals to illustrate the read- Conference, New Orleans, pp. 139-142. 2006. revolution, Kepler, Newton, graphs with a time axes, etc. [3] ing order. For example, Read uses arrows to mark the true 2For example, “In practically all the sonatas of the earlier period the [17] James, S. “Multidimensional Data Sets: exposition is repeated, as is indicated by the repeat-sign at its end, which reading order (see Figure 1) [15]. This approach defines Traversing Sound Synthesis, Sound Sculpture, is also helpful for the reader in finding the end of the exposition ...” [2] both the syntax and meaning. and Scored Composition.” Proceedings of the Australasian Computer Music Conference, Auckland. 2011. 84 | 2013 ICMC idea | LONG PAPERS 85 | 2013 ICMC idea | LONG PAPERS However, the visual definition is incomplete; there are cases where we are not sure which production to use. One such case is nested repeats where there are two ways of grouping the right repeat with the left repeat. In common practice music, we often restrict the number of levels of repeats to be 1, which excludes the nested repeat prob- lem. In more modern and flexible music practice, there are nested structures, text annotations [8] and repetitions conditioned on actual time [17]. There lacks an unam- Figure 4: Symbolizing the score: this score is converted biguous way to formalize the syntax and the meaning of to the following list of strings to represent control flow: such notation. While nested repeats are relatively simple (block, 0, 4) Fine : (block, 4, 4) : : (block, 8, 4) [ ∥ ∥∥ to formalize, we argue that the general problem of formal- (block, 12, 4) ] : [ (block, 16, 4) ] DC.Fine izing control flow is non-obvious, interesting, and useful. ∥ In this paper, we present a new framework for formal- izing control flow notation based on formal grammar and 5. MODEL FOR EXTENDED MUSIC PRACTICE compilation. One core feature of this framework is that it can be easily designed by humans and applied automati- The framework for defining music