Sound Theatre.” Organised Sound, 15(3), [18] James, S. “From Autonomous to Performative FORMAL SEMANTICS FOR MUSIC NOTATION CONTROL FLOW 2010. Control of Timbral Spatialisation.” Proceedings of the Australasian Computer [4] Smalley, D. “Space-Form and the Acousmatic Music Conference, Brisbane. 2012. Zeyu Jin Roger Dannenberg Image.” Organised Sound, 12(1), 2007. [19] Oram, D. “Oramics,” Carnegie Mellon University Carnegie Mellon University [5] Wishart, T. On Sonic Art. Harwood Academic, http://daphneoram.org/oramarchive/oramics/ Amsterdam, 1996. School of Music, Pittsburgh, PA Computer Science Department, Pittsburgh, PA [20] Penrose, C. “HyperUpic” (1991, no longer [email protected] [email protected] [6] Sazdov, R., Paine, G., Stevens, K. “Perceptual publicly available) Investigation into envelopment, spatial clarity, http://www.music.princeton.edu/~penrose/soft/ and engulfment in reproduced multi-channel audio”, AES 31st International Conference, [21] UISoftware. MetaSynth. ABSTRACT 2007. http://www.uisoftware.com/MetaSynth Music notation includes a specification of control flow, [7] Chowning, John. “The Simulation of Moving [22] Marino, G., Serra, MH., and Raczinski, JM, which governs the order in which the score is read us- Sound Sources.” Journal of the Audio “The UPIC System: Origins and Innovations,” ing constructs such as repeats and endings. Music theory Engineering Society 19, 1971. Perspectives of New Music 31(1): 258-269. provides only an informal description of control flow no- 1993. [8] Smalley, D. “Spectromorphology: Explaining tation and its interpretation, but interactive music systems Sound Shapes.” Organised Sound: Vol. 2(2), [23] Lyon, E. “Image-Based Spatialization,” need unambiguous models of the relationships between 1997. Proceedings of the International Computer the static score and its performance. A framework is in- Music Conference, Slovenia. 2012. troduced to describe music control flow semantics using [9] Ando, Y. Architectural Acoustics: Blending theories of formal languages and compilers. A formaliza- Figure 1: Control flow definition in Read’s book Sound Sources, Sound Fields, and Listeners. [24] Barreiro, D. L. 2010. “Considerations on the tion of control flow answers several critical questions: Are Springer- Verlag, New York, 1998. Handling of Space in Multichannel the control flow indications in a score valid? What do the Electroacoustic Works”, Organised Sound, [10] Griesinger, D. “The psychoacoustics of control flow indications mean? What is the mapping from a gap between often simplified theoretical ideals and ac- 15(03). 2010. apparent source width, spaciousness and performance location to static score location? Conven- tual practice, especially in modern works. In practice, we envelopment in performance spaces,’’ [25] Interview with Davies, 15 October 2006, tional notation is extended to handle practical problems, find nested repeats, exceptions and special cases indicated Acustica, 83, 1997. Reverie, southern Quebec. and an implementation, Live Score Display, is offered as by textual annotations, multiple endings, and symbols for rearrangement. [11] Rumsey, F. “Spatial Quality Evaluation for [26] James, S. Particle I. Perf. Stuart James. Stasis a component for interactive music display. reproduced Sound: Terminology, Meaning, Ecstatic, Heartless Robot, 2012 We encountered this gap between theory and practice and a Scene-based Paradigm." Journal of the in the implementation of music notation display software. [27] Hope, C. and Vickery, L. The Talking Board. 1. INTRODUCTION AES, 50(9), 2002. We needed a formal (computable) way to relate notation 2012. to its performance, and we found conventional notions too [12] Kendall, G. and A. Cabrera. “Why Things Music notation has been evolving for centuries, creating [28] Francis, M. When Traffic Rises. 2012. limiting to express what we found in actual printed scores. Don’t Work: What you need to know about a symbolic system to convey music information. Early music notation contained only lines and notes, which are To address this problem, we developed new theoretical spatial audio”, Proceedings of the 2011 [29] James, S. Veden Ja Tulen Elementit. 2012. foundations based upon models of formal language and International Computer Music Conference, sufficient for communicating pitches and durations. It was [30] Hofmann, B. “Spatial Aspects in Xenakis’ compilation, and we applied these developments to the Huddersfield, UK. 2011. later that bar lines and time signatures emerged, grouping Instrumental Works,” Definitive Proceedings music into measures and introducing the idea of beats.1 implementation of a flexible music display system. [13] Ligeti, G. “Metamorphoses of Musical Form,” of the International Symposium Iannis The notation for music control flow, like repeats and co- Music control flow is the reading order of measures Die Reihe, (ed.) K. Stockhausen and H. Xenakis, Athens, 2005. das, came even later. Control flow helps to identify repeat- affected by control symbols including the time signature, Eimert, (trans.) by C. Cardew, Bryn Mawr: [31] Schaeffer, P. Traité des objets musicaux. 1977. ing structures of music and eliminates duplication in the measures, repeats, endings, etc. It can also be viewed for- Theodore Presser, 1965. printed score. In the Classical period, control flow nota- mally as a function f that maps the performed beat k to a [32] James, S. “Developing a Flexible and [14] Torchia R, and Lippe, C. “Techniques for tion is closely tied to music forms such as binary, ternary location of a score,
little conflict among definitions. However, traditional mu- or “performance score.” Audio recordings and MIDI se- [15] Kim-Boyle, D. “Spectral Spatialization - An quences are both in the order of the flattened representa- Overview”, Proceedings of the 2008 sic theory has not explored the possibilities of expanded tion of the corresponding score. International Computer Music Conference, or enriched representations for control flow, and there is Existing music theory devotes little attention to con- Belfast. 2008. 1 Far beyond formalizing the notion of beats, music notation led to trol flow, and in fact, there does not seem to be even a [16] Kim-Boyle, D. “Spectral and Granular the “discovery” of time as an independent dimension that did not de- pend upon physical actions. In particular the musical rest is the first standard term for the concept of control flow. To define Spatialization with Boids”, Proceedings of the direct representation of “nothingness” existing over time, or of time it- the meaning of control flow symbols, the conventional 2006 International Computer Music self. Composers developed this concept centuries before the scientific practice is to use words and visuals to illustrate the read- Conference, New Orleans, pp. 139-142. 2006. revolution, Kepler, Newton, graphs with a time axes, etc. [3] ing order. For example, Read uses arrows to mark the true 2For example, “In practically all the sonatas of the earlier period the [17] James, S. “Multidimensional Data Sets: exposition is repeated, as is indicated by the repeat-sign at its end, which reading order (see Figure 1) [15]. This approach defines Traversing Sound Synthesis, Sound Sculpture, is also helpful for the reader in finding the end of the exposition ...” [2] both the syntax and meaning. and Scored Composition.” Proceedings of the Australasian Computer Music Conference, Auckland. 2011.
84 | 2013 ICMC idea | LONG PAPERS 85 | 2013 ICMC idea | LONG PAPERS However, the visual definition is incomplete; there are cases where we are not sure which production to use. One such case is nested repeats where there are two ways of grouping the right repeat with the left repeat. In common practice music, we often restrict the number of levels of repeats to be 1, which excludes the nested repeat prob- lem. In more modern and flexible music practice, there are nested structures, text annotations [8] and repetitions conditioned on actual time [17]. There lacks an unam- Figure 4: Symbolizing the score: this score is converted biguous way to formalize the syntax and the meaning of to the following list of strings to represent control flow: such notation. While nested repeats are relatively simple (block, 0, 4) Fine : (block, 4, 4) : : (block, 8, 4) [ ∥ ∥∥ to formalize, we argue that the general problem of formal- (block, 12, 4) ] : [ (block, 16, 4) ] DC.Fine izing control flow is non-obvious, interesting, and useful. ∥ In this paper, we present a new framework for formal- izing control flow notation based on formal grammar and 5. MODEL FOR EXTENDED MUSIC PRACTICE compilation. One core feature of this framework is that it can be easily designed by humans and applied automati- The framework for defining music notation control flow cally by computer. As an example, we show a formal def- takes three steps: First, abstract the score into symbols; inition of control flow notation that unifies conventional next, define the syntax for a well-formed score based on music notation and some of the most frequent notations Figure 2: Control flow notation and the corresponding formal grammars; finally, define the meaning of each gram- used in modern music practice. We evaluate the ambiguity reading order in common music practice Figure 3: Control flow notation in modern music practice mar using meta expressions. From this, we obtain the and completeness of this definition within our framework. mapping function f (k). Finally, we show the implementation of this method and Book [8]. The six examples6 contain four typical notations its application. sentation. This work also draws attention to arrangement in which performance order is specified as a list of sec- in modern music practice: 5.1. Symbolize the Score tion labels (e.g. “play the intro, verse 1, verse 2, chorus, 2. RELATED WORK Nested repeats (example 1 and 5). For both convenience and clarity, we first convert the vi- chorus”), a common practice in popular music. However, sual score to a list of symbols, each corresponds to a no- the algorithms in this work do not handle nested repeats Multiple and nested DS/DC-coda notation: In Example The score representation and interpretation is essential to tation or a block of notations in the score. Table 1 shows or provide a systematic way to validate the correctness of 3, the notation is complicated. The relation among score writing, printing, auto-accompaniment and conduct- a list of these symbols. To separate the problem of con- notation. Overall, previous work has made great contri- D.C. al coda, to coda and coda signs are connected ing systems. Most works in this field focus mainly on stor- trol flow semantics from the (much) larger and more gen- butions to the notational presentation of the score, while by lines. ing and visualizing the score so as to accommodate both eral problem of music notation semantics, we simply la- the basic questions of music control flow, such as “what traditional and modern music practice, and, as a minor (Example 2): re-order the section bel each block of notation between control symbols and notation is right” and “what is the correct performance or- Re-arranged sections concern, provide room for implementing correct and user- marks to create a different reading order of the score. assume the meaning and duration of a block is known. der given this notation” are still unanswered. In this work, friendly score play-back systems. Early studies such as The procedure is to read the score from left to right, we provide a systematic scheme to define the “right” no- MuseData [9] and Common Music Notation [16] mainly Word annotation (Example 1, 4 and 6): omitting or play- write down each control flow symbol and name the inter- tation and produce the flattened score representation with- concern the encoding of conventional measure-based scores ing some measures conditioned on different rounds vening blocks as shown in Figure 4. out ambiguity. in their original notational presentation. To further en- of repetition. code logical structure of the score and support a wide Table 1: Symbols used to label Control flow notation 3. COMMON MUSIC PRACTICE One could dismiss these particular scores as anomolies, range of modern presentations, structured music descrip- but we find that in music, the exception proves the rule. tion languages like MusicXML [12] offer standards for Frequently used control flow symbols in common practice Existing commercial software interprets Example 5 in- Symbol Meaning score sharing and archiving. More recently, commercial 3 4 music are summarized in Figure 2. To avoid ambiguity, consistently. In Finale 2012 and Sibelius 7, the default (b, k, l) A block of score, starting score printing software like Finale and Sibelius , and 7 8 we often make the following restrictions: reading order is a-b-b-c-b-b-c-d , but in MuseScore , the from beat k with length l score accompaniment software like SmartMusic5 can gen- order is a-b-b-c-a-b-c-d. The correct order is a-b-b-c-a- : and : Left Repeat and Right Repeat erate a music performance from notation. However, the No control flow symbols are allow within the blank b-b-c-d, which differs from both software interpretations. ∥ ∥ algorithms that map the score to performance are prob- • [ Beginning of an ending staff in Rule 1, 2 and 3. Clearly, we need a more sophisticated approach that offers ] Endpoint of an ending lematic when nested repeats are involved. Given that pop- a syntax and meaning that can be shared by both humans ular score editors attempt to perform scores by following There can only be at most one type of DC/DS sym- DC dal Capo • and computers. In the next section, we show such an ap- bols in the entire music. DC.Coda D.C. al Coda control flow, it is surprising that they do not define a syn- proach based on formal language and attribute grammars. DC.Fine D.C al Fine tax for control flow, check scores for validity, or offer a In Section 6, we show an even more flexible method “the Left and right repeats can be used within the blank DS Del Segno consistent meaning for control flow notation. • intermediate notation” that supports extensions based on staff in Rules 4-9. DS.Coda D.S. al Coda More recent study with an emphasis on music control textual annotations. flow modeling led us to develop a score player for HCMP DS.Fine D.S. al Fine 6 [7]. This work provides data structures and algorithms for From 1 to 6: “Follow Your Heart” by J. Mclaughlin pp.154–155; Segno the Segno sign 4. EXTENDED MUSIC PRACTICE “Forest Flower” by Charles Lloyd p.158; “Good Evening Mr. & Mrs. finding the dynamic score (similar to the “flattened score” America” by John Guerin p.176; “Group your own” by Keith Jarrett Coda the Coda sign in this work) by unfolding the static (original) score pre- In modern music practice, the control flow notation is much p.182; Little Niles by Raudy Weston p.267; “Mallet Man” by Gordon ToCoda To Coda (or Coda symbol) Boek p.282 Fine Fine 3Finale music notation software: http://www.finalemusic.com/ more flexible; there are many examples undefined in the 7One can notate nested repeat by manually specifying which measure 4Sibelius music notation software: http://www.sibelius.com/ conventional notation framework. Figure 3 shows some the right repeat goes to. 5SmartMusic software: http://www.smartmusic.com/ of the many unconventional examples found in The Real 8MuseScore music notation software: http://musescore.org/
86 | 2013 ICMC idea | LONG PAPERS 87 | 2013 ICMC idea | LONG PAPERS However, the visual definition is incomplete; there are cases where we are not sure which production to use. One such case is nested repeats where there are two ways of grouping the right repeat with the left repeat. In common practice music, we often restrict the number of levels of repeats to be 1, which excludes the nested repeat prob- lem. In more modern and flexible music practice, there are nested structures, text annotations [8] and repetitions conditioned on actual time [17]. There lacks an unam- Figure 4: Symbolizing the score: this score is converted biguous way to formalize the syntax and the meaning of to the following list of strings to represent control flow: such notation. While nested repeats are relatively simple (block, 0, 4) Fine : (block, 4, 4) : : (block, 8, 4) [ ∥ ∥∥ to formalize, we argue that the general problem of formal- (block, 12, 4) ] : [ (block, 16, 4) ] DC.Fine izing control flow is non-obvious, interesting, and useful. ∥ In this paper, we present a new framework for formal- izing control flow notation based on formal grammar and 5. MODEL FOR EXTENDED MUSIC PRACTICE compilation. One core feature of this framework is that it can be easily designed by humans and applied automati- The framework for defining music notation control flow cally by computer. As an example, we show a formal def- takes three steps: First, abstract the score into symbols; inition of control flow notation that unifies conventional next, define the syntax for a well-formed score based on music notation and some of the most frequent notations Figure 2: Control flow notation and the corresponding formal grammars; finally, define the meaning of each gram- used in modern music practice. We evaluate the ambiguity reading order in common music practice Figure 3: Control flow notation in modern music practice mar using meta expressions. From this, we obtain the and completeness of this definition within our framework. mapping function f (k). Finally, we show the implementation of this method and Book [8]. The six examples6 contain four typical notations its application. sentation. This work also draws attention to arrangement in which performance order is specified as a list of sec- in modern music practice: 5.1. Symbolize the Score tion labels (e.g. “play the intro, verse 1, verse 2, chorus, 2. RELATED WORK Nested repeats (example 1 and 5). For both convenience and clarity, we first convert the vi- chorus”), a common practice in popular music. However, sual score to a list of symbols, each corresponds to a no- the algorithms in this work do not handle nested repeats Multiple and nested DS/DC-coda notation: In Example The score representation and interpretation is essential to tation or a block of notations in the score. Table 1 shows or provide a systematic way to validate the correctness of 3, the notation is complicated. The relation among score writing, printing, auto-accompaniment and conduct- a list of these symbols. To separate the problem of con- notation. Overall, previous work has made great contri- D.C. al coda, to coda and coda signs are connected ing systems. Most works in this field focus mainly on stor- trol flow semantics from the (much) larger and more gen- butions to the notational presentation of the score, while by lines. ing and visualizing the score so as to accommodate both eral problem of music notation semantics, we simply la- the basic questions of music control flow, such as “what traditional and modern music practice, and, as a minor (Example 2): re-order the section bel each block of notation between control symbols and notation is right” and “what is the correct performance or- Re-arranged sections concern, provide room for implementing correct and user- marks to create a different reading order of the score. assume the meaning and duration of a block is known. der given this notation” are still unanswered. In this work, friendly score play-back systems. Early studies such as The procedure is to read the score from left to right, we provide a systematic scheme to define the “right” no- MuseData [9] and Common Music Notation [16] mainly Word annotation (Example 1, 4 and 6): omitting or play- write down each control flow symbol and name the inter- tation and produce the flattened score representation with- concern the encoding of conventional measure-based scores ing some measures conditioned on different rounds vening blocks as shown in Figure 4. out ambiguity. in their original notational presentation. To further en- of repetition. code logical structure of the score and support a wide Table 1: Symbols used to label Control flow notation 3. COMMON MUSIC PRACTICE One could dismiss these particular scores as anomolies, range of modern presentations, structured music descrip- but we find that in music, the exception proves the rule. tion languages like MusicXML [12] offer standards for Frequently used control flow symbols in common practice Existing commercial software interprets Example 5 in- Symbol Meaning score sharing and archiving. More recently, commercial 3 4 music are summarized in Figure 2. To avoid ambiguity, consistently. In Finale 2012 and Sibelius 7, the default (b, k, l) A block of score, starting score printing software like Finale and Sibelius , and 7 8 we often make the following restrictions: reading order is a-b-b-c-b-b-c-d , but in MuseScore , the from beat k with length l score accompaniment software like SmartMusic5 can gen- order is a-b-b-c-a-b-c-d. The correct order is a-b-b-c-a- : and : Left Repeat and Right Repeat erate a music performance from notation. However, the No control flow symbols are allow within the blank b-b-c-d, which differs from both software interpretations. ∥ ∥ algorithms that map the score to performance are prob- • [ Beginning of an ending staff in Rule 1, 2 and 3. Clearly, we need a more sophisticated approach that offers ] Endpoint of an ending lematic when nested repeats are involved. Given that pop- a syntax and meaning that can be shared by both humans ular score editors attempt to perform scores by following There can only be at most one type of DC/DS sym- DC dal Capo • and computers. In the next section, we show such an ap- bols in the entire music. DC.Coda D.C. al Coda control flow, it is surprising that they do not define a syn- proach based on formal language and attribute grammars. DC.Fine D.C al Fine tax for control flow, check scores for validity, or offer a In Section 6, we show an even more flexible method “the Left and right repeats can be used within the blank DS Del Segno consistent meaning for control flow notation. • intermediate notation” that supports extensions based on staff in Rules 4-9. DS.Coda D.S. al Coda More recent study with an emphasis on music control textual annotations. flow modeling led us to develop a score player for HCMP DS.Fine D.S. al Fine 6 [7]. This work provides data structures and algorithms for From 1 to 6: “Follow Your Heart” by J. Mclaughlin pp.154–155; Segno the Segno sign 4. EXTENDED MUSIC PRACTICE “Forest Flower” by Charles Lloyd p.158; “Good Evening Mr. & Mrs. finding the dynamic score (similar to the “flattened score” America” by John Guerin p.176; “Group your own” by Keith Jarrett Coda the Coda sign in this work) by unfolding the static (original) score pre- In modern music practice, the control flow notation is much p.182; Little Niles by Raudy Weston p.267; “Mallet Man” by Gordon ToCoda To Coda (or Coda symbol) Boek p.282 Fine Fine 3Finale music notation software: http://www.finalemusic.com/ more flexible; there are many examples undefined in the 7One can notate nested repeat by manually specifying which measure 4Sibelius music notation software: http://www.sibelius.com/ conventional notation framework. Figure 3 shows some the right repeat goes to. 5SmartMusic software: http://www.smartmusic.com/ of the many unconventional examples found in The Real 8MuseScore music notation software: http://musescore.org/
86 | 2013 ICMC idea | LONG PAPERS 87 | 2013 ICMC idea | LONG PAPERS 5.2. Context-free Grammar languages. However, not all context-free grammars can DS.al.coda in the DS/DC family and limit our attention be parsed by LR(1). Since LR(1) parsers are balanced in to the “.code” attribute only. A context free grammar (CFG) [1] is used to formalize generality and computational efficiency, we often design Define nonterminal S as the score, L as the left-most the valid sequences of music symbols. A grammar is a set grammars acceptable to LR(1) parsers. part of the score, E as score element, LEND as the ending of rules that describe how a string (or a sequence of sym- within L and ND as the ending. Add two additional sym- bols) is formed. A context-free grammar (CFG) consists 5.3. Ambiguity and Error Handling bols to the original score: # at the beginning and $ at the of terminal, nonterminal, productions, and a start symbol. end. In a CFG, one nonterminal is distinguished as the start The problem of ambiguity arises when there are multi- The score S: to be consistent with case 1, the score symbol. The start symbol is replaced according to pro- ple parse trees for the same the sequence of symbols and needs to be decomposed into an L, the leftmost part that ductions. After replacement, any remaining nonterminal the same grammar. For example, if we change production could have multiple right repeats, and an E followed by can be replaced according to productions, and so on, until (G2) to E E: , we get an ambiguous grammar. The → | the ending sign $. This gives us only terminals remain. score b : b: can be generated two ways: | | For example, we can define S as a whole score and E Figure 6: Data flow in syntax directed translation S -> L $ { as a score element. Then we can define the following: A S => E => E E => E :| E :| => b :| b :| S.code = L.code; } S => E => E :| => E E :| => score S is a sequence of 1 or more musical elements (E), S -> L E $ { E b :| => E :| b :| => b :| b :| S.code = L.code | E.code; } optionally followed by a right repeat. A grammar for this Rule (G1) says the flattened representation of a score can be a flattened score element with the same starting S -> L Segno E ToCoda E DS.coda Coda E $ { language (set of scores) is: The first derivation (line 1) implies a sequence of two S.code = L.code | E0.code | E1.code | beat and duration. Rule (G2) says if the score has a re- repeated blocks. The second derivation (line 2) implies E0.code | E2.code; } Starting: S peat, then the flattened score is two copies of the score nested repeats. Notice that while semantics are not in- (G1) S -> E element before the repeat sign with the starting beat equal The leftmost group L: If there are no non-paired right (G2) S -> E :| herent in formal grammars, we often associate semantics to that of the score element and the duration equal to two repeats in the score, L should be an E. Or this L can pro- (G3) E -> E E closely with productions and parse trees. (G4) E -> b times the duration of the score element. Rule (G3) says duce more L, right repeat sign followed by an E and also if a score element is formed by two sub-score elements, an ending structure, the leftmost multiple endings. 5.4. Syntax Directed Translation and Attribute Gram- Here, we consider “b” to be a terminal denoting any then the flattened element is made by concatenating the block, although we could add productions that expand mar L -> L :| E { flattened representation of sub-elements, and the duration L.code = L1.code | L1.code | E.code; } “b” (now a nonterminal) to terminals of the form “(block A syntax-directed definition is a context free grammar to- is the sum of the lengths of the sub-elements. Finally, if L -> # E { start duration).” Based on this CFG, we are able to tell if gether with attributes and rules. The attributes are asso- an element is a single block, then all the attributes of the L.code = E.code; } a sequence of symbols is formed from this grammar us- ciated with grammar symbols and the rules are used to block are copied to the element (G4). L -> L :| { ing derivation. For example, (b,0,4)(b,4,4): is L.code = L1.code | L1.code; } ∥ define the relation among symbols [13]. For any termi- Based on the parse tree, we can apply rules from leaves L -> LEND E { well-defined from the grammar because we can derive it nal or nonterminal X, denote X.a as some attribute of to root in order to synthesize the attributes of upper levels. L.code = LEND.code | E.code; } from S using the productions: X. For a production like X YZ, the rule can be X.a = For the example shown in Figure 5, the attribute flow is L -> LEND { → L.code = LEND.code; } S => E :| op(Y.b,Z.c) which contains some attribute symbol, an equal shown in Figure 6. sign and some expression op of Y.b and Z.c. Finally the flattened score is in s.code and the dura- => E E :| [use E -> E E] The score element E can be a simple block, a repeated => (b, 0, 4) (b, 4, 4) :| [use E -> b (twice)] Define attribute code as the flattened presentation for tion of the entire score is 16 beats. Although this is a structure, an ending or a DS.al.Coda structure. all the terminals and nonterminals in our formal control simple example, notice that we have precisely and un- We say that a sequence of symbols is a well defined flow framework. Define “ ” as the concatenation opera- ambiguously specified how to interpret score control flow E -> b { | score if it can be derived using a grammar. The result of tor that links two lists of symbols to be one. For exam- notation. Moreover, we have introduced a meta-notation E.code = (b, b.beat, b.dur); } a derivation is a tree structure where each score symbol is ple, if E1.code = (b,0,4) (b,4,4); and E2.code = : (b,8,4) based on attribute grammars that allows us to define other E -> |: E :| { ∥ E.code = E1.code | E1.code; } a leaf and where each parent node and its children corre- : , then E1.code E2.code = (b,0,4) (b,4,4) : (b,8,4) : . interpretations of score control flow. ∥ | ∥ ∥ E -> |: E ND [ E ] { sponds to a production in the grammar. The leaves from Furthermore, define attributes “beat” and “dur” to be the For n = 1 to ND.count left to right (or more precisely in preorder traversal) will starting beat and duration for any terminal or nontermi- 5.5. The Mapping Function E0.code = E0.code | E1.code generate the input sequence. The tree is called a parse nal. For any block b, assume b.beat is the “starting beat” | ND.ending[n]; tree in compilation theory. As an example, the tree for One goal of defining control flow is to obtain a mapping E0.code = E0.code | E1.code | E2.code; } of b and len is the “length” of the block. The syntax di- E -> E E { (b,0,4)(b,0,4): is shown in the Figure 5. from beats in a performance of the score (i.e. the flattened ∥ rected definition for the simple grammar (G1)–(G4) can E0.code = E1.code | E2.code; } A program that converts a sequence of symbols into a be defined as follows: score) to positions or blocks in the original score. We E -> E Segno E ToCoda E DS.Coda Coda E $ { parse tree is called a parser. A parser essentially runs the denote the mapping as f (k), where k is the beat position E.code = E1.code | E2.code | E3.code | grammar “backwards,” reducing a string of nonterminals Starting: S in the flattened score (S.code) and f (k) is the position in E2.code | E4.code; } to the start symbol by applying productions in reverse. (G1) S -> E { the score. The map can be constructed by summing beat (1.1) S.code = E.code ND and LEND: Because of the complication of this Many parsing algorithms have been developed for gram- durations in the flattened representation S.code to obtain (1.2) S.beat = E.beat structure, we need to record each ending to the top level mars of different complexity. LR(1) [10] and LALR(1) k for each block in the original score. (1.3) S.dur = E.dur } by creating a new attribute ND.ending with a list structure. [14] are the most frequently used parsers for programming (G2) S -> E :| { (2.1) S.code = E.code | E.code We also need to use a loop to pair different endings with 5.6. Well Defined Music Control Flow (2.2) S.beat = E.beat the constant part. (2.3) S.dur = E.dur + E.dur } As a demonstration of how we can use this framework to (G3) E -> E E { LEND -> # E ND [ E ] { (3.1) E.code = E1.code | E2.code formalize music control flow for extended music practice, For n = ND.count downto 1 (3.2) E.beat = E1.beat the following grammar supports nested repeats and un- LEND.code = LEND.code | E0.code (3.3) E.dur = E1.dur + E2.dur } limited endings. An even more sophisticated implemen- | ND.ending[n]; (G4) E -> b { tation is used in a newly implemented score display sys- LEND.code = LEND.code | E0.code | E1.code; } (4.1) E.code = (b, b.beat, b.dur) ND -> ND [ E :| { (4.2) E.beat = b.beat tem which can model word annotation and section-based ND.count = ND1.count | 1; Figure 5: Parse tree for (b,0,4) (b,4,4) : ∥ (4.3) E.dur = b.dur } arrangement. Because space is limited, we only show ND.ending[ND.count] = E.code; }
88 | 2013 ICMC idea | LONG PAPERS 89 | 2013 ICMC idea | LONG PAPERS 5.2. Context-free Grammar languages. However, not all context-free grammars can DS.al.coda in the DS/DC family and limit our attention be parsed by LR(1). Since LR(1) parsers are balanced in to the “.code” attribute only. A context free grammar (CFG) [1] is used to formalize generality and computational efficiency, we often design Define nonterminal S as the score, L as the left-most the valid sequences of music symbols. A grammar is a set grammars acceptable to LR(1) parsers. part of the score, E as score element, LEND as the ending of rules that describe how a string (or a sequence of sym- within L and ND as the ending. Add two additional sym- bols) is formed. A context-free grammar (CFG) consists 5.3. Ambiguity and Error Handling bols to the original score: # at the beginning and $ at the of terminal, nonterminal, productions, and a start symbol. end. In a CFG, one nonterminal is distinguished as the start The problem of ambiguity arises when there are multi- The score S: to be consistent with case 1, the score symbol. The start symbol is replaced according to pro- ple parse trees for the same the sequence of symbols and needs to be decomposed into an L, the leftmost part that ductions. After replacement, any remaining nonterminal the same grammar. For example, if we change production could have multiple right repeats, and an E followed by can be replaced according to productions, and so on, until (G2) to E E: , we get an ambiguous grammar. The → | the ending sign $. This gives us only terminals remain. score b : b: can be generated two ways: | | For example, we can define S as a whole score and E Figure 6: Data flow in syntax directed translation S -> L $ { as a score element. Then we can define the following: A S => E => E E => E :| E :| => b :| b :| S.code = L.code; } S => E => E :| => E E :| => score S is a sequence of 1 or more musical elements (E), S -> L E $ { E b :| => E :| b :| => b :| b :| S.code = L.code | E.code; } optionally followed by a right repeat. A grammar for this Rule (G1) says the flattened representation of a score can be a flattened score element with the same starting S -> L Segno E ToCoda E DS.coda Coda E $ { language (set of scores) is: The first derivation (line 1) implies a sequence of two S.code = L.code | E0.code | E1.code | beat and duration. Rule (G2) says if the score has a re- repeated blocks. The second derivation (line 2) implies E0.code | E2.code; } Starting: S peat, then the flattened score is two copies of the score nested repeats. Notice that while semantics are not in- (G1) S -> E element before the repeat sign with the starting beat equal The leftmost group L: If there are no non-paired right (G2) S -> E :| herent in formal grammars, we often associate semantics to that of the score element and the duration equal to two repeats in the score, L should be an E. Or this L can pro- (G3) E -> E E closely with productions and parse trees. (G4) E -> b times the duration of the score element. Rule (G3) says duce more L, right repeat sign followed by an E and also if a score element is formed by two sub-score elements, an ending structure, the leftmost multiple endings. 5.4. Syntax Directed Translation and Attribute Gram- Here, we consider “b” to be a terminal denoting any then the flattened element is made by concatenating the block, although we could add productions that expand mar L -> L :| E { flattened representation of sub-elements, and the duration L.code = L1.code | L1.code | E.code; } “b” (now a nonterminal) to terminals of the form “(block A syntax-directed definition is a context free grammar to- is the sum of the lengths of the sub-elements. Finally, if L -> # E { start duration).” Based on this CFG, we are able to tell if gether with attributes and rules. The attributes are asso- an element is a single block, then all the attributes of the L.code = E.code; } a sequence of symbols is formed from this grammar us- ciated with grammar symbols and the rules are used to block are copied to the element (G4). L -> L :| { ing derivation. For example, (b,0,4)(b,4,4): is L.code = L1.code | L1.code; } ∥ define the relation among symbols [13]. For any termi- Based on the parse tree, we can apply rules from leaves L -> LEND E { well-defined from the grammar because we can derive it nal or nonterminal X, denote X.a as some attribute of to root in order to synthesize the attributes of upper levels. L.code = LEND.code | E.code; } from S using the productions: X. For a production like X YZ, the rule can be X.a = For the example shown in Figure 5, the attribute flow is L -> LEND { → L.code = LEND.code; } S => E :| op(Y.b,Z.c) which contains some attribute symbol, an equal shown in Figure 6. sign and some expression op of Y.b and Z.c. Finally the flattened score is in s.code and the dura- => E E :| [use E -> E E] The score element E can be a simple block, a repeated => (b, 0, 4) (b, 4, 4) :| [use E -> b (twice)] Define attribute code as the flattened presentation for tion of the entire score is 16 beats. Although this is a structure, an ending or a DS.al.Coda structure. all the terminals and nonterminals in our formal control simple example, notice that we have precisely and un- We say that a sequence of symbols is a well defined flow framework. Define “ ” as the concatenation opera- ambiguously specified how to interpret score control flow E -> b { | score if it can be derived using a grammar. The result of tor that links two lists of symbols to be one. For exam- notation. Moreover, we have introduced a meta-notation E.code = (b, b.beat, b.dur); } a derivation is a tree structure where each score symbol is ple, if E1.code = (b,0,4) (b,4,4); and E2.code = : (b,8,4) based on attribute grammars that allows us to define other E -> |: E :| { ∥ E.code = E1.code | E1.code; } a leaf and where each parent node and its children corre- : , then E1.code E2.code = (b,0,4) (b,4,4) : (b,8,4) : . interpretations of score control flow. ∥ | ∥ ∥ E -> |: E ND [ E ] { sponds to a production in the grammar. The leaves from Furthermore, define attributes “beat” and “dur” to be the For n = 1 to ND.count left to right (or more precisely in preorder traversal) will starting beat and duration for any terminal or nontermi- 5.5. The Mapping Function E0.code = E0.code | E1.code generate the input sequence. The tree is called a parse nal. For any block b, assume b.beat is the “starting beat” | ND.ending[n]; tree in compilation theory. As an example, the tree for One goal of defining control flow is to obtain a mapping E0.code = E0.code | E1.code | E2.code; } of b and len is the “length” of the block. The syntax di- E -> E E { (b,0,4)(b,0,4): is shown in the Figure 5. from beats in a performance of the score (i.e. the flattened ∥ rected definition for the simple grammar (G1)–(G4) can E0.code = E1.code | E2.code; } A program that converts a sequence of symbols into a be defined as follows: score) to positions or blocks in the original score. We E -> E Segno E ToCoda E DS.Coda Coda E $ { parse tree is called a parser. A parser essentially runs the denote the mapping as f (k), where k is the beat position E.code = E1.code | E2.code | E3.code | grammar “backwards,” reducing a string of nonterminals Starting: S in the flattened score (S.code) and f (k) is the position in E2.code | E4.code; } to the start symbol by applying productions in reverse. (G1) S -> E { the score. The map can be constructed by summing beat (1.1) S.code = E.code ND and LEND: Because of the complication of this Many parsing algorithms have been developed for gram- durations in the flattened representation S.code to obtain (1.2) S.beat = E.beat structure, we need to record each ending to the top level mars of different complexity. LR(1) [10] and LALR(1) k for each block in the original score. (1.3) S.dur = E.dur } by creating a new attribute ND.ending with a list structure. [14] are the most frequently used parsers for programming (G2) S -> E :| { (2.1) S.code = E.code | E.code We also need to use a loop to pair different endings with 5.6. Well Defined Music Control Flow (2.2) S.beat = E.beat the constant part. (2.3) S.dur = E.dur + E.dur } As a demonstration of how we can use this framework to (G3) E -> E E { LEND -> # E ND [ E ] { (3.1) E.code = E1.code | E2.code formalize music control flow for extended music practice, For n = ND.count downto 1 (3.2) E.beat = E1.beat the following grammar supports nested repeats and un- LEND.code = LEND.code | E0.code (3.3) E.dur = E1.dur + E2.dur } limited endings. An even more sophisticated implemen- | ND.ending[n]; (G4) E -> b { tation is used in a newly implemented score display sys- LEND.code = LEND.code | E0.code | E1.code; } (4.1) E.code = (b, b.beat, b.dur) ND -> ND [ E :| { (4.2) E.beat = b.beat tem which can model word annotation and section-based ND.count = ND1.count | 1; Figure 5: Parse tree for (b,0,4) (b,4,4) : ∥ (4.3) E.dur = b.dur } arrangement. Because space is limited, we only show ND.ending[ND.count] = E.code; }
88 | 2013 ICMC idea | LONG PAPERS 89 | 2013 ICMC idea | LONG PAPERS ND -> [ E :| { The intermediate grammar can be used to annotate 7.1. Score Annotation ND.count = 1; word-defined scores as in Figure 3. It is very useful to ND.ending[1] = E; } compile the original score to this intermediate form and Ideally, scores would all be machine readable with con- then use the built-in translator to further compile it to a sistent control flow notation. In reality, we often work It can be shown that this grammar can be parsed by flattened score. The compiler for the intermediate gram- with scanned score images, and optical music recognition LR(1) without ambiguity (“Reduce-reduce” error or “Shift- mar has sophisticated functions such as loop mapping, (OMR) would at best require extensive manual editing to shift” error). “Reduce-shift” errors occur but can be solved which helps to relate static score positions back to mul- produce usable data. Our solution is to import scanned by “preferring reduce then shift.” tiple performance positions (k in f (k)). For example, one or photographed score images and manually annotate the might want to select score position as it occurs in the “2nd elements that are critical for control flow and display. 6. IMPLEMENTATION repeat after the D.S.” For example, consider the score The score annotation interface is provided to import (&,A,1) |: |: (b,0,4) :| (&,A,2) (b,4,4) :| score images and annotate the layout of the score and the We implemented a Java-based API called MCFC (music control flow symbols. This interface is shown in Figure which is translated into intermediate presentation control flow compiler) that creates a score compiler from 7a. There are four main panels: the leftmost contains the a user-defined lexical definition and grammars at start-up (&,A,1) (loop,0) (loop,1) (b,0,4) score structure tools, which are used to draw the system (rep,1,2) (&,A,2) (b,4,4) (rep,0,2) and translates string-format score symbols to a flattened boundary, barlines and places between the barlines where score based on the given grammar. Unlike lex and yacc and finally to the flattened score control flow symbols occur. These places and barlines are [11], which are used frequently for generating code frame- (&,A,1) (b,0,4) (b,0,4) (&,A,2) (b,4,4) called time points. Next to the structure toolbar is the works for making compilers, MCFC generates the entire (b,0,4) (b,0,4) (&,A,2) (b,4,4) symbol toolbar, which is used to place symbols on time compiler directly from the CFG and attribute grammar, points. In the middle is the notation panel where users add Output symbols are marked with labels (below) called flags (a) Notating Mode and users can switch among different grammars without annotations. The rightmost panel is the navigator where that show the mapping from the symbol to loop. The for- closing the application. This feature is especially useful a small-size score is shown along with the string-format mat is [L1,count1;L2,count2,...] where countn is the num- when dealing with notation from different domains. symbols that are used in the score compiler. ber of repetitions of loop Ln. Notice that (b,0,4) appears MCFC does not assume any model of the score and with four distinct loop labels To compile the score and enter the arrangement mode, works in the string representation. The first step for any one can press the “GoLive” menu on the menubar. If application to have this API embedded is to convert the [] [L0,1;L1,1] [L0,1;L1,2] [L0,1] [L0,1] [L0,2;L1,1] [L0,2;L1,2] [L0,2] [L0,2] there are notation errors, a prompt window will display score to a list of symbols as demonstrated in Section 5.1. an error message and suggestion. To switch between dif- To make the compiler understand the semantics of these 6.3. Rearrangement ferent grammars, one can find “switch grammar file” and symbols, the user needs to provide an additional file declar- “switch lex file” in the “GoLive” menu. ing all the symbols, which is known as lexical definition, The other feature of this built-in translator is its ability to and is accomplished using regular expressions similar to handle rearrangement. In the example above, the score is those used in lex [11]. separated into subsections by section marks. 7.2. Arrangement Mode A1-[] (b,0,4) (b,0,4) 6.1. Grammar Definition A2-[L0,1] (b,4,4) (b,0,4) (b,0,4) When the score is compiled, the program enters “arrange- A2-[L0,2] (b,4,4) ment mode” as shown in Figure 7b. The arrangement The grammar definition is put in an additional file with ex- One can input a rearrangement based on the section marks editor features a navigator which highlights the selected tension “.g.” The content of the grammar definition is es- and the loop mapping. section in the “section selector” panel. In the bottom is sentially identical to the attribute grammar shown in Sec- A1-[] A2-[L0,2] A1-[] the section panel where one can insert or delete more sec- tion 5.6. tions. The sections are shown in small square widgets (b) Arragement Mode Then the flattened score is with the name and corresponding loop marks (see Sec- 6.2. The Intermediate Grammar A1-[] (b,0,4) (b,0,4) tion 6.2). One can also add dynamic controls between A2-[L0,2] (b,4,4) sections, which are special repeat signs that can be dy- The score compiler supports a built-in grammar that uses A1-[] (b,0,4) (b,0,4) namically controlled in the performance mode to model a small set of instructions to make control flow more flex- In this way the rearrangement notation used in Figure 3 “repeat until cue” directions. ible. The basic instructions are shown in Table 2. can be compiled.
: Default Intermediate Language Grammar Table 2 7. APPLICATION 7.3. Performance Mode
The score compiler is used in a music score display ap- After the score is arranged, the user can enter Live mode Symbol Meaning plication that, in turn, can be used as part of a human- and set up an HCMP conductor [4, 5] to play with a band. (&,
90 | 2013 ICMC idea | LONG PAPERS 91 | 2013 ICMC idea | LONG PAPERS ND -> [ E :| { The intermediate grammar can be used to annotate 7.1. Score Annotation ND.count = 1; word-defined scores as in Figure 3. It is very useful to ND.ending[1] = E; } compile the original score to this intermediate form and Ideally, scores would all be machine readable with con- then use the built-in translator to further compile it to a sistent control flow notation. In reality, we often work It can be shown that this grammar can be parsed by flattened score. The compiler for the intermediate gram- with scanned score images, and optical music recognition LR(1) without ambiguity (“Reduce-reduce” error or “Shift- mar has sophisticated functions such as loop mapping, (OMR) would at best require extensive manual editing to shift” error). “Reduce-shift” errors occur but can be solved which helps to relate static score positions back to mul- produce usable data. Our solution is to import scanned by “preferring reduce then shift.” tiple performance positions (k in f (k)). For example, one or photographed score images and manually annotate the might want to select score position as it occurs in the “2nd elements that are critical for control flow and display. 6. IMPLEMENTATION repeat after the D.S.” For example, consider the score The score annotation interface is provided to import (&,A,1) |: |: (b,0,4) :| (&,A,2) (b,4,4) :| score images and annotate the layout of the score and the We implemented a Java-based API called MCFC (music control flow symbols. This interface is shown in Figure which is translated into intermediate presentation control flow compiler) that creates a score compiler from 7a. There are four main panels: the leftmost contains the a user-defined lexical definition and grammars at start-up (&,A,1) (loop,0) (loop,1) (b,0,4) score structure tools, which are used to draw the system (rep,1,2) (&,A,2) (b,4,4) (rep,0,2) and translates string-format score symbols to a flattened boundary, barlines and places between the barlines where score based on the given grammar. Unlike lex and yacc and finally to the flattened score control flow symbols occur. These places and barlines are [11], which are used frequently for generating code frame- (&,A,1) (b,0,4) (b,0,4) (&,A,2) (b,4,4) called time points. Next to the structure toolbar is the works for making compilers, MCFC generates the entire (b,0,4) (b,0,4) (&,A,2) (b,4,4) symbol toolbar, which is used to place symbols on time compiler directly from the CFG and attribute grammar, points. In the middle is the notation panel where users add Output symbols are marked with labels (below) called flags (a) Notating Mode and users can switch among different grammars without annotations. The rightmost panel is the navigator where that show the mapping from the symbol to loop. The for- closing the application. This feature is especially useful a small-size score is shown along with the string-format mat is [L1,count1;L2,count2,...] where countn is the num- when dealing with notation from different domains. symbols that are used in the score compiler. ber of repetitions of loop Ln. Notice that (b,0,4) appears MCFC does not assume any model of the score and with four distinct loop labels To compile the score and enter the arrangement mode, works in the string representation. The first step for any one can press the “GoLive” menu on the menubar. If application to have this API embedded is to convert the [] [L0,1;L1,1] [L0,1;L1,2] [L0,1] [L0,1] [L0,2;L1,1] [L0,2;L1,2] [L0,2] [L0,2] there are notation errors, a prompt window will display score to a list of symbols as demonstrated in Section 5.1. an error message and suggestion. To switch between dif- To make the compiler understand the semantics of these 6.3. Rearrangement ferent grammars, one can find “switch grammar file” and symbols, the user needs to provide an additional file declar- “switch lex file” in the “GoLive” menu. ing all the symbols, which is known as lexical definition, The other feature of this built-in translator is its ability to and is accomplished using regular expressions similar to handle rearrangement. In the example above, the score is those used in lex [11]. separated into subsections by section marks. 7.2. Arrangement Mode A1-[] (b,0,4) (b,0,4) 6.1. Grammar Definition A2-[L0,1] (b,4,4) (b,0,4) (b,0,4) When the score is compiled, the program enters “arrange- A2-[L0,2] (b,4,4) ment mode” as shown in Figure 7b. The arrangement The grammar definition is put in an additional file with ex- One can input a rearrangement based on the section marks editor features a navigator which highlights the selected tension “.g.” The content of the grammar definition is es- and the loop mapping. section in the “section selector” panel. In the bottom is sentially identical to the attribute grammar shown in Sec- A1-[] A2-[L0,2] A1-[] the section panel where one can insert or delete more sec- tion 5.6. tions. The sections are shown in small square widgets (b) Arragement Mode Then the flattened score is with the name and corresponding loop marks (see Sec- 6.2. The Intermediate Grammar A1-[] (b,0,4) (b,0,4) tion 6.2). One can also add dynamic controls between A2-[L0,2] (b,4,4) sections, which are special repeat signs that can be dy- The score compiler supports a built-in grammar that uses A1-[] (b,0,4) (b,0,4) namically controlled in the performance mode to model a small set of instructions to make control flow more flex- In this way the rearrangement notation used in Figure 3 “repeat until cue” directions. ible. The basic instructions are shown in Table 2. can be compiled.
: Default Intermediate Language Grammar Table 2 7. APPLICATION 7.3. Performance Mode
The score compiler is used in a music score display ap- After the score is arranged, the user can enter Live mode Symbol Meaning plication that, in turn, can be used as part of a human- and set up an HCMP conductor [4, 5] to play with a band. (&,
90 | 2013 ICMC idea | LONG PAPERS 91 | 2013 ICMC idea | LONG PAPERS 8. SUMMARY AND CONCLUSIONS [4] R. B. Dannenberg, “New Interfaces for Popular Mu- TRACKING RESTS AND TEMPO CHANGES: IMPROVED SCORE sic Performance,” in International Conference on The interpretation of control flow symbols in music no- New Interfaces for Musical Expression, New York FOLLOWING WITH PARTICLE FILTERS tation is an intricate problem that existing music theory University, New York, Jun. 2005, pp. 130–135. solves only in a highly simplified and informal manner. Filip Korzeniowski, Florian Krebs, Andreas Arzt, Gerhard Widmer We have described a framework that borrows from formal [5] ——, “A Virtual Orchestra for Human-Computer languages, formal semantics, and compiler theory. The Music Performance,” in Proceedings of the Interna- Johannes Kepler University Linz, framework allows us to describe precisely what scores tional Computer Music Conference 2011, San Fran- cisco, Aug. 2011, pp. 185–188. Department of Computational Perception, Linz, Austria have valid control flow directives and what these direc- filip.korzeniowski,florian.krebs,andreas.arzt,gerhard.widmer @jku.at tives mean. Furthermore, we showed that we can model [6] T. Gerou and L. Lusk, “Repeat Signs,” in Essen- { } modern practices such as nested repeats and textual di- tial Dictionary of Music Notation. Alfred Pub Co, rectives that have no conventional control flow notation. 1996, p. 110. We can also model the practice we call “arrangement” in ABSTRACT The results seem promising enough to encourage fu- which additional directives override control flow conven- [7] N. Gold and R. Dannenberg, “A reference archi- ture work. So far mostly basic models were used, which, tions to perform the score in a new sequence, including tecture and score representation for popular music In this paper we present a score following system based depending on the complexity of score and performance, on-the-fly arranging such as vamping a section until cued human-computer music performance systems,” in on a Dynamic Bayesian Network, using particle filtering might compromise alignment quality. In this paper we in- or taking an optional cut. International Conference on New Interfaces for Mu- as inference method. The proposed model sets itself apart troduce a real-time score following system, showing how meaningful extensions to the standard position/tempo Beyond theory, we created a flexible implementation sical Expression, Oslo, Jun. 2011, pp. 36–39. from existing approaches by including two new exten- sions: model can improve the alignment. for displaying music notation in live performance, map- [8] Hal Leonard Corporation, The Real Book: Sixth Edi- A multi-level tempo model to improve alignment qual- The contributions of this paper are the extensions of ping the performance position to the score position, look- tion, 6th ed. Hal Leonard Corporation, 2004. ing ahead to page turns and non-sequential jumps in the ity of performances with challenging tempo changes, and the standard position/velocity model typically used so far, score notation, and allowing the user to add annotations [9] W. B. Hewlett, “MuseData : multipurpose represen- an extension to reflect different expressive characteristics especially designed to resolve alignment problems emerg- and arrangements. The Live Score Display system has a tation,” in Beyond MIDI. MIT Press Cambridge, of notated rests. ing when working with complex musical performances. meta-notation system in which new control flow syntax Oct. 1997, pp. 402–447. Both extensions are evaluated against a dataset of clas- Specifically, we will describe an extension to handle rests, can be introduced. sical piano music. As the results show, the extensions im- robust to reverberation and the artist’s expressiveness. We [10] D. Knuth, “On the translation of languages from left prove both the accuracy and the robustness of the algo- will also introduce a multi-level tempo model based on the In the absence of interactive computer music, control Information and control to right,” , vol. 9, pp. 707– rithm. work in [11] to better represent performances with strong flow semantics is mostly a theoretical problem. Regard- 639, 1965. less of theory, the “meaning” of a score is really whatever tempo fluctuations. the performers choose to perform. However, the problem [11] J. Levine, T. Mason, D. Brown, and B. Cupac, Lex 1. INTRODUCTION & yacc, 2nd ed. O’Reilly Media, Oct. 1992. is much more concrete in the context of interactive sys- 2. SYSTEM DESCRIPTION tems. Computers and humans must agree on control flow [12] Michael Good, “MusicXML for Notation and Anal- Score following systems, which listen to a (live) music semantics if we expect to perform conventional scores with ysis,” in The Virtual Score: Representation, Re- performance through a microphone and at any time rec- The overall structure of our system, as depicted in Fig- computers. Computers can “understand” scores or we can trieval, Restoration,, Walter B. Hewlett and Eleanor ognize the current position in the score, facilitate a wide ure 1, is similar to other score followers, and can be di- “tell” computers what they mean. Either way, we need Selfridge-Field, Ed. Cambridge, MA: MIT Press, range of applications. Given robust and accurate score fol- vided in three parts. The only off-line part is the score formal descriptive models to express our intentions. This 2001, vol. 12, pp. 113–124. lowing, the computer can serve as a musical partner to the modeller, which reads the notes from a symbolic repre- work offers an initial step toward this goal. performer(s) by, e.g., automatically accompanying them, sentation of the score and creates a model for each one. [13] J. Paakki, “Attribute Grammar Paradigms Language interacting with them, giving audio-visual feedback, or The feature calculator computes features on the incoming Implementation A High-Level Methodology,” ACM 9. ACKNOWLEDGEMENTS simply turning the pages for them. audio, capturing different kinds of information present in Computing Surveys (CSUR) , vol. 27, pp. 195–255, The task of score following is far from trivial, as such the signal. The matcher connects the outputs of the two The authors would like to thank Nicolas Gold, Gus Xia, 1995. a system has to be able to cope with (possibly extreme) de- parts and computes and estimates the current score posi- and Dawen Liang for discussions and earlier work that set [14] D. Pager, “A practical general method for construct- viations from the score (e.g., in tempo and timing, loud- tion. the stage for this research. This work was partially sup- ing LR(k) parsers,” Acta Informatica, vol. 7, no. 3, ness, structure, left-out/added/changed notes). Classical ported by the National Science Foundation under Grant pp. 249–268, 1977. music, the focus of the proposed system, allows a lot of Score Audio No. 0855958. expressive freedom, and as a consequence these deviation [15] G. Read, Music notation a manual of modern prac- happen constantly. Score Modeller Feature Calculator tice. Boston: Allyn and Bacon, Inc., 1964, pp. 224– Over time, various approaches to this task have been 10. REFERENCES 231. presented, ranging from simple string matching techniques Notes Note Models Spectrum Onsets Loudness [7] to systems based on dynamic time warping with vari- [1] A. Aho, M. Lam, R. Sethi, and J. Ullman, Com- [16] B. Schottstaedt, “Common music notation,” in Be- ous extensions [1, 2], and sophisticated probabilistic mod- pilers: Principles, Techniques and Tools, 2nd ed. yond MIDI. MIT Press Cambridge, Oct. 1997, pp. Matcher els [5, 14]. Prentice Hall, 2007. 217–221. In recent years, several authors proposed the applica- [17] K. Stone, Music notation in the twentieth century: a Score Position [2] W. Apel, “Harvard Dictionary of Music,” in Harvard tion of particle filters to estimate the current performance practical guidebook. W. W. Norton & Company, University Press. Harvard University Press, 1970, position in a continuous “score space” measured in beats. 1980. Figure 1. Overall structure of our score following system p. 696. While [10] and [12] adapt a standard tracking network to the score following problem, [13] presents a multi-level [18] M. Wright, “Open Sound Control: an enabling tech- tracker, switching between position and tempo prediction The different parts are described in detail in the fol- [3] A. W. Crosby, The Measure of Reality: Quantifica- nology for musical networking,” Organised Sound, depending on its tracking confidence. lowing. tion and Western Society, 1250-1600. Cambridge vol. 10, no. 3, p. 193, Nov. 2005. University Press, 1997.
92 | 2013 ICMC idea | LONG PAPERS 93 | 2013 ICMC idea | LONG PAPERS