order, to affect the trajectories of the emitted particles. [8] C. W. Reynolds, “Flocks, herds and schools: A FORMAL SEMANTICS FOR MUSIC NOTATION CONTROL FLOW As an example, particle behaviours might include: distributed behavioral model,” in ACM 1. Bounded spaces and edge effects (e.g. SIGGRAPH Computer Graphics, 1987, vol. 21, ‘bouncing’ trajectories) pp. 25–34. Zeyu Jin Roger Dannenberg 2. Interparticle attraction or repulsion [9] T. Blackwell and M. Young, “Self-organised Carnegie Mellon University Carnegie Mellon University 3. Randomized motion (e.g. drunken walks) music,” Organised Sound, vol. 9, no. 2, pp. 123– 4. Dynamical non-linear systems (e.g. Lorenz 136, 2004. School of Music, Pittsburgh, PA Computer Science Department, Pittsburgh, PA attractors, cellular automata) [10] Wilson, Scott, “Spatial Swarm Granulation.” [email protected] [email protected] 5. Flocking systems (boids, swarms) [Online]. Available: 6. Vortex and other ‘wind’ motions (may not be so http://eprints.bham.ac.uk/237/1/cr1690.pdf. effective in audio) [Accessed: 07-Feb-2013]. ABSTRACT Two other developments that would reap artistic [11] D. Smalley, “Spectromorphology: explaining benefits: sound-shapes,” Organised sound, vol. 2, no. 2, pp. Music notation includes a specification of control flow, 1. A non-linear method for creating grains instead 107–126, 1997. which governs the order in which the score is read us- of the current fixed ‘hop-size with variance’. [12] V. Pulkki, “Virtual sound source positioning using ing constructs such as repeats and endings. Music theory We imagine some kind of ‘swarm’ or ‘burst’ vector base amplitude panning,” Journal of the provides only an informal description of control flow no- algorithm could be applied to the creation times Audio Engineering Society, vol. 45, no. 6, 1997. tation and its interpretation, but interactive music systems
of grains to further simulate environmental need unambiguous models of the relationships between effects the static score and its performance. A framework is in- 2. Quantization of the readhead to the nearest troduced to describe music control flow semantics using attack transient to avoid the traditional theories of formal languages and compilers. A formaliza- Figure 1: Control flow definition in Read’s book ‘smoothness’ of granular envelope attack portions in most granular synthesis algorithms tion of control flow answers several critical questions: Are Finally, we intend to implement a multi-voice version the control flow indications in a score valid? What do the a gap between often simplified theoretical ideals and ac- that will manage multiple soundfiles/buffers control flow indications mean? What is the mapping from tual practice, especially in modern works. In practice, we simultaneously, each buffer being fed to an performance location to static score location? Conven- find nested repeats, exceptions and special cases indicated independently moving emitter. This would allow for a tional notation is extended to handle practical problems, truly ‘symphonic’ approach to spatialized, trajectorial and an implementation, Live Score Display, is offered as by textual annotations, multiple endings, and symbols for granular synthesis. Such an approach would have strong a component for interactive music display. rearrangement. implications for a macrostructural use of spin/drift, by We encountered this gap between theory and practice in the implementation of music notation display software. taking into account the tensional and teleological aspects 1. INTRODUCTION of varying spatial motions, energies and articulations. We needed a formal (computable) way to relate notation Music notation has been evolving for centuries, creating to its performance, and we found conventional notions too The spindrift~ object and sample Max/MSP patch will a symbolic system to convey music information. Early limiting to express what we found in actual printed scores. be available for download from http://michaelnorris.info music notation contained only lines and notes, which are To address this problem, we developed new theoretical in late 2013. sufficient for communicating pitches and durations. It was foundations based upon models of formal language and later that bar lines and time signatures emerged, grouping compilation, and we applied these developments to the 8. REFERENCES music into measures and introducing the idea of beats.1 implementation of a flexible music display system. The notation for music control flow, like repeats and co- Music control flow is the reading order of measures [1] W. T. Reeves, “Particle systems—a technique for das, came even later. Control flow helps to identify repeat- affected by control symbols including the time signature, modeling a class of fuzzy objects,” in ACM ing structures of music and eliminates duplication in the measures, repeats, endings, etc. It can also be viewed for- SIGGRAPH Computer Graphics, 1983, vol. 17, printed score. In the Classical period, control flow nota- mally as a function f that maps the performed beat k to a pp. 359–375. tion is closely tied to music forms such as binary, ternary location of a score,
308 | 2013 ICMC idea | SHORT PAPERS 309 | 2013 ICMC idea | SHORT PAPERS However, the visual definition is incomplete; there are cases where we are not sure which production to use. One such case is nested repeats where there are two ways of grouping the right repeat with the left repeat. In common practice music, we often restrict the number of levels of repeats to be 1, which excludes the nested repeat prob- lem. In more modern and flexible music practice, there are nested structures, text annotations [8] and repetitions conditioned on actual time [17]. There lacks an unam- Figure 4: Symbolizing the score: this score is converted biguous way to formalize the syntax and the meaning of to the following list of strings to represent control flow: such notation. While nested repeats are relatively simple (block, 0, 4) Fine : (block, 4, 4) : : (block, 8, 4) [ to formalize, we argue that the general problem of formal- (block, 12, 4) ] : [ (block, 16, 4) ] DC.Fine izing control flow is non-obvious, interesting, and useful. In this paper, we present a new framework for formal- izing control flow notation based on formal grammar and 5. MODEL FOR EXTENDED MUSIC PRACTICE compilation. One core feature of this framework is that it can be easily designed by humans and applied automati- The framework for defining music notation control flow cally by computer. As an example, we show a formal def- takes three steps: First, abstract the score into symbols; inition of control flow notation that unifies conventional next, define the syntax for a well-formed score based on music notation and some of the most frequent notations Figure 2: Control flow notation and the corresponding formal grammars; finally, define the meaning of each gram- used in modern music practice. We evaluate the ambiguity reading order in common music practice Figure 3: Control flow notation in modern music practice mar using meta expressions. From this, we obtain the and completeness of this definition within our framework. mapping function f (k). Finally, we show the implementation of this method and Book [8]. The six examples6 contain four typical notations its application. sentation. This work also draws attention to arrangement in which performance order is specified as a list of sec- in modern music practice: 5.1. Symbolize the Score tion labels (e.g. “play the intro, verse 1, verse 2, chorus, 2. RELATED WORK Nested repeats (example 1 and 5). For both convenience and clarity, we first convert the vi- chorus”), a common practice in popular music. However, sual score to a list of symbols, each corresponds to a no- the algorithms in this work do not handle nested repeats In Example The score representation and interpretation is essential to Multiple and nested DS/DC-coda notation: tation or a block of notations in the score. Table 1 shows or provide a systematic way to validate the correctness of 3, the notation is complicated. The relation among score writing, printing, auto-accompaniment and conduct- a list of these symbols. To separate the problem of con- notation. Overall, previous work has made great contri- D.C. al coda, to coda and coda signs are connected ing systems. Most works in this field focus mainly on stor- trol flow semantics from the (much) larger and more gen- butions to the notational presentation of the score, while by lines. ing and visualizing the score so as to accommodate both eral problem of music notation semantics, we simply la- the basic questions of music control flow, such as “what traditional and modern music practice, and, as a minor bel each block of notation between control symbols and notation is right” and “what is the correct performance or- Re-arranged sections (Example 2): re-order the section concern, provide room for implementing correct and user- marks to create a different reading order of the score. assume the meaning and duration of a block is known. der given this notation” are still unanswered. In this work, friendly score play-back systems. Early studies such as The procedure is to read the score from left to right, we provide a systematic scheme to define the “right” no- MuseData [9] and Common Music Notation [16] mainly Word annotation (Example 1, 4 and 6): omitting or play- write down each control flow symbol and name the inter- tation and produce the flattened score representation with- concern the encoding of conventional measure-based scores ing some measures conditioned on different rounds vening blocks as shown in Figure 4. out ambiguity. in their original notational presentation. To further en- of repetition. code logical structure of the score and support a wide Table 1: Symbols used to label Control flow notation 3. COMMON MUSIC PRACTICE One could dismiss these particular scores as anomolies, range of modern presentations, structured music descrip- but we find that in music, the exception proves the rule. tion languages like MusicXML [12] offer standards for Frequently used control flow symbols in common practice Existing commercial software interprets Example 5 in- Symbol Meaning score sharing and archiving. More recently, commercial 3 4 music are summarized in Figure 2. To avoid ambiguity, consistently. In Finale 2012 and Sibelius 7, the default (b, k, l) A block of score, starting score printing software like Finale and Sibelius , and 7 8 we often make the following restrictions: reading order is a-b-b-c-b-b-c-d , but in MuseScore , the from beat k with length l score accompaniment software like SmartMusic5 can gen- order is a-b-b-c-a-b-c-d. The correct order is a-b-b-c-a- : and : Left Repeat and Right Repeat erate a music performance from notation. However, the No control flow symbols are allow within the blank b-b-c-d, which differs from both software interpretations. algorithms that map the score to performance are prob- • [ Beginning of an ending staff in Rule 1, 2 and 3. Clearly, we need a more sophisticated approach that offers ] Endpoint of an ending lematic when nested repeats are involved. Given that pop- a syntax and meaning that can be shared by both humans ular score editors attempt to perform scores by following There can only be at most one type of DC/DS sym- DC dal Capo • and computers. In the next section, we show such an ap- DC.Coda D.C. al Coda control flow, it is surprising that they do not define a syn- bols in the entire music. proach based on formal language and attribute grammars. DC.Fine D.C al Fine tax for control flow, check scores for validity, or offer a In Section 6, we show an even more flexible method “the Left and right repeats can be used within the blank DS Del Segno consistent meaning for control flow notation. • intermediate notation” that supports extensions based on staff in Rules 4-9. DS.Coda D.S. al Coda More recent study with an emphasis on music control textual annotations. DS.Fine D.S. al Fine flow modeling led us to develop a score player for HCMP 6 [7]. This work provides data structures and algorithms for From 1 to 6: “Follow Your Heart” by J. Mclaughlin pp.154–155; Segno the Segno sign 4. EXTENDED MUSIC PRACTICE “Forest Flower” by Charles Lloyd p.158; “Good Evening Mr. & Mrs. finding the dynamic score (similar to the “flattened score” America” by John Guerin p.176; “Group your own” by Keith Jarrett Coda the Coda sign in this work) by unfolding the static (original) score pre- In modern music practice, the control flow notation is much p.182; Little Niles by Raudy Weston p.267; “Mallet Man” by Gordon ToCoda To Coda (or Coda symbol) Boek p.282 Fine Fine 3Finale music notation software: http://www.finalemusic.com/ more flexible; there are many examples undefined in the 7One can notate nested repeat by manually specifying which measure 4Sibelius music notation software: http://www.sibelius.com/ conventional notation framework. Figure 3 shows some the right repeat goes to. 8 5SmartMusic software: http://www.smartmusic.com/ of the many unconventional examples found in The Real MuseScore music notation software: http://musescore.org/
310 | 2013 ICMC idea | SHORT PAPERS 311 | 2013 ICMC idea | SHORT PAPERS However, the visual definition is incomplete; there are cases where we are not sure which production to use. One such case is nested repeats where there are two ways of grouping the right repeat with the left repeat. In common practice music, we often restrict the number of levels of repeats to be 1, which excludes the nested repeat prob- lem. In more modern and flexible music practice, there are nested structures, text annotations [8] and repetitions conditioned on actual time [17]. There lacks an unam- Figure 4: Symbolizing the score: this score is converted biguous way to formalize the syntax and the meaning of to the following list of strings to represent control flow: such notation. While nested repeats are relatively simple (block, 0, 4) Fine : (block, 4, 4) : : (block, 8, 4) [ to formalize, we argue that the general problem of formal- (block, 12, 4) ] : [ (block, 16, 4) ] DC.Fine izing control flow is non-obvious, interesting, and useful. In this paper, we present a new framework for formal- izing control flow notation based on formal grammar and 5. MODEL FOR EXTENDED MUSIC PRACTICE compilation. One core feature of this framework is that it can be easily designed by humans and applied automati- The framework for defining music notation control flow cally by computer. As an example, we show a formal def- takes three steps: First, abstract the score into symbols; inition of control flow notation that unifies conventional next, define the syntax for a well-formed score based on music notation and some of the most frequent notations Figure 2: Control flow notation and the corresponding formal grammars; finally, define the meaning of each gram- used in modern music practice. We evaluate the ambiguity reading order in common music practice Figure 3: Control flow notation in modern music practice mar using meta expressions. From this, we obtain the and completeness of this definition within our framework. mapping function f (k). Finally, we show the implementation of this method and Book [8]. The six examples6 contain four typical notations its application. sentation. This work also draws attention to arrangement in which performance order is specified as a list of sec- in modern music practice: 5.1. Symbolize the Score tion labels (e.g. “play the intro, verse 1, verse 2, chorus, 2. RELATED WORK Nested repeats (example 1 and 5). For both convenience and clarity, we first convert the vi- chorus”), a common practice in popular music. However, sual score to a list of symbols, each corresponds to a no- the algorithms in this work do not handle nested repeats In Example The score representation and interpretation is essential to Multiple and nested DS/DC-coda notation: tation or a block of notations in the score. Table 1 shows or provide a systematic way to validate the correctness of 3, the notation is complicated. The relation among score writing, printing, auto-accompaniment and conduct- a list of these symbols. To separate the problem of con- notation. Overall, previous work has made great contri- D.C. al coda, to coda and coda signs are connected ing systems. Most works in this field focus mainly on stor- trol flow semantics from the (much) larger and more gen- butions to the notational presentation of the score, while by lines. ing and visualizing the score so as to accommodate both eral problem of music notation semantics, we simply la- the basic questions of music control flow, such as “what traditional and modern music practice, and, as a minor bel each block of notation between control symbols and notation is right” and “what is the correct performance or- Re-arranged sections (Example 2): re-order the section concern, provide room for implementing correct and user- marks to create a different reading order of the score. assume the meaning and duration of a block is known. der given this notation” are still unanswered. In this work, friendly score play-back systems. Early studies such as The procedure is to read the score from left to right, we provide a systematic scheme to define the “right” no- MuseData [9] and Common Music Notation [16] mainly Word annotation (Example 1, 4 and 6): omitting or play- write down each control flow symbol and name the inter- tation and produce the flattened score representation with- concern the encoding of conventional measure-based scores ing some measures conditioned on different rounds vening blocks as shown in Figure 4. out ambiguity. in their original notational presentation. To further en- of repetition. code logical structure of the score and support a wide Table 1: Symbols used to label Control flow notation 3. COMMON MUSIC PRACTICE One could dismiss these particular scores as anomolies, range of modern presentations, structured music descrip- but we find that in music, the exception proves the rule. tion languages like MusicXML [12] offer standards for Frequently used control flow symbols in common practice Existing commercial software interprets Example 5 in- Symbol Meaning score sharing and archiving. More recently, commercial 3 4 music are summarized in Figure 2. To avoid ambiguity, consistently. In Finale 2012 and Sibelius 7, the default (b, k, l) A block of score, starting score printing software like Finale and Sibelius , and 7 8 we often make the following restrictions: reading order is a-b-b-c-b-b-c-d , but in MuseScore , the from beat k with length l score accompaniment software like SmartMusic5 can gen- order is a-b-b-c-a-b-c-d. The correct order is a-b-b-c-a- : and : Left Repeat and Right Repeat erate a music performance from notation. However, the No control flow symbols are allow within the blank b-b-c-d, which differs from both software interpretations. algorithms that map the score to performance are prob- • [ Beginning of an ending staff in Rule 1, 2 and 3. Clearly, we need a more sophisticated approach that offers ] Endpoint of an ending lematic when nested repeats are involved. Given that pop- a syntax and meaning that can be shared by both humans ular score editors attempt to perform scores by following There can only be at most one type of DC/DS sym- DC dal Capo • and computers. In the next section, we show such an ap- DC.Coda D.C. al Coda control flow, it is surprising that they do not define a syn- bols in the entire music. proach based on formal language and attribute grammars. DC.Fine D.C al Fine tax for control flow, check scores for validity, or offer a In Section 6, we show an even more flexible method “the Left and right repeats can be used within the blank DS Del Segno consistent meaning for control flow notation. • intermediate notation” that supports extensions based on staff in Rules 4-9. DS.Coda D.S. al Coda More recent study with an emphasis on music control textual annotations. DS.Fine D.S. al Fine flow modeling led us to develop a score player for HCMP 6 [7]. This work provides data structures and algorithms for From 1 to 6: “Follow Your Heart” by J. Mclaughlin pp.154–155; Segno the Segno sign 4. EXTENDED MUSIC PRACTICE “Forest Flower” by Charles Lloyd p.158; “Good Evening Mr. & Mrs. finding the dynamic score (similar to the “flattened score” America” by John Guerin p.176; “Group your own” by Keith Jarrett Coda the Coda sign in this work) by unfolding the static (original) score pre- In modern music practice, the control flow notation is much p.182; Little Niles by Raudy Weston p.267; “Mallet Man” by Gordon ToCoda To Coda (or Coda symbol) Boek p.282 Fine Fine 3Finale music notation software: http://www.finalemusic.com/ more flexible; there are many examples undefined in the 7One can notate nested repeat by manually specifying which measure 4Sibelius music notation software: http://www.sibelius.com/ conventional notation framework. Figure 3 shows some the right repeat goes to. 8 5SmartMusic software: http://www.smartmusic.com/ of the many unconventional examples found in The Real MuseScore music notation software: http://musescore.org/
310 | 2013 ICMC idea | SHORT PAPERS 311 | 2013 ICMC idea | SHORT PAPERS 5.2. Context-free Grammar languages. However, not all context-free grammars can DS.al.coda in the DS/DC family and limit our attention be parsed by LR(1). Since LR(1) parsers are balanced in to the “.code” attribute only. A context free grammar (CFG) [1] is used to formalize generality and computational efficiency, we often design Define nonterminal S as the score, L as the left-most the valid sequences of music symbols. A grammar is a set grammars acceptable to LR(1) parsers. part of the score, E as score element, LEND as the ending of rules that describe how a string (or a sequence of sym- within L and ND as the ending. Add two additional sym- bols) is formed. A context-free grammar (CFG) consists 5.3. Ambiguity and Error Handling bols to the original score: # at the beginning and $ at the of terminal, nonterminal, productions, and a start symbol. end. In a CFG, one nonterminal is distinguished as the start The problem of ambiguity arises when there are multi- : to be consistent with case 1, the score symbol. The start symbol is replaced according to pro- ple parse trees for the same the sequence of symbols and The score S needs to be decomposed into an L, the leftmost part that ductions. After replacement, any remaining nonterminal the same grammar. For example, if we change production can be replaced according to productions, and so on, until (G2) to E E: , we get an ambiguous grammar. The could have multiple right repeats, and an E followed by → | only terminals remain. score b : b: can be generated two ways: the ending sign $. This gives us | | For example, we can define S as a whole score and E Figure 6: Data flow in syntax directed translation S->L${ S=>E=>EE=>E:|E:|=>b:|b:| as a score element. Then we can define the following: A S.code = L.code; } S=>E=>E:|=>EE:|=> S->LE${ score S is a sequence of 1 or more musical elements (E), Eb:|=>E:|b:|=>b:|b:| optionally followed by a right repeat. A grammar for this Rule (G1) says the flattened representation of a score S.code = L.code | E.code; } S -> L Segno E ToCoda E DS.coda Coda E $ { language (set of scores) is: The first derivation (line 1) implies a sequence of two can be a flattened score element with the same starting S.code = L.code | E0.code | E1.code | repeated blocks. The second derivation (line 2) implies beat and duration. Rule (G2) says if the score has a re- E0.code | E2.code; } Starting: S (G1) S -> E nested repeats. Notice that while semantics are not in- peat, then the flattened score is two copies of the score (G2) S -> E :| herent in formal grammars, we often associate semantics element before the repeat sign with the starting beat equal The leftmost group L: If there are no non-paired right (G3) E -> E E closely with productions and parse trees. to that of the score element and the duration equal to two repeats in the score, L should be an E. Or this L can pro- (G4) E -> b times the duration of the score element. Rule (G3) says duce more L, right repeat sign followed by an E and also Here, we consider “b” to be a terminal denoting any 5.4. Syntax Directed Translation and Attribute Gram- if a score element is formed by two sub-score elements, an ending structure, the leftmost multiple endings. block, although we could add productions that expand mar then the flattened element is made by concatenating the flattened representation of sub-elements, and the duration L->L:|E{ “b” (now a nonterminal) to terminals of the form “(block A syntax-directed definition is a context free grammar to- L.code = L1.code | L1.code | E.code; } is the sum of the lengths of the sub-elements. Finally, if start duration).” Based on this CFG, we are able to tell if gether with attributes and rules. The attributes are asso- L->#E{ an element is a single block, then all the attributes of the L.code = E.code; } a sequence of symbols is formed from this grammar us- ciated with grammar symbols and the rules are used to block are copied to the element (G4). L->L:|{ ing derivation. For example, (b,0,4)(b,4,4): is define the relation among symbols [13]. For any termi- L.code = L1.code | L1.code; } Based on the parse tree, we can apply rules from leaves well-defined from the grammar because we can derive it nal or nonterminal X, denote X.a as some attribute of L -> LEND E { to root in order to synthesize the attributes of upper levels. from S using the productions: X. For a production like X YZ, the rule can be X.a = L.code = LEND.code | E.code; } → L -> LEND { op(Y.b,Z.c) which contains some attribute symbol, an equal For the example shown in Figure 5, the attribute flow is S=>E:| shown in Figure 6. L.code = LEND.code; } => E E :| [use E -> E E] sign and some expression op of Y.b and Z.c. Finally the flattened score is in s.code and the dura- => (b, 0, 4) (b, 4, 4) :| [use E -> b (twice)] Define attribute code as the flattened presentation for The score element E can be a simple block, a repeated tion of the entire score is 16 beats. Although this is a all the terminals and nonterminals in our formal control structure, an ending or a DS.al.Coda structure. We say that a sequence of symbols is a well defined flow framework. Define “ ” as the concatenation opera- simple example, notice that we have precisely and un- | score if it can be derived using a grammar. The result of tor that links two lists of symbols to be one. For exam- ambiguously specified how to interpret score control flow E->b{ a derivation is a tree structure where each score symbol is ple, if E1.code = (b,0,4) (b,4,4); and E2.code = : (b,8,4) notation. Moreover, we have introduced a meta-notation E.code = (b, b.beat, b.dur); } E->|:E:|{ a leaf and where each parent node and its children corre- : , then E1.code E2.code = (b,0,4) (b,4,4) : (b,8,4) : . based on attribute grammars that allows us to define other | E.code = E1.code | E1.code; } sponds to a production in the grammar. The leaves from Furthermore, define attributes “beat” and “dur” to be the interpretations of score control flow. E->|:END[E]{ left to right (or more precisely in preorder traversal) will starting beat and duration for any terminal or nontermi- For n = 1 to ND.count generate the input sequence. The tree is called a parse nal. For any block b, assume b.beat is the “starting beat” 5.5. The Mapping Function E0.code = E0.code | E1.code | ND.ending[n]; tree in compilation theory. As an example, the tree for of b and len is the “length” of the block. The syntax di- (b,0,4)(b,0,4): is shown in the Figure 5. One goal of defining control flow is to obtain a mapping E0.code = E0.code | E1.code | E2.code; } rected definition for the simple grammar (G1)–(G4) can E->EE{ from beats in a performance of the score (i.e. the flattened A program that converts a sequence of symbols into a be defined as follows: E0.code = E1.code | E2.code; } parse tree is called a parser. A parser essentially runs the score) to positions or blocks in the original score. We E -> E Segno E ToCoda E DS.Coda Coda E $ { grammar “backwards,” reducing a string of nonterminals Starting: S denote the mapping as f (k), where k is the beat position E.code = E1.code | E2.code | E3.code | to the start symbol by applying productions in reverse. (G1) S -> E { in the flattened score (S.code) and f (k) is the position in E2.code | E4.code; } (1.1) S.code = E.code Many parsing algorithms have been developed for gram- the score. The map can be constructed by summing beat (1.2) S.beat = E.beat ND and LEND: Because of the complication of this mars of different complexity. LR(1) [10] and LALR(1) (1.3) S.dur = E.dur } durations in the flattened representation S.code to obtain structure, we need to record each ending to the top level [14] are the most frequently used parsers for programming (G2) S -> E :| { k for each block in the original score. (2.1) S.code = E.code | E.code by creating a new attribute ND.ending with a list structure. (2.2) S.beat = E.beat We also need to use a loop to pair different endings with (2.3) S.dur = E.dur + E.dur } 5.6. Well Defined Music Control Flow the constant part. (G3) E -> E E { (3.1) E.code = E1.code | E2.code As a demonstration of how we can use this framework to LEND->#END[E]{ (3.2) E.beat = E1.beat formalize music control flow for extended music practice, For n = ND.count downto 1 (3.3) E.dur = E1.dur + E2.dur } (G4) E -> b { the following grammar supports nested repeats and un- LEND.code = LEND.code | E0.code (4.1) E.code = (b, b.beat, b.dur) limited endings. An even more sophisticated implemen- | ND.ending[n]; (4.2) E.beat = b.beat tation is used in a newly implemented score display sys- LEND.code = LEND.code | E0.code | E1.code; } Figure 5: Parse tree for (b,0,4) (b,4,4) : (4.3) E.dur = b.dur } ND->ND[E:|{ tem which can model word annotation and section-based ND.count = ND1.count | 1; arrangement. Because space is limited, we only show ND.ending[ND.count] = E.code; }
312 | 2013 ICMC idea | SHORT PAPERS 313 | 2013 ICMC idea | SHORT PAPERS 5.2. Context-free Grammar languages. However, not all context-free grammars can DS.al.coda in the DS/DC family and limit our attention be parsed by LR(1). Since LR(1) parsers are balanced in to the “.code” attribute only. A context free grammar (CFG) [1] is used to formalize generality and computational efficiency, we often design Define nonterminal S as the score, L as the left-most the valid sequences of music symbols. A grammar is a set grammars acceptable to LR(1) parsers. part of the score, E as score element, LEND as the ending of rules that describe how a string (or a sequence of sym- within L and ND as the ending. Add two additional sym- bols) is formed. A context-free grammar (CFG) consists 5.3. Ambiguity and Error Handling bols to the original score: # at the beginning and $ at the of terminal, nonterminal, productions, and a start symbol. end. In a CFG, one nonterminal is distinguished as the start The problem of ambiguity arises when there are multi- : to be consistent with case 1, the score symbol. The start symbol is replaced according to pro- ple parse trees for the same the sequence of symbols and The score S needs to be decomposed into an L, the leftmost part that ductions. After replacement, any remaining nonterminal the same grammar. For example, if we change production can be replaced according to productions, and so on, until (G2) to E E: , we get an ambiguous grammar. The could have multiple right repeats, and an E followed by → | only terminals remain. score b : b: can be generated two ways: the ending sign $. This gives us | | For example, we can define S as a whole score and E Figure 6: Data flow in syntax directed translation S->L${ S=>E=>EE=>E:|E:|=>b:|b:| as a score element. Then we can define the following: A S.code = L.code; } S=>E=>E:|=>EE:|=> S->LE${ score S is a sequence of 1 or more musical elements (E), Eb:|=>E:|b:|=>b:|b:| optionally followed by a right repeat. A grammar for this Rule (G1) says the flattened representation of a score S.code = L.code | E.code; } S -> L Segno E ToCoda E DS.coda Coda E $ { language (set of scores) is: The first derivation (line 1) implies a sequence of two can be a flattened score element with the same starting S.code = L.code | E0.code | E1.code | repeated blocks. The second derivation (line 2) implies beat and duration. Rule (G2) says if the score has a re- E0.code | E2.code; } Starting: S (G1) S -> E nested repeats. Notice that while semantics are not in- peat, then the flattened score is two copies of the score (G2) S -> E :| herent in formal grammars, we often associate semantics element before the repeat sign with the starting beat equal The leftmost group L: If there are no non-paired right (G3) E -> E E closely with productions and parse trees. to that of the score element and the duration equal to two repeats in the score, L should be an E. Or this L can pro- (G4) E -> b times the duration of the score element. Rule (G3) says duce more L, right repeat sign followed by an E and also Here, we consider “b” to be a terminal denoting any 5.4. Syntax Directed Translation and Attribute Gram- if a score element is formed by two sub-score elements, an ending structure, the leftmost multiple endings. block, although we could add productions that expand mar then the flattened element is made by concatenating the flattened representation of sub-elements, and the duration L->L:|E{ “b” (now a nonterminal) to terminals of the form “(block A syntax-directed definition is a context free grammar to- L.code = L1.code | L1.code | E.code; } is the sum of the lengths of the sub-elements. Finally, if start duration).” Based on this CFG, we are able to tell if gether with attributes and rules. The attributes are asso- L->#E{ an element is a single block, then all the attributes of the L.code = E.code; } a sequence of symbols is formed from this grammar us- ciated with grammar symbols and the rules are used to block are copied to the element (G4). L->L:|{ ing derivation. For example, (b,0,4)(b,4,4): is define the relation among symbols [13]. For any termi- L.code = L1.code | L1.code; } Based on the parse tree, we can apply rules from leaves well-defined from the grammar because we can derive it nal or nonterminal X, denote X.a as some attribute of L -> LEND E { to root in order to synthesize the attributes of upper levels. from S using the productions: X. For a production like X YZ, the rule can be X.a = L.code = LEND.code | E.code; } → L -> LEND { op(Y.b,Z.c) which contains some attribute symbol, an equal For the example shown in Figure 5, the attribute flow is S=>E:| shown in Figure 6. L.code = LEND.code; } => E E :| [use E -> E E] sign and some expression op of Y.b and Z.c. Finally the flattened score is in s.code and the dura- => (b, 0, 4) (b, 4, 4) :| [use E -> b (twice)] Define attribute code as the flattened presentation for The score element E can be a simple block, a repeated tion of the entire score is 16 beats. Although this is a all the terminals and nonterminals in our formal control structure, an ending or a DS.al.Coda structure. We say that a sequence of symbols is a well defined flow framework. Define “ ” as the concatenation opera- simple example, notice that we have precisely and un- | score if it can be derived using a grammar. The result of tor that links two lists of symbols to be one. For exam- ambiguously specified how to interpret score control flow E->b{ a derivation is a tree structure where each score symbol is ple, if E1.code = (b,0,4) (b,4,4); and E2.code = : (b,8,4) notation. Moreover, we have introduced a meta-notation E.code = (b, b.beat, b.dur); } E->|:E:|{ a leaf and where each parent node and its children corre- : , then E1.code E2.code = (b,0,4) (b,4,4) : (b,8,4) : . based on attribute grammars that allows us to define other | E.code = E1.code | E1.code; } sponds to a production in the grammar. The leaves from Furthermore, define attributes “beat” and “dur” to be the interpretations of score control flow. E->|:END[E]{ left to right (or more precisely in preorder traversal) will starting beat and duration for any terminal or nontermi- For n = 1 to ND.count generate the input sequence. The tree is called a parse nal. For any block b, assume b.beat is the “starting beat” 5.5. The Mapping Function E0.code = E0.code | E1.code | ND.ending[n]; tree in compilation theory. As an example, the tree for of b and len is the “length” of the block. The syntax di- (b,0,4)(b,0,4): is shown in the Figure 5. One goal of defining control flow is to obtain a mapping E0.code = E0.code | E1.code | E2.code; } rected definition for the simple grammar (G1)–(G4) can E->EE{ from beats in a performance of the score (i.e. the flattened A program that converts a sequence of symbols into a be defined as follows: E0.code = E1.code | E2.code; } parse tree is called a parser. A parser essentially runs the score) to positions or blocks in the original score. We E -> E Segno E ToCoda E DS.Coda Coda E $ { grammar “backwards,” reducing a string of nonterminals Starting: S denote the mapping as f (k), where k is the beat position E.code = E1.code | E2.code | E3.code | to the start symbol by applying productions in reverse. (G1) S -> E { in the flattened score (S.code) and f (k) is the position in E2.code | E4.code; } (1.1) S.code = E.code Many parsing algorithms have been developed for gram- the score. The map can be constructed by summing beat (1.2) S.beat = E.beat ND and LEND: Because of the complication of this mars of different complexity. LR(1) [10] and LALR(1) (1.3) S.dur = E.dur } durations in the flattened representation S.code to obtain structure, we need to record each ending to the top level [14] are the most frequently used parsers for programming (G2) S -> E :| { k for each block in the original score. (2.1) S.code = E.code | E.code by creating a new attribute ND.ending with a list structure. (2.2) S.beat = E.beat We also need to use a loop to pair different endings with (2.3) S.dur = E.dur + E.dur } 5.6. Well Defined Music Control Flow the constant part. (G3) E -> E E { (3.1) E.code = E1.code | E2.code As a demonstration of how we can use this framework to LEND->#END[E]{ (3.2) E.beat = E1.beat formalize music control flow for extended music practice, For n = ND.count downto 1 (3.3) E.dur = E1.dur + E2.dur } (G4) E -> b { the following grammar supports nested repeats and un- LEND.code = LEND.code | E0.code (4.1) E.code = (b, b.beat, b.dur) limited endings. An even more sophisticated implemen- | ND.ending[n]; (4.2) E.beat = b.beat tation is used in a newly implemented score display sys- LEND.code = LEND.code | E0.code | E1.code; } Figure 5: Parse tree for (b,0,4) (b,4,4) : (4.3) E.dur = b.dur } ND->ND[E:|{ tem which can model word annotation and section-based ND.count = ND1.count | 1; arrangement. Because space is limited, we only show ND.ending[ND.count] = E.code; }
312 | 2013 ICMC idea | SHORT PAPERS 313 | 2013 ICMC idea | SHORT PAPERS
ND->[E:|{ The intermediate grammar can be used to annotate ND.count = 1; word-defined scores as in Figure 3. It is very useful to SONICMAPS: CONNECTING THE RITUAL OF THE ND.ending[1] = E; } compile the original score to this intermediate form and CONCERT HALL WITH A LOCATIVE AUDIO URBAN then use the built-in translator to further compile it to a EXPERIENCE It can be shown that this grammar can be parsed by flattened score. The compiler for the intermediate gram- LR(1) without ambiguity (“Reduce-reduce” error or “Shift- mar has sophisticated functions such as loop mapping, shift” error). “Reduce-shift” errors occur but can be solved which helps to relate static score positions back to mul- Ignacio Pecino Ricardo Climent by “preferring reduce then shift.” tiple performance positions (k in f (k)). For example, one NOVARS Research Centre, NOVARS Research Centre, might want to select score position as it occurs in the “2nd University of Manchester, UK University of Manchester, UK 6. IMPLEMENTATION repeat after the D.S.” For example, consider the score (&,A,1) |: |: (b,0,4) :| (&,A,2) (b,4,4) :| We implemented a Java-based API called MCFC (music which is translated into intermediate presentation control flow compiler) that creates a score compiler from a user-defined lexical definition and grammars at start-up (&,A,1) (loop,0) (loop,1) (b,0,4) ABSTRACT This situation could be described as Augmented (rep,1,2) (&,A,2) (b,4,4) (rep,0,2) and translates string-format score symbols to a flattened Virtuality. score based on the given grammar. Unlike lex and yacc and finally to the flattened score Physical space is often used as a means for sound organization. Furthermore, our environment can be [11], which are used frequently for generating code frame- (&,A,1) (b,0,4) (b,0,4) (&,A,2) (b,4,4) sonically augmented, highlighting hidden aspects of it or works for making compilers, MCFC generates the entire (b,0,4) (b,0,4) (&,A,2) (b,4,4) proposing a whole new reinterpretation. On this basis, compiler directly from the CFG and attribute grammar, Output symbols are marked with labels (below) called flags we present SonicMaps, a new locative audio tool and and users can switch among different grammars without that show the mapping from the symbol to loop. The for- complete solution for sound geolocation. Using a closing the application. This feature is especially useful mat is [L1,count1;L2,count2,...] where countn is the num- number of sensors on mobile devices, the SonicMaps when dealing with notation from different domains. ber of repetitions of loop Ln. Notice that (b,0,4) appears application virtually places sounds into real space MCFC does not assume any model of the score and providing panning and amplitude information. These with four distinct loop labels Figure 1. The Virtuality Continuum. works in the string representation. The first step for any sounds are played back as we walk into predefined areas application to have this API embedded is to convert the [] [L0,1;L1,1] [L0,1;L1,2] [L0,1] [L0,1] [L0,2;L1,1] [L0,2;L1,2] [L0,2] [L0,2] which can be overlapped into layers according to score to a list of symbols as demonstrated in Section 5.1. compositional purposes. A custom Dynamic Content On the other hand, real space can also be augmented To make the compiler understand the semantics of these Server (DCS) automatically updates audio files for 6.3. Rearrangement incorporating elements from the digital domain, which symbols, the user needs to provide an additional file declar- selected locations according to a set of environment are not originally present but are displayed using some ing all the symbols, which is known as lexical definition, The other feature of this built-in translator is its ability to variables (time, temperature, etc.) creating a dynamic kind of interface (Augmented Reality). This is the and is accomplished using regular expressions similar to handle rearrangement. In the example above, the score is piece of music that emphasizes the link between sound baseline for the sound geolocation tool we will be those used in lex [11]. separated into subsections by section marks. materials and their assigned physical location. describing here. A1-[] (b,0,4) (b,0,4) Additionally, a OSC module has been implemented in order to send users positions and sensor data to a remote 1.2. Locative Audio 6.1. Grammar Definition A2-[L0,1] (b,4,4) (b,0,4) (b,0,4) A2-[L0,2] (b,4,4) server. This data can then be musically interpreted in a concert hall to establish creative links between distant In that context, Locative Audio can be defined as a type The grammar definition is put in an additional file with ex- One can input a rearrangement based on the section marks spaces. of Augmented Reality based on sonic experiences tension “.g.” The content of the grammar definition is es- and the loop mapping. sentially identical to the attribute grammar shown in Sec- (Augmented Aurality). It works by adding a layer of A1-[] A2-[L0,2] A1-[] tion 5.6. 1. INTRODUCTION auditory information to our real-world environment and Then the flattened score is consequently modifying or enhancing our perception 1.1. Extending Space and understanding of a particular place. Nowadays, this 6.2. The Intermediate Grammar A1-[] (b,0,4) (b,0,4) can be achieved by using existing sensors on mobile A2-[L0,2] (b,4,4) By definition, space is a boundless, three-dimensional A1-[] (b,0,4) (b,0,4) devices, like GPS or a solid state compass, which track The score compiler supports a built-in grammar that uses extent in which objects and events occur and have our position/orientation and render this layer a small set of instructions to make control flow more flex- In this way the rearrangement notation used in Figure 3 relative position and direction. We usually use this term accordingly. Frauke Behrendt discussion about ‘the ible. The basic instructions are shown in Table 2. can be compiled. referring to the physical extent in which we live, and is sound of locative media’ sets a clear framework to subject to Newtonian laws. Such an extent and explain how we can listen to space by walking and thus : Default Intermediate Language Grammar everything that exists within, is what we perceive as Table 2 7. APPLICATION remixing a number of placed sounds [1] . Such an Reality. However, the introduction of computer- approach contrasts with other geotagged audio projects, generated 3D environments, although being simulated The score compiler is used in a music score display ap- where a collection of field recordings are just displayed Symbol Meaning and not necessarily governed by these natural laws, also plication that, in turn, can be used as part of a human- on a website using the Google Maps API or other offers an instance of space that conforms to the given (&,
314 | 2013 ICMC idea | SHORT PAPERS 315 | 2013 ICMC idea | SHORT PAPERS