Leon 1 Encoding Non-Standard Forms of Music Notation Using MEI (May 2016) Matthew Leon, Music & Computer Science, University of Virginia Class of 2017

I. ABSTRACT II. MEI Since the 1950’s, computers have been utilized MEI is an open-source Extensible Markup heavily to further the study of music. Now, music is Language (XML) vocabulary for musical notation. conceived, recorded, produced, stored, and XML, a hierarchical encoding system, can be used consumed almost entirely with the assistance of to represent virtually any type of document in a computers. Though music and computers are deeply simple, machine-readable format. MEI utilizes intertwined in the modern age, there is still a XML to encode symbolic musical information in a disconnect between music notation and computer way that can be easily parsed and utilized by representation. As an avid lover of both music and computers. For example, the simple musical phrase computer science, I became intrigued with this void shown in Figure 1 can be represented as MEI code on the left. in . What is the proper way to store and symbolize musical scores? Is there a way to encode scores that do not follow standardized rules of notation? Inspired by both the old and new methods of computer music composition and technology, this project seeks to overcome the boundaries between music notation and digital encoding.

Though several established formats for standard digital music encoding exist, it becomes quite difficult to use these formats to encode graphic Figure 1 - A musical phrase represented as MEI markup. notation. Scores like György Ligeti’s ‘,’ Each piece of information associated with the George Crumb’s ‘ Galaxy,’ Baude Cordier’s musical phrase is encoded as MEI data. Though the ‘Belle, Bonne, Sage,’ ’s ‘Music for musical notation and MEI code do not look very Airports,’ and ’s similar, they both are equivalently representing the ‘Elektronische Studie II’ all contain unique forms of same musical phrase. Similarly, entire scores can be notation which are nearly impossible to represent translated and represented using MEI, as using standard means of encoding. How can an demonstrated in Appendix A. Encoding in this unusual shape be encoded in the same way that a format allows more functionality and interaction with the score, as I attempted to execute in this standard chord or note would be encoded? Using project. Stockhausen’s 1954 piece as a proof-of-concept, this paper explores the limitations and capabilities of MEI (Music Encoding Initiative) as a tool for III. MEI’S LIMITATIONS encoding non-standard graphic scores. One of the first issues I encountered with MEI was

its inability to represent non-standard scores in a Using a combination of Python, MEI, and suitable format. The glaring issue that stood out to JavaScript, I detail below a method for the encoding me was the lack of a standard framework for and representation of graphic musical notation as encoding pieces with non-conventional notation, well as the creation of an interactive score utilizing like those found in Appendix B. Since MEI does not this encoding. currently allow for an easy encoding of this type of

Leon 2 score, I attempted to devise a framework for implementing my plan much more difficult. First, encoding a non-standard score. the score was produced after the piece was written, and was not created by Ligeti himself, which means IV. PROJECT OVERVIEW the score is not a perfect representation of the piece. After understanding the limitations of MEI, I Without a straightforward and exact representation wanted to create a project that pushed the of the piece, an interactive and playable score boundaries in order to adapt to these limitations. would be virtually impossible. Furthermore, the First, after finding a suitable score, I planned to piece is laden with ambiguity, since there are many formulate a method for encoding each bit of images overlaid atop each other. Using computer information in the piece. Using this method, the graphics image tracing, isolating individual shapes entire score would be encoded using MEI. Once would be very difficult, and I found it to be a represented as MEI, a website would be created that fruitless task. Finding this piece too difficult to could interpret this data and display it as an encode, I decided to pursue another path. interactive visual score. Armed with a long-term plan, I set out to find a fitting score.

V. LIGETI’S ‘ARTIKULATION’ The first piece I examined was György Ligeti’s 1958 piece ‘Artikulation.’ This piece is accompanied by Rainer Wehinger’s 1970 graphic notation which was created several years after the composition of the piece, and breaks the piece into several different symbols. The graphic score, shown in Figure 2, is vibrant and interesting to look at, and I attempted to use this piece as my sample encoding. Figure 2 - Rainer Wehinger's graphic score for György Ligeti's 1958 piece 'Artikulation.' Initially I intended to use scans of the graphic score that would be processed using Adobe Illustrator’s VI. STOCKHAUSEN’S ‘STUDIE II’ computer graphics to generate Scalable Vector Graphics (SVG) files which could then be linked to After much searching, I settled on Karlheinz MEI. Once encoded, I wanted to create a timeline Stockhausen’s 1954 piece, ‘Elektronische Studie view of the score which would link each SVG II,’ pictured in Figure 3. I chose this piece for object to a sound from the original piece. After the several reasons. Stockhausen is one of the most SVG objects were linked, the score could be influential and important electronic composers of th interactive and playable, displaying a description of the 20 Century, and several of his works have been each note event so that information would be critical in fueling my interest in . In available for every note. Each note could also be addition, Stockhausen’s piece was one of the first synthesized in order to hear the note individually. early electronic music compositions to utilize this type of graphic score. I began the task of analyzing each note of the score, using the visual programming language Max/MSP The piece was formulaically composed, with to process an image of the score and convert the relatively distinct and separate segments, allowing image using spectral analysis, but ultimately this me to write a framework for encoding the piece in a method generated too much noise and was unable to much easier way than Ligeti’s piece allowed. create a clean analysis of the piece. Furthermore, Stockhausen’s piece is confusing to understand simply by looking at it; a deep I attempted a few other methods to encode the understanding of the piece is necessary in order to Ligeti piece, but a few issues arose that made

Leon 3 interpret the score, so encoding the piece would lowest and highest frequency of each note mixture, allow for easier interpretation. the rest of the component frequencies can be calculated. Initially, I attempted to calculate each frequency from the lowest and highest frequencies VII. ELEMENTS OF ‘STUDIE II’ listed on the vertical axis, but my results were not After choosing my piece, I confronted the initial consistent with Stockhausen’s results due to problem with this score: its underlying meaning is Stockhausen’s use of gratuitous rounding. Since obscured by the unique nature of the notation. The oscillators were not very advanced in 1954, score consists of rectangles denoting note mixtures, Stockhausen often rounded, resulting in varying and triangles denoting dynamics. Though the score effects. In order to alleviate this issue, I created an looks quite different than standard notation, both array containing each frequency in the score, and hold essentially the same information. Every used the index of the highest and lowest frequencies rectangle is essentially a chord containing five to reference the frequencies stored in the array. The notes, and each triangle details the attack or decay next element, volume, is indicated by the triangles of the chord. on the bottom of the score. Each shape correlates directly with the envelope of volume for each note, Each note mixture has three defining characteristics: and thus the increase or decrease in volume over pitch, volume, and duration. Pitch is indicated on time can be calculated by determining the side of the vertical axis, shown by the numbers on the left the triangle. Finally, the duration of each note is that correspond to frequencies in hertz. The scale of indicated by the numbers along the middle of the frequencies is based on an interval between each score. Each value is shown in centimeters, which step, which consists of the frequency proportion of corresponds to the length of tape played at 76.2 the 25th root of 5, as compared to the traditional centimeters per second. Since Stockhausen chose tempered tuning system with the proportion of the the duration of each note mixture by cutting strips 12th root of 2. Each rectangular note mixture of , each number indicates the length consists of five component frequencies, each of tape rather than the duration of the mixture in equally spaced using this scale. By determining the time. In order to calculate the duration of each note

Figure 3 - Karlheinz Stockhausen's graphic score for the formulaically composed 1954 piece 'Elektronische Studie II'.

Leon 4 mixture in seconds, the tape length value can be definitions are shown in Appendix E. Finally, the divided by 76.2. calculated note mixtures are appended to the XML tree, and exported as an MEI file. The code for XML appending is shown in Appendix F, and a VIII. ENCODING ‘STUDIE II’ portion of the rendered MEI file is shown in After understanding each independent element of Appendix G. the score, I set about manually encoding the score by identifying five pieces of information for each note mixture: the bottom frequency index, top IX. MAKING THE WEB APP frequency index, starting position in centimeters, Now that the MEI data was correctly generated, I duration in centimeters, and the starting or ending began the task of doing something useful with the position of the envelope in decibels. Once I had file. I decided to create an interactive version of the derived each piece of information, I entered them score, hosted as a web application. Since this was into a spreadsheet (see Appendix C), which was my first time doing any sort of web development, I converted into a comma separated value (CSV) file. first took several online tutorials in HTML, CSS, Each frequency represented in the score was also and JavaScript. Using the JavaScript library jQuery, entered into CSV file, used to populate an array for I first embarked on the task of parsing MEI, using pitch calculations. Next, the data was parsed by a jQuery’s parseXML() function. Once the XML was python script that uses each piece of information parsed correctly, I included the image of the score from the CSV encoding to calculate useful values as the background of the web page, and overlaid which are then rendered as an MEI file. The python invisible buttons on top of each note mixture. I code first builds an XML tree with each appropriate created a class called hover that changed the element, in preparation for each note mixture’s transparency and color of the buttons when information to be appended after the CSV file is hovering on top of them, resulting in an interactive read. experience with the score. I then linked each button to the corresponding note mixture, using jQuery’s MEI does not contain all elements required to dialog function to display the information parsed follow my data classification system, so it was from MEI. This implementation, while it looked necessary to create a few new elements. Beginning adequate on the surface, had several issues. One with the standard MEI score, the appropriate title, major issue with this implementation was the hard- composer, and header information is applied to the coded nature of each button. Since I hard-coded document object model (DOM). Next, a new body, each pixel value on the image, the buttons appeared mdiv, score, section, measure, staff, and layer are in different locations depending on the screen size, created. Inside the layer, each note mixture is and would move outside of the proper locations represented as a chord. This chord contains the when zoomed out. In addition, the buttons were slope represented as ‘envelope,’ and the starting difficult to overlay on top of the envelope triangles, position as ‘position.’ Using MEI’s ‘tstamp_gs’ so I decided to pursue another method. attribute, we represent the duration of the chord in seconds. Contained within each chord are the five Using an HTML image map generator and James frequencies in hertz, shown using the ‘freq’ trait. Treworgy’s ImageMapster plugin, I was able to link Next, the frequency list is populated by iterating every object on the score and create a more through each frequency in the CSV file. interactive experience. Using this plugin, the website allows the selection of both note mixtures After that, each piece of information from the CSV and envelopes, as well as a link between each note file of note mixtures is parsed and sent to a function mixture and its corresponding envelope. More than which calculates and creates a note mixture object one dialog can be open at one time, allowing note to be added to an array of notes, called song. The mixtures to be compared and to be understood more code for CSV parsing code is shown in Appendix deeply. Once the image mapping was implemented, D, and the code for note mixture and song class I added audio functionality by embedding the

Leon 5

X. ISSUES There are a few issues I encountered in the playback of the score that I was unable to correct. Due to inconsistency between Stockhausen’s recording and the score, some note mixtures are slightly skewed. This becomes evident during playback of some note mixtures that contain pieces of other note mixtures that should not be overlapping. Even with scrupulous editing of the sound file, overlapping still occurred. I have not yet resolved a way to fix this problem, other than cutting the recording into smaller sound files to alleviate the issue.

XI. CONCLUSION Currently, only one page of the twenty-six-page score is encoded and interactive. With a bit of time

Figure 4 - A screenshot of the web app, demonstrating a highlighted it would be relatively easy to encode the full piece, note mixture and its accompanying dialog. The dialog displays each since a framework for encoding has already been component frequency along with envelope, position and duration. established. In the future, I would like to add more The ‘Play’ button plays the note mixture in the context of the original recording, beginning at the position parameter and playing for the interactive synthesis elements, such as selectable duration listed in the dialog. The ‘Synthesize All’ button plays a pitches so that the score can be heard with and synthesized version of all the component frequencies together, while without certain frequencies. I would also like to add the following ‘Synthesize’ buttons play the synthesized component frequencies individually. play-through functionality so that the entire piece can be automated and each note mixture will highlight as the piece plays. This project serves as a original audio file into the website and using proof-of-concept of the capabilities of MEI. Though HTML’s audio player so that Stockhausen’s 1954 this piece was more straightforward to encode than realization can be played and understood in the some other graphic scores, by breaking any score context of the score. I then added more interactive into its fundamental pieces and creating a program elements in the dialog box, allowing audio playback to encode each piece, any score has the potential to and synthesis. Using the duration and starting be encoded using MEI. In the future, I would like to position elements from the MEI file, each note attempt to encode a different score, possibly one mixture has a starting point and duration that is with more difficult elements and characteristics. linked to the recording. When the ‘play’ button is clicked, the note mixture is played from the context of the recording, beginning at the starting position and ending after the duration of the note mixture has passed. Using HTML’s AudioContext, I added synthesis capabilities, so that each note mixture can be synthesized independently from the recording. Each frequency parsed from MEI can be synthesized one-by-one, or all frequencies can be synthesized together to give an impression of the whole note mixture. An example of a highlighted note mixture and its accompanying dialog is shown in Figure 4. The web application can be accessed at the following link: www4.iath.virginia.edu/mei/ML

Leon 6

Appendix A

Figure A.1 - An example score in standard musical notation (on the left) and a “full” encoding of that score in MEI markup (on the right). The MEI encoding is “full” in the sense that, if done properly, it captures the components of the score that can be considered to precisely represent the intended music.

Figure A.2 - 'Spiral Galaxy' from Volume I of George Crumb's 1972 Figure A.3 - ‘Belle, Bonne, Sage,’ composed by Baude composition 'Makrokosmos.' This piece's spiral orientation is quite Cordier features different colorations and a heart-shaped interesting, with features conventional notation simply twisted in a spiral score. The red notes indicate variations in note lengths from pattern. Since the musical semantics intended by the spiral pattern are not their normal lengths. The piece, focusing on the theme of clear, I moved on to examine other potential pieces. love, has heart-shaped note clusters, and would also be interesting to encode. Yet the piece, like Crumb’s, uses conventional notation in an unconventional manner that is understood to be beyond the scope of MEI encoded.

Leon 7

Figure A.4 - A diagram of Brian Eno's 1978 ambient album 'Music for Airports.' Despite my fondness for the recordings and the intriguing accompanied diagram, it is difficult to interpret as a score since each diagram represents a system used to make music, rather than individual notes. Using this system, it would be very difficult to encode based on the massive amount of controlled randomness.

Appendix B

Figure B.1 - A spreadsheet including the encoded data from the first page of Studie II. The first column indicates the bottom index of the note mixture, which corresponds to the index in an array containing the note mixture’s bottom Figure B.2 - The python code to parse through both data files. First, the frequency list file is frequency. The second column indicates the starting position parsed, populating an array with each frequency indicated in Stockhausen’s instructions. Next, a of the note mixture in centimeters. The third column indicates song() object is created to hold each note mixture. For each line in the encoded-data spreadsheet the top index of the note mixture, which similarly (see Figure B.1), a new note mixture is created, with each relevant piece of information corresponds to the index in an array containing the note calculated by the calc_frequencies() method. Finally, the new note mixture is added to the song mixture’s top frequency. The fourth column indicates the object. duration of the note mixture in centimeters. The fifth column indicates the dB of the ending envelope slope.

Leon 8

Figure B.3 - The python code’s class definitions for note_mixture and song objects. Each note_mixture object has five frequencies, duration, envelope slope, and starting position. These parameters are calculated and set in the calc_frequencies() method, which takes the raw data from the ‘markup.csv’ file and performs the appropriate calcuations and value assignments for each piece of information.

Figure B.4 -The python code for creating the XML tree. For each note_mixture in the song, a chord is created which contains duration as tstamp_gs, envelope, position, and each frequency as a note. The tree is then exported as ‘meifile.mei.’

Leon 9

Figure B.5 - The rendered MEI file, containing the pertinent information for each note mixture and for the piece of Studie II shown in Figure 1.

Leon 10

WORKS CITED

Ligeti, György, and Ranier Wehinger. Artikulation: Elektronische Musik = Electronic Music. Mainz:

Schtt, 1994. Musical Score.

Madden, Blake. "Brian Eno – “Ambient 1: Music For Airports”." Trust Me I'm a Scientist. N.p., 3 Mar.

2014. Web. 07 June 2016.

music-for-airports/>.

Muchmore, Pat. "Scoring Outside the Lines." Opinionator Scoring Outside the Lines Comments. N.p., 3

Aug. 2011. Web. 12 May 2016.

lines/>.

"Music Encoding Initiative." MEI. Music Encoding Initiative, 2016. Web. 12 May 2016.

encoding.org/>.

Stamp, Jimmy. "5 1/2 Examples of Notation." Smithsonian. N.p., 5 June 2013. Web.

12 May 2016.

notation-92223646/?no-ist>.

Stockhausen, Karlheinz, Nr. 3 Elektronische Studien. Studie II. Wien: Universal Edition, 1956. Print.