Dynamic Generation of Musical Notation from MusicXML Input on an Android Tablet
THESIS
Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in the Graduate School of The Ohio State University
By
Laura Lynn Housley
Graduate Program in Computer Science and Engineering
The Ohio State University
2012
Master's Examination Committee:
Rajiv Ramnath, Advisor
Jayashree Ramanathan
Copyright by
Laura Lynn Housley
2012
Abstract
For the purpose of increasing accessibility and customizability of sheet music, an application on an Android tablet was designed that generates and displays sheet music from a MusicXML input file. Generating sheet music on a tablet device from a
MusicXML file poses many interesting challenges. When a user is allowed to set the size and colors of an image, the image must be redrawn with every change. Instead of zooming in and out on an already existing image, the positions of the various musical symbols must be recalculated to fit the new dimensions. These changes must preserve the relationships between the various musical symbols. Other topics include the laying out and measuring of notes, accidentals, beams, slurs, and staffs. In addition to drawing a large bitmap, an application that effectively presents sheet music must provide a way to scroll this music across a small tablet screen at a specified tempo. A method for using animation on Android is discussed that accomplishes this scrolling requirement. Also a generalized method for writing text-based documents to describe notations similar to musical notation is discussed. This method is based off of the knowledge gained from using MusicXML.
3
Dedication
This document is dedicated to my family.
4
Acknowledgments
I thank my advisor, Rajiv Ramnath, for all his help and guidance in the completion of this project and thesis. I also thank Peter Rogers and Tony Wells for their vision and support. I additionally thank Michael Jewitt for his role as my teammate for the initial part of my project.
5
Vita
May 2007 ...... Avon Lake High School
June 2011 ...... B.S. Computer and Information Science,
The Ohio State University
September 2011 - present ...... Graduate Student, Department of Computer
Science and Engineering, The Ohio State
University
Fields of Study
Major Field: Computer Science and Engineering
6
Table of Contents
Abstract ...... 3
Dedication ...... 4
Acknowledgments...... 5
Vita ...... 6
Fields of Study ...... 6
Table of Contents ...... 7
List of Tables and Figures...... 9
Chapter 1: Introduction ...... 10
1.1: Organization of this Document ...... 12
Chapter 2: Background Information ...... 13
2.1: Basic Sheet Music Description ...... 13
2.2: Android ViewGroups and Layouts ...... 16
2.3: Visual Impairments and Music ...... 17
Chapter 3: Related Research ...... 19
Chapter 4: Implementation ...... 23
4.1: Music Data Structure...... 24 7
4.2: Layout/ViewGroup Hierarchy ...... 25
4.3: Measure Width ...... 26
4.4: Note Placement ...... 26
4.5: Laying Out and Measuring Accidentals ...... 28
4.6: Laying Out Beams...... 29
4.7: Laying Out Slurs and Ties...... 30
4.8: Multiple-Staff Layout and Overlap ...... 32
4.9: Maximizing Tablet Screen Space...... 33
4.10: Method for Nesting Inside Android Relative Layouts ...... 35
4.11: Drawing Musical Symbols ...... 37
4.12: Color Changes ...... 38
4.13: Smooth Scrolling ...... 38
Chapter 5: Insights in Text to Image Generation ...... 40
Chapter 6: Conclusions ...... 44
6.1: Contributions ...... 44
6.2: Future Work ...... 44
Works Cited ...... 46
8
List of Tables and Figures
Figure 1: Simple Note and Rest Durations ...... 14
Figure 2: Beamed Eighth Notes ...... 15
Figure 3: Labeled Music ...... 16
Figure 4: Music Data Structure ...... 24
Figure 5: Layout Hierarchy ...... 25
Figure 6: Distance Between Lines is 1/14th of the Height ...... 27
Figure 7: Accidentalview ...... 28
Figure 8: Beamviews ...... 29
Figure 9: Overlapping whitespace ...... 32
Figure 10: Two Parts Versus One Part ...... 34
Figure 11: Dotted Half Note, Rest, Petal, Fermata, Treble clef...... 37
Figure 12: MusicXML Excerpt ...... 41
9
Chapter 1: Introduction
It is currently standard for sheet music to be created in a one-size-fits-all paper format. This music is printed and distributed in music stores and available online for musicians to print out. It contains black print on white paper, and is generally printed on
9x12 inch paper with staff lines 1/16th of an inch apart. While this may be an acceptable format for the average music reader, it is not adequate for readers with low vision. Also some people prefer music in a different color scheme than black on white. Some people with visual impairments prefer white on black or yellow on black. People with cataracts read faster with white text on a black background (Elliott). These issues motivate a solution of dynamically generating sheet music in a desired size and color set by the user.
To make this process easier for the user, an application that runs on a tablet device can be used to allow the user to set these parameters and immediately see the regenerated music.
This thesis focuses on the process of generating musical notation on an Android tablet from a text-based input and displaying it to the user. This thesis describes the algorithms and data structures used to create an application that inputs a MusicXML file and displays the corresponding musical notation on an Android 3.0 tablet in a size and color scheme specified at run time.
This thesis has several goals. The first is to create an application that is capable of reading in the MusicXML format on an Android tablet and displaying the notes and symbols in musical notation. The second is that the music be aesthetically pleasing and resemble regular sheet music. The third goal is that the music scrolls across the screen at
10 a specified tempo. A final goal is that the techniques used for this application can apply to other musical formats besides MusicXML.
This application contributes to the field in that it meets the goals specified above and is the first application to do so. Currently visually impaired musicians have several options available to access music, but many are seen as undesirable. Sheet music can be enlarged to three times its original size and printed, but this requires frequent page turnings while playing. This application does not require any page turnings because it scrolls through the music. Other mechanical devices used to aid visually impaired musicians are expensive and heavy. For example, the Lime Lighter is a standalone tablet-like device that displays enlarged music that retails around $3,995. This price does not include the software that costs several hundred more dollars (Lime). Creating an application that can work on a multi-functional tablet has a greater accessibility than an expensive device that only has music-related purposes.
Other less-altruistic contributions include the knowledge gained from the process of moving from text to musical notation. For instance, the limitations of looking at a technical image as a tree-like structure are discussed. Additionally, experience dealing with the issues in using the MusicXML file format have given insight on the process and requirements of describing an image in words.
11
1.1: Organization of this Document
Chapter 2 gives a brief overview of the musical notation used in this discussion.
It outlines the types of symbols that will be used and their meanings. It also explains how
Android lays out and measures views in a Layout and ViewGroup hierarchy. Types of low vision and their effect on reading sheet music are discussed. Chapter 3 surveys related research in the fields covered. Chapter 4 discusses the implementation of the project. Chapter 5 discusses the insight gained on text to image generation. Chapter 6 includes conclusions and future work to be done.
12
Chapter 2: Background Information
Any discussion of the details of programming sheet music requires a basic understanding of the pieces that compose it and how they are structured to form sheet music. An explanation of dynamically programming graphics on Android using
ViewGroups and Layouts is also helpful to the discussion of drawing the music. Finally, visual impairments and their relationship to sheet music are discussed.
2.1: Basic Sheet Music Description
Sheet Music is a representation of a piece of music using symbols and accepted musical notation so that two users can read the same piece of sheet music and output the same song. There may be some room for interpretation for the musician, such as how loud to play the music, but the time and pitch at which each note should be played is explicitly stated. This allows many musicians to play the same music at the same time in a group setting such as an orchestra. It is read left to right.
Sheet Music is separated into multiple staffs(staves) to provide a convenient way to illustrate that multiple notes are being played at the same time. Each staff has five staff lines. These lines are used to show the pitches of the notes. If two notes are in the same horizontal position, they should be played concurrently. These staffs are representative of one or more parts. For example, a musical piece with three staffs often represents two parts: the first part containing one staff with the melody and lyrics, and the second containing two staffs for the piano to play. In this instance, someone could sing the song while using the second part to play the piano.
13
The length of time a note should be played is indicated by its shading, stem, and flags. Whole notes are not shaded and have no stem. Half notes are held for half as long as whole notes, and quarter notes are held for half as long as half notes. Half notes are not shaded but have a stem. Quarter notes are shaded and also have a stem. Eighth, sixteenth, and thirty-second notes follow the same pattern. For each type of note, there is a corresponding type of rest. Rests occur when no note is being played for the specified duration.
Figure 1: Simple Note and Rest Durations Notes whose lengths are in between the simple durations are marked with a dot.
When a note has a dot next to it, its length is 1.5 times as long as it would be without a dot. For example, a dotted quarter note has the length of one quarter note plus one eighth note.
The vertical position of a note in relation to the staff lines indicates the pitch of the note. Within a single staff, a higher note represents a higher pitch. Sometimes notes need flat, natural, or sharp signs next to them to show more information about the pitch.
14
An understanding of the specific pitches in a piece of music is not necessary for the discussion of how to display sheet music.
Left to right, the music is divided into measures. Measures help the reader keep track of the beat. Each piece of music has a time signature which describes how many notes of each type are included in each measure. Time signatures consist of two numbers: the bottom represents the type of note the signature is based on, and the bottom number represents how many of this base note are in each measure. For instance, a time signature of ¾ indicates that there will be three quarter notes per measure. A 3/2 time signature would mean there are three half notes per measure. This is just the length of each measure, so these three quarter notes could be written as six eighth notes or one half note and one quarter note. All that matters is that the total duration of the notes and rests adds to three quarter notes for a ¾ measure.
Beams are used to connect notes and prevent a piece of music from being filled up with flags off of stems. When several eighth notes are to be played in a row, they are connected together to show they are played as a group. Eighth notes use single beams, sixteenth notes use double beams and thirty-second notes use triple beams.
Figure 2: Beamed Eighth Notes (Eighth) There are many other symbols found in musical notation, but an understanding of how notes and rests are displayed is enough to follow the general procedure for
15 displaying music. In Figure 3, some of the discussed musical symbols are labeled in a sample piece of music. This piece has two parts – one of which has one staff and the other has two staves. Three and a half measures are at least partially visible in this screen capture.
Figure 3: Labeled Music
2.2: Android ViewGroups and Layouts
Drawing custom graphics in Android can be done by using the View class and extending its onDraw method. Views can be contained inside already existing layout classes such as RelativeLayout or inside custom ViewGroups. The latter gives more control on laying out and measuring the child views inside the ViewGroup.
16
When extending the ViewGroup class, the onMeasure and onLayout methods should be overridden. In the onMeasure method, a ViewGroup needs to measure out each of its children. A ViewGroup can tell what its own width and height are here, and it is able to give each of its children some fraction of that height in size. If a ViewGroup measures a child with a greater size than its own, the child view’s edges will be cut off when they are displayed. This is similar to laying a small picture frame on top of a large picture. The onLayout measure tells Android where the child views should be drawn within the ViewGroup. This is called after onMeasure, so a ViewGroup can use information it saved on what size the child views were drawn if needed. In onLayout, the
ViewGroup lays out each child at specific coordinates within the ViewGroup. This tells the view the borders of where it is allowed to draw.
Custom Views need to override the onMeasure and onDraw methods. Typically the onMeasure method is just used so that the view can store how big it should be. The onDraw method contains all the work for drawing the view. The method onDraw has a parameterized canvas on which it can draw circles, lines, and other shapes in different colors. The size of this canvas has been set by the parent, so the view is free to fill up the entire width and height it has been given.
2.3: Visual Impairments and Music
There are several types of visual impairments that impact a person’s ability to read regular sheet music. These impairments can cause low vision, small field of vision, and lack of focus. Some visual impairments are severe enough to require braille for
17 reading sheet music. In these cases, a music application on a tablet device would not be helpful.
Low vision describes a wide range of impairments and can be severe or mild.
Visual acuity is measured with two numbers in the form XX/YY, where XX represents the distance of the test, and YY represents the distance that a person with average vision can see the letters. For instance, 20/20 vision means that the 20/20 vision line on the vision chart can be read at 20 feet away. 20/60 vision means that the person can read the lines from 20 feet away that a person with 20/20 vision could read from 60 feet away
(Watt). Mild low vision starts at 20/30, and near-total blindness is defined as anything worse than 20/1000. Legal blindness is defined as anything worse than having no eye better than 20/200 or having a field of vision of less than or equal to 20 degrees (Low
2006).
Low vision takes many forms, and does not always mean that the person’s vision is all faded equally. For example, some people lose their central vision, peripheral vision, or night vision. Other examples of issues causing low vision are tunnel vision, distortion, and multiple field loss. Multiple field loss means that there are multiple spots missing from a person’s field of vision (Low 2010). People with low vision are defined as such when their impairments are not correctable.
There are many possible causes of low vision. These include macular degeneration, diabetic retinopathy, retinal detachment, cataracts, and glaucoma. Different diseases may lead to different impairments. For instance, macular degeneration causes blurred vision and possibly causes a blind spot, while glaucoma usually produces
18 symptoms of night blindness and loss of peripheral vision (Low 2006). Others with low vision have issues seeing different color combinations. For example, yellow on black might be preferable to some people, but other people would be unable to distinguish between these colors enough to read the music.
These visual impairments require changes to sheet music. Based on visual acuity alone, low vision requires that sheet music be magnified larger than it is normally printed.
Issues with field of vision present challenges for scrolling the music. Because a person with blind spots cannot see the entire screen at once, it is important that the music be scrolled in a way that people can continuously read the music. Continuous smooth- scrolling would achieve this affect, since the music that is being played is always in the same position. Shifting the music several measures at a time would probably not be acceptable for people with a low field of vision, because they would not be able to follow where their eye should look next. Shifting instead of scrolling could be better for people who have a hard time reading scrolling/moving music. Color issues require customizability of the colors of the sheet music. Some would prefer a light background with dark text, and others would prefer a dark background with light text.
Chapter 3: Related Research
Currently, software exists that generates sheet music in a format similar to the paper format. Most notably, Finale can generate paper music from a MusicXML file
(Finale). This runs on a desktop computer, and it does not have a version for tablets.
Sheet music could be generated into an image using Finale and then transferred to a tablet. However, the ability to resize, hide parts, and change colors would be lost as soon
19 as that transfer was made. Also, there would be no way to scroll the music across the tablet screen automatically. Sheet music created by Finale is intended to be printed and read. Because larger sheet music takes up many pages and necessitates frequent page turns, this is not sufficient for the needs of those with low vision.
Research has been done on general image resizing for people with low vision.
People desiring a larger image do not generally want everything to be resized at the same rate. For example, the whitespace in between two tables need not be resized. If whitespace is resized at the same rate as everything else, more paper than necessary is used because fewer items can fit on a page. This explains why enlarged sheet music is not accepted as a great solution. Page turns have to happen too frequently. In order to resize a document properly, logical relationships between different elements in the document need to be known. This way, the logical relationships can be preserved. This is rarely the case for documents being resized. Therefore, document resizing for the visually impaired is often done with a human intervening and making adjustments during the process (Connolly 2010). Conveniently, MusicXML provides information on the logical relationships between elements, so it is capable of being resized properly. This understanding of relationships between elements means that sheet music can be automatically drawn at a larger size without the need for human intervention.
Other work has been done in Adaptive Document Layout (ADL) to address the challenges in dynamically laying out documents on tablet-like devices. Tablets create a unique opportunity for dynamic document layout, and this has not yet been taken advantage of as much as is possible. Most of what has been done has focused on laying
20 out text. However, other items in documents such as images can be dynamically placed based on screen size as well (Jacobs, 2003; Jacobs, 2004).
To fulfill the purpose of captions, images and captions need to be kept together.
Resizing and laying out these items together must be done to preserve this relationship.
This can be done by nesting these objects so that a caption is a child of an image.
Documents are usually viewed as a grid, and items with a parent-child relationship like images and their captions are nested within the grid. Other items such as an article title and the body of an article have a relationship that requires the title go at the top of the page and the body go somewhere below the title. This is also done using a grid and requires that information about all the objects be used to determine the size and location of each object in the grid (Jacobs, 2003: Jacobs, 2004).
This thesis connects the dots in the areas of research of music generation and document resizing. Music is a perfect candidate for this because of its finite set of symbols and defined relationships between elements. Therefore, it expands upon the research done on resizing documents for the visually impaired that concluded that human involvement was important in resizing. This may be true for other documents, but music does not need a human to approve the resizing and laying out of elements; it can be done purely with automation. This thesis illustrates the complexities in doing this automatically, and succeeds in accomplishing this for many elements of musical notation.
This thesis also serves to extend the ideas discussed in Adaptive Document
Layout. Musical notation differs from document layout because there are stricter rules on the location of most items. While a document can display its images in different
21 locations, music requires exact coordinates for its notes. However, some items within music can be shifted without losing meaning. For example, lyrics do not have to exactly under the notes they are tied to for their meaning to be known. Lyrics and notes have a similar relationship to images and captions in ADL. These types of relationships can use the idea of nesting objects to keep them together in dynamically creating musical notation. Additionally, music can be viewed as a grid-like structure when considering the layout and positions of the staffs and parts. The vertical spacing between different staffs is similar to the vertical spacing between different elements in a document like sections of an article or images.
22
Chapter 4: Implementation
The application runs in three stages: parsing the XML into the music data structure, building an Android ViewGroup hierarchy from the music data structure, and scrolling the bitmap created from the ViewGroup hierarchy across the screen at a set tempo. This chapter focuses on the challenges from these three stages. Because of the close connection between building the music data structure and the layout of the music, these stages will be discussed together multiple times to show the challenges of building different musical elements, such as notes, accidentals, beams, slurs, ties and the staffs that hold them. Additionally, methods for drawing musical symbols using the graphics available on Android are explained.
23
4.1: Music Data Structure
The music data structure is represented by a class called Score. Its basic structure is based off of the MusicXML document. A Score consists of multiple Parts, and each
Part consists of multiple Measures. Each Measure contains multiple Notes. Because the
MusicXML contains display information as well as pure musical information, this display information is also captured in the Score class. These structures derive their names from the MusicXML document. The next figure shows the general but simplified structure of the music (What).
Figure 4: Music Data Structure
24
4.2: Layout/ViewGroup Hierarchy
The Score class is “printed” into a layout hierarchy by giving all of the higher- level classes print methods that return a ViewGroup or RelativeLayout. For example, the printScore function creates a RelativeLayout and calls printPart on each of the Parts within that Score class. It then adds the RelativeLayouts returned by the calls to printPart to its own RelativeLayout. While doing this, it tells the smaller RelativeLayouts their size and positions within the larger RelativeLayout. That way, each RelativeLayout only needs to worry about the space it is given instead of worrying about relative coordinates between the other layouts. For example, the state of a part’s RelativeLayout contains no information about other parts being drawn above or below it. The following figure shows the basic ViewGroup/Layout hierarchy, while leaving out a few symbols such as fermatas that do not add to the discussion.
Figure 5: Layout Hierarchy
25
4.3: Measure Width
MusicXML files often give the width of each measure. This usually differs from measure to measure since the MusicXML is set up to allow close to exact reproduction of the original printed sheet music. This is fine when the music is stationary and scrolling is done with the eyes, but poses a problem when scrolling is done automatically. Because each measure should be on the screen for the same amount of time, a fixed measure width is required. Otherwise the speed of scrolling would need to constantly change. Being able to change the width of the measures is a requirement of the application, so this number is not used at all to set the width. Instead the application defaults to three measures on the screen at a time, and the user may increase or decrease this number of measures. A whole number of measures needs to be on the screen at any time to ensure that the notes currently being played are on the screen at the correct moment.
When the user changes the width of the measures, the entire layout needs to be recreated with the new width values. This width value used for all of the measures is stored in a Singleton class. This allows any view to access this value at any time.
4.4: Note Placement
As shown in the previous figure, notes are laid out within a single staff of a single measure. When they are laid out, they are centered on an x-coordinate within the measure. Some MusicXML files provide this x-coordinate directly. When this is the case, the x-coordinate assumes the specified measure width, which was ignored. The actual x-coordinate within the measure is found by multiplying the given x-coordinate by
26 the user-set width and dividing by the MusicXML width. This scales the coordinate to the wider or skinnier measure.
Notes are given an x-coordinate, but they are also given horizontal space for room for things like accidentals. To make things easier, they are given the entire width of the measure. Because lyrics are attached to notes, a long one-syllable word could fill up a large portion of the measure, so in that instance giving most of the measure would be needed anyway. Notes are given the entire height of their parent, a single staff of a single measure. This height can be used by the NoteViewGroup to do all of the needed measuring and laying out. The distance between two staff lines is 1/14th of the height of the staff. This allows for the four spaces in between the lines on the staff and five spaces above and below. In the next image, the lines marked with the red borders show the actual staff lines drawn for the staff, and the lines above and below illustrate how much extra room there is for higher and lower notes.
Figure 6: Distance Between Lines is 1/14th of the Height This 1/14th ratio is used by the NoteViewGroup to set the width and height of its children. For instance, a NoteHeadView gets exactly the distance between two lines as its width and height. This distance is commonly referred to as the line to line height.
27
4.5: Laying Out and Measuring Accidentals
Oftentimes, notes are marked with an accidental symbol such as a sharp, flat, or natural sign. These change the pitch of the note, so they are vital for properly displaying the music. They are listed inside the MusicXML as tags under the note they modify, and they are stored within the NoteViewGroup because their position and size is completely based on the position and size of the note head they modify. Typically they sit directly to the left of the note head. Because they require drawing to the canvas, a class called
AccidentalView was created to draw them.
An AccidentalView is laid out by the NoteViewGroup that contains it so that it may draw across its entire width and height. In the following example, a natural sign is drawn. The NoteViewGroup measures the natural sign so that it has the same height as the NoteViewGroup and a width of the line to line height. It lays the AccidentalView out in a space directly to the left of the note head. The right border of this space is the x- coordinate of the note, and the left border of this space can be calculated by subtracting the line to line height.
Figure 7: AccidentalView
28
4.6: Laying Out Beams
As demonstrated in Figure 5, BeamViews are contained inside the RelativeLayout for a Measure. As a reminder, measures within a part may have multiple staffs. It may seem curious that BeamViews are at such a high level in the hierarchy, but the reason is that beams can cross between two staffs. Typically a Measure’s vertical space is divided evenly among its SingleStaffFromSingleMeasureViewGroups. Beams, however, require the vertical space of an entire Measure to accommodate this cross-staff possibility.
When the RelativeLayout for a Measure is being created, an ArrayList of
BeamViews is used to give each BeamView information about the notes that it beams together. The algorithm starts by creating the empty ArrayList of BeamViews. For each note in the measure, a check is performed to see if it contained a beam tag. If the note begins a beam, a BeamView is created. This BeamView is passed the coordinates of that note. Then this BeamView is added into the ArrayList. If the note continues a beam, the beam it continues is found in the ArrayList, and this note’s information is added to that beam. If this note ends a beam, the corresponding BeamView it ends is removed from the ArrayList. This note’s information is added to the BeamView, and the BeamView is added to the Measure’s layout. Once this has happened, that BeamView is out of the list and it cannot interfere with future beams with the same id number.
Figure 8 shows the resulting BeamViews. Because they are given so much space to draw themselves, they must be given the actual stem heights of the notes they connect.
Figure 8: BeamViews
29
4.7: Laying Out Slurs and Ties Slurs and ties are similar to beams in that they connect notes, but different because they can cross measures. This requirement means that their views must be inside the Part’s RelativeLayout instead of the Measure’s RelativeLayout. Ties are simpler than slurs because they always connect two notes that are next to each other and that have the same pitch. Both algorithms are similar to the algorithm for BeamViews, but the difference occur mainly when a measure is crossed. Because of the similarities between slurs and ties, only ties will be discussed.
Ties are kept in the Measure class, while TieViews are created in the Part class.
This distinction occurs because the tags indicating that notes are tied are all of the way down in the note elements of the XML. Instead of pulling them all the way to the Part class immediately, it is easier to process them and access their notes while they are still in the Measure class. The Measure class contains an ArrayList of Ties. When a Measure object is being created, each note is checked to see if it contains a tie tag. If it contains the beginning of a tie, a Tie object is created and passed information about that note.
This Tie is then added to the ArrayList of Ties. If the note ends a tie, then it either ends a tie from the previous measure or one from the current measure. If it is from the previous measure, a new Tie is created with the note’s information and marked as being from an old measure. If it is from the current measure, then the Tie is found from the ArrayList of
Ties and the note’s information is added. When the Part is being parsed and created, it then checks each measure to see if it has any ties that need to be merged. If a measure does have a tie without an end, it goes into the next measure and takes that note’s information and adds it to the Tie. 30
The algorithm for creating TieViews works by going measure by measure through a part. If the measure contains any ties, then the ArrayList of Ties from that Measure is pulled up into the Part. These ties are iterated through and a TieView is created for each one of them. Each of the TieViews gets the information about the Notes from the corresponding Tie. Then the TieViews know the exact coordinates of where they should draw themselves.
31
4.8: Multiple-Staff Layout and Overlap
If this application had no concerns about making the notes as large as possible, it would just divide the vertical space evenly between the different parts and staffs.
However, the discussion earlier demonstrated that there must be a significant amount of whitespace above and below the staff lines to fit in notes of a large range. Most of the time, this whitespace goes unused. Therefore, a compromise was reached allowing these whitespace areas to overlap. As demonstrated in Figure 9, the usage of overlap allows for significantly larger note font. This is desirable in an application geared toward the visually impaired. However, it can lead to note collisions if a note from a lower staff lands in the same place as one from the staff above it. Instead of restricting the note size based on this possibility, a decision was made to allow the user to increase the height to the point of a collision if desired. Some users may decide that larger notes are worth one or two collisions.
Figure 9: Overlapping Whitespace The height of the music set by the user and the amount of overlap are stored as two separate doubles in a Height Singleton class. A height fraction of 1.0 means that the
32 music should take up the entire music display area. A height fraction of 0.5 means that the music should take up 50% of the display area. An overlap fraction of 1.0 would put two staffs exactly on top of each other. For this reason, the maximum overlap fraction is
0.5. This means that 50% of each of the staffs are on top of each other. Because overlap should only be used when necessary, any height increases requested by the user will first bring the height fraction toward 1.0 before raising the overlap fraction above
0.0. When a height decrease is requested, the opposite applies. The overlap fraction is first brought down to zero before the music starts taking up less than 100% of the display area. This height singleton is called upon while the Score and Parts lay out their children, so they can overlap them. With overlap, the height of all the part’s RelativeLayouts added together is greater than the total height of the music display area. The height of all of the SingleStaffFromSingleMeasureViewGroups also adds to a number greater than the height of a part.
4.9: Maximizing Tablet Screen Space
Additional efforts besides overlapping parts were made to maximize the screen space of the tablet. Primary testing of this application was done on a Motorola Xoom which has a 10.1 inch diagonal. It has a resolution of 1280x800, so maximizing vertical and horizontal space was important. First, the signature measure was fixed to the left of the music display area. Signature measures contain the time and key signature, and are necessary so that the user can check which notes should be modified and how many beats are in a measure. The signature measure was made to be a fraction of the width of a regular measure, so that it takes up as little horizontal room as is necessary. Music
33 readers tend to look ahead from the measure they are currently playing, so more measures on the screen at a time is desirable, and a smaller signature measure accomplishes this.
Additionally, the ability to hide certain parts was added. As shown in Figure 10, this significantly increases the size of the notes. Oftentimes a user would only play one part of the song at a time, so the ability to remove an unneeded part was required. This was implemented in the layout stage of the application. If a part was marked as hidden, the Score did not add it to its RelativeLayout. It also made the other part’s
RelativeLayout larger by informing the height singleton that fewer total staffs were being drawn.
Figure 10: Two Parts Versus One Part Horizontal space was also saved by being careful to include a minimum number of buttons on the screen. Width, height, tempo, restart, and a link to the options menu were determined to be the bare minimum of buttons needed on the score page. 85% of the height was given to the music display area, while 15% of the height went to the button area. Any smaller and the font of the buttons would have been too small to read, especially for the visually impaired.
34
4.10: Method for Nesting Inside Android Relative Layouts
Several methods for laying out Views and ViewGroups inside other custom
ViewGroups in Android have been discussed. Dynamically laying out ViewGroups inside a RelativeLayout is a different process. This process is not stated in Android documentation primarily because the assumption is made that xml files are mostly used to place things inside RelativeLayouts. Dynamically doing this is briefly mentioned but not sufficiently explained.
RelativeLayout.LayoutParams are required to manipulate the size and placement of something inside a RelativeLayout. Because of the repeated creation of this type of parameter, this application contains a helper class called LayoutHelper to create them.
This class contains an overloaded static function that returns a
RelativeLayout.LayoutParams. The first step of this function is to create a new
RelativeLayout.LayoutParams object with the specified width and height of the child that is going inside of the RelativeLayout. If these are not given, the object is created to match the size of the parent using RelativeLayout.LayoutParams.MATCH_PARENT.
Next, the margins on all sides of the object are set. By default, we set these margins to be zero. Margins of size zero mean that the entire space of the RelativeLayout will be filled by the child. If a different, smaller rectangle needs to be used, these margins are set to accommodate this. The margins are not coordinates, they are the size of the space between the rectangle and the edge of the RelativeLayout on each side. Finally, four rules are added to the RelativeLayout.LayoutParams so that the child aligns with the parent. This is done using one call for each direction similar to
35 params.addRule(RelativeLayout.ALIGN_PARENT_RIGHT). Without these rules, the child layout will not be placed in the parent RelativeLayout in the way the programmer would expect. Using this helper class streamlines the process for adding children to
RelativeLayouts and produces consistent results.
36
4.11: Drawing Musical Symbols
Until this point, measuring and laying out musical symbols has been discussed in detail, but the actual onDraw methods of the Views have been almost ignored. All symbols are drawn on Android’s canvas using Android’s 2D graphics. Symbols are drawn as a combination of lines, circles, and arcs. Figure 11 shows some close up examples of the notes and symbols featured in the application. Looking closely at them, it is easy to see the small shapes that create them. For example, the treble clef contains four arcs, one line and a shaded circle. These small shapes are carefully connected using geometry to create a larger image. The note heads are drawn as circles, but in reality note heads on regular sheet music are not perfect circles. The thickness of the lines is calculated as a fraction of the line to line height.
Figure 11: Dotted half note, rest, petal, fermata, treble clef
37
4.12: Color Changes
The ability to change the colors of the music is actually very simple. The options menu allows the user to set a background and foreground color. An options singleton object is also kept that contains these colors. When the user changes one of these colors, the singleton is updated. Because any change requires that the RelativeLayout structure be recreated, the custom Views are able to pull these new colors in their onDraw methods. The notes, symbols, and text are drawn with the foreground color. The options menu does not allow the user to select a similar background and foreground color. This limit stops the user from being able to change the menu buttons to black on black or white on white and lose the ability to change them back. Currently white, black, yellow, and blue are available as colors for the background and foreground because they were requested by people with visual impairments.
4.13: Smooth Scrolling
Scrolling smoothly across the tablet remains a partially unsolved requirement.
The scrolling is fairly smooth, but experiences occasional jumps due to unresolved performance issues. Nevertheless, the method used to scroll is discussed here.
A bitmap is first created from the RelativeLayout and View structure discussed previously and then added to a MusicAnimationView. On Android, the bitmap creation is accomplished by creating a canvas from a bitmap and drawing the RelativeLayout to that canvas. The MusicAnimationView extends SurfaceView, and interacts with a running thread that constantly updates a sprite that takes up the entire screen. Whenever the run method of the thread is called, it tells the sprite to update itself. The sprite does
38 this by calculating the position the music should be at on the screen and moving there.
The desired position is calculated by storing a starting position, time, and tempo. This tempo is converted to pixels/ms. When the update call comes through to the sprite, it checks the current time and compares it to the initial time. Using the tempo, a distance from the initial position is calculated. Moving the bitmap is done by shifting the rectangle in which it is displayed on the screen.
Currently this method is getting around 30 fps, which is fast enough to appear smooth. However, the scrolling experiences occasional jumps due to lack of performance. A few times a second, there is a longer gap between two frames, which causes the music to have to jump a longer distance at once. This long distance does not appear smooth to the eye, and it almost looks as if it is moving backwards for a moment.
No solution has yet been found, but it is most likely a performance issue somewhere in the code of the application instead of the Android operating system.
39
Chapter 5: Insights in Text to Image Generation
Music provides an interesting example of something that is best viewed as an image but can be described in text that is also readable by people. In fact, it is fairly simple to read a MusicXML file and draw the sheet music out by hand. This is because it is easy for a person to switch back and forth between drawing notes and the beams that connect them. Many types of notations are simpler to draw than music, such as mathematical notation. In mathematical notation, elements are grouped by their parentheses and elements do not cross between parenthesized groups, so it is easy to picture them as a tree-like structure. Other types of technical documents are much more complicated, such as technical drawings. They differ from music notation in that they have many types of large, overlapping symbols. Music has a small, finite set of symbols that generally don’t overlap, but have connectors running between them to show their relationships. This makes it a good candidate for storing in a human-readable text document. Other notations with this type of structure include structural formulas for chemical compounds and state diagrams for grammars.
The basic idea behind MusicXML is that it lists out each individual element with the connections it has underneath that element. The next figure shows an example of a note. This note has its own information such as pitch, duration, and type, but it also has two beams and a slur connected to it. The other notes connected to those beams and slur would have similar tags.
40
Figure 12: MusicXML Excerpt MusicXML does separate the notes into parts and measures, but this is for readability. It would be more consistent with the typical format if part and measure were marked by tags like the
MusicXML so that it is not a totally flat structure of notes. For this discussion, we will focus on the notes and their tags instead of going up a level and analyzing the measures and parts.
Another noteworthy item from the figure is that MusicXML lists relative coordinates for notes within a measure. This information immediately tells the system the precise location of the note, but it is actually unnecessary. The main purpose of this
41 is to allow the regeneration of a close-to exact replica of the original sheet music.
However, a knowledge of the duration of notes and the time signature make it simple to evenly place the notes throughout the measure. For example, four quarter notes should be spread out evenly across a 4/4 measure. Other types of notations being drawn would face this same choice. They could be listed with exact coordinates, or they could rely on an understanding of the spatial relationships between the different elements.
This pattern of listing out all the notes and tagging them with their connectors can serve as a general strategy for creating textual representations of similar notations who have a finite set of elements to be drawn. First, each element should be listed out. The order for this list depends on the notation. For instance, music is read left to right, so it makes sense to order the notes in the order they are played. One can imagine a type of drawing that was ordered by distance from a central point, or one with no order at all. In the latter case, any ordering of elements in the list would be sufficient. Next, additional information should be listed under each of the elements. In the music example, the pitch and duration would be listed here. If the notation had different sized elements, a size could be listed for each. Any of the information required to draw an element on its own would be tagged directly under it. Finally, the connectors would be created. In the case of music, which is read left to right, connectors are given the same identifiers repeatedly.
Once a beam labeled “1” ends, the label “1” is free to use again. This works in a left to right scenario, because it is clear which notes are beamed together. However, in the most general case where there is no real ordering, each connector should have a unique identifier. This allows any item to be connected to any other item(s). With no real
42 ordering, there might not be a notion of a connector “starting” and “ending” because connectors need not have direction. Because of this, while parsing it will be unclear until the end when a connector has all of its elements unless that type of connector always has a fixed number of elements. If this result is undesirable, the first instance of a connector in the list could include a tag that says how many elements will be under that connector.
That way, once all of the elements are found the system can process the connector as it wishes without waiting for the entire document or structure to be parsed.
Parsing and drawing the type of document described would be similar to the process described for MusicXML. A list of connectors would need to be kept at the same level as the set of elements. That way, a connector could be given the information it needs about each of the elements it connects as they are created.
43
Chapter 6: Conclusions
The application described was created and successfully displays the notes and most of the symbols from the MusicXML input document. It runs on Android 3.0 and is compatible with the Motorola Xoom. It fulfills the requirements of resizing and color selection. It meets the goal of scrolling at a specified tempo.
6.1: Contributions
1. Implemented the first known application on Android 3.0 that generates sheet music from an XML document and allows scrolling at a tempo and user customization of size and color.
2. Generalized a method for writing human-readable text documents that describe graphical notations based on the structure of MusicXML.
3. Generalized a method for the proper dynamic layout and measuring of ViewGroups inside RelativeLayouts on Android.
6.2: Future Work
The application could benefit from several improvements. Techniques for smoothly scrolling a bitmap at a specified tempo need to be researched further for
Android and applied to this application. The drawing of musical symbols and notes could be improved so that the generated sheet music looks closer to printed sheet music.
Some symbols are still missing all together, and most do not look identical to their printed counterparts. For example, the note heads should not be circles. Additionally, different input file formats could be used. MusicXML is only one way of storing music, and other formats could be explored.
44
It would also be beneficial to the visually impaired community to develop the application for iOS to run on an iPad. This would increase accessibility so that the user could buy the tablet of their choice instead of being forced into an Android tablet.
The generalized method of creating human-readable documents that describe notations could be expanded upon by creating an application that displays another type of notation. This application could start with the general method in mind and make improvements upon it if it did not properly capture some of the relationships between items.
45
Works Cited
1. Charles Jacobs, Wil Li, and David Salesin. Adaptive Document Layout Via Manifold
Content. In Proceedings of Workshop on Web Document Analysis, Edinburgh, 2003.
2. Charles Jacobs , Wil Li , Evan Schrier , David Bargeron , David Salesin, Adaptive
Document Layout, Communications of the ACM, v.47 n.8, August 2004
3. Connolly, Michael, Christof Lutteroth, and Beryl Plimmer. "Document Resizing for
Visually Impaired Students." ACM Digital Library. 2010. Web. 04 May 2012.
4. "Eighth Notes." Wikipedia. Wikimedia Foundation, 03 Jan. 2012. Web. 04 May 2012.
5. Elliott, David B., Bhavesh Patel, and David Whitaker. "Development of a Reading
Speed Test for Potential-Vision Measurements." Investigative Ophthalmology & Visual
Science. July 2001. Web. 04 May 2012.
6. Finale Music Composing & Notation Software. Web. 04 May 2012.
7. "The Lime Lighter Low Vision Music-Reading Device." The Lime Lighter. Web. 04
May 2012.
8. "Low Vision." American Optometric Association. 2006. Web. 30 May 2012.
9. "Low Vision Library: About Low Vision." About Low Vision. 5 Jan. 2010. Web. 30
May 2012.
46
10. "Note Values and Durations." How to Learn Piano. Web.
11. Watt, Wendy S. "How Visual Acuity Is Measured." How Visual Acuity Is Measured.
Macular Degeneration Support, Oct. 2003. Web. 30 May 2012.
12. "What Is MusicXML?" MakeMusic, Inc. Web. 04 May 2012.
47