Efficient Optical Music Recognition Validation using MIDI Sequence Data by Janelle C. Sands Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Master of Engineering in Electrical Engineering and Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY May 2020 ○c Massachusetts Institute of Technology 2020. All rights reserved. Author................................................................ Department of Electrical Engineering and Computer Science May 12, 2020 Certified by. Michael Scott Cuthbert Associate Professor Thesis Supervisor Accepted by . Katrina LaCurts Chair, Master of Engineering Thesis Committee ii Efficient Optical Music Recognition Validation using MIDI Sequence Data by Janelle C. Sands Submitted to the Department of Electrical Engineering and Computer Science on May 12, 2020, in partial fulfillment of the requirements for the degree of Master of Engineering in Electrical Engineering and Computer Science Abstract Despite advances in optical music recognition (OMR), resultant scores are rarely error-free. The power of these OMR systems to automatically generate searchable and editable digital representations of physical sheet music is lost in the tedious manual effort required to pinpoint and correct these errors post-OMR, or evento just confirm no errors exist. To streamline post-OMR error correction, I developeda corrector to automatically identify discrepancies between resultant OMR scores and corresponding Musical Instrument Digital Interface (MIDI) scores and then either automatically fix errors, or in ambiguous cases, notify the user to manually fix errors. This tool will be open source, so anyone can contribute to further improving the accuracy of OMR tools and expanding the amount of trusted digitized music. Thesis Supervisor: Michael Scott Cuthbert Title: Associate Professor iii iv Acknowledgments Thank you to my advisor, Professor Cuthbert, for sharing your enthusiasm, expertise, and encouragement throughout this project. I am grateful for the invaluable learning experience and opportunity to engage in such an exciting interdisciplinary field under your advising. I’m grateful to all of my professors and teachers for all you have taught me. Thank you for imparting a joy in learning and pushing me to grow. I’d especially like to thank the many music educators and musicians who have inspired and instilled in me this love of music. What you have taught me is not only integral to this work, but also my identity. I’d also like to thank the incredible network of friends who have been there for me through this crazy and wonderful MIT adventure. Your pep talks, kindness, listening ears, advice, cooking, and humor made even the most stressful times okay. Finally, thank you to my family for your love and support. I have leaned on your unwavering encouragement and vast collective knowledge in every area this project touches. I couldn’t imagine a better family to be my side through this project and life. Thank you. v vi Contents 1 Introduction 1 1.1 The Overture: An Illustrative Example . .1 1.2 The Problem: Trading Off Time versus Accuracy . .4 1.3 The Overlooked Opportunity: Improving OMR with MIDI . .5 2 Existing Digital Music Technology 7 2.1 History of Optical Music Recognition (OMR) . .7 2.2 Demystifying OMR Systems . .7 2.3 A Post-OMR Step . .8 2.4 Algorithms for Aligning Two Different Versions of the Same Score .9 2.5 Symbolic Music Encoding . 10 2.6 Human in the Loop . 11 2.7 Useful Tools . 12 2.7.1 music21 Toolkit . 12 2.7.2 SmartScore: Accessible OMR . 12 2.7.3 Viewing and Editing Music with MuseScore . 13 2.7.4 IMSLP: Hundreds of Thousands of Music Scans . 13 2.7.5 Online Classical Archives: Countless Works Without Visual Notation . 13 3 Design and Implementation of a System for Efficient OMR Output Correction and Validation 15 3.1 Goals of OMR Validation . 15 vii 3.1.1 Goal 1: Accuracy is Guaranteed . 16 3.1.2 Goal 2: Manual Intervention is Efficient . 16 3.1.3 Goal 3: Automatic Correction is Reasonably Efficient . 17 3.1.4 Goal 4: Easy for Developers to Expand . 17 3.2 System Overview . 18 3.2.1 An Accessible System . 18 3.2.2 Automatic Fixing . 20 3.2.3 Manual Fixing . 20 3.2.4 Expected Output . 22 3.3 Intended Inputs . 23 3.3.1 One Part at a Time . 23 3.3.2 Input Correctness Assumptions . 25 3.3.3 Supported Instruments . 26 3.4 Preliminary Fixing . 27 3.4.1 Measure Alignment . 27 3.4.2 Holistic Fixing . 28 3.5 Tracking the Correction History . 28 3.5.1 Aligning OMR and MIDI Parts . 29 3.5.2 Fixing a Correction Generation . 30 3.5.3 Creating the Next Generation . 31 3.6 Automatic Fixing . 32 3.6.1 The Process . 32 3.6.2 Challenges . 33 3.7 Manual Fixing and Approval . 34 3.7.1 Correction Statuses For Display . 35 3.7.2 Display Layout . 36 3.7.3 Editing the Display . 38 3.7.4 Rebuilding The Corrected Score . 40 viii 4 Error Fixers 43 4.1 Identifying Errors . 44 4.1.1 Challenges With Existing Visualizations . 44 4.1.2 A Better Way To Display Discrepancies . 45 4.2 Classifying Errors . 48 4.3 Initial Fixers for Measure Alignment . 48 4.3.1 Shifting to Account for a Pickup Measure . 49 4.3.2 Collapsing Repeats . 52 4.3.3 Fixing Misread Multirests . 54 4.4 Other Initial Fixers . 55 4.4.1 Key Signature Resolution . 55 4.4.2 Time Signature Resolution . 57 4.4.3 Replacing Intruding Guitar Tablatures . 59 4.5 Iterative Fixers . 60 4.5.1 Rest Representation Discrepancies . 60 4.5.2 Misread Accidentals . 61 4.5.3 Undoing the Rhythmic Harm of a Misread Dot . 65 4.5.4 Articulations Marked By Different Rhythms . 66 4.5.5 Finding Missing MIDI Notes . 66 4.5.6 Ornaments . 67 4.6 Developers Guide to Fixers . 74 4.7 Existing Fixers Fix Some, but Not All Discrepancies . 75 5 Evaluating Usefulness 77 5.1 Evaluation Procedure . 77 5.2 Metrics for Evaluation . 78 5.2.1 Quantity of Content to Review . 78 5.2.2 Accuracy Improvement due to Automatic Correction . 78 5.3 Applying the Corrector to Quartet Music . 79 5.3.1 Haydn’s First Quartet . 79 ix 5.3.2 Mozart’s First Quartet . 87 5.4 Generalizing Results . 90 5.4.1 Changes in Accuracy From Automatic Fixing . 90 5.4.2 Discrepancies Can be Resolved and Some Errors Fixed . 92 5.4.3 Effectively Reducing the Quantity of Content to Review . 93 5.5 The Role of Human Error in MIDI Files . 94 5.6 Truths Are Not Necessarily Ground Truths . 94 5.7 Unlocking a New Piece of Music . 94 6 Conclusion 97 A Example Corrector Display Score: Haydn Quartet 99 A.1 Haydn Quartet Op.1, No.1, i: Violin 1 . 100 A.2 Haydn Quartet Op.1, No.1, i: Violin 1 . 101 A.3 Haydn Quartet Op.1, No.1, i: Violin 1 . 102 A.4 Haydn Quartet Op.1, No.1, i: Violin 1 . 103 A.5 Haydn Quartet Op.1, No.1, i: Violin 2 . 105 A.6 Haydn Quartet Op.1, No.1, i: Violin 2 . 106 A.7 Haydn Quartet Op.1, No.1, i: Violin 2 . 107 A.8 Haydn Quartet Op.1, No.1, i: Violin 2 . 108 A.9 Haydn Quartet Op.1, No.1, i: Viola . 110 A.10 Haydn Quartet Op.1, No.1, i: Viola . 111 A.11 Haydn Quartet Op.1, No.1, i: Viola . 112 A.12 Haydn Quartet Op.1, No.1, i: Viola . 113 A.13 Haydn Quartet Op.1, No.1, i: Cello . 115 A.14 Haydn Quartet Op.1, No.1, i: Cello . 116 A.15 Haydn Quartet Op.1, No.1, i: Cello . 117 B Example Corrector Display Score: Mozart Quartet 119 B.1 Mozart Quartet No.1, ii: Violin 1 . 120 B.2 Mozart Quartet No.1, ii: Violin 1 . 121 x B.3 Mozart Quartet No.1, ii: Violin 1 . 122 B.4 Mozart Quartet No.1, ii: Violin 1 . 123 B.5 Mozart Quartet No.1, ii: Violin 1 . 124 B.6 Mozart Quartet No.1, ii: Violin 1 . 125 B.7 Mozart Quartet No.1, ii: Violin 2 . 126 B.8 Mozart Quartet No.1, ii: Violin 2 . 127 B.9 Mozart Quartet No.1, ii: Violin 2 . 128 B.10 Mozart Quartet No.1, ii: Violin 2 . 129 B.11 Mozart Quartet No.1, ii: Violin 2 . 130 B.12 Mozart Quartet No.1, ii: Violin 2 . 131 B.13 Mozart Quartet No.1, ii: Viola . 132 B.14 Mozart Quartet No.1, ii: Viola . 133 B.15 Mozart Quartet No.1, ii: Viola . 134 B.16 Mozart Quartet No.1, ii: Cello . 135 B.17 Mozart Quartet No.1, ii: Cello . 136 B.18 Mozart Quartet No.1, ii: Cello . 137 xi xii List of Figures 1-1 Excerpt of original sheet music for the viola part from the beginning of Haydn’s Op. 1 No. 1 Quartet movement 1 . .1 1-2 Excerpt from the.