Stephanie Van Laere Lead Sheet Generation with Deep Learning
Total Page:16
File Type:pdf, Size:1020Kb
Lead Sheet Generation with Deep Learning Stephanie Van Laere Supervisor: Prof. dr. ir. Bart Dhoedt Counsellors: Ir. Cedric De Boom, Dr. ir. Tim Verbelen Master's dissertation submitted in order to obtain the academic degree of Master of Science in Computer Science Engineering Department of Information Technology Chair: Prof. dr. ir. Bart Dhoedt Faculty of Engineering and Architecture Academic year 2017-2018 Lead Sheet Generation with Deep Learning Stephanie Van Laere Supervisor: Prof. dr. ir. Bart Dhoedt Counsellors: Ir. Cedric De Boom, Dr. ir. Tim Verbelen Master's dissertation submitted in order to obtain the academic degree of Master of Science in Computer Science Engineering Department of Information Technology Chair: Prof. dr. ir. Bart Dhoedt Faculty of Engineering and Architecture Academic year 2017-2018 Permission of use The author gives permission to make this master dissertation available for con- sultation and to copy parts of this master dissertation for personal use. In the case of any other use, the copyright terms have to be respected, in particular with regard to the obligation to state expressly the source when quoting results from this master dissertation. Stephanie Van Laere Ghent, May 2018 I Acknowledgements When I stumbled on this project, I immediately thought this would be the perfect fit for me. Being a musician and a composer myself, I saw this asthebest opportunity to combine two passions of mine: computer science and music. At the final stages of one of the biggest projects of my life, I am overwhelmed by gratefulness to all the people that have helped me through this project. Firstly, I would like to thank my counselors, dr. ir. Cedric De Boom and dr. ir. Tim Verbelen, for your enormous support. Through the hours of feedback sessions, this master dissertation wouldn’t be what it is now without you. I would also like to thank my supervisor, prof. dr. ir. Bart Dhoedt, that I was able to work on this project. I truly couldn’t have imagined a better subject than this one. I would like to thank all my friends from BEST, who provided me great laugh- ter through moments I truly needed it. I would especially like to thank Karim, Margot and Alexander, for guidance and occasional distraction that was invalu- able to me. I would also like to thank Esti for our little thesis working moments and our friendship throughout these past six years. Laurence, thank you for the coffee and Skype dates and of course, our 11-year long friendship. Janneken, thank you for singing every song I ever wrote and making my love for music grow even stronger. Lars, for helping and supporting me. Finally, I would like to thank everybody who filled in the survey and sent it to their friends. I never expected 177 responses in such a short amount of time and it helped immensely. I would like to thank my parents for introducing me to the piano and all types of music throughout my youth. You have always supported me through thick and thin, with some advice, a great meal, a hug or a joke. My brother, for your innovative, creative mind and confidence. My sister, for your generous and altru- istic heart. My grandparents, whom I’ve never seen without the biggest smiles. Finally, my great-grandmother, for your inspiration. I will always remember you playing ‘La tartine de beurre’ on the piano in our living room. Stephanie Van Laere II Lead Sheet Generation with Deep Learning Stephanie Van Laere Supervisor: Prof. dr. ir. Bart Dhoedt Counsellors: Ir. Cedric De Boom, Dr. ir. Tim Verbelen Master’s dissertation submitted in order to obtain the academic degree of Master of Science in Computer Science Engineering Department of Information Technology Chair: Prof. dr. ir. Bart Dhoedt Faculty of Engineering and Architecture – Ghent University Academic year 2017-2018 Abstract The Turing test of music, the idea that a computer creates music where it be- comes indistinguishable from a human-composed music piece, has been fascinating researches for decades now. Most explorations on music generation has focused on classical music, but some research has also been done regarding modern music, partially about lead sheet music. A lead sheet is a format of music representa- tion that is especially popular in jazz and pop music. The main elements are melody notes, chords and optional lyrics. In the field of lead sheet generation, Markov models have been mostly explored, whilst a lot of the research in the classical field incorporates Recurrent Neural Networks (RNNs). We would liketo use these RNNs to generate modern-age lead sheet music by using the Wikifonia dataset. Specifically, we use a model with two components. The first component generates the chord scheme, the backbone of a lead sheet, together with the du- ration of the melody notes. The second component generates the pitches on this chord scheme. We evaluated this model through a survey with 177 participants. Keywords: music generation, recurrent neural networks, deep learning, lead sheet generation III Lead Sheet Generation with Deep Learning Stephanie Van Laere Supervisor(s): Bart Dhoedt, Cedric De Boom, Tim Verbelen Abstract— The Turing test of music, the idea that a computer II. RELATED WORKS creates music where it becomes indistinguishable from a human- Music can be represented in two ways: (i) a signal or audio composed music piece, has been fascinating researches for decades representation, which uses raw audio or waveforms, or (ii) a now. Most explorations on music generation has focused on classical music, but some research has also been done regarding symbolic representation, which will note music through modern music, partially about lead sheet music. A lead sheet is a discrete events, such as notes, chord or rhythms. We will focus format of music representation that is especially popular in jazz on symbolic music generation. and pop music. The main elements are melody notes, chords and optional lyrics. In the field of lead sheet generation, Markov Researchers have been working on music generation models have been mostly explored, whilst a lot of the research in problems for decades now. From the works of Bharucha and the classical field incorporates Recurrent Neural Networks Todd in 1989 using neural networks (NNs) [1] to working with (RNNs). We would like to use these RNNs to generate modern-age more complex models such as the convolutional generative lead sheet music by using the Wikifonia dataset. Specifically, we adversarial network (GAN) from Yang et al. in 2017 [2], the use a model with two components. The first component generates topic clearly still has a lot left to explore. the chord scheme, the backbone of a lead sheet, together with the Research specifically for lead sheet generation has also been duration of the melody notes. The second component generates the conducted. FlowComposer [3] is part of the FlowMachines [4] pitches on this chord scheme. We evaluated this model through a survey with 177 participants. project and generates lead sheets in the style of the corpus selected by the user. The user can enter a partial lead, which is Keywords— music generation, recurrent neural networks, deep then completed by the model. If needed, the model can also learning, lead sheet generation generate from scratch. Some meter-constrained Markov chains represent the lead sheets. Pachet et al. have also conducted I. INTRODUCTION some other research within the FlowMachines project What is art? That is a question that can be thoroughly regarding lead sheets. In [5], they use the Mongeau & Sankoff discussed for hours by any art lover. Can the name ‘art’ only be similarity measure in order to generate similar melody themes used when it is made by humans, or is it also art if a human to the imposed theme. This is definitely relevant in any type of doesn’t recognize computer generated ‘art’? We want to focus music, since a lot of melody themes are repeated multiple times on research that makes this question so enticing: improving during pieces. Ramona, Cabral and Pachet have also created the computer-generated music that resembles the work of humans. ReChord Player, which generates musical accompaniments for Specifically, we want to focus on lead sheet generation by any song [6]. Even though most research on modern lead sheet using Recurrent Neural Networks (RNNs). A lead sheet is a generation makes use of Markov models, we want to focus on format of music representation that is especially popular in jazz using Recurrent Neural Networks (RNNs) for lead sheet and pop music. The main elements are melody notes, chords generation using the Wikifonia dataset1. and optional lyrics. An example can be found in Figure 1. III. WIKIFONIA DATA PREPROCESSING AND AUGMENTATION The pieces from the Wikifonia dataset were all in MusicXML (MXL) format [7]. To make it easier to process, we transformed the pieces into a Protocol Buffer format2, inspired by Google's Magenta [8]. Some preprocessing steps were taken. Polyphonic notes, which means that multiple notes are played at the same time, are deleted until only the highest notes, the melody notes, Figure 1 A (partial) example of a lead sheet. remain. Anacruses, which are sequences of notes that precede the downbeat of the first measure, are omitted. Ornaments, We divided the problem in two components. The first which are decorative notes that are played rapidly in front of component generated the chords together with the rhythm or the central note, are also left out. duration of the melody notes. The second component generated In order for the model to know which measures need to be the pitches of the melody notes, based on the chords and the played after which measures, repeat signs need to be rhythm.