Conditioning Deep Generative Raw Audio Models for Structured Automatic Music
Total Page:16
File Type:pdf, Size:1020Kb
CONDITIONING DEEP GENERATIVE RAW AUDIO MODELS FOR STRUCTURED AUTOMATIC MUSIC Rachel Manzelli∗ Vijay Thakkar∗ Ali Siahkamari Brian Kulis ∗Equal contributions ECE Department, Boston University fmanzelli, thakkarv, siaa, [email protected] ABSTRACT In order to create sound, these methods often require an in- Existing automatic music generation approaches that fea- termediate step of interpretation of the output by humans, ture deep learning can be broadly classified into two types: where the symbolic representation transitions to an audio raw audio models and symbolic models. Symbolic mod- output in some way. els, which train and generate at the note level, are cur- An alternative is to train on and produce raw audio rently the more prevalent approach; these models can cap- waveforms directly by adapting speech synthesis models, ture long-range dependencies of melodic structure, but fail resulting in a richer palette of potential musical outputs, to grasp the nuances and richness of raw audio genera- albeit at a higher computational cost. WaveNet, a model tions. Raw audio models, such as DeepMind’s WaveNet, developed at DeepMind primarily targeted towards speech train directly on sampled audio waveforms, allowing them applications, has been applied directly to music; the model to produce realistic-sounding, albeit unstructured music. is trained to predict the next sample of 8-bit audio (typi- In this paper, we propose an automatic music generation cally sampled at 16 kHz) given the previous samples [25]. methodology combining both of these approaches to cre- Initially, this was shown to produce rich, unique piano ate structured, realistic-sounding compositions. We con- music when trained on raw piano samples. Follow-up sider a Long Short Term Memory network to learn the work has developed faster generation times [16], generated melodic structure of different styles of music, and then use synthetic vocals for music using WaveNet-based architec- the unique symbolic generations from this model as a con- tures [3], and has been used to generate novel sounds and ditioning input to a WaveNet-based raw audio generator, instruments [8]. This approach to music generation, while creating a model for automatic, novel music. We then eval- very new, shows tremendous potential for music genera- uate this approach by showcasing results of this work. tion tools. However, while WaveNet produces more real- istic sounds, the model does not handle medium or long- 1. INTRODUCTION range dependencies such as melody or global structure in music. The music is expressive and novel, yet sounds un- The ability of deep neural networks to generate novel mu- practiced in its lack of musical structure. sical content has recently become a popular area of re- search. Many variations of deep neural architectures have Nonetheless, raw audio models show great potential for generated pop ballads, 1 helped artists write melodies, 2 the future of automatic music. Despite the expressive na- and even have been integrated into commercial music gen- ture of some advanced symbolic models, those methods re- eration tools. 3 quire constraints such as mood and tempo to generate cor- Current music generation methods are largely focused responding symbolic output [22]. While these constraints on generating music at the note level, resulting in outputs can be desirable in some cases, we express interest in gen- consisting of symbolic representations of music such as se- erating structured raw audio directly due to the flexibility quences of note numbers or MIDI-like streams of events. and versatility that raw audio provides; with no specifica- These methods, such as those based on Long Short Term tion, these models are able to learn to generate expression Memory networks (LSTMs) and recurrent neural networks and mood directly from the waveforms they are trained on. (RNNs), are effective at capturing medium-scale effects We believe that raw audio models are a step towards less in music, can produce melodies with constraints such as guided, unsupervised music generation, since they are un- mood and tempo, and feature fast generation times [14,22]. constrained in this way. With such tools for generating raw audio, one can imagine a number of new applications, such 1 http://www.flow-machines.com/ as the ability to edit existing raw audio in various ways. 2 https://www.ampermusic.com/ 3 https://www.jukedeck.com/ Thus, we explore the combination of raw audio and symbolic approaches, opening the door to a host of new possibilities for music generation tools. In particular, c Rachel Manzelli, Vijay Thakkar, Ali Siahkamari, Brian we train a biaxial Long Short Term Memory network Kulis. Licensed under a Creative Commons Attribution 4.0 International to create novel symbolic melodies, and then treat these License (CC BY 4.0). Attribution: Rachel Manzelli, Vijay Thakkar, Ali Siahkamari, Brian Kulis. “Conditioning Deep Generative Raw Audio melodies as an extra conditioning input to a WaveNet- Models for Structured Automatic Music”, 19th International Society for based model. Consequently, the LSTM model allows us Music Information Retrieval Conference, Paris, France, 2018. to represent long-range melodic structure in the music, 182 Proceedings of the 19th ISMIR Conference, Paris, France, September 23-27, 2018 183 while the WaveNet-based component interprets and ex- pands upon the generated melodic structure in raw audio form. This serves to both eliminate the intermediate inter- pretation step of the symbolic representations and provide structure to the output of the raw audio model, while main- taining the aforementioned desirable properties of both models. We first discuss the tuning of the original unconditioned WaveNet model to produce music of different instruments, Figure 1: A stack of dilated causal convolutions as used styles, and genres. Once we have tuned this model appro- by WaveNet, reproduced from [25]. priately, we then discuss our extension to the conditioned case, where we add a local conditioning technique to the 256 possible values. The WaveNet model uses a deep raw audio model. This method is comparable to using a neural network to model the conditional probabilities text-to-speech method within a speech synthesis model. p(x jx ; :::; x ). The model is trained by predicting val- We first generate audio from the conditioned raw audio t 1 t−1 ues of the waveform at step t and comparing them to the model using well-known melodies (e.g., a C major scale true value x , using cross-entropy as a loss function; thus, and the Happy Birthday melody) after training on the Mu- t the problem simply becomes a multi-class classification sicNet dataset [24]. We also discuss an application of our problem (with 256 classes) for each timestep in the wave- technique to editing existing raw audio music by changing form. some of the underlying notes and re-generating selections The modeling of conditional probabilities in WaveNet of audio. Then, we incorporate the LSTM generations as a utilizes causal convolutions, similar to masked convolu- unique symbolic component. We demonstrate results of tions used in PixelRNN and similar image generation net- training both the LSTM and our conditioned WaveNet- works [7]. Causal convolutions ensure that the prediction based model on corresponding training data, as well as for time step t only depends on the predictions for previ- showcase and evaluate generations of realistic raw audio ous timesteps. Furthermore, the causal convolutions are melodies by using the output of the LSTM as a unique lo- dilated; these are convolutions where the filter is applied cal conditioning time series to the WaveNet model. over an area larger than its length by skipping particular This paper is an extension of an earlier work originally input values, as shown in Figure 1. In addition to dilated published as a workshop paper [19]. We augment that causal convolutions, each layer features gated activation work-in-progress model in many aspects, including more units and residual connections, as well as skip connections concrete results, stronger evaluation, and new applications. to the final output layers. 2. BACKGROUND 2.2 Symbolic Audio Models We elaborate on two prevalent deep learning models for music generation, namely raw audio models and symbolic Most deep learning approaches for automatic music gen- models. eration are based on symbolic representations of the mu- sic. MIDI (Musical Instrument Digital Interface), 4 for ex- 2.1 Raw Audio Models ample, is a ubiquitous standard for file format and proto- col specification for symbolic representation and transmis- Initial efforts to generate raw audio involved models used sion. Other representations that have been utilized include primarily for text generation, such as char-rnn [15] and the piano roll representation [13]—inspired by player pi- LSTMs. Raw audio generations from these networks are ano music rolls—text representations (e.g., ABC nota- often noisy and unstructured; they are limited in their ca- tion 5 ), chord representations (e.g., Chord2Vec [18]), and pacity to abstract higher level representations of raw audio, lead sheet representations. A typical scenario for produc- mainly due to problems with overfitting [21]. ing music in such models is to train and generate on the In 2016, DeepMind introduced WaveNet [25], a gen- same type of representation; for instance, one may train on erative model for general raw audio, designed mainly for a set of MIDI files that encode melodies, and then generate speech applications. At a high level, WaveNet is a deep new MIDI melodies from the learned model. These mod- learning architecture that operates directly on a raw audio els attempt to capture the aspect of long-range dependency waveform. In particular, for a waveform modeled by a vec- in music. tor x = fx1; :::; xT g (representing speech, music, etc.), the joint probability of the entire waveform is factorized as a A traditional approach to learning temporal dependen- product of conditional probabilities, namely cies in data is to use recurrent neural networks (RNNs).