Transcription of the Singing Melody in Polyphonic Music

Matti Ryynanen¨ and Anssi Klapuri Institute of Signal Processing, Tampere University Of Technology P.O.Box 553, FI-33101 Tampere, Finland {matti.ryynanen, anssi.klapuri}@tut.fi

Abstract ACOUSTIC MODEL This paper proposes a method for the automatic transcrip- INPUT: AUDIO SIGNAL NOTE MODEL REST MODEL tion of singing melodies in polyphonic music. The method HMM GMM is based on multiple-F0 estimation followed by acoustic and musicological modeling. The acoustic model consists of EXTRACT COMPUTE OBSERVATION separate models for singing notes and for no-melody seg- FEATURES LIKELIHOODS FOR MODELS ments. The musicological model uses key estimation and SELECT SINGING note bigrams to determine the transition probabilities be- NOTE RANGE CHOOSE FIND OPTIMAL PATH tween notes. Viterbi decoding produces a sequence of notes BETWEEN−NOTE THROUGH THE MODELS ESTIMATE TRANSITION and rests as a transcription of the singing melody. The per- MUSICAL KEY PROBABILITIES formance of the method is evaluated using the RWC popular OUTPUT: MUSICOLOGICAL MODEL SEQUENCE OF music database for which the recall rate was 63% and pre- NOTES AND RESTS cision rate 46%. A significant improvement was achieved Figure 1. The block diagram of the transcription method. compared to a baseline method from MIREX05 evaluations. Keywords: singing transcription, acoustic modeling, musicological modeling, key estimation, HMM the proposed method which was evaluated second best in 1. Introduction the Music Information Retrieval Evaluation eXchange 2005 (MIREX05) 1 audio-melody extraction contest. Ten state- Singing melody transcription refers to the automatic extrac- of-the-art melody transcription methods were evaluated in tion of a parametric representation (e.g., a MIDI file) of the this contest where the goal was to estimate the F0 trajectory singing performance within a polyphonic music excerpt. A of the melody within polyphonic music. Our related work melody is an organized sequence of consecutive notes and includes monophonic singing transcription [8]. rests, where a note has a single pitch (a note name), a begin- Figure 1 shows a block diagram of the proposed method. ning (onset) time, and an ending (offset) time. Automatic First, an audio signal is frame-wise processed with two fea- transcription of singing melodies provides an important tool ture extractors, including a multiple-F0 estimator and an ac- for MIR applications, since a compact MIDI file of a singing cent estimator. The acoustic modeling uses these features melody can be efficiently used to identify the song. to derive a hidden Markov model (HMM) for note events Recently, melody transcription has become an active re- and a Gaussian mixture model (GMM) for singing rest seg- search topic. The conventional approach is to estimate the ments. The musicological model uses the F0s to determine fundamental frequency (F0) trajectory of the melody within the note range of the melody, to estimate the musical key, polyphonic music, such as in [1], [2], [3], [4]. Another class and to choose between-note transition probabilities. A stan- of transcribers produce discrete notes as a representation of dard Viterbi decoding finds the optimal path through the the melody [5], [6]. The proposed method belongs to the models, thus producing the transcribed sequence of notes latter category. and rests. The decoding simultaneously resolves the note The proposed method resembles our polyphonic music onsets, the note offsets, and the note pitch labels. transcription method [7] but here it has been tailored for singing melody transcription and includes improvements, For training and testing our transcription system, we use such as an acoustic model for rest segments in singing. As the RWC (Real World Computing) Popular Music Database a baseline in our simulations, we use an early version of which consists of 100 acoustic recordings of typical pop songs [9]. For each recording, the database includes a reference MIDI file which contains a manual annotation of the Permission to make digital or hard copies of all or part of this work for singing-melody notes. The annotated melody notes are here personal or classroom use is granted without fee provided that copies referred to as the reference notes. Since there exist slight are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. 1 The evaluation results and extended abstracts are available at c 2006 University of Victoria www.music-ir.org/evaluation/mirex-results/audio-melody time deviations between the recordings and the reference a) F0 estimates xit (intensity by sit) 72 notes, all the notes within one reference file are collectively Ref. notes time-scaled to synchronize them with the acoustic signal. 70 68 The synchronization could be performed reliably for 96 of 66 the songs and the first 60 seconds of each song are used. On 64 MIDI notes the average, each song excerpt contains approximately 84 62 reference melody notes. b) Onsetting F0 estimates yit (intensity by dit) 72 2. Feature Extraction 70 Ref. notes The front-end of the method consists of two frame-wise fea- 68 ture extractors: a multiple-F0 estimator and an accent esti- 66 64 mator. The input for the extractors is a monophonic audio MIDI notes signal. For stereophonic input audio, the two channels are 62 summed together and divided by two, prior to the feature c) Accent signal at extraction. 2

2.1. Multiple-F0 Estimation 1 We use the multiple-F0 estimator proposed in [10] in a fash- 0 ion similar to [7]. The estimator applies an auditory model Ref. note onsets where an input signal is passed through a 70-channel band- −1 48 50 52 54 56 pass filterbank and the subband signals are compressed, half- time (s) wave rectified, and lowpass filtered. STFTs are computed within the bands and the magnitude spectra are summed Figure 2. The features extracted from a segment of song RWC- across channels to obtain a summary spectrum for subse- MDB-P-2001 No. 14. See text for details. quent processing. Periodicity analysis is then carried out by simulating a bank of comb filters in the frequency domain. F0s are estimated one at a time, the found sounds are can- challenging for singing voice. Therefore, we add the accent celed from the mixture, and the estimation is repeated for signal feature which has been successfully used in singing the residual. transcription [8]. The accent estimation method proposed We use the estimator to analyze audio signal in overlap- in [11] is used to produce accent signals at four frequency ping 92.9 ms frames with 23.2 ms interval between the be- channels. The bandwise accent signals are then summed ginnings of successive frames. As an output, the estima- across the channels to obtain the accent signal at which is tor produces four feature matrices X, S, Y , and D of size decimated by factor 4 to match the frame rate of the multiple- 6 × tmax (tmax is the number of analysis frames): F0 estimator. Again, logarithm is applied to the accent signal and the signal is normalized to zero mean and unit vari- F0 estimates in matrix and their salience values in • X ance. matrix S. For a F0 estimate x = [X] , the salience it it Figure 2 shows an example of the features compared to value sit = [S]it roughly expresses how prominent reference notes. Panels a) and b) show the F0 estimates xit xit is in the analysis frame t. and the onsetting F0s yit with the reference notes, respec- • Onsetting F0 estimates in matrix Y and their onset tively. The gray level indicates the salience values sit in strengths in matrix D. If a sound with F0 estimate panel a) and the onset strengths dit in panel b). Panel c) shows the accent signal at and the note onsets in the refer- yit = [Y ]it sets on in frame t, the onset strength value ence melody. dit = [D]it is high. The F0 values in both X and Y are expressed as unrounded 3. Acoustic and Musicological Modeling MIDI note numbers by 69 + 12 log2(F0/440). Logarithm is taken from the elements of S and D to compress their dy- Our method uses two different abstraction levels to model namic range, and the values in these matrices are normalized singing melodies: low-level acoustic modeling and high- over all elements to zero mean and unit variance. level musicological modeling. The acoustic modeling aims at capturing the acoustic content of singing whereas the mu- 2.2. Accent Signal for Note Onsets sicological model employs information about typical me- The accent signal at indicates the degree of phenomenal ac- lodic intervals. This approach is analogous to speech recog- cent in frame t, and it is here used to indicate the poten- nition systems where the acoustic model corresponds to a tial note onsets. There was room for improvement in the word model and the musicological model to a “language note-onset transcription of [7], and the task is even more model”, for example. 3.1. Note Event Model parameters are then obtained using the Baum-Welch algo- Note events are modeled with a 3-state left-to-right HMM. rithm. The observation likelihood distributions are modeled The note HMM state qi, 1 ≤ i ≤ 3, represents the typical with a four-component GMM. values of the features in the i:th temporal segment of note 3.2. Rest Model events. The model allocates one note HMM for each MIDI We use a GMM for modeling the time segments where no note in the estimated note range (explained in Section 3.3). singing-melody notes are sounding, that is, rests. Rests are Given the extracted features X, S, Y , D, and at, the ob- 5 clearly defined for monophonic singing melodies, and there- servation vector o ∈ R is defined for a note HMM with n,t fore, we can now train an acoustic rest model instead of us- nominal pitch n in frame t as ing artificial rest-state probabilities derived from note-model probabilities as in [7]. The observation vector or,t for rest on,t = (∆xn,t,sjt, ∆yn,t, dkt,at) , (1) consists of the maximum salience and onset strength in each where frame t, i.e.,

or,t = (max{sit}, max{djt}) . (6) ∆xn,t = xjt − n , (2) i j ∆yn,t = ykt − n . (3) The model itself is a four-component GMM (analogous to a Index j is obtained using 1-state HMM) trained on the frames of the no-melody segments. The logarithmic observation likelihoods of the rest m = arg max {sit} , (4) model are scaled to the same dynamic range with those of i the note model by multiplying with an experimentally-found m , if |x − n| ≤ 1 j = mt (5) constant. arg mini {|xit − n|} , otherwise. 3.3. Note Range Estimation Index k is chosen similarly to (4)–(5) by substituting k, yit, The note range estimation aims at constraining the possible and dit in place of j, xit, and sit, respectively. pitch range of the transcribed notes. Since singing melodies An observation vector thus consists of five features: (i) usually lie within narrow note ranges, the selection makes the F0 difference ∆xn,t between the measured F0 and the the system more robust against spurious too-high notes and nominal pitch n of the modeled note and (ii) its correspond- the interference of prominent bass line notes. This also re- ing salience value sjt; (iii) the onsetting F0 difference ∆yn,t duces the computational load due to the smaller amount of and (iv) its strength dkt; and (v) the accent signal value at. note models that need to be evaluated. If the note range esti- For a note model n, the maximum-salience F0 estimate and mation is disabled, we use a note range from MIDI note 44 its salience value are associated with the note if the absolute to 84. F0 difference is less or equal to one semitone (see (4)–(5)), The proposed procedure takes the maximum-salience F0 otherwise the nearest F0 estimate is used. A similar selec- estimate in each frame. If an estimate is on MIDI note tion is performed to choose index k for the onsetting F0s. range 50–74 and its salience value is above a threshold 1.0, We use the F0 difference as a feature instead of the abso- the estimate is considered as valid. Then we calculate the lute F0 value so that only one set of note-HMM parameters salience-weighted mean of the valid F0s to obtain the note- needs to be trained. In other words, we have a distinct note range mean, i.e., nmean = h( i xisi)/( i si)i, where op- HMM for each nominal pitch n but they all share the same erator h·i is the nearest integerP function,Pxi is a valid F0 trained parameters. This can be done since the observation estimate, and si its salience. The note range is then set to vector (1) is tailored to be different for each note model n. be nmean ± 12, i.e., a two octave range centered around the Since the F0 difference varies a lot for singing voice, we mean. use the maximum-salience F0 in contrast to the nearest F0 In 95% of the songs, all reference notes are covered by used in [7]. For the same reason, the onset strength values the estimated ranges, and even in the worst case over 80% are slightly increased during singing notes, and therefore, of notes are covered. Averaged over all songs, the estimated we decided to use the onsetting F0s and their strengths sim- note ranges cover over 99.5% of the reference notes. ilarly to normal F0 measurements. The note model is trained as follows. For the time region 3.4. Key Estimation and Note Bigrams of a reference note with nominal pitch n, the observation The musicological model controls transitions between the vectors (1) constitute a training sequence. Since for some note models and the rest model in a manner similar to that reference notes there are no reliable F0 measurements avail- used in [7]. The musicological model is based on the fact able, the observation sequence is accepted for the training that some note sequences are more common than others in a only if the median of the absolute F0 differences |∆xn,t| certain musical key. A musical key is roughly defined by the during the note is less than one semitone. The note HMM basic note scale used in a song. A major key and a minor key eut ie h rvosnt n h otpoal relati pair probable most key a the large and As note a previous [12]. the in given from reported result, estimated as melodies, were monophonic which of database bigrams note by fined music. work. future popular for assumption, left of unrealistic is excerpts estimation an key short Time-adaptive is for this acceptable general, mu- however, the In during changed current excerpt. not The is sic key model. the rest that the probabil- assumes transition and method choose models to note used between recording then ities whole is prob- pair the most key for The this estimated and zero). is use pair larger we relative-key is (here able value threshold salience fixed which a for than numbers) note MIDI est ,r i, hc sfeunl curn ntemsclky rest- note A a key. with musical sequence the note in a occurring end frequently to is and which start to both able note h ehdpoue ieiod o ifrn ao and major estimates different F0 those for from keys likelihoods minor produces [8] method method estimation The key musical a using pair relative-key minor). A the and major C the with scales (e.g., of notes consist same they if pair relative-key a called are minor. A / major C pair relative-key the for one over probabilities transition Musicological 3. Figure From note h rniinpoaiiisbtennt Msaede- are HMMs note between probabilities transition The h uiooia oe sue hti smr prob- more is it that assumes model musicological The h uiooia oe rtfid h otprobable most the finds first model musicological The ) G#4 C#4 Bb4 Eb4 F#4 0.05 0.15 G4 C5 D4 C4 B4 A4 E4 F4 j ie rniinpoaiiyt oefo note from move to probability transition a gives 0.1 . 0 C4 C4 r h oebga probability bigram note the , C#4 C#4 Note −>restnotetransitionprobabilities D4 D4 Eb4 Eb4 E4 E4 F4 F4 to note F#4 F#4 x G4 G4 it abilities prob- Note-transition ruddt h near- the to (rounded G#4 G#4 P ( n A4 A4 t P = Bb4 Bb4 ( n j t | | B4 B4 n n t octave t − − C5 C5 1 i 1 the ) ve- to = . h oe,teetontsaemre soewihsat at starts which one as merged are notes two these notes, the olwdb ogrnt with note longer a by followed ms 200 first 203]. the p. fre- during [14, fundamental pitch note a note the of the and to low) matched (too is quency flat begin notes often long of which beginning pitch. the at note employed nominal usually is the Glissando to refer slide glissando fundamental-frequency term a to The correction. glissando rule-based example an MIDI path. shows the the the 5 finding by Figure exits after determined model. path transcription note is the the label the of when pitch number enters off note note path sets The optimal and note state. model the last the a when and of on state onsets, sets first note note the A labels, offsets. pitch i.e., note rests, pr the and simultaneously duces this notes that of through Notice melody. sequence path transcribed a optimal the produce the to find network to the We algorithm 4. Viterbi Figure in the illustrated use is by This controlled are model. transitions musicological rest ofthe network and a note form the model where rest models the and models event Post-processing note and The Path Optimal the Finding 3.5. the If minor. used. A are ov transitions / distributions all uniform major disabled, C is model pair musicological relative-key the in proba- rest-to-not sitions and transition note-to-rest, musicological between-note, ac- the for into shows bilities taken 3 is Figure pair transit relative-key count. probable rest-to-note most The the the and that 67]. proba- so note-to-rest p. as the distributions for [13, these bilities music applies classical model of musicological amount estimate key, large musical a the to from respect with note notes probabilitie a different occurrence of ending the to reported Krumhansl sequence corresponds sequence. note transition a note-to-rest starting a and to corresponds transition to-note MIDI NOTES iue4 h ewr fnt oesadters model. rest the and models note of network The 4. Figure fatasrbdnt hre hn10m simmediately is ms 180 than shorter note transcribed a If simple a use may we step, post-processing optional an As MUSICOLOGICAL MODEL TIME +1 or +2 nevlbetween interval NOTE MODEL REST MODEL tran- e ions o- er d s s Table 2. Results with perfect note range, perfect key, and worst B4 case key (%). A4 Method R P F M

G4 Perfect note range estimation 64 47 53 53 F#4 Perfect key estimation 63 45 51 53 Worst-case key estimation 37 29 32 57 E4 Note names (in D major) D4 Ref. notes Trans. notes melody-transcription method in the MIREX05 evaluations. C#4 48 50 52 54 56 The baseline method is a slight modification of the poly- time (s) phonic music transcription method proposed in [7], and it Figure 5. The transcription of the melody from song RWC- uses multiple-F0 estimation (two F0s per frame), note event MDB-P-2001 No. 14. Figure 2 shows the features for this time modeling, and note bigrams with key estimation. segment. The proposed transcription method reached recall rate 63%, precision rate 48%, f-measure 53%, and mean overlap ratio 54% when for the baseline method the correspond- Table 1. Simulation results summary (%). ing results were 56%, 28%, 37%, and 51%. The rest model Method R P F M significantly improves the precision compared to the baseline method. By adding note-range estimation, the recall MIREX05 method (baseline) 56 28 37 51 and precision rates are slightly increased. Using key esti- Acoustic models (notes, rest) 60 42 48 54 mation with note bigrams further improves both recall and + Note-range estimation 61 43 49 54 precision rates. Finally, using simple post-processing to cor- + Key estimation and note bigrams 63 45 51 53 rect glissandi, precision rate is increased, since it reduces the + Glissando correction 63 48 53 54 number of incorrectly transcribed notes. The balance of recall and precision rates can be adjusted with the weighting of the rest model. the first note onset, and has the MIDI note number and the We studied the influence of imperfections in the note offset of the latter note. range estimation and in the key estimation to the overall performance of the method. The results are summarized 4. Simulation Results in Table 2. We used the method with all the other com- The melody transcription method was evaluated using three- ponents but the post processing (the results on the second fold cross-validation on the 96 songs in RWC popular mu- last line in Table 1). By using this method but setting the sic database. We used the performance criteria proposed note range limits according to the minimum and maximum in [7], including the recall rate (R), the precision rate (P ), of the reference notes, the recall and precision rates increase and mean overlap ratio (M). The recall rate denotes how by one and two percentage units, respectively. However, many of the reference notes were correctly transcribed and no improvement is obtained from using manually annotated the precision rate how many of the transcribed notes are cor- key signatures instead of the estimated keys (see key esti- rect. A reference note is correctly transcribed by a note in mation results in Sec. 4.2). This suggests that small errors the transcription if their MIDI note numbers are equal, the in key-estimation are not crucial to the overall performance. absolute difference between their onset times is less than We also simulated the worst-case scenario of key estimation 150 ms, and the transcribed note is not already associated by converting every reference key into a worst-case key by with another reference note. The mean overlap ratio mea- shifting its tonic by a tritone (e.g., C major key is shifted sures the average temporal overlap between transcribed and to F♯ major). This dropped the recall and precision rates reference notes. In addition, we report the f-measure F = to 37% and 29%, respectively, thus indicating that the key 2RP/(R + P ) to give an overall measure of performance. estimation plays a major role in the method. The recall rate, the precision rate, the f-measure, and the The perceived quality of the transcriptions is rather good. mean overlap ratio are calculated separately for the tran- Due to the expressive nature of singing, the transcriptions scriptions of each recording, and the average over all the include additional notes resulting from glissandi and vibrato. transcriptions for each criterion are reported. The additional notes sound rather natural although they are erroneous according to the evaluation criteria. Demonstra- 4.1. Transcription Results tions of the singing melody transcriptions done with the pro- Table 1 summarizes the melody transcription results for dif- posed method are available at http://www.cs.tut.fi/sgn/ ferent simulation setups. As a baseline method, we use our arg/matti/demos/melofrompoly. References Table 3. Key estimation results. [1] J. Eggink and G. J. Brown, “Extracting melody lines from Distance on the 0 1 2 3 ≥ 3 complex audio,” in Proc. 5th International Conference on circle of fifths Music Information Retrieval, Oct. 2004. % of songs 76.6 12.8 4.26 4.26 2.13 [2] M. Goto, “A real-time music-scene-description system: Predominant-F0 estimation for detecting melody and bass lines in real-world audio signals,” Speech Communication, 4.2. Key Estimation Results vol. 43, no. 4, pp. 311–329, 2004. [3] M. Marolt, “Audio melody extraction based on timbral simi- We also evaluated the performance of the key estimation larity of melodic fragments,” in Proc. EUROCON 2005, Nov. method. We manually annotated key signatures for 94 songs 2005. of the dataset (for two songs, the key was considered too am- [4] K. Dressler, “Extraction of the melody pitch contour from biguous). As an evaluation criterion, we use the key signa- polyphonic audio,” in Proc. 6th International Conference ture distance on the circle of fifths between the reference key on Music Information Retrieval, Sept. 2005. MIREX05 and the estimated relative-key pair. This distance is equal to extended abstract, available online http://www.music- the absolute difference in the number of accidentals (sharps ir.org/evaluation/mirex-results/articles/melody/dressler.pdf. ♯ and flats ♭). For example, if the reference key is A ma- [5] G. E. Poliner and D. P. W. Ellis, “A classification approachto jor and the key estimator correctly produces a relative-key melody transcription,” in Proc. 6th International Conference on Music Information Retrieval, pp. 161–166, Sept. 2005. pair A major / F♯ minor, the distance is zero (three sharps [6] R. P. Paiva, T. Mendes, and A. Cardoso, “On the detection for both keys). If the reference key is E minor (one sharp) of melody notes in polyphonic audio,” in Proc. 6th Interna- and the estimated relative-key pair is F major / D minor (one tional Conference on Music Information Retrieval, pp. 175– flat), the distance is two. 182, Sept. 2005. Table 3 shows the evaluation results for the key estima- [7] M. P. Ryynanen¨ and A. Klapuri, “Polyphonic music tran- tion method by using the introduced distance. The method scription using note event modeling,” in Proc. 2005 IEEE correctly estimates the relative-key pair (distance zero) for Workshop on Applications of Signal Processing to Audio and over 76% of the songs. For approximately 90% of the songs, Acoustics, pp. 319–322, Oct. 2005. the key estimation method produces correct or a perfect fifth [8] M. Ryynanen,¨ “Singing transcription,” in Signal Processing key (i.e., distance one). Methods for Music Transcription (A. Klapuri and M. Davy, eds.), pp. 361–390, Springer Science + Business Media LLC, 2006. 5. Conclusions and Future Work [9] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC music database: Popular, classical, and jazz music This paper described a method for the automatic transcrip- databases,” in Proc. 3rd International Conference on Music tion of singing melodies in polyphonic music. The method Information Retrieval, Oct. 2002. was evaluated with realistic popular music and showed a sig- [10] A. Klapuri, “A perceptually motivated multiple-F0 estima- nificant improvement in transcription accuracy compared to tion method,” in Proc. IEEE Workshop on Applications of our previous method. This was mainly due to the acoustic Signal Processing to Audio and Acoustics, pp. 291–294, Oct. modeling of no-melody (i.e., rest) segments. 2005. There is still room for improvement. One possible ap- [11] A. P. Klapuri, A. J. Eronen, and J. T. Astola, “Analysis of proach to enhance the transcription accuracy would be to the meter of acoustic musical signals,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, pp. 342– elaborate timbre information to discriminate singing notes 355, Jan. 2006. from notes played with other instruments. We did some pre- [12] M. P. Ryynanen¨ and A. Klapuri, “Modelling of note events liminary tests to include sound source separation in our tran- for singing transcription,” in Proc. ISCA Tutorial and Re- scription system. Briefly, we first generated a large set of search Workshop on Statistical and Perceptual Audio, Oct. note candidates by iteratively decoding several possible note 2004. paths. The note candidates covered approximately 80% of [13] C. Krumhansl, Cognitive Foundations of Musical Pitch. Ox- the reference notes. We then run a sound-source separation ford University Press, 1990. algorithm on these notes, calculate MFCCs on the separated [14] J. Sundberg, “The perception of singing,” in The Psychology notes, model the MFCCs of the correctly transcribed candi- of Music (D. Deutsch, ed.), ch. 6, pp. 171–214, Academic dates with a GMM to derive a timbre model, and then run the Press, second ed., 1999. Viterbi decoding again with the timbre model. Yet this approach did not perform any better than the proposed system in the preliminary simulations. However, we believe that using timbre in singing melody transcription from polyphonic music is worth further study and has the potential of improv- ing the results in instrument specific transcription tasks.