Outlines of Burcas : A simple concatenation-based MIDI-to-singing voice synthesis system
Uneson, Marcus
Published in: Proceedings of Fonetik 2002 (TMH - QPSR 43)
2002
Link to publication
Citation for published version (APA): Uneson, M. (2002). Outlines of Burcas : A simple concatenation-based MIDI-to-singing voice synthesis system. In Proceedings of Fonetik 2002 (TMH - QPSR 43) (Vol. 43). Speech, music and hearing, KTH, Stockholm, Sweden.
Total number of authors: 1
General rights Unless other specific re-use rights are stated the following general rights apply: Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal
Read more about Creative commons licenses: https://creativecommons.org/licenses/ Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
LUND UNIVERSITY
PO Box 117 221 00 Lund +46 46-222 00 00 Dept. for Speech, Music and Hearing Quarterly Progress and Status Report
Outlines of Burcas - A simple concatenation-based MIDI-to-singing voice synthesis system
Uneson, M.
journal: Proceedings of Fonetik, TMH-QPSR volume: 44 number: 1 year: 2002 pages: 133-136
http://www.speech.kth.se/qpsr
TMH-QPSR Vol. 44 – Fonetik 2002
Outlines of Burcas – a simple concatenation- based MIDI-to-singing voice synthesis system
Marcus Uneson Department of Linguistics and Phonetics, Lund University
Abstract The present paper outlines a simple system (yet to be completed) for concatena- tion-based singing synthesis in Swedish. The system, called Burcas, takes as input a MIDI file (possibly holding multiple parts) for melody and a text file for lyrics, and it produces standard audio files as output. For the digital signal processing, the MBROLA speech generator is employed. Burcas consists of an input-parsing interface to three independent phonetic modules: a letter-to-sound converter, a segment duration model, and a data base of sung diphones. In this paper, some considerations of the system as a whole and of its phonetic modules are presented, with emphasis on characteristics of concatena- tion-based synthesis of singing as opposed to synthesis of speech.
Introduction speech. However, there are also important differ- ences—most notably, in singing, the immense This paper introduces Burcas, a (not yet com- problems of reliably modelling intonation and pleted) system for singing synthesis that will syllable length are already solved (or rather by- 1 produce audio files from MIDI data and arbi- passed) by the composer. trary lyrics in Swedish. The system relies on For the purposes of a concatenated singing the MBROLA speech generator for all digital system, at least the following additional proper- signal processing (DSP). ties of singing need consideration: First, a few general issues on the difference between synthesizing speech and synthesizing 1) In singing, it is normal for vowels to be singing are considered, earlier works on the sustained subject are mentioned, and the MBROLA pro- Thus, the segment duration model must be able to ject is presented. Thereafter, the three phonetic handle not only unusually short but also very long modules of Burcas are briefly commented on: syllables. Also sonorant consonants may have the letter-to-sound (LTS) converter, the seg- very large durations. ment duration model, and the data base of sung diphones. 2) In singing, the pitch range is wider and Note-to-frequency assignment and other higher than in speech pitch-associated questions are not addressed in For that reason, the pitch scaling methods must this paper, although phonetically relevant. One perform well also at high scaling factors, or else such non-addressed issue is that of bringing the data base(s) must contain various similar to- about smooth, portamento-like transitions from kens recorded at different pitch levels, the most one note to another. For a more complete ac- appropriate among which to be chosen during count, see the Burcas web page (Burcas www). synthesis.
Singing vs speech in concatenative synthesis 3) In singing, the musical quality of the voice The task of synthesizing singing has, of course, is more critical than the intelligibility of the a lot in common with that of synthesizing lyrics The identity of an individual vowel is often sec- ondary to its intonation and timbre (in particular, 1The Musical Instrument Digital Interface (MIDI) standard is an industry-standard protocol for con- female singers, especially sopranos, often modify trolling electronic music instruments; it is supported the lip, tongue, and jaw position so that formants by practically all applications for music, like nota- coincide with partials in the source spectrum). tion software and sequencers. Perceived quality differences between vowels
133 tend to disappear anyway as pitch is raised and synthetic, concatenated speech. For the purposes the spectral envelope of vocal tract resonances of this project, the MBROLA speech generator is is sampled, as it were, more sparsely. As a regarded as a black box with three control pa- consequence, the naturalness of vowels is more rameters: phoneme, duration, and pitch. critical than that of consonants for naturally sounding singing synthesis; however, the exact Overview quality of a vowel is less important. Other phonological contrasts may also be diminished or lost. Input and output. Interface The differences mentioned concern in par- As input for melody, Burcas takes a MIDI file, ticular the singing style of (educated) opera which may contain multiple parts.2 Lyrics are singers, which is the style that departs most given in an ordinary text file, with (manually) from ordinary speech and also is the most in- inserted hyphens for separating syllables and the vestigated. However, at least the first two individual notes of melismatic vowels (as in any items of the list are valid for other genres as sheet of vocal music). Notes and text are aligned well. And all of them are relevant to the pho- syllable-wise. Each voice outputs an audio file in netic modules of Burcas. standard format (*.wav, *.au, or *.aiff).
Outlook Letter-to-sound conversion Experiments on concatenative synthesis of The daunting task of transducing a given input singing are apparently scarce (although some text in the normal orthography of a language into attempts have been made with other methods, a corresponding phoneme sequence is well- such as formant synthesis). However, a very known from TTS systems, and, for most lan- interesting project for English is known as guages, nowhere near to be solved at a more gen- Lyricos (Macon 1996, Macon et al 1997, Lyri- eral level. cos www). It uses sinusoidal waveform model Each orthographic system provides its own set parameters as concatenation units, selected of difficulties for a TTS system. For Swedish, during synthesis by a specially devised opti- they include absence of orthographical markings mizing algorithm. The data inventory is col- of lexical stress, of morpheme boundaries in lected from a professional singer. The sinusoi- compounds, and of word accent; loan words re- dal model permits convenient modification in taining their original spelling (bourgogne, chianti, the frequency domain of spectral properties rave, aficionado, nachspiel); various pronuncia- like spectral tilt. Lyricos takes MIDI as input, tions of especially the
134 TMH-QPSR Vol. 44 – Fonetik 2002
Two requirements should be fulfilled, If ρ > 1, that is, if the syllable is sustained, the though: any graphotactically acceptable input nucleus is prolonged. The MBROLA speech gen- string, although containing unknown words, erator is not meant for singing and cannot handle must produce a valid phoneme sequence with very long segments; however, this deficiency can the correct number of syllables; and any cor- be worked around by concatenating several to- rections must be easy to make. kens of the vowel, each of 200-300 ms. (Splitting Given these specifications, a simple rule- a long vowel into several, incidentally, also pro- based LTS converter will be used for Burcas, vides a convenient way of emulating a LFO for basically formalizing well-known reading rules less static timbre, by letting the duration and/or and using a small dictionary of exceptions. A frequency input of each vowel vary quasi- mixture of phonetic symbols and normal or- periodically. This feature is not yet implemented.) thography will be allowed as input, to facilitate If 0 ≤ ρ ≤ 1, the duration D of each segment is corrections. It is yet to be completed. calculated as Dmin + ρ(Dord - Dmin). A negative value of ρ raises an error. Segment duration modelling The model, although simple, allows for differ- The syllable can be regarded as a ”quantal ent compression rates of, say, nasals (whose dura- unit” of rhythm, in speech as well as in vocal tions ordinarily are 100-120 ms, but in fast sing- music. In singing, each syllable of lyric is as- ing may be 50-60 ms) versus plosives (which also sociated with a number of notes of the mel- typically are 100-130 ms, but seldom shorter than ody—one, in syllabic singing, or several, in 90-100 ms). More extensive duration data of sing- melismatic. ing are under preparation. The rhythmic information of a MIDI file is A more sophisticated approach would of no more than a set of ”note-on/note-off”- in- course be to consider quantity as well, absent in structions, with associated timepoints. For syl- the model above. That would make the task of the lable-timepoint alignment, any given timepoint LTS considerably more difficult and is currently must be anchored to a specific location in the not an issue. It should be noted that the quantity syllable. An appropriate anchor is the CV- opposition is not always present in singing; for a border between onset and nucleus—perceptual long syllable (of which there are plenty in sing- experiments have shown that listeners reliably ing), the complementary V:C vs VC: syllable place the beat of a syllable, its ”perceptual cen- structure of central Swedish is obviously difficult to retain. Apparently, the distinction may be lost ter”, at that point (Macon et al 1997). 3 The segment duration model of Burcas (a in more speech-like tempos as well. modified version of the model presented in Macon et al 1997) is straightforward. For a The diphone database given syllable, the duration of its associated The diphone data base currently used (“Ofelia”; note or notes is divided between the nucleus of female speaker, south Swedish dialect) was pro- the syllable, the coda of the syllable, and the duced from and primarily for spoken language. onset of the subsequent syllable. Each segment Although it works tolerably as a testing tool, it has a tabulated value for minimum and ’ordi- does have some drawbacks. The timbre is of nary’ duration. For each syllable, a scaling fac- course not very much like singing. More critical tor ρ is calculated: is that the [æ] and [œ] allophones have no transi- tions but /r/; they thus cannot be employed for sustained vowels. Nn N ph − Recording of a sung data base (male non- ∑ Li ∑ Dmin j = = trained singer, bass, central Swedish dialect) is ρ = i 1 j 1 , N ph scheduled to May, 2002. A possible future exten- − sion of the system is the recording of data bases at ∑(Dordk Dmink) k=1 other pitches as well (preferably by different peo- ple); in that way, undesirably high pitch-scaling where Nph is the number of phonemes of the syllable, Dmin and Dord their minimum and or- 3 dinary duration, respectively, and Nn the num- For instance, duration measurements of lament sing- ber of notes with duration L associated with ing in Estonian, where disyllabic words exhibit a three- the syllable. way quantity opposition, have shown that the acoustic correlates of quantity present in spoken language largely are lost in song (Ross & Lehiste 1994).
135 factors could be avoided. Clearly, a good start instance jazz, ethnic music, lullaby, rap). For such is to include a soprano, an alto, and a tenor. purposes, a non-specialized voice quality might be preferable to a more bel-canto style of singing. Aims, and not aims Also, such a timbre is better suited for multi- voiced pieces (and, apparently, adding voices in similar rhythms does a lot for the perceived natu- Timbre—or the lack of it ralness—probably just by drawing the listener’s Burcas is a small, zero-budget project. Many attention away from the static spectral characteris- interesting questions are not addressed at pre- tics). sent—more specifically, anything that cannot With a (much) more developed interface, Bur- be controlled by the three parameters men- cas might in a distant future prove handy as a tool tioned is silently ignored, as are any MIDI pa- for arrangers of vocal music, in that it can offer rameters relating to spectrum or intensity. synthesized versions of draft arrangements with In particular, it should be stressed that tim- arbitrary lyrics. bre is not an issue. Sampled sounds, when looped, are often perceived as lifeless and rigid Final remark by human listeners, and this is particularly true for sampled voices. For one thing, they lack This paper has presented the outlines of Burcas, a the natural pitch fluctuations that are typical of MIDI-to-singing synthesis system for Swedish the human voice; for another, they have noth- employing the MBROLA speech generator. At ing corresponding to the increase of vocal ef- the time of writing, the system has not advanced fort which a listener connects with singing very far, and it cannot yet be evaluated. However, louder or at a higher pitch (one acoustic corre- any progress will be reported on the project’s web late of which is a decreased downward tilt of page (Burcas www). the vocal spectrum). The simplest forms of concatenative methods are very rigid; a lot of References low-level difficulties are worked around by [Internet resources as for March 2002] handling sampled, predesigned building- Burcas www blocks. Once a data base is prepared, little can http://www.ling.lu.se/persons/Marcusu/music/ bur- be done to control the spectrum of a given cas/index.html phoneme independently of the fundamental. Flinger www Quasi-random pitch fluctuations can be emu- http://cslu.cse.ogi.edu/tts/flinger/ lated with a simple low-frequency oscillator Lyricos www http://cslu.cse.ogi.edu/tts/research/sing/sing.html (LFO) mechanism, but characteristics like Macon M (1996). Speech Synthesis Based on Sinusoi- spectral tilt cannot be controlled. In the long dal Modeling. PhD thesis, Georgia Institute of Tech- run, the more sophisticated approach of Lyri- nology. cos approach is certainly worth considering. Macon M, Jensen-Link L, Oliverio J, Clements M & Even disregarding signal processing issues, George E B (1997). Concatenation-based MIDI-to- the timbre of Burcas will be rather unexpres- singing voice synthesis. In: 103rd Meeting of the Audio Engineering Society, New York. sive for another reason. The planned diphone MBROLA www data base will be built on recordings not of a http://tcts.fpms.ac.be/synthesis/mbrola.html trained singer, but of an ordinary choir- Ross J & Lehiste I (1994). Lost prosodic oppositions: member. Where the former usually aims at a A study of contrastive duration in Estonian funeral personal and powerful voice, the latter rather laments. Language and Speech 37, 407-424. strives for anonymity.
Possible applications Burcas does not imitate the performance of a trained singer, nor is it meant to. Given the current limitations, the timbre of the produced singing will be uninteresting at best. However, even so, it might eventually be of use as a re- search tool, e. g. for studying temporal aspects of phrasing in different musical genres (for
136