Neural Text-To-Speech with a Modeling-By-Generation Excitation Vocoder
Total Page:16
File Type:pdf, Size:1020Kb
INTERSPEECH 2020 October 25–29, 2020, Shanghai, China Neural Text-to-Speech with a Modeling-by-Generation Excitation Vocoder Eunwoo Song1, Min-Jae Hwang2, Ryuichi Yamamoto3, Jin-Seob Kim1, Ohsung Kwon1, and Jae-Min Kim1 1NAVER Corp., Seongnam, Korea 2Search Solutions Inc., Seongnam, Korea 3LINE Corp., Tokyo, Japan Abstract propagated throughout the synthesis process. It is therefore cru- cial to model the interactions between the acoustic and vocod- This paper proposes a modeling-by-generation (MbG) ex- ing elements during the training process in order to achieve the citation vocoder for a neural text-to-speech (TTS) system. Re- best complete performance of the TTS system. cently proposed neural excitation vocoders can realize quali- In this paper, we propose a neural excitation model based fied waveform generation by combining a vocal tract filter with on modeling-by-generation (MbG) in which the spectral param- a WaveNet-based glottal excitation generator. However, when eters generated from the acoustic model are utilized in the neu- these vocoders are used in a TTS system, the quality of syn- ral vocoder’s training process. Specifically, the target excitation thesized speech is often degraded owing to a mismatch be- is defined as a combination of the prediction errors from the tween training and synthesis steps. Specifically, the vocoder LP analysis and those from the acoustic model. The vocoding is separately trained from an acoustic model front-end. There- model is then optimized to learn the distribution of the target fore, estimation errors of the acoustic model are inevitably excitation while compensating for the errors from the acoustic boosted throughout the synthesis process of the vocoder back- model. It has been reported elsewhere that training the neural end. To address this problem, we propose to incorporate an vocoder with generated acoustic parameters improves synthetic MbG structure into the vocoder’s training process. In the pro- quality [11]. Although the MbG method is similar to this ap- posed method, the excitation signal is extracted by the acoustic proach, there are also clear differences in that MbG aligns even model’s generated spectral parameters, and the neural vocoder the target excitation signal with the acoustic model’s generated is then optimized not only to learn the target excitation’s dis- spectral parameters. tribution but also to compensate for the estimation errors oc- We investigated the effectiveness of the proposed method curring from the acoustic model. Furthermore, as the gener- by conducting subjective evaluation tasks. The MbG structure ated spectral parameters are shared in the training and synthe- can be extended to any neural excitation vocoder that uses LP sis steps, their mismatch conditions can be reduced effectively. coefficients, but the focus here is on the WaveNet-based Ex- The experimental results verify that the proposed system pro- citNet vocoder [7]. The experimental results show that a TTS vides high-quality synthetic speech by achieving a mean opin- system with the proposed MbG-ExcitNet vocoder provides sig- ion score of 4:57 within the TTS framework. nificantly better perceptual quality than a similarly configured Index Terms: neural text-to-speech, WaveNet, ExcitNet, system with a conventional vocoder. In particular, our TTS modeling-by-generation vocoder framework achieves 4:57 mean opinion score (MOS). 1. Introduction 2. Related work Generative models for raw speech waveform have significantly The idea of using an MbG structure is not new. In a study of improved the quality of neural text-to-speech (TTS) systems parametric glottal vocoders, Juvela et al. [12] first proposed the [1,2]. Specifically, by conditioning acoustic features to the net- closed-loop extraction of glottal excitation from the generated work input, neural vocoding models such as WaveNet, Wav- spectral parameters, and our own previous work proposed the eRNN, and WaveGlow successfully generate a time-sequence MbG structure to compensate for missing noise components of speech signal [2–5]. More recently, neural excitation in generated glottal signals [13]. However, it was not possi- vocoders such as GlotNet, ExcitNet, LP-WaveNet and LPC- ble to fully utilize the effectiveness of the MbG training strat- Net [6–10] have exploited the advantages of linear prediction egy because our experiments were only performed with sim- (LP)-based parametric vocoders. In this type of vocoder, an ple deep learning models including stacked feed-forward and/or adaptive predictor is used to decouple the formant-related spec- long short-term memory (LSTM) networks. tral structure from the input speech signal, and the probability Our aim here was to extend the usage of the MbG structure distribution of its residual signal (i.e. the excitation signal) is to recently proposed neural excitation models (e.g. ExcitNet) then modeled by the vocoding network. As variation in the ex- with autoregressive acoustic models (e.g. Tacotron) [11,14,15]. citation signal is only constrained by vocal cord movement, the As the accuracy of acoustic models has been significantly im- training and generation processes become much more efficient. proved, it is now possible to extract stable excitation signals However, because the vocoding and acoustic models have from the generated spectral parameters. Furthermore, the Ex- been trained separately, it is not known whether or not com- citNet vocoder directly models the time-domain excitation se- bining them within the TTS framework would benefit synthesis quence which enables straightforward application of the MbG quality. Furthermore, as parameters estimated from the acoustic structure to the training process. As a result, the entire model model are used as a direct input of the vocoding model in the can be stably and easily trained while the perceptual quality of synthesis step, estimation errors of the acoustic features can be the synthesized speech is significantly improved. Copyright © 2020 ISCA 3570 http://dx.doi.org/10.21437/Interspeech.2020-2116 (a) (b) (c) Figure 1: An ExcitNet vocoder for a TTS system: (a) conventional training; (b) proposed MbG training; and (c) synthesis methods. quence of the excitation signal. Finally, the speech signal is re- constructed by passing the generated excitation signal through the LP synthesis filter. 3.2. MbG-structured ExcitNet vocoders To further improve the quality of the synthesized speech, we propose the incorporation of an MbG structure into the train- ing process of the ExcitNet vocoder. As illustrated in Fig. 1a, conventional vocoding models are trained separately from the acoustic model, even though the generated acoustic parameters, which contain estimation errors, are used as direct conditional inputs (Fig. 1c). This inevitably causes quality degradation of the synthesized speech as the estimation errors from the acous- Figure 2: Negative log-likelihood (NLL) obtained during the tic model are boosted non-linearly throughout the synthesis pro- training process with respect to the plain ExcitNet and MbG- cess in the vocoder back-end. based ExcitNet (MbG-ExcitNet) training methods. Fig. 1b shows the proposed MbG training method which uses closed-loop extraction1 of the excitation signal. To mini- mize the mismatch between the training and the generation pro- 3. ExcitNet TTS systems cesses, the LP coefficients in the training step are replaced with those generated by the pre-trained acoustic model as follows: 3.1. ExcitNet vocoders p The basic WaveNet framework is an autoregressive network X e^n = xn − α^kxn−k; (3) which generates a probability distribution of discrete speech k=1 symbols from a fixed number of past samples [16]. The Excit- Net vocoder is an advanced version of this network which takes where fα^1; :::; α^pg denotes the generated LP coefficients. By advantages of both the LP vocoder and the WaveNet structure. combining equations (2) and (3), the excitation sequence can be In an ExcitNet framework, an LP-based adaptive predictor is represented as follows: used to decouple the spectral formant structure from the input am speech signal (Fig. 1a). The WaveNet model is then used to e^n = en + en ; (4) train the distribution of the prediction residuals (i.e. excitation) am as follows: where en denotes an intermediate prediction defined as fol- N lows: Y p p(ejh) = p(enje1; :::; en−1; h); (1) am X en = (αk − α^k)xn−k: (5) n=1 k=1 p Using the excitation signal (i.e. e^ ) as the training target means X n en = xn − αkxn−k; (2) that it becomes possible to guide the model to learn the distribu- k=1 tions of the true excitation signal (i.e. en) as well as compensate am th for the acoustic model’s estimation errors (i.e. e ). Further- where xn and en denote the n sample of speech and excita- n th more, because the training and synthesis processes share the tion, respectively; αk denotes the k LP coefficient with the order p; h denotes the conditional inputs composed of acoustic same LP coefficients, it is also possible to minimize any mis- parameters. match. In the speech synthesis step (Fig. 1c), the acoustic parame- 1This extraction method has been adopted in analysis-by-synthesis ters of the given input text are generated by a pre-trained acous- speech coding frameworks [17,18] where the encoder and decoder share tic model. These parameters are then used as conditional inputs the same quantized filter parameters for minimizing their mismatch con- for the WaveNet model to generate the corresponding time se- ditions. 3571 Figure 3: Acoustic model consisting of three sub-modules: context analysis, context embedding, and Tacotron decoding. The merits of the proposed method are presented in Fig. 2 In context embedding, the linguistic features were trans- which shows the negative log-likelihood obtained from the formed into high-level context vectors. The module here con- training and validation sets. The proposed MbG-ExcitNet sisted of three convolution layers with a 10×1 kernel and 512 model enables a reduction in both training and validation er- channels per layer, a bi-directional LSTM network with 512 rors as compared to a plain ExcitNet approach.