arXiv:1711.00354v1 [cs.CL] 28 Oct 2017 od,adtae-oanadpeeetutrne.We utterances. precedent and loa travel-domain as such and utterances, words, different-domain conventional includes prosody by and it phonemes measured Also, as not such are representation, which rea intermediate individual Japanese, and in all characters ings have daily-use to designed of is corpus pronunciations Uni- The the corpus.” Tokyo) Laboratory, of Saruwatari “JSUT versity of the corpus named speech corpus, (Japanese speech Japanese large-scale free, no are there purpose. this However, for [9], e.g., synthesis. corpora, corpus speech existing speech end-to-end research Japanese on related a accelerate as would that available freely expect is We that grapheme-to-phonem dif- [8]. and more parsing conversion a semantic is e.g., Japanese task, for ficult known processing is language it However, natural 7]. in that 6, phonemes, [5, e.g., German as knowledge, and such Spanish, linguistic representations English, intermediate use of not use re an do no synthesis is speech that speech on methods to studies ported text Some from task. or particu- targeted text actively In to speech 4]. end-t from research, 3, conversion text-to-speech 2, and [1, speech-to-text in accelerated lar, have speech on studies end-to-end sis, cor- online. The available w corpus. freely paper, the is analyzed pus this and In designed pronun- we main how characters. the describe Japanese of daily-use all speech covers of reading-style and ciations of transcription hours its 10 and of data consists corpus synth The speech end-to-end “JSUT sis. achieving we the at aimed named paper, is corpus, this that speech corpus,” In Japanese novel exist. a not designed does for corpus synthesis a speech such However, Japanese role. important commercial an tha and has institutions corpus companies academic speech between large-scale shared be free can a in- learning, techniques learning deep machine cluding in improvements to Thanks nti ae,w ecieterslso osrciga constructing of results the describe we paper, this In techniques, learning deep in developments to Thanks Terms Index rdaeSho fIfrainSineadTcnlg,The Technology, and Science Information of School Graduate STCRU:FE AG-CL AAEESEC CORPUS SPEECH JAPANESE LARGE-SCALE FREE CORPUS: JSUT — .INTRODUCTION 1. pehcru,Jpns,sec synthe- speech Japanese, corpus, speech ABSTRACT ysk ooe hnoueTkmci n ioh Saruwat Hiroshi and Takamichi, Shinnosuke Sonobe, Ryosuke -- og ukok,Tko1385,Japan 133–8656, Tokyo Bunkyo-ku, Hongo 3-7-1 O N-OEDSEC SYNTHESIS SPEECH END-TO-END FOR o-end such n- d- e- e e - t . aaee 2136 Japanese, basic5000 2.2.1. Components 2.2. teacso h sub-corpus. the of utterances h olwn iesbcroa hi aei omte as formatted is name Their includes intermedi- [NAME][NUMBER] corpus sub-corpora. The cover nine following to phonemes. pronunciations the as not main such the characters, representations of ate Japanese all cover daily-use to of is corpus JSUT the Structures 2.1. freely is data, speech and [10]. online text available Japanese Japanese statistics. native including speech a and corpus, by linguistic read its analyzed data and speech speaker of hours 10 recorded • hsi h ansbcru fteJU ops In corpus. JSUT the of sub-corpus main the is This below. sub-corpora nine the designed we how describe We • • • • • • • • oaclrt n-oedrsac,temi ups of purpose main the research, end-to-end accelerate To travel1000 repeat500 (onomatopoeia). onomatopee onomatopee300 [11]. actresses voice Japanese paraphrase. its with replaced voiceactress100 is text of piece a of phrase utparaphrase512 nouns. or verbs loanword128 suffixes. counter of ings countersuffix26 characters. Japanese daily-use of nunciations basic5000 precedent130 kanji eetdysoe utterances. spoken repeatedly : rvldmi utterances. travel-domain : . teacst oe l ftemi pro- main the of all cover to utterances ... .CRU DESIGN CORPUS 2. teacsicuiglawrs e.g., loanwords, including utterances : rcdn-oanutterances. precedent-domain : . [NUMBER] aasec o recru of corpus free a for para-speech : teacsicuigidvda read- individual including utterances : hrces( characters teacsicuigfmu Japanese famous including utterances : nvriyo Tokyo, of University teacsfrwihawr or word a which for utterances : ari kanji niae h ubrof number the indicates r h logographic the are The characters used in the modern Japanese writing system) are Japanese is rich in onomatopoeia words. We crowdsourced officially defined as daily-use characters [12], and each char- 300 sentences having individual onomatopoeia words. acter has individualpronunciationsconsisting of its individual kunyomi (Chinese readings) and onyomi (Japanese readings). 2.2.7. repeat500 For example, we pronounce “一”(one in English) as “ichi,” “itsu,” “hito,” and “hito (tsu).” we collected 5000 sentences Human speech production is not deterministic, i.e., speech from Wikipedia [13] and the TANAKA corpus [14] so that waveforms always differ even if we try to reproduce the same all pronunciations of the daily-use kanji characters could be linguistic and para-linguistic information. Takamichi et al. covered. Some of the pronunciations cannot be found in these [3] proposed moment matching network-based speech syn- corpora, therefore, we manually made additional sentences to thesis that synthesizes speech with natural randomness within cover the remaining readings. the same contexts. To quantify randomness, we recorded ut- terances spoken repeatedly by a single speaker. The speaker made utterances 5 times for each of the 100 sentences of the 2.2.2. countersuffix26 Voice Actress Corpus. In Japanese, numerals cannot quantify nouns by them- selves, and the pronunciation of the numerals changes de- 2.2.8. travel1000 and precedent138 pending on the suffix. For example, “二” (“two” in English) is pronounced “ni” with “個” (ko) as the suffix and “futa” We further constructed sentences whose domain differed with “つ” (tsu) . We crowdsourced 26 sentences including from the above corpora. 1000 travel-domain sentences were such counter suffixes. collected from English-Japanese Translation Alignment Data [19]. Also, 138 copyright-free precedent sentences were col- 2.2.3. loanword128 lected from [20]. The words and phrases of the precedentsen- tences were significantly different from the above corpora, but Japanese sentences spoken daily have many loanwords, some sentences are too difficult to read. Therefore, we man- e.g., verbs and nouns, for example, “ググる (guguru)” is ually removed and modified these sentences to make reading ー a verb meaning to Google, and “ディズニ (dyizunii)” easier. means Disney. The pronunciations and accents of loanwords are a curious task in spoken language processing [15]. We 3. RESULTS OF DATA COLLECTION crowdsourced such words and sentences. Also, we collected sentences from Wikipedia that included pronunciations not 3.1. Corpus specs included in the modern Japanese system, for example, sen- tences that had a Japanese-accented foreign proper name. We hired a female native Japanese speaker and recorded her voice in our anechoic room. She was not a professional speaker but had experience working with her voice. The 2.2.4. utparaphrase512 recordings were made in February, March, September, and Paraphrasing, e.g., lexical simplification, is a technique October of 2017 for a few hours each day. The speaker made that substitutes a word or phrase into another sentence [16, the recordings herself with our recording system. The speech 17]. It can support the reading comprehensionof a wide range data was sampled at 48 kHz. We used Lancers [21] to collect of readers in speech communication. The SNOW E4 corpus several kinds of Japanese sentences. The total duration was [17, 18] includes sentences and a list of its paraphrasedwords. 10 hours including small amounts of the non-speech region. We chose one paraphrasedword per sentence, and constructed The 16 bit/sample RIFF WAV format was used. Sentences 256 sentences and paraphrased sentences. The total number (transcriptions) were encoded in UTF-8. of sentences was 512. The distributed corpora included UTF-8-encoded sen- tences, 48-kHz speech, and recording information. Because 2.2.5. voiceactress100 the recording period was comparably long and the objective The Voice Actress Corpus [11] is a free speech corpus of scores among the recording days varied as shown below, the professional Japanese voice actresses and includes not only recording information shows what day the speech data was neutral but also emotional voices. Collecting para-speech for recorded. The power of the speech data was normalized, this speech corpus is very helpful to build attractive and emo- but basically we made no additional modifications. Commas tional speech synthesis systems. We used sentences from this were added between breath groups. The positions of the corpus and manually modified the pause positions. commas were manually annotated.

2.2.6. Onomatopee300 3.2. Analysis Onomatopee (onomatopoeia) has an important role in We analyzed the linguistic and speech information of connecting speech and non-speech sounds in nature, and the constructed corpus. Note that not all of the data was 0.040 5.42 0.035 0.030 5.40 0.025 5.38 0.020 0.015 5.36 Frequency 0.010

0.005 Mean of 5.34log F0 0.000 0 20 40 60 80 100 120 140 5.32 Number of moras in one utterance

1st 5th 9th 15th16th42th43th53th54th 160th161st162th163th164th165th166th173th174th Fig. 1. Histogram of number of moras (sub-syllables) in Recording day one utterance. Minimum, mean, and maximum values are 7, 37.14, and 133, respectively. Fig. 3. Mean of log-scaled F0 for each recording day. Ordinal number of x-axis means how much time passed from “1st” recording day. For example, “5th” means 4 days after 1st 0.08 recording day. 0.07 0.06 4. CONCLUSION 0.05 0.04 In this paper, we constructed a free, large-scale Japanese 0.03 speech corpus (JSUT corpus) for end-to-end speech synthe- Frequency 0.02 sis research. The corpus was designed to have all pronuncia- 0.01 tions of daily-use kanji characters of Japanese and sentences 0.00 0 10 20 30 40 50 60 70 of several domains. The corpus may be used for research by Number of words in one utterance academic institutions and non-commercial research including research conducted within commercial organizations. Acknowledgements: Part of this work was supported by Fig. 2. Histogram of number of words in one utterance. Min- the SECOM Science and Technology Foundation. We thank imum, mean, and maximum values are 2, 18.03, and 70, re- Dr. Masahiro Mizukami of the Nara Institute of Science spectively. and Technology for the fruitful discussion on the paraphrase corpus, Assistant Prof. Kazuhide Yamamoto of the Nagaoka University of Technology and Tomoyuki Kajiwara of the Tokyo Metropolitan University for the use of the SNOW E4 used for the analysis to shorten the computation time. First, corpus, and the person in charge of the Voice Actress Corpus we counted the number of moras (sub-syllables) and words for the use of their corpus. within one utterance by using MeCab [22] and NEologd [23, 24]. The utterance length is the important factor in 5. REFERENCES speech synthesis using the sequence-to-sequencemechanisms [25, 26]. Fig. 1 and Fig. 2 show histograms of the moras [1] G. Hinton, L. Deng, D. Yu, G. Dahl, A. r. Mo- and words, respectively. As we can see, the corpus included hamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, a variety of lengths, from short utterances (a few words and T. Sainath, and B. Kingsbury, “Deep neural networks moras) to long utterances (70 words and 140 moras). for acoustic modeling in speech recognition: The shared views of four research groups,” Signal Processing Mag- Next, we analyzed the changes in speech statistics per azine of IEEE, vol. 29, no. 6, pp. 82–97, 2012. recording day. Speech data recorded during long periods causes objective and subjective differences among recording [2] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, days [27]. The Mean of log F0 was calculated for each O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, recording day. F0 was extracted by using the WORLD and K. Kavukcuoglu, “WaveNet: A generative model analysis-synthesis system [28]. Fig. 3 shows the result. for raw audio,” vol. abs/1609.03499, 2016. There was no special tendency in the first half of the record- ings, but we can see that the log F0 increased for the days of [3] S. Takamichi, K. Tomoki, and H. Saruwatari, the second half. “Sampling-based speech parameter generation using moment-matching network,” in Proc. INTERSPEECH, [17] K. Tomoyuki and Y. Kazuhide, “Evaluation dataset and Stockholm, Sweden, Aug. 2017. system for japanese lexical simplification,” in Proceed- ings of the ACL-IJCNLP 2015 Student Research Work- [4] Y. Saito, S. Takamichi, and H. Saruwatari, “Training al- shop, Beijing, China, July 2015, pp. 35–40. gorithm to deceive anti-spoofing verification for DNN- based speech synthesis,” in Proc. ICASSP, Orleans, [18] “SNOW E4: evaluation data set of japanese lexical sim- U.S.A., Mar. 2017. plification,” http://www.jnlp.org/SNOW/E4, 2010. [5] Y. Wang, RJ Skerry-Ryan, D. Stanton, Y. Wu, Ron J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Ben- [19] M. Utiyama and M. Takahashi, “English- gio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. japanese translation alignment data,” Saurous, “Tacotron: Towards end-to-end speech syn- http://www2.nict.go.jp/astrec-att/member/mutiyama/align/index.html, thesis,” vol. abs/1609.03499, 2017. 2003.

[6] S. Jose, M. Soroush, K. Kundan, S. Jo˜ao F., K. Kyle, [20]“COURTS IN JAPAN,” http://www.courts.go.jp/app/hanrei_jp/search1 C. Aaron, and B. Yoshua, “Char2Wav: End-to- . end speech synthesis,” in International Conference [21] “Lancers http://www.lancers.jp,” . on Learning Representations (Workshop Track), April 2017. [22] T. Kudo, K. Yamamoto, and Y. Matsumoto, “Apply- ing conditional random fields to Japanese morphologi- [7] O. Watts, “Unsupervised learning for text-to-speech cal analysis,” in Proc. EMNLP, Barcelona, Spain, Jul. synthesis,” Ph. D thesis of the University of Edinburgh, 2004, pp. 230–237. 2012. [23] T. Sato, T. Hashimoro, and M. Okumura, “Implemen- [8] K. Kubo, S. Sakti, G. Neubig, T. Toda, and S. Naka- tation of a word segmentation dictionary called mecab- mura, “Narrow adaptive regularization of weights for ipadic-neologdand study on how to use it effectively for grapheme-to-phoneme conversion,” in Proc. ICASSP, information retrieval (in Japanese),” in Proceedings of Florence, Italy, May 2014. the Twenty-three Annual Meeting of the Association for Natural Language Processing, 2017, pp. NLP2017–B6– [9] M. Abe, Y. Sagisaka, T. Umeda, and H. Kuwabara, 1. “ATR technical report,” , no. TR-I-0166M, 1990. [24] T. Sato, “Neologism dictionary based on the language [10] “JSUT: Japanese speech corpus of Saruwatari resources on the web for Mecab,” 2015. Lab, the University of Tokyo corpus,” https://sites.google.com/site/shinnosuketakamichi/p[25] W. Wang, S.ublication/jsut Xu, and B. Xu, “First. step towards end- toend parametric TTS synthesis: Generating spectral [11] y benjo and MagnesiumRibbon, “Voice-actress corpus,” parameters with neural attention,” in Proc. INTER- http://voice-statistics.github.io/. SPEECH, San Francisco, U.S.A., Sep. 2016, pp. 2243– 2247. [12] Governments of Japan Agency for Cul- tural Affairs, “List of daily-use kanjis [26] H. Miyoshi, Y. Saito, S. Takamichi, and H. Saruwatari, http://www.bunka.go.jp/kokugo_nihongo/sisaku/joho/j“Voice conversionoho/kijun/naikaku/kanji/index.html using sequence-to-sequence learning ,” 2010. of context posterior probabilities,” in Proc. INTER- SPEECH, Stockholm, Sweden, Aug. 2017, pp. 1268– [13] “Wikipedia,” https://ja.wikipedia.org/. 1272.

[14] Y. Tanaka, “Compilation of a multilingual parallel cor- [27] H. Kawai, T. Toda, J. Ni, M. Tsuzaki, and K. Tokuda., pus,” in Proc. Pacling2001, 2001. “XIMERA: a new TTS from ATR based on corpus- based technologies,” in Proc. SSW5, Pittsburgh, USA, [15] H. Kubozono, “Where does loanword prosody come June 2004, pp. 179–184. from?: A case study of Japanese loanword accent,” Lin- gua, vol. 116, no. 7, pp. 1140–1170, 2006. [28] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoder-based high-quality speech synthesis system for [16] M. Moku, K. Yamamoto, and A. Makabi, “Automatic real-time applications,” IEICE transactions on infor- easy Japanese translation for information accessibility mation and systems, vol. E99-D, no. 7, pp. 1877–1884, of foreigners,” in the Workshop on Speech and Lan- 2016. guage Processing Tools in Education, 2012, pp. 85–90.