Neural Machine Translation of Logographic Languages Using Sub-character Level Information
Longtu Zhang Mamoru Komachi Tokyo Metropolitan University Tokyo Metropolitan University 6-6 Asahigaoka, Hino, 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan Tokyo 191-0065, Japan [email protected] [email protected]
Abstract model better encode the source sentence and de- code the target sentence, particularly when the Recent neural machine translation (NMT) source and target languages share some similar- systems have been greatly improved by ities. encoder-decoder models with attention mech- anisms and sub-word units. However, im- Almost all of the methods used to improve portant differences between languages with NMT systems were developed for alphabetic lan- logographic and alphabetic writing systems guages such as English, French, and German as have long been overlooked. This study fo- either the source or target language, or both. An cuses on these differences and uses a simple alphabetic language typically uses an alphabet: a approach to improve the performance of NMT small set of letters (basic writing symbols) that systems utilizing decomposed sub-character level information for logographic languages. each roughly represents a phoneme in the spo- Our results indicate that our approach not ken language. Words are composed by ordered only improves the translation capabilities of letters, and sentences are composed by space- NMT systems between Chinese and English, segmented ordered words. However, in other but also further improves NMT systems be- major writing systems—namely, logographic (or tween Chinese and Japanese, because it uti- character-based) languages such as Chinese, lizes the shared information brought by simi- Japanese, and traditional Korean—strokes are lar sub-character units. used to construct ideographs; ideographs are used 1 Introduction to construct characters, which are the basic units for meaningful words. Words can then further Neural machine translation (Cho et al., 2014) compose sentences. In alphabetic languages, (NMT) systems based on sequence-to-sequence sub-word units are easy to identify, whereas in models (Sutskever et al., 2014) have recently be- logographic languages, a similar effect can be come the de facto standard architecture. The achieved only if sub-character level information models use attention mechanisms (Bahdanau is taken into consideration.1 et al., 2015; Luong et al., 2015) to keep records Having noticed this significant difference of all encoding results, and can focus on particu- between these two writing systems, Shi et lar parts of these results during decoding, so that al. (2015), Liu et al. (2017), Peng et al. (2017), the model can produce longer and more accurate and Cao et al. (2017) used stroke-level informa- translations. Sub-word units are another tech- tion for logographic languages when constructing nique first introduced by Sennrich’s (2016) appli- word embeddings; Toyama et al. (2017) used vi- cation of the byte pair encoding (BPE) algorithm, sual information for strokes and Japanese Kanji and are used to break up words in both source and target sentences into sequences of smaller units, 1Taking the ASPEC corpus as an example, the average learned without supervision. This alleviates the word lengths are roughly 1.5 characters (Chinese words, to- kenized by Jieba tokenizer), 1.7 characters (Japanese words, risk of producing
Proceedings of the Third Conference on Machine Translation17 (WMT), Volume 1: Research Papers, pages 17–25 Belgium, Brussels, October 31 - Novermber 1, 2018. c 2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64003 radicals in a text classification task.2 2. We facilitate the encoding or decoding pro- Some studies have performed NMT tasks using cess by using sub-character sequences on ei- various sub-word “equivalents”. For instance, ther the source or target side of the NMT Du and Way (2017) trained factored NMT mod- system. This will improve translation perfor- els using “Pinyin”3 sequences on the source side. mance; if sub-character information is shared Unfortunately, they did not apply a BPE algo- between the encoder and decoder, it will fur- rithm during training, and their model also cannot ther benefit the NMT system. perform factored decoding. Wang et al. (2017) 4 directly applied a BPE algorithm to character se- 3. Specifically, Chinese ideograph data and quences before building NMT models. However, Japanese stroke data are the best choices for they did not take advantage of sub-character level relevant NMT tasks. information during the training of sub-word and 2 Background NMT models. Kuang and Han (2018) also at- tempted to use a factored encoder for Chinese 2.1 NMT with Attention Mechanisms and NMT systems using radical data. It is worth not- Sub-word Units ing that although the idea of using ideographs and In this study, we applied a sequence-to-sequence strokes in NLP tasks (particularly in NMT tasks) model with an attention mechanism (Bahdanau is not new, no previous NMT research has fo- et al., 2015). The basic recurrent unit is the “long cused on the decoding process. If it is also possi- short-term memory” (Hochreiter and Schmidhu- ble to construct an ideograph/stroke decoder, we ber, 1997) unit. Because of the nature of the can further investigate translations between lo- sequence-to-sequence model, the vocabulary size gographic languages. Additionally, no NMT re- must be limited for the computational efficiency search has previously used stroke data. of the Softmax function. In such cases, the de- To summarize, there are three potential in- coder outputs an
18 Semantic Phonetic Word Meaning Ideographs Character Pinyin ideograph ideograph 树木 Wood 木对木 驰 run 马 horse 也 chí 森林 Forest 木木木木木 池 pool 水(氵) water 也 chí 施 impose 方 direction 也 sh Table 2: Examples of multi-character words in Chi- 弛 loosen 弓 bow 也 chí nese and their ideograph sequences. 地 land 土 soil 也 dì 驱 drive 马 horse 区 q tively). A few ideographs can also be treated as Table 1: Examples of decomposed ideographs of Chi- standalone characters. nese characters. The composing ideographs of differ- To the best of our knowledge, however, no re- ent functionality might be shared across different char- search has been performed on logographic lan- acters. guage NMT beyond character-level data, except in the work of Du and Way (2017), who attempted models in English/Japanese NMT tasks. The sub- to use Pinyin sequences instead of character se- word units were trained on character (Japanese quences in Chinese–English NMT tasks. Consid- Kanji and Hiragana/Katakana) sequences. Sim- ering the fact that there are a large number of ho- ilarly, Wang et al. (2017) attempted to compare mophones and homonyms in Chinese languages, the effects of different segmentation methods on it was difficult for this method to be used to re- NMT tasks, including “BPE” units trained on construct characters in the decoding step. Chinese character sequences. 3 NMT Using Sub-character Level Units 2.2 Sub-character Units in NLP 3.1 Ideograph Information In alphabetic languages, the smallest unit for When building NMT vocabulary, the use of sub- sub-word unit training is the letter; in character- characters (instead of words, characters, and char- based languages, the smallest units should be sub- acter level sub-words) can greatly condense vo- character units, such as ideographs or strokes. cabulary size. For example, a vocabulary can be Because sub-character units are shared across dif- decreased from 6,000 to 10,000 character types7 ferent characters and have similar meanings, it to hundreds 8 of ideographs. Table 2 presents two is possible to build a significantly smaller vocab- Chinese words composed of four different char- ulary to cover a large amount of training data. acters that have very close meanings. Character- This has been researched quite extensively within based NMT models treat these characters sep- tasks such as word embeddings, as mentioned arately as one-hot vectors. In contrast, if the previously. two words are broken down into ideograph se- As we can see from the examples in Table 1, quences, they overlap significantly. Then, only there are several independent Chinese charac- two ideographs are needed to compose the vocab- ters. Each character can be split into at least ulary of the two words. The computational load two ideographs: a semantic ideograph and a pho- will be reduced, and the chances of training neu- 6 netic ideograph. More importantly, the same rons responsible for low-frequency vocabularies ideograph can be shared by different characters will increase. denoting similar meanings. For example, the Moreover, sub-character units can serve as first five characters (驰, 池, 施, 弛 and 地) have building blocks for constructing characters that similar pronunciation (and they are written sim- are not present in the training data, because all ilarly in Pinyin) because they share the same CJK characters are designed to be composed of a phonetic ideograph “也”. Similarly, semantic limited number of ideographs in UNICODE stan- ideographs can be shared across characters and dards. denote a similar semantic meaning. For exam- ple, the first character “驰” and the last char- 3.2 Stroke Information acter “驱” share same semantic ideograph “马” All ideographs can be further decomposed into (meaning “horse”); and their semantic meanings strokes, which are smaller units and have an even are closely related (“run” and “drive”, respec- 7According to the ASPEC corpus. 6Semantic ideographs denote the meaning of a character, 8214 as defined in UNICODE 10.0 standard and 517 as whereas phonetic ideographs denote the pronunciation. defined in CNS11643 charset.
19 smaller number of types. Therefore, we also Language Word propose training our model on stroke sequences. JP-character 風 景 There are five basic stroke types for Chinese char- JP-ideograph 几一虫 日亠口小_1 acters and Japanese Kanji: “horizontal” (一), 丿㇈㇐㇑㇕㇐㇑㇀㇔ JP-stroke “vertical” (丨), “right falling” (㇃), “left falling” ㇑㇕㇐㇐㇔㇐㇑㇆㇐㇚㇒㇔_1 (丿), and “break” (㇕). Each stroke type can be CN-character 风 景 further sub-categorized into several stroke varia- CN-ideograph 几㐅 日亠口小_1 tions. For example, left falling strokes contain 丿㇈㇒㇔ CN-stroke both long and short left fallings (㇓ and ㇒), while ㇑㇕㇐㇐㇔㇐㇑㇆㇐㇚㇒㇔_1 a break contains many more variations, such as EN landscape ㇄, ㇅, ㇆, and ㇉ (details can be found in Ap- pendix A). Table 3: The Japanese word 風景 and Chinese word In practice, the CNS11643 charset9 is used to 风景 both mean “landscape” in English, and they only transform each character into a stroke sequence, differ in the middle part of the first character. Note that where unfortunately only “stroke-type” informa- there are “_1” tags at the ends of some decomposed se- quences to distinguish between possible duplications. tion is available. In this study, we manually tran- scribed all ideographs into stroke sequences using 33 pre-defined strokes. cating sub-character structures (Ideographic De- scription Characters), which provide a clearer in- 3.3 Character Decomposition dication of character compositions. We will make The CNS11643 charset is used to facilitate char- further use of this information in future studies. acter decomposition, where Chinese, Japanese, To ensure that there are no duplicated ideo- and Korean characters are merged into a sin- graph and stroke sequences for different charac- gle character type based on similarities in their ters and multi-character words, we post-tag the se- forms and meanings. This is potentially bene- quences on the duplicated ones using “_1”, “_2”, ficial; for example, if Chinese and Japanese vo- etc. Table 3 shows an example of character de- cabularies are built, they will authentically share composition in Chinese and Japanese11. some common types. There are 517 so-called “components” (i.e., ideographs) pre-defined in 4 Experiments on CNS11643. This ensures that all characters can Chinese–Japanese–English Translation be divided into certain sequences of components. To answer our research questions, we set up a For example, the character “可” can be split into series of experiments to compare NMT mod- “丁” and “口”; and the character “君” can be els of logographic languages trained on word split into “尹” and “口”. Details can be found sequences, character-level sub-word unit se- 10 on the CNS11643 website . Using this ideo- quences, and ideograph- and stroke-level sub- graph decomposition information, all Chinese word unit sequences. and Japanese sentence data can be transformed We performed two lines of experiments: into new ideograph sequences; then, using the manually transcribed stroke decomposition data 1. We trained NMT models between logo- introduced in Section 3.2, we can also obtain new graphic language and alphabetic language stroke sequences. combinations, i.e., Japanese/Chinese and Note that although there are no clear indica- English. In each model, we varied the data tions of how the components/strokes are struc- granularity for the logographic language, tured together, the sequence potentially contains using “character level” or “sub-character structural information, because the writing of level” (ideograph level and stroke level) characters always follows a certain order, such as granularities. We used the character level
“up-down”, “outside-in”, etc. We also note that 11For example, the ideograph and stroke sequences for UNICODE 10.0 has introduced symbols indi- character 景 are the same as those for character 晾 (meaning “to dry in the sun”). However, these two characters have dif- 9The CNS11643 charset is published and maintained by ferent architectures (“top-down” vs. “left-right”). Post-tags the Taiwan government. are thus appended in order to distinguish them. Similarly, -have the same ideograph and stroke se ذ http://www.cns11643.gov.tw/AIDB/welcome_en.do characters 风 and 10http://www.cns11643.gov.tw/search.jsp?ID=13 quences, and thus must be post-tagged.
20 NMT models as our baselines, and investi- we made random selections from the develop- gated whether the sub-character level NMT ment set and test set of 1,000 sentences each. models could outperform the baseline mod- els. 4.2 Settings Different pre-tokenization methods were applied 2. We trained NMT models between combina- to the data in three languages (if applicable). A tions of two logographic languages, i.e., Chi- Moses tokenizer was applied to the English data; nese and Japanese. Similarly, we used data a Jieba13 tokenizer using the default dictionary sets with different granularities: 1) Models was applied to the Chinese data; and a MeCab14 lacking sub-character level data. 2) Mod- tokenizer using the IPA dictionary was applied to els having sub-character level data on both the Japanese data. For the Pinyin baseline, the sides (to confirm the results of the previ- pypinyin15 Python library was used to transcribe ous experiment). For the experiments, the the Chinese character sequence into a Pinyin se- models will have both source and target quence. sides. The models will use sub-character In both of the experiment lines discussed level data with/without shared vocabularies above, data at the “word”, “character”, “ideo- (namely, ideograph models, stroke models, graph”, and “stroke” levels were used in combi- ideograph-stroke models, stroke-ideograph nations. For “word” level data, only dictionary- models, and ideograph/stroke models with based segmentation was applied; for the other shared vocabularies). 3) Pinyin baselines ac- three levels of data, the byte pair encoding (BPE) cording to (Du and Way, 2017), where both models were trained and applied, with a vocabu- Pinyin word sequences with tones and char- lary size of 8,000. In the second line of exper- acter sequences with Pinyin factors are used iments, where both the source and target sides with the encoder. were logographic languages, we added “charac- ter” level data without BPE (“char”) for com- 4.1 Dataset parison. Additionally, shared vocabularies were We trained our baselines and experiments using applied when both the source and target had the Chinese, Japanese, and English. The Asian Sci- same data granularity level (meaning that both the entific Paper Excerpt Corpus (ASPEC (Nakazawa source and target side would have the same vocab- et al., 2016)) and Casia201512 corpus were used ulary)16. for this purpose. A basic RNNsearch model (Bahdanau et al., ASPEC contains a Japanese–English paper 2015) with two layers of long short-term mem- abstract corpus of 3 million parallel sentences ory (LSTM) units was used. The hidden size was (ASPEC-JE) and a Japanese–Chinese paper 512. A normalized Bahdanau attention mecha- excerpt corpus of 680,000 parallel sentences nism was applied at the output layer of the de- (ASPEC-JC). We used the first million con- coder. We developed our model based on Ten- fidently aligned parallel sentences in ASPEC- sorFlow17 and its neural machine translation tu- JE and all of the ASPEC-JC data to cover torial18. Japanese–English and Japanese–Chinese lan- The model was trained on a single GeForce guage pairs. The Casia2015 corpus contains ap- GTX TITAN X GPU. During training, the SGD proximately 1 million parallel Chinese–English optimizer was used, and the learning rate was sentences. All data in the Casia2015 corpus were set at 1.0. The size of the training batch was used to cover Chinese–English language pairs. set to 128, and the total global training step was During training, the maximum length hyperpa- 250,000. We also decayed the learning rate as the rameter was adjusted to ensure 90% coverage of training progressed: after two-thirds of the train- the training data. For development and testing, 13 the ASPEC corpus has an official split between https://github.com/fxsjy/jieba 14http://taku910.github.io/mecab/ the development set and test set; however, be- 15https://github.com/mozillazg/python-pinyin cause the Casia2015 corpus is not similarly split, 16The shared vocabulary can be trained by a BPE model on a concatenated corpus of source and target sentences. 12http://nlp.nju.edu.cn/cwmt-wmt/, provided by the Insti- 17https://github.com/tensorflow tute of Automation, Chinese Academy of Sciences. 18https://github.com/tensorflow/nmt
21 English-Japanese NMT BLEU Section 5. EN_word JP_word 36.1 EN_word JP_character 38.3 4.3.2 NMT of Logographic Language Pairs EN_word JP_ideograph 40.3∗ EN_word JP_stroke 41.3∗ Table 5 shows the results for all baselines and pro- Japanese-English NMT BLEU posed models. Among the character-level base- JP_word EN_word 25.5 lines, the “char” models and “bpe” models out- JP_character EN_word 26.3 performed the “word” models in both translation JP_ideograph EN_word 26.8∗ directions. When we applied a shared vocabu- JP_stroke EN_word 27.0∗ lary to the “bpe” models, the models achieved the English-Chinese NMT BLEU best BLEU scores in both translation directions. EN_word CN_word 11.8 EN_word CN_character 10.3 These character-level baselines conform with pre- EN_word CN_ideograph 14.6∗ vious studies indicating that sub-word units im- EN_word CN_stroke 14.1∗ prove the performance of NMT systems, and that Chinese-English NMT BLEU whenever both the source and target side data CN_word EN_word 14.7 have similarities in their writing systems, shared CN_character EN_word 14.5 vocabularies will further enhance performance. CN_ideograph EN_word 15.6∗ Sub-character level models aim to replicate CN_stroke EN_word 15.5∗ similar results to those presented in Section 4.3.1, Table 4: Experimental results (BLEU scores) of NMT because only one side of these models uses sub- systems for Japanese/English and Chinese/English character level data. For Japanese–Chinese trans- language pairs. All the scores are statistically signifi- lation directions, half of the models showed a sig- cant at p = 0.0001 (marked by ). ∗ nificant improvement over the baselines, whereas for Chinese–Japanese translation directions, five ing steps, we set the learning rate to be four times out of six models showed significant improve- smaller until the end of training. Additionally, we ments. set the drop-out rate to 0.2 during training. When both the source and target side used the BLEU was used as the evaluation metric in same sub-character level data (either ideograph or our experiments. For Chinese and Japanese data, stroke data), the experimental results also showed a KyTea tokenization was applied before we ap- significant improvement over character baselines. plied BLEU, following the WAT (Workshop on Additionally, the ideograph models outperformed Asian Translation) leaderboard standard. To val- stroke models. When shared vocabularies were idate the significance of our results, we ran boot- applied to the models, the ideograph models ex- strap re-sampling (Koehn, 2004) for all results us- hibited slight performance improvements (0.1 ∼ ing Travatar (Neubig, 2013) at a significance level 0.4 BLEU point), and the stroke models exhib- of p = 0.0001. ited dramatically decreased performance (0.9 ∼ 1.1 BLEU points). However, no model here out- 4.3 Results performed the sub-character baselines. 4.3.1 NMT of Logographic and Alphabetic To further exploit the power of sub-character Language Pairs units, the last models having different levels of Table 4 shows the experimental results for the sub-character units on the source and target side Japanese/English and Chinese/English language were trained. The results conform with what we pairs in both translation directions. Generally, found in Section 4.3.1: the models using Chinese for each of the experiment settings, the mod- ideograph data and Japanese stroke data exhib- els using ideograph and stroke data outperformed ited the best performance, regardless of whether the baseline systems, regardless of the language they were applied at the source or target side. For pair or translation direction. However, for the Japanese–Chinese translations, the best BLEU Japanese/English language pair, the stroke se- score was 33.8, which was produced by the quence models performed better. For the Chi- Japanese-stroke and Chinese-ideograph model; nese/English language pairs, the ideograph se- for Chinese–Japanese translation, the best BLEU quence models worked better. The reason for score was 43.9, which was produced by the these differences will be discussed in detail in Chinese-ideograph and Japanese-stroke model.
22 JP-CN NMT CN_word CN_char CN_bpe CN_ideograph CN_stroke JP_word 29.6 -- 30.8 30.3 JP_char - 31.6 - 32.0∗ 32.1∗ JP_bpe -- 31.5 (31.7) 31.6 31.7 JP_ideograph 30.4 33.1∗ 33.3∗ 32.0∗ (32.4∗) 33.4∗ JP_stroke 30.3 33.4∗ 32.6∗ 33.8∗ 32.1∗ (31.2) CN-JP NMT JP_word JP_char JP_bpe JP_ideograph JP_stroke CN_word 40.0 (40.0)-- 40.5 40.1 CN_char 42.1 (40.4) 41.7 - 43.1∗ 42.2∗ CN_bpe 42.1 - 42.0 (42.3) 43.1∗ 42.2∗ CN_ideograph 43.2∗ 43.5∗ 43.0∗ 42.6∗ (42.7∗) 43.9∗ CN_stroke 43.0∗ 43.3∗ 42.5∗ 42.9∗ 42.2∗ (41.1)
Table 5: Experimental results (BLEU scores) for Japanese/Chinese NMT systems. The row headers and column headers indicate which source and target data were used in the training. In particular, “word” and “char” are character level data without BPE segmentation, while “bpe” (character level), “ideograph”, and “stroke” (sub- character level) are data with BPE segmentation. The scores in parentheses indicate the models that had a shared vocabulary, whenever applicable. The italic numbers represent the two Pinyin baselines used for comparative purposes, namely the “WdPyT” model, which uses Pinyin words with tones as the source data, and the “factored- NMT” model, which uses Pinyin characters as factors (Du and Way, 2017). Note that these two baselines can only have Chinese data on the encoder side. The superscripts indicate that a score is significantly better than the best baseline result. ∗
5 Discussions worked better for Chinese and strokes worked better for Japanese. This difference might be be- 5.1 Translation Examples cause of the differences in the writing systems. In Table 6 shows some of the translation exam- addition to Kanji (Chinese characters), Japanese ples. There is a rare proper noun “松下電器 uses Hiragana and Katakana, which are stan- (Matsushita Electric)” (OOV) in the source sen- dalone alphabets. tence. The word baseline model cannot decode Moreover, as described in Section 4, stroke this; therefore, an
23 Model Sentence 松下 電器 グループ で は , 経営 理念 の 基本 と し て 1991 年 に 「 環境 Source 宣言 」 を 制定 し た 。 Reference 作为 经营 理念 , 松下电气集团 于 1991年 制定 了 《 环境 宣言 》 。 Baseline 在
Table 6: Translation examples of Japanese-Chinese NMT systems. Note that “松下电器” as a proper noun, could be handled properly in sub-character based translation systems. words. As a result, during training, the neurons Many questions remain to be answered in fu- responsible for high-frequency words might be ture work. The first question involves the relative updated many times, while the neurons respon- benefits of ideograph data and stroke data, and the sible for low-frequency words might be updated effects of shared vocabularies. We have not yet only a very limited number of times. This will explained why there are considerable differences potentially harm translation performance for low- in performance. In particular, for NMT models frequency words. in which both sides have stroke data, why does However, this problem can be alleviated by ap- performance drop dramatically when shared vo- plying sub-character units. Because ideographs cabularies are applied? We discussed the possi- and strokes are repeatedly shared by different ble reasons for this phenomenon in Section 5.2; characters, no items occur with very low frequen- however, further investigation is needed. cies. More instances can be found in the train- Another important issue is as follows: when ing data, even for the least frequent sub-character characters are transformed into ideographs and items. Therefore, the translation performance for strokes, no structural information is considered. low-frequency items could be much better. This causes repetitions in data, and we must add tags at the end of each sequence to differentiate 6 Conclusions and Future Work them. A better way to solve this problem would This study was the first attempt to use sub- be to have structural information directly encoded character units in NMT models. Our results in the data. not only confirmed the positive effects of using ideograph and stroke sequences in NMT tasks, Acknowledgments but also indicated that different logographic lan- This work was partially supported by JSPS Grant- guages actually preferred different sub-character in-Aid for Young Scientists (B) Grant Number granularities (namely, ideograph for Chinese and JP16K16117. stroke for Japanese). Finally, this paper presented a simple method for extending the available cor- pus from the character level to the sub-character References level. During this process, we maintained a one- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua to-one relationship between the original charac- Bengio. 2015. Neural machine translation by ters and transformed sub-character sequences. As jointly learning to align and translate. In ICLR a result, this simple and straightforward method 2015. achieved consistently better results for NMT sys- Shaosheng Cao, Wei Lu, Jun Zhou, and Xiaolong Li. tems used to translate logographic languages, and 2017. Investigating stroke-level information for could be easily applied to similar scenarios. learning Chinese word embeddings. In ISWC-16.
24 Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- Yota Toyama, Makoto Miwa, and Yutaka Sasaki. cehre, Dzmitry Bahdanau, Fethi Bougares, Hol- 2017. Utilizing visual forms of Japanese characters ger Schwenk, and Yoshua Bengio. 2014. Learn- for neural review classification. In IJCNLP 2017, ing phrase representations using RNN encoder–de- pages 378–382. coder for statistical machine translation. In EMNLP 2014, pages 1724–1734. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Jinhua Du and Andy Way. 2017. Pinyin as subword Kaiser, and Illia Polosukhin. 2017. Attention is all unit for Chinese-sourced neural machine transla- you need. In NIPS 2017, pages 6000–6010. tion. In Irish Conference on Artificial Intelligence and Cognitive Science, page 11. Yining Wang, Long Zhou, Jiajun Zhang, and Chengqing Zong. 2017. Word, subword or char- Sepp Hochreiter and Jürgen Schmidhuber. 1997. acter? an empirical study of granularity in Chinese- Long short-term memory. Neural computation, English NMT. In China Workshop on Machine 9(8):1735–1780. Translation, pages 30–42. Springer. Philipp Koehn. 2004. Statistical significance tests for Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. machine translation evaluation. In EMNLP 2014. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Shaohui Kuang and Lifeng Han. 2018. Apply Chinese Macherey, Jeff Klingner, Apurva Shah, Melvin radicals into neural machine translation: Deeper Johnson, Xiaobing Liu, ukasz Kaiser, Stephan than character level. In 30Th European Summer Gouws, Yoshikiyo Kato, Taku Kudo, Hideto School In Logic, Language And Information. Kazawa, Keith Stevens, George Kurian, Nishant Taku Kudo. 2018. Subword regularization: Improv- Patil, Wei Wang, Cliff Young, Jason Smith, Ja- ing neural network translation models with multiple son Riesa, Alex Rudnick, Oriol Vinyals, Greg Cor- subword candidates. In ACL 2018. rado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridg- Frederick Liu, Han Lu, Chieh Lo, and Graham Neu- ing the gap between human and machine transla- big. 2017. Learning character-level composition- tion. arXiv:1609.08144 [cs]. ality with visual features. In ACL 2017, pages 2059–2068. Appendix A Strokes in CNS11643 Charset Thang Luong, Hieu Pham, and Christopher D Man- ning. 2015. Effective approaches to attention-based Type Strokes neural machine translation. In EMNLP 2015, pages 1412–1421. Horizontal Vertical Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchi- Left-falling moto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. ASPEC: Right-falling Asian scientific paper excerpt corpus. In LREC-9, pages 2204–2208. Break Graham Neubig. 2013. Travatar: A forest-to-string machine translation engine based on tree transduc- ers. In ACL 2013: System Demonstrations, pages 91–96. Haiyun Peng, Erik Cambria, and Xiaomei Zou. 2017. Radical-based hierarchical embeddings for Chinese sentiment analysis at sentence level. In FLAIRS- 30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL 2016, pages 1715–1725. Xinlei Shi, Junjie Zhai, Xudong Yang, Zehua Xie, and Chao Liu. 2015. Radical embedding: Delving deeper to Chinese radicals. In ACL-IJCNLP 2015, pages 594–598. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural net- works. In NIPS, pages 3104–3112.
25