Convergence of Translation Memory and Statistical Machine Translation
Philipp Koehn Jean Senellart University of Edinburgh Systran 10 Crichton Street La Grande Arche Edinburgh, EH8 9AB 1, Parvis de la Defense´ Scotland, United Kingdom 92044 Paris, France [email protected] [email protected]
Abstract tems (Koehn, 2010) are built by fully automati- cally analyzing translated text and learning the rules. We present two methods that merge ideas SMT has been embraced by the academic and com- from statistical machine translation (SMT) mercial research communities as the new dominant and translation memories (TM). We use a TM paradigm in machine translation. Almost all re- to retrieve matches for source segments, and cently published papers on machine translation are replace the mismatched parts with instructions to an SMT system to fill in the gap. We published on new SMT techniques. The method- show that for fuzzy matches of over 70%, one ology has left the research labs and become the method outperforms both SMT and TM base- basis of successful companies such as Language lines. Weaver and the highly visible Google and Microsoft web translation services. Even traditional rule-based companies such as Systran have embraced statisti- 1 Introduction cal methods and integrated them into their systems Two technological advances in the field of au- (Dugast et al., 2007). tomated language translation, translation memory The two technologies have not touched much in (TM) and statistical machine translation (SMT), the past not only because of the different devel- have seen vast progress over the last decades, but opment communities (software suppliers to transla- they have been developed very much in isolation. tion agencies vs. mostly academic research labs). The reason for this is that different communities Another factor is that TM and SMT have recently played a role in each technology’s development. addressed different translation challenges. While TMs are a tool for human translators. Since many TM have addressed the need of translation agencies translation needs are highly repetitive (translation of to produce high-quality translations of often repet- updated product manuals, or several drafts of leg- itive material, SMT has set itself the challenge of islation), being able to find existing translations of open domain translations such as news stories and is segments of the source language text, alleviates the mostly satisfied with translation quality that is good need to carry out redundant translation. In addition, enough for gisting, i.e., transmitting the meaning of finding close matches (so-called fuzzy matches), the source text to a target language speaker (consider may dramatically reduce the translation workload. web page translation or information gathering by in- Various commercial vendors offer TM software and telligence agencies). the technology is in wide use by translation agen- Currently, SMT receives increasing attention by cies. translation agencies, who would like to employ the Instead of building machine translation systems technology in a workflow of first automatic transla- by manually writing translation rules, SMT sys- tion and then human post-editing. One possible user
Ventsislav Zhechev (ed.): Proceedings of the Second Joint EM+/CNGL Workshop “Bringing MT to the User: Research on Integrating MT in the Translation Industry” (JEC ’10), pp. 21–31. Denver, CO, 4 November 2010. 21 JEC 2010 “Bringing MT to the User” Denver, CO scenario is to offer a human translator a fuzzy match 2.1 Example from a TM, or an SMT translation, or both. What To illustrate the process (see also Figure 1), let us to show may be decided by an automatic classifier first go over one example. Let us say that the fol- (Simard and Isabelle, 2009; Soricut and Echihabi, lowing source segment needs translation: 2010; He et al., 2010) or may be based on fuzzy The second paragraph of Article 21 is deleted . match score or SMT confidence measures (Specia et al., 2009). The TM does not contain this source segment, but it contains something very similar: In this paper, we argue that the two technologies have much more in common that commonly per- The second paragraph of Article 5 is deleted . ceived. We present a method that integrates TM In the TM, this segment is translated as: and SMT and that outperforms either technology A` l’ article 5 , le texte du deuxieme´ alinea´ est supprime´ . for fuzzy matches of more than 80%. In a second method, we encode TM matches as very large trans- The mismatch between our source segment and lation rules and outperform all other methods for the TM source segment is the word 21 or 5, respec- fuzzy match ranges over 70%. tively. By letting the SMT system translate the true source word 21, but otherwise trusting the target side of the TM match, we construct the following (sim- 2 XML Method plified) XML frame:
22 November 4th, 2010 Philipp Koehn and Jean Senellart number of deletions, insertions, and substitutions a specified translation, and instead the source word needed to edit the source segment to the TM seg- is inserted in their place. ment. A mismatch may consist of a sequence of multiple Our implementation of the fuzzy match score words in source, TM source, or TM target. We treat uses word-based string edit distance, and uses let- such a block the same way we treat single words: ter string edit distance as a tie breaker. We define they are removed as a block from the TM target and the fuzzy match score as: the source block is inserted. There may be multiple non-neighboring mis- edit-distance(source, tm-source) FMS = 1 − matched sequences. We treat each separately and max(|source|, |tm-source|) perform the subtraction process for each.
For our example, the word-based string edit distance 2.5 Special Cases is 89% (one substitution for 9 words), and the letter- There are a number of special cases (also illustrated based string edit distance is 95% (one substitution in Figure 2): and one deletion for 37 letters, not counting spaces). Pure insertion: If the source segment has an in- 2.3 Word Alignment of TM serted block that is not present in the TM source The mismatch between the source segment and the segment, we add these source words to the TM source segment is easy to detect. As with our XML frame. The location of the words is after fuzzy match metric, we compute the string edit dis- the TM target word aligned to the TM source tance between the two, which detects the inserted, word just prior to the insertion point. deleted and substituted words. Pure deletion: If the TM source segment has addi- A harder problem is to determine which target tar- tional words, then these are removed from the get words are affected by the mismatch. For this, specified translation in the XML frame. we need a word alignment between the source words and the target words in the TM segment. Unaligned mismatched words: If the mismatched Word alignment is a standard problem in SMT, source words are unaligned, then we find the and many algorithms have been proposed and im- closest aligned previous source word and spec- plemented. Most commonly used is the implemen- ify them after this target position. tation of the IBM Models (Brown et al., 1993) in the Non-contiguous alignments: If a TM source seg- toolkit GIZA++ (Och et al., 1999), combined with ment has non-contiguous alignments to the TM symmetrization heuristics. This toolkit is also used target segment, we first try to resolve this by by the Moses SMT training pipeline. So, if we build splitting up blocks. If this does not help, then an SMT system on the TM data, then the word align- the mismatched source words are placed at the ment falls out as a by-product. position of the first aligned TM target block. 2.4 Construction of XML Frame The basic principles of the heuristic are: all mis- We now have all the necessary ingredients to con- matched source words are inserted; all TM target struct the XML frame that we will pass to the SMT words aligned to mismatched TM source words are decoder. See Figure 1 for an illustration of our ex- removed; if the alignment to the TM target words ample. Given the string edits and the word align- fails, then go to the previous TM source word and ment, we can track the mismatched source word to follow its alignment. the TM target word. The detailed pseudo-code of the algorithm is We construct the XML frame by subtraction. All given in Figure 3. of the TM target segment is passed as a specified 3 Experiments translation to the SMT decoder, except for the sub- traction of the mismatch. The TM target word We carried out experiments using two data sets: an aligned to the mismatched source word is not part of English–Italian commercial product manual corpus
23 JEC 2010 “Bringing MT to the User” Denver, CO
Source The second paragraph of Article 21 is deleted . String Edit TM Source The second paragraph of Article 5 is deleted .
Word Alignment
TM Target À l' article 5 , le texte du deuxième alinéa est supprimé .
XML Frame <À l' article> 21 <, le texte du deuxième alinéa est supprimé .>
Figure 1: Construction of the XML frame: The mismatched source word is tracked to the TM target word.
Insertion Deletion Unaligned Multiple Alignments
Source the big fish the fish the green fish joe will eat String Edit TM Source the fish the big fish the big fish joe does not eat
Word Alignment
TM Target les poissons les gros poissons les poissons joe ne mange pas
XML Frame
Figure 2: Special Cases: The basic principles are: all mismatched source words are inserted, all TM target words aligned to mismatched TM source words are removed, if the alignment to the target words fails, go to previous word and follow its alignment.
Acquis Corpus Test Product corpus is more representative of the type of segments 1,165,867 4,107 data used in TM systems. It is much smaller (around English words 24,069,452 129,261 a million words), with shorter segments (average 12 French words 25,533,259 135,224 words per segments). We are especially interested in the performance of Product Corpus Test the methods for sentences for which we find highly segments 83,461 2,000 scoring fuzzy matches, since we do not expect fuzzy English words 1,038,762 24,643 matches to be very useful otherwise. See Table 2 French words 1,110,284 26,248 for statistics on the subsets based on fuzzy match ranges. The sentences with 100% fuzzy match are Table 1: Statistics of the corpus used in experiments much shorter, but otherwise there is no strong cor- relation between fuzzy match score and sentence (Product) and the English–French part of the pub- length. Note that we do not break out a 100% fuzzy licly available JRC-Acquis corpus1 (Acquis), for match subset for the Product corpus, since the test which we use the same test set as Koehn et al. data is drawn from a TM, so we do not expect to (2009). See Table 1 for basic corpus statistics. find valid 100% matches (since we exclude the test The Acquis corpus is a collection of laws and reg- data from the TM used in training and testing). ulations that apply to all member countries of the European Union. It has more repetitive content than 3.1 SMT Training the parallel corpora that are more commonly used in We train SMT systems using the Moses toolkit. We machine translation research. Still, the commercial use a standard setup with lexicalized reordering and 1http://wt.jrc.it/lt/Acquis/ (Steinberger et al., 2006) 5-gram language model.
24 November 4th, 2010 Philipp Koehn and Jean Senellart
Input: match path M, 4: matched-target(t) = false; input i, 5: end for TM source s, 6: end for TM target t, 7: // find insertion position alignment a 8: inst = min( {t ∈ a([starts; ends])} ) Output: xml frame 9: while inst undefined do 1: global M, i, s, t, a, insertion 10: inst = min( {t ∈ a(--starts)} ) 2: posi = 0; 11: end while 3: poss = 0; 12: if inst undefined then 4: matching = false; 13: inst = -1 // sentence start 5: matched-target([0; |t|]) = true // matched words 14: inst = |i| if endi == |i| // end 6: for all edit e ∈ M do 15: end if 7: if matching AND e != ’match’ then 16: // locate inserted words 8: // beginning of mismatch 17: insertion[inst] = i[starti;endi] 9: start = pos ; i i function construct-xml: 10: starts = poss; 11: matching = false; 1: xml = ”” 12: else if !matching AND e==’match’ then 2: included = false − 13: // end of mismatch 3: startt = 1 4: for all target positions t ∈ [0; |t|[ do 14: mismatch(starti,posi-1, starts,poss-1) 15: matching = true; 5: if !included AND matched-target(t) then − 16: end if 6: startt = 1 7: included = true 17: posi++ unless edit == ’deletion’ 8: else if included AND (!matched-target(t) OR 18: poss++ unless edit == ’insertion’ 19: end for insertion[t]) then ≥ 20: if !matching then 9: if startt 0 then 10: xml += ”
Figure 3: Algorithm to construct the XML frame: The main function first detects all mismatched sequences in the string edit distance path. Mismatched sequences (function mismatch) trigger de-activation of aligned target words (variable matched-target) and an insertion record for mismatched input words (variable insertion). During the XML construction, matched target words are included as markup (lines 10–12 in function construct-xml), in addition to inserted input words (line 16, ibid.).
25 JEC 2010 “Bringing MT to the User” Denver, CO
Acquis Sentences Words W/S sentence which have fuzzy matches of at least 80%, 100% 1395 14,559 10.4 SMT is best below this threshold. 90-99% 433 12,775 29.5 80-89% 154 5,347 34.7 3.3 Multiple Fuzzy Matches 70-79% 250 6,767 27.1 When we described our method in Section 2, we skipped over some choices we have to make when Product Sentences Words W/S constructing an XML frame: 95-99% 230 3,006 13.1 90-94% 225 2,968 13.2 1. If the fuzzy match score to the source is the 85-89% 177 2,000 11.3 same for several TM source segments, which 80-84% 185 1,950 10.5 TM match should be picked? 75-79% 152 1,350 8.9 2. If the source has multiple targets, which target 70-74% 98 987 10.1 should be picked? Note that this happens typ- ically in a parallel corpus such as the Acquis Table 2: Composition of the test subsets based on fuzzy data set, less so in a real TM. match scores (letter-based string edit distance). 3. If we run the decoder with an XML frame what translation should be picked?
To increase word-level matches between input In the basic results, we answered these questions and TM, we use an aggressive tokenization scheme as follows: that separates out hyphens as separate tokens and splits up letter/number combinations (e.g., XR300 1. randomly chosen from best matches according becomes XR ∼@∼ 300). to letter string edit distance To improve word alignment quality of the Product 2. most frequent system, we add training data from Europarl when 3. highest model score running GIZA++. We only use the data to obtain better word alignments, we do not use it in the trans- By choosing the most frequent target translation lation model or as TM. of a TM source segment, we do better than choosing When using the XML frames, we specify its use one at random (for 100% matches the BLEU score to the decoder as exclusive, i.e., the decoder may not is 79.9 vs. 78.7 on Acquis). override it with its own translations. But would we also do better if we did not chose a TM match at random, but find some other criterion 3.2 Basic Method at a later stage to pick a translation. Maybe the SMT system offers scoring functions that would aid a de- We compare our method using XML frames against cision? Experience in SMT research shows that an two baselines: (a) the basic SMT system, and (b) us- integrated approach with a global scoring function ing TM matches unmodified. The latter is only rea- almost always outperforms a pipelined approach. sonable, if we find sufficiently good fuzzy matches. To keep with our three questions, we would like We expect the methods to perform differently in to now attempt the following answers: different match ranges. If we find a very close match, we would expect the TM to produce a very 1. all matches with best word string edit distance good translation, and hopefully the XML method an 2. all even better one. If no close match exists, we expect 3. language model score / full decoder score the baseline SMT system to outperform the other methods. Not only do we now use all source matches and all This expectation is confirmed by our experiments. their target translations and a 10-best list of transla- In Figure 4, we plot the BLEU scores for subsets of tions. We pick the best translation based on (a) the the test sets where we have a find a fuzzy match of language model score and (b) the full decoder score. at least 70%. Our XML method performs best for The full decoder score also contains the word count
26 November 4th, 2010 Philipp Koehn and Jean Senellart
Acquis
Product
Figure 4: Basic Results: BLEU scores for different fuzzy match ranges. Our XML method performs best for sentence which have fuzzy matches of at least 80%, SMT is best below this threshold.
27 JEC 2010 “Bringing MT to the User” Denver, CO
Model 70s 80s 90s 100 Hierarchical phrase-based rules are constructed SMT 59.5 62.0 66.6 75.8 by removing a smaller phrase pairs from a larger XML baseline 58.5 62.7 71.6 79.9 phrase pairs. For instance, by taking the phrase pair: XML + LM score 59.0 63.6 71.2 77.7 ( the big fish ; les gros poissons ) XML + SMT score 59.6 63.3 72.1 66.7 and removing the phrase pair Table 3: Using SMT scores to decide among multiple XML frames and translations ( big ; gros ) we create the rule feature. Also, if there are different XML frames, dif- ( the X fish ; les X poissons ) ferent words are translated, hence we have different The symbol X is called a non-terminal, since the translation model scores. translation rule is viewed as a synchronous context- Table 3 shows the results with these variations. free grammar rule. In essence it is a place-holder for Except for the 100% matches, one of the multiple recursively nested sub-phrases. XML frame methods outperforms the basic XML Hierarchical rules require a different decoding method and the SMT system, but the differences are algorithm that is typically drawn from syntactic not very large. parsing methods, but otherwise use a very similar The decoder could also use additional scoring training, tuning, and testing pipeline as traditional functions in its model, such as the fuzzy match phrase-based models (Hoang et al., 2009). scores. The XML frames could also be scored with methods used in the SMT system, such as lexical 4.2 TM Matches as Very Large Rules smoothing. But before we go further down the road, How does that relate to our XML frames? Recall the let us shortly reflect into which direction we are XML frame that we constructed in Section 2.1: heading.
28 November 4th, 2010 Philipp Koehn and Jean Senellart Acquis
Figure 5: TM matches as very large rules (VLR): Encoding TM match frames as very large hierarchical grammar rules (VLR) outperforms all previous methods.
4.3 Results tem on the 70-79% match range, where our XML We train a hierarchical phrase-based model with method scores worse. Moses, which has very similar performance as the 5 User Interaction phrase-based model used in the previous experi- ments. Instead of using XML frames, we create very The translations generated by our methods may be large rules (VLR) that we store in a second trans- presented to a human translator the same way cur- lation table. Thus, the VLR rule table has its own rently TM matches are presented. However, we may feature functions. We also do this for the tuning decorate them additionally for added benefit. Since data, hence optimizing the weighting between the we know which parts of the translation were gener- two rule tables. ated by the SMT system, we can highlight it to alert This approach uses the translator to potential flaws. We did not carry out a user study to demonstrate 1. all TM matches with best word string edit dis- how much productivity increase can be expected tance along with the improved translation performance 2. all target translations of each TM match that we measured with the BLEU score. However, 3. integrated scoring of VLR rules and basic results suggest that these improvements correspond translation rules to a 5–10% increase in fuzzy match score. On the Product and Acquis corpus, the BLEU In other words, this is a fully integrated approach score for our method on each of the 70-75% and 75- that uses all available choices at every stage. 80% subsets is as good as a the TM score for the Results are presented in Figure 5. The VLR ranges with 10% higher fuzzy match. For subsets approach outperms all other methods on all fuzzy with higher fuzzy match scores, results are almost match ranges. It even outperforms the SMT sys- as good. We suspect that the BLEU scores actually
29 JEC 2010 “Bringing MT to the User” Denver, CO underestimate these improvements, since they tend Chiang, David. 2007. Hierarchical Phrase- to be too literal and do not sufficiently appreciate al- Based Translation. Computational Linguistics, ternate acceptable translations. 33(2):201–228. We would like to evaluate these user issues in Dugast, Lo¨ıc, Jean Senellart, and Philipp Koehn. forthcoming work. 2007. Statistical Post-Editing on SYSTRAN’s 6 Conclusion Rule-Based Translation System. In Proceedings of the Second Workshop on Statistical Machine We presented two methods that convergence ideas Translation, pages 220–223, Prague, Czech Re- from TM and SMT. The first method constructed public, June. Association for Computational Lin- XML frames that provided specified translations for guistics. part of the segments (the matched part) and rely in He, Yifan, Yanjun Ma, Andy Way, and Josef van the SMT decoder to fill in the remainder (the un- Genabith. 2010. Integrating N-best SMT Out- matched part). We showed experimentally on two puts into a TM System. In Coling 2010: Posters, data sets that we outperform those technologies for pages 374–382, Beijing, China, August. Coling relatively high fuzzy match ranges (80-99%). Either 2010 Organizing Committee. XML or SMT always outperform TM matches. The second method encodes these XML frames as Hoang, Hieu, Philipp Koehn, and Adam Lopez. very large hierarchical phrase rules, and uses these 2009. A unified framework for phrase-based, hi- in a secondary rule table of a hierarchical phrase- erarchical, and syntax-based statistical machine based model. This method outperforms SMT on all translation. In Proceedings of the Interna- tested fuzzy match ranges (≥70%). tional Workshop on Spoken Language Translation These results suggest that fuzzy TM matches can (IWSLT), pages 152–159, December. be seen as just another way to generate translation Koehn, Philipp, Hieu Hoang, Alexandra Birch, rules for an SMT system, and that there is little value Chris Callison-Burch, Marcello Federico, Nicola is using TM without SMT. Bertoldi, Brooke Cowan, Wade Shen, Chris- We would like to evaluate the utility of our meth- tine Moran, Richard Zens, Christopher J. Dyer, ods in user studies in future work. Ondrejˇ Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit Acknowledgement for Statistical Machine Translation. In Proceed- This work was partly supported by the EuroMatrix- ings of the 45th Annual Meeting of the Associ- Plus project funded by the European Commission ation for Computational Linguistics Companion (7th Framework Programme). Volume Proceedings of the Demo and Poster Ses- sions, pages 177–180, Prague, Czech Republic, References June. Association for Computational Linguistics. Bic¸ici, Ergun and Marc Dymetman. 2008. Dy- Koehn, Philipp, Alexandra Birch, and Ralf Stein- namic Translation Memory: Using Statistical Ma- berger. 2009. 462 Machine Translation Systems chine Translation to Improve Translation Mem- for Europe. In Proceedings of the Twelfth Ma- ory. In Gelbukh, Alexander F., editor, Proceed- chine Translation Summit (MT Summit XII). In- ings of the 9th Internation Conference on Intelli- ternational Association for Machine Translation. gent Text Processing and Computational Linguis- Koehn, Philipp. 2010. Statistical Machine Transla- tics (CICLing), volume 4919 of Lecture Notes in tion. Cambridge University Press. Computer Science, pages 454–465. Springer Ver- Lopez, Adam. 2007. Hierarchical Phrase-Based lag. Translation with Suffix Arrays. In Proceedings Brown, Peter F., Stephen A. Della-Pietra, Vincent J. of the 2007 Joint Conference on Empirical Meth- Della-Pietra, and Robert L. Mercer. 1993. The ods in Natural Language Processing and Com- Mathematics of Statistical Machine Translation. putational Natural Language Learning (EMNLP- Computational Linguistics, 19(2):263–313. CoNLL), pages 976–985.
30 November 4th, 2010 Philipp Koehn and Jean Senellart
Och, Franz Josef, Christoph Tillmann, and Her- mann Ney. 1999. Improved Alignment Models for Statistical Machine Translation. In Proceed- ings of the Joint Conference of Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP-VLC), pages 20–28. Simard, Michel and Pierre Isabelle. 2009. Phrase-based Machine Translation in a Computer- assisted Translation Environment. In Proceedings of the Twelfth Machine Translation Summit (MT Summit XII). International Association for Ma- chine Translation. Smith, James and Stephen Clark. 2009. EBMT for SMT: A New EBMT-SMT Hybrid. In Forcada, Mikel L. and Andy Way, editors, Proceedings of the 3rd International Workshop on Example- Based Machine Translation, pages 3–10. Soricut, Radu and Abdessamad Echihabi. 2010. TrustRank: Inducing Trust in Automatic Trans- lations via Ranking. In Proceedings of the 48th Annual Meeting of the Association for Computa- tional Linguistics, pages 612–621, Uppsala, Swe- den, July. Association for Computational Linguis- tics. Specia, Lucia, Marco Turqui, Zhuoran Wang, John Shawe-Taylor, and Craig Saunders. 2009. Im- proving the Confidence of Machine Translation Quality Estimates. In Proceedings of the Twelfth Machine Translation Summit (MT Summit XII). International Association for Machine Transla- tion. Steinberger, Ralf, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaz Erjavec, Dan Tufis, and Daniel Varga. 2006. The JRC-Acquis: A multilin- gual aligned parallel corpus with 20+ languages. In 5th Edition of the International Conference on Language Resources and Evaluation (LREC ’06), pages 2142–2147. Zhechev, Ventsislav and Josef van Genabith. 2010. Seeding Statistical Machine Translation with Translation Memory Output through Tree-Based Structural Alignment. In Proceedings of the 4th Workshop on Syntax and Structure in Statistical Translation, pages 43–51, Beijing, China, Au- gust. Coling 2010 Organizing Committee.
31