Linguistic Evaluation of Support Verb Constructions by Openlogos and Google Translate

Linguistic Evaluation of Support Verb Constructions by OpenLogos and Google Translate

A. Barreiro1, J. Monti2, B. Orliac3, S. Preuß4, K. Arrieta3, W. Ling1,5, F. Batista1,6, I. Trancoso1

1 INESC-ID, Rua Alves Redol 9, 1000-029, Lisboa, Portugal 2 University of Sassari, Italy 3 Logos Institute, USA 4 University of Saarlandes, Germany 5 Carnegie Mellon University - Instituto Superior Tecnico,´ USA-Portugal 6 ISCTE-IUL - Instituto Universitario´ de Lisboa, Portugal [email protected], [email protected]

Abstract This paper presents a systematic human evaluation of translations of English support verb constructions produced by a rule-based machine translation (RBMT) system (OpenLogos) and a statistical machine translation (SMT) system (Google Translate) for five languages: French, German, Italian, Portuguese and Spanish. We classify support verb constructions by means of their syntactic structure and semantic behavior and present a qualitative analysis of their translation errors. The study aims to verify how machine translation (MT) systems translate fine-grained linguistic phenomena, and how well-equipped they are to produce high-quality translation. Another goal of the linguistically motivated quality analysis of SVC raw output is to reinforce the need for better system hybridization, which leverages the strengths of RBMT to the benefit of SMT, especially in improving the translation of multiword units. Taking multiword units into account, we propose an effective method to achieve MT hybridization based on the integration of semantico-syntactic knowledge into SMT.

Keywords: Machine Translation, MT Evaluation, Support Verb Constructions, Multiword Units, Semantico-Syntactic Knowledge, MT Hybridization.

1. Introduction evaluation efforts involving MT systems of different nature was our main motivation for evaluating the performance of MT researchers and developers have a common goal of cre- RBMT and SMT when dealing with a very specific linguis- ating robust translation systems that can produce high qual- tic phenomenon. ity translation. However, MT is a work-in-progress. Af- This paper describes an evaluation exercise that consisted ter six decades of research on MT models, tools and lin- of the linguistic analysis of the translations of 100 sup- guistic resources, translations produced by widely used MT port verb constructions (SVC), such as make a presenta- systems show unfortunate errors which require significant tion. The sentences in our SVC corpus were randomly post-editing effort if the result is to be used by professional collected from the news and the Internet and hand-picked translators or for purposes other than gisting. A notice- for the task. The corpus was translated into 5 languages: able trend is that of linguistically enhancing SMT mod- French, German, Italian, Portuguese and Spanish by the els, to produce systems that combine linguistic resources OpenLogos (OL) and the Google Translate (GT) MT sys- and analysis with statistical techniques. While this is a tems. Five MT expert linguists, native speakers of the target promising direction, making significant advances towards languages, evaluated the translations of the SVC consider- hybrid MT (HMT) systems requires a deep understanding ing their meaning in context, and classified the translation of different approaches, their weaknesses and strengths. errors based on a SVC typology, as illustrated in Table 1. Bringing different approaches together, contrasting them The remaining of the paper is organized as follows: Section and measuring which modules need improvement seems to 2. presents the state-of-the-art HMT. Section 3. describes be an effective method for achieving the desired end result. the main characteristics of the OL and the GT models. Sec- In pursuing the creation of an HMT model and improving tion 4. describes the corpus, datasets, and the evaluation existing translation technology, a systematic fine-grained task. Section 5. discusses the linguistic challenges found linguistic analysis of the performance of individual models in the translations for the 5 language pairs contemplated appears to us as an important first step. To our knowledge, in this research. Section 7. proposes a method to achieve no major effort has been made to combine the strengths MT hybridization based on the integration of semantico- of different approaches with the purpose of overcoming syntactic knowledge into SMT. Finally, Section 8. presents known weaknesses on the basis of a joint linguistic eval- the main conclusions and points to future work, namely the uation of those weaknesses, as used in this paper. In ad- inclusion of linguistic expertise in MT evaluation. dition, state-of-the-art quality metrics and estimation have been targeting human-factors tasks such as measuring post- editing time and effort in terms of keystrokes, etc., not di- 2. Hybrid Machine Translation agnosing fine-grained linguistic errors to improve syntac- A noticeable trend in current MT research is that of HMT tic structure and meaning. The lack of such qualitative models that combine linguistic knowledge with statistical Nominal Support Verb Construction (NSVC) make a presentation Non-contiguous nominal (NON-CONT NSVC) have [ADV+ADJ-particularly good] links Prepositional nominal (PREPNSVC) give an illustration of Non-contiguous prepositional nominal (NON-CONT PREPNSVC) be the [ADJ-immediate] cause of Idiomatic nominal (IDIOM NSVC) set in motion, place at risk, go on strike Idiomatic prepositional nominal (IDIOM PREPNSVC) earn an income of Non-contiguous idiomatic nominal (NON-CONT IDIOM NSVC) hold [NP-the option] in place, be of [ADJ-practical] value Non-contiguous idiomatic prepositional nominal (NON-CONT IDIOM PREPNSVC) give [PRO-us] a [bird’s-eye] view of, be [ADV-clearly] at odds with, open talks [May 14] with Adjectival Support Verb Construction (ADJSVC) be meaningful Non-contiguous adjectival (NON-CONT ADJSVC) be [ADV-extremely] selective Prepositional adjectival (PREPADJSVC) be known as; be involved in Non-contiguous prepositional adjectival (NON-CONT PREPADJSVC) fall [ADV-so far] short of

Table 1: Major categories of support verb construction in our corpus techniques. HMT systems attempt to combine RBMT sys- the coverage of a RBMT system. A similar method us- tems, such as the work presented by (Scott, 2003), with ing example-based MT for the same end has been proposed data-driven MT systems, such as phrase-based SMT pro- in (Sanchez-Mart´ ´ınez et al., 2009). Finally, the translation posed by (Koehn et al., 2007). System combination often quality of RBMT output can also be improved by using leads to improvements in the final translation quality, as statistical post-editing methods (cf. (Simard et al., 2007; different systems tend to address different translation chal- Elming, 2006; Dugast et al., 2007; Terumasa, 2007)). The lenges. opposite has also been proposed, where RBMT systems In general, SMT methods learn the generalizations of the are used to enhance data-driven approaches. (Shirai et al., translation process using parallel corpora, which are sets of 1997) use an example-based MT system (Brown, 1996) to translated sentences pairs. In language pairs where there create an initial translation template, and use a RBMT sys- are abundant amounts of parallel corpora, such as English- tem to translate individual words and phrases according to Mandarin, these models tend to perform better than RBMT this template. To the best of our knowledge, it is still not ob- approaches. However, when parallel data is scarce, such vious which HMT approach will be the most efficient one as for the Spanish-Basque language pair, it is less likely and will lead to higher quality translation in the long run. that the SMT methods will have enough data to properly learn such generalizations (Labaka et al., 2007). Morpho- 3. The OpenLogos and the Google Translate logically rich languages require more data to learn accu- Models rate translations, since there are more word types due to OL is an open source copy of the commercial Logos Sys- different word forms that can be generated for the same tem (Scott, 2003; Barreiro et al., 2011). The system ad- stem. While more complex SMT models for morphologi- dresses morphology, syntax, and semantics, it has robust cally rich languages have been proposed (Chahuneau et al., parsers, sets of semantico-syntactic rules, terminology sets 2013), RBMT systems where the morphology of the lan- and tools. Unlike most other RBMT systems, the OL model guage is encoded manually, represent a strong alternative is closer in spirit to the SMT approach in that both method- for the translation of such languages. ologies are pattern-based, with the additional advantage of Many methods to combine RBMT with SMT have been including semantic understanding. The use of an inter- proposed. One straightforward way to create an HMT sys- mediate language (SAL) to encode linguistic information tem is to combine the translations of the same text by two and process text contributes to OL high quality translation different systems (Heafield and Lavie, 2011; Eisele et al., and lessens one of the main problems in SMT, viz., the 2008). Other approaches attempt to use data-driven tech- sparseness in linguistic examples. OL linguistic knowledge niques to improve RBMT systems. For example, (Eisele et databases have not been developed for over 10 years. al., 2008) use phrase pair extraction in phrase-based MT to GT is one of the most widely used online MT systems. It is extract phrasal translations entries that are used to improve an SMT system that benefits from the huge amount of parallel data that Google collects from the web. In March 2014, Lang. pair System OK ERR Agreem Other it was set to account for 80 language pairs. As is typical GT 64 32 4 - EN-FR of SMT, the translation quality is highly dependent on the OL 51 48 1 - language pair, producing much better results for close lan- GT 37 46 3 14 EN-GE guage pairs (e.g. Portuguese and Spanish) and languages OL 60 33 1 6 for which large amounts of parallel data are available. GT GT 61 31 - 8 EN-IT is a closed system, however, no knowledge of semantic un- OL 43 52 - 5 derstanding is known to exist in GT. GT 68 27 5 - EN-PT OL 41 58 1 - 4. Corpus, datasets, and evaluation task GT 51 41 6 2 EN-ES The corpus used in this research task contains 100 English OL 25 70 3 2 SVC that appear in sentences collected from the news and the Internet. A SVC is a multiword or complex predi- Table 2: Performance of the OL and GT MT systems for cate consisting of a semantically weak verb (the support 100 SVC. verb), and a predicate noun (most commonly), a predicate adjective, or a predicate adverb (Barreiro, 2009). For ex- SVC under evaluation. The linguists evaluating the transla- ample, make a presentation is a SVC made of the support tion quality were also instructed to provide a more compre- verb make and the predicate noun presentation, and make hensive qualitative evaluation of mistranslations according it simple is a SVC made of the support verb make and to the different types of SVC (Section 6.). None of the sys- the predicate adjective simple. We have selected SVC as tems was specifically trained for the task, as the texts were the object of our evaluation for two main reasons. Firstly, not domain specific. SVC have been studied sistematically and in depth within the Lexicon-Grammar Theory (Gross, 1987) and have been 5. Quantitative Results recognized and processed computationally in general and specific-purpose corpora for several languages. The scien- A systematic linguistic evaluation of the translations of the tific study of SVC eliminates subjectivity concerns for the SVC provided by the GT and OL MT systems showed that, evaluation task and allows evalution to be done by only one for the corpus used, GT translated more SVC correctly than evaluator per language. Secondly, SVC occur abundantly in OL due to its richer lexical knowledge. However, OL struc- texts and most MT systems still fail at addressing the com- tural analysis is a powerful feature that could turn OL per- positional aspect of multiword units. SVC When translated formance for the Romance languages as satisfactory as that incorrectly, SVC have a negative impact in the understand- of the German language given the fact that the structural ability and quality of translations (Barreiro et al., 2013). analysis is the same for all language pairs. OL use of lin- Like other multiword units (e.g. phrasal verbs, such as set guistic knowledge in its structural analysis ability, such as up, prepositional predicates, such as look after, SVC can be the SEMTAB semantico-syntactic knowledge representa- non-contiguous, i.e., the individual elements that compose tion and its ability to translate different surface structure the unit are placed apart in the sentence, with a smaller of a sentence, combined with GT rich word selection in the or greater number of inserts. We considered an insert to transfer powered by sophisticated SMT methods to extract be any word in between elements of the multiword unit knowledge from vast amounts of parallel empirical data can other than a definite or an indefinite article before a pred- be a powerful combination that would certainly result in icate noun. Non-contiguous SVC are extremely difficult better translation quality. Table 2 presents the quantitative to align in SMT, remaining one of the key cross-language results for the translation of the 100 SVC in our corpus for challenges for MT. French, German, Italian, Portuguese, and Spanish with the In our research, each SVC under evaluation was anno- OL and the GT MT systems. tated in the context of its sentence and classified according to the taxonomy presented in Table 1, prior to the trans- 6. Linguistic Evaluation lation process so to ease the translation evaluation task. In this section, we will discuss the linguistic challenges The English SVC-annotated corpus was then translated into found in the translations for the 5 language pairs contem- French, German, Italian, Portuguese and Spanish, using the plated in our research, individually and with detailed exam- OL and the GT systems. Native linguists, who are also ples. We believe that such qualitative evaluation is valuable MT experts, evaluated the translation quality of the SVC to the research community. for each target language (one evaluator per language pair), and classified the errors found for each of the target lan- 6.1. English-French guages, according to a binary evaluation metrics: OK for Of the 100 SVC in the English test corpus, 64 were trans- correct translations and ERR for incorrect ones. In addition lated correctly by GT and 51 by OL. The majority of the to ERR, which was used for semantically incorrect transla- translation errors (64% for GT, 69% for OL) involved in- tions of the SVC or serious syntactic problems within the correct lexical choice for some or all of the elements of SVC translation, evaluators have also classified errors of the SVC, resulting in non-idiomatic expressions. Exam- Agreement (Agreem in Table 2) and Other to distinguish ples of literal translations of SVC include the economist is morphologically-related or other problems, such as incor- equally correct in generalizing translated as l’economiste´ rect prepositions or wrong word order affecting directly the est tout aussi correct de gen´ eraliser´ (GT) and we are taking a growing interest in translated as nous prenons un inter´ etˆ SVC elements, such as in Canada was falling so far short croissant pour (OL). A number of interesting errors by GT of translated as Canada stava cadendo finora breve di, to involved idiomatic translations which were not accurate in wrong selections of the meaning of the SVC elements, such the context (e.g. value judgments [...] come into the picture as in These specifications gave insight into space translated translated as les jugements de valeur [...] entrent en scene` ) as Queste specifiche hanno spaccato lo spazio. The second or were only partly generated (e.g. cause-and-effect rela- major category of mistranslations corresponds to 8 wrong tionships are typically not self-evident translated as les rela- agreements (e.g. Il processo dell’economista [...] e` fondato tions de cause a` effet ne sont gen´ eralement´ pas de soi). Er- sui; quando due insiemi di dati sono direttamente corre- rors in source sentence analysis (36% for GT, 31% for OL) late). Similarly to GT, OL mistranslations are mainly due accounted for the remainder of the SVC errors. For GT, to incorrect lexical selection (70%), such as in This is an- these errors showed mostly as lack of subject-verb agree- other way of poking fun at translated as questo e` un altro ment (e.g. The mass market banks then put in place punitive modo di conficcare il divertimento in, and wrong agree- measures translated as Les banques du marche´ de masse ment (10%). Of the 100 SVC, 51 were non-contiguous, alors mettre en place des mesures punitives) whereas the 23 being translated incorrectly by GT and 36 by OL. In GT, analysis of errors in OL was more varied and the result of mistranslations of this typology of SVC are mainly due to missing dictionary entries (e.g. It’s a lot more helpful to wrong lexical choices (15 instances) and affect both prepo- them translated as Il est un lot plus utile a` eux) or incor- sitional nominal SVC (7 instances) and prepositional ad- rect noun phrase analysis, especially when involving a nu- jectival ones (6). OL shows translation problems mainly merical expression (e.g. It also posted a $1.3 billion gain with non-contiguous idiomatic SVC (7), non-contiguous translated as Il a egalement´ poste´ un $1.3 milliard le gain). prepositional nominal SVC (11 instances), and finally, non- contiguous prepositional adjectival SVC (8). In OL, mis- 6.2. English-German translation due to wrong lexical choices are more frequent The analysis of the 100 translations of SVC in the test cor- in comparison with other error typologies. Of the 20 prepo- pus provided the following picture: OL has a higher num- sitional adjectival SVC construction, OL and GT performed ber of correct SVC translations (60) than GT (46). The differently, i.e. OL translated 8 constructions incorrectly, incorrect translations in both German corpora were catego- whereas GT translated 6 constructions incorrectly. rized with respect to the error type: (i) lexical (L) for incorrect word choice, including prepositions (e.g. has par- 6.4. English-Portuguese ticularly good links with was translated as hat besonders gute Verbindungen mit instead of zu, (ii) order (O) for in- Of the 100 English SVC, 68 were translated correctly by correct word order, including incorrect clause segmentation GT and 40 by OL. Of the 32 incorrect translations by GT, (e.g. Is it directly usable or does the user have to do addi- 5 were related to agreement between the subject and the tional data manipulation before one can make use of it? predicate adjective of the SVC (e.g. To be meaningful, facts where the German non-finite tun was incorrectly scram- must be was translated as Para ser significativa, os fatos de- bled into the subordinate clause Ist es direkt verwendbar vem ser instead of Para serem significativos, os fatos devem oder hat der Benutzer zusatzliche¨ Datenmanipulation bevor ser), and between the subject and the verb of the SVC (e.g., man tun kann, davon Gebrauch machen?, (iii) morphology protests will have no effect on was translated as os protestos (M) for incorrect word form, including the choice between nao˜ tera´ nenhum efeito sobre instead of os protestos nao˜ bare-infinitive and to-infinitive, (iv) ellipsis (E) for missing terao˜ nenhum efeito sobre). The SVC translations obtained word, mainly auxiliary verb and main verb (e.g. I hope to from OL did not present any agreement errors. Of the 100 be worthy of it translated as ich hoffe, wurdig¨ , instead of ich SVC, 51 were non-contiguous. The most idiomatic SVC hoffe, dessen wurdig¨ zu sein). When applicable, incorrect were translated correctly by both systems in 3 cases (to be SVC translations were categorized with more than one er- in a position; to set in motion and to take on duties), and ror type. For the 40 incorrect OL translations, there were incorrectly in 5 cases (to come to a rest; to open talks, to 33 lexical errors, and 7 word order errors. For the 54 incor- put in place, to fall short of, to have a spotty record). In the rect GT translations, there were 37 lexical errors, 3 agree- remaining cases, GT performed better than OL. GT missed ment errors, and 14 other errors. In general, GT has more to hold in place, to be in charge of, to be on guard. OL structural (word order and morphology) and missing words missed to come into the picture, to place at risk, to put un- errors than OL. Surprisingly, the performance of GT is also der the microscope, to be on strike, to be at odds with, to poor with regards to lexical coverage of contiguous SVC. earn an income, and to give a bird’s-eye view. In the case of Finally, GT does not translate well the German verb split, less idiomatic SVC, both systems presented problems with even though there seems to be a reordering component. (i) prepositions (e.g. was responsible for was translated as foi responsavel´ para, instead of foi responsavel´ por); (ii) 6.3. English-Italian literal translation of the support verb (e.g. makes it possible Of the 100 English SVC, 61 were translated correctly by was translated as faz poss´ıvel instead of torna poss´ıvel); and GT and 43 by OL. Of the 39 incorrect translations by GT, (iii) wrong lexical choice for the predicate noun (e.g. gave 22 (56%) are related to wrong lexical choices, which range insight into was translated as deu uma visao˜ para, intead from untranslated SVC elements such as in Economists of deram um esclarecimento sobre). In general, SVC prob- have had a spotty record in translated as Gli economisti lems by GT were more structural, while SVC problems by hanno avuto un record spotty nel, literal translation of the OL were more lexical. 6.5. English-Spanish rules are called after dictionary look-up and during the exe- Of the 100 examples of SVC, there are 41 examples in cution of target transfer rules (TRAN rules) to solve ambi- which GT and OL exhibited different behavior. Of these guity problems, such as verb dependencies and multiwords 41 cases, GT got 33 correct translations and 8 incorrect of different nature, overriding the default dictionary trans- translations of SVC, and OL the reverse distribution. Both fer. When a sentence is being parsed by TRAN, OL sends systems encountered the same types of problem: either the the SAL patterns to the SEMTAB database to look for a system recognizes the SVC or it does not. Beyond that they rule match. If the rule exists for a linguistic string, TRAN both presented problems related to translating prepositions uses that rule and overrides the dictionary transfer for that correctly or assigning the correct choice of a determiner. string. In the case of a SVC, such as to apply paint to, One interesting example is To be meaningful, facts must a SEMTAB rule can maintain the SVC structure or para- be, which was translated as Para que tenga sentido (GT) phrase it, transforming it into the verb to paint. When and Para ser de mucho sentido (OL), where the transla- TRAN finds a SEMTAB rule that transforms and translates tion provided by GT is correct and that provided by OL the SVC into another language, such as Portuguese, it can is incorrect. OL would easily translate the frozen SVC cor- choose to override the dictionary entries for the string [to rectly, provided it added it to its dictionary. Another in- + apply + paint + to] into the SVC aplicar tinta a or into teresting example is the SVC We have a lot of worldwide the verb pintar. The SEMTAB rule allows for the different experience in translated as Tenemos mucha experiencia en surface structures of the SVC and any insert specified in the by GT, and Tenemos mucha experiencia mundial by OL. rule, such as aplicaram tinta vermelha a (they applied red In this case, OL is able to resolve the SVC internal modi- paint to) or iria eventualmente pintar (he/she would even- fier adjective worldwide and, therefore, able to generate the tually paint). As long as the SEMTAB rule exists in the intended meaning. GT drops the adjective, removing mean- semantico-syntactic database, OL can process and translate ing from the source in the translation. Finally, the SVC a correctly the SVC in our corpus, which were incorrectly desire to give priority to was translated as dar prioridad al translated in the OL and the GT systems. Therefore, the OL desarrollo by GT, and dar la prioridad al desarrollo by OL. method can overcome the structural problems presented by While both translations are correct, the example illustrates SMT, not only the contiguous, but also the non-contiguous the fact that, for a good amount of SVC, both MT systems SVC, independently of how near or how remotely they oc- can benefit from free rides, where only minor corrections cur in the sentence. The OL methodology applies to any of determiners would be needed. other type of multiword unit and allows the translation of other context-sensitive challenges. 7. Proposal for Semantico-Syntactic Knowledge Integration into SMT 8. Conclusions and Future Work The OL method for semantico-syntactic integration of mul- Two main conclusions can be extracted from our evalua- tiwords, including SVC, is one of the most interesting ap- tion. Our first conclusion refers to the importance to de- proaches to multiword processing in MT. In OL, semantico- velop systematic linguistic quality evaluation metrics with syntactic information is encoded in the lexicon, both in the a phased error categorization task where specific linguis- dictionary entries and in the rules. Any linguistic element tic phenomena can be evaluated individually in stages by in the OL system is represented in an abstraction language MT expert linguists. Independently of the approach, there with ontological properties called SAL1. SAL is an hier- is still significant work to be done as far as evaluation is archical taxonomy made up of supersets, sets and subsets concerned. For example, each subtype of multiword unit and represents the heart of OL, accounting for its relative needs to be tested individually as we have done for sup- effectiveness in parsing and semantic understanding. port verb constructions. Fine-grained error categorization Upon entering the OL system, all natural language in- can contribute to more controlled and systematic evalua- put sentences are converted into SAL patterns, which rep- tion tasks. In addition, for the task of error categorization resent not only the semantico-syntactic features of each to be successfully accomplished, grammar specific evalu- word, but also its morphology. SAL elements interact with ation corpora need to be developed and used to evaluate semantico-syntactic rules, or tables as they are known in each group of linguistic error and identify which system has OL, called SEMTAB. SEMTAB rules represent the mean- more difficulties translating each particular type of linguis- ing of words on the basis of their association with other tic challenge. Future comparative evaluation tasks require words (context); disambiguate the meanings of words in the construction of corpora to test grammatical correctness the source text by identifying the syntactic structures un- addressing individual linguistic phenomena, such as cor- derlying each meaning; and provide the target language pora of different types of multiword units. In this case, the equivalents of each identified meaning of a source lan- translation of each particular type of multiword unit would guage. SEMTAB rules are conceptual and encode deep be evaluated autonomously. Similarly, grammar targeted structure relations that are not sensitive to differences at evaluation would be done by using corpora of relative con- the surface syntactic level (e.g. they raised their salaries structions, passives, pronouns, determiners, locative prepo- and salaries were raised) or morphological level (the rais- sitions, and so on and so forth. No effective hybridization ing of the salary and salary raising policies). SEMTAB can take place before linguistic evaluation of the results provided by different approaches is successfully accom- 1freely available at https://www.l2f.inesc-id.pt/ plished. Therefore, we believe that the question “how effec- abarreiro/openlogos-tutorial/newA2menu.htm tively can rule-based and statistical MT be combined?” can only be answered after linguistic quality evaluation metrics Dictionaries and Automata in Computational Linguis- are developed and validated by the MT community. tics, pages 34–50. Kenneth Heafield and Alon Lavie. 2011. Cmu system Acknowledgements combination in wmt 2011. In Proceedings of the Sixth Anabela Barreiro was funded by FCT Fundaçao˜ para a Workshop on Statistical Machine Translation, WMT ’11, Cienciaˆ e Tecnologia (post-doctoral grant SFRH / BPD / pages 145–151, Stroudsburg, PA, USA. Association for 91446 / 2012). This research was also supported by FCT Computational Linguistics. Fundaçao˜ para a Cienciaˆ e Tecnologia, under project PEst- Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris OE/EEI/LA0021/2013. Callison-Burch, Marcello Federico, Nicola Bertoldi, Autorship contribution is as follows: Anabela Barreiro is Brooke Cowan, Wade Shen, Christine Moran, Richard author of the Abstract and Sections 1., 5., 6., 6.4., 7. and Zens, Chris Dyer, Ondrejˇ Bojar, Alexandra Constantin, 8.; Johanna Monti of Sections 4. and 6.3.; Brigitte Orliac of and Evan Herbst. 2007. Moses: Open source toolkit Section 6.1.; Susanne Preuß of Section 6.2.; Kutz Arrieta for statistical machine translation. In Proceedings of the of Section 6.5.; Wang Ling of Section 2.; and Fernando 45th Annual Meeting of the ACL on Interactive Poster Batista of Section 3.. and Demonstration Sessions, ACL ’07, pages 177–180, Stroudsburg, PA, USA. Association for Computational 9. References Linguistics. Anabela Barreiro, Bernard Scott, Walter Kasper, and Bernd Gorka Labaka, Nicolas Stroppa, Andy Way, and Kepa Sara- Kiefer. 2011. Openlogos rule-based machine transla- sola. 2007. Comparing rule-based and data-driven ap- tion: Philosophy, model, resources and customization. proaches to spanish-to-basque machine translation. In Machine Translation, 25(2):107–126. Machine Translation Summit XI, Copenhagen, Denmark. Anabela Barreiro, Johanna Monti, Brigitte Orliac, and Fer- Felipe Sanchez-Mart´ ´ınez, Mikel L. Forcada, and Andy nando Batista. 2013. When multiwords go bad in ma- Way. 2009. Hybrid rule-based - example-based mt: chine translation. In Proceedings of the Workshop on feeding apertium with sub-sentential translation units. In Multi-word Units in Machine Translation and Transla- EBMT 2009 - 3rd Workshop on Example-Based Machine tion Technology, Machine Translation Summit XIV. Translation, Dublin, Ireland. DORAS. Anabela Barreiro. 2009. Make it Simple with Paraphrases: Bernard (Bud) Scott. 2003. The logos model: An historical Automated Paraphrasing for Authoring Aids and Ma- perspective. Machine Translation, 18(1):1–72. chine Translation. Ph.D. thesis, Universidade do Porto, Satoshi Shirai, Francis Bond, and Yamato Takahashi. 1997. Porto, Portugal. A hybrid rule and example-based method for machine Ralf D. Brown. 1996. Example-based machine translation translation. In Recent Advances in Example-Based Ma- in the pangloss system. In COLING, volume 1, pages chine Translation, pages 211–224. Kluwer Academic 169–174. Publishers. Victor Chahuneau, Noah A. Smith, and Chris Dyer. 2013. Michel Simard, Nicola Ueffing, Pierre Isabelle, and Roland Knowledge-rich morphological priors for bayesian lan- Kuhn. 2007. Rule-based translation with statistical guage models. In HLT-NAACL, pages 1206–1215. The phrase-based post-editing. In Proceedings of the Sec- Association for Computational Linguistics. ond Workshop on Statistical Machine Translation, pages Lo¨ıc Dugast, Jean Senellart, and Philipp Koehn. 2007. Sta- 203–206, Prague, Czech Republic, June. Association for tistical post-editing on systran’s rule-based translation Computational Linguistics. system. In Proceedings of the Second Workshop on Sta- Ehara Terumasa. 2007. Rule based machine translation tistical Machine Translation, StatMT ’07, pages 220– combined with statistical post editor for japanese to en- 223, Stroudsburg, PA, USA. Association for Computa- glish patent translation. In Proceedings of the MT Sum- tional Linguistics. mit XI Workshop on Patent Translation, volume 11, Andreas Eisele, Christian Federmann, Hans Uszkoreit, pages 13–18. Herve´ Saint-Amand, Martin Kay, Michael Jellinghaus, Sabine Hunsicker, Teresa Herrmann, and Yu Chen. 2008. Hybrid machine translation architectures within and beyond the euromatrix project. In J. Hutchins and W.V. Hahn, editors, Hybrid MT Methods in Practice: Their Use in Multilingual Extraction, Cross-Language Information Retrieval, Multilingual Summarization, and Applications in Hand-Held Devices.Proceedings of the European Machine Translation Conference, pages 27– 34. European Association for Machine Translation, HITEC e.V. Jakob Elming. 2006. Transformation-based correction of rule-based mt. In Proceedings of EAMT 2006, pages 219–226. Maurice Gross. 1987. The use of finite automata in the lexical representaion of natural language. In Electronic