Linguistic Evaluation of Support Verb Constructions by Openlogos and Google Translate
Total Page：16
File Type：pdf, Size：1020Kb
Linguistic Evaluation of Support Verb Constructions by OpenLogos and Google Translate A. Barreiro1, J. Monti2, B. Orliac3, S. Preuß4, K. Arrieta3, W. Ling1,5, F. Batista1,6, I. Trancoso1 1 INESC-ID, Rua Alves Redol 9, 1000-029, Lisboa, Portugal 2 University of Sassari, Italy 3 Logos Institute, USA 4 University of Saarlandes, Germany 5 Carnegie Mellon University - Instituto Superior Tecnico,´ USA-Portugal 6 ISCTE-IUL - Instituto Universitario´ de Lisboa, Portugal [email protected], [email protected] Abstract This paper presents a systematic human evaluation of translations of English support verb constructions produced by a rule-based ma- chine translation (RBMT) system (OpenLogos) and a statistical machine translation (SMT) system (Google Translate) for five languages: French, German, Italian, Portuguese and Spanish. We classify support verb constructions by means of their syntactic structure and semantic behavior and present a qualitative analysis of their translation errors. The study aims to verify how machine translation (MT) systems translate fine-grained linguistic phenomena, and how well-equipped they are to produce high-quality translation. Another goal of the linguistically motivated quality analysis of SVC raw output is to reinforce the need for better system hybridization, which leverages the strengths of RBMT to the benefit of SMT, especially in improving the translation of multiword units. Taking multiword units into ac- count, we propose an effective method to achieve MT hybridization based on the integration of semantico-syntactic knowledge into SMT. Keywords: Machine Translation, MT Evaluation, Support Verb Constructions, Multiword Units, Semantico-Syntactic Knowledge, MT Hybridization. 1. Introduction evaluation efforts involving MT systems of different nature was our main motivation for evaluating the performance of MT researchers and developers have a common goal of cre- RBMT and SMT when dealing with a very specific linguis- ating robust translation systems that can produce high qual- tic phenomenon. ity translation. However, MT is a work-in-progress. Af- This paper describes an evaluation exercise that consisted ter six decades of research on MT models, tools and lin- of the linguistic analysis of the translations of 100 sup- guistic resources, translations produced by widely used MT port verb constructions (SVC), such as make a presenta- systems show unfortunate errors which require significant tion. The sentences in our SVC corpus were randomly post-editing effort if the result is to be used by professional collected from the news and the Internet and hand-picked translators or for purposes other than gisting. A notice- for the task. The corpus was translated into 5 languages: able trend is that of linguistically enhancing SMT mod- French, German, Italian, Portuguese and Spanish by the els, to produce systems that combine linguistic resources OpenLogos (OL) and the Google Translate (GT) MT sys- and analysis with statistical techniques. While this is a tems. Five MT expert linguists, native speakers of the target promising direction, making significant advances towards languages, evaluated the translations of the SVC consider- hybrid MT (HMT) systems requires a deep understanding ing their meaning in context, and classified the translation of different approaches, their weaknesses and strengths. errors based on a SVC typology, as illustrated in Table 1. Bringing different approaches together, contrasting them The remaining of the paper is organized as follows: Section and measuring which modules need improvement seems to 2. presents the state-of-the-art HMT. Section 3. describes be an effective method for achieving the desired end result. the main characteristics of the OL and the GT models. Sec- In pursuing the creation of an HMT model and improving tion 4. describes the corpus, datasets, and the evaluation existing translation technology, a systematic fine-grained task. Section 5. discusses the linguistic challenges found linguistic analysis of the performance of individual models in the translations for the 5 language pairs contemplated appears to us as an important first step. To our knowledge, in this research. Section 7. proposes a method to achieve no major effort has been made to combine the strengths MT hybridization based on the integration of semantico- of different approaches with the purpose of overcoming syntactic knowledge into SMT. Finally, Section 8. presents known weaknesses on the basis of a joint linguistic eval- the main conclusions and points to future work, namely the uation of those weaknesses, as used in this paper. In ad- inclusion of linguistic expertise in MT evaluation. dition, state-of-the-art quality metrics and estimation have been targeting human-factors tasks such as measuring post- editing time and effort in terms of keystrokes, etc., not di- 2. Hybrid Machine Translation agnosing fine-grained linguistic errors to improve syntac- A noticeable trend in current MT research is that of HMT tic structure and meaning. The lack of such qualitative models that combine linguistic knowledge with statistical Nominal Support Verb Construction (NSVC) make a presentation Non-contiguous nominal (NON-CONT NSVC) have [ADV+ADJ-particularly good] links Prepositional nominal (PREPNSVC) give an illustration of Non-contiguous prepositional nominal (NON-CONT PREPNSVC) be the [ADJ-immediate] cause of Idiomatic nominal (IDIOM NSVC) set in motion, place at risk, go on strike Idiomatic prepositional nominal (IDIOM PREPNSVC) earn an income of Non-contiguous idiomatic nominal (NON-CONT IDIOM NSVC) hold [NP-the option] in place, be of [ADJ-practical] value Non-contiguous idiomatic prepositional nominal (NON-CONT IDIOM PREPNSVC) give [PRO-us] a [bird’s-eye] view of, be [ADV-clearly] at odds with, open talks [May 14] with Adjectival Support Verb Construction (ADJSVC) be meaningful Non-contiguous adjectival (NON-CONT ADJSVC) be [ADV-extremely] selective Prepositional adjectival (PREPADJSVC) be known as; be involved in Non-contiguous prepositional adjectival (NON-CONT PREPADJSVC) fall [ADV-so far] short of Table 1: Major categories of support verb construction in our corpus techniques. HMT systems attempt to combine RBMT sys- the coverage of a RBMT system. A similar method us- tems, such as the work presented by (Scott, 2003), with ing example-based MT for the same end has been proposed data-driven MT systems, such as phrase-based SMT pro- in (Sanchez-Mart´ ´ınez et al., 2009). Finally, the translation posed by (Koehn et al., 2007). System combination often quality of RBMT output can also be improved by using leads to improvements in the final translation quality, as statistical post-editing methods (cf. (Simard et al., 2007; different systems tend to address different translation chal- Elming, 2006; Dugast et al., 2007; Terumasa, 2007)). The lenges. opposite has also been proposed, where RBMT systems In general, SMT methods learn the generalizations of the are used to enhance data-driven approaches. (Shirai et al., translation process using parallel corpora, which are sets of 1997) use an example-based MT system (Brown, 1996) to translated sentences pairs. In language pairs where there create an initial translation template, and use a RBMT sys- are abundant amounts of parallel corpora, such as English- tem to translate individual words and phrases according to Mandarin, these models tend to perform better than RBMT this template. To the best of our knowledge, it is still not ob- approaches. However, when parallel data is scarce, such vious which HMT approach will be the most efficient one as for the Spanish-Basque language pair, it is less likely and will lead to higher quality translation in the long run. that the SMT methods will have enough data to properly learn such generalizations (Labaka et al., 2007). Morpho- 3. The OpenLogos and the Google Translate logically rich languages require more data to learn accu- Models rate translations, since there are more word types due to OL is an open source copy of the commercial Logos Sys- different word forms that can be generated for the same tem (Scott, 2003; Barreiro et al., 2011). The system ad- stem. While more complex SMT models for morphologi- dresses morphology, syntax, and semantics, it has robust cally rich languages have been proposed (Chahuneau et al., parsers, sets of semantico-syntactic rules, terminology sets 2013), RBMT systems where the morphology of the lan- and tools. Unlike most other RBMT systems, the OL model guage is encoded manually, represent a strong alternative is closer in spirit to the SMT approach in that both method- for the translation of such languages. ologies are pattern-based, with the additional advantage of Many methods to combine RBMT with SMT have been including semantic understanding. The use of an inter- proposed. One straightforward way to create an HMT sys- mediate language (SAL) to encode linguistic information tem is to combine the translations of the same text by two and process text contributes to OL high quality translation different systems (Heafield and Lavie, 2011; Eisele et al., and lessens one of the main problems in SMT, viz., the 2008). Other approaches attempt to use data-driven tech- sparseness in linguistic examples. OL linguistic knowledge niques to improve RBMT systems. For example, (Eisele et databases have not been developed for over 10 years. al., 2008) use phrase pair extraction in phrase-based MT to GT is one of the most widely used online MT systems. It is extract phrasal translations entries that are used to improve an SMT system that benefits from the huge amount of paral- lel data that Google collects from the web. In March 2014, Lang. pair System OK ERR Agreem Other it was set to account for 80 language pairs. As is typical GT 64 32 4 - EN-FR of SMT, the translation quality is highly dependent on the OL 51 48 1 - language pair, producing much better results for close lan- GT 37 46 3 14 EN-GE guage pairs (e.g. Portuguese and Spanish) and languages OL 60 33 1 6 for which large amounts of parallel data are available.