DOCSLIB.ORG

Semeval-2017 Task 9: Abstract Meaning Representation Parsing and Generation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Semeval-2017 Task 9: Abstract Meaning Representation Parsing and Generation SemEval-2017 Task 9: Abstract Meaning Representation Parsing and Generation Jonathan May and Jay Priyadarshi Information Sciences Institute Computer Science Department University of Southern California [email protected], [email protected] (f / fear-01 Abstract :polarity "-" :ARG0 ( s / soldier ) In this report we summarize the results :ARG1 ( d / die-01 of the 2017 AMR SemEval shared task. :ARG1 s )) The task consisted of two separate yet The soldier was not afraid of dying. related subtasks. In the parsing sub- The soldier was not afraid to die. task, participants were asked to produce The soldier did not fear death. Abstract Meaning Representation (AMR) (Banarescu et al., 2013) graphs for a set Figure 1: An Abstract Meaning Representation of English sentences in the biomedical do- (AMR) with several English renderings. Example main. In the generation subtask, partici- borrowed from Pust et al.(2015). pants were asked to generate English sen- tences given AMR graphs in the news/fo- rum domain. A total of five sites partici- formal compared to some of those evaluated in last pated in the parsing subtask, and four par- year’s task, they are also very complex, and have ticipated in the generation subtask. Along many terms unique to the domain. An example with a description of the task and the par- is shown in Figure2. We continue to use Smatch ticipants’ systems, we show various score (Cai and Knight, 2013) as a metric for AMR pars- ablations and some sample outputs. ing, but we perform additional ablative analysis using the approach proposed by Damonte et al. 1 Introduction (2016). Abstract Meaning Representation (AMR) is a Along with parsing into AMR, it is important compact, readable, whole-sentence semantic an- to encourage improvements in automatic genera- notation (Banarescu et al., 2013). It includes entity tion of natural language (NL) text from AMR. Hu- identification and typing, PropBank semantic roles mans favor communication in NL. An AI that is (Kingsbury and Palmer, 2002), individual entities able to parse text into AMR at a quality level in- playing multiple roles, as well as treatments of distinguishable from humans may be said to un- modality, negation, etc. AMR abstracts in numer- derstand NL, but without the ability to render its ous ways, e.g., by assigning the same conceptual own semantic representations into NL no human structure to fear (v), fear (n), and afraid (adj). Fig- will ever be able to appreciate this. ure1 gives an example. The advent of several systems that generate En- In 2016 an AMR parsing shared task was held at glish text from AMR input (Flanigan et al., 2016b; SemEval (May, 2016). Task participants demon- Pourdamghani et al., 2016) inspired us to conduct strated several new directions in AMR parsing a generation-based shared task from AMRs in the technology and also validated the strong perfor- news/discussion forum domain. For the gener- mance of existing parsers. We sought, in 2017, to ation subtask, we solicited human judgments of focus AMR parsing performance on the biomed- sentence quality. We followed the precedent es- ical domain, for which a not insignificant but tablished by the Workshop in Machine Transla- still relatively small training corpus had been pro- tion (Bojar et al., 2016) and used the Appraise so- duced. While sentences from this domain are quite licitation system (Federmann, 2012), lightly mod- 536 Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 536–545, Vancouver, Canada, August 3 - 4, 2017. c 2017 Association for Computational Linguistics Interestingly, serpinE2 mRNA and protein were also markedly enhanced in human CRC pervised by Ulf Hermjakob at USC/ISI, is an cells exhibiting mutation in extension of previous releases (LDC2015E86, <i>KRAS </i>and <i>BRAF</i>. LDC2014E41 and LDC2014T12). It contains (e / enhance-01 :li 2 39,260 sentences (subsuming, in turn, the 19,572 :ARG1 (a3 / and AMRs from LDC2015E86, the 18,779 AMRs :op1 (n6 / nucleic-acid from LDC2014E41, and the 13,051 AMRs from :name (n / name :op1 "mRNA") :ARG0-of (e2 / encode-01 LDC2014T12), partitioned into training, develop- :ARG1 p)) ment, and test splits, from a variety of news and :op2 (p / protein discussion forum sources. Participants in the gen- :name (n2 / name :op1 "serpinE2"))) :manner (m / marked) eration task only were provided with AMRs for :mod (a2 / also) an additional 1,293 sentences for evaluation; the :location (c / cell :ARG0-of (e3 / exhibit-01 original sentences were also provided, as needed, :ARG1 (m2 / mutate-01 to human evaluators during the human evaluation :ARG1 (a4 / and phase of the generation subtask (see Section 5.2). :op1 (g / gene :name (n4 / name :op1 "KRAS")) These sentences and their corresponding AMRs :op2 (g2 / gene were sequestered and never released as data :name (n5 / name :op1 "BRAF"))))) before the evaluation phase. :mod (h / human) :mod (d / disease We also made available the Bio-AMR corpus :name (n3 / name :op1 "CRC"))) version 0.8, which consists of 6,452 AMR anno- :manner (i / interesting)) tations of sentences from cancer-related PubMed articles, covering 3 full papers1 as well as the re- Figure 2: One of the simpler biomedical domain sult sections of 46 additional PubMed papers. The sentences and its AMR. Note the italics markers corpus also includes about 1000 sentences each in the original sentence are preserved, as they are from the BEL BioCreative training corpus and the semantically important to the sentence’s Chicago Corpus. The Bio-AMR corpus was par- understanding. titioned into training, development, and test splits. An additional 500 sentences and their AMRs were ified, to gather human rankings, then TrueSkill sequestered until the evaluation phase, at which (Sakaguchi et al., 2014) to elicit an overall system point the sentences were provided to parsing task ranking. participants only. Table1 summarizes the avail- Since the same training data and tools are avail- able data, including the split sizes. able to both subtasks (though, in the case of the 3 Other Resources generation subtask, the utility of the Bio-AMR corpus is unclear), we will describe all the re- We made the following resources available to par- sources for both subtasks in Sections2 and3 but ticipants: then will handle descriptions and ablations for the The tokenizer (from Ulf Hermjakob) used to parsing and generation subtasks separately, in, re- • produce the tokenized sentences in the train- spectively, Sections4 and5. Readers interested ing corpus.2 in only one of these subtasks should not feel com- pelled to read the other section. We will reconvene The AMR specification, used by annotators • in Section6 to conclude and discuss hardware, as in producing the AMRs.3 we continue the tradition established last year in 4 the awarding of trophies to the declared winners The JAMR (Flanigan et al., 2014) and • 5 of each subtask. CAMR (Wang et al., 2015a) parsers, as strong parser baselines. 2 Data 1PMIDs 24651010, 11777939, and 15630473 2 http://alt.qcri.org/semeval2016/ LDC released a new corpus of AMRs task8/data/uploads/tokenizer.tar.gz (LDC2016E25), created as part of the DARPA 3 https://github.com/ kevincrawfordknight/amr-guidelines/blob/ DEFT program, in March of 2016. The new master/amr.md corpus, which was annotated by teams at SDL, 4 https://github.com/jflanigan/jamr LDC, and the University of Colorado, and su- 5 https://github.com/c-amr/camr 537 Corpus Domain Train Dev Test Eval LDC2016E25 News/Forum 36,521 1,368 1,371 N/A Bio-AMR v0.8 Biomedical 5,452 500 500 N/A Parsing evaluation set Biomedical N/A N/A N/A 500 Generation evaluation set (LDC2016R33) News/Form N/A N/A N/A 1,293 Table 1: A summary of data used in this task; split sizes indicate the number of AMRs per sub-corpus. Figure 3: The Appraise interface, adapted for AMR generation evaluation. The JAMR (Flanigan et al., 2016b) genera- asked to produce AMR graphs. Main results and • tion system, as a strong generation baseline. ablative results are shown in Table2. An unsupervised AMR-to-English aligner 4.1 Systems • (Pourdamghani et al., 2014).6 Five teams participated in the task, a noticeable The same Smatch (Cai and Knight, 2013) decline from last year’s task, which saw eleven full • scoring script used in the evaluation.7 participants. One team submitted two systems for a total of six distinct systems. Two teams were re- A Python AMR manipulation library, from peats from last year: CMU and RIGOTRIO (previ- • Nathan Schneider.8 ously RIGA). Below are brief descriptions of each of the various systems, based on summaries pro- 4 Parsing Sub-Task vided by the system authors. Readers are encour- aged to consult individual system description pa- In the parsing sub-task, participants were given pers or relevant conference paper descriptions for 500 previously sequestered Bio-AMRs and were more details. 6 http://isi.edu/˜damghani/papers/ Aligner.zip 4.1.1 The Meaning Factory 7 https://github.com/snowblink14/ (van Noord and Bos, 2017) smatch 8 https://github.com/nschneid/ This team submitted two parsers. TMF-1 is a amr-hackathon character-level sequence-to-sequence deep learn- 538 Smatch Unlab. No WSD NER Wiki TMF-1 0.46 0.5 0.46 0.51 0.46 TMF-2 0.58 0.63 0.58 0.58 0.4 UIT-DANGNT-CLNLP 0.61 0.65 0.61 0.66 0.35 Oxford 0.59 0.63 0.59 0.66 0.18 CMU 0.44 0.47 0.44 0.48 0.59 RIGOTRIO 0.54 0.59 0.54 0.46 0 (a) Main parsing results and four ablations Smatch Neg. Concepts Reent.
Load more

Recommended publications
  • Semantic Role Labeling: an Introduction to the Special Issue
    Semantic Role Labeling: An Introduction to the Special Issue Llu´ıs Marquez` ∗ Universitat Politecnica` de Catalunya Xavier Carreras∗∗ Massachusetts Institute of Technology Kenneth C. Litkowski† CL Research Suzanne Stevenson‡ University of Toronto Semantic role labeling, the computational identification and labeling of arguments in text, has become a leading task in computational linguistics today. Although the issues for this task have been studied for decades, the availability of large resources and the development of statistical machine learning methods have heightened the amount of effort in this field. This special issue presents selected and representative work in the field. This overview describes linguistic background of the problem, the movement from linguistic theories to computational practice, the major resources that are being used, an overview of steps taken in computational systems, and a description of the key issues and results in semantic role labeling (as revealed in several international evaluations). We assess weaknesses in semantic role labeling and identify important challenges facing the field. Overall, the opportunities and the potential for useful further research in semantic role labeling are considerable. 1. Introduction The sentence-level semantic analysis of text is concerned with the characterization of events, such as determining “who” did “what” to “whom,” “where,” “when,” and “how.” The predicate of a clause (typically a verb) establishes “what” took place, and other sentence constituents express the participants in the event (such as “who” and “where”), as well as further event properties (such as “when” and “how”). The primary task of semantic role labeling (SRL) is to indicate exactly what semantic relations hold among a predicate and its associated participants and properties, with these relations Departament de Llenguatges i Sistemes Informatics,` Universitat Politecnica` de Catalunya, Jordi Girona ∗ Salgado 1–3, 08034 Barcelona, Spain.
    [Show full text]
  • Ten Years of Babelnet: a Survey
    Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21) Survey Track Ten Years of BabelNet: A Survey Roberto Navigli1 , Michele Bevilacqua1 , Simone Conia1 , Dario Montagnini2 and Francesco Cecconi2 1Sapienza NLP Group, Sapienza University of Rome, Italy 2Babelscape, Italy froberto.navigli, michele.bevilacqua, [email protected] fmontagnini, [email protected] Abstract to integrate symbolic knowledge into neural architectures [d’Avila Garcez and Lamb, 2020]. The rationale is that the The intelligent manipulation of symbolic knowl- use of, and linkage to, symbolic knowledge can not only en- edge has been a long-sought goal of AI. How- able interpretable, explainable and accountable AI systems, ever, when it comes to Natural Language Process- but it can also increase the degree of generalization to rare ing (NLP), symbols have to be mapped to words patterns (e.g., infrequent meanings) and promote better use and phrases, which are not only ambiguous but also of information which is not explicit in the text. language-specific: multilinguality is indeed a de- Symbolic knowledge requires that the link between form sirable property for NLP systems, and one which and meaning be made explicit, connecting strings to repre- enables the generalization of tasks where multiple sentations of concepts, entities and thoughts. Historical re- languages need to be dealt with, without translat- sources such as WordNet [Miller, 1995] are important en- ing text. In this paper we survey BabelNet, a pop- deavors which systematize symbolic knowledge about the ular wide-coverage lexical-semantic knowledge re- words of a language, i.e., lexicographic knowledge, not only source obtained by merging heterogeneous sources in a machine-readable format, but also in structured form, into a unified semantic network that helps to scale thanks to the organization of concepts into a semantic net- tasks and applications to hundreds of languages.
    [Show full text]
  • ACNLP at Semeval-2020 Task 6: a Supervised Approach for Definition
    ACNLP at SemEval-2020 Task 6: A Supervised Approach for Definition Extraction Fabien Caspani Pirashanth Ratnamogan Mathis Linger Mhamed Hajaiej BNP Paribas, France ffabien.caspani, pirashanth.ratnamogan, mathis.linger, [email protected] Abstract We describe our contribution to two of the subtasks of SemEval 2020 Task 6, DeftEval: Extracting term-definition pairs in free text. The system for Subtask 1: Sentence Classification is based on a transformer architecture where we use transfer learning to fine-tune a pretrained model on the downstream task, and the one for Subtask 3: Relation Classification uses a Random Forest classifier with handcrafted dedicated features. Our systems respectively achieve 0.830 and 0.994 F1-scores on the official test set, and we believe that the insights derived from our study are potentially relevant to help advance the research on definition extraction. 1 Introduction SemEval 2020 Task 6 (Spala et al., 2020) is a definition extraction problem divided into three different subtasks, in which the objective is to find the term-definition pairs in free text, possibly spanning across sentences boundaries. In Subtask 1 the system classifies a sentence as containing a definition or not. Subtask 2 consists of labeling each token with their corresponding BIO tags. Finally, in Subtask 3 the system pairs tagged entities, and labels the relation existing between them. The Definition Extraction from Texts (DEFT) corpus (Spala et al., 2019) is the dataset used for this task, and consists of human-annotated data from textbooks on a variety of subjects. Definition extraction methods in the research literature fall into one of the following categories.
    [Show full text]
  • Semeval 2020 Task 1: Unsupervised Lexical Semantic Change Detection
    SemEval 2020 Task 1: Unsupervised Lexical Semantic Change Detection November 11, 2020 Dominik Schlechtweg,| Barbara McGillivray,};~ Simon Hengchen,♠ Haim Dubossarsky,~ Nina Tahmasebi♠ [email protected] |University of Stuttgart, }The Alan Turing Institute, ~University of Cambridge ♠University of Gothenburg 1 Introduction I evaluation is currently our most pressing problem in LSC detection 1 I SemEval is competition-style semantic evaluation series I SemEval 2020 Task 1 on Unsupervised Lexical Semantic Change Detection (Schlechtweg, McGillivray, Hengchen, Dubossarsky, & Tahmasebi, 2020)2 I datasets for 4 languages with 100,000 human judgments I 2 subtasks I 33 teams submitted 186 systems 1https://semeval.github.io/ 2 https://languagechange.org/semeval/ 2 Tasks I comparison of two time periods t1 and t2 (i) reduces the number of time periods for which data has to be annotated (ii) reduces the task complexity I two tasks: I Subtask 1 { Binary classification: for a set of target words, decide which words lost or gained senses between t1 and t2, and which ones did not. I Subtask 2 { Ranking: rank a set of target words according to their degree of LSC between t1 and t2. I defined on word sense frequency distributions 3 Sense Frequency Distributions (SFDs) Figure 1: An example of a sense frequency distribution for the word cell in two time periods. 4 Corpora t1 t2 English CCOHA 1810-1860 CCOHA 1960-2010 German DTA 1800-1899 BZ+ND 1946-1990 Latin LatinISE -200{0 LatinISE 0{2000 Swedish Kubhist 1790-1830 Kubhist 1895-1903 Table 1: Time-defined subcorpora for each language. 5 Annotation 1.
    [Show full text]
  • Semeval-2013 Task 4: Free Paraphrases of Noun Compounds
    SemEval-2013 Task 4: Free Paraphrases of Noun Compounds Iris Hendrickx Zornitsa Kozareva Radboud University Nijmegen & University of Southern California Universidade de Lisboa [email protected] [email protected] Preslav Nakov Diarmuid OS´ eaghdha´ QCRI, Qatar Foundation University of Cambridge [email protected] [email protected] Stan Szpakowicz Tony Veale University of Ottawa & University College Dublin Polish Academy of Sciences [email protected] [email protected] Abstract The frequency spectrum of compound types fol- lows a Zipfian distribution (OS´ eaghdha,´ 2008), so In this paper, we describe SemEval-2013 Task many NC tokens belong to a “long tail” of low- 4: the definition, the data, the evaluation and frequency types. More than half of the two-noun the results. The task is to capture some of the types in the BNC occur exactly once (Kim and Bald- meaning of English noun compounds via para- phrasing. Given a two-word noun compound, win, 2006). Their high frequency and high produc- the participating system is asked to produce tivity make robust NC interpretation an important an explicitly ranked list of its free-form para- goal for broad-coverage semantic processing of En- phrases. The list is automatically compared glish texts. Systems which ignore NCs may give up and evaluated against a similarly ranked list on salient information about the semantic relation- of paraphrases proposed by human annota- ships implicit in a text. Compositional interpretation tors, recruited and managed through Ama- is also the only way to achieve broad NC coverage, zon’s Mechanical Turk. The comparison of because it is not feasible to list in a lexicon all com- raw paraphrases is sensitive to syntactic and morphological variation.
    [Show full text]
  • Detecting Semantic Difference: a New Model Based on Knowledge and Collocational Association
    DETECTING SEMANTIC DIFFERENCE: A NEW MODEL BASED ON KNOWLEDGE AND COLLOCATIONAL ASSOCIATION Shiva Taslimipoor1, Gloria Corpas2, and Omid Rohanian1 1Research Group in Computational Linguistics, University of Wolverhampton, UK 2Research Group in Computational Linguistics, University of Wolverhampton, UK and University of Malaga, Spain {shiva.taslimi, omid.rohanian}@wlv.ac.uk [email protected] Abstract Semantic discrimination among concepts is a daily exercise for humans when using natural languages. For example, given the words, airplane and car, the word flying can easily be thought and used as an attribute to differentiate them. In this study, we propose a novel automatic approach to detect whether an attribute word represents the difference between two given words. We exploit a combination of knowledge-based and co- occurrence features (collocations) to capture the semantic difference between two words in relation to an attribute. The features are scores that are defined for each pair of words and an attribute, based on association measures, n-gram counts, word similarity, and Concept-Net relations. Based on these features we designed a system that run several experiments on a SemEval-2018 dataset. The experimental results indicate that the proposed model performs better, or at least comparable with, other systems evaluated on the same data for this task. Keywords: semantic difference · collocation · association measures · n-gram counts · word2vec · Concept-Net relations · semantic modelling. 1. INTRODUCTION Semantic modelling in natural language processing requires attending to both semantic similarity and difference. While similarity is well-researched in the community (Mihalcea & Hassan, 2017), the ability of systems in discriminating between words is an under-explored area (Krebs et al., 2018).
    [Show full text]
  • Semeval-2017 Task 2: Multilingual and Cross-Lingual Semantic Word Similarity
    SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity Jose Camacho-Collados*1, Mohammad Taher Pilehvar*2, Nigel Collier2 and Roberto Navigli1 1Department of Computer Science, Sapienza University of Rome 2Department of Theoretical and Applied Linguistics, University of Cambridge 1 collados,navigli @di.uniroma1.it { 2 mp792,nhc30}@cam.ac.uk { } Abstract word representation, a research field that has re- cently received massive research attention mainly This paper introduces a new task on Multi- as a result of the advancements in the use of neural lingual and Cross-lingual Semantic Word networks for learning dense low-dimensional se- Similarity which measures the semantic mantic representations, often referred to as word similarity of word pairs within and across embeddings (Mikolov et al., 2013; Pennington five languages: English, Farsi, German, et al., 2014). Almost any application in NLP that Italian and Spanish. High quality datasets deals with semantics can benefit from efficient se- were manually curated for the five lan- mantic representation of words (Turney and Pan- guages with high inter-annotator agree- tel, 2010). ments (consistently in the 0.9 ballpark). However, research in semantic representation These were used for semi-automatic con- has in the main focused on the English language struction of ten cross-lingual datasets. 17 only. This is partly due to the limited availabil- teams participated in the task, submitting ity of word similarity benchmarks in languages 24 systems in subtask 1 and 14 systems in other than English. Given the central role of subtask 2. Results show that systems that similarity datasets in lexical semantics, and given combine statistical knowledge from text the importance of moving beyond the barriers of corpora, in the form of word embeddings, the English language and developing language- and external knowledge from lexical re- independent and multilingual techniques, we felt sources are best performers in both sub- that this was an appropriate time to conduct a task tasks.
    [Show full text]
  • Semeval-2016 Task 1: Semantic Textual Similarity, Monolingual And
    SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation Eneko Agirrea, Carmen Baneab, Daniel Cerd, Mona Diabe, Aitor Gonzalez-Agirrea, Rada Mihalceab, German Rigaua, Janyce Wiebef aUniversity of the Basque Country bUniversity of Michigan Donostia, Basque Country Ann Arbor, MI dGoogle Inc. eGeorge Washington University f University of Pittsburgh Mountain View, CA Washington, DC Pittsburgh, PA Abstract ranges from complete semantic equivalence to com- plete semantic dissimilarity. The intermediate levels Semantic Textual Similarity (STS) seeks to capture specifically defined degrees of partial sim- measure the degree of semantic equivalence ilarity, such as topicality or rough equivalence, but between two snippets of text. Similarity is ex- with differing details. The snippets being scored are pressed on an ordinal scale that spans from approximately one sentence in length, with their as- semantic equivalence to complete unrelated- ness. Intermediate values capture specifically sessment being performed outside of any contextu- defined levels of partial similarity. While alizing text. While STS has previously just involved prior evaluations constrained themselves to judging text snippets that are written in the same lan- just monolingual snippets of text, the 2016 guage, this year’s evaluation includes a pilot subtask shared task includes a pilot subtask on com- on the evaluation of cross-lingual sentence pairs. puting semantic similarity on cross-lingual text snippets. This year’s traditional mono- The systems and techniques explored as a part of lingual subtask involves the evaluation of En- STS have a broad range of applications including glish text snippets from the following four do- Machine Translation (MT), Summarization, Gener- mains: Plagiarism Detection, Post-Edited Ma- ation and Question Answering (QA).
    [Show full text]
  • Semeval 2017 Task 10: Scienceie - Extracting Keyphrases and Relations from Scientific Publications
    SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications Isabelle Augenstein1, Mrinal Das2, Sebastian Riedel1, Lakshmi Vikraman2, and Andrew McCallum2 1Department of Computer Science, University College London (UCL), UK 2College of Information and Computer Sciences, University of Massachusetts Amherst, USA Abstract classification and relation extraction. However, keyphrases are much more challenging to identify We describe the SemEval task of extract- than e.g. person names, since they vary signifi- ing keyphrases and relations between them cantly between domains, lack clear signifiers and from scientific documents, which is cru- contexts and can consist of many tokens. For this cial for understanding which publications purpose, a double-annotated corpus of 500 pub- describe which processes, tasks and ma- lications with mention-level annotations was pro- terials. Although this was a new task, we duced, consisting of scientific articles of the Com- had a total of 26 submissions across 3 eval- puter Science, Material Sciences and Physics do- uation scenarios. We expect the task and mains. the findings reported in this paper to be Extracting keyphrases and relations between relevant for researchers working on under- them is of great interest to scientific publishers as standing scientific content, as well as the it helps to recommend articles to readers, high- broader knowledge base population and light missing citations to authors, identify poten- information extraction communities. tial reviewers for submissions, and analyse re- search trends over time. Note that organising 1 Introduction keyphrases in terms of synonym and hypernym re- Empirical research requires gaining and maintain- lations is particularly useful for search scenarios, ing an understanding of the body of work in spe- e.g.
    [Show full text]
  • A Machine Translation Approach to Cross-Lingual Word Sense Disambiguation (Semeval-2013 Task 10)
    NRC: A Machine Translation Approach to Cross-Lingual Word Sense Disambiguation (SemEval-2013 Task 10) Marine Carpuat National Research Council Ottawa, Canada [email protected] Abstract (WSD) tasks: as a tool to automatically create train- ing data (Guo and Diab, 2010, for instance) ; as This paper describes the NRC submission to a source of parallel data that can be used to train the Spanish Cross-Lingual Word Sense Dis- ambiguation task at SemEval-2013. Since this WSD systems (Ng and Chan, 2007; van Gompel, word sense disambiguation task uses Spanish 2010; Lefever et al., 2011); or as an application translations of English words as gold annota- which can use the predictions of WSD systems de- tion, it can be cast as a machine translation veloped for SemEval tasks (Carpuat and Wu, 2005; problem. We therefore submitted the output of Chan et al., 2007; Carpuat and Wu, 2007). This Se- a standard phrase-based system as a baseline, mEval shared task gives us the opportunity to com- and investigated ways to improve its sense dis- pare the performance of machine translation systems ambiguation performance. Using only local context information and no linguistic analy- with other submissions which use very different ap- sis beyond lemmatization, our machine trans- proaches. Our goal is to provide machine transla- lation system surprisingly yields top precision tion output which is representative of state-of-the-art score based on the best predictions. However, approaches, and provide a basis for comparing its its top 5 predictions are weaker than those strength and weaknesses with that of other systems from other systems.
    [Show full text]
  • L2 Translation Assistance by Emulating the Manual Post-Editing Process
    Sensible: L2 Translation Assistance by Emulating the Manual Post-Editing Process Liling Tan, Anne-Kathrin Schumann, Jose M.M. Martinez and Francis Bond1 Universitat¨ des Saarland / Campus, Saarbrucken,¨ Germany Nanyang Technological University1 / 14 Nanyang Drive, Singapore [email protected], [email protected], [email protected], [email protected] Abstract The main difference is that in the context of post- editing the source text is provided. A transla- This paper describes the Post-Editor Z sys- tion workflow that incorporates post-editing be- tem submitted to the L2 writing assis- gins with a source sentence, e.g. “I could still tant task in SemEval-2014. The aim of sit on the border in the very last tram to West task is to build a translation assistance Berlin.” and the human translator is provided with system to translate untranslated sentence a machine-generated translation with untranslated fragments. This is not unlike the task fragments such as the previous example and some- of post-editing where human translators times “fixing” the translation would simply re- improve machine-generated translations. quire substituting the appropriate translation for Post-Editor Z emulates the manual pro- the untranslated fragment. cess of post-editing by (i) crawling and ex- tracting parallel sentences that contain the 2 Related Tasks and Previous untranslated fragments from a Web-based Approaches translation memory, (ii) extracting the pos- sible translations of the fragments indexed The L2 writing assistant task lies between the by the translation memory and (iii) apply- lines of machine translation and crosslingual word ing simple cosine-based sentence similar- sense disambiguation (CLWSD) or crosslingual ity to rank possible translations for the un- lexical substitution (CLS) (Lefever and Hoste, translated fragment.
    [Show full text]
  • SAARSHEFF at Semeval-2016 Task 1: Semantic Textual Similarity With
    SAARSHEFF at SemEval-2016 Task 1: Semantic Textual Similarity with Machine Translation Evaluation Metrics and (eXtreme) Boosted Tree Ensembles Liling Tan1, Carolina Scarton2, Lucia Specia2 and Josef van Genabith1, 3 Universitat¨ des Saarlandes1 / Campus A2.2, Saarbrucken,¨ Germany University of Sheffield2 / Regent Court, 211 Portobello, Sheffield, UK Deutsches Forschungszentrum fur¨ Kunstliche¨ Intelligenz3 / Saarbrucken,¨ Germany [email protected], [email protected], [email protected], josef.van [email protected] Abstract ing results (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015). This paper describes the SAARSHEFF sys- At the pilot English STS-2012 task, Rios et al. tems that participated in the English Seman- (2012) trained a Support Vector Regressor using the tic Textual Similarity (STS) task in SemEval- lexical overlaps between the surface strings, named 2016. We extend the work on using ma- entities and semantic role labels and the BLEU (Pa- chine translation (MT) metrics in the STS task by automatically annotating the STS datasets pineni et al., 2002) and METEOR (Banerjee and with a variety of MT scores for each pair of Lavie, 2005; Denkowski and Lavie, 2010) scores be- text snippets in the STS datasets. We trained tween the text snippets and their best system scored our systems using boosted tree ensembles and a Pearson correlation mean of 0.3825. The system achieved competitive results that outperforms underperformed compared to the organizers’ base- he median Pearson correlation scores from all line system1 which scored 0.4356. participating systems. For the English STS-2013 task, Barron-Cede´ no˜ et al.
    [Show full text]