DOCSLIB.ORG

Recognizing Textual Entailment Using

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Recognizing Textual Entailment Using RECOGNIZING TEXTUAL ENTAILMENT USING DESCRIPTION LOGIC AND SEMANTIC RELATEDNESS Reda Siblini A thesis in The Department of Computer Science and Software Engineering Presented in Partial Fulfillment of the Requirements For the Degree of Doctor of Philosophy Concordia University Montreal,´ Quebec,´ Canada April 2014 c Reda Siblini, 2014 Concordia University School of Graduate Studies This is to certify that the thesis prepared By: Mr. Reda Siblini Entitled: Recognizing Textual Entailment using Description Logic and Semantic Relatedness and submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Science) complies with the regulations of this University and meets the accepted standards with respect to originality and quality. Signed by the final examining committee: Dr. A. Hammad, Chair Dr. H. Saggion, External Examiner Dr. L. Kosseim, Thesis Supervisor Dr. V. Haarslev, Examiner Dr. S. Bergler, Examiner Dr. F. Khendek, Examiner Approved Chair of Department or Graduate Program Director 20 Dr. Christopher Trueman, Dean Faculty of Engineering and Computer Science Abstract Recognizing Textual Entailment using Description Logic and Semantic Relatedness Reda Siblini, Ph.D. Concordia University, 2014 Textual entailment (TE) is a relation that holds between two pieces of text where one read- ing the first piece can conclude that the second is most likely true. Accurate approaches for textual entailment can be beneficial to various natural language processing (NLP) ap- plications such as: question answering, information extraction, summarization, and even machine translation. For this reason, research on textual entailment has attracted a signif- icant amount of attention in recent years. A robust logical-based meaning representation of text is very hard to build, therefore the majority of textual entailment approaches rely on syntactic methods or shallow semantic alternatives. In addition, approaches that do use a logical-based meaning representation, require a large knowledge base of axioms and inference rules that are rarely available. The goal of this thesis is to design an efficient description logic based approach for recognizing textual entailment that uses semantic re- latedness information as an alternative to large knowledge base of axioms and inference rules. In this thesis, we propose a description logic and semantic relatedness approach to textual entailment, where the type of semantic relatedness axioms employed in aligning the descrip- tion logic representations are used as indicators of textual entailment. In our approach, the text and the hypothesis are first represented in description logic. The representations are enriched with additional semantic knowledge acquired by using the web as a corpus. The iii hypothesis is then merged into the text representation by learning semantic relatedness axioms on demand and a reasoner is then used to reason over the aligned representation. Finally, the types of axioms employed by the reasoner are used to learn if the text entails the hypothesis or not. To validate our approach we have implemented an RTE system named AORTE, and evaluated its performance on recognizing textual entailment using the fourth recognizing textual entailment challenge. Our approach achieved an accuracy of 68.8% on the two way task and 61.6% on the three way task which ranked the approach as 2nd when compared to the other participating runs in the same challenge. These results show that our description logical based approach can effectively be used to recognize textual entailment. iv Acknowledgments This thesis reflects my long and rewarding research journey, during which I came across many wonderful people to whom I express my thanks. First, I would like to express my deepest graduate to my supervisor, Dr. Leila Kos- seim, whose spirit of quest in regard to research inspired me to freely explore new ideas. This thesis would not have been possible without her persistent support and guidance. In addition, I wish to extend my appreciation for the members of my doctoral commit- tee, Professors Volker Haarslev, Sabine Bergler, Ferhat Khendek, and Amin Hammad for their valuable time and feedback. I am particularly thankful to Professor Bergler, whose enthusiasm for linguistics is contagious, for her thorough comments on my the- sis. I am also grateful to Professor Haarslev, who introduced me and gave me access to his work in Description Logic. I want to specially thank the external examiner, Professor Horacio Saggion for his participation in my defense committee and for his valuable feedback on my thesis. Finally, I would like to thank my colleagues at the CLaC Lab, my family, and loved ones for their continuous moral support and encouragement throughout my studies. v Contents List of Figures x List of Tables xii 1 Introduction 1 1.1 Problem Statement . 2 1.2 Motivation . 5 1.3 Overview of the Thesis . 7 1.4 Intended Contributions . 9 1.5 Thesis Organization . 11 2 State of the Art 13 2.1 Approaches to Recognizing Textual Entailment . 14 2.1.1 Lexical Based Methods . 15 2.1.2 Syntax Based Methods . 18 2.1.3 Semantic Based Methods . 20 2.1.4 Logical Form Based Methods . 21 2.1.5 Hybrid Methods . 23 2.1.6 Analysis . 24 2.2 Recognizing Textual Entailment Benchmarks . 26 2.2.1 The First Recognizing Textual Entailment Challenge (RTE1) . 26 vi 2.2.2 The Second Recognizing Textual Entailment Challenge (RTE2) 31 2.2.3 The Third Recognizing Textual Entailment Challenge (RTE3) 32 2.2.4 The Fourth Recognizing Textual Entailment Challenge (RTE4) 33 2.2.5 The Fifth Recognizing Textual Entailment Challenge (RTE5) 35 2.2.6 The Student Response Analysis and The Eighth Recognizing Textual Entailment Challenge (SRA-RTE8) . 36 2.2.7 Analysis of Benchmarking . 37 2.3 Resources used in Recognizing Textual Entailment . 38 2.3.1 Acronym Lists . 39 2.3.2 Nominalization Databases . 40 2.3.3 Gazetteers . 42 2.3.4 N-gram Models . 42 2.3.5 Semantic Networks . 43 2.3.6 Frames . 46 2.3.7 Thesauri and Encyclopaedias . 49 2.3.8 Ontologies . 51 2.3.9 Inference Rules . 53 2.4 Conclusion . 55 3 Baseline Knowledge Querying Approach 56 3.1 Overview of the Baseline Knowledge Querying Approach . 57 3.2 Representation . 61 3.2.1 Syntactic Analysis . 65 3.2.2 Semantic Analysis . 70 3.2.3 Ontological Analysis . 74 3.3 Comparison . 84 3.3.1 Query Creation . 85 vii 3.3.2 Querying with a Reasoner . 88 3.4 Decision . 90 3.5 Evaluation and Analysis . 91 3.5.1 Evaluation of the Knowledge Querying Approach . 91 3.5.2 Evaluation of the Recognizing Textual Entailment Approach . 93 3.5.3 Analysis . 94 3.6 Conclusion . 95 4 Knowledge Alignment Approach 96 4.1 Overview of the Knowledge Alignment Approach . 97 4.2 Representation . 101 4.2.1 Named Entity Recognition Using the Web . 101 4.2.2 Evaluation of the Named Entity Recognizer . 107 4.2.3 Named Entity Recognizer and RTE . 110 4.3 Comparison . 111 4.4 Decision . 116 4.5 Evaluation and Analysis . 120 4.6 Conclusion . 123 5 Recognizing Textual Entailment and Lexical Semantic Relatedness 124 5.1 Lexical Semantic Relatedness . 125 5.1.1 Our Approach to Semantic Relatedness . 127 5.2 Intrinsic Evaluation of Semantic Relatedness . 133 5.2.1 Evaluation using Human Ratings . 133 5.2.2 Evaluation using Synonym Tests . 136 5.2.3 Evaluation on a Word-Phrase Semantic Relatedness . 138 5.3 Knowledge Alignment Approach to RTE Revisited . 140 viii 5.4 Evaluation and Analysis . 143 5.5 Conclusion . 144 6 Conclusion and Future Work 146 6.1 Main Findings and Contributions of the Thesis . 147 6.2 Directions for Future Research . 150 Bibliography 152 A Sample OWL Generated Representation 178 ix List of Figures 1 Baseline Knowledge Querying Approach . 59 2 Prototype of the Baseline Knowledge Querying Approach . 60 3 Syntax and Semantics of SHOINQ Description Logic from [Horrocks and Patel-Schneider, 2004] . 63 4 A Graphical Illustration of Part of the Representation Created for the Text \Jurassic Park is a novel written by Michael Crichton and Pub- lished in 1990." with WordNet axioms. 66 5 Syntax and Semantics of OWL DL Entities [Horrocks, Patel-Schneider, and Van Harmelen, 2003] . 77 6 Syntax and Semantics of OWL DL Axioms and Facts [Horrocks, Patel- Schneider, and Van Harmelen, 2003] . 78 7 A Graphical Illustration of Part of the Representation Created for the Text \Jurassic Park is a novel written by Michael Crichton and pub- lished in 1990.".............................. 80 8 Knowledge Alignment Approach . 98 9 Prototype of the Knowledge Alignment Approach . 99 10 A Graphical Illustration of representation-T Created for the Hypothe- sis \Jurassic Park is a novel written by Michael Crichton" . 114 11 A Graphical Illustration of representation-H Created for the Hypothe- sis \Michael Crichton crafted the book Jurassic Park" . 115 x 12 A Graphical Illustration of an Alignment representation-A . 116 13 Part of the Resulting Decision Tree . 119 14 Example of the Semantic Network Around the Word car. 128 15 Lowest Cost Path Between the Words Monk and Oracle. 132 16 Knowledge Alignment Approach Augmented with New Semantic Re- latedness Calculator . 142 xi List of Tables 1 Dependency Parse of \Jurassic Park is a novel written by Michael Crichton and Published in 1990.".................... 67 2 Result of the Syntactic Analysis for \Jurassic Park is a novel written by Michael Crichton and Published in 1990. 69 3 Result of the Semantic Analysis for the \Jurassic Park is a novel writ- ten by Michael Crichton and Published in 1990. 74 4 Result of the Semantic Analysis for \Michael Crichton crafted Jurassic Park in 1990." .............................. 85 5 Examples of semantic mapping for the predicate structure has-crafter (Craft, Author) .............................. 87 6 Examples of Predicate Structure Patterns and Corresponding nRQL Queries . 89 7 Sample of the Evaluation Results .
Load more

Recommended publications
  • Are Annotators' Word-Sense-Disambiguation
    J. Hlavácovᡠ(Ed.): ITAT 2017 Proceedings, pp. 5–14 CEUR Workshop Proceedings Vol. 1885, ISSN 1613-0073, c 2017 S. Cinková, A. Vernerová Are Annotators’ Word-Sense-Disambiguation Decisions Affected by Textual Entailment between Lexicon Glosses? Silvie Cinková, Anna Vernerová Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostranské námestíˇ 25 118 00 Praha 1 Czech Republic ufal.mff.cuni.cz {cinkova,vernerova}@ufal.mff.cuni.cz Abstract: We describe an annotation experiment com- verb on the one hand, and those related to the lexicograph- bining topics from lexicography and Word Sense Disam- ical design of PDEV, such as semantic relations between biguation. It involves a lexicon (Pattern Dictionary of En- implicatures within a lemma or denotative similarity of the glish Verbs, PDEV), an existing data set (VPS-GradeUp), verb arguments, on the other hand (see Section 3 for defi- and an unpublished data set (RTE in PDEV Implicatures). nitions and examples). The aim of the experiment was twofold: a pilot annota- This paper focuses on a feature related to PDEV’s de- tion of Recognizing Textual Entailment (RTE) on PDEV sign (see Section 2.3), namely on textual entailment be- implicatures (lexicon glosses) on the one hand, and, on tween implicatures in pairs of patterns of the same lemma the other hand, an analysis of the effect of Textual Entail- entry (henceforth colempats, see Section 3.1 for definition ment between lexicon glosses on annotators’ Word-Sense- and more detail). Disambiguation decisions, compared to other predictors, We pairwise compare all colempats, examining their such as finiteness of the target verb, the explicit presence scores in the graded decision annotation with respect to of its relevant arguments, and the semantic distance be- how much they compete to become the most appropriate tween corresponding syntactic arguments in two different pattern, as well as the scores of presence of textual entail- patterns (dictionary senses).
    [Show full text]
  • Textual Inference for Machine Comprehension Martin Gleize
    Textual Inference for Machine Comprehension Martin Gleize To cite this version: Martin Gleize. Textual Inference for Machine Comprehension. Computation and Language [cs.CL]. Université Paris Saclay (COmUE), 2016. English. NNT : 2016SACLS004. tel-01317577 HAL Id: tel-01317577 https://tel.archives-ouvertes.fr/tel-01317577 Submitted on 18 May 2016 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Le cas échéant, logo de l’établissement co-délivrant le doctorat en cotutelle internationale de thèse , sinon mettre le logo de l’établissement de préparation de la thèse (UPSud, HEC, UVSQ, UEVE, ENS Cachan, Polytechnique, IOGS, …) NNT : 2016SACLS004 THÈSE DE DOCTORAT DE L’UNIVERSITÉ PARIS-SACLAY PRÉPARÉE À L'UNIVERSITÉ PARIS-SUD ECOLE DOCTORALE N° 580 Sciences et technologies de l'information et de la communication (STIC) Spécialité de doctorat : Informatique Par M. Martin Gleize Textual Inference for Machine Comprehension Thèse présentée et soutenue à Orsay, le 7 janvier 2016 Composition du Jury : M. Yvon François Professeur, Université Paris-Sud Président, Examinateur Mme Gardent Claire Directeur de recherche CNRS, LORIA Rapporteur M. Magnini Bernardo Senior Researcher, FBK Rapporteur M. Piwowarski Benjamin Chargé de recherche CNRS, LIP6 Examinateur Mme Grau Brigitte Professeur, ENSIIE Directeur de thèse Abstract With the ever-growing mass of published text, natural language understanding stands as one of the most sought-after goal of artificial intelligence.
    [Show full text]
  • Repurposing Entailment for Multi-Hop Question Answering Tasks
    Repurposing Entailment for Multi-Hop Question Answering Tasks Harsh Trivedi|, Heeyoung Kwon|, Tushar Khot♠, Ashish Sabharwal♠, Niranjan Balasubramanian| | Stony Brook University, Stony Brook, U.S.A. fhjtrivedi,heekwon,[email protected] ♠ Allen Institute for Artificial Intelligence, Seattle, U.S.A. ftushark,[email protected] Abstract Question Answering (QA) naturally reduces to an entailment problem, namely, verifying whether some text entails the answer to a ques- tion. However, for multi-hop QA tasks, which require reasoning with multiple sentences, it remains unclear how best to utilize entailment models pre-trained on large scale datasets such as SNLI, which are based on sentence pairs. We introduce Multee, a general architecture Figure 1: An example illustrating the challenges in us- that can effectively use entailment models for ing sentence-level entailment model for multi-sentence multi-hop QA tasks. Multee uses (i) a local reasoning needed for QA, and the high-level approach module that helps locate important sentences, used in Multee. thereby avoiding distracting information, and level, whereas question answering requires verify- (ii) a global module that aggregates informa- ing whether multiple sentences, taken together as tion by effectively incorporating importance weights. Importantly, we show that both mod- a premise, entail a hypothesis. ules can use entailment functions pre-trained There are two straightforward ways to address on a large scale NLI datasets. We evaluate per- this mismatch: (1) aggregate independent entail- formance on MultiRC and OpenBookQA, two ment decisions over each premise sentence, or (2) multihop QA datasets. When using an entail- make a single entailment decision after concate- ment function pre-trained on NLI datasets, nating all premise sentences.
    [Show full text]
  • Techniques for Recognizing Textual Entailment and Semantic Equivalence
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by EPrints Complutense Techniques for Recognizing Textual Entailment and Semantic Equivalence Jes´us Herrera, Anselmo Pe˜nas, Felisa Verdejo Departamento de Lenguajes y Sistemas Inform´aticos Universidad Nacional de Educaci´ona Distancia Madrid, Spain {jesus.herrera, anselmo, felisa}@lsi.uned.es Abstract. After defining what is understood by textual entailment and semantic equivalence, the present state and the desirable future of the systems aimed at recognizing them is shown. A compilation of the currently implemented techniques in the main Recognizing Textual Entailment and Semantic Equivalence systems is given. 1 Introduction The concept “textual entailment” is used to indicate the state in which the semantics of a natural language written text can be inferred from the semantics of another one. More specifically, if the truth of an enunciation entails the truth of another enunciation. For example, given the texts: 1. The three-day G8 meeting will take place in Scotland. 2. The Group of Eight summit will last three days. it is clear that the semantics of the second one can be inferred from the semantics of the first one; then, it is said that textual entailment exists between both texts. Textual entailment is a directional relationship: in the example above, the first statement entails the second one, but this entailment is not given in the opposite direction. The recognition of textual entailment requires a processing at the lexical level (for example, synonymy between meeting and summit or between G8 and Group of Eight), as well as at the syntactic level and the sentence semantic level.
    [Show full text]
  • Automatic Evaluation of Summary Using Textual Entailment
    Automatic Evaluation of Summary Using Textual Entailment Pinaki Bhaskar Partha Pakray Department of Computer Science & Department of Computer Science & Engineering, Jadavpur University, Engineering, Jadavpur University, Kolkata – 700032, India Kolkata – 700032, India [email protected] [email protected] summary is very much needed when a large Abstract number of summaries have to be evaluated, spe- cially for multi-document summaries. For sum- This paper describes about an automatic tech- mary evaluation we have developed an automat- nique of evaluating summary. The standard ed evaluation technique based on textual entail- and popular summary evaluation techniques or ment. tools are not fully automatic; they all need Recognizing Textual Entailment (RTE) is one some manual process. Using textual entail- of the recent research areas of Natural Language ment (TE) the generated summary can be evaluated automatically without any manual Processing (NLP). Textual Entailment is defined evaluation/process. The TE system is the as a directional relationship between pairs of text composition of lexical entailment module, lex- expressions, denoted by the entailing “Text” (T) ical distance module, Chunk module, Named and the entailed “Hypothesis” (H). T entails H if Entity module and syntactic text entailment the meaning of H can be inferred from the mean- (TE) module. The syntactic TE system is ing of T. Textual Entailment has many applica- based on the Support Vector Machine (SVM) tions in NLP tasks, such as Summarization, In- that uses twenty five features for lexical simi- formation Extraction, Question Answering, In- larity, the output tag from a rule based syntac- formation Retrieval. tic two-way TE system as a feature and the outputs from a rule based Chunk Module and 2 Related Work Named Entity Module as the other features.
    [Show full text]
  • Dependency-Based Textual Entailment
    Dependency-based Textual Entailment Vasile Rus Department of Computer Science Institute for Intelligent Systems The University of Memphis Memphis, TN 38120, USA [email protected] Abstract and Hypothesis are mapped into two sets of dependen- cies: the T-set and H-set, respectively. A score is then This paper studies the role of dependency information computed that measures the degree to which the depen- for the task of textual entailment. Both the Text and Hypothesis of an entailment pair are mapped into sets dencies in the H-set are present in the T-set. Based on of dependencies and a score is computed that measures the score, an entailment decision is made. Dependen- the similarity of the two sets. Based on the score an cies are labelled relations among words in a sentence. entailment decision is made. Two experiments are con- For example, there is a subj dependency between cap- ducted to measure the impact of dependencies on the ital and Besancon in the previous example which we entailment task. In one experiment we compare the represent as the triplet (subj, capital, Besancon). The dependency-based approach with a baseline approach first term in the triplet is the name or label of the re- on a standard data set. In a second experiment, we lation, the second term represents the head word, and measure the performance on a subset of the standard the last term is the dependent word. We use MINIPAR data set. The subset is so selected to minimize the (Lin 1998), an automated dependency parser, as our effect of other factors, such as word-level information, in the performance measurement process.
    [Show full text]
  • Parser Evaluation Using Textual Entailments
    Language Resources and Evaluation manuscript No. (will be inserted by the editor) Parser Evaluation Using Textual Entailments Deniz Yuret · Laura Rimell · Aydın Han Received: date / Accepted: date Abstract Parser Evaluation using Textual Entailments (PETE) is a shared task in the SemEval-2010 Evaluation Exercises on Semantic Evaluation. The task involves recognizing textual entailments based on syntactic information alone. PETE introduces a new parser evaluation scheme that is formalism independent, less prone to annotation error, and focused on semantically rele- vant distinctions. This paper describes the PETE task, gives an error analysis of the top-performing Cambridge system, and introduces a standard entail- ment module that can be used with any parser that outputs Stanford typed dependencies. Keywords Parsing · Textual Entailments 1 Introduction Parser Evaluation using Textual Entailments (PETE) is a shared task in the SemEval-2010 Evaluation Exercises on Semantic Evaluation that involves rec- ognizing textual entailments to evaluate parser performance. Given two text Deniz Yuret and Aydın Han Ko¸cUniversity Rumelifeneri Yolu 34450 Sarıyer, Istanbul,˙ Turkey Tel.: +90-212-338-1724 Fax: +90-212-338-1548 E-mail: dyuret,[email protected] Laura Rimell Computer Laboratory William Gates Building 15 JJ Thomson Avenue Cambridge CB3 0FD, UK Tel.: +44 (0)1223 334696 E-mail: [email protected] 2 Deniz Yuret et al. fragments called “text” (T) and “hypothesis” (H), recognizing textual entail- ment (RTE) is the task of determining whether the meaning of the hypothesis is entailed (can be inferred) from the text. In contrast to general RTE tasks (Dagan et al 2009) PETE is a targeted textual entailment task that focuses on syntactic entailments: Text: The man with the hat was tired.
    [Show full text]
  • Textual Entailment Recognition Based on Dependency Analysis and Wordnet
    Textual Entailment Recognition Based on Dependency Analysis and WordNet Jesus´ Herrera, Anselmo Penas,˜ Felisa Verdejo Departamento de Lenguajes y Sistemas Informaticos´ Universidad Nacional de Educacion´ a Distancia Madrid, Spain jesus.herrera, anselmo, felisa @lsi.uned.es f g Abstract. The Recognizing Textual Entailment System shown here is based on the use of a broad-coverage parser to extract dependency relationships; in addi- tion, WordNet relations are used to recognize entailment at the lexical level. The work investigates whether the mapping of dependency trees from text and hy- pothesis give better evidence of entailment than the matching of plain text alone. While the use of WordNet seems to improve system's performance, the notion of mapping between trees here explored (inclusion) shows no improvement, sug- gesting that other notions of tree mappings should be explored such as tree edit distances or tree alignment distances. 1 Introduction Textual Entailment Recognition (RTE) aims at deciding whether the truth of a text en- tails the truth of another text called hypothesis. This concept has been the basis for the PASCAL1 RTE Challenge [3]. The system presented here is aimed at validating the hypothesis that (i) a certain amount of semantic information could be extracted from texts by means of the syntactic structure given by a dependency analysis, and that (ii) lexico-semantic information such as WordNet relations can improve RTE. In short, the techniques involved in this system are the following: – Dependency analysis of texts and hypothesises. – Lexical entailment between dependency tree nodes using WordNet. – Mapping between dependency trees based on the notion of inclusion.
    [Show full text]
  • A Probabilistic Classification Approach for Lexical Textual Entailment
    A Probabilistic Classification Approach for Lexical Textual Entailment Oren Glickman and Ido Dagan and Moshe Koppel Computer Science Department , Bar Ilan University Ramat-Gan, 52900, Iarael {glikmao, dagan, koppel}@cs.biu.ac.il Abstract The uncertain nature of textual entailment calls for its The textual entailment task – determining if a given text explicit modeling in probabilistic terms. In this paper we entails a given hypothesis – provides an abstraction of first propose a general generative probabilistic setting for applied semantic inference. This paper describes first a textual entailment. This probabilistic framework may be general generative probabilistic setting for textual considered analogous to (though different than) the entailment. We then focus on the sub-task of recognizing probabilistic setting defined for other phenomena, like whether the lexical concepts present in the hypothesis are language modeling and statistical machine translation. entailed from the text. This problem is recast as one of text Obviously, such frameworks are needed to allow for categorization in which the classes are the vocabulary principled (fuzzy) modeling of the relevant language words. We make novel use of Naïve Bayes to model the problem in an entirely unsupervised fashion. Empirical tests phenomenon, compared to utilizing ad-hoc ranking scores suggest that the method is effective and compares favorably which have no clear probabilistic interpretation. We with state-of-the-art heuristic scoring approaches. suggest that the proposed setting may provide a unifying framework for modeling uncertain semantic inferences from texts. Introduction An important sub task of textual entailment, which we term lexical entailment , is recognizing if the concepts in a Many Natural Language Processing (NLP) applications hypothesis h are entailed from a given text t, even if the need to recognize when the meaning of one text can be relations between these concepts may not be entailed from expressed by, or inferred from, another text.
    [Show full text]
  • Frame Semantics Annotation Made Easy with Dbpedia
    Frame Semantics Annotation Made Easy with DBpedia Marco Fossati, Sara Tonelli, and Claudio Giuliano ffossati, satonelli, [email protected] Fondazione Bruno Kessler - via Sommarive, 18 - 38123 Trento, Italy Abstract. Crowdsourcing techniques applied to natural language pro- cessing have recently experienced a steady growth and represent a cheap and fast, albeit valid, solution to create benchmarks and training data. Nevertheless, some particularly complex tasks such as semantic role an- notation have been rarely conducted in a crowdsourcing environment, due to their intrinsic difficulty. In this paper, we present a novel ap- proach to accomplish this task by leveraging information automatically extracted from DBpedia. We show that replacing role definitions, typi- cally meant for expert annotators, with a list of DBpedia types, makes the identification and assignment of role labels more intuitive also for non-expert workers. Results prove that such strategy improves on the standard annotation workflow, both in terms of accuracy and of time consumption. Keywords: Natural Language Processing, Frame Semantics, En- tity Linking, DBpedia, Crowdsourcing, Task Modeling 1 Introduction Frame semantics [6] is one of the theories that originate from the long strand of linguistic research in artificial intelligence. A frame can be informally defined as an event triggered by some term in a text and embedding a set of partic- ipants. For instance, the sentence Goofy has murdered Mickey Mouse evokes the Killing frame (triggered by murdered) together with the Killer and Vic- tim participants (respectively Goofy and Mickey Mouse). Such theory has led to the creation of FrameNet [2], namely an English lexical database containing manually annotated textual examples of frame usage.
    [Show full text]
  • Constructing a Textual Semantic Relation Corpus Using a Discourse Treebank
    Constructing a Textual Semantic Relation Corpus Using a Discourse Treebank Rui Wang and Caroline Sporleder Computational Linguistics Saarland University frwang,[email protected] Abstract In this paper, we present our work on constructing a textual semantic relation corpus by making use of an existing treebank annotated with discourse relations. We extract adjacent text span pairs and group them into six categories according to the different discourse relations between them. After that, we present the details of our annotation scheme, which includes six textual semantic relations, backward entailment, forward entailment, equality, contradiction, overlapping, and independent. We also discuss some ambiguous examples to show the difficulty of such annotation task, which cannot be easily done by an automatic mapping between discourse relations and semantic relations. We have two annotators and each of them performs the task twice. The basic statistics on the constructed corpus looks promising: we achieve 81.17% of agreement on the six semantic relation annotation with a .718 kappa score, and it increases to 91.21% if we collapse the last two labels with a .775 kappa score. 1. Introduction TRADICTION, and UNKNOWN. ENTAILMENT is the same Traditionally, the term ‘semantic relation’ refers to rela- as the previous YES; but NO is divided into CONTRADIC- tions that hold between the meanings of two words, e.g., TION and UNKNOWN, to differentiate cases where T and H synonymy, hypernymy, etc. These relations are usually are contradictory to each other from all the other cases. The situation-independent. Discourse relations, on the other RTE data are acquired from other natural language process- hand, tend to depend on the wider context of an utterance.
    [Show full text]
  • PATTY: a Taxonomy of Relational Patterns with Semantic Types
    PATTY: A Taxonomy of Relational Patterns with Semantic Types Ndapandula Nakashole, Gerhard Weikum, Fabian Suchanek Max Planck Institute for Informatics Saarbrucken,¨ Germany {nnakasho,weikum,suchanek}@mpi-inf.mpg.de Abstract Yet, existing large-scale knowledge bases are mostly limited to abstract binary relationships be- This paper presents PATTY: a large resource for textual patterns that denote binary relations tween entities, such as “bornIn” (Auer 2007; Bol- between entities. The patterns are semanti- lacker 2008; Nastase 2010; Suchanek 2007). These cally typed and organized into a subsumption do not correspond to real text phrases. Only the Re- taxonomy. The PATTY system is based on ef- Verb system (Fader 2011) yields a larger number of ficient algorithms for frequent itemset mining relational textual patterns. However, no attempt is and can process Web-scale corpora. It har- made to organize these patterns into synonymous nesses the rich type system and entity popu- patterns, let alone into a taxonomy. Thus, the pat- lation of large knowledge bases. The PATTY taxonomy comprises 350,569 pattern synsets. terns themselves do not exhibit semantics. Random-sampling-based evaluation shows a Goal. Our goal in this paper is to systematically pattern accuracy of 84.7%. PATTY has 8,162 compile relational patterns from a corpus, and to im- subsumptions, with a random-sampling-based pose a semantically typed structure on them. The precision of 75%. The PATTY resource is freely available for interactive access and result we aim at is a WordNet-style taxonomy of download. binary relations. In particular, we aim at patterns that contain semantic types, such as hsingeri sings 1 Introduction hsongi.
    [Show full text]