Dependency-Based Textual Entailment
Total Page:16
File Type:pdf, Size:1020Kb
Dependency-based Textual Entailment Vasile Rus Department of Computer Science Institute for Intelligent Systems The University of Memphis Memphis, TN 38120, USA [email protected] Abstract and Hypothesis are mapped into two sets of dependen- cies: the T-set and H-set, respectively. A score is then This paper studies the role of dependency information computed that measures the degree to which the depen- for the task of textual entailment. Both the Text and Hypothesis of an entailment pair are mapped into sets dencies in the H-set are present in the T-set. Based on of dependencies and a score is computed that measures the score, an entailment decision is made. Dependen- the similarity of the two sets. Based on the score an cies are labelled relations among words in a sentence. entailment decision is made. Two experiments are con- For example, there is a subj dependency between cap- ducted to measure the impact of dependencies on the ital and Besancon in the previous example which we entailment task. In one experiment we compare the represent as the triplet (subj, capital, Besancon). The dependency-based approach with a baseline approach first term in the triplet is the name or label of the re- on a standard data set. In a second experiment, we lation, the second term represents the head word, and measure the performance on a subset of the standard the last term is the dependent word. We use MINIPAR data set. The subset is so selected to minimize the (Lin 1998), an automated dependency parser, as our effect of other factors, such as word-level information, in the performance measurement process. A brief dis- primary source of dependency information. Its output cussion compares the dependency-based approach to is further processed to obtain dependencies triplets, and other, similar approaches. to filter out irrelevant information. Details about how to map the Hypothesis and Text into sets of dependen- cies are provided in subsequent sections. Introduction The major advantage of using dependencies over The task of textual entailment is to decide whether a word-based similarity measures, such as the one de- text fragment the size of a sentence, called the Text scribed in (Monz & de Rijke 2001), is that they capture (T), can logically infer another text of same or smaller syntactic information which is important in fine lan- size, called the Hypothesis (H). guage understanding tasks, such as textual entailment. Entailment has received a lot of attention since it Syntactic relations are important to decide that Yahoo was proposed under the Recognizing Textual Entail- took over Overture entails Yahoo bought Overture (ex- ment (RTE) Challenge (Dagan, Glickman, & Magnini ample from first RTE Challenge task description) and 2004 2005) in 2004. In our experiments presented here, does not entail Overture bought Yahoo. Using depen- we use the standard data set that RTE offers for devel- dencies the latter is rejected simply because the subject opment and comparison purposes. Below, we illustrate of buying is Overture which contradicts the Text where the Text and Hypothesis of pair 2028 from the RTE Overture is the object of took over while the subject is test. Yahoo. Text: Besancon is the capital of France’s watch and Another advantage of using a dependency-parser in- clock-making industry and of high precision stead of a phrase parser (which returns phrases hierar- engineering. chically organized in a parse tree) is its applicability to Hypothesis: Besancon is the capital of France. a larger variety of languages. It is known that a phrase- based parser is not applicable to free-order languages. For this particular pair the entailment decision is For our dependency-based representation of syntactic FALSE since H cannot be logically inferred from T. information all it takes to port the system to a new Textual Entailment is a hard task that requires lin- language is to train the dependency extractor for that guistic, world and domain knowledge to be solved. In particular language and plug-it in our system, by sim- this paper, we study the impact of dependency rela- ply replacing MINIPAR. tions on the task of textual entailment. Both the Text To preview our results we show that dependencies Copyright c 2006, American Association for Artificial In- lead to good results for data where solely lexical/word telligence (www.aaai.org). All rights reserved. information is not enough. 110 The rest of the paper is structured as follows. The template. next section presents related work on the entailment Pazienza and colleagues (Pazienza, Pennacchiotti, & task. Section The Dependency-based Approach de- Zanzotto 2005) use syntactic graph distance approach scribes our approach to solving the entailment task. for the task of textual entailment. Their approach is Section Experiments and Results outlines the two ex- closest to ours. By comparison, we use a different scor- periments we conducted and the results we obtained. ing mechanism and a different set of syntactic relations. A comparative discussion is also included in this sec- Our score is simple and thus easy to compute and easy tion. The paper ends with Conclusions. to interpret. The set of syntactic relations we use is a reduced set of the relations handled by MINIPAR. Related Work Vanderwende and colleagues (Vanderwende, Cough- lin, & Dolan 2005) looked at T-H pairs from the RTE A great number of approaches to entailment have been data that could be solved using “solely” syntactic infor- taken since the RTE (Dagan, Glickman, & Magnini mation. Their broad definition of the meaning of syntax 2004 2005) data was made available, most of them af- spans over things traditional considered in other lin- ter 2004. They range from shallow approaches such as guistic subareas (pronoun resolution is usually part of weighted-bag of words to deeper approaches that rely discourse and not syntax). For example, it includes ar- on theorem proving and world knowledge. gument structure, pronoun resolution and alternations. It is not the purpose of this paper to carefully re- They claim that a ’large proportion of the data’, i.e. view these approaches. Rather, we briefly present at- 37% of the test pairs, can be handled with syntax alone tempts prior to the RTE Challenge and discuss work and that adding a general-purpose thesaurus can boost that presents detailed analysis of RTE-like entailment, the performance to 49%. The claims are based on two noting that it is difficult to get a comprehensive analysis human annotators who examined the data manually. of particular aspects of recently analyzed entailment. Bar-Haim and colleagues (Bar-Haim, Szpektor, & In one of the earliest explicit treatments of entailment Glickman 2005) present a very interesting conceptual Monz and de Rijke (Monz & de Rijke 2001) proposed analysis of entailment at lexical and syntactic level. a weighted bag of words approach to entailment. They They adopt same broad definition of syntax as Van- argued that traditional inference systems based on first derwende and colleagues. They found that paraphrases order logic are restricted to yes/no decisions when it are important and that a lexico-syntactic model out- comes to entailment tasks while their approach deliv- performs a lexical model. Both the lexico-syntactic and ered ’graded outcomes’. They established entailment lexical models have low recall. Their work too, is based relations among larger pieces of text - on average seg- on manual processing of the test data. ments of 4 sentences - than the proposed RTE setup We present here a fully automated system that im- where the text size is a sentence (seldom two) or part plements a dependency-based approach to entailment. of a sentence (phrase). We answer two questions: what is the performance of a A closely related effort is presented in (Moldovan dependency-based approach and what is the impact of & Rus 2001). They show how to use unification and dependencies on the performance of the system on top matching to address the answer correctness problem. of word-similarity approaches. Answer correctness can be viewed as entailment: Is a candidate answer entailing the ideal answer to the The Dependency-based Approach question? Initially, the question is paired with an an- Before we proceed let us remind the reader that the swer from a list of candidate answers (obtained through entailment task as defined by RTE is a binary classifi- some keyword proximity and shallow semantics meth- cation task in which the output can have two values: ods). The resulted pair is mapped into a first-order TRUE, meaning the Text entails the Hypothesis, or logic representation and a unification process between FALSE, otherwise. the question and the answer follows. As a back-off step, Our approach starts by mapping both the Text and for the case when no full unification is possible, the an- the Hypothesis into sets of dependencies. It continues swer with highest unification score is top ranked. The with a step in which it computes how many dependen- task they describe is different than the RTE task be- cies in the Hypothesis are present in the Text. The re- cause a list of candidate answers to rank are available. sult is normalized by the total number of dependencies The granularity of candidate answers and questions is in the Hypothesis leading to an entailment score. If all similar to the RTE data. H-dependencies are present, it means the Hypothesis is Recently, Dagan and Glickman (Dagan & Glickman syntactically (and lexically since the related words in a 2004) presented a probabilistic approach to textual en- dependency need to match also) contained by the Text tailment based on lexico-syntactic structures. They use and thus we can conclude the Text entails the Hypothe- a knowledge base with entailment patterns and a set of sis.