Finding Contradictions in Text
Total Page:16
File Type:pdf, Size:1020Kb
Finding Contradictions in Text Marie-Catherine de Marneffe, Anna N. Rafferty and Christopher D. Manning Linguistics Department Computer Science Department Stanford University Stanford University Stanford, CA 94305 Stanford, CA 94305 [email protected] {rafferty,manning}@stanford.edu Abstract matics where protein-protein interaction is widely studied, automatically finding conflicting facts about Detecting conflicting statements is a foun- such interactions would be beneficial. dational text understanding task with appli- Here, we shed light on the complex picture of con- cations in information analysis. We pro- pose an appropriate definition of contradiction tradiction in text. We provide a definition of contra- for NLP tasks and develop available corpora, diction suitable for NLP tasks, as well as a collec- from which we construct a typology of con- tion of contradiction corpora. Analyzing these data, tradictions. We demonstrate that a system for we find contradiction is a rare phenomenon that may contradiction needs to make more fine-grained be created in different ways; we propose a typol- distinctions than the common systems for en- ogy of contradiction classes and tabulate their fre- tailment. In particular, we argue for the cen- quencies. Contradictions arise from relatively obvi- trality of event coreference and therefore in- corporate such a component based on topical- ous features such as antonymy, negation, or numeric ity. We present the first detailed breakdown mismatches. They also arise from complex differ- of performance on this task. Detecting some ences in the structure of assertions, discrepancies types of contradiction requires deeper inferen- based on world-knowledge, and lexical contrasts. tial paths than our system is capable of, but (1) Police specializing in explosives defused the rock- we achieve good performance on types arising ets. Some 100 people were working inside the plant. from negation and antonymy. (2) 100 people were injured. This pair is contradictory: defused rockets cannot go 1 Introduction off, and thus cannot injure anyone. Detecting con- In this paper, we seek to understand the ways con- tradictions appears to be a harder task than detecting tradictions occur across texts and describe a system entailments. Here, it is relatively easy to identify the for automatically detecting such constructions. As a lack of entailment: the first sentence involves no in- foundational task in text understanding (Condoravdi juries, so the second is unlikely to be entailed. Most et al., 2003), contradiction detection has many possi- entailment systems function as weak proof theory ble applications. Consider applying a contradiction (Hickl et al., 2006; MacCartney et al., 2006; Zan- detection system to political candidate debates: by zotto et al., 2007), but contradictions require deeper drawing attention to topics in which candidates have inferences and model building. While mismatch- conflicting positions, the system could enable voters ing information between sentences is often a good to make more informed choices between candidates cue of non-entailment (Vanderwende et al., 2006), and sift through the amount of available informa- it is not sufficient for contradiction detection which tion. Contradiction detection could also be applied requires more precise comprehension of the conse- to intelligence reports, demonstrating which infor- quences of sentences. Assessing event coreference mation may need further verification. In bioinfor- is also essential: for texts to contradict, they must 1039 Proceedings of ACL-08: HLT, pages 1039–1047, Columbus, Ohio, USA, June 2008. c 2008 Association for Computational Linguistics refer to the same event. The importance of event useful, a looser definition that more closely matches coreference was recognized in the MUC information human intuitions is necessary; contradiction occurs extraction tasks in which it was key to identify sce- when two sentences are extremely unlikely to be true narios related to the same event (Humphreys et al., simultaneously. Pairs such as Sally sold a boat to 1997). Recent work in text understanding has not John and John sold a boat to Sally are tagged as con- focused on this issue, but it must be tackled in a suc- tradictory even though it could be that each sold a cessful contradiction system. Our system includes boat to the other. This definition captures intuitions event coreference, and we present the first detailed of incompatiblity, and perfectly fits applications that examination of contradiction detection performance, seek to highlight discrepancies in descriptions of the on the basis of our typology. same event. Examples of contradiction are given in table 1. For texts to be contradictory, they must in- 2 Related work volve the same event. Two phenomena must be con- Little work has been done on contradiction detec- sidered in this determination: implied coreference tion. The PASCAL Recognizing Textual Entailment and embedded texts. Given limited context, whether (RTE) Challenges (Dagan et al., 2006; Bar-Haim two entities are coreferent may be probable rather et al., 2006; Giampiccolo et al., 2007) focused on than certain. To match human intuitions, compatible textual inference in any domain. Condoravdi et al. noun phrases between sentences are assumed to be (2003) first recognized the importance of handling coreferent in the absence of clear countervailing ev- entailment and contradiction for text understanding, idence. In the following example, it is not necessary but they rely on a strict logical definition of these that the woman in the first and second sentences is phenomena and do not report empirical results. To the same, but one would likely assume it is if the two our knowledge, Harabagiu et al. (2006) provide the sentences appeared together: first empirical results for contradiction detection, but (1) Passions surrounding Germany’s final match turned they focus on specific kinds of contradiction: those violent when a woman stabbed her partner because featuring negation and those formed by paraphrases. she didn’t want to watch the game. They constructed two corpora for evaluating their (2) A woman passionately wanted to watch the game. system. One was created by overtly negating each We also mark as contradictions pairs reporting con- entailment in the RTE2 data, producing a bal- tradictory statements. The following sentences refer anced dataset (LCC negation). To avoid overtrain- to the same event (de Menezes in a subway station), ing, negative markers were also added to each non- and display incompatible views of this event: entailment, ensuring that they did not create con- (1) Eyewitnesses said de Menezes had jumped over the tradictions. The other was produced by paraphras- turnstile at Stockwell subway station. ing the hypothesis sentences from LCC negation, re- (2) The documents leaked to ITV News suggest that moving the negation (LCC paraphrase): A hunger Menezes walked casually into the subway station. strike was not attempted → A hunger strike was called off. They achieved very good performance: This example contains an “embedded contradic- accuracies of 75.63% on LCC negation and 62.55% tion.” Contrary to Zaenen et al. (2005), we argue on LCC paraphrase. Yet, contradictions are not lim- that recognizing embedded contradictions is impor- ited to these constructions; to be practically useful, tant for the application of a contradiction detection any system must provide broader coverage. system: if John thinks that he is incompetent, and his boss believes that John is not being given a chance, 3 Contradictions one would like to detect that the targeted information in the two sentences is contradictory, even though 3.1 What is a contradiction? the two sentences can be true simultaneously. One standard is to adopt a strict logical definition of contradiction: sentences A and B are contradictory 3.2 Typology of contradictions if there is no possible world in which A and B are Contradictions may arise from a number of different both true. However, for contradiction detection to be constructions, some overt and others that are com- 1040 ID Type Text Hypothesis 1 Antonym Capital punishment is a catalyst for more crime. Capital punishment is a deterrent to crime. 2 Negation A closely divided Supreme Court said that juries and The Supreme Court decided that only not judges must impose a death sentence. judges can impose the death sentence. 3 Numeric The tragedy of the explosion in Qana that killed more An investigation into the strike in Qana than 50 civilians has presented Israel with a dilemma. found 28 confirmed dead thus far. 4 Factive Prime Minister John Howard says he will not be Australia withdraws from Iraq. swayed by a warning that Australia faces more terror- ism attacks unless it withdraws its troops from Iraq. 5 Factive The bombers had not managed to enter the embassy. The bombers entered the embassy. 6 Structure Jacques Santer succeeded Jacques Delors as president Delors succeeded Santer in the presi- of the European Commission in 1995. dency of the European Commission. 7 Structure The Channel Tunnel stretches from England to The Channel Tunnel connects France France. It is the second-longest rail tunnel in the and Japan. world, the longest being a tunnel in Japan. 8 Lexical The Canadian parliament’s Ethics Commission said The Canadian parliament’s Ethics former immigration minister, Judy Sgro, did nothing Commission accuses Judy Sgro. wrong and her staff had put her into a conflict of in- terest. 9 Lexical In the election, Bush called for U.S. troops to be with- He cites such missions as an example of drawn from the peacekeeping mission in the Balkans. how America must “stay the course.” 10 WK Microsoft Israel, one of the first Microsoft branches Microsoft was established in 1989. outside the USA, was founded in 1989. Table 1: Examples of contradiction types. plex even for humans to detect. Analyzing contra- to find the contradiction in example 8 (table 1), diction corpora (see section 3.3), we find two pri- it is necessary to learn that X said Y did nothing mary categories of contradiction: (1) those occur- wrong and X accuses Y are incompatible.