It’s a Contradiction—No, it’s Not: A Case Study using Functional Relations Alan Ritter, Doug Downey, Stephen Soderland and Oren Etzioni Turing Center Department of Computer Science and Engineering University of Washington Box 352350 Seattle, WA 98195, USA {aritter,ddowney,soderlan,etzioni}@cs.washington.edu Abstract 1.1 Contradictions and World Knowledge Contradiction Detection (CD) in text is a Our paper is motivated in part by de Marneffe et al.’s difficult NLP task. We investigate CD work, but with some important differences. First, over functions (e.g., BornIn(Person)=Place), we introduce a simple logical foundation for the CD and present a domain-independent algorithm task, which suggests that extensive world knowl- that automatically discovers phrases denoting edge is essential for building a domain-independent functions with high precision. Previous work CD system. Second, we automatically generate a on CD has investigated hand-chosen sentence large corpus of apparent contradictions found in ar- pairs. In contrast, we automatically harvested bitrary Web text. We show that most of these appar- from the Web pairs of sentences that appear contradictory, but were surprised to find that ent contradictions are actually consistent statements most pairs are in fact consistent. For example, due to meronyms (Alan Turing was born in London “Mozart was born in Salzburg” does not con- and in England), synonyms (George Bush is mar- tradict “Mozart was born in Austria” despite ried to both Mrs. Bush and Laura Bush), hypernyms the functional nature of the phrase “was born (Mozart died of both renal failure and kidney dis- in”. We show that background knowledge ease), and reference ambiguity (one John Smith was about meronyms (e.g., Salzburg is in Austria), born in 1997 and a different John Smith in 1883). synonyms, functions, and more is essential for success in the CD task. Next, we show how background knowledge enables a CD system to discard seeming contradictions and focus on genuine ones. 1 Introduction and Motivation De Marneffe et al. introduced a typology of con- tradiction in text, but focused primarily on contra- Detecting contradictory statements is an important dictions that can be detected from linguistic evi- and challenging NLP task with a wide range of dence (e.g. negation, antonymy, and structural or potential applications including analysis of politi- lexical disagreements). We extend their analysis to cal discourse, of scientific literature, and more (de a class of contradictions that can only be detected Marneffe et al., 2008; Condoravdi et al., 2003; utilizing background knowledge. Consider for ex- Harabagiu et al., 2006). De Marneffe et al. present a ample the following sentences: model of CD that defines the task, analyzes different types of contradictions, and reports on a CD system. 1) “Mozart was born in Salzburg.” They report 23% precision and 19% recall at detect- 2) “Mozart was born in Vienna.” ing contradictions in the RTE-3 data set (Voorhees, 3) “Mozart visited Salzburg.” 2008). Although RTE-3 contains a wide variety of 4) “Mozart visited Vienna.” contradictions, it does not reflect the prevalence of Sentences 1 & 2 are contradictory, but 3 & 4 are seeming contradictions and the paucity of genuine not. Why is that? The distinction is not syntactic. contradictions, which we have found in our corpus. Rather, sentences 1 and 2 are contradictory because 11 Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 11–20, Honolulu, October 2008. c 2008 Association for Computational Linguistics the relation expressed by the phrase “was born in” delegated to TEXTRUNNER or simply ignored. Nev- can be characterized here as a function from peo- ertheless, extracted tuples are a convenient approxi- ple’s names to their unique birthplaces. In contrast, mation of sentence content, which enables us to fo- “visited” does not denote a functional relation.1 cus on function detection and function-based CD. We cannot assume that a CD system knows, in Our contributions are the following: advance, all the functional relations that might ap- • We present a novel model of the Contradiction pear in a corpus. Thus, a central challenge for a Detection (CD) task, which offers a simple log- function-based CD system is to determine which re- ical foundation for the task and emphasizes the lations are functional based on a corpus. Intuitively, central role of background knowledge. we might expect that “functional phrases” such as “was born in” would typically map person names • We introduce and evaluate a new EM-style al- to unique place names, making function detection gorithm for detecting whether phrases denote easy. But, in fact, function detection is surprisingly functional relations and whether nouns (e.g., difficult because name ambiguity (e.g., John Smith), “dad”) are ambiguous, which enables a CD sys- common nouns (e.g., “dad” or “mom”), definite de- tem to identify functions in arbitrary domains. scriptions (e.g., “the president”), and other linguistic • We automatically generate a corpus of seem- phenomena can mask functions in text. For example, ing contradictions from Web text, and report the two sentences “John Smith was born in 1997.” on a set of experiments over this corpus, which and “John Smith was born in 1883.” can be viewed provide a baseline for future work on statistical as either evidence that “was born in” does not de- function identification and CD. 2 note a function or, alternatively, that “John Smith” is ambiguous. 2 A Logical Foundation for CD On what basis can a CD system conclude that two 1.2 A CD System Based on Functions statements T and H are contradictory? Logically, We report on the AUCONTRAIRE CD system, which contradiction holds when T |= ¬H. As de Marneffe addresses each of the above challenges. First, AU- et al. point out, this occurs when T and H contain CONTRAIRE identifies “functional phrases” statis- antonyms, negation, or other lexical elements that tically (Section 3). Second, AUCONTRAIRE uses suggest that T and H are directly contradictory. But these phrases to automatically create a large cor- other types of contradictions can only be detected pus of apparent contradictions (Section 4.2). Fi- with the help of a body of background knowledge nally, AUCONTRAIRE sifts through this corpus to K: In these cases, T and H alone are mutually con- find genuine contradictions using knowledge about sistent. That is, synonymy, meronymy, argument types, and ambi- guity (Section 4.3). T |=\ ¬H ∧ H |=\ ¬T Instead of analyzing sentences directly, AUCON- T H TRAIRE relies on the TEXTRUNNER Open Informa- A contradiction between and arises only in tion Extraction system (Banko et al., 2007; Banko the context of K. That is: and Etzioni, 2008) to map each sentence to one or more tuples that represent the entities in the sen- ((K ∧ T ) |= ¬H) ∨ ((K ∧ H) |= ¬T ) tences and the relationships between them (e.g., was born in(Mozart,Salzburg)). Using extracted tu- Consider the example of Mozart’s birthplace in ples greatly simplifies the CD task, because nu- the introduction. To detect a contradiction, a CD merous syntactic problems (e.g., anaphora, rela- system must know that A) “Mozart” refers to the tive clauses) and semantic challenges (e.g., quantifi- same entity in both sentences, that B) “was born in” cation, counterfactuals, temporal qualification) are denotes a functional relation, and that C) Vienna and Salzburg are inconsistent locations. 1Although we focus on function-based CD in our case study, we believe that our observations apply to other types of CD as 2The corpus is available at http://www.cs. well. washington.edu/research/aucontraire/ 12 Of course, world knowledge, and reasoning about tional relation appear non-functional. Example B text, are often uncertain, which leads us to associate depicts a distribution of y values that appears less probabilities with a CD system’s conclusions. Nev- functional due to the fact that “John Adams” refers ertheless, the knowledge base K is essential for CD. to multiple, distinct real-world individuals with that We now turn to a probabilistic model that helps name. Finally, example C exhibits evidence for a us simultaneously estimate the functionality of re- non-functional relation. lations (B in the above example) and ambiguity of A. was born in(Mozart, PLACE): argument values (A above). Section 4 describes the Salzburg(66), Germany(3), Vienna(1) remaining components of AUCONTRAIRE. B. was born in(John Adams, PLACE): 3 Detecting Functionality and Ambiguity Braintree(12), Quincy(10), Worcester(8) This section introduces a formal model for comput- C. lived in(Mozart, PLACE): ing the probability that a phrase denotes a function Vienna(20), Prague(13), Salzburg(5) based on a set of extracted tuples. An extracted tuple takes the form R(x, y) where (roughly) x is the sub- Figure 1: Functional relations such as example A have a ject of a sentence, y is the object, and R is a phrase different distribution of y values than non-functional rela- denoting the relationship between them. If the re- tions such as C. However, an ambiguous x argument as in lation denoted by R is functional, then typically the B, can make a functional relation appear non-functional. object y is a function of the subject x. Thus, our dis- cussion focuses on this possibility, though the anal- 3.1 Formal Model of Functions in Text ysis is easily extended to the symmetric case. Logically, a relation R is functional in a vari- To decide whether R is functional in x for all x, able x if it maps it to a unique variable y: we first consider how to detect whether R is lo- ∀x, y1, y2 R(x, y1) ∧ R(x, y2) ⇒ y1 = y2. Thus, cally functional for a particular value of x.