Using Coreference for Question Answering Thomas S. Morton Department of Computer and Information Science University of Pennslyvania t sm6rton@cis, upenn, edu Abstract Sentences are then raaked and presented to the We present a system which retrieves answers to user. We only examine queries to which an- queries based on coreference relationships be- swers are likely to be stated in a sentence or tween entities and events in the query and doc- noun phrase since answers which are typically uments. An evaluation of this system is given longer are can be difficult to annotate reliably. which demonstrates that the the amount of in- This technology differs from the standard docu- formation that the user must process on aver- ment ranking task in that, if successful the user age, tQ find an answer to their query, is reduced will likely not need to examine any of the re- by an order of magnitude. trieved documents in their entirety. This also differs from the document summarization, pro- 1 Introduction vided by many Search engines today, in that the Search engines have become ubiquitous as a sentences selected are influenced by the query means for accessing information. When a rank- and are selected across multiple documents. ing of documents is returned by a search en- We view a system such as ours as providing gine the information retrieval task is usually not a secondary level of processing after a small set complete. The document, as a unit of informa- of documents, which the user believes contain tion, is often too large for many users informa- the information desired, have been found. This tion needs and finding information within the first step would likely be provided by a tradi- set of returned documents poses a burden of its tional search engine, thus this technology serves own. Here we examine a technique for extract- as an enhancement to an existing document re- ing sentences from documents which attempts trieval systems rather than a replacement. Ad- to satisfy the users information needs by provid- vancements in document retrieval would only ing an answer to the query presented. The sys- help the performance of a system such as ours tem does this by modeling coreference relation- as these improvements would increase the like- ships between entities and events in the query lihood that the answer to the user's query is in and documents. An evaluation of this system is one of the top ranked documents returned. given which demonstrates that it performs bet- ter than using a standard tf. idf weighting and 3 Approach that the amount of information that the user A query is viewed as identifying a relation to must process on average, to find an answer to which a user desires a solution. This relation their query, is reduced by an order of magnitude will most likely involve events and entities, and over document ranking alone. an answer to this relation will involve the same events and entities. Our approach attempts to 2 Problem Statement find coreference relationships between the enti- A query indicates an informational need by the ties and events evoked by the query and those user to the search engine. The information re- evoked in the document. Based on these rela- quired may take the form of a sentence or even tionships, sentences are ranked, and the highest a noun phrase. Here the task is to retrieve the ranked sentences are displayed to the user. passage of text which contains the answer to The coreference relationships that are mod- the query from a small collection of documents. eled by this system include identity, part-whole, 85 and synonymy relations. Consider the following 4 Implementation query and answer pairs. The relationships above are captured by a num- Query: What did Mark McGwire say ber of different techniques which can be placed about child abuse? in essentially two categories. The first group Sentence: "What kills me is that you finds identity relationships between different in- know there are kids over there who vocations of the same entity in a document. The are being abused or neglected, you second identifies more loosely defined relation- just don't know which ones" McGwire ships such as part-whole and synonymy. Each of says. the relationships identified is given a weight and based on the weights and relationships them- In the above query answer pair the system at- selves sentences are ranked and presented to the tempts to capture the identity relationship be- user. tween Mark McGwire and McGwire by deter- mining that the term McGwire in this sentence 4.1 Identity Relationships is coreferent with a mention of Mark McGwire earlier in the document.. This allows the sys- Identity relationships are first determined be- tem to rank this sentence equivalently to a sen- tween the string instantiations of entities in sin- tence mentioning the full name. The system gle documents. This is done so that the dis- also treats the term child abuse as a nominaliza- course context in which these strings appear tion which allows it to speculate that the term can be taken into account. The motivation for abused in the sentence is a related event. Finally this comes in part from example texts where the the verb neglect occurs frequently within doc- same last name will be used to refer to differ- uments which contain the verb abuse, which is ent individuals in the same family. This is of- nominalized in the query, so this term is treated ten unambiguous because full names are used in as a related event. The system does not cur- previous sentences, however this requires some rently have a mechanism which tries to capture modeling of which entities are most salient in the relationship between kids and children. the discourse. These relations are determined using techniques described in (Baldwin et al., Query: Why did the U.S. bomb Su- 1998). dan? Another source of identity relationships Sentence: Last month, the United is morphological and word order variations. States launched a cruise missile at- Within noun phrases in the query the sys- tack against the Shifa Pharmaceuti- tem constructs other possible word combina- cal Industries plant in Khartoum, al- tions which contain the head word of the noun leging that U.S. intelligence agencies phrase. For example a noun phrase such as "the have turned up evidence - including photographed little trouper" would be extended soil samples - showing that the plant to include "the photographed trouper", "the lit- was producing chemicals which could tle tropper", and "the trouper" as well as vari- be used to make VX, a deadly nerve ations excluding the determiner. Each of the gas. variations is given a weight based on the ratio of the score that the new shorter term would have In this example one of the entity-based relation- received if it had appeared in the query and the ships of interest is the identity relationship be- actual noun phrase that occured. The morpho- tween U.S. and United States. Also of interest is logical roots of single word variations are also the part-whole relationship between Sudan and added to the list a possible terms which refer Khartoum, it's capital. Finally the bomb event to the entity or event with no additional deduc- is related to the launch/attack event. The sys- tion in weighting. Finally query entities which tem does not currently have a mechanism which are found in an acronym database are added to tries to capture the relationship between Why the list of corefering terms as well with a weight and alleging or evidence. of 1. 86 4.2 Part-Whole and Synonymy where weightwl is the weight assigned during Relationships one of the previous term expansion phases and The system captures part-wt~ole and synonymy idf is defined above. The weightwl function is relationships by examining co-occurrence statis- defined to be 0 for any term w2 for which no tics between certain classes of words. Specif- expansion took place. The score for the a par- ically co-occurrence statistics are gathered on ticular entity or event in the document with re- verbs and nominalization which co-occur much spect to an entity or event in the query is the more often then one would expect based on maximum value of S(Wl,W2) over all values of chance alone. This is also done for proper Wl and w2 for that entity or event. A particular nouns. For each verbal pair or proper noun pair sentence's score is computed as the sum of the the mutual information between the two is com- scores of the set of entities and events it evokes. puted as follows: For the purpose of evaluation a baseline sys- tem was also constructed. This system fol- I(wl, w2) " " p(Wl' w2) lowed a more standard information retrieval ap- = ,ogtf l)p -C2 )) proach to text ranking described in (Salton, 1989). Each token in the the query is assigned where Wl and w2 are words and an event is de- an idf score also based on the same corpus of fined as a word occuring in a document. All Wall Street Journal articles as used with the words w2 for which I(wl, w2) exceeds a thresh- other system. Query expansion simply con- old where Wl is a query term are added to the sisted of stemming the tokens using a version Of list of terms with which the query term can be the Porter stemmer and sentences were scored referred to. This relationship is given with a as a sum of all matching terms, giving the fa- weight of I(wl, w2)/N where N is a normaliza- miliar t f .