TALN ’99, The importance of annotated corpora for NLP 1 The importance of annotated corpora for NLP: the cases of anaphora resolution and clause splitting Ruslan Mitkov, Constantin Orasan and Richard Evans SLES, University of Wolverhampton, Stafford Street, Wolverhampton, WV1 1SB {r.mitkov, in6093, in6087}@wlv.ac.uk http://www.wlv.ac.uk/sles/compling/ Abstract In this paper we present two applications that depend on annotated corpora for their implementation, evaluation and improvement. The first is an automatic anaphora resolution system. After describing the algorithm we discuss the importance of corpora for the tasks of evaluation and automatic scoring and the development of a coreferentially annotated corpus. We go on to look ahead at the role of corpora in optimisation and semi-automatic annotation. The second task investigates the use of an annotated corpus with a machine learning algorithm for clause splitting. We show that the method minimises the number of hand made rules necessary to achieve a good result. 1. Introduction Corpus-based research has been the major driving force in recent NLP developments. Computational Linguists use corpora among other things, to observe (and propose) linguistic hypotheses (rules), to optimise them and to finally evaluate them (or the approaches based on those rules). This paper will discuss two NLP applications which heavily exploit corpora: anaphora resolution and clause splitting. The paper is structured as follows. The first section will address the use of corpora in anaphora resolution. To start with, our robust, knowledge-poor anaphora resolution approach will be briefly outlined. The core of this approach is the set of so-called antecedent indicators - anaphora resolution factors, which have been derived from corpus-based empirical analysis. Section 2 will also propose a new evaluation package for anaphora resolution and will discuss the evaluation of the approach carried out so far. It will also explain how annotated corpora will be used to optimise the performance of the system: the availability of coreferentially annotated corpora is vital for automatic optimisation and evaluation, and this section will describe our work on the development of a coreferentially annotated corpus. Section 3 of the paper will outline how we use corpora for the task of clause splitting. We have recently launched a project on corpus-based clause splitting and we shall report the progress of our program for automatic clause segmentation, its preliminary evaluation and plans for optimisation. 2 Ruslan Mitkov, Constantin Orasan and Richard Evans 2. Anaphora resolution 2.1 Our robust, knowledge-poor approach: a brief outline With a view to avoiding complex syntactic, semantic and discourse analysis (which is vital for real-world applications), we developed a robust, knowledge-poor and corpus-based ap- proach to pronoun resolution which does not parse and analyse the input in order to identify antecedents of anaphors1. Our robust approach works as follows: it takes as an input the output of a text processed by a part-of-speech tagger, identifies the noun phrases which precede the anaphor within a distance of 2 sentences, checks them for gender and number agreement with the anaphor and then applies the so-called antecedent indicators to the remaining candidates by assigning a positive or negative score (-1, 0, 1 or 2). The noun phrase with the highest aggregate score is proposed as antecedent. Some indicators give NPs a bonus and are therefore called boosting indicators (e.g. first noun phrases, lexical reiteration, section heading, collocation pattern preference, immediate reference, referential distance, term preference), whereas others penalise certain NPs and are referred to as impeding indicators (indefiniteness, non-prepositional noun phrase). Most of the indicators are genre- independent and related to coherence phenomena (such as salience and distance) or to structural matches, whereas others are genre-specific (term preference). For instance, indefiniteness considers definite noun phrases preceding the anaphor better candidates for antecedent than indefinite ones and therefore, indefinite noun phrases are penalised by a score of –1. Also, first noun phrases in previous sentences/clauses are deemed good candidates for antecedents and score 1. As a further example, the collocation pattern preference favours candi- dates which have an identical collocation pattern with a pronoun and awards them a score of 2 (e.g. in the example “Press the keyi down and turn the volume up... Press iti again”, the candidate “the key” is a better candidate than “the volume” because of its occurrence in the same collocation pattern as the anaphor “it”). For more information on the indicators and their patterns see (Mitkov 1998). The methodology for selecting the current set of indicators is purely empirical and is based on observation from corpora. So far, we have considered almost exclusively texts from the genre of technical manuals. This variety was chosen because it is relatively homogenous in terms of structure. We started from a hypothetical set of indicators and then observed the syntactic environments associated with pronominal anaphors. In addition, we looked at the error cases when the anaphora resolver was not able to suggest the correct antecedent. Each time a new indicator was proposed, it was tested rigorously against a number of anaphors. The indicators' scores have been determined experimentally and are constantly being updated. The scores used so far should be regarded as preliminary: even though we carried out some analysis with a view to determining more accurate scores, they are far from being optimal. The corpus analysis looked at the ratios: 1. The number of cases in which an indicator X is applied to an NP and this NP is the antecedent, to the number of all applications of the indicator X (for boosting indicators) and 2. The number of cases in which an indicator X is applied to an NP and this NP is not the antecedent, to the number of all applications of the indicator X (for impeding indicators). 1 At the moment pleonastic pronouns are removed manually, but we are working on a program to identify those automatically TALN ’99, The importance of annotated corpora for NLP 3 This analysis helped us to identify indicators with highest relative importance. For instance, the ratio for “immediate reference” was 14/14, for “collocation pattern preference” – 11/12, for “non-prepositional noun phrase” 46/52, for “indefiniteness” – 40/51 and for “section heading preference” – 15/22. Another factor which played a role in determining whether newly proposed indicators should be added to the set of indicators was their frequency of application (Mitkov & Stys, 1997). 2.2 Evaluation As in any other NLP task, evaluation is of crucial importance in anaphora resolution. The MUC (Message Understanding Conference) initiatives suggested the measures "recall" and "precision" be used for evaluating the performance of coreference resolution. It is felt, however, that evaluation in anaphora resolution needs further attention. Measuring the success rate of an anaphora resolution system in terms of "recall" and "precision" is undoubtedly an important (and consistent) step towards assessing the efficiency of anaphora resolution approaches, but "recall" and "precision" cannot be seen as distinct measures for robust systems2. In addition, it appears that they alone cannot provide a comprehensive overall assessment of an approach. In order to see how much a certain approach is "worth", it would be necessary to evaluate it against other "benchmarks", e.g. against other existing or baseline models. In order to evaluate the effectiveness of the approach and explore whether it is superior to the baseline models for anaphora resolution and if so by how much, we also tested all sample texts on (i) a Baseline Model which checks agreement in number and gender and, where more than one candidate remains, picks as antecedent the most recent subject matching the gender and number of the anaphor (we shall refer to it as "Baseline Subject") and (ii) a Baseline Model which picks out as antecedent the most recent noun phrase that matches the gender and number of the anaphor (we shall refer to it as "Baseline Most Recent"). We have also introduced the measure "critical success rate" which exclusively accounts for the performance of the antecedent indicators since it is associated only with those anaphors which still have more than one candidate for antecedent after gender and number filters i.e. anaphors whose antecedents can be tracked down only on the basis of the antecedent indicators. The evaluation shows that the results are comparable to and even better than syntax-based methods (Lappin & Leass 1994). The evaluation results also show superiority over other knowledge-poor methods (Baldwin 1997; see also below)3. We believe that the good success rate is due to the fact that a number of antecedent indicators are taken into account and no factor is given absolute preference. In particular, this strategy can often override incorrect decisions linked with strong centering preference or syntactic and semantic parallelism preferences (Mitkov 1998) The evaluation corpus included texts from different technical manuals (Minolta Photocopier, Portable Style Writer, Alba Twin Speed Video Recorder, Seagate Medalist Hard 2 In anaphora resolution, the recall would be defined as the ratio number of correctly resolved anaphors / number of all anaphors