Improving coreference resolution with automatically predicted prosodic information Ina Rosiger¨ ∗, Sabrina Stehwien∗, Arndt Riester, Ngoc Thang Vu Institute for Natural Language Processing University of Stuttgart, Germany roesigia,stehwisa,arndt,thangvu @ims.uni-stuttgart.de { } Abstract there exist a few systems for pronoun resolution in transcripts of spoken text (Strube and Muller¨ , Adding manually annotated prosodic in- 2003; Tetreault and Allen, 2004). It has been formation, specifically pitch accents and shown that there are differences between written phrasing, to the typical text-based feature and spoken text that lead to a drop in performance set for coreference resolution has previ- when coreference resolution systems developed ously been shown to have a positive effect for written text are applied on spoken text (Amoia on German data. Practical applications on et al., 2012). For this reason, it may help to use spoken language, however, would rely on additional information available from the speech automatically predicted prosodic informa- signal, for example prosody. tion. In this paper we predict pitch ac- In West-Germanic languages, such as English cents (and phrase boundaries) using a con- and German, there is a tendency for coreferent volutional neural network (CNN) model items, i.e. entities that have already been intro- from acoustic features extracted from the duced into the discourse (their information sta- speech signal. After an assessment of the tus is given), to be deaccented, as the speaker as- quality of these automatic prosodic anno- sumes the entity to be salient in the listener’s dis- tations, we show that they also signifi- course model (cf. Terken and Hirschberg(1994); cantly improve coreference resolution. Baumann and Riester(2013); Baumann and Roth (2014)). We can make use of this fact by providing 1 Introduction prosodic information to the coreference resolver. Noun phrase coreference resolution is the task of Example (2), this time marked with prominence grouping noun phrases (NPs) together that refer information, shows that prominence can help us resolve cases where the transcription is potentially to the same discourse entity in a text or dialogue. 2 In Example (1), taken from Umbach(2002), the ambiguous . question for the coreference resolver, besides link- (2) John 1 has an old cottage 2. ing the anaphoric pronoun he back to John, is to { } { } a. Last year he reconstructed the decide whether an old cottage and the shed refer { }1 { SHED . to the same entity. }3 b. Last year he 1 reconSTRUCted the { } (1) John has an old cottage . shed 2. { }1 { }2 } Last year he reconstructed the shed . { }1 { }? The pitch accent on shed in (2a) leads to the in- Coreference resolution is an active NLP research terpretation that the shed and the cottage refer to area, with its own track at most NLP conferences different entities, where the shed is a part of the and several shared tasks such as the CoNLL or cottage (they are in a bridging relation). In con- SemEval shared tasks (Pradhan et al., 2012; Re- trast, in (2b), the shed is deaccented, which sug- casens et al., 2010) or the CORBON shared task gests that the shed and the cottage corefer. 20171. Almost all work is based on text, although A pilot study byR osiger¨ and Riester(2015) has *The two first authors contributed equally to this work. 2The anaphor under consideration is typed in boldface, its 1http://corbon.nlp.ipipan.waw.pl/ antecedent is underlined. Accented syllables are capitalised. 78 Proceedings of the First Workshop on Speech-Centric Natural Language Processing, pages 78–83 Copenhagen, Denmark, September 7–11, 2017. c 2017 Association for Computational Linguistics shown that enhancing the text-based feature set 3 Data for a coreference resolver, consisting of e.g. auto- To ensure comparability, we use the same dataset matic part-of-speech (POS) tags and syntactic in- as in the pilot study, namely the DIRNDL cor- formation, with pitch accents and prosodic phras- pus (Eckart et al., 2012; Bjorkelund¨ et al., 2014), ing information helps to improve coreference res- a German radio news corpus annotated with both olution of German spoken text. The prosodic la- manual coreference and manual prosody labels. bels used in the experiments were annotated man- We adopt the official train, test and development ually, which is not only expensive but not applica- split4 designed for research on coreference res- ble in an automatic pipeline setup. In our paper, olution. The recorded news broadcasts in the we present an experiment in which we replicate DIRNDL-anaphora corpus were spoken by 13 the main results from the pilot study by annotating male and 7 female speakers, in total roughly 5 the prosodic information automatically, thus omit- hours of speech. The prosodic annotations follow ting any manual annotations from the feature set. the GToBI(S) standard for pitch accent types and We show that adding prosodic information signif- boundary tones and are described in Bjorkelund¨ icantly helps in all of our experiments. et al.(2014). In this study we make use of two 2 Prosodic features for coreference class labels of prosodic events: all accent types resolution (marked by the standard ToBI *) grouped into a single class (pitch accent presence) and the same Similar to the pilot study, we make use of pitch ac- for intonational phrase boundaries (marked by %). cents and prosodic phrasing. We predict the pres- 3 ence of a pitch accent and use phrase boundaries 4 Automatic prosodic information to derive nuclear accents, which are taken to be the last (and perceptually most prominent) accent In this section we describe the prosodic event de- in an intonation phrase. This paper tests whether tector used in this work. It is a binary classifier previously reported tendencies for manual labels that is trained separately for either pitch accents are also observable for automatic labels, namely: or phrase boundaries and predicts for each word, whether it carries the respective prosodic event. Short NPs Since long, complex NPs almost al- ways have at least one pitch accent, the presence 4.1 Model and the absence of a pitch accent is more helpful We apply a convolutional neural network (CNN) for shorter phrases. model, illustrated in Figure1. The input to the Long NPs For long, complex NPs, we look for CNN is a matrix spanning the current word and its nuclear accents that indicate the phrase’s overall right and left context word. The input matrix is prominence. If the NP contains a nuclear accent, a frame-based representation of the speech signal. it is assumed to be less likely to take part in The signal is divided into overlapping frames for coreference chains. each 20 ms with a 10 ms shift and are represented by a 6-dimensional feature vector for each frame. We test the following features that have proven We use acoustic features as well as position in- beneficial in the pilot study. These features are de- dicator features following Stehwien and Vu(2017) rived for each NP. that are simple and fast to obtain. The acoustic features were extracted from the speech signal us- Pitch accent presence focuses on the presence ing the OpenSMILE toolkit (Eyben et al., 2013). of a pitch accent, disregarding its type. If one ac- The feature set consists of 5 features that comprise cent is present in the NP, this boolean feature gets acoustic correlates of prominence: smoothed fun- assigned the value true, and false otherwise. damental frequency (f0), RMS energy, PCM loud- Nuclear accent presence is a boolean feature ness, voicing probability and Harmonics-to-Noise comparable to pitch accent presence. It gets as- Ratio. The position indicator feature is appended signed the value true if there is a nuclear accent as an extra feature to the input matrices (see Fig- present in the NP, false otherwise. ure1) and aids the modelling of the acoustic con- 3We do not predict the pitch accent type (e.g. fall H*L or 4http://www.ims.uni-stuttgart.de/ rise L*H) as this distinction was not helpful in the pilot study forschung/ressourcen/korpora/dirndl. and is generally more difficult to model. en.html 79 News Corpus (Ostendorf et al., 1995) that contains speech from 3 female and 2 male speakers and that includes manually labelled pitch accents and into- national phrase boundary tones. Hence, both cor- pora consist of read speech by radio news anchors. The prediction accuracy on the DIRNDL anaphora corpus is 81.9% for pitch accents and 85.5% for intonational phrase boundary tones6. The speaker- independent performance of this model on the Boston dataset is 83.5% accuracy for pitch accent detection and 89% for phrase boundary detection. We conclude that the prosodic event detector gen- eralises well to the DIRNDL dataset and the ob- tained accuracies are appropriate for our experi- ments. 5 Coreference resolution In this section, we describe the coreference re- solver used in our experiments and how it was ap- plied to create the baseline system using only au- tomatic annotations. 5.1 IMS HotCoref DE Figure 1: CNN for prosodic event recognition with The IMS HotCoref DE coreference resolver is an input window of 3 successive words and posi- a state-of-the-art tool for German7 (Rosiger¨ and tion indicating features. Kuhn, 2016). It is data-driven, i.e. it learns from annotated data with the help of pre-defined fea- text by indicating which frames belong to the cur- tures using a structured perceptron that models rent word or the neighbouring words. coreference within a document as a directed tree. We apply two convolution layers in order to ex- This way, it can exploit the tree structure to create pand the input information and then use max pool- non-local features (features that go beyond a pair ing to find the most salient features.