Collocation Extraction: Needs, Feeds and Results of an Extraction System for German

Julia Ritz Institut für Maschinelle Sprachverarbeitung (IMS) Universität Stuttgart Azenbergstr. 12 70174 Stuttgart Germany [email protected]

Abstract in a collocation) (adapted from (Hausmann, 1979; Hausmann, 1989; Hausmann, 2003)). This paper provides a specification of requirements for collocation extraction sys- b. A collocation is a word combination whose tems, taking as an example the extraction semantic and/or syntactic properties cannot of noun + verb collocations from German be fully predicted from those of its compo- texts. A hybrid approach to the extrac- nents, and which therefore has to be listed in tion of habitual collocations and idioms a lexicon (Evert, 2004). is presented, aiming at a detailed description of collocations and their morphosyn- We argue that linguistic knowledge could not tax for natural language generation sys- only improve results (Krenn, 2000b; Smadja, tems as well as to support learner lexicog- 1993) but is essential when extracting colloca- raphy. tions from certain languages: this knowledge provides other applications (or a lexicon user, respec- 1 Introduction tively) with a fine-grained description of how the Since Firth first described collocations as habit- extracted collocations are to be used in context. ual word combinations in the 1950ies (cf. Firth, Additional requirements resulting from the 1968), a number of papers focusing on collocation needs of dictionary users are described in (Haus- extraction have been published (see the overviews mann, 2003; Heid and Gouws, 2005) and are of in (Evert, 2004; Bartsch, 2004)). Most studies interest not only in lexicography but can also be concentrate on the extraction from English. How- transferred to the field of natural language gener- ever, the procedures proposed in these studies can- ation. These requirements influence the develop- not necessarily be applied to other languages as ment of collocation extraction systems, which mo- English stands out, e.g. with respect to configu- tivates this paper. rationality. They rely on the fact that the syntax The structure of the paper is as follows: In chap- of English (and of all configurational languages) ter 2, the requirements, depending on factors like provides positional clues to the grammatical func- the targeted language, are presented. We then dis- tion of noun phrases, and they exploit this con- cuss and suggest methods to meet the given needs. cept by means of window-based, adjacency-based A documentation of ongoing work on the extrac- or pattern-based extraction, combined with associ- tion of noun + verb collocations from German ation measures to identify co-occurrences that are texts is given in chapter 3. Chapter 4 gives a con- more frequent than statistically expectable. What clusion and an outlook on work still to be done. these procedures do not cover is semantic-oriented 2 Collocation Extraction Tools: definitions like (a) and (b). Requirements a. A collocation is a combination of a free (’au- The development of a collocation extraction tool tosematic’) element (the base) and a lexically depends on the following conditions: determined (’synsemantic’) element (the collocate, which may lose (some of) its meaning 1. properties of the targeted language

41 2. the targeted application of statistical analyses are little reliable. To allow a grouping of words sharing the same lemma, lem- 3. the kinds of collocations to be extracted matisation is crucial. 4. the degree of detail 2.2 Application factors Whereas issues 1 to 3 deal with the collocation Other important factors are the targeted applica- itself, issue 4 is focused at the collocation in con- tion (i.e. analysis vs. generation) and, to some ex- text, i.e. its behaviour (from a syntagmatic analy- tent resulting from it, factors (3.) and (4.), above. sis point of view) or, respectively, its use (from a Depending on the purpose of the tool (or lexicon, generation perspective). respectively), the collocation definition chosen as an outline may vary, e.g. including transparent and 2.1 Language factors regular collocations (cf. (Tutin, 2004)) for genera- One of the most important factors is, of course, the tion purposes, but excluding them for analysis pur- targeted language and its main characteristics with poses. In addition, a more detailed description of respect to word formation and word order. De- the use of collocations in context (e.g. information pending on word and constituent order, the pros about preferences with respect to the determiner, and cons of positional vs. relational extraction etc.) is needed for generation purposes than for patterns need to be considered. Positional pat- text analysis. terns (based on adjacency or a ’window’) are ad- equate for configurational languages, but in lan- 2.3 Factors of collocation definition guages with rather free word order, words belong- Collocations can be distinguished on two levels: ing to a phrase or collocation do not necessarily the formal level and the content level. On the for- occur within a predefined span1. mal level, a collocation can be classified accord- Extracting word combinations using relational ing to the structural relation between its elements. patterns (represented by part of speech (PoS) tags Typical patterns are shown in table 12 (taken from or dependency rules) offers a higher level of ab- (Heid and Gouws, 2005)). straction and improves the results (cf. (Krenn, On the content level, there are regular, transpar- 2000b; Smadja, 1993)). However, this requires ent, and opaque collocations (according to (Tutin, part of speech tagging and possibly partial parsing. 2004)) and, taking definition (b) into account, id- A system extracting word combinations by apply- ioms as well. However, as a classification at the ing relational patterns, obviously profits from lan- content level needs detailed semantic description, guage specific knowledge about phrase and sen- we see no means of accomplishing this goal other tence structure and word formation. One example than manually at the moment. is the order of adjective + noun pairs: in English and German, the adjective occurs left of the noun, 2.4 Contextual factors whereas in French, the adjective can occur left or (Hausmann, 2003; Heid and Gouws, 2005; Evert right of the noun. Another example is compound- et al., 2004) argue that collocations have strong ing, handled differently in different languages: preferences with respect to their morphosyntax noun + noun in English, typically separated by (see examples (1) and (2)) and may be combined a white space (e.g. death penalty) vs. noun + (see example (3)). The collocation in example (1) prepositional phrase in French (e.g. peine de mort) (’to charge somebody’) is restricted with respect vs. compound noun in German (e.g. Todesstrafe). to the determiner (null determiner) of the base, Consequently, language specific word formation whereas the same base shows a strong preference rules need to be considered when designing ex- for a (definite or indefinite) determiner when used traction patterns. For languages with a rich in- 2Abbreviations in table 1: flectional morphology where the individual word advl - adverbial forms are rather rare, frequency counts and results prd - predicative subj - subject 1In German, e.g., in usual verb second constructions with obj - object a full verb in the left sentence bracket (topological field the- pobj - prepositional object ory see (Wöllstein-Leisten et al., 1997)), particles of particle dat - dative case verbs appear in the right sentence bracket. The middle field gen - genitive case (containing arguments and possibly adjuncts of the verb) is quant - quantifying of undetermined length.

42 No. Type Example German newspaper texts. 1 N + Adj tiefer Schlaf The syntactic patterns for the extraction of these 2 Adj + Adv tief rot combinations concentrate on verb-ﬁnal construc- 3 V + Adv tief schlafen tions as in example (4) and verb second con-

4 V + NP ¢¡¤£¦¥ Bauklötze staunen structions with a modal verb in the left sentence

5 V + N §©¨ Frage + sich stellen bracket according to the topological ﬁeld theory

6 V + N ¡ Anforderungen + genü- (see (Wöllstein-Leisten et al., 1997)) as in exam- gen ple (5). The reason is that, in these constructions,

7 V + N Frage + aufwerfen the particle forms one word with the verb (see ex-

8 V + PP zu + Darstellung + gelan- ample (6)), as opposed to usual verb second con-

gen structions (see example (7)). Thus, we need not ¡ 9 V + Adj verrückt spielen recombine verb + particle groups that appear sep-

10 N + N ¦ Einreichung des Antrags aratedly.

¨ 11 N + N ein Schwarm Heringe (4) ... wenn Wien einen Antrag auf Vollmitglied- the category containing the base is underlined. schaft stellt. Table 1: Collocational patterns (’if Vienna an application for full membership puts’) (if Vienna applies for full membership) with a different collocate (example (2), ’to drop a lawsuit’). Example (3) shows two collocations (5) ... kann Wien einen Antrag auf Vollmitglied- sharing the base can form a collocational sequence schaft stellen. (example taken from (Heid and Gouws, 2005)). (’might Vienna an application for full membership put.’) (1) ø Anklage erheben (Vienna might apply for full membership.) (2) die/eine Anklage fallenlassen (6) ..., daß er ein Schild auf stellt. (3) Kritik üben + scharfe Kritik (’that he a sign upputs’) scharfe Kritik üben (that he puts up a sign) For both natural language generation systems (7) Er stellt ein Schild auf. and lexicography, such information is highly rel- (’He puts a sign up.’) evant. Therefore, the extraction of contextual in- (He puts up a sign.) formation (called ’context parameters’ in the fol- Preprocessing lowing) should be integrated into the collocation As data, we used a collection of 300 million extraction process. words from German newspaper texts dating from 3 Extracting noun + verb collocations 1987 to 1993. The corpus is tokenized and PoS- from German tagged by the Treetagger (Schmid, 1994), then chunk annotated by YAC (Kermes, 2003). The The standard architecture for collocation extrac- chunker YAC determines phrase boundaries and tion systems contains three stages (cf. (Krenn, heads, and disambiguates agreement information 2000)): a more or less detailed linguistic analysis as far as possible. It is based on the corpus query of the corpus text (preprocessing), an extraction language cqp (Christ et al., 1999)3, which can in step and a statistic filtering of the extracted word turn be used to query the chunk annotations. combinations. We follow this architecture (see figure 1). However, our hypothesis differs from other Data Extraction approaches. Collocations are often restricted with The syntactic patterns used to extract verb + respect to their morphosyntax. We test to what ex- subject/object combinations are based on PoS tags tent they can be identified via these restrictions. and chunk information. These patterns are represented using cqp macros (see figure 2). The cqp 3.1 Approach syntax largely overlaps with regular expressions. In an experiment, we extracted relational word 3http://www.ims.uni- combinations (verb + subject/object pairs) from stuttgart.de/projekte/CorpusWorkbench/

43 preprocessing collocation identification postprocessing

analysis extraction interpretation

semantic classification corpus TreeTagger YAC cqp morphology macros NLP resources stat. filtering

data tokenizing, partial parsing extraction of noun + verb database (newspaper texts) PoS tagging (annotation of pairs and their features (STTS tagset) phrase boundaries, wrt. morphosyntax agreement lexicons

IMSLex morphology

lemma and agreement information

Figure 1: Tool architecture

np of measure (meas, line (9), see example (9)), ( 1) MACRO n_vfin(0) nor must its head be a cardinal number (card, line ( 2) ( (10), see example (10)). An arbitrary number of ( 3) [pos = "(KOUS|VMFIN)"] words may follow the np (punctuation marks (PoS

( 4) []* tagged $.), subordinating conjunctions and ﬁnite ( 5) np modal verbs excluded). At least one verb is re- ( 6) [!pp quired (line (13), all PoS tags for verbs start with ( 7) & _.np_f not contains "ne" a capital ’V’4). Line (14) indicates the end of the ( 8) & _.np_f not contains "pron" subclause or sentence. ( 9) & _.np_f not contains "meas"

(10) & _.np_h != "@card@"]+ (8) ... kann [zur Verfügung]¦ gestellt werden.

(11) /np (12)[pos != "($.|KOUS|VMFIN)"]* (9) ... weil davon jährlich [3,5 Tonnen] ! #"¢$ (13)[pos = "V.*"]+ eingeführt werden.

(14)[pos = "($.|KON)"] (10) ... obwohl er [1989] &%©"'¤( noch dort (15)) arbeitete. (16); By applying the macro to the corpus, all se- Figure 2: sample macro quences of words matching the pattern are extracted. Line (1) of ﬁgure 2 contains the name of the From these sequences, the following informa- macro and the number of its parameters. In line tion is made explicit (cf. (Heid and Ritz, 2005)):

(3), a word PoS tagged KOUS (subordinating con- ) lemma of the noun (potential base) junction) or VMFIN (ﬁnite modal verb) is re- quested, followed by an arbitrary number (’*’) of ) lemma of the verb (potential collocate) words without any restrictions (line (4)). Line (5) indicates the start of a nominal phrase (np), line ) number of the noun (singular, plural)

(11) its end. The elements within this np (one or ) more words, as indicated by ’+’) must not be part case of the noun of a prepositional phrase (pp) to avoid the extrac- 4The search condition is underspeciﬁed with respect to tion of pp + verb (line (6), see example (8)). In ad- the ﬁniteness and the role of the verb (auxiliary, modal or full verb). Thus, line (13) matches verbal complexes. It also cov- dition, the np must be neither a named entity (ne, ers cases where full verbs are accidentally PoS tagged modal see line (7)) nor a pronoun (pron, line (8)) nor an or auxiliary verbs.

44 * determination of the noun (definite, indefinite, null, demonstrative, quantifier) SELECT COUNT(*) AS f, n_lemma, v_lemma * modification of the noun (adjective, cardinal FROM comfea1 number, genitive np, compound noun etc.) WHERE neg = ’+’ GROUP BY n_lemma, v_lemma * negation (yes/no) ORDER BY f DESC;

* auxiliaries and modal verbs f | n_lemma | v_lemma

Figure 3: sample query frequency of 10. Sample results are shown in figure 65. We evaluated collocation candidates with a fre- Filtering quency of at least 100. Within the 323 most fre- The instances extracted in the previous step are quent collocation candidates, we found 213 collo- grouped according to noun and verb lemmas, i.e. cations (including 11 idioms). This corresponds to instances of the same lemma pair form one group. a precision of 66% (see table 26). As a compari- Within these groups, a relative frequency distribu- son, a window-based study was carried out on the tion is computed for each of the features. For que- same (PoS-tagged) data. In this study, the window riability reasons, the results of this postprocessing was defined in a way that up to two tokens (exclud- are also stored in the database, as shown in figure ing sentence boundaries and finite full verbs) were 1. A word combination is chosen as a collocation allowed to appear between a noun (PoS tagged candidate if a preference (specified by a threshold NN) and a finite full verb (PoS tagged VVFIN). of e.g. 60% of the occurrences) for a certain fea- Log-likelihood7 was used as an association mea- ture value (singular / plural, presence / absence of sure. The precision of this approach is 41%8. a determiner, definite / indefinite / demonstrative 5 / possessive / quantifying determiner, presence of Abbreviations in figure 6: c - rated as a collocation in evaluation modifying elements) is discovered. i - rated as an idiom in evaluation. For chosing collocation candidates, a threshold of 60% is 3.2 Results used. However, additional preferences are displayed for val- ues greater than 50%. From 300 million words, we extracted more than 6Abbreviations in table 2: 1.3 million noun + verb combinations, the in- log-l - window-based approach using log-likelihood feat - pattern-based approach using morphosyntactic features stances of 726,488 different lemma pairs. 10,934 7www.collocations.de of these lemma pairs appeared with a minimum 8Note that partial matches, such as Verfügung + stellen

45 log-l feat SELECT COUNT(*) AS f, collocation candidates 700 323 n_lemma, v_lemma collocations (manually verified) 290 213 FROM comfea1 precision 41% 66% WHERE cas = ’Akk’ GROUP BY n_lemma, v_lemma Table 2: evaluation results ORDER BY f DESC; For evaluation purposes, combinations judged | | f n_lemma v_lemma collocations by either (or both) of the definitions 507 | Beitrag | leisten were marked as correct matches. In cases like ex- 237 | Antrag | stellen ample (11), combinations were marked as correct 173 | Eindruck | erwecken matches if no alternative collocate existed for de- 173 | Weg | finden scribing the denoted situation or event. 167 | Umsatz | steigern 145 | Hut | nehmen (11) Chance + haben (’to have a/the chance’) 140 | Bericht | vorlegen 135 | Betrieb | aufnehmen (12) Anteil + nehmen (’to commiserate’) 121 | Sprung | schaffen 120 | Ausschlag | geben 4 Conclusion and Outlook 116 | Mut | haben 111 | Sitz | haben We presented a system for collocation extraction 106 | Weg | ebnen that takes into account the behaviour or use of 105 | Zuschlag | erhalten collocations in context. Profiting from linguis- 104 | Platz | finden tic information (PoS tagging, chunking), the tool 100 | Anspruch | haben reaches a precision of 66% on the top 323 candi- 94 | Tod | feststellen dates by frequency. On the same data, a window- 93 | Zusammenhang | geben based approach relying only on PoS information 90 | Vertrag | unterzeichnen reached a precision of 41%. 90 | Riegel | vorschieben As the extracted word combinations as well as their context parameters (including the original ev- idence from the corpus) are stored in a database, Figure 5: sample query: verb + accusative object the tool also supports explorative research in lexicography. However, there are some enhancements worth However, the evaluation criteria from defini- doing: Especially when dealing with low frequen- tions (a) and (b) remain vague or even contradic- cies, relative frequencies lack reliability. There- tory for some of the results. First, there is the fore, we suggest computing a confidence interval problem of semantic equivalence: does the com- as proposed in (Evert, 2004b; Heid and Ritz, 2005; bination express more than its elements (consider Ritz, 2005). example (11))? Secondly, definitions (a) and (b) As indicated in figure 1, several postprocessing may judge the same example differently: Anteil steps can be added to the system, e.g. enabling a nehmen (example (12)) is usually agreed upon to sorting of collocation candidates with compound be a support verb construction, but the distinction nouns by the morphological heads of their base. of the noun Anteil as the base (making the main In order to get more data, the extraction from contribution to the meaning) is questionable. On verb first and verb second constructions is also the other hand, its unpredictable syntactic proper- possible. To complete the tool, extraction patterns ties (e.g. null determiner) and semantics (partial for collocations of different syntactic relations (cf. loss of meaning of the collocate nehmen) make it table 1) could be designed. clear that this combination has to be listed in a lexicon.

(without the corresponding preposition), have been treated as correct matches in 72 cases.

Figure 6: sample results

47 References Hannah Kermes. 2003. Off-line (and On-line) Text Analysis for Computational Lexicography. Ar- Sabine Bartsch. 2004. Structural and functional prop- beitspapiere des Instituts für Maschinelle Sprachver- erties of collocations in English. A corpus study arbeitung (AIMS):9(3). Stuttgart. of lexical and pragmatic constraints on lexical co- occurrence Narr, Tübingen. Brigitte Krenn. 2000. The Usual Suspects: Data Ori- ented Models for the Identiﬁcation and Representa- Oliver Christ, Bruno M. Schulze, Anja Hofmann and tion of Lexical Collocations. PhD thesis. DFKI und Esther König. 1999. The IMS Corpus Workbench: Universität des Saarlandes, Saarbrücken. Corpus Query Processor (CQP). User’s manual. In- stitut für maschinelle Sprachverarbeitung, Univer- Brigitte Krenn. 2000. Collocation Mining: Ex- sität Stuttgart. ploiting Corpora for Collocation Identiﬁcation and Representation. W. Zühlke and E. G. Schukat- Jonathan Crowther, Sheila Dignen and Diana Lea. Talamazzini (eds.): Proceedings of KONVENS 2000, 2002. Oxford Collocations Dictionary for students p. 209-214. Ilmenau, Deutschland. of English. Oxford University Press. Stefan Evert. 2004. The Statistics of Word Cooc- Julia Ritz. 2005. Entwicklung eines Systems zur curences - Word Pairs and Collocations. PhD thesis. Extraktion von Kollokationen mittels morphosyn- Institut für Maschinelle Sprachverarbeitung (IMS), taktischer Features. Diploma Thesis. Institut für Universität Stuttgart. Maschinelle Sprachverarbeitung (IMS), Universität Stuttgart. Stefan Evert, Ulrich Heid and Kristina Spranger. 2004. Identifying morphosyntactic preferences in colloca- Helmut Schmid. 1994. Probabilistic part-of-speech tions. In M. T. Lino, M. F. Xavier, F. Ferreira, R. tagging using decision trees. D. Jones and H. Costa and R. Silva (eds.): Proceedings of the 4th In- Somers (eds.): Proceedings of the International ternational Conference on Language Resources and Conference on New Methods in Language Process- Evaluation (LREC 2004), p. 907 - 910. Lisbon, Por- ing (NeMLaP). Manchester, U.K. tugal. Frank Smadja. 1993. Retrieving Collocations from Stefan Evert. 2004. The statistical analysis of mor- Text: Xtract. Computational Linguistics (19), p. phosyntactic distributions. In M. T. Lino et al. 143-177. Manchester, U.K. (eds.): Proceedings of the 4th International Confer- Agnès Tutin. 2004. Pour une modélisation dynamique ence on Language Resources and Evaluation (LREC des collocations dans les textes 2004), p. 1539 - 1542. Lisbon, Portugal. . G. Williams and S. Vessier (eds.): Proceedings of the Eleventh EU- John R. Firth. 1968. F.R. Palmer (ed.): Selected Papers RALEX International Congress. Lorient, France. of J.R. Firth 1952-59. Longman, London. Angelika Wöllstein-Leisten, Axel Heilmann, Peter Franz Josef Hausmann. 1979. Un dictionnaire de col- Stepan and Sten Vikner. 1997. Deutsche Satzstruk- locations est-il possible? Travaux de Linguistique tur. Stauffenburg, Tübingen. et de Litterature XVII(1), p. 187-195. Centre de philologie et de littérature romanes de l’université de Strasbourg. Franz Josef Hausmann. 1989. Le dictionnaire de collocations. In F. J. Hausmann et al. (eds.): Wörter- bücher, Dictionaries, Dictionnaires, p. 1010 - 1019. De Gruyter, Berlin. Franz Josef Hausmann. 2003. Was sind eigentlich Kollokationen? In K. Steyer (ed.): Wortverbindun- gen - mehr oder weniger fest. Jahrbuch des Instituts für Deutsche Sprache 2003:309 – 334. De Gruyter, Berlin. Ulrich Heid and Rufus H. Gouws. 2005. A model for a multifunctional electronic dictionary of collocations. Institut für Maschinelle Sprachverar- beitung (IMS), Universität Stuttgart, submitted to EURALEX 2006. Ulrich Heid and Julia Ritz. 2005. Extracting collocations and their contents from corpora. F. Kiefer et al. (eds.): Papers in Computational Lexicography: Complex 2005. Hungarian Academy of Sciences, Budapest.