Term and collocation extraction by means of complex linguistic web services
Ulrich Heid1, Fabienne Fritzinger1, Erhard Hinrichs2, Marie Hinrichs2, Thomas Zastrow2
1Universitat¨ Stuttgart Universitat¨ Tubingen¨ Institut fur¨ maschinelle Sprachverarbeitung Seminar fur¨ Sprachwissenschaft Azenbergstr. 12 Wilhelmstr. 19 D 70174 Stuttgart D 72074 Tubingen¨ [heid,fritzife]ims.uni-stuttgart.de [eh,meh]sfs.uni-tuebingen.de
Abstract We present a web service-based environment for the use of linguistic resources and tools to address issues of terminology and language varieties. We discuss the architecture, corpus representation formats, components and a chainer supporting the combination of tools into task-specific services. Integrated into this environment, single web services also become part of complex scenarios for web service use. Our web services take for example corpora of several million words as an input on which they perform preprocessing, such as tokenisation, tagging, lemmatisation and parsing, and corpus exploration, such as collocation extraction and corpus comparison. Here we present an example on extraction of single and multiword items typical of a specific domain or typical of a regional variety of German. We also give a critical review on needs and available functions from a user’s point of view. The work presented here is part of ongoing experimentation in the D-SPIN project, the German national counterpart of CLARIN.
1. Introduction ploration (data extraction). Section 3. is devoted to our pre- This paper presents a web-based service environment for processing web services: architecture, formats used for text the use of linguistic resources and tools, focusing on early encoding, and in particular WebLicht, the web-based lin- results of a part of the German D-SPIN project1. guistic tool chainer (cf. (Hinrichs et al., 2009), (Hinrichs The scenario underlying the present study is targeted at et al., 2010)) which allows the user to combine pipelines of linguists, philologists, terminologists and translators inter- linguistic processing modules without need for installing or ested in an analysis of text resources they may have col- adapting existing tools. We present its infrastructure func- lected, with a view to lexis: single and multiword items tion, its user interface and the range of preprocessing op- typical of the specialised language of a given domain, or tions at hand. In section 4., we turn to terminology and typical of a regional variety of German. variety analyses, and discuss web service pipelines for col- Identifying such lexical specificities is a complex task, location extraction and corpus comparison (e.g. Swiss vs. which involves several computational linguistic processing German news texts). Section 5. is a critical review of the steps and the combination of tools which not every user current state, from the user’s point of view: available func- may have available. Thus, an objective of D-SPIN, as of tions vs. needs for future improvements. the EU project CLARIN, is (i) to make the necessary tools available in a service-oriented architecture, (ii) to provide 2. Scenarios and targeted phenomena an environment which allows the user to apply them to Term candidate extraction targets both single word and his/her own texts (or to corpora made available for the pur- multiword items (e.g. Rechtsbeschwerde ‘appeal’, Recht- pose), and (iii) to combine them as needed. snachfolger ‘legal successor’, Recht(e) geltend machen ‘as- In this paper, we present some exemplary corpus prepro- sert one’s rights’). Similarly, our comparison of regional cessing tools, the retrieval of lexical collocations, and the varieties also targets single words and multiwords, in this comparison of data from two corpora. These functions are case mainly collocations: in line with the pluricentricity useful for finding specialised terminology and/or for iden- hypothesis (cf. (Ammon, 2001)), we check Swiss (CH) tifying lexical regionalisms. Obviously, many more types and Austrian (AT) newspaper texts for region-specific lexis of linguistic, terminological and philological analyses can (not dialects). Examples of single words include AT Krida in principle be supported by the tools. ‘bankrupcy’ for DE Bankrott, Insolvenz, or CH Zustupf In section 2., we sketch the application scenarios and illus- ‘(financial) contribution’ for DE Zuschuss. These spe- trate the kinds of data to be extracted from texts. We dis- cific items may be part of multiwords (betrugerische¨ Krida tinguish between corpus preprocessing (tokenisation, tag- ‘fraudulent bankrupcy’, Zustupf leisten ‘give a contribu- ging/lemmatisation and possibly parsing) and corpus ex- tion’). But also non-specific lexical items present in all varieties may occur in region-specific collocations (e.g. 1 D-SPIN stands for Deutsche SPrachressourcen-INfrastruktur; CH markanter Anstieg ‘marked increase’ for DE deutlicher the D-SPIN project is financed by the German Federal Min- Anstieg). istry of Research and Education, BMBF; it is a national German Such phenomena are among the data we assume that lin- complement to the EU-project CLARIN. See the URLs http: //www.sfs.uni-tuebingen.de/dspin and http:// guists, lexicographers, terminologists or translators may www.clarin.eu for details. wish to extract from texts. This extraction requires a certain
3215 amount of preprocessing of the texts, as well as additional resources themselves need not be modified with respect to steps: extraction by syntactic patterns (e.g. noun + adjec- their stand-alone versions when being used in the web ser- tive, verb + object noun, noun + genitive complement) – vice setup. So far, we have integrated linguistic corpus pro- frequency counts and a calculation of cooccurrence signif- cessing tools provided by four German research institutes icance – a frequency-based comparison of data from two (see section 3.2.1., below). Wrappers may interact with corpora, e.g. specialised vs. general, or Swiss vs. German. service components: e.g. when a tool provided by one in- In the following sections, we discuss to what extent the stitution uses a resource provided by another one. tools needed for these tasks and the work flows resulting Components of the infrastructural layer are used, among from their combination can be supported by linguistic web others, to call converters, e.g. for transforming the D-SPIN- services. internal text corpus format into an exchange format (see be- low, section 3.1.). User interaction, as well as calls of the 3. Linguistic Web Services for German web services from applications (e.g. other linguistic pro- Linguistic web services can be word-oriented or text- cessing tools) are directed to the infrastructure layer. In our oriented. Word-oriented web services rely on lexical or setup, interactive calls are channeled through the WebLicht corpus resources; users query the web service with a par- tool (Hinrichs et al., 2009), a linguistic tool chainer for ticular item and get back all data associated with the item composing task-specific web services. The chain builder in the respective resource (see e.g. Wortschatz2). function will be presented below, in section 3.2.. The implementation uses both Perl/Python and Java com- We will in the following sections concentrate on text- 3 oriented web services: the user uploads a text (corpus) and ponents; it is based on the REST architecture (Richardson the service calls one or more computational linguistic tools and Ruby, 2007), and on an Apache web server. As the which are applied to the text (corpus) and deliver annotation whole setup is experimental, we have so far not worked results. As a variant, the tools may perform calculations in detail on authentification, access rights, billing, etc. The (e.g. of frequency or significance) and deliver the results to EU project CLARIN has an ongoing work package devoted the user. to these issues, the results of which will be used, before our Obviously, web services may be called interactively or via web services will be made available at large. APIs, from tools. We focus on the former interaction, here, 3.1. Formats used internally and for exchange even though both are possible in our setup. In our experiments, we implement an architecture consist- Internal communication. For communication between ing of four layers (cf. Fig. 1, from bottom to top): (i) tools, different web services, we use an XML-based text cor- (ii) wrappers, services and converters, (iii) the web service pus format which is defined by an XML schema expressed infrastructure, as well as (iv) clients. in Relax NG. It uses additional constraints expressed in Schematron. A sample encoding taken from the current im- plementation is reproduced in Figure 2. Basically, the format contains both a metadata section, to encode sources, tools, the language of the document, etc., and a multi-layered token and region annotation. Figure 2
Application Webservice Application Clients shows a part of the internal encoding of the sentence die Composition zweite Studie lieferte ahnliche¨ Ergebnisse4 parsed with the BitPar constituent parser (Schmid, 2004); we reproduce Web Service Infrastructure Infrastructure only some tokens and nodes of the parse tree, skipping metadata and lemmas (cf. the dependency tree in Figure 5, below). Transformer Service XML Wrappers Exchange formats – long-term view. The current text
Wrapper Wrapper Wrapper Wrapper corpus format was designed with a view to efficiency: in our tests, we processed among others the 10 million words EMEA corpus together with a 40 million words newspaper Tool A Tool B Tools Res. Res. corpus (Frankfurter Rundschau, 1992/93) through one in- stance of the pipeline (see section 4.). The annotated results lead to rather large amounts of data which have to be trans- Figure 1: Layers of the web service architecture ported through the web based pipeline. Thus, the definition of the D-SPIN text corpus format aims at keeping the XML overhead low (for details, see (Heid et al., 2010)). Tools and resources are embedded each in a wrapper, Obviously, there exist standard proposals for corpus encod- which acts as an interface for their input and output and ing formats, e.g. from the language resources standard- thereby supports communication between different tools in isation work groups of ISO5. The ISO proposals cover a a pipeline. By using wrappers, we ensure that the tools and 3Implemented on Apache Web server and Tomcat application 2http://wortschatz.uni-leipzig.de/axis/ server. servlet/ServiceOverviewServlet, an extensive 4EN: the second study provided similar results. linguistic database of the University of Leipzig. 5ISO TC 37/SC-4 www.tc37sc4.org
3216
3217 Lemmatizer 4.1. Scenarios revisited: tools GermaNet Results Leipzig With the help of WebLicht services, we can extract Tuebingen Tokenizer frequency-wise prominent single words and their lexi- Text2Dspin Results Text Berlin Tagger Parser Berlin cal collocations (e.g. in the sense of Bartsch (2004):76 Leipzig Berlin Stuttgart (Bartsch, 2004)), from texts of a specialised domain, or Tuebingen Results Stuttgart Stuttgart from regional texts (cf. section 2.). Frequency Analyzer NER Collocation candidate extraction. Collocation extrac- Leipzig Results Berlin tion amounts to the extraction of all lemma pairs of a given syntactic pattern (e.g. adjective + noun, verb + ob- ject noun) and subsequent sorting of the candidates by their text-specific association value (for details, cf. (Weller and Heid, 2010)). The calculation of association values (e.g. Figure 3: Extract of the WebLicht architecture: Log Likelihood) is done by means of a (web service) call to preprocessing components Evert’s UCS toolkit (Evert, 2005), which implements over thirty different association measures. Moreover, different workflows may require preprocessing of different depth: if a user only needs to extract adjec- Corpus I Collocation Calculation of Significant Parsing Collocations tive+noun pairs from his texts, there is no need for parsing, Corpus I (parsed) Extraction associative strength Collocations whereas for German verb+object pairs, results improve if parsed data are used (Ivanova et al., 2008). Figure 3 shows part of the preprocessing workflows currently supported by Figure 4: Collocation extraction procedure the WebLicht chainer ((Hinrichs et al., 2009), (Hinrichs et al., 2010)). Figure 4 schematises a simple collocation candidate extrac- 3.2.3. WebLicht as a user interface tion pipeline. Parsing, the extraction of lemma pairs and the The WebLicht platform is language-independent. The user calculation of associative strength are modules provided as is able to select tools and resources for a specific language web services, cf. (Fritzinger et al., 2009). These are three by the choice options in a language selection field. This separate models, which we “package” into two web ser- Language field currently allows the selection of e.g. Ger- vices, for reasons of practicality. man, English, Italian, French, etc. For identifying verb + object pairs, we rely on text parsed Language input to the web services can be plain text to be with the dependency parser FSPar (Schiehlen, 2003). Its inserted by the user in a plain text field or by uploading a output encodes grammatical relations between governor plain text file whose source location can be specified. Al- and governed items. As a sample, the analysis of Die zweite ternatively, various format converters are offered to bring Studie lieferte ahnliche¨ Ergebnisse is reproduced in Fig- input text into the proper format used by WebLicht. ure 5. A Selected Tools field displays all web services that have already been entered into the web service chain. A Next lieferte . Tool Choises field then offers the set of tools that can be TOP TOP entered into the chain next. The user often has a choice Studie Ergebnisse of alternative tools – sometimes a wide variety of services NP:nom NP:akk are offered as candidates. Obviously, only those tools are die zweite ähnliche offered as options for further processing that are compatible SPEC ADJ ADJ with the results of the processing steps already completed. (a) Tree representation For more information on the WebLicht user interface see (Hinrichs et al., 2010). 0 Die ART d | 2 SPEC 1 zweite ADJA 2. | 2 ADJ 2 Studie NN Studie Nom:F:Sg 3 NP:1 4. Complex scenarios 3 lieferte VVFIN liefern 3:Sg:Past:Ind* −1 TOP 4 ähnliche ADJA ähnlich | 5 ADJ for Web Services use 5 Ergebnisse NN Ergebnis Akk:N:Pl 3 NP:8 6 . $. . | −1 TOP Above, the standard WebLicht setup has been described. It is characterised by its flexibility and by the possibility (b) Parsing output to process corpora of non-trivial size. Obviously, in such a case the processing may take some time, but users may prefer to wait several hours for a processing result over hav- Figure 5: Output of FSPAR: dependency analysis ing their own computers blocked by ongoing processing, or having to install and adapt tools. FSPar internally encodes the dependency tree in linear form In the following, we turn to the experimental applications (Fig. 5 (b)), with several columns: the first column con- from terminology and language variety research sketched tains the position of the word form in the parsed sentence, in section 2., now from a more technical perspective. the second the word form itself, followed by POS-tag and
3218 (a) Only EMEA (not FR). ports for pharmaceuticals, made available by EMEA, the term candidates f (abs.) European Medicines Agency (ca. 10 million words). The Durchstechflasche 5638 Injektionsstelle 3489 data have been collected by Jorg¨ Tiedemann and made Pharmakokinetik 3426 available on his OPUS web site12 This corpus is compared Hamoglobinwert¨ 3395 Fertigspritze 3271 to newspaper text of Frankfurter Rundschau, consisting of Ribavirin 3234 ca. 40 million words, in order to identify specialised termi- Gebrauchsinformation 2801 Dosisanpassung 2580 nology. Epoetin 2302 In the experiments on the regional varieties of German, we Hydrochlorothiazid 2128 use newspaper texts from the DeReKo corpus, Deutsches (b) EMEA and FR Referenzkorpus13. These are grouped according to coun- term candidates weirdness f (abs.) Filmtablette 25522 6389 tries. We use Austrian and Swiss material and compare Injektionslosung¨ 19854 4970 it with newspaper texts from Germany, such as Frank- Packungsbeilage 14710 7365 Niereninsuffizienz 14233 3563 furter Allgemeine Zeitung, Frankfurter Rundschau, Die Verkehrstuchtigkeit¨ 13558 3394 Zeit, Stuttgarter Zeitung, Handelsblatt. Leberfunktion 8385 2099 Hypoglykamie¨ 8353 2091 Sample results. Table 2, above, contains a few typical Toxizitat¨ 7957 1992 single word candidates from the EMEA corpus: items only Einnehmen 7035 7045 Hypotonie 6823 1708 contained in EMEA (part (a)) and items significantly more frequent in EMEA than in the Frankfurter Rundschau, used Table 2: Single word term candidates – Top 10 as a “general language” text, for comparison purposes. The items are sorted by frequency (2(a)) and by the RS/RG quo- tient (2(b)), i.e. in decreasing order of relevance for the lemma. The fifth column contains the morphosyntactic fea- specialised text. tures of the word form, e.g. case, number, person, etc. Table 3 contains a few verb + object collocations which are Columns 6 and 7 are to be interpreted together: the num- found in Swiss news texts but not in German news. ber indicates the position of the word’s governor, and the last column indicates the grammatical relation between the Abklarung¨ treffen 96 word and its governor. Thus, in Figure 5, the word form Abklarung¨ vornehmen 91 Ergebnisse is coded as plural accusative (line 5, col. 5) and Anlaß besuchen 73 as an accusative object (NP:8, col. 7) of the verb liefern Anlaß durchfuhren¨ 199 (col. 7: value “3”, pointing to lieferte, which is on line Anlaß organisieren 367 3). The actual collocation extraction is implemented as Beschwerde gutheißen 88 a pattern-matching over FSPar-output, e.g. for verbs and Bilanz deponieren 82 their objects, (e.g. Perl/Python scripts). Busse aussprechen 72 Defizit budgetieren 94 Extraction of prominent single word items. The identi- Einsitz nehmen 295 fication of typical single word items, be it from specialised Einsprache erheben 262 or regional texts, relies on an implementation of Ahmad et Entscheid fallen¨ 79 al. (1992)’s “weirdness” measure (Ahmad et al., 1992). Gegensteuer geben 143 The intuition is that terms from the specialised language Gesuch bewilligen 90 of a given domain are much more frequent in the domain specific text than elsewhere. Some specialised items do not show up at all in general language, others are found Table 3: Typical CH verb+object pairs in both, but more frequently in the specialised text. A tool for identifying specialised items in this way simply has to An adjective + noun case of the same type is tiefer Preis determine the relative frequency of each item from the spe- ‘low price’. In German texts from Germany, we find this cialised text (RS), calculate its relative frequency in a gen- combination almost never; manual inspection shows that eral text used for comparison (RG), and determine the re- instead, German texts contain niedriger Preis. The tools lationship between both (RS/RG). It produces two kinds of provide the most significant collocations with tief in texts term candidate output: a list of words never found in the from Switzerland (Table 4 (a)) and those from texts from general texts (cf. Table 2 (a) for an example from a phar- Germany (Table 4 (b)), with their absolute frequency in maceutical corpus, we indicate absolute frequency in the both corpora. While tiefer Zins ‘low interest’, tiefer Preis, specialized corpus), and a list of words found more often in tiefe Inflation are prominent in Swiss texts, these combina- the specialised than in the general texts (cf. Table 2 (b) for tions are rare in texts from Germany. However, the (figura- examples from the same corpus, with the weirdness figure tive) use of tief in texts from Germany, which is top ranked RS/RG and absolute frequency in the specialized text). 12http://urd.let.rug.nl/tiedeman/OPUS/ 13 4.2. Corpus data used for experimentation This corpus was collected jointly by Institut fur¨ deutsche and sample results Sprache, Mannheim, and by the Universities of Tubingen¨ and Stuttgart, in the framework of a project financed by the Corpus data. For the terminology extraction work, we Land Baden-Wurttemberg,¨ http://www.ids-mannheim. use the German part of the multilingual corpus of test re- de/projekte/dereko_I.
3219 by frequency in our corpora (tiefe Krise ‘deep crisis’, tiefer
Tagging Corpus II Corpus II (tagged) Moreover, it seems necessary to be able to parametrise web services and to allow users to set parameters, before the processing chain is entered. In WebLicht, this is done by se- Figure 6: Extraction of specialised collocations lecting a given service. But in the medium term, we would like to consider a setup where users rather select in terms of properties of the expected output than of the tools used. 4.4. A format for multiword data Taking up collocation extraction, Figure 8 symbolises a few For the web-service-internal encoding of multiword data, of the parameters we envisage users may wish to set (from as produced by the term candidate and the collocation ex- left to right): which grammar to use for parsing, which syn- traction services, we devised an XML encoding which is tactic type or types of collocations to extract, which associ- inspired by the same principles as our corpus encoding for- ation measure(s) to use, how to package (e.g. by syntactic mat. It is also meant to be a “slim” XML format, which can type) and how to sort and lexicographically display the re- in the long run be mapped onto LMF (ISO 24613:2008), the sults. We are still far from this flexibility, but the internal Lexical Markup Framework. An example of the encoding format used in our web services allows us to “transport” a of two word pairs is displayed in figure 7. parameter through the pipeline.
3220 Siebert, and Jorg¨ Didakowski. 2009. Weblicht: Web-
Collocator Basis Workshop on lin- Chi−squared based LRT services for German. In ADJ+NN Dependency T−Score Dice alphabetical LFG separated guistic processing pipelines, GSCL Jahrestagung 2009, Input Associative Parsing Collocation Output Sorting text selection strength Result Potsdam. all−in−one Statistical V+Nakk Fisher’s Test Significance Marie Hinrichs, Thomas Zastrow, and Erhard Hinrichs. NN+NNgen LogLikelihood Chi−2 T−Score LogL Dice 2010. WebLicht: Web-based LRT Services in a Dis- Fisher tributed eScience Infrastructure. In Proceedings of Figure 8: Possible points for user interaction LREC 2010, Malta. Nancy Ide and Laurent Romary. 2004. International stan- dard for a linguistic annotation framework. Journal of Finally, we have shown the need for and the usefulness of Natural Language Engineering, 10:3–4. web services that can take more than one input. Future Kremena Ivanova, Ulrich Heid, Sabine Schulte im Walde, work will elaborate on the challenges mentioned here, be- Adam Kilgarriff, and Jan Pomikalek.´ 2008. Evaluating a sides the inclusion of additional tools, work towards full German Sketch Grammar: A case study on noun phrase standards compatibility and more extensive experimenta- case. In Proceedings of the VIth International Confer- tion with web service-based linguistic processing chains. ence on Language Resources and Evaluation (LREC Acknowledgements 2008), Marrakesh, Morocco. CD-ROM. Manuel Kountz, Ulrich Heid, and Kerstin Eckart. 2008. We gratefully acknowledge the support of the BMBF for A LAF/GrAF based encoding scheme for underspeci- the D-SPIN project. We should also like to thank our col- fied representations of dependency structures. In Pro- leagues from Leipzig and Berlin for making their tools ceedings of the VIth International Conference on Lan- available for integration into WebLicht, and for discussing guage Resources and Evaluation (LREC 2008), Mar- related issues with us. Special thanks to Helmut Schmid rakesh, Morocco. CD-ROM. and Max Kisselew (IMS Stuttgart) for their helpful com- Leonard Richardson and Sam Ruby. 2007. RESTful Web ments on components and procedures described in sections Services. O’Reilly. 3 and 4. Michael Schiehlen. 2003. A cascaded finite-state parser for German. In Proceedings of the Research Note Sessions 6. References of the 10th Conference of the European Chapter of the Khurshid Ahmad, Andrea Davies, Heather Fulford, and Association for Computational Linguistics (EACL’03), Margaret Rogers. 1992. What is a term? the semi- Budapest, April. automatic extraction of terms from text. In Mary Snell- Anne Schiller, Simone Teufel, Christine Stockert,¨ and Hornby, Franz Pochhacker,¨ and Klaus Kaindl, editors, Christine Thielen. 1995. Vorlaufige¨ Guidelines fur¨ das Translation Studies - an interdiscipline. John Benjamins, Tagging deutscher Textcorpora mit STTS. Technical Amsterdam/Philadelphia. report, Universitat¨ Stuttgart, Institut fur¨ maschinelle Ulrich Ammon. 2001. Die Plurizentrizitat¨ des Deutschen, Sprachverarbeitung, and Seminar fur¨ Sprachwis- oder: Wer sagt, was gutes Deutsch ist? In Kurt Egger senschaft, Universitat¨ Tubingen.¨ and Franz Lanthaler, editors, Die deutsche Sprache in Helmut Schmid. 2004. Efficient parsing of highly ambigu- Sudtirol:¨ Einheitssprache oder regionale Vielfalt, pages ous context-free grammars with bit vectors. In Proceed- 11–26. Folio, Wien/Bozen. ings of the 20th International Conference on Computa- Sabine Bartsch. 2004. Structural and functional properties tional Linguistics (COLING 2004), Geneva, Switzerland. of collocations in English. In A corpus study of lexi- Marion Weller and Ulrich Heid. 2010. Extraction of Ger- cal and pragmatic constraints on lexical co-occurrence. man multiword expressions from parsed corpora using Narr, Tubingen.¨ context features. In Proceedings of LREC 2010, Malta. Stefan Evert. 2005. The statistics of word cooccurrences: word pairs and collocations. Ph.D. thesis, Universitat¨ Stuttgart. Fabienne Fritzinger, Max Kisselew, Ulrich Heid, Andreas Madsack, and Helmut Schmid. 2009. Werkzeuge zur extraktion von signifikanten wortpaaren als web service. In GSCL-Symposium Sprachtechnologie und eHumani- ties, Technischer Bericht Nr.2009-01, Universitat¨ Duis- burg Essen, Duisburg. Ulrich Heid, Helmut Schmid, Kerstin Eckart, and Erhard Hinrichs. 2010. A Corpus Representation Format for Linguistic Web Services: the D-SPIN Text Corpus For- mat and its Relationship with ISO Standards. In Pro- ceedings of LREC 2010, Malta. Erhard Hinrichs, Marie Hinrichs, Thomas Zastrow, Ger- hard Heyer, Volker Boehlke, Uwe Quasthoff, Helmut Schmid, Ulrich Heid, Fabienne Fritzinger, Alexander
3221