arXiv:cmp-lg/9707010v1 21 Jul 1997 faot45M opldPoo oe(o count- (not system code lexicons) stand-alone Prolog the a compiled ing is MB 4.5 version ver- about UNIX this the of in on GTU concentrate will sion. we UNIX. paper under Prolog this SICStus In in and OS/2, and DOS a Koblenz of as Zurich. Universities com- and used the in at successfully courses linguistics syntax been putational supplement has Throughout to it tool years tutoring formats. 7 various last in the test- grammars and flexible of development a the ing for as tool developed user-friendly was and environment) test gram- mar Grammatik-Testumgebung; (German: GTU Introduction 1 lexicons: o ohSNO . n U S5.x. OS SUN MB and 7 4.x OS about SUN total requirements both for memory actual the tem T a enipeetdi rt rlgunder Prolog Arity in implemented been has GTU xeine ihteGUgamrdvlpetenvironment development grammar GTU the with Experiences 1 codn orarneet fteoeaigsys- operating the of rearrangements to According o uuegamrworkbench. grammar necessary future features a the for From derive will researchers. we CL these ex- for an as tool and students perimental for GTU tool the tutoring used a of we as when description gained a we is experiences follows inter- What lexicon integrated face. an via to attached grammar be a can that lexicons types German three set of as well a various as covering phenomena contains syntactic sentences test it German Additionally, of in- user for- terface. window-oriented grammar a four under environ- malisms supports GTU test gram- ment). Grammatik- grammar development language (German: Testumgebumg; the natural GTU experi- called for of our mars tool testing describe a and we with paper ences this In eateto optrScience Computer of Department opttoa Linguistics Computational 1 T neat ih3German 3 with interacts GTU . itrhrrt.190 Winterthurerstr. [email protected] nvriyo Zurich of University Abstract H85 Zurich CH-8057 atnVolk Martin [email protected] nttt fCmuainlLinguistics Computational of Institute okec Kpa n awl,19).From grammar future workbenches. for 1996)). recommendations derive Maxwell, we this and LFG Xerox (Kaplan (mostly the and to workbench 1996) access al., have et (Gaizauskas we on GATE that concentrate workbenches some we to other paper them comparing this features, GTU’s In evaluating and Jung, (Volk, 1995)). or Richarz, 1994) Volk, and Richarz, sentence. (Jung, a parsing when sen- can off a module and of) This on reading switched (a anomalous. be if semantically compute is lexi- to tence hand-coded used be the can module of f- con another restrictions an In selectional for rules. the expressions semantic using logical structure computes a seman- the Thus module way the tic specify f-structures. may LFG developer grammar of processing semantic .LxclFntoa rma LG (Kaplan (LFG, Grammar Grammar Functional Lexical Structure 4. Phrase Generalized Precedence 3. Linear / Dominance Immediate 2. and (Pereira (DCG, Grammar Clause Definite 1. .PO,afl-omlxcnta a enderived been has that lexicon full-form a PLOD, 3. anal- morphology fast a 1994), (Oy, GerTWOL 2. vocabu- whose stem-lexicon hand-coded small a 1. T’ etrshv enpbihdbfr (see before published been have features GTU’s towards step first a provides GTU Additionally, T uprsgamr ne orformalisms: four under grammars supports GTU nvriyo Koblenz-Landau of University n rsa,1982)). Bresnan, and 1985)), al., et (Gazdar (GPSG, GPSG), of subset a (ID/LP; Grammar struc- feature with tures, augmented 1987)) Shieber, rmteCLXlxcldtbs (Baayen, database 1995). Rijn, lexical van and CELEX Piepenbrock, the from and program, ysis adjectives.), and nouns restric- its selectional all for contains tions also lexicon sentences test (This the towards tailored been has lary -67 Koblenz D-56075 ikRicharz Dirk hia 1 Rheinau 2 GTU - its merits and its limits ble grammar rules and lexicon interface rules under this formalism. Grammar rule notation Writing large grammars with GTU has sometimes One of the primary goals in the GTU project was to lead to problems in navigation through the grammar support a grammar rule notation that is as close as files. A grammar browser could be used to alliviate possible to the one used in the linguistics literature. these problems. The Xerox LFG-WB contains such This has been a general guideline for every formal- a browser. It consists of a clickable index of all rule ism added to the GTU system. Let us give some heads (i.e. all defined constituent symbols). Via this examples. Typical ID-rules in GTU are: index the grammar developer can comfortably access the rule definitions for a given constituent. (1)S ->NP[X], VP[X] | X = [kas=nom]. Static grammar checks (2) NP[kas=K] -> Det[kas=K, num=N], For the different formalisms in GTU, different types (AdjP[kas=K, num=N]), of parsers are produced. GPSG grammars are pro- N[kas=K, num=N]. cessed by a bottom-up chart parser, DCG and LFG Rule (1) says, that a constituent of type S con- grammars are processed by top-down depth-first sists of constituents of type NP and VP. The feature parsers. All parsers have specific problems with structures are given in square brackets. A capital some structural properties of a grammar, e.g. top- letter in a feature structure represents a variable. down depth-first parsers may run into infinite loops Identical variables within a rule stand for shared val- if the grammar contains (direct or indirect) left re- ues. Hence, the feature structures for NP and VP in cursive rules. rule (1) are declared to be identical. In addition the Therefore GTU provides a static check for detect- feature structure equation behind the vertical bar ing left recursions. This is done by building up a | specifies that X must be unified with the feature graph structure. After processing all grammar rules structure [kas=nom]. Rule (2) says that an NP con- and inserting all possible edges into the graph, the sists of a Det, an optional AdjP and an N. It also says grammar contains a possible left recursion if this that the features kas and num are set to be identi- graph contains at least one cycle. In a similar man- cal across constituents while only the feature kas is ner we can detect cycles within transitive LP rules passed on to the NP-node. or within alias definitions. There are further means for terminal symbols These checks have shown to be very helpful in un- within a grammar and a reserved word representing covering structural problems once a grammar has an empty constituent. grown to more than two dozen rules. The static In our experience the grammar rule notation helps checks in GTU have to be explicitly called by the the students in getting acquainted with the system. grammar developer. It would be better to perform But students still need some time in understanding these checks automatically any time a grammar is the syntax. In particular they are sometimes misled loaded into the system. by the apparent similarity of GTU’s ID-rules to Pro- A model for the employment of grammar checks log DCG-rules. While in Prolog constituent symbols is the workbench for affix grammars introduced by are atoms and are usually written with lower case (Nederhof et al., 1992), which uses grammar checks letters, GTU requires upper case letters as is custom- in order to report on inconsistencies (conflicts with ary in the linguistic literature. In addition students well-formedness conditions such as that every non- need a good understanding of feature structure uni- terminal should have a definition), properties (such fication to be able to manipulate the grammatical as LL(1)), and information on the overall grammar features within the grammar rules. structure (such as the is-called-by relation). For writing grammar rules GTU has an inte- grated editor that facilitates loading the grammar Output in different granularities into GTU’s database. A grammar thus becomes One of GTU’s main features is the graphics display immediately available for testing. Loading a gram- of parsing results. All constituent structures can be mar involves the translation of a grammar rule into displayed as parse trees. For LFG-grammars GTU Prolog. This is done by various grammar proces- additionally outputs the f-structure. For DCG and sors (one for each formalism). The grammar pro- GPSG the parse tree is also displayed in an indented cessors are SLR parsers generated from metagram- fashion with all features used during the parsing pro- mars. There is one metagrammar for each gram- cess. Output can be directed into one or multiple mar formalism describing the format of all admissi- windows. The multiple window option facilitates the comparison of the tree structures on screen. Pars- with the saved structures and to inform the user that ing results can also be saved into files in order to there now is an additional reading. In our compari- use them in documentations or for other evaluation son tool series of comparisons for multiple sentences purposes. can be run in the background. Their results are dis- The automatic graphic display of parsing results played in a table which informs about the numbers is an important feature for using GTU as a tutoring of readings for every sentence. tool. For students this is the most striking advantage This comparison tool is considered very helpful, over coding the grammar directly in a programming once the user understands how to use it. It should language. The GTU display works with structures be complemented with the option to compare the of arbitrary size. But a structure that does not fit output structures of two readings of the same input on the screen requires extensive scrolling. A zoom sentence. option could remedy this problem. Zooming into output structures is nicely inte- Tracing the parsing process grated into the Xerox LFG-WB. Every node in the Within GTU the parsing of natural language input parse tree output can be enlarged by a mouse click can be traced on various levels. It can be traced to its complete feature structure. Every label on a • chart edge output can be displayed with its internal during the lexicon lookup process displaying the tree structure and with its feature structure. morpho-syntactical information for every word,

Automatic comparison of output structures • during the evaluation of the lexicon interface rules displaying the generated lexical rules for a When developing a grammar it often happens that given word, the parser finds multiple parses for a given sentence. Sometimes these parses differ only by a single feature • during the application of the grammar or se- which may be hard to detect by a human. Automatic mantic rules. comparison of the parses is needed. This can also be used to compare the parses of a given sentence before For GPSG grammars GTU presents every edge and after a grammar modification. produced by the bottom-up chart parser. For DCG It is difficult to assess the effects of a grammar and LFG grammars GTU shows ENTRY, EXIT, modification. Often it is necessary to rerun long FAIL and REDO ports for a predicate, as in a Pro- series of tests. In these tests one wants to save log development environment. But GTU does not the parse structure(s) for a given test sentence if provide options for selectively skipping the trace for a certain level of coverage and correctness has been a particular category or for setting special interrupt reached. Should a modification of the grammar be- points that allow more goal-oriented tracing. Fur- come necessary, the newly computed parse structure thermore, the parser cannot be interrupted by an can be automatically compared to the saved struc- abort option in trace mode. These problems lead to ture. We have included such a tool in GTU. a reluctance in using the trace options since most of The comparison tool works through three subse- the time too much information is presented on the quent levels. First, it checks whether the branching screen. Only elaborate trace options are helpful in structures of two parse trees are identical, then it writing sizable grammars. compares the node names (the constituent symbols), and finally it detects differences in the feature struc- Lexicon interface tures. The procedure stops when it finds a difference The flexible lexicon interface is another of GTU’s and reports this to the user. core elements. With special lexicon interface rules Implementing such a comparison tool is not too that are part of every grammar formalism the gram- difficult, but integrating it into the testing module mar developer can specify which lexicon information of a grammar workbench is a major task, if this mod- the grammar needs and how this information should ule supports different types of tests (single sentence be structured and named. tests and series of tests; manual input and selections For each word a lexicon provides information from the test suite). At the same time one needs about the possible part of speech and morpho- to ensure that the module’s functionality is trans- syntactical information. Lexicon interface rules de- parent and its handling is easy. For example, what termine how this information is passed to the gram- should happen if a sentence had two readings before mar. a grammar modification and has three readings now? A lexicon interface rule contains a test criterion We decided to compare the first two new structures and a specification and has the following format: if in lex (test criterion) then in gram tax’ was subdivided into ’simple verb groups’, ’com- (specification) . plex verb groups’, and ’verb groups with separated prefixes’. The sentences were attached to the phe- The test criterion is a list of feature-value pairs nomena they represented. In this representation the to be checked against a word’s lexical information. grammar developer can select a phenomenon result- Additionally, constraints are allowed that check if ing in the display of the set of subsumed sentences. some feature has a value for the given word. For If multiple phenomena are selected the intersection example, the test of the sets is displayed. (pos=verb, !tense, ~reflexive) It turned out that the latter representation was will only succeed for irreflexive finite verbs2. hardly used by our students. It seems that gram- mar writing itself is such a complex process that a While it is necessary that the test contains only user does not want to bother with the complexities features available in the lexicon, the specification of navigating through a phenomena tree. The other, part may add new information to the information simple representation of sentence classes in files is found in the lexicon. For example, the specification often used and much appreciated. It is more trans- case = #kasus, number = #numerus, person = 3 parent, easier to select from, and easier to modify assigns the value of the feature kasus found in the (i.e. it is easier to add new test sentences). lexicon (which is indicated by #) to a feature named Few other grammar workbenches include an elab- case (and the like for number). Additionally, a new orate test module and only PAGE (Oepen, 1997) feature person is added with the value 3. In this way comprises a test suite which is integrated similarly every noun may get a specification for the person to GTU. PAGE’s test suite, however, is more com- feature. prehensive than GTU’s since it is based on the The specification part defines how lexicon infor- TSNLP (Test Suites for Natural Language Process- mation shall be mapped to a syntactic category in ing) database. TSNLP provides more than 4000 test case the test criterion is met. While the format of items for English, French and German each. We are the test criterion is the same for all formalisms, the not aware of any reports of this test suite’s usability format of the specification has been adjusted to the and acceptability in PAGE. format of every grammar formalism. In this way the definition of lexical entries can be adapted to a gram- Output of recognized fragments in case of mar formalism while reusing the lexical resources. ungrammaticality Writing lexicon interface rules requires a good un- In case a parser cannot process the complete natural derstanding of the underlying lexicon. And some- language input, it is mandatory that the grammar times it is difficult to see if a problem with lexical developer gets feedback about the processed frag- features stems from the lexicon or is introduced by ments. GTU presents the largest recognized frag- the interface rules. But overall this lexicon inter- ments. That is, starting from the beginning of the face has been successful. With its simple format of sentence it takes the longest fragment, from the end rules with conditions and constraints it can serve as of this fragment it again takes the longest fragment a model for interfacing other modules to a grammar and so on. If there is more than one fragment of workbench. the same length, only the last one parsed is shown. The fragments are retrieved from the chart (GPSG) Test suite administration or from a well-formed substring table (DCG, LFG). GTU contains a test suite with about 300 sentences Obviously, such a display is sometimes misleading annotated with their syntactic properties. We have since the selection is not based on linguistic criteria. experimented with two representations of the test As an alternative we have experimented with dis- suite (Volk, 1995). One representation had every playing the shortest paths through the chart (i.e. sentence assigned to a phenomenon class and every the paths from the beginning to the end of the in- class in a separate file. Each sentence class can be put with the least number of edges). In many cases loaded into GTU and can be separately tested. In a such a path is a candidate close to a parsing solution. second representation the sentences were organized In general, it fares better than the longest fragments as leaves of a hierarchical tree of syntactic phenom- but again it suffers from a lack of linguistic insight. ena. That is, a phenomenon like ’verb group syn- Yet another way is to pick certain combinations 2’!feature’ means that the feature must have some of constituents according to predefined patterns. It value, while ’∼feature’ prohibits any value on the is conceivable that the grammar developer specifies feature. an expected structure for a given sentence and that the system reports on the parts it has found. Or the should be allowed to modify the basic lexicon ac- display system may use the grammar rules for se- cording to suggestions sent to him by the grammar lecting the most promising chart entries. Displaying developers. the complete chart, as done in the Xerox LFG-WB, will help only for small grammars. For any sizable Combination of lexical resources grammar this kind of display will overwhelm the user GTU currently does not support the combination with hundreds of edges. of lexical resources. Every lexical item is taken Selecting and displaying chart fragments is an from the one lexicon selected by the user. Miss- interesting field where more research is urgently ing features cannot be complemented by combining needed, especially with respect to treating the re- lexicons. This is a critical aspect because none of sults of parsing incomplete or ill-formed input. the lexicons contains every information necessary. While GerTWOL analyzes a surprising variety of Lexicon extension module words and returns morphological information with high precision, it does not provide any syntactical in- When writing grammars for real natural language formation. In particular it does not provide a verb’s sentences, every developer will soon encounter words subcategorization. This information can be found in that are not in the lexicon, whatever size it has. the PLOD/CELEX lexicon to some degree. For ex- Since GTU was meant as a tutoring tool it contains ample, the grammar developer can find out whether only static lexicons. In fact, its first lexicon was tai- a verb requires a prepositional object, but he cannot lored towards the vocabulary of the test suite. GTU find out which preposition the phrase has to start does not provide an extension module for any of the with.4 attached lexical resources. The grammar developer has to use the information as is. Adding new features Clear modularization can only be done by inserting them in lexicon inter- face rules or grammar rules. Words can be added as The development of a large grammar - like a large terminal symbols in the grammar. software system - makes it necessary to split the work into modules. GTU supports such modular- This is not a satisfactory solution. It is not only isation into files that can be loaded and tested inde- that one wants to add new words to the lexicon pendently. But GTU only recommends to divide a but also that lexicon entries need to be corrected grammar into modules, it does not enforce modular- and that new readings of a word need to be en- isation. For a consistent development of large gram- tered. In that respect using GerTWOL is a draw- mars, especially if distributed over a group of people, back, since it is a closed system which cannot be we believe that a grammar workbench should sup- modified. (Though its developers are planning on 3 port more engineering aspects we know from soft- extending it with a module to allow adding words. ) ware development environments such as a module The other lexicons within GTU could in principle concept with clear information hiding, visualisation be modified, and they urgently need a user inter- of call graphs on various levels, or summarisation of face to support this. This is especially important for selected rule properties. the PLOD-lexicon derived from the CELEX lexical database, which contains many errors and omissions. General remarks on GTU Models for lexicon extension modules can be GTU focuses on grammar writing. It does not in- found in the latest generation of commercial machine clude any means to influence parsing efficiency. But translation systems such as IBM’s Personal Trans- parsing efficiency is another important aspect of lator or Langenscheidts T1. Lexicon extension in learning to deal with grammars and to write NLP these systems is made easy by menus asking only systems. It would therefore be desirable to have a for part of speech and little inflectional information. system with parameterizable parsers. On the other The entry word is then classified and all inflectional hand this might result in an unmanageable degree forms are made available. of complexity for the user and - like with the alter- Of course in a multi-user system these modifica- native test suite - we will end up with a nice feature tions need to be organized with access restrictions. that nobody wants to use. Every developer should be able to have his own sub- The GTU system has been implemented with lexicon where lexicon definitions of any basic lexi- great care. Over time more than a dozen program- con can be superseded. But only a designated user 4The next version of CELEX will contain such prepo- 3Personal communication with Ari Majorin of Ling- sitional requirements. (Personal communication with soft, Helsinki, in December 1996. CELEX manager Richard Piepenbrock in April 1997) mers have contributed modules to the overall sys- NLP environments are designed as platforms for tem. The robust integration of these modules was the development of multi-module NLP systems. possible since the core programmers did not change. Rather than being a closed system they provide They had documented their code in an exemplary a shell for combining multiple linguistic mod- way. Still, the problem of interfacing new modules ules such as tokenizers, taggers, morphology an- has worsened. A more modular approach seems de- alyzers, parsers (with grammars) and so on. A sirable for building large workbenches. grammar workbench is a tool to develop such a module. All the modules can be tailored and 3 Different grammar development tuned to the specific needs of the overall sys- environments tem. We consider ALEP (Simpkins, 1994) and GATE (Gaizauskas et al., 1996) to be examples In order to position GTU within the context of gram- of such environments. Although it seems logi- mar development environments, let us classify them cal and desirable that NLP environments should according to their purpose. provide for the delivery of stand-alone systems this aspect has been neglected so far. In par- Tutoring environments are designed for learning ticular we suspect that the interface format, as to write grammars. They must be robust and required e.g. between GATE modules, will have easy to use (including an intuitive format for negative effects on the processing efficiency of grammar rules and an intuitive user interface). the complete system.5 The grammar developer should be able to fo- cus on grammar writing. Lexicon and test suite GTU was designed as a tutorial system for gram- should be hidden. Tutoring environments there- mar development. Over time it has grown into a fore should contain a sizable lexicon and a test system that supports most functions of experimenta- suite with a clear organisation. They should tion environments. Its main limitations are its closed provide for easy access to and intuitive display architecture and the inability to use the grammars of intermediate and final parsing results. They outside the system. Many of its modules can be em- need not bother with efficiency considerations ployed by an NLP environment. GTU’s most suc- of processing a natural language input. GTU is cessful modules are its flexible lexicon interface, the an example of such a system. tight integration of the test suite and the module for comparison of output structures. Experimentation environments are An NLP environment should be an open platform designed for professional experimentation and rather than a closed workbench, as is the core con- demonstration. They must also be robust but cept of ALEP and GATE. This is needed to allow they may require advanced engineering and lin- special treatment for special linguistic problems. For guistic skills. They should provide for check- instance, the treatment of separable prefix verbs in ing the parsing results. They must support German is so specific that it could be tackled by the grammars and parsers to be used outside a preprocessor before parsing starts. Only after the the development system. We think that Alvey- separated prefix and the main verb have been recom- GDE (Carroll, Briscoe, and Grover, 1991) and pounded the verb’s subcategorization can be deter- Pleuk (Calder and Humphreys, 1993) are good mined. examples of such environments. They allow the Another specific problem of German is the reso- tuning of the parser (Alvey) and even redefin- lution of elliptical coordinated compounds (e.g. In- ing the grammar formalism (Pleuk). The Xerox und Ausland standing for Inland und Ausland). If LFG-WB is partly a tutoring environment (es- such ellipses are filled in before parsing starts such pecially with its grammar index and zoom-in a coordination does not need special grammar rules. displays) and partly an experimentation envi- Other peculiarities such as date, time, currency, dis- ronment since it lacks a test suite and a lexicon. tance expressions will also need special modules. In Note that the systems also differ in the num- this way only the processing of the core syntactic ber of grammar formalisms they support. The phenomena is left to the parser. Alvey-GDE (for GPSG) and the Xerox LFG- An NLP environment should allow parametrisa- WB work only for one designated formalism. tion of parsers or multiple parsers of different pro- GTU has built-in processors for three for- 5GATE requires modules to communicate via a so malisms, and Pleuk supports whatever formal- called CREOLE interface, which is a layer wrapped ism one defines. around an existing module. cessing strategies (e.g. a combination of symbolic velopment environment for large natural language and statistic parsing) and processing depths (e.g. grammars. Technical report, University of Cam- shallow parsing if no complete parse can be found). bridge Computer Laboratory. 4 Conclusions [Gaizauskas et al.1996] Gaizauskas, R., H. Cunning- ham, Y. Wilks, P. Rodgers, and K. Humphreys. Tools like GTU are well suited for learning to de- 1996. GATE: an environment to support research velop grammars, for experimenting with grammar and development in natural language engineering. formalisms, and for demonstrating the work of com- In Proc. of the 8th IEEE Conf. on tools with AI putational linguistics. The use of GTU as an ed- (ICTAI-96). ucational tool in computational linguistics courses [Gazdar et al.1985] Gazdar, Gerald, Ewan Klein, Ge- has been very successful. In a recent project GTU’s offrey Pullum, and Ivan Sag. 1985. Generalized flexibility is being challenged in a joint project with phrase structure grammar. Harvard University the Institute of Germanic Language at the Univer- Press, Cambridge,MA. sity of Koblenz. In this project we examine the use of GTU for the benefit of courses in German as a [Jung, Richarz, and Volk1994] Jung, Michael, Dirk foreign language. Richarz, and Martin Volk. 1994. GTU - Eine For the development of large grammars in combi- Grammatik-Testumgebung. In Proceedings of KONVENS-94, pages 427–430, Wien. nation with large linguistic resources and for pro- cessing them efficiently, GTU is less suited. We [Kaplan and Maxwell1996] Kaplan, R.M. and are now convinced that we need an open platform J.T. Maxwell III, 1996. LFG Grammar Writer’s that provides a framework for combining modules Workbench (Version 3.1). Xerox Corporation. for such a task. For this it is necessary to develop in- terface standards for different types of modules (tag- [Kaplan and Bresnan1982] Kaplan, Ronald and Joan gers, grammars, lexicons, test suites etc.). Bresnan. 1982. Lexical-functional grammar. A formal system for grammatical representation. In Finally, we should keep in mind that a com- Joan Bresnan, editor, The Mental Representa- putational environment for grammar development tion of Grammatical Relations. MIT Press, Cam- offers help in engineering NLP modules for well- bridge,MA. understood phenomena. The real hard problems in NLP (most importantly the resolution of ambigu- [Nederhof et al.1992] Nederhof, M.J., C.H.A. Koster, ity) need to be solved by bringing to bear the right C. Dekkers, and A. van Zwol. 1992. The gram- information at the right time. But this is of yet a mar workbench: A first step towards lingware complex area with many questions that have not re- engineering. In W. ter Stal, A. Nijholt, and ceived a theoretical answer let alone an engineering R. op den Akker, editors, Proceedings of Second Twente Workshop on Language Technology, pages solution. 103–115, Twente, NL. 5 Acknowledgements [Oepen1997] Oepen, Stephan. 1997. PAGE. We would like to thank Diego Moll`a Aliod and Platform for Advanced Grammar Engineer- Gerold Schneider for providing background informa- ing. WWW page (http://cl-www.dfki.uni- sb.de/cl/systems /page), April. tion on some grammar workbenches. [Oy1994] Oy, Lingsoft. 1994. Gertwol. Questionnaire for Morpholympics 1994. LDV-Forum, 11(1):17– References 29. [Baayen, Piepenbrock, and van Rijn1995] [Pereira and Shieber1987] Pereira, Fernando C.N. Baayen, R. H., R. Piepenbrock, and H. van Rijn. and Stuart M. Shieber. 1987. Prolog and Natural- 1995. The CELEX lexical database (CD-ROM). Language Analysis, volume 10 of CSLI Lecture Linguistic Data Consortium, University of Penn- Notes. University of Chicago Press, Stanford. sylvania. [Calder and Humphreys1993] Calder, J. [Simpkins1994] Simpkins, N.K. 1994. An open and K. Humphreys. 1993. Pleuk overview. Tech- architecture for language engineering. The Ad- nical report, University of Edinburgh. Centre for vanced Language Engineering Platform (ALEP). Cognitive Science. In Proceedings of Language Engineering Conven- tion, Paris. European Network in Language and [Carroll, Briscoe, and Grover1991] Carroll, Speech, Centre for Cognitive Science, Edinburgh, John, Ted Briscoe, and Claire Grover. 1991. A de- pages 129–136. [Volk, Jung, and Richarz1995] Volk, M., M. Jung, and D. Richarz. 1995. GTU - A workbench for the development of natural language grammars. In Proc. of the Conference on Practical Applica- tions of Prolog, pages 637–660, Paris. [Volk1995] Volk, Martin. 1995. Einsatz einer Testsatzsammlung im Grammar Engineering, vol- ume 30 of Sprache und Information. Niemeyer Verlag, T¨ubingen.