A New Language for Constraint Grammar: Estonian∗ Kaili M¨u¨urisep Tiina Puolakainen Kadri Muischnek Institute of Cybernetics Institute of Estonian Language Department of Estonian and Tallinn Technical University 10136 Tallinn, Estonia Finno-Ugric Linguistics 12168 Tallinn, Estonia [email protected] University of Tartu [email protected] 50409 Tartu, Estonia [email protected]

Mare Koit and Tiit Roosmaa and Heli Uibo Institute of Computer Science University of Tartu 50409 Tartu, Estonia {koit,roosmaa,heli u}@ut.ee

Abstract problem exists also in Estonian, but the number The Constraint Grammar of Estonian presented of choices is much greater due to the richness of in the paper is the first attempt in automatic forms. syntactic analysis of Estonian. The grammar consists of 1,240 morphological disambiguation The ratio of ambiguous words varies greatly rules, 47 clause boundary detection rules, 180 from language to language: for example, in En- morphosyntactic mapping rules and 1,118 syn- glish, Swedish, and Finnish the ratio of words tactic constraints. The rules have been devised using a training corpus of 20,300 words and have with multiple morphological interpretation is been tested on a benchmark corpus of 10,000 40%, over 60%, and 11%, respectively (Karls- words. As the result of tests, 86.6% of words become morphologically unambiguous, and the son et al. 95). In Estonian literary texts more error rate of the morphological disambiguator than 45% of words are ambiguous. Estonian, un- is 1.8%. The results of the full analysis demon- like Germanic languages, is not subject-centered. strate the ambiguity rate of 83% and error rate of 3.5%. There are a number of non-elliptical sentences in Estonian with no subject. Before we started our project, an automatic 1 Introduction morphological analyzer for Estonian had already The Estonian language is a Finno-Ugric language been created (Kaalep 96). It was our task to elab- that has a rich structure of declensional and con- orate a grammar suitable for the automatic syn- jugational forms, and also a relatively free order tactic analysis of Estonian, and to compile the of sentence constituents. In these respects it dif- program for the syntactic analysis. To accomplish fers considerably from English. There are 14 cases this task, we had to choose a suitable grammar in Estonian, but due to the free word order it is model for Estonian, and analyze the available Es- difficult to determine the syntactic functions of tonian texts, primarily from the Corpus of Writ- these cases. Furthermore, there is no grammati- ten Estonian Texts (Hennoste et al. 98), in order cal gender. The person agreement (1st, 2nd, and to use the established regularities for wording the 3rd person in singular and plural) is common in rules and writing the syntactic parser. finite verbs in all forms and tenses. The major- Our grammar has been composed on the for- ity of grammatical categories are implemented by malism of the Constraint Grammar (Karlsson et means of morphology. al. 95). The main idea of the Constraint Gram- Estonian is characterised by a wide extent and mar (CG) is that it determines the surface-level variety of grammatical homonymy that makes the syntactic analysis of the text, which has gone automatic analysis of Estonian a difficult task. In through prior morphological analysis. The pro- the case of English, the main difficulty lies in de- cess of syntactic analysis consists of three stages: termining the correct part of speech. The same morphological disambiguation, identification of ∗ This work was supported by the Estonian Science clause boundaries, and identification of syntactic Foundation under the grants No. 3314 and No. 4605. functions of words. The underlying principle in determining both (came) the morphological interpretation and the syntac- tuli+0 // S com sg nom // (the light) tic functions is the same: first all the possible $. labels are attached to words, and then the ones that do not fit the context are removed by ap- Let us note that the word forms in this sentence plying special rules called constraints. Constraint can be sequenced in 3! = 6 different ways, and all Grammar consists of hand written rules, which by the resulting sentences will be correct and under- checking the context decide whether an interpre- standable for a native speaker of Estonian due to tation is correct or has to be removed. the free word order. The Constraint Grammar parser of Estonian In analyzing this sentence, the correct interpre- exists as two separate programs: the morphologi- tation of the word form tuli is found by applying cal disambiguator (Puolakainen 01) and the syn- the following constraint: remove the finite form tactic analyzer in a narrower sense (M¨u¨urisep00). of the verb from the cohort (in the present case The basic differences of our grammar from the verb tule+i // V main indic impf ps3 sg ps af #Intr //) standard one are the following: if a given word is immediately preceded by a fi- nite form of verb which is the only interpretation • the assumed clause boundaries are also used; of the word form (in the present case kustu+s // V • the referenced context conditions can be rep- main indic impf ps3 sg ps af #Intr // ). resented in two ways: the appropriate cohort After having added the syntactic tags we have is searched either up to the very end of the the sentence in the following form: possible context, or it is searched up to the Aknas first appropriate elements/filler; aken+s // S com sg in **CLB // @ADVL @ • it is possible to remove morphological inter- kustus pretations during the syntactic analysis. kustu+s // V main indic impf ps3 sg ps af #Intr // The next sections of the paper will provide an @+FMV overview of the Constraint Grammar of Estonian tuli (EstCG) and problems that cropped up in its cre- tuli+0 // S com sg nom // @SUBJ @OBJ @ADVL ation. We think that other researches who set the @NN> @object (@OBJ), adverbial (@ADVL), pre- modifing attribute (@NN>), or postmodifing at- 2 Motives for Selecting Constraint tribute (@genitive case. that leading position from its introduction to the Among the attribute rules, the simplest are present day. those seeking whether in the left or right context there exists at all a word they may complement, 3 Method of Elaborating Rules as well as numerous rules checking the agreement or non-agreement. A number of rules are clearly To elaborate the morphological disambigua- of heuristic nature – the rule might not be 100% tion constraints, we established the more fre- true but its proficiency rate is very high, com- quent groups of ambiguities. We found both more pared to the number of errors. Several rules have frequent ambiguous word forms as well as the am- been compiled solely on the statistical informa- biguous grammatical categories (the past partici- tion. While observing the word order in the sen- ple with the interpretation of the adjective either tence, it became obvious that such combinations in singular or plural, the noun or the verb; noun in as 1) object in the nominative or genitive case – the nominative, genitive and partitive case; noun predicate – subject in the nominative case, or 2) in the genitive, partitive and aditive; adverb and object in the nominative or genitive case – sub- adjective in the ablative case, etc.) (Puolakainen ject in the nominative case – predicate occur very 01). This frequency table indicated which phe- rarely. nomena needed to be handled first of all. For each We tried to group the rules in such a way case samples were collected from text corpora and that the most reliable ones or those that cause upon these observations tentative rules were com- least errors are in the main part of the grammar; piled. The rules were tested against a manually the heuristic rules in turn have been divided into tagged corpus where the correct interpretations groups based on their reliability. provided by linguists had the corresponding tags The rules have been written in a way to reduce attached to them. Depending on the results the the amount of errors in the training corpus. The rules were revised and tested again until their im- rule may go wrong only in exceptional cases or pact was satisfactory. due to a mistake caused by another rule. While It was quite easy to compile the syntactic writing the rules we had in mind the objective of mapping rules. For the majority of rules it obtaining 99% recall of the parser on the training was necessary to transform the linguistic rules corpus. taken from the Grammar of the Estonian Lan- guage (Erelt et al. 93) into a suitable computa- 4 Rules in the Constraint Grammar tional grammar form. Then the manually tagged of Estonian text and the parser output were compared and most of the exceptions detected. 4.1 Rules of morphological Compiling the syntactic constraints (which disambiguation remove the added tags) was not that simple. The There are 1,240 rules in the morphological disam- first rules were derived from the grammar books: biguation grammar half of which treat concrete all kinds of rules of agreement check the agree- word forms. The order of applying the rules is ment of the predicate with the potential subject important. For example, it is essential to apply in person and number, and the agreement of at- the phrasal verb rules before applying other rules tributes with their heads in the case. Also a num- as these determine most appropriately the part of ber of rules checking the case of the object were speech of doubtful adverbs. As it is not possible based on grammars. to present the whole list of rules in this paper, we When the grammar books were exhausted the will bring some examples from different groups. rest of the rules had to be generated on the basis Phrasal verb rules of intuition: to examine a partly analyzed text The gist of the problem lies in the fact that many adverbs (affixed adverbs) in the phrasal Choose the conjunction interpretation if kui is verb may in certain context be either adpositions immediately preceded by a comma, the last but (¨ule – adverb in the phrase ¨uleelama /to survive/ one word form is siis /then/ or if the right context and preposition in the phrase ¨ulej˜oe /across the contains the word form siis. For example: Kui river/), or coincide with some case of the noun (Conj) homme sajab, siis j¨atanminemata /If it (k¨atte – adverb in the phrase k¨attejuhatama /to rains tomorrow then I’ll not go/, as well as Siis, indicate to/ and the noun k¨asi /a hand/ in the kui (Conj) homme sajab, j¨atanminemata. inessive case). This group is represented, for ex- Choose the adverb interpretation if the follow- ample, by the rule: select the adverb reading of ing word is one of the list: mitu /a number of/, the word form k¨atte if either to the left or to the v¨ahe /little, few/, palju /a lot of, much/, hea right of this word in the clause occur forms of /good/, halb /bad/, h¨asti /well/, kaua /long/, the verbs v˜otma or juhatama. E.g., juhatas tee vana /old/, noor /young/, raske /difficult/, kerge k¨atte /indicated the way/ (adverb) but pani kin- /easy/. For example, Kui (adverb) kaua see dad k¨atte /put the gloves on/ (noun). kestab, ei tea keegi /Nobody knows how long it Determining the case of a declinable word will last/. based on the preceding numeral 4.2 Clause boundary detection rules The cardinal (except ¨uks /one/) as well the word mitu /several/ in the nominative case is followed The clause boundaries are determined by rules by a noun, pronoun or adjective in the partitive based on conjunctions, punctuation marks, and case, in other cases there has to be agreement verbs. The basic rule runs as follows: if a word is between these words. For example, kaks ( N card preceded by a punctuation mark, and/or the word sg nom) maja ( S sg nom gen part) /two houses/ itself is a conjunction, and in the right and left and kahe ( N card sg gen) maja ( S sg nom gen contexts there is a conjugable form of a verb, then part). the word is the first word of a clause. In EstCG Selecting between the noun and the adjec- there are 47 clause boundary detection rules, a tive number of them for very specific cases. This very frequent ambiguity is typically repre- 4.3 Morphosyntactic mapping rules sented by words like haige /ill/, vana /old/; kao- 27 syntactic tags of EstCG represent syntactic taja /looser/, v˜oitja /winner/; all the past par- functions of traditional Estonian grammar (Erelt ticiples. et al. 93), although there are some modifications Choose the part of speech noun if there is a considering the specialities of Constraint Gram- preceding disambiguated adjective in the clause, mar: CG annotates every word with some syntac- there are no punctuation marks nor a verb allow- tic label while a linguistic grammar has a more ing a predicate and no other nouns or pronouns general view treating multiple words as units. in the clause, e.g. rasket ( A sg part) haiget ( S The syntax used in CG is word based, i.e., no sg part) /gravely ill/. hierarchical phrase structure is constructed. The The following is a selection of rules pertaining phrasal heads are labelled as subjects, objects, to certain word forms. adverbials, or predicatives (complements of pred- The word form tuli /fire, came, had to/ is icate). The modifiers have tags that indicate the provided by two interpretations by the morpho- direction where the head of a phrase could be logical analyzer: verb form (came, had to) and found, but the modifiers and heads are not for- noun (fire). The word form is very probably a mally connected. The verb chain is marked by verb if there is either a preceding or a following five labels: finite or infinite, auxiliary or main da-infinitive form in the clause, eg tuli ( V main verb, and a label for negation. indic impf ps3 sg ps af) lugeda ( V main inf) Syntactic tags are added to words by 180 mor- /one had to read/. phosyntactic mapping rules. While compiling The word form kui /if, as, how/ may be ei- these it was taken into account that each rule ther adverb (how) or conjunction (if, as) . In should be optimal – there should be a correct tag disambiguating this form we proceed mainly from among the tags added by the rule, and the rule the fact that the adverb kui must be followed by should avoid adding rarely occurring tags to all either an adverb or an adjective. words. This has been achieved by adding con- text conditions to the rules. Thus, the adpo- the subject. The adverbs functioning as quan- sition tags @P> or @

comparison (there are no such words as nagu The grammar of syntactic analyzer was tested on /as/, kui /as/, otsekui /as if/, justkui /as if/ to two types of corpora: the first one was manu- its left). ally morphologically disambiguated, i.e. all the errors of previous stages of analysis were fixed 5 Results and morphological ambiguities removed by lin- guists. This type of corpus helped to determine 5.1 The corpora used the problems that are specific for syntactic dis- To compile and test the EstCG, a 20,314 word ambiguation only. The second corpus consists of corpus of Estonian was used that was manually same texts but the preceding analysis is achieved tagged for morphological and syntactic features automatically. (hencetoforth referred to as the training corpus). While compiling the grammar the goal was set The corpus consists of six original fiction texts, to obtain as disambiguous analysis as possible on each from a different author (12,223 words alto- the training corpus, at the same time maintaining gether, derived from the Corpus of Written Esto- a high rate of recall (at least 98.5%). The recall of nian), a 6,373 word translation of a fiction text all texts did not fall below 98.9% in the training (”1984” by G. Orwell), and a 1,719 word newspa- corpus, whereas the ratio of unambiguous anal- per text. yses differs more than 7%, from 86.1% to 93.4% To assess the effectiveness of the parser, a man- (precision is 89.61% for the whole corpus). It is ually tagged 9,663 word test corpus was used that lowest for the newspaper text which is stylistically had not been applied for optimizing nor evaluat- a lot different from the others. ing the grammar rules. The results of syntactic disambiguation in the test corpus were a bit less promising but the dif- 5.2 Results of morphological ference was not too big: 0.7% reduction in re- disambiguation call and 2% reduction in precision. When ana- lyzing the results of the test corpus it should be The use of the disambiguator reduced the per- noted that the journalistic text is more difficult centage of ambiguous word forms 4 times in both for the parser. The remaining ambiguity mostly the training and the test corpus. consists of two tags, but single words may have The results of morphological disambiguation in from 3 to 5 syntactic tags. The largest class of the training corpus are the following: precision ambiguities is formed by adverbials and adver- is 83.39-89.68%, recall is 97.87-99.16%, and the bial attributes. This is almost the same problem percentage of words with one reading is 88.67- as PP-attachment in English, but additionally it 91.74%. is possible to use both premodifying and post- The results of morphological disambiguation modifying adverbial attributes in Estonian. Of in the test corpus are the following: precision course, the PP-attachment problem is also exis- is 85.49-89.16%, recall is 97.95-98.36%, and the tent. The other complicated problem is the dis- percentage of words with one reading is 88.67- tinction of genitive attributes and objects, which 91.96%. are followed by any other noun. To make things Most of the mistakes occurred when the cases of even worse the morphological disambiguator often nouns, adjectives, and pronouns were determined: fails to solve the morphological ambiguities in the the difficulties occur in differentiating the nomi- same position: noun in genitive case is frequently native, genitive, partitive, and aditive cases. It ambiguous between nominative or partitive case. was also the leading ambiguity type in the initial So the most difficult problem in the Estonian text and in remaining ambiguities after disam- language appears to be determining the borders biguation as well. The second place in the errors of noun phrases. It is often hard to decide which toplist was taken by the problem of determining adjacent nouns belong to a common noun phrase, whether past participles were used as adjectives or and which form separate noun phrases. verbs, and the third is closely related to the sec- Having analyzed the errors, we found that ond one – determining whether the form of the there were no systematic misinterpretations in the verb olema /to be/ is the main or auxiliary verb. grammar. The errors are mostly caused by ellip- sis, some errors occurred during determination of than the nominative, genitive, and partitive. apposition and the third biggest group of errors The problem of complex and phrasal verbs exists in sentences where one clause divides the needs solving as well. At the moment their other into two parts. Most of the errors can be nominal and adverbial components are ana- avoided by refining the contextual conditions of lyzed according to their morphological form, the rules. The rate of erroneous analyses is as but it would be necessary to relate them to well below 2% that was set as the goal in compil- the verb. To do so, we need exhaustive lists ing the grammar. of complex and phrasal verbs (this work has The results obtained from the automatically already been launched, cf. (Kaalep & Muis- morphologically disambiguated corpora are sup- chnek 02)). posedly worse. In the test corpus, the recall of syntactic disambiguation was 95.50-96.85%, pre- • The volume of the tagged training corpus has cision 76.36-79.22%, and the unambiguity 81.34- to be increased. 83.63%. • To experiment with different statistical The analysis of errors on that type of corpus methods for the automatic generation of demonstrates that the majority of the additional grammar on the bigger training corpus. errors have directly been caused by the errors of the morphological disambiguator and by words • To expand the amount of syntactic tags. A unrecognised by the morphological analyzer. more detailed tagging should considerably in- crease the efficiency of the object analysis. 6 Conclusion • In the longer perspective we foresee a need Constraint grammars have been written for for transition to a deeper description of the the Basque, English, Norwegian, Portuguese, syntax, e.g., to take over the principles of the Swedish, Turkish, and now for the Estonian lan- Functional Dependency Grammar (J¨arvinen guage. & Tapanainen 97). The computational grammar and the parser elaborated during our project are the first at- tempts to automate the syntactic analysis of Es- References tonian. The 2,500-odd rules of the Constraint (Bangalore 97) Srinivas Bangalore. Complexity of Lexical Descrip- tions and its Relevance to Partial Parsing. Ph.d. diss., Univer- Grammar of Estonian have been formed on the sity of Pennsylvania, 1997. basis of the standard linguistic grammar of Esto- (Erelt et al. 93) Mati Erelt, Reet Kasik, Helle Metslang, Henno Rajandi, Kristiina Ross, Henn Saari, Kaja Tael, and Silvi Vare. nian and the texts from the corpus of the written Eesti keele grammatika. II S¨untaks. Eesti TA Keele ja Kirjan- Estonian. The parser has been successfully used duse Instituut, Tallinn, 1993. in two prototypes of practical applications: recog- (Hennoste et al. 98) Tiit Hennoste, Mare Koit, Tiit Roosmaa, and Madis Saluveer. Structure and usage of the Tartu University nizing the noun phrases and the automatic gen- corpus of written Estonian. International Journal of Corpus eration of text summaries. The promising fields Linguistics, 3(2):279–304, 1998. of application include information retrieval (tak- (J¨arvinen& Tapanainen 97) Timo J¨arvinenand Pasi Tapanainen. A dependency parser for English. TR 1, Department of General ing into account the typical patterns of sentences Linguistics, University of Helsinki, 1997. while dealing with a specific topic, e.g., (Banga- (Kaalep & Muischnek 02) Heiki-Jaan Kaalep and Kadri Muischnek. Using the text corpus to create a comprehensive list of phrasal lore 97), text-speech generation (partial syntac- verbs. In Proceedings of the Third International Conference on Language Resources and Evaluation, pages 101–105, Las Pal- tic analysis for determining the intonation of the mas, 2002. sentence), grammar checking in text editing, and (Kaalep 96) Heiki-Jaan Kaalep. Estmorf. a morphological analyzer translator aids (e.g., search for standard sentence for Estonian. In Estonian in the Changing World, pages 43–98. Tartu, 1996. constructions from a parallel corpus). (Karlsson et al. 95) Fred Karlsson, Atro Voutilainen, Juha Heikkil¨a, The directions of our future work are as follows. and Arto Anttila. Constraint Grammar: a Language Indepen- dent System for Parsing Unrestricted Text. Mouton de Gruyter, • The share of the lexicon in the grammar Berlin and New York, 1995. (M¨u¨urisep 00) Kaili M¨u¨urisep. Eesti keele arvutigrammatika: must be increased. We need a lexicon con- s¨untaks. Ph.d. diss., University of Tartu, 2000. taining certain semantic information cover- (Puolakainen 01) Tiina Puolakainen. Eesti keele arvutigrammatika: ing, first of all, noun quantifiers and nouns morfoloogiline ¨uhestamine. Ph.d. diss., University of Tartu, that can fulfill the functions of adverbials of 2001. time and manner occurring in other cases