Inferring morphological rules from small examples using 0/1 linear programming Ann Lilliestrom,¨ Koen Claessen, Nicholas Smallbone Chalmers University of Technology Gothenburg, Sweden fannl,koen,[email protected] Abstract Here, +’s∗ represents the suffix ’s, and +a+ repre- 1 We show how to express the problem of sents an infix a. The table shows that both ’s and finding an optimal morpheme segmentation an apostrophe can mark the genitive case; the sec- from a set of labelled words as a 0/1 linear ond column means that the genitive was marked programming problem, and how to build by ’s only in singular nouns, and by an apostrophe on this to analyse a language’s morphology. only in plural nouns. An s suffix marks plural, and The approach works even when there is because of the tiny input data it was only seen in very little training data available. genitive nouns. Plural can be marked by an inner vowel change from a to e (indicated by the arrow), 1 Introduction or by changing a final y to ies, in which case the Many recent tools for morphological analysis use singular form is marked by a or y. statistical approaches such as neural networks (Cot- The tool also segments the input words into mor- terell et al., 2018). These approaches can profitably phemes consistent with the table of rules (the in- use huge amounts of training data, which makes ferred stem is marked in bold): them ideal for high-resource languages. But if there standard inflected is little training data available, statistical methods womjajnj ? womjejnj ? can struggle to learn an accurate model. And when Sg Nom Pl Nom babjyj ? babjiesj ’ they learn the wrong model, it is difficult to diag- Sg Nom Pl Gen dogj ? dogjsj ’ nose the error, because the model is a black box. Sg,Nom Pl Gen This paper presents a new approach to catj ? catj ’s j? Sg,Nom Gen Sg morphological analysis that produces human- lorrjyj ? lorrjiesj ? understandable models and works even if only a Sg Nom Pl Nom Our key idea is a novel morphological segmen- few training words are given. The user gives a list tation algorithm used to produce the segmentation of inflected words together with each word’s mor- above (section 3). Once we have a segmentation, phosyntactic features and standard form (lemma): we use it to compute the rules on the left (section 4). standard inflected features In this example, the rules inferred by our tool pre- woman women Pl;Nom cisely capture the morphology of the input words, baby babies’ Pl;Gen and the segmentation shows which rules are most dog dogs’ Pl;Gen common. We claim that, together, they can be cat cat’s Sg;Gen used to perform high-quality morphological analy- lorry lorries Pl;Nom sis. We validate this claim in section 5 by showing Our tool then proposes, for each feature, the affixes how to build on our tool to do reinflection. and morphological rules which mark that feature. For the example above it suggests the following: 2 Related Work feature context morpheme Gen Sg +’s∗ Morphological segmentation is a well-studied prob- Gen Pl +’∗ lem in NLP, with applications including machine Pl Gen +s∗ translation (Green and DeNero, 2012), speech Pl Nom +a+ ! +e+ 1The tool also supports prefixes such as ∗un+, and cir- Pl +y∗ ! +ies∗ cumfixes such as ∗ge+t∗ in the German gerannt (run; past Sg +y∗, +a+, ? participle). We explain the meaning of ∗ and + in section 3.3. recognition (Rajagopal Narasimhan et al., 2014) of multiple discontinuous pieces, or be empty. In and information retrieval (Turunen and Kurimo, the segmentation on page 1, for example, the word 2008); for a more detailed overview we refer women is segmented by assigning Pl the segment the reader to Ruokolainen et al. (2016) or Ham- e, Nom the null segment, and woman the discon- marstrom¨ and Borin (2011). Unsupervised learning tinuous segment wom*n. In general, the segment (Harris, 1955; Creutz and Lagus, 2007; Goldsmith, assigned to the lemma feature represents the stem 2001; Johnson, 2008; Poon et al., 2009) relies on of the word. unannotated text, and is perhaps the most popular We say a segmentation is valid if for each word: approach, because of the large amount of unanno- tated text available for many languages, but it can • Every letter is in exactly one segment. suffer from low accuracy (Hammarstrom¨ and Borin, • Each feature labels exactly one segment. 2011). One way to improve accuracy is to exploit semantic information (Sakakini et al., 2017; Vulic´ • Each segment is labelled with exactly one fea- et al., 2017; Schone and Jurafsky, 2000; Soricut and ture.2 Och, 2015). Another is minimally-supervised learn- ing (Monson et al., 2007; Kohonen et al., 2010; There are many valid segmentations, some good, Ruokolainen et al., 2013; Gronroos¨ et al., 2014; some bad. We consider a segmentation good if it is Sirts and Goldwater, 2013; Ahlberg et al., 2014), parsimonious: each morpheme should be marked which combines a large amount of unannotated by as few features as possible. Note that different text with a small amount of annotated text, and forms of a word may be assigned different stems potentially provides high accuracy at a low cost. (e.g. go and went), but parsimony dictates that we Silfverberg and Hulden (2017) observe that in share stems wherever reasonable. Perhaps surpris- the Universal Dependencies project (Nivre et al., ingly, finding the most parsimonious segmentation 2017), each word is annotated with its lemma and is an NP-hard problem, by reduction from set cover. features, but not segmented. They study the prob- lem of finding a segmentation for these words. Our 3.1 Segmentation as a constraint problem segmentation algorithm solves the same problem, We solve the segmentation problem using zero-one and can be used in their setting. We improve on linear programming (0/1 LP). A 0/1 LP problem their solution by using a constraint solver to achieve consists of a set of variables (e.g. x; y; z; : : :) and high accuracy even with limited data, and using a a set of linear inequalities over those variables (e.g. precise language for expressing affixes and rules, 2x + 3y ≥ z and x ≤ 2y + 3). Given such a which allows us to use the resulting model to pre- problem, a 0/1 LP solver finds an assignment of cisely analyse new words. values to the variables that makes all the inequali- Luo et al. (2017) use Integer Linear Program- ties hold, and where all variables have the value 0 ming for unsupervised modelling of morphological or 1. A 0/1 LP problem also specifies a linear term families. They use ILP as a component of a larger whose value should be minimised (e.g. 4x+y −z), training algorithm, so unlike our work they do not called the objective function. A 0/1 LP solver is attempt to find a globally optimal solution. ILP guaranteed to find the solution that minimises the has been used in other NLP applications outside objective function, if a solution exists. The solver of morphology (Berant et al., 2011; Roth and Yih, that we use is CPLEX (ILOG, 2009). 2005; Clarke and Lapata, 2008). In this section we assume that we are given, in addition to the labelled words, a (possibly large) 3 Morphological segmentation set of allowable segments for each feature, which we call candidate morphemes. How candidate mor- The input to our tool is a list of words, each anno- phemes are automatically generated is explained in tated with its lemma and morphosyntactic features. section 3.2. From now on, we model the lemma as simply an- Our encoding uses the following variables. If m other feature; for example, the word “women” has is a candidate morpheme of feature f, the variable the features Nom, Pl and woman. The goal of this section is to divide each word 2This is an unrealistic assumption, but also fairly harmless: when a morpheme marks (e.g.) genitive plural, it will be into segments, and assign each feature of the word assigned one of those features, but the inferred rules will show to one of those segments. A segment may consist that it only occurs together with the other feature. Sf;m should be 1 if m is used to mark f, and 0 Example For the above example, the first rule otherwise. If f is a feature with a candidate mor- generates the following constraints, which force an pheme m that occurs in word w, and P is the set S-variable to 1 when one of its M-variables is 1: of positions in w occupied by m, then the vari- S ≥ M able Mw;P;f;m should be 1 if this occurrence of m hast,h¨ ast¨ hast,1–4,h¨ ast,h¨ ast¨ marks feature f, and 0 otherwise. SPl,ar ≥ Mhast,5–6,Pl,ar¨ Example Suppose that we are given the prob- SPl,na ≥ Mhast,7–8,Pl,na¨ lem of segmenting the Swedish word hastarna¨ (the SDef,ar ≥ Mhast,5–6,Def,ar¨ horses). For simplicity, suppose that we know that SDef,na ≥ Mhast,7–8,Def,na¨ the stem of hastarna¨ is hast¨ , and that the possible affixes are ar and na. (For now, we ignore the fact The second rule generates the following con- that these should be suffixes; we deal with this in straints, since letters 1 to 4 can only be covered by section 3.3.) This results in the following candidate hast¨ , letters 5 to 6 by ar (either as Pl or Def), and morphemes: letters 7 to 8 by na: feature candidate morphemes Mhast,1–4,h¨ ast,h¨ ast¨ = 1 Pl ar, na M + M = 1 Def ar, na hast,5–6,Pl,ar¨ hast,5–6,Def,ar¨ hast¨ hast¨ Mhast,7–8,Pl,na¨ + Mhast,7–8,Def,na¨ = 1 The string ar appears at positions 5–6 The third rule generates the following con- in hastarna¨ , and na appears at positions straints, stating that hast¨ , Pl and Def must be 7–8.