Visualising Finnish Word Formation in a Browser-Based Reading Assistant

Show, Don’t Tell: Visualising Finnish Word Formation in a Browser-Based Reading Assistant

Frankie Robertson University of Jyvaskyl¨ a¨ [email protected]

Abstract is to avoid presentations which tell learners about word-level grammatical features such as technical This paper presents the NiinMikaOli?!¨ read- linguistic language including Latinate names for ing assistant for Finnish. The focus is upon the simplified presentation and visualisation of nominal cases, rather opting to highlight their sur- a wide range of word-level linguistic phenom- face forms. The user interface visualises the con- ena of the Finnish language in a unified form nection between surface forms, analytic forms and so as to benefit language learners. The system definitions. is available as a browser extension, intended Described first is the construction of the base- to be used in-context, with authentic texts, in line reading assistant system. The system is order to encourage free reading in language learners. by and large similar to existing systems such as GLOSSER (Nerbonne et al., 1998) or the reading assistant features of SMILLE (Zilio et al., 2017) or Revita (Katinskaia et al., 2017) – or even very 1 Introduction widely used systems such as the WordNet-based This paper presents an intelligent reading assistant alternative translations shown when a single word for Finnish. The system, NiinMikaOli?!¨ (English: is selected in Google Translate. Notable as an TheWhatNow?!), presents word and idiom defini- improvement over some of those systems is Ni- tions in-context. The system can be used through a inMikaOli?!¨ ’s use of a full scale Word Sense Dis- web interface either as a dictionary or by manually ambiguation (WSD) system. The rest of this paper entering or copying text into a text field, or ide- describes the motivation behind and implementa- ally, as a browser extension to assist with reading tion of NiinMikaOli?!¨ ’s visualisations and its ex- Finnish web pages. When used as a browser ex- haustive treatment of complex lexical items such tension, NiinMikaOli?!¨ presents word definitions as derived words, compounds and multiword ex- in a sidebar. pressions. There is increasing interest in contextualised learning of vocabulary (Godwin-Jones, 2018), and 2 Baseline system NiinMikaOli?!¨ aims to facilitate this in the context of web pages. NiinMikaOli?!¨ can be clas- A combined lexical resource of Finnish was cre- sified as an ATICALL (Authentic Text Intelligent ated by combining two sources: FinnWordNet Computer Aided Language Learning) system, de- (Linden´ and Carlson, 2010) and Wiktionary. The Wiktionary definitions were extracted from pub- fined by Meurers et al. (2010a) as software which 1 produces enhanced input based on real texts. licly available dumps, using a Python script . The The focus of this paper is upon NiinMikaOli?!¨ ’s Python script parses MediaWiki markup into word mwparserfromhell2 simplified, unified view of the Finnish language, senses using . which uses information visualisation techniques to At least one definition was extracted from “show rather than tell” learners about morpholog- 99.8% of a total of 153 196 Wiktionary pages con- ical and word formation features. A principle aim 1Available at https://github.com/frankier/ This work is licensed under a Creative Commons Attri- wikiparse. bution 4.0 International Licence. Licence details: http: 2https://github.com/earwig/ //creativecommons.org/licenses/by/4.0/. mwparserfromhell

Frankie Robertson 2020. Show, don’t tell: Visualising Finnish word formation in a browser-based reading assistant. Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020). Linköping Electronic Conference Proceedings 175: 37–45. 37 1.0 3 Remaining proportion taining Finnish as a section heading . Of these, Compounds per token

90 653 are lemmas rather than inflected forms. For 0.8 comparison, FinnWordNet contains 139 871 headwords, which are mostly lemmas but include oc- 0.6 casional idiomatic word forms such as humalassa Proportion (literally “in hops”). 0.4 FinnWordNet is modelled after WordNet, and as such has very fine-grained sense distinc- 0.2 tions. This results in potentially overwhelming the 0.0 learner with too much information. Furthermore, 100 101 102 103 104 105 106 107 Rank some Wiktionary senses are likely to essentially duplicate FinnWordNet. Thus, similar definitions Figure 1: Proportions related to words unknown to a should be clustered together and only the best def- simplified model of a language learner. The x-axis gives inition displayed by default. the rank of the word that the learner has learned all words up The clustering and alignment was created using to. Remaining proportion is the proportion of words in run- affinity propagation (Frey and Dueck, 2007). The ning text unknown to the language learner. Compounds per token is the proportion of unknown words which are com- distances graph is constructed by taking cosine pounds. distances between definitions, represented as vec- tors based on the English text of their definitions according to the English sentence similarity model 3 Linguistic rationale of Reimers and Gurevych (2019). This model is 4 based on pretrained English BERT models fine- Finnish is morphologically rich . Substantives tuned on a semantic similarity task. The pretrained are declined for case and number and verbs are bert-large-nli-stsb-mean-tokens conjugated for person, tense and voice. Finnish 5 model is used. Links between Wiktionary defini- word formation is also rich . It includes a number tions with distinct etymologies are then removed of highly productive derivational morphemes, in- and extra weight is given to Wiktionary definitions cluding many deverbal morphemes which is char- so as to encourage them to become exemplars of acteristic of the language. Compounding also clusters. The resulting system obtains adjusted plays a major role in Finnish word formation, with rand index scores of 0.48 on a gold standard many of the compounds being semantically trans- of WordNet verbs grouped by PropBank sense parent. Finnish also has a number of enclitic obtained from Predicate Matrix (de Lacalle et al., particles such as the question forming “-ko/-ko”.¨ 2016), and a score of 0.52 on a manually created Finally, it has MWEs (Multi-Word Expressions) clustering of 128 Wiktionary and WordNet noun such as idioms. In Finnish these may take the form definitions. The scikit-learn (Pedregosa et al., of syntactic frames, treated here as gapped MWEs 2011) implementation of affinity propagation is e.g. “pitaä¨ -sta”, which could occur in a form used. such as “pidan¨ voileipakakusta”¨ (English: I like sandwich cake) distinguished from e.g. “pidan¨ WSD is performed using UKB (Agirre et al., voileipakakun”¨ (English: I keep sandwich cake). 2014). Since UKB is a graph based WSD al- Why bother going to the effort of making a gorithm, it only operates on definitions from comprehensive treatment of word formation and FinnWordNet, which are connected by the se- complex word types? After all, these lexical items mantic links from Princeton WordNet. In order occur relatively infrequently in running text and to compare WordNet definitions with Wiktionary so it may seem like a poor allocation of effort to definitions, the clustering is used. Clusters are spend time dealing with them. One assumption then ranked using their best WordNet definition as here is that these elements become more impor- a representative. Since Wiktionary definitions are tant after the beginner stage of language learn- usually better for learners, they are pushed to the ing. If we assume a very simplified model of top of each cluster in the user interface. lexical acquisition where words are learnt in de-

4See for example Karlsson (2015). 3In a Wiktionary dump from 6/4/2019. 5See for example Hyvarinen¨ (2019).

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 38 scending order of frequency, we can analyse prop- they draw attention away from the comprehen- erties of words that the language learner does not sible input needed for true language acquisition. know and therefore may like to look up. Figure 1 This large amount of extra material can lead to shows two such properties varying as the number reduced confidence from learners. Secondly, due of words the learner knows increases: the propor- to Finnish’s agglutinative morphology, contrasted tion of all words seen which are unknown, and with a fusional language like Latin, it is simply the proportion of unknown words which are com- not necessary to add this extra layer of analyti- pounds. The data is based on 1.5 billion tokens cal language, since many Finnish inflectional mor- of analysed Finnish text from the Turku Internet phemes occur in the same form or an easily recog- Parsebank (Laippala and Ginter, 2014). Taken as a nisable form at all times, and they can thus be re- whole, the corpus is 9.8% compounds. Supposing ferred to by their normalised form. Consider for that an intermediate learner may know somewhere example, the Finnish system of locative case end- between 1000 and 10 000 words. After learning ings. These have a fairly good correspondence 1000 words, 24% of unknown words would be in terms of function with prepositions in English. compounds, and after 10 000, it would be 42%. Imagine if, when teaching English, every preposi- Thus quite a large proportion of words unknown tion was also given a name to describe it so that to intermediate level learners are compounds. It is we would always refer to “from” as “the elative assumed that other complex lexical items such as preposition”. It is hard to imagine that a learner MWEs follow a similar pattern. Here we refer to would be well served by this extra indirection! any item which can be given a definition, includ- This principle is somewhat flexible, however, and ing lemmas, individual morphemes and MWEs as the names of the most common case endings — headwords. partitive and genitive — are shown on the basis that their usage is more grammatical. They are Admitting these items are frequent, the next more often obligated by context rather than used question becomes, why is simple lemma ex- with the intention of conveying extra information. traction not sufficient? One argument against Plural is referred to by-name since it is likely to be performing lemma extraction and simply show- familiar. ing lemmas comes from the noticing hypothe- sis (Schmidt, 1990) which states that without at- 4 Implementation tention to form (Lightbown and Spada, 2013, pp. 168–175), second language learners in partic- The pipeline from running text to analytical seg- ular are prone to not acquiring fine-grained gram- ments, described in this section, is shown in Fig- matical knowledge. Following this concept, sys- ure 2. tems such as those of Meurers et al. (2010b) and Reynolds et al. (2014) were created to automati- 4.1 MWE lexicon & extraction cally enhance input in web pages in order to pro- FinnMWE (Robertson, 2020) is used as a lexicon mote noticing of, for example, parts of speech. Ni- of MWEs. In order to extract MWEs from running inMikaOli?!¨ follows a similar direction, but in- text, for each MWE is indexed by all possible lem- stead focusses on morphemes, drawing attention mas of a key token. In case the head is known, it to the connection and overlap between analytic is used as the key token, otherwise the rarest to- and surface forms, so to promote learning of mor- ken based on wordfreq (Speer et al., 2018) is used. phology, as well as attention to the formation of MWEs are then extracted from dependency trees the word itself. obtained using the Turku neural dependency pars- Why analyse words using only normalised seg- ing pipeline (Kanerva et al., 2018). First, all key ments, rather than — as a lot of reference mate- matches are found simultaneously by looking up rial for the Finnish language does — using gram- all lemmas in the dependency tree. These candi- matical descriptions, such as Latinate names for dates are filtered by trying to match each remain- case endings. The reason for this is twofold. ing token in the MWE against any neighbour, ex- Firstly, as Bleyhl (2009) notes, treatments of lan- tending the neighbourhood in the process until the guage which are heavy on grammatical analysis whole MWE is matched or it is impossible to pro- and the associated linguistic terminology can be ceed. When the MWE key is its head, its parent counter-productive in language instruction since is excluded from the neighbourhood of potential

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 39 Running text surface morphemes. Turku-neural- Tokenisation Wiktionary contains various template tags parser-pipeline which give information about word formation. This data is scraped into a database so that each Reuse Dependency tree anchor etymology section can give a derivation for its Reuse headword. Template tags are normalised into ei- anchor Extraction ther inflections, derivations or compounds consist- MWE ing of normalised segments. For compounds and Reuse most derivations, normalised segments are directly anchor available as arguments to the template tag. How- Split ever, the agent noun of template tag, for ex- Token ample, must be manually mapped to “-ja”. Finally, Analytical the form of template tag makes use of gram- Force-Align segmentation matical terms such as elative, which are mapped step to normalised segments such as “-sta”. Headword Analytical Force-Align segmentation 4.3 Building segmentation derivations step Complex words may have several levels of com- Analytical segment pounding or word derivation and inflection. Thus, we may have to make use of several lookups to Figure 2: Diagram showing processing pipeline from surface forms to analytical segments. Objects indicated fully segment a word form. We also want to make in yellow are linked within the user interface during the sure a completely segmented word form can be as- hover brushing interaction. Dotted lines indicate how spans sociated with all lexical items that make it up. We corresponding to the in-node are found in the out-node. thus shift our perspective to think of these analyses as rules and the segmenter as a rule engine which applies them to produce derivations subject to con- matching tokens. MWE tokens without a lemma straints. Each rule can match any single segment act as wildcards, and can match multiple tokens, and produce many segments. but they must be connected within the dependency The basic rule engine operates by recursively tree.6 applying rules. It keeps track of the current front 4.2 Analytical segmentation data of the derivation tree. At each iteration, each node from the front is considered and one or more steps The approach to analytical segmentation pur- consisting of applying one or more rules are taken sued here is to combine analyses from the to create child nodes, creating a new front. There Omorfi morphological analyser (Pirinen, 2015) may be multiple rules which can match a segment. and information from Wiktionary together to In this case, all combinations of rules matching produce analytical segmentations. Omorfi pro- each matchable segment are applied. When either duces analyses in its own format, which there are no more rules which match, or there is has some degree of compatibility with tags a match which does not expand any segments, the from the Universal Dependencies (UD) project node is marked as terminal. (Pyysalo et al., 2015). As an example, A simple approach would be to allow all rules [WORD ID=kakku] kakusta may be analysed as to apply at once. However, Omorfi analyses do not [UPOS=NOUN][NUM=SG][CASE=ELA] . Mor- work very well as rules as-is in our case, for ex- phological tags are mapped to analytical mor- ample for voileipakakusta¨ Omorfi produces three [CASE=ELA] phemes so that e.g. is output as different analyses of different levels of decom- WORD ID -sta, while is passed through. The or- pounding of voileipakakku.¨ If we were to apply der in which the tags appear is the same order as each of these analyses as rules we would end up the surface morphemes appear, meaning our an- with 3 final segmentations. However, for our pur- alytical morphemes are in the same order as the poses, they should all be subsumed under the same 6The MWE extraction code is made available at https: derivation. Therefore we take the following ap- //github.com/frankier/lextract proach:

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 40 1. First, apply Wiktionary based rules recursively. v o i d a m a 2. Fetch all Omorﬁ rules resulting from looking up the whole word form cost: 4 voima kas sti mpi 3. While there are Omorﬁ rules left:

(a) Remove any Omorfi rules already sub- voimakascost: 0 mmin cost: N/A sumed by the current derivation. (b) If any rules remain, apply the one pro- ducing the least new segments and dis- voimakkaammin cost: 13 card. (c) Apply Wiktionary based rules recur- Figure 3: Alignment within derivation tree of sively. voimakkaammin. Dark yellow portions denote surface spans, while each of the whole yellow portions including 4. Apply any retrofitting rules, which exist to dark and light denote the whole logical span. The cost deal with occasional cases of fusional Finnish of each alignment according to Force-Align is shown next to morphology. the parent segment. The dashed lines indicate the alignment is not produced by Force-Align but instead obtained from the For example voileipakakusta¨ (English: out underlying rule. In this case: the synthetic rule -mmin → -sti of/from sandwich cake) would produce the follow- -mpi. ing derivation: Example 1: voileipakakusta¨ → voileipakakku¨ sta Omorfi: voileipakakusta¨ so here inflected forms are treated as having the → voileipa¨ kakku sta Wiktionary: voileipakakakku¨ POS of their lemma. → voi leipa¨ kakku sta Wiktionary: voileipa¨ The permissible compound POS patterns can While voimakkaammin (English: more power- then be produced by a list of production rules, ob- fully) would produce the following: tained by studying Hyvarinen¨ (2019): Example 2: voimakkaammin Verb → Noun Verb (e.g., koe+lentaä)¨ → voimakkaasti mpi Wiktionary: voimakkaammin → → Verb Adverb Verb (e.g., edes+auttaa) voimakas sti mpi Wiktionary: voimakkaasti Noun → Noun Noun (e.g., voi+leipa)¨ → voima kas sti mpi Wiktionary: voimakas → → Noun Adjective Noun (e.g., puna+viini) voida ma kas sti mpi Wiktionary: voima Adjective → Adjective Adjective (e.g., hyvan+n¨ ak¨ oinen)¨ 7 In this case a retrofitting rule mmin → sti mpi can We start by treating the whole token as having Un- be applied: known POS, meaning we can match any POS. At Example 3: voimakas sti mpi any time a segment is constrained to having one of ← voimakas mmin Retrofit: mmin ← voimakkaammin Connect to parent a set of POS tags. For compounds of more than two parts, we can obtain the possible POS patterns 4.4 Constraints upon rules by expanding the production rules given above. Applied as-is, this scheme will produce impossi- Referring back to Example 1, a Wiktionary rule ble segmentations. However, if we consider the allows the analysis of voi (Verb) as voida (3rd POS (Part-Of-Speech) of each segment, we can pers.), however voileipakakku¨ is known to be a place constraints to ameliorate this. Noun, meaning according to the above rules, voi We use a simple set of POS tags based on Word- must be either a Noun or an Adjective, meaning Net: Verb, Noun, Adverb, Adjective & Unknown. this rule cannot be applied. The UD POS tags used by Omorfi and the Wik- tionary POS headings are mapped into this com- 4.5 Obtaining alignments from derivations mon scheme. The mapping is lossy, for example, In order to find correspondences between indi- UD adpositions are mapped onto the Adverb POS. vidual characters in analytical segments, surface All closed classes, interjections and affixes are forms and headwords, we apply the Force-Align mapped to Unknown. Note that constituent words procedure at each step of the analytical segmen- of Finnish compounds can be inflected words, and tation derivation. Each child segment is given a 7“-sti” is adverb forming morpheme like “-ly”, while “- span into the parent segment. These are ordered mpi” is a comparative forming morpheme like “-er”. and non-overlapping. Matches are performed af-

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 41 Function FORCE-ALIGN(parent string p, array of of the characters in the current child segment. The child strings c1 ...cn ) procedure ends when there is a complete solution returns alignment a Create a bounded priority queue pq with the at the front of the queue. Each child segment’s lowest cost partial solution at its front full span covers the characters from the beginning Add an empty partial solution into pq while the head of pq is not complete do of its ﬁrst character match to just before the ﬁrst Pop partial solution j from front of pq character match of the next child segment, or until /* Make a match */ the end of the parent segment in the case of the last if j’s cursor into c has not reached end and j’s cursor into p has not reached end and child segment.

p(j’s cursor into p) = c(j’s cursor into c) then Add copy of j into pq with its cursors A span of a segment onto any ancestor seg- into p and c incremented ment can be found by following a simple rule at end each step: shift the whole span rightwards by far /* Skip a parent character */ if j’s cursor into p is not at beginning or enough to fit any new segments to the left, and ex- end then tend the right edge to the end of the child span Add copy of j into pq with its cursor into p and its parent characters onto the parent segment while the current child skipped incremented segment is the rightmost. As an example, con- end sider Figure 3 and the analytical morpheme -kas. /* Skip the rest of the current child segment */ It begins on the right edge. We consider its align- if j’s cursor into p is not at beginning and ment onto voimakas and find we must shift its left j’s cursor into c has not reached end then edge by five characters to make space for voima. Add copy of j into pq with its cursor into c and its child characters We replace its right edge with that of its parent, skipped incremented which does not change the span. -kas is still on end the right edge of voimakas when we consider the end a := alignment formed by solution at front of pq alignment of voimakas and voimakkaammin. At end this step, we do not have to shift the left edge since Algorithm 1: The Force-Align procedure to find there are no new segments to the left. The right an alignment between a parent string and its seg- edge is replaced with the right edge of the align- mented children strings. ment of voimakas onto voimakkaammin, shifting it right by one character. The final span contains the characters ‘kkaa’ — which is the allomorph ter normalisation. All strings are lowercased and corresponding to ‘kas’, as required. the front vowels a,¨ o¨ and y are mapped to the respective back vowels a, o and u. We aim to minimise a cost defined as the sum of the square When the user hovers over a segment of a seg- of the number of parent characters skipped and mentation in the user interface, the corresponding square of the number of child characters skipped. surface form of the same segment should high- An example showing the type of alignments pro- light. The highlighting consists of a strong high- duced by Force-Align applied to the derivation of light for that part of the surface form which has voimakkaammin is shown in Figure 3. overlapping text with the analytic form, and a Force-Align is implemented as a dynamic pro- weak highlight for that part which is grammati- gramming style procedure, given as pseudocode cally part of the same morpheme but does not liter- in Algorithm 1. At each step, Force-Align keeps ally match. The strong highlight is found by find- track of candidate solutions in a priority queue, ing the longest match between the child segment with the lowest cost partial solution always being and its span within the parent segment, while the at the front. The priority queue has bounded length weak highlight is made from any part of the span to bound the running time — making the proce- which is left over. dure a form of beam search. Whenever a partial candidate solution is taken from the front of the Special consideration is given to wildcards, queue, up to three new partial solutions are cre- such as -sta. In this case, matching is performed ated and added back to the priority queue: making right to left, and the wildcard is weakly matched a single character match; skipping a single char- against that which remains after all other analytic acter from the parent string; and skipping the rest segments are aligned.

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 42 Figure 4: A screenshot showing the Finnish Wikipedia page Turku being read using the browser extension.

5 Visualising Finnish word formation

A screenshot of the user interface is shown in Fig- ure 4. Definitions are grouped by normalised segmentation. Within each normalised segmentation there are defined headwords, each corresponding to one or more of the normalised segments. They are ordered in decreasing order of coverage of the normalised segmentation, meaning those definitions which define the meaning of the surface form most closely appear closest to the top. Within each defined headword appears one or more clusters of definitions, each with an exemplar. Figure 5: A composite screenshot showing different To bring attention back to surface forms from stages of the interaction resulting when a user brushes the normalised forms, the interface highlights the over segments in the text analyser. surface forms as the learner hovers over the segmented forms, as shown in Figure 5. The interaction recalls a one dimensional “hover scrub” ac- nical linguistic jargon, following the principle of tion. Initially, the whole word or phrase is lightly “show, don’t tell”. highlighted. As the learner scrubs over analytic Clearly, the question of whether systems such morphemes, the corresponding spans in the sur- as NiinMikaOli?!¨ truly help language learners is a face form are highlighted. pertinent one. Quantitative user evaluation to vali- To show the connection between the normalised date existing features and point to new ones is thus segmentation and its definitions, parts of the de- an important piece of future work. fined headwords are highlighted when normalised NiinMikaOli?!¨ gives definitions in English. segments are hovered over, as shown in Fig- Adding common L1 languages of Finnish learn- ures 4 & 5. The whole interaction serves to ers such as Swedish, Russian or Arabic, as well as link the different views of surface form, analyti- Finnish itself could be a useful addition. 8 cal form and headwords. The current analytical segmenter is rule based, and thus cannot handle out of vocabulary words. 6 Conclusions and Future Work A machine learning approach such as that of Kann This paper presented NiinMikaOli?!¨ The sys- et al. (2016) could be combined with the data de- tem streamlines the experience of using reference veloped here to address this. material by presenting it in-context, emphasising A future direction for all reading assistants is the most important parts and presenting simplified better prediction of language learner needs, which grammatical analyses which do not rely upon tech- would lead to a system which knows beforehand which types of reading assistance would be best 8The NiinMikaOli?!¨ browser extension and website to offer either by explicitly requesting information are available at https://niinmikaoli.fi/, while the analytical segmentation code is available at https:// from the learner, or implicitly using information github.com/frankier/asafi. from previous interactions with the software.

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 43 References P. M. Lightbown and N. Spada. 2013. How languages are learned, 3rd ed edition. Oxford handbooks for Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa. language teachers. Oxford University Press. 2014. Random walks for knowledge-based word sense disambiguation. Computational Linguistics, Krister Linden´ and Lauri Carlson. 2010. Finnwordnet– 40(1):57–84. ﬁnnish wordnet by translation. LexicoNordica– Nordic Journal of Lexicography, 17:119–140. Werner Bleyhl. 2009. The hidden paradox of foreign language instruction. or: Which are the real foreign Detmar Meurers, Ramon Ziai, Luiz Amaral, Adriane language learning processes. Input Matters in SLA. Boyd, Aleksandar Dimitrov, Vanessa Metcalf, and Clevedon: Multilingual Matters, pages 137–55. Niels Ott. 2010a. Enhancing authentic web pages for language learners. In Proceedings of the NAACL Brendan J. Frey and Delbert Dueck. 2007. Clustering HLT 2010 Fifth Workshop on Innovative Use of NLP by passing messages between data points. Science, for Building Educational Applications, pages 10–18. 315(5814):972–976. Association for Computational Linguistics.

Robert Godwin-Jones. 2018. Contextualized vocabu- Detmar Meurers, Ramon Ziai, Luiz Amaral, Adriane lary learning. Language Learning & Technology, Boyd, Aleksandar Dimitrov, Vanessa Metcalf, and 22(3):1–19. Niels Ott. 2010b. Enhancing authentic web pages for language learners. In Proceedings of the NAACL Irma Hyvarinen.¨ 2019. Compounds and multi-word ex- HLT 2010 Fifth Workshop on Innovative Use of NLP pressions in Finnish: Compounds and Multi-Word for Building Educational Applications, pages 10–18. Expressions, pages 307–336. De Gruyter. Association for Computational Linguistics.

Jenna Kanerva, Filip Ginter, Niko Miekka, Akseli J. Nerbonne, D. Dokter, and P. Smit. 1998. Morpho- Leino, and Tapio Salakoski. 2018. Turku neural logical processing and computer-assisted language parser pipeline: An end-to-end system for the conll learning. Computer Assisted Language Learning, 2018 shared task. In Proceedings of the CoNLL 11(5):543–559. 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 133–142. As- Fabian Pedregosa, Gael¨ Varoquaux, Alexandre Gram- sociation for Computational Linguistics. fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Katharina Kann, Ryan Cotterell, and Hinrich Schutze.¨ Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: 2016. Neural morphological analysis: Encoding- Machine learning in python. Journal of machine decoding canonical segments. In Proceedings of learning research, 12(Oct):2825–2830. the 2016 Conference on Empirical Methods in Nat- Tommi A Pirinen. 2015. Development and use of com- ural Language Processing, pages 961–967, Austin, putational morphology of ﬁnnish in the open source Texas. Association for Computational Linguistics. and open science era: Notes on experiences with omorﬁ development. SKY Journal of Linguistics, Fred Karlsson. 2015. Finnish: an essential grammar. 28:381–393. Routledge. Sampo Pyysalo, Jenna Kanerva, Anna Missila,¨ Anisia Katinskaia, Javad Nouri, and Roman Yangarber. Veronika Laippala, and Filip Ginter. 2015. Univer- 2017. Revita: a system for language learning and sal dependencies for Finnish. In Proceedings of the supporting endangered languages. In Proceedings 20th Nordic Conference of Computational Linguis- of the joint workshop on NLP for Computer Assisted tics (NODALIDA 2015), pages 163–172, Vilnius, Language Learning and NLP for Language Acquisi- Lithuania. Linkoping¨ University Electronic Press, tion, pages 27–35, Gothenburg, Sweden. LiU Elec- Sweden. tronic Press. Nils Reimers and Iryna Gurevych. 2019. Sentence- Maddalen Lopez de Lacalle, Egoitz Laparra, Itziar Ald- BERT: Sentence embeddings using Siamese BERT- abe, and German Rigau. 2016. A multilingual pred- networks. In Proceedings of the 2019 Conference on icate matrix. In Proceedings of the Tenth Inter- Empirical Methods in Natural Language Process- national Conference on Language Resources and ing and the 9th International Joint Conference on Evaluation (LREC’16), pages 2662–2668, Portoroz,ˇ Natural Language Processing (EMNLP-IJCNLP), Slovenia. European Language Resources Associa- pages 3982–3992, Hong Kong, China. Association tion (ELRA). for Computational Linguistics.

Veronika Laippala and Filip Ginter. 2014. Syntactic Robert Reynolds, Eduard Schaf, and Detmar Meurers. n-gram collection from a large-scale corpus of inter- 2014. A VIEW of Russian: Visual input enhance- net ﬁnnish. In Human Language Technologies-The ment and adaptive feedback. In Proceedings of the Baltic Perspective: Proceedings of the Sixth Inter- third workshop on NLP for computer-assisted lan- national Conference Baltic HLT, volume 268, page guage learning, pages 98–112, Uppsala, Sweden. 184. LiU Electronic Press.

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 44 Frankie Robertson. 2020. Filling the -s in ﬁnnish mwe lexicons. In Joint Workshop on Multiword Expressions and Electronic Lexicons (MWE-LEX 2020), Online. Association for Computational Lin- guistics. Richard W. Schmidt. 1990. The Role of Consciousness in Second Language Learning. Applied Linguistics, 11(2):129–158. Robyn Speer, Joshua Chin, Andrew Lin, Sara Jew- ett, and Lance Nathan. 2018. Luminosoin- sight/wordfreq: v2.2. Leonardo Zilio, Rodrigo Wilkens, and Cedrick´ Fairon. 2017. Using NLP for enhancing second language acquisition. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 839–846, Varna, Bulgaria. INCOMA Ltd.

Proceedings of the 9th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2020) 45